scispace - formally typeset
Search or ask a question
Book ChapterDOI

A Comparative Study of Community Detection Techniques for Large Evolving Graphs

TL;DR: The results show that no single best performing community detection technique exists, but rather, the choice of the method depends on the objective and dataset characteristics.
Abstract: Community detection has recently received increased attention due to its wide range of applications in many fields. While at first most techniques were focused on discovering communities in static networks, lately the focus has shifted toward evolving networks because of their high relevance in real-life problems. Given the increasing number of the methods being proposed, this paper explores the current availability of empirical comparative studies of dynamic methods and also provides its own qualitative and quantitative comparison with the aim of gaining more insight in the performance of available methods. The results show that no single best performing community detection technique exists, but rather, the choice of the method depends on the objective and dataset characteristics.

Summary (2 min read)

1 Introduction

  • Community detection techniques in complex networks are a well-covered topic in academic literature nowadays as identifying meaningful substructures in complex networks has numerous applications in a vast variety of fields ranging from biology, mathematics, and computer science to finance, economics and sociology.
  • Nonetheless, newly proposed community detection methods for dynamic graphs are typically compared with only very few methods in settings aiming to demonstrate superiority of the proposed method.
  • The setup and results of these comparisons might contain an unconscious bias towards one’s own algorithm.
  • This is not surprising given the many different aspects which come into play when comparing DCD methods: different underlying network models, different community definitions, different temporal segments used for detecting communities, different community evolution events tracked etc.
  • Results showcase that no single best performing community technique exists.

2.1 Algorithm selection

  • Given the soundness and completeness of Cazabet and Rossetti’s classification framework [1], the authors opt for using this framework as a steering wheel in the process of method selection.
  • In an applied setting, neglecting previous states of the network oftentimes leads to sub-optimal solutions.
  • For each subcategory, at least one representative is chosen.
  • Additionally, the list of compared algorithms is complemented by more recently published techniques, which, in turn, are also classified in the four previously mentioned categories.
  • Secondly, the algorithm would preferably be able to detect overlapping communities to ensure a realistic partitioning in social network problems.

2.3 Empirical analysis setup

  • Given their different characteristics, to provide a fair comparison, selectedDCDmethods are benchmarked based on both synthetic and real-life datasets.
  • Larger makes the sizes of communities relatively larger, more dispersed, while smaller makes the differences between community sizes smaller, more uniform.
  • The rate of node appearance and vanishing is fixed to 0.05 and 0.02 respectively.
  • To measure the relative performance of the different algorithms, two metrics were chosen.

3.1 Algorithm selection

  • It is more time-efficient than its modularity-based peers that do not rely on community updating.
  • The algorithms belonging to this category all consider a list of historic network changes in order to update the network’s partitioning.
  • FacetNet is used as a benchmark approach in many papers introducing algorithms with similar capabilities.
  • DEMON introduced in [23] is a technique that is able to hierarchically detect overlapping communities but cannot, unlike all previous methods, identify community evolution events.

3.2 Qualitative results

  • The first aspect that stands out is the larger presence of algorithms that update communities by a set of rules (2.2) not only in their final selection, but, likewise, among the more recently proposed methods, such as AFOCS, HOCTracker, OLCPM and DOCET which are also more focused on performing in dynamic social environments.
  • Finally, TILES, AFOCS, HOCTracker and DOCET appear to be the most complex algorithms as their computation time is expected to grow quadratically with the number of nodes, which is particularly problematic for large graphs.
  • Firstly, it is remarkable that the event resurgence cannot be detected by any of the selected algorithms, nor by any of the other algorithms that were analyzed, even though the event has been included in the literature, among others by [1].
  • It might be the case that continue is implied/detected when no event occurs and is therefore not mentioned by the authors.
  • Secondly, the algorithms, such as OLCPM, HOCTracker and DOCET, that were included in addition to the survey by [1] because they were more recent and possessed good features for social network community detection, can detect most of the events community evolution events.

5 Conclusion

  • Dynamic community detection has numerous applications in different fields and as such is extensively studied in the current literature.
  • The qualitative analysis included an overall set of characteristics relevant for community detection such as community definition used, the ability to track community life-cycle events, overlapping and hierarchical communities, and the time complexity.
  • For the empirical analysis, several limiting factors such as unavailable/poorly documented source code and inability to runmethods on large graphs led to a narrower set of compared methods.
  • Nevertheless, 900 synthetic, evolving graphs of various sizes and community size distributions and the most frequently used real-world DBLP dataset were used for a thorough analysis.

Did you find this useful? Give us your feedback

Figures (8)

Content maybe subject to copyright    Report

A comparative study of community detection techniques
for large evolving graphs
Lauranne Coppens, Jonathan De Venter, Sandra Mitrović
[0000−0002−5697−5865]
, and
Jochen De Weerdt
[0000−0001−6151−0504]
Research Center for Information Systems Engineering (LIRIS)
KU Leuven, Leuven, Belgium
sandra.mitrovic@kuleuven.be
Abstract. Community detection has recently received increased attention due to
its wide range of applications in many fields. While at first most techniques were
focused on discovering communities in static networks, lately the focus has shifted
toward evolving networks because of their high relevance in real-life problems.
Given the increasing number of the methods being proposed, this paper explores
the current availability of empirical comparative studies of dynamic methods and
also provides its own qualitative and quantitative comparison with the aim of gain-
ing more insight in the performance of available methods. The results show that
no single best performing community detection technique exists, but rather, the
choice of the method depends on the objective and dataset characteristics.
Keywords: Dynamic Community Detection · Large Evolving Graphs · RDyn.
1 Introduction
Community detection techniques in complex networks are a well-covered topic in aca-
demic literature nowadays as identifying meaningful substructures in complex networks
has numerous applications in a vast variety of fields ranging from biology, mathematics,
and computer science to finance, economics and sociology. A majority of the literature
covers static community detection algorithms, i.e. algorithms used to uncover communi-
ties in static networks. However, real-world networks often possess temporal properties
as nodes and edges can appear and disappear, potentially resulting in a changed commu-
nity structure. Consequently, researchers have recently taken a keen interest in commu-
nity detection algorithms that can tackle dynamic networks. Given the increasing num-
ber of the methods being proposed, a systematic comparison of both their algorithmic
and performance differences is required so as to be able to select a suitable method for a
particular community discovery problem. Nonetheless, newly proposed community de-
tection methods for dynamic graphs are typically compared with only very few methods
in settings aiming to demonstrate superiority of the proposed method. Consequently, the
setup and results of these comparisons might contain an unconscious bias towards one’s
own algorithm. As such, a well-founded and extensive comparative analysis of dynamic
community detection (DCD) techniques is missing in the current literature. This is not
surprising given the many different aspects which come into play when comparing DCD

2 L. Coppens et al.
methods: different underlying network models, different community definitions, differ-
ent temporal segments used for detecting communities, different community evolution
events tracked etc.
To bridge this literature gap, we perform a qualitative and quantitative comparison
of DCD techniques. To this end, we adopt the classification system of Cazabet and Ros-
setti [1] to provide a concise framework within which the comparison is framed. For our
qualitative comparison, we focus on relevant community detection characteristics like
community definition used, ability to detect different type of communities and commu-
nity evolution events as well as computational complexity. For quantitative analysis we
report computational time and partition quality in terms of NF1 statistics, on 900 syn-
thetic RDyn [7] and one real-world DBLP dataset [25]. Results showcase that no single
best performing community technique exists. Instead, the choice of the method should
adapt to the dataset and the final objective.
2 Methodology for Unbiased Comparison of Dynamic Community
Detection Methods
In this section we provide details of our comparative study which basically consists of
three parts: first, shortlisting candidate algorithms to be compared, second, analyzing
their algorithmic characteristics and, third, performing the empirical analysis.
2.1 Algorithm selection
Given the soundness and completeness of Cazabet and Rossetti’s classification frame-
work [1], we opt for using this framework as a steering wheel in the process of method
selection. Within this framework, three large types of dynamic algorithms for searching
communities are distinguished: 1) those that only consider the current state of the net-
work (instant-optimal); 2) those that only consider past and present clustering and past
instances of the network topology (temporal trade-off); 3) those that consider the entire
network evolution available in the data, both past and future clustering (cross-time).
In an applied setting, neglecting previous states of the network oftentimes leads to
sub-optimal solutions. Additionally, it is realistic to assume that communities will be
updated using data that become available periodically. Consequently, no future infor-
mation will be available at time 𝑡. With this in mind, we opt for focusing on temporal
trade-off algorithms. Within Cazabet and Rossetti’s framework, these are further subdi-
vided into four categories: Global Optimization (denoted originally as 2.1), Rule-Based
Updates (2.2), Multi-Objective Optimization (2.3) and Network Smoothing (2.4). For
each subcategory, at least one representative is chosen. Additionally, the list of com-
pared algorithms is complemented by more recently published techniques, which, in
turn, are also classified in the four previously mentioned categories.
Moreover, three characteristics are instrumental in the selection. Firstly, the algo-
rithm has to be able to detect communities in evolving graphs. Secondly, the algorithm
would preferably be able to detect overlapping communities to ensure a realistic par-
titioning in social network problems. Thirdly, the capability of extracting community

A comparative study of community detection techniques for large evolving graphs 3
evolution events is a desired trait with the goal of having realistic partitions that incor-
porate as much available information as possible in the partitioning process. Finally,
some algorithms will be included as benchmark algorithms in order to compare results
with previously performed comparative analyses.
2.2 Qualitative analysis
The qualitative analysis is based on the comparison of algorithmic characteristics. In
particular, comparison is performed with respect to the following six questions:
1. How does the method search for communities? In other words, which of the cate-
gories within the framework of Cazabet and Rossetti does it fit in (if any)?
2. What community definition is adopted (modularity, density, conductance...)?
3. How efficient the method is? That is, what is its computational complexity?
4. Which community evolution events can the method track (birth, death, merge, split,
growth, contraction, continuation, resurgence)?
5. Can the method find overlapping communities?
6. Can the method find hierarchical communities?
2.3 Empirical analysis setup
Given their different characteristics, to provide a fair comparison, selected DCD methods
are benchmarked based on both synthetic and real-life datasets. As synthetic datasets, 9
different RDyn graphs [7] were created by varying the number of nodes to 1000, 2000
and 4000 and the communities size distribution parameter 𝛼 to be 2.5, 3 and 3.5. Larger
𝛼 makes the sizes of communities relatively larger, more dispersed, while smaller makes
the differences between community sizes smaller, more uniform. The rate of node ap-
pearance and vanishing is fixed to 0.05 and 0.02 respectively. The appearance rate is
slightly larger than the vanishing rate in an attempt to mimic a slowly growing graph
which could resemble, for instance, a customer base where customers enter, remain for
a (long) while, and churn. For each of the 9 different RDyn graphs, 100 RDyn instances
are created, yielding 900 graphs in total. The specific number 100 was arbitrarily chosen
but is used to account for variations in the results.
As for the real-life dataset, the co-authorship graph from [25] was used. This dataset
was originally extracted from DBLP database and for purposes of this analysis, was
further limited to data from 1971 to 2002. Resulting dataset has 850 875 nodes, which
represent 303 628 unique authors, and 1 656 873 edges.
To measure the relative performance of the different algorithms, two metrics were
chosen. On the one side, the quality of the partition is measured by Normalized F1-
statistics (NF1) and on the other side, the efficiency of the algorithm is reported in terms
of the computation time.
3 Results
In this section we provide results of each of the three phases of our comparative analysis:
algorithm selection, qualitative and quantitative analysis.

4 L. Coppens et al.
3.1 Algorithm selection
For the broad selection, the initial list of 51 papers on DCD methods was used [6,
13–18, 20–23, 27–65]. It was obtained by supplementing 32 temporal trade-off algo-
rithms [6, 13–15,17,21, 22, 24, 27–50] from [1] with 19 algorithms not included in the
aforementioned survey [16,18, 20, 23,51–65] that nonetheless possess interesting char-
acteristics with regards to community and evolution extraction. Figure 1 illustrates the
relevance of adding those 19 papers as it ensures the inclusion of more recent methods.
After this list of algorithms was compiled, the three algorithm-specific character-
istics mentioned before were compared in order to select the approximately ten most
promising algorithms that will be compared qualitatively and empirically. Following
the analysis mentioned above, 13 algorithms were selected to be compared, as follows.
Partition update by global optimization (2.1) This category contains algorithms that in-
crementally update and find communities by globally optimizing a metric such as mod-
ularity, density or other utility functions. Two methods represent this category in the
analysis. Firstly, D-GT is a game-theory based algorithm proposed in [13] for dynamic
social networks. The technique considers nodes as rational agents, maximizes a utility
function and finds the optimal structure when a Nash equilibrium is reached. Secondly,
Updated BGL is a modularity based incremental algorithm designed by [14]. It is more
time-efficient than its modularity-based peers that do not rely on community updating.
Both D-GT and Updated BGL are capable of tracking community evolution events.
Partition update by set of rules (2.2) This category seems to be the most promising
in terms of efficiency and accuracy for algorithms that take into account past informa-
tion. The algorithms belonging to this category all consider a list of historic network
changes in order to update the network’s partitioning. AFOCS is an algorithm designed
for performing well in mobile networks, such as online social networks, wireless sensor
networks and Mobile Ad Hoc Networks [15]. The technique is able to uncover overlap-
ping communities in an efficient way by incrementally updating the communities based
on past information. It avoids the recalculation of communities at each time step, by
identifying community evolution events based on four network changes, namely the ap-
pearance of a new node or edge and the removal of an edge or node. The algorithm
applies different rules on how to update communities depending on which events occur.
HOCTracker is a technique designed to detect hierarchical and overlapping communi-
ties in online social networks [16]. The approach detects community evolution by com-
paring significant evolutionary changes between consecutive time steps, reducing the
number of operations to be performed by the algorithm. The algorithm identifies active
nodes, which are nodes that (dis)appear or are linked to an edge that (dis)appears, and
compares those nodes’ neighborhoods with their previous time step to reassign nodes
to new communities if necessary. TILES, is an online algorithm that identifies overlap-
ping communities by iteratively recomputing a node’s community membership in case
of a new interaction [17]. The approach is capable of singling out community evolu-
tion events such as birth, death, merge, split, growth and contraction. OLCPM is an
online, deterministic and DCD method based on clique percolation and label propaga-
tion [18]. OLCPM, unlike CPM (Clique Percolation Method) [19], works by updating

A comparative study of community detection techniques for large evolving graphs 5
communities by looking at some predefined events resulting in improved computation
times. OLCPM is able to detect overlapping communities in temporal networks. Finally,
DOCET [20] incrementally updates overlapping dynamic communities after it finds an
initial community structure. It can track community evolution events.
Informed CD by multi-objective optimization (2.3) The two previous categories updated
partitions by looking at past communities. Informed community detection algorithms,
on the other hand, calculate the communities from scratch in each time step. The al-
gorithm tries to balance partition quality and temporal partition coherence or in other
words, the current network structure and past partitions. A disadvantage of these kinds
of approaches is the computational power necessary to execute the algorithm. An ad-
vantage is its temporal independence, potentially resulting in more stable outcomes. In
informed community detection by multi-objective optimization, the partition at time 𝑡
is detected by optimizing a certain metric, e.g. modularity, density. Two algorithms will
represent this category in the evaluation. FacetNet was a pioneer in detecting communi-
ties in an unified process, in contrast with a two-step approach, where evolution events
can be uncovered together with the partitioning [6]. Consequently, FacetNet is used as
a benchmark approach in many papers introducing algorithms with similar capabilities.
The approach finds communities based on non-negative matrix factorization and itera-
tively updates the network structure to balance the current partitioning fit and historical
cost function. A disadvantage of the technique is that the number of communities is fixed
and should be determined by the user. DYNMOGA [21], unlike FacetNet, balances the
Fig. 1. Analyzed papers by year.
current partitioning fit and cost function simultaneously and, therefore, does not need
a preference parameter with regard to maximizing partition quality and minimizing the
historical cost or clustering drift. It optimizes a multi-objective problem and automat-
ically determines the optimal trade-off between cluster accuracy and clustering drift.
Neither FacetNet or DYNMOGA are capable of detecting overlapping communities.
Informed CD by network smoothing (2.4) ECSD proposed by [22] is a particle-and-
density based evolutionary clustering method that is capable of determining the num-
ber of communities itself. The method detects the network’s structure and evolutionary

Citations
More filters
Journal ArticleDOI
TL;DR: In this paper, the authors compare six algorithms for dynamic community detection in terms of instantaneous and longitudinal similarity with the planted ground truth, smoothness of dynamic partitions, and scalability.
Abstract: Many algorithms have been proposed in the last ten years for the discovery of dynamic communities. However, these methods are seldom compared between themselves. In this article, we propose a generator of dynamic graphs with planted evolving community structure, as a benchmark to compare and evaluate such algorithms. Unlike previously proposed benchmarks, it is able to specify any desired evolving community structure through a descriptive language, and then to generate the corresponding progressively evolving network. We empirically evaluate six existing algorithms for dynamic community detection in terms of instantaneous and longitudinal similarity with the planted ground truth, smoothness of dynamic partitions, and scalability. We notably observe different types of weaknesses depending on their approach to ensure smoothness, namely Glitches, Oversimplification and Identity loss. Although no method arises as a clear winner, we observe clear differences between methods, and we identified the fastest, those yielding the most smoothed or the most accurate solutions at each step.

10 citations

Journal ArticleDOI
TL;DR: Wang et al. as discussed by the authors proposed a dynamic community detection algorithm called, Dynamic Community Detection based on the Matthew effect (DCDME), which employs a batch processing method to reveal communities incrementally in each network snapshot.

7 citations

Posted Content
TL;DR: This article empirically evaluate six existing algorithms for dynamic community detection in terms of instantaneous and longitudinal similarity with the planted ground truth, smoothness of dynamic partitions and scalability, and identifies the fastest, those yielding the most smoothed or the most accurate solutions at each step.
Abstract: Many algorithms have been proposed in the last ten years for the discovery of dynamic communities. However, these methods are seldom compared between themselves. In this article, we propose a generator of dynamic graphs with planted evolving community structure, as a benchmark to compare and evaluate such algorithms. Unlike previously proposed benchmarks, it is able to specify any desired evolving community structure through a descriptive language, and then to generate the corresponding progressively evolving network. We empirically evaluate six existing algorithms for dynamic community detection in terms of instantaneous and longitudinal similarity with the planted ground truth, smoothness of dynamic partitions, and scalability. We notably observe different types of weaknesses depending on their approach to ensure smoothness, namely Glitches, Oversimplification and Identity loss. Although no method arises as a clear winner, we observe clear differences between methods, and we identified the fastest, those yielding the most smoothed or the most accurate solutions at each step.

7 citations


Cites methods from "A Comparative Study of Community De..."

  • ...To the best of our knowledge, a single paper has been published so far comparing empirically dynamic community detection algorithms: in [7], 5 methods have been tested on RDyn benchmark [28]....

    [...]

References
More filters
Journal ArticleDOI
TL;DR: It is demonstrated that the algorithms proposed are highly effective at discovering community structure in both computer-generated and real-world network data, and can be used to shed light on the sometimes dauntingly complex structure of networked systems.
Abstract: We propose and study a set of algorithms for discovering community structure in networks-natural divisions of network nodes into densely connected subgroups. Our algorithms all share two definitive features: first, they involve iterative removal of edges from the network to split it into communities, the edges removed being identified using any one of a number of possible "betweenness" measures, and second, these measures are, crucially, recalculated after each removal. We also propose a measure for the strength of the community structure found by our algorithms, which gives us an objective metric for choosing the number of communities into which a network should be divided. We demonstrate that our algorithms are highly effective at discovering community structure in both computer-generated and real-world network data, and show how they can be used to shed light on the sometimes dauntingly complex structure of networked systems.

12,882 citations


"A Comparative Study of Community De..." refers background in this paper

  • ...Although themost prominently used benchmark graphs GirvanNewman (GN) [4] and Lancichinetti-Fortunato-Radicchi (LFR) [2] are not suited for temporal community discovery, to this end, their extensions in [6] and [5] respectively, were proposed....

    [...]

Journal ArticleDOI
09 Jun 2005-Nature
TL;DR: After defining a set of new characteristic quantities for the statistics of communities, this work applies an efficient technique for exploring overlapping communities on a large scale and finds that overlaps are significant, and the distributions introduced reveal universal features of networks.
Abstract: A network is a network — be it between words (those associated with ‘bright’ in this case) or protein structures. Many complex systems in nature and society can be described in terms of networks capturing the intricate web of connections among the units they are made of1,2,3,4. A key question is how to interpret the global organization of such networks as the coexistence of their structural subunits (communities) associated with more highly interconnected parts. Identifying these a priori unknown building blocks (such as functionally related proteins5,6, industrial sectors7 and groups of people8,9) is crucial to the understanding of the structural and functional properties of networks. The existing deterministic methods used for large networks find separated communities, whereas most of the actual networks are made of highly overlapping cohesive groups of nodes. Here we introduce an approach to analysing the main statistical features of the interwoven sets of overlapping communities that makes a step towards uncovering the modular structure of complex systems. After defining a set of new characteristic quantities for the statistics of communities, we apply an efficient technique for exploring overlapping communities on a large scale. We find that overlaps are significant, and the distributions we introduce reveal universal features of networks. Our studies of collaboration, word-association and protein interaction graphs show that the web of communities has non-trivial correlations and specific scaling properties.

5,217 citations

Journal ArticleDOI
TL;DR: This work introduces a class of benchmark graphs, that account for the heterogeneity in the distributions of node degrees and of community sizes, and uses this benchmark to test two popular methods of community detection, modularity optimization, and Potts model clustering.
Abstract: Community structure is one of the most important features of real networks and reveals the internal organization of the nodes. Many algorithms have been proposed but the crucial issue of testing, i.e., the question of how good an algorithm is, with respect to others, is still open. Standard tests include the analysis of simple artificial graphs with a built-in community structure, that the algorithm has to recover. However, the special graphs adopted in actual tests have a structure that does not reflect the real properties of nodes and communities found in real networks. Here we introduce a class of benchmark graphs, that account for the heterogeneity in the distributions of node degrees and of community sizes. We use this benchmark to test two popular methods of community detection, modularity optimization, and Potts model clustering. The results show that the benchmark poses a much more severe test to algorithms than standard benchmarks, revealing limits that may not be apparent at a first analysis.

2,772 citations


"A Comparative Study of Community De..." refers background in this paper

  • ...Although themost prominently used benchmark graphs GirvanNewman (GN) [4] and Lancichinetti-Fortunato-Radicchi (LFR) [2] are not suited for temporal community discovery, to this end, their extensions in [6] and [5] respectively, were proposed....

    [...]

Journal ArticleDOI
TL;DR: It is found that the most accurate methods tend to be more computationally expensive, and that both aspects need to be considered when choosing a method for practical purposes.
Abstract: We compare recent approaches to community structure identification in terms of sensitivity and computational cost. The recently proposed modularity measure is revisited and the performance of the methods as applied to ad hoc networks with known community structure, is compared. We find that the most accurate methods tend to be more computationally expensive, and that both aspects need to be considered when choosing a method for practical purposes. The work is intended as an introduction as well as a proposal for a standard benchmark test of community detection methods.

2,630 citations

Journal ArticleDOI
TL;DR: Three recent algorithms introduced by Rosvall and Bergstrom and Ronhovde and Nussinov have an excellent performance, with the additional advantage of low computational complexity, which enables one to analyze large systems.
Abstract: Uncovering the community structure exhibited by real networks is a crucial step toward an understanding of complex systems that goes beyond the local organization of their constituents. Many algorithms have been proposed so far, but none of them has been subjected to strict tests to evaluate their performance. Most of the sporadic tests performed so far involved small networks with known community structure and/or artificial graphs with a simplified structure, which is very uncommon in real systems. Here we test several methods against a recently introduced class of benchmark graphs, with heterogeneous distributions of degree and community size. The methods are also tested against the benchmark by Girvan and Newman [Proc. Natl. Acad. Sci. U.S.A. 99, 7821 (2002)] and on random graphs. As a result of our analysis, three recent algorithms introduced by Rosvall and Bergstrom [Proc. Natl. Acad. Sci. U.S.A. 104, 7327 (2007); Proc. Natl. Acad. Sci. U.S.A. 105, 1118 (2008)], Blondel et al. [J. Stat. Mech.: Theory Exp. (2008), P10008], and Ronhovde and Nussinov [Phys. Rev. E 80, 016109 (2009)] have an excellent performance, with the additional advantage of low computational complexity, which enables one to analyze large systems.

2,113 citations

Frequently Asked Questions (2)
Q1. What are the contributions mentioned in the paper "A comparative study of community detection techniques for large evolving graphs" ?

Given the increasing number of the methods being proposed, this paper explores the current availability of empirical comparative studies of dynamic methods and also provides its own qualitative and quantitative comparison with the aim of gaining more insight in the performance of available methods. 

For future work, the authors envision an even more extensive empirical evaluation.