The ground truth about metadata and community detection in networks
Reads0
Chats0
TLDR
It is proved that no algorithm can uniquely solve community detection, and a general No Free Lunch theorem for community detection is proved, which implies that there can be no algorithm that is optimal for all possible community detection tasks.Abstract:
Across many scientific domains, there is a common need to automatically extract a simplified view or coarse-graining of how a complex system's components interact. This general task is called community detection in networks and is analogous to searching for clusters in independent vector data. It is common to evaluate the performance of community detection algorithms by their ability to find so-called ground truth communities. This works well in synthetic networks with planted communities because these networks' links are formed explicitly based on those known communities. However, there are no planted communities in real-world networks. Instead, it is standard practice to treat some observed discrete-valued node attributes, or metadata, as ground truth. We show that metadata are not the same as ground truth and that treating them as such induces severe theoretical and practical problems. We prove that no algorithm can uniquely solve community detection, and we prove a general No Free Lunch theorem for community detection, which implies that there can be no algorithm that is optimal for all possible community detection tasks. However, community detection remains a powerful tool and node metadata still have value, so a careful exploration of their relationship with network structure can yield insights of genuine worth. We illustrate this point by introducing two statistical techniques that can quantify the relationship between metadata and community structure for a broad class of models. We demonstrate these techniques using both synthetic and real-world networks, and for multiple types of metadata and community structures.read more
Citations
More filters
Posted ContentDOI
Identifying flow modules in ecological networks using Infomap
TL;DR: In this article, the authors present Infomap, an established method based on network flows to the field of ecological networks, which can identify modules in virtually any type of ecological network and is particularly useful for directed, weighted and multilayer networks.
Journal ArticleDOI
Communities as Vague Operators: Epistemological Questions for a Critical Heuristics of Community Detection Algorithms
TL;DR: In this article , the authors analyse the nature and epistemic consequences of what figures in network science as patterns of nodes and edges called "communities" and propose to describe the concept of community as a "vague operator", a variant of Susan Leigh Star's notion of the boundary object.
Posted Content
Multiresolution Consensus Clustering in Networks.
TL;DR: A hierarchical consensus clustering procedure, based on a modified modularity, that allows one to construct a hierarchical consensus structure given a set of input partitions and can be applied to the output of any clustering algorithm.
Posted ContentDOI
Community level individual-based plant-pollinator networks unveil pollen flow dynamics and plant reproductive success
Alfonso Allen-Perkins,Alfonso Allen-Perkins,Maria Hurtado,David García-Callejas,David García-Callejas,Oscar Godoy,Ignasi Bartomeus +6 more
TL;DR: In this article, a mechanistic multilayer framework based on the frequency of insect visits to plant individuals belonging to different species is introduced to depict from the network structure the potential conspecific and heterospecific pollen flows among plant individuals.
Book ChapterDOI
Metrics Matter in Community Detection.
TL;DR: A critical evaluation of normalized mutual information (NMI) as an evaluation metric for community detection is presented and one-sided AMI under the $\mathbb{M}_{\mathrm{all}}$ model (all partitions of $n$ nodes) is advised.
References
More filters
Journal ArticleDOI
Community structure in social and biological networks
Michelle Girvan,Mark Newman +1 more
TL;DR: This article proposes a method for detecting communities, built around the idea of using centrality indices to find community boundaries, and tests it on computer-generated and real-world graphs whose community structure is already known and finds that the method detects this known structure with high sensitivity and reliability.
Journal ArticleDOI
Normalized cuts and image segmentation
Jianbo Shi,Jitendra Malik +1 more
TL;DR: This work treats image segmentation as a graph partitioning problem and proposes a novel global criterion, the normalized cut, for segmenting the graph, which measures both the total dissimilarity between the different groups as well as the total similarity within the groups.
Proceedings ArticleDOI
Normalized cuts and image segmentation
Jianbo Shi,Jitendra Malik +1 more
TL;DR: This work treats image segmentation as a graph partitioning problem and proposes a novel global criterion, the normalized cut, for segmenting the graph, which measures both the total dissimilarity between the different groups as well as the total similarity within the groups.
Journal ArticleDOI
Community detection in graphs
TL;DR: A thorough exposition of community structure, or clustering, is attempted, from the definition of the main elements of the problem, to the presentation of most methods developed, with a special focus on techniques designed by statistical physicists.
Journal ArticleDOI
Community detection in graphs
TL;DR: A thorough exposition of the main elements of the clustering problem can be found in this paper, with a special focus on techniques designed by statistical physicists, from the discussion of crucial issues like the significance of clustering and how methods should be tested and compared against each other, to the description of applications to real networks.