The modern science of networks has brought significant advances to our understanding of complex systems. One of the most relevant features of graphs representing real systems is community structure, or clustering, i. e. the organization of vertices in clusters, with many edges joining vertices of the same cluster and comparatively few edges joining vertices of different clusters. Such clusters, or communities, can be considered as fairly independent compartments of a graph, playing a similar role like, e. g., the tissues or the organs in the human body. Detecting communities is of great importance in sociology, biology and computer science, disciplines where systems are often represented as graphs. This problem is very hard and not yet satisfactorily solved, despite the huge effort of a large interdisciplinary community of scientists working on it over the past few years. We will attempt a thorough exposition of the topic, from the definition of the main elements of the problem, to the presentation of most methods developed, with a special focus on techniques designed by statistical physicists, from the discussion of crucial issues like the significance of clustering and how methods should be tested and compared against each other, to the description of applications to real networks.

/pdf/community-detection-in-graphs-34rogm3r6p.pdf

Community detection in graphs

Although widely used in practice, the behavior and accuracy of the popular module identification technique called modularity maximization is not well understood in practical contexts. Here, we present a broad characterization of its performance in such situations. First, we revisit and clarify the resolution limit phenomenon for modularity maximization. Second, we show that the modularity function Q exhibits extreme degeneracies: it typically admits an exponential number of distinct high-scoring solutions and typically lacks a clear global maximum. Third, we derive the limiting behavior of the maximum modularity Q(max) for one model of infinitely modular networks, showing that it depends strongly both on the size of the network and on the number of modules it contains. Finally, using three real-world metabolic networks as examples, we show that the degenerate solutions can fundamentally disagree on many, but not all, partition properties such as the composition of the largest modules and the distribution of module sizes. These results imply that the output of any modularity maximization procedure should be interpreted cautiously in scientific contexts. They also explain why many heuristics are often successful at finding high-scoring partitions in practice and why different heuristics can disagree on the modular structure of the same network. We conclude by discussing avenues for mitigating some of these behaviors, such as combining information from many degenerate solutions or using generative models.

Performance of modularity maximization in practical contexts

Networks have proved to be useful representations of complex systems. Within these networks, there are typically a number of subsystems defined by only a subset of nodes and edges. Detecting these structures often provides important information about the organization and functioning of the overall network. Here, progress towards quantifying medium- and large-scale structures within complex networks is reviewed.

/pdf/communities-modules-and-large-scale-structure-in-networks-4dmkk9s9td.pdf

Communities, modules and large-scale structure in networks

In this paper, we use a partition of the links of a network in order to uncover its community structure. This approach allows for communities to overlap at nodes so that nodes may be in more than one community. We do this by making a node partition of the line graph of the original network. In this way we show that any algorithm that produces a partition of nodes can be used to produce a partition of links. We discuss the role of the degree heterogeneity and propose a weighted version of the line graph in order to account for this.

/pdf/line-graphs-link-partitions-and-overlapping-communities-m1ntm5cex7.pdf

Line graphs, link partitions, and overlapping communities.

In many networks, it is of great interest to identify communities, unusually densely knit groups of individuals. Such communities often shed light on the function of the networks or underlying properties of the individuals. Recently, Newman suggested modularity as a natural measure of the quality of a network partitioning into communities. Since then, various algorithms have been proposed for (approximately) maximizing the modularity of the partitioning determined. In this paper, we introduce the technique of rounding mathematical programs to the problem of modularity maximization, presenting two novel algorithms. More specifically, the algorithms round solutions to linear and vector programs. Importantly, the linear programing algorithm comes with an a posteriori approximation guarantee: by comparing the solution quality to the fractional solution of the linear program, a bound on the available “room for improvement” can be obtained. The vector programming algorithm provides a similar bound for the best partition into two communities. We evaluate both algorithms using experiments on several standard test cases for network partitioning algorithms, and find that they perform comparably or better than past algorithms, while being more efficient than exhaustive techniques.

/pdf/modularity-maximizing-graph-communities-via-mathematical-3nk1f06tpt.pdf

Modularity-Maximizing Graph Communities via Mathematical Programming

In many networks, it is of great interest to identify "communities", unusually densely knit groups of individuals. Such communities often shed light on the function of the networks or underlying properties of the individuals. Recently, Newman suggested "modularity" as a natural measure of the quality of a network partitioning into communities. Since then, various algorithms have been proposed for (approximately) maximizing the modularity of the partitioning determined. 
In this paper, we introduce the technique of rounding mathematical programs to the problem of modularity maximization, presenting two novel algorithms. More specifically, the algorithms round solutions to linear and vector programs. Importantly, the linear programing algorithm comes with an a posteriori approximation guarantee: by comparing the solution quality to the fractional solution of the linear program, a bound on the available "room for improvement" can be obtained. The vector programming algorithm provides a similar bound for the best partition into two communities. We evaluate both algorithms using experiments on several standard test cases for network partitioning algorithms, and find that they perform comparably or better than past algorithms.

Modularity-Maximizing Network Communities via Mathematical Programming

Cloud computing is one of the most commonly used infrastructures for carrying out activities using virtual machines known as processing units. One of the most fundamental issues with cloud computing is task scheduling. The optimal determination of scheduling criteria in cloud computing is a non-deterministic polynomial-time (NP)-complete optimization problem, and several procedures to manage this problem have been suggested by researchers in the past. Among these methods, the Heterogeneous Earliest Finish Time (HEFT) algorithm is recognized to produce optimal outcomes in a shorter time period for scheduling tasks in a heterogeneous environment. Literature shows that HEFT gives extraordinary results in terms of quality of schedule and execution time. However, in some cases, the average computation cost and selection of the first idle slot may not produce a good solution. Therefore, here we propose modified versions of the HEFT algorithm that can obtain improved results. In the rank generation phase, we implement different methodologies for calculating ranks, while in the processor selection phase, we modify the way of selecting idle slots for scheduling the tasks. This paper suggests enhanced versions of the HEFT algorithm under user-required financial constraints to minimize the makespan of a specified workflow submission on virtual machines. Our findings also suggest that enhanced versions of the HEFT algorithm perform better than the basic HEFT method in terms of lesser schedule length of the workflow problems running on various virtual machines.

/pdf/efficient-prioritization-and-processor-selection-schemes-for-3qicrbsf.pdf

Efficient Prioritization and Processor Selection Schemes for HEFT Algorithm: A Makespan Optimizer for Task Scheduling in Cloud Environment

In an embodiment, characteristics of an email thread are analyzed to find related email threads. Email threads are combined to identify duplicate emails and to generate a superset thread, which maintains the context of combined email threads. The superset thread is displayed to a reviewer for review, wherein each unique email is displayed only once. In an embodiment, a system for presenting a plurality of email threads includes a thread analyzer, a database manager, and a superset thread generator. The thread analyzer analyzes characteristics of an email thread. The database manager indexes and identifies email threads from networked databases. The superset thread generator combines the email threads to determine duplicate emails and generates a superset thread, which maintains the context of each of the combined email thread.

System and Method for Presenting A Plurality of Email Threads for Review

In an electronic discovery search tool, non-substantive information, such as signatures in e-mail, can bias a search tool and add processing time. A method and system for identifying recurring non-substantive text in documents has been developed so that non-substantive text may be processed or ignored by the search tool, as needed.

Gaurav Agarwal

Papers

Modularity-Maximizing Graph Communities via Mathematical Programming

Modularity-Maximizing Network Communities via Mathematical Programming

Efficient Prioritization and Processor Selection Schemes for HEFT Algorithm: A Makespan Optimizer for Task Scheduling in Cloud Environment

System and Method for Presenting A Plurality of Email Threads for Review

Signature Detection in E-Mails