Communication complexity

About: Communication complexity is a research topic. Over the lifetime, 3870 publications have been published within this topic receiving 105832 citations.

Integrated Model, Batch, and Domain Parallelism in Training Neural Networks

Abstract: We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-memory computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using P processes. Our method is inspired by the communication-avoiding algorithms in numerical linear algebra. We see P processes as logically divided into a P_r x P_c grid where the P_r dimension is implicitly responsible for model/domain parallelism and the P_c dimension is implicitly responsible for batch parallelism. In practice, the integrated matrix-based parallel algorithm encapsulates these types of parallelism automatically. We analyze the communication complexity and analytically demonstrate that the lowest communication costs are often achieved neither with pure model nor with pure data parallelism. We also show how the domain parallel approach can help in extending the theoretical scaling limit of the typical batch parallel method.

A flow control mechanism to avoid message deadlock in k-ary n-cube networks

Abstract: We propose a flow control algorithm for k-ary n-cube networks which avoids the deadlock problems without using virtual channels. Some basic definitions and theorems are proposed in order to establish the necessary and sufficient conditions to verify that an algorithm is deadlock-free. Our proposal is based on a restriction of the virtual cut-through flow control rather than of the routing algorithm and it can be applied both over central buffers or edge buffers. A minimum free buffer space of two packets is required. The implementation complexity of the router according to Chien's (1993) model, is much easier and faster than using virtual channels. Network simulations considering the router complexity show the performance achieved by this new algorithm. The results display a latency improvement of 20% to 35% compared with the use of virtual channels depending on the load of the network.

Learning complexity vs communication complexity

Abstract: This paper has two main focal points. We first consider an important class of machine learning algorithms: large margin classifiers, such as Support Vector Machines. The notion of margin complexity quantifies the extent to which a given class of functions can be learned by large margin classifiers. We prove that up to a small multiplicative constant, margin complexity is equal to the inverse of discrepancy. This establishes a strong tie between seemingly very different notions from two distinct areas. In the same way that matrix rigidity is related to rank, we introduce the notion of rigidity of margin complexity. We prove that sign matrices with small margin complexity rigidity are very rare. This leads to the question of proving lower bounds on the rigidity of margin complexity. Quite surprisingly, this question turns out to be closely related to basic open problems in communication complexity, e.g., whether PSPACE can be separated from the polynomial hierarchy in communication complexity. Communication is a key ingredient in many types of learning. This explains the relations between the field of learning theory and that of communication complexity [6, l0, 16, 26]. The results of this paper constitute another link in this rich web of relations. These new results have already been applied toward the solution of several open problems in communication complexity [18, 20, 29].

Carrier Frequency Offset Estimation Using ESPRIT for Interleaved OFDMA Uplink Systems

Abstract: In this paper, a new carrier frequency offset (CFO) estimator based on the system of estimation of signal parameters via rotational invariance technique is proposed for interleaved orthogonal frequency division multiple access uplink systems. This new estimator performs better in relatively low signal-to-noise region than a CFO estimator recently proposed by Cao and Yao, and has a much lower computational complexity as well. Simulation results show several performance examples for the proposed CFO estimator which demonstrate the advantages of the proposed estimator over the Cao and Yao's version.

Lower Bounds on Near Neighbor Search via Metric Expansion

Abstract: In this paper we show how the complexity of performing nearest neighbor (NNS) search on a metric space is related to the expansion of the metric space. Given a metric space we look at the graph obtained by connecting every pair of points within a certain distance $r$ . We then look at various notions of expansion in this graph relating them to the cell probe complexity of NNS for randomized and deterministic, exact and approximate algorithms. For example if the graph has node expansion $\Phi$ then we show that any deterministic $t$-probe data structure for $n$ points must use space $S$ where $(St/n)^t > \Phi$. We show similar results for randomized algorithms as well. These relationships can be used to derive most of the known lower bounds in the well known metric spaces such as $l_1$, $l_2$, $l_\infty$ by simply computing their expansion. In the process, we strengthen and generalize our previous results (FOCS 2008). Additionally, we unify the approach in that work and the communication complexity based approach. Our work reduces the problem of proving cell probe lower bounds of near neighbor search to computing the appropriate expansion parameter. In our results, as in all previous results, the dependence on $t$ is weak; that is, the bound drops exponentially in $t$. We show a much stronger (tight) time-space tradeoff for the class of dynamic low contention data structures. These are data structures that supports updates in the data set and that do not look up any single cell too often.