5分で分かる!? 有名論文ナナメ読み：Jacot, Arthor, Gabriel, Franck and Hongler, Clement : Neural Tangent Kernel : Convergence and Generalization in Neural Networks

Communication Complexity: Basics

Data structures

We show how to efficiently simulate the sending of a message M to a receiver who has partial information about the message, so that the expected number of bits communicated in the simulation is close to the amount of additional information that the message reveals to the receiver. This is a generalization and strengthening of the Slepian-Wolf theorem, which shows how to carry out such a simulation with low amortized communication in the case that M is a deterministic function of X. A caveat is that our simulation is interactive. 
As a consequence, we prove that the internal information cost (namely the information revealed to the parties) involved in computing any relation or function using a two party interactive protocol is exactly equal to the amortized communication complexity of computing independent copies of the same relation or function. We also show that the only way to prove a strong direct sum theorem for randomized communication complexity is by solving a particular variant of the pointer jumping problem that we define. Our work implies that a strong direct sum theorem for communication complexity holds if and only if efficient compression of communication protocols is possible.

Information Equals Amortized Communication

Confidentiality of training data induced by releasing machine-learning models, and has recently received increasing attention. Motivated by existing MI attacks and other previous attacks that turn out to be MI "in disguise," this paper initiates a formal study of MI attacks by presenting a game-based methodology. Our methodology uncovers a number of subtle issues, and devising a rigorous game-based definition, analogous to those in cryptography, is an interesting avenue for future work. We describe methodologies for two types of attacks. The first is for black-box attacks, which consider an adversary who infers sensitive values with only oracle access to a model. The second methodology targets the white-box scenario where an adversary has some additional knowledge about the structure of a model. For the restricted class of Boolean models and black-box attacks, we characterize model invertibility using the concept of influence from Boolean analysis in the noiseless case, and connect model invertibility with stable influence in the noisy case. Interestingly, we also discovered an intriguing phenomenon, which we call "invertibility interference," where a highly invertible model quickly becomes highly non-invertible by adding little noise. For the white-box case, we consider a common phenomenon in machine-learning models where the model is a sequential composition of several sub-models. We show, quantitatively, that even very restricted communication between layers could leak a significant amount of information. Perhaps more importantly, our study also unveils unexpected computational power of these restricted communication channels, which, to the best of our knowledge, were not previously known.

A Methodology for Formalizing Model-Inversion Attacks

We develop a new local characterization of the zero-error information complexity function for two-party communication problems, and use it to compute the exact internal and external information complexity of the 2-bit AND function: IC(AND,0) = C∧≅ 1.4923 bits, and ICext(AND,0) = log2 3 ≅ 1.5839 bits. This leads to a tight (upper and lower bound) characterization of the communication complexity of the set intersection problem on subsets of {1,...,n} (the player are required to compute the intersection of their sets), whose randomized communication complexity tends to C∧⋅ n pm o(n) as the error tends to zero.The information-optimal protocol we present has an infinite number of rounds. We show this is necessary by proving that the rate of convergence of the r-round information cost of AND to IC(AND,0)=C∧ behaves like Θ(1/r2), i.e. that the r-round information complexity of AND is C∧+Θ(1/r2).We leverage the tight analysis obtained for the information complexity of AND to calculate and prove the exact communication complexity of the set disjointness function Disjn(X,Y) = - vi=1n AND(xi,yi) with error tending to 0, which turns out to be = CDISJ⋅ n pm o(n), where CDISJ≅ 0.4827. Our rate of convergence results imply that an asymptotically optimal protocol for set disjointness will have to use ω(1) rounds of communication, since every r-round protocol will be sub-optimal by at least Ω(n/r2) bits of communication.We also obtain the tight bound of 2/ln2 k pm o(k) on the communication complexity of disjointness of sets of size ≤ k. An asymptotic bound of Θ(k) was previously shown by Hastad and Wigderson.

From information to exact communication

Motivated by recent Linear Programming solvers, we design dynamic data structures for maintaining the inverse of an $n\times n$ real matrix under $\textit{low-rank}$ updates, with polynomially faster amortized running time. Our data structure is based on a recursive application of the Woodbury-Morrison identity for implementing $\textit{cascading}$ low-rank updates, combined with recent sketching technology. Our techniques and amortized analysis of multi-level partial updates, may be of broader interest to dynamic matrix problems. 
This data structure leads to the fastest known LP solver for general (dense) linear programs, improving the running time of the recent algorithms of (Cohen et al.'19, Lee et al.'19, Brand'20) from $O^*(n^{2+ \max\{\frac{1}{6}, \omega-2, \frac{1-\alpha}{2}\}})$ to $O^*(n^{2+\max\{\frac{1}{18}, \omega-2, \frac{1-\alpha}{2}\}})$, where $\omega$ and $\alpha$ are the fast matrix multiplication exponent and its dual. Hence, under the common belief that $\omega \approx 2$ and $\alpha \approx 1$, our LP solver runs in $O^*(n^{2.055})$ time instead of $O^*(n^{2.16})$.

Faster Dynamic Matrix Inverse for Faster LPs.

We give exponentially small upper bounds on the success probability for computing the direct product of any function over any distribution using a communication protocol. Let suc(μ, f, C) denote the maximum success probability of a 2-party communication protocol for computing the boolean function f(x, y) with C bits of communication, when the inputs (x, y) are drawn from the distribution μ. Let μn be the product distribution on n inputs and fn denote the function that computes n copies of f on these inputs. We prove that if T log3/2 T ≪ (C - 1)√n and suc(μ, f, C) <; 2/3, then suc(μn, fn, T) ≤ exp(-Ω(n)). When μ is a product distribution, we prove a nearly optimal result: as long as T log2 T ≪ Cn, we must have suc(μn, fn, T) ≤ exp(-Ω(n)).

Direct Products in Communication Complexity.

The celebrated PPAD hardness result for finding an exact Nash equilibrium in a two-player game initiated a quest for finding approximate Nash equilibria efficiently, and is one of the major open questions in algorithmic game theory.We study the computational complexity of finding an e-approximate Nash equilibrium with good social welfare. Hazan and Krauthgamer and subsequent improvements showed that finding an e-approximate Nash equilibrium with good social welfare in a two player game and many variants of this problem is at least as hard as finding a planted clique of size O(log n) in the random graph G(n, 1/2).We show that any polynomial time algorithm that finds an e-approximate Nash equilibrium with good social welfare refutes (the worst-case) Exponential Time Hypothesis by Impagliazzo and Paturi, confirming the recent conjecture by Aaronson, Impagliazzo and Moshkovitz. Specifically it would imply a 2O(n1/2) algorithm for SAT.Our lower bound matches the quasi-polynomial time algorithm by Lipton, Markakis and Mehta for solving the problem.Our key tool is a reduction from the PCP machinery to finding Nash equilibrium via free games, the framework introduced in the recent work by Aaronson, Impagliazzo and Moshkovitz. Techniques developed in the process may be useful for replacing planted clique hardness with ETH-hardness in other applications.

Approximating the best nash equilibrium in no(log n)-time breaks the exponential time hypothesis

/pdf/direct-products-in-communication-complexity-4ga76datbv.pdf

Omri Weinstein

Papers

From information to exact communication

Faster Dynamic Matrix Inverse for Faster LPs.

Direct Products in Communication Complexity.

Approximating the best nash equilibrium in no(log n)-time breaks the exponential time hypothesis

Direct Products in Communication Complexity