Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.

/pdf/sequence-to-sequence-learning-with-neural-networks-4slgp125no.pdf

Sequence to Sequence Learning with Neural Networks

Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT'14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.

We will review some of the major results in random graphs and some of the more challenging open problems. We will cover algorithmic and structural questions. We will touch on newer models, including those related to the WWW.

Random graphs

This beginning graduate textbook describes both recent achievements and classical results of computational complexity theory. Requiring essentially no background apart from mathematical maturity, the book can be used as a reference for self-study for anyone interested in complexity, including physicists, mathematicians, and other scientists, as well as a textbook for a variety of courses and seminars. More than 300 exercises are included with a selected hint set.

/pdf/computational-complexity-a-modern-approach-5gtk8urev9.pdf

Computational Complexity: A Modern Approach

A major consideration we had in writing this survey was to make it accessible to mathematicians as well as to computer scientists, since expander graphs, the protagonists of our story, come up in numerous and often surprising contexts in both fields But, perhaps, we should start with a few words about graphs in general They are, of course, one of the prime objects of study in Discrete Mathematics However, graphs are among the most ubiquitous models of both natural and human-made structures In the natural and social sciences they model relations among species, societies, companies, etc In computer science, they represent networks of communication, data organization, computational devices as well as the flow of computation, and more In mathematics, Cayley graphs are useful in Group Theory Graphs carry a natural metric and are therefore useful in Geometry, and though they are “just” one-dimensional complexes, they are useful in certain parts of Topology, eg Knot Theory In statistical physics, graphs can represent local connections between interacting parts of a system, as well as the dynamics of a physical process on such systems The study of these models calls, then, for the comprehension of the significant structural properties of the relevant graphs But are there nontrivial structural properties which are universally important? Expansion of a graph requires that it is simultaneously sparse and highly connected Expander graphs were first defined by Bassalygo and Pinsker, and their existence first proved by Pinsker in the early ’70s The property of being an expander seems significant in many of these mathematical, computational and physical contexts It is not surprising that expanders are useful in the design and analysis of communication networks What is less obvious is that expanders have surprising utility in other computational settings such as in the theory of error correcting codes and the theory of pseudorandomness In mathematics, we will encounter eg their role in the study of metric embeddings, and in particular in work around the Baum-Connes Conjecture Expansion is closely related to the convergence rates of Markov Chains, and so they play a key role in the study of Monte-Carlo algorithms in statistical mechanics and in a host of practical computational applications The list of such interesting and fruitful connections goes on and on with so many applications we will not even

/pdf/expander-graphs-and-their-applications-wbsb8gxfw2.pdf

Expander graphs and their applications

We prove that for k ≪ 4√n regular resolution requires length nΩ(k) to establish that an Erdős–Renyi graph with appropriately chosen edge density does not contain a k-clique. This lower bound is opt...

Clique Is Hard on Average for Regular Resolution

In the context of proving lower bounds on proof space in k-DNF resolution, [Ben-Sasson and Nordstrom 2009] introduced the concept of minimally unsatisfiable sets of k-DNF formulas and proved that a minimally unsatisfiable k-DNF set with m formulas can have at most O((mk)^(k+1)) variables. They also gave an example of such sets with Omega(mk^2) variables. 
In this paper we significantly improve the lower bound to Omega(m)^k, which almost matches the upper bound above. Furthermore, we show that this implies that the analysis of their technique for proving time-space separations and trade-offs for k-DNF resolution is almost tight. This means that although it is possible, or even plausible, that stronger results than in [Ben-Sasson and Nordstrom 2009] should hold, a fundamentally different approach would be needed to obtain such results.

On Minimal Unsatisfiability and Time-Space Trade-offs for k-DNF Resolution

/pdf/o-3-4-probleme-turana-s-zapreshchennymi-podgrafami-on-turan-27i7o3wxjj.pdf

О $(3,4)$-проблеме Турана с запрещенными подграфами@@@On Turán's $(3,4)$-Problem with Forbidden Subgraphs

A two server private information retrieval (PIR) scheme allows a user U to retrieve the i-th bit of an n-bit string x replicated between two servers while each server individually learns no information about i The main parameter of interest in a PIR scheme is its communication complexity, namely the number of bits exchanged by the user and the servers A large amount of effort has been invested by researchers over the last decade in search for efficient PIR schemes A number of different schemes ((B Chor O Goldreich E Kushilevitz and M Sudan, 1998), (A Beimel and Y Ishai, 2001) ,(D Woodruff and S Yekhanin, 2005)) have been proposed, however all of them ended up with the same communication complexity of O(n1/3) The best known lower bound to date is 5 log n by (S Wehner and R de Wolf, 2005)  The tremendous gap between upper and lower bounds is the focus of our paper We show an Omega(n1/3) lower bound in a restricted model that nevertheless captures all known upper bound techniques Our lower bound applies to bilinear group based PIR schemes A bilinear PIR scheme is a one round PIR scheme, where user computes the dot product of servers' responses to obtain the desired value of the i-th bit Every linear scheme can be turned into a bilinear one with an asymptotically negligible communication overhead A group based PIR scheme is a PIR scheme that involves servers representing database by a function on a certain finite group G, and allows user to retrieve the value of this function at any group element using the natural secret sharing scheme based on G Our proof relies on representation theory of finite groups

An Omega(n^{1/3}) Lower Bound for Bilinear Group Based Private Information Retrieval.

Misha Alekhnovich was born in Moscow, USSR, on October 26, 1978. In 1995 Misha began his studies at Moscow State University and graduated in 2000 with the equivalent of a B.A. in mathematics. In 2000-2001 Misha participated in the Special Year on Computational Complexity at the Institute for Advanced Study, and in 2001-2003 he attended graduate school at MIT. Upon receiving his PhD in 2003, Misha returned to IAS as a postdoctoral member and spent the next two years, 2003-2005, there. In 2005 he moved on to a faculty position at the University of California San Diego.

Alexander A. Razborov

Papers

Clique Is Hard on Average for Regular Resolution

On Minimal Unsatisfiability and Time-Space Trade-offs for k-DNF Resolution

О $(3,4)$-проблеме Турана с запрещенными подграфами@@@On Turán's $(3,4)$-Problem with Forbidden Subgraphs

An Omega(n^{1/3}) Lower Bound for Bilinear Group Based Private Information Retrieval.

Eulogy: Michael (Misha) Alekhnovich 1978-2006