Showing papers on "Time complexity published in 2020"

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting.

Abstract: Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a $ProbSparse$ self-attention mechanism, which achieves $O(L \log L)$ in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Abstract: Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L log L) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

A Multi-Stream Feature Fusion Approach for Traffic Prediction

Abstract: Accurate and timely traffic flow prediction is crucial for intelligent transportation systems (ITS). Recent advances in graph-based neural networks have achieved promising prediction results. However, some challenges remain, especially regarding graph construction and the time complexity of models. In this paper, we propose a multi-stream feature fusion approach to extract and integrate rich features from traffic data and leverage a data-driven adjacent matrix instead of the distance-based matrix to construct graphs. We calculate the Spearman rank correlation coefficient between monitor stations to obtain the initial adjacent matrix and fine-tune it while training. As to the model, we construct a multi-stream feature fusion block (MFFB) module, which includes a three-channel network and the soft-attention mechanism. The three-channel networks are graph convolutional neural network (GCN), gated recurrent unit (GRU) and fully connected neural network (FNN), which are used to extract spatial, temporal and other features, respectively. The soft-attention mechanism is utilized to integrate the obtained features. The MFFB modules are stacked, and a fully connected layer and a convolutional layer are used to make predictions. We conduct experiments on two real-world traffic prediction tasks and verify that our proposed approach outperforms the state-of-the-art methods within an acceptable time complexity.

Ultra-Scalable Spectral Clustering and Ensemble Clustering

Abstract: This paper focuses on scalability and robustness of spectral clustering for extremely large-scale datasets with limited resources. Two novel algorithms are proposed, namely, ultra-scalable spectral clustering (U-SPEC) and ultra-scalable ensemble clustering (U-SENC). In U-SPEC, a hybrid representative selection strategy and a fast approximation method for $K$ K -nearest representatives are proposed for the construction of a sparse affinity sub-matrix. By interpreting the sparse sub-matrix as a bipartite graph, the transfer cut is then utilized to efficiently partition the graph and obtain the clustering result. In U-SENC, multiple U-SPEC clusterers are further integrated into an ensemble clustering framework to enhance the robustness of U-SPEC while maintaining high efficiency. Based on the ensemble generation via multiple U-SEPC's, a new bipartite graph is constructed between objects and base clusters and then efficiently partitioned to achieve the consensus clustering result. It is noteworthy that both U-SPEC and U-SENC have nearly linear time and space complexity, and are capable of robustly and efficiently partitioning 10-million-level nonlinearly-separable datasets on a PC with 64 GB memory. Experiments on various large-scale datasets have demonstrated the scalability and robustness of our algorithms. The MATLAB code and experimental data are available at https://www.researchgate.net/publication/330760669 .

Multipole Graph Neural Operator for Parametric Partial Differential Equations

Abstract: One of the main challenges in using deep learning-based methods for simulating physical systems and solving partial differential equations (PDEs) is formulating physics-based data in the desired structure for neural networks. Graph neural networks (GNNs) have gained popularity in this area since graphs offer a natural way of modeling particle interactions and provide a clear way of discretizing the continuum models. However, the graphs constructed for approximating such tasks usually ignore long-range interactions due to unfavorable scaling of the computational complexity with respect to the number of nodes. The errors due to these approximations scale with the discretization of the system, thereby not allowing for generalization under mesh-refinement. Inspired by the classical multipole methods, we purpose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Our multi-level formulation is equivalent to recursively adding inducing points to the kernel matrix, unifying GNNs with multi-resolution matrix factorization of the kernel. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.

Practical Locally Private Heavy Hitters

Abstract: We present new practical local differentially private heavy hitters algorithms achieving optimal or near-optimal worst-case error and running time -- TreeHist and Bitstogram. In both algorithms, server running time is $\tilde O(n)$ and user running time is $\tilde O(1)$, hence improving on the prior state-of-the-art result of Bassily and Smith [STOC 2015] requiring $O(n^{5/2})$ server time and $O(n^{3/2})$ user time. With a typically large number of participants in local algorithms ($n$ in the millions), this reduction in time complexity, in particular at the user side, is crucial for making locally private heavy hitters algorithms usable in practice. We implemented Algorithm TreeHist to verify our theoretical analysis and compared its performance with the performance of Google's RAPPOR code.

Large-Scale Multi-View Subspace Clustering in Linear Time

Abstract: A plethora of multi-view subspace clustering (MVSC) methods have been proposed over the past few years. Researchers manage to boost clustering accuracy from different points of view. However, many state-of-the-art MVSC algorithms, typically have a quadratic or even cubic complexity, are inefficient and inherently difficult to apply at large scales. In the era of big data, the computational issue becomes critical. To fill this gap, we propose a large-scale MVSC (LMVSC) algorithm with linear order complexity. Inspired by the idea of anchor graph, we first learn a smaller graph for each view. Then, a novel approach is designed to integrate those graphs so that we can implement spectral clustering on a smaller graph. Interestingly, it turns out that our model also applies to single-view scenario. Extensive experiments on various large-scale benchmark data sets validate the effectiveness and efficiency of our approach with respect to state-of-the-art clustering methods.

Data Complexity in Pattern Recognition

Abstract: Theory and Methodology.- Measures of Geometrical Complexity in Classification Problems.- Object Representation, Sample Size, and Data Set Complexity.- Measures of Data and Classifier Complexity and the Training Sample Size.- Linear Separability in Descent Procedures for Linear Classifiers.- Data Complexity, Margin-Based Learning, and Popper's Philosophy of Inductive Learning.- Data Complexity and Evolutionary Learning.- Classifier Domains of Competence in Data Complexity Space.- Data Complexity Issues in Grammatical Inference.- Applications.- Simple Statistics for Complex Feature Spaces.- Polynomial Time Complexity Graph Distance Computation for Web Content Mining.- Data Complexity in Clustering Analysis of Gene Microarray Expression Profiles.- Complexity of Magnetic Resonance Spectrum Classification.- Data Complexity in Tropical Cyclone Positioning and Classification.- Human-Computer Interaction for Complex Pattern Recognition Problems.- Complex Image Recognition and Web Security.

Finding key players in complex networks through deep reinforcement learning.

Abstract: Finding an optimal set of nodes, called key players, whose activation (or removal) would maximally enhance (or degrade) certain network functionality, is a fundamental class of problems in network science1,2. Potential applications include network immunization3, epidemic control4, drug design5, and viral marketing6. Due to their general NP-hard nature, those problems typically cannot be solved by exact algorithms with polynomial time complexity7. Many approximate and heuristic strategies have been proposed to deal with specific application scenarios1,2,8-12. Yet, we still lack a unified framework to efficiently solve this class of problems. Here we introduce a deep reinforcement learning framework FINDER, which can be trained purely on small synthetic networks generated by toy models and then applied to a wide spectrum of influencer finding problems. Extensive experiments under various problem settings demonstrate that FINDER significantly outperforms existing methods in terms of solution quality. Moreover, it is several orders of magnitude faster than existing methods for large networks. The presented framework opens up a new direction of using deep learning techniques to understand the organizing principle of complex networks, which enables us to design more robust networks against both attacks and failures.

Warm-starting quantum optimization

Abstract: There is an increasing interest in quantum algorithms for problems of integer programming and combinatorial optimization. Classical solvers for such problems employ relaxations, which replace binary variables with continuous ones, for instance in the form of higher-dimensional matrix-valued problems (semidefinite programming). Under the Unique Games Conjecture, these relaxations often provide the best performance ratios available classically in polynomial time. Here, we discuss how to warm-start quantum optimization with an initial state corresponding to the solution of a relaxation of a combinatorial optimization problem and how to analyze properties of the associated quantum algorithms. In particular, this allows the quantum algorithm to inherit the performance guarantees of the classical algorithm. We illustrate this in the context of portfolio optimization, where our results indicate that warm-starting the Quantum Approximate Optimization Algorithm (QAOA) is particularly beneficial at low depth. Likewise, Recursive QAOA for MAXCUT problems shows a systematic increase in the size of the obtained cut for fully connected graphs with random weights, when Goemans-Williamson randomized rounding is utilized in a warm start. It is straightforward to apply the same ideas to other randomized-rounding schemes and optimization problems.

Stepwise Reasoning for Multi-Relation Question Answering over Knowledge Graph with Weak Supervision

Abstract: Knowledge Graph Question Answering aims to automatically answer natural language questions via well-structured relation information between entities stored in knowledge graphs. When faced with a multi-relation question, existing embedding-based approaches take the whole topic-entity-centric subgraph into account, resulting in high time complexity. Meanwhile, due to the high cost for data annotations, it is impractical to exactly show how to answer a complex question step by step, and only the final answer is labeled, as weak supervision. To address these challenges, this paper proposes a neural method based on reinforcement learning, namely Stepwise Reasoning Network, which formulates multi-relation question answering as a sequential decision problem. The proposed model performs effective path search over the knowledge graph to obtain the answer, and leverages beam search to reduce the number of candidates significantly. Meanwhile, based on the attention mechanism and neural networks, the policy network can enhance the unique impact of different parts of a given question over triple selection. Moreover, to alleviate the delayed and sparse reward problem caused by weak supervision, we propose a potential-based reward shaping strategy, which can accelerate the convergence of the training algorithm and help the model perform better. Extensive experiments conducted over three benchmark datasets well demonstrate the effectiveness of the proposed model, which outperforms the state-of-the-art approaches.

Backward Feature Correction: How Deep Learning Performs Deep Learning

Abstract: How does a 110-layer ResNet learn a high-complexity classifier using relatively few training examples and short training time? We present a theory towards explaining this in terms of hierarchical learning. We refer hierarchical learning as the learner learns to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning efficiently and automatically by applying SGD. On the conceptual side, we present, to the best of our knowledge, the FIRST theory result indicating how deep neural networks can be sample and time efficient on certain hierarchical learning tasks, when NO KNOWN non-hierarchical algorithms (such as kernel method, linear regression over feature mappings, tensor decomposition, sparse coding, and their simple combinations) are efficient. We establish a principle called "backward feature correction", where training higher layers in the network can improve the features of lower level ones. We believe this is the key to understand the deep learning process in multi-layer neural networks. On the technical side, we show for every input dimension $d > 0$, there is a concept class consisting of degree $\omega(1)$ multi-variate polynomials so that, using $\omega(1)$-layer neural networks as learners, SGD can learn any target function from this class in $\mathsf{poly}(d)$ time using $\mathsf{poly}(d)$ samples to any $\frac{1}{\mathsf{poly}(d)}$ error, through learning to represent it as a composition of $\omega(1)$ layers of quadratic functions. In contrast, we present lower bounds stating that several non-hierarchical learners, including any kernel methods, neural tangent kernels, must suffer from $d^{\omega(1)}$ sample or time complexity to learn this concept class even to $d^{-0.01}$ error.

Fast Differentiable Sorting and Ranking

Abstract: The sorting operation is one of the most commonly used building blocks in computer programming. In machine learning, it is often used for robust statistics. However, seen as a function, it is piecewise linear and as a result includes many kinks where it is non-differentiable. More problematic is the related ranking operator, often used for order statistics and ranking metrics. It is a piecewise constant function, meaning that its derivatives are null or undefined. While numerous works have proposed differentiable proxies to sorting and ranking, they do not achieve the $O(n \log n)$ time complexity one would expect from sorting and ranking operations. In this paper, we propose the first differentiable sorting and ranking operators with $O(n \log n)$ time and $O(n)$ space complexity. Our proposal in addition enjoys exact computation and differentiation. We achieve this feat by constructing differentiable operators as projections onto the permutahedron, the convex hull of permutations, and using a reduction to isotonic optimization. Empirically, we confirm that our approach is an order of magnitude faster than existing approaches and showcase two novel applications: differentiable Spearman's rank correlation coefficient and least trimmed squares.

Private Stochastic Convex Optimization: Optimal Rates in Linear Time

Abstract: We study differentially private (DP) algorithms for stochastic convex optimization: the problem of minimizing the population loss given i.i.d. samples from a distribution over convex loss functions. A recent work of Bassily et al. (2019) has established the optimal bound on the excess population loss achievable given $n$ samples. Unfortunately, their algorithm achieving this bound is relatively inefficient: it requires $O(\min\{n^{3/2}, n^{5/2}/d\})$ gradient computations, where $d$ is the dimension of the optimization problem. We describe two new techniques for deriving DP convex optimization algorithms both achieving the optimal bound on excess loss and using $O(\min\{n, n^2/d\})$ gradient computations. In particular, the algorithms match the running time of the optimal non-private algorithms. The first approach relies on the use of variable batch sizes and is analyzed using the privacy amplification by iteration technique of Feldman et al. (2018). The second approach is based on a general reduction to the problem of localizing an approximately optimal solution with differential privacy. Such localization, in turn, can be achieved using existing (non-private) uniformly stable optimization algorithms. As in the earlier work, our algorithms require a mild smoothness assumption. We also give a linear-time algorithm achieving the optimal bound on the excess loss for the strongly convex case, as well as a faster algorithm for the non-smooth case.

Power Control in Cellular Massive MIMO With Varying User Activity: A Deep Learning Solution

Abstract: This paper considers the sum spectral efficiency (SE) optimization problem in multi-cell Massive MIMO systems with a varying number of active users. This is formulated as a joint pilot and data power control problem. Since the problem is non-convex, we first derive a novel iterative algorithm that obtains a stationary point in polynomial time. To enable real-time implementation, we also develop a deep learning solution. The proposed neural network, PowerNet, only uses the large-scale fading information to predict both the pilot and data powers. The main novelty is that we exploit the problem structure to design a single neural network that can handle a dynamically varying number of active users; hence, PowerNet is simultaneously approximating many different power control functions with varying number inputs and outputs. This is not the case in prior works and thus makes PowerNet an important step towards a practically useful solution. Numerical results demonstrate that PowerNet only loses 2% in sum SE, compared to the iterative algorithm, in a nine-cell system with up to 90 active users per in each coherence interval, and the runtime was only 0.03 ms on a graphics processing unit (GPU). When good data labels are selected for the training phase, PowerNet can yield better sum SE than by solving the optimization problem with one initial point.

Interpretable and Efficient Heterogeneous Graph Convolutional Network

Abstract: Graph Convolutional Network (GCN) has achieved extraordinary success in learning effective task-specific representations of nodes in graphs. However, regarding Heterogeneous Information Network (HIN), existing HIN-oriented GCN methods still suffer from two deficiencies: (1) they cannot flexibly explore all possible meta-paths and extract the most useful ones for a target object, which hinders both effectiveness and interpretability; (2) they often need to generate intermediate meta-path based dense graphs, which leads to high computational complexity. To address the above issues, we propose an interpretable and efficient Heterogeneous Graph Convolutional Network (ie-HGCN) to learn the representations of objects in HINs. It is designed as a hierarchical aggregation architecture, i.e., object-level aggregation first, followed by type-level aggregation. The novel architecture can automatically extract useful meta-paths for each object from all possible meta-paths (within a length limit), which brings good model interpretability. It can also reduce the computational cost by avoiding intermediate HIN transformation and neighborhood attention. We provide theoretical analysis about the proposed ie-HGCN in terms of evaluating the usefulness of all possible meta-paths, its connection to the spectral graph convolution on HINs, and its quasi-linear time complexity. Extensive experiments on three real network datasets demonstrate the superiority of ie-HGCN over the state-of-the-art methods.

An aggregative learning gravitational search algorithm with self-adaptive gravitational constants

Abstract: The gravitational search algorithm (GSA) is a meta-heuristic algorithm based on the theory of Newtonian gravity. This algorithm uses the gravitational forces among individuals to move their positions in order to find a solution to optimization problems. Many studies indicate that the GSA is an effective algorithm, but in some cases, it still suffers from low search performance and premature convergence. To alleviate these issues of the GSA, an aggregative learning GSA called the ALGSA is proposed with a self-adaptive gravitational constant in which each individual possesses its own gravitational constant to improve the search performance. The proposed algorithm is compared with some existing variants of the GSA on the IEEE CEC2017 benchmark test functions to validate its search performance. Moreover, the ALGSA is also tested on neural network optimization to further verify its effectiveness. Finally, the time complexity of the ALGSA is analyzed to clarify its search performance.

Yin-Yang firefly algorithm based on dimensionally Cauchy mutation

Abstract: Firefly algorithm (FA) is a classical and efficient swarm intelligence optimization method and has a natural capability to address multimodal optimization. However, it suffers from premature convergence and low stability in the solution quality. In this paper, a Yin-Yang firefly algorithm (YYFA) based on dimensionally Cauchy mutation is proposed for performance improvement of FA. An initial position of fireflies is specified by the good nodes set (GNS) strategy to ensure the spatial representativeness of the firefly population. A designed random attraction model is then used in the proposed work to reduce the time complexity of the algorithm. Besides, a key self-learning procedure on the brightest firefly is undertaken to strike a balance between exploration and exploitation. The performance of the proposed algorithm is verified by a set of CEC 2013 benchmark functions used for the single objective real parameter algorithm competition. Experimental results are compared with those of other the state-of-the-art variants of FA. Nonparametric statistical tests on the results demonstrate that YYFA provides highly competitive performance in terms of the tested algorithms. In addition, the application in constrained engineering optimization problems shows the practicability of YYFA algorithm.

Twin-width I: tractable FO model checking

Abstract: Inspired by a width invariant defined on permutations by Guillemot and Marx [SODA '14], we introduce the notion of twin-width on graphs and on matrices. Proper minor-closed classes, bounded rank-width graphs, map graphs, $K_t$-free unit $d$-dimensional ball graphs, posets with antichains of bounded size, and proper subclasses of dimension-2 posets all have bounded twin-width. On all these classes (except map graphs without geometric embedding) we show how to compute in polynomial time a sequence of $d$-contractions, witness that the twin-width is at most $d$. We show that FO model checking, that is deciding if a given first-order formula $\phi$ evaluates to true for a given binary structure $G$ on a domain $D$, is FPT in $|\phi|$ on classes of bounded twin-width, provided the witness is given. More precisely, being given a $d$-contraction sequence for $G$, our algorithm runs in time $f(d,|\phi|) \cdot |D|$ where $f$ is a computable but non-elementary function. We also prove that bounded twin-width is preserved by FO interpretations and transductions (allowing operations such as squaring or complementing a graph). This unifies and significantly extends the knowledge on fixed-parameter tractability of FO model checking on non-monotone classes, such as the FPT algorithm on bounded-width posets by Gajarský et al. [FOCS '15].

Fast gap-affine pairwise alignment using the wavefront algorithm.

Abstract: MOTIVATION Pairwise alignment of sequences is a fundamental method in modern molecular biology, implemented within multiple bioinformatics tools and libraries. Current advances in sequencing technologies press for the development of faster pairwise alignment algorithms that can scale with increasing read lengths and production yields. RESULTS In this article, we present the wavefront alignment algorithm (WFA), an exact gap-affine algorithm that takes advantage of homologous regions between the sequences to accelerate the alignment process. As opposed to traditional dynamic programming algorithms that run in quadratic time, the WFA runs in time O(ns), proportional to the read length n and the alignment score s, using O(s2) memory. Furthermore, our algorithm exhibits simple data dependencies that can be easily vectorized, even by the automatic features of modern compilers, for different architectures, without the need to adapt the code. We evaluate the performance of our algorithm, together with other state-of-the-art implementations. As a result, we demonstrate that the WFA runs 20-300× faster than other methods aligning short Illumina-like sequences, and 10-100× faster using long noisy reads like those produced by Oxford Nanopore Technologies. AVAILABILITY AND IMPLEMENTATION The WFA algorithm is implemented within the wavefront-aligner library, and it is publicly available at https://github.com/smarco/WFA.

Strong mixed-integer programming formulations for trained neural networks

Abstract: We present an ideal mixed-integer programming (MIP) formulation for a rectified linear unit (ReLU) appearing in a trained neural network. Our formulation requires a single binary variable and no additional continuous variables beyond the input and output variables of the ReLU. We contrast it with an ideal “extended” formulation with a linear number of additional continuous variables, derived through standard techniques. An apparent drawback of our formulation is that it requires an exponential number of inequality constraints, but we provide a routine to separate the inequalities in linear time. We also prove that these exponentially-many constraints are facet-defining under mild conditions. Finally, we study network verification problems and observe that dynamically separating from the exponential inequalities (1) is much more computationally efficient and scalable than the extended formulation, (2) decreases the solve time of a state-of-the-art MIP solver by a factor of 7 on smaller instances, and (3) nearly matches the dual bounds of a state-of-the-art MIP solver on harder instances, after just a few rounds of separation and in orders of magnitude less time.

A Proof of the CSP Dichotomy Conjecture

Abstract: Many natural combinatorial problems can be expressed as constraint satisfaction problems. This class of problems is known to be NP-complete in general, but certain restrictions on the form of the constraints can ensure tractability. The standard way to parameterize interesting subclasses of the constraint satisfaction problem is via finite constraint languages. The main problem is to classify those subclasses that are solvable in polynomial time and those that are NP-complete. It was conjectured that if a constraint language has a weak near-unanimity polymorphism then the corresponding constraint satisfaction problem is tractable; otherwise, it is NP-complete. In the article, we present an algorithm that solves Constraint Satisfaction Problem in polynomial time for constraint languages having a weak near unanimity polymorphism, which proves the remaining part of the conjecture.1

Effects of Hidden Layers on the Efficiency of Neural networks

Abstract: Hidden layers play a vital role in the performance of Neural network especially in the case of complex problems where the accuracy and the time complexity are the main constraints. The process of deciding the number of hidden layers and number of neurons in each hidden layer is still confusing. In this article, we reviewed different impacts of Hidden layers on the network which provides an overview of using three numbers of hidden layers that were found to be optimal in terms of reducing the time complexity and getting the qualified accuracy. The techniques implementing less than three number of hidden layers mostly had a loss in accuracy while the architecture implementing more than three numbers of hidden layers were found not to be optimal in terms of time complexity. Usually implementing three numbers of hidden layers give the optimal performance in terms of time complexity and accuracy. We had a survey on recent work about the Neural network based on the Empirical observations, in which if the number of hidden layers is reduced it has a direct impact on the accuracy of the network as with the complex problem having less number of hidden layers it might be possible that network will not be trained properly. On the other hand when the number of hidden layers cross the optimal number of hidden layers (three layers), time complexity increases in orders of magnitude as compared to the accuracy gain.

From Learning to Meta-Learning: Reduced Training Overhead and Complexity for Communication Systems

Abstract: Machine learning methods adapt the parameters of a model, constrained to lie in a given model class, by using a fixed learning procedure based on data or active observations. Adaptation is done on a per-task basis, and retraining is needed when the system configuration changes. The resulting inefficiency in terms of data and training time requirements can be mitigated, if domain knowledge is available, by selecting a suitable model class and learning procedure, collectively known as inductive bias. However, it is generally difficult to encode prior knowledge into an inductive bias, particularly with black-box model classes such as neural networks. Meta-learning provides a way to automatize the selection of an inductive bias. Meta-learning leverages data or active observations from tasks that are expected to be related to future, and a priori unknown, tasks of interest. With a meta-trained inductive bias, training of a machine learning model can be potentially carried out with reduced training data and/or time complexity. This paper provides a high-level introduction to meta-learning with applications to communication systems.

Dynamic Flow Migration for Embedded Services in SDN/NFV-Enabled 5G Core Networks

Abstract: Software defined networking (SDN) and network function virtualization (NFV) are key enabling technologies in fifth generation (5G) communication networks for embedding service-level customized network slices in a network infrastructure, based on statistical resource demands to satisfy long-term quality of service (QoS) requirements. However, traffic loads in different slices are subject to changes over time, resulting in challenges for consistent QoS provisioning. In this paper, a dynamic flow migration problem for embedded services is studied, to meet end-to-end (E2E) delay requirements with time-varying traffic. A multi-objective mixed integer optimization problem is formulated, addressing the trade-off between load balancing and reconfiguration overhead. The problem is transformed to a tractable mixed integer quadratically constrained programming (MIQCP) problem. It is proved that there is no optimality gap between the two problems; hence, we can obtain the optimum of the original problem by solving the MIQCP problem with some post-processing. To reduce time complexity, a heuristic algorithm based on redistribution of hop delay bounds is proposed to find an efficient solution. Numerical results are presented to demonstrate the aforementioned trade-off, the benefit from flow migration in terms of E2E delay guarantee, as well as the effectiveness and efficiency of the heuristic solution.

Deep Inductive Graph Representation Learning

Abstract: This paper presents a general inductive graph representation learning framework called $\text{DeepGL}$ DeepGL for learning deep node and edge features that generalize across-networks. In particular, $\text{DeepGL}$ DeepGL begins by deriving a set of base features from the graph (e.g., graphlet features) and automatically learns a multi-layered hierarchical graph representation where each successive layer leverages the output from the previous layer to learn features of a higher-order. Contrary to previous work, $\text{DeepGL}$ DeepGL learns relational functions (each representing a feature) that naturally generalize across-networks and are therefore useful for graph-based transfer learning tasks. Moreover, $\text{DeepGL}$ DeepGL naturally supports attributed graphs, learns interpretable inductive graph representations, and is space-efficient (by learning sparse feature vectors). In addition, $\text{DeepGL}$ DeepGL is expressive, flexible with many interchangeable components, efficient with a time complexity of $\mathcal {O}(|E|)$ O ( | E | ) , and scalable for large networks via an efficient parallel implementation. Compared with recent methods, $\text{DeepGL}$ DeepGL is (1) effective for across-network transfer learning tasks and large (attributed) graphs, (2) space-efficient requiring up to 6x less memory, (3) fast with up to 106x speedup in runtime performance, and (4) accurate with an average improvement in AUC of 20 percent or more on many learning tasks and across a wide variety of networks.

Joint DNN Partition Deployment and Resource Allocation for Delay-Sensitive Deep Learning Inference in IoT

Abstract: Nowadays, the widely used Internet-of-Things (IoT) mobile devices (MDs) generate huge volumes of data, which need analyzing and extracting accurate information in real time by compute-intensive deep learning (DL) inference tasks. Due to its multilayer structure, the deep neural network (DNN) is appropriate for the mobile-edge computing (MEC) environment, and the DL tasks can be offloaded to DNN partitions deployed in MEC servers (MECSs) for speed-up inference. In this article, we first assume the arrival process of DL tasks as Poisson distribution and develop a tandem queueing model to evaluate the end-to-end (E2E) inference delay of DL tasks in multiple DNN partitions. To minimize the E2E delay, we develop a joint optimization problem model of partition deployment and resource allocation in MECSs (JPDRA). Since the JPDRA is a mixed-integer nonlinear programming (MINLP) problem, we decompose the original problem into a computing resource allocation (CRA) problem with fixed partition deployment decision and a DNN partition deployment (DPD) problem that optimizes the optimal-delay function related to the CRA problem. Next, we design a CRA algorithm based on Markov approximation and a low-complexity DPD algorithm to obtain the near-optimal solution in the polynomial time. The simulation results demonstrate that the proposed algorithms are more efficient and can reduce the average E2E delay by 25.7% with better convergence performance.

Quantum Attacks without Superposition Queries: the Offline Simon's Algorithm

Abstract: In symmetric cryptanalysis, the model of superposition queries has led to surprising results, with many constructions being broken in polynomial time thanks to Simon's period-finding algorithm. But the practical implications of these attacks remain blurry. In contrast, the results obtained so far for a quantum adversary making classical queries only are less impressive. In this paper, we introduce a new quantum algorithm which uses Simon's subroutines in a novel way. We manage to leverage the algebraic structure of cryptosystems in the context of a quantum attacker limited to classical queries and offline quantum computations. We obtain improved quantum-time/classical-data tradeoffs with respect to the current literature, while using only as much hardware requirements (quantum and classical) as a standard exhaustive search with Grover's algorithm. In particular, we are able to break the Even-Mansour construction in quantum time $\tilde{O}(2^{n/3})$, with $O(2^{n/3})$ classical queries and $O(n^2)$ qubits only. In addition, we improve some previous superposition attacks by reducing the data complexity from exponential to polynomial, with the same time complexity. Our approach can be seen in two complementary ways: \emph{reusing} superposition queries during the iteration of a search using Grover's algorithm, or alternatively, removing the memory requirement in some quantum attacks based on a collision search, thanks to their algebraic structure. We provide a list of cryptographic applications, including the Even-Mansour construction, the FX construction, some Sponge authenticated modes of encryption, and many more.