We explore in this chapter questions of existence and uniqueness for solutions to stochastic differential equations and offer a study of their properties. This endeavor is really a study of diffusion processes. Loosely speaking, the term diffusion is attributed to a Markov process which has continuous sample paths and can be characterized in terms of its infinitesimal generator.

Stochastic Differential Equations

As companies strive to develop artefacts intended for services instead of traditional sell-off, new challenges in the product development process arise to promote continuous improvement and increasing market profits. This creates a focus on product life-cycle components as companies then make life-cycle commitments, where they are responsible for the function availability during the extent of the life-cycle, i.e. functional products. One of these life-cycle components is manufacturing; therefore, companies search for new approaches of success during manufacturability evaluation already in engineering design. Efforts have been done to support early engineering design, as this phase sets constraints and opportunities for manufacturing. These efforts have turned into design for manufacturing methods and guidelines. A further step to improve the life-cycle focus during early engineering design is to reuse results and use experience from earlier projects. However, because results and experiences created during project work are often not documented for reuse, only remembered by some people, there is a need for design support. Knowledge engineering (KE) is a methodology for creating knowledge-based systems, e.g. systems that enable reuse of earlier results and make available both explicit and tacit corporate knowledge, enabling the automated generation and evaluation of new engineering design solutions during early product development. There are a variety of KE-approaches, such as knowledge-based engineering, case-based reasoning and programming, which have been used in research to develop design for manufacturing methods and applications. There are, however, opportunities for research where several approaches and their interdependencies, to create a transparent picture of how KE can be used to support engineering design, are investigated. The aim of the research presented in this thesis is to create new methods for design for manufacturing, by using several approaches of KE, and find the beneficial and less beneficial aspects of these methods in comparison to each other and earlier research. This thesis presents methods and applications for design for manufacturing using KE. KE has been employed in several ways, namely rule-based, rule-, programmingand finite element analysis (FEA)-based, and ruleand plan-based, which are tested and compared with each other. Results show that KE can be used to generate information about manufacturing in several ways. The rule-based way is suitable for supporting life-cycle commitments, as engineering design and manufacturing can be integrated with maintenance and performance predictions during early engineering design, though limited to the firing of production rules. The rule-, programmingand FEA-based way can be used to integrate computer-aided design tools and virtual manufacturing for non-linear stress and displacement analysis. This way may also bridge the gap between engineering designers and computational experts, even though this way requires a larger effort to program than the rule-based. The ruleand planbased way can enable design for manufacturing in two fashions – based on earlier manufacturing plans and based on rules. Because earlier manufacturing plans, together with programming algorithms, can handle knowledge that may be more intricate to capture as rules, as opposed to the time demanding routine work that is often automated by means of rules, several opportunities for designing for manufacturing exist.

Methods and Applications

We construct a stream-cipher S whose implementation is secure even if a bounded amount of arbitrary (adversarially chosen) information on the internal state ofS is leaked during computation. This captures all possible side-channel attacks on S where the amount of information leaked in a given period is bounded, but overall can be arbitrary large. The only other assumption we make on the implementation of S is that only data that is accessed during computation leaks information. The stream-cipher S generates its output in chunks K1, K2, . . . and arbitrary but bounded information leakage is modeled by allowing the adversary to adaptively chose a function fl : {0,1}* rarr {0, 1}lambda before Kl is computed, she then gets fl(taul) where taul is the internal state ofS that is accessed during the computation of Kg. One notion of security we prove for S is that Kg is indistinguishable from random when given K1,..., K1-1,f1(tau1 ),..., fl-1(taul-1) and also the complete internal state of S after Kg has been computed (i.e. S is forward-secure). The construction is based on alternating extraction (used in the intrusion-resilient secret-sharing scheme from FOCS'07). We move this concept to the computational setting by proving a lemma that states that the output of any PRG has high HILLpseudoentropy (i.e. is indistinguishable from some distribution with high min-entropy) even if arbitrary information about the seed is leaked. The amount of leakage lambda that we can tolerate in each step depends on the strength of the underlying PRG, it is at least logarithmic, but can be as large as a constant fraction of the internal state of S if the PRG is exponentially hard.

/pdf/leakage-resilient-cryptography-15i0byrphr.pdf

Leakage-Resilient Cryptography

Network management requires accurate estimates of metrics for traffic engineering (e.g., heavy hitters), anomaly detection (e.g., entropy of source addresses), and security (e.g., DDoS detection). Obtaining accurate estimates given router CPU and memory constraints is a challenging problem. Existing approaches fall in one of two undesirable extremes: (1) low fidelity general-purpose approaches such as sampling, or (2) high fidelity but complex algorithms customized to specific application-level metrics. Ideally, a solution should be both general (i.e., supports many applications) and provide accuracy comparable to custom algorithms. This paper presents UnivMon, a framework for flow monitoring which leverages recent theoretical advances and demonstrates that it is possible to achieve both generality and high accuracy. UnivMon uses an application-agnostic data plane monitoring primitive; different (and possibly unforeseen) estimation algorithms run in the control plane, and use the statistics from the data plane to compute application-level metrics. We present a proof-of-concept implementation of UnivMon using P4 and develop simple coordination techniques to provide a ``one-big-switch'' abstraction for network-wide monitoring. We evaluate the effectiveness of UnivMon using a range of trace-driven evaluations and show that it offers comparable (and sometimes better) accuracy relative to custom sketching solutions.

https://dl.acm.org/doi/pdf/10.1145/2934872.2934906

One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon

Over the last decade, there has been considerable interest in designing algorithms for processing massive graphs in the data stream model. The original motivation was two-fold: a) in many applications, the dynamic graphs that arise are too large to be stored in the main memory of a single machine and b) considering graph problems yields new insights into the complexity of stream computation. However, the techniques developed in this area are now finding applications in other areas including data structures for dynamic graphs, approximation algorithms, and distributed and parallel computation. We survey the state-of-the-art results; identify general techniques; and highlight some simple algorithms that illustrate basic ideas.

/pdf/graph-stream-algorithms-a-survey-1m1oq4kh5y.pdf

Graph stream algorithms: a survey

Model compression provides a means to efficiently deploy deep neural networks (DNNs) on devices that limited computation resources and tight power budgets, such as mobile and IoT (Internet of Things) devices. Consequently, model compression is one of the most critical topics in modern deep learning. Typically, the state-of-the-art model compression methods suffer from a big limitation: they are only based on heuristics rather than theoretical foundation and thus offer no worst-case guarantees. To bridge this gap, Baykal et. al. [2018a] suggested using a coreset, a small weighted subset of the data that provably approximates the original data set, to sparsify the parameters of a trained fully-connected neural network by sampling a number of neural network parameters based on the importance of the data. However, the sampling procedure is data-dependent and can only be only be performed after an expensive training phase. 
We propose the use of data-independent coresets to perform provable model compression without the need for training. We first prove that there exists a coreset whose size is independent of the input size of the data for any neuron whose activation function is from a family of functions that includes variants of ReLU, sigmoid and others. We then provide a compression-based algorithm that constructs these coresets and explicitly applies neuron pruning for the underlying model. We demonstrate the effectiveness of our methods with experimental evaluations for both synthetic and real-world benchmark network compression. In particular, our framework provides up to 90% compression on the LeNet-300-100 architecture on MNIST and actually improves the accuracy.

On Activation Function Coresets for Network Pruning

Spectral functions of large matrices contains important structural information about the underlying data, and is thus becoming increasingly important. Many times, large matrices representing real-world data are \emph{sparse} or \emph{doubly sparse} (i.e., sparse in both rows and columns), and are accessed as a \emph{stream} of updates, typically organized in \emph{row-order}. In this setting, where space (memory) is the limiting resource, all known algorithms require space that is polynomial in the dimension of the matrix, even for sparse matrices. We address this challenge by providing the first algorithms whose space requirement is \emph{independent of the matrix dimension}, assuming the matrix is doubly-sparse and presented in row-order. Our algorithms approximate the Schatten $p$-norms, which we use in turn to approximate other spectral functions, such as logarithm of the determinant, trace of matrix inverse, and Estrada index. We validate these theoretical performance bounds by numerical experiments on real-world matrices representing social networks. We further prove that multiple passes are unavoidable in this setting, and show extensions of our primary technique, including a trade-off between space requirements and number of passes.

/pdf/schatten-norms-in-matrix-streams-hello-sparsity-goodbye-4xmvwmtf2m.pdf

Schatten Norms in Matrix Streams: Hello Sparsity, Goodbye Dimension

Single-cell RNA-sequencing (scRNA-seq) analyses typically begin by clustering a gene-by-cell expression matrix to empirically define groups of cells with similar expression profiles. We describe new methods and a new open source library, minicore, for efficient k-means++ center finding and k-means clustering of scRNA-seq data. Minicore works with sparse count data, as it emerges from typical scRNA-seq experiments, as well as with dense data from after dimensionality reduction. Minicore's novel vectorized weighted reservoir sampling algorithm allows it to find initial k-means++ centers for a 4-million cell dataset in 1.5 minutes using 20 threads. Minicore can cluster using Euclidean distance, but also supports a wider class of measures like Jensen-Shannon Divergence, Kullback-Leibler Divergence, and the Bhattacharyya distance, which can be directly applied to count data and probability distributions. Further, minicore produces lower-cost centerings more efficiently than scikit-learn for scRNA-seq datasets with millions of cells. With careful handling of priors, minicore implements these distance measures with only minor (k-means++, localsearch++ and mini-batch k-means can cluster a 4-million cell dataset in minutes, using less than 10GiB of RAM. This memory-efficiency enables atlas-scale clustering on laptops and other commodity hardware. Finally, we report findings on which distance measures give clusterings that are most consistent with known cell type labels. Availability: The open source library is at https://github.com/dnbaker/minicore. Code used for experiments is at https://github.com/dnbaker/minicore-experiments.

https://dl.acm.org/doi/pdf/10.1145/3459930.3469523

Fast and memory-efficient scRNA-seq k-means clustering with various distances

In their seminal work, Alon, Matias, and Szegedy introduced several sketching techniques, including showing that 4-wise independence is sufficient to obtain good approximations of the second frequency moment. In this work, we show that their sketching technique can be extended to product domains $[n]^k$ by using the product of 4-wise independent functions on $[n]$. Our work extends that of Indyk and McGregor, who showed the result for $k = 2$. Their primary motivation was the problem of identifying correlations in data streams. In their model, a stream of pairs $(i,j) \in [n]^2$ arrive, giving a joint distribution $(X,Y)$, and they find approximation algorithms for how close the joint distribution is to the product of the marginal distributions under various metrics, which naturally corresponds to how close $X$ and $Y$ are to being independent. By using our technique, we obtain a new result for the problem of approximating the $\ell_2$ distance between the joint distribution and the product of the marginal distributions for $k$-ary vectors, instead of just pairs, in a single pass. Our analysis gives a randomized algorithm that is a $(1 \pm \epsilon)$ approximation (with probability $1-\delta$) that requires space logarithmic in $n$ and $m$ and proportional to $3^k$.

Measuring $k$-Wise Independence of Streaming Data

Given a stream $S$ of length $m$ with element frequencies $f_i$, $i\in[n]$, and a function ${g:\mathbb{N}\to\mathbb{R}}$, we consider the problem computing a $(1\pm\epsilon)$-approximation to $\sum g(f_i)$. Most previous efforts have focused on specific functions, for example the $p$-th moment $g(x)=|x|^p$ or other norms. The main contributions of this paper are the complete classification of the space necessary for approximating periodic and decreasing functions, up to polylogarithmic factors, and a sublinear space algorithm for non-monotone functions satisfying a relatively simple sufficient condition. This is the first streaming complexity characterization for functions that are not just monotonically increasing, and it is the first improvement on the complexity of general streaming frequency sums since 2010. Our sharpest results characterize nonincreasing functions in terms of the domain size \emph{and} stream length. In contrast to approximations of the frequency moments, it turns out the storage required to approximate a decreasing function depends delicately on the length of the stream. We apply our results to derive the first sublinear-space approximations to the harmonic mean, geometric mean, and the cumulative histogram of the frequencies.

Vladimir Braverman

Papers

On Activation Function Coresets for Network Pruning

Schatten Norms in Matrix Streams: Hello Sparsity, Goodbye Dimension

Fast and memory-efficient scRNA-seq k-means clustering with various distances

Measuring $k$-Wise Independence of Streaming Data

Streaming sums in sublinear space.