Testing for forbidden order patterns in an array
16 Jan 2017-pp 1582-1597
TL;DR: It is shown that adaptivity can make a big difference in testing non-monotone patterns, and an adaptive algorithm is developed that for any π ∈ 𝔖3, tests π-freeness by making (ϵ−1 log n)O(1) queries.
Abstract: In this paper, we study testing of sequence properties that are defined by forbidden order patterns. A sequence f : {1, . . . , n} → ℝ of length n contains a pattern π ∈ 𝔖k (𝔖k is the group of permutations of k elements), iff there are indices i1 f(iy) whenever π(x) > π(y). If f does not contain π, we say f is π-free. For example, for π = (2, 1), the property of being π-free is equivalent to being non-decreasing, i.e. monotone. The property of being (k, k − 1, . . . , 1)-free is equivalent to the property of having a partition into at most k − 1 non-decreasing subsequences.Let π ∈ 𝔖k, k constant, be a (forbidden) pattern. Assuming f is stored in an array, we consider the property testing problem of distinguishing the case that f is π-free from the case that f differs in more than ϵn places from any π-free sequence. We show the following results: There is a clear dichotomy between the monotone patterns and the non-monotone ones:• For monotone patterns of length k, i.e., (k, k − 1, . . . , 1) and (1, 2, . . . , k), we design non-adaptive one-sided error ϵ-tests of (ϵ−1 log n)O(k2) query complexity.• For non-monotone patterns, we show that for any size-k non-monotone π, any non-adaptive one-sided error ϵ-test requires at least [EQUATION] queries. This general lower bound can be further strengthened for specific non-monotone k-length patterns to Ω(n1−2/(k+1)).On the other hand, there always exists a non-adaptive one-sided error ϵ-test for π ∈ 𝔖k with O(ϵ−1/kn1−1/k) query complexity. Again, this general upper bound can be further strengthened for specific non-monotone patterns. E.g., for π = (1, 3, 2), we describe an ϵ-test with (almost tight) query complexity of [EQUATION].Finally, we show that adaptivity can make a big difference in testing non-monotone patterns, and develop an adaptive algorithm that for any π ∈ 𝔖3, tests π-freeness by making (ϵ−1 log n)O(1) queries.For all algorithms presented here, the running times are linear in their query complexity.
Citations
More filters
Journal Article•
TL;DR: Improved algorithms for testing monotonicity of functions are presented, given the ability to query an unknown function f: Σ n ↦ Ξ, and the test always accepts a monotone f, and rejects f with high probability if it is e-far from being monotones.
Abstract: We present improved algorithms for testing monotonicity of functions. Namely, given the ability to query an unknown function f: Σ n ↦ Ξ, where Σ and Ξ are finite ordered sets, the test always accepts a monotone f, and rejects f with high probability if it is e-far from being monotone (i.e., every monotone function differs from f on more than an e fraction of the domain). For any e > 0, the query complexity of the test is O((n/e) · log ∣Σ ∣ · log ∣Ξ∣). The previous best known bound was \(\tilde{O}((n^2/\epsilon) \cdot \vert\Sigma\vert^2 \cdot \vert\Xi\vert)\).
152 citations
Journal Article•
TL;DR: A new technique for proving lower bounds in property testing is developed, by showing a strong connection between testing and communication complexity, and significantly strengthens the best known bounds.
Abstract: We develop a new technique for proving lower bounds in property testing, by showing a strong connection between testing and communication complexity. We give a simple scheme for reducing communication problems to testing problems, thus allowing us to use known lower bounds in communication complexity to prove lower bounds in testing. This scheme is general and implies a number of new testing bounds, as well as simpler proofs of several known bounds. For the problem of testing whether a boolean function is k-linear (a parity function on k variables), we achieve a lower bound of Omega(k) queries, even for adaptive algorithms with two-sided error, thus confirming a conjecture of Goldreich (2010). The same argument behind this lower bound also implies a new proof of known lower bounds for testing related classes such as k-juntas. For some classes, such as the class of monotone functions and the class of s-sparse GF(2) polynomials, we significantly strengthen the best known bounds.
107 citations
TL;DR: This work proposes new algorithms for counting patterns and provides a nearly linear time computation of a statistic by Yanagimoto (1970), Bergsma and Dassios (2010), which yields a natural and strongly consistent variant of Hoeffding's test for independence of X and Y.
Abstract: A sample of n generic points in the xy-plane defines a permutation that relates their ranks along the two axes. Every subset of k points similarly defines a pattern, which occurs in that permutation. The number of occurrences of small patterns in a large permutation arises in many areas, including nonparametric statistics. It is therefore desirable to count them more efficiently than the straightforward ~O(n^k) time algorithm.
This work proposes new algorithms for counting patterns. We show that all patterns of order 2 and 3, as well as eight patterns of order 4, can be counted in nearly linear time. To that end, we develop an algebraic framework that we call corner tree formulas. Our approach generalizes the existing methods and allows a systematic study of their scope.
Using the machinery of corner trees, we find twenty-three independent linear combinations of order-4 patterns, that can be computed in time ~O(n). We also describe an algorithm that counts one of the remaining 4-patterns, and hence all 4-patterns, in time ~O(n^(3/2)).
As a practical application, we provide a nearly linear time computation of a statistic by Yanagimoto (1970), Bergsma and Dassios (2010). This statistic yields a natural and strongly consistent variant of Hoeffding's test for independence of X and Y, given a random sample as above. This improves upon the so far most efficient ~O(n^2) algorithm.
20 citations
Proceedings Article•
07 Jan 2018TL;DR: In this paper, the authors consider the problem of testing for π-freeness with one-sided error, and show that for most permutations π of length k, any non-adaptive onesided ϵ-test requires ϵ−1/(k−Θ(1))n1−1/k) queries.
Abstract: A sequence f: {1, . . . , n} → R contains a permutation π of length k if there exist i1 ik such that, for all x, y, f(ix) f(iy) if and only if π(x) π(y); otherwise, f is said to be π-free. In this work, we consider the problem of testing for π-freeness with one-sided error, continuing the investigation of [Newman et al., SODA'17]. We demonstrate a surprising behavior for non-adaptive tests with one-sided error: While a trivial sampling-based approach yields an ϵ-test for π-freeness making Θ(ϵ−1/kn1−1/k) queries, our lower bounds imply that this is almost optimal for most permutations! Specifically, for most permutations π of length k, any non-adaptive one-sided ϵ-test requires ϵ−1/(k−Θ(1))n1−1/(k−Θ(1)) queries; furthermore, the permutations that are hardest to test require Θ(ϵ−/(k−1)n1−1/(k−1)) queries, which is tight in n and ϵ. Additionally, we show two hierarchical behaviors here. First, for any k and l ≤ k − 1, there exists some π of length k that requires [EQUATION] non-adaptive queries. Second, we show an adaptivity hierarchy for π = (1, 3, 2) by proving upper and lower bounds for (one- and two-sided) testing of π-freeness with r rounds of adaptivity. The results answer open questions of Newman et al. and [Canonne and Gur, CCC'17].
17 citations
01 Dec 2016
TL;DR: It is proved that convexity can be tested with \(O({Epsilon }^{-1})\) queries by testers that can make queries of their choice while uniform testers for this property require \(\varOmega ({\epsilon )^{-5/4}\) samples.
Abstract: We investigate testing of properties of 2-dimensional figures that consist of a black object on a white background. Given a parameter epsilon in (0,1/2), a tester for a specified property has to accept with probability at least 2/3 if the input figure satisfies the property and reject with probability at least 2/3 if it does not. In general, property testers can query the color of any point in the input figure.
We study the power of testers that get access only to uniform samples from the input figure. We show that for the property of being a half-plane, the uniform testers are as powerful as general testers: they require only O(1/epsilon) samples. In contrast, we prove that convexity can be tested with O(1/epsilon) queries by testers that can make queries of their choice while uniform testers for this property require Omega(1/epsilon^{5/4}) samples. Previously, the fastest known tester for convexity needed Theta(1/epsilon^{4/3}) queries.
16 citations
Cites background from "Testing for forbidden order pattern..."
...Many types of objects have been investigated in the property testing framework, including graphs [26, 21, 1], functions [12, 25, 16, 20, 28, 40, 3, 35], distributions [6, 49, 44, 13, 24], strings [5, 32], arrays [18, 38, 34] and geometric objects [15, 14]....
[...]
References
More filters
Proceedings Article•
31 Jul 1994TL;DR: Preliminary experiments with a dynamic programming approach to pattern detection in databases, based on the dynamic time warping technique used in the speech recognition field, are described.
Abstract: Knowledge discovery in databases presents many interesting challenges within the context of providing computer tools for exploring large data archives. Electronic data repositories are growing quickly and contain data from commercial, scientific, and other domains. Much of this data is inherently temporal, such as stock prices or NASA telemetry data. Detecting patterns in such data streams or time series is an important knowledge discovery task. This paper describes some preliminary experiments with a dynamic programming approach to the problem. The pattern detection algorithm is based on the dynamic time warping technique used in the speech recognition field.
3,229 citations
"Testing for forbidden order pattern..." refers background in this paper
...Another motivation comes from the study of patterns and motifs in time series analysis [BC94; KLC02; Pat+02], for example, series of measurements from sensors, stock market data or data of an electrocardiogram....
[...]
TL;DR: The characterizations provide results in the area of coding theory by giving extremely fast and efficient error-detecting schemes for some well-known codes and play a crucial role in subsequent results on the hardness of approximating some NP-optimization problems.
Abstract: The study of self-testing and self-correcting programs leads to the search for robust characterizations of functions. Here the authors make this notion precise and show such a characterization for polynomials. From this characterization, the authors get the following applications. Simple and efficient self-testers for polynomial functions are constructed. The characterizations provide results in the area of coding theory by giving extremely fast and efficient error-detecting schemes for some well-known codes. This error-detection scheme plays a crucial role in subsequent results on the hardness of approximating some NP-optimization problems.
954 citations
"Testing for forbidden order pattern..." refers methods in this paper
...Property testing was introduced by Rubinfeld and Sudan [41]....
[...]
...Property testing was introduced by Rubinfeld and Sudan [RS96]....
[...]
21 Apr 2022
TL;DR: This book discusses Permutations as Genome Rearrangements, algorithms and permutations, and the proof of the Stanley-Wilf Conjecture.
Abstract: In One Line and Close. Permutations as Linear Orders. Descents Alternating Runs Alternating Subsequences In One Line and Anywhere. Permutations as Linear Orders. Inversions. Inversions Inversion in Permutations of Multisets In Many Circles. Permutations as Products of Cycles. Decomposing a Permutation into Cycles Type and Stirling Numbers Cycle Decomposition versus Linear Order Permutations with Restricted Cycle Structure In Any Way but This. Pattern Avoidance. The Basics. The Notion of Pattern Avoidance Patterns of Length Three Monotone Patterns Patterns of Length Four The Proof of the Stanley-Wilf Conjecture In This Way but Nicely. Pattern Avoidance. Follow-Up. Polynomial Recurrences Containing a Pattern Many Times Containing a Pattern a Given Number of Times Mean and Insensitive. Random Permutations. The Probabilistic Viewpoint Expectation Variance and Standard Deviation An Application: Longest Increasing Subsequences Permutations versus Everything Else. Algebraic Combinatorics of Permutations. The Robinson-Schensted-Knuth Correspondence Posets of Permutations Simplicial Complexes of Permutations Get Them All. Algorithms and Permutations. Generating Permutations Stack Sorting Permutations Variations of Stack Sorting How Did We Get Here? Permutations as Genome Rearrangements. Introduction Block Transpositions Block Interchanges Block Transpositions Revisited Solutions to Odd-Numbered Exercises References List of Frequently Used Notation Index Exercises, Problems, and Problem Solutions appear at the end of each chapter.
529 citations
TL;DR: This paper examines the extremal problem of how many 1-entries an n × n 0-1 matrix can have that avoids a certain fixed submatrix P and proves a linear bound for any permutation matrix P.
505 citations
23 Jul 2002
TL;DR: A novel technique is introduced that defines a pattern surprising if the frequency of its occurrence differs substantially from that expected by chance, given some previously seen data.
Abstract: The problem of finding a specified pattern in a time series database (i.e. query by content) has received much attention and is now a relatively mature field. In contrast, the important problem of enumerating all surprising or interesting patterns has received far less attention. This problem requires a meaningful definition of "surprise", and an efficient search technique. All previous attempts at finding surprising patterns in time series use a very limited notion of surprise, and/or do not scale to massive datasets. To overcome these limitations we introduce a novel technique that defines a pattern surprising if the frequency of its occurrence differs substantially from that expected by chance, given some previously seen data.
451 citations
"Testing for forbidden order pattern..." refers background in this paper
...27 Another motivation comes from the study of patterns and motifs in time series analysis [8, 33, 40], for 28 example, series of measurements from sensors, stock market data or data of an electrocardiogram....
[...]