Database management systems will continue to manage large data volumes. Thus, efficient algorithms for accessing and manipulating large sets and sequences will be required to provide acceptable performance. The advent of object-oriented and extensible database systems will not solve this problem. On the contrary, modern data models exacerbate the problem: In order to manipulate large sets of complex objects as efficiently as today's database systems manipulate simple records, query-processing algorithms and software will become more complex, and a solid understanding of algorithm and architectural issues is essential for the designer of database management software. This survey provides a foundation for the design and implementation of query execution facilities in new database management systems. It describes a wide array of practical query evaluation techniques for both relational and postrelational database systems, including iterative execution of complex query evaluation plans, the duality of sort- and hash-based set-matching algorithms, types of parallel query execution and their implementation, and special operators for emerging database application domains.

/pdf/query-evaluation-techniques-for-large-databases-45td1anbls.pdf

Query evaluation techniques for large databases

Aggregation in traditional database systems is performed in batch mode: a query is submitted, the system processes a large volume of data over a long period of time, and, eventually, the final answer is returned. This archaic approach is frustrating to users and has been abandoned in most other areas of computing. In this paper we propose a new online aggregation interface that permits users to both observe the progress of their aggregation queries and control execution on the fly. After outlining usability and performance requirements for a system supporting online aggregation, we present a suite of techniques that extend a database system to meet these requirements. These include methods for returning the output in random order, for providing control over the relative rate at which different aggregates are computed, and for computing running confidence intervals. Finally, we report on an initial implementation of online aggregation in POSTGRES.

Online aggregation

This introductory text on the design and analysis of sample surveys emphasizes the practical aspects of survey problems. It begins with brief chapters on the role of sample surveys in the modern world. Thereafter, each chapter introduces a sample survey design or estimation procedure by describing the pertinent practical problem. The authors describe the methodology proposed for solving the problem and provide the details of the estimation procedure, including a compact presentation of the formulas needed to complete the analysis. Then, a practical example is worked out in complete detail. At the end of each chapter, a wealth of exercises gives students ample opportunity to practice the techniques and stretch their grasp of ideas.

Elementary Survey Sampling

This chapter contains sections titled: Introduction, The Idea Behind View Maintenance, Using Full Information, Using Partial Information, Open Problems, Acknowledgments

Maintenance of materialized views: problems, techniques, and applications

How many kinds are there? Suppose that a population is partitioned into C classes. In many situations interest focuses not on estimation of the relative sizes of the classes, but on estimation of C itself. For example, biologists and ecologists may be interested in estimating the number of species in a population of plants or animals, numismatists may be concemed with estimating the number of dies used to produce an ancient coin issue, and linguists may be interested in estimating the size of an author's vocabulary. In this article we review the problem of statistical estimation of C. Many approaches have been proposed, some purely data-analytic and others based in sampling theory. In the latter case numerous variations have been considered. The population may be finite or infinite. If finite, samples may be taken with replacement (multinomial sampling) or without replacement (hypergeometric sampling), or by Bernoulli sampling; if infinite, sampling may be multinomial or Bernoulli, or the sample may be th...

Estimating the Number of Species: A Review

Present database systems process all the data related to a query before giving out responses. As a result, the size of the data to be processed becomes excessive for real-time/time-constrained environments. A new methodology is needed to cut down systematically the time to process the data involved in processing the query. To this end, we propose to use data samples and construct an approximate synthetic response to a given query.In this paper, we consider only COUNT(E) type queries, where E is an arbitrary relational algebra expression. We make no assumptions about the distribution of attribute values and ordering of tuples in the input relations, and propose consistent and unbiased estimators for arbitrary COUNT(E) type queries. We design a sampling plan based on the cluster sampling method to improve the utilization of sampled data and to reduce the cost of sampling. We also evaluate the performance of the proposed estimators.

/pdf/statistical-estimators-for-relational-algebra-expressions-29cd6qmkhw.pdf

Statistical estimators for relational algebra expressions

We consider those database environments in which queries have strict timing constraints, and develop a time-constrained query evaluation methodology. For aggregate relational algebra queries, we describe a time constrained query evaluation algorithm. The algorithm, which is implemented in our prototype DBMS, iteratively samples from input relations, and evaluates the associated estimators developed in our previous work, until a stopping criterion (e.g., a time quota or a desired error range) is satisfied.To determine sample sizes at each stage of the iteration (so that the time quota will not be overspent) we need to have (a) accurate sample selectivity estimations of the RA operators in the query, (b) precise time cost formulas, and (c) good time-control strategies. To estimate the sample selectivities of RA operators, we use a runtime sample selectivity estimation and improvement approach which is flexible. For query time estimations, we use time-cost formulas which are adaptive and precise. To use the time quota efficiently, we propose statistical and heuristic time-control strategies to control the risk of overspending the time quota. Preliminary evaluation of the implemented prototype is also presented.

Processing aggregate relational queries with hard time constraints

In thk paper, we discuss the problem of estimating a COUNT(E) query having a guaranteed error bound with a certain confidence level, where E is an arbitrary relational algebra (RA) expression with Select, Join and Intersection operations. Such a~ estimate is useful in query optimization as well as in real-time databases as it takes significantly less time to obtain compared with evaluating COUNT(E) completely. To guarantee the desired error bound with the given confidence level, we use double sampling (or two-phase sampling) where in the first stage preliminary information is obtained from a small pilot sample taken from the input relations. And then, the sample size for the second stage is deter-

Error-constrained COUNT query evaluation in relational databases

This paper discusses the estimation of COUNT(E) queries by sampling, where E is an arbitrary relational algebra expression. Consistent and unbiased statistical estimators for COUNT(E) are proposed without any assumptions on the distributions of attribute values or the orderings of tuples in the operand relatlons of E. We present a set of COUNT’(E) estimator evaluation algorithms, all based on simple random sampling, and each suitable for a different type of relational algebra expression E. To improve the efficiency, we propose three enhancements, and revise the estimator evaluation algorithms to incorporate the enhancements. One of the enhancements m the use of cluster sampling. Estimator evaluation algorithms with the enhancements have been incorporated into a prototype DBMS. We present the performance evaluation of the estimators using the implemented prototype DBMS.

Wen-Chi Hou

Papers

Statistical estimators for relational algebra expressions

Processing aggregate relational queries with hard time constraints

Error-constrained COUNT query evaluation in relational databases

Statistical estimators for aggregate relational algebra queries

Proceedings of the 11th International Conference on Scientific and Statistical Database Management