Motivation: The enormous amount of short reads generated by the new DNA sequencing technologies call for the development of fast and accurate read alignment programs. A first generation of hash table-based methods has been developed, including MAQ, which is accurate, feature rich and fast enough to align short reads from a single individual. However, MAQ does not support gapped alignment for single-end reads, which makes it unsuitable for alignment of longer reads where indels may occur frequently. The speed of MAQ is also a concern when the alignment is scaled up to the resequencing of hundreds of individuals.

Results: We implemented Burrows-Wheeler Alignment tool (BWA), a new read alignment package that is based on backward search with Burrows–Wheeler Transform (BWT), to efficiently align short sequencing reads against a large reference sequence such as the human genome, allowing mismatches and gaps. BWA supports both base space reads, e.g. from Illumina sequencing machines, and color space reads from AB SOLiD machines. Evaluations on both simulated and real data suggest that BWA is ~10–20× faster than MAQ, while achieving similar accuracy. In addition, BWA outputs alignment in the new standard SAM (Sequence Alignment/Map) format. Variant calling and other downstream analyses after the alignment can be achieved with the open source SAMtools software package.

Availability: http://maq.sourceforge.net 

Contact: [email protected]

/pdf/fast-and-accurate-short-read-alignment-with-burrows-wheeler-43uqhgr33c.pdf

Fast and accurate short read alignment with Burrows–Wheeler transform

The increasing volume of data in modern business and science calls for more complex and sophisticated tools. Although advances in data mining technology have made extensive data collection much easier, it's still always evolving and there is a constant need for new techniques and tools that can help us transform this data into useful information and knowledge. Since the previous edition's publication, great advances have been made in the field of data mining. Not only does the third of edition of Data Mining: Concepts and Techniques continue the tradition of equipping you with an understanding and application of the theory and practice of discovering patterns hidden in large data sets, it also focuses on new, important topics in the field: data warehouses and data cube technology, mining stream, mining social networks, and mining spatial, multimedia and other complex data. Each chapter is a stand-alone guide to a critical topic, presenting proven algorithms and sound implementations ready to be used directly or with strategic modification against live data. This is the resource you need if you want to apply today's most powerful data mining techniques to meet real business challenges. * Presents dozens of algorithms and implementation examples, all in pseudo-code and suitable for use in real-world, large-scale data mining projects. * Addresses advanced topics such as mining object-relational databases, spatial databases, multimedia databases, time-series databases, text databases, the World Wide Web, and applications in several fields. *Provides a comprehensive, practical look at the concepts and techniques you need to get the most out of real business data

/pdf/data-mining-concepts-and-techniques-4dtvdfkvmi.pdf

Data Mining: Concepts and Techniques

Data Mining: Practical Machine Learning Tools and Techniques offers a thorough grounding in machine learning concepts as well as practical advice on applying machine learning tools and techniques in real-world data mining situations. This highly anticipated third edition of the most acclaimed work on data mining and machine learning will teach you everything you need to know about preparing inputs, interpreting outputs, evaluating results, and the algorithmic methods at the heart of successful data mining. Thorough updates reflect the technical changes and modernizations that have taken place in the field since the last edition, including new material on Data Transformations, Ensemble Learning, Massive Data Sets, Multi-instance Learning, plus a new version of the popular Weka machine learning software developed by the authors. Witten, Frank, and Hall include both tried-and-true techniques of today as well as methods at the leading edge of contemporary research. *Provides a thorough grounding in machine learning concepts as well as practical advice on applying the tools and techniques to your data mining projects *Offers concrete tips and techniques for performance improvement that work by transforming the input or output in machine learning methods *Includes downloadable Weka software toolkit, a collection of machine learning algorithms for data mining tasks-in an updated, interactive interface. Algorithms in toolkit cover: data pre-processing, classification, regression, clustering, association rules, visualization

Data Mining: Practical Machine Learning Tools and Techniques

More than twelve years have elapsed since the first public release of WEKA. In that time, the software has been rewritten entirely from scratch, evolved substantially and now accompanies a text on data mining [35]. These days, WEKA enjoys widespread acceptance in both academia and business, has an active community, and has been downloaded more than 1.4 million times since being placed on Source-Forge in April 2000. This paper provides an introduction to the WEKA workbench, reviews the history of the project, and, in light of the recent 3.6 stable release, briefly discusses what has been added since the last stable version (Weka 3.4) released in 2003.

/pdf/the-weka-data-mining-software-an-update-3zs9onav7o.pdf

The WEKA data mining software: an update

Machine Learning is the study of methods for programming computers to learn. Computers are applied to a wide range of tasks, and for most of these it is relatively easy for programmers to design and implement the necessary software. However, there are many tasks for which this is difficult or impossible. These can be divided into four general categories. First, there are problems for which there exist no human experts. For example, in modern automated manufacturing facilities, there is a need to predict machine failures before they occur by analyzing sensor readings. Because the machines are new, there are no human experts who can be interviewed by a programmer to provide the knowledge necessary to build a computer system. A machine learning system can study recorded data and subsequent machine failures and learn prediction rules. Second, there are problems where human experts exist, but where they are unable to explain their expertise. This is the case in many perceptual tasks, such as speech recognition, hand-writing recognition, and natural language understanding. Virtually all humans exhibit expert-level abilities on these tasks, but none of them can describe the detailed steps that they follow as they perform them. Fortunately, humans can provide machines with examples of the inputs and correct outputs for these tasks, so machine learning algorithms can learn to map the inputs to the outputs. Third, there are problems where phenomena are changing rapidly. In finance, for example, people would like to predict the future behavior of the stock market, of consumer purchases, or of exchange rates. These behaviors change frequently, so that even if a programmer could construct a good predictive computer program, it would need to be rewritten frequently. A learning program can relieve the programmer of this burden by constantly modifying and tuning a set of learned prediction rules. Fourth, there are applications that need to be customized for each computer user separately. Consider, for example, a program to filter unwanted electronic mail messages. Different users will need different filters. It is unreasonable to expect each user to program his or her own rules, and it is infeasible to provide every user with a software engineer to keep the rules up-to-date. A machine learning system can learn which mail messages the user rejects and maintain the filtering rules automatically. Machine learning addresses many of the same research questions as the fields of statistics, data mining, and psychology, but with differences of emphasis. Statistics focuses on understanding the phenomena that have generated the data, often with the goal of testing different hypotheses about those phenomena. Data mining seeks to find patterns in the data that are understandable by people. Psychological studies of human learning aspire to understand the mechanisms underlying the various learning behaviors exhibited by people (concept learning, skill acquisition, strategy change, etc.).

Machine learning

We introduce an efficient new algorithm for dy­ namic Huffman coding, called Algorithm V. It performs one-pass coding and transmission in real-time, and uses at most one more bit per letter than does the standard two­ pass Huffman algorithm; this is optimum in the worst case among all one-pass schemes. We also analyze the dynamic Huffman algorithm due to Faller, Gallager, and Knuth. In each algorithm, both the sender and the receiver maintain equivalent dynamically varying Huffman trees. The pro­ cessing time required to encode and decode a letter whose node in the dynamic Huffman tree is currently on the fth level is O(f); hence, the processing can be done in real time. Empirical tests show that Algorithm V performs quite well in practice, often better than the two-pass method. The proposed algorithm is well-suited for file compression and online encoding/decoding in data networks.

Design and Analysis of Dynamic Huffman Coding* (extended abstract)

We consider the natural extension of the single disk caching problem to parallel disk I/O model. We close the existing gap between lower and upper bounds and achieve optimal competitive ratio of O(√D) when lookahead is more than the memory size M. When lookahead is smaller, we derive various upper bounds and lower bounds on the competitive ratio under various adversarial models.

On competitive online read-many parallel disks scheduling

Databases and information retrieval systems storing data from the World Wide Web require new optimization strategies to deal with the different types of data (textual, semi-structured and relational), the massiveness of the data, and the dynamic nature of the data. This dissertation investigates on-line methods for the optimization of databases storing web data. On-line methods avoid access to the underlying data, avoid the rebuilding of data structures from scratch, and adapt to changes in the data and query workload characteristics. On-line techniques are therefore especially suited to web data. 
In this dissertation, we present on-line techniques for two general problems: how to update inverted indexes and how to estimate the selectivity or result size of queries in database systems. For the index update problem, we present the landmark-diff method for updating the inverted index in response to changes in previously indexed documents. The landmark-diff method allows indexes to be updated incrementally without a complete rebuild. 
For selectivity estimation, we propose three on-line techniques that address the selectivity estimation problem in three different context. In the context of relational databases, we present SASH, a Self-Adapting Set of Histograms, that uses probabilistic graphical models to address the following issues in an on-line manner: which sets of attributes to build histograms on, how to build these histograms without looking at data, and how much memory should be allocated to these histograms. 
For XML native databases, we present XPathLearner, an on-line workload-aware method for estimating the result size of given XPath queries. XPathLearner learns the path statistics from query feedback (past query answers) in an on-line manner, and adapts itself to changes in the data and the query workload characteristics. 
In a more general context, we present CXHist, an on-line classification based histogram for estimating the selectivity of a broad class of queries. CXHist models queries instead of data and stores the mapping between queries and their selectivity using a naive Bayesian classifier. We show that CXHist is very accurate in estimating the selectivity of exact match and substring predicates on leaf values reachable by a given XPath in XML databases.

/pdf/on-line-methods-for-database-optimization-1upxdu3ty4.pdf

On-line methods for database optimization

We present a new complexity-theoretic approach to incremental computation. We define complexity classes that capture the intuitive notion of incremental efficiency and study their relation to existing complexity classes. We show that all common logs p-complete problems for poly are incrplt-complete for poly. This suggests that non-redundant problems that seem inherently unparallelizable also seem hard to dynamize. We show that a form of transitive closure is complete under incremental reduction for nlgsp and give similar problems which are incrementally complete for the classes logsp and non-uniform nick. We show that under certain restrictions problems which have efficient dynamic solutions also have efficient parallel solutions. We also look at the time complexity of circuit value and network stability problems restricted to comparator gates. We show that the dynamic version of the comparator-circuit value problem and ``Lex-First Maximal Matching'''' problem is in logsp and that of the comparator-network stability problem and ``Man-Optimal Stable Marriage Problem'''' is in logsp given an oracle which operates in nlgsp. This shows that the dynamic versions of these problems are solvable quickly in parallel even though there are no known nick algorithms to solve them from scratch.

A Complexity-Theoretic Approach to Incremental Computation

Chien et al [1, 2] introduced the geometric Burrows-Wheeler transform (GBWT) as the first succinct text index for I/O-efficient pattern matching in external memory; it operates by transforming a text T into point set S in the two-dimensional plane In this paper we introduce a practical succinct external memory text index, called mKD-GBWT We partition S into ς2 subregions by partitioning the x-axis into ς intervals using the suffix ranges of characters of T and partitioning the y-axis into ς intervals using characters of T, where ς is the alphabet size of T In this way, we can represent a point using fewer bits and perform a query in a reduced region so as to improve the space usage and I/Os of GBWT in practice In addition, we plug a crit-bit tree into each node of string B-trees to represent variable-length strings stored Experimental results show that mKD-GBWT provides significant improvement in space usage compared with the state-of-the-art indexing techniques The source code is available online [3]

Jeffrey Scott Vitter

Papers

Design and Analysis of Dynamic Huffman Coding* (extended abstract)

On competitive online read-many parallel disks scheduling

On-line methods for database optimization

A Complexity-Theoretic Approach to Incremental Computation

Practical Succinct Text Indexes in External Memory