The paper aims to shed some light on the question why clustering algorithms, despite being quantitative and hence supposedly objective in nature, yield different and varied results. To do that, we took 10 common clustering algorithms and tested them over four known datasets, used in the literature as baselines with agreed upon clusters. One additional method, Binary-Positive, developed by our team, was added to the analysis. The results affirm the unpredictable nature of the clustering process, point to different assumptions taken by different methods. One conclusion of the study is to carefully choose the appropriate clustering method for any given application.

Investigating diversity of clustering methods: An empirical comparison

A knowledge tool, which includes a binary dataset for representing relationship patterns between objects and methods of its use. The use of the binary representation is based on an algorithm of data clustering according to binary similarity indices, which are derived from the binary matrix. Applications which are based on the binary representation and its compression capability include data mining, text mining, search engines, pattern recognition, enhancing data exchange rate between computerized devices, database implementation on hardware, saving storage space and adaptive network addressing.

Unified binary model and methodology for knowledge representation and for data and information mining

Bitmap indices are widely used for large read-only repositories in data warehouses and scientific databases. Their binary representation allows for the use of bitwise operations and specialized run-length compression techniques. Due to a trade-off between compression and query efficiency, bitmap compression schemes are aligned using a fixed encoding length size (typically the word length) to avoid explicit decompression during query time. In general, smaller encoding lengths provide better compression, but require more decoding during query execution. However, when the difference in size is considerable, it is possible for smaller encodings to also provide better execution time. We posit that a tailored encoding length for each bit vector will provide better performance than a one-size-fits-all approach. We present a framework that optimizes compression and query efficiency by allowing bitmaps to be compressed using variable encoding lengths while still maintaining alignment to avoid explicit decompression. Efficient algorithms are introduced to process queries over bitmaps compressed using different encoding lengths. An input parameter controls the aggressiveness of the compression providing the user with the ability to tune the tradeoff between space and query time. Our empirical study shows this approach achieves significant improvements in terms of both query time and compression ratio for synthetic and real data sets. Compared to 32-bit WAH, VAL-WAH produces up to 1.8× smaller bitmaps and achieves query times that are 30% faster.

A tunable compression framework for bitmap indices

Data Warehouses are the core of the modern systems for decision making They store integrated information extracted from various and heterogeneous data sources, making it available in multidimensional form for analyses aimed at improving the users’ knowledge of their business Though the first use of the term dates back to the 80s, only during the late 90s data warehousing has emerged as a research area on its own, though in strict correlation with several other research topics as database integration, view materialization, data visualization, etc This paper surveys more than 20 years of research on data warehouse systems, from their early relational implementations (still widely adopted in corporate environments), to the new architectures solicited by Business Intelligence 20 scenarios during the last decade, and up to the exciting challenges now posed by the integration with big data settings The timeline of research is organized into three interrelated tracks: techniques, architectures, and methodologies

From Star Schemas to Big Data: 20+ Years of Data Warehouse Research

Big Data Preprocessing : Enabling Smart Data

A binary approach to data storage and retrieval is introduced. It views the data base as a two-dimensional matrix that relates entities to all possible values the attributes of these entities may take. As such, it provides a unified solution to the two conflicting types of data base transactions—operational and managerial. An analytical investigation of the feasibility of binary storage and a compression method for reducing meaningless areas of the matrix are presented. Storage efficiencies of binary and conventional inverted file methods are compared and evaluated. An analysis of retrieval considerations associated with the binary matrix is given, particularly the issue of going from high to low orders of compression. Results of these analyses indicate that the binary data base's efficiency increases with increases in query complexity. Future research directions are sited and discussed.

Rafi Maayan

Papers

Storage and retrieval considerations of binary data bases