Applications that require real-time processing of high-volume data steams are pushing the limits of traditional data processing infrastructures. These stream-based applications include market feed processing and electronic trading on Wall Street, network and infrastructure monitoring, fraud detection, and command and control in military environments. Furthermore, as the "sea change" caused by cheap micro-sensor technology takes hold, we expect to see everything of material significance on the planet get "sensor-tagged" and report its state or location in real time. This sensorization of the real world will lead to a "green field" of novel monitoring and control applications with high-volume and low-latency processing requirements.Recently, several technologies have emerged---including off-the-shelf stream processing engines---specifically to address the challenges of processing high-volume, real-time data without requiring the use of custom code. At the same time, some existing software technologies, such as main memory DBMSs and rule engines, are also being "repurposed" by marketing departments to address these applications.In this paper, we outline eight requirements that a system software should meet to excel at a variety of real-time stream processing applications. Our goal is to provide high-level guidance to information technologists so that they will know what to look for when evaluation alternative stream processing solutions. As such, this paper serves a purpose comparable to the requirements papers in relational DBMSs and on-line analytical processing. We also briefly review alternative system software technologies in the context of our requirements.The paper attempts to be vendor neutral, so no specific commercial products are mentioned.

/pdf/the-8-requirements-of-real-time-stream-processing-47v1h05t4e.pdf

The 8 requirements of real-time stream processing

In the course of the evolution of management support towards corporate wide Business Intelligence infrastructures, the integration of components for handling unstructured data comes into focus. In this paper, three types of approaches for tackling the respective challenges are distinguished. The approaches are mapped to a three layer BI framework and discussed regarding challenges and business potential. The application of the framework is exemplified for the domains of Competitive Intelligence and Customer Relationship Management.

Management Support with Structured and Unstructured Data-An Integrated Business Intelligence Framework

Improved method of and apparatus for joining and aggregating data elements integrated within a relational database management system (RDBMS) using a non-relational multi-dimensional data structure (MDD). The improved RDBMS system of the present invention can be used to realize achieving a significant increase in system performance (e.g. deceased access/search time), user flexibility and ease of use. The improved RDBMS system of the present invention can be used to realize an improved Data Warehouse for supporting on-line analytical processing (OLAP) operations or to realize an improved informational database system or the like.

Relational database management system having integrated non-relational multi-dimensional data store of aggregated data elements

Online Analytical Processing (OLAP) systems considerably ease the process of analyzing business data and have become widely used in industry. Such systems primarily employ multidimensional data models to structure their data. However current multidimensional data models fall short in their abilities to model the complex data found in some real world application domains. The paper presents nine requirements to multidimensional data models, each of which is exemplified by a real world, clinical case study. A survey of the existing models reveals that the requirements not currently met include support for many-to-many relationships between facts and dimensions, built-in support for handling chance and time, and support for uncertainty as well as different levels of granularity in the data. The paper defines an extended multidimensional data model, and an associated algebra, which address all nine requirements.

/pdf/multidimensional-data-modeling-for-complex-data-455yiyk3rf.pdf

Multidimensional data modeling for complex data

Disclosed is a method, system, and program for query processing. Metadata for a facts metadata object and one or more dimension metadata objects that are associated with the facts metadata object is stored. A view with columns for one or more measures in the facts metadata object and one or more attributes in the one or more dimension metadata objects is constructed. Additional metadata that describes roles of columns in the fact and dimension metadata objects is generated. Also disclosed is a computer-readable medium for storing data for access by a program. A data structure stored in the computer-readable medium includes data for use by the program. The data includes a cube model metadata object that includes a facts metadata object, one or more dimension metadata objects, and one or more join metadata objects that describe how one or more tables in the facts metadata object and one or more tables in the one or more dimension metadata objects are joined. The data also includes a cube metadata object that represents a subset of the cube model metadata object and comprises a view with columns for one or more measures of one of the facts metadata objects and one or more attributes of one or more of the dimension metadata objects and a document that describes roles of columns in the facts metadata object and the one or more dimension metadata objects.

Method, system, and program for use of metadata to create multidimensional cubes in a relational database

Overview Recently, there has been a great deal of discussion in the trade press and elsewhere regarding the coexistence of so-called transaction databases with decision support systems. These discussions usually revolve around the argument that the physical design required for acceptable performance of each is incompatible and that therefore, data should be stored redundantly in multiple enterprise databases: one for transaction processing, and the other for decision support type activities. Also, these same arguments usually confuse physical schema with logical and conceptual schema.

/pdf/providing-olap-to-user-analysts-an-it-mandate-1klqncel3i.pdf

C.T. Salley

Papers

Providing OLAP to User-Analysts: An IT Mandate