Scientific facilities such as the Advanced Light Source (ALS) and Joint Genome Institute and projects such as the Materials Project have an increasing need to capture, store, and analyze dynamic semi-structured data and metadata. A similar growth of semi-structured data within large Internet service providers has led to the creation of NoSQL data stores for scalable indexing and MapReduce for scalable parallel analysis. MapReduce and NoSQL stores have been applied to scientific data. Hadoop, the most popular open source implementation of MapReduce, has been evaluated, utilized and modified for addressing the needs of different scientific analysis problems. ALS and the Materials Project are using MongoDB, a document oriented NoSQL store. However, there is a limited understanding of the performance trade-offs of using these two technologies together.In this paper we evaluate the performance, scalability and fault-tolerance of using MongoDB with Hadoop, towards the goal of identifying the right software environment for scientific data analysis.

/pdf/performance-evaluation-of-a-mongodb-and-hadoop-platform-for-uga3ttt9iy.pdf

Performance evaluation of a MongoDB and hadoop platform for scientific data analysis

While Big Data gradually become a hot topic of research and business and has been everywhere used in many industries, Big Data security and privacy has been increasingly concerned. However, there is an obvious contradiction between Big Data security and privacy and the widespread use of Big Data. In this paper, we firstly reviewed the enormous benefits and challenges of security and privacy in Big Data. Then, we present some possible methods and techniques to ensure Big Data security and privacy.

Big Data security and privacy: A review

We are constantly being told that we live in the Information Era - the Age of BIG data. It is clearly apparent that organizations need to employ data-driven decision making to gain competitive advantage. Processing, integrating and interacting with more data should make it better data, providing both more panoramic and more granular views to aid strategic decision making. This is made possible via Big Data exploiting affordable and usable Computational and Storage Resources. Many offerings are based on the Map-Reduce and Hadoop paradigms and most focus solely on the analytical side. Nonetheless, in many respects it remains unclear what Big Data actually is, current offerings appear as isolated silos that are difficult to integrate and/or make it difficult to better utilize existing data and systems. Paper addresses this lacunae by characterising the facets of Big Data and proposing a framework in which Big Data applications can be developed. The framework consists of three Stages and seven Layers to divide Big Data application into modular blocks. The aim is to enable organizations to better manage and architect a very large Big Data application to gain competitive advantage by allowing management to have a better handle on data processing.

Big Data Framework

With the advancement in the database technology NoSQL databases are becoming more and more popular now a days. The cloud-based NoSQL database MongoDB is one of them. As one can know the data modeling strategies for relational and non-relational databases differ drastically. Modeling of data in MongoDB database depends on the data and characteristics of MongoDB. Due to variation in data models MongoDB based application performance gets affected. In the present paper we have applied two different modeling styles as embedding of documents and normalization on collections. With the embedding feature we may face situation where documents grow in size after creation which may degrade the performance of database. The maximum document size allowed in MongoDB is limited. With references we get maximum flexibility than embedding but client-side applications must issue follow-up queries to resolve the references. The joins in this case cannot be effectively used. Hence there is need for defining the strategy of extent of normalization and embedding to get better performance in the mixed situation. The paper discussed here shows the variation in the performance along with the change in the modeling style with reference to normalization and embedding and it gives the base to find the extent of normalization and embedding for reducing query execution time.

A study of normalization and embedding in MongoDB

With the emergence of Big Data, the use of NoSQL (Not only SQL) technology is rising rapidly among internet companies and other enterprises. Benefits include simplicity of design, horizontal scaling and finer control over availability. NoSQL databases are increasingly considered a viable alternative to relational databases, as more organizations recognize that its schema less data model is a better method for handling the large volumes of structured, semi structured and unstructured data, being captured and processed today. For example NoSQL databases are often used to collect and store social media data. This paper aims to introduce the concepts behind NoSQL, provides a review of relevant literature, highlights the different NoSQL database types, and provide arguments for and against adopting NoSQL. A small prototype application has been developed to assess the stated NoSQL benefits and illustrate the differences between the SQL and NoSQL approaches. The last section of the paper offers some conclusions and recommendations for further research to expand upon our research work.

Handling Big Data Using NoSQL

Data aggregation is one of the key features used in databases, especially for Business Intelligence (e.g., ETL, OLAP) and analytics/data mining. When considering SQL databases, aggregation is used to prepare and visualize data for deeper analyses. However, these operations are often impossible on very large volumes of data regarding memory-and-time-consumption. In this paper, we show how NoSQL databases such as MongoDB and its key-value stores, thanks to the native MapReduce algorithm, can provide an efficient framework to aggregate large volumes of data. We provide basic material about the MapReduce algorithm, the different NoSQL databases (read intensive vs. write intensive). We investigate how to efficiently modelize the data framework for BI and analytics. For this purpose, we focus on read intensive NoSQL databases using MongoDB and we show how NoSQL and MapReduce can help handling large volumes of data.

/pdf/reduce-you-say-what-nosql-can-do-for-data-aggregation-and-bi-4tomi8sghl.pdf

Laurent Bonnet

Papers

Reduce, You Say: What NoSQL Can Do for Data Aggregation and BI in Large Repositories