Technologies for Big Data

TLDR: This chapter provides a review and analysis of several key Big Data technologies, including Map-Reduce, NOSQL technology, MPP (Massively Parallel Processing), and In Memory Databases technologies.

Abstract: This chapter provides a review and analysis of several key Big Data technologies. Currently, there are many Big Data technologies in development and implementation; hence, a comprehensive review of all of these technologies is beyond the scope of this chapter. This chapter focuses on the most popularly accepted technologies. The key Big Data technologies to be discussed include: Map-Reduce, NOSQL technology, MPP (Massively Parallel Processing), and In Memory Databases technologies. For each of these Big Data technologies, the following subtopics are discussed: the history and genesis of the Big Data technologies, problem set that this technology solves for Big Data analytics, the details of the technologies, including components, technical architecture, and theory of operations. This is followed by technical operation and infrastructure (compute, storage, and network), design considerations, and performance benchmarks. Finally, this chapter provides an integrated approach to the above-mentioned Big Data technologies. INTRODUCTION: THE CHALLENGE OF BIG DATA The amount of data in the world is being collected and stored at unprecedented rates. A study by IDC Gantz & Reinsel, (2011) indicates that the world’s information is doubling every two years. Also the IDC study by Gantz & Reinsel (2011), mentions that the world created a staggering 1.8 zettabytes of information (a zettabyte is 1000 exabytes), and projections suggest that by 2020, we’ll generate will generate 50 times that amount. Big Data has been defined as, when data sets get so large, that traditional technologies, Kapil Bakshi Cisco Systems Inc., USA