The traditional databases are not capable of handling unstructured data and high volumes of real-time datasets Diverse datasets are unstructured lead to big data, and it is laborious to store, manage, process, analyze, visualize, and extract the useful insights from these datasets using traditional database approaches However, many technical aspects exist in refining large heterogeneous datasets in the trend of big data This paper aims to present a generalized view of complete big data system which includes several stages and key components of each stage in processing the big data In particular, we compare and contrast various distributed file systems and MapReduce-supported NoSQL databases concerning certain parameters in data management process Further, we present distinct distributed/cloud-based machine learning (ML) tools that play a key role to design, develop and deploy data models The paper investigates case studies on distributed ML tools such as Mahout, Spark MLlib, and FlinkML Further, we classify analytics based on the type of data, domain, and application We distinguish various visualization tools pertaining three parameters: functionality, analysis capabilities, and supported development environment Furthermore, we systematically investigate big data tools and technologies (Hadoop 30, Spark 23) including distributed/cloud-based stream processing tools in a comparative approach Moreover, we discuss functionalities of several SQL Query tools on Hadoop based on 10 parameters Finally, We present some critical points relevant to research directions and opportunities according to the current trend of big data Investigating infrastructure tools for big data with recent developments provides a better understanding that how different tools and technologies apply to solve real-life applications

The big data system, components, tools, and technologies: a survey

Cipherbase is a comprehensive database system that provides strong end-to-end data confidentiality through encryption. Cipherbase is based on a novel architecture that combines an industrial strength database engine (SQL Server) with lightweight processing over encrypted data that is performed in secure hardware. The overall architecture provides significant benefits over the state-of-the-art in terms of security, performance, and functionality. This paper presents a prototype of Cipherbase that uses FPGAs to provide secure processing and describes the system engineering details implemented to achieve competitive performance for transactional workloads. This includes hardware-software co-design issues (e.g. how to best offer parallelism), optimizations to hide the latency between the secure hardware and the main system, and techniques to cope with space inefficiencies. All these optimizations were carefully designed not to affect end-to-end data confidentiality. Our experiments with the TPC-C benchmark show that in the worst case when all data are strongly encrypted, Cipherbase achieves 40% of the throughput of plaintext SQL Server. In more realistic cases, if only critical data such as customer names are encrypted, the Cipherbase throughput is more than 90% of plaintext SQL Server.

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/cipherbase.pdf

Transaction processing on confidential data using cipherbase

Data confidentiality is one of the main concerns for users of public cloud services. The key problem is protecting sensitive data from being accessed by cloud administrators who have root privileges and can remotely inspect the memory and disk contents of the cloud servers. While encryption is the basic mechanism that can leveraged to provide data confidentiality, providing an efficient database-as-a-service that can run on encrypted data raises several interesting challenges. In this demonstration we outline the functionality of Cipherbase --- a full fledged SQL database system that supports the full generality of a database system while providing high data confidentiality. Cipherbase has a novel architecture that tightly integrates custom-designed trusted hardware for performing operations on encrypted data securely such that an administrator cannot get access to any plaintext corresponding to sensitive data.

Secure database-as-a-service with Cipherbase

Internet of Things (IoT) devices have grown up to comprise embedded systems and sensors with the ability to connect, collect, and transmit data over the Internet. Although, solutions to secure IoT systems exist, Class-0 IoT devices with insufficient resources to support such solutions are considered too resource constrained for a secure communication. This paper provides a distributed security mechanism that targets Class-0 IoT devices. The research goal is to secure the entire data path in two segments; device-to-gateway and gateway-to-server data communications. The main concern in the provided solution is that lighter security operations with minimal resource requirements are performed in the device, while heavier tasks are performed in the gateway side. The proposed mechanism utilizes a symmetric encryption for data objects combined with the native wireless security to offer a layered security mechanism between the device and the gateway. In the offered solution, the IoT gateways provide additional protection by securing data using Transport Layer Security (TLS). The real-time experimental evaluations have proven the applicability of the proposed mechanism pertaining to the security assurance and the consumed resources of the target IoT devices.

/pdf/a-distributed-security-mechanism-for-resource-constrained-4c3b5ycb9q.pdf

A Distributed Security Mechanism for Resource-Constrained IoT Devices

Data security is a serious concern when we migrate data to a cloud DBMS. Database encryption, where sensitive columns are encrypted before they are stored in the cloud, has been proposed as a mechanism to address such data security concerns. The intuitive expectation is that an adversary cannot "learn" anything about the encrypted columns, since she does not have access to the encryption key. However, query processing becomes a challenge since it needs to "look inside" the data. This tutorial explores the space of designs studied in prior work on processing queries over encrypted data. We cover approaches based on both classic client-server and involving the use of a trusted hardware module where data can be securely decrypted. We discuss the privacy challenges that arise in both approaches and how they may be addressed. Briefly, supporting the full complexity of a modern DBMS including complex queries, transactions and stored procedures leads to significant challenges that we survey and open problems which we highlight.

/pdf/querying-encrypted-data-1iygqgj9b6.pdf

Querying encrypted data

Pro TSQL Programmers Guide is your guide to making the best use of the powerful, Transact-SQL programming language that is built into Microsoft SQL Server's database engine. This edition is updated to cover the new, in-memory features that are part of SQL Server 2014. Discussing new and existing features, the book takes you on an expert guided tour of TransactSQL functionality. Fully functioning examples and downloadable source code bring technically accurate and engaging treatment of TransactSQL into your own hands. Stepbystep explanations ensure clarity, and an advocacy of bestpractices will steer you down the road to success.TransactSQL is the language developers and DBAs use to interact with SQL Server. Its used for everything from querying data, to writing stored procedures, to managing the database. Support for in-memory stored procedures running queries against in-memory tables is new in the language and gets coverage in this edition. Also covered are must-know features such as window functions and data paging that help in writing fast-performing database queries. Developers and DBAs alike can benefit from the expressive power of T-SQL, and Pro T-SQL Programmer's Guide is your roadmap to success in applying this increasingly important database language to everyday business and technical tasks.Covers the newly-introduced, in-memory database featuresShares the best practices used by experienced professionalsGoes deeply into the subject matter an advanced book for the serious readerWhat youll learnGain massive performance boosts from the in-memory database featuresApply window functions to write reporting queries with unparalleled easeMove business logic into the database via procedures and functionsDevelop using advanced features such as encryption and full-text searchStore and retrieve XML data using XQuery and XPathDebug and Optimize T-SQL ExecutionBuild ADO.NET Client ApplicationsWho this book is for Pro TSQL Programmers Guide is written for SQL Server and T-SQL developers with a desire to write advanced-level code and take full advantage of all that SQL Server has to offer. Microsoft .NET programmers will find the book helpful in showing how to run .NET code within SQL Server. Database administrators who need to write triggers and the occasional stored procedure will also benefit from the book. Table of Contents1. Foundations of T-SQL2. Tools of the Trade3. Procedural Code and CASE Expressions4. User Defined Functions5. Stored Procedures6. In-memory Database Programming7. Triggers8. Encryption9. Common Table Expressions10. Advanced Features11. Integrated Full-Text Search12. XML13. XQuery and XPath14. Catalog Views and Dynamic Management Views15. SQL CLR Programming16. .NET Client Programming17. Data Services18. Error Handling and Dynamic SQL19. Performance Monitoring and Tuning20. Appendix A21. Appendix B22. Appendix C23. Appendix D

Pro T-SQL Programmer's Guide

Full-text search (FTS) is a powerful SQL Server feature allowing for advanced searches using multiple languages to find information in documents as well as document properties. FTS is tightly integrated with SQL Server 2012 and can be easily managed with SQL Server Management Studio (SSMS) and monitored with standard dynamic management views. FTS broadens the scope of what is thought of as a T-SQL search by providing meaningful results from sometimes seemingly unstructured textual data. SQL Server 2012 also introduces statistical semantics which allow for searching on document meaning as opposed to simply searching content. Based on word distributions and other factors, statistical semantics allows you to find documents with similar contents.

Full-Text Search

One of your tasks in building enterprise ETL solutions is making it ready to deal with unforeseen events—the kind that may or may not happen, as Dan Quayle so eloquently points out. Some of this planning comes in the form of infrastructure planning (such as hardware sizing and network and storage capacity planning), but a robust ETL system requires that you also focus attention on building in error handling. Although much of your attention will be focused on issues you can anticipate, unforeseen events can cause the biggest heartburn. This chapter covers SSIS events and event handling, as well as error handling and recovery.

Events and Error Handling

Indexing, searching, and sorting data are related functions that all have requirements that contradict the needs of secure data encryption. Indexing, searching, and related functionality requires you to assign order to your data. Order, in turn, depends on recognizing patterns in your data. As I discussed in Chapter 1, the security of encryption algorithms depends on eliminating recognizable patterns from your data. This renders encrypted data essentially useless in terms of efficient search and sort.

Indexing Encrypted Data

Projects that require bringing together data from multiple sources—for example, data warehouse, data mart, or operational data store (ODS) projects—are extremely common. You could spend months gathering business requirements, putting together technical specifications, designing target databases, and coding and testing your ETL process. You could spend an eternity in “ad hoc maintenance mode” rewriting large sections of code that don’t handle unanticipated bad or nonconforming data. This scenario is the result of a failure to properly plan and execute data intergration projects—a phenomenon known as code, load, and explode.

Michael Coles

Papers

Pro T-SQL Programmer's Guide

Full-Text Search

Events and Error Handling

Indexing Encrypted Data

Data Profiling and Scrubbing