The traditional databases are not capable of handling unstructured data and high volumes of real-time datasets Diverse datasets are unstructured lead to big data, and it is laborious to store, manage, process, analyze, visualize, and extract the useful insights from these datasets using traditional database approaches However, many technical aspects exist in refining large heterogeneous datasets in the trend of big data This paper aims to present a generalized view of complete big data system which includes several stages and key components of each stage in processing the big data In particular, we compare and contrast various distributed file systems and MapReduce-supported NoSQL databases concerning certain parameters in data management process Further, we present distinct distributed/cloud-based machine learning (ML) tools that play a key role to design, develop and deploy data models The paper investigates case studies on distributed ML tools such as Mahout, Spark MLlib, and FlinkML Further, we classify analytics based on the type of data, domain, and application We distinguish various visualization tools pertaining three parameters: functionality, analysis capabilities, and supported development environment Furthermore, we systematically investigate big data tools and technologies (Hadoop 30, Spark 23) including distributed/cloud-based stream processing tools in a comparative approach Moreover, we discuss functionalities of several SQL Query tools on Hadoop based on 10 parameters Finally, We present some critical points relevant to research directions and opportunities according to the current trend of big data Investigating infrastructure tools for big data with recent developments provides a better understanding that how different tools and technologies apply to solve real-life applications

The big data system, components, tools, and technologies: a survey

Cipherbase is a comprehensive database system that provides strong end-to-end data confidentiality through encryption. Cipherbase is based on a novel architecture that combines an industrial strength database engine (SQL Server) with lightweight processing over encrypted data that is performed in secure hardware. The overall architecture provides significant benefits over the state-of-the-art in terms of security, performance, and functionality. This paper presents a prototype of Cipherbase that uses FPGAs to provide secure processing and describes the system engineering details implemented to achieve competitive performance for transactional workloads. This includes hardware-software co-design issues (e.g. how to best offer parallelism), optimizations to hide the latency between the secure hardware and the main system, and techniques to cope with space inefficiencies. All these optimizations were carefully designed not to affect end-to-end data confidentiality. Our experiments with the TPC-C benchmark show that in the worst case when all data are strongly encrypted, Cipherbase achieves 40% of the throughput of plaintext SQL Server. In more realistic cases, if only critical data such as customer names are encrypted, the Cipherbase throughput is more than 90% of plaintext SQL Server.

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/cipherbase.pdf

Transaction processing on confidential data using cipherbase

Data confidentiality is one of the main concerns for users of public cloud services. The key problem is protecting sensitive data from being accessed by cloud administrators who have root privileges and can remotely inspect the memory and disk contents of the cloud servers. While encryption is the basic mechanism that can leveraged to provide data confidentiality, providing an efficient database-as-a-service that can run on encrypted data raises several interesting challenges. In this demonstration we outline the functionality of Cipherbase --- a full fledged SQL database system that supports the full generality of a database system while providing high data confidentiality. Cipherbase has a novel architecture that tightly integrates custom-designed trusted hardware for performing operations on encrypted data securely such that an administrator cannot get access to any plaintext corresponding to sensitive data.

Secure database-as-a-service with Cipherbase

Internet of Things (IoT) devices have grown up to comprise embedded systems and sensors with the ability to connect, collect, and transmit data over the Internet. Although, solutions to secure IoT systems exist, Class-0 IoT devices with insufficient resources to support such solutions are considered too resource constrained for a secure communication. This paper provides a distributed security mechanism that targets Class-0 IoT devices. The research goal is to secure the entire data path in two segments; device-to-gateway and gateway-to-server data communications. The main concern in the provided solution is that lighter security operations with minimal resource requirements are performed in the device, while heavier tasks are performed in the gateway side. The proposed mechanism utilizes a symmetric encryption for data objects combined with the native wireless security to offer a layered security mechanism between the device and the gateway. In the offered solution, the IoT gateways provide additional protection by securing data using Transport Layer Security (TLS). The real-time experimental evaluations have proven the applicability of the proposed mechanism pertaining to the security assurance and the consumed resources of the target IoT devices.

/pdf/a-distributed-security-mechanism-for-resource-constrained-4c3b5ycb9q.pdf

A Distributed Security Mechanism for Resource-Constrained IoT Devices

Data security is a serious concern when we migrate data to a cloud DBMS. Database encryption, where sensitive columns are encrypted before they are stored in the cloud, has been proposed as a mechanism to address such data security concerns. The intuitive expectation is that an adversary cannot "learn" anything about the encrypted columns, since she does not have access to the encryption key. However, query processing becomes a challenge since it needs to "look inside" the data. This tutorial explores the space of designs studied in prior work on processing queries over encrypted data. We cover approaches based on both classic client-server and involving the use of a trusted hardware module where data can be securely decrypted. We discuss the privacy challenges that arise in both approaches and how they may be addressed. Briefly, supporting the full complexity of a modern DBMS including complex queries, transactions and stored procedures leads to significant challenges that we survey and open problems which we highlight.

/pdf/querying-encrypted-data-1iygqgj9b6.pdf

Querying encrypted data

Until this point, I’ve discussed the wide assortment of cryptographic functionality built in to SQL Server. The tools I’ve talked about include symmetric and asymmetric encryption functions, encryption key management, hashing, EKM, and TDE. These tools provide an impressive level of cryptographic functionality that can help make SQL Server databases more secure than ever.

SQL CLR Cryptography

Ac couple of key advantages provided by enterprise systems, which are often overlooked until the auditors start poking around, are process logging, auditing, and data lineage. The conceptsare simple: when you move data from system to system and manipulate it along the way, you might need to keep track of where it came from, maintain summary information about data processing, and know which processes touched that data along the way. There are a few reasons for keeping track of this additional information: 


You may need to keep it for troubleshooting purposes, to trace data issues backward through your processes to their source.


The legal department may have mandated a new legal requirement to document every step of processing in real-time in the event of issues.


Your business may fall under a regulation such as Sarbanes-Oxley Act, SOX, or Health Insurance Portability and Accountability Act, HIPAA, that requires high levels of security and controls around confidential medical or consumer data.

Logging and Auditing

In very high security situations laws, regulations and IT policies impose very strict prohibitions againststoring encryption keys in the same database (or even on the same physical device) as the datawhich it protects. To address this need, SQL Server 2008 provides an option known as extensible keymanagement (EKM). EKM allows you to create, store, manage, and access encryption keys on dedicated encryption appliances separate from your SQL Server databases. In this chapter, I will use the Luna SA hardware security module (HSM) provided by Safe Net to demonstrate EKM, although the concepts and tools apply to all SQL Server-enabled HSMs. SQL Server 2008 introduces new T-SQL statements, and new options for existing statements, designed to take advantage of EKM functionality. I’ll discuss these T-SQL language extensions in this chapter.

Extensible Key Management

As we have seen, encrypting sensitive data in SQL Server 2008 can be done at many different levels. Data can be encrypted on disk via an encrypted backup file. Data can also be encrypted where it is stored in a table or index via Transparent Data Encryption (TDE) that will be accessed via a stored procedure. You can even encrypt the stored procedure itself. These encryption techniques work primarily with data at rest. It goes without saying that this “resting” data will be accessed and the methods described thus far for encrypting and decrypting this data afford you a layer of protection at the instance level and on disk. It will not be easy for hackers to restore or otherwise interrogate an encrypted backup file, for example.

Michael Coles

Papers

SQL CLR Cryptography

Logging and Auditing

Extensible Key Management

Encrypting Connections to SQL Server 2008