Today, organizations in every industry are being showered with imposing quantity of new information. As there are many more data channels and categories available along with traditional sources. Therefore the rate of data growth is increasing more and more which results in a very large volume of data. These vastly larger volumes and new assets are known as Big Data. Technologies such as MapReduce & Hadoop are used to extract value from Big Data. Hadoop is well adopted, standard-based, open source software framework build on the foundation of Google’s MapReduce. There are also new data storage techniques that have arisen to bolster these new architectures, including very large file system running on commodity hardware. This new data storage technology is HDFS. This file system is meant to support enormous amount of structured as well as unstructured data.

http://www.ijsrd.com/articles/IJSRDV3I2571.pdf

A Review Paper on Big Data and Hadoop

The amount of data processed and stored in the cloud is growing dramatically The traditional storage devices at both hardware and software levels cannot meet the requirement of the cloud This fact motivates the need for a platform which can handle this problem Hadoop is a deployed platform proposed to overcome this big data problem which often uses MapReduce architecture to process vast amounts of data of the cloud system Hadoop has no strategy to assure the safety and confidentiality of the files saved inside the Hadoop distributed File system(HDFS) In the cloud, the protection of sensitive data is a critical issue in which data encryption schemes plays avital rule This research proposes a hybrid system between two well-known asymmetric key cryptosystems (RSA, and Paillier) to encrypt the files stored in HDFS Thus before saving data in HDFS, the proposed cryptosystem is utilized for encrypting the data Each user of the cloud might upload files in two ways, non-safe or secure The hybrid system shows higher computational complexity and less latency in comparison to the RSA cryptosystem alone

/pdf/an-approach-for-enhancing-data-confidentiality-in-hadoop-2355mpxmnz.pdf

An approach for enhancing data confidentiality in Hadoop

Distributed denial of service (DDoS) is an immense threat for Internet based-applications and their resources. It immediately floods the victim system by transmitting a large number of network packets, and due to this, the victim system resources become unavailable for legitimate users. Therefore, this attack is claimed to be a dangerous attack for Internet-based applications and their resources. Several security approaches have been proposed in the literature to protect Internet-based applications from this type of threat. However, the frequency and strength of DDoS attacks are increasing day-by-day. Further, most of the traditional and distributed processing frameworks-based DDoS attack detection systems analyzed network flows in offline batch processing. Hence, they failed to classify network flows in real-time. This paper proposes a novel Spark Streaming and Kafka-based distributed classification system, named by SSK-DDoS, for classifying different types of DDoS attacks and legitimate network flows. This classification approach is implemented using a distributed Spark MLlib machine learning algorithms on a Hadoop cluster and deployed on the Spark streaming platform to classify streams in real-time. The incoming streams consume by Kafka's topic to perform preprocessing tasks such as extracting and formulating features for classifying them into seven groups: Benign, DDoS-DNS, DDoS-LDAP, DDoS-MSSQL, DDoS-NetBIOS, DDoS-UDP, and DDoS-SYN. Further, the SSK-DDoS classification system stores formulated features with their predicted class into the HDFS that will help to retrain the distributed classification approach using a new set of samples. The proposed SSK-DDoS classification system has been validated using the recent CICDDoS2019 dataset. The results show that the proposed SSK-DDoS efficiently classified network flows into seven classes and stored formulated features with the predicted value of each incoming network flow into HDFS. 

/pdf/ssk-ddos-distributed-stream-processing-framework-based-4t9edojm.pdf

SSK-DDoS: distributed stream processing framework based classification system for DDoS attacks

The amount of data processed and stored in the cloud is growing dramatically. The traditional storage devices at both hardware and software levels cannot meet the requirement of the cloud. This fact motivates the need for a platform which can handle this problem. Hadoop is a deployed platform proposed to overcome this big data problem which often uses MapReduce architecture to process vast amounts of data of the cloud system. Hadoop has no strategy to assure the safety and confidentiality of the files saved inside the Hadoop distributed File system (HDFS). In the cloud, the protection of sensitive data is a critical issue in which data encryption schemes plays avital rule. This research proposes a hybrid system between two well-known asymmetric key cryptosystems (RSA, and Paillier) to encrypt the files stored in HDFS. Thus before saving data in HDFS, the proposed cryptosystem is utilized for encrypting the data. Each user of the cloud might upload files in two ways, non-safe or secure. The hybrid system shows higher computational complexity and less latency in comparison to the RSA cryptosystem alone.

Enhancing approach using hybrid pailler and RSA for information security in bigdata

Distributed frameworks for detecting distributed denial of service attacks: A comprehensive review, challenges and future directions

Recently Cloud based Hadoop has gained a lot of interest that offer ready to use Hadoop cluster environment for processing of Big Data, eliminating the operational challenges of on-site hardware investment, IT support, and installing, configuring of Hadoop components such as HDFS and MapReduce. On demand Hadoop as a service helps the industries to focus on business growth and based on pay per use model for Big Data processing with auto-scaling of Hadoop cluster feature. In this paper implementation of various MapReduce jobs like Pi, TeraSort, WordCount has been done on cloud based Hadoop deployment by using Microsoft Azure cloud services. Performance of MapReduce jobs has been evaluated with respect to CPU execution time with varying size of Hadoop cluster. From the experimental result, it is found that CPU execution time to finish the jobs decrease as the number of Data Nodes in HDInsight cluster increases and indicates the good response time with increase in performance as well as more customer satisfaction.

Vineet Kumar Singh

Papers

Analyzing BigData with Hadoop cluster in HDInsight azure Cloud