Computer clusters with the shared-nothing architecture are the major computing platforms for big data processing and analysis. In cluster computing, data partitioning and sampling are two fundamental strategies to speed up the computation of big data and increase scalability. In this paper, we present a comprehensive survey of the methods and techniques of data partitioning and sampling with respect to big data processing and analysis. We start with an overview of the mainstream big data frameworks on Hadoop clusters. The basic methods of data partitioning are then discussed including three classical horizontal partitioning schemes: range, hash, and random partitioning. Data partitioning on Hadoop clusters is also discussed with a summary of new strategies for big data partitioning, including the new Random Sample Partition (RSP) distributed model. The classical methods of data sampling are then investigated, including simple random sampling, stratified sampling, and reservoir sampling. Two common methods of big data sampling on computing clusters are also discussed: record-level sampling and block-level sampling. Record-level sampling is not as efficient as block-level sampling on big distributed data. On the other hand, block-level sampling on data blocks generated with the classical data partitioning methods does not necessarily produce good representative samples for approximate computing of big data. In this survey, we also summarize the prevailing strategies and related work on sampling-based approximation on Hadoop clusters. We believe that data partitioning and sampling should be considered together to build approximate cluster computing frameworks that are reliable in both the computational and statistical respects.

/pdf/a-survey-of-data-partitioning-and-sampling-methods-to-5anrhdwver.pdf

A survey of data partitioning and sampling methods to support big data analysis

Internet of Things (IoT)-generated data are characterized by its continuous generation, large amount, and unstructured format. The existing relational database technologies are inadequate to handle such IoT-generated data due to the limited processing speed and the significant storage-expansion cost. Thus, big data processing technologies, which are normally based on distributed file systems, distributed database management, and parallel processing technologies, have arisen as a core technology to implement IoT-generated data repositories. In this paper, we propose a sensor-integrated radio frequency identification (RFID) data repository-implementation model using MongoDB, the most popular big data-savvy document-oriented database system now. First, we devise a data repository schema that can effectively integrate and store the heterogeneous IoT data sources, such as RFID, sensor, and GPS, by extending the event data types in electronic product code information services standard, a de facto standard for the information exchange services for RFID-based traceability. Second, we propose an effective shard key to maximize query speed and uniform data distribution over data servers. Last, through a series of experiments measuring query speed and the level of data distribution, we show that the proposed design strategy, which is based on horizontal data partitioning and a compound shard key, is effective and efficient for the IoT-generated RFID/sensor big data.

MongoDB-Based Repository Design for IoT-Generated RFID/Sensor Big Data

For over forty years, relational databases have been the leading model for data storage, retrieval and management. However, due to increasing needs for scalability and performance, alternative systems have emerged, namely NoSQL technology. The rising interest in NoSQL technology, as well as the growth in the number of use case scenarios, over the last few years resulted in an increasing number of evaluations and comparisons among competing NoSQL technologies. While most research work mostly focuses on performance evaluation using standard benchmarks, it is important to notice that the architecture of real world systems is not only driven by performance requirements, but has to comprehensively include many other quality attribute requirements. Software quality attributes form the basis from which software engineers and architects develop software and make design decisions. Yet, there has been no quality attribute focused survey or classification of NoSQL databases where databases are compared with regards to their suitability for quality attributes common on the design of enterprise systems. To fill this gap, and aid software engineers and architects, in this article, we survey and create a concise and up-to-date comparison of NoSQL engines, identifying their most beneficial use case scenarios from the software engineer point of view and the quality attributes that each of them is most suited to.

/pdf/choosing-the-right-nosql-database-for-the-job-a-quality-5c09lm08lp.pdf

Choosing the right NoSQL database for the job: a quality attribute evaluation

A scalable distributed machine learning approach for attack detection in edge computing environments

In this article, a comprehensive overview of the Crow Search Algorithm (CSA) is introduced with detailed discussions, which is intended to keep researchers interested in swarm intelligence algorithms and optimization problems. CSA is a new swarm intelligence algorithm recently developed, which simulates crow behavior in storing excess food and retrieving it when needed. In the optimization theory, the crow is the searcher, the surrounding environment is the search space, and randomly storing the location of food is a feasible solution. Among all food locations, the location where the most food is stored is considered to be the global optimal solution, and the objective function is the amount of food. By simulating the intelligent behavior of crows, CSA tries to find optimal solutions to various optimization problems. It has gained a considerable interest worldwide since its advantages like simple implementation, a few numbers of parameters, flexibility, etc. This survey introduces a comprehensive variant of CSA, including hybrid, modified, and multi-objective versions. Furthermore, based on the analyzed papers published in the literature by some publishers such as IEEE, Elsevier, and Springer, the comprehensive application scenarios of CSA such as power, computer science, machine learning, civil engineering have also been reviewed. Finally, the advantages and disadvantages of CSA have been discussed by conducting some comparative experiments with other similar published peers.

/pdf/crow-search-algorithm-theory-recent-advances-and-20yq33540d.pdf

Crow Search Algorithm: Theory, Recent Advances, and Applications

Various organizations are dependent on structured database like Oracle, MySQL etc. but these database systems do not harness the requirements of all organizations like scalability, availability etc. When data and number of requests increases, structure database cannot handle huge data and requests efficiently. One of the solution to overcome these issues is to shift datacenters on NoSQL unstructured databases. In this paper we explain few NoSQL unstructured databases and present performance analysis of MongoDB. We compare the time required to insertion in different databases as well as searching with different number of threads in database with different number of entries. This work also studies the importance of the Sharding and Configuration of the cluster for MongoDB.

Performance evaluation of unstructured NoSQL data over distributed framework

Cloud computing paradigm is being used because of its low up-front cost. In recent years, even mobile phone users store their data at Cloud. Customer information stored at Cloud needs to be protected against potential intruders as well as cloud service provider. There is threat to the data in transit and data at cloud due to different possible attacks. Organizations are transferring important information to the Cloud that increases concern over security of data. Cryptography is common approach to protect the sensitive information in Cloud. Cryptography involves managing encryption and decryption keys. In this paper, we compare key management methods, apply key management methods to various cloud environments and analyze symmetric key cryptography algorithms.

Key Management for Cloud Data Storage: Methods and Comparisons

Cloud Computing is a successful paradigm for deploying scalable and highly available web applications at low cost. In real life scenarios, the applications are expected to be scalable and consistent. Data partitioning is a commonly used technique for improving scalability. Traditional horizontal partitioning techniques are not capable of tracking the data access patterns of web applications. The development of novel, scalable workload-driven data partitioning is a requirement for improving scalability. This paper proposes a novel workload-aware approach, with scalable workload-driven data partitioning based on data access patterns of web applications for transaction processing. It is specially designed to scale out using NoSQL data stores. In contrast to the existing static approaches, this approach offers high throughput, lower response time, and a less number of distributed transactions. Further, implementation and validation of scalable workload-driven partitioning scheme is carried out through experimentation over cloud data stores such as Hadoop HBase and Amazon SimpleDB. An experimental results of the concerned partitioning scheme is conducted using the industry standard TPC-C benchmark. Analytical and experimental results are observed and it shows that scalable workload-driven data partitioning outperforms the schema level and graph partitioning in terms of throughput, response time and distributed transactions.

/pdf/scalable-transactions-in-cloud-data-stores-4x5pojpfam.pdf

Scalable transactions in cloud data stores

Nowadays, virtual machine migration (VMM) is a trending research since it helps in balancing the load of the Cloud effectively. Several VMM-based strategies defined in the literature have considere...

Energy-aware VM migration using dragonfly–crow optimization and support vector regression model in Cloud

Virtual machines use are increasing day by day as people are using many smart gadgets which are highly computing and for operating them smoothly, required virtual machines. Virtualization is the basic technique of creating various resources from the available physical infrastructure. It is the backbone of cloud computing technology. This paper try to deal with studying various techniques, models, algorithms, for efficient green cloud computing by using virtualization techniques. It mainly involves virtual machines(VMs) consolidation. Power utilization can be reduced by simply deactivating and reactivating physical machines to meet the current workload. Power awareness in distributed system can be defined as identifying the parameters which minimize the energy consumption along with better QOS, SLA. The various strategies discussed here are mainly focusing on saving power and making data centers more efficient by considering server and network as its scope, however in future due to requirement of high bandwidth and network connectivity of data centers, power requirement of data center will go beyond our imagination. To handle the above mentioned concern require actual study of how power consumption happens in data centers, most of the part of power are consumed by servers, CPU and switches i.e. network devices. There still lots of research need to be done to design such modern algorithm and techniques which achieve better energy efficiency along with consideration of QOS, SLA, and VM consolidation by defining VM topologies which are fault tolerant and also helps nature, mankind by minimizing Co 2 emission.

Rajesh Ingle

Papers

Performance evaluation of unstructured NoSQL data over distributed framework

Key Management for Cloud Data Storage: Methods and Comparisons

Scalable transactions in cloud data stores

Energy-aware VM migration using dragonfly–crow optimization and support vector regression model in Cloud

Challenges in green computing for energy saving techniques