Next Generation Sequencing (NGS) technology has resulted in massive amounts of proteomics and genomics data. This data is of no use if it is not properly analyzed. ETL (Extraction, Transformation, Loading) is an important step in designing data analytics applications. ETL requires proper understanding of features of data. Data format plays a key role in understanding of data, representation of data, space required to store data, data I/O during processing of data, intermediate results of processing, in-memory analysis of data and overall time required to process data. Different data mining and machine learning algorithms require input data in specific types and formats. This paper explores the data formats used by different tools and algorithms and also presents modern data formats that are used on Big Data Platform. It will help researchers and developers in choosing appropriate data format to be used for a particular tool or algorithm.

/pdf/modern-data-formats-for-big-bioinformatics-data-analytics-53gwd0kc6e.pdf

Modern Data Formats for Big Bioinformatics Data Analytics

Multi-modal Multimedia Big Data Analyzing Architecture and Resource Allocation on Cloud Platform

Distributed Computing has achieved tremendous development since cloud computing was proposed in 2006, and played a vital role promoting rapid growth of data collecting and analysis models, e.g., Internet of things, Cyber-Physical Systems, Big Data Analytics, etc. Hadoop has become a data convergence platform for sensor networks. As one of the core components, MapReduce facilitates allocating, processing and mining of collected large-scale data, where speculative execution strategies help solve straggler problems. However, there is still no efficient solution for accurate estimation on execution time of run-time tasks, which can affect task allocation and distribution in MapReduce. In this paper, task execution data have been collected and employed for the estimation. A two-phase regression (TPR) method is proposed to predict the finishing time of each task accurately. Detailed data of each task have drawn interests with detailed analysis report being made. According to the results, the prediction accuracy of concurrent tasks’ execution time can be improved, in particular for some regular jobs.

https://www.mdpi.com/1424-8220/16/9/1386/pdf

Estimation Accuracy on Execution Time of Run-Time Tasks in a Heterogeneous Distributed Environment

The objective of this paper is to improve the overall performance of distributed encoding, with a particular focus on encoding speed. The proposed scheme consists of content-aware video segmentation and scheduling that consists of two major parts. In the first part, segmentation is carried out with greater efficiency by considering changes in the video content, and in the second part, the segment assignment process is carried out using an efficient scheduling scheme that changes the encoding order of the segments. We measured the content similarity by using the sum of absolute difference algorithm and then applied a threshold to define the degree of change in similarity. The video was segmented based on the extent to which the similarity had changed, and the encoding order of the segments was rearranged to perform distributed encoding. Finally, this paper introduces the MapReduce-based distributed video encoding, using the content-aware video segmentation and scheduling described above, and presents the results of the performance using this scheme, which indicate that the proposed scheme increases the bitrate by a maximum of 2.9% over existing segmentation schemes, and also increases the speed by a maximum of 15.3%.

MapReduce-Based Distributed Video Encoding Using Content-Aware Video Segmentation and Scheduling

Multimedia applications are gradually becoming an essential--and flourishing--part of our daily lives. The area of wireless sensor networks is not an exception to this trend, as multimedia data have attracted the attention of researchers. Their importance is due to the shift of focus from traditional scalar sensors to sensors equipped with multimedia devices, as well as to the next-generation wireless multimedia sensor networks (WMSNs). The highly sensitive nature of multimedia traffic makes data routing with security even more important in WMSNs. This paper discusses the challenges of secure routing in WMSNs, where most of the proposed works in the literature deal with routing and security as individual issues. A critical and comprehensive review of state-of-the-art routing and security approaches for WMSNs is presented, followed by the discussion of their limitations and features. It is hoped that this extensive review and discussion ultimately identifies the future research directions and the open challenges of secure routing approaches in WMSNs.

A review of secure routing approaches for current and next-generation wireless multimedia sensor networks

The next-generation wireless sensor network (WSN) has the capability of carrying hundreds of high-definition video streams, beside the feature of massive employment of energy-efficient nodes. However, several challenges are identified with respect to the video bearing, such as the different video formats, enormous size of “raw” video, and compatibility with heterogeneous terminal devices. The video transcoding system (VTS) is widely believed to address these challenges. This paper introduces a cloud-based, more specifically, Hadoop-based, video transcoding system to fulfill the vision of bearing hundreds of HD video streams in the next generation WSN, with a discussion on optimization of several significant parameters. This paper obtains three remarkable results: (1) there is an optimal value of the number of Mappers; (2) the optimal value is closely related to the file size; (3) the transcoding time depends principally on the duration of video files rather than their sizes.

Design and Experiment Analysis of a Hadoop-Based Video Transcoding System for Next-Generation Wireless Sensor Networks

Foreign Object Debris (FOD) on the runways or taxiways may damage airplanes and injure personnel in the airport. There are two main types of commercial FOD detection systems, i.e. Fixed and mobile. Nowadays, only one of these two main types of FOD detection systems is deployed in some airports. In this paper, we propose a total solution to support both of FOD detection system types based on Shanghai Hongqiao International Airport FOD prevention's requirements, i.e. Mobile Ad-Hoc Network (MANET) based FOD detection system. In this paper, a novel FOD detection system architecture is proposed. Then we study the feasibility of MANET applied to the proposed FOD detection system. In this proposed MANET, Wi-Fi and IEEE 802.11 are introduced to physical and MAC layer of wireless node, OLSR is used to provide network routing function, in addition, a novel multi-path data fusion mechanism is proposed to decrease the packet loss ratio.

A Study on the Feasibility of Mobile Ad-Hoc Networks Applied to Foreign Objective Debris Detection System

The microgrid system is an energy network with renewable energy as the core, multi-energy network coupling, and multi-interest subject synergy. The balance of constraints between multiple agents is the key to realize the optimal operation of the system. Driven by different interest goals, there is an urgent need to establish a microgrid multi-corporate interaction mechanism and corresponding optimization methods. Therefore, this paper firstly constructs a multi- agent game interaction mechanism including microgrid operators, energy storage operators and users, and establishes a multi- agent game optimization model with one master and many subordinates by fully considering the economic interest game among the three and the uncertainty of both sides of the source and load. Then, this paper adopts a distributed equilibrium solution method of particle swarm algorithm combined with CPLEX solver to solve the model, and illustrates with practical examples that the multi-agent game model established in this paper can maximize the interests of each agent, and verifies the feasibility and applicability of the proposed method.

Multi- agent Collaborative Optimal Operation Strategy of Microgrid Based on Master-Slave Game

In order to improve the power output stability and frequency stability when large-scale new energy is integrated into the grid, large-scale new energy base must consider the configuration of energy storage systems with a certain capacity. Facing the demand under the background of new energy development, this paper analyzes the positive impact of energy storage to new energy base. The configuration method of energy storage capacity is proposed, and furthermore, the proposed method is used to calculate the capacity of the energy storage system required to be configured in a typical new energy base, and the power suppression and stability improvement effects brought by the energy storage are evaluated. The case analysis results show that the required energy storage capacity of a new energy base is about 10% of its total wind power and photovoltaic capacity. This configuration ratio can basically achieve the stability of the new energy output fluctuation, and at the same time provides the same frequency regulation capability with conventional generator set of equal capacity.

Liangyou Wang

Papers

Design and Experiment Analysis of a Hadoop-Based Video Transcoding System for Next-Generation Wireless Sensor Networks

A Study on the Feasibility of Mobile Ad-Hoc Networks Applied to Foreign Objective Debris Detection System

Multi- agent Collaborative Optimal Operation Strategy of Microgrid Based on Master-Slave Game

Research on Energy Storage Capacity Configuration Method and Performance Evaluation in New Energy Base