On robust estimation of the location parameter

Deep learning is currently widely used in a variety of applications, including computer vision and natural language processing. End devices, such as smartphones and Internet-of-Things sensors, are generating data that need to be analyzed in real time using deep learning or used to train deep learning models. However, deep learning inference and training require substantial computation resources to run quickly. Edge computing, where a fine mesh of compute nodes are placed close to end devices, is a viable way to meet the high computation and low-latency requirements of deep learning on edge devices and also provides additional benefits in terms of privacy, bandwidth efficiency, and scalability. This paper aims to provide a comprehensive review of the current state of the art at the intersection of deep learning and edge computing. Specifically, it will provide an overview of applications where deep learning is used at the network edge, discuss various approaches for quickly executing deep learning inference across a combination of end devices, edge servers, and the cloud, and describe the methods for training deep learning models across multiple edge devices. It will also discuss open challenges in terms of systems performance, network technologies and management, benchmarks, and privacy. The reader will take away the following concepts from this paper: understanding scenarios where deep learning at the network edge can be useful, understanding common techniques for speeding up deep learning inference and performing distributed training on edge devices, and understanding recent trends and opportunities.

https://ieeexplore.ieee.org/ielaam/5/8789751/8763885-aam.pdf

Deep Learning With Edge Computing: A Review

Nowadays, 360° video/image has been increasingly popular and drawn great attention. The spherical viewing range of 360° video/image accounts for huge data, which pose the challenges to 360° video/image processing in solving the bottleneck of storage, transmission, etc. Accordingly, the recent years have witnessed the explosive emergence of works on 360° video/image processing. In this article, we review the state-of-the-art works on 360° video/image processing from the aspects of perception, assessment and compression. First, this article reviews both datasets and visual attention modelling approaches for 360° video/image. Second, we survey the related works on both subjective and objective visual quality assessment (VQA) of 360° video/image. Third, we overview the compression approaches for 360° video/image, which either utilize the spherical characteristics or visual attention models. Finally, we summarize this overview article and outlook the future research trends on 360° video/image processing.

State-of-the-Art in 360° Video/Image Processing: Perception, Assessment and Compression

Quadtree with nested multi-type tree (QTMT) partition structure is an efficient improvement in versatile video coding (VVC) over the quadtree (QT) structure in the advanced high-efficiency video coding (HEVC) standard. With the exception of the recursive QT partition structure, recursive multi-type tree partition is applied to each leaf node, which generates more flexible block sizes. Besides, intra prediction modes are extended from 35 to 67 so as to satisfy various texture patterns. These newly developed techniques achieve high coding efficiency but also result in very high computational complexity. To tackle this problem, we propose a fast intra-coding algorithm consisting of low-complexity coding tree units (CTU) structure decision and fast intra mode decision in this paper. The contributions of the proposed algorithm lie in the following aspects: 1) the new block size and coding mode distribution features are first explored for a reasonable fast coding scheme; 2) a novel fast QTMT partition decision framework is developed, which can determine the partition decision on both QT and multi-type tree with a novel cascade decision structure; and 3) fast intra mode decision with gradient descent search is introduced, while the best initial search point and search step are also investigated in this paper. The simulation results show that the complexity reduction of the proposed algorithm is up to 70% compared to VVC reference software (VTM), and averagely 63% encoding time saving is achieved with 1.93% BDBR increasing. Such results demonstrate that our method yields a superior performance in terms of computational complexity and compression quality compared to the state-of-the-art methods.

Low-Complexity CTU Partition Structure Decision and Fast Intra Mode Decision for Versatile Video Coding

The intensive computation of High Efficiency Video Coding (HEVC) engenders challenges for the hardwired encoder in terms of the hardware overhead and the power dissipation. On the other hand, the constrains in hardwired encoder design seriously degrade the efficiency of software oriented fast coding unit (CU) partition mode decision algorithms. A fast algorithm is attributed as VLSI friendly, when it possesses the following properties. First, the maximum complexity of encoding a coding tree unit (CTU) could be reduced. Second, the parallelism of the hardwired encoder should not be deteriorated. Third, the process engine of the fast algorithm must be of low hardware- and power-overhead. In this paper, we devise the convolution neural network based fast algorithm to decrease no less than two CU partition modes in each CTU for full rate-distortion optimization (RDO) processing, thereby reducing the encoder’s hardware complexity. As our algorithm does not depend on the correlations among CU depths or spatially nearby CUs, it is friendly to the parallel processing and does not deteriorate the rhythm of RDO pipelining. Experiments illustrated that, an averaged 61.1% intraencoding time was saved, whereas the Bjontegaard-Delta bit-rate augment is 2.67%. Capitalizing on the optimal arithmetic representation, we developed the high-speed [714 MHz in the worst conditions (125 °C, 0.9 V)] and low-cost (42.5k gate) accelerator for our fast algorithm by using TSMC 65-nm CMOS technology. One accelerator could support HD1080p at 55 frames/s real-time encoding. The corresponding power dissipation was 16.2 mW at 714 MHz. Finally, our accelerator is provided with good scalability. Four accelerators fulfill the throughput requirements of UltraHD-4K at 55 frames/s.

CU Partition Mode Decision for HEVC Hardwired Intra Encoder Using Convolution Neural Network

This paper explores gaze prediction in dynamic 360° immersive videos, i.e., based on the history scan path and VR contents, we predict where a viewer will look at an upcoming time. To tackle this problem, we first present the large-scale eye-tracking in dynamic VR scene dataset. Our dataset contains 208 360° videos captured in dynamic scenes, and each video is viewed by at least 31 subjects. Our analysis shows that gaze prediction depends on its history scan path and image contents. In terms of the image contents, those salient objects easily attract viewers' attention. On the one hand, the saliency is related to both appearance and motion of the objects. Considering that the saliency measured at different scales is different, we propose to compute saliency maps at different spatial scales: the sub-image patch centered at current gaze point, the sub-image corresponding to the Field of View (FoV), and the panorama image. Then we feed both the saliency maps and the corresponding images into a Convolutional Neural Network (CNN) for feature extraction. Meanwhile, we also use a Long-Short-Term-Memory (LSTM) to encode the history scan path. Then we combine the CNN features and LSTM features for gaze displacement prediction between gaze point at a current time and gaze point at an upcoming time. Extensive experiments validate the effectiveness of our method for gaze prediction in dynamic VR scenes.

/pdf/gaze-prediction-in-dynamic-360-immersive-videos-4rsesbfy8g.pdf

Gaze Prediction in Dynamic 360° Immersive Videos

The high efficiency video coding (HEVC) standard has highly improved the coding efficiency by adopting hierarchical structures of coding unit (CU), prediction unit (PU), and transform unit (TU). However, enormous computational complexity is introduced due to the recursive rate-distortion optimization (RDO) process on all CUs, PUs and TUs. In this paper, we propose a fast and efficient mode decision algorithm based on the Neyman-Pearson rule, which consists of early SKIP mode decision and fast CU size decision. First, the early mode decision is modeled as a binary classification problem of SKIP/non-SKIP or split/unsplit. The Neyman-Pearson-based rule is employed to balance the rate-distortion (RD) performance loss and the complexity reduction by minimizing the missed detection with a constrained incorrect decision rate. A nonparametric density estimation scheme is also developed to calculate the likelihood function of the statistical parameters. Furthermore, an online training scheme is employed to periodically update the probability density distributions for different quantization parameters (QPs) and CU depth levels. The experimental results show that the proposed overall algorithm can save 65% and 58% computational complexity on average with a 1.29% and 1.08% Bjontegaard Delta bitrate (BDBR) increase for various test sequences under random access and low delay P conditions, respectively. The proposed overall scheme also has the advantage that it can make the trade-off between the RD performance and time saving by setting different values for the incorrect decision rate.

Neyman-Pearson-Based Early Mode Decision for HEVC Encoding

The High Efficiency Video Coding (HEVC) adopts a hierarchical quad-tree based coding unit (CU) partitioning structure, which allows to recursively split a block into four equally sized blocks. At each depth level, there are up to 35 intra prediction modes. Therefore, enormous computational complexity is introduced due to the recursive Rate-Distortion Optimization (RDO) process on all possible depth levels and prediction modes to find the optimal CU partitions. In this paper, a fast mode decision algorithm is proposed for HEVC intra coding. A logistic regression classifier is introduced to early terminate CU splitting decision process, which is formulated as a binary classification problem. An offline training scheme is applied to obtain a set of efficient coefficients for the logistic regression classifier. Efficient and computationally-friendly features are extracted based on an F-score approach for different QPs and CU depth levels. The experimental results show the proposed algorithm can reduce computational complexity by 55.5% on average with 1.29% Bjontegaard Delta bitrate (BDBR) increase for various test sequences under “encoder_intra_main” condition compared with that of HEVC reference test model.

Fast HEVC intra mode decision based on logistic regression classification

The new video coding standard, H.264/AVC, has achieved significant compression gain by utilizing several advanced techniques. Block-Matching Motion Estimation is one of the most important elements to reduce the motion residual. However, it results in heavy computational burden and limits the application for real-time video service. In this paper, a fast motion estimation algorithm called Simulated Annealing Adaptive Search (SAAS) is proposed to reduce the computational load. The basic idea of the proposed scheme is based on adjusting search pattern not only for each frame, but also for each block. Initially, the adaptive search pattern is performed by statistical analysis of previous frame's Motion Vector Correlation. Then the search pattern is adjusted for each block according to Predicted Motion Vector. According to motion vector correlation statistics information, search region is adaptively divided and Simulated Annealing (SA) mechanism is adopted to select search power for each region and to avoid trapping into local minima. Experimental results indicate that the proposed algorithm offers considerable improvement in computing time and motion search points at the same rate-distortion performance, compared to the conventional fast motion estimation algorithm.

Adaptive direction search algorithms based on motion correlation for block motion estimation

Motion estimation is the most computationally intensive element in a block based video codec. In this paper, a novel Predictive Intensive Direction Searching (PIDS) algorithm is proposed to reduce the computational load of H.264/AVC video codec. Based on the direction of predicted motion vector, the search area is adaptively divided into one region of intensive-direction-searching and several regions of' coarse-direction-searching. Then the extended hexagon search is used to refine the final optimal motion vector. The experimental results indicate that the proposed algorithm achieves a reduction of 15% motion estimation time, while incurring only 0.5% increment on the bite rate compared with that of UMHexagonS algorithm, which is adopted by the H.264/AVC reference software.

Zhiru Shi

Papers

Gaze Prediction in Dynamic 360° Immersive Videos

Neyman-Pearson-Based Early Mode Decision for HEVC Encoding

Fast HEVC intra mode decision based on logistic regression classification

Adaptive direction search algorithms based on motion correlation for block motion estimation

A motion estimation algorithm based on Predictive Intensive Direction Search for H.264/AVC