This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach provides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

3D High-Efficiency Video Coding for Multi-View Video and Depth Data

Since the popularisation of the Internet in the 1990s, the cyberspace has kept evolving. We have created various computer-mediated virtual environments including social networks, video conferencing, virtual 3D worlds (e.g., VR Chat), augmented reality applications (e.g., Pokemon Go), and Non-Fungible Token Games (e.g., Upland). Such virtual environments, albeit non-perpetual and unconnected, have bought us various degrees of digital transformation. The term `metaverse' has been coined to further facilitate the digital transformation in every aspect of our physical lives. At the core of the metaverse stands the vision of an immersive Internet as a gigantic, unified, persistent, and shared realm. While the metaverse may seem futuristic, catalysed by emerging technologies such as Extended Reality, 5G, and Artificial Intelligence, the digital `big bang' of our cyberspace is not far away. This survey paper presents the first effort to offer a comprehensive framework that examines the latest metaverse development under the dimensions of state-of-the-art technologies and metaverse ecosystems, and illustrates the possibility of the digital `big bang'. First, technologies are the enablers that drive the transition from the current Internet to the metaverse. We thus examine eight enabling technologies rigorously - Extended Reality, User Interactivity (Human-Computer Interaction), Artificial Intelligence, Blockchain, Computer Vision, IoT and Robotics, Edge and Cloud computing, and Future Mobile Networks. In terms of applications, the metaverse ecosystem allows human users to live and play within a self-sustaining, persistent, and shared realm. Therefore, we discuss six user-centric factors -- Avatar, Content Creation, Virtual Economy, Social Acceptability, Security and Privacy, and Trust and Accountability. Finally, we propose a concrete research agenda for the development of the metaverse.

All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda

Driven by the emergence of new compute-intensive applications and the vision of the Internet of Things (IoT), it is foreseen that the emerging 5G network will face an unprecedented increase in traffic volume and computation demands. However, end users mostly have limited storage capacities and finite processing capabilities, thus how to run compute-intensive applications on resource-constrained users has recently become a natural concern. Mobile edge computing (MEC), a key technology in the emerging fifth generation (5G) network, can optimize mobile resources by hosting compute-intensive applications, process large data before sending to the cloud, provide the cloud computing capabilities within the radio access network (RAN) in close proximity to mobile users, and offer context-aware services with the help of RAN information. Therefore, MEC enables a wide variety of applications, where the real-time response is strictly required, e.g., driverless vehicles, augmented reality, robotics, and immerse media. Indeed, the paradigm shift from 4G to 5G could become a reality with the advent of new technological concepts. The successful realization of MEC in the 5G network is still in its infancy and demands for constant efforts from both academic and industry communities. In this survey, we first provide a holistic overview of MEC technology and its potential use cases and applications. Then, we outline up-to-date researches on the integration of MEC with the new technologies that will be deployed in 5G and beyond. We also summarize testbeds and experimental evaluations, and open source activities, for edge computing. We further summarize lessons learned from state-of-the-art research works as well as discuss challenges and potential future directions for MEC research.

A Survey of Multi-Access Edge Computing in 5G and Beyond: Fundamentals, Technology Integration, and State-of-the-Art

Rate control is a useful tool for video coding, especially in real-time communication applications. Most of existing rate control algorithms are based on the R-Q model, which characterizes the relationship between bitrate R and quantization Q , under the assumption that Q is the critical factor on rate control. However, with the video coding schemes becoming more and more flexible, it is very difficult to accurately model the R-Q relationship. In fact, we find that there exists a more robust correspondence between R and the Lagrange multiplier λ . Therefore, in this paper, we propose a novel λ -domain rate control algorithm based on the R-λ model, and implement it in the newest video coding standard high efficiency video coding (HEVC). Experimental results show that the proposed λ -domain rate control can achieve the target bitrates more accurately than the original rate control algorithm in the HEVC reference software as well as obtain significant R-D performance gain. Thanks to the high accurate rate control algorithm, hierarchical bit allocation can be enabled in the implemented video coding scheme, which can bring additional R-D performance gain. Experimental results demonstrate that the proposed λ -domain rate control algorithm is effective for HEVC, which outperforms the R-Q model based rate control in HM-8.0 (HEVC reference software) by 0.55 dB on average and up to 1.81 dB for low delay coding structure, and 1.08 dB on average and up to 3.77 dB for random access coding structure. The proposed λ -domain rate control algorithm has already been adopted by Joint Collaborative Team on Video Coding and integrated into the HEVC reference software.

λ domain rate control algorithm for high efficiency video coding.

Wireless Sensor Networks (WSNs) are crucial in supporting continuous environmental monitoring, where sensor nodes are deployed and must remain operational to collect and transfer data from the environment to a base-station. However, sensor nodes have limited energy in their primary power storage unit, and this energy may be quickly drained if the sensor node remains operational over long periods of time. Therefore, the idea of harvesting ambient energy from the immediate surroundings of the deployed sensors, to recharge the batteries and to directly power the sensor nodes, has recently been proposed. The deployment of energy harvesting in environmental field systems eliminates the dependency of sensor nodes on battery power, drastically reducing the maintenance costs required to replace batteries. In this article, we review the state-of-the-art in energy-harvesting WSNs for environmental monitoring applications, including Animal Tracking, Air Quality Monitoring, Water Quality Monitoring, and Disaster Monitoring to improve the ecosystem and human life. In addition to presenting the technologies for harvesting energy from ambient sources and the protocols that can take advantage of the harvested energy, we present challenges that must be addressed to further advance energy-harvesting-based WSNs, along with some future work directions to address these challenges.

https://dl.acm.org/doi/pdf/10.1145/3183338

Energy-Harvesting Wireless Sensor Networks (EH-WSNs): A Review

Joint video/depth rate allocation is an important optimization problem in 3D video coding. To address this problem, this paper proposes a distortion model to evaluate the synthesized view without access to the captured original view. The proposed distortion model is an additive model that accounts for the video-coding-induced distortion and the depth-quantization-induced distortion, as well as the inherent geometry distortion. Depth-quantization-induced distortion not only considers the warping error distortion, which is described by a piecewise linear model with the video power spectral property, but also takes into account the warping error correlation distortion between two sources reference views. Geometry distortion is approximated from that of the adjacent view synthesis. Based on the proposed distortion model, a joint rate allocation method is proposed to seek the optimal trade-off between video bit-rate and depth bit-rate for maximizing the view synthesis quality. Experimental results show that the proposed distortion model is capable of approximately estimating the actual distortion for the synthesized view, and that the proposed rate allocation method can almost achieve the identical rate allocation performance as the full-search method at less computational cost. Moreover, the proposed rate allocation method consumes less computational cost than the hierarchical-search method at high bit-rates while providing almost the equivalent rate allocation performance.

http://www.jdl.ac.cn/doc/2009/Image%20Communication_Joint%20Video-Depth%20Rate%20Allocation%20for%203D%20Video%20Coding%20based%20on%20View%20Synthesis%20Distortion%20Model.pdf

Joint video/depth rate allocation for 3D video coding based on view synthesis distortion model

Panoramic virtual reality video (PVRV) is becoming increasingly popular since it offers a true immersive experience. However, the ultra-high resolution of PVRV requires significant bandwidth and ultra-low latency for PVRV streaming, something that makes challenging the extension of this application to mobile networks. Besides bandwidth, the frequent perspective viewport rendering induces a heavy computational load on battery-constrained mobile devices. To attack these problems jointly, this paper proposes a PVRV streaming system that is designed for modern multiconnectivity-based millimeter wave (mmWave) cellular networks in conjunction with mobile edge computing (MEC). First, mmWave is deployed to support the high bandwidth needs of PVRV streaming. Next, the multiple mmWave links that tend to suffer from outages are coupled with a sub-6 GHz link to ensure disruption-free wireless communication. With the help of an MEC server, the tradeoff among link adaptation, transcoding-based chunk quality adaptation, and viewport rendering offloading is sought to improve the wireless bandwidth utilization and mobile device's energy efficiency. Simulation results show that the proposed scheme can improve the streaming performance in both energy efficiency and the quality of received viewport over the state-of-the-art schemes.

MEC-Assisted Panoramic VR Video Streaming Over Millimeter Wave Mobile Networks

A wireless video surveillance system consists of three major components: 1) the video capture and preprocessing; 2) the video compression and transmission in wireless sensor networks; and 3) the video analysis at the receiving end. A myriad of research works have been dedicated to this field due to its increasing popularity in surveillance applications. This survey provides a comprehensive overview of existing state-of-the-art technologies developed for wireless video surveillance, based on the in-depth analysis of the requirements and challenges in current systems. Specifically, the physical network infrastructure for video transmission over wireless channel is analyzed. The representative technologies for video capture and preliminary vision tasks are summarized. For video compression and transmission over the wireless networks, the ultimate goal is to maximize the received video quality under the resource limitation. This is also the main focus of this survey. We classify different schemes into categories including unequal error protection, error resilience, scalable video coding, distributed video coding, and cross-layer control. Cross-layer control proves to be a desirable measure for system-level optimal resource allocation. At the receiver's end, the received video is further processed for higher-level vision tasks, and the security and privacy issues in surveillance applications are also discussed.

Wireless Video Surveillance: A Survey

This paper presents a novel rate control technique for multiview video plus depth (MVD) based 3D video coding. In the proposed rate control technique, an image-stitching method is first utilized to simultaneously encode video and depth, and then a joint rate control algorithm for MVD is presented. The joint rate control algorithm is performed on three levels, namely view level, video/depth level and frame level. In the view level, different proportions of rates are allocated for different types of views according to the pre-statistical rate allocation. In the video/depth level, the target rates for video and depth are discriminatorily assigned to guarantee the high quality of video for the backward-compatible display. In the frame level, the hierarchical rate allocation is used to regulate the target bits for each frame. In addition to the above mentioned rate control strategies, according to the special characteristics of multiview hypothetical reference decoder (HRD), the buffer-related rate control is also considered to prevent the decoder buffer from overflow or underflow even outputting multiple views. Experimental results show that the proposed rate control technique can accurately control the bit-rate to satisfy the requirements of 3D video systems.

A Novel Rate Control Technique for Multiview Video Plus Depth Based 3D Video Coding

In the recent years, energy harvesting technology has been integrated into wireless sensor networks for resolving the energy "bottleneck" problems caused by the limited-capacity batteries equipped in traditional sensor nodes. However, due to the limited capacity of energy buffers equipped for storing the harvested energy, excessive harvested energy is indiscreetly discarded while sensor nodes are staying at the status of energy saturation. To address this issue, we present a topology control approach, for the energy harvesting wireless sensor network, that allows each node to adaptively adjust its transmission power for utilizing the harvested energy effectively. Specifically, we first model the behaviors of sensor nodes as an ordinal potential game where the high harvesting power nodes cooperate with the low harvesting power nodes to maintain the connectivity of the whole network. And we theoretically prove the existence of a Nash equilibrium for the game. Then, a polynomial-time algorithm is proposed to achieve the Nash equilibrium. Simulation results show that our proposed scheme improves the energy availability and outperforms existing energy-aware schemes in terms of energy conservation and balanced distribution.

Yanwei Liu

Papers

Joint video/depth rate allocation for 3D video coding based on view synthesis distortion model

MEC-Assisted Panoramic VR Video Streaming Over Millimeter Wave Mobile Networks

Wireless Video Surveillance: A Survey

A Novel Rate Control Technique for Multiview Video Plus Depth Based 3D Video Coding

Energy harvesting aware topology control with power adaptation in wireless sensor networks