Edge computing has emerged as a trend to improve scalability, overhead, and privacy by processing large-scale data, e.g., in deep learning applications locally at the source. In IoT networks, edge devices are characterized by tight resource constraints and often dynamic nature of data sources, where existing approaches for deploying Deep/Convolutional Neural Networks (DNNs/CNNs) can only meet IoT constraints when severely reducing accuracy or using a static distribution that cannot adapt to dynamic IoT environments. In this paper, we propose DeepThings, a framework for adaptively distributed execution of CNN-based inference applications on tightly resource-constrained IoT edge clusters. DeepThings employs a scalable Fused Tile Partitioning (FTP) of convolutional layers to minimize memory footprint while exposing parallelism. It further realizes a distributed work stealing approach to enable dynamic workload distribution and balancing at inference runtime. Finally, we employ a novel work scheduling process to improve data reuse and reduce overall execution latency. Results show that our proposed FTP method can reduce memory footprint by more than 68% without sacrificing accuracy. Furthermore, compared to existing work sharing methods, our distributed work stealing and work scheduling improve throughput by $1.7\times -2.2\times$ with multiple dynamic data sources. When combined, DeepThings provides scalable CNN inference speedups of $1.7\times$ – $3.5\times$ on 2–6 edge devices with less than 23 MB memory each.

DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters

Emergence of Internet-of-Things brings a whole new class of applications and higher efficiency for existing services. Application-specific requirements, as well as connectivity and communication ability of devices have introduced new challenges for IoT applications. This paper provides an overview of IoT technologies required from an embedded design perspective and specific properties associated with IoT in embedded systems' landscape. We investigate essential technologies for development of IoT systems, existing trends, and its distinguishing properties. By discussing the key characteristics, main application domains, and major research issues in IoT, this paper provides a comprehensive IoT perspective for embedded system design.

IoT technologies for embedded computing: a survey

Background modeling methods in video analysis: A review and comparative evaluation

Radio Frequency IDentification (RFID) has attracted considerable attention in recent years for its low cost, general availability, and location sensing functionality. Most existing schemes require the tracked persons to be labeled with RFID tags. This requirement may not be satisfied for some activity sensing applications due to privacy and security concerns and uncertainty of objects to be monitored, e.g., group behavior monitoring in warehouses with privacy limitations, and abnormal customers in banks. In this paper, we propose TASA-Tag-free Activity Sensing using RFID tag Arrays for location sensing and frequent route detection. TASA relaxes the monitored objects from attaching RFID tags, online recovers and checks frequent trajectories by capturing the Received Signal Strength Indicator (RSSI) series for passive RFID tag arrays where objects traverse. In order to improve the accuracy for estimated trajectories and accelerate location sensing, TASA introduces reference tags with known positions. With the readings from reference tags, TASA can locate objects more accurately. Extensive experiment shows that TASA is an effective approach for certain activity sensing applications.

https://www.cs.sjtu.edu.cn/~guo-my/PDF/Journals/J04.pdf

TASA: Tag-Free Activity Sensing Using RFID Tag Arrays

Battery lifetime is critical for wireless video sensors. To enable battery-powered wireless video sensors, low-power design is required. In this paper, we consider applying multi-view summarization to wireless video sensors to remove redundant contents such that the compression and transmission power can be reduced. A low-complexity online multi-view video summarization scheme is proposed. Experiments show that the proposed summarization method successfully reduces the video content while keeping important events. A power analysis of the system also shows that a significant amount of energy can be saved.

On-Line Multi-View Video Summarization for Wireless Video Sensor Network

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multibackground model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multibackground model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

Video Object Segmentation and Tracking Framework With Improved Threshold Decision and Diffusion Distance

In the next-generation visual surveillance systems, content analysis tools will be integrated. In this paper, to accelerate these tools, it is proposed to integrate a hardware content analysis engine into a smart camera system-on-a-chip (SoC). A smart camera SoC hardware architecture with the proposed visual content analysis engine is first presented. This engine consists of dedicated accelerators and a programmable morphology coprocessor. Stream processing design concept, frame-level pipelining, and subword level parallelism are employed together to efficiently utilize the bandwidth of the system bus and achieve high throughput. The implementation results show that, with 168 K logic gates and 40.63 Kb on-chip memory, a processing speed of 30 640 x 480 frames/s can be achieved, while the operations of video object segmentation, object description and tracking, and face detection and scoring are supported.

Efficient Content Analysis Engine for Visual Surveillance Network

There are four major components in application systems with internet-of-things (IoT): sensors, communications, computation and service, where large amount of data are acquired for ultra-big data analysis to discover the context information and knowledge behind signals. To support such large-scale data size and computation tasks, it is not feasible to employ centralized solutions on cloud servers. Thanks for the advances of silicon technology, the cost of computation become lower, and it is possible to distribute computation on every node in IoT. In this paper, we take video sensing network as an example to show the idea of distributed computing in IoT. Existing related works are reviewed and the architecture of a system-on-a-chip solution for distributed smart cameras is proposed with coarse-grained reconfigurable image stream processing architecture. It can accelerate various computer vision algorithms for distributed smart cameras in IoT.

Distributed computing in IoT: System-on-a-chip for smart cameras as an example

Background subtraction video segmentation is the important first step for video surveillance applications with fixed camera. There are many existing methods in the literature. However, most of them are either too simple to handle complex environment, such as dynamic background, or too complex to be executed in real-time. Based on the proposed multi-background registration technique, this paper presents a real-time video object segmentation algorithm. The proposed algorithm can better handle non-static background cases compared with original single background registration segmentation. Compared with other works that can handle dynamic background cases, it is efficient in memory usage.

Real-Time Memory-Efficient Video Object Segmentation in Dynamic Background with Multi-Background Registration Technique

Segmentation, tracking, and description extraction are important operations in smart camera surveillance systems. In this paper, a robust segmentation-and-descriptor based tracking algorithm is proposed. Segmentation is applied first, and description for each connected component is extracted for object classification to generate the video object masks. It can do segmentation, tracking, and description extraction with a single algorithm without redundant computation. In addition, a new descriptor for human objects, Human Color Structure Descriptor (HCSD), is also proposed for this algorithm. Experimental results show that the proposed algorithm can provide precise video object masks and trajectories. It is also shown that the proposed descriptor, HCSD, can achieve better performance than Scalable Color Descriptor and Color Structure Descriptor of MPEG-7 for human objects.

/pdf/human-object-tracking-algorithm-with-human-color-structure-59s4m575x5.pdf

Wei-Kai Chan

Papers

Video Object Segmentation and Tracking Framework With Improved Threshold Decision and Diffusion Distance

Efficient Content Analysis Engine for Visual Surveillance Network

Distributed computing in IoT: System-on-a-chip for smart cameras as an example

Real-Time Memory-Efficient Video Object Segmentation in Dynamic Background with Multi-Background Registration Technique

Human Object Tracking Algorithm with Human Color Structure Descriptor for Video Surveillance Systems