Edge computing has emerged as a trend to improve scalability, overhead, and privacy by processing large-scale data, e.g., in deep learning applications locally at the source. In IoT networks, edge devices are characterized by tight resource constraints and often dynamic nature of data sources, where existing approaches for deploying Deep/Convolutional Neural Networks (DNNs/CNNs) can only meet IoT constraints when severely reducing accuracy or using a static distribution that cannot adapt to dynamic IoT environments. In this paper, we propose DeepThings, a framework for adaptively distributed execution of CNN-based inference applications on tightly resource-constrained IoT edge clusters. DeepThings employs a scalable Fused Tile Partitioning (FTP) of convolutional layers to minimize memory footprint while exposing parallelism. It further realizes a distributed work stealing approach to enable dynamic workload distribution and balancing at inference runtime. Finally, we employ a novel work scheduling process to improve data reuse and reduce overall execution latency. Results show that our proposed FTP method can reduce memory footprint by more than 68% without sacrificing accuracy. Furthermore, compared to existing work sharing methods, our distributed work stealing and work scheduling improve throughput by $1.7\times -2.2\times$ with multiple dynamic data sources. When combined, DeepThings provides scalable CNN inference speedups of $1.7\times$ – $3.5\times$ on 2–6 edge devices with less than 23 MB memory each.

DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters

Emergence of Internet-of-Things brings a whole new class of applications and higher efficiency for existing services. Application-specific requirements, as well as connectivity and communication ability of devices have introduced new challenges for IoT applications. This paper provides an overview of IoT technologies required from an embedded design perspective and specific properties associated with IoT in embedded systems' landscape. We investigate essential technologies for development of IoT systems, existing trends, and its distinguishing properties. By discussing the key characteristics, main application domains, and major research issues in IoT, this paper provides a comprehensive IoT perspective for embedded system design.

IoT technologies for embedded computing: a survey

Background modeling methods in video analysis: A review and comparative evaluation

Radio Frequency IDentification (RFID) has attracted considerable attention in recent years for its low cost, general availability, and location sensing functionality. Most existing schemes require the tracked persons to be labeled with RFID tags. This requirement may not be satisfied for some activity sensing applications due to privacy and security concerns and uncertainty of objects to be monitored, e.g., group behavior monitoring in warehouses with privacy limitations, and abnormal customers in banks. In this paper, we propose TASA-Tag-free Activity Sensing using RFID tag Arrays for location sensing and frequent route detection. TASA relaxes the monitored objects from attaching RFID tags, online recovers and checks frequent trajectories by capturing the Received Signal Strength Indicator (RSSI) series for passive RFID tag arrays where objects traverse. In order to improve the accuracy for estimated trajectories and accelerate location sensing, TASA introduces reference tags with known positions. With the readings from reference tags, TASA can locate objects more accurately. Extensive experiment shows that TASA is an effective approach for certain activity sensing applications.

https://www.cs.sjtu.edu.cn/~guo-my/PDF/Journals/J04.pdf

TASA: Tag-Free Activity Sensing Using RFID Tag Arrays

Battery lifetime is critical for wireless video sensors. To enable battery-powered wireless video sensors, low-power design is required. In this paper, we consider applying multi-view summarization to wireless video sensors to remove redundant contents such that the compression and transmission power can be reduced. A low-complexity online multi-view video summarization scheme is proposed. Experiments show that the proposed summarization method successfully reduces the video content while keeping important events. A power analysis of the system also shows that a significant amount of energy can be saved.

On-Line Multi-View Video Summarization for Wireless Video Sensor Network

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multibackground model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multibackground model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

Video Object Segmentation and Tracking Framework With Improved Threshold Decision and Diffusion Distance

In the next-generation visual surveillance systems, content analysis tools will be integrated. In this paper, to accelerate these tools, it is proposed to integrate a hardware content analysis engine into a smart camera system-on-a-chip (SoC). A smart camera SoC hardware architecture with the proposed visual content analysis engine is first presented. This engine consists of dedicated accelerators and a programmable morphology coprocessor. Stream processing design concept, frame-level pipelining, and subword level parallelism are employed together to efficiently utilize the bandwidth of the system bus and achieve high throughput. The implementation results show that, with 168 K logic gates and 40.63 Kb on-chip memory, a processing speed of 30 640 x 480 frames/s can be achieved, while the operations of video object segmentation, object description and tracking, and face detection and scoring are supported.

Efficient Content Analysis Engine for Visual Surveillance Network

There are four major components in application systems with internet-of-things (IoT): sensors, communications, computation and service, where large amount of data are acquired for ultra-big data analysis to discover the context information and knowledge behind signals. To support such large-scale data size and computation tasks, it is not feasible to employ centralized solutions on cloud servers. Thanks for the advances of silicon technology, the cost of computation become lower, and it is possible to distribute computation on every node in IoT. In this paper, we take video sensing network as an example to show the idea of distributed computing in IoT. Existing related works are reviewed and the architecture of a system-on-a-chip solution for distributed smart cameras is proposed with coarse-grained reconfigurable image stream processing architecture. It can accelerate various computer vision algorithms for distributed smart cameras in IoT.

Distributed computing in IoT: System-on-a-chip for smart cameras as an example

A 5.877 TOPS/W Reconfigurable Smart-camera Stream Processor is implemented in 90nm CMOS technology. A reconfigurable hardware architecture with heterogeneous stream processing and subword-level parallelism is implemented to accelerate the vision processing for smart-camera applications. The area efficiency reaches 111.329 GOPS/mm2. The power efficiency and area efficiency are 4.5x to 33.0x and 3.8x to 74.2x better than the state-of-the-art works, respectively.

ReSSP: A 5.877 TOPS/W Reconfigurable Smart-camera Stream Processor

Constructions are direct form-meaning pairs with possible schematic slots. These slots are simultaneously constrained by the embedded construction itself and the sentential context. We propose that the constraint could be described by a conditional probability distribution. However, as this conditional probability is inevitably complex, we utilize language models to capture this distribution. Therefore, we build CxLM, a deep learning-based masked language model explicitly tuned to constructions’ schematic slots. We first compile a construction dataset consisting of over ten thousand constructions in Taiwan Mandarin. Next, an experiment is conducted on the dataset to examine to what extent a pretrained masked language model is aware of the constructions. We then fine-tune the model specifically to perform a cloze task on the opening slots. We find that the fine-tuned model predicts masked slots more accurately than baselines and generates both structurally and semantically plausible word samples. Finally, we release CxLM and its dataset as publicly available resources and hope to serve as new quantitative tools in studying construction grammar.

Yu-Hsiang Tseng

Papers

Video Object Segmentation and Tracking Framework With Improved Threshold Decision and Diffusion Distance

Efficient Content Analysis Engine for Visual Surveillance Network

Distributed computing in IoT: System-on-a-chip for smart cameras as an example

ReSSP: A 5.877 TOPS/W Reconfigurable Smart-camera Stream Processor

CxLM: A Construction and Context-aware Language Model