Diffusion models have emerged as a powerful new family of deep generative models with record-breaking performance in many applications, including image synthesis, video generation, and molecule design. In this survey, we provide an overview of the rapidly expanding body of work on diffusion models, categorizing the research into three key areas: efficient sampling, improved likelihood estimation, and handling data with special structures. We also discuss the potential for combining diffusion models with other generative models for enhanced results. We further review the wide-ranging applications of diffusion models in fields spanning from computer vision, natural language generation, temporal data modeling, to interdisciplinary applications in other scientific disciplines. This survey aims to provide a contextualized, in-depth look at the state of diffusion models, identifying the key areas of focus and pointing to potential areas for further exploration. Github: https://github.com/YangLing0818/Diffusion-Models-Papers-Survey-Taxonomy.

Diffusion Models: A Comprehensive Survey of Methods and Applications

In recent years, due to the rapidly growing capacities of physical layer, device-free passive detection holds great importance for a broad range of application. Most recent works focus on motion detection, intrusion detection, and vital sign with commodity WiFi devices in the indoor environment. Conventional device-free motion detection techniques, which utilize received signal strength (RSS), may suffer from coarse granularity and high variability problems. In resorting to the finer-grained channel state information (CSI), we propose PhaseMode, a novel approach for device-free motion detection leveraging CSI phase difference data between adjacent antenna pairs. We implement our approach on commercial WiFi devices and validate its performance. We conduct experiments in different test periods of three indoor environments; the results show that the proposed scheme achieves an average accuracy over 99.4% of motion detection in different scenarios.

/pdf/a-robust-passive-intrusion-detection-system-with-commodity-398bhkv0ff.pdf

A Robust Passive Intrusion Detection System with Commodity WiFi Devices

Crowdsourced business intelligence (CrowdBI), which leverages the crowdsourced user-generated data to extract useful knowledge about business and create marketing intelligence to excel in the business environment, has become a surging research topic in recent years. Compared with the traditional business intelligence that is based on the firm-owned data and survey data, CrowdBI faces numerous unique issues, such as customer behavior analysis, brand tracking, and product improvement, demand forecasting and trend analysis, competitive intelligence, business popularity analysis and site recommendation, and urban commercial analysis. This paper first characterizes the concept model and unique features and presents a generic framework for CrowdBI. It also investigates novel application areas as well as the key challenges and techniques of CrowdBI. Furthermore, we make discussions about the future research directions of CrowdBI. INDEX TERMS Crowdsourced business intelligence, consumer behaviors, competitive intelligence, crowd intelligence, commercial site recommendation, brand trending.

https://hal.archives-ouvertes.fr/hal-02321021/document

Harnessing the Power of the General Public for Crowdsourced Business Intelligence: A Survey

Diffusion models have achieved remarkable success in text-to-image generation, enabling the creation of high-quality images from text prompts or other modalities. However, existing methods for customizing these models are limited by handling multiple personalized subjects and the risk of overfitting. Moreover, their large number of parameters is inefficient for model storage. In this paper, we propose a novel approach to address these limitations in existing text-to-image diffusion models for personalization. Our method involves fine-tuning the singular values of the weight matrices, leading to a compact and efficient parameter space that reduces the risk of overfitting and language drifting. We also propose a Cut-Mix-Unmix data-augmentation technique to enhance the quality of multi-subject image generation and a simple text-based image editing framework. Our proposed SVDiff method has a significantly smaller model size compared to existing methods (approximately 2,200 times fewer parameters compared with vanilla DreamBooth), making it more practical for real-world applications.

SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

Diffusion models have found widespread adoption in various areas. However, their sampling process is slow because it requires hundreds to thousands of network evaluations to emulate a continuous process defined by differential equations. In this work, we use neural operators, an efficient method to solve the probability flow differential equations, to accelerate the sampling process of diffusion models. Compared to other fast sampling methods that have a sequential nature, we are the first to propose parallel decoding method that generates images with only one model forward pass. We propose \textit{diffusion model sampling with neural operator} (DSNO) that maps the initial condition, i.e., Gaussian distribution, to the continuous-time solution trajectory of the reverse diffusion process. To model the temporal correlations along the trajectory, we introduce temporal convolution layers that are parameterized in the Fourier space into the given diffusion model backbone. We show our method achieves state-of-the-art FID of 4.12 for CIFAR-10 and 8.35 for ImageNet-64 in the one-model-evaluation setting.

Fast Sampling of Diffusion Models via Operator Learning

This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Our key insight is -- learning diffusion models for marginal, conditional, and joint distributions can be unified as predicting the noise in the perturbed data, where the perturbation levels (i.e. timesteps) can be different for different modalities. Inspired by the unified view, UniDiffuser learns all distributions simultaneously with a minimal modification to the original diffusion model -- perturbs data in all modalities instead of a single modality, inputs individual timesteps in different modalities, and predicts the noise of all modalities instead of a single modality. UniDiffuser is parameterized by a transformer for diffusion models to handle input types of different modalities. Implemented on large-scale paired image-text data, UniDiffuser is able to perform image, text, text-to-image, image-to-text, and image-text pair generation by setting proper timesteps without additional overhead. In particular, UniDiffuser is able to produce perceptually realistic samples in all tasks and its quantitative results (e.g., the FID and CLIP score) are not only superior to existing general-purpose models but also comparable to the bespoken models (e.g., Stable Diffusion and DALL-E 2) in representative tasks (e.g., text-to-image generation).

One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

Vision transformers (ViT) have shown promise in various vision tasks while the U-Net based on a convolutional neural network (CNN) remains dominant in diffusion models. We design a simple and general ViT-based architecture (named U-ViT) for image generation with diffusion models. U-ViT is characterized by treating all inputs including the time, condition and noisy image patches as tokens and employing long skip connections between shallow and deep layers. We evaluate U-ViT in unconditional and class-conditional image generation, as well as text-to-image generation tasks, where U-ViT is comparable if not superior to a CNN-based U-Net of a similar size. In particular, latent diffusion models with U-ViT achieve record-breaking FID scores of 2.29 in class-conditional image generation on ImageNet 256x256, and 5.48 in text-to-image generation on MS-COCO, among methods without accessing large external datasets during the training of generative models. Our results suggest that, for diffusion-based image modeling, the long skip connection is crucial while the down-sampling and up-sampling operators in CNN-based U-Net are not always necessary. We believe that U-ViT can provide insights for future research on backbones in diffusion models and benefit generative modeling on large scale cross-modality datasets.

All are Worth Words: A ViT Backbone for Diffusion Models

Customer’s product preference provides how a customer collects products or prefers one collection over another. Understanding customer’s product preference can provide retail store owner and librarian valuable insight to adjust products and service. Current solutions offer a certain convenience over common approaches such as questionnaire and interviews. However, they either require video surveillance or need wearable sensor which are usually invasive or limited to additional device. Recently, researchers have exploited physical layer information of wireless signals for robust device-free human detection, ever since Channel State Information (CSI) was reported on commodity WiFi devices. Despite of a significant amount of progress achieved, there are few works studying customer’s product preference. In this paper, we propose a customer’s product preference analysis system, PreFi, based on Commercial Off-The-Shelf (COTS) WiFi-enabled devices. The key insight of PreFi is to extract the variance features of the fine-grained time-series CSI, which is sensitively affected by customer activity, to recognize what is the customer doing. First, we conduct Principal Component Analysis (PCA) to smooth the preprocessed CSI values since general denoising method is insufficient in removing the bursty and impulse noises. Second, a sliding window-based feature extraction method and majority voting scheme are adopted to compare the distribution of activity profiles to identify different activities. We prototype our system on COTS WiFi-enabled devices and extensively evaluate it in typical indoor scenarios. The results indicate that PreFi can recognize a few representative customer activity with satisfied accuracy and robustness.

Analyzing Customer’s Product Preference Using Wireless Signals

Real-time video conferencing systems have recently become indispensable tools. However, existing commodity video conferencing systems often fail to deliver a satisfactory quality of experience (QoE) due to discrepancies between the actual network throughput and the average bitrate of encoded videos. In this paper, we present a system called Tyrus that aims to achieve efficient real-time video conferencing. Tyrus adapts its delivery of B-frames based on the real network bandwidth and the average bitrate of encoded videos, effectively addressing the QoE loss caused by mismatch issues. This system can be easily implemented on top of prevalent video conferencing systems that utilize standard video encoders. Unlike traditional approaches that treat all frame types equally, Tyrus enables adaptive delivery of B-frames, prioritizing the allocation of bandwidth to other crucial video frames. By doing so, it reduces end-to-end conferencing latency, especially when the network throughput experiences significant fluctuations. Additionally, Tyrus defines the playing deadline for B-frames by considering factors such as delivery time, buffering time, and the progress of preceding frames. It accurately classifies frame types according to video coding standards and estimates bandwidth allocation on a frame-by-frame basis. Moreover, Tyrus proactively handles potential errors in B-frame playing deadline estimation, minimizing their impact on video conferencing performance. Our evaluation results, obtained through real implementation on top of WebRTC, demonstrate the effectiveness of Tyrus. On average, it reduces per-frame latency by 19.03%, video stalls by 22.36%, and improves bandwidth utilization by at least 10.29%. 

Efficient real-time video conferencing with adaptive frame delivery

The unabating growth of the memory needs of emerging datacenter applications has exacerbated the scalability bottleneck of virtual memory. However, reducing the excessive overhead of address translation will remain onerous until the physical memory contiguity predicament gets resolved. To address this problem, this paper presents Contiguitas, a novel redesign of memory management in the operating system and hardware that provides ample physical memory contiguity. We identify that the primary cause of memory fragmentation in Meta's datacenters is unmovable allocations scattered across the address space that impede large contiguity from being formed. To provide ample physical memory contiguity by design, Contiguitas first separates regular movable allocations from unmovable ones by placing them into two different continuous regions in physical memory and dynamically adjusts the boundary of the two regions based on memory demand. Drastically reducing unmovable allocations is challenging because the majority of unmovable pages cannot be moved with software alone given that access to the page cannot be blocked for a migration to take place. Furthermore, page migration is expensive as it requires a long downtime to (a) perform TLB shootdowns that scale poorly with the number of victim TLBs, and (b) copy the page. To this end, Contiguitas eliminates the primary source of unmovable allocations by introducing hardware extensions in the last-level cache to enable the transparent and efficient migration of unmovable pages even while the pages remain in use. We build the operating system component of Contiguitas into the Linux kernel and run our experiments in a production environment at Meta's datacenters. Our results show that Contiguitas's OS component successfully confines unmovable allocations, drastically reducing unmovable 2MB blocks from an average of 31% scattered across the address space down to 7% confined in the unmovable region, leading to significant performance gains. Specifically, we show that for three major production services, Contiguitas achieves end-to-end performance improvements of 2--9% for partially fragmented servers, and 7--18% for highly fragmented servers, which account for nearly a quarter of Meta's fleet. We further use full-system simulations to demonstrate the effectiveness of the hardware extensions of Contiguitas. Our evaluation shows that Contiguitas-HW enables the efficient migration of unmovable allocations, scales well with the number of victim TLBs, and does not affect application performance. We are currently in the process of upstreaming Contiguitas into Linux.

Kaiwen Xue

Papers

One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

All are Worth Words: A ViT Backbone for Diffusion Models

Analyzing Customer’s Product Preference Using Wireless Signals

Efficient real-time video conferencing with adaptive frame delivery

Contiguitas: The Pursuit of Physical Memory Contiguity in Datacenters