High resolution, wide field of view video generated from multiple camera feeds has many use cases. However, processing the different steps of a panorama video pipeline in real-time is challenging due to the high data rates and the stringent requirements of timeliness. We use panorama video in a sport analysis system where video events must be generated in real-time. In this respect, we present a system for real-time panorama video generation from an array of low-cost CCD HD video cameras. We describe how we have implemented different components and evaluated alternatives. We also present performance results with and without co-processors like graphics processing units (GPUs), and we evaluate each individual component and show how the entire pipeline is able to run in real-time on commodity hardware.

Efficient implementation and processing of a real-time panorama video pipeline with emphasis on background subtraction

Video processing is a resource demanding task. While todays high–end machines are able to process and encode video at reasonable speeds, they are usually not capable of doing this in real–time. In this thesis, we investigate and implement a distributed version of the real–time panorama video processing pipeline from the Bagadus project. The pipeline consists of several processing steps. Images are first captured from five individual cameras, and then grouped into sets. These sets are converted from the Bayer image format to YUV444 format, before they are stitched into a large panorama image. The stitched panorama video is encoded into H.264 format and stored on disk. It is also possible to enable HDR mode in the pipeline, which creates a video with more details visible in shadows and light areas. We initially created a simple distribution setup, allowing individual processing steps to be run on separate machines. To improve the performance of this setup, we have implemented a more advanced setup. The improved setup removed bottlenecks and adds support for Nvidia GPUDirect, for minimal latency GPU–to–GPU memory copies between machines. This enables a large number of setups, with minimal delay added by the distribution.

Implementation of a real–time distributed video processing pipeline

Great technological advances in the last decades have enabled the creation of multimedia systems that capture high resolution images at high frame rates. As more and more digital video content is generated, we strive for ways to automatically analyze and understand video through computer vision, for categorization, searchable video and other purposes. Object recognition forms an important part in this evolution, where objects in video are found and identified. In this thesis, we introduce a real-time distributed video pipeline that produces high resolution panorama video. Created as a diverse tool for capture and analysis of sports footage, the system allows for real-time generation and viewing of a cylindrical panorama covering an entire soccer field using five industrial cameras. In three case studies, we investigate the potential for ball and object tracking within such a panorama, by installing the system in both indoor and outdoor locations using different configurations. By selecting three current state-of-the-art object tracking algorithms and through implementation of three of our own tracker algorithms, we evaluate the performance and feasibility of object tracking in a real-time video pipeline. We detail and evaluate the performance of these six trackers in all case studies, and learn that there is often a trade-off between tracking robustness and execution speed. Trackers that maintain adaptive appearance models to track unknown objects, struggle with handling the workload of the high resolution video produced by Bagadus. By developing tracking algorithms specific to each sequence, we are able to achieve usable performance by reducing computations, at the sacrifice of portability. We recognize that especially ball tracking in team sports is a very challenging case, and hope to provide knowledge useful for further research.

Øystein Skonseng Landsverk

Papers

Investigating the performance and applicability of object tracking in a real-time distributed video pipeline