Vegetation phenology controls the seasonality of many ecosystem processes, as well as numerous biosphere-atmosphere feedbacks. Phenology is also highly sensitive to climate change and variability. Here we present a series of datasets, together consisting of almost 750 years of observations, characterizing vegetation phenology in diverse ecosystems across North America. Our data are derived from conventional, visible-wavelength, automated digital camera imagery collected through the PhenoCam network. For each archived image, we extracted RGB (red, green, blue) colour channel information, with means and other statistics calculated across a region-of-interest (ROI) delineating a specific vegetation type. From the high-frequency (typically, 30 min) imagery, we derived time series characterizing vegetation colour, including “canopy greenness”, processed to 1- and 3-day intervals. For ecosystems with one or more annual cycles of vegetation activity, we provide estimates, with uncertainties, for the start of the “greenness rising” and end of the “greenness falling” stages. The database can be used for phenological model validation and development, evaluation of satellite remote sensing data products, benchmarking earth system models, and studies of climate change impacts on terrestrial ecosystems.

Tracking vegetation phenology across diverse North American biomes using PhenoCam imagery: A new, publicly-available dataset

Motivated by recent work on deep neural network (DNN)-based image compression methods showing potential improvements in image quality, savings in storage, and bandwidth reduction, we propose to perform image understanding tasks such as classification and segmentation directly on the compressed representations produced by these compression methods Since the encoders and decoders in DNN-based compression methods are neural networks with feature-maps as internal representations of the images, we directly integrate these with architectures for image understanding This bypasses decoding of the compressed representation into RGB space and reduces computational cost Our study shows that accuracies comparable to networks that operate on compressed RGB images can be achieved while reducing the computational complexity up to $2\times$ Furthermore, we show that synergies are obtained by jointly training compression networks with classification networks on the compressed representations, improving image quality, classification accuracy, and segmentation performance We find that inference from compressed representations is particularly advantageous compared to inference from compressed RGB images for aggressive compression rates

Towards Image Understanding from Deep Compression without Decoding

Moving object segmentation and classification from compressed video plays an important role in intelligent video surveillance. Compared with H.264/AVC, High Efficiency Video Coding (HEVC) introduces a host of new coding features that can be further exploited for moving object segmentation and classification. In this paper, we present a real-time approach to segment and classify moving objects using unique features directly extracted from the HEVC compressed domain for video surveillance. In the proposed method, first, motion vector (MV) interpolation for intra-coded prediction unit (PU) and MV outlier removal are employed for preprocessing. Second, blocks with nonzero MVs are clustered into the connected foreground regions using the four-connectivity component labeling algorithm. Third, object region tracking based on temporal consistency is applied to the connected foreground regions to remove the noise regions. The boundary of moving object region is further refined by the coding unit size and PU size. Finally, a person–vehicle classification model using bag of spatial–temporal HEVC syntax words is trained to classify the moving objects, either persons or vehicles. The experimental results demonstrate that the proposed method provides solid performance and can classify moving persons and vehicles accurately.

Real-Time Moving Object Segmentation and Classification From HEVC Compressed Surveillance Video

Image and video analysis requires rich features that can characterize various aspects of visual information. These rich features are typically extracted from the pixel values of the images and videos, which require huge amount of computation and seldom useful for real-time analysis. On the contrary, the compressed domain analysis offers relevant information pertaining to the visual content in the form of transform coefficients, motion vectors, quantization steps, coded block patterns with minimal computational burden. The quantum of work done in compressed domain is relatively much less compared to pixel domain. This paper aims to survey various video analysis efforts published during the last decade across the spectrum of video compression standards. In this survey, we have included only the analysis part, excluding the processing aspect of compressed domain. This analysis spans through various computer vision applications such as moving object segmentation, human action recognition, indexing, retrieval, face detection, video classification and object tracking in compressed videos.

A survey on compressed domain video analysis techniques

Motivated by recent work on deep neural network (DNN)-based image compression methods showing potential improvements in image quality, savings in storage, and bandwidth reduction, we propose to perform image understanding tasks such as classification and segmentation directly on the compressed representations produced by these compression methods. Since the encoders and decoders in DNN-based compression methods are neural networks with feature-maps as internal representations of the images, we directly integrate these with architectures for image understanding. This bypasses decoding of the compressed representation into RGB space and reduces computational cost. Our study shows that accuracies comparable to networks that operate on compressed RGB images can be achieved while reducing the computational complexity up to $2\times$. Furthermore, we show that synergies are obtained by jointly training compression networks with classification networks on the compressed representations, improving image quality, classification accuracy, and segmentation performance. We find that inference from compressed representations is particularly advantageous compared to inference from compressed RGB images for aggressive compression rates.

/pdf/towards-image-understanding-from-deep-compression-without-flv8mkm1do.pdf

. Continuous monitoring of climate indicators is important for understanding the dynamics and trends of the climate system. Lake ice has been identified as one such indicator, and has been included in the list of Essential Climate Variables (ECVs). Currently there are two main ways to survey lake ice cover and its change over time, in-situ measurements and satellite remote sensing. The challenge with both of them is to ensure sufficient spatial and temporal resolution. Here, we investigate the possibility to monitor lake ice with video streams acquired by publicly available webcams. Main advantages of webcams are their high temporal frequency and dense spatial sampling. By contrast, they have low spectral resolution and limited image quality. Moreover, the uncontrolled radiometry and low, oblique viewpoints result in heavily varying appearance of water, ice and snow. We present a workflow for pixel-wise semantic segmentation of images into these classes, based on state-of-the-art encoder-decoder Convolutional Neural Networks (CNNs). The proposed segmentation pipeline is evaluated on two sequences featuring different ground sampling distances. The experiment suggests that (networks of) webcams have great potential for lake ice monitoring. The overall per-pixel accuracies for both tested data sets exceed 95 %. Furthermore, per-image discrimination between ice-on and ice-off conditions, derived by accumulating per-pixel results, is 100 % correct for our test data, making it possible to precisely recover freezing and thawing dates.

/pdf/lake-ice-monitoring-with-webcams-233n168nej.pdf

Lake ice monitoring with webcams

This paper discusses a novel high-speed approach for human action recognition in H.264/AVC compressed domain. The proposed algorithm utilizes cues from quantization parameters and motion vectors extracted from the compressed video sequence for feature extraction and further classification using Support Vector Machines (SVM). The ultimate goal of the proposed work is to portray a much faster algorithm than pixel domain counterparts, with comparable accuracy, utilizing only the sparse information from compressed video. Partial decoding rules out the complexity of full decoding, and minimizes computational load and memory usage, which can result in reduced hardware utilization and faster recognition results. The proposed approach can handle illumination changes, scale, and appearance variations, and is robust to outdoor as well as indoor testing scenarios. We have evaluated the performance of the proposed method on two benchmark action datasets and achieved more than 85 % accuracy. The proposed algorithm classifies actions with speed (?>?2,000 fps) approximately 100 times faster than existing state-of-the-art pixel-domain algorithms.

Compressed domain human action recognition in H.264/AVC video streams

. Monitoring and analyzing the (decreasing) trends in lake freezing provides important information for climate research. Multi-temporal satellite images are a natural data source to survey ice on lakes. In this paper, we describe a method for lake ice monitoring, which uses low spatial resolution (250 m–1000 m) satellite images to determine whether a lake is frozen or not. We report results on four selected lakes in Switzerland: Sihl, Sils, Silvaplana and St. Moritz. These lakes have different properties regarding area, altitude, surrounding topography and freezing frequency, describing cases of medium to high difficulty. Digitized Open Street Map (OSM) lake outlines are back-projected on to the image space after generalization. As a pre-processing step, the absolute geolocation error of the lake outlines is corrected by matching the projected outlines to the images. We define the lake ice detection as a two-class (frozen, non-frozen) semantic segmentation problem. Several spectral channels of the multi-spectral satellite data are used, both reflective and emissive (thermal). Only the cloud-free (clean) pixels which lie completely inside the lake are analyzed. The most useful channels to solve the problem are selected with xgboost and visual analysis of histograms of reference data, while the classification is done with non-linear support vector machine (SVM). We show experimentally that this straight-forward approach works well with both MODIS and VIIRS satellite imagery. Moreover, we show that the algorithm produces consistent results when tested on data from multiple winters.

/pdf/lake-ice-detection-in-low-resolution-optical-satellite-4njw1g8isp.pdf

Lake Ice Detection in Low-Resolution Optical Satellite Images

This paper discusses a novel fast approach for moving object detection in H.264/AVC compressed domain for video surveillance applications. The proposed algorithm initially segments out edges from regions with motion at macroblock level by utilizing the gradient of quantization parameter over 2D-image space. A spatial median filtering of the segmented edges followed by weighted temporal accumulation accounts for whole object segmentation. To attain sub-macroblock (4×4) level precision, the size of macroblocks (in bits) is interpolated using a two tap filter. Partial decoding rules out the complexity involved in full decoding and gives fast foreground segmentation results. Compared to other compressed domain techniques, the proposed approach allows the video streams to be encoded with different quantization parameters across macroblocks thereby increasing flexibility in bit rate adjustment.

Manu Tom

Papers

A survey on compressed domain video analysis techniques

Lake ice monitoring with webcams

Compressed domain human action recognition in H.264/AVC video streams

Lake Ice Detection in Low-Resolution Optical Satellite Images

Fast moving-object detection in H.264/AVC compressed domain for video surveillance