Recent advances in clothes recognition have been driven by the construction of clothes datasets. Existing datasets are limited in the amount of annotations and are difficult to cope with the various challenges in real-world applications. In this work, we introduce DeepFashion1, a large-scale clothes dataset with comprehensive annotations. It contains over 800,000 images, which are richly annotated with massive attributes, clothing landmarks, and correspondence of images taken under different scenarios including store, street snapshot, and consumer. Such rich annotations enable the development of powerful algorithms in clothes recognition and facilitating future researches. To demonstrate the advantages of DeepFashion, we propose a new deep model, namely FashionNet, which learns clothing features by jointly predicting clothing attributes and landmarks. The estimated landmarks are then employed to pool or gate the learned features. It is optimized in an iterative manner. Extensive experiments demonstrate the effectiveness of FashionNet and the usefulness of DeepFashion.

/pdf/deepfashion-powering-robust-clothes-recognition-and-2snhosfcs7.pdf

DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations

Digital processing of speech signals

Newness and distinctiveness is claimed in the features of ornamentation as shown inside the broken line circle in the accompanying representation.

Display screen or portion thereof with graphical user interface

We present our study of audio content analysis for classification and segmentation, in which an audio stream is segmented according to audio type or speaker identity. We propose a robust approach that is capable of classifying and segmenting an audio stream into speech, music, environment sound, and silence. Audio classification is processed in two steps, which makes it suitable for different applications. The first step of the classification is speech and nonspeech discrimination. In this step, a novel algorithm based on K-nearest-neighbor (KNN) and linear spectral pairs-vector quantization (LSP-VQ) is developed. The second step further divides nonspeech class into music, environment sounds, and silence with a rule-based classification scheme. A set of new features such as the noise frame ratio and band periodicity are introduced and discussed in detail. We also develop an unsupervised speaker segmentation algorithm using a novel scheme based on quasi-GMM and LSP correlation analysis. Without a priori knowledge, this algorithm can support the open-set speaker, online speaker modeling and real time segmentation. Experimental results indicate that the proposed algorithms can produce very satisfactory results.

Content analysis for audio classification and segmentation

In today's fast-paced world, while the number of channels of television programming available is increasing rapidly, the time available to watch them remains the same or is decreasing. Users desire the capability to watch the programs time-shifted (on-demand) and/or to watch just the highlights to save time. In this paper we explore how to provide for the latter capability, that is the ability to extract highlights automatically, so that viewing time can be reduced.We focus on the sport of baseball as our initial target—it is a very popular sport, the whole game is quite long, and the exciting portions are few. We focus on detecting highlights using audio-track features alone without relying on expensive-to-compute video-track features. We use a combination of generic sports features and baseball-specific features to obtain our results, but believe that may other sports offer the same opportunity and that the techniques presented here will apply to those sports. We present details on relative performance of various learning algorithms, and a probabilistic framework for combining multiple sources of information. We present results comparing output of our algorithms against human-selected highlights for a diverse collection of baseball games with very encouraging results.

/pdf/automatically-extracting-highlights-for-tv-baseball-programs-baac9me619.pdf

Automatically extracting highlights for TV Baseball programs

Methods and integrated systems for camera motion analysis and moving object analysis and methods of extracting semantics mainly from camera motion parameters in videos and video segments without shot changes are described. Typical examples of such videos are a home video taken by a digital camera and a segment, or clip, of a professional video or film. The extracted semantics can be directly used in a number of video/image understanding and management applications, such as annotation, browsing, editing, frame enhancement, key-frame extraction, panorama generation, printing, retrieval, summarization. Automatic methods of detecting and tracking moving objects that do not rely on a priori knowledge of the objects are also described. The methods can be executed in real time.

Video content understanding through real time video motion analysis

Face-based image clustering systems and methods are described. In one aspect, face regions (58) are detected in images (20). At least one respective parameter value (53) is extracted from each of the face regions (58). Ones of the face regions (58) associated with parameter values (53) satisfying a cluster seed predicate are classified as cluster seed face regions (38). The cluster seed face regions (38) are clustered into one or more clusters (44, 48). A respective face model (24) is built for each of the clusters (44, 48). The face models (24) are stored. In another aspect, face regions (58) are detected in images (20). At least one respective parameter value (53) is extracted from each of the face regions (58). The face regions (58) are ranked based on the extracted parameter values (53). The face regions (58) are clustered in rank order into one or more clusters (44, 48). Representations of ones of the clusters (44, 48) are rendered on a display (116).

Face-based image clustering

A method and system for automatically summarizing a video document (20). The video document (20) is decomposed into scenes (22), shots (24) and frames (26), and an importance value is assigned to each scene, shot and frame. Keyframes are allocated among the shots (24) based on the importance value of each shot (24). The allocated number of keyframes are then selected from each shot(24) . The number of keyframes may be altered for greater or lesser detail in response to user input.

Scalable video summarization and navigation system and method

Automatic clothes search in consumer photos is not a trivial problem as photos are usually taken under completely uncontrolled realistic imaging conditions. In this paper, a novel framework is presented to tackle this issue by leveraging low-level features (e.g., color) and high-level features (attributes) of clothes. First, a content-based image retrieval(CBIR) approach based on the bag-of-visual-words (BOW) model is developed as our baseline system, in which a codebook is constructed from extracted dominant color patches. A reranking approach is then proposed to improve search quality by exploiting clothes attributes, including the type of clothes, sleeves, patterns, etc. The experiments on photo collections show that our approach is robust to large variations of images taken in unconstrained environment, and the reranking algorithm based on attribute learning significantly improves retrieval performance in combination with the proposed baseline.

Clothes search in consumer photos via color matching and attribute learning

A real-time audio segmentation and indexing scheme is presented in this paper. Audio recordings are segmented and classified into basic audio types such as silence, speech, music, song, environmental sound, speech with the music background, environmental sound with the music background, etc. Simple audio features such as the energy function, the average zero-crossing rate, the fundamental frequency, and the spectral peak track are adopted in this system to ensure on-line processing. Morphological and statistical analysis for temporal curves of these features are performed to show differences among different types of audio. A heuristic rule-based procedure is then developed to segment and classify audio signals by using these features. The proposed approach is generic and model free. It can be applied to almost any content-based audio management system. It is shown that the proposed scheme achieves an accuracy rate of more than 90% for audio classification. Examples for segmentation and indexing of accompanying audio signals in movies and video programs are also provided.

Tong Zhang

Papers

Video content understanding through real time video motion analysis

Face-based image clustering

Scalable video summarization and navigation system and method

Clothes search in consumer photos via color matching and attribute learning

Heuristic approach for generic audio data segmentation and annotation