In this paper, a novel approach to video temporal decomposition into semantic units, termed scenes, is presented. In contrast to previous temporal segmentation approaches that employ mostly low-level visual or audiovisual features, we introduce a technique that jointly exploits low-level and high-level features automatically extracted from the visual and the auditory channel. This technique is built upon the well-known method of the scene transition graph (STG), first by introducing a new STG approximation that features reduced computational cost, and then by extending the unimodal STG-based temporal segmentation technique to a method for multimodal scene segmentation. The latter exploits, among others, the results of a large number of TRECVID-type trained visual concept detectors and audio event detectors, and is based on a probabilistic merging process that combines multiple individual STGs while at the same time diminishing the need for selecting and fine-tuning several STG construction parameters. The proposed approach is evaluated on three test datasets, comprising TRECVID documentary films, movies, and news-related videos, respectively. The experimental results demonstrate the improved performance of the proposed approach in comparison to other unimodal and multimodal techniques of the relevant literature and highlight the contribution of high-level audiovisual features toward improved video segmentation to scenes.

https://mklab.iti.gr/files/csvt11_preprint.pdf

Temporal Video Segmentation to Scenes Using High-Level Audiovisual Features

Many applications such as media monitoring are experiencing a large expansion as a consequence of the different emerging media sources and can benefit dramatically by using automatic transcription of audio data. In this paper, we describe the development of a speech recognition engine, AUDIMUS.MEDIA used in the Broadcast News domain. Additionally we describe recent improvements that permitted a relative recognition error decrease of more than 20% and a 4x speed-up.

/pdf/audimus-media-a-broadcast-news-speech-recognition-system-for-v5fgvqvdes.pdf

AUDIMUS.MEDIA: a broadcast news speech recognition system for the european portuguese language

Audio event detection is one of the tasks of the European project VIDIVIDEO. This paper focuses on the detection of non-speech events, and as such only searches for events in audio segments that have been previously classified as non-speech. Preliminary experiments with a small corpus of sound effects have shown the potential of this type of corpus for training purposes. This paper describes our experiments with SVM and HMM-based classifiers, using a 290-hour corpus of sound effects. Although we have only built detectors for 15 semantic concepts so far, the method seems easily portable to other concepts. The paper reports experiments with multiple features, different kernels and several analysis windows. Preliminary experiments on documentaries and films yielded promising results, despite the difficulties posed by the mixtures of audio events that characterize real sounds.

Non-speech audio event detection

This paper focuses on the tasks of recovering capitalization and punctuation marks from texts without that information, such as spoken transcripts, produced by automatic speech recognition systems. These two practical rich transcription tasks were performed using the same discriminative approach, based on maximum entropy, suitable for on-the-fly usage. Reported experiments were conducted both over Portuguese and English broadcast news data. Both force aligned and automatic transcripts were used, allowing to measure the impact of the speech recognition errors. Capitalized words and named entities are intrinsically related, and are influenced by time variation effects. For that reason, the so-called language dynamics have been addressed for the capitalization task. Language adaptation results indicate, for both languages, that the capitalization performance is affected by the temporal distance between the training and testing data. In what regards the punctuation task, this paper covers the three most frequent punctuation marks: full stop, comma, and question marks. Different methods were explored for improving the baseline results for full stop and comma. The first uses punctuation information extracted from large written corpora. The second applies different levels of linguistic structure, including lexical, prosodic, and speaker related features. The comma detection improved significantly in the first method, thus indicating that it depends more on lexical features. The second method provided even better results, for both languages and both punctuation marks, best results being achieved mainly for full stop. As for question marks, there is a small gain, but differences are not very significant, due to the relatively small number of question marks in the corpora.

Bilingual Experiments on Automatic Recovery of Capitalization and Punctuation of Automatic Speech Transcripts

http://www.inesc-id.pt/pt/indicadores/Ficheiros/4830.pdf

Recovering capitalization and punctuation marks for automatic speech recognition: Case study for Portuguese broadcast news

This paper describes ongoing work on selective dissemination of broadcast news. Our pipeline system includes several modules: audio preprocessing, speech recognition, and topic segmentation and indexation. The main goal of this work is to study the impact of earlier errors in the last modules. The impact of audio preprocessing errors is quite small on the speech recognition module, but quite significant in terms of topic segmentation. On the other hand, the impact of speech recognition errors on the topic segmentation and indexation modules is almost negligible. The diagnostic of the errors in these modules is a very important step for the improvement of the prototype of a media watch system described in this paper.

/pdf/a-prototype-system-for-selective-dissemination-of-broadcast-25gn709py1.pdf

A Prototype System for Selective Dissemination of Broadcast News in European Portuguese

The media monitoring activity is undergoing a large expansion as a consequence of the different emerging media sources. This is pushing the development of automatic systems for selective dissemination of multimedia information. In this paper we present the development of a prototype system able to scan multimedia data, specifically TV broadcasts, and to generate alert messages to users about the relevant information to them. The system makes use of advanced processing technologies for content-based indexing of multimedia information. We use large vocabulary speech recognition system, associated with audio segmentation, and automatic topic indexing and segmentation, to generate category information as semantic markup of multimedia data. The system service is based on a web interface design offering new views of these marked documents and providing useful end-user services based on the content multimedia exploitation.

/pdf/a-system-for-selective-dissemination-of-multimedia-4r26k4juz1.pdf

A system for selective dissemination of multimedia information resulting from the alert project

This paper describes a topic segmentation and indexation system for TV broadcast news programs spoken in European Portuguese. The system is integrated in an alert system for selective dissemination of multimedia information developed in the scope of an European Project. The goal of this work is to enhance the retrieval of specific spoken documents that have been automatically transcribed, using speech recognition. Our segmentation algorithm is based on simple heuristics related with anchor detection. The indexation is based on hierarchical concept trees (thesaurus), containing 22 main thematic domains, for which Hidden Markov models and topic language models were created. On-going experiments related to multiple topic indexing are also described, where a confidence measure based on the likelihood ratio test is used as the hypothesis test.

/pdf/topic-indexing-of-tv-broadcast-news-programs-2y4uufzytj.pdf

Topic indexing of TV broadcast news programs

This paper describes the latest progress in our work on Broadcast News for European Portuguese. The central modules of our media watch system that matches the topic of each news story against the user preferences registered in the system are: audio pre-processing, speech recognition and topic segmentation and indexation. The main focus of the paper is on the impact of the errors made by the earlier modules in the last ones. This impact is in our opinion an essential diagnostic tool for the improvement of the overall pipeline system.

/pdf/automatic-vs-manual-topic-segmentation-and-indexation-in-3ub6qiusfb.pdf

Automatic vs. manual topic segmentation and indexation in broadcast news

This paper describes the evaluation of the system for selective dissemination of Broadcast News that we developed in the context of the European project ALERT. Each component of the main processing block of our system was evaluated separately, using the ALERT corpus. Likewise, the user interface was also evaluated separately. Besides this modular evaluation which will be briefly mentioned here, as a reference, the system can also be evaluated as a whole, in a field trial from the point of view of a potential user. This is the main topic of this paper. The analysis of the main sources of problems hinted at a large number of issues that must be dealt with in order to improve the performance. In spite of these pending problems, we believe that having a fully operational system is a must for being able to address user needs in the future in this type of service.

/pdf/evaluation-of-an-alert-system-for-selective-dissemination-of-1n1tfh3cgc.pdf

Rui Amaral

Papers

A Prototype System for Selective Dissemination of Broadcast News in European Portuguese

A system for selective dissemination of multimedia information resulting from the alert project

Topic indexing of TV broadcast news programs

Automatic vs. manual topic segmentation and indexation in broadcast news

Evaluation of an alert system for selective dissemination of broadcast news