Fuzzy color histogram-based video segmentation

TL;DR: Experimental results show that the proposed fuzzy color histogram-based shot-boundary detection algorithm effectively detects shot boundaries and reduces false alarms as compared to the state-of-the-art shot- boundary detection algorithms.

Abstract: We present a fuzzy color histogram-based shot-boundary detection algorithm specialized for content-based copy detection applications The proposed method aims to detect both cuts and gradual transitions (fade, dissolve) effectively in videos where heavy transformations (such as cam-cording, insertions of patterns, strong re-encoding) occur Along with the color histogram generated with the fuzzy linking method on L*a*b* color space, the system extracts a mask for still regions and the window of picture-in-picture transformation for each detected shot, which will be useful in a content-based copy detection system Experimental results show that our method effectively detects shot boundaries and reduces false alarms as compared to the state-of-the-art shot-boundary detection algorithms

Summary

1. Introduction

• Recent developments in multimedia technology with the significant growth of media resources introduced content-based copy detection (CBCD) as a new research field alternative to the watermarking approach for identification of video sequences.
• An abrupt transition, also known as hard-cut, is the most common and easy to detect transition type.
• The paper is organized as follows: Section 2 discusses related work.
• Section 3 explains the video transformations used for ll rights reserved.
• Section 4 describes the proposed shot-boundary detection algorithm, as well as the methods for detecting frame-dropping and picture-in-picture transformations, noise detection, and mask generation.

2. Related work

• Studies on shot-boundary detection are typically based on extracting visual features (color, edge, motion, and interest points) and comparing them among successive frames.
• Dailianas et al. [9], Boreczky and Rothe authors [10] compare early shot-boundary detection algorithms.
• In recent years, researchers focus on detecting gradual transitions effectively and avoiding the false alarms caused by flashlight and the motion of large objects in the scene, since the recognition of hard-cuts is very reliable for most of the methods.
• Das et al. [18] define a unified interval type-2 fuzzy rule based model using fuzzy histogram and fuzzy co-occurrence matrix to detect cuts and various types of gradual transitions.
• The authors also preferred extracting a fixed number of frames per time interval in their CBCD system [22].

3. Video transformations

• In order to develop a shot-boundary detection algorithm specialized for CBCD applications, the authors need to understand the effects of transformations used for modifying videos.
• Here, the authors discuss the negative effects of video transformations, and possible corrective actions taken by their method: i Frame dropping: Dropped frames should be ignored or estimated; otherwise the shot-boundary detection algorithm decides each blank frame as a cut.
• Such frames have the mean of intensity values near to zero.
• With the extracted window, foreground and background frames can be handled separately.
• A mask for still regions, which includes the inserted pattern or text, will increase the effectiveness of a CBCD system.

4.1. Detection of frame-dropping transformation

• Handling frame-dropping transformation is one of the key features of a shot-boundary detection system specialized for CBCD applications; since most of the proposed algorithms consider missing frames as hard-cuts.
• A dropped frame is either exactly or nearly a blank frame, which has a small overall intensity (less than thbf = 0.0039).

4.2. Noise detection

• CBCD applications should handle query videos with heavy noise transformations.
• For their algorithms to work properly, noisy frames/shots should be identified before any further operation that is based on edge detection or use standard deviation of pixel intensity values.
• In image processing, a nonlinear median filter is preferred over a linear filter for cleaning salt and pepper and white noise.
• Otherwise, when the average intensity change exceeds a threshold thn, it is regarded as noisy.
• The authors evaluate the noise detection method and the impact of the parameters (the size of the median filter and the threshold value) in Section 5.

4.3. Mask generation

• When a video segment is transformed with various types of transformations summarized in Table 1, it clearly changes the content of the frames regarding color, edge, and shape information.
• A content-based copy detection system should cut out the artificially inserted texts, patterns, logos, etc., if possible.
• Besides, it should ignore the bordering black areas produced by shift, crop, and letterbox transformations.
• As a result, the probability of matching with the original video segment is increased.
• The authors create a mask of standard deviations greater than the threshold thsr = 0.01 for each shot representing still regions while detecting shot boundaries.

4.4. Detection of picture-in-picture transformation

• In order to detect the window of picture-in-picture transformation, black framings on the sides of the video segment generated by cam-cording, crop, or shift transformations should be extracted first.
• The authors mark each row and column starting from the beginning and from the end as border rows if 1 w Xw c¼1 rshotði; cÞ < thsr ð9Þ holds for that row.
• The authors crop out the borders from Mshot, and then find the derivatives with a first-order difference from both + and x-axis using the Prewitt edge detector: Eshot ¼ 1 0 1 1 0 1 1 0 1 2 64 3 75 Mshot ð10Þ Strong vertical edges are extracted from Eshot using Hough lines [29].
• Only vertical lines are selected, and compared in order to form a rectangular window.
• The candidate window(s) and the border information for each shot are stored.

4.5. Shot-boundary detection

• The authors use a color histogram-based method generated with the fuzzy linking method on L*a*b* color space.
• L* is divided into 3 regions (black, gray, white), a* is divided into 5 regions (green, greenish, middle, reddish, red), and b* also is divided into 5 regions (blue, bluish, middle, yellowish, yellow).
• The main advantage of the proposed fuzzy color histogram over a conventional color histogram is its accuracy.
• The fuzzy color histogram of a dropped frame is predicted by averaging the features of the previous two frames:.
• The essential idea of using color histogram for shot-boundary detection is that color content does not change rapidly within a shot.

5. Evaluations and discussion

• Three main parts of the proposed system, noise detection, shotboundary detection, and window extraction of picture-in-picture transformation, are evaluated.
• The authors test dataset is composed of query videos provided for TRECVID 2008 content-based copy detection task.
• There are total of 2010 MPEG-1 videos, which is about 80 h of video segments with various transformations applied (see Table 1).
• The important events that occur in query videos are shown in Fig.

5.1. Evaluation of noise detection

• The authors used the query video set of TRECVID 2008 CBCD task, and extracted 1 frame per 2 s for each of 2010 videos.
• After decoding the videos, 33,478 images are manually labeled as 1 or 0, indicating that the frame is highly noisy or not.
• Median filters of different sizes are evaluated and compared in an ROC curve (see Fig. 9).
• Most of the false alarms are caused by query videos that have noise originally, but not as a transformation.
• Method Recall Precision F1 RGB CH 0.6284 0.6862 0.6560 that frames with sea, wavy water, or a textured background generally give high noise detection outputs.

5.2. Evaluation of picture-in-picture transformation detection

• Out of 2010 query videos, 545 of them include picture-in-picture transformation.
• The authors obtained the scale and offset information of all picture-in-picture transformations by processing the groundtruth data used for generating query videos.
• Missed picture-in-picture transformations are generally caused by complex transformations, i.e., (8)–(10) in Table 1.

5.3. Evaluation of shot-boundary detection algorithm

• The authors selected a set of shot-boundary detection algorithms from the literature for a comparative evaluation of their method.
• The algorithms selected for comparison cover most of the major techniques listed in [8,10,7].
• It should be noted that the methods selected for comparison could perform much better for detecting shot-boundaries of videos on which none of the transformations listed in Table 1 are applied.
• For most of the transformations, the proposed fuzzy color histogram-based method performs better than other techniques.

Figures

Table 2 The parameters of the algorithm.

Fig. 5. Fuzzy membership functions for the inputs (L*, a*, b*) and output.

Fig. 6. The structure of the fuzzy color histogram.

Fig. 9. The ROC curve of the noise detection method with different median filter settings.

Fig. 8. The important events that occur in query videos: successive frames with frame-dropping transformation (first row), cut and dissolve transitions (second row), fade-in transition (third row), shot-boundaries for a picture-in-picture transformation-applied video, foreground changes at 1087, background at 1779 (fourth row), fast moving object in the scene (last row). Shot-boundaries during cut/gradual transitions (rows 2–3), and for background and foreground videos (row 4) are detected, while dropped frames (row 1) and fast object movements are ignored.

Fig. 7. Two successive frames in a gradual transition: (a) ith frame; (b) the gray-scale histogram of the ith frame; (c) the fuzzy color histogram of the ith frame; (d) (i + 1)th frame; (e) the gray-scale histogram of the (i + 1)th frame; (f) the fuzzy color histogram of the (i + 1)th frame.

Table 1 The list of transformations used in the CBCD task.

Fig. 1. Transformations: (a) original frame, (b) picture-in-picture Type 1, (c) insertion of pattern, (d) strong re-encoding, (e) change of gamma, (f) letterbox, (g) white noise, (h) crop, (i) shift, (j) caption/text insertion, (k) flip, and (l) picture-in-picture Type 2.

Fig. 2. The overview of the proposed algorithm.

Fig. 10. Recall values of shot-boundary detection algorithms for different transformation types.

Fig. 4. Detected borders and windows for query frames. The borders are shown in red and the window frames are shown as green rectangles. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 3. The detection of borders: (a) first frame of a query video shot on which both the picture-in-picture and crop transformations are applied, (b) the standard deviation of the shot, and (c) the border shown in red. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Frequently asked questions

Q1. What are the contributions in "Fuzzy color histogram-based video segmentation" ?

The authors present a fuzzy color histogram-based shot-boundary detection algorithm specialized for contentbased copy detection applications. The proposed method aims to detect both cuts and gradual transitions ( fade, dissolve ) effectively in videos where heavy transformations ( such as cam-cording, insertions of patterns, strong re-encoding ) occur. 

Q2. What have the authors stated for future works in "Fuzzy color histogram-based video segmentation" ?

As a future work the authors will use the detected shot boundaries, masks of still regions, picture-in-picture window boundaries, and the fuzzy color histogram method in their content-based copy detection system. 

Q3. How does the method determine whether there is a shot change?

By matching the same objects and scenes using contrast context histogram (CCH) in two adjacent frames, the method decides that there is no shot change. 

Q4. What is the common approach for CBCD?

In the field of CBCD, representing video with a set of keyframes (one or more representative frame for each shot) is a common approach. 

Q5. What is the definition of a fuzzy logic approach for detecting cut?

In [17], histogram differences of consecutive frames are characterized as fuzzy terms, such as small, significant and large, and fuzzy rules for detecting abrupt and gradual transitions are formulated in a fuzzy-logic-based framework for segmentation of video sequences. 

Q6. How did the authors obtain the scale and offset information of all picture-in-picture transformations?

The authors obtained the scale and offset information of all picture-in-picture transformations by processing the groundtruth data used for generating query videos. 

Q7. What is the basic idea of the keypoint matching-based shot-boundary detection methods?

iv Local keypoint matching (KM): Recognizing the objects and scenes throughout the video is the basic idea of the keypoint matching-based shot-boundary detection methods. 

Q8. How many frames per second do Douze et al. extract?

Douze et al. prefer extracting 2.5 frames per second for query videos, and extracting only a few representative keyframes for the dataset [21]. 

Q9. What is the method for detecting shot-boundaries?

Their tests with 50 query videos, which represents each transformation type with at least 4 videos, showed that fuzzy color histogram-based shot-boundary detection method can achieve higher accuracy values, while reducing false alarms. 

Q10. How does the fuzzy color histogram method work?

The authors propose a fuzzy color histogram-based shot-boundary detection method for the videos where heavy transformations (such as cam-cording, insertions of patterns, strong re-encoding) occur. 

Q11. How does the fuzzy color histogram method achieve a lower false alarm rate?

Their method also achieves a lower false alarm rate with a precision of 93.67%, whereas the precision values of the other methods could only reach up to 53.21%. 

Q12. What is the effect of a mask for still regions?

Although the insertion of a pattern or text does not affect the shot-boundary detection process strongly, a mask for still regions, which includes the inserted pattern or text, will increase the effectiveness of a CBCD system.