With the proliferation of online services and mobile technologies, the world has stepped into a multimedia big data era. A vast amount of research work has been done in the multimedia area, targeting different aspects of big data analytics, such as the capture, storage, indexing, mining, and retrieval of multimedia big data. However, very few research work provides a complete survey of the whole pine-line of the multimedia big data analytics, including the management and analysis of the large amount of data, the challenges and opportunities, and the promising research directions. To serve this purpose, we present this survey, which conducts a comprehensive overview of the state-of-the-art research work on multimedia big data analytics. It also aims to bridge the gap between multimedia challenges and big data solutions by providing the current big data frameworks, their applications in multimedia analyses, the strengths and limitations of the existing methods, and the potential future directions in multimedia big data analytics. To the best of our knowledge, this is the first survey that targets the most recent multimedia management techniques for very large-scale data and also provides the research studies and technologies advancing the multimedia analyses in this big data era.

Multimedia Big Data Analytics: A Survey

Big data has attracted much attention from academia and industry. But the discussion of big data is disparate, fragmented and distributed among different outlets. This paper conducts a systematic a...

Big data analytics and big data science: a survey

In certain application fields digital audio recordings contain sensitive content. Examples are historical archival material in public archives that preserve our cultural heritage, or digital evidence in the context of law enforcement and civil proceedings. Because of the powerful capabilities of modern editing tools for multimedia such material is vulnerable to doctoring of the content and forgery of its origin with malicious intent. Also inadvertent data modification and mistaken origin can be caused by human error. Hence, the credibility and provenience in terms of an unadulterated and genuine state of such audio content and the confidence about its origin are critical factors.

To address this issue, this PhD thesis proposes a mechanism for verifying the integrity and authenticity of digital sound recordings. It is designed and implemented to be insensitive to common post-processing operations of the audio data that influence the subjective acoustic perception only marginally (if at all). Examples of such operations include lossy compression that maintains a high sound quality of the audio media, or lossless format conversions. It is the objective to avoid de facto false alarms that would be expectedly observable in standard crypto-based authentication protocols in the presence of these legitimate post-processing. For achieving this, a feasible combination of the techniques of digital watermarking and audio-specific hashing is investigated.

At first, a suitable secret-key dependent audio hashing algorithm is developed. It incorporates and enhances so-called audio fingerprinting technology from the state of the art in contentbased audio identification. The presented algorithm (denoted as ”rMAC” message authentication code) allows ”perception-based” verification of integrity. This means classifying integrity breaches as such not before they become audible. As another objective, this rMAC is embedded and stored silently inside the audio media by means of audio watermarking technology. This approach allows maintaining the authentication code across the above-mentioned admissible post-processing operations and making it available for integrity verification at a later date. For this, an existent secret-key ependent audio watermarking algorithm is used and enhanced in this thesis work. 

To some extent, the dependency of the rMAC and of the watermarking processing from a secret key also allows authenticating the origin of a protected audio. To elaborate on this security aspect, this work also estimates the brute-force efforts of an adversary attacking this combined rMAC-watermarking approach. The experimental results show that the proposed method provides a good distinction and classification
performance of authentic versus doctored audio content. It also allows the temporal localization of audible data modification within a protected audio file. The experimental evaluation finally provides recommendations about technical configuration settings of the combined watermarking-hashing approach. 

Beyond the main topic of perception-based data integrity and data authenticity for audio, this PhD work provides new general findings in the fields of audio fingerprinting and digital watermarking. The main contributions of this PhD were published and presented mainly at conferences about multimedia security. These publications were cited by a number of other authors and hence had some impact on their works.

Digital Watermarking for Verification of Perception-based Integrity of Audio Data

In order to satisfy the requirements of retrieval time-efficiency and security for encrypted speech data retrieval in the cloud environment, and to improve the impact of noise on the robustness and discrimination for the speech perceptual hashing scheme, an encrypted speech retrieval algorithm based on Chirp-Z transform and perceptual hashing second feature extraction is proposed in this paper. The speech owner first processes the original speech file by pre-processing, framing, and adding window. The features of the original speech file is extracted by Chirp-Z transform combined with the sparse random matrix to construct a hash sequence. Then encrypt the original speech file based on the m sequence and upload it to the cloud to ensure the security of information. By processing the speech perceptual hashing feature, the speech features are re-extracted, and the speech is evenly classified by k-means clustering technique. The binary string of several hundred bits is converted into a decimal number. Finally, the second feature is stored in system hash index table of the cloud. When the user retrieves, the query speech is denoised and the hash sequence is extracted. Then the secondary features of the hash sequence are extracted and matched with the encrypted speech features in the cloud system hash index table to obtain the retrieval result. The experimental results show that the proposed algorithm greatly compresses the information capacity of speech features, significantly improves the retrieval time-efficiency, with strong robustness and discrimination, and has a good retrieval effect on noisy speech.

An encrypted speech retrieval algorithm based on Chirp-Z transform and perceptual hashing second feature extraction

Vector quantization is an essential tool for tasks involving large scale data, for example, large scale similarity search, which is crucial for content-based information retrieval and analysis. In this paper, we propose a novel vector quantization framework that iteratively minimizes quantization error. First, we provide a detailed review on a relevant vector quantization method named residual vector quantization (RVQ). Next, we propose generalized residual vector quantization (GRVQ) to further improve over RVQ. Many vector quantization methods can be viewed as special cases of our proposed method. To enable GRVQ on billion scale data, we introduce a nonexhaustive search scheme named aggregating tree (A-Tree) for high dimensional data that uses GRVQ encodings to build a radix tree and perform the nearest neighbor search by beam search. To search accurately and efficiently, VQ-encodings should satisfy locally aggregating encoding criterion: For any node of the corresponding A-Tree, neighboring vectors should aggregate in fewer subtrees to make beam search efficient. We show that the proposed GRVQ encodings best satisfy the suggested criterion, and the joint use of GRVQ and A-Tree shows significantly better performances on billion scale datasets. Our methods are validated on several standard benchmark datasets. Experimental results and empirical analysis show the superior efficiency and effectiveness of our proposed methods compared to the state-of-the-art for large scale search.

Generalized Residual Vector Quantization and Aggregating Tree for Large Scale Search

Fast audio retrieval is crucial for many important applications and yet demanding due to the high dimension nature and increasingly larger volume of audios on the Internet. Although audio fingerprinting can greatly reduce its dimension while keeping audio identifiable, the dimension for audio fingerprints is still too high to scale up for big audio data. The tradeoff between accuracy (measured by precision and recall rate) and efficiency (measured by retrieval time) prevents further reduction in the dimension of fingerprints. This paper shows that a multi-stage filtering strategy can achieve both speedup and high accuracy, with the beginning stages focusing on speedup and the end stage emphasizing accuracy. With this strategy, an efficient cascaded filtering retrieval method is proposed that consists of filtering with Fibonacci hashing, the middle fingerprint, thresholds to quickly select candidate audios, and refining with an accurate and robust fingerprint on the candidate audios. Experiments with 500 000 audios show that the proposed method can achieve a speed gain more than 28 K times that of the Fibonacci hashing retrieval. After applying MP3 conversion, resampling, white noise addition, and background noise addition, the recall rates of the method are all above 99.45%, and the precision is the same as the Philips audio fingerprint, which is close to 100%.

An Efficient Cascaded Filtering Retrieval Method for Big Audio Data

It is challenging to retrieve audio clips from large audio datasets not only due to the high dimensionality of audio but also due to the large number of audios. Fingerprinting methods primarily focus on the use of semantic-level techniques to speed up retrieval and neglect low-level support. This paper shows that the performance of audio retrieval can be exploited by properly organizing and manipulating audio fingerprint data. The proposed sampling and counting method markedly improves the retrieval speed while maintaining a high recall rate and high precision for short audio clips. The proposed inverted index structure for fingerprints quickly shrinks the scope of the candidate set while requiring considerably less memory. The experiments show that the proposed method is faster and yields more consistent performance in terms of recall rate and precision than do the state-of-the-art methods. The proposed method is scalable for big audio data and is robust to background noise, resampling, MP3 conversion, white noise addition, bandpass filtering, chorus, echo, flanger, gsm compression, and tremolo. The average precision and recall rates are 99.995% and 99.78%, respectively, for dataset 1 and average 98.97% and 95.15%, respectively, for dataset 2. Specifically, the precision and recall rates for query audio clips of 3 s in length in the targeted dataset 1 are 99.94% and 99.64%, respectively.

Audio Identification by Sampling Sub-fingerprints and Counting Matches

Fast audio retrieval is crucial for many important applications and yet demanding due to the high dimension nature and increasingly larger volume of audios in the internet. Although audio fingerprinting can greatly reduce its dimension while keeping audio identifiable, the dimension of audio fingerprints is still too high to scale up for big audio data. The tradeoff between the accuracy and the efficiency prevents the further reducing of the dimension of fingerprints. This paper proposes a multi-stage filtering strategy for audio retrieval, with the beginning stages focusing on speed up by using a middle fingerprint with much smaller size to quickly filtering the most likely audios, and the ending stages emphasizing on accuracy by applying an accurate and robust fingerprint on the small set of the most likely audios. A notion called middle fingerprint is devised with considerable small dimension for quickly filtering out most irrelevant audios. A matching algorithm is developed to reduce the computational complexity by comparing the samples at fixed interval of two audios with thresholds. By using the middle fingerprint, audio retrieval can get a speed gain of 12 times on average compared with the Fibonacci Hashing retrieval. By combing the Fibonacci hashing algorithm with the middle filtering retrieval and the matching algorithm, we propose an efficient cascaded filtering retrieval methods, which can further improve the retrieval by 250 times on average. After applying MP3 conversion, resampling, and random shearing, the recall rates of the method are all above 99.47%, and the theoretical accuracy is close to 100%.

A step type voice frequency retrieval method comprises the steps that the Fibonacci Hash algorithm is used to establish a Hash index table for a voice frequency original fingerprint database; the voice frequency original fingerprint database is converted through the BoF algorithm to generate a voice frequency intermediate fingerprint database corresponding to the voice frequency original fingerprint database; by screening for three times, a third serial number set of a possible outcome is screened out, and the retrieval range is narrowed once again; in original fingerprints corresponding to the third serial number set, accurate match retrieval is conducted with the original fingerprints of a voice frequency clip to be retrieved, and the final retrieval result is obtained. By means of the step type voice frequency retrieval method, when the quick voice frequency retrieval is conducted, on the premise that the precision is not lowered, the calculated amount can be reduced, it is achieved that the efficiency is improved, and the use of internal storage can be reduced.

Step type voice frequency retrieval method

An ideal audio retrieval method should be not only highly efficient in identifying an audio track from a massive audio dataset, but also robust to any distortion. Unfortunately, none of the audio retrieval methods is robust to all types of distortions. An audio retrieval method has to do with both the audio fingerprint and the strategy, especially how they are combined. We argue that the Sampling and Counting Method (SC), a state-of-the-art audio retrieval method, would be promising towards an ideal audio retrieval method, if we could make it robust to time-stretch and pitch-stretch. Towards this objective, this paper proposes a turning point alignment method to enhance SC with resistance to time-stretch, which makes Philips and Philips-like fingerprints resist to time-stretch. Experimental results show that our approach can resist to time-stretch from 70% to 130%, which is on a par to the state-of-the-art methods. It also marginally improves the retrieval performance with various noise distortions.

Shanshan Yao

Papers

An Efficient Cascaded Filtering Retrieval Method for Big Audio Data

Audio Identification by Sampling Sub-fingerprints and Counting Matches

An Efficient Cascaded Filtering Retrieval Method for Big Audio Data

Step type voice frequency retrieval method

Enhancing Sampling and Counting Method for Audio Retrieval with Time-Stretch Resistance