Machine learning techniques for more efficient video compression and video enhancement have been developed thanks to breakthroughs in deep learning. The new techniques, considered as an advanced form of Artificial Intelligence (AI), bring previously unforeseen capabilities. However, they typically come in the form of resource-hungry black-boxes (overly complex with little transparency regarding the inner workings). Their application can therefore be unpredictable and generally unreliable for large-scale use (e.g. in live broadcast). The aim of this work is to understand and optimise learned models in video processing applications so systems that incorporate them can be used in a more trustworthy manner. In this context, the presented work introduces principles for simplification of learned models targeting improved transparency in implementing machine learning for video production and distribution applications. These principles are demonstrated on video compression examples, showing how bitrate savings and reduced complexity can be achieved by simplifying relevant deep learning models.

Towards Transparent Application of Machine Learning in Video Processing.

AbstractAutomatic image colourisation methods applied independently to each video frame usually lead to flickering artefacts or propagation of errors because of differences between neighbouring frames. While this can be partially solved using optical flow methods, complex scenarios such as the appearance of new objects in the scene limit the efficiency of such solutions. To address this issue, we propose application of blind temporal consistency, learned during the inference stage, to consistently adapt colourisation to the given frames. However, training at test time is extremely time-consuming and its performance is highly dependent on the content, motion, and length of the input video, requiring a large number of iterations to generalise to complex sequences with multiple shots and scene changes. This paper proposes a generalised framework for colourisation of complex videos with an optimised few-shot training strategy to learn scene-aware video priors. The proposed architecture is jointly trained to stabilise the input video and to cluster its frames with the aim of learning scene-specific modes. Experimental results show performance improvement in complex sequences while requiring less training data and significantly fewer iterations.KeywordsVideo colourisationTemporal consistencyDeep video priorFew-shot learning 

Scene-Adaptive Temporal Stabilisation for Video Colourisation Using Deep Video Priors

Numerous breakthroughs in multimedia signal processing are being enabled thanks to applications of machine learning in tasks such as multimedia creation, enhancement, classification and compression [1]. Notably, in the context of production and distribution of television programmes, it has been successfully demonstrated how Artificial Intelligence (AI) can support innovation in the creative sector. In the context of delivering TV programmes of stunning visual quality, the applications of deep learning have enabled significant advances when the original content is of poor quality / resolution, or when delivery channels are very limited. Examples when the enhancement of originally poor quality is needed include new content forms (e.g. user generated content) and historical content (e.g. archives), while limitations of delivery channels can, first of all, be addressed by improving content compression. As a state-of-the-art example, the benefits of deep-learning solutions have been recently demonstrated within an end-to-end platform for management of user generated content [2], where deep learning is applied to increase video resolution, evaluate video quality and enrich the video by providing automatic metadata. Within this particular application space where large amount of user generated content is available, the progress has also been made in addressing visual story editing using social media data in automatic ways, making programmes from large amount of content faster [3]. Broadcasters are also interested in restauration of historical content more cheaply. For example, adding colour to "black and white" content has until now been an expensive and time-consuming task. However, recently new algorithms have been developed to perform the task more efficiently. Generative Adversarial Networks (GANs) have become the baseline for many image-to-image translation tasks, including image colourisation. Aiming at the generation of more naturally coloured images from "black and white" sources, newest algorithms are capable of generalisation of the colour of natural images, producing realistic and plausible results [4]. In the context of content delivery, new generations of compression standards enable significant reduction of required bandwidth [5], however, with a cost of increased computational complexity. This is another area where AI can be utilised for better efficiency - either in its simple forms as decision trees [6,7] or more advanced deep convolutional neural networks [8]. Looking forward, this penetration of AI opens new challenges, such as interpretability of deep learning (to enable use AI in an accountable way as well as to enable AI-inspired low-complexity algorithms) and applicability in systems which require low-complexity solutions and/or do not have enough training data. However, overall further benefits of these new approaches include automatization of many traditional production tasks which has the potential to transform the way content providers make their programmes in cheaper and more effective ways.

Marta Mrak

Papers

AI Gets Creative