Synchronization of sensory effects with multimedia content is a nontrivial and error-prone task that can discourage authoring of mulsemedia applications. Although there are authoring tools that perform some automatic authoring of sensory effect metadata, the analysis techniques that they use are not generally enough to identify complex components that may be related to sensory effects. In this article, we present a new method, which allows the semiautomatic definition of sensory effects in an authoring tool. We outline a software component to be integrated into authoring tools that uses content analysis assistance to indicate moments of sensory effects activation, according to author preferences. The proposed method was implemented in the STEVE 2.0 authoring tool and an evaluation was performed to assess the precision of the generated sensory effects in comparison with human authoring. This solution is expected to considerably reduce the effort of synchronizing audiovisual content with sensory effects—in particular, by easing the author’s repetitive task of synchronizing recurring effects with lengthy media.

Toward Content-Driven Intelligent Authoring of Mulsemedia Applications

Software‐defined network (SDN) is a programmable networking paradigm that enables logically centralized network management. In order to overcome the shortcomings of traditional networks, SDN has come into the picture. SDN is a recent and fast‐forwarding technique that involves separating the network's control plane from the data plane. This network also allows us to directly communicate with the applications through application programming interfaces. A network administrator can directly modify and update the routing policies and rules in SDN. Apart from the advantages given by the SDN, it has also been found that separation of control plane directly affects the network performance. Finding the optimum number of controllers and their locations to improve the performance is one of the fundamental research problems in SDN which is also commonly known as the controller placement problem (CPP). In this article, we have addressed the CPP in SDN in detail and investigated the optimization objectives that influence CPP. The state‐of‐the‐art of this article is to review almost all the optimization objective along with their mathematical formulation and survey the CPP solutions proposed by the other researchers'. Classified the CPP into three parts based on the objectives and solutions of CPP and reviewed the impact of these objectives on network performance. This article gives a comprehensive overview of current strategies on CPP in SDN and at last, we give some research issues that can be further explored by the researchers.

A comprehensive study of different objectives and solutions of controller placement problem in software‐defined networks

The use of multiple senses in interactive applications has become increasingly feasible due to the upsurge of commercial, off-the-shelf devices to produce sensory effects. Creating Multiple Sensorial Media (MulSeMedia) immersive systems requires understanding their digital ecosystem. Mulsemedia systems encompass a set of applications, and devices of different types assembled to communicate or express feelings from the virtual world to the real world. Despite existing standards, tools, and recent research devoted to them, there is still a lack of formal and explicit representation of what mulse-media is. Misconceptions could eventually lead to the construction of solutions that might not take into account reuse, integration, standardization, among other design features. In this paper, we propose to establish a common conceptualization about mulsemedia systems through a reference ontology, named MulseOnto, covering their main notions. To evaluate it, we applied ontology verification and validation techniques, including assessment by humans and a data-driven approach. The results showed that MulseOnto can be used as a consensual conceptual model for exploring the knowledge about the whole chain of mulsemedia systems.

/pdf/towards-a-reference-ontology-on-mulsemedia-systems-39bjx7omip.pdf

Towards a reference ontology on mulsemedia systems

Currently, most declarative languages used to define multimedia documents do not support the specification of an interactive application with multiple users and multimodal interaction. To handle multimodal and multi-user interaction in these languages, the author of multimedia applications must write custom code in an imperative language to be included in the declarative document. To fill in this gap, this work proposes an extension of NCM (Nested Context Language), NCL (Nested Context Language) and Ginga-NCL to provide high-level abstraction multimodal and multi-user interaction specifications. As a proof of concept, we developed a multimedia application that uses multi-user voice interaction.

Extending Ginga-NCL to Specify Multimodal Interactions With Multiple Users

Although several studies discuss rendering sensory effects in multimedia applications, only a few of them discuss the definition of spatial and temporal synchronization relationships among multimedia content and sensory effects. In order to fill in this gap, this paper proposes an approach for authoring mulsemedia applications based on events. This approach involves the proposal of a mulsemedia conceptual model, called MultiSEM (Multimedia Sensory Effect Model), for supporting the development of graphical authoring environments based on temporal view. The approach also includes the implementation of MultiSEM in a graphical authoring environment, named STEVE 2.0, as proof of concept. STEVE 2.0 is designed for users with few or no knowledge of programming, multimedia/mulsemedia languages and standards. Among existing mulsemedia solutions, our work is unique in that it explores the application level proposing a novel mulsemedia conceptual model.

An Approach for Authoring Mulsemedia Documents Based on Events

Multimedia applications are usually composed by audiovisual content. Traditional multimedia conceptual models, and consequently declarative multimedia authoring languages, do not support the definition of multiple sensory effects. Multiple sensorial media (mulsemedia) applications consider the use of sensory effects that can stimulate touch, smell and taste, in addition to hearing and sight. Therefore, mulsemedia applications have been usually developed using general-purpose programming languages. In order to fill in this gap, this paper proposes an approach for modeling sensory effects as first-class entities, enabling multimedia applications to synchronize sensorial media to interactive audiovisual content in a high-level specification. Thus, complete descriptions of mulsemedia applications will be made possible with multimedia models and languages. In order to validate our ideas, an interactive mulsemedia application example is presented and specified with NCL (Nested Context Language) and Lua. Lua components are used for translating sensory effect high-level attributes to MPEG-V SEM (Sensory Effect Metadata) files. A sensory effect simulator was developed to receive SEM files and simulate mulsemedia application rendering.

Modeling sensory effects as first-class entities in multimedia applications

Synchronization of sensory effects with multimedia content is a non-trivial and error-prone task that can discourage authoring of mulsemedia applications. Although there are authoring tools that assist in the specification of sensory effect metadata in an automated way, the forms of analysis used by them are not general enough to identify complex components that may be related to sensory effects. In this work, we present an intelligent component, which allows the semi-automatic definition of sensory effects. This component uses a neural network to extract information from video scenes. This information is used to set sensory effects synchronously to related videos. The proposed component was implemented in STEVE 2.0 authoring tool, helping the authoring of sensory effects in a graphical interface.

Semi-automatic synchronization of sensory effects in mulsemedia authoring tools

This proposal consists of adding a new language module named Users to provide multi-user support in NCL, proposing three new NCL elements: userBase, userAgent and userProfile. user-Base is used for declaring user and their profiles. userAgent identifies a user and userProfile represents a description of the user’s profile. Each userAgent is associated with only one userProfile, and together, they allow representing a unique user. The user description with its specific characteristics can be loaded from a profile specified with the Resource Description Framework (RDF) standard, for example.

/pdf/providing-multi-user-in-ncl-with-useragent-and-userprofile-53t60kubvy.pdf

Raphael Abreu

Papers

Modeling sensory effects as first-class entities in multimedia applications

Toward Content-Driven Intelligent Authoring of Mulsemedia Applications

Extending Ginga-NCL to Specify Multimodal Interactions With Multiple Users

Semi-automatic synchronization of sensory effects in mulsemedia authoring tools

Providing multi-user in NCL with userAgent and UserProfile