Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - MECANEX (Multimedia Content Annotations for Rapid Exploitation in Multi-Screen Environments)

Teaser

Every day, the need for enriched multimedia content is becoming ever more prevalent. Modern audiences appear willing to immerse into more interactive and personalised viewing experiences, through second-screen environment and enriched video content. With production houses...

Summary

Every day, the need for enriched multimedia content is becoming ever more prevalent. Modern audiences appear willing to immerse into more interactive and personalised viewing experiences, through second-screen environment and enriched video content. With production houses, broadcasters, advertising agencies and online publishing companies seeking to better reach their audience with relevant, highly targeted content, the need for innovative tools that aid them in this task is a requirement rather than a desire. The adaption of enriched multimedia content to multi-screen environments, which enable the automatic porting to different target platforms, such as regular web pages, mobile pages and mobile apps, as well as TV applications, is a significant asset for the global market of enriched multimedia content. The objective of MECANEX project is to provide the necessary solutions to these problems, to devise tools for automatic annotation and editorial support of multimedia content, extraction of personalized information and adaption of enriched multimedia content in multi-screen environments. Furthermore, the major novelty of MECANEX is the creation of SaaS (REST based) toolkit that provides the aforementioned functionalities by bringing together complementary expertise and technology of different European companies and research institutes, in order to provide a cost effective multi-screen video metadata enrichment solution for SMEs. Such tools will be adopted by production and post-production companies, as well as by content providers or relevant service oriented companies and will be adapted into the creation process. MECANEX will offer SMEs the ability to semi-automatically enrich video collections for multi-screen application, as well rapid prototype multi-screen concepts and test related interaction models that eventually will improve the chance for SMEs to develop more solid business models, marketing and advertising campaigns. Finally, the developed social and personalization mechanisms, allows for the collection of consumers’ preferences and provide feedback and input to the aforementioned tools. In a nutshell, automatic annotation, editorial, multi-screen and social and personalization tools have been integrated into a working MECANEX toolkit ready for the production of multimedia content, as well as the advertising agencies.

The strategic key objective of MECANEX is to provide innovative tools and methods, towards:
➢ enabling the automatic annotation and editing during the creation, production and post-production of multimedia content that will allow SMEs, such as production houses, broadcasters, advertising agencies, online publishing companies, etc. to create and enrich multimedia content ready for multi-screen environments as well as content ready for marketing campaigns.
➢ supporting search and retrieval mechanisms from libraries with existing enriched multimedia content that could be used as building blocks of a fast innovative creative process.
➢ pairing created metadata with multimedia content allowing easy creation and access to relevant educational and recreational content, e.g. wikipedia pages of landmarks appeared in a video, images of actors or staff participating, relevant trivia, historic events affecting the story etc.
➢ providing a multi-screen toolkit for the development of prototypes that enable an automatic porting to different target platforms, such as regular web pages, mobile pages and mobile apps as well as TV applications.
➢ gathering from end users/consumers, processing and delivering personalized information, which will facilitate the production and editing process, while at the same time will allow advertising & marketing brokers to automatically ingest relevant ads and marketing data via coupling user preferences with the details of the playing multimedia content. End users participate within this process providing feedback towards both the enrichment of metadata by manually anno

Work performed

The concept of the MECANEX emanates from the need of relevant SMEs to slim line the production both of annotation streams and personalization data towards the rapid development of new products as well as the enhancement of existing ones. Towards this direction our efforts focused on producing a toolkit collection of the appropriate tools allowing easy integration and exploitation of MECANEX results from the participating SMEs in the near future. We have developed the MECANEX concept into five parallel pillars as follows:

A. Multimedia content analysis, automatic annotation tools and mechanisms, aiming into developing REST based web services for the automatic annotation of existing or even newly created content in a general way, as well as in in specific way for multi-screen context. Such tools allow the adaptation of automatic annotation processes into the creative process from Production and Post-Production Companies as well as Content Providers or relevant service oriented companies. Search and retrieval tools aim to build upon the automatic annotation tools by providing a REST based web service for the search and retrieval of content. Content can be retrieved based on specific criteria characteristics of the video allowing the fast creation of informational and demo clips based on existing productions. The Automatic Annotation tool enables the analysis and automatic annotation of multimedia content that was collected within the project from various partners\' sources (archiving multimedia content). Advanced automatic annotation algorithms constitute the Automatic Annotation tool (AAT), which enable the enrichment of multimedia content with automatic annotations, related text, as well as multimodal information.

B. Editorial tool provides the enhanced flexibility to edit the already enriched metadata via supporting functionalities like accepting, ignoring, extending them. The editorial tool ensures the quality of the newly created meta-data. The output of editorial tool provides feedback to the multimedia content analysis and automatic annotation processes. This tool supports mainly the needs of online publishing companies and also production houses and broadcasters by its provided feedback.

C. Multi-screen tool provides a decision engine, which takes the enriched metadata, advertisements, live detected social data and information of the current context and turn it into a scenario to be displayed on the available screens at any moment. This is an active cycle, so the multi-screen tool sends signals of usage back to the social and personalization tool in order to be enhanced further while watching and interacting with the content and other viewers.The Multi-screen tool performs synthesizing and previewing of the enriched multimedia content with the respective annotations and metadata in multi-screen environments.

D. Personalization data aggregation and relevance feedback, aims to provide a framework for connecting users’ responses and comments with content streams and annotation data. The provided tools allow easy and fast interconnection of media streams with users clusters and preferences allowing SMEs to provide customizable personalized services through promotions, advertisements and marketing campaigns as well as personalized multi-screen services. Moreover, social and personalization tool provides feedback to the automatic annotation, search and retrieval processes, as well as to the multi-screen tool. The Social and Personalization (SP) tool is designed towards assessing users’ experience with the enriched multimedia content, as well as its success in the corresponding advertising and marketing campaign. User-generated tags, personal information provided by the end user and relevance feedback data are collected towards constructing users’ profiles and facilitating a novel personalization mechanism. Two basic workflows of the SP tool are designed which are differentiated based on the type of end user, i.e. Busi

Final results

Key advances contributed by MECANEX are briefly summarized in the following technological areas and categories:

• Multimedia Content Analysis
Multimedia content analysis functionalities will allow “small pieces” of information from different modalities to complement each other at various levels of the content analysis process in order to provide reliable input for reasoning at the highest level. Moreover, these functionalities will support the combination of features from different modalities in multi-modal content models via extracting content-related information from multimedia data. These functionalities will facilitate the production process within production houses and broadcasters, as well as will allow the fast and easy creation of marketing campaigns by the advertising agencies.

• Automatic Annotation
Automatic annotation functionalities will enable the enrichment of the multimedia content with supplementary information, based on advanced algorithms, techniques and methodologies. The multimedia content will be fully annotated, thus providing the ability to the production houses, the broadcasters and the advertising agencies to maximize their profit and enhance their production and advertising process, as well as the end users to exploit and utilized the annotations in order to improve their experience.

• Search and Retrieval of Multimedia Content
Search and retrieval functionalities will support the fast and easy creation of new multimedia content by the production houses and the broadcasters, as well as the creation of marketing campaigns by the advertising agencies. Based on these functionalities the existing enriched multimedia content by the annotation process will act as building block in order to create new content. Therefore, the production process will be facilitated and accelerated, thus providing great benefits and profit to the related industrial area.

• Editorial selection and enhancement of meta-data
Having editorial tools as part of the enrichment workflow will bridge the gap between fully automatic systems that are nice for research but can’t be trusted by content creators to be taken into production and manual enrichment tools that have a too high a overhead for content creators. The novel element is having a manual (editorial) phase that is much quicker by having the automatic system act as a helper not a replacement for editors.

• Multi-screen Environments
The developed toolkit does not need to be adapted for different output devices but adapts dynamically to them. The novelty is that it also adapts to the available devices during the consuming of the content. It allows for the first time to create multi-screen applications that react to changes in their environment like people joining/leaving with their phones, tablets and other devices.

• Relevance Feedback,Personalization and Social Recommendation
MECANEX provides a social and personalization tool, which will support the advanced mechanisms of relevance feedback, personalization, and social recommendations. The output of this tool will be personalized information, which will be utilized by production houses’, broadcasters’, advertising agencies’ and annotation and online publishing companies in order to enhance the functionalities and the whole process during the production and editorial process within multi-screen environments. The above mentioned type of companies and the corresponding market, where these companies target to, will have great benefits and profit by exploiting the personalized provided information by the SP tool, due to the fact that they will be able to target to the specific needs, preferences and habits of users.

Overall MECANEX is expected to contribute to improving innovation capacity and integration of new knowledge in European creative industries, strengthening their competitiveness in the European and international markets. Specifically:
a) reinforces the position of the European ICT and digit