#	Pagina
attuale pagina	/open-h2020/projects/194225/results.html

Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - AutoPost (Deformable Surface Tracking and Alpha Matting for the Automation of Post-production Workflows)

Teaser

Summary

The H2020 EU project â€œDeformable Surface Tracking and Alpha Matting for the Automation of Post-production Workflowsâ€ aims to improve the efficiency of labour-intensive VFX tasks, automating major parts of the daily workload in current post-production and visual effects workflows. The goal of the AutoPost project is to automate major parts of the daily workload in small and medium post houses and with it to make post-production more efficient by reducing time-consuming and costly manual processing. This objective will be achieved by bridging the existing gap between commercially-available post-production tools and state-of-the-art research in the domain of deformable surface tracking and natural video matting, alleviating the operator workload on material that lacks planning with respect to post-production requirements.
The AutoPost project aims to develop new methods applicable to general movie productions filmed without specific constraints, since they constitute most of the problematic, labour intensive footage encountered in daily post-production tasks of small and medium post-houses. To ensure industry adoption, the algorithms will be integrated into â€œThe AutoPost Plugin Suiteâ€, an extensible, close-to-market multi-platform post-production tool targeting regular 2D and 3D productions, Visual effects and 2D-to-3D conversion.

The technical and operational objectives of the project are:
O1. To define the requirements of the tools as determined by the industry needs.
O2. To develop deformable surface tracking methods that estimate temporally consistent surface motion, deformation, and shading changes, even in presence of temporary occlusions under real-world conditions.
O3. To develop natural video matting methods that provide accurate and more realistic mattes for VFX and post-production processes with particular attention to motion blur and deformable surfaces under real-world conditions.
O4. To develop Software Development Kits (SDKs) for the tracking and matting algorithms. The SDKs will be the basis for the development of the Plugin Suite and will come along with functional prototypes that demonstrate their use.
O5. To develop The AutoPost Plugin Suite that will include tracking and matting plugins for industry-standard tools. The Plugin Suite will provide data interfaces that are widely used and accepted by VFX and 2D-to-3D conversion artists and user interaction tools.
O6. To evaluate and demonstrate the developed plugins under real working conditions by SME project partners from creative industries, developers of related post-production tools as well as post-houses and end-users.
O7. To define the strategies and policies for managing and exploiting the intellectual property derived from the activities of the project and to define and implement a business exploitation model and its corresponding business plan.

Work performed

The main scientific and technical results of the AUTOPOST project can be grouped in the following categories:

Functional and technical requirements
------------------------------------------------------
The main objective in this area was the definition of requirements related to the tracking and matting tools from a userâ€™s perspective as well as the specification of interoperability interfaces with the aim to guarantee that the solutions produced could be integrated into existing postproduction platforms and are effective in real productions.
As a first stage, the most important use cases where efficient matting and tracking tools are required for visual effects were identified and reviewed. The use cases were derived from common visual effects tasks and actual projects that have been executed by the post-production partners. In order to come up with a representative list of use cases the world-wide connections with other post-production companies were also exploited. Identified use cases included the tracking of rigid objects (e.g. changing a logo on a coffee mug), or deforming surfaces (e.g. adding a tattoo on a body part or digital make-up), extracting people with motion blur and heads with hair, enhancing or replacing the background of a scene (e.g. adding snow to a landscape), or selecting objects for colour grading and finishing.
Based on these use cases the functional and technical requirements and the initial specification of the AutoPost surface tracking and alpha matting tools was derived. Taking advantage of the consortium members experience with existing tools and methods, as well as research on a wide scope of potential applications, typical user requirements were identified and transferred into required features and specifications of the interfaces and data formats. These were reviewed during the project based on systematic user evaluations of existing tools and workflows, discussions and collaborative work between the consortium members and advancements in the tracking and matting algorithms/plugins.

Extension and adaptation of state-of-the-art algorithms
-------------------------------------------------------------------------------
Existing state-of-the-art algorithms for deformable surface tracking and natural video matting were redesigned, extended and adapted to meet the user needs and requirements defined in the project in order to make them suitable for post-production environments and tasks. Data interfaces were designed to conform to established industry standards for easy adoption. Furthermore, programming interfaces were designed to allow for easy integration into SDKs and eventually plugins, including means for user interaction. Incremental versions of both tracking and matting algorithms were developed and implemented into the corresponding SDKs.
An improved algorithmic tracking framework, providing fully deformable surface tracking functionality as well as low level programming interfaces for configuration and data interaction, was developed to improve both the effectiveness and the efficiency of tracking tasks. The new algorithm operates on arbitrary continuous surfaces and is not limited to planarity or rigid motion. Also, shading and motion blur, two of the most common changes in scene content, are incorporated into the pure geometric displacement estimation. Last but not least, a variety of parameters and interactions are available allowing to achieve better results by bringing the user into the loop.
An improved algorithmic matting framework was implemented to improve and extend the state-of-the-art in natural matting for video sequences. This framework uses effectively motion information to automatically propagate predefined trimaps to assist the user through the whole sequence, to incorporate temporal coherence between alpha mattes and to deal with blurred mattes produced by objects in motion. Effective interaction tools allow for semiautomatic correction of possible errors in propa

Final results

The AutoPost Plugin Suite aims to boost the competitiveness of the European post-production industry in the global market, targeting specifically small and medium post-production companies. AutoPost has the ambition of improving the efficiency of labour-intensive visual effects task through the development of a multi-platform plugin suite by integrating advanced scientific state-of-the-art solutions for tracking deformable surfaces and natural video matting into common workflows and production pipelines of visual effect artists.
AutoPost has been highly successful in its efforts to meet the complex and ambitious scientific, technological and industrial challenges, and has produced the AutoPost Plugin Suite, a set of close-to-market matting and tracking plugins that go beyond the current state-of the art of the industry and that can be integrated into existing post-production platforms (such as the widely used Nuke), extending them with more sophisticated tools for improved quality and efficiency.
The AutoPost Plugin Suite brings together several innovative features that address specifically the needs of European post houses. These functionalities make the matting and tracking plugins suitable for postproduction workflows and a large variety of visual effects including object and skin manipulation, scene extension and replacement, appearance modification (digital makeup, ageing), compositing and object extraction from arbitrary and natural backgrounds and object selection for colour grading and finishing. In particular, the plugins are applicable to general movie productions filmed without specific constraints such as markers, availability of 3D-models, individual object layers, green screens, which currently constitute most of the problematic, labour intensive footage encountered in daily post-production tasks of small and medium post-houses.
The AutoPost plugins automate and improve the efficiency of visual effects and 2D-to-3D conversion workflows, providing time savings in labour-intensive manual process. Evaluation results have shown that in average an 80% of time savings can be achieved with AutoPost tools, leading to a 20% reduction of the postproduction costs.
AutoPost will not only help post-houses to reduce post-production costs but will also help them to produce higher-quality results thanks to the integration of new advanced technologies. This fact has been validated in tests with real-users, achieving operator approval in more than 80% of the tests.
Furthermore, providing the developed tools as plugins facilitates standardization, minimizes the risk of excessive solution diversification and makes possible a fast and natural arrival to the end users.
The impact from the use of AutoPost tools is expected to go beyond the local markets of European post-houses and reach the global market where infrastructure and labour costs can dictate where films are finally produced. AutoPost technology will help to reduce the weight of the labour cost factor, thus emphasizing the other decision-making factors in which the European postproduction industry is strong.
Indeed, it is expected that the impact on the small and medium post-production companies in Europe also creates a positive effect for the European post-production industry at a general level, given the fact, that the European postproduction landscape is primarily composed of small and medium facilities.
Apart from its potential impact in the competitiveness of the post-production industry, AutoPost also has a positive impact on the scientific community as well as the industry and the market. As AutoPost is an innovation project, the positive aspects amongst the scientific community do not reside particularly in the advancement of the state of the art but in the opportunity to convert research results into solutions for the industry and the strong collaborative relationship that is established between ICT R&D organisations and the media industry. The R&D tasks carri