Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - EUROBENCH (EUropean ROBotic framework for bipedal locomotion bENCHmarking)

Teaser

Human-centred robots such as prostheses, exoskeletons and humanoids, are becoming increasingly relevant worldwide. Many prototypes are moving out of the lab into everyday applications in a wide range of market domains. Several roadblocks exist in this process. Some of these...

Summary

Human-centred robots such as prostheses, exoskeletons and humanoids, are becoming increasingly relevant worldwide. Many prototypes are moving out of the lab into everyday applications in a wide range of market domains. Several roadblocks exist in this process. Some of these are technical, while others are related to the lack of reliable performance/safety indicators for these devices to meet international certifications and standardization requirements. The Robotic Research Agenda has emphasized benchmarking as an important instrument to assess the Technology Readiness Level (TRL) and to quantify how robotic solutions match user needs. Recent international efforts (e.g. RockIn, Robocup, Cybathlon, DARPA Robotic Challenge) have confirmed the interest of the scientific and industrial communities in evaluating (and comparing) the performance of robotic systems in real-life environments.
However, a consolidated benchmarking methodology for Robotics does not exist yet. In bipedal robotics, current benchmarking approaches are focused on generic, goal-level performance indicators. They are usually based on competitions, which can hardly describe performance in sufficient detail to allow recognising and quantifying the technical causes of performance. When existing, metrics are numerous and overlapping, and experimental procedures are defined from each lab independently and chosen according to lab-specific functionalities. This situation undermines the basis of scientific (and industrial) progress, i.e. the evaluation of the state of the art, and building on other’s research works. This situation is aggravated by the lack of standard terminology, databases, regulations across domains and research areas.
In order to fill these gaps, EUROBENCH aims to create the first European framework for the application of benchmarking methodology on robotic systems. The framework, specifically focused on bipedal robotic technologies, will include methods and tools to measure system abilities on a rigorous, quantitative and replicable way. The successful achievement of this goal will put Europe at the forefront of the evaluation of robotic systems, facilitating the process of bringing innovative robotic technologies forward to market.

Work performed

The EUROBENCH project achieved the following results from the beginning of the project:
- End-users and stakeholders’ needs survey completed.
- In-depth analysis of the international State of the Art of the assessment of robotics technologies.
- Priorities on the system ability levels established.
- FSTP procedures created, taking into account the results of the survey and SoA analysis and the experience of other Robotics Cascade Funding initiatives.
- Collaboration and interaction with other EU initiatives initiated.
- Technical and business success indicators defined.
- Technical specifications of the benchmarks outlined.
- Contents and software architecture of the benchmarking algorithms defined, and a first part implemented.
- Existing human reference data for the relevant motions identified and made available to the project.
- Development of algorithms for the calculation of walking Performance Indicators initiated.
- Data Format for recorded variables defined.
- A first mock-up of the user interface and first version of the database created.
- FSTP-1 Open Call successfully closed, 17 projects selected and initiated.
- Development of testbeds from both UROBENCH Consortium and Third Parties planned and initiated.
- Assembly of the humanoid robot Reem-C finalized.
- Facility of Wearable Robots designed and in process of construction.
- Facility of Humanoids designed and in process of preparation.

Final results

The EUROBENCH project aims to reach the following global objectives, each representing an advance on the State of the Art in the evaluation of robotic performance:

1. Achieving an international consensus in the industrial and academic community on the benchmarking requirements. The involvement of the international community is a required step to convert benchmarks into future standards. The EUROBENCH project is leading several efforts to create a worldwide network of industrial and academic entities in discussing on the specific requirements on standardization and terminology of robotic systems. This is expected to facilitate the inclusion of good practice and regulations for the smooth introduction of robotics systems in the society.

2. Identifying a scientific method to quantitatively assess the system abilities of bipedal robots and establish the causal relationships with the technologies composing them. The EUROBENCH is gathering together algorithms and data from many Third Parties involved through the cascade funding Open Calls. This massive software and data sharing has been never achieved before in the bipedal locomotion field, and is expected to advance the knowledge and habilitate to application of artificial intelligence algorithms to establish reliable predictors of robot performance, and help developers identify technical shortcomings efficiently.

3. Providing practical tools to industry and academy to efficiently test robotic platforms at any stage of development, from early prototyping to commercial products. The novel value proposition of EUROBENCH stands on three main pillars, i.e. facilities, benchmarking algorithms, and datasets, which are being designed to be easy accessible by anyone worldwide. Current Testing facilities (in Europe and US) only provide very simple benchmarks, based on completion time and do/not-do metrics. EUROBENCH, instead, will include a comprehensive set of test benches and metrics to evaluate bipedal robotic performance under a multi-faceted perspective. This will increase the adoption of benchmarking methods as a common practice, and therefore permit to achieve systems with higher quality and lower costs.

4. Achieving an effective utilisation of the benchmarking methodology in the bipedal robotics community and an efficient transferability to other application domains. The main advance in this respect will be to have flexible tools, focused on sub-tasks, which can be used across technological fields and application domains. For this reason the Benchmarking Software and Database will be modular and prone to be extended over time. This approach is not in conflict with the competition approaches, but represents a complementary tool that can help researchers to find causal relationships between the performance during competition and the performance in each of the sub-functions, taken separately.

5. Create a sustainable “benchmarking infrastructure” for the European robotics community. The EUROBENCH project contains a sustainability plan that aims to convert the EUROBENCH outcomes (facilities, software, database) into a business model, based on pay-per-use model, which will offer users different kinds of services. This model, partly based on existing Testing Facilities (e.g. NIST-RACE), includes additional services U8e.g. calculation of Perfromance Indicators, or access to a common Database) which have never been offered before by any other facility for Robot testing worldwide.

Website & more info

More info: http://eurobench2020.eu/.