Benchmarking is increasingly important to autonomous robotics. To go out of the laboratories and become real products, robots need benchmarks: standardised, objective ways to characterise, measure and compare their performance in a modular and composable way.

Users -from system integrators to consumers- need objective evaluation of components to choose products that meet their needs; industry needs composable benchmarks to predict the performance of component-based solutions at design time; research needs benchmarks to compare novel approaches with established references.

Plug&Bench logo

Plug&Bench expands the RobMoSys Ecosystem with new elements for the definition of standardized and easy­ to ­use performance benchmarks. Plug&Bench's models let experimenters define, implement and run benchmark by building on a formalised framework. This eases their work, avoids ad hoc solutions and opens the way to a modular and composable evaluation of systems.

Execution of a proper benchmark is a scientific experiment: a benchmarking experiment. The reproducibility and repeatability of the benchmark make different executions (possibly by different people, at different times, on different systems) comparable. Shared metamodels and models are important to achieve this result.

The methodological foundations of Plug&Bench are described in this document. They inherit from a successful line of European projects about robot benchmarking comprising RAWSEEDS (FP6), RoCKIn (FP7), RockEU2 (H2020) and SciRoc (H2020).

The Plug&Bench Benchmark Metamodel (Figure 1) is a new element in the set of RobMoSys Composition Structures. It defines the Benchmarking Component, i.e. an extension of the Component-Definition Metamodel providing all the elements needed to describe and execute a benchmark in a standardised manner.

You can download the Plug&Bench Benchmark Metamodel from this repository. This note explains its contents, while the accompanying document describes the metamodel.

Figure 1: ecore class diagram of the Plug&Bench benchmark metamodel

Plug&Bench also adds a new element to the set of RobMoSys Roles in the Ecosystem: the Benchmark Developer. A description of all the connections between Plug&Bench and the RobMoSys Ecosystem is provided by a specific document.

Plug&Bench provides a Benchmark Engineering Tool supporting the Benchmark Developer in building benchmark models based on the Plug&Bench Benchmark Metamodel.

A Virtual Machine with all prequisites already installed for the integration with SmartMDSD is also available on https://owncloud.fraunhofer.de/index.php/s/N9PblHWzYKmBcCj

Yes. By using the Benchmark Engineering Tool, three benchmark models have been defined:

• Screw-hole Localizer benchmark

• Trajectory Planner benchmark

• Trajectory Follower benchmark (see Figure 2)

Figure 2: structure of the Trajectory Follower benchmark

Plug&Bench benchmark models can be downloaded from this repository.

Plug&Bench is an Integrated Technical Project (ITP) of EU H2020 project RobMoSys robmosys.eu. The Consortium of Plug&Bench comprises Politecnico di Milano https://www.polimi.it/en/ and Fraunhofer IPA https://www.ipa.fraunhofer.de/en.html.

This activity has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 732410.