Develop a component-based, composable benchmark system within the RobMoSys ecosystem
Plug&Bench’s main goal is to provide the RobMoSys framework with new elements enabling the definition of standardized and easy to use performance benchmarks.
Benchmarks defined using Plug&Bench models and tools will, in turn, allow meaningful comparisons between components offering the same functionality, and act as a foundation for the benchmarking of complete robot systems. The first result of Plug&Bench is a Benchmark Meta-Model integrated with the other Meta-Models in the RobMoSys EcoSystem, on which all other project results will be built.These will be validated first, developing demonstrator benchmarks and components, then by actually benchmarking the latter using the former.
Plug&Bench defines 3 categories of benchmarks:
- Type 1, i.e. dataset-driven (non-interactive)
- Type 2, i.e. simulations
- Type 3, i.e. physical
Demonstrator benchmarks developed by Plug&Bench will cover only 2 of the previous.
Plug&Bench is the direct descendant of a chain of successful European projects focusing on benchmarking, including RAWSEEDS (FP6), RoCKIn (FP7) and RockEU2 (H2020): the latter for what concerns the European Robotics League benchmarking competition.
By bringing the model-driven approach of RobMoSys to the field of robot benchmarking, Plug&Bench aims at providing it with a set of useful tools and a well-defined approach to benchmark design. On the other hand, Plug&Bench augments the RobMoSys EcoSystem with the elements required for performance evaluation and comparison of components and systems.
