Survey and Inventory of Current Efforts in
Comparative Robotics Research

• Introduction
• Organized Workshops
• Robot competitions
• Benchmarking Initiatives in Europe
• Related Workshops
• References

GEM: Good Experimental Methodology

Introduction

This initiative aims at promoting better experimental practice in robotics research wand help pave the way for the widespread practice of benchmarking in robotics. Often, the current practice of publishing research results in robotics makes it extremely difficult not only to compare results of different approaches, but also to assess the quality of the research presented by the authors. Frequently, when researchers claim that their particular algorithm or system is capable of achieving some performance, those claims are intrinsically unverifiable, either because it is their unique system or just because a lack of experimental details, including working hypothesis. Often papers published in robotics journals and generally considered as good would not meet the minimum requirements in domains in which good practice calls for the inclusion of a detailed section describing the materials and experimental methods that support the authors' claims. This topic raised interesting discussions about the way that researches in robotics perform the experiments and the validity that they have. This is very important in other disciplines such as medicine and some ideas could be imported. It was suggested to create a protocol of "writing" the papers involving a given protocol to "experiment" with the robots. The idea was to implement this in the future papers trying to win homogeneity and rigorous validation processes.

This initiative has advanced considerably in 2007 and 2008 with increasing support from EURON community. Two major actions have taken place:

• The topic of the EURON 2007 and 2008 workshops at the Annual Meeting this year included Experimental Practices and benchmarking. A full account of the topics discussed in these workshops can be found above in this document.
• A proposal for a Special Interest Group (SIG) in EURON was accepted with the focus of increasing the quality of experimental methodology practiced in robotics by sharing good practice via educational workshops, summer schools, email discussion and web presentation; by providing assistance to journal and conference reviewers and editors concerning what constitutes experimental robotics and good practice in that sub-discipline; by encouraging the principled replication and comparison of results; and by encouraging the development and use of appropriate systems benchmarks and standard evaluation procedures.

A number of SIG meetings were held to work on these issues, namely:

• Benicassim, Spain, 9/1/2008, collocated with RISE08
• Genova, Italy, 31/1/2008,
• Zaragoza, Spain, 4/3/2008
• Prague, 28/3/2008, as part of the EURON Annual meeting
• Valencia, Spain, 30/4/2008

The main result of these efforts was a document entitled General Guidelines for Robotics Papers Involving Experiments enumerating recommended quality criteria for Robotics Journal/Conference reviews that the community expects a proper high-quality experimental robotics paper have to satisfy. This is formulated as a reviewer checklist and circulated to Journal Editors and Conference Programme Committees. This documente can be downloaded from here. It has already been sent to the Editor in Chief of the highest impact robotics journal: the IEEE Transactions on Robotics. The TRO Senior Editorial Board discussed it in its last meeting in Pasadena (USA) in May 2008. We plan to send to other journal editors.

 

© Euron - European Robotics Search Network