This was the main joint event in 2006 devoted to benchmarking in Europe. It was held in Palermo on March 18, 2006, taking advantage of the First European Robotics Symposium (EUROS) and the Annual EURON Meeting. It was well attended with over 25 participants.
The main purpose of the workshop was to provide an informal forum for participants to exchange their ideas about benchmarking in robotics, as well as presenting and discussing on-going relevant initiatives.
The workshop was intended as a working meeting wit the following structure:
Speaker: Angel P. Del Pobil
Title: The need of Benchmarks in Robotics Research
Abstract:
Current practice of publishing research results in robotics makes it extremely difficult not only to compare results of different approaches, but also to asses the quality of the research presented by the authors. This state of affairs cannot be changed in the short term, but some steps can be taken in the right direction by studying the ways in which research results in robotics can be assessed and compared.
Speaker: Javier Minguez, University of Zaragoza, Spain
Title: Metrics to evaluate motion techniques from the resulting motion point of view (mobile robotics perspective)
Abstract:
Historically, many techniques have been designed to address autonomous collision-free motion (sensor-based motion with obstacle avoidance). For example (Khatib, O.,1986), (Borenstein, J. & Koren, Y., 1991), (Fox, D.; Burgard, W. & Thrun, S., 1997), (Simmons, R., 1996), (Fiorini, P. & Shiller, Z. ,1998), (Minguez, J. & Montano, L., 2004) among many others. It is clear that under the same conditions each technique generates a different motion. Nevertheless, questions like: which is the most robust one? or which of them behaves better in a determined context or condition? They cannot be answered neither from a scientific nor technological point of view. In other words, once we face a mobile robotics application, selection of a motion technique among all the existing ones is a matter of specialists and not accessible to everybody. This is because there is no objective comparisons of methods neither quantitative (in terms robustness or action parameters of such as the time or the total distance travelled) nor qualitative (in terms of security of the motion). At present, there is only an experimental comparison (Manz, A.; Liscano, R. & Green, D., 1991). Nevertheless, this comparison is very old, and thus, it does not include the advances in this subject in the last 15 years. Furthermore it is based on the observation and does not present a rigorous and objective methodology to address this objective. The scientific objective of this part is the development of a procedure to evaluate the motion techniques from a quantitative and/or qualitative perspective of the motion generated in different contexts.
Speaker: John Hallam, Syddansk Universitet, Denmark
Title: Experimental Robotics
Abstract:
The purpose of the talk was to promote good experimental methodology among roboticists. This would contribute to help pave the way for the widespread practice of benchmarking in robotics.
Speaker: Domenico G. Sorrenti, University of Milano-Bicocca, Italy
Title: Image processing and computer vision benchmarking in robotics
Abstract:
Computer vision (and image processing algorithms) is very important in robotics, as at the moment cameras seem both prospectively as low-cost as required and capable to supply rich data about the robot workspace. Therefore benchmarking of computer vision (and image processing algorithms) is important for roboticists; we think it is important to have a distributed awareness about the pros, cons and actual expectable capabilities of each algorithm in each situation. In some we probably have plenty of alternatives, in some just one and in other we have no viable algorithm for robotic use. The underlying issue is that generation of sensed data for our robots is done most of times by a processing pipelline of gigo type (garbage-in-garbage-out, at each step). We propose to prepare datasets, for relevant sub-domains, relevant algorithms, etc so that the community can ask new robotics algorithms, as described in the papers, to be validated with those datasets. In particular we are interested in omnidirectional vision and also in robustness to light changes, highlights, strong shadows, etc.
Speaker: Domenico Prattichizzo, Università di Siena, Italy
Title: Remote Visual Servoing Benchmarks Sharing the Same HW Setup
Abstract:
We describe current work aimed at the extension of our Automatic Control TeleLab so that it can be used as a tool for benchmarking visual servoing techniques by sharing remotely the same hardware set-up.
Speaker: Claudio Melchiorri, University of Genoa, Italy
Title: Benchmarks for Manipulation and Grasping
Abstract:
We present results of discussions and some proposals for benchmarking in the context of robot manipulation and grasping
Speaker: Javier Mínguez, University of Zaragoza, Spain
Title: Metrics to evaluate the motion techniques based on the biomedical response of the user (biomedical perspective)
Abstract:
In the healthcare or clinical fields, any proposed intervention (therapeutic or preventive) has to take into account the impact over the patient quality of life. This is not only to improve their health, but also the subjective perception of welfare. In this sense, the scenarios have to be conceived and validated explicitly in terms of compatibility with the users-patients welfare. Thus, for example, a motion technique can be robust and move the vehicle with optimal trajectories in time, distance, etc. But when a robot guide or transports a human, a basic question is how does the user feel during the movement? In the development of healthcare technologies the user satisfaction is an essential specification. Therefore, it is required to measure and to evaluate systematically the quality of the user-device interaction and the reaction of the individual to a motion technique particularly. This fact has a vital importance in the growing demand of autonomous robots for human transportation, specially for those designed for aging users and/or that spend a great time using the vehicles. The objective is the development of a procedure to evaluate the reaction of the user to the motion generated by the vehicle. In biomedical sciences, one way to evaluate the human response to external events is to study the biomedical (physical and psychological) activity during the development of a task. Other methodologies include the filmed observation, speech techniques, etc. By using these methodologies, the objective is to develop a way to evaluate and thus to create a metric for the existing motion techniques obtained from the user reactions.
Speaker: Stefano Caselli, University of Parma, Italy.
Title: The case for repeated research in Robotics
Abstract:
The main point of my presentation is that repeated experimental research should be appreciated and properly rewarded. Among its benefits we have:
The discussions focused on four axis:
Participants in general found the workshop very interesting in that it added significant insight to their perspective of benchmarking. The only thing that some people missed was the discussion in different working areas. That was planned but turned out to be impossible with an audience composed of many researches of very different topics. However, since small groups would generate very interesting discussions for interested and motivated people, who are the intended structure for more focused events.
Also it seems difficult to consider benchmarking in the same way for "low level" control issues and more "high level" planning and decision processes (as collision avoidance, vision, localization...). It was felt that implementation on a real experimental platform is required to compare real performances of "low level" control schemes (and not only Simulink simulation, even though obviously it gives significant information). But real implementation costs a lot in terms of equipments and time, and it requires very dedicated expertise. Another limitation is that it is robot dependant (frictions and other nonlinearities), thus in many cases the benchmark should be done on the same platform to get relevant and comparable results.