The robotics community has always recognized as a key area that of the interaction of a robotic manipulation system with its environment. Although this is a very interesting and central issue, the involved problems, both at the technical and methodological level, still represent a limiting factor in a number of important cases not only in standard industrial applications but also in non-conventional uses (e.g. space, soft material, dexterous manipulation, etc.). Among the relevant research fields interested in this area, one can enumerate the following: advanced sensors (e.g. tactile, force/torque, vision, etc.), mechanical structures and devices (e.g. parallel, redundant, flexible, etc.), planning, control, optimization. Due to these reasons, the complexity to advance in the definition of benchmarks is acknowledged as a very difficult task, which is also the case for other areas of robotics, though manipulation and grasping is an area that has also been suggested by the EURON research roadmap, since it is crucial both for advanced production systems and household robots.
Nevertheless, the majority of experts in this field agree that such a definition would be desirable and less difficult to achieve if we try to define a benchmark for each elemental topic within this very large domain: such as mechanical design, control algorithms, sensors, artificial intelligence strategies, etc.
In particular, these tentative benchmarks should include well-defined tasks and rules, methodologies, design for experiments, calibration process for instrumental devices, etc. Moreover, the algorithmic performance should be tested on a variety of different hardware platforms and vice versa. For instance, a hand can perform better than another one, also with the same control algorithm, since it is more suitable for a given task, and so on.
Finally, some ideas about specific benchmarks have been proposed related with the following three tests:
There has been further recent progress based on the test 2 above, particularly in the context of visually guided grasping. This initiative has been developed at Universitat Jaume I and offers a description of a set of experiments on visually-guided grasping of planar objects with a Barret hand. They are made available to the community as a set of standard experiments on which benchmarks and associated performance metrics can be proposed. A first experimental protocol for such a benchmark has been also proposed.
The goal of the experiments is to test different implemented procedures for visually-guided grasping. That is, by only using visual information about planar objects a set of 3-finger feasible grasps are determined and executed to lift the object. The experimental setup for the experiments consists of a stereo head with two gray-scale cameras and a robotic arm with a three-fingered Barrett Hand, which is commercially available and widely used for robotic grasping research.
Some problems are included as part of these experiments: such as constraining of a general three-finger grasp to the particular kinematics of the 4 d.o.f. Barrett Hand or the use of stereo images to locate an object placed in front of the robot, and to obtain the contour of such object.
A second step in this initiative puts forward a protocol based on the previous experiments, once the object is lifted successfully. The robot applies several sequences of shaking movements to the object consisting of rapid accelerations and decelerations of the hand. The result of this stability test, permit to asses the grasp quality in five different categories.
This initiative is described here.
Also additional technical information is included in this document as Appendix X. Part of these ideas are going to be implemented in the context of the new FP7 project GRASP "Emergence of Cognitive Grasping through Emulation, Introspection, and Surprise" in which Antonio Morales from UJI (Spain) is one of the partners.
Benchmarks for manipulation and grasping is also an important issue other new FP7 projects such as DEXMART "DEXterous and autonomous dual-arm/hand robotic manipulation with sMART sensory-motor skills: A bridge from natural to artificial cognition". In this Integrated Project there is a complete workpackage devoted to benchmarking and experiments, as discussed by the WP leader Gerhard Grunwald from the DLR (Germany) at the CogGEMBench'08 Workshop on Good Experimental Methodology & Benchmarks in Cognitive Robotics in Kalrsruhe.