This area is a comparatively young field in which many new ideas and researchers are being incorporated and, at the same time, it is an area of increasing importance and rapid expansion in terms of people and funding as suggested by the EURON research roadmap.
Defining benchmarks and their associated metrics is perceived by experts in the field as both very interesting and certainly possible, since there are so many proposals for system designs of networked robots, that it would be mandatory to compare performances. Otherwise everybody would get lost in all the available design proposals. This initiative was received as very timely as this area is expanding to accommodate or integrate with other areas as well, such as ambient intelligence.
These benchmarks should include both datasets available in a public repository together with well-defined tasks and associated rules to be performed in a way similar to robot competitions. Also, suitability of the one —datasets— or the other —competitions— mainly depends on the given tasks. It is felt that the issue deserves further discussion. The emphasis should be in the comparison of different approaches/algorithms running on different hardware. Interest is increasing to form a working group or forum on networked robotics to deal with these issues.
Since networked robotics covers a number of areas, so a number of different benchmarks might be appropriate —e.g. online robots. In this sub-area one performance-oriented benchmark would be the time required for a remote robot system to respond to an instruction from an operator. A server could be set up that aimed to 'attack' a remote robot site with instructions (good and bad) and get a measure of reaction/return to the operator’s site. Such a site could be set up for different types of networked robotics scenarios. In this domain, performance is very much influenced by link bandwidth and inherent link delays, a fact that should clearly be taken into account by the benchmarking procedure in order not to render the validity of the benchmark useless.
This was also the topic of one of the workshops at the last ICRA'08 in Pasadena, entitled: Network Robot Systems: benchmarks and platforms toward Human-Robot Interaction, as a continuation of a successful discussion in the workshop in IROS'07 for the establishment of common testbeds or benchmarks, to extend this discussion to bring together the technological aspects of 'network robot systems' (such as a common testbed) with the human-centered aspects of Human-Robot Interaction.
For further details see the official website.