Realization of purposive goals in complex environments, especially in applied robotics (e.g. service, manufacturing, and unmanned vehicle applications), impose stringent demands on robots in terms of being able to cope with and react to dynamic situations using limited on-board computational resources, incomplete and often corrupted sensor data. High-level cognitive competencies encompassing knowledge representation, perception, control, and learning are considered essential elements that will allow robots to perform complex tasks within highly uncertain environments. New more successful implementations of concepts already presented in literature, but not implemented with exhaustive experimental methodology, run the risk of being ignored, if appropriate benchmarking procedures, allowing to compare the actual practical results with reference to standard accepted procedures, are not in place.
It is a well-known fact that current robotics research makes it difficult not only to compare results of different approaches, but also to assess the quality of individual research work. Some steps have been taken to address this problem by studying the ways in which research results in robotics can be assessed and compared. In this context the European Robotics Research Network EURON has as one of its major goal the definition and promotion of benchmarks for robotics research. As part of this goal, it has funded a Special Interest Group on Good Experimental Methodology and Benchmarking in Robotics. Similarly, the Performance Metrics for Intelligent Systems (PerMIS) Workshop series has been dealing with similar issues in the context of intelligent systems. The main purpose of this workshop is to contribute to the progress of performance evaluation and benchmarking, focusing in intelligent robots and systems, by providing a forum for participants to exchange their on-going work and ideas in this regard. These objectives are pursued in a context where there is not a wide agreement on the concepts like ‘autonomy’, ‘cognition’ and ‘intelligence’.
The emphasis of the workshop will be, then, on cognitive solutions to practical problems. These cognitive approaches should enable an “intelligent” system to behave appropriately in real-world scenarios in various unstructured application domains. In the context of this workshop, we define intelligence as “the ability to act appropriately in an uncertain environment, where appropriate action is that which increases the probability of success, and success is the achievement of behavioral goals” (J. Albus, “Outline for a Theory of Intelligence”, IEEE Trans. on Systems, Man, an Cybernetics, Vol. 21, No. 3, May/June 1991).
Another key issue will be a capability-led understanding of cognitive robots: how to define shared ontologies or dictionaries to discuss robotic cognitive systems in terms of their performance, relationships between different cognitive robotics capabilities, requirements, theories, architectures, models and methods that can be applied across multiple engineering and application domains, detailing and understanding better the requirements for robots in terms of performance, the approaches to meeting these requirements and the trade-offs in terms of performance. The proper definition of benchmarking is related to the problem of measuring capabilities of robots in a context in which, in many cases, the ‘robotics experiments’ themselves are difficult to ‘replicate’.
We welcome any topic relevant to benchmarking and performance evaluation in the context of cognitive solutions to practical problems, such as:
The workshop is the third in the series after two successful and well attended workshops in IROS’06 and IROS’07. The primary audience of the workshop is intended to be researchers and practitioners both from academia and industry with an interest in cognitive robotics and how these approaches can be utilized in generating intelligent behaviors amidst uncertainty for robots in the service and commercial sectors. The workshop is also aimed at benchmarking and objectively evaluating performance of such robots. Accordingly, it is envisioned to be useful for anyone who has an interest in quantitative performance evaluation of robots and/or robot algorithms.