This half-day workshop was held on April 1st, 2008 in Karlsruhe, Germany. It was originally submitted and accepted as part of the Program of the International Conference on Cognitive Systems (CogSys 2008), however the organization decided to cancel all workshops and replaced them by project presentations. We decided to hold the workshop one day before the conference in a nearby hotel room.
As the complexity of current embodied cognitive systems grows, it is more and more necessary to define proper experimental approaches and benchmarking procedures. On the one hand, reliable benchmarks are called for in order to allow the comparison of the many research results in embodied cognitive systems, so that their potential application is eventually possible. On the other hand, if cognitive robotics aims to be regarded as serious science, replication of experiments deserves conscientious attention to verify if and by which measure new procedures and algorithms are real progress. New implementations of concepts in the literature, but not implemented with exhaustive experimental methodology, risk to be ignored, if appropriate benchmarking procedures are not in place, allowing to compare the actual practical results with reference to standard accepted procedures. Both replication and benchmarking are needed to foster a cumulative advancement of our knowledge of intelligent physical agents and even to correctly appreciate disruptive innovation in the science and technology of robots. In order to address these needs the European Robotics Network of Excellence (EURON) has funded a Special Interest Group on Good Experimental Methodology and Benchmarking. This workshop aims to provide a discussion forum on these topics and to identify guidelines for the future.
All science proceeds from experiment, which motivates the creation of new theory and establishes the limits and validity of the existing theoretical basis. Individual branches of science conduct experiments differently, depending on the topic of investigation, but all have in common a body of knowledge concerning experimental methodology that specifies how to design and conduct good experiments in that discipline. If cognitive robotics aims to be serious science, serious attention must be paid to experimental methods. During the last four years different initiatives have addressed benchmarking in a number of robotics areas such as visual servoing, motion planning, or SLAM. However, performance metrics for the cognitive aspects of embodied systems has been somehow neglected by both the robotics and cognitive systems communities. By getting together experts in both fields we plan to foster further progress in that direction. The research activities in this field are huge as it is the number of published papers. In order to allow the exploitation of the many results obtained it is at least necessary to be able to validate the results by replicating them or compare the results in terms of the selected performance criteria. Although some work is already carried on, many open issues are still ahead.
The emphasis of the workshop will be on cognitive solutions to both theoretical and practical problems. These cognitive approaches should enable an intelligent system to behave appropriately in real-world scenarios in various application domains. We also propose to discuss the distinction between autonomy and intelligence (if any) and how one influences the other. We welcome any topic relevant to benchmarking and performance evaluation in the context of cognitive solutions to practical problems.