The RAWSEEDS project is and SSA (Specific Support Action) in the EU 6th Frame Program aiming at stimulating and supporting progress in autonomous robotics by providing a comprehensive, high-quality benchmarking toolkit for SLAM (Simultaneous Localization And Map building).
One of the most substantial limitations to the development of mobile autonomous robotics is the sheer arduousness and cost of performing repeatable and reliable tests of systems and algorithms. In addition, it is difficult to quantitatively assess the performance of a system in ways which are meaningful to people outside the group who designed the system; and it is often impossible to compare the results obtained with different solutions or by different research teams. This has a stifling effect on the whole robotics field, especially where industrial research policies are concerned: no one is happy to operate in a field where marketable applications abound, but heavy investments are needed to simply check if an idea is a good one, before any design, engineering or industrial effort can even begin.
This absence of standard benchmarks is a widely acknowledged problem in the robotics field, and this is doubly harmful to it: firstly, because it prevents recognition of scientific and technical progress, thus discouraging research and development; and secondly, because it prevents new actors (and particularly SMEs) from entering the robotic sector, as heavy investments are needed to compensate for that absence.
The problem described above has two causes: first, the experiments needed to test an algorithm or robotic system are extremely difficult and costly to set up; second, the unavailability of benchmarks to quantitatively evaluate the performances of such algorithms and systems makes the experiments nearly useless for groups different from the one which performed them in the first place. Rawseeds strives to tackle and eliminate both causes of the problem. The benchmarking toolkit that RAWSEEDS will create includes: high-quality multisensor data sets, benchmark problems based on them, state-of-the-art solutions to these problems in the form of algorithms and software, and methodologies for the assessment of algorithms. Rawseeds' datasets are sets of time-synced data streams, generated by the sensors aboard a robot platform when it moves through an environment. The datasets are gathered in real-world locations.
For researchers, Rawseeds will speed up considerably the production of innovative, high-quality results (i.e., algorithms, software and complete robotic applications). If, as we hope, the Rawseeds Benchmarking Toolkit will be internationally acknowledged as a standard benchmark for the evaluation of algorithms and software systems (in the fields of mapping, localization and SLAM), its use will let research groups evaluate each other’s work and compare it with their own. The best solutions will be easily and rapidly singled out. Highly innovative but "far out" ideas will no more be neglected for the sheer cost of testing out them, as that cost will be heavily cut down. Even the smallest research groups will have the opportunity to publicize the results of their work, using the Rawseeds website: so, if worthwhile, this work will be immediately acknowledged by all the robotics community.
For groups with an interest into autonomous robotics but not yet involved, Rawseeds will represent the force that will drive them in. Knowing that their ideas and solutions can be tested with a small expense of time and money, the main reason for them to wait will disappear. This is especially true for companies (and even more for SMEs, i.e. Small and Medium Enterprises), both because they are especially sensible to cost issues and because they will have access to an easy mean to compare the performance of their (possible) products with those of competitors already entered into robotics or with state-of-the-art research. Moreover, Rawseeds will let companies make their achievements known to all the robotics community.
The presence on the Rawseeds website of a corpus of already working algorithms means that no one will ever need to "start from scratch".
Rawseeds will generate and publish two categories of structured benchmarks:
The set of sensor data (or dataset) associated to each BP is subject to a validation procedure prior to publication, to ensure absence of defects (such as data drop-outs) and correct time-syncing between separate sensor data streams. In addition to that, each dataset includes the corresponding ground truth, time-synced with the sensor data.
The complete set of BPs and BSs published by Rawseeds is called the Rawseeds Benchmarking Toolkit.
The main use of a BP is to test existing algorithms and compare their performance with that of alternative algorithms. The fact that a common ground for comparison exists is assured by the use of the rating methodology defined by the BP itself.
A BS can be used in many ways, as it is possible to:
The Rawseeds Project will generate the BPs and a set of BSs based on state-of-the-art robotics algorithms, but users of the Rawseeds website are encouraged to contribute to Rawseeds by submitting their own Benchmark Solutions for publication. We hope that a vital community of Rawseeds users and contributors will build up, using our Forum to communicate. Rawseeds includes mechanisms to safeguard the intellectual property of the contributors.
It is also possible for Rawseeds users to submit new Benchmark Problems for publication. However, they will be accepted only if the associated datasets have been subjected to the same exacting validation procedures that we applied on our own datasets, and if an associated ground truth with sufficient accuracy is present.
Further information can be found at the official website. Additional technical information can be found in the papers: