RoboChallenge: Large-scale Real-robot Evaluation of Embodied Policies
Testing on real machines is indispensable for robotic control algorithms. In the context of learning-based algorithms, especially VLA models, demand for large-scale evaluation, i.e. testing a large number of models on a large number of tasks, is becoming increasingly urgent. However, doing this right is highly non-trivial, especially when scalability and reproducibility is taken into account. In this report, we describe our methodology for constructing RoboChallenge, an online evaluation system to test robotic control algorithms, and our survey of recent state-of-the-art VLA models using our initial benchmark Table30. As vision-language-action models (VLA) become increasingly successful in robotic tasks [14, 6, 7, 8, 2, 5], the problem of benchmarking emerged. The evaluation methodology needs to be fair enough for stable results, scalable enough to cover a wide range of tasks, and robust enough for public access. Great effort has been put into simulator-based benchmarks [12, 9, 11, 10]