PhAIL Benchmarks Robot AI on Hardware
PhAIL is an open benchmark for robot AI on real commercial hardware, focused on bin-to-bin order picking on the DROID platform. It evaluates four vision-language-action models under blind conditions using production-style metrics like Units Per Hour and Mean Time Between Failures, with synchronized video and telemetry for every run. The headline result is stark: the best model reaches only about 5% of human hand throughput and needs human intervention roughly every four minutes, while teleoperation on the same robot is still far ahead of autonomous policies.
Strong work. This is the kind of benchmark robotics has been missing because it measures deployment economics, not just demo success.
- –The framing is compelling: same hardware, same task, blind evaluation, and metrics ops teams actually care about.
- –The gap is still huge: OpenPI and GR00T are the best AI entries, but they are far below teleop and human performance.
- –MTBF is the more important signal here than raw UPH, because frequent assists make “autonomy” operationally expensive.
- –The open dataset, scripts, and public run videos make the benchmark more credible and easier to challenge or extend.
- –The main limitation is scope: one task, one hardware setup, and known objects, so this is a strong baseline, not a general manipulation verdict.
DISCOVERED
54d ago
2026-04-02
PUBLISHED
55d ago
2026-04-02
RELEVANCE
AUTHOR
svertix
