LeWM study cracks JEPA long-horizon planning

The widespread belief that world models struggle primarily with compounding predictive errors is challenged by these findings—the real bottleneck in JEPA systems lies in search topology and goal representation. Decoupling agent state from target objectives and structuring latent space as a navigable graph unlocks dramatic performance leaps without modifying a single model weight.

• –Learned physics matches ground truth: Over 25-step horizons, vanilla LeWM achieved a 78% success rate compared to 77% when using the simulator's exact physics, confirming that model capacity is not the limiting factor.
• –Hierarchical latent memory graphs: Chaining 25-step local searches across an embedded demonstration graph with penalized jumps surged 75-step task completion from 22% to 85%, matching an oracle planner fed expert subgoals.
• –Spatial transformations solve data sparsity: Shifting, rotating, and mirroring demonstration segments into uncovered latent areas restored success from 32% to 85% on out-of-distribution goals, allowing 1% of demonstration data to rival 10%.
• –Incidental goal penalties: Scoring plans against raw goal images mistakenly forces robots to replicate incidental end-effector positions; eliminating pusher coordinates from the objective quadrupled success rates on difficult goals (9/50 to 39/50).
• –Architecture implications: Validates Yann LeCun's joint-embedding thesis for robotics while highlighting that latent world models require topological high-level planners rather than brute-force forward trajectory rollouts.