ACTION-CONDITIONED WORLD MODELING · LATENT PLANNING

ActWorld–JEPA

Action-Conditioned JEPA World Models Meet Latent Planning for End-to-End Autonomous Driving

Ziying Song^1,† · Shengkai Zhang^2,† · Caiyan Jia² · Lei Yang¹ · Lin Liu³
Yuchen Liu⁴ · Zirui Li¹ · Li Wang⁵ · Yadan Luo⁶ · Gang Wang⁷ · Chen Lv^1,*

¹Nanyang Technological University ²Beijing Jiaotong University
³Dalian University of Technology ⁴North University of China
⁵Beijing Institute of Technology ⁶The University of Queensland
⁷NingboTech University

† Intern of Horizon Robotics * Corresponding author

Source code ↗Checkpoints ↗Main results ↓Qualitative results ↓

Candidate plans shape the predicted future.
The predicted future informs the next plan.

NAVSIM VIDEO DEMO

See the scene. Follow the plan.

31.5 seconds · three continuous turning clips · source cadence 2 Hz. Offline replay of saved ActWorld-JEPA predictions, not closed-loop driving. BEV actors are dataset annotations; predicted trajectories are unmodified. Camera privacy masking applied.

Data: nuPlan/Motional and OpenScene. Derived media: CC BY-NC-SA 4.0 and nuPlan non-commercial dataset terms. Provenance, license and rendering instructions ↗

MOTIVATION

Predict a future that matters for planning.

A future representation becomes useful to a planner when it is connected to a candidate action. ActWorld-JEPA makes that connection explicit through action-conditioned latent prediction.

Paper motivation comparing JEPA, Drive-JEPA and ActWorld-JEPA — From visual future prediction to action-conditioned planning.

ActWorld-JEPA architecture: perception, plan-conditioned latent rollout, compatibility modeling and future-consistent ranking — Condition on candidate actions. Predict in latent space. Plan with the predicted future.

THE IDEA

World modeling, conditioned on the plan.

ActWorld-JEPA connects a perception-based driving planner with action-conditioned latent future prediction. The base planners are iPad for NAVSIM and MomAD for nuScenes. Candidate trajectories guide the JEPA rollout; current and predicted-future representations inform plan compatibility and trajectory selection.

Plan-conditioned JEPA

Candidate actions condition latent dynamics, connecting future prediction to the decisions the planner needs to make.

Relational residual reranking

Plan–future compatibility and candidate relationships refine the base planner’s decisions.

Trajectory selection and refinement

Use candidate-conditioned predictions to guide the final trajectory, with the selection or refinement pipeline specified for each benchmark.

MAIN EXPERIMENTS

Main results.

Benchmark	Metric	Result	Protocol
NAVSIM v1 NAVTEST	PDMS ↑	90.07	Leaderboard evaluation
NAVSIM v2 NAVTEST	EPDMS ↑	91.6	NAVTEST evaluation
NAVSIM v2 NAVHARD	Combined EPDMS ↑	42.7	Two-stage evaluation
nuScenes · 6 s	L2 (m) ↓	2.3945	Validation split

Results use the released benchmark-specific configurations. Evaluation details ↗

nuScenes · complete 1–6 s results

Metric	1 s	2 s	3 s	4 s	5 s	6 s	Mean
L2 (m) ↓	0.2780	0.5380	0.8851	1.3135	1.8181	2.3945	1.2045
Box collision (%) ↓	0.012	0.136	0.344	0.723	1.334	2.034	0.7638

QUALITATIVE RESULTS

Explore the scene and the planned trajectory.

Front-view visualizations and bird’s-eye-view plans from NAVSIM and nuScenes. Select a collection; open an image to inspect it at full resolution.

NAVSIM front-view heatmaps with corresponding BEV trajectories — NAVSIM · scene understanding and trajectory planning.

Additional NAVSIM traffic scenes and plans — More NAVSIM scenes · open the image for full resolution.

nuScenes six-second planned trajectories and ground truth in BEV — nuScenes · six-second trajectories.

REPRODUCE ACTWORLD-JEPA

From the result to your first evaluation.

Choose a benchmark, prepare its environment and data, then download the matching weights and run evaluation. The released main models do not require retraining.

01 / CHOOSE