Self-play is not enough: human imitation improves self-play autonomy in the wild

Marc Heim^*,1,‡, Bassam Helou^*,1,‡, Rohit Satishkumar^*,1,2,§, Momchil S. Tomov^*,1, Hansford C. Hendargo¹, Nicolas Rey¹, Gianmarco Bernasconi^1,‡, Joseph Baker¹, Alexander Boule¹, Rafael DaSilva¹, Abed Ghanbari¹, Cedric Warny¹, Yi Huo¹, Phuc Nguyen¹, Neşet Ünver Akmandor¹, Sarjak Thakkar¹, Daphne Cornelisse³, Eugene Vinitsky³, Forbes Howington¹, Boaz Floor¹, Yiluan Guo¹, Samuel Findler^1,†, Sourabh Vora^1,†

¹ Motional | ² Carnegie Mellon University | ³ NYU Tandon School of Engineering

^* Equally contributing first authors ^† Equally contributing senior authors ^‡ Work done while at Motional ^§ Work done during an internship at Motional

CoRL 2026

Paper

TL;DR

Self-play RL makes driving policies robust but not human-like. SHIFT (Self-play with Human Imitation Fine-Tuning) pretrains with large-scale self-play, then fine-tunes on human driving logs in closed-loop log replay, using the same PPO loop with one added human-similarity reward and a KL anchor to the frozen self-play policy.

How SHIFT works

SHIFT training pipeline: stage 1 self-play pretraining, stage 2 human imitation fine-tuning

Figure 1: SHIFT training pipeline.

Most hybrid methods learn from human data first and then use it to anchor self-play. SHIFT flips the order: it pretrains a robust foundation policy with large-scale self-play, then fine-tunes it on human driving. Fine-tuning takes hours rather than days, so adding a city or targeting a new capability doesn't mean re-running self-play.

Both stages use the same network and the same PPO algorithm. Fine-tuning adds three things:

Log replay. The policy drives through real driving logs while every other road user replays what it actually did in the log.
A human-similarity reward. One extra reward term for staying close to the human driver's trajectory.
A KL anchor. Keeps the policy close to the frozen self-play policy, so it doesn't forget how to drive safely.

On-road videos

Example drives on public roads in Pittsburgh, PA, with self-play on the left and SHIFT on the right.

Self-play

SHIFT

Scenario 1 · Close proximity

Self-play slows down near other vehicles before moving forward, whereas SHIFT naturally moves past them.

Scenario 1 — Self-play

Scenario 1 — SHIFT

Scenario 2 · Unprotected right turn

Self-play is very conservative and takes the right turn at a low speed, whereas SHIFT takes the turn naturally.

Scenario 2 — Self-play

Scenario 2 — SHIFT

Scenario 3 · Close proximity

Self-play stops abruptly when other vehicles are close by, whereas SHIFT drives through naturally.

Scenario 3 — Self-play

Scenario 3 — SHIFT

This is a research project in Motional and is not representative of Motional's production AV's

Results on nuPlan

−15%

ADE vs. self-play

Self-play 9.41 → SHIFT 8.04 m

−39%

Collisions vs. self-play

Self-play 0.064 → SHIFT 0.039

−31%

Brake taps vs. self-play

Self-play 0.288 → SHIFT 0.197

−53%

Oscillations vs. self-play

Self-play 0.090 → SHIFT 0.042

We fine-tune on ∼92K scenes of human urban driving from 1,712 logs and evaluate on the held-out closed-loop nuPlan Val14 benchmark (1,115 scenarios, non-reactive agents). SHIFT outperforms the self-play baseline on 9 of 10 metrics, with the largest gains in human-likeness (ADE −15%, FDE −13%) and comfort (brake taps −31%, oscillations −53%). It is also safer: collisions −39%, drivable-area violations −32%, red-light violations −39%.

Radar chart of SHIFT's relative improvement over self-play on ten nuPlan metrics

Figure 2: Relative improvement of SHIFT over self-play on nuPlan Val14 (CLS-NR).

Metric	Self-play	SHIFT	Extra self-play	BC only	BC+RL	TFISH	HR-PPO
Human-likeness
ADE (m) ↓	9.406	8.041	8.936	26.229	25.305	11.915	9.645
FDE (m) ↓	21.347	18.542	20.050	72.606	57.800	30.522	21.914
Safety
Collisions ↓	0.064	0.039	0.057	0.552	0.230	0.193	0.151
Drivable area violations ↓	0.025	0.017	0.022	0.414	0.336	0.039	0.017
Red light violations ↓	0.016	0.010	0.014	0.169	0.007	0.013	0.015
Speed limit violations ↓	0.074	0.076	0.082	0.451	0.065	0.080	0.094
Progress
Progress along expert route ↑	0.762	0.797	0.801	0.666	0.276	0.662	0.810
Speed vs. expert (m/s) ↑	−0.894	−0.620	−0.581	2.588	−3.992	−1.719	−0.663
Comfort
Brake taps ↓	0.288	0.197	0.285	0.124	0.639	0.100	0.427
Oscillations ↓	0.090	0.042	0.091	0.012	0.030	0.132	0.224

Table 1: Model comparison on the held-out nuPlan Val14 benchmark (CLS-NR). Bold is best per row.

Scatter plots of collisions, drivable-area violations, progress and brake taps against ADE for each method

Figure 3: Key safety, progress and comfort metrics compared to ADE.

Baselines. Extra self-play gives only modest gains. BC only is comfortable but unsafe, with 14× more collisions, 24× more drivable-area violations and 17× more red-light violations than SHIFT. Naively combining BC+RL from scratch does worse still, which points to the role of self-play pretraining in this limited-data regime. BC-regularized self-play (TFISH and HR-PPO), the conventional IL→RL ordering, does not match SHIFT.

Example nuPlan scenario comparing self-play, SHIFT and BC only trajectories with velocity, acceleration and steering profiles

Figure 4: Example nuPlan scenario. Self-play is safe but unhuman-like; SHIFT tracks the human expert with smooth kinematics; BC only is comfortable but drives off-route.

Fine-tuning for specific capabilities

Because fine-tuning is cheap, the same foundation policy can be specialized quickly. Fine-tuning on a dataset of nominal driving already improves substantially over self-play. A curated close-proximity dataset helps most in dense traffic, where the policy keeps up with expert speed and brakes less. Fine-tuning on both datasets in sequence gives the best of both: it is best on six of ten metrics overall, with collisions −44% and brake taps −42% relative to self-play.

Hyperparameter sweeps

Sweeps over training cycles, similarity weight and KL weight on five nuPlan metrics

Figure 6: Sweeps over training cycle (top), similarity weight (middle) and KL regularization (bottom) around the main configuration (1,500 cycles, w_similarity = 10⁻², β = 10⁻²).

Most of the human-likeness gain arrives in the first few hundred PPO cycles, while safety stays flat or improves across all 1,500. Raising the similarity weight from pure RL toward pure BC improves human-likeness and comfort, but safety worsens past 10⁻². With too little KL, fine-tuning overwrites the self-play policy and behaves like BC only; with too much, the policy stops learning.

Driving on public roads

2.6×

fewer safety takeovers than self-play

One every 1.46 mi with SHIFT vs. every 0.57 mi with self-play

14×

fewer discomfort events than self-play

One every 13.1 mi with SHIFT vs. every 0.92 mi with self-play

We deployed the self-play baseline and SHIFT, fine-tuned on either the nominal dataset or both datasets, on autonomous vehicles on public roads in Las Vegas, with a safety driver, on loops around the Motional Las Vegas office. Both SHIFT variants needed fewer safety takeovers and improved on every discretionary metric (stuck, slow driving, discomfort, bad driving). To our knowledge, these are the first on-vehicle results for a policy pretrained with self-play and fine-tuned with imitation learning.

Metric (miles per event ↑)	Self-play	SHIFT (nominal)	SHIFT (both)
Total auto-miles	14.8	18.5	13.1
Discretionary metrics
Takeovers	0.568	1.322	1.456
Stuck	0.739	1.851	1.871
Slow driving	0.314	0.544	0.655
Discomfort	0.924	1.157	13.100
Bad driving	0.141	0.220	0.279
Objective metrics
Brake taps	1.478	2.644	0.819
Jerk violations	0.139	0.180	0.109
Acceleration violations	0.119	0.123	0.098
Oscillations	2.956	1.322	4.367
Steering rate violations	0.591	0.561	0.570

Table 3: Public-road evaluation. Routes were similar but not identical, so mileage differs.

Limitations

Non-reactive log replay limits episodes to 6.6 seconds; beyond that, replayed agents collide with the ego once its trajectory diverges from the log. Rolling out and supervising all agents during fine-tuning could remove this limit. Some edge cases involving yellow lights and stop signs degrade after fine-tuning, which suggests residual conflict between imitation and strict rule-following even with KL regularization. Our public-road tests also cover limited mileage, so further on-road evaluation is needed.

Code

^* Coming soon, check back on Nov 15.

The released code is a public reimplementation of the method described in the paper. Results may differ from those reported.

Citation

If you find this work useful in your research, please consider citing:

@inproceedings{heim2026shift, title={Self-play is not enough: human imitation improves self-play autonomy in the wild}, author={Heim, Marc and Helou, Bassam and Satishkumar, Rohit and Tomov, Momchil S. and Hendargo, Hansford C. and Rey, Nicolas and Bernasconi, Gianmarco and Baker, Joseph and Boule, Alexander and DaSilva, Rafael and Ghanbari, Abed and Warny, Cedric and Huo, Yi and Nguyen, Phuc and Akmandor, Ne{\c{s}}et {\"U}nver and Thakkar, Sarjak and Cornelisse, Daphne and Vinitsky, Eugene and Howington, Forbes and Floor, Boaz and Guo, Yiluan and Findler, Samuel and Vora, Sourabh}, booktitle={Conference on Robot Learning (CoRL)}, year={2026} }