NeurIPS 2026 · Spotlight

The Invisible Hand of Physics

When Video Diffusion Models Know More Than They Show

*Equal contribution  ·  †Equal supervision

Paper arXiv Code BibTeX

Intuitive physics is not unique to self-supervised encoders like V-JEPA. Video diffusion models, trained only to generate, encode it too: whether a real video is physically plausible can be read linearly from their hidden states, on par with or better than V-JEPA 2 and VideoMAE. The signal is absent from the VAE latents the models operate on and emerges inside the denoising transformer.

Diagram. Forward sampling integrates dZ/dt = v(Z, t) from noise to a clean video. Reverse sampling integrates the negated velocity from the clean video back to noise; the velocity network's internal states along this path feed a probe that outputs plausible or implausible.
Reverse sampling and probing. We encode a real video and integrate the model's learned velocity field backwards, from the clean latent to noise. The transformer's activations at every block and noise level along this path are probed for physical plausibility.

Abstract

Modern video diffusion models generate increasingly realistic and temporally coherent videos, motivating their use as candidate world simulators. Yet it remains unclear whether these models internally encode physical structure, or merely reproduce motion patterns seen during training. We study this question by probing video diffusion models along latent trajectories corresponding to real videos with known physical plausibility. To obtain such trajectories, we approximately invert the deterministic sampling process by integrating the learned velocity field backward from a clean video latent to noise, giving access to the model's intermediate states and attention maps. Using these recovered trajectories, we show that physical plausibility is linearly decodable from diffusion transformer states across IntPhys and InfLevel, reaching around 81.27% average accuracy and outperforming dedicated representation-learning baselines such as V-JEPA and VideoMAE. Surprisingly, this signal is absent from the VAE latent input and emerges inside the denoising transformer itself, despite the model not being trained with a self-supervised predictive objective. These findings suggest that physically meaningful representations can arise as a byproduct of generative denoising.

The tasks

What the probe has to tell apart

IntPhys and InfLevel follow the violation-of-expectation paradigm from infant cognition research. Every implausible clip is matched with a plausible one that shares its setup, and the probe must label each clip on its own, without seeing its partner. PhyWorld asks for numbers instead of a label.

Rendered scenes from the IntPhys development set (Riochet et al., 2018). Many violations in the benchmark happen behind an occluder and cannot be spotted from local motion; the examples below happen in view so they are easy to see.

Object permanence

Objects do not vanish or appear from nothing.

Plausible
Implausible

The object vanishes in plain view.

Plausible
Implausible

A cone appears out of nowhere in mid-air.

Shape constancy

Objects keep their shape.

Plausible
Implausible

A cube turns into a cone in mid-air.

Plausible
Implausible

Objects switch between a sphere and a cube in flight.

Spatio-temporal continuity

Objects move along continuous paths.

Plausible
Implausible

The ball jumps partway along its flight.

Plausible
Implausible

The ball's path between the two screens is broken.

Method

Getting inside the model on real videos

A flow-based video model generates by integrating a learned velocity field, \(\tfrac{d\mathbf{Z}_t}{dt} = v_\theta(\mathbf{Z}_t, t)\), from noise at \(t=0\) to a clean latent at \(t=1\). A real video has no such trajectory, so the model's internal states for it are never computed. We recover one by running the sampler in reverse.

  1. 1

    Invert

    Encode the video with the model's VAE, \(\mathbf{Z}_1 = \mathcal{E}(X)\), and integrate backwards to noise. The exact inverse of an Euler step is implicit, so we evaluate the velocity at the known endpoint instead:

    \[\mathbf{Z}_{t_k} = \mathbf{Z}_{t_{k+1}} - \Delta t_k \, v_\theta(\mathbf{Z}_{t_{k+1}},\, t_{k+1})\]

    One network evaluation per step, the same cost as generation. Its deviation from the exact inverse shrinks linearly with step size, and the recovered noise regenerates the original video up to minor artifacts.

  2. 2

    Probe

    Along the 100-step trajectory, record every transformer block's output at 11 noise levels, mean-pool it over tokens, and fit one linear classifier per block and noise level:

    \[\hat{y} = \operatorname{softmax}\!\big(W_{b,t}\,\bar{\mathbf{h}}_{b,t} + \mathbf{c}_{b,t}\big)\]

    Held-out accuracy on a 60/40 split, averaged over five seeds. Each clip is classified on its own.

  3. 3

    Intervene

    During generation, corrupt one block's output at every denoising step, re-invert the result, and score it with the probe's surprise \(\psi = \text{logit}_\text{implausible} - \text{logit}_\text{plausible}\):

    \[\tilde{\mathbf{h}} = \mathbf{h} + \alpha\, \boldsymbol{\sigma}(\mathbf{h}) \odot \boldsymbol{\epsilon}, \qquad \Delta_b = \bar{\psi}(V_b) - \bar{\psi}(V_\text{base})\]

    A positive \(\Delta_b\) means corrupting block \(b\) makes the generated video look less plausible to the probe.

Diffusion backbones WAN-1.3BCogVideoX-2BLTX-2B
Baselines V-JEPA 2 ViT-L / ViT-gVideoMAE-Large / GiantVAE latents

Results

What the hidden states know

Key result

Intuitive physics is not unique to V-JEPA-style encoders: video diffusion models have it too

The evidence so far for intuitive physics in video models comes from self-supervised encoders such as V-JEPA, trained to predict masked parts of a video in representation space. Video diffusion models are trained for something else entirely: generating video. Yet linear probes on their hidden states separate plausible from implausible clips as well as, and often better than, V-JEPA 2 and VideoMAE, including their largest ViT-g and Giant versions.

On IntPhys, all three diffusion models score above every encoder; on InfLevel, the two families are on par. Physical structure is not the product of one training objective. It also emerges from learning to generate.

Bar charts of per-video accuracy for seven models. On IntPhys, WAN-1.3B (about 0.90), LTX-2B (0.83) and CogVideoX-2B (0.77) are all above V-JEPA 2 ViT-L, VideoMAE-Large, V-JEPA 2 ViT-g and VideoMAE-Giant (0.63 to 0.71). On InfLevel, all seven models fall between about 0.66 and 0.72.
Per-video accuracy on IntPhys and InfLevel: each clip is classified on its own. Shaded bars are the diffusion models, probed at the midpoint of the reverse trajectory (\(t = 0.5\)). Best block per seed; mean and standard error over five seeds.

Where it emerges

Physics emerges at intermediate depth, across the whole trajectory

Probing every block at every noise level, the best block scores above 76% at every noise level except the clean endpoint, for all three models. WAN-1.3B's richest block reaches 88.9%. The signal is localized in depth, not in time: the best blocks cluster in the middle third of the network, the same intermediate-depth emergence reported for V-JEPA.

Heatmaps of per-video accuracy over transformer block (x-axis) and noise level (y-axis) for WAN-1.3B, CogVideoX-2B and LTX-2B, with stars marking the best block per noise level; best values 88.9%, 79.2% and 76.4%. A fourth panel plots per-block accuracy for V-JEPA 2 and VideoMAE-Large, peaking in middle blocks.
Where the signal lives. Per-video accuracy for every transformer block (x-axis) and noise level along the reverse trajectory (y-axis); ☆ marks the best block at each noise level. Right: per-block accuracy of the representation encoders on the same scale.

What drives it

Early blocks inject the physics; middle blocks expose it

Corrupting one block at a time while generating plausible IntPhys scenes shows which blocks the probe-assessed plausibility depends on. The effect is spread across depth but structured: early blocks dominate for WAN-1.3B and CogVideoX-2B, while LTX-2B is bimodal, with early and late blocks both sensitive. The most decodable blocks are not the most causal ones. Physical structure appears to be set up early and propagated through the network, becoming linearly readable only at intermediate depth.

Bar charts of mean probe-surprise shift per intervened block for WAN-1.3B, CogVideoX-2B and LTX-2B. The top eight blocks are colored: blocks 3 to 7 and a few late blocks for WAN, mostly blocks 0 to 8 for CogVideoX, and early blocks plus 25 to 27 for LTX.
Block interventions. Mean probe-surprise shift \(\Delta_b\) when block \(b\) is corrupted (\(\alpha = 0.5\)), scored with the probe at 50% noise. Colored bars are each model's eight most sensitive blocks.

Beyond plausibility

Beyond plausibility: the initial conditions are in there too

On PhyWorld parabolas, a linear regressor on a single block recovers the ball's initial position with R² = 1.00 for every model, and its initial velocity with R² between 0.88 (WAN-1.3B) and 0.99, on par with V-JEPA 2 and VideoMAE-Large. The hidden states carry not just that a trajectory obeys gravity, but the specific numbers that generated it. Neither is supervised during diffusion training.

Scatter plots of predicted versus true initial position (top row) and initial velocity (bottom row) for five models. Points lie on the diagonal; initial position R squared is 1.00 for all, initial velocity R squared is 0.88 for WAN-1.3B and 0.99 to 1.00 for the others.
Regressing physical parameters. Predicted vs. true initial position \(x_0\) (top) and initial velocity \(v_0\) (bottom), using each model's best single block.

Model capacity

Narrower transformers give a cleaner readout

Best per-video accuracy falls as the DiT's hidden width grows, from 1536 (WAN) to 1920 (CogVideoX) to 2048 (LTX). The smallest and most compressed model reads out physics best.

One reading is capacity. A narrow model must spend its width on the coarse spatiotemporal structure needed for coherent denoising, while a wide one also spends it on texture, diluting the physical signal. The same trade-off fits the regression result above, where WAN-1.3B is the weakest at the fine-grained velocity. We treat the trend as suggestive: the three models differ in more than width.

Bubble chart of best per-video accuracy against hidden dimension: WAN-1.3B at 1536 near 0.91, CogVideoX-2B at 1920 near 0.79, LTX-2B at 2048 near 0.76, with a downward trend line.
Accuracy vs. width. Best-block per-video accuracy against DiT hidden dimension; bubble size shows parameter count.

The trajectory matters

Physics lives in the flow, not at its endpoints

Inverting with 20 steps instead of 100 still recovers noise that regenerates a faithful video, yet probe accuracy collapses from 0.82 to 0.57, close to chance. A more accurate integrator at the same 20 steps recovers part of it. Decodability tracks how closely the discrete path follows the model's continuous ODE: a coarse trajectory lands in the right place, but through states the model would never visit.

Line chart of per-video accuracy against total inversion steps for WAN-1.3B: 0.57 at 20 steps, 0.76 at 40, 0.81 at 60, 0.82 at 80 and 100. Insets show that the 20-step and 100-step reconstructions look alike.
Reconstruction succeeds where decoding fails. Both the 20-step and 100-step inversions reconstruct the video; only the finer one preserves the physical signal.
Bar chart comparing Euler and Heun integrators at 20 steps across permanence, shape constancy and continuity; mean accuracy rises from 0.57 with Euler to 0.62 with Heun.
Better integration, better physics. At 20 steps, Heun lifts mean accuracy from 0.57 to 0.62 over Euler.

BibTeX

@inproceedings{esmati2026invisible,
  title     = {The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show},
  author    = {Esmati, Parsa and Nath, Somjit and Hofmann, Katja and Nowrouzezahrai, Derek and Kahou, Samira Ebrahimi and Mirmehdi, Majid},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  eprint    = {2606.05328},
  archivePrefix = {arXiv}
}

Acknowledgements

This work was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) under grant EP/W524414/1. The authors acknowledge the use of resources provided by the Isambard-AI National AI Research Resource (AIRR). Isambard-AI is operated by the University of Bristol and is funded by the UK Government's Department for Science, Innovation and Technology (DSIT) via UK Research and Innovation, and the Science and Technology Facilities Council [ST/AIRR/I-A-I/1023].