Samuele Punzo*,
Niccolò Caselli*,
Ippokratis Pantelidis*,
Francesco Massafra*,
Salvatore Lo Sardo*,
Mohammadreza Salehi†
University of Amsterdam
* Equal contribution † Supervisor
Probe4Physics · Official implementation accompanying the paper.
Published June 8, 2026
Adapting MVP to video-only physical-plausibility probing (Figure 5).
Video frames adapted from MVP (Krojer et al., 2025).
This work studies the accessibility of physical-plausibility information in frozen video foundation model representations. It provides a shared pipeline for dataset preparation, feature extraction, probe training, and evaluation across predictive joint-embedding, masked-reconstruction, and video-diffusion models. Separate probes operate on intermediate representations, allowing comparisons across network depth, backbone variants, and readout capacity without updating the pretrained backbone.
| Component | Supported scope |
|---|---|
| Benchmarks | Minimal Video Pairs (MVP), IntPhys2 |
| Backbones | V-JEPA, V-JEPA 2, V-JEPA 2.1, VideoMAE, VideoMAE-v2, LTX-Video |
| Probes | Linear, MLP, temporal attentive |
| Primary metrics | MVP pair consistency; IntPhys2 violation-of-expectation (VOE) accuracy |
| Controls | Temporal displacement, repeated single frame, structured random labels |
MVP is adapted to a video-only binary plausibility task. IntPhys2 uses supervised scene-level train/validation/test partitions. Scores therefore belong to this probing protocol and should not be interpreted as directly comparable to the original benchmarks' published evaluations. Probe performance measures decodability; it does not establish that a backbone actively uses the recovered information for physical reasoning.
Run commands from the repository root. Initialize the recorded upstream versions:
git submodule update --init --recursive
conda env create -f environment.yml
conda activate probe4physics
python run.py help
python run.py health.layers strict_exit=truePython 3.11 is the environment target. The standard environment supports CPU probe training and result analysis. For CUDA workloads, use the separate GPU environment, which specifies PyTorch 2.6.0 and CUDA 12.4 wheels:
conda env create -f environment-gpu.yml
conda activate probe4physics-gpuChoose an environment compatible with the execution host. GPU feature extraction and token-level attentive training require substantially more memory than pooled linear or MLP probing. The environment definitions are dependency specifications, not exact historical locks. See the validation environment record for the versions actually checked during release preparation.
Prepare MVP videos using the upstream benchmark's setup instructions. The
init.mvp command can obtain annotations, applies the existing binary
plausibility selection, and creates grouped splits; it does not download the
complete video collection. IntPhys2 has a dedicated downloader:
python run.py download.intphys2
python run.py init.intphys2
python run.py init.mvpDefault video roots are data/videos/mvp and data/videos/intphys2. Supply
videos_root=/absolute/path when using an existing collection. Feature caches
normally live under artifacts/features/<dataset>; override feature_cache.dir
when necessary. Environment variables provide equivalent defaults, documented
in data preparation.
Backbone variants, input sizes, local checkpoint filenames, and Hugging Face
identifiers are defined in configs/backbones.yaml.
JEPA checkpoints must be obtained from their upstream releases; the expected
variants and filenames are listed in the backbone configuration. VideoMAE and LTX adapters resolve the configured remote
models when extraction is run. Initial model access requires network availability
and compliance with each model's terms. Explicit checkpoints can be supplied
with +backbone.kwargs.checkpoint_path=/absolute/path where supported.
Raw datasets, pretrained weights, feature caches, and trained probe checkpoints are not bundled in this checkout.
Reproduction starts from the datasets and pretrained checkpoints, rather than bundled measurement workbooks. Prepare the videos, extract frozen features, and run hyperparameter searches or fixed-configuration probes using the Python CLI. The experiment guide covers extraction, Optuna searches, fixed-config seed runs, controls, and collection of newly generated metrics.
The public results/ directory contains selected configuration records and run
manifests, together with the source CSV rows required to validate their hashes.
These are execution inputs, not a distribution of the paper's measurement tables.
See the configuration inventory for their roles and limits.
Verify these inputs before running a historical configuration:
python scripts/verify_release_inputs.py
python scripts/run_seed_manifest_row.py \
--manifest results/seed_runs/seed_manifest_layerwise_main_linear_mlp_v1.csv \
--probe linear --task-index 0 --dry-runSupply local cache, checkpoint, and output paths before executing the selected row. Keep the recorded seeds, splits, preprocessing, and selected settings when comparing new measurements with the paper. Generic defaults alone do not specify the complete experiment matrix, and not every historical run has complete source artifacts. Full reruns require sufficient CPU/GPU resources and external assets. The reproduction guide describes the supported workflow, generated outputs, and remaining limits.
Historical measurement workbooks and performance-analysis notebooks are retained on the development branch. They are not required to execute the training and evaluation pipeline and are not bundled with the public release. The included parameter-count notebook operates on model configuration without those workbooks:
python scripts/reproduce_notebooks.py \
--output-dir output/reproduction/parameters \
--notebooks parameter_countsThis writes an executed notebook, exported metadata tables, and an execution report into a fresh output directory while preserving its source. It does not regenerate the paper's performance figures. New evaluation measurements are written by the training/evaluation commands and can be collected independently.
After preparing MVP videos and downloading a V-JEPA checkpoint, the following runs a linear probe on layer 24 of the ViT-H encoder:
python run.py init.mvp
python run.py extract.mvp \
backbone.name=jepa_v1 +backbone.kwargs.variant=vith16_384 \
+backbone.kwargs.checkpoint_path=/absolute/path/to/vith16-384.pth.tar \
backbone.kwargs.device=cuda
python run.py train_eval.probe.mvp \
backbone.name=jepa_v1 +backbone.kwargs.variant=vith16_384 \
+backbone.kwargs.checkpoint_path=/absolute/path/to/vith16-384.pth.tar \
backbone.kwargs.device=cuda \
probe.name=linear probe.layer=24 probe.device=cpuUse the same backbone overrides when loading the cache as during extraction.
Splits are written below data/splits/; caches contain an index, feature tensors
or chunks, and a manifest. Probe outputs include probe_best.pt,
probe_last.pt, training summaries, predictions, and benchmark metrics under
artifacts/. Output directories are configurable. Cache validation prevents
silently combining incompatible representations. Consult the experiment guide
before changing paths for an existing resumable extraction.
benchmarks/ Dataset selection, grouped splits, loading, and scoring
models/ Frozen backbone adapters and feature contracts
probes/ Trainable readouts
training/ Extraction and probe orchestration
configs/ Dataset and backbone configuration
experiments/ Recipes and readiness checks
scripts/ Reproduction and result collection tools
notebooks/ Model metadata inspection
results/ Experiment configurations and required run manifests
docs/ Detailed instructions and provenance
tests/ CPU tests and fixtures
third_party/ Recorded upstream implementations
Generated feature caches, checkpoints, executed notebooks, figures, and logs
belong in ignored artifacts/, output/, or logs/ directories. The structure
above lists the shared paper workflow; development branches may also contain
additional research and cluster utilities.
python -m unittest discover -s tests -q
python scripts/verify_release_inputs.py
python run.py exp.listCPU CI runs the retained tests and checks the public command interface and layer
configuration. python run.py health inspects dataset and model readiness;
python run.py health.features checks existing feature caches. These checks may
report missing external assets on a fresh checkout. Tests use CPU fixtures and
mocked backbones; they do not validate full GPU extraction or training. See the
reproduction index for artifact availability and limits.
Citation metadata is also available in CITATION.cff.
@inproceedings{punzo2026probe4physics,
title = {How Do Video Foundation Models Encode Intuitive Physics?
Probing Across Pretraining Paradigms},
author = {Punzo, Samuele and Caselli, Niccolò and Pantelidis, Ippokratis
and Massafra, Francesco and Lo Sardo, Salvatore
and Salehi, Mohammadreza},
booktitle = {NeurIPS 2026 Workshop on World Models in Physical AI},
year = {2026}
}This work uses MVP, IntPhys2, V-JEPA, VideoMAE, and LTX-Video. Dataset, model, and upstream code attribution is recorded in third-party documentation. These assets retain their respective licenses and access conditions. A license for this project’s own code has not yet been selected.
