Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

How Do Video Foundation Models Encode Intuitive Physics?
Probing Across Pretraining Paradigms

NeurIPS 2026 Workshop on World Models in Physical AI

Samuele Punzo*, Niccolò Caselli*, Ippokratis Pantelidis*,
Francesco Massafra*, Salvatore Lo Sardo*, Mohammadreza Salehi†

University of Amsterdam
* Equal contribution   † Supervisor

Reproduce results Citation Python 3.11 PyTorch 2.6

Probe4Physics · Official implementation accompanying the paper.
Published June 8, 2026

MVP adaptation from text-conditioned VideoQA to frozen-feature probing with binary physical-plausibility labels

Adapting MVP to video-only physical-plausibility probing (Figure 5).
Video frames adapted from MVP (Krojer et al., 2025).


Overview

This work studies the accessibility of physical-plausibility information in frozen video foundation model representations. It provides a shared pipeline for dataset preparation, feature extraction, probe training, and evaluation across predictive joint-embedding, masked-reconstruction, and video-diffusion models. Separate probes operate on intermediate representations, allowing comparisons across network depth, backbone variants, and readout capacity without updating the pretrained backbone.

Component Supported scope
Benchmarks Minimal Video Pairs (MVP), IntPhys2
Backbones V-JEPA, V-JEPA 2, V-JEPA 2.1, VideoMAE, VideoMAE-v2, LTX-Video
Probes Linear, MLP, temporal attentive
Primary metrics MVP pair consistency; IntPhys2 violation-of-expectation (VOE) accuracy
Controls Temporal displacement, repeated single frame, structured random labels

MVP is adapted to a video-only binary plausibility task. IntPhys2 uses supervised scene-level train/validation/test partitions. Scores therefore belong to this probing protocol and should not be interpreted as directly comparable to the original benchmarks' published evaluations. Probe performance measures decodability; it does not establish that a backbone actively uses the recovered information for physical reasoning.

Installation

Run commands from the repository root. Initialize the recorded upstream versions:

git submodule update --init --recursive
conda env create -f environment.yml
conda activate probe4physics
python run.py help
python run.py health.layers strict_exit=true

Python 3.11 is the environment target. The standard environment supports CPU probe training and result analysis. For CUDA workloads, use the separate GPU environment, which specifies PyTorch 2.6.0 and CUDA 12.4 wheels:

conda env create -f environment-gpu.yml
conda activate probe4physics-gpu

Choose an environment compatible with the execution host. GPU feature extraction and token-level attentive training require substantially more memory than pooled linear or MLP probing. The environment definitions are dependency specifications, not exact historical locks. See the validation environment record for the versions actually checked during release preparation.

Data and checkpoints

Prepare MVP videos using the upstream benchmark's setup instructions. The init.mvp command can obtain annotations, applies the existing binary plausibility selection, and creates grouped splits; it does not download the complete video collection. IntPhys2 has a dedicated downloader:

python run.py download.intphys2
python run.py init.intphys2
python run.py init.mvp

Default video roots are data/videos/mvp and data/videos/intphys2. Supply videos_root=/absolute/path when using an existing collection. Feature caches normally live under artifacts/features/<dataset>; override feature_cache.dir when necessary. Environment variables provide equivalent defaults, documented in data preparation.

Backbone variants, input sizes, local checkpoint filenames, and Hugging Face identifiers are defined in configs/backbones.yaml. JEPA checkpoints must be obtained from their upstream releases; the expected variants and filenames are listed in the backbone configuration. VideoMAE and LTX adapters resolve the configured remote models when extraction is run. Initial model access requires network availability and compliance with each model's terms. Explicit checkpoints can be supplied with +backbone.kwargs.checkpoint_path=/absolute/path where supported.

Raw datasets, pretrained weights, feature caches, and trained probe checkpoints are not bundled in this checkout.

Reproducing the paper

Reproduction starts from the datasets and pretrained checkpoints, rather than bundled measurement workbooks. Prepare the videos, extract frozen features, and run hyperparameter searches or fixed-configuration probes using the Python CLI. The experiment guide covers extraction, Optuna searches, fixed-config seed runs, controls, and collection of newly generated metrics.

The public results/ directory contains selected configuration records and run manifests, together with the source CSV rows required to validate their hashes. These are execution inputs, not a distribution of the paper's measurement tables. See the configuration inventory for their roles and limits. Verify these inputs before running a historical configuration:

python scripts/verify_release_inputs.py
python scripts/run_seed_manifest_row.py \
  --manifest results/seed_runs/seed_manifest_layerwise_main_linear_mlp_v1.csv \
  --probe linear --task-index 0 --dry-run

Supply local cache, checkpoint, and output paths before executing the selected row. Keep the recorded seeds, splits, preprocessing, and selected settings when comparing new measurements with the paper. Generic defaults alone do not specify the complete experiment matrix, and not every historical run has complete source artifacts. Full reruns require sufficient CPU/GPU resources and external assets. The reproduction guide describes the supported workflow, generated outputs, and remaining limits.

Historical measurement workbooks and performance-analysis notebooks are retained on the development branch. They are not required to execute the training and evaluation pipeline and are not bundled with the public release. The included parameter-count notebook operates on model configuration without those workbooks:

python scripts/reproduce_notebooks.py \
  --output-dir output/reproduction/parameters \
  --notebooks parameter_counts

This writes an executed notebook, exported metadata tables, and an execution report into a fresh output directory while preserving its source. It does not regenerate the paper's performance figures. New evaluation measurements are written by the training/evaluation commands and can be collected independently.

Running an experiment

After preparing MVP videos and downloading a V-JEPA checkpoint, the following runs a linear probe on layer 24 of the ViT-H encoder:

python run.py init.mvp
python run.py extract.mvp \
  backbone.name=jepa_v1 +backbone.kwargs.variant=vith16_384 \
  +backbone.kwargs.checkpoint_path=/absolute/path/to/vith16-384.pth.tar \
  backbone.kwargs.device=cuda
python run.py train_eval.probe.mvp \
  backbone.name=jepa_v1 +backbone.kwargs.variant=vith16_384 \
  +backbone.kwargs.checkpoint_path=/absolute/path/to/vith16-384.pth.tar \
  backbone.kwargs.device=cuda \
  probe.name=linear probe.layer=24 probe.device=cpu

Use the same backbone overrides when loading the cache as during extraction. Splits are written below data/splits/; caches contain an index, feature tensors or chunks, and a manifest. Probe outputs include probe_best.pt, probe_last.pt, training summaries, predictions, and benchmark metrics under artifacts/. Output directories are configurable. Cache validation prevents silently combining incompatible representations. Consult the experiment guide before changing paths for an existing resumable extraction.

Repository structure

benchmarks/   Dataset selection, grouped splits, loading, and scoring
models/       Frozen backbone adapters and feature contracts
probes/       Trainable readouts
training/     Extraction and probe orchestration
configs/      Dataset and backbone configuration
experiments/  Recipes and readiness checks
scripts/      Reproduction and result collection tools
notebooks/    Model metadata inspection
results/      Experiment configurations and required run manifests
docs/         Detailed instructions and provenance
tests/        CPU tests and fixtures
third_party/  Recorded upstream implementations

Generated feature caches, checkpoints, executed notebooks, figures, and logs belong in ignored artifacts/, output/, or logs/ directories. The structure above lists the shared paper workflow; development branches may also contain additional research and cluster utilities.

Validation and development

python -m unittest discover -s tests -q
python scripts/verify_release_inputs.py
python run.py exp.list

CPU CI runs the retained tests and checks the public command interface and layer configuration. python run.py health inspects dataset and model readiness; python run.py health.features checks existing feature caches. These checks may report missing external assets on a fresh checkout. Tests use CPU fixtures and mocked backbones; they do not validate full GPU extraction or training. See the reproduction index for artifact availability and limits.

Citation

Citation metadata is also available in CITATION.cff.

@inproceedings{punzo2026probe4physics,
  title = {How Do Video Foundation Models Encode Intuitive Physics?
           Probing Across Pretraining Paradigms},
  author = {Punzo, Samuele and Caselli, Niccolò and Pantelidis, Ippokratis
            and Massafra, Francesco and Lo Sardo, Salvatore
            and Salehi, Mohammadreza},
  booktitle = {NeurIPS 2026 Workshop on World Models in Physical AI},
  year = {2026}
}

Acknowledgments and licensing

This work uses MVP, IntPhys2, V-JEPA, VideoMAE, and LTX-Video. Dataset, model, and upstream code attribution is recorded in third-party documentation. These assets retain their respective licenses and access conditions. A license for this project’s own code has not yet been selected.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages