TPIPS — Text-Prompted Image Perceptual Similarity

Project Page | Paper | Demo | Documentation

Sheng-Yu Wang¹, Yotam Nitzan², Aaron Hertzmann², Jun-Yan Zhu¹,
Eli Shechtman², Alexei A. Efros³, Richard Zhang².
Carnegie Mellon University¹, Adobe Research², UC Berkeley³
In ArXiv, 2026.

Installation and quickstart

TPIPS supports Python 3.10 and later. Install matching PyTorch and torchvision builds from the official PyTorch installer, then install TPIPS:

pip install tpips

Start with the recommended embedding model. The factor argument is free-form text, specifying the visual aspect that similarity is conditioned on (e.g., "lighting", "the pose of the person", "number of ducks"). Pass factor="overall" (or None, "") for prompt-independent similarity.

import torch
import tpips
from PIL import Image

model = tpips.load_model("embedding", device="cuda")

image_a = Image.open("a.jpg").convert("RGB")
image_b = Image.open("b.jpg").convert("RGB")

with torch.inference_mode():
    similarity = model.similarity(image_a, image_b, factor="lighting")
    distance = model.distance(image_a, image_b, factor="lighting")

print(similarity.item())  # higher means more similar
print(distance.item())    # lower means more similar

The first call downloads the selected TPIPS checkpoint.

TPIPS also take PyTorch tensors, by default assuming channel-first (CHW or BCHW), [0, 1] tensors:

tensor_a = torch.zeros(3, 512, 512)  # CHW or BCHW, float in [0, 1]
tensor_b = torch.ones(3, 512, 512)

with torch.inference_mode():
    similarity = model.similarity(tensor_a, tensor_b, factor="the window on the left")
    distance = model.distance(tensor_a, tensor_b, factor="the window on the left")

You can backpropagate through the images by dropping torch.inference_mode(). For more argument details, see the tpips.load_model, model.similarity, and model.distance references.

Note: CUDA GPU is recommended. FlashAttention is optional: TPIPS automatically uses the newest compatible FlashAttention backend and falls back to PyTorch SDPA when none is available.

Table of contents

• Models
• Datasets
• Running locally
  • Install from source
  • Run pairwise inference
  • Launch the interactive demo
  • Download the datasets
  • Dataset format
  • Evaluate on TPIPS datasets
  • Train TPIPS
• Frequently Asked Questions
• Citation
• License

Models

Model Type	Supported APIs	Base Model	Ckpt	Odd-One-Out Score	2AFC Score
embedding	similarity, distance, embed	Qwen3VL-8B-Embedding	link	64.1%	77.9%
early_fusion	similarity, distance	Qwen3VL-8B-Embedding	link	64.7%	79.3%
activation_dist	distance	Qwen3VL-8B-Embedding	link	63.8%	75.9%

See the tpips.load_model reference for more detail.

Datasets

Dataset	Splits	Size	HF Link
Odd-One-Out	Train/Val/Test	26G	sywang/tpips-odd-one-out
Cross-Algorithm 2AFC	Test (unseen algorithms)	1.7G	sywang/tpips-2afc

Running locally

The commands below are for the evaluation and demo utilities included in the source repository.

Install from source

Clone the repository and install the full local requirements:

git clone https://github.com/adobe-research/TPIPS.git
cd TPIPS
pip install -r requirements.txt

If you need a specific CUDA build, install matching versions of PyTorch and torchvision first. FlashAttention is optional; TPIPS automatically falls back to PyTorch SDPA when no compatible FlashAttention installation is available.

Run pairwise inference

scripts/inference.py compares two images over one or more comma-separated factors:

python scripts/inference.py \
  --model-type embedding \
  --img-a images/cow_ref.jpg \
  --img-b images/cow_0.jpg \
  --factors "overall,subject color,cow statue pose,ground surface,flag pattern"

Use --model-type early_fusion or --model-type activation_dist to select another released model. Pass --model-path to use a local checkpoint directory or another Hugging Face repository.

Launch the interactive demo

The embedding-only web demo has Image Pair and Video tabs. It reports raw cosine similarity for image conditions and plots frame similarity against actual video time. Video analysis is limited to the first 10 seconds by default.

python scripts/demo.py --gpus 0

Image Mode ↑

Video Mode ↑

Download the datasets

Download and safely extract both official datasets into data/:

python scripts/download_data.py

This creates data/odd_one_out/ and data/2afc/. Use --output-dir to choose another parent directory.

Dataset format

Each odd-one-out JSONL record contains three image paths and one or more factor-specific human judgments:

{
  "p0_path": "data/odd_one_out/images/002004_0.png",
  "p1_path": "data/odd_one_out/images/002004_1.png",
  "p2_path": "data/odd_one_out/images/002004_2.png",
  "factors": [
    {
      "name": "overall",
      "label": "p1",
      "probs": [0.2, 0.4, 0.4]
    },
    {
      "name": "plate color",
      "label": "p1",
      "probs": [0.0, 1.0, 0.0]
    }
  ]
}

For odd-one-out, label is the human-majority odd image and probs is ordered as [p0, p1, p2]. For 2AFC, p0 is the reference, p1 and p2 are the two candidates, and the two-class probabilities are ordered as [p1, p2].

Evaluate on TPIPS datasets

Run odd-one-out evaluation on one GPU with the default downloaded test split:

python scripts/evaluate.py \
  --task odd_one_out \
  --model-type embedding \
  --output results/odd_one_out_embedding.json

Run the same task on multiple gpus with Accelerate:

accelerate launch --multi_gpu --num_processes <num_gpus> scripts/evaluate.py \
  --task odd_one_out \
  --model-type embedding \
  --output results/odd_one_out_embedding.json

Run 2AFC evaluation:

# Run on single GPU
python scripts/evaluate.py \
  --task 2afc \
  --model-type embedding \
  --output results/2afc_embedding.json

# Run on multiple GPUs with accelerate
accelerate launch --multi_gpu --num_processes <num_gpus> scripts/evaluate.py \
  --task 2afc \
  --model-type embedding \
  --output results/2afc_embedding.json

Use --data for another annotation JSONL, --image-root to change the base directory for image paths, and --model-path for custom weights. Each Accelerate process evaluates a separate shard; rank zero merges the results.

Train TPIPS

Train the recommended embedding model on one GPU:

python scripts/train.py --config configs/embedding.yaml

Run the equivalent training job on multiple GPUs:

accelerate launch --multi_gpu --num_processes <num_gpus> scripts/train.py \
  --config configs/embedding.yaml

The same commands accept configs/early_fusion.yaml and configs/activation_dist.yaml. Override any configuration field with dotted key=value arguments, for example:

python scripts/train.py --config configs/embedding.yaml \
  run_name=embedding_2epochs \
  train.num_epochs=2 \
  train.gradient_checkpointing=true

Runs are written under checkpoints/<timestamp>_<run_name>/ by default, and the completed run is in final/. Set output_dir=/path/to/checkpoints to change the parent directory. Set train.gradient_checkpointing=true if you encounter out-of-memory issues during training.

Frequently Asked Questions

Do I need FlashAttention?

No. TPIPS uses a compatible FlashAttention backend when one is available and otherwise falls back to PyTorch SDPA. If a FlashAttention installation is incompatible with your Torch or CUDA build, remove the explicit backend override or run with TPIPS_ATTN_IMPL=sdpa.

Error: "Warning: CUDA initialization: The NVIDIA driver on your system is too old..."

This usually means that the installed PyTorch wheel was compiled for a newer CUDA version than the NVIDIA driver supports. Compare the maximum CUDA version reported by nvidia-smi with the build reported by python -c "import torch; print(torch.__version__, torch.version.cuda)". Install matching PyTorch and torchvision builds from the official PyTorch installer whose CUDA version is supported by the driver, or upgrade the NVIDIA driver. Errors such as ProcessGroupNCCL is only supported with GPUs, no GPUs found may appear afterward because the CUDA mismatch prevented PyTorch from detecting the GPUs.

Why does Transformers warn about an incorrect Mistral tokenizer regex?

TPIPS uses a Qwen3-VL tokenizer, not a Mistral tokenizer. Some Transformers versions misclassify the TPIPS checkpoint because its loader-specific config.json does not contain the standard model metadata used by the tokenizer heuristic. The bundled tokenizer matches the official Qwen3-VL tokenizer and the warning does not affect TPIPS results. Do not apply fix_mistral_regex=True, because that can change tokenization relative to the tokenizer used for TPIPS training.

Why does activation_dist.similarity(...) raise NotImplementedError?

activation_dist is a distance-only model. Use distance(...), where lower values mean the images are more similar. Use embedding or early_fusion when you need cosine similarity.

Where are the models downloaded?

By default, both the released TPIPS checkpoint and its Qwen backbone are downloaded to the standard Hugging Face cache (~/.cache/huggingface/hub). Pass cache_dir= to tpips.load_model(...) to choose a cache for that call, or set HF_HOME to relocate the Hugging Face cache globally. The two resources are stored in separate model directories inside the same cache and are reused on later runs, so they are not downloaded again unless the cache is missing or the requested revision changes.

Citation

If you find TPIPS useful for your research, please cite:

@article{wang2026tpips,
      title={The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric},
      author={Wang, Sheng-Yu and Nitzan, Yotam and Hertzmann, Aaron and Zhu, Jun-Yan and Shechtman, Eli and Efros, Alexei A. and Zhang, Richard},
      journal={arXiv preprint arXiv:2607.18237},
      year={2026}
}

License

TPIPS is provided under the Adobe Research License for noncommercial research use. See the license for the complete terms.