Doubt of underlying bugs in the training pipeline

[ryan186868]

Hi @rizavelioglu, thanks for your amazing work. I have been running the code following your instructions in [Paper 1], and noticed that for this test case, the prediction matches the ground truth very closely, including small details like the hidden cloth tag. I just wanted to check if there might be an underlying bug causing some overlap between the training and test data. Have you observed the same phenomenon on your side?

[rizavelioglu]

Hi @ryan186868, nice catch!
This is not a bug in our implementation. This issue originates from the original VITON-HD dataset, which contains duplicate and more importantly, leaked samples. We describe this in Supplementary Material A of the, recently updated, paper1). I have now pushed a Jupyter Notebook demonstrating how we found these issues using simple image hashing.
For the specific case you mentioned (vitonhd/test/cloth/00034_00.jpg), its hash matches closely with two samples in the training set (02197_00.jpg & 13665_00.jpg):

Notice how all garments have the exact same wrinkles, shading, and even the cloth tag, which would be highly unlikely in the real-world. I'm assuming that the dataset creators used tools like Photoshop to recolor the same garment, which appears to be quite common thing to do (see videos on youtube).
As a result, the model may 'remember' these details (cloth tag) from training and reproduce them during inference.

The code to compute similar matches for this particular sample:

import os
from PIL import Image
import imagehash
from tqdm import tqdm
import matplotlib.pyplot as plt

def calculate_phashes(folder_path):
    phashes = {}
    for image_name in os.listdir(folder_path):
        image_path = os.path.join(folder_path, image_name)
        image = Image.open(image_path)
        phash = imagehash.phash(image)
        phashes[image_name] = phash
    return phashes

threshold=10
target_name = "00034_00.jpg"
target_path = f"./vitonhd/test/cloth/{target_name}"
target_hash = imagehash.phash(Image.open(target_path))
target_img = Image.open(target_path).resize((128, 170))

# Compute phashes for train set
train_dir = "./vitonhd/train/cloth/"
phashes1 = calculate_phashes(train_dir)

# Find similar images by Hamming distance
similar_images = [
    (name, target_hash - h)
    for name, h in tqdm(phashes1.items())
    if (target_hash - h) <= threshold
]

# Plot images side-by-side
for train_name, distance in similar_images:
    train_img = os.path.join(train_dir, train_name)
    train_img = Image.open(train_img).resize((128, 170))

    plt.figure(figsize=(10, 5))
    plt.subplot(1, 2, 1) ; plt.imshow(train_img)  ; plt.title(f"{'train: '}{train_name} (dist={distance})") ; plt.axis('off')
    plt.subplot(1, 2, 2) ; plt.imshow(target_img) ; plt.title(f"{'test: '}{target_name}") ; plt.axis('off')
    plt.show()

A note on thresholds: in our original cleaning pipeline, we used a conservative value of threshold=5 to reduce false positives. This particular case only appears when using threshold=10, which is why it was not automatically filtered.