Data Processing for PixelArena

Project page: https://pixelarena.reify.ing/project

Web viewer for the results: https://pixelarena.reify.ing/

The project page includes links to the paper, code, and results gallery.

Usage

Setup project:

0. Clone the repository: git clone https://github.com/ifsheldon/mllm-semantic-segmentation.git
1. (Optional) setup submodules: git submodule update --init --recursive
2. Install uv: https://docs.astral.sh/uv/getting-started/installation/
3. Run uv sync to install dependencies.
4. Run uv run poe setup-frontend to install frontend dependencies.
5. (Optional) install oxen: https://docs.oxen.ai/getting-started/install
6. (Optional) run oxen clone https://hub.oxen.ai/ifsheldon/mllm-segmentation-data to get all results.
   • remember to run ln -s mllm-segmentation-data/results results if you need to run the frontend.

Alternatively, extract a local results.zip containing a top-level results/ directory from the repository root: unzip results.zip -x '__MACOSX/*' '*/.DS_Store'. The frontend reads predictions from results/celeb and results/coco, and images and reference masks from eval-set/, all at the repository root. Extracted results and the root results.zip archive are ignored by Git.

Run frontend: uv run poe run-frontend

Host with Docker Compose

With Docker Engine and Docker Compose installed, prepare results/ as described above and keep eval-set/ at the repository root. Both directories are mounted read-only into the container and must exist before startup. The image contains only the frontend; the datasets and results.zip are not included in the build context.

From the repository root, build and start the service in the background:

sudo docker compose up -d --build --wait

Open http://localhost:3011 or use the host's address on port 3011. Stop any existing local server on that port first. The container uses restart: unless-stopped, so it restarts after a crash or Docker restart and remains stopped after an explicit stop. Docker must start on boot for the service to return after a host reboot.

sudo docker compose logs -f frontend  # View logs
sudo docker compose stop            # Stop until explicitly started again
sudo docker compose up -d           # Start the existing image
sudo docker compose down            # Remove the container and its network

Rerun sudo docker compose up -d --build --wait after code or dependency updates. Removing the container preserves the host's datasets. Omit sudo if your user already has Docker access.

The Docker build installs dependencies with Bun's frozen lockfile and enables Next.js standalone output through NEXT_OUTPUT_STANDALONE=1. Local bun run build and bun run start-serving continue to use the regular production build.

Eval Set

CelebAMask-HQ

A random subset (500 images) of the CelebAMask-HQ dataset is used for evaluation.

• Images: eval-set/celeb/images, 512x512
• Images (150): eval-set/celeb/images-150, 512x512, a subset (150) of the images
• Reference masks: eval-set/celeb/masks-512, 512x512
• Upscaled reference masks: eval-set/celeb/masks-1024, 1024x1024

Test results should be saved in results/celeb directory. For Gemini and GPT generated masks, the naming convention is <id>.mask.[0-4].{raw.jpeg, raw.png, pred.png}. [0-4] is attempt index (total 5 attempts). raw.{jpeg, png} means the colorful mask images generated by Gemini/GPT, pred.png means the P-mode png converted from the colorful jpeg.

COCO

A random subset (150 images) of the COCO dataset is used for evaluation.

• Images (150): eval-set/coco/images-150, 512x512, a subset (150) of the images
• Reference masks: eval-set/coco/masks-1024, 1024x1024

Test results should be saved in results/coco directory. For Gemini and GPT generated masks, the naming convention is <id>.mask.[0-4].{raw.jpeg, raw.png, pred.png}. [0-4] is attempt index (total 5 attempts). raw.{jpeg, png} means the colorful mask images generated by Gemini/GPT, pred.png means the P-mode png converted from the colorful jpeg.

Results

Results are tracked by oxen, a version control system for large datasets.