Skip to content

AI v2 B2: retrain the vehicle mesh segmentation on generated (TRELLIS/TripoSG) vehicles + rig-prior labels from the vehicle template #1051

Description

@fernandotonon

Part of #818 (Track B2). Follow-up to #1015 (vehicle plumbing shipped, model trained on synthetic blobs only) and #1050 (vehicle skeleton template + rigid one-bone binding).

Why now

meshseg_vehicle.onnx (vehicle_body / wheel / window / wing / rotor) is trained exclusively on the synthetic make_vehicle generator in scripts/export-meshseg-onnx.py — boxes with blob wheels. Real cars (fused low-poly bodies like ugly_car.obj, detailed multi-part models like the Buick) look nothing like that, and the model has never seen one. Two things changed that make a real retrain cheap:

  1. We can generate vehicles on demand. qtmesh generate3d --prompt "…" (TRELLIS.2 / TripoSG / TripoSR) produces cars, trucks, buses, planes, helicopters, motorbikes, wheels-only, wing-only meshes — unlimited, varied, and license-clean (our own outputs of MIT-licensed models; no ShapeNet/PartNet non-commercial data).
  2. The vehicle rig gives EXACT body/wheel labels for free. --skeleton vehicle binds every vertex to exactly one of Chassis / FrontAxle / RearAxle / 4 wheel bones (feat(#1013): vehicle skeleton template + rigid one-bone binding #1050). That is a per-vertex label. The existing rig-prior miner (qtmesh segment --dump-training-data, AutoRig::rigPriorPartLabelsMeshSegmenter::partForBoneName) already turns bone weights into exact training samples — it only knows body-part bone names today.

Plan

  • Miner: extend partForBoneName with the vehicle vocabulary — Chassis/*Axlevehicle_body, *Wheelwheel (also common artist names: wheel_fl, tire, rim, hub; wing, rotor, propeller, blade). Make --dump-training-data write the sample's category (schema bump qtmesh-meshseg-training-v2, v1 = body) so the export script routes it to the right model.
  • Pipeline script scripts/mine-vehicle-segdata.py: prompt list × seeds → generate3d (TRELLIS.2 when available, else TripoSG; --target-tris 25000) → rig --skeleton vehicle --rigidsegment --dump-training-data. Reject samples whose wheel snap failed (a wheel bone with < N vertices, or wheels not at the four corners — reuse the fitVehicle quadrant check). Also mine the CC0 vehicle packs already used for the body miner (Quaternius) and the two local test cars.
  • Parts as standalone meshes: run PartOps split (segment --split-parts) on the rigged cars and export each part alone (wheel, body) — the Auto category classifier (meshseg_category.onnx) must still say vehicle for a lone wheel or a wing, which is the verified failure mode noted in the export script (detached parts at odd offsets).
  • Window / wing / rotor labels: the rig gives body vs wheel only. Options, in order of preference: (a) keep the synthetic generator for these three classes and mix; (b) label windows on generated cars via the baked texture (dark/low-roughness glass texels → window) — cheap but noisy, gate it; (c) SAM 2 multiview (AI v2 B2: sam2seg — zero-shot mesh segmentation via multiview SAM 2 #1014) if it lands first. Do NOT block the retrain on this — body/wheel is where the real-mesh gap is.
  • Retrain meshseg_vehicle.onnx with --real-data ./mined_vehicle/ mixed into the synthetic set (yaw/tilt/jitter aug as for body) and retrain the category classifier with the generated vehicles + lone parts. Upload via the existing HF flow (segment/ on QtMeshEditor-models), update THIRD_PARTY_AI_MODELS.md provenance (generated-data statement).
  • Evaluation (held out, never trained on): ugly_car.obj, the Buick, 3–5 Quaternius vehicles, 3 generated vehicles from prompts not in the training list. Metrics: per-vertex accuracy + wheel IoU vs the rig-prior labels; category classifier accuracy on whole vehicles and on lone parts. Record the numbers in docs/MESH_SEGMENTATION_STRATEGY.md. Acceptance: wheel IoU ≥ 0.8 on the two real cars (today the geometric fallback's "lowest 30 %" rule is the bar to beat).
  • Verify in-app: Edit Mode → "Select by Part (AI)" on an unrigged generated car selects a wheel cleanly; qtmesh segment car.glb --category vehicle per-part counts look right.

Notes / risks

  • Generated meshes are single fused shells: wheel arches merge into the body, exactly like ugly_car.obj — good, that IS the hard case the synthetic set lacks. Discard generations where the rig's wheel snap fails rather than training on bad labels.
  • Training samples are point clouds + labels, so TRELLIS textures are irrelevant except for the optional window heuristic.
  • Keep the one-model-per-category design (decision record in docs/MESH_SEGMENTATION_STRATEGY.md); this retrains two files, adds none.
  • Vehicle-rig-derived labels only cover 4-wheel ground vehicles; planes/helicopters/bikes need the synthetic classes or SAM 2 until a matching rig template exists.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai-assistLocal-AI-assisted 3D workflows (epic prefix: AI:)enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions