AI Agent: end-to-end FLUX.2 → image-to-3D → rig/skin/animate workflow and agent benchmark

[fernandotonon]

Child of #1000. Integration milestone for #1001–#1006.

Goal

Prove the harness with a real QtMeshEditor workflow and create a repeatable benchmark for comparing local orchestrator models.

Reference workflow

Prompt example:

“Create a cartoon armored ogre suitable for a game, generate a clean full-body reference, turn it into a 3D model, keep it reasonably low-poly, rig and skin it, apply an idle animation, and inspect the final result.”

Expected agent behavior:

1. identify required capabilities;
2. call existing FLUX.2 image generation with image-to-3D-oriented constraints;
3. if a VLM is available, verify full-body framing/background/pose suitability and retry generation when clearly unsuitable;
4. run the configured image-to-3D backend;
5. validate mesh properties;
6. simplify/repair only if required by postconditions;
7. rig and skin using the existing preferred paths;
8. apply an idle animation/template;
9. validate skeleton/skinning/animation;
10. render/take screenshot and optionally perform final visual QA;
11. report artifacts, actions, warnings and any compromises to the user.

Benchmark suite

Create ~20–30 task cases covering:

• simple scene edits;
• material creation/application;
• file discovery/import/export;
• image generation;
• image-to-3D;
• repair/simplify;
• rig + skin;
• animation;
• mixed multi-step workflows;
• intentional missing-resource/tool failures;
• visual verification cases when a VLM is available.

Record at minimum: task completion, invalid tool call count, retries, steps, wall time, model load overhead, final validation outcome and whether human intervention was required.

Use this suite to compare the current 3B model with candidate 4–8B orchestrators/VLMs rather than relying on generic LLM benchmarks.

Acceptance

• Reference FLUX.2-to-animated-3D workflow completes end-to-end on at least one supported local setup.
• Text-only mode can execute the workflow with visual checks skipped/degraded.
• VLM mode can reject at least one deliberately unsuitable generated/reference image and replan.
• Benchmark runner/report format exists and can compare two configured local models.
• Sentry breadcrumbs make failed steps diagnosable without storing image/prompt contents unnecessarily.

Parent: #1000