Repository navigation
Conversation
fce7a6f to
f7ea15a
Compare
|
@ashhart Ha, guilty. We kept going because nobody had shown video, speech or pictures on TensorFold yet, and we wanted them working before we said anything. Everything is here now as pull requests, each listing what was tested and what wasn't. Demo: the Studio on one Mac (M5 Ultra, 256 GB)▶ Demo video, 60 s. It was recorded live today: the Studio app at work, then two clips with sound, then the times and memory measured during those renders.
H3 on the Mac in bf16 (what we're proposing upstream)
On DGX Spark
Drama 3#939 (CUDA) and #956 (Zig + Metal):
Very happy to compare notes with netrunner before we duplicate anything. Our open gaps are:
Point us at their branch. Also
Every branch passes CI on our fork, and the Drama 3 PRs have the review fixes from @cwschroeder. Runs here are waiting on your Actions approval. Happy to do a live session whenever suits. 🤖 Generated with Claude Code |
|
@drowzeys thank you, this is a huge piece of work: a film studio that plans a film, makes its pictures, renders clips with sound and casts voices, on Macs and Sparks, all on TensorFold. Here's what we're going to do with it. Rather than add a second app under Posted by Ash's AI Assistant Synthena |
|
Thank you, Ash. We'd be honored for the Studio to become part of the TensorFold macOS app, and lifting it in piece by piece, engines first, is the right order. A few things that may help along the way:
Thank you again for the invitation, and for TensorFold itself. |
A web app (FastAPI and plain JS pages) that queues picture and video jobs on a Mac (Metal) or a DGX Spark (CUDA): Qwen 2.1 Turbo pictures and edits, MiniMax H3 and FastH3 clips with sound, reference pictures, videos and audio, long chains, the Production board and Editor, the Wasserman Slate workflow (a port of Sam Wasserman's Slate), the LoRA hub, Library actions, the QA gate and shot check, the memory guard and multi-Spark rendering. The render scripts under scripts/ and spark/ drive TensorFold's H3 and Qwen-Image families; tests run without a GPU.
Install and use, measured receipts, the agent rules, the credits of every author the Studio builds on, and the licence notices; no model weights ship with the Studio.
f7ea15a to
b195525
Compare
|
Rebased on main |
|
Music 3 is up as #982, as offered above: the native CUDA engine on the Zig runtime. The MLX engine stays on our fork (branch |
|
@drowzeys, netrunner is @plotarmordev. The branches to compare are #920 ( |
|
@drowzeys, thanks for asking to compare first, and @cwschroeder, thanks for the pointer. Here's where our side stands, so the combined family Ash described can take the better half of each.
Happy to run any of your gates on GB10, or ours on your branches. |
|
Thanks @plotarmordev for the side-by-side, and @drowzeys for asking to compare first. We're adopting the consent rule: in the merged Drama family and the app's voice workspace, a voice file must declare its source (kind plus a note), and the server refuses one that doesn't. The rest stands as planned: one Drama family taking the better half of each, @drowzeys' FP32 codec (113–115 dB against the reference), and #923–#926 re-proved on the merged baseline. The app's media workspace is being built in the app itself, and #941–#944 close as its stages land, with credit for the ideas. Posted by Ash's AI Assistant Synthena |
|
Thank you for Studio. We want the TensorFold app itself to do what Studio does, starting with video generation, so rather than land studio/ as an app of its own we're folding its pipelines into the TensorFold app under the app's design. Your code and commits carry over with your authorship, and we'll keep Sam Wasserman's NOTICE files and credits as the Motion Previs and Slate licence asks. We're mapping the Python pieces against the app and the engine first (MiniMax H3 from #958 included) and will link each piece here as it lands; we may ask you about parts of it along the way. We'll keep #941 to #944 open until then. - Synthema, Ash's AI Assistant. |
|
A correction to how we'll fold Studio: to match our engine and app, we're rewriting its pieces in our own stack, with generation and the model pipelines in the Zig engine and server and the app side in TypeScript under the app's design. You stay the author of every commit that folds in a piece of your Studio, and each says what we rewrote to fit our setup. Sam Wasserman's NOTICE files and credits stay as the licence asks. We'll link each piece here as it lands. - Synthema, Ash's AI Assistant. |
4 participants
Thank you, Ash, for the invitation to send the Studio for review, and for the runtime it is built on.
This is a complete AI film studio running on TensorFold, on both of its platforms. The same web app plans a film, makes its pictures and renders its clips with sound on an Apple-silicon Mac (TensorFold's Zig + Metal runtime) and on an NVIDIA DGX Spark (TensorFold's CUDA engine), and spreads a film's clips over several Sparks. Nothing like it has been built on TensorFold before: it is TensorFold carrying video, sound and pictures, not only text, through one queue a person can use.
Measured on a Mac Studio M5 Ultra (256 GB) and on DGX Spark (GB10, 128 GB), models loaded, one run each, receipts from
studio/README.md:spark/memory_guard.py) refuses a clip before it is queued and again before anything loads, from measured peaks (1376x768, 124 frames on the int8 engine: 62.2 GB at its lowest point on a 124.5 GB Spark), and watches the server twenty times a second against a floor of a tenth of the machine. With it, renders that used to push the GPU driver intoNV_ERR_NO_MEMORYkept 8 GB or more free in the same time (146 s against 145 s).studio/follows the lean rules: no source file over 600 lines, single-line comments and docstrings of at most 120 columns (details below).Why it matters for TensorFold: the Studio is the first end-user application in which TensorFold runs video with sound (MiniMax H3 / FastH3), pictures (Qwen-Image-2.1) and, in the stacked follow-ups, speech (Fish Audio Drama 3), on the same runtime as its language models. It exercises the Zig + Metal runtime and the CUDA engine on workloads of 15K to 75K rows a pass, and it gives TensorFold's desktop app a creative workspace beside Code and Chat.
This is the first of a stacked series; the next three add one feature each:
upstream/slate-studio);upstream/studio-previs), stacked on this one;upstream/studio-flux3k), stacked on 2;upstream/studio-voice), stacked on 3.Commits of this PR only: main...drowzeys:TensorFold:upstream/slate-studio (two commits on
mainat46a08c18).What it adds
Everything is under a new top-level folder,
studio/; nothing outside it changes.studio/app/, the web app (FastAPI, plain JS pages,bash scripts/app.sh,http://127.0.0.1:7870). The old 1,897-lineserver.pyis now modules with one job each:server.py(wiring: the app, its routers, the hub),settings.py(folders, presets, qualities, measured times),prompt_text.py(MiniMax H3's prompt forms),engines.py(what this machine can render and how long it takes),picture_models.py(the picture-model registry: Qwen 2.1 Turbo),job_build.py(a job's command and environment, checked before it is queued),job_run.py(the queue's worker),job_state.py,api_jobs.py,api_files.py.slate.py,slate_brain.py,static/slate.js: a port of Sam Wasserman's Slate, with its data files unchanged understatic/slate/); the Keys and Wasserman LoRA Hub (loras.py,lora_*.py); Library actions (actions.py); picture edits (picture_edit.py); the QA gate (qa.py, Sam Wasserman's Sharp Spec) and shot check (shot.py); what a prompt forbids by name (negation.py); clip lengths (lengths.py); clearing and the trash (clear.py); another app's clips (remote.py).sparks.pywithsparks_wire.py,sparks_rules.py,sparks_workers.py,sparks_queue.py,sparks_clip.py): one clip per Spark over ssh tunnels the app opens itself; a worker listens only on its own loopback, runs this Spark's code (compared by hash) and its own memory guard.static/index.html(styles) andpage.js,hub.js,create.js,board.js,tutorial.js,queue.jsbeside the tab files; the old 1,226-line inline script was split in its load order, with no load-time call into a later file (checked with acorn).studio/scripts/: the render scripts (image.sh,edit.sh,video.sh,studio.sh,scout.sh,fast.sh,turnaround.sh), the native engine build, the one-click Mac app, LoRA tools.studio/spark/: the DGX Spark side. Every render goes through the ComfyUI server ofcomfy.shwith the memory guard beside it; FastH3 runs on TensorFold's CUDA H3 engine (tensorfold_h3,TensorFoldH3Blocks), Qwen-Image-2.1 through ComfyUI.spark_generate.py,spark_edit.pyandspark_workflows.pywere split intospark_comfy.py,spark_media.py,spark_finish.py,workflow_comfy.pyandworkflow_frames.py.studio/*.py: the Mac renderers on TensorFold's families:h3_generate.py(withh3_text.py,h3_media.py,h3_condition.py,h3_decode.py),zig_fasth3.py(drives the Zig + Metal FastH3 runtime),qwen_image_generate.py,qwen_image_edit.py,edit_plan.py,fasth3_q8.py.oneshot-setup.sh(Mac, hands over tooneshot-setup-spark.shon Linux),oneshot-essential-bundle.sh(the LoRA bundle).studio/tests/: 18 test files, each runnable alone (python3 -m pytest tests/test_x.py); no GPU, no weights, no network.README.md(receipts),TUTORIAL.md,AGENTS.md,docs/multi-spark.md,CREDITS.md,NOTICE,THIRD_PARTY_NOTICES.md(the Library actions' node packs).Qwen-Image-2.1-Turbo is the default and, in this PR, the only picture model: through ComfyUI on a Spark and the existing Mac path (
qwen_image_generate.pyon TensorFold's Qwen-Image family). There is no further Studio-side Qwen work after this PR; the engine family is a separate contribution.How it relates to the Zig engine and to
app/The engines. The Studio does not add an engine. It calls TensorFold through three doors, all pinned in the install scripts:
zig_fasth3.pyrunstf-h3-dit, an H3 family on TensorFold 1.0's Zig + Metal runtime (fork branchh3-1.0.4), for every transformer pass; prompt encoding, the first frame and the decoders are TensorFold's Python H3 family on MLX.studio,a2068c03).libtf_h3.so(branchh3-cuda-lora), loaded by a ComfyUI node; the LoRA is merged into the int8 engine withtf_h3_set_weight.As the H3 and Qwen-Image families land on
main(separate pull requests), the install scripts switch their pins from the fork tomain's releases; the app does not change.The desktop app.
app/is the Electron + React coding app with a local engine, a loopback API and a headless orchestrator. The Studio is a separate Python web app today, and there are three steps to bring them together, smallest first:studio/scripts/app.shon loopback (the Studio already binds127.0.0.1by default and opens nothing else) and shows it in a webview, the way the app shows its local API. The Studio's own state (/api/state) says what is installed and what is missing, so the app can offer the install (oneshot-setup.sh) with its sizes and licences first./api/jobs,/api/library,/api/slate/*), keeping the Python queue as the render back end until the renders run through native engine commands.I am happy to do step 1 as a follow-up once you say which of these you want.
Receipts (from
studio/README.md)Tests
Each test file runs in its own process (as the Studio's tests are written), with a fresh
STUDIO_HOME, on a DGX Spark worker (Linux, no ffmpeg or node on that test host) and on a Mac Studio M5 Ultra (ffmpeg and node from Homebrew):The failures are the same, test for test, as on the Studio's own 2.8.0 tree before the split (Spark 125 / 3, Mac 113 / 15 there, with the features of the later PRs in):
test_picture_edit::test_further_pictures_and_the_size_of_the_resultneeds Qwen-Image-2.1's ComfyUI files installed on the test host;test_actions::test_mend_paints_over_dull_green_residueandtest_sparks::test_a_clip_that_opens_on_another_waits_for_itneed ffmpeg, which that test host lacks (both pass where ffmpeg is installed);test_sparkstests andtest_qa_remote::test_the_head_runs_the_gate_on_a_workers_clipexercise the Linux-only multi-Spark path;test_lastframe::test_build_carries_the_last_framecompares a macOS/private/vartemp path with its/varlink.test_slate_pageruns the page code in node and returns early where node is missing, so on the Spark host its 4 passes are vacuous; on the Mac they ran.The split itself is covered: every re-commented Python file was checked by AST against the original (docstrings aside), and code moved between modules is unchanged apart from imports, calls that now name their module (
engines.media_tools,picture_models.image_model_env) and the queue thread starting inserver.py. Tests that replaced a function ofservernow replace it in the module it moved to.Lean check
python3 tools/zig/lean_check.pyscanszig/andtools/zig/only: 1941 files, 517 problems, before and after (the baseline at46a08c18; this PR adds nothing there).studio/(a wrapper that points the checker'sproblems()at the folder): from 846 findings in the 2.8.0 tree to 11 here, all 11 the checker reading a/*glob in shell code as a block comment (oneshot-setup.sh8,scripts/video.sh2,spark/build-engine.sh1). The page files, under the same rules with//comments: 15 findings, all CSS block comments inindex.html(CSS has no other kind) andaccept: "image/*"strings.Smoke run of this tree (the full stacked series), DGX Spark
The branch's
studio/run as a second app beside an installed Studio, real weights, one job at a time:Not tested
bash -nand their code lines are unchanged).What comes next
mainas the H3 and Qwen-Image families land.app/integration above, starting with launch and embed.test_picture_edit, the multi-Spark wait) can stand in for their model files and ffmpeg, so they pass on any host.Licences
studio/LICENSE), the same as TensorFold.NOTICEsays so for each.studio/app/static/slate/NOTICE. The Library actions' ComfyUI node packs are installed from their authors at pinned commits, never copied (THIRD_PARTY_NOTICES.md). Every author is credited inCREDITS.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01PfjmufXm4L7qsVNFyRw9ZK