diff --git a/README.md b/README.md index b6d40ed..4e2bb9e 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,19 @@ For example, Remix's entry file should be located at: ~/.claude/skills/remix/SKILL.md ``` -After installation, check the skill's README for additional dependencies and setup. If the skill does not appear, start a new agent session. When updating, back up any local edits and replace the existing skill folder rather than nesting another copy inside it. +After installation, check the skill's README for additional dependencies and setup. If the skill does not appear, start a new agent session. + +## Check the version and update + +A skill's version is `metadata.version` at the top of its `SKILL.md`. + +To update, ask your agent: + +```text +Update the Remix skill from https://github.com/SigmaMatrix-AI/tapvid +``` + +Or download this repository again and copy the skill's folder over the installed one. Back up any local edits first, and replace the existing folder rather than nesting another copy inside it. Then start a new agent session. ## Service access diff --git a/skills/remix/README.md b/skills/remix/README.md index 188ef7d..0b1ccb2 100644 --- a/skills/remix/README.md +++ b/skills/remix/README.md @@ -70,7 +70,7 @@ When available, setup uses Homebrew on macOS or winget on Windows to install mis Set `TAPVID_VIDEO_HOME` to place runtime data, caches, and custom providers elsewhere. `PLAYWRIGHT_BROWSERS_PATH` controls the browser cache. Neither the checkout nor an installed skill needs `node_modules`. -Updating the installed skill preserves existing TapVid clone projects, provider settings, and stored credentials in their current formats and locations. Keep your projects and `~/.tapvid-video` when replacing the skill folder. +To check your version or update, see [Check the version and update](https://github.com/SigmaMatrix-AI/tapvid#check-the-version-and-update). Updating the installed skill preserves existing TapVid clone projects, provider settings, and stored credentials in their current formats and locations. Keep your projects and `~/.tapvid-video` when replacing the skill folder. ## Media providers and MCP diff --git a/skills/remix/SKILL.md b/skills/remix/SKILL.md index d0edda6..104b236 100644 --- a/skills/remix/SKILL.md +++ b/skills/remix/SKILL.md @@ -2,6 +2,8 @@ name: remix description: Clone a reference video into a reusable template (React boards, transition table, narrative structure, reference clips per media slot, sound effects) and replicate it on the user's own topic — rewrite the copy, generate or find new footage that follows the original shots with the user's people and products swapped in, voice it at the original's pace, score it and render tapvid.mp4, all locally, at a 1:1 `high` tier (default) or a fast `low` tier. Use when the user asks (in any language) to clone, replicate or remake a reference video (a local file, a link, or keywords to search for one), or to reuse a template made earlier on a new topic. allowed-tools: Read, Write, Edit, Bash, Agent, Task, WebSearch, WebFetch +metadata: + version: "1.0.1" --- # Remix a video @@ -132,9 +134,10 @@ generation starts. Deliver with the same report, and say which choices you made. 1. `node "$SKILL_ROOT/scripts/ensure-runtime.mjs"` installs whatever is missing (Homebrew on macOS, winget on Windows) and the runtime; the first run downloads Chromium and can take minutes — say so. Only a `missing: …` last line (no package manager, or a manual step) needs the user: show what it printed. -2. `node "$SKILL_ROOT/scripts/media.mjs" providers --json` — read it with `references/providers.md`, quietly. No - transcription provider ready → `node "$SKILL_ROOT/scripts/ensure-runtime.mjs" --tool whisper-cli` (local, free; - the model download falls back to a mirror on a slow network by itself). +2. `node "$SKILL_ROOT/scripts/media.mjs" providers --json` — read it with `references/providers.md`, quietly. `whisper-local` + not ready → `node "$SKILL_ROOT/scripts/ensure-runtime.mjs" --tool whisper-cli` (local, free, first in the + transcription order even when another vendor has a key; the model download falls back to a mirror on a slow + network by itself). Keys and generation providers are not raised now; they come up in step 4, where they matter. ## Step 1 — inputs @@ -225,7 +228,7 @@ and these references cover what you need, so do not read plugin sources. 7. `node "$SKILL_ROOT/scripts/refs.mjs" "$PROJECT"` for all slots; write one English `describe` per slot in `template/refs/refs.json`. 8. Narrative structure `template/skill/SKILL.md` and the rest of `template.json` (`references/structure.md`). -9. Swap test (second topic, min / max list counts, Chinese and 1.5× English text). +9. Swap test (second topic, min / max list counts, Chinese and 1.5× English text, every line half its length). 10. Acceptance (`references/acceptance.md`): full `build-boards.mjs`, then in one command (they are independent, run them in the background together) `clock.mjs "$PROJECT"`, `compare.mjs "$PROJECT" --fail-under 0.95 --timing` (sound effects included) and @@ -302,7 +305,8 @@ Then: `$REPLICA/elements/`; presenter slots as digital human (narration mode only). **At `low`** each layer is one slot: `presenter` stays a digital human; a pixel layer stays `ai` / `reference` with `provider: "tapvid"`, `model: "minimax-h3"` when TapVid is ready (else leave both - `null`), the replacements in `replace` and the prompt from `references/replica.md` §6 Low tier; a screen + `null`), the replacements in `replace` and the prompt from `references/replica.md` §6 Low tier, and goes to + keyframe when the identity gate in step 7 fails; a screen layer becomes `user` (their recording or screenshot) when it must be readable, otherwise it is treated like a pixel layer; a code layer is an element — write `$REPLICA/elements/..jsx` for the new topic inside the region its field's `desc` names. The `texts` items are the new copy, one per original @@ -393,11 +397,19 @@ rebuild, ask again. Wait for an explicit yes. returning pass, or the problem and a clean span: extra hands, wrong fingers, a morphing product, garbled text, clipping, the subject drifting back to the original, a digital human's pace drifting within its clip. Trim to a clean span that covers the slot; otherwise state the cost and regenerate once with the prompt fixed. + **Identity gate, on every clip that shows a person** (`references/replica.md` §9): the original's person must + not appear in any frame. `inspect.mjs "$REPLICA" --identity`, look at every frame, then + `identity.mjs pass|fail "$REPLICA" --saw ""`; `render.mjs` refuses the film + until `identity.mjs status "$REPLICA"` is clear. A clip that fails is never trimmed or kept: it goes to keyframe, + and the other reference clips of the film lose their verdict and are read again. Do not route a generated clip + through `source: user`: that slot leaves the gate. Contact sheets and overviews do not count as this check. | Failure | Do | |---|---| | Network / 5xx | The runner already retried once; rerun `run` later | | Moderation | Rewrite the prompt without brands, celebrities, copyrighted elements; retry once; a refused reference video switches that slot to keyframe | +| A first frame with a realistic face flagged as sensitive (Seedance) | Move that slot to another first-frame model (`veo-3.1-fast`); rewording does not help | +| The same error returns seconds after a rerun, with no new vendor job | The rerun returned the earlier result of an identical request: change the model or the prompt, then run it | | A reference picture refused by the vendor | Retry that request without the reference (prompt only); a portrait reframe falls back to `reframe.mjs` | | Unsupported / invalid | The runner moved to the next model or vendor; nothing to do unless all failed | | Bad key / no balance | That vendor is off for this run; tell the user (new key via `set-key.mjs`, or top up) | @@ -419,10 +431,13 @@ Read `references/audio.md` and, in narration mode, `references/captions.md`. 1. `node "$SKILL_ROOT/scripts/project.mjs" "$REPLICA" --final`. 2. `node "$SKILL_ROOT/scripts/validate.mjs" project "$REPLICA/project"` — fix every error. 3. `node "$SKILL_ROOT/scripts/frames.mjs" "$REPLICA"`, then fast sub-agents (one per group of scenes) look at - every scene midpoint, every transition's start / middle / end and the contact sheet — text overflow, captions covered, a pop-up over a face, jumps at + every scene midpoint, the frames around every line boundary, every transition's start / middle / end and the contact sheet — text overflow, two texts overlapping (captions included; `frames.mjs` also scans the film for this and exits 1 listing each pair), captions covered, a pop-up over a face, jumps at boundaries, sound vs picture. Fix and repeat, at most 3 rounds. 4. `node "$SKILL_ROOT/scripts/render.mjs" "$REPLICA"` → `$REPLICA/out/tapvid.mp4` (1080p). Check the render: its - length against the original's, the pace line `timing.mjs` printed, and frames sampled across the film. + length against the original's, the pace line `timing.mjs` printed, and frames sampled across the film, + including the first second of every scene that shows a person, frame by frame at full width: a midpoint never + sees an original person who is only in a scene's opening frames. A slot that went back to generation + counts as done only when its request completed: a failed request leaves the old clip under the same file name. Alternative versions the user asks for go to `out/tapvid-.mp4`; `tapvid.mp4` is always the current one. 5. `node "$SKILL_ROOT/scripts/review.mjs" "$PROJECT" ` again: the Final cut tab plays the film beside the original with a version picker, the Assets tab every generated clip with its vendor and cost. diff --git a/skills/remix/assets/schemas/replica.json b/skills/remix/assets/schemas/replica.json index dd79ce3..903f77a 100644 --- a/skills/remix/assets/schemas/replica.json +++ b/skills/remix/assets/schemas/replica.json @@ -32,6 +32,10 @@ "text" ] }, + "reference_ok": { + "type": "string", + "minLength": 1 + }, "ref": { "type": [ "string", diff --git a/skills/remix/assets/template/src/components/Captions.jsx b/skills/remix/assets/template/src/components/Captions.jsx index 619e347..720e052 100644 --- a/skills/remix/assets/template/src/components/Captions.jsx +++ b/skills/remix/assets/template/src/components/Captions.jsx @@ -12,6 +12,7 @@ // `captions` renders as before. `style` in the tokens is the recommended style name; the drawn // style is always `variant` (replica.audio.captions). import React from 'react'; +import { phrases } from './captionPhrases.js'; // Plate styles: a phrase holds at most 28 characters, two short lines at 4.2 % of the height, bold, // on the plate, its bottom 7 % above the frame's bottom edge. @@ -24,19 +25,6 @@ const PLATE_STYLE = { background: 'rgba(0,0,0,0.55)', borderRadius: '1.2cqh', pa const clamp = (v, a, b) => Math.min(b, Math.max(a, v)); -function phrases(words, maxChars) { - const out = []; - let cur = []; - let len = 0; - for (const w of words) { - if (cur.length && len + w.text.length > maxChars) { out.push(cur); cur = []; len = 0; } - cur.push(w); - len += w.text.length; - } - if (cur.length) out.push(cur); - return out.map((ws) => ({ words: ws, start: ws[0].start, end: ws[ws.length - 1].end })); -} - export function Captions({ t, words, variant, Design }) { if (!words || !words.length) return null; if (variant === 'creator') return ; diff --git a/skills/remix/assets/template/src/components/captionPhrases.js b/skills/remix/assets/template/src/components/captionPhrases.js new file mode 100644 index 0000000..429a889 --- /dev/null +++ b/skills/remix/assets/template/src/components/captionPhrases.js @@ -0,0 +1,19 @@ +// Caption phrases: words grouped up to maxChars, and never across a silence, so a phrase is gone when +// its words end. Muted lines (a board's meta.captions.off) leave such a silence in the words; without +// the break the phrase before one stayed on screen until the phrase after it. +// Sentence pauses in narration run up to about 0.4 s; a muted line is rarely shorter than a second. +const MAX_WORD_GAP_S = 0.6; + +export function phrases(words, maxChars) { + const out = []; + let cur = []; + let len = 0; + for (const w of words) { + const gap = cur.length ? w.start - cur[cur.length - 1].end : 0; + if (cur.length && (len + w.text.length > maxChars || gap > MAX_WORD_GAP_S)) { out.push(cur); cur = []; len = 0; } + cur.push(w); + len += w.text.length; + } + if (cur.length) out.push(cur); + return out.map((ws) => ({ words: ws, start: ws[0].start, end: ws[ws.length - 1].end })); +} diff --git a/skills/remix/references/acceptance.md b/skills/remix/references/acceptance.md index 31fa139..437032b 100644 --- a/skills/remix/references/acceptance.md +++ b/skills/remix/references/acceptance.md @@ -151,6 +151,8 @@ fixed: | Idle motion matches | a sticker that boils in the original stands still | the measured `poses_per_s`, which parts move and by how much | | Moving content is a video slot | a GIF or a screen recording kept as a still | field type `video`, the reference clip from `refs.mjs` | | Captions match | a dark plate where the original has plate-less words | `Design.captions` from the measurements | +| Captions stand where the original's stand | `centerY` copied from a title, so the captions land on the headline | measure on a frame with spoken words (`references/captions.md` §1) | +| Lines keep their titles inside them | a title timed in the original's seconds, landing in the next line once the copy is shorter | times as fractions of the segment (`references/boards.md` §2) | | Captions show where the original shows them | captions under a kinetic title that already says the line; full-frame captions at the split-screen height | the board's `meta.captions` (`false`, `off`, a `centerY`) | | Elements behind the person stay behind | a sunburst drawn over the presenter's hair, or cut by a hand-drawn mask | `behind` + `media.cutout` + `Cutout` (`references/boards.md` §8) | | Sound effects are cued | an entrance that pops in the original is silent | `data-sfx` on the element; `compare.mjs --timing` lists missing cues | diff --git a/skills/remix/references/boards.md b/skills/remix/references/boards.md index d454d5f..f7bc32f 100644 --- a/skills/remix/references/boards.md +++ b/skills/remix/references/boards.md @@ -36,9 +36,11 @@ exported function. Record each board's layering tier in `report.md`: - `t` is scene time in seconds and the frame is a pure function of it: no `Math.random` (use a seeded hash), no `Date.now`, no CSS transitions or animations, no timers, no own clock. Seeking to `t` always gives the same frame. -- Unsegmented board: every time is a fraction of `duration`. Segmented board: elements are timed in seconds - from their segment's `start`, found by key (`segments.find(s => s.key === "points.1.say")`); `duration` still - spans the whole scene. Never seconds from the scene's start. +- Unsegmented board: every time is a fraction of `duration`. Segmented board: an element's times are fractions + of its own segment, found by key (`const s = segments.find(s => s.key === "points.1.say")`, then + `s.start + 0.78 * (s.end - s.start)`), so it stays inside the line however short the new copy is; `duration` + still spans the whole scene. Only an entrance under 0.5 s may be in seconds. Never the original's seconds, + never seconds from the scene's start (`validate.mjs template` warns on a segment key followed by 0.5 s or more). - Geometry in `cqw` / `cqh` only, on a root with `containerType: "size"`. Measure in source pixels, divide by the source width / height. - Colours and fonts only from `Design` (`const { palette, font, fontHead } = Design`); no hex in a board. diff --git a/skills/remix/references/captions.md b/skills/remix/references/captions.md index 6b4559b..9bfba13 100644 --- a/skills/remix/references/captions.md +++ b/skills/remix/references/captions.md @@ -32,6 +32,13 @@ default: | `wordFadeS` | 0.08 | 0.08 | one word's fade, seconds | | `style` | — | — | the style the measurements belong to (a record; `replica.audio.captions` picks the style) | +`centerY` comes from frames where the words are being spoken (`source/timeline.json` words), through +`measure.mjs grid`, on at least three lines of the film: a title or sticker on the same spot is not the caption. +The comparison cannot catch a wrong value, because it draws no template captions and the original's are burned +in. So write each measured value next to the crop it came from in `report.md`, and check the band (`centerY` ± `size`) +against every title and sticker box of each board that shows captions: where they collide in the original too, +the board's `meta.captions` moves or mutes them. `validate.mjs template` warns on such a collision. + The render project mounts one caption overlay above every scene (`src/components/Captions.jsx`), with words in scene seconds. One look, one size, one position across the film: a caption that changes between cuts reads as a bug. Boards never draw captions and keep the lower 20 % of the frame free of key content where @@ -81,7 +88,9 @@ readable: On the frames from `frames.mjs`: -- the caption never covers a face, the product, or a board's text; +- the caption never covers a face, the product, or a board's text, and no two texts overlap, at line boundaries + too (`frames.mjs` saves the frames around them and scans the film, exit 1 per overlapping pair); +- a phrase leaves when its words end: it never waits on screen through a muted line; - no phrase overruns the plate on two lines; - CJK punctuation never starts a line; no lone glyph on a line; - the caption at a transition belongs to the scene whose voice is playing. diff --git a/skills/remix/references/providers.md b/skills/remix/references/providers.md index 71c4b6c..29da810 100644 --- a/skills/remix/references/providers.md +++ b/skills/remix/references/providers.md @@ -11,7 +11,7 @@ no MCP server is involved. | `video.generate` | AI video | TapVid → MiniMax → fal → Ark → Kling → Runway → Luma → Gemini → Replicate | | `image.generate` | AI image | TapVid → fal → OpenAI → Gemini → MiniMax → Ark → Replicate | | `speech.synthesize`, `speech.voices` | voice-over, voice list | TapVid → ElevenLabs → MiniMax → OpenAI → Gemini | -| `speech.transcribe` | transcribing the original | ElevenLabs → local Whisper → OpenAI | +| `speech.transcribe` | transcribing the original | local Whisper → ElevenLabs → OpenAI | | `video.understand` | AI watching the original | Gemini | | `music.generate` | AI music | ElevenLabs → MiniMax | | `music.search` | royalty-free music | Jamendo | @@ -64,6 +64,9 @@ its default model that fits. "models": { "video.generate": { "fal": "bytedance/seedance-2.5/reference-to-video" } } } ``` +Generation goes to TapVid first unless the user names another vendor: never pin or suggest fal, MiniMax or any +other vendor on your own while TapVid is ready. + A vendor the user picks for one slot goes into that slot's `provider` (and `model`) in `replica.json` and wins over routing. A vendor the user prefers for a whole capability goes into `config.json` `routing`. diff --git a/skills/remix/references/replica.md b/skills/remix/references/replica.md index cf52fa4..b2d528b 100644 --- a/skills/remix/references/replica.md +++ b/skills/remix/references/replica.md @@ -138,6 +138,15 @@ Each media entry: } ``` +**A slot with a person starts as `keyframe` at `high`** (`replica-init.mjs` writes it so, and `validate.mjs replica` +rejects `reference` there). A reference video carries the original's person, and a model can copy that person +however the prompt and the replacement image read. Showing a real person's likeness in the film is a fault no +quality can offset, so `reference` stays for slots with no recognisable person: objects, places, and with a stated +`reference_ok` ("a silhouette, no one recognisable") a person who cannot be told apart. For a keyframe, point `frame_t` of the slot in `analysis/boards.json` at the window's start (its +default is the middle, which may be another moment of a moving shot) and run `refs.mjs --only `; `validate.mjs +replica` then asks for the slot's sketch (`references/sketches.md`), its fallback if generation fails. The +trade-off is that the camera move then follows the description, not the reference clip: say so in the delivery report. + `visual` is the one line the user reads in the table (their language); `prompt` is English for the provider; `provider` / `model` pin a vendor the user chose (otherwise routing picks); `sketch` is set for `code` slots and for `ai` slots without a reference. @@ -175,6 +184,10 @@ screen), describe its new-topic version in the prompt instead and drop that clau > {Subject and action}. {Framing and camera move}. {Light, palette matching the film}. Style as in image 1. No > text, no logos. +Models generate at least 5 s on most vendors, whatever the shot needs. A generated clip longer than its scene is +sped up by `project.mjs` to fit, at most 2.5×, and past that the end is cut: write one quick action for a shot +under 2 s, because only its first 2.5 × shot seconds show. A reference clip under 2 s is slowed to 2 s when it is uploaded. + Rules: one action per shot; at most two hands in frame and say what each holds; the product's shape and colour in words even when an image is given; describe what to show, never "do not show the old product"; keep areas the board covers with text empty ("leave the left third clear"). @@ -189,7 +202,8 @@ replacements (the user's photo, a product shot, a screenshot of their app): > the reference video; never its people, {phone / screen} content, text or setting. No captions, no subtitles, > no text overlays. -MiniMax H3 keeps the person in image 1 recognisable and copies the reference's handheld move and framing; a +MiniMax H3 copies the reference's handheld move and framing. It can keep the person in image 1 recognisable, and +it can also keep the reference's person instead, so every such clip passes the identity gate (§9); a screen in the shot comes out as a likeness of the screenshot whose small text is not readable. When the screen must be readable, make that slot `user` (a capture or the user's screenshot) instead. A layer with `cutout` needs only its people to read well: ask for a plain, evenly lit background behind them. @@ -245,4 +259,13 @@ own voice is the take (`references/audio.md` §2). morphing product, garbled text, clipping through objects, the subject drifting back to the original or into someone else. A clean span that covers the slot is trimmed and used; otherwise state the cost and regenerate once with the prompt fixed (fewer hands, one action, a keyframe). +- **Identity gate, every clip that shows a person.** Beside the replacement image and the original's subject + frame, read `inspect.mjs "$REPLICA" --identity` (the first second at 8 fps, the rest at 2 fps, at full width, + because a person can be a few dozen pixels in a wide or overhead shot; zoom a crop when the face still cannot be + told apart). Record what you saw: `identity.mjs pass|fail "$REPLICA" --saw ""`. Wrong person, or a + person you cannot identify, is a failure: state nothing as swapped until you have seen the face. Do not trim + around it: regenerate as `keyframe`. A fail takes the verdict off all the film's `reference` clips, since one leak says + that mode is copying people on this source; a regenerated clip loses its own. `render.mjs` refuses while + `identity.mjs status "$REPLICA"` lists anything, with no flag to skip it. A digital human and any slot whose + reference or `replace` names a `person` subject is covered, so give every person a subject in `analysis/boards.json`. - A slot that still fails becomes its sketch (`source: "code"`) and is listed in the delivery report. diff --git a/skills/remix/scripts/frames.mjs b/skills/remix/scripts/frames.mjs index a71e350..730ddf3 100644 --- a/skills/remix/scripts/frames.mjs +++ b/skills/remix/scripts/frames.mjs @@ -1,28 +1,34 @@ #!/usr/bin/env node // Self-check frames (spec §4 steps 5 and 9.3), captured from a built project. // node frames.mjs [--preview] [--no-build] -// every scene's midpoint, start / middle / end of every transition (or just before / at / -// after a hard cut) into review/frames/, and a 1-frame-per-second contact sheet -// review/contact-sheet.jpg. Uses the final project when replica.resolved covers every -// scene, else (or with --preview) the preview project. +// every scene's midpoint, just before / after every narration line's boundary, start / middle / +// end of every transition (or just before / at / after a hard cut) into review/frames/, and a +// 1-frame-per-second contact sheet review/contact-sheet.jpg. Scans the film for two texts +// overlapping and exits 1 listing them. Uses the final project when replica.resolved covers +// every scene, else (or with --preview) the preview project. // node frames.mjs --sketch [--duration s] // 25 / 50 / 85 % frames of one sketch into review/sketches/-.jpg. import { mkdirSync, rmSync } from 'node:fs'; import { basename, join, resolve } from 'node:path'; import { fail, fromPosixRel, parseArgs, readJson, run, toPosixRel } from './lib/common.mjs'; -import { openProject } from './lib/capture.mjs'; +import { openProject, overlappingTexts, visibleTextParts } from './lib/capture.mjs'; import { buildProject, generateProject, generateSketchProject, projectDirFor } from './lib/project.mjs'; import { loadReplica, sceneKey } from './lib/replica.mjs'; const SKETCH_MARKS = [0.25, 0.5, 0.85]; const SHEET_COLUMNS = 6; const SHEET_TILE_WIDTH = 270; +// How far either side of a line boundary to grab: past a fade, inside the neighbouring line. +const LINE_BOUNDARY_S = 0.15; +// Overlap scan rate: the shortest title in the measured originals stays about 0.8 s. +const SCAN_HZ = 4; const args = parseArgs(); if (!args._[0]) fail('usage: frames.mjs [--preview] [--no-build] | --sketch [--duration s]', 2); const ctx = loadReplica(args._[0]); const review = join(ctx.dir, 'review'); const saved = []; +let overlaps = []; async function grabAll(meta, marks, outDir) { mkdirSync(outDir, { recursive: true }); @@ -42,6 +48,27 @@ async function grabAll(meta, marks, outDir) { } } +// Every pair of overlapping texts, once per pair, with when it starts. Transitions are left out: two +// scenes are on screen by design there. +async function scanOverlaps(meta) { + const cap = await openProject(meta.dir, meta.canvas, { scale: 0.5 }); + const found = new Map(); + try { + for (let t = 0; t < cap.duration; t += 1 / SCAN_HZ) { + const scene = meta.scenes.filter((s) => s.start <= t + 1e-6).pop(); + if (t < scene.start + (scene.transitionIn?.duration || 0)) continue; + await cap.seek(t); + for (const [a, b] of overlappingTexts(await cap.page.evaluate(visibleTextParts))) { + const pair = `${scene.key}: ${a.part} "${a.text}" over ${b.part} "${b.text}"`; + if (!found.has(pair)) found.set(pair, t); + } + } + } finally { + await cap.close(); + } + return [...found].map(([pair, t]) => `${pair} at ${t.toFixed(2)}s`); +} + // Media slot of a sketch file: its aspect and duration. function slotOf(file) { for (const scene of ctx.replica.scenes) { @@ -70,6 +97,11 @@ try { const items = meta.scenes; const step = 1 / meta.fps; const marks = items.map((it) => [it.start + it.duration / 2, `${it.key}-mid`]); + for (const it of items) { + (it.segments || []).slice(1).forEach((seg, j) => { + marks.push([it.start + seg.start - LINE_BOUNDARY_S, `${it.key}-line-${j + 1}-before`], [it.start + seg.start + LINE_BOUNDARY_S, `${it.key}-line-${j + 1}-after`]); + }); + } for (let k = 1; k < items.length; k++) { const it = items[k]; const T = it.transitionIn?.duration || 0; @@ -87,8 +119,10 @@ try { '-vf', `scale=${SHEET_TILE_WIDTH}:-1,tile=${SHEET_COLUMNS}x${Math.ceil(n / SHEET_COLUMNS)}:padding=4:color=black`, '-frames:v', '1', join(review, 'contact-sheet.jpg')]); rmSync(sheetDir, { recursive: true, force: true }); saved.push({ t: null, path: 'review/contact-sheet.jpg' }); + overlaps = await scanOverlaps(meta); } } catch (e) { fail(e.stderr?.toString() || e.message); } console.log(JSON.stringify(saved, null, 2)); +if (overlaps.length) fail(`texts overlap:\n ${overlaps.join('\n ')}`); diff --git a/skills/remix/scripts/identity.mjs b/skills/remix/scripts/identity.mjs new file mode 100644 index 0000000..062c83e --- /dev/null +++ b/skills/remix/scripts/identity.mjs @@ -0,0 +1,27 @@ +#!/usr/bin/env node +// Step 7, the identity gate (references/replica.md §9): the verdict on a generated clip of a person. +// node inspect.mjs --identity read its frames first +// node identity.mjs pass --saw "" +// node identity.mjs fail --saw "" +// node identity.mjs status exit 1 while anything blocks render.mjs +import { fail, parseArgs } from './lib/common.mjs'; +import { identityProblems, signOff } from './lib/identity.mjs'; +import { loadReplica } from './lib/replica.mjs'; + +const USAGE = 'usage: identity.mjs pass|fail --saw "" | identity.mjs status '; +const args = parseArgs(); +const [command, dir, id] = args._; +if (!dir || !['pass', 'fail', 'status'].includes(command) || (command !== 'status' && !id)) fail(USAGE, 2); + +const ctx = loadReplica(dir); +if (command !== 'status') { + try { + signOff(ctx, id, command, { saw: args.saw }); + } catch (e) { + fail(e.message); + } +} +const problems = identityProblems(loadReplica(dir)); +for (const p of problems) console.log(p); +if (command === 'status' && problems.length) process.exit(1); +if (!problems.length) console.log('identity gate clear'); diff --git a/skills/remix/scripts/inspect.mjs b/skills/remix/scripts/inspect.mjs index cdaf456..b71ca36 100644 --- a/skills/remix/scripts/inspect.mjs +++ b/skills/remix/scripts/inspect.mjs @@ -1,15 +1,20 @@ #!/usr/bin/env node // Step 7 (spec §4): frames of a generated clip for the anatomy / product / subject-drift check. // Frames are written one per file, never tiled, at a size where fingers can be counted. -// node inspect.mjs [--fps 2] [--width 720] +// node inspect.mjs [--fps 2] [--width 720] [--identity] +// --identity is the read the identity gate asks for (lib/identity.mjs): the first second at 8 fps, the +// rest at 2 fps, at full width, because a person can be a few dozen pixels in a wide or overhead shot. // Prints JSON: { clip, duration, frames: [{ time, path }] } with paths relative to the replica. import { existsSync, mkdirSync, readdirSync, rmSync } from 'node:fs'; import { basename, join, resolve } from 'node:path'; import { fail, fromPosixRel, parseArgs, readJson, run, toPosixRel } from './lib/common.mjs'; +import { recordInspection } from './lib/identity.mjs'; + +const IDENTITY = { headS: 1, headFps: 8, restFps: 2, width: 1920 }; const args = parseArgs(); const [dirArg, target] = args._; -if (!dirArg || !target) fail('usage: inspect.mjs [--fps 2] [--width 720]', 2); +if (!dirArg || !target) fail('usage: inspect.mjs [--fps 2] [--width 720] [--identity]', 2); const dir = resolve(dirArg); const rate = Number(args.fps || 2); const width = Number(args.width || 720); @@ -23,9 +28,22 @@ const id = fromTask ? target : basename(clip).replace(/\.[^.]+$/, ''); const out = join(dir, 'review', 'gen', id); rmSync(out, { recursive: true, force: true }); mkdirSync(out, { recursive: true }); -run('ffmpeg', ['-v', 'error', '-y', '-i', clip, '-vf', `fps=${rate},scale=${width}:-2`, '-q:v', '3', join(out, '%02d.jpg')]); - const duration = Number(Number(run('ffprobe', ['-v', 'error', '-show_entries', 'format=duration', '-of', 'csv=p=0', clip])).toFixed(3)); +const grab = (seek, fps, count, name) => run('ffmpeg', ['-v', 'error', '-y', ...(seek ? ['-ss', String(seek)] : []), '-i', clip, + '-vf', `fps=${fps},scale=${args.identity ? IDENTITY.width : width}:-2`, '-q:v', '3', ...(count ? ['-frames:v', String(count)] : []), join(out, name)]); + +let times; +if (args.identity) { + const head = Math.ceil(Math.min(duration, IDENTITY.headS) * IDENTITY.headFps); + const rest = Math.ceil(Math.max(0, duration - IDENTITY.headS) * IDENTITY.restFps); + grab(0, IDENTITY.headFps, head, 'a%02d.jpg'); + if (rest) grab(IDENTITY.headS, IDENTITY.restFps, rest, 'b%02d.jpg'); + times = (i) => (i < head ? i / IDENTITY.headFps : IDENTITY.headS + (i - head) / IDENTITY.restFps); +} else { + grab(0, rate, 0, '%02d.jpg'); + times = (i) => i / rate; +} const frames = readdirSync(out).filter((f) => f.endsWith('.jpg')).sort() - .map((f, i) => ({ time: Number((i / rate).toFixed(2)), path: toPosixRel(dir, join(out, f)) })); + .map((f, i) => ({ time: Number(times(i).toFixed(3)), path: toPosixRel(dir, join(out, f)) })); +if (args.identity && fromTask) recordInspection(dir, id, clip); console.log(JSON.stringify({ clip: toPosixRel(dir, clip), duration, frames }, null, 2)); diff --git a/skills/remix/scripts/lib/capture-page.mjs b/skills/remix/scripts/lib/capture-page.mjs index a221673..ba7ca7e 100644 --- a/skills/remix/scripts/lib/capture-page.mjs +++ b/skills/remix/scripts/lib/capture-page.mjs @@ -84,6 +84,23 @@ export function installCapturePage({ fake, mask, cursorId, rippleMs }) { }, true); } +// Runs in the page: changes the black background invisibly on every frame for `ms`, ending on black. +export function warmRecorder(ms) { + return new Promise((resolve) => { + const { style } = document.body; + const end = performance.now() + ms; + const tick = () => { + if (performance.now() >= end) { + style.background = '#000'; + return resolve(); + } + style.background = style.background === 'rgb(1, 1, 1)' ? '#000' : '#010101'; + requestAnimationFrame(tick); + }; + tick(); + }); +} + // Runs in the page: a full-viewport white marker with a black centre square for at least // `durationMs` on the black start page; resolves the wall-clock ms at which it entered the // DOM, from a task like any page update, so it reaches the video the way the page's changes do. diff --git a/skills/remix/scripts/lib/capture-record.mjs b/skills/remix/scripts/lib/capture-record.mjs index bffc875..dd48440 100644 --- a/skills/remix/scripts/lib/capture-record.mjs +++ b/skills/remix/scripts/lib/capture-record.mjs @@ -5,7 +5,7 @@ import { tmpdir } from 'node:os'; import { join, resolve } from 'node:path'; import { HOME, launchBrowser, readJson, runAsync, writeJson } from './common.mjs'; import { ffmpeg, probe, videoInfo } from './video.mjs'; -import { flashMarker, installCapturePage, parseFake, smoothScroll } from './capture-page.mjs'; +import { flashMarker, installCapturePage, parseFake, smoothScroll, warmRecorder } from './capture-page.mjs'; export const AUTH_DIR = join(HOME, 'auth'); // Saved sessions are keyed by host; a port's colon is not allowed in a Windows file name. @@ -42,10 +42,13 @@ const CURSOR_ID = '__tapvid-cursor'; // for three recording intervals, with a 250 ms margin for a busy compositor at high fps. const FLASH_MIN_MS = 250; const FLASH_RECORDING_FRAMES = 3; -// The recorder starts a few frames after the page opens; a flash shown at once was found already -// running in the first recorded frame (its start lost, the alignment off by those frames). Half a -// second of black page before it leaves a wide margin. +// The recorder only gets a frame when the page changes: after a still black page the flash could be the +// first recorded frame, however long the wait (under CPU load, 16 of 48 at 500 ms, 19 of 48 at 2 s). A page +// changing for this long before the flash put a frame ahead of it every time (0 of 96). const RECORDER_WARMUP_MS = 500; +// The flash then waits on a still page: right after that changing page it reached the video about 50 ms +// later than the page's later changes do, which shifted every logged time by that much. +const RECORDER_SETTLE_MS = 250; // The flash is the first thing recorded; it is searched for in this many seconds. const FLASH_SEARCH_S = 10; // The flash search decodes the recording at this size: enough to tell the black square from the white. @@ -296,7 +299,8 @@ export async function record(replicaArg, storyboardFile) { await context.addInitScript(installCapturePage, { fake: sb.fake, mask: sb.mask, cursorId: CURSOR_ID, rippleMs: RIPPLE_MS }); const page = await context.newPage(); await page.setContent(''); - await sleep(RECORDER_WARMUP_MS); + await page.evaluate(warmRecorder, RECORDER_WARMUP_MS); + await sleep(RECORDER_SETTLE_MS); const flashWall = await page.evaluate(flashMarker, Math.max(FLASH_MIN_MS, 1000 * FLASH_RECORDING_FRAMES / fps)); const { act, trail } = driver(page, viewport, fps); // Shots keep the drawn cursor: hiding it for the screenshot would blank it in the video too. diff --git a/skills/remix/scripts/lib/capture.mjs b/skills/remix/scripts/lib/capture.mjs index 760955f..bac66f4 100644 --- a/skills/remix/scripts/lib/capture.mjs +++ b/skills/remix/scripts/lib/capture.mjs @@ -51,6 +51,41 @@ export async function openProject(dir, canvas, { scale = 1, browser, page: pageF }; } +// In-page: every visible [data-part] with text of its own, as { part, text, box } with the box tight around +// the text (an element's own box can be a whole region), leaving out a part that holds another text part. +export function visibleTextParts() { + const all = [...document.querySelectorAll('[data-part]')].filter((el) => el.textContent.trim()); + const out = []; + for (const el of all) { + if (all.some((o) => o !== el && el.contains(o))) continue; + if (getComputedStyle(el).visibility === 'hidden') continue; + let opacity = 1; + for (let n = el; n && n.nodeType === 1; n = n.parentElement) opacity *= Number(getComputedStyle(n).opacity); + if (opacity <= 0.05) continue; + const range = document.createRange(); + range.selectNodeContents(el); + const r = range.getBoundingClientRect(); + if (r.width > 0 && r.height > 0) out.push({ part: el.dataset.part, text: el.textContent.trim().slice(0, 40), box: [r.left, r.top, r.right, r.bottom] }); + } + return out; +} + +// Pairs of texts sharing more than this share of the smaller one: line boxes touch without the words colliding. +const MIN_OVERLAP = 0.05; + +export function overlappingTexts(parts) { + const out = []; + parts.forEach((a, i) => { + for (const b of parts.slice(i + 1)) { + const w = Math.min(a.box[2], b.box[2]) - Math.max(a.box[0], b.box[0]); + const h = Math.min(a.box[3], b.box[3]) - Math.max(a.box[1], b.box[1]); + const area = ([l, t, r, d]) => (r - l) * (d - t); + if (w > 0 && h > 0 && (w * h) / Math.min(area(a.box), area(b.box)) > MIN_OVERLAP) out.push([a, b]); + } + }); + return out; +} + // In-page: every [data-sfx] element present and visible now (non-zero box, not // visibility:hidden, opacity through its ancestors > 0.05) as { id, kind, gain, offset, index }: // id is data-sfx-id or null, offset the data-time-offset of its scene wrapper (projectPlayer.js: diff --git a/skills/remix/scripts/lib/fit-clip.mjs b/skills/remix/scripts/lib/fit-clip.mjs new file mode 100644 index 0000000..3a0d997 --- /dev/null +++ b/skills/remix/scripts/lib/fit-clip.mjs @@ -0,0 +1,30 @@ +// A generated clip against the time its slot is on screen. Models have a minimum length (5 s on most), so a +// shot of 2 s gets a 5 s clip: played as it is, only its opening shows and its move never happens. The clip +// is sped up to fit, and past the speed limit the rest is cut. +import { existsSync } from 'node:fs'; +import { run } from './common.mjs'; +import { probeDuration } from './replica.mjs'; + +// Above this a generated move stops reading as camera or hand motion. It lets a 5 s clip finish in a 2 s shot. +const MAX_CLIP_SPEED = 2.5; +// A clip this close to its slot plays as it is: the scene's end cuts the difference. +const SPEED_TOLERANCE = 1.05; + +// → null when the clip plays as it is, else { speed, cutS }: cutS seconds of the clip's end never show. +export function clipFit(clipS, targetS) { + if (!(clipS > 0) || !(targetS > 0) || clipS / targetS <= SPEED_TOLERANCE) return null; + const speed = Math.min(clipS / targetS, MAX_CLIP_SPEED); + return { speed: Number(speed.toFixed(3)), cutS: Number(Math.max(0, clipS - targetS * speed).toFixed(3)) }; +} + +// → { file, fit }: the copy of `file` retimed to targetS, next to it, or `file` itself when fit is null. +export function fitClip(file, targetS) { + const fit = clipFit(probeDuration(file), targetS); + if (!fit) return { file, fit }; + const out = `${file.replace(/\.[^.]+$/, '')}.fit-${targetS.toFixed(3)}.mp4`; + if (!existsSync(out)) { + run('ffmpeg', ['-y', '-v', 'error', '-i', file, '-an', '-vf', `setpts=PTS/${fit.speed}`, '-t', String(targetS), + '-c:v', 'libx264', '-preset', 'veryfast', '-crf', '17', '-pix_fmt', 'yuv420p', out]); + } + return { file: out, fit }; +} diff --git a/skills/remix/scripts/lib/identity.mjs b/skills/remix/scripts/lib/identity.mjs new file mode 100644 index 0000000..42633e7 --- /dev/null +++ b/skills/remix/scripts/lib/identity.mjs @@ -0,0 +1,78 @@ +// The identity gate (references/replica.md §9): a generated clip showing a person reaches the film +// only after someone looked at it frame by frame and saw the replacement, not the original's person. +// No script can tell who a face is, so the gate enforces the looking: `inspect.mjs --identity` must +// have read this very clip before a verdict is taken, a regenerated clip voids its verdict, and +// render.mjs refuses a film with an open slot. Records live in /identity.json. +import { createHash } from 'node:crypto'; +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { fromPosixRel, readJson, writeJson } from './common.mjs'; +import { slotRequestId } from './media/plan.mjs'; +import { taskFiles } from './replica.mjs'; + +const recordsFile = (dir) => join(dir, 'identity.json'); +const readRecords = (dir) => readJson(recordsFile(dir), { clips: {} }); +const sha256 = (file) => createHash('sha256').update(readFileSync(file)).digest('hex'); + +export const templateTier = (ctx) => readJson(join(ctx.cloneRoot, 'analysis', 'brief.json'), {})?.tier; + +const personSubjects = (refs) => new Set(Object.values(refs || {}) + .flatMap((r) => r.subjects || []).filter((s) => s.kind === 'person').map((s) => s.id)); + +// A reference clip with a person in it carries that person into the model's input. +export const refShowsPerson = (ref) => (ref?.subjects || []).some((s) => s.kind === 'person'); + +export function slotShowsPerson(ctx, m) { + if (m.kind === 'avatar' || refShowsPerson(ctx.refs?.[m.ref])) return true; + const people = personSubjects(ctx.refs); + return (m.replace || []).some((id) => people.has(id)); +} + +// The generated clips the gate covers. The user's own files are theirs; a request that did not +// complete has no clip, and its slot falls back to the sketch. +export function identitySlots(ctx) { + const { replica } = ctx; + return replica.scenes.flatMap((scene, index) => Object.entries(scene.media || {}).flatMap(([path, m]) => { + if (m.source !== 'ai' || !['video', 'avatar'].includes(m.kind) || !slotShowsPerson(ctx, m)) return []; + const id = slotRequestId(replica.slug, index, scene.board, path); + const file = taskFiles(ctx.tasks, id)[0]; + return file ? [{ id, mode: m.mode, clip: fromPosixRel(ctx.dir, file.path) }] : []; + })); +} + +// inspect.mjs --identity has read this clip: the verdict may now be given. +export function recordInspection(dir, id, clip) { + const records = readRecords(dir); + writeJson(recordsFile(dir), { clips: { ...records.clips, [id]: { sha256: sha256(clip), inspected: true, verdict: null } } }); +} + +export function signOff(ctx, id, verdict, { saw } = {}) { + const slot = identitySlots(ctx).find((s) => s.id === id); + if (!slot) throw new Error(`${id}: not a generated clip of a person in this replica`); + const records = readRecords(ctx.dir); + const record = records.clips[id]; + if (!record?.inspected || record.sha256 !== sha256(slot.clip)) { + throw new Error(`${id}: run inspect.mjs ${ctx.dir} ${id} --identity and look at every frame first`); + } + if (!saw) throw new Error(`${id}: --saw is required: say whose face you saw and where in the clip`); + const clips = { ...records.clips, [id]: { ...record, verdict, saw } }; + // One leak says reference mode is copying people on this source: its other clips are read again. + if (verdict === 'fail') { + for (const other of identitySlots(ctx)) { + if (other.id !== id && other.mode === 'reference') delete clips[other.id]; + } + } + writeJson(recordsFile(ctx.dir), { clips }); +} + +// What still stops the render; empty when every clip of a person passed. +export function identityProblems(ctx) { + const { clips } = readRecords(ctx.dir); + return identitySlots(ctx).flatMap((slot) => { + const record = clips[slot.id]; + if (!record?.verdict) return [`${slot.id}: no identity check (inspect.mjs --identity, then identity.mjs pass or fail)`]; + if (record.sha256 !== sha256(slot.clip)) return [`${slot.id}: the clip changed since its identity check; check it again`]; + if (record.verdict !== 'pass') return [`${slot.id}: failed the identity check (${record.saw}); regenerate it as keyframe`]; + return []; + }); +} diff --git a/skills/remix/scripts/lib/media/adapt.mjs b/skills/remix/scripts/lib/media/adapt.mjs index 1a834af..0ace3b7 100644 --- a/skills/remix/scripts/lib/media/adapt.mjs +++ b/skills/remix/scripts/lib/media/adapt.mjs @@ -73,6 +73,7 @@ export function adaptToModel(capability, params, model) { Object.assign(resolved, fitResolution(params.min_short_edge, model.resolutions)); } if (refVideos.length && model.refVideoMaxS) resolved.refVideoMaxS = model.refVideoMaxS; + if (refVideos.length && model.refVideoMinS) resolved.refVideoMinS = model.refVideoMinS; return { ok: true, resolved }; } diff --git a/skills/remix/scripts/lib/media/ctx.mjs b/skills/remix/scripts/lib/media/ctx.mjs index 85e1817..afb2bfa 100644 --- a/skills/remix/scripts/lib/media/ctx.mjs +++ b/skills/remix/scripts/lib/media/ctx.mjs @@ -127,19 +127,22 @@ export function createCtx({ provider, secrets, fetchImpl = globalThis.fetch, cac return { duration_s: Number(out.format?.duration) || null, width: v.width ?? null, height: v.height ?? null }; } - // Cut a reference video to the model's limits (length and size), re-encoded to H.264. - function trim(file, { maxS, maxBytes } = {}) { + // Fit a reference video to the model's limits, re-encoded to H.264: cut when too long or too big, slowed + // when too short (the original's move stays whole; looping it would add a jump back to the start). + function trim(file, { minS, maxS, maxBytes } = {}) { const src = abs(file); const info = probe(src); const tooLong = maxS && info.duration_s > maxS; const tooBig = maxBytes && statSync(src).size > maxBytes; - if (!tooLong && !tooBig) return src; - const out = join(cacheDir, 'trim', `${sha256File(src).slice(0, 16)}-${maxS || 'full'}-${maxBytes || 'any'}.mp4`); + // A hair over the minimum: the encoded length rounds to whole frames. + const stretch = minS && info.duration_s < minS ? (minS * 1.02) / info.duration_s : null; + if (!tooLong && !tooBig && !stretch) return src; + const out = join(cacheDir, 'trim', `${sha256File(src).slice(0, 16)}-${minS || 'any'}-${maxS || 'full'}-${maxBytes || 'any'}.mp4`); if (existsSync(out)) return out; mkdirSync(dirname(out), { recursive: true }); const args = ['-y', '-v', 'error', '-i', src]; if (tooLong) args.push('-t', String(maxS)); - args.push('-c:v', 'libx264', '-preset', 'veryfast', '-crf', tooBig ? '28' : '20', '-vf', 'scale=-2:min(ih\\,1080)', '-an', out); + args.push('-c:v', 'libx264', '-preset', 'veryfast', '-crf', tooBig ? '28' : '20', '-vf', `${stretch ? `setpts=PTS*${stretch.toFixed(4)},` : ''}scale=-2:min(ih\\,1080)`, '-an', out); run('ffmpeg', args); return out; } diff --git a/skills/remix/scripts/lib/media/router.mjs b/skills/remix/scripts/lib/media/router.mjs index 6e0736e..6bdd007 100644 --- a/skills/remix/scripts/lib/media/router.mjs +++ b/skills/remix/scripts/lib/media/router.mjs @@ -11,7 +11,8 @@ export const DEFAULT_ORDER = { 'image.generate': ['tapvid', 'fal', 'openai', 'gemini', 'minimax', 'ark', 'replicate'], 'speech.synthesize': ['tapvid', 'elevenlabs', 'minimax', 'openai', 'gemini'], 'speech.voices': ['tapvid', 'elevenlabs', 'minimax', 'openai', 'gemini'], - 'speech.transcribe': ['elevenlabs', 'whisper-local', 'openai'], + // Local first: a transcript only drives timing and reading the script, so it need not cost anything. + 'speech.transcribe': ['whisper-local', 'elevenlabs', 'openai'], 'video.understand': ['gemini'], 'music.generate': ['elevenlabs', 'minimax'], 'music.search': ['jamendo'], diff --git a/skills/remix/scripts/lib/project.mjs b/skills/remix/scripts/lib/project.mjs index 74a4c72..7221d3e 100644 --- a/skills/remix/scripts/lib/project.mjs +++ b/skills/remix/scripts/lib/project.mjs @@ -12,6 +12,7 @@ import { spawnSync } from 'node:child_process'; import { basename, extname, join } from 'node:path'; import { ASSETS_DIR, PLUGIN, fromPosixRel, run, runtimePath, toFrame, toPosixRel, writeJson } from './common.mjs'; import { cutoutFresh, cutoutPath } from './matte.mjs'; +import { fitClip } from './fit-clip.mjs'; import { fieldForPath, fieldPaths, replaceRequestId, slotRequestId } from './media/plan.mjs'; import { wordTokens } from './pace.mjs'; import { layoutNarration, scaleSourceSegments } from './timing.mjs'; @@ -77,7 +78,7 @@ function chipsFor(ctx, ref, subjects) { } // One scene's media: the JSX expression for each media path plus imports and asset files. -function sceneMedia(ctx, mode, scene, index, key, out) { +function sceneMedia(ctx, mode, scene, index, key, out, visibleS) { const imports = []; const assets = {}; const exprs = {}; @@ -153,7 +154,14 @@ function sceneMedia(ctx, mode, scene, index, key, out) { const task = taskFiles(ctx.tasks, slotRequestId(ctx.replica.slug, index, scene.board, path))[0]; // timing.mjs may have sped a self-voiced avatar clip up to the film's tempo: that copy plays. const sped = ctx.replica.resolved?.[key]?.media?.[path]; - const gen = (sped && fromPosixRel(ctx.dir, sped)) || userFile || (task && fromPosixRel(ctx.dir, task.path)); + let gen = (sped && fromPosixRel(ctx.dir, sped)) || userFile || (task && fromPosixRel(ctx.dir, task.path)); + // A generated clip longer than its scene is retimed so its move finishes on screen: every clip runs on the + // scene's clock, a slot shown for part of the scene too. Not one the user chose or one that loops. + if (m.kind === 'video' && !userFile && gen && existsSync(gen) && !loops(path)) { + const fitted = fitClip(gen, visibleS); + gen = fitted.file; + if (fitted.fit?.cutS) warnings.push(`${key}.${path}: the clip is too long for ${visibleS}s even at ${fitted.fit.speed}×; its last ${fitted.fit.cutS}s are cut`); + } // Once generated, the preview shows the new material; the reference / sketch is only a stand-in. if (mode === 'preview' && gen && existsSync(gen)) { exprs[path] = asset(path, gen, tag, slotAttrs(path, `gen:${kind}`, [], m.prompt)); @@ -347,7 +355,9 @@ export function generateProject(replicaDirArg, { mode, out: outArg } = {}) { timing = { duration: r.duration, segments: r.segments || [], words: r.words, flag: null }; } else timing = estimateScene(ctx, mode, scene, transitionIn, occ[i]); const duration = toFrame(timing.duration, fps); - const media = sceneMedia(ctx, mode, scene, i, key, out); + // The outgoing scene stays on screen through the next boundary's transition. + const overlap = incomingTransition(template, order, i + 1)?.duration || 0; + const media = sceneMedia(ctx, mode, scene, i, key, out, round3(duration + overlap)); Object.assign(manifest, media.assets); warnings.push(...media.warnings); const segments = (timing.segments || []).map((s) => ({ key: s.key, start: round3(s.start), end: round3(s.end) })); diff --git a/skills/remix/scripts/lib/validate-replica.mjs b/skills/remix/scripts/lib/validate-replica.mjs index c3d6db8..0800d78 100644 --- a/skills/remix/scripts/lib/validate-replica.mjs +++ b/skills/remix/scripts/lib/validate-replica.mjs @@ -3,6 +3,7 @@ import { existsSync } from 'node:fs'; import { join } from 'node:path'; import { fromPosixRel, readJson } from './common.mjs'; +import { refShowsPerson, templateTier } from './identity.mjs'; import { check } from './schema.mjs'; import { PACE_TOLERANCE, speechRate, textSyllables } from './pace.mjs'; import { loadReplica, mediaPaths, narrationLines, occurrences, sourceWindow } from './replica.mjs'; @@ -60,6 +61,9 @@ function checkMedia(ctx, scene, key, fields, errors, warnings) { const ref = m.ref ? refs[m.ref] : null; if (m.ref && !ref) errors.push(`${at}: ref ${m.ref} is not in refs.json`); const hasClip = Boolean(ref?.clip); + if (m.source === 'ai' && m.mode === 'reference' && refShowsPerson(ref) && !m.reference_ok && templateTier(ctx) === 'high') { + errors.push(`${at}: its reference clip shows a person, and a reference clip can carry that person into the film: use mode keyframe, or say in reference_ok why no one is recognisable (a silhouette)`); + } if (m.source === 'ai' && m.mode === 'reference' && !hasClip) errors.push(`${at}: mode reference needs a ref with a clip`); if (m.source === 'ai' && m.mode === 'keyframe' && !ref?.frame) errors.push(`${at}: mode keyframe needs a ref with a frame`); const needsSketch = m.source === 'code' && m.kind !== 'element' diff --git a/skills/remix/scripts/lib/validate-template.mjs b/skills/remix/scripts/lib/validate-template.mjs index e0f3de7..b204f65 100644 --- a/skills/remix/scripts/lib/validate-template.mjs +++ b/skills/remix/scripts/lib/validate-template.mjs @@ -153,6 +153,31 @@ export function checkBoardCaptions(name, meta, templateCaptions) { .map((k) => `board ${name}: meta.captions.off "${k}" is not a narration field (${lines.join(', ') || 'none'})`); } +// A segmented board's times are fractions of their line: the original's seconds land outside a shorter one. +// Only an entrance nudge may stay in seconds, and those measure well under this. +const MAX_SEGMENT_SECONDS = 0.5; + +export function checkSegmentSeconds(name, meta, source) { + const out = []; + for (const { key } of meta.example_segments || []) { + const escaped = key.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + for (const m of source.matchAll(new RegExp(`["'\`]${escaped}["'\`]\\)?\\s*[,+-]\\s*(\\d*\\.?\\d+)`, 'g'))) { + if (Number(m[1]) >= MAX_SEGMENT_SECONDS) out.push(`board ${name}: ${m[1]} s after line "${key}" looks like the original's timing — use a fraction of the line (references/boards.md §2)`); + } + } + return out; +} + +// The caption band of a board that shows captions, against its title and sticker boxes (0–1 of the frame). +export function checkCaptionBand(name, rule, design, elements) { + if (rule === false) return []; + const style = { ...design?.captions, ...(rule && typeof rule === 'object' ? rule : {}) }; + if (style.centerY == null || style.size == null) return []; + const [top, bottom] = [(style.centerY - style.size) / 100, (style.centerY + style.size) / 100]; + return elements.filter((e) => e.board === name && ['title', 'sticker'].includes(e.kind) && e.box[1] < bottom && e.box[1] + e.box[3] > top) + .map((e) => `board ${name}: captions at ${style.centerY}% run over ${e.id} (references/captions.md §1)`); +} + // A media.cutout field `host` reaches the board as a second prop `hostCutout` (the same clip with only the // people opaque), which the board draws over the elements behind the person. export function checkCutoutDrawn(name, fields, source) { @@ -299,6 +324,7 @@ export async function validateTemplate(templateDir) { const boardsDir = join(templateDir, 'boards'); const listed = Array.isArray(template.boards) ? template.boards : []; const boardFields = {}; + const captionRules = {}; const onDisk = existsSync(boardsDir) ? readdirSync(boardsDir).filter((f) => f.endsWith('.json')).map((f) => f.slice(0, -5)) : []; for (const extra of onDisk.filter((n) => !listed.includes(n))) warnings.push(`boards/${extra}.json is not in template.json boards`); for (const name of new Set([...listed, ...onDisk])) { @@ -314,8 +340,15 @@ export async function validateTemplate(templateDir) { if (schemaErrors.length) continue; errors.push(...checkSegments(name, meta), ...checkExample(name, meta.schema.fields, meta.example_data), ...checkMediaKeys(name, meta.schema.fields), ...checkBoardCaptions(name, meta, template.audio?.captions)); - if (existsSync(jsxPath)) errors.push(...checkCutoutDrawn(name, meta.schema.fields, readFileSync(jsxPath, 'utf-8'))); - if (listed.includes(name)) boardFields[name] = meta.schema.fields; + if (existsSync(jsxPath)) { + const source = readFileSync(jsxPath, 'utf-8'); + errors.push(...checkCutoutDrawn(name, meta.schema.fields, source)); + warnings.push(...checkSegmentSeconds(name, meta, source)); + } + if (listed.includes(name)) { + boardFields[name] = meta.schema.fields; + captionRules[name] = meta.meta?.captions; + } } const analysisDir = join(dirname(templateDir), 'analysis'); @@ -330,6 +363,9 @@ export async function validateTemplate(templateDir) { errors.push(...checkElements({ inventory, boardFields, design, motion: readJson(join(analysisDir, 'motion.json'), null), faces: readJson(join(templateDir, 'fonts.json'), { faces: [] }).faces || [] })); + if (template.audio?.captions) { + for (const name of Object.keys(captionRules)) warnings.push(...checkCaptionBand(name, captionRules[name], design, inventory.elements || [])); + } } } diff --git a/skills/remix/scripts/providers/_template.mjs b/skills/remix/scripts/providers/_template.mjs index 3d706fe..bc761e4 100644 --- a/skills/remix/scripts/providers/_template.mjs +++ b/skills/remix/scripts/providers/_template.mjs @@ -5,7 +5,7 @@ // meta name, displayName, docs, secrets[], concurrency, pollIntervalS?, timeoutS?, // capabilities: { '': { models: [model…] } } — model fields in // scripts/lib/media/adapt.mjs (aspects, duration/durations, resolutions, -// firstFrame, lastFrame, maxRefs, maxRefVideos, refVideoMaxS, modes, maxChars, price) +// firstFrame, lastFrame, maxRefs, maxRefVideos, refVideoMinS, refVideoMaxS, modes, maxChars, price) // verify(ctx) optional: cheapest authenticated call; throw MediaError('auth') on a bad key. // May return { unavailable: ['music.generate'] } for capabilities the account lacks. // estimate(cap, req) optional: { usd } | { credits }; default estimateFromMeta() on the model's price. @@ -16,7 +16,7 @@ // // req = { id, capability, params, model, resolved_params, out }. params follow // schemas/capabilities/.json; resolved_params hold what the runner fitted to the -// model (model, resolution, duration, aspect, crop, refVideoMaxS). Map them to the API's names; +// model (model, resolution, duration, aspect, crop, refVideoMinS, refVideoMaxS). Map them to the API's names; // throw MediaError('unsupported') for anything the API cannot do — never drop it silently. // // result = { files: [{ path, mime, duration_s?, width?, height?, words? }] } or { items: [...] }, @@ -25,7 +25,7 @@ // // ctx: secret(name), json(url, opts), bytes(url, opts), fetch(url, init), download(url, outPrefix), // writeOut(outPrefix, buffer, ext), upload(file, via), dataUri(file), blob(file), readFile(file), -// probe(file), trim(file, { maxS, maxBytes }), log(msg). json/bytes throw a classified MediaError +// probe(file), trim(file, { minS, maxS, maxBytes }), log(msg). json/bytes throw a classified MediaError // on non-2xx (401/403 auth, 402/429 quota, 400/422 invalid, moderation text → moderation, else transient). import { MediaError } from '../lib/media/errors.mjs'; import { estimateFromMeta } from '../lib/media/adapt.mjs'; diff --git a/skills/remix/scripts/providers/fal.mjs b/skills/remix/scripts/providers/fal.mjs index 9d7c69c..5295a3a 100644 --- a/skills/remix/scripts/providers/fal.mjs +++ b/skills/remix/scripts/providers/fal.mjs @@ -36,7 +36,7 @@ export const meta = { aspects: ['21:9', '16:9', '4:3', '1:1', '3:4', '9:16'], duration: { min: 4, max: 30 }, resolutions: [{ name: '480p', shortEdge: 480 }, { name: '720p', shortEdge: 720 }], - firstFrame: true, lastFrame: true, maxRefs: 30, maxRefVideos: 10, refVideoMaxS: 30, + firstFrame: true, lastFrame: true, maxRefs: 30, maxRefVideos: 10, refVideoMinS: 2, refVideoMaxS: 30, price: { usdPerSecond: { '480p': 0.2205, '720p': 0.473 } }, }], }, @@ -133,7 +133,7 @@ async function videoInput(req, ctx) { // 30 s combined across the clips; split it evenly (each clip ≥ 2 s). const perClip = Math.max(2, Math.floor(r.refVideoMaxS / refVideos.length)); input.video_urls = []; - for (const ref of refVideos) input.video_urls.push(await upload(ctx, ctx.trim(ref.file, { maxS: perClip, maxBytes: MAX_UPLOAD_BYTES }))); + for (const ref of refVideos) input.video_urls.push(await upload(ctx, ctx.trim(ref.file, { minS: r.refVideoMinS, maxS: perClip, maxBytes: MAX_UPLOAD_BYTES }))); } return { endpoint: `${r.model}/reference-to-video`, input }; } diff --git a/skills/remix/scripts/providers/minimax.mjs b/skills/remix/scripts/providers/minimax.mjs index 7193a50..13ddd0b 100644 --- a/skills/remix/scripts/providers/minimax.mjs +++ b/skills/remix/scripts/providers/minimax.mjs @@ -33,15 +33,15 @@ export const meta = { models: [ { id: 'MiniMax-H3', default: true, aspects: H3_ASPECTS, duration: { min: 4, max: 15 }, resolutions: [{ name: '768P', shortEdge: 768 }, { name: '2K', shortEdge: 1440 }], - firstFrame: true, lastFrame: true, maxRefs: 9, maxRefVideos: 3, refVideoMaxS: REF_VIDEO_TOTAL_S, + firstFrame: true, lastFrame: true, maxRefs: 9, maxRefVideos: 3, refVideoMinS: 2, refVideoMaxS: REF_VIDEO_TOTAL_S, price: { usdPerSecond: { '768P': 0.08, '2K': 0.13 } } }, { id: 'MiniMax-H3-Highspeed', aspects: H3_ASPECTS, duration: { min: 4, max: 15 }, resolutions: [{ name: '768P', shortEdge: 768 }], - firstFrame: true, lastFrame: true, maxRefs: 9, maxRefVideos: 3, refVideoMaxS: REF_VIDEO_TOTAL_S, + firstFrame: true, lastFrame: true, maxRefs: 9, maxRefVideos: 3, refVideoMinS: 2, refVideoMaxS: REF_VIDEO_TOTAL_S, price: { usdPerSecond: { '768P': 0.06 } } }, { id: 'MiniMax-H3-Lite', aspects: H3_ASPECTS, duration: { min: 4, max: 15 }, resolutions: [{ name: '480P', shortEdge: 480 }, { name: '768P', shortEdge: 768 }], - firstFrame: true, lastFrame: true, maxRefs: 9, maxRefVideos: 3, refVideoMaxS: REF_VIDEO_TOTAL_S, + firstFrame: true, lastFrame: true, maxRefs: 9, maxRefVideos: 3, refVideoMinS: 2, refVideoMaxS: REF_VIDEO_TOTAL_S, price: { usdPerSecond: { '480P': 0.025, '768P': 0.04 } } }, ], }, @@ -113,7 +113,7 @@ function videoBody(req, ctx) { // Split the 15 s combined allowance between the clips (each must stay ≥ 2 s). const perClip = refVideos.length ? Math.max(2, Math.floor(REF_VIDEO_TOTAL_S / refVideos.length)) : 0; for (const ref of refVideos) { - const file = ctx.trim(ref.file, { maxS: perClip, maxBytes: 30 * 1024 * 1024 }); + const file = ctx.trim(ref.file, { minS: r.refVideoMinS, maxS: perClip, maxBytes: 30 * 1024 * 1024 }); content.push({ type: 'video_url', video_url: { url: inline(ctx, file) }, role: 'reference_video' }); } // Image-to-video always follows the frame's aspect; the ratio is ignored there (adaptive). diff --git a/skills/remix/scripts/providers/tapvid.mjs b/skills/remix/scripts/providers/tapvid.mjs index c7bee61..7994b7b 100644 --- a/skills/remix/scripts/providers/tapvid.mjs +++ b/skills/remix/scripts/providers/tapvid.mjs @@ -42,11 +42,11 @@ export const meta = { { id: 'seedance-2.0-fast', default: true, aspects: ASPECTS_16_9, duration: { min: 5, max: 15 }, resolutions: res('480p', '720p'), firstFrame: true, maxRefs: 0, maxRefVideos: 0, price: { creditsPerSecond: { '480p': 8, '720p': 18 } } }, { id: 'seedance-2.5', aspects: ASPECTS_16_9, duration: { min: 5, max: 15 }, resolutions: res('480p', '720p', '1080p'), - maxRefs: 30, maxRefVideos: 10, refVideoMaxS: 15, billsReferenceVideo: true, price: { creditsPerSecond: { '480p': 16, '720p': 34, '1080p': 82 } } }, + maxRefs: 30, maxRefVideos: 10, refVideoMinS: 2, refVideoMaxS: 15, billsReferenceVideo: true, price: { creditsPerSecond: { '480p': 16, '720p': 34, '1080p': 82 } } }, { id: 'seedance-2.0', aspects: ASPECTS_16_9, duration: { min: 5, max: 15 }, resolutions: res('480p', '720p', '1080p'), - maxRefs: 9, maxRefVideos: 3, refVideoMaxS: 15, billsReferenceVideo: true, price: { creditsPerSecond: { '480p': 10, '720p': 22, '1080p': 48 } } }, + maxRefs: 9, maxRefVideos: 3, refVideoMinS: 2, refVideoMaxS: 15, billsReferenceVideo: true, price: { creditsPerSecond: { '480p': 10, '720p': 22, '1080p': 48 } } }, { id: 'minimax-h3', aspects: ASPECTS_16_9, duration: { min: 5, max: 15 }, resolutions: res('480P', '768P', '2K'), - maxRefs: 9, maxRefVideos: 3, refVideoMaxS: 15, freeRefs: 5, creditsPerExtraRef: 6, price: { creditsPerSecond: { '480P': 4, '768P': 5, '2K': 10 } } }, + maxRefs: 9, maxRefVideos: 3, refVideoMinS: 2, refVideoMaxS: 15, freeRefs: 5, creditsPerExtraRef: 6, price: { creditsPerSecond: { '480P': 4, '768P': 5, '2K': 10 } } }, { id: 'veo-3.1', aspects: ASPECTS_16_9, durations: [4, 6, 8], resolutions: res('720p', '1080p'), maxRefs: 3, maxRefVideos: 0, price: { creditsPerSecond: { '720p': 29, '1080p': 29 } } }, { id: 'veo-3.1-fast', aspects: ASPECTS_16_9, durations: [4, 6, 8], resolutions: res('720p', '1080p'), @@ -118,7 +118,7 @@ async function videoBody(req, ctx, fresh) { if (p.first_frame) references.push({ kind: 'image', documentId: await upload(ctx, p.first_frame, fresh) }); for (const ref of p.refs || []) references.push({ kind: 'image', documentId: await upload(ctx, ref.file, fresh) }); for (const ref of p.ref_videos || []) { - const file = r.refVideoMaxS ? ctx.trim(ref.file, { maxS: r.refVideoMaxS }) : ref.file; + const file = r.refVideoMaxS ? ctx.trim(ref.file, { minS: r.refVideoMinS, maxS: r.refVideoMaxS }) : ref.file; references.push({ kind: 'video', documentId: await upload(ctx, file, fresh) }); } return { diff --git a/skills/remix/scripts/render.mjs b/skills/remix/scripts/render.mjs index 098a3d6..d921efa 100644 --- a/skills/remix/scripts/render.mjs +++ b/skills/remix/scripts/render.mjs @@ -7,6 +7,7 @@ import { cpus } from 'node:os'; import { existsSync, mkdirSync, rmSync, writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { fail, fromPosixRel, launchBrowser, parseArgs, readJson, run } from './lib/common.mjs'; +import { identityProblems } from './lib/identity.mjs'; import { firstCues, openProject, sfxTracker } from './lib/capture.mjs'; import { buildProject, generateProject, projectDirFor } from './lib/project.mjs'; import { loadReplica } from './lib/replica.mjs'; @@ -24,6 +25,9 @@ const args = parseArgs(); if (!args._[0]) fail('usage: render.mjs [--workers N] [--no-build]', 2); const ctx = loadReplica(args._[0]); const { replica } = ctx; +// A real person's likeness must not reach the film: no flag skips this (references/replica.md §9). +const unchecked = identityProblems(ctx); +if (unchecked.length) fail(`identity gate: the film cannot be rendered yet\n${unchecked.join('\n')}`); let meta; try { meta = args['no-build'] ? { dir: projectDirFor(ctx.dir, 'final'), ...readJson(join(projectDirFor(ctx.dir, 'final'), 'project.json')) } : generateProject(ctx.dir, { mode: 'final' }); diff --git a/skills/remix/scripts/replica-init.mjs b/skills/remix/scripts/replica-init.mjs index 36ea809..c750800 100644 --- a/skills/remix/scripts/replica-init.mjs +++ b/skills/remix/scripts/replica-init.mjs @@ -9,6 +9,7 @@ import { pathToFileURL } from 'node:url'; import { cloneDir, fail, parseArgs, replicaDir, writeJson } from './lib/common.mjs'; import { check } from './lib/schema.mjs'; import { loadTemplate, mediaPaths } from './lib/replica.mjs'; +import { refShowsPerson, templateTier } from './lib/identity.mjs'; const canvasAspect = (canvas) => (canvas.width > canvas.height ? '16:9' : '9:16'); @@ -20,7 +21,7 @@ function mediaEntry(ctx, board, path, field, type, mode) { const entry = { kind: presenter ? 'avatar' : type, source: 'ai', - mode: ref?.clip ? 'reference' : 'text', + mode: ref?.clip ? (refShowsPerson(ref) && templateTier(ctx) === 'high' ? 'keyframe' : 'reference') : 'text', ref: ref ? key : null, replace: (ref?.subjects || []).map((s) => s.id), prompt: '', diff --git a/tests/remix/identity.test.mjs b/tests/remix/identity.test.mjs new file mode 100644 index 0000000..57f9c73 --- /dev/null +++ b/tests/remix/identity.test.mjs @@ -0,0 +1,176 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { execFileSync, spawnSync } from 'node:child_process'; +import { readdirSync } from 'node:fs'; +import { join } from 'node:path'; +import { PLUGIN, readJson, writeJson } from '../../skills/remix/scripts/lib/common.mjs'; +import { identityProblems, identitySlots, signOff } from '../../skills/remix/scripts/lib/identity.mjs'; +import { slotRequestId } from '../../skills/remix/scripts/lib/media/plan.mjs'; +import { loadReplica } from '../../skills/remix/scripts/lib/replica.mjs'; +import { validateReplica } from '../../skills/remix/scripts/lib/validate-replica.mjs'; +import { buildReplica } from '../../skills/remix/scripts/replica-init.mjs'; +import { clip, fakeTasks, makeClone, makeReplica } from './render-helpers.mjs'; + +const SCRIPTS = join(PLUGIN, 'scripts'); +const COVER = slotRequestId('demo', 0, 'Title01', 'cover'); + +// A clone whose cover slot shows a person, as a replacement target. +function personClone({ tier } = {}) { + const clone = makeClone(); + clip(join(clone, 'template', 'refs', 'Title01.cover.mp4'), 1); + const refsPath = join(clone, 'template', 'refs', 'refs.json'); + const refs = readJson(refsPath); + refs['Title01.cover'] = { ...refs['Title01.cover'], clip: 'Title01.cover.mp4', subjects: [{ id: 'presenter', kind: 'person', image: 'subjects/presenter.png' }] }; + writeJson(refsPath, refs); + if (tier) writeJson(join(clone, 'analysis', 'brief.json'), { tier }); + return clone; +} + +function generatedReplica({ tier, edit } = {}) { + const dir = makeReplica(personClone({ tier }), { + edit: (r) => { + r.replace = [{ subject: 'presenter', with: { generate: 'A fictional woman.' } }]; + Object.assign(r.scenes[0].media.cover, { mode: 'keyframe', replace: ['presenter'], prompt: 'A woman.' }); + edit?.(r); + }, + }); + fakeTasks(dir, { cover: 2 }); + return dir; +} + +const inspect = (dir, id, ...flags) => JSON.parse(execFileSync(process.execPath, [join(SCRIPTS, 'inspect.mjs'), dir, id, ...flags], { encoding: 'utf-8' })); + +test('a generated clip of a person is listed and blocks until it is signed off', () => { + const ctx = loadReplica(generatedReplica()); + assert.deepEqual(identitySlots(ctx).map((s) => s.id), [COVER]); + const problems = identityProblems(ctx); + assert.equal(problems.length, 1); + assert.match(problems[0], /no identity check/); +}); + +test('sign-off is refused until the clip was inspected with --identity', () => { + const dir = generatedReplica(); + assert.throws(() => signOff(loadReplica(dir), COVER, 'pass', { saw: 'the replacement woman' }), /inspect\.mjs .* --identity/); + inspect(dir, COVER); + assert.throws(() => signOff(loadReplica(dir), COVER, 'pass', { saw: 'the replacement woman' }), /--identity/, 'a plain inspect does not count'); +}); + +test('--identity reads the first second at 8 fps and the rest at 2 fps, at full width', () => { + const dir = generatedReplica(); + const out = inspect(dir, COVER, '--identity'); + const times = out.frames.map((f) => f.time); + assert.deepEqual(times.slice(0, 8), [0, 0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875]); + assert.deepEqual(times.slice(8), [1, 1.5]); + const width = execFileSync('ffprobe', ['-v', 'error', '-select_streams', 'v', '-show_entries', 'stream=width', '-of', 'csv=p=0', join(dir, out.frames[0].path)], { encoding: 'utf-8' }).trim(); + assert.equal(width, '1920'); +}); + +test('a pass needs what was seen, then clears the gate', () => { + const dir = generatedReplica(); + inspect(dir, COVER, '--identity'); + assert.throws(() => signOff(loadReplica(dir), COVER, 'pass', {}), /--saw/); + signOff(loadReplica(dir), COVER, 'pass', { saw: 'the replacement woman in every frame' }); + assert.deepEqual(identityProblems(loadReplica(dir)), []); +}); + +test('a regenerated clip voids its sign-off', () => { + const dir = generatedReplica(); + inspect(dir, COVER, '--identity'); + signOff(loadReplica(dir), COVER, 'pass', { saw: 'the replacement woman' }); + const ctx = loadReplica(dir); + clip(identitySlots(ctx)[0].clip, 3); + const problems = identityProblems(loadReplica(dir)); + assert.equal(problems.length, 1); + assert.match(problems[0], /changed since/); +}); + +test('a failed clip blocks, and voids the passes of every reference clip of the film', () => { + const second = slotRequestId('demo', 2, 'Title01', 'cover'); + const third = slotRequestId('demo', 3, 'Title01', 'cover'); + const dir = generatedReplica({ + edit: (r) => { + const copy = (mode) => ({ ...structuredClone(r.scenes[0]), media: { cover: { ...r.scenes[0].media.cover, mode } } }); + r.scenes.push(copy('reference'), copy('keyframe')); + }, + }); + const tasks = readJson(join(dir, 'tasks.json')); + for (const [id, name] of [[second, 'b'], [third, 'c']]) { + clip(join(dir, 'assets', 'gen', `${name}.mp4`), 2); + tasks.tasks[id] = { id, status: 'completed', result: { files: [{ path: `assets/gen/${name}.mp4`, mime: 'video/mp4' }] } }; + } + writeJson(join(dir, 'tasks.json'), tasks); + for (const id of [COVER, second, third]) { + inspect(dir, id, '--identity'); + signOff(loadReplica(dir), id, 'pass', { saw: 'the replacement woman' }); + } + signOff(loadReplica(dir), COVER, 'fail', { saw: 'the original man in the opening frames' }); + const problems = identityProblems(loadReplica(dir)).join('\n'); + assert.match(problems, new RegExp(`${COVER}: failed`)); + assert.match(problems, new RegExp(`${second}: no identity check`), 'the reference clip must be checked again'); + assert.doesNotMatch(problems, new RegExp(third), 'a keyframe clip keeps its own check'); +}); + +test('slots without a person, the user\'s own files and ungenerated slots are not gated', () => { + const clone = makeClone(); + const plain = makeReplica(clone, { edit: (r) => { r.scenes[0].media.cover.replace = []; } }); + fakeTasks(plain, { cover: 2 }); + assert.deepEqual(identityProblems(loadReplica(plain)), [], 'no person in the slot'); + + const user = generatedReplica({ edit: (r) => { + Object.assign(r.scenes[0].media.cover, { source: 'user', file: 'assets/gen/mine.mp4', mode: undefined }); + } }); + assert.deepEqual(identitySlots(loadReplica(user)), []); + + const failed = generatedReplica(); + const tasks = readJson(join(failed, 'tasks.json')); + tasks.tasks[COVER].status = 'failed'; + writeJson(join(failed, 'tasks.json'), tasks); + assert.deepEqual(identityProblems(loadReplica(failed)), [], 'no clip reaches the film: it falls back to the sketch'); +}); + +test('a digital human is always gated', () => { + const dir = generatedReplica({ edit: (r) => { r.scenes[0].media.cover.kind = 'avatar'; } }); + assert.equal(identityProblems(loadReplica(dir)).length, 1); +}); + +test('render.mjs refuses a film with an unchecked person, before any build', () => { + const dir = generatedReplica(); + const res = spawnSync(process.execPath, [join(SCRIPTS, 'render.mjs'), dir], { encoding: 'utf-8' }); + assert.notEqual(res.status, 0); + assert.match(res.stderr, /identity/); + assert.match(res.stderr, new RegExp(COVER)); + assert.deepEqual(readdirSync(dir).filter((f) => f === 'project' || f === 'out'), []); +}); + +test('the identity CLI signs off and reports status', () => { + const dir = generatedReplica(); + const run = (...a) => spawnSync(process.execPath, [join(SCRIPTS, 'identity.mjs'), ...a], { encoding: 'utf-8' }); + assert.equal(run('status', dir).status, 1); + inspect(dir, COVER, '--identity'); + assert.equal(run('pass', dir, COVER, '--saw', 'the replacement woman').status, 0); + assert.equal(run('status', dir).status, 0); +}); + +test('validate: at high a person in a reference slot is an error unless it gives a reason', () => { + const reference = (tier, extra = {}) => { + const dir = makeReplica(personClone({ tier }), { + edit: (r) => { + r.replace = [{ subject: 'presenter', with: { generate: 'A fictional woman.' } }]; + Object.assign(r.scenes[0].media.cover, { mode: 'reference', replace: ['presenter'], prompt: 'A woman.', ...extra }); + }, + }); + return validateReplica(dir).errors.filter((e) => /person|schema/.test(e)); + }; + assert.equal(reference('high').length, 1); + assert.match(reference('high')[0], /keyframe/); + assert.deepEqual(reference('low'), []); + assert.deepEqual(reference(undefined), []); + assert.deepEqual(reference('high', { reference_ok: 'a silhouette, no recognisable person' }), []); +}); + +test('replica-init: at high a slot with a person starts as keyframe, at low as reference', () => { + const modeAt = (tier) => buildReplica(personClone({ tier }), 'demo', { topic: 'Nova', language: 'en' }).scenes[0].media.cover.mode; + assert.equal(modeAt('high'), 'keyframe'); + assert.equal(modeAt('low'), 'reference'); +}); + diff --git a/tests/remix/media.test.mjs b/tests/remix/media.test.mjs index fb01ff9..6ea497e 100644 --- a/tests/remix/media.test.mjs +++ b/tests/remix/media.test.mjs @@ -79,6 +79,8 @@ test('providerOrder puts tapvid first and appends custom providers', () => { const reg = registryOf(fakeProvider('fal'), fakeProvider('tapvid'), custom); assert.deepEqual(providerOrder('video.generate', reg, {}), ['tapvid', 'fal', 'mine']); assert.deepEqual(providerOrder('video.generate', reg, { routing: { 'video.generate': ['fal'] } }), ['fal', 'tapvid', 'mine']); + const transcribers = new Map(['elevenlabs', 'whisper-local', 'openai'].map((name) => [name, { meta: { name, capabilities: { 'speech.transcribe': {} } } }])); + assert.deepEqual(providerOrder('speech.transcribe', transcribers, {}), ['whisper-local', 'elevenlabs', 'openai']); }); test('route skips unconfigured providers and explains why', () => { diff --git a/tests/remix/render-project.test.mjs b/tests/remix/render-project.test.mjs index de06a0b..036ce30 100644 --- a/tests/remix/render-project.test.mjs +++ b/tests/remix/render-project.test.mjs @@ -1,14 +1,21 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; -import { existsSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import { execFileSync } from 'node:child_process'; +import { existsSync, mkdtempSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { readJson, writeJson } from '../../skills/remix/scripts/lib/common.mjs'; import { loadReplica } from '../../skills/remix/scripts/lib/replica.mjs'; import { boardCaptions, buildProject, generateProject } from '../../skills/remix/scripts/lib/project.mjs'; -import { checkBoardCaptions, checkCutoutDrawn } from '../../skills/remix/scripts/lib/validate-template.mjs'; +import { checkBoardCaptions, checkCaptionBand, checkCutoutDrawn, checkSegmentSeconds } from '../../skills/remix/scripts/lib/validate-template.mjs'; +import { overlappingTexts } from '../../skills/remix/scripts/lib/capture.mjs'; +import { phrases } from '../../skills/remix/assets/template/src/components/captionPhrases.js'; +import { clipFit, fitClip } from '../../skills/remix/scripts/lib/fit-clip.mjs'; +import { createCtx } from '../../skills/remix/scripts/lib/media/ctx.mjs'; +import { probeDuration } from '../../skills/remix/scripts/lib/replica.mjs'; import { resolveTiming } from '../../skills/remix/scripts/lib/timing.mjs'; import { lintDeterminism, validateProject } from '../../skills/remix/scripts/lib/validate-project.mjs'; -import { clip, fakeTasks, makeClone, makeReplica, runtimeReady } from './render-helpers.mjs'; +import { clip, fakeTasks, makeClone, makeReplica, probe, runtimeReady } from './render-helpers.mjs'; const read = (dir, ...p) => readFileSync(join(dir, ...p), 'utf-8'); @@ -80,6 +87,99 @@ test('boardCaptions: false drops a board\'s captions, off mutes its lines, other assert.deepEqual(boardCaptions({ off: ['items.title'], centerY: 80 }, words, segments), { words: [words[2]], style: { centerY: 80 } }); }); +const incomingTransitionS = (ctx) => { + const [a, b] = ctx.replica.scenes; + return (ctx.template.transitions || []).find((x) => x.from === a.board && x.to === b.board)?.duration || 0; +}; + +test('clipFit speeds a clip into its scene, and cuts what the speed limit leaves over', () => { + assert.equal(clipFit(5, 5), null); + assert.equal(clipFit(5, 4.9), null); + assert.equal(clipFit(5, 8), null); + assert.deepEqual(clipFit(5, 2.5), { speed: 2, cutS: 0 }); + assert.deepEqual(clipFit(5, 2), { speed: 2.5, cutS: 0 }); + assert.deepEqual(clipFit(5, 1), { speed: 2.5, cutS: 2.5 }); +}); + +test('fitClip writes a clip of the scene\'s length and keeps the original', { skip: !runtimeReady }, () => { + const dir = mkdtempSync(join(tmpdir(), 'tapvid-fit-')); + const src = join(dir, 'clip.mp4'); + execFileSync('ffmpeg', ['-y', '-v', 'error', '-f', 'lavfi', '-i', 'testsrc=size=160x90:rate=30:duration=5', '-pix_fmt', 'yuv420p', src]); + const { file, fit } = fitClip(src, 2.5); + assert.deepEqual(fit, { speed: 2, cutS: 0 }); + assert.notEqual(file, src); + assert.ok(Math.abs(probeDuration(file) - 2.5) < 0.1); + assert.ok(Math.abs(probeDuration(src) - 5) < 0.1); + assert.deepEqual(fitClip(src, 6), { file: src, fit: null }); +}); + +test('final: a generated clip longer than its scene is sped up, then cut; one that fits plays as it is', () => { + const fit = (cover) => { + const dir = makeReplica(makeClone()); + fakeTasks(dir, { cover }); + const ctx = loadReplica(dir); + writeJson(ctx.path, { ...ctx.replica, resolved: resolveTiming(ctx).resolved }); + const meta = generateProject(dir, { mode: 'final' }); + const target = meta.scenes[0].duration + incomingTransitionS(ctx); + return { placed: probe(join(meta.dir, 'assets', 'media', '01-Title01-cover.mp4')), target, cut: meta.warnings.filter((w) => /too long/.test(w)) }; + }; + const sped = fit(5); + assert.ok(sped.target > 2 && sped.target < 2.5, `scene ${sped.target}s`); + assert.ok(Math.abs(sped.placed - sped.target) < 0.1, `placed ${sped.placed}s for ${sped.target}s`); + assert.deepEqual(sped.cut, []); + const cut = fit(10); + assert.ok(Math.abs(cut.placed - cut.target) < 0.1, `placed ${cut.placed}s for ${cut.target}s`); + assert.match(cut.cut[0], /01-Title01\.cover: the clip is too long .* at 2\.5×; its last [\d.]+s are cut/); + const fits = fit(1); + assert.ok(Math.abs(fits.placed - 1) < 0.1); + assert.deepEqual(fits.cut, []); +}); + +test('ctx.trim slows a reference clip up to the model\'s minimum and leaves a long enough one alone', { skip: !runtimeReady }, () => { + const dir = mkdtempSync(join(tmpdir(), 'tapvid-trim-')); + const short = join(dir, 'short.mp4'); + clip(short, 0.9, '160x90'); + const ctx = createCtx({ provider: 'x', secrets: { secret: () => '' }, cacheDir: dir }); + const stretched = ctx.trim(short, { minS: 2, maxS: 15 }); + assert.notEqual(stretched, short); + assert.ok(probe(stretched) >= 2 && probe(stretched) < 2.2, `${probe(stretched)}s`); + assert.equal(ctx.trim(short, { minS: 0.5, maxS: 15 }), short); +}); + +test('phrases break at a silence: a muted line leaves no phrase waiting across it', () => { + const word = (text, start, end) => ({ text, start, end }); + const words = [word('Revisions', 0, 0.5), word('drag.', 0.5, 0.9), word('Now', 5, 5.2), word('fly.', 5.2, 5.5)]; + const [{ words: muted }] = [boardCaptions({ off: ['middle'] }, [...words.slice(0, 2), word('hidden', 1, 4.9), ...words.slice(2)], [{ key: 'middle', start: 1, end: 4.95 }])]; + assert.deepEqual(muted, words); + assert.deepEqual(phrases(muted, 38).map((g) => [g.start, g.end]), [[0, 0.9], [5, 5.5]]); + assert.deepEqual(phrases(words.slice(0, 2), 38).map((g) => g.words.length), [2]); + assert.equal(phrases(words.slice(0, 2), 10).length, 2); +}); + +test('checkSegmentSeconds warns on the original\'s seconds, not on a fraction or a short entrance', () => { + const meta = { example_segments: [{ key: 'leadSay', start: 0, end: 2 }, { key: 'compareSay', start: 2, end: 4 }] }; + assert.match(checkSegmentSeconds('E', meta, 'title(q, [1,2,3,4], 5.4, "leadSay", 1.637, 0.85)')[0], /1\.637 s after line "leadSay"/); + assert.equal(checkSegmentSeconds('E', meta, 'const a = local("compareSay") - 2.987;').length, 1); + assert.deepEqual(checkSegmentSeconds('E', meta, 'const a = local("compareSay") - 0.087; const s = seg("leadSay"); at(s, 0.78 * span);'), []); +}); + +test('checkCaptionBand finds captions on a title; a board without captions or another centerY passes', () => { + const elements = [{ id: 'E.question', board: 'E', kind: 'title', box: [0.12, 0.06, 0.76, 0.07] }, { id: 'E.cap', board: 'E', kind: 'caption', box: [0.1, 0.07, 0.8, 0.06] }]; + const design = { captions: { centerY: 9, size: 4 } }; + assert.match(checkCaptionBand('E', undefined, design, elements)[0], /captions at 9% run over E\.question/); + assert.equal(checkCaptionBand('E', undefined, design, elements).length, 1); + assert.deepEqual(checkCaptionBand('E', false, design, elements), []); + assert.deepEqual(checkCaptionBand('E', { centerY: 40 }, design, elements), []); +}); + +test('overlappingTexts pairs texts that collide, not ones that only touch', () => { + const part = (text, box) => ({ part: 'Heading', text, box }); + const title = part('Title', [100, 100, 500, 200]); + assert.equal(overlappingTexts([title, part('Caption', [150, 120, 400, 180])]).length, 1); + assert.deepEqual(overlappingTexts([title, part('Below', [100, 198, 500, 300])]), []); + assert.deepEqual(overlappingTexts([title, part('Far', [600, 100, 900, 200])]), []); +}); + test('checkBoardCaptions: the rule is required once the template has captions; off names narration fields', () => { const meta = readJson(join(makeClone(), 'template', 'boards', 'Points02.json')); assert.deepEqual(checkBoardCaptions('Points02', meta, null), []);