Repository navigation
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Split from #218, where it was left as "not solved by a check". A design question, to decide before anything is built.
The problem
For four content types the source holds numbers that only make sense against a file:
| Type | What the model writes | Against |
|---|---|---|
| Interactive Video | when each interaction, bookmark and the summary shows, in seconds | a video |
| Image Hotspots | each hotspot's x, y in percent |
a picture |
| Find the Hotspot | each area's x, y, width, height in percent |
a picture |
| Game Map | each stage's place on the map | a picture |
The model often has not looked at the file. It then writes plausible numbers: a hotspot named "river mouth" at 70, 65, a question at 45 s. The package builds, passes the verifier, and is wrong. Only a person who plays it finds out.
What exists today
- The skill tells it to look. Interactive Video's type file: "Know the video's length, and keep every time within it." Image Hotspots': "Look at the picture to place them; the preview shows where they landed."
authoring.md: "Positions on a picture are guesses until seen: look at the picture if you can, and ask the user to check them in the preview." - The build knows some facts, after the source is written. It reads a picture's size, and an mp4's length, sound and codec (feat(author): check a video's and a recording's bytes, and read an mp4's index (#218) #228). A time past an mp4's end is refused with both numbers. So a model learns the length from a refusal, one round late.
- Nothing is known of a WebM's length, and nothing of a hosted video's. Vimeo's oEmbed answer, which the build already fetches since feat(author): ask YouTube and Vimeo whether they have a hosted video #258, carries
duration(62 for the test video); YouTube's carries none. - Nothing says what is in the file: what is on screen at 45 s, where the river mouth is.
What is missing, by kind
- Facts a tool can read: a video's length, a picture's size. Cheap, exact, and today only reachable through a failed build.
- What the file shows: only a model that can open the picture, or a person, knows. For a video no chat sandbox has a decoder, so in practice only the person, or a transcript with times.
Options
- A. A command that says what a file is, before the source is written.
h5p-author media clip.mp4 map.png(name to decide) prints length, sound and codec for a video, and width and height for a picture, with the same readers the build uses. For a Vimeo address it could print the length from oEmbed. An MCP tool beside it. Small, and it answers kind 1 fully for local files. - B. The build says the length whenever it builds an Interactive Video, as a line of its output, so a later change starts from it. Nearly free; does not help the first source.
- C. A grid over the picture. The same command writes a copy of the picture with a labelled percent grid (10 by 10). A model that can view images places hotspots by cell, which is a known way to make its coordinates better than guessing from the bare picture. Needs an image encoder in the tool (PNG is small to write; drawing labels is the work), and helps only chats that can open an image file from their sandbox.
- D. Ask the person for the moments. For a video, the skill's one question becomes "at which moments, and what happens there?" when the user gave none, or it takes them from a transcript or a caption file (
.vtthas times). Costs a question, which the skill is built to avoid; it is the only source of truth for a video. - E. Say what was guessed. Where positions or times were written unseen, the hand-over says so in a fixed line: "the hotspots' places are guesses: check them in the preview". Today this is a sentence in
authoring.mdthat a weak model drops. - F. Let the person fix them on the build page, by dragging. That is editing in place, a larger direction of its own (Edit text in place, in the playing content embed-my/creator#4), and not proposed here.
What I would do first
A and B together (the facts), then E (honesty about the rest), then measure before C or D:
- Run Haiku and a larger model on an Image Hotspots task with the picture attached, three ways: no help, "look at the picture", and a gridded copy. Count hotspots that land on what they name. C is worth building only if the grid clearly wins.
- For Interactive Video, check what chats do today when the user gives a video and no moments, and whether a transcript changes it.
To decide
- Whether A is a command of its own or part of
build --check's output. - Whether D's question is acceptable for Interactive Video only, as the one type where nothing else can know.
- Whether a guess may be handed over at all without E's line.
Not in scope
Item 6 of #218, the runs with media attached in a chat's sandbox, stays there: it tests the routes, not this question.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity