Skip to content

h5p-author: the model writes times and positions for media it has not seen #260

Open

Description

@alekswebnet

Split from #218, where it was left as "not solved by a check". A design question, to decide before anything is built.

The problem

For four content types the source holds numbers that only make sense against a file:

Type What the model writes Against
Interactive Video when each interaction, bookmark and the summary shows, in seconds a video
Image Hotspots each hotspot's x, y in percent a picture
Find the Hotspot each area's x, y, width, height in percent a picture
Game Map each stage's place on the map a picture

The model often has not looked at the file. It then writes plausible numbers: a hotspot named "river mouth" at 70, 65, a question at 45 s. The package builds, passes the verifier, and is wrong. Only a person who plays it finds out.

What exists today

  • The skill tells it to look. Interactive Video's type file: "Know the video's length, and keep every time within it." Image Hotspots': "Look at the picture to place them; the preview shows where they landed." authoring.md: "Positions on a picture are guesses until seen: look at the picture if you can, and ask the user to check them in the preview."
  • The build knows some facts, after the source is written. It reads a picture's size, and an mp4's length, sound and codec (feat(author): check a video's and a recording's bytes, and read an mp4's index (#218) #228). A time past an mp4's end is refused with both numbers. So a model learns the length from a refusal, one round late.
  • Nothing is known of a WebM's length, and nothing of a hosted video's. Vimeo's oEmbed answer, which the build already fetches since feat(author): ask YouTube and Vimeo whether they have a hosted video #258, carries duration (62 for the test video); YouTube's carries none.
  • Nothing says what is in the file: what is on screen at 45 s, where the river mouth is.

What is missing, by kind

  1. Facts a tool can read: a video's length, a picture's size. Cheap, exact, and today only reachable through a failed build.
  2. What the file shows: only a model that can open the picture, or a person, knows. For a video no chat sandbox has a decoder, so in practice only the person, or a transcript with times.

Options

  • A. A command that says what a file is, before the source is written. h5p-author media clip.mp4 map.png (name to decide) prints length, sound and codec for a video, and width and height for a picture, with the same readers the build uses. For a Vimeo address it could print the length from oEmbed. An MCP tool beside it. Small, and it answers kind 1 fully for local files.
  • B. The build says the length whenever it builds an Interactive Video, as a line of its output, so a later change starts from it. Nearly free; does not help the first source.
  • C. A grid over the picture. The same command writes a copy of the picture with a labelled percent grid (10 by 10). A model that can view images places hotspots by cell, which is a known way to make its coordinates better than guessing from the bare picture. Needs an image encoder in the tool (PNG is small to write; drawing labels is the work), and helps only chats that can open an image file from their sandbox.
  • D. Ask the person for the moments. For a video, the skill's one question becomes "at which moments, and what happens there?" when the user gave none, or it takes them from a transcript or a caption file (.vtt has times). Costs a question, which the skill is built to avoid; it is the only source of truth for a video.
  • E. Say what was guessed. Where positions or times were written unseen, the hand-over says so in a fixed line: "the hotspots' places are guesses: check them in the preview". Today this is a sentence in authoring.md that a weak model drops.
  • F. Let the person fix them on the build page, by dragging. That is editing in place, a larger direction of its own (Edit text in place, in the playing content embed-my/creator#4), and not proposed here.

What I would do first

A and B together (the facts), then E (honesty about the rest), then measure before C or D:

  • Run Haiku and a larger model on an Image Hotspots task with the picture attached, three ways: no help, "look at the picture", and a gridded copy. Count hotspots that land on what they name. C is worth building only if the grid clearly wins.
  • For Interactive Video, check what chats do today when the user gives a video and no moments, and whether a transcript changes it.

To decide

  • Whether A is a command of its own or part of build --check's output.
  • Whether D's question is acceptable for Interactive Video only, as the one type where nothing else can know.
  • Whether a guess may be handed over at all without E's line.

Not in scope

Item 6 of #218, the runs with media attached in a chat's sandbox, stays there: it tests the routes, not this question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions