diff --git a/SECURITY.md b/SECURITY.md index abff0fa2..2dd4ed9a 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -79,6 +79,8 @@ model may have written: name and then asks, so it guards against a mistaken or planted address, not a hostile network. A video on YouTube, Vimeo, Panopto or Echo360 is not downloaded: its address is kept, and the player loads it wherever it points. + A YouTube or Vimeo address is sent to that site's oEmbed endpoint, which says + whether it has the video. - The MCP server always builds so, in the folders the person gave it, else the app's roots, as they are given. Given none, it uses Desktop, Documents and Downloads, and never the home folder whole. It never writes over a file, and diff --git a/design/author.md b/design/author.md index a6b0f4d8..504d25b9 100644 --- a/design/author.md +++ b/design/author.md @@ -92,7 +92,33 @@ text to the content check's rules, and `apps/skills` plays every example untrust Echo360 address stays an address, as the editor embeds it, and only where H5P.Video's own pattern for that host takes it, since an address none matches goes to its HTML5 player, which cannot play a page: a Short or a live stream is written - as its video's watch address, and another form on those hosts is refused by name. + as its video's watch address, and another form on those hosts is refused by name. A YouTube or + Vimeo address is asked of the host's oEmbed, which answers anyone, a page too (#218): an address + holds to its form only, and a made-up id built and showed a dead player. What a status means is + each host's own list (`answers` in `media.mjs`), as seen on real videos on 2026-11-20. YouTube: + 400 for an id it never had and 404 for a video taken down, both a problem; 403 for a private + video, a note, since its owner can open it; 200 for one that asks a viewer's age; 401 was not + seen, is said to be embedding turned off, and is a note. Vimeo: 404 for no such video, a + private one, and an unlisted one given without the rest of its link, a problem that says so; a + video behind a password or let to some sites only answers 200 (Vimeo's help centre), and its + 403 is Vimeo refusing the asker's network, so it says nothing. + - **Only the host's own answer counts.** A sandbox's proxy or a school's filter answers for the + host, and would otherwise refuse a build with no way round, or put a note on every video. + Both hosts send a status that is no video in a way of their own (`sends`): YouTube that + status's words alone ("Bad Request", "Forbidden", "Not Found", and "Unauthorized" expected + of the 401 that was not seen) under its JSON type, Vimeo a page of them ("404 Not Found", + `text/html`). A status that means something counts only when + it comes so, its own words and the host's type; a page of the box's own, the words under + another type as a small proxy sends them, or another status's words say nothing, as do any + other status, no network, and no answer in ten seconds, and the build goes on. A box that + copies both the words and the type would still be taken for the host: none was seen. If a + host changes how it answers, the check goes quiet, not wrong. + - **A host that does not answer costs ten seconds once.** Videos are asked one at a time, and + after the first with no answer that host is not asked again in that build; the other still + is. One video named twice is asked once. YouTube is asked by the video's watch address, since its oEmbed answers 404 for + the `/embed/` and `/v/` forms of a video it has. + - Panopto and Echo360 answer no one and are not asked. A read asks nothing. The tests never + ask: a setup file answers for both hosts (`tests/helpers/oembed.ts`). `build.test.ts` holds each accepted form to the patterns in H5P.Video's scripts. A package with media goes through the normalizer's `normalizeArchive`, stored and laid out to stream. An addon whose pattern the text matches, H5P.MathDisplay for LaTeX, is carried as a site with it diff --git a/packages/author/README.md b/packages/author/README.md index c763b2b0..d5db4a30 100644 --- a/packages/author/README.md +++ b/packages/author/README.md @@ -86,7 +86,11 @@ What `params` takes is what H5P's editor would let an author save in the same fo and, where it gives one, its length, past which an Interactive Video's times are refused. The package is then laid out to stream, media stored and an mp4's index first. A YouTube, Vimeo, Panopto or Echo360 address stays where it is, if it is one H5P.Video plays: a Short's is taken - as its video's, and another address on those sites is refused. A download is given up once + as its video's, and another address on those sites is refused. A YouTube or Vimeo video is + asked of the site's oEmbed, which is sent the video's address: one the site has no video at is + refused, one YouTube says is private is a note, as is its 401, taken to mean that embedding is + turned off (not seen yet), and with no answer of the site's own in ten seconds, behind a proxy + that answers for it or with no network, the build goes on without one. A download is given up once nothing has arrived for a minute, however long it runs. - **Interface texts** (Check, Retry…): leave them out; they come from the libraries' translations, and the command lists what stayed English. A source may still set one by name. @@ -233,7 +237,8 @@ const { output, source: built, builtWith, manifest, content, libraries, install, and throws a `SourceError` whose `problems` are `{ path, message }`; with `check` it builds and writes no package and takes no output (`buildPackage(source, undefined, { check: true })`), though a media file named by web address is still downloaded, to a temporary folder removed after; a video on -YouTube, Vimeo, Panopto or Echo360 stays an address, as in a build. For a build run +YouTube, Vimeo, Panopto or Echo360 stays an address, as in a build, a YouTube or Vimeo one asked +of the site with the same `fetch`. For a build run on what a model wrote: `within` is a list of the folders the output and every local file the source names must be inside, on their real paths, and its problems then name no path but the one the source wrote. It does not cover a file named by web address, which is downloaded with the `fetch` diff --git a/packages/author/lib/media.mjs b/packages/author/lib/media.mjs index 9362b739..a39b0226 100644 --- a/packages/author/lib/media.mjs +++ b/packages/author/lib/media.mjs @@ -47,8 +47,13 @@ const under = (host, domains) => domains.some((domain) => host === domain || hos * an address by its own pattern for each host, and gives one no pattern matches to its HTML5 * player, which cannot play a page. So `plays` takes only addresses H5P.Video 1.6's pattern for * the host takes as well, and any other address on the host (`on`, told by its name, with no - * pattern to backtrack through its labels) is refused, naming the form. - * @type {{ host: string, mime: string, plays: RegExp, on: (url: URL) => boolean, form: string }[]} + * pattern to backtrack through its labels) is refused, naming the form. `oembed` is where the + * host says whether it has the video, for the two hosts that answer anyone (#218), and `answers` + * what each status it answers there means, as seen on 2026-11-20: a problem (`gone`) or a note + * (`closed`), each with the words for it. A status not listed says nothing. `sends` is how the + * host itself sends such a status, its body and the type it gives it, by which its own answer is + * told from one a box in between gives for it (`#ask`). + * @type {{ host: string, mime: string, plays: RegExp, on: (url: URL) => boolean, form: string, oembed?: (address: string) => string, sends?: { type: string, body: (status: number) => string }, answers?: Record }[]} */ const HOSTED = [ { @@ -56,14 +61,41 @@ const HOSTED = [ mime: 'video/Vimeo', plays: /^https?:\/\/(?:[\w-]+\.)*vimeo\.com\/(?:channels\/[A-Za-z_]+\/|groups\/[A-Za-z_]+\/videos\/)?\d/, on: (url) => under(url.hostname, ['vimeo.com']), - form: 'https://vimeo.com/ and the video\'s number' + form: 'https://vimeo.com/ and the video\'s number', + // Every form it plays, a channel's and a group's too, and an unlisted video's with its hash. + oembed: (address) => `https://vimeo.com/api/oembed.json?url=${encodeURIComponent(address)}`, + // Its 404 is a page of the words alone. + sends: { type: 'text/html', body: (status) => `${status} ${STATUS_WORDS[status]}` }, + // 404 for no such video, a private one, and an unlisted one asked of without the rest of its + // link (Vimeo's help centre). A video behind a password or let to some sites only answers + // 200, and a 403 is Vimeo refusing the asker's network, which says nothing of the video. + answers: { + 404: { gone: 'has no video there that others can play: give https://vimeo.com/ and the number of a public video, or all of an unlisted video\'s link, which has more after the number' } + } }, { host: 'YouTube', mime: 'video/YouTube', plays: /^https?:\/\/(?:[\w-]+\.)*(?:youtube\.com\/(?:watch\?(?:[^#\s]*&)?v=|embed\/|v\/)|youtu\.be\/)[\w-]{11}/i, on: (url) => under(url.hostname, ['youtube.com', 'youtube-nocookie.com', 'youtu.be']), - form: 'https://www.youtube.com/watch?v= and the video\'s id' + form: 'https://www.youtube.com/watch?v= and the video\'s id', + // Asked by its watch address: YouTube's oEmbed answers 404 for /embed/ and /v/ addresses of a + // video it has (2026-11-20). + oembed: (address) => { + const id = YOUTUBE_ID.exec(address)?.[1] + return id ? `https://www.youtube.com/oembed?format=json&url=${encodeURIComponent(`https://www.youtube.com/watch?v=${id}`)}` : '' + }, + // The status's words alone, under the type of the answer it would have given. + sends: { type: 'application/json', body: (status) => STATUS_WORDS[status] }, + // Seen: 400 for an id it never had, 404 for a video taken down, 403 for a private one, and + // 200 for one that asks a viewer's age. 401 was not seen: it is said to be a video whose + // owner has turned embedding off. + answers: { + 400: { gone: 'has no video there: give https://www.youtube.com/watch?v= and the id of a public or unlisted video' }, + 404: { gone: 'has no video there any more: give https://www.youtube.com/watch?v= and the id of a public or unlisted video' }, + 403: { closed: 'says the video is private: nobody but its owner can play it until it is made public or unlisted' }, + 401: { closed: 'would not say the video may be embedded, which is taken to mean its owner has turned embedding off: play it where it will be used' } + } }, { host: 'Panopto', @@ -81,6 +113,21 @@ const HOSTED = [ } ] +/** A YouTube video's id, in an address H5P.Video plays. */ +const YOUTUBE_ID = /(?:[?&]v=|\/embed\/|\/v\/|youtu\.be\/)([\w-]{11})/i + +/** How long a host may take to say whether it has a video, before the build goes on without it. */ +const ASK_MS = 10_000 + +/** + * A status's words, which YouTube and Vimeo send alone with an answer that is no video ("Bad + * Request" under YouTube's JSON type, "404 Not Found" as Vimeo's page): each host's `sends`. A box + * between the build and the host, a sandbox's proxy or a school's filter, answers with a page of + * its own or under a type of its own, and so does Vimeo where it refuses the asker's network: none + * of those is the host speaking of the video. + */ +const STATUS_WORDS = /** @type {Record} */ ({ 400: 'Bad Request', 401: 'Unauthorized', 403: 'Forbidden', 404: 'Not Found' }) + /** A YouTube Short or live stream, by its video's id, which H5P.Video plays at the watch address. */ const YOUTUBE_SHORT = /^https?:\/\/(?:[\w-]+\.)*youtube\.com\/(?:shorts|live)\/([\w-]{11})(?![\w-])/i @@ -263,6 +310,10 @@ export class MediaStore { this.names = new Set() /** @type {Map} a video's or a recording's length in seconds, by its path in the package, where its index gave one */ this.lengths = new Map() + /** @type {Map>} what a host answered of a video, by the address asked, so one video named twice is asked of once */ + this.asked = new Map() + /** @type {Set} the hosts that gave no answer at all, none of which is asked again in this build */ + this.unanswered = new Set() } /** @@ -293,7 +344,10 @@ export class MediaStore { const short = YOUTUBE_SHORT.exec(source) const address = short ? `https://www.youtube.com/watch?v=${short[1]}` : source const hosted = HOSTED.find(({ plays }) => plays.test(address)) - if (hosted) return { path: address, mime: hosted.mime, copyright } + if (hosted) { + await this.#there(hosted, address, source, `${at}.path`, report, note) + return { path: address, mime: hosted.mime, copyright } + } const url = URL.canParse(address) ? new URL(address) : null const other = url ? HOSTED.find(({ on }) => on(url)) : undefined if (other) { @@ -432,6 +486,62 @@ export class MediaStore { return { from: found.file, extension, stem: name.slice(0, name.length - extension.length - 1) } } + /** + * Whether a host has a video, asked of its oEmbed endpoint where it has one (#218): an address + * holds to its form only, so a made-up id built and showed a dead player. What an answer means + * is the host's own list (`answers`): a video that is not there is a problem, and one nobody + * else can play yet a note, since its owner can open it. Any other answer, one that is not the + * host's own (`#ask`), no network, or no answer in ten seconds says nothing either way, and the + * build goes on without it. + * @param {(typeof HOSTED)[number]} hosted + * @param {string} address as H5P.Video will play it + * @param {string} source as written + * @param {string} at + * @param {(path: string, message: string) => void} report + * @param {(path: string, message: string) => void} note + */ + async #there(hosted, address, source, at, report, note) { + const ask = hosted.oembed?.(address) + if (!ask || !hosted.answers) return + let answer = this.asked.get(ask) + if (!answer) { + // One at a time, and a host that does not answer costs its ten seconds once, not a video. + if (this.unanswered.has(hosted.host)) return + answer = this.#ask(ask, hosted) + this.asked.set(ask, answer) + } + const status = await answer + if (status === 0) this.unanswered.add(hosted.host) + const meaning = hosted.answers[status] + if (meaning?.gone) report(at, `${source}: ${hosted.host} ${meaning.gone} (it answered ${status})`) + else if (meaning?.closed) note(at, `${source}: ${hosted.host} ${meaning.closed} (it answered ${status})`) + } + + /** + * The status a host's oEmbed endpoint answers; 0 where it does not answer in time, and -1 where + * a status that would say something of the video is not sent as the host sends it (`sends`): + * that status's own words and nothing else, under the host's type. + * @param {string} url + * @param {(typeof HOSTED)[number]} hosted + * @returns {Promise} + */ + async #ask(url, hosted) { + try { + const response = await this.fetch(url, { signal: AbortSignal.timeout(ASK_MS) }) + const { status } = response + if (!hosted.answers || !hosted.sends || !Object.hasOwn(hosted.answers, status)) { + await response.body?.cancel().catch(() => {}) + return status + } + const type = (response.headers.get('content-type') ?? '').split(';')[0].trim().toLowerCase() + const own = type === hosted.sends.type && (await response.text()).trim() === hosted.sends.body(status) + if (!own) await response.body?.cancel().catch(() => {}) + return own ? status : -1 + } catch { + return 0 + } + } + /** * A web address downloaded and kept, its type taken from its name or, failing that, from what * the server says it is. diff --git a/packages/author/lib/node.mjs b/packages/author/lib/node.mjs index 2d3f4d7a..0587ebc6 100644 --- a/packages/author/lib/node.mjs +++ b/packages/author/lib/node.mjs @@ -41,7 +41,8 @@ import { SourceError } from './source-error.mjs' * list even of one; `check`, to build and write no package: no output is needed or looked at, and * the result's `output` is empty, but a media file named by web address is still downloaded, to a * temporary folder removed after, as the build checks it; a video on YouTube, Vimeo, Panopto or - * Echo360 stays an address, as in a build + * Echo360 stays an address, as in a build, a YouTube or Vimeo one asked of the site's oEmbed with + * `fetch` as a build asks it * @typedef {{ output: string, source: import('./build.mjs').Source, files: string[], notes: Problem[] }} ReadResult * `output`, the source's path; `files`, the media written beside it, by the paths the source * names them by, from its folder; `notes`, what was left out or could not be read back as it is diff --git a/packages/author/tests/build.test.ts b/packages/author/tests/build.test.ts index 195d3dcd..16153190 100644 --- a/packages/author/tests/build.test.ts +++ b/packages/author/tests/build.test.ts @@ -840,6 +840,75 @@ describe('media', () => { ]) }) + it('asks YouTube and Vimeo whether they have a hosted video, once each, and says what they answered (#218)', async () => { + const video = (path: string) => ({ content: { library: 'H5P.Video', params: { sources: { path } } } }) + const column = (paths: string[]) => ({ ...META, type: 'H5P.Column', params: { content: paths.map(video) } }) + const at = (i: number) => `params.content[${i}].content.params.sources.path` + const asked: string[] = [] + const json = 'application/json; charset=UTF-8' + const page = 'text/html; charset=UTF-8' + // Each host's own answer for a status, as it sent it on 2026-11-20, by the video asked of: + // YouTube the status's words under its JSON type, Vimeo a page of them. + const answers: Record = { + aaaaaaaaaaa: [400, 'Bad Request', json], bbbbbbbbbbb: [404, 'Not Found', json], ccccccccccc: [403, 'Forbidden', json], ddddddddddd: [401, 'Unauthorized', json], + eeeeeeeeeee: [500, 'Internal Server Error', json], '999999999': [404, '404 Not Found', page], '888888888': [403, '403 Forbidden', page], + // A box between the build and the host, answering for it: with a page of its own, with the + // status's words alone under its own type, as a small proxy does, and with another status's. + fffffffffff: [403, 'BlockedThis site is not allowed on this network.', page], ggggggggggg: [403, 'Forbidden\n', 'text/plain; charset=utf-8'], + hhhhhhhhhhh: [404, 'Forbidden', json], '777777777': [404, '404 page not found', 'text/plain; charset=utf-8'], '666666666': [404, 'Not Found', page] + } + const fetch = async (input: string | URL | Request) => { + const url = String(input) + asked.push(url) + const [status, body, type] = Object.entries(answers).find(([id]) => url.includes(id))?.[1] ?? [200, '{}', json] + return new Response(body, { status, headers: { 'content-type': type } }) + } + const youtube = (id: string) => `https://www.youtube.com/oembed?format=json&url=${encodeURIComponent(`https://www.youtube.com/watch?v=${id}`)}` + const vimeo = (address: string) => `https://vimeo.com/api/oembed.json?url=${encodeURIComponent(address)}` + + const { notes } = await build(column([ + 'https://www.youtube.com/embed/dQw4w9WgXcQ', 'https://youtu.be/dQw4w9WgXcQ', 'https://vimeo.com/channels/staffpicks/76979871', + 'https://video.example.edu/Panopto/Pages/Viewer.aspx?id=0b6d3a5e', 'https://www.youtube.com/watch?v=ccccccccccc', 'https://www.youtube.com/watch?v=ddddddddddd', + 'https://www.youtube.com/watch?v=eeeeeeeeeee', 'https://www.youtube.com/watch?v=fffffffffff', 'https://www.youtube.com/watch?v=ggggggggggg', + 'https://www.youtube.com/watch?v=hhhhhhhhhhh', 'https://vimeo.com/888888888', 'https://vimeo.com/777777777', 'https://vimeo.com/666666666' + ]), { fetch }) + // A YouTube video by its watch address, which its oEmbed answers for any form; once for the two + // forms of one video; Vimeo's as written; Panopto's not, since it answers no one. + expect(asked).toEqual([ + youtube('dQw4w9WgXcQ'), vimeo('https://vimeo.com/channels/staffpicks/76979871'), youtube('ccccccccccc'), youtube('ddddddddddd'), youtube('eeeeeeeeeee'), + youtube('fffffffffff'), youtube('ggggggggggg'), youtube('hhhhhhhhhhh'), vimeo('https://vimeo.com/888888888'), vimeo('https://vimeo.com/777777777'), vimeo('https://vimeo.com/666666666') + ]) + // A video nobody else can play yet is a note. Nothing is said of a status that means nothing + // for the host (a 500; Vimeo's 403, which refuses the asker's network), nor of one that is not + // sent as the host sends it: other words, another status's, or under another type. + expect(notes).toEqual([ + { path: at(4), message: 'https://www.youtube.com/watch?v=ccccccccccc: YouTube says the video is private: nobody but its owner can play it until it is made public or unlisted (it answered 403)' }, + { path: at(5), message: 'https://www.youtube.com/watch?v=ddddddddddd: YouTube would not say the video may be embedded, which is taken to mean its owner has turned embedding off: play it where it will be used (it answered 401)' } + ]) + expect(await problems(column(['https://youtu.be/aaaaaaaaaaa', 'https://www.youtube.com/watch?v=bbbbbbbbbbb', 'https://vimeo.com/999999999']), { fetch })).toEqual([ + `${at(0)}: https://youtu.be/aaaaaaaaaaa: YouTube has no video there: give https://www.youtube.com/watch?v= and the id of a public or unlisted video (it answered 400)`, + `${at(1)}: https://www.youtube.com/watch?v=bbbbbbbbbbb: YouTube has no video there any more: give https://www.youtube.com/watch?v= and the id of a public or unlisted video (it answered 404)`, + `${at(2)}: https://vimeo.com/999999999: Vimeo has no video there that others can play: give https://vimeo.com/ and the number of a public video, or all of an unlisted video's link, which has more after the number (it answered 404)` + ]) + + // With no network the build goes on, and asks each host once, not once a video. + const tried: string[] = [] + const offline = async (input: string | URL | Request) => { + tried.push(new URL(String(input)).hostname) + throw new TypeError('fetch failed') + } + expect((await build(column(['https://youtu.be/aaaaaaaaaaa', 'https://vimeo.com/999999999', 'https://youtu.be/bbbbbbbbbbb', 'https://vimeo.com/888888888']), { fetch: offline })).notes).toEqual([]) + expect(tried).toEqual(['www.youtube.com', 'vimeo.com']) + // And one host that does not answer leaves the other asked: its video that is not there is still refused. + const vimeoDown = async (input: string | URL | Request) => { + if (String(input).startsWith('https://vimeo.com/')) throw new TypeError('fetch failed') + return fetch(input) + } + expect(await problems(column(['https://vimeo.com/999999999', 'https://youtu.be/aaaaaaaaaaa']), { fetch: vimeoDown })).toEqual([ + `${at(1)}: https://youtu.be/aaaaaaaaaaa: YouTube has no video there: give https://www.youtube.com/watch?v= and the id of a public or unlisted video (it answered 400)` + ]) + }) + it('reads a video\'s and a recording\'s first bytes, and refuses one that is not what the field takes (#218)', async () => { await writeFile(join(dir, 'page.mp4'), 'clip') await writeFile(join(dir, 'song.mp3'), 'not a recording at all') diff --git a/packages/author/tests/helpers/oembed.ts b/packages/author/tests/helpers/oembed.ts new file mode 100644 index 00000000..913641a3 --- /dev/null +++ b/packages/author/tests/helpers/oembed.ts @@ -0,0 +1,8 @@ +// The tests never ask YouTube or Vimeo whether they have a video (#218): a build of a hosted +// video's address hears that they do, unless a test passes a `fetch` of its own to say otherwise. +const ask = /^https:\/\/(?:www\.youtube\.com\/oembed|vimeo\.com\/api\/oembed\.json)\?/ +const real = globalThis.fetch +globalThis.fetch = (input, init) => { + const url = input instanceof Request ? input.url : String(input) + return ask.test(url) ? Promise.resolve(Response.json({ type: 'video' })) : real(input, init) +} diff --git a/packages/author/vitest.config.ts b/packages/author/vitest.config.ts index 64c4d80a..e3be2b59 100644 --- a/packages/author/vitest.config.ts +++ b/packages/author/vitest.config.ts @@ -35,6 +35,7 @@ export default defineConfig({ environment: 'node', include: ['tests/**/*.test.ts'], exclude: [...configDefaults.exclude, 'tests/browser/**'], + setupFiles: ['tests/helpers/oembed.ts'], // The package's limit, and no test's own (design/testing.md, #255). These tests build // packages from the library pack and write every content type's bundle, which a CI // runner does two to four times slower than a laptop: on vitest's 5 s a test, three diff --git a/packages/mcp/README.md b/packages/mcp/README.md index 5c481fdc..95557185 100644 --- a/packages/mcp/README.md +++ b/packages/mcp/README.md @@ -58,7 +58,8 @@ The build downloads each picture, video, recording or other file your content na address, and refuses to download from an address on your local network: a guard against a mistaken or planted address, not against a hostile network. A video on YouTube, Vimeo, Panopto or Echo360 is not downloaded: it stays an address, which the player loads wherever it points, as H5P does -anywhere. Math is drawn by the package's own copy of MathJax, which loads nothing from elsewhere. +anywhere. A YouTube or Vimeo address is sent to that site's oEmbed, to ask whether it has the +video. Math is drawn by the package's own copy of MathJax, which loads nothing from elsewhere. The check and the preview play the package from your computer. Your app sends your conversation to its model, as it does without this server.