From 31f2da4d97e9822287e02ec32feab8d967c52a6e Mon Sep 17 00:00:00 2001 From: Carlos Montero Date: Tue, 20 Oct 2026 08:01:23 +0200 Subject: [PATCH] =?UTF-8?q?fix(journey):=20on=20Windows=20the=20org=20scop?= =?UTF-8?q?e=20reads=20a=20file=20id=20through=20a=20handle=20=E2=80=94=20?= =?UTF-8?q?on=20Node=2022.16.x=20(libuv=201.49.2)=20a=20local=20path=20bes?= =?UTF-8?q?ide=20a=20workspace=20named=20through=20a=20share=20was=20never?= =?UTF-8?q?=20placed,=20and=20acme=20read=20beta's=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit crawlScopeRefusal placed a local path in a workspace named through a share (\myhost\ws, a drive mapped to \localhost\C$\ws) by a file id read with statSync. libuv answers a stat of a local path by its name, and 1.49.0 to 1.50.x report there the upper half of the file's 128-bit id as the device, 0 on NTFS, while the workspace through its share is opened and reports the serial: no local directory was the workspace's own, so beta's C:\ws\orgs\beta\... read as outside the workspace, nobody's part, and an acme journey loaded it (fails open). A file id is now read through a handle (openSync, fstatSync, closeSync), as server/store/paths.mjs does since #165; the three are injectable in place of statSync. tools/test-journey-scope.mjs injects the ntfs fixture's handles, and a new test loops libuv 1.51.0 and 1.49.2; at the parent it fails on 1.49.2 alone. VENDOR-MANIFEST.json is unchanged (exports are). Co-Authored-By: Claude Opus 5.5 --- docs/CHANGELOG.md | 3 +++ tools/lib/journey.mjs | 28 ++++++++++++++-------- tools/test-journey-scope.mjs | 45 +++++++++++++++++++++++++++++++----- 3 files changed, 61 insertions(+), 15 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 96c133c6..aa266fb2 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -2,6 +2,9 @@ ## Unreleased +### Follow-up `journey-scope-through-handle`: on Windows a journey's org scope reads a file id through a handle +- **On Node 22.16.x a studio-run journey's org scope places a local path in a workspace named through a share again** (a fix; `tools/lib/journey.mjs`; Linux and macOS compare exactly as before). While the workspace is named through a share this host serves (`OBSERVOGRAM_WORKSPACE=\\myhost\ws`, or a drive mapped to `\\localhost\C$\ws`), `crawlScopeRefusal` places a path on a local drive outside it by its nearest directory with the workspace's file id, the volume serial's low 32 bits and the file's id, which it read with `statSync`. libuv answers a stat of a local path by its name (`GetFileInformationByName`, Windows 11 24H2 and Server 2025), and 1.49.0 to 1.50.x (Node 22.16.x ships 1.49.2) report there the upper half of the file's 128-bit id as the device, 0 on NTFS, while the workspace through its share is opened and reports the serial: no local directory was the workspace's own, so acme's journey read beta's `C:\ws\orgs\beta\packs\b1.pack.yaml` as a file outside the workspace, nobody's part, and loaded it (`server/store/paths.mjs` had the same read, fixed in `paths-volume-through-handle`). On Windows a file id is now read through a handle opened on the path (`openSync`, `fstatSync`, `closeSync`; libuv opens a directory too), which reports the serial on every libuv, every handle closed before the comparison goes on; a path that does not open is passed over as one that did not stat was. `crawlScopeRefusal` and `crawlScopeTest` take `openSync`, `fstatSync` and `closeSync` in their host in place of `statSync`. One more test in `tools/test-journey-scope.mjs` (Suite total 1485 → 1486): on libuv 1.51.0 and 1.49.2, with the workspace named `\\myhost\ws` or `Z:\` (mapped to `\\localhost\C$\ws`), beta's file in every local spelling refused to acme, acme's own and `C:\repo\…` read, and every handle closed; at the parent commit it fails on 1.49.2 alone (beta's file read). Every scope the suite builds injects the fixture's handles. + ### Follow-up `exporter-metrics`: the crawler learns metrics from prebuilt exporters and from an OTel collector's scrape jobs - **A metrics manifest is an artefact** (`tools/lib/exporter-metrics.mjs`, new and listed: `readMetricsManifest`, `manifestFromExposition`, `parseExposition`, `safeSourceUrl`, `resolveExporterCatalogue`, `asResolvedCatalogue`, `matchExporterConfig`, `mayBeExporterConfig`, `inferExporterFamilies`, `isMetricsManifestPath`, `isExpositionSnapshotPath`, `isTestOrFixturePath`, `EXPOSITION_TYPES`). `metrics-manifest.json` and `*.metrics-manifest.json` are the kind `metrics-manifest`. Each entry is a metric with `origin_kind: exporter-manifest`, `confidence: declared`, and the exporter as its service. A file that is not a manifest (none of its text quoted: the JSON parser's message would quote its start), or entries without a name and an exposition type, are named in a warning. A manifest or exporter config saved with a leading BOM is read. - **A saved exposition is read** (**a declared default change**: `SCAN_EXT` adds `.prom` and `.metrics`, so the CLI walk, the studio's folder picker and a journey's crawl read them; `server/github-crawl.mjs` fetches a manifest with the files it fetched before, and a saved exposition, any `config.yaml` and a chart's `files/*.yaml` only after them (`isExporterCandidateFile`, `selectCrawlerFiles`): a monorepo's many `config.yaml` files never use up the 200-file budget its rule, Alertmanager or collector files had). The kind is `exposition-snapshot`: `# HELP` / `# TYPE` and the label names of the samples, label values never kept, `origin_kind: exporter-snapshot`. A family is named as Prometheus stores it: an OpenMetrics counter `jobs` is `jobs_total`, an info `build` is `build_info`. It is never read from a folder named by a test, spec, fixture, mock or test-data word, alone or joined to others by `-` / `_` (`test/`, `__fixtures__/`, `testFixtures/`, `__mocks__/`, `e2e/`, `integration-tests/`, `integrationTest/`, `unit_tests/`, `unittests/`, `testing/`, `test-data/`, `__snapshots__/`, …), a camelCase test source set (`componentTest/`, `smokeTest/`, `testUtils/`) or Maven's `src/it/`, nor from a `*.test.*`, `*.spec.*`, `*_test.*` or `*_spec.*` file (`isTestOrFixturePath`). diff --git a/tools/lib/journey.mjs b/tools/lib/journey.mjs index b2841a43..a7dd6e43 100644 --- a/tools/lib/journey.mjs +++ b/tools/lib/journey.mjs @@ -77,7 +77,7 @@ // total loss of the observation point is a point in the drift history, // not a hole in it. -import { readFileSync, writeFileSync, mkdirSync, readdirSync, statSync, unlinkSync, openSync, readSync, closeSync, realpathSync } from 'node:fs'; +import { readFileSync, writeFileSync, mkdirSync, readdirSync, statSync, unlinkSync, openSync, readSync, closeSync, fstatSync, realpathSync } from 'node:fs'; import { resolve, join, dirname, extname, basename, sep } from 'node:path'; import { fileURLToPath } from 'node:url'; import { isDeepStrictEqual } from 'node:util'; @@ -539,9 +539,9 @@ const MAX_TOTAL_BYTES = 16 * 1024 * 1024; // with \ separators and case folded. That form only compares: a refusal // names the path as written. Elsewhere the comparison is the realpath's // spelling, as it always was. host = { windows, path, realpathSync, -// statSync } (the platform, node:path's, node:fs's realpathSync with its -// .native and node:fs's statSync) is injectable -// (tools/test-journey-scope.mjs); the default is this host. +// openSync, fstatSync, closeSync } (the platform, node:path's, node:fs's +// realpathSync with its .native, openSync, fstatSync and closeSync) is +// injectable (tools/test-journey-scope.mjs); the default is this host. // // A network share is one more name the native realpath keeps: a path reached // through one (\\host\share\…, \\?\UNC\…, a mapped drive) resolves to @@ -555,15 +555,22 @@ const MAX_TOTAL_BYTES = 16 * 1024 * 1024; // there (C:\ws\orgs\beta) is one more name for it, which no realpath maps // back. So while the workspace is named through a share, a path on a local // drive outside it is placed by its nearest directory that is the -// workspace's own — the same file id on the same volume (statSync, with the -// volume serial's low 32 bits: all a share reports) — and judged there. +// workspace's own — the same file id on the same volume (the volume serial's +// low 32 bits: all a share reports) — and judged there. A file id is read +// through a handle opened on the path (openSync, fstatSync, closeSync: libuv +// opens a directory too), which reports the serial on every libuv, never a +// stat by name, as server/store/paths.mjs reads it: libuv answers one of a +// local path by its name (GetFileInformationByName, Windows 11 24H2 and +// Server 2025), and 1.49.0 to 1.50.x (Node 22.16.x's 1.49.2) report there no +// volume (0) — no local directory was the workspace's own, and beta's +// C:\ws\orgs\beta\… read as outside it, nobody's part. const NODE_PATH = Object.freeze({ resolve, join, dirname, basename, sep }); const ON_WINDOWS = process.platform === 'win32'; const LOCAL_DRIVE = /^[a-z]:\\/; // a compared form on a local drive (folded) const IN_ANOTHER_ORGS_PART = 'belongs to another org\'s part of the workspace — refused'; const NOT_ON_A_LOCAL_DRIVE = 'is not a path on a local drive, and a network share or a device path may lead into another org\'s part of the workspace — refused; ' + 'name it by its path on a local drive (copy it there if it has none)'; -export function crawlScopeRefusal(scope, { windows = ON_WINDOWS, path = NODE_PATH, realpathSync: fsRealpath = realpathSync, statSync: fsStat = statSync } = {}) { +export function crawlScopeRefusal(scope, { windows = ON_WINDOWS, path = NODE_PATH, realpathSync: fsRealpath = realpathSync, openSync: fsOpen = openSync, fstatSync: fsFstat = fstatSync, closeSync: fsClose = closeSync } = {}) { if (!scope) return null; const realpath = windows ? fsRealpath.native : fsRealpath; const realOrResolved = (p) => { @@ -581,8 +588,11 @@ export function crawlScopeRefusal(scope, { windows = ON_WINDOWS, path = NODE_PAT const orgsDir = path.join(base, 'orgs'); // A workspace named through a share: a local path's place in it, by its // nearest directory that is the workspace's own, or null (above). - const fileId = (p) => { - try { const st = fsStat(p, { bigint: true }); return `${st.dev & 0xffffffffn}:${st.ino}`; } catch { return null; } + const fileId = (p) => { // through a handle opened on p, closed before it answers, or null when p cannot be opened + try { + const fd = fsOpen(p, 'r'); + try { const st = fsFstat(fd, { bigint: true }); return `${st.dev & 0xffffffffn}:${st.ino}`; } finally { fsClose(fd); } + } catch { return null; } }; const baseId = windows && !LOCAL_DRIVE.test(base) ? fileId(path.resolve(scope.base)) : null; const isBase = new Map(); // a path or its ancestor (native realpath) → whether it is the workspace's own directory diff --git a/tools/test-journey-scope.mjs b/tools/test-journey-scope.mjs index ca0decb4..8d97a93e 100644 --- a/tools/test-journey-scope.mjs +++ b/tools/test-journey-scope.mjs @@ -11,8 +11,9 @@ // the volume behind it, so on Windows a path outside the workspace is // nobody's part only on a local drive — and, while the workspace is itself // named through a share, only if none of its directories is the workspace's -// own by file id (the volume's statSync): a share this host serves is stored -// on a local drive, and its path there is one more name for the workspace. +// own by file id (read through a handle on the volume, as every libuv reports +// it): a share this host serves is stored on a local drive, and its path there +// is one more name for the workspace. // On Linux and macOS the comparison is // unchanged: the A-24 cases in tools/test-journey.mjs and // server/test-tenancy.mjs hold it. The last test runs the default host on a @@ -41,9 +42,11 @@ const VOLUME = ntfsVolume([ shares: { '\\\\localhost\\C$': 'C:\\', '\\\\127.0.0.1\\C$': 'C:\\', '\\\\myhost.corp.example\\C$': 'C:\\', '\\\\myhost\\ws': 'C:\\ws' }, mapped: { 'Z:': '\\\\localhost\\C$\\ws' }, }); -const WINDOWS = { windows: true, path: win32, realpathSync: VOLUME.realpathSync, statSync: VOLUME.statSync }; +// A volume's realpath and the handles a file id is read through. +const onVolume = ({ realpathSync, openSync, fstatSync, closeSync }) => ({ path: win32, realpathSync, openSync, fstatSync, closeSync }); +const WINDOWS = { windows: true, ...onVolume(VOLUME) }; // The comparison this replaces on Windows: the spelling JS realpathSync keeps. -const BY_SPELLING = { windows: false, path: win32, realpathSync: VOLUME.realpathSync, statSync: VOLUME.statSync }; +const BY_SPELLING = { windows: false, ...onVolume(VOLUME) }; const ACME = { base: 'C:\\ws', ownRoot: 'C:\\ws\\orgs\\acme' }; const DEFAULT_ORG = { base: 'C:\\ws', ownRoot: 'C:\\ws' }; @@ -157,7 +160,7 @@ test('a workspace named through a share this host serves (\\\\myhost\\ws, Z: map assert.equal(local.ino, shared.ino); assert.notEqual(local.dev, shared.dev); assert.equal(local.dev & 0xffffffffn, shared.dev); - const noFileId = { ...WINDOWS, statSync: () => { throw Object.assign(new Error('ENOENT: no such file or directory'), { code: 'ENOENT' }); } }; + const noFileId = { ...WINDOWS, openSync: () => { throw Object.assign(new Error('ENOENT: no such file or directory'), { code: 'ENOENT' }); } }; for (const [base, ownRoot] of [['\\\\myhost\\ws', '\\\\myhost\\ws\\orgs\\acme'], ['Z:\\', 'Z:\\orgs\\acme'], ['z:/', 'z:/ORGS/Acme'], ['\\\\MYHOST\\WS\\', '//myhost/ws/orgs/acme/']]) { const refusal = crawlScopeRefusal({ base, ownRoot }, WINDOWS); // Beta's file, a crawl root and a file not written yet there, the default org's part, the workspace itself, orgs\acme2. @@ -178,6 +181,36 @@ test('a workspace named through a share this host serves (\\\\myhost\\ws, Z: map for (const p of ['C:\\ws\\packs\\demo.pack.yaml', 'c:/WS/Packs/demo.pack.yaml', 'C:\\WS~1', 'C:\\repo\\catalog.pack.yaml']) assert.equal(dflt(p), '', `read: ${p}`); }); +// libuv answers a stat by the path's name (GetFileInformationByName, Windows 11 24H2 and Server 2025) unless the path is +// reached through a share or its last name is a reparse point, which it opens: 1.51.0 and later read all 64 bits of the +// volume's serial by name, 1.49.0 to 1.50.x (Node 22.16.0's 1.49.2) the upper half of the file's 128-bit id — 0 on +// NTFS — while a handle reports the serial's low 32 bits on every one (server/fixtures/ntfs.mjs, fastDev). +const LIBUVS = [['1.51.0', null], ['1.49.2', 0n]]; + +test('on every libuv a local path is placed in a workspace named through a share by its file id read through a handle — on 1.49.2 a stat by name reports no volume, and acme read beta\'s file on the local drive', () => { + for (const [libuv, fastDev] of LIBUVS) { + const volume = ntfsVolume(['C:\\ws\\orgs\\acme\\packs\\own.pack.yaml', 'C:\\ws\\orgs\\beta\\packs\\b1.pack.yaml', 'C:\\repo\\catalog.pack.yaml'], { + short: { 'C:\\ws': 'WS~1' }, + shares: { '\\\\myhost\\ws': 'C:\\ws', '\\\\localhost\\C$': 'C:\\' }, + mapped: { 'Z:': '\\\\localhost\\C$\\ws' }, + fastDev, + }); + const serial = (p) => { const fd = volume.openSync(p); try { return volume.fstatSync(fd).dev; } finally { volume.closeSync(fd); } }; + assert.equal(serial('C:\\ws'), serial('\\\\myhost\\ws'), `libuv ${libuv}: through a handle C:\\ws and \\\\myhost\\ws report one serial`); + assert.equal(volume.statSync('C:\\ws').dev === 0n, fastDev === 0n, `libuv ${libuv}: by name C:\\ws reports ${fastDev === 0n ? 'no volume' : 'its volume'}`); + // A stat by name is on offer too, so a file id read by name again fails here on 1.49.2 alone. + const host = { windows: true, ...onVolume(volume), statSync: volume.statSync }; + for (const [base, ownRoot] of [['\\\\myhost\\ws', '\\\\myhost\\ws\\orgs\\acme'], ['Z:\\', 'Z:\\orgs\\acme']]) { + const refusal = crawlScopeRefusal({ base, ownRoot }, host); + for (const p of [...BETA_FILE, 'c:/WS~1/orgs/beta', 'C:\\ws\\orgs\\beta\\new\\x.pack.yaml']) { + assert.equal(refusal(p), IN_ANOTHER_ORGS_PART, `libuv ${libuv}, ${base}: not acme's: ${p}`); + } + for (const p of [...ACME_FILE, 'C:\\repo\\catalog.pack.yaml']) assert.equal(refusal(p), '', `libuv ${libuv}, ${base}: read: ${p}`); + } + assert.equal(volume.opened(), 0, `libuv ${libuv}: every handle opened is closed`); + } +}); + test('a workspace on a share: its own share name decides as a drive path does, another name of its server is refused, a local drive outside its storage is read', () => { // The share's server keeps the workspace at N:\obs on its own disk (the model's volume stands in for it); this host // reaches it as \\nas01\obs and by two more names of that server, and maps Y: to it. @@ -189,7 +222,7 @@ test('a workspace on a share: its own share name decides as a drive path does, a shares: { '\\\\nas01\\obs': 'N:\\obs', '\\\\nas01.corp.example\\obs': 'N:\\obs', '\\\\10.0.0.5\\obs': 'N:\\obs' }, mapped: { 'Y:': '\\\\nas01\\obs' }, }); - const host = { windows: true, path: win32, realpathSync: nas.realpathSync, statSync: nas.statSync }; + const host = { windows: true, ...onVolume(nas) }; const refusal = crawlScopeRefusal({ base: '\\\\nas01\\obs', ownRoot: '\\\\nas01\\obs\\orgs\\acme' }, host); // Where this host reaches the share's disk itself (here the model's one volume), N:\obs is one more name of the workspace. for (const p of ['\\\\nas01\\obs\\orgs\\beta\\packs\\b1.pack.yaml', '\\\\NAS01\\OBS\\ORGS\\BETA', '//nas01/obs/orgs/beta/', 'Y:\\orgs\\beta\\packs\\b1.pack.yaml',