Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,78 @@ move: the JSON report's `schemaVersion` and the baseline file's. Both are bumped
a field is removed, renamed, or changes meaning — new fields may appear without one, so
consumers must ignore what they do not recognise.

## Unreleased

### Fixed

- **`audit --browser` audited nothing at all unless it was given a directory.** The
runner takes the build directory so it can serve the pages over loopback; passing
nothing is how a caller says these pages came off a running site and already have
somewhere to be fetched from. The audit command passed the directory *the user typed* —
which under auto-detection is nothing, because auto-detection is the thing that works it
out. So `eaa-kit audit --browser`, the form the README leads with, skipped the server and
navigated Chromium to `/home/you/site/dist/index.html`: a filesystem path, not a URL.
Every page came back `Cannot navigate to invalid URL`.

The collection stage now reports the directory it settled on, and both commands pass that
along. A real crawl still passes nothing, because serving a crawled page back out of a
copy on disk would audit the markup with the server that produced it cut out of the
picture. `--browser` with an explicit directory was never affected, which is why the
suite did not see it: every browser test named one.

- **A run where every page failed reported "No violations".** The count in that sentence was
of pages attempted, not pages audited, so a run that could read none of them opened its
summary with a clean result in green — the exact fail-open the rest of the tool is built
to refuse — with the error line beneath it. It now says nothing was audited, and where
only some pages failed the counts are over the pages that were actually read, so
"no violations across 2 pages" cannot describe a run that opened one.

- **A baseline stopped suppressing a barrier when anything else on the page changed**, and
`diff` reported that same untouched barrier as **fixed**. The identity of a violating
element hashed the element's whole outer markup, and for the document-level rules —
`html-has-lang`, `document-title`, and every other rule that fails against `<html>` — the
element's outer markup is the entire page. Adding one paragraph gave them a new identity,
and all three consumers of that identity believed it: the build went red on barriers a
team had accepted, `diff` announced a missing `<title>` as fixed while the page still had
none, and SARIF churned its fingerprints so code scanning closed an alert and opened an
identical one on every edit.

The hash now covers the element's opening tag and not its descendants. Where two elements
share an opening tag, axe-core's selector already tells them apart — it qualifies an
ambiguous match with `:nth-child(…)` — so nothing that could be distinguished before
stops being distinguishable, and a barrier that is genuinely fixed still changes identity.
This moves both file contracts; see **Changed**.

- **A browser run reported criteria as unevaluable and advised `--browser`.** The coverage
view consulted the table of rules jsdom is structurally blind to without asking which
engine had run, so a report produced in real Chromium said colour contrast and target
size were beyond it, and closed by suggesting the flag that run had been given. It now
believes a browser run's own results: a rule reaches no verdict there only if that run
said so.

- **A filter that excluded every page was reported as an empty build directory**, together
with framework advice naming another directory to audit — a fix for a path that was never
wrong. `--include`/`--exclude` matching nothing now says so, and echoes the patterns it
was given.

- One unreachable page was counted in the singular and conjugated in the plural: "1 page
could not be reached, and were not audited".

### Changed

- **The baseline file's `schemaVersion` moves to 2** and the JSON report's to 2, because
`fingerprint` changed meaning in both. Neither is read across the boundary: a baseline
written by 0.4.0 records identities under the old rule, and matching them against the new
one would suppress nothing while looking as though it had, so it is refused with the
command that rewrites it. `diff` likewise refuses to compare a 0.4.0 report against a
newer one — that comparison is precisely the one that reports every document-level
barrier as both new and fixed. SARIF's partial fingerprint key moves to `eaaKit/v2` for
the same reason, which is what tells code scanning these are a new scheme rather than
defects that moved.

Re-record a baseline with `eaa-kit baseline`, and read the new file before committing it:
it lists what this run found, which is not necessarily what the old one accepted.

## 0.4.0 — 2026-09-01

### Added
Expand Down
16 changes: 12 additions & 4 deletions docs/baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,13 @@ to become one.

```jsonc
{
"schemaVersion": 1,
"schemaVersion": 2,
"createdOn": "2026-08-27",
"entries": [
{
"page": "index.html",
"ruleId": "image-alt",
"fingerprint": "f0e17d2582e9a5b3", // rule + selector + markup, not the path
"fingerprint": "f0e17d2582e9a5b3", // rule + selector + opening tag, not the path
"selector": "img", // the rest is for whoever reads the file
"help": "Images must have alternative text",
"impact": "critical",
Expand All @@ -70,8 +70,16 @@ to become one.

The fingerprint deliberately excludes the page path — it is the same one SARIF uses — so
moving a page does not invalidate the entry's identity, though the `page` field does have
to match. Entries are sorted, so the file diffs cleanly and two people regenerating it get
the same result.
to match. It also excludes anything nested inside the failing element, so an entry for a
document-level rule such as `html-has-lang` survives an edit elsewhere on the page.
Entries are sorted, so the file diffs cleanly and two people regenerating it get the same
result.

A baseline written by an earlier version is refused rather than read: `schemaVersion` 1
recorded fingerprints under the old rule, and matching them against the new one would
suppress nothing while looking as though it had. Record it again with `eaa-kit baseline`,
then read the new file before committing it — it lists what this run found, which is not
necessarily what the old one accepted.

## In code scanning

Expand Down
31 changes: 19 additions & 12 deletions docs/reports.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ shareable ones.

### Compatibility

- `schemaVersion` is an integer, currently `1`.
- `schemaVersion` is an integer, currently `2`.
- It is bumped only when a field is **removed, renamed, or changes meaning**.
- New fields may be added without a bump, so **consumers must ignore fields they do not
recognise**.
Expand All @@ -33,7 +33,7 @@ A complete generated document is checked in at

```jsonc
{
"schemaVersion": 1,
"schemaVersion": 2,
"tool": {
"name": "eaa-kit",
"version": "0.1.0",
Expand Down Expand Up @@ -182,18 +182,25 @@ at its limit. A tool deciding whether a clean report means anything should read
fetched; an errored one was fetched and then could not be audited. They are different
failures with different fixes, and summing them would name neither.

`completeness` was added after `schemaVersion` 1 and does not move it: new fields may
`completeness` was added without moving `schemaVersion`: new fields may
appear without a bump, and consumers must ignore what they do not recognise. A consumer
written against version 1 that has never seen this field should treat its absence as
unknown rather than as a complete run.
written against an earlier version that has never seen this field should treat its absence
as unknown rather than as a complete run.

### `fingerprint`

Each node carries the same identity the baseline file records and SARIF sends as a partial
fingerprint: a hash of the rule, the selector and the element's own markup, and deliberately
not of the page it was found on. Two consumers already depended on it agreeing — SARIF, so
that moving a page does not close one code-scanning alert and open an identical one, and
the baseline, so an accepted violation stays accepted when the surrounding page changes.
fingerprint: a hash of the rule, the selector and the element's own opening tag, and
deliberately not of the page it was found on. Two consumers already depended on it
agreeing — SARIF, so that moving a page does not close one code-scanning alert and open an
identical one, and the baseline, so an accepted violation stays accepted when the
surrounding page changes.

The tag, not the element's whole markup. axe-core reports the failing element's outerHTML,
which for `<html>` — the element every document-level rule fails against — is the entire
page. Hashing that made `html-has-lang` and `document-title` change identity on any edit
to the page they were on, which is precisely what these two consumers exist to survive.
Where two elements share an opening tag, axe-core's selector already tells them apart.

It is emitted so a third does not have to reimplement the hash and risk disagreeing with
them. `eaa-kit diff` matches on it.
Expand All @@ -213,7 +220,7 @@ the site — which is the same mistake as adding `inapplicable` to `passes`, and
`notEvaluated` criteria a `--browser` run would decide. The remainder need a person
whatever engine runs.

Added after `schemaVersion` 1 and does not move it.
Added without moving `schemaVersion`.

## Comparing two runs

Expand Down Expand Up @@ -285,7 +292,7 @@ Real results from the fixture run above, abbreviated:
}
}
],
"partialFingerprints": { "eaaKit/v1": "4c2a13ab4c8c0365" }
"partialFingerprints": { "eaaKit/v2": "b6128eff391a4be6" }
},
{
"ruleId": "image-alt",
Expand All @@ -300,7 +307,7 @@ Real results from the fixture run above, abbreviated:
}
}
],
"partialFingerprints": { "eaaKit/v1": "f0e17d2582e9a5b3" }
"partialFingerprints": { "eaaKit/v2": "f0e17d2582e9a5b3" }
}
],
"invocations": [{ "executionSuccessful": true, "toolExecutionNotifications": [] }],
Expand Down
6 changes: 3 additions & 3 deletions examples/baseline.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 10 additions & 10 deletions examples/report.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions examples/report.sarif

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 16 additions & 3 deletions src/audit/baseline.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,18 @@ import { type Finding, type FindingNode, findingElements, type PageAudit } from
* anything. Nothing here is permanent unless somebody keeps deciding it is.
*/

/** Bumped only when an existing field is removed, renamed, or changes meaning. */
export const BASELINE_SCHEMA_VERSION = 1
/**
* Bumped only when an existing field is removed, renamed, or changes meaning.
*
* 2: `fingerprint` changed meaning. It used to hash the failing element's whole
* outerHTML, which for the document-level rules is the entire page — so every
* entry for `html-has-lang` or `document-title` stopped matching the moment
* anybody edited the page it was on, and the build went red on barriers that
* had been accepted. Entries written under 1 cannot be matched against 2, and
* silently accepting the file would suppress nothing while looking as though it
* had, so it is refused with the command that rewrites it.
*/
export const BASELINE_SCHEMA_VERSION = 2

/** Default filename, used by the CLI when no path is given. */
export const DEFAULT_BASELINE_FILE = 'eaa-baseline.json'
Expand Down Expand Up @@ -261,7 +271,10 @@ export async function readBaseline(file: string, cwd = process.cwd()): Promise<B
}
if (result.data.schemaVersion !== BASELINE_SCHEMA_VERSION) {
throw new BaselineError(
`${path.basename(target)} has schemaVersion ${result.data.schemaVersion}; this version of eaa-kit reads ${BASELINE_SCHEMA_VERSION}`,
`${path.basename(target)} has schemaVersion ${result.data.schemaVersion}; this version of eaa-kit reads ${BASELINE_SCHEMA_VERSION}.\n` +
` Record it again on the current build: eaa-kit baseline\n` +
' Read the new file before committing it: it lists what this run found, which is\n' +
' not necessarily what the old one accepted.',
)
}

Expand Down
23 changes: 23 additions & 0 deletions src/audit/collect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,29 @@ export async function collectPages(
return pages
}

/**
* Whether the directory holds any HTML at all, ignoring include and exclude.
*
* Asked when a run collected nothing, to tell two different mistakes apart: a
* build directory with no pages in it, and a directory full of pages that the
* caller's own filters excluded. The advice for the first is wrong for the
* second — it names another directory to audit — so the question has to be
* answered before it is given.
*
* Globs rather than collecting: this only needs to know whether one file
* exists, and reading every page of a large build to answer that on a path that
* is about to print a warning and stop would be work for nothing.
*/
export async function holdsHtml(dir: string): Promise<boolean> {
const found = await glob([...DEFAULT_INCLUDE], {
cwd: path.resolve(dir),
ignore: [...DEFAULT_EXCLUDE],
onlyFiles: true,
dot: false,
})
return found.length > 0
}

async function assertDirectory(root: string, original: string): Promise<void> {
let stats: Awaited<ReturnType<typeof stat>>
try {
Expand Down
Loading
Loading