Skip to content

ci(fuzz): choose and record a cadence for the fuzz job as the target matrix grows #594

Description

@justin13888

Context

extended.yml's fuzz job now runs nine targets, each for -max_total_time=600, on every push
to the default branch and on manual dispatch (PR #568 added six of the nine). The cadence was never
chosen — it is inherited from the job's first three targets, when the matrix was small and the
targets were law targets over normalised inputs.

Why the cadence is now a question

  • Cost scales with the matrix. Nine parallel ten-minute runners per push, and the design of the
    job is explicitly that adding a crate adds a row. The next few entry points in
    docs/testing.md's table (PngDecoder, JpegDecoder, WebpDecoder, IccProfile::parse,
    RiffReader, decode, parse, read_irb) would roughly double it.
  • Ten minutes from a cold, empty corpus is not the same search twice. Each run starts from the
    committed seeds only: the engine's accumulated corpus is deliberately not persisted (it is in
    .gitignore, and the README explains why — the corpus is a search aid, not the regression
    record). So sixty ten-minute runs are sixty shallow searches, not one ten-hour one. A nightly run
    with a cached corpus would search deeper for less total compute; a per-push run catches a
    regression sooner.
  • Per-push is a regression signal; scheduled is an exploration signal. They are different jobs
    wearing one name, and the tier currently only does the first badly and the second not at all.

The question

Pick, and record, a cadence for the fuzz job. Candidates:

  1. Keep per-push at 600 s. Simplest; a defect introduced by a push is caught by that push's
    own run. Cost grows linearly with the matrix.
  2. Nightly schedule: at a longer budget, per-push dropped. Deeper search, cheaper overall,
    regression detection delayed by up to a day.
  3. Both, split: a short per-push smoke run (60–120 s) plus a nightly deep run. Two rows per
    target, more YAML, and the drift guard
    (tooling/gamut-fuzz/check-targets.sh) has to reconcile two matrices instead of one.
  4. Per-push, but only for targets whose crates the push touched. Best cost/signal ratio, most
    machinery, and it needs a path filter that cannot go stale — which is the same class of problem
    the drift guard exists to solve.

Whether to cache and persist the engine corpus between runs (actions/cache keyed per target) is
part of this decision: it is what makes options 2 and 3 worth more than option 1, and it is in
tension with the "the corpus is not the regression record" rule, which is about the repository,
not about a cache.

Not urgent — nine targets at ten minutes is affordable today. It should be decided before the
matrix doubles.

Refs #264, #311, #568.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions