Skip to content

feat: test metadata reporting and a unit-aware metric engine - #316

Draft
roxblnfk wants to merge 6 commits into
1.xfrom
feature/sandbox-test-metadata
Draft

roxblnfk wants to merge 6 commits into
1.xfrom
feature/sandbox-test-metadata

Conversation

@roxblnfk

@roxblnfk roxblnfk commented Sep 3, 2026

Copy link
Copy Markdown
Member

🔍 What was changed

  • #[TestMetadata] in testo/test: a repeatable attribute that attaches a testMetadata value (number, text, link, image, artifact) to a test. The interceptor is registered by the default TestPlugin, so any suite reports it to TeamCity out of the box. Relative image and artifact paths resolve against the test file's directory.
  • Bench metrics are reported as a bench.<case>.<columnGroup>.<column> table instead of a flat list, with the same column groups the terminal table prints. TeamCity and JUnit output of bench metrics changes; the HTML and JSON structured map does not.
  • New metric engine in Testo\Metric: #[Dimension] on a unit enum names what it measures, #[Factor] on a case gives its multiplier to the base unit. Units reflects both once per enum and does the conversion; Metric::to() and Metric::compact() sit on top. The concrete families Time, Memory, Percent, Scalar stay in Testo\Core\Metric.
  • Memory uses binary units with honest names: Bytes, Kibibytes, Mebibytes, Gibibytes with symbols B, KiB, MiB, GiB. The factors were already 1024-based.
  • Reflection::fetchEnumCaseAttributes() returns the attributes of every enum case keyed by case name.

How it works

  • A unit enum implements Unit via the UnitConversion trait and declares its cases with #[Factor]; the case without one is the base. Conversion is a ratio of factors and stays integral while every operand is an integer.
  • compact() picks the largest unit the value still reaches at least one of, derived from the factors alone, so a family needs no hook of its own.
  • Reporters convert at their own boundary: TeamCity scales every time unit to ms and memory to bytes, JUnit appends the unit symbol to the flat property name (…mean.us, …memory.B).

Why?

A value should carry its unit instead of hiding it in a key name. Reporters were each parsing units back out of leaf names, and the bench mapper rounded once for everyone. With the unit on the Metric, each consumer renders it natively and the source keeps its precision.

Checklist

  • How was this tested:
    • Tested manually
    • Unit tests added
    • composer psalm passes locally; composer test:ci -- --json --type=!bench passes except BenchAttr::rangeSum, which trips the local Xdebug nesting limit on the recursive bench and is unrelated to this branch

Review notes

  • Unit families are one enum per physical dimension. Splitting time into two enums for sub-second and wall-clock ranges was considered and rejected, since values could no longer convert between them.
  • Skills do not yet describe Testo\Metric or #[TestMetadata]; that is left for a follow-up.

@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

@roxblnfk
roxblnfk force-pushed the feature/sandbox-test-metadata branch from 65884cc to b489981 Compare September 29, 2026 17:55
@roxblnfk
roxblnfk force-pushed the feature/sandbox-test-metadata branch from b489981 to 137e4b2 Compare November 17, 2026 19:27
Every case is present in the result, in declaration order, so a caller can walk an enum once and fall back to a default where a case carries nothing.

Assisted-By: Claude Opus 5.5
…nit families

A `Metric` carries its number in the unit it was measured in, so each reporter renders the unit its own way instead of parsing it back out of a key. A family is a backed enum: `#[Dimension]` names what it measures, `#[Factor]` on a case gives its multiplier to the base unit, and `Units` reads both once per enum. Conversion stays integral while every operand is an integer, so `2 KiB` is `2048` bytes rather than `2048.0`; `compact()` picks the most readable unit from the factors alone.

`Testo\Core\Metric` ships the built-in families: `Time`, `Memory` (binary `KiB`/`MiB`/`GiB`), `Percent`, `Scalar`.

Assisted-By: Claude Opus 5.5
Bench metrics spell `bench.<case>.<columnGroup>.<column>` instead of the flat `bench.<case>.<metric>`, so a consumer that understands the four-segment `testMetadata` convention rebuilds them as a table with the same column groups the terminal prints. Every case emits every column and `bench.iterations` folds into each row's setup group, keeping the grid rectangular and the key depth uniform; a mixed depth drops a consumer back to a flat list.

Values are `Metric`s, so the unit no longer hides in the leaf name. JUnit, having no type attribute, suffixes the property name with the native unit symbol (`…mean.us`); TeamCity converts at its boundary through `TeamcityMetricType` to the number types it charts, time to milliseconds and memory to bytes. The HTML/JSON structured map is unchanged.

Assisted-By: Claude Opus 5.5
…ata]`

The interceptor is registered by the default `TestPlugin`, so any suite can report numbers, text, links, images and artifacts between `testStarted` and `testFinished`. It builds the `##teamcity[testMetadata …]` message itself rather than through the core formatter, since that format is TeamCity's stable protocol and the plugin then depends on nothing internal. A relative image or artifact path resolves against the test file's directory.

Assisted-By: Claude Opus 5.5
A `metadata` group on the showcase and the bench self-test lets the metadata-reporting tests run as a set. The showcase covers a chartable table, a graph-plottable table, text, links, and images and artifacts from local files and URLs.

Assisted-By: Claude Opus 5.5
@roxblnfk
roxblnfk force-pushed the feature/sandbox-test-metadata branch from 6d0448a to 402e89a Compare November 17, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant