Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,29 @@ Versioning follows [Semantic Versioning](https://semver.org/).
[`docs/v1-scope.md`](docs/v1-scope.md#zeromq-transport).
- The whole-packet links (KISS, ZMQ) share one routing and delivery implementation
(`transport/whole_packet.py`); KISS behaviour is unchanged.
- `run --html PATH`: a self-contained HTML report of the run, as one offline file for a CI
artifact or a colleague. It links each assertion to the frame its value was decoded
from (a failing assertion leads straight to the frame that carried the wrong value),
shows the decoded frame timeline with the scenario steps and dropped packets and filters
by direction, node, port and text, and plots every numeric signal over virtual time with
the assertions marked. It needs no server and no network, is readable with scripting
off, is written even when assertions fail or the run aborts, composes with `--json`,
`--quiet` and `--trace`, and is byte-identical for the same run. See
[`docs/v1-scope.md`](docs/v1-scope.md#html-report).
- `--trace` shows a whole-packet frame as one line (`link=serial|zmq|local`).
- `run` warns on stderr, as for SocketCAN, when a `kiss` or `zmq` transport is used
without `--realtime`.
- `TransportAdapter.new_sender()` and `take_drops()`: how a packet becomes frames, and
what a link discarded, are now the transport's business, so the runner and OBC
Peer work unchanged over CAN and serial.

### Changed (internal)

- The `--trace` line is now rendered from a structured `FrameRecord`
(`transport.tracing.describe_frame` / `format_frame_record`) instead of being built as
text directly; the output is unchanged. `TracingTransportAdapter` takes an optional
`observer`, and its `stream` is now optional.

### Fixed

- A configured command whose target is a `software`/`hardware` node no longer aborts
Expand Down
28 changes: 26 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,29 @@ it, not a parallel report. It goes to `stderr`, so it composes with `--json`
and `--quiet`, and it works on SocketCAN too, where it is the only way to see
what your testbed actually put on the wire.

**8. A report you can attach to a CI run**

Add `--html report.html` and the run also writes one self-contained HTML file:

```sh
cubesat-testbed run --example default --html report.html
```

It needs nothing to open (no server, no network, no installed testbed) and shows what
`--trace` shows, in a form you can send to a colleague or keep as a CI artifact:

- the **assertions**, each linked to the frame its value was decoded from, so a failing
assertion takes you straight to the frame that carried the wrong value;
- the **frame timeline**, decoded as in the trace, interleaved with the scenario steps
and any dropped packets, with filters by direction, node, port and text;
- **plots** of every numeric signal over virtual time, with the assertions marked on them;
- the setup (transport and nodes) and, if the run aborted, the error and everything
recorded up to it.

It is written even when assertions fail or the run aborts, and it composes with
`--json`, `--quiet` and `--trace`. The file contains no wall-clock time, so the same run
produces the same bytes.

## Hardware in the loop

Point the same scenario at a real bus: switch the node under test to
Expand Down Expand Up @@ -300,8 +323,9 @@ sudo ip link set up vcan0
USB-serial or a pty) and ZeroMQ (a libcsp `zmqproxy` hub; needs the `zmq`
extra: `pip install 'cubesat-testbed[zmq]'`).
- Deterministic virtual-time engine, TOML setup + YAML scenarios, PASS/FAIL
CLI report with CI-friendly exit codes, JUnit XML output and a decoded
wire-level frame trace (`--trace`).
CLI report with CI-friendly exit codes, JUnit XML output, a decoded
wire-level frame trace (`--trace`) and a self-contained HTML report
(`--html`).

Full detail, constraints, and what's deliberately out of scope for v1:
[`docs/v1-scope.md`](docs/v1-scope.md).
Expand Down
2 changes: 2 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ src/cubesat_testbed/
transport/kiss.py # serial KISS adapter (whole CSP packets, local delivery)
transport/serial_line.py # termios serial line (tty or pty), standard library only
transport/whole_packet.py # routing shared by the whole-packet links (KISS, ZMQ)
report/recorder.py # observer that records a run for the HTML report
report/html_report.py # renders the recording as one self-contained HTML file
transport/zmq.py # ZeroMQ adapter: a client of a libcsp zmqproxy hub
transport/zmq_link.py # pyzmq PUB/SUB connection (optional `zmq` extra)
protocol/csp_v2_packet.py # whole-packet frame and 6-byte header shared by KISS and ZMQ
Expand Down
5 changes: 4 additions & 1 deletion docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ unblocked that user, highest first.
## Landed since v1.1.0 (unreleased)

On `main`, not yet in a published release; see the `[Unreleased]` section of
[`CHANGELOG.md`](../CHANGELOG.md). Each item had its golden vectors committed and
[`CHANGELOG.md`](../CHANGELOG.md). Items 1-3 had their golden vectors committed and
verified in CI before any codec or adapter code.

1. [x] **CFP2 fragmentation and reassembly.** A packet longer than one CAN frame is
Expand All @@ -95,6 +95,9 @@ verified in CI before any codec or adapter code.
3. [x] **ZeroMQ transport.** The testbed as a client of a libcsp `zmqproxy` hub, so
an unmodified libcsp process can exchange packets with a simulated node; needs
the optional `zmq` extra.
4. [x] **Self-contained HTML run report.** `run --html OUT.html` writes one offline
file with the assertions linked to the frames they were judged on, the decoded
frame timeline with filters, signal plots and dropped packets.

## Next

Expand Down
52 changes: 52 additions & 0 deletions docs/v1-scope.md
Original file line number Diff line number Diff line change
Expand Up @@ -661,6 +661,58 @@ touch the assertion path, and enabling it does not change scenario results or
determinism. From the Python API, pass a stream as `trace=` to `run_scenario`,
`run_scenario_files` or `build_runtime`.

### HTML report

`--html PATH` writes one self-contained HTML file for the run, in addition to the
normal output:

```sh
cubesat-testbed run --example default --html report.html
```

**What it contains.**

- **Assertions.** Result, expected and actual value, when it was evaluated and an
*evidence* link: the frame (the last fragment, for a multi-frame packet) from which the
signal's latest value was decoded when the assertion was evaluated. For a passing
assertion that is the frame that satisfied it, for a failing one the frame whose value
did not match; a signal never seen on the bus says so and has no link. Failed
assertions are repeated in a list at the top.
- **Setup.** The transport and every node (address, mode, module, `max_packet_bytes`).
- **Signals.** A step plot of each numeric telemetry signal over virtual time, with a
line where an assertion on it was evaluated; a table of value changes for the others
(enumerations, booleans).
- **Timeline.** Every frame that crossed the transport boundary, decoded exactly as
`--trace` prints it (both are rendered from the same `FrameRecord`), interleaved with
the scenario steps and the dropped packets. On the in-memory bus a frame appears twice,
`TX` when sent and `RX` when read back. Filters: direction, node, port, the three kinds
of row, and free text. A multi-frame packet shows every fragment, the earlier ones
dimmed, and the last one with the reassembled payload.

**Properties.**

- *Self-contained:* inline stylesheet, script and SVG; no font, image or network fetch.
Everything is readable with scripting off: only the filters need it, and the evidence
links are plain `#f<id>` anchors. Following one to a row a filter has hidden clears the
filters.
- *Passive:* it is an observer on the same transport wrapper as `--trace`. It changes
neither scenario semantics, ordering nor determinism, and a run gives the same result
with and without it.
- *Deterministic:* the file is a pure function of the run, with no wall-clock time, host
name, absolute path or random id, so the same inputs give the same bytes.
- *Always written:* also when assertions fail (exit code 1) and when the run aborts
(exit code 2), the latter with the error and everything recorded up to it. If the file
cannot be written the run is an execution error (exit code 2).
- *Composes* with `--json`, `--quiet` and `--trace`; the only extra output is a note on
`stderr` unless `--quiet`.

The timestamps in it are the virtual ones described under
[Timestamps of frames from a real peer](#timestamps-of-frames-from-a-real-peer).

From Python, pass a `cubesat_testbed.report.RunRecorder` as `recorder=` to
`run_scenario`, `run_scenario_files` or `build_runtime`, then
`cubesat_testbed.report.write_report(path, recorder, result)`.

### Hardware-in-the-loop from the CLI

`--realtime` runs the scenario on a `RealTimeClock` instead of the default
Expand Down
35 changes: 35 additions & 0 deletions src/cubesat_testbed/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
available_examples,
get_example,
)
from cubesat_testbed.report import RunRecorder, write_report
from cubesat_testbed.scenario import (
ScenarioRunResult,
build_obc_rules_from_file,
Expand Down Expand Up @@ -123,6 +124,17 @@ def _build_parser() -> argparse.ArgumentParser:
"CSP header fields, raw bytes and the decoded command/telemetry signal"
),
)
run_parser.add_argument(
"--html",
type=Path,
metavar="PATH",
help=(
"write a self-contained HTML report of the run to PATH: assertions linked to the "
"frames they were judged on, the decoded frame timeline with filters, signal "
"plots over virtual time, dropped packets and the scenario steps. One offline "
"file, written even when the run aborts; composes with --json and --quiet"
),
)
run_parser.add_argument(
"--quiet",
action="store_true",
Expand Down Expand Up @@ -189,6 +201,7 @@ def _run_command(args: argparse.Namespace) -> int:
suppress_assertion_output = bool(args.quiet or args.json)
output = io.StringIO() if suppress_assertion_output else None
clock: Clock | None = RealTimeClock() if args.realtime else None
recorder = RunRecorder() if args.html is not None else None

started = time.perf_counter()
try:
Expand All @@ -211,9 +224,17 @@ def _run_command(args: argparse.Namespace) -> int:
# The trace goes to stderr precisely so it stays out of the way of
# --json/--quiet stdout, which is why it ignores both flags.
trace=sys.stderr if args.trace else None,
recorder=recorder,
)
except Exception as exc: # noqa: BLE001
# CLI contract: execution errors map to 2; assertion failures map to 1 via result.passed.
if recorder is not None and args.html is not None:
# A run that aborts is when the report is most wanted; it shows what was
# recorded up to the error. Failing to write it must not hide the real error.
try:
write_report(args.html, recorder, None, error=str(exc))
except OSError as write_error:
print(f"warning: could not write the HTML report: {write_error}", file=sys.stderr)
if args.junit_xml is not None:
_write_junit_error_xml(
args.junit_xml, str(exc), kind="execution_error", started=started
Expand All @@ -232,6 +253,20 @@ def _run_command(args: argparse.Namespace) -> int:
format_junit_xml(result, wall_time_seconds=elapsed), encoding="utf-8"
)

if recorder is not None and args.html is not None:
try:
write_report(args.html, recorder, result)
except OSError as exc:
_print_error(
f"could not write the HTML report to {args.html}: {exc}",
json_output=args.json,
exit_code=_EXIT_EXECUTION_ERROR,
kind="report_error",
)
return _EXIT_EXECUTION_ERROR
if not args.quiet:
print(f"html report written to {args.html}", file=sys.stderr)

if not result.assertions:
print("warning: 0 assertions in scenario", file=sys.stderr)

Expand Down
24 changes: 24 additions & 0 deletions src/cubesat_testbed/report/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
"""Run recording and the self-contained HTML report built from it."""

from cubesat_testbed.report.html_report import render_report, write_report
from cubesat_testbed.report.recorder import (
RecordedAssertion,
RecordedDrop,
RecordedFrame,
RecordedStep,
RunRecorder,
describe_step,
format_virtual_time,
)

__all__ = [
"RecordedAssertion",
"RecordedDrop",
"RecordedFrame",
"RecordedStep",
"RunRecorder",
"describe_step",
"format_virtual_time",
"render_report",
"write_report",
]
Loading
Loading