Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,21 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- `TransportAdapter.new_sender()` and `take_drops()`: how a packet becomes frames, and
what a link discarded, are now the transport's business, so the runner and OBC
Peer work unchanged over CAN and serial.
- Bus fault injection. Faults used to stop at the module boundary; now the in-memory
bus can fail too. `[bus_faults]` in the setup (a `seed` and `[[bus_faults.rules]]`) and
`inject_bus_fault` scenario steps apply `drop`, `delay`, `corrupt`, `duplicate` and
`reorder` rules to CAN frames, on send (`direction = "tx"`) or on receive (`"rx"`),
selected by `port`, `source_port`, `source`, `destination` and the `nth` matching
frame (with `count`) or a `probability`. A rule on a port reaches every fragment of a
multi-frame packet, so `nth` can drop exactly one fragment and the receiver reports
the loss as `frame_lost`/`timeout`. Delays are virtual-time events; every random
choice comes from a generator seeded from `seed` and the rule's position, so the same
seed gives a byte-identical `--trace`. Every injected fault is reported: `FAULT`
lines in `--trace` (right after the `TX` line of the frame concerned) and in the text
output, `bus_faults` and `bus_fault_seed` in `--json` and on the run result, rows in
the HTML report. In-memory transport only: the setup is rejected on `socketcan`, `kiss`
and `zmq`, which run on wall-clock time. See
[`docs/v1-scope.md`](docs/v1-scope.md#bus-faults).

### Changed (internal)

Expand Down Expand Up @@ -91,6 +106,10 @@ Versioning follows [Semantic Versioning](https://semver.org/).

### Changed

- A telemetry payload the signal's codec cannot read (an enum value with no label, say a
byte a bus fault corrupted) is now a reported dropped packet (`reason=undecodable`,
in `--trace`, the text output, `--json` and the HTML report) and the run goes on, instead
of ending with an execution error.
- A command `payload_hex` or telemetry layout longer than 4 bytes is accepted;
the former 4-byte limit is replaced by the sending node's `max_packet_bytes`.
- `ObcPeerNamedCommand.encode()` returns the tuple of frames carrying the command
Expand Down
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,25 @@ It is written even when assertions fail or the run aborts, and it composes with
`--json`, `--quiet` and `--trace`. The file contains no wall-clock time, so the same run
produces the same bytes.

**9. Break the bus itself**

`inject_fault` breaks a module. The bus can fail too: drop the Nth frame on a port,
delay a frame by a virtual duration, flip a byte, duplicate a frame or deliver it after
the next one, on send or on receive. Add the rule to the setup, or as a scenario step:

```yaml
- action: inject_bus_fault
type: drop
port: 10 # destination port; the 2nd frame addressed to it is lost
nth: 2
```

The randomness (`probability`, the byte a `corrupt` rule picks) is seeded
(`[bus_faults] seed = 42`), delays are virtual time, so two runs with one seed give
byte-identical traces, and every fault is marked in `--trace`, the text and `--json`
output and the HTML report, never silent. In-memory transport only. See
[`docs/v1-scope.md`](docs/v1-scope.md#bus-faults).

## Hardware in the loop

Point the same scenario at a real bus: switch the node under test to
Expand Down
8 changes: 8 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ src/cubesat_testbed/
transport/base.py # Bus adapter interface
transport/factory.py # Config-to-adapter construction
transport/in_memory.py # CI/local in-memory bus
transport/faults.py # bus fault decorator: drop/delay/corrupt/duplicate/reorder frames
transport/socketcan.py # Linux SocketCAN adapter
transport/kiss.py # serial KISS adapter (whole CSP packets, local delivery)
transport/serial_line.py # termios serial line (tty or pty), standard library only
Expand Down Expand Up @@ -125,6 +126,13 @@ the next event time instead of spinning in idle loops.
- `signal_override` targets `.telemetry.` paths and spoofs outgoing telemetry.
- `named_fault` activates a module-defined fault flag.

`FaultInjectingTransport` (`transport/faults.py`) is the same kind of passive
executor one level down: it breaks the *bus* instead of a module, applying explicit
`[bus_faults]` rules and `inject_bus_fault` steps to the frames of the in-memory
transport (drop the Nth frame on a port, delay, corrupt, duplicate, reorder). Its
randomness is seeded and its delays are virtual-time events. See
[Bus faults](v1-scope.md#bus-faults).

The OBC Peer is the active decision maker. It listens to telemetry and applies
rules such as:

Expand Down
5 changes: 5 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,11 @@ verified in CI before any codec or adapter code.
4. [x] **Self-contained HTML run report.** `run --html OUT.html` writes one offline
file with the assertions linked to the frames they were judged on, the decoded
frame timeline with filters, signal plots and dropped packets.
5. [x] **Bus fault injection.** Faults no longer stop at the module boundary: on the
in-memory bus a scenario can drop the Nth frame on a port, delay a frame by a
virtual duration, flip a byte, duplicate a frame or deliver it after the next
one, on send or on receive, with a seed. Two runs with one seed give
byte-identical traces and `--trace` marks every injected fault.

## Next

Expand Down
94 changes: 92 additions & 2 deletions docs/v1-scope.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,92 @@ setup against several different FDIR rule sets without editing it. A rules
file reuses the exact same schema, keyed by node name at the top level (see
[`configs/schema/module_schema.md`](../configs/schema/module_schema.md#obc-peer-rules)).

### Bus faults

`FaultInjectionEngine` breaks a *module*. Bus faults break the *bus*: the in-memory
transport is wrapped in a decorator (`transport/faults.py`,
`FaultInjectingTransport`) that applies explicit rules to the CAN frames passing
through it. It is passive like the engine above: it decides nothing, it applies the
rules it was given. A rule says which frames, when, and what happens:

```toml
[bus_faults]
seed = 42 # every random choice derives from this; default 0

[[bus_faults.rules]]
type = "drop" # drop | delay | corrupt | duplicate | reorder
port = 10 # destination port; also source_port, source, destination
nth = 2 # the 2nd matching frame (1-based)
```

The same fields work as a scenario step, which puts the rule in force from that
instant on and starts counting `nth` from there:

```yaml
- action: inject_bus_fault
type: delay
port: 20
nth: 1
delay: 300ms
```

| field | meaning |
| --- | --- |
| `type` | `drop` loses the frame; `delay` holds it for `delay` of virtual time; `corrupt` XORs `mask` (default `0xFF`) into data byte `byte` (default: one drawn from the seed); `duplicate` delivers it twice; `reorder` holds it until the next frame in that direction has gone by, or for `hold` (default `1s`) if none comes |
| `direction` | `tx` (default) acts when a node sends, before the frame reaches the bus; `rx` acts on what the receiver reads off it. Write two rules for both |
| `port`, `source_port`, `source`, `destination` | selectors; a frame must match all that are given. `port` is the CSP destination port |
| `nth`, `count` | fire on the `nth` matching frame and the next `count - 1` (default 1). Without `nth` the rule fires on every matching frame, at most `count` times if given |
| `probability` | fire on each matching frame with this probability, drawn from the seeded generator; not combinable with `nth` |
| `name` | label in the trace and the result; default `<type>-<position>` |

Things worth knowing:

- **Frames, not packets.** A multi-frame packet is several frames, so `nth` can drop
exactly one fragment. Only a packet's first frame carries its ports and source
address, so the decorator remembers each first frame's header and matches the
continuation fragments by the packet they belong to; a rule on `port = 10` therefore
reaches every fragment of a packet to port 10. The receiver then reports the loss
like any other (`frame_lost`, `timeout`, ...), as dropped packets.
- **`byte` indexes the frame's CAN data field**, as `--trace` prints it in `data=`
(the first frame's first four bytes are the CSP header). Flipping a header byte
can make the frame unroutable or undecodable, which the trace then says. A `byte`
past the end of a short frame corrupts nothing; the fault is still reported, with
`skipped` in its detail.
- **One rule per frame.** Rules are tried in the order they were added (setup first,
then scenario steps); the first one that fires acts, and a rule counts only the
frames that reached it.
- **Delay is virtual.** A held frame is released by a scheduled event at its
virtual time and delivered in that same instant; nothing sleeps.
- **Determinism.** Each rule has its own generator, seeded from `seed` and the rule's
position, and nothing reads the wall clock, so the same seed, rules and scenario
give a byte-identical `--trace`. Adding a rule does not change the draws of the
others. The seed is part of the result (`bus_fault_seed`).
- **Visible.** `--trace` prints a `FAULT` line right after the `TX` line of the frame
it concerns (an `rx` fault, which has no such line, is printed where it happens):

```text
trace t=3000000 TX can_id=0x10006083 dlc=5 pri=2 src=1 dst=3 dport=10 ... command obc.payload_power_off->payload
trace FAULT t=3000000 rule=drop-1 action=drop dir=TX match=1 can_id=0x10006083 src=1 dst=3 dport=10 sport=10 data=0004a28000
```

A `delay` fault carries `until=<virtual time>`, a `corrupt` fault the byte, mask and
before/after values, a `reorder` fault its `hold`. The same events are `FAULT` lines
in the text output, `bus_faults`/`bus_fault_seed` in `--json`, rows in the HTML
report, and `bus_faults` on the run result.
- **What `TX` and `RX` mean once rules exist.** The tracing adapter sits above the
decorator, so `TX` is a frame a node *tried* to send and `RX` a frame that
*reached* the receiver. A dropped frame is a `TX` with a `FAULT drop` and no `RX`;
a duplicated one is one `TX` and two `RX`.
- **`rx` faults act on the bus monitor stream**, which is where the runner decodes
every frame for every node; per-endpoint streams pass through unchanged.

**In-memory transport only.** A real bus, a serial line and a ZeroMQ hub deliver on
wall-clock time, so a delay or an ordering there could not be reproduced from a seed,
and a fault can only touch what the testbed itself sends or reads, not traffic between
two real devices. `[bus_faults]` and `inject_bus_fault` are therefore rejected when
the setup's transport is not `in-memory`. The decorator is always present on the
in-memory bus and is a plain pass-through until a rule exists.

## Config and scenario files

Static CubeSat/testbed setup uses TOML; scenario scripts use YAML. Both are
Expand Down Expand Up @@ -234,8 +320,9 @@ A receiving node discards a packet, as libcsp does, when a frame is missing or
out of order (`frame_lost`), the packet outgrows the node's limit (`overflow`),
a continuation frame has no packet in progress (`orphan_fragment`), a new first
frame arrives mid-packet (`restarted`), the frame or header is invalid
(`malformed`), or no frame arrives for one virtual second (`timeout`). Nothing is
discarded silently:
(`malformed`), or no frame arrives for one virtual second (`timeout`). A complete
packet whose telemetry payload the signal's codec cannot read (a value with no enum
label, a corrupted byte) is discarded as `undecodable`. Nothing is discarded silently:

- `run` prints a `DROP t=... node=... reason=... detail=...` line per dropped
packet before the summary (`SUMMARY` gains `dropped_packets=N`);
Expand Down Expand Up @@ -651,6 +738,9 @@ consequences worth knowing:
decodes it. That round trip is precisely what makes an assertion an
observation of the wire rather than of module state (see
[Telemetry wire encoding](#telemetry-wire-encoding)).
- With bus fault rules in force, `TX` means a frame a node tried to send and `RX`
one that reached the receiver; injected faults are `FAULT` lines between them
(see [Bus faults](#bus-faults)).
- On SocketCAN the adapter does not receive its own messages, so outgoing
frames appear only as `TX` and everything the peer sends appears as `RX`.
This is the only place an outgoing HIL command is visible at all.
Expand Down
10 changes: 10 additions & 0 deletions src/cubesat_testbed/config/__init__.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
"""Configuration and scenario schema parsing."""

from cubesat_testbed.config.parser import (
BUS_FAULT_DEFAULT_HOLD_US,
AssertionOperator,
AssertStep,
BusFaultRule,
BusFaultsConfig,
BusFaultType,
CommandMapping,
ConfigParseError,
FaultType,
InjectBusFaultStep,
InjectFaultStep,
InMemoryTransportConfig,
KissTransportConfig,
Expand Down Expand Up @@ -39,12 +44,17 @@
)

__all__ = [
"BUS_FAULT_DEFAULT_HOLD_US",
"AssertStep",
"AssertionOperator",
"BusFaultRule",
"BusFaultType",
"BusFaultsConfig",
"CommandMapping",
"ConfigParseError",
"FaultType",
"InMemoryTransportConfig",
"InjectBusFaultStep",
"InjectFaultStep",
"KissTransportConfig",
"ModuleType",
Expand Down
Loading
Loading