From 62557d665087c920b37270062d56551c49bfb8ba Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 17:36:21 +0800 Subject: [PATCH 01/14] feat: add topic authorization for v0.2.0 --- .github/workflows/release.yml | 4 +- CHANGELOG.md | 24 +++- README.md | 34 ++++-- SECURITY.md | 4 +- docs/architecture.md | 28 ++++- docs/authorization.md | 104 +++++++++++++++++ docs/cli.md | 4 + docs/compatibility.md | 4 + docs/getting-started.md | 12 +- docs/linux-service.md | 6 +- docs/protocol.md | 5 + docs/release.md | 21 ++-- docs/releases/v0.2.0.md | 60 ++++++++++ docs/security.md | 16 ++- docs/transports.md | 4 +- docs/tutorials.md | 19 ++- library.json | 2 +- library.properties | 2 +- packaging/linux/README.md | 8 +- python/README.md | 12 +- python/pyproject.toml | 2 +- python/src/opennet/__init__.py | 5 +- python/src/opennet/server.py | 35 ++++++ python/tests/certs/README.md | 5 + python/tests/certs/ca.crt | 19 +++ python/tests/certs/client.crt | 20 ++++ python/tests/certs/client.key | 28 +++++ python/tests/certs/server.crt | 20 ++++ python/tests/certs/server.key | 28 +++++ python/tests/test_integration.py | 195 ++++++++++++++++++++++++++++++- 30 files changed, 675 insertions(+), 55 deletions(-) create mode 100644 docs/authorization.md create mode 100644 docs/releases/v0.2.0.md create mode 100644 python/tests/certs/README.md create mode 100644 python/tests/certs/ca.crt create mode 100644 python/tests/certs/client.crt create mode 100644 python/tests/certs/client.key create mode 100644 python/tests/certs/server.crt create mode 100644 python/tests/certs/server.key diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f1f1d04..efa8b63 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -22,11 +22,13 @@ jobs: cache: pip - name: Verify, test, and build run: | - python -m pip install -e "./python[dev]" build + python -m pip install -e "./python[dev]" build platformio python -m pytest python python -m ruff check python benchmarks scripts python -m mypy python/src python scripts/check_docs.py + platformio test -e native + platformio run -e esp32dev python -m build python python scripts/build_release.py --expected-version "${GITHUB_REF_NAME#v}" python scripts/build_linux_bundle.py --expected-version "${GITHUB_REF_NAME#v}" diff --git a/CHANGELOG.md b/CHANGELOG.md index 6aadb8f..21b2561 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,27 @@ All notable changes are documented here. OpenNet follows ## [Unreleased] +## [0.2.0] - 2026-07-29 + +### Added + +- Optional synchronous or asynchronous server authorization before DATA handlers + and acknowledgements. +- TLS state and verified peer-certificate access through the Python `Peer` API. +- Fail-closed authorization behavior, a denial counter, mutual-TLS integration + coverage, and test-only certificate fixtures. +- A practical certificate-to-topic authorization guide and v0.2.0 release notes. + +### Changed + +- Release verification now includes the native Arduino tests and ESP32 build. +- Current install examples and package metadata target v0.2.0. + +### Compatibility + +- ONP/1 framing and typed-value encodings are unchanged. +- The Python server additions are optional and backward compatible. + ## [0.1.1] - 2026-07-29 ### Added @@ -38,6 +59,7 @@ All notable changes are documented here. OpenNet follows - Arduino IDE, PlatformIO, and Python examples. - Cross-platform tests, CI, packaging, and contributor documentation. -[Unreleased]: https://github.com/devkyato/OpenNet/compare/v0.1.1...HEAD +[Unreleased]: https://github.com/devkyato/OpenNet/compare/v0.2.0...HEAD +[0.2.0]: https://github.com/devkyato/OpenNet/compare/v0.1.1...v0.2.0 [0.1.1]: https://github.com/devkyato/OpenNet/compare/v0.1.0...v0.1.1 [0.1.0]: https://github.com/devkyato/OpenNet/releases/tag/v0.1.0 diff --git a/README.md b/README.md index 5668fdf..f8e22f8 100644 --- a/README.md +++ b/README.md @@ -7,19 +7,26 @@ [![License: MIT](https://img.shields.io/badge/license-MIT-4a638f.svg)](LICENSE) [![Protocol](https://img.shields.io/badge/protocol-ONP%2F1-4a638f.svg)](docs/protocol.md) -OpenNet is a dependency-free typed messaging protocol for direct communication -between ESP32 devices, Raspberry Pi computers, and backend services. It sends +OpenNet is a small, dependency-free typed messaging protocol I built for direct +communication between ESP32 devices, Raspberry Pi computers, and backend +services. It sends JSON, text, signed integers, finite doubles, booleans, null, and arbitrary binary data over TCP/TLS or another reliable ordered stream. -Version 0.1.1 is an alpha release for real projects and learning. It has a +Version 0.2.0 is an alpha release for real projects and learning. It has a documented wire format, bounded resource use, retries and duplicate suppression, native Arduino protocol tests, Python tests across 3.9–3.14, reproducible release -packages, measured local performance, and security guidance. It does not claim -that a radio or internet route exists when the underlying hardware/network is -unavailable. +packages, measured local performance, and an optional topic-authorization hook. +It does not claim that a radio or internet route exists when the underlying +hardware or network is unavailable. -## Why OpenNet +## What I wanted from OpenNet + +I kept coming back to one idea: a sensor message should have the same bytes and +the same meaning whether it travels through an ESP32 Wi-Fi client, a serial +stream, or a Python TLS connection. I thought about reliability on that point +too, which is why retries keep the same message ID and receivers can suppress +recent duplicates without pretending the message is durably stored. - One small API and one frame format across ESP32 and Python. - Typed values without a JSON dependency for every scalar or binary payload. @@ -30,6 +37,8 @@ unavailable. SPP on compatible ESP32 hardware. - Strict control frames, CRC-32 corruption detection, payload limits, bounded queues, connection limits, and partial transport-write handling. +- Optional Python server authorization using the topic and verified TLS peer + certificate before a handler runs or an ACK is sent. - CLI tools to serve, send, ping, and benchmark. - Arduino IDE, PlatformIO/VS Code, Python, Linux/systemd, and security tutorials. @@ -39,7 +48,7 @@ Python wheel from the release page: ```sh python -m pip install \ - https://github.com/devkyato/OpenNet/releases/download/v0.1.1/opennet_protocol-0.1.1-py3-none-any.whl + https://github.com/devkyato/OpenNet/releases/download/v0.2.0/opennet_protocol-0.2.0-py3-none-any.whl opennet --version ``` @@ -47,15 +56,15 @@ For an isolated command: ```sh pipx install \ - https://github.com/devkyato/OpenNet/releases/download/v0.1.1/opennet_protocol-0.1.1-py3-none-any.whl + https://github.com/devkyato/OpenNet/releases/download/v0.2.0/opennet_protocol-0.2.0-py3-none-any.whl ``` -For Raspberry Pi/Linux, extract `OpenNet-linux-0.1.1.tar.gz` and run +For Raspberry Pi/Linux, extract `OpenNet-linux-0.2.0.tar.gz` and run `sudo ./install.sh`. It creates a hardened, unprivileged systemd service with a -loopback-only default. For Arduino IDE, install `OpenNet-0.1.1.zip` through +loopback-only default. For Arduino IDE, install `OpenNet-0.2.0.zip` through **Sketch > Include Library > Add .ZIP Library**. -The Python distribution is installable with pip, but v0.1.1 is distributed from +The Python distribution is installable with pip, but v0.2.0 is distributed from GitHub Releases rather than the public PyPI index. ## Five-minute local test @@ -100,6 +109,7 @@ device identity matters—outside isolated local testing. - [Measured TCP/TLS and 20-client load results](docs/benchmarks.md) - [Comparison with raw TCP, MQTT, WebSocket, and HTTP](docs/comparison.md) - [Architecture, ACK retry, and defensive boundaries](docs/architecture.md) +- [Certificate-to-topic authorization](docs/authorization.md) - [Actual GitHub adoption counters and analytics limits](docs/adoption-analytics.md) - [Transport support, including Bluetooth boundaries](docs/transports.md) diff --git a/SECURITY.md b/SECURITY.md index 10c2050..98e24d2 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -22,7 +22,9 @@ coordinate disclosure after a fix is available. - Plain TCP provides no confidentiality, peer identity, or replay protection. - Use TLS with certificate validation across untrusted networks. - Do not use `setInsecure()` in production. -- Applications must authorize topics and validate payloads. +- Applications must authorize topics and validate payloads. Python servers can + use the optional pre-handler `authorizer`; other deployments must enforce an + equivalent application policy. - The 16 MiB frame limit reduces memory-exhaustion risk; deployments should choose a smaller application limit appropriate for their hardware. diff --git a/docs/architecture.md b/docs/architecture.md index 13114a1..92e3754 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -3,6 +3,9 @@ OpenNet separates ONP/1 framing from the byte stream carrying it. Both endpoints must provide a reliable, ordered stream; Wi-Fi itself is not required. +I made that separation deliberately. A transport should only have to move bytes; +it should not get to redefine what a topic, type, ACK, or message ID means. + ```mermaid flowchart LR A["Application values
JSON · text · numbers · bytes"] --> B["OpenNet API"] @@ -19,7 +22,7 @@ server. The Arduino implementation accepts either a `Client` (`WiFiClient`, `BluetoothSerial`). The frame bytes are identical across transports. BLE GATT is packet-oriented, not a continuous byte stream. It needs an adapter -that fragments and reassembles ONP frames; v0.1.1 does not claim direct BLE +that fragments and reassembles ONP frames; v0.2.0 does not claim direct BLE support. ESP32-C3/S3 boards also do not provide Bluetooth Classic SPP. ## Delivery and retry @@ -41,6 +44,27 @@ execution within a configurable bounded window. An ACK proves that the peer parsed and accepted the frame; it does not make delivery durable across power loss. Durable applications must persist outbound messages and application state. +## Authorization path + +The optional Python authorizer runs after frame validation and before duplicate +handling, the application handler, or an ACK. I thought about this ordering +carefully: even a repeated message ID must pass the current topic policy. + +```mermaid +flowchart LR + A["Validated DATA frame"] --> B{"Authorizer configured?"} + B -->|No| D["Duplicate check"] + B -->|Yes| C{"Allowed?"} + C -->|No| E["Generic ERROR and close"] + C -->|Yes| D + D --> F["Application handler"] + F --> G["ACK when requested"] +``` + +The policy can inspect `Peer.tls_enabled`, `Peer.tls_peer_certificate`, and the +validated frame. This is an application API, not an ONP/1 wire-format change. +See [Authorizing topics](authorization.md). + ## Defensive boundaries - Payload, topic, receive-queue, connection, and duplicate-window bounds prevent @@ -50,6 +74,8 @@ loss. Durable applications must persist outbound messages and application state. - Remote CLI connections require TLS unless plaintext is explicitly enabled. - TLS provides confidentiality and server identity; mutual TLS can also identify clients. +- Optional authorization can reject a peer/topic pair before application code + or acknowledgement. See [Security](security.md) for the trust model and [Transports](transports.md) for supported connection types. diff --git a/docs/authorization.md b/docs/authorization.md new file mode 100644 index 0000000..0aee2a0 --- /dev/null +++ b/docs/authorization.md @@ -0,0 +1,104 @@ +# Authorizing topics + +TLS answers one question: “Which certificate reached this connection?” The +application still has to answer another one: “What may that identity do?” + +I thought about putting permissions into ONP/1 itself, but that would make the +wire protocol choose an identity model for every project. OpenNet v0.2.0 keeps +the frame format unchanged and gives the Python server an optional `authorizer` +instead. That keeps a small home sensor setup simple while still giving a +mutual-TLS deployment one clear place to enforce its rules. + +## How the decision works + +For every DATA frame, the server follows this order: + +1. Parse and validate the ONP/1 frame. +2. Call the synchronous or asynchronous authorizer, when configured. +3. Stop on denial before duplicate handling, application code, or ACK. +4. Suppress an already-seen authorized message ID. +5. Run the application handler for a new authorized message. +6. Send an ACK when the frame requested one. + +Oh! The order on step 3 matters. Authorization runs even when a message ID looks +like a duplicate, so a peer cannot reuse an accepted ID to sneak a different +topic past the policy. + +A denial, or an exception raised by the authorizer, fails closed. The server +sends a generic `message not authorized` ERROR, increments +`ServerStats.authorization_denials`, and closes that connection. It does not +send private policy details to the peer. + +## Mutual-TLS example + +The example below allows one certificate common name to publish device status +and nothing else: + +```python +from __future__ import annotations + +import asyncio +import ssl + +from opennet import Frame, OpenNetServer, Peer + + +def common_name(peer: Peer) -> str | None: + certificate = peer.tls_peer_certificate + if certificate is None: + return None + subject = { + key: value + for relative_name in certificate.get("subject", ()) + for key, value in relative_name + } + return subject.get("commonName") + + +def authorize(peer: Peer, frame: Frame) -> bool: + return ( + peer.tls_enabled + and common_name(peer) == "greenhouse-01" + and frame.topic.startswith("greenhouse/status/") + ) + + +async def handle(_peer: Peer, frame: Frame) -> None: + print(frame.topic, frame.payload) + + +async def main() -> None: + tls = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + tls.load_cert_chain("server.crt", "server.key") + tls.load_verify_locations("devices-ca.crt") + tls.verify_mode = ssl.CERT_REQUIRED + + server = OpenNetServer( + handle, + host="0.0.0.0", + ssl_context=tls, + authorizer=authorize, + ) + await server.serve_forever() + + +asyncio.run(main()) +``` + +`tls_peer_certificate` is the dictionary returned by Python’s TLS layer after +the configured certificate checks. It is `None` on plaintext connections and +when no peer certificate is available. A production policy can match a subject, +issuer, serial number, or another certificate field, but it should use a stable +identity your own certificate process controls. + +## Boundaries + +- The authorizer is a Python server API; it adds no ONP/1 bytes or negotiation. +- Certificate identity depends on a correctly configured `SSLContext`. +- Topic authorization does not validate payload contents. +- Common names are convenient for the example, not a universal identity scheme. +- Revocation, certificate rotation, audit storage, and durable operation IDs + remain deployment responsibilities. + +See the [security guide](security.md) for the broader threat model and the +[protocol specification](protocol.md) for the unchanged ONP/1 frame rules. diff --git a/docs/cli.md b/docs/cli.md index cc88934..5f3fe3f 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -12,6 +12,10 @@ Useful controls include `--max-payload`, `--max-connections`, `--deduplication-window`, `--tls-cert`, `--tls-key`, and `--tls-client-ca`. Remote plaintext listeners require `--allow-plaintext`. +The general-purpose CLI server logs or echoes messages; it does not accept a +custom topic policy on the command line. Use the Python `OpenNetServer` +`authorizer` API when certificate-to-topic rules are required. + ## Send ```sh diff --git a/docs/compatibility.md b/docs/compatibility.md index f7e5c4b..ac84200 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -14,6 +14,10 @@ supported CPython version and provide the chosen connection. It does not imply that every historical Pi image, radio, or adapter has been physically tested. +The v0.2.0 authorization callback and TLS peer-certificate properties are Python +server features. They do not change ONP/1 compatibility and do not imply that +the Arduino client performs server-side policy enforcement. + ## Not claimed - ESP8266, RP2040/Pico W, Arduino UNO WiFi, or non-ESP32 Arduino boards. diff --git a/docs/getting-started.md b/docs/getting-started.md index d5dbb78..0c4d18b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -6,6 +6,10 @@ One machine listens as the server; other devices connect as clients. On a home o lab network, a Raspberry Pi is a convenient always-on server. OpenNet does not bypass routers, firewalls, captive portals, or internet service outages. +I like starting on loopback first because it separates application mistakes from +Wi-Fi, certificates, and firewall rules. Once that works, move the exact same +ONP/1 messages onto the real transport. + ## Raspberry Pi or backend server OpenNet supports Python 3.9 or newer on operating systems supported by Python. @@ -14,7 +18,7 @@ supported Python version. ```bash python -m pip install \ - https://github.com/devkyato/OpenNet/releases/download/v0.1.1/opennet_protocol-0.1.1-py3-none-any.whl + https://github.com/devkyato/OpenNet/releases/download/v0.2.0/opennet_protocol-0.2.0-py3-none-any.whl opennet serve ``` @@ -48,7 +52,7 @@ For another project: ```ini lib_deps = - https://github.com/devkyato/OpenNet.git#v0.1.1 + https://github.com/devkyato/OpenNet.git#v0.2.0 ``` ## Sending values @@ -68,4 +72,6 @@ Validate incoming JSON against an application schema and bound binary payload si for the receiving device. Continue with the [step-by-step tutorials](tutorials.md) and -[development tips](tips.md). +[development tips](tips.md). When a server accepts more than one device, add +[topic authorization](authorization.md) instead of treating a valid certificate +as permission to use every topic. diff --git a/docs/linux-service.md b/docs/linux-service.md index 19b2b18..48e4b48 100644 --- a/docs/linux-service.md +++ b/docs/linux-service.md @@ -6,9 +6,9 @@ library `venv` module. ## Install ```sh -grep 'OpenNet-linux-0.1.1.tar.gz' SHA256SUMS | sha256sum -c - -tar -xzf OpenNet-linux-0.1.1.tar.gz -cd OpenNet-linux-0.1.1 +grep 'OpenNet-linux-0.2.0.tar.gz' SHA256SUMS | sha256sum -c - +tar -xzf OpenNet-linux-0.2.0.tar.gz +cd OpenNet-linux-0.2.0 sudo ./install.sh ``` diff --git a/docs/protocol.md b/docs/protocol.md index 981c4ca..28faf7c 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -99,3 +99,8 @@ confidentiality, peer authentication, and tamper resistance. Senders must use version 1. A version-1 receiver must ignore non-zero reserved header bytes for forward compatibility but reject reserved flag bits. New value types or frame kinds require a protocol revision or an extension specification. + +OpenNet v0.2.0 adds an optional Python server authorization callback without +changing the ONP/1 frame, kinds, flags, types, or connection lifecycle. A denied +DATA frame receives the existing ERROR control frame and the connection closes; +it is not acknowledged. diff --git a/docs/release.md b/docs/release.md index c717ed7..8aa593a 100644 --- a/docs/release.md +++ b/docs/release.md @@ -1,12 +1,17 @@ # Release process +I keep releases intentionally mechanical. The interesting decisions belong in +the code and changelog; the tag should only package a commit that has already +passed every check. + 1. Confirm CI passes on `main`. 2. Update `CHANGELOG.md` and versions in `library.properties`, `library.json`, and `python/pyproject.toml`. 3. Create and push an annotated `vX.Y.Z` tag. -4. The release workflow verifies that all versions match, runs tests, builds the - Python wheel/source distribution, Arduino ZIP, Linux/systemd bundle, and - SHA-256 checksums, then creates a GitHub release. +4. The release workflow verifies that all versions match, runs Python and native + Arduino tests, compiles the ESP32 example, builds the Python wheel/source + distribution, Arduino ZIP, Linux/systemd bundle, and SHA-256 checksums, then + creates a GitHub release. 5. A maintainer checks installation from the published artifacts and edits the generated release notes if needed. @@ -14,12 +19,12 @@ PyPI publication is intentionally not automatic until the owner configures trust publishing. Release assets remain installable and reproducible without a registry credential. -## v0.1.1 assets +## v0.2.0 assets | Asset | Use | |---|---| -| `OpenNet-0.1.1.zip` | Arduino IDE library | -| `opennet_protocol-0.1.1-py3-none-any.whl` | Direct pip/pipx install | -| `opennet_protocol-0.1.1.tar.gz` | Python source distribution | -| `OpenNet-linux-0.1.1.tar.gz` | `sudo` Linux/systemd install | +| `OpenNet-0.2.0.zip` | Arduino IDE library | +| `opennet_protocol-0.2.0-py3-none-any.whl` | Direct pip/pipx install | +| `opennet_protocol-0.2.0.tar.gz` | Python source distribution | +| `OpenNet-linux-0.2.0.tar.gz` | `sudo` Linux/systemd install | | `SHA256SUMS` | Artifact integrity verification | diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md new file mode 100644 index 0000000..3a45dff --- /dev/null +++ b/docs/releases/v0.2.0.md @@ -0,0 +1,60 @@ +# OpenNet v0.2.0 + +OpenNet v0.2.0 adds one boundary I wanted to make explicit: a valid connection +is not automatically permission to use every topic. + +I thought about putting identity and permissions directly into ONP/1, but that +would force one authorization model onto small home projects, classroom labs, +and certificate-managed deployments alike. Instead, this release keeps every +ONP/1 byte unchanged and adds an optional policy callback to the Python server. + +## What is new + +- `OpenNetServer(..., authorizer=...)` accepts a synchronous or asynchronous + decision function. +- `Peer.tls_enabled` reports whether the connection uses TLS. +- `Peer.tls_peer_certificate` exposes the certificate dictionary supplied by + Python's verified TLS layer. +- `ServerStats.authorization_denials` counts fail-closed policy decisions. +- The release workflow now runs native Arduino tests and the ESP32 build before + packaging a tag. + +Oh! The ordering is the important part here. The authorizer runs after strict +frame validation but before duplicate suppression, the application handler, and +the ACK. A denied frame receives a generic ERROR and the connection closes. An +authorizer exception follows the same path without exposing its details. + +## Compatibility + +ONP/1 remains version 1. The 24-byte header, frame kinds, flags, value types, +CRC-32, ACK behavior for accepted messages, and Arduino frame bytes are +unchanged. Existing Python servers do not need to provide an authorizer, and the +Arduino API has no compatibility change in this release. + +## Install + +```sh +python -m pip install \ + https://github.com/devkyato/OpenNet/releases/download/v0.2.0/opennet_protocol-0.2.0-py3-none-any.whl +``` + +Arduino IDE users can install `OpenNet-0.2.0.zip`. Linux or Raspberry Pi users +can extract `OpenNet-linux-0.2.0.tar.gz` and run `sudo ./install.sh`. Check every +download against `SHA256SUMS`. + +## Verification + +- 46 Python tests, including allow, deny, fail-closed, plaintext, and real + loopback mutual-TLS authorization paths. +- Strict Ruff and mypy checks. +- Native C++ protocol tests and the `esp32dev` firmware build. +- Local documentation links and reproducible Arduino/Linux bundles. + +These tests do not turn into a hardware field claim. The existing measured +loopback numbers remain labeled as v0.1.1 results, and direct BLE GATT remains +unsupported. + +For the narrative walkthrough, see +[Authorizing topics](https://github.com/devkyato/OpenNet/blob/v0.2.0/docs/authorization.md). +The exact security boundary remains in the +[security guide](https://github.com/devkyato/OpenNet/blob/v0.2.0/docs/security.md). diff --git a/docs/security.md b/docs/security.md index 3dd43c2..a1df051 100644 --- a/docs/security.md +++ b/docs/security.md @@ -37,6 +37,11 @@ For mutual TLS, add `--tls-client-ca devices-ca.crt` to the server and `--cert device.crt --key device.key` to the client. Issue a distinct certificate per device so one device can be revoked without replacing every key. +Oh! Mutual TLS identifies a certificate, but it does not decide which topics +that certificate may use. Python servers can attach an `authorizer` that checks +the verified certificate and topic before the handler or ACK. The complete +example is in [Authorizing topics](authorization.md). + ## Application responsibilities - Authorize which device may publish or receive each topic. @@ -49,8 +54,9 @@ per device so one device can be revoked without replacing every key. ## Current limits -ONP/1 has no message-level signature or session authorization handshake. -TLS protects network streams, but a peer with an accepted certificate can still -send any topic unless the application checks it. Bluetooth pairing is not a -replacement for application authorization. OpenNet has not been independently -audited or certified. +ONP/1 has no message-level signature or session authorization handshake. TLS +protects network streams, and the optional Python authorizer provides a place +for application policy, but OpenNet does not invent that policy. Without an +authorizer or handler-level checks, a peer with an accepted certificate can send +any topic. Bluetooth pairing is not a replacement for application +authorization. OpenNet has not been independently audited or certified. diff --git a/docs/transports.md b/docs/transports.md index a3fc6a0..1279ba9 100644 --- a/docs/transports.md +++ b/docs/transports.md @@ -3,14 +3,14 @@ ONP/1 needs a reliable, ordered sequence of bytes. It does not require a specific radio. -| Connection | v0.1.1 path | Security | Notes | +| Connection | v0.2.0 path | Security | Notes | |---|---|---|---| | Wi-Fi/Ethernet/LAN | TCP | TLS recommended | Python and ESP32 | | Internet/VPN | TCP | Verified TLS required by CLI | Routers/firewalls still apply | | ESP32 Wi-Fi | `WiFiClient` / `WiFiClientSecure` | Prefer `WiFiClientSecure` | All targets need a compatible ESP32 Arduino core | | UART/USB serial | Arduino `Stream` | Physical/access controls | Already-open stream | | Bluetooth Classic SPP | ESP32 `BluetoothSerial` stream | Pairing plus app controls | Original ESP32; not C3/S3 | -| BLE GATT | Not direct in v0.1.1 | Pairing is not app authorization | Needs fragmentation/reassembly adapter | +| BLE GATT | Not direct in v0.2.0 | Pairing is not app authorization | Needs fragmentation/reassembly adapter | An internet connection does not make arbitrary devices discoverable or reachable. Peers still need an address, route, open firewall, and listening service. Offline diff --git a/docs/tutorials.md b/docs/tutorials.md index 1b45e88..21c092c 100644 --- a/docs/tutorials.md +++ b/docs/tutorials.md @@ -2,13 +2,13 @@ ## 1. Local Python round trip -Install the v0.1.1 wheel directly from the GitHub release: +Install the v0.2.0 wheel directly from the GitHub release: ```sh python -m venv .venv source .venv/bin/activate python -m pip install \ - https://github.com/devkyato/OpenNet/releases/download/v0.1.1/opennet_protocol-0.1.1-py3-none-any.whl + https://github.com/devkyato/OpenNet/releases/download/v0.2.0/opennet_protocol-0.2.0-py3-none-any.whl ``` Terminal one: @@ -27,7 +27,7 @@ opennet send device/state '{"online":true,"battery":91}' --type json ## 2. Raspberry Pi service -Download and extract `OpenNet-linux-0.1.1.tar.gz`, then: +Download and extract `OpenNet-linux-0.2.0.tar.gz`, then: ```sh sudo ./install.sh @@ -40,7 +40,7 @@ TLS before the first start. See [Security](security.md). ## 3. ESP32 over a trusted development LAN -1. Install `OpenNet-0.1.1.zip` through Arduino IDE's **Add .ZIP Library**. +1. Install `OpenNet-0.2.0.zip` through Arduino IDE's **Add .ZIP Library**. 2. Open **File > Examples > OpenNet > TelemetryClient**. 3. Enter the Wi-Fi details and the server's LAN address. 4. Start a development server with @@ -71,3 +71,14 @@ opennet benchmark --host raspberrypi.local --ca lab-ca.crt \ Repeat at different distances and busy periods. Record device versions, RSSI, payload, sample count, reconnects, p50, p95, and maximum. A good report includes failures and configuration, not only the best number. + +## 6. Authorize a device topic + +Once mutual TLS is working, the Python server can check the verified client +certificate and topic in an `authorizer`. I thought about putting this check in +each handler, but one callback is easier to audit and it runs before any handler +or ACK. + +Start with the complete [certificate-to-topic example](authorization.md). Give +each device certificate a stable identity, map that identity to a narrow topic +prefix, and test both an allowed and denied topic before deployment. diff --git a/library.json b/library.json index 7f563ab..f25cc4e 100644 --- a/library.json +++ b/library.json @@ -1,6 +1,6 @@ { "name": "OpenNet", - "version": "0.1.1", + "version": "0.2.0", "description": "Typed ONP/1 messaging over TCP/TLS and reliable Arduino streams.", "keywords": ["esp32", "raspberry-pi", "iot", "tcp", "tls", "bluetooth", "messaging"], "repository": { diff --git a/library.properties b/library.properties index a9799c8..72812c5 100644 --- a/library.properties +++ b/library.properties @@ -1,5 +1,5 @@ name=OpenNet -version=0.1.1 +version=0.2.0 author=devkyato maintainer=devkyato sentence=Typed ONP/1 messaging over Wi-Fi, Ethernet, Bluetooth serial, and UART streams. diff --git a/packaging/linux/README.md b/packaging/linux/README.md index b893083..29c6bcd 100644 --- a/packaging/linux/README.md +++ b/packaging/linux/README.md @@ -4,15 +4,15 @@ This bundle installs a dedicated, unprivileged `opennet` service under `/opt/opennet` and keeps its settings in `/etc/opennet/opennet.env`. ```sh -tar -xzf OpenNet-linux-0.1.1.tar.gz -cd OpenNet-linux-0.1.1 +tar -xzf OpenNet-linux-0.2.0.tar.gz +cd OpenNet-linux-0.2.0 sudo ./install.sh ``` The default listener is loopback-only. Read the -[security guide](https://github.com/devkyato/OpenNet/blob/v0.1.1/docs/security.md) +[security guide](https://github.com/devkyato/OpenNet/blob/v0.2.0/docs/security.md) before exposing it to a network. The complete -[service/TLS guide](https://github.com/devkyato/OpenNet/blob/v0.1.1/docs/linux-service.md) +[service/TLS guide](https://github.com/devkyato/OpenNet/blob/v0.2.0/docs/linux-service.md) shows a hardened remote setup. Use `sudo ./install.sh --no-start` to inspect or customize the unit before it starts. diff --git a/python/README.md b/python/README.md index 15c1827..6a9a416 100644 --- a/python/README.md +++ b/python/README.md @@ -1,10 +1,11 @@ # OpenNet for Python -The reference ONP/1 client and server for Raspberry Pi and backend systems. +The reference ONP/1 client and server I use for Raspberry Pi and backend +projects. ```bash python -m pip install \ - https://github.com/devkyato/OpenNet/releases/download/v0.1.1/opennet_protocol-0.1.1-py3-none-any.whl + https://github.com/devkyato/OpenNet/releases/download/v0.2.0/opennet_protocol-0.2.0-py3-none-any.whl opennet serve ``` @@ -23,5 +24,10 @@ The complete documentation, ESP32 Arduino library, examples, protocol specification, and security guidance are in the [OpenNet repository](https://github.com/devkyato/OpenNet). -The wheel is distributed through GitHub Releases for v0.1.1; it is not yet +The server also accepts an optional sync or async `authorizer`. It sees the +validated frame and the peer's TLS certificate before the handler or ACK, which +keeps topic rules in one explicit place. See +[Authorizing topics](https://github.com/devkyato/OpenNet/blob/v0.2.0/docs/authorization.md). + +The wheel is distributed through GitHub Releases for v0.2.0; it is not yet published on the public PyPI index. diff --git a/python/pyproject.toml b/python/pyproject.toml index f66c1a1..65468bd 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "opennet-protocol" -version = "0.1.1" +version = "0.2.0" description = "Dependency-free typed ONP/1 messaging for Raspberry Pi, ESP32, and backend systems" readme = "README.md" requires-python = ">=3.9" diff --git a/python/src/opennet/__init__.py b/python/src/opennet/__init__.py index 381ff4e..f37ea9f 100644 --- a/python/src/opennet/__init__.py +++ b/python/src/opennet/__init__.py @@ -11,9 +11,10 @@ decode_value, encode_value, ) -from .server import OpenNetServer, Peer, ServerStats +from .server import Authorizer, OpenNetServer, Peer, ServerStats __all__ = [ + "Authorizer", "DeliveryTimeout", "Flags", "Frame", @@ -28,4 +29,4 @@ "encode_value", ] -__version__ = "0.1.1" +__version__ = "0.2.0" diff --git a/python/src/opennet/server.py b/python/src/opennet/server.py index 51467d5..730765b 100644 --- a/python/src/opennet/server.py +++ b/python/src/opennet/server.py @@ -23,6 +23,7 @@ from .stream import read_frame, write_frame Handler = Callable[["Peer", Frame], Union[None, Awaitable[None]]] +Authorizer = Callable[["Peer", Frame], Union[bool, Awaitable[bool]]] class SocketLike(Protocol): @@ -41,6 +42,19 @@ class Peer: def address(self) -> object: return self.writer.get_extra_info("peername") + @property + def tls_enabled(self) -> bool: + """Return whether this peer is connected through TLS.""" + return self.writer.get_extra_info("ssl_object") is not None + + @property + def tls_peer_certificate(self) -> Optional[dict[str, Any]]: + """Return the verified peer certificate, when the TLS layer provides one.""" + ssl_object = self.writer.get_extra_info("ssl_object") + if ssl_object is None: + return None + return cast(Optional[dict[str, Any]], ssl_object.getpeercert()) + async def send_frame(self, frame: Frame) -> None: await write_frame(self.writer, frame, self.max_payload) @@ -75,6 +89,7 @@ class ServerStats: received_messages: int = 0 duplicate_messages: int = 0 handler_errors: int = 0 + authorization_denials: int = 0 class OpenNetServer: @@ -87,6 +102,7 @@ def __init__( host: str = "127.0.0.1", port: int = 8765, ssl_context: Optional[ssl.SSLContext] = None, + authorizer: Optional[Authorizer] = None, max_payload: int = DEFAULT_MAX_PAYLOAD, max_connections: int = 128, deduplication_window: int = 4096, @@ -103,6 +119,7 @@ def __init__( self.host = host self.port = port self.ssl_context = ssl_context + self.authorizer = authorizer self.max_payload = max_payload self.max_connections = max_connections self.deduplication_window = deduplication_window @@ -176,6 +193,24 @@ async def _accept( if frame.kind is not FrameKind.DATA: continue self.stats.received_messages += 1 + if self.authorizer is not None: + try: + decision = self.authorizer(peer, frame) + if inspect.isawaitable(decision): + decision = await decision + authorized = decision is True + except Exception: + authorized = False + if not authorized: + self.stats.authorization_denials += 1 + await peer.send_frame( + Frame( + FrameKind.ERROR, + ValueType.UTF8, + payload=b"message not authorized", + ) + ) + break duplicate = frame.message_id in seen if duplicate: self.stats.duplicate_messages += 1 diff --git a/python/tests/certs/README.md b/python/tests/certs/README.md new file mode 100644 index 0000000..c985051 --- /dev/null +++ b/python/tests/certs/README.md @@ -0,0 +1,5 @@ +# Test-only TLS certificates + +These certificates and private keys exist only for the loopback mutual-TLS +integration test. They are public fixtures, they identify no real device, and +they must never be reused outside the test suite. diff --git a/python/tests/certs/ca.crt b/python/tests/certs/ca.crt new file mode 100644 index 0000000..5c6f11a --- /dev/null +++ b/python/tests/certs/ca.crt @@ -0,0 +1,19 @@ +-----BEGIN CERTIFICATE----- +MIIDJTCCAg2gAwIBAgIUHlZxOdlfnEPFNqNdwiQNtJKLfQ0wDQYJKoZIhvcNAQEL +BQAwGjEYMBYGA1UEAwwPT3Blbk5ldCBUZXN0IENBMB4XDTI2MDcyOTA2MzExM1oX +DTM2MDcyNjA2MzExM1owGjEYMBYGA1UEAwwPT3Blbk5ldCBUZXN0IENBMIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0wKMfxy32PpC4UA5N7Um+ETcDBqG +UjeWAdQ9D5BK1NqNULQ/g9JZnHRNeYYoGYUxZJrQ9zVMRlKU6s06iGEBtxW6iJ+h +jwdJH7qQZHGkdzfMvxySX/yveTp39dYG5iBsIBf97CmZjW0P2iTYsb+/eDn7S7Ls +KLS3/brwBtNrLGSjh5+JDwCX4Jn6BeclRg1bPUFpzhZxDJ+oqp4ZaWAFc4JzPMW/ +OrU3NO81HS47M9L4aL2xhG6wB9nlQNZpXs7dSs5OBkLM6U/WA0DrlUbzXKbqTds6 +Y9I3GDeyS7eT44tDzlsp5IlUuJ3UHFVUbM8skN1cVuprXcy/QvCHIn9p3QIDAQAB +o2MwYTAdBgNVHQ4EFgQUDMP51NCmT8d3vAaEI9aARKDlwqEwHwYDVR0jBBgwFoAU +DMP51NCmT8d3vAaEI9aARKDlwqEwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8E +BAMCAQYwDQYJKoZIhvcNAQELBQADggEBAB4uKLd4aql7pbnvstvT0PIcm/H4NDSM +/P+yoroTXrOmGjLMabJNtesbdYsa2zXgaDo0aJ/NPNRaHkNaknxNX99Nh71T2Bee +Z046FK68v9D+QXxvXQWN/2rUJtwbkloI5qDroeHnVTKpbfxMOtArrc005ISUsjSM +C/lvOjanu07f/UZ97/dsqatDFPluG5gU8E9IzOPUA1XFtFzqzer/yZAArEcKS5cJ +avUa5o5oQi7j9M9VAC5wQ5Z7qvFsRkmjIdauauyx2a1jmUZGVe0C199XPi/W8UU1 +j6c03aPQ+Os6qn7pVAvugWK0C8j02RgtKmm6yg5eqOhUFuskfe5YWGo= +-----END CERTIFICATE----- diff --git a/python/tests/certs/client.crt b/python/tests/certs/client.crt new file mode 100644 index 0000000..1dd34fa --- /dev/null +++ b/python/tests/certs/client.crt @@ -0,0 +1,20 @@ +-----BEGIN CERTIFICATE----- +MIIDOzCCAiOgAwIBAgIUOBPIKWfVEkcv2sEMfhfm6bQu4hMwDQYJKoZIhvcNAQEL +BQAwGjEYMBYGA1UEAwwPT3Blbk5ldCBUZXN0IENBMB4XDTI2MDcyOTA2MzExNVoX +DTM2MDcyNjA2MzExNVowHjEcMBoGA1UEAwwTb3Blbm5ldC10ZXN0LWNsaWVudDCC +ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOudkU5SgQ6481j6axXL+S+9 +4P6l/3MZzaZjg16JFG2YuYfiU9kAZ3NkRgcrAmj+4u8U7QCb+40hUQiLbhHg0sVp +zB66QKUSPEtkKzH8ZPCdbqEoanGds4ZJnN2lS0d60zjKFIXMuyp3+7C4ofASvq7n +BeF/J4R86dQfF+iedpEB3FdGjJn0iiU/IxPGRBATj8KqpmRe4C8o0TBKIjT2Riez +xtJPAWmZtIKo/FBBpgsBbJpR1zP4ruaMkTatgTRI/ApBUiHQqscJdHm/IPn2SCpO +O58bS2mkFOTdPJXmIa4axtATq3Beq3bm/oR+UXgtB/YSrYUqVsLEFQE6CRotsbsC +AwEAAaN1MHMwDAYDVR0TAQH/BAIwADAOBgNVHQ8BAf8EBAMCB4AwEwYDVR0lBAww +CgYIKwYBBQUHAwIwHQYDVR0OBBYEFN+lgonZX6LNcXuI/MXMlT4Ad25mMB8GA1Ud +IwQYMBaAFAzD+dTQpk/Hd7wGhCPWgESg5cKhMA0GCSqGSIb3DQEBCwUAA4IBAQAv +aj4gxhWGJJ+Vti03n9MxzNRIxCt+AGeC0YERlZJCGrND7Iyd7ATntCINChzgva2I +vvD4skjlnQ5JtHTBSzJcfkTW8Uf/sKg+VriabyaNNBMzx8ktKkGMc+/WEt+6TAcl +HpPF5zunQE55dDZPoL/6Z1A6sTOZJioCKwsNf4Hi89WC0GVOhyVGgTsngPfZ5SvT +fu5BXecY0Rcywxlwo+h8dJXeTBUef4P2KpXN/ClubrH+v1W7I9/m0iC1LmwuV4yw +q2dzl4IucQt++CzOPtWz1k89vm2rZGHG4ghMn1LkMNdNUSnns51nl9X3Z6bmKzjb +2Q6EPqC9pIBoC3wGkZsJ +-----END CERTIFICATE----- diff --git a/python/tests/certs/client.key b/python/tests/certs/client.key new file mode 100644 index 0000000..acac29d --- /dev/null +++ b/python/tests/certs/client.key @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDrnZFOUoEOuPNY ++msVy/kvveD+pf9zGc2mY4NeiRRtmLmH4lPZAGdzZEYHKwJo/uLvFO0Am/uNIVEI +i24R4NLFacweukClEjxLZCsx/GTwnW6hKGpxnbOGSZzdpUtHetM4yhSFzLsqd/uw +uKHwEr6u5wXhfyeEfOnUHxfonnaRAdxXRoyZ9IolPyMTxkQQE4/CqqZkXuAvKNEw +SiI09kYns8bSTwFpmbSCqPxQQaYLAWyaUdcz+K7mjJE2rYE0SPwKQVIh0KrHCXR5 +vyD59kgqTjufG0tppBTk3TyV5iGuGsbQE6twXqt25v6EflF4LQf2Eq2FKlbCxBUB +OgkaLbG7AgMBAAECggEAIvs/FnFSaZ9VEJoGvx2MbtBSsOFwG/ncmhO+mTu3SuRO +G294wnL4aQieCwoRhAmKeN5S/RsA4z3AkbOzJmK0+GFnpbeXycpQ0GKxX8JY3Dk2 +MnrVKZmKjL6uKbbqW9T6ZOfqhHOr5YzF02w+LW4jCHBoDkg6KPCrfiEVrtBOyzgX +ZO3oD90JxrbLIT8wNWbeafpnelhdFTKVSIKTVczVLwJNM18S2zJYY5IncMmG4PAj +DJ8ybbGpW2r9oMOYnsmAiVRPcgG1kqxfga/OohogkWYAXgjH9n6CFyTLEKH9oCPQ +0J9Vz6EbA1rS4yYTDxcMZkvUgImvNlnLU9wV1TrMqQKBgQD6nVVBKNR8keRy5OKk +aX/MB6jOeHQblODEqkZl0s6lZtXx0kkGUyMuQmsIMFnyGT6/XPp/g5FEw5B+Rt4Y +su4caxLSBjih5TaZgmnKUJuzEWDJ5nqi7KXxX2Rzb45ixVYKSf2ojOxc+X8kMCbK +BeCz23jqlwNs04GiashSkIrimQKBgQDwrbjypKaLtAHmk+/t2IdpmwBm5XQQWOs4 +HviZAjPT0PxzhqleRgENfrWcYe9iq2clEfVZkZe5AMJrP4O9UYQTO7C21TcTfPUK +VMqHFly3wpFW3dkLd72CtOqDgUS2ZQFqxYzj21mBeIVM6hBfcBpTqAYLlrBCGkaw +BY2mLGB/cwKBgFYlvyim4GPxwpW8GCxsF4ILH/vZbBAPHTR9U7WswVwZ/XEi1/dU +nMNKEYC/HiwagXdGftVWZJk+oGzH75CN8UvTgqSvfsgoApzCN5tUGjFzx9GfaIiY +0HIoWF9V+Zv6OZlP1eIajyVmnk3SP4MyggtRZj89qe++xNEP48x940ABAoGAWV1H +xOCZ7lmCMylO9xNlBDNHbO1ZsRAzFn15dOa6c7WvZv3jOVvo3CfmKxC6H/rgq4UF +gqJqoYjEp6nHsq3nynRpYxm7/4JwQcvF/26wMpMOACETjAyD01p1kSqYZtkOM6Ty +cVBq5YXoiOyH88feHp3O0NK8026KPKVzOPZVRj0CgYBMavdJGgmFgeQ4BqnKK9U8 +wgCPIAiz0nofvZcTlmYMsMtzNETcO94zc8JI5jLVoIsfDMTbfeEI/jP/pLURjAVe +f1FENlpe3qEZrrqgQcfY6OXV13Ng0pWwgBfvZlO63iqu8z1hYyB6m1S7G6nFh/lo +any4aV5BEykPb30uBR3ogQ== +-----END PRIVATE KEY----- diff --git a/python/tests/certs/server.crt b/python/tests/certs/server.crt new file mode 100644 index 0000000..0ba2565 --- /dev/null +++ b/python/tests/certs/server.crt @@ -0,0 +1,20 @@ +-----BEGIN CERTIFICATE----- +MIIDTzCCAjegAwIBAgIUOBPIKWfVEkcv2sEMfhfm6bQu4hIwDQYJKoZIhvcNAQEL +BQAwGjEYMBYGA1UEAwwPT3Blbk5ldCBUZXN0IENBMB4XDTI2MDcyOTA2MzExNFoX +DTM2MDcyNjA2MzExNFowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG +9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu9SetszEoBB4+LnXCH0kXMHH6CokZi306UzG +bcWvhqphr2qc/ZcZ6+6LiaNCAS4DgMaE6k1ZBlsXv4SjHaAaR7YAYgbLYDlXeE9c +rMNaF6zgkZ0181XarLbuJdbsPrzWg9+fB7JgxGfROG+nNzJ2xZImf5Rpep4QyLhB +VSTWsbOm/VvQX5h4DYoJdEV+TfpkWgDvZZB5ePkA63QI1X0R081Icl2VwWmuXsbt +JZ+CHNtxGbpQnXnWZttG+5mA7655KkZKxtkmafCKej5n/rB7f1qjMbK8MQkneRT4 +1r64iJfKTmPAzPIWtPLzQPFHbma309kHe9FAG/MoJNPaPe7vtwIDAQABo4GSMIGP +MAwGA1UdEwEB/wQCMAAwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF +BwMBMBoGA1UdEQQTMBGCCWxvY2FsaG9zdIcEfwAAATAdBgNVHQ4EFgQUDrT7vLT/ +frdlTwzTIVx/dG5FO9swHwYDVR0jBBgwFoAUDMP51NCmT8d3vAaEI9aARKDlwqEw +DQYJKoZIhvcNAQELBQADggEBAGaK4aeCkE1Gov6e0HdOV33e/+0SuZZUFWRTc1zr +7uMzQkI/2DdWtYDf+3loJlF1BWBOw3+xtnMxOBYptSWSLRA7icjGD0Dp7WSTVQdo +sigTfe6SFnAOha06l0DWxVqy+g2X4xZKvDQB7fm0O/G+xi5ZQOqLuoojskkozyBI +IdZRcaxDKOJemy0QjTVkE9bfP1rKY6/6mWUuteKyLBLee1AG11go1AbuU/vKSvRS +r5aoLzmKPT5JiZTvdJzUrcsxi/wx9I3UGPCQ1Xh36v6t8CcI/YqRBh3KroCDZeaD +tQjQeNddfbvEc/gU3A05mQo/R+u1v7gdBXXCYJlqjqLNuXk= +-----END CERTIFICATE----- diff --git a/python/tests/certs/server.key b/python/tests/certs/server.key new file mode 100644 index 0000000..582268c --- /dev/null +++ b/python/tests/certs/server.key @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC71J62zMSgEHj4 +udcIfSRcwcfoKiRmLfTpTMZtxa+GqmGvapz9lxnr7ouJo0IBLgOAxoTqTVkGWxe/ +hKMdoBpHtgBiBstgOVd4T1ysw1oXrOCRnTXzVdqstu4l1uw+vNaD358HsmDEZ9E4 +b6c3MnbFkiZ/lGl6nhDIuEFVJNaxs6b9W9BfmHgNigl0RX5N+mRaAO9lkHl4+QDr +dAjVfRHTzUhyXZXBaa5exu0ln4Ic23EZulCdedZm20b7mYDvrnkqRkrG2SZp8Ip6 +Pmf+sHt/WqMxsrwxCSd5FPjWvriIl8pOY8DM8ha08vNA8UduZrfT2Qd70UAb8ygk +09o97u+3AgMBAAECggEAKJ1KEGbooV/OeKHeoGRG55OzQT1Tj5CiY4wJYBUd5jWT +V9Xqyl7Q6RY0r/dQP2cOtNq4RN2iwWATG4N3reE/rX/qR0gG0/CyPD6U4HqBUZs9 +F99xmgNXfH81sZLulIZKsEs7XPOyPuXf9J8X/jR4QDJhBrc1s6DYKj7FSb7keZnr +Ff4PikSYqdLbVuEWGAiP20syoANsEP0osCcCv4KVakS/htkR16WQvpmWAM+5cGUR +IyqvnNq9V4Yoy+TJao2SuTmU1QccyIiuWDCypUTsF/donQPk810stSAnyvd/Kx+l +GsfTMyj5YypsKNstC8MBMM408PWtjoSnnhH/9wAOuQKBgQDqr1Dsq5sQ2g/yazl7 +lW1fJk4R+KiCzIyBRSu50ltjGZ8no/hPjeL5HlyySp0PQ0vGHKfi3JhsIjax/kvW +tx19MfOeU2/pSXquKo7FVwcOLMwBsbdLsJ2m5E5ImAiin/oB1fmCCYFPQZ44rAhD +Wx6avYP5VdRUCBdwzLDKu4WURQKBgQDM4+O7nFu42PrIVKQ6MlDLG3jzJ2QRsQIe +i1EXDinQKxoZAmmGbX1X3s2vbBOyPO2PfUPN9UyJkfPKn6lgKoes/kLGfVkOTvzT +rbImPc7xDhA8ubDeFGN65z7gdVpUp81hYij+mLtJn/SCKNIltq3knokflALWk5xO +LKlIinM5ywKBgQDmQVQ4oLXndsumoSUo55cDf+WhJELQ8jf4rREVpBodxQmuLuZ1 +x1Ql73ArTaGDVBeILJ1bq+uwTHE1ebaW19qK9vN7J4npbTp8nyys2ZIGilUuZFre +XE6Ra/aSqvft9t+n5XSD7Sd9x3ehSshgrYn9cGuHuno0aCv3dB0RC6MnsQKBgGpf +kndv6Srh8+mYEB9qygl4/OV4uPp6ZLhI6lPG9axvBEChwPma1K2PLkh6Gwu5mDib +91zxksgFr4WuOPiFfCUzaKW2pErKdXbMwiYahtsdyw5L9eiGu1MuxbUxGDHFYNrD +ZLcxwmKqGf6NhPUxj7yXFmf/py2SO19Wzpir0MDbAoGBALCCSGTOYPBb8a5gzB15 +MNZVCK9UCb/BNpnigTtziyXnCMdA9MGzQdWy5xMPKDfIPxLu7Hid7wSZU3HDxmLk +miZ1pPFXRbM5q9i7tT2SKQOwAteQPiyYrjfRDoxOQBvQThBonqscfueMeuJp43x1 +SihsS3ouvgudpqgd5hJRwKF3 +-----END PRIVATE KEY----- diff --git a/python/tests/test_integration.py b/python/tests/test_integration.py index 89882a3..21a2ce9 100644 --- a/python/tests/test_integration.py +++ b/python/tests/test_integration.py @@ -1,10 +1,14 @@ import asyncio +import ssl +from pathlib import Path import pytest -from opennet import Flags, Frame, FrameKind, OpenNetClient, OpenNetServer, decode_value +from opennet import Flags, Frame, FrameKind, OpenNetClient, OpenNetServer, Peer, decode_value from opennet.stream import read_frame, write_frame +CERTS = Path(__file__).with_name("certs") + async def test_client_server_ack_and_values(): received: asyncio.Queue[tuple[str, object]] = asyncio.Queue() @@ -30,13 +34,24 @@ async def handler(_peer, frame: Frame): async def test_duplicate_data_is_delivered_once_and_acknowledged_twice(): calls = 0 + authorization_calls = 0 ack_count = 0 + def authorizer(_peer, _frame): + nonlocal authorization_calls + authorization_calls += 1 + return True + async def handler(_peer, _frame): nonlocal calls calls += 1 - server = OpenNetServer(handler, host="127.0.0.1", port=0) + server = OpenNetServer( + handler, + host="127.0.0.1", + port=0, + authorizer=authorizer, + ) await server.start() port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] reader, writer = await asyncio.open_connection("127.0.0.1", port) @@ -55,6 +70,7 @@ async def handler(_peer, _frame): assert ack.kind is FrameKind.ACK ack_count += 1 assert calls == 1 + assert authorization_calls == 2 assert ack_count == 2 finally: writer.close() @@ -148,3 +164,178 @@ async def handler(_peer, frame): writer.close() await writer.wait_closed() await server.close() + + +async def test_authorizer_allows_before_handler_and_ack(): + events: list[str] = [] + + async def authorizer(_peer: Peer, frame: Frame) -> bool: + events.append(f"authorize:{frame.topic}") + await asyncio.sleep(0) + return frame.topic.startswith("allowed/") + + async def handler(_peer: Peer, frame: Frame) -> None: + events.append(f"handle:{frame.topic}") + + server = OpenNetServer( + handler, + host="127.0.0.1", + port=0, + authorizer=authorizer, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + try: + async with OpenNetClient("127.0.0.1", port) as client: + await client.send("allowed/temperature", 24.5) + assert events == [ + "authorize:allowed/temperature", + "handle:allowed/temperature", + ] + assert server.stats.authorization_denials == 0 + finally: + await server.close() + + +async def test_authorizer_denial_sends_error_without_handler_or_ack(): + handled = False + + def authorizer(_peer: Peer, _frame: Frame) -> bool: + return False + + async def handler(_peer: Peer, _frame: Frame) -> None: + nonlocal handled + handled = True + + server = OpenNetServer( + handler, + host="127.0.0.1", + port=0, + authorizer=authorizer, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + reader, writer = await asyncio.open_connection("127.0.0.1", port) + try: + await write_frame( + writer, + Frame( + FrameKind.DATA, + topic="private/command", + message_id=1, + flags=Flags.ACK_REQUIRED, + ), + 1024, + ) + response = await asyncio.wait_for(read_frame(reader), 1) + assert response.kind is FrameKind.ERROR + assert response.payload == b"message not authorized" + assert await asyncio.wait_for(reader.read(), 1) == b"" + assert not handled + assert server.stats.authorization_denials == 1 + finally: + writer.close() + await writer.wait_closed() + await server.close() + + +async def test_authorizer_exception_fails_closed(): + def authorizer(_peer: Peer, _frame: Frame) -> bool: + raise RuntimeError("private policy detail") + + server = OpenNetServer( + lambda _peer, _frame: pytest.fail("handler must not run"), + host="127.0.0.1", + port=0, + authorizer=authorizer, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + reader, writer = await asyncio.open_connection("127.0.0.1", port) + try: + await write_frame( + writer, + Frame(FrameKind.DATA, topic="private/command", message_id=1), + 1024, + ) + response = await asyncio.wait_for(read_frame(reader), 1) + assert response.kind is FrameKind.ERROR + assert response.payload == b"message not authorized" + assert b"policy" not in response.payload + assert server.stats.authorization_denials == 1 + finally: + writer.close() + await writer.wait_closed() + await server.close() + + +async def test_plaintext_peer_has_no_tls_certificate(): + observed: asyncio.Future[tuple[bool, object]] = ( + asyncio.get_running_loop().create_future() + ) + + def authorizer(peer: Peer, _frame: Frame) -> bool: + observed.set_result((peer.tls_enabled, peer.tls_peer_certificate)) + return True + + server = OpenNetServer( + lambda _peer, _frame: None, + host="127.0.0.1", + port=0, + authorizer=authorizer, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + try: + async with OpenNetClient("127.0.0.1", port) as client: + await client.send("public/status", "online") + assert await asyncio.wait_for(observed, 1) == (False, None) + finally: + await server.close() + + +async def test_mutual_tls_certificate_can_authorize_a_topic(): + observed_common_names: list[str] = [] + + def authorizer(peer: Peer, frame: Frame) -> bool: + certificate = peer.tls_peer_certificate + assert peer.tls_enabled + assert certificate is not None + subject = { + key: value + for relative_name in certificate["subject"] + for key, value in relative_name + } + common_name = subject.get("commonName", "") + observed_common_names.append(common_name) + return common_name == "opennet-test-client" and frame.topic.startswith( + "device/status" + ) + + server_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + server_context.load_cert_chain(CERTS / "server.crt", CERTS / "server.key") + server_context.load_verify_locations(CERTS / "ca.crt") + server_context.verify_mode = ssl.CERT_REQUIRED + + client_context = ssl.create_default_context(cafile=CERTS / "ca.crt") + client_context.load_cert_chain(CERTS / "client.crt", CERTS / "client.key") + + server = OpenNetServer( + lambda _peer, _frame: None, + host="127.0.0.1", + port=0, + ssl_context=server_context, + authorizer=authorizer, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + try: + async with OpenNetClient( + "127.0.0.1", + port, + ssl_context=client_context, + ) as client: + await client.send("device/status", "online") + assert observed_common_names == ["opennet-test-client"] + finally: + await server.close() From 97a8981a0582659e30db37c8290b6ba7d0b915bb Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 17:38:12 +0800 Subject: [PATCH 02/14] docs: sharpen ONP/1 project scope --- README.md | 15 ++++++-- docs/comparison.md | 14 ++++++-- docs/design-rationale.md | 78 ++++++++++++++++++++++++++++++++++++++++ library.json | 2 +- library.properties | 4 +-- 5 files changed, 105 insertions(+), 8 deletions(-) create mode 100644 docs/design-rationale.md diff --git a/README.md b/README.md index f8e22f8..0f7fc2d 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,10 @@ [![License: MIT](https://img.shields.io/badge/license-MIT-4a638f.svg)](LICENSE) [![Protocol](https://img.shields.io/badge/protocol-ONP%2F1-4a638f.svg)](docs/protocol.md) -OpenNet is a small, dependency-free typed messaging protocol I built for direct -communication between ESP32 devices, Raspberry Pi computers, and backend -services. It sends +OpenNet is a focused, dependency-free messaging protocol I built for direct +links between embedded devices and a Python host—especially an ESP32 talking to +a Raspberry Pi or backend service. It is not a general cross-platform +communication layer or a replacement for MQTT, HTTP, or WebSocket. It sends JSON, text, signed integers, finite doubles, booleans, null, and arbitrary binary data over TCP/TLS or another reliable ordered stream. @@ -28,6 +29,13 @@ stream, or a Python TLS connection. I thought about reliability on that point too, which is why retries keep the same message ID and receivers can suppress recent duplicates without pretending the message is durably stored. +Why make ONP/1 when other protocols already exist? I wanted the framing, typing, +ACK behavior, limits, and cross-language test vectors to stay small enough to +read end to end. MQTT is the stronger choice for brokered fleets and retained +messages; HTTP and WebSocket are stronger when web ecosystems matter. ONP/1 is +for the narrower space where one embedded peer and one host need a direct, +inspectable contract without a broker. + - One small API and one frame format across ESP32 and Python. - Typed values without a JSON dependency for every scalar or binary payload. - Optional application ACKs, retry with stable message IDs, and bounded @@ -108,6 +116,7 @@ device identity matters—outside isolated local testing. - [Measured TCP/TLS and 20-client load results](docs/benchmarks.md) - [Comparison with raw TCP, MQTT, WebSocket, and HTTP](docs/comparison.md) +- [Why ONP/1 exists and what it deliberately does not do](docs/design-rationale.md) - [Architecture, ACK retry, and defensive boundaries](docs/architecture.md) - [Certificate-to-topic authorization](docs/authorization.md) - [Actual GitHub adoption counters and analytics limits](docs/adoption-analytics.md) diff --git a/docs/comparison.md b/docs/comparison.md index 53ee2aa..d51de97 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -1,7 +1,9 @@ # Protocol comparison -OpenNet is a direct typed messaging option, not a universal replacement for -established web or broker protocols. +OpenNet is a direct typed messaging option for embedded-to-host links, not a +universal replacement for established web or broker protocols. Its most natural +shape is an ESP32 or similar embedded peer exchanging small typed messages with +a Raspberry Pi or Python service. | Capability | OpenNet ONP/1 | Raw TCP | MQTT 5 | WebSocket | HTTP | |---|---|---|---|---|---| @@ -33,6 +35,8 @@ represent as one honest number. ## Choosing deliberately +- Choose **nothing new** when an existing protocol already fits the system and + its operational cost is acceptable. - Choose **MQTT** when you need a broker, retained messages, subscriptions, or standardized QoS across a device fleet. - Choose **WebSocket** when browser support is central. @@ -43,6 +47,12 @@ represent as one honest number. - Choose **OpenNet** for direct, dependency-light typed messages shared by ESP32 and Python code, with a small API and documented frame format. +That last choice is intentionally narrow. ONP/1 is useful when learning or +building the communication boundary itself is part of the project: framing, +byte order, validation, retries, duplicate handling, resource limits, and TLS +remain visible rather than disappearing behind a broker. It is not a reason to +replace a working MQTT or HTTP deployment. + Primary specifications: [MQTT 5.0 (OASIS)](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html), [WebSocket RFC 6455](https://www.rfc-editor.org/rfc/rfc6455.html), and [HTTP Semantics RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html). diff --git a/docs/design-rationale.md b/docs/design-rationale.md new file mode 100644 index 0000000..491591b --- /dev/null +++ b/docs/design-rationale.md @@ -0,0 +1,78 @@ +# Why ONP/1 exists + +There are already many good communication protocols. OpenNet does not exist +because MQTT, HTTP, WebSocket, or raw TCP are missing. It exists because I wanted +one small protocol whose complete communication contract could be read, tested, +and discussed across an embedded device and a Python host. + +The concrete use case is intentionally specialized: an ESP32 sends typed sensor +data, status, or commands directly to a Raspberry Pi or backend service, and the +same frame can also travel over another reliable Arduino stream. A project that +needs a broker, browser interoperability, retained state, discovery, or fleet +management should normally choose an established protocol that already owns +those responsibilities. + +## The gap ONP/1 explores + +Raw TCP gives an application an ordered byte stream, but it does not define +message boundaries, types, topics, limits, corruption checks, or application +acknowledgements. A small project has to invent those rules somewhere. + +ONP/1 makes those rules explicit: + +- one fixed 24-byte header; +- seven value types with exact encodings; +- UTF-8 topics and bounded lengths; +- CRC-32 for accidental corruption; +- optional ACKs with stable IDs for retries; +- strict malformed-frame behavior; +- identical test vectors in Python and Arduino. + +This is not the smallest possible byte count. It is a trade: some framing +overhead buys a contract that can be inspected and implemented consistently. + +## Why not just use MQTT? + +MQTT is usually the better answer for brokered publish/subscribe, retained +messages, fleet fan-out, offline sessions, and established operational tooling. +OpenNet deliberately has no broker and no subscription protocol. Its direct +client/server topology is simpler only when the application actually wants a +direct connection. + +## Why not HTTP or WebSocket? + +HTTP is the natural fit for web APIs, proxies, caches, and request/response +integrations. WebSocket is the natural fit for browser-compatible full-duplex +communication. OpenNet has no browser-native API and does not inherit those +ecosystems. + +## Why not stay with raw TCP? + +Raw TCP is appropriate when an application genuinely needs its own framing and +is prepared to specify and test it. ONP/1 is the reusable result of making those +decisions once for a narrow ESP32/Python use case. + +## Learning value + +I think this is where the project can help most honestly: it facilitates focused +learning about communication design. The code keeps byte order, framing, +partial reads and writes, validation, ACK meaning, retry identity, duplicate +suppression, resource bounds, and TLS responsibilities visible. + +That educational value does not prove adoption, hardware reach, performance, or +security. Those claims still require evidence. It only means the repository is +small enough to trace a value from an Arduino call, through ONP/1 bytes, into a +Python handler—and then question every design choice along the way. + +## Deliberate non-goals + +- General cross-platform RPC or service discovery. +- Brokered pub/sub, retained messages, or offline fleet queues. +- Browser-native communication. +- Making unreliable datagrams behave like streams. +- Hiding application authorization or durable-operation design. +- Replacing a protocol that already fits a deployment. + +See the [protocol comparison](comparison.md) for a capability table, the +[transport contract](transports.md) for the byte-stream boundary, and the +[architecture](architecture.md) for how the pieces fit together. diff --git a/library.json b/library.json index f25cc4e..b27d09a 100644 --- a/library.json +++ b/library.json @@ -1,7 +1,7 @@ { "name": "OpenNet", "version": "0.2.0", - "description": "Typed ONP/1 messaging over TCP/TLS and reliable Arduino streams.", + "description": "Direct typed ONP/1 messaging between ESP32 devices and Python hosts.", "keywords": ["esp32", "raspberry-pi", "iot", "tcp", "tls", "bluetooth", "messaging"], "repository": { "type": "git", diff --git a/library.properties b/library.properties index 72812c5..23d4e84 100644 --- a/library.properties +++ b/library.properties @@ -2,8 +2,8 @@ name=OpenNet version=0.2.0 author=devkyato maintainer=devkyato -sentence=Typed ONP/1 messaging over Wi-Fi, Ethernet, Bluetooth serial, and UART streams. -paragraph=OpenNet sends JSON, text, numeric values, and binary data with acknowledgements and partial-write safety over TCP, TLS, and reliable Arduino streams. +sentence=Direct typed ONP/1 messaging between ESP32 devices and Python hosts. +paragraph=OpenNet provides a small inspectable frame format for JSON, text, numeric values, and binary data over TCP, TLS, and reliable Arduino streams. category=Communication url=https://github.com/devkyato/OpenNet architectures=esp32 From 6a840df3cd8c647be70e973949544a5f6370a455 Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 17:39:17 +0800 Subject: [PATCH 03/14] docs: define the reliable stream boundary --- CHANGELOG.md | 2 ++ docs/architecture.md | 2 ++ docs/comparison.md | 2 ++ docs/protocol.md | 5 +++++ docs/transports.md | 37 +++++++++++++++++++++++++++++++++++++ 5 files changed, 48 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 21b2561..0d34d11 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,8 @@ All notable changes are documented here. OpenNet follows - Release verification now includes the native Arduino tests and ESP32 build. - Current install examples and package metadata target v0.2.0. +- Project positioning now emphasizes direct embedded-to-host communication, and + the reliable-stream boundary explicitly documents why raw UDP is unsupported. ### Compatibility diff --git a/docs/architecture.md b/docs/architecture.md index 92e3754..3d25d3a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -5,6 +5,8 @@ must provide a reliable, ordered stream; Wi-Fi itself is not required. I made that separation deliberately. A transport should only have to move bytes; it should not get to redefine what a topic, type, ACK, or message ID means. +The tradeoff is equally deliberate: ONP/1 does not make an unreliable datagram +transport reliable. That responsibility belongs in an adapter below the codec. ```mermaid flowchart LR diff --git a/docs/comparison.md b/docs/comparison.md index d51de97..9be3450 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -44,6 +44,8 @@ represent as one honest number. matter more than a compact persistent channel. - Choose **raw TCP** when you are prepared to design, test, and maintain every application framing and typing rule. +- Choose **UDP** when datagram semantics and application-controlled loss behavior + are requirements; raw UDP cannot be passed directly to ONP/1. - Choose **OpenNet** for direct, dependency-light typed messages shared by ESP32 and Python code, with a small API and documented frame format. diff --git a/docs/protocol.md b/docs/protocol.md index 28faf7c..0299863 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -7,6 +7,11 @@ than a broker protocol: peers establish a reliable byte stream (normally TCP or TLS), exchange frames, and decide at the application layer which topics are allowed. +A transport must preserve every byte in order for the lifetime of the +connection. Datagram loss, reordering, fragmentation, and session semantics are +outside ONP/1. An adapter for a datagram transport must satisfy the stream +contract before passing bytes to this protocol. + ## Byte order and limits All multibyte integers and floating-point values use network byte order (big diff --git a/docs/transports.md b/docs/transports.md index 1279ba9..730ed27 100644 --- a/docs/transports.md +++ b/docs/transports.md @@ -3,6 +3,12 @@ ONP/1 needs a reliable, ordered sequence of bytes. It does not require a specific radio. +That sentence is the transport contract, not just a recommendation. ONP/1 frames +may arrive in many partial reads, but their bytes must arrive once and in order. +TCP, TLS, UART with an appropriate link, USB serial, and Bluetooth Classic SPP +can provide that shape. The quality of reconnects, buffering, and the physical +link still belongs to the chosen transport. + | Connection | v0.2.0 path | Security | Notes | |---|---|---|---| | Wi-Fi/Ethernet/LAN | TCP | TLS recommended | Python and ESP32 | @@ -16,6 +22,37 @@ An internet connection does not make arbitrary devices discoverable or reachable Peers still need an address, route, open firewall, and listening service. Offline radio links work only while both devices are in range. +## What a stream adapter must provide + +An adapter is responsible for: + +- preserving byte order; +- not silently dropping bytes inside a connection; +- exposing partial reads until one complete ONP/1 frame can be assembled; +- completing or clearly failing partial writes; +- defining when a connection or stream session has ended. + +ONP ACKs sit above that contract. They confirm that a complete DATA frame was +parsed and accepted; they do not repair missing bytes in the middle of a stream. +Reconnect policy and replay of unacknowledged application messages remain +application decisions. + +## UDP and other datagram transports + +Raw UDP is not an ONP/1 transport. Datagrams may be lost, duplicated, reordered, +or truncated, and their boundaries are different from a continuous byte stream. +Passing a UDP socket directly to the current codec would violate its assumptions. + +A future datagram adapter would need its own documented layer for packet +fragmentation, reassembly, ordering, loss detection, retransmission, duplicate +handling, size limits, and session identity. That layer could carry unchanged +ONP/1 frame bytes after reassembly, but its behavior would need separate tests +and an extension specification before the repository claimed support. + +Applications can also build that adapter themselves. In that case, OpenNet +begins only after the adapter exposes a reliable ordered stream; the adapter's +reliability and performance are not OpenNet guarantees. + ## Arduino stream use ```cpp From add5409ae6a9e8e18eb03cdd9187c624c0ca284f Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 17:42:01 +0800 Subject: [PATCH 04/14] perf: reuse Arduino receive buffers --- CHANGELOG.md | 2 + README.md | 1 + docs/embedded-memory.md | 95 ++++++++++++++++++++++++++++++++++ docs/releases/v0.2.0.md | 7 +++ docs/tips.md | 5 ++ src/OpenNet.cpp | 9 +++- src/OpenNet.h | 8 +++ test/test_native/test_main.cpp | 14 +++-- 8 files changed, 137 insertions(+), 4 deletions(-) create mode 100644 docs/embedded-memory.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d34d11..479c026 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,8 @@ All notable changes are documented here. OpenNet follows - Current install examples and package metadata target v0.2.0. - Project positioning now emphasizes direct embedded-to-host communication, and the reliable-stream boundary explicitly documents why raw UDP is unsupported. +- Arduino receive handling now moves the validated body buffer into the callback + message instead of allocating and copying a second payload vector. ### Compatibility diff --git a/README.md b/README.md index 0f7fc2d..c39aab5 100644 --- a/README.md +++ b/README.md @@ -121,6 +121,7 @@ device identity matters—outside isolated local testing. - [Certificate-to-topic authorization](docs/authorization.md) - [Actual GitHub adoption counters and analytics limits](docs/adoption-analytics.md) - [Transport support, including Bluetooth boundaries](docs/transports.md) +- [Arduino memory model and payload sizing](docs/embedded-memory.md) Measured loopback results are software baselines, not invented Wi-Fi or hardware claims. Hardware field reports are welcome through the diff --git a/docs/embedded-memory.md b/docs/embedded-memory.md new file mode 100644 index 0000000..d0eae99 --- /dev/null +++ b/docs/embedded-memory.md @@ -0,0 +1,95 @@ +# Embedded memory model + +Memory on the embedded side deserves a concrete explanation because “small API” +does not automatically mean “small object.” OpenNet uses bounded frame content, +but it also uses Arduino `String`, `std::vector`, and `std::function`; their +object sizes, capacity growth, allocator metadata, and fragmentation behavior +depend on the board core and toolchain. + +I do not publish one universal `sizeof(OpenNetClient)` number for that reason. A +native desktop value would not describe an ESP32 build, and even an ESP32 value +would omit the Wi-Fi, Bluetooth, or TLS stack surrounding it. + +## Fixed-width protocol fields + +The Arduino API declares these representations explicitly: + +| Field | Representation | +|---|---:| +| Receive header buffer | 24 bytes | +| `OpenNetFrameKind` | 1 byte | +| `OpenNetValueType` | 1 byte | +| `OpenNetError` | 1 byte | +| Message ID | 4 bytes | +| Flags | 1 byte | +| INT64/FLOAT64 temporary send buffer | 8 bytes | + +Compile-time assertions protect the three one-byte enums. C++ may still insert +padding inside structs, so the table must not be added together and presented as +the total object size. + +## Receive allocation + +The client validates the 24-byte header before allocating a frame body. The body +contains: + +```text +topic bytes + payload bytes +``` + +The topic is bounded to 1,024 bytes. Payload bytes are bounded by the +constructor's `maxPayload`, which defaults to 4,096 bytes. At the default, the +largest accepted body therefore contains 5,120 bytes. Vector capacity and +allocator bookkeeping may be larger than the logical content. + +After CRC and type validation, OpenNet moves that same body allocation into the +`OpenNetMessage` payload used by the callback. It shifts payload bytes over the +topic prefix but does not allocate and copy a second payload vector. After the +callback, the client takes the vector back so its capacity can be reused by the +next frame. + +The topic is still copied into an Arduino `String` for the callback. Calling +`OpenNetMessage::text()` creates another `String` containing a UTF-8 or JSON +payload. The typed scalar accessors read the eight-byte or one-byte payload +without creating a text copy. + +## Send allocation + +Sending does not assemble a complete frame in one heap buffer. The client writes +a 24-byte stack header, the caller's topic bytes, and the caller's payload bytes +in sequence. Partial transport writes are completed before moving to the next +part. Typed integers and doubles use an eight-byte stack buffer. + +## Choosing a payload limit + +Set `maxPayload` to the largest payload the device must actually receive, plus no +speculative margin. For example: + +```cpp +WiFiClient transport; +OpenNetClient client(transport, 512); +``` + +That changes the accepted payload bound to 512 bytes; the protocol topic bound +remains 1,024 bytes. Short application topics reduce real allocations even +though the protocol permits longer ones. + +Also account separately for: + +- Wi-Fi, TCP, TLS, Bluetooth, or serial buffers; +- application objects retained by the callback; +- JSON parsing performed by the application; +- task stacks and other firmware components; +- allocator fragmentation over the intended run time. + +OpenNet v0.2.0 does not provide a no-heap Arduino profile, a fixed caller-owned +receive buffer, or a guarantee that an allocation failure can be recovered on +every Arduino core. Those would be separate API designs with their own tests. + +## Measuring a real firmware + +PlatformIO's build summary reports static RAM and flash for the whole firmware, +not OpenNet alone. Use it as a reproducible whole-image baseline, then measure +free heap and largest free block on the exact board, core, transport, TLS mode, +payload mix, and run duration. Report those conditions with the numbers; do not +generalize one sketch's result to every ESP32 target. diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index 3a45dff..780acb4 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -16,6 +16,8 @@ ONP/1 byte unchanged and adds an optional policy callback to the Python server. - `Peer.tls_peer_certificate` exposes the certificate dictionary supplied by Python's verified TLS layer. - `ServerStats.authorization_denials` counts fail-closed policy decisions. +- Arduino receive callbacks reuse the validated frame allocation for payloads + instead of copying into a second vector. - The release workflow now runs native Arduino tests and the ESP32 build before packaging a tag. @@ -31,6 +33,11 @@ CRC-32, ACK behavior for accepted messages, and Arduino frame bytes are unchanged. Existing Python servers do not need to provide an authorizer, and the Arduino API has no compatibility change in this release. +The enum wire-facing types remain explicitly one byte, with compile-time checks. +The C++ object size itself remains toolchain-dependent; the documented memory +model separates bounded frame bytes from `String`, `vector`, allocator, transport, +and TLS overhead. + ## Install ```sh diff --git a/docs/tips.md b/docs/tips.md index 83d343a..672b15b 100644 --- a/docs/tips.md +++ b/docs/tips.md @@ -5,12 +5,17 @@ in the value, not the topic. - Set the smallest realistic `max_payload`. An ESP32 should not inherit a backend server's multi-megabyte allowance. +- Remember that an Arduino receive buffer can hold `maxPayload + 1024` bytes of + frame body. See the [embedded memory model](embedded-memory.md) before choosing + a limit. - Call Arduino `poll()` every loop iteration. Long `delay()` calls postpone ACKs, pings, and incoming messages. - Retry idempotent operations. For actions such as “unlock door,” include an application operation ID and persist processed IDs. - Use JSON for evolving records, typed numbers for frequent scalar telemetry, and bytes for already encoded data. +- Prefer `asInt`, `asDouble`, and `asBool` when possible. Calling `text()` creates + a new Arduino `String` containing the payload. - Do not base64-encode binary data; ONP carries bytes directly. - Use `opennet benchmark` on the deployment network and report p95/p99, not only the fastest observation. diff --git a/src/OpenNet.cpp b/src/OpenNet.cpp index 81795d1..4b7e6da 100644 --- a/src/OpenNet.cpp +++ b/src/OpenNet.cpp @@ -410,7 +410,13 @@ void OpenNetClient::handleFrame() { for (size_t i = 0; i < expectedTopicLength_; ++i) { message.topic += static_cast(body_[i]); } - message.payload.assign(payload, payload + expectedPayloadLength_); + if (expectedPayloadLength_ > 0) { + memmove(body_.data(), payload, expectedPayloadLength_); + body_.resize(expectedPayloadLength_); + } else { + body_.clear(); + } + message.payload = std::move(body_); if (message.valueType == OpenNetValueType::Float64) { double value = 0; if (!message.asDouble(value)) { @@ -422,6 +428,7 @@ void OpenNetClient::handleFrame() { if (handler_) { handler_(message); } + body_ = std::move(message.payload); if ((message.flags & kAckRequired) != 0) { sendFrame(OpenNetFrameKind::Ack, OpenNetValueType::Null, "", nullptr, 0, message.messageId, 0); diff --git a/src/OpenNet.h b/src/OpenNet.h index ff46928..fc97dd4 100644 --- a/src/OpenNet.h +++ b/src/OpenNet.h @@ -25,6 +25,11 @@ enum class OpenNetValueType : uint8_t { Null = 6, }; +static_assert(sizeof(OpenNetFrameKind) == sizeof(uint8_t), + "OpenNet frame kinds must remain one byte"); +static_assert(sizeof(OpenNetValueType) == sizeof(uint8_t), + "OpenNet value types must remain one byte"); + struct OpenNetMessage { OpenNetFrameKind kind; OpenNetValueType valueType; @@ -51,6 +56,9 @@ enum class OpenNetError : uint8_t { ChecksumMismatch, }; +static_assert(sizeof(OpenNetError) == sizeof(uint8_t), + "OpenNet errors must remain one byte"); + class OpenNetClient { public: using MessageHandler = std::function; diff --git a/test/test_native/test_main.cpp b/test/test_native/test_main.cpp index 7f5304c..853d092 100644 --- a/test/test_native/test_main.cpp +++ b/test/test_native/test_main.cpp @@ -106,9 +106,9 @@ void testIncomingDataIsDeliveredAndAcknowledged() { FakeClient transport; OpenNetClient client(transport); assert(client.connect("127.0.0.1", 8765)); - bool handled = false; + int handled = 0; client.onMessage([&handled](const OpenNetMessage& message) { - handled = true; + ++handled; assert(message.messageId == 9); assert(std::strcmp(message.topic.c_str(), "server") == 0); assert(std::strcmp(message.text().c_str(), "ok") == 0); @@ -118,7 +118,15 @@ void testIncomingDataIsDeliveredAndAcknowledged() { "4f4e0101010100060000000900000002cd83e74a000000007365727665726f6b")); client.poll(); - assert(handled); + assert(handled == 1); + assert(transport.outbound == + fromHex("4f4e01020006000000000009000000000000000000000000")); + + transport.outbound.clear(); + transport.receive(fromHex( + "4f4e0101010100060000000900000002cd83e74a000000007365727665726f6b")); + client.poll(); + assert(handled == 2); assert(transport.outbound == fromHex("4f4e01020006000000000009000000000000000000000000")); } From 19b659e9652b8dda600cd88b46f8ef9aab481340 Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:23:26 +0800 Subject: [PATCH 05/14] docs: define ONP delivery semantics --- CHANGELOG.md | 3 +++ README.md | 5 +++++ docs/architecture.md | 24 +++++++++++++++++++----- docs/protocol.md | 24 ++++++++++++++++++++---- docs/security.md | 2 ++ 5 files changed, 49 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 479c026..35449aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,9 @@ All notable changes are documented here. OpenNet follows the reliable-stream boundary explicitly documents why raw UDP is unsupported. - Arduino receive handling now moves the validated body buffer into the callback message instead of allocating and copying a second payload vector. +- ACK semantics now explicitly mean validated transport acceptance before + handler completion; delivery guarantees and timeout/overload policies are + documented without changing ONP/1 bytes. ### Compatibility diff --git a/README.md b/README.md index c39aab5..3729ee3 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,11 @@ Measured loopback results are software baselines, not invented Wi-Fi or hardware claims. Hardware field reports are welcome through the [compatibility process](docs/compatibility.md). +ACKs mean validated transport acceptance, not successful application side +effects. Duplicate suppression is bounded and connection-local; applications +must use durable operation IDs when replay after reconnect could repeat an +unsafe action. + ## Learn and build - [Tutorials](docs/tutorials.md) diff --git a/docs/architecture.md b/docs/architecture.md index 3d25d3a..ade238c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -42,9 +42,23 @@ sequenceDiagram ``` Python retries reuse the message ID, and the server suppresses duplicate handler -execution within a configurable bounded window. An ACK proves that the peer -parsed and accepted the frame; it does not make delivery durable across power -loss. Durable applications must persist outbound messages and application state. +execution within a configurable bounded window. The server ACKs after validation, +authorization, and duplicate admission, before it dispatches the application +handler. An ACK therefore proves transport acceptance, not handler success or +durability across power loss. Durable applications must persist outbound +messages, operation IDs, and application state. + +## Delivery terms + +- **Accepted at most once in a retained live-connection window:** retrying one + message ID does not run its handler twice while that ID remains in the bounded + window. +- **Not exactly once across reconnects:** message IDs are connection-local and + the in-memory window is lost when the connection ends. +- **Potentially at least once across uncertain failure:** an application that + replays after reconnect may cause the same logical operation again. +- **Application idempotency is required:** durable operation IDs and persisted + results are needed when repeating an action would be unsafe. ## Authorization path @@ -59,8 +73,8 @@ flowchart LR B -->|Yes| C{"Allowed?"} C -->|No| E["Generic ERROR and close"] C -->|Yes| D - D --> F["Application handler"] - F --> G["ACK when requested"] + D --> F["ACK when requested"] + F --> G["Bounded handler dispatch"] ``` The policy can inspect `Peer.tls_enabled`, `Peer.tls_peer_certificate`, and the diff --git a/docs/protocol.md b/docs/protocol.md index 0299863..50120ee 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -51,7 +51,10 @@ as an optional hierarchy separator. | 5 | CLOSE | optional UTF-8 reason | | 6 | ERROR | UTF-8 diagnostic; must not contain secrets | -Unknown frame kinds must produce ERROR and close the connection. A peer receiving +When a receiver can safely trust the frame boundary, an unsupported frame kind +should produce ERROR and close the connection. A receiver may close immediately +when the magic, version, kind, lengths, or other framing fields are invalid +enough that continuing or writing a response would be unsafe. A peer receiving PING should promptly return PONG. ## Flags @@ -60,9 +63,16 @@ PING should promptly return PONG. - Bit 1 (`0x02`), `DUPLICATE`: retransmission of the same message ID. - Bits 2–7 are reserved and must be zero when sending. -ACK confirms that the receiver parsed and accepted the frame, not that application -side effects completed. Message IDs are scoped to one connection. Receivers should -deduplicate repeated IDs for the duration of the connection. +ACK confirms transport-level acceptance. For DATA, acceptance means that the +receiver validated the complete frame, applied configured limits and +authorization, and admitted the message ID to its bounded duplicate window. The +ACK is sent before application handler completion. It does not prove that a +handler started, succeeded, persisted data, or completed a physical side effect. + +Message IDs are scoped to one connection. Receivers should deduplicate repeated +IDs for the duration of the connection. A repeated accepted ID receives an ACK +with `DUPLICATE`; its handler is not run again inside the retained live-connection +window. ## Value types @@ -99,6 +109,12 @@ confidentiality, peer authentication, and tamper resistance. 5. Reconnect with exponential backoff and jitter. Application code decides whether to replay unacknowledged messages. +Implementations should bound idle time, incomplete-header time, incomplete-body +time, TLS handshake time, and writes. These are local transport policies rather +than ONP/1 fields. When an application receive queue or handler dispatcher is +full, a receiver must apply an explicit bounded policy; it must not allow DATA +backpressure to prevent ACK, PING, PONG, CLOSE, or ERROR processing indefinitely. + ## Compatibility Senders must use version 1. A version-1 receiver must ignore non-zero reserved diff --git a/docs/security.md b/docs/security.md index a1df051..d9a98a8 100644 --- a/docs/security.md +++ b/docs/security.md @@ -51,6 +51,8 @@ example is in [Authorizing topics](authorization.md). - Persist operation IDs when duplicate physical actions would be dangerous. - Rate-limit expensive handlers and monitor `ServerStats`. - Keep payload and connection limits appropriate for available RAM and CPU. +- Configure idle, incomplete-frame, TLS-handshake, and write timeouts for the + deployment rather than allowing slots to remain occupied indefinitely. ## Current limits From 3563dff1251f50703f849584ca7524bfed23953f Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:27:37 +0800 Subject: [PATCH 06/14] fix: separate Python control frame dispatch --- CHANGELOG.md | 9 +++ docs/architecture.md | 12 +++ docs/tips.md | 3 + python/src/opennet/__init__.py | 4 + python/src/opennet/client.py | 98 ++++++++++++++++++------ python/src/opennet/protocol.py | 25 ++++++- python/tests/test_integration.py | 125 ++++++++++++++++++++++++++++++- python/tests/test_protocol.py | 6 ++ 8 files changed, 255 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 35449aa..2c33cb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,15 @@ All notable changes are documented here. OpenNet follows handler completion; delivery guarantees and timeout/overload policies are documented without changing ONP/1 bytes. +### Fixed + +- Python PING no longer cycles forever when DATA arrives before PONG. +- A full Python application queue no longer blocks ACK and control processing; + overflow now sends ERROR and closes explicitly. +- Python message-ID wrap skips IDs with pending ACK futures. +- Python JSON decoding rejects non-standard `NaN` and infinity constants. +- Python client writes now have a configurable timeout. + ### Compatibility - ONP/1 framing and typed-value encodings are unchanged. diff --git a/docs/architecture.md b/docs/architecture.md index ade238c..65a6cb9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -60,6 +60,18 @@ messages, operation IDs, and application state. - **Application idempotency is required:** durable operation IDs and persisted results are needed when repeating an action would be unsafe. +## Python client dispatch + +The Python client reader handles ACK, PING, PONG, CLOSE, and ERROR control frames +without waiting for application DATA consumption. DATA enters a bounded queue +with a deliberate overload policy: when the queue is already full, the client +sends a generic `receive queue full` ERROR and closes instead of blocking the +reader and hiding later control frames. + +PING uses dedicated PONG waiters, so DATA received before a PONG remains in the +application queue rather than being repeatedly re-read. Outbound writes have a +configurable timeout, and message-ID allocation skips IDs still waiting for ACK. + ## Authorization path The optional Python authorizer runs after frame validation and before duplicate diff --git a/docs/tips.md b/docs/tips.md index 672b15b..59e69bf 100644 --- a/docs/tips.md +++ b/docs/tips.md @@ -10,6 +10,9 @@ a limit. - Call Arduino `poll()` every loop iteration. Long `delay()` calls postpone ACKs, pings, and incoming messages. +- Size Python `receive_queue_size` for the application's burst behavior and keep + consuming it. Queue overflow deliberately sends ERROR and closes so control + frames never wait forever behind DATA backpressure. - Retry idempotent operations. For actions such as “unlock door,” include an application operation ID and persist processed IDs. - Use JSON for evolving records, typed numbers for frequent scalar telemetry, and diff --git a/python/src/opennet/__init__.py b/python/src/opennet/__init__.py index f37ea9f..078d206 100644 --- a/python/src/opennet/__init__.py +++ b/python/src/opennet/__init__.py @@ -7,6 +7,8 @@ Frame, FrameKind, ProtocolError, + ReceiveQueueFull, + RemoteError, ValueType, decode_value, encode_value, @@ -23,6 +25,8 @@ "OpenNetServer", "Peer", "ProtocolError", + "ReceiveQueueFull", + "RemoteError", "ServerStats", "ValueType", "decode_value", diff --git a/python/src/opennet/client.py b/python/src/opennet/client.py index fecb6e6..2d00e47 100644 --- a/python/src/opennet/client.py +++ b/python/src/opennet/client.py @@ -5,9 +5,8 @@ import asyncio import contextlib import ssl -from collections import deque from types import TracebackType -from typing import Any, AsyncIterator, Deque, Optional, Type +from typing import Any, AsyncIterator, Optional, Type from .protocol import ( DEFAULT_MAX_PAYLOAD, @@ -16,6 +15,9 @@ Frame, FrameKind, ProtocolError, + ReceiveQueueFull, + RemoteError, + ValueType, encode_value, ) from .stream import read_frame, write_frame @@ -33,13 +35,14 @@ def __init__( max_payload: int = DEFAULT_MAX_PAYLOAD, ack_timeout: float = 5.0, connect_timeout: float = 10.0, + write_timeout: float = 5.0, receive_queue_size: int = 256, ) -> None: if not 1 <= port <= 65535: raise ValueError("port must be between 1 and 65535") if max_payload < 0: raise ValueError("max_payload must be non-negative") - if ack_timeout <= 0 or connect_timeout <= 0: + if ack_timeout <= 0 or connect_timeout <= 0 or write_timeout <= 0: raise ValueError("timeouts must be positive") if receive_queue_size <= 0: raise ValueError("receive_queue_size must be positive") @@ -49,12 +52,13 @@ def __init__( self.max_payload = max_payload self.ack_timeout = ack_timeout self.connect_timeout = connect_timeout + self.write_timeout = write_timeout self._reader: Optional[asyncio.StreamReader] = None self._writer: Optional[asyncio.StreamWriter] = None self._next_id = 1 self._pending: dict[int, asyncio.Future[None]] = {} self._messages: asyncio.Queue[Frame] = asyncio.Queue(maxsize=receive_queue_size) - self._backlog: Deque[Frame] = deque() + self._pong_waiters: list[asyncio.Future[None]] = [] self._reader_task: Optional[asyncio.Task[None]] = None self._closed_event = asyncio.Event() self._reader_error: Optional[BaseException] = None @@ -84,7 +88,7 @@ async def close(self) -> None: if writer is None: return with contextlib.suppress(Exception): - await write_frame(writer, Frame(FrameKind.CLOSE), self.max_payload) + await self._write_frame(writer, Frame(FrameKind.CLOSE)) writer.close() with contextlib.suppress(Exception): await writer.wait_closed() @@ -128,7 +132,7 @@ async def send( frame = Frame( FrameKind.DATA, value_type, topic, payload, message_id, flags ) - await write_frame(writer, frame, self.max_payload) + await self._write_frame(writer, frame) if future is None: return message_id try: @@ -148,8 +152,6 @@ async def send( raise AssertionError("unreachable") async def receive(self, timeout: Optional[float] = None) -> Frame: - if self._backlog: - return self._backlog.popleft() if self._closed_event.is_set() and self._messages.empty(): raise self._connection_error() @@ -174,23 +176,21 @@ async def receive(self, timeout: Optional[float] = None) -> Frame: task.cancel() async def messages(self) -> AsyncIterator[Frame]: - while self.connected or self._backlog or not self._messages.empty(): + while self.connected or not self._messages.empty(): yield await self.receive() async def ping(self, timeout: float = 5.0) -> None: if timeout <= 0: raise ValueError("timeout must be positive") writer = self._require_writer() - await write_frame(writer, Frame(FrameKind.PING), self.max_payload) - deadline = asyncio.get_running_loop().time() + timeout - while True: - remaining = deadline - asyncio.get_running_loop().time() - if remaining <= 0: - raise asyncio.TimeoutError - frame = await self.receive(remaining) - if frame.kind is FrameKind.PONG: - return - self._backlog.append(frame) + waiter: asyncio.Future[None] = asyncio.get_running_loop().create_future() + self._pong_waiters.append(waiter) + try: + await self._write_frame(writer, Frame(FrameKind.PING)) + await asyncio.wait_for(asyncio.shield(waiter), timeout) + finally: + if waiter in self._pong_waiters: + self._pong_waiters.remove(waiter) async def __aenter__(self) -> "OpenNetClient": await self.connect() @@ -210,7 +210,12 @@ def _require_writer(self) -> asyncio.StreamWriter: return self._writer def _next_message_id(self) -> int: - message_id = self._next_id + start = self._next_id + message_id = start + while message_id in self._pending: + message_id = 1 if message_id == 0xFFFFFFFF else message_id + 1 + if message_id == start: + raise RuntimeError("all ONP/1 message IDs are pending") self._next_id = 1 if message_id == 0xFFFFFFFF else message_id + 1 return message_id @@ -224,15 +229,46 @@ async def _read_loop(self) -> None: if future is not None and not future.done(): future.set_result(None) elif frame.kind is FrameKind.PING: - await write_frame(self._writer, Frame(FrameKind.PONG), self.max_payload) + await self._write_frame(self._writer, Frame(FrameKind.PONG)) + elif frame.kind is FrameKind.PONG: + if self._pong_waiters: + waiter = self._pong_waiters.pop(0) + if not waiter.done(): + waiter.set_result(None) elif frame.kind is FrameKind.CLOSE: return - else: - await self._messages.put(frame) - except (asyncio.IncompleteReadError, ConnectionError, ProtocolError) as exc: + elif frame.kind is FrameKind.ERROR: + reason = frame.payload.decode("utf-8", errors="replace") + raise RemoteError(reason or "unspecified error") + elif frame.kind is FrameKind.DATA: + try: + self._messages.put_nowait(frame) + except asyncio.QueueFull: + with contextlib.suppress(Exception): + await self._write_frame( + self._writer, + Frame( + FrameKind.ERROR, + value_type=ValueType.UTF8, + payload=b"receive queue full", + ), + ) + raise ReceiveQueueFull( + "OpenNet receive queue is full" + ) from None + except ( + asyncio.IncompleteReadError, + asyncio.TimeoutError, + ConnectionError, + ProtocolError, + ) as exc: self._reader_error = exc self._fail_pending(exc) finally: + if self._reader_error is None: + self._reader_error = ConnectionError("OpenNet peer closed") + self._fail_pending(self._reader_error) + self._fail_pings(self._reader_error) if self._writer is not None: self._writer.close() self._closed_event.set() @@ -248,3 +284,17 @@ def _connection_error(self) -> ConnectionError: error = ConnectionError("OpenNet connection closed") error.__cause__ = self._reader_error return error + + async def _write_frame( + self, writer: asyncio.StreamWriter, frame: Frame + ) -> None: + await asyncio.wait_for( + write_frame(writer, frame, self.max_payload), + timeout=self.write_timeout, + ) + + def _fail_pings(self, exc: BaseException) -> None: + for waiter in self._pong_waiters: + if not waiter.done(): + waiter.set_exception(exc) + self._pong_waiters.clear() diff --git a/python/src/opennet/protocol.py b/python/src/opennet/protocol.py index 10ba6e2..16bf995 100644 --- a/python/src/opennet/protocol.py +++ b/python/src/opennet/protocol.py @@ -33,6 +33,18 @@ def __init__(self, message_id: int, attempts: int) -> None: ) +class ReceiveQueueFull(ConnectionError): + """Raised when DATA backpressure reaches the configured queue bound.""" + + +class RemoteError(ConnectionError): + """Raised when the peer sends an ONP/1 ERROR control frame.""" + + def __init__(self, reason: str) -> None: + self.reason = reason + super().__init__(f"OpenNet peer error: {reason}") + + class FrameKind(IntEnum): DATA = 1 ACK = 2 @@ -268,7 +280,10 @@ def decode_value(value_type: ValueType, payload: bytes) -> Any: if value_type is ValueType.UTF8: return payload.decode("utf-8") if value_type is ValueType.JSON: - return json.loads(payload.decode("utf-8")) + return json.loads( + payload.decode("utf-8"), + parse_constant=_reject_json_constant, + ) if value_type is ValueType.INT64: if len(payload) != 8: raise ProtocolError("INT64 payload must be 8 bytes") @@ -280,6 +295,12 @@ def decode_value(value_type: ValueType, payload: bytes) -> Any: if not math.isfinite(value): raise ProtocolError("FLOAT64 must be finite") return value - except (UnicodeDecodeError, json.JSONDecodeError) as exc: + except ProtocolError: + raise + except (UnicodeDecodeError, ValueError) as exc: raise ProtocolError(str(exc)) from exc raise ProtocolError(f"unsupported value type {value_type!r}") + + +def _reject_json_constant(value: str) -> None: + raise ValueError(f"non-standard JSON constant {value}") diff --git a/python/tests/test_integration.py b/python/tests/test_integration.py index 21a2ce9..cca7a80 100644 --- a/python/tests/test_integration.py +++ b/python/tests/test_integration.py @@ -4,7 +4,17 @@ import pytest -from opennet import Flags, Frame, FrameKind, OpenNetClient, OpenNetServer, Peer, decode_value +from opennet import ( + Flags, + Frame, + FrameKind, + OpenNetClient, + OpenNetServer, + Peer, + ReceiveQueueFull, + ValueType, + decode_value, +) from opennet.stream import read_frame, write_frame CERTS = Path(__file__).with_name("certs") @@ -110,6 +120,119 @@ async def accept(reader, writer): await server.wait_closed() +async def test_ping_dispatches_data_arriving_before_pong(): + async def accept(reader, writer): + ping = await read_frame(reader) + assert ping.kind is FrameKind.PING + data = Frame( + FrameKind.DATA, + ValueType.UTF8, + topic="status/update", + payload=b"ready", + message_id=41, + ) + writer.write(data.to_bytes() + Frame(FrameKind.PONG).to_bytes()) + await writer.drain() + await read_frame(reader) + writer.close() + await writer.wait_closed() + + server = await asyncio.start_server(accept, "127.0.0.1", 0) + port = server.sockets[0].getsockname()[1] + try: + client = OpenNetClient("127.0.0.1", port) + await client.connect() + await client.ping(timeout=1) + data = await client.receive(timeout=1) + assert data.kind is FrameKind.DATA + assert data.topic == "status/update" + assert data.payload == b"ready" + await client.close() + finally: + server.close() + await server.wait_closed() + + +async def test_full_application_queue_does_not_block_ack_dispatch(): + async def accept(reader, writer): + outbound = await read_frame(reader) + queued = Frame( + FrameKind.DATA, + ValueType.UTF8, + topic="queued/data", + payload=b"x", + message_id=99, + ) + ack = Frame(FrameKind.ACK, message_id=outbound.message_id) + writer.write(queued.to_bytes() + ack.to_bytes()) + await writer.drain() + await read_frame(reader) + writer.close() + await writer.wait_closed() + + server = await asyncio.start_server(accept, "127.0.0.1", 0) + port = server.sockets[0].getsockname()[1] + try: + client = OpenNetClient( + "127.0.0.1", + port, + ack_timeout=0.5, + receive_queue_size=1, + ) + await client.connect() + assert await client.send("outbound/data", "value") == 1 + queued = await client.receive(timeout=1) + assert queued.topic == "queued/data" + await client.close() + finally: + server.close() + await server.wait_closed() + + +async def test_receive_queue_overflow_closes_with_explicit_error(): + async def accept(reader, writer): + first = Frame(FrameKind.DATA, topic="queue/one", message_id=1) + second = Frame(FrameKind.DATA, topic="queue/two", message_id=2) + writer.write(first.to_bytes() + second.to_bytes()) + await writer.drain() + response = await read_frame(reader) + assert response.kind is FrameKind.ERROR + assert response.payload == b"receive queue full" + assert await reader.read() == b"" + writer.close() + await writer.wait_closed() + + server = await asyncio.start_server(accept, "127.0.0.1", 0) + port = server.sockets[0].getsockname()[1] + try: + client = OpenNetClient("127.0.0.1", port, receive_queue_size=1) + await client.connect() + first = await client.receive(timeout=1) + assert first.topic == "queue/one" + with pytest.raises(ConnectionError) as result: + await client.receive(timeout=1) + assert isinstance(result.value.__cause__, ReceiveQueueFull) + finally: + server.close() + await server.wait_closed() + + +async def test_message_id_wrap_skips_pending_ids(): + client = OpenNetClient("127.0.0.1") + loop = asyncio.get_running_loop() + maximum = loop.create_future() + first = loop.create_future() + client._pending[0xFFFFFFFF] = maximum + client._pending[1] = first + client._next_id = 0xFFFFFFFF + + assert client._next_message_id() == 2 + assert client._next_id == 3 + + maximum.cancel() + first.cancel() + + async def test_receive_raises_when_remote_closes(): async def accept(_reader, writer): writer.close() diff --git a/python/tests/test_protocol.py b/python/tests/test_protocol.py index b0e12d2..39597df 100644 --- a/python/tests/test_protocol.py +++ b/python/tests/test_protocol.py @@ -150,3 +150,9 @@ def test_received_data_payload_must_match_declared_type(): encoded[5] = int(ValueType.BOOL) with pytest.raises(ProtocolError, match="BOOL"): Frame.from_parts(bytes(encoded[: HEADER.size]), bytes(encoded[HEADER.size :])) + + +@pytest.mark.parametrize("constant", (b"NaN", b"Infinity", b"-Infinity")) +def test_json_rejects_non_standard_numeric_constants(constant): + with pytest.raises(ProtocolError, match="non-standard JSON"): + decode_value(ValueType.JSON, constant) From dff3d3ee9c91f41ff03b786ab62118aa7a20ba9d Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:33:00 +0800 Subject: [PATCH 07/14] fix: keep server protocol progress bounded --- CHANGELOG.md | 5 + docs/architecture.md | 12 +++ docs/releases/v0.2.0.md | 4 +- python/src/opennet/server.py | 118 +++++++++++++++++++--- python/src/opennet/stream.py | 25 ++++- python/tests/test_integration.py | 163 +++++++++++++++++++++++++++++++ 6 files changed, 308 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c33cb9..1999322 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,11 @@ All notable changes are documented here. OpenNet follows - Python message-ID wrap skips IDs with pending ACK futures. - Python JSON decoding rejects non-standard `NaN` and infinity constants. - Python client writes now have a configurable timeout. +- Python servers ACK accepted DATA before bounded handler dispatch, so slow + handlers no longer block PING or turn completed side effects into ACK timeout + ambiguity. +- Python servers bound idle/header/body/TLS/write/handler waits and reject a full + handler set explicitly. ### Compatibility diff --git a/docs/architecture.md b/docs/architecture.md index 65a6cb9..a5e92ec 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -72,6 +72,18 @@ PING uses dedicated PONG waiters, so DATA received before a PONG remains in the application queue rather than being repeatedly re-read. Outbound writes have a configurable timeout, and message-ID allocation skips IDs still waiting for ACK. +The Python server admits new authorized DATA into a bounded per-connection +handler set, sends any requested ACK, and then dispatches the handler. Async +handlers run as bounded tasks; synchronous handlers run off the event loop. +Handler exceptions and timeouts increment counters without retracting an ACK +that has already reported transport acceptance. If the handler set is full, the +server sends a generic `handler queue full` ERROR and closes. + +Server defaults also bound idle waits, incomplete headers, incomplete bodies +with a payload-size allowance, TLS handshakes, writes, and handler duration. +These values are configurable because a serial bridge and a local TCP service +can have very different safe timing envelopes. + ## Authorization path The optional Python authorizer runs after frame validation and before duplicate diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index 780acb4..35e98de 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -24,7 +24,9 @@ ONP/1 byte unchanged and adds an optional policy callback to the Python server. Oh! The ordering is the important part here. The authorizer runs after strict frame validation but before duplicate suppression, the application handler, and the ACK. A denied frame receives a generic ERROR and the connection closes. An -authorizer exception follows the same path without exposing its details. +authorizer exception follows the same path without exposing its details. An +accepted frame is ACKed before bounded handler dispatch; handler completion is +an application result rather than a transport-delivery signal. ## Compatibility diff --git a/python/src/opennet/server.py b/python/src/opennet/server.py index 730765b..467b359 100644 --- a/python/src/opennet/server.py +++ b/python/src/opennet/server.py @@ -37,6 +37,7 @@ class Peer: reader: asyncio.StreamReader writer: asyncio.StreamWriter max_payload: int + write_timeout: float = 5.0 @property def address(self) -> object: @@ -56,7 +57,10 @@ def tls_peer_certificate(self) -> Optional[dict[str, Any]]: return cast(Optional[dict[str, Any]], ssl_object.getpeercert()) async def send_frame(self, frame: Frame) -> None: - await write_frame(self.writer, frame, self.max_payload) + await asyncio.wait_for( + write_frame(self.writer, frame, self.max_payload), + timeout=self.write_timeout, + ) async def send( self, topic: str, payload: bytes, message_id: int, value_type: ValueType @@ -90,6 +94,9 @@ class ServerStats: duplicate_messages: int = 0 handler_errors: int = 0 authorization_denials: int = 0 + overloaded_messages: int = 0 + timed_out_connections: int = 0 + handler_timeouts: int = 0 class OpenNetServer: @@ -106,6 +113,14 @@ def __init__( max_payload: int = DEFAULT_MAX_PAYLOAD, max_connections: int = 128, deduplication_window: int = 4096, + max_handler_tasks: int = 16, + idle_timeout: float = 60.0, + header_timeout: float = 5.0, + body_timeout: float = 5.0, + min_body_bytes_per_second: float = 1024.0, + write_timeout: float = 5.0, + handler_timeout: float = 30.0, + ssl_handshake_timeout: float = 10.0, ) -> None: if not 0 <= port <= 65535: raise ValueError("port must be between 0 and 65535") @@ -115,6 +130,21 @@ def __init__( raise ValueError("max_connections must be positive") if deduplication_window <= 0: raise ValueError("deduplication_window must be positive") + if max_handler_tasks <= 0: + raise ValueError("max_handler_tasks must be positive") + if any( + value <= 0 + for value in ( + idle_timeout, + header_timeout, + body_timeout, + min_body_bytes_per_second, + write_timeout, + handler_timeout, + ssl_handshake_timeout, + ) + ): + raise ValueError("timeouts and minimum body rate must be positive") self.handler = handler self.host = host self.port = port @@ -123,9 +153,18 @@ def __init__( self.max_payload = max_payload self.max_connections = max_connections self.deduplication_window = deduplication_window + self.max_handler_tasks = max_handler_tasks + self.idle_timeout = idle_timeout + self.header_timeout = header_timeout + self.body_timeout = body_timeout + self.min_body_bytes_per_second = min_body_bytes_per_second + self.write_timeout = write_timeout + self.handler_timeout = handler_timeout + self.ssl_handshake_timeout = ssl_handshake_timeout self.stats = ServerStats() self._server: Optional[asyncio.AbstractServer] = None self._peers: set[asyncio.StreamWriter] = set() + self._connection_tasks: set[asyncio.Task[None]] = set() @property def sockets(self) -> list[SocketLike]: @@ -136,7 +175,13 @@ async def start(self) -> None: if self._server is not None: return self._server = await asyncio.start_server( - self._accept, self.host, self.port, ssl=self.ssl_context + self._accept, + self.host, + self.port, + ssl=self.ssl_context, + ssl_handshake_timeout=( + self.ssl_handshake_timeout if self.ssl_context is not None else None + ), ) async def serve_forever(self) -> None: @@ -156,6 +201,10 @@ async def close(self) -> None: await asyncio.gather( *(writer.wait_closed() for writer in writers), return_exceptions=True ) + current = asyncio.current_task() + tasks = [task for task in self._connection_tasks if task is not current] + if tasks: + await asyncio.gather(*tasks, return_exceptions=True) async def __aenter__(self) -> "OpenNetServer": await self.start() @@ -174,17 +223,28 @@ async def _accept( await writer.wait_closed() return self._peers.add(writer) + connection_task = asyncio.current_task() + if connection_task is not None: + self._connection_tasks.add(connection_task) self.stats.accepted_connections += 1 self.stats.active_connections += 1 sock = writer.get_extra_info("socket") if sock is not None: sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) - peer = Peer(reader, writer, self.max_payload) + peer = Peer(reader, writer, self.max_payload, self.write_timeout) seen: set[int] = set() seen_order: Deque[int] = deque() + handler_tasks: set[asyncio.Task[None]] = set() try: while True: - frame = await read_frame(reader, self.max_payload) + frame = await read_frame( + reader, + self.max_payload, + idle_timeout=self.idle_timeout, + header_timeout=self.header_timeout, + body_timeout=self.body_timeout, + min_body_bytes_per_second=self.min_body_bytes_per_second, + ) if frame.kind is FrameKind.CLOSE: break if frame.kind is FrameKind.PING: @@ -215,25 +275,20 @@ async def _accept( if duplicate: self.stats.duplicate_messages += 1 else: - seen.add(frame.message_id) - seen_order.append(frame.message_id) - if len(seen_order) > self.deduplication_window: - seen.discard(seen_order.popleft()) - if not duplicate: - try: - result = self.handler(peer, frame) - if inspect.isawaitable(result): - await result - except Exception: - self.stats.handler_errors += 1 + if len(handler_tasks) >= self.max_handler_tasks: + self.stats.overloaded_messages += 1 await peer.send_frame( Frame( FrameKind.ERROR, ValueType.UTF8, - payload=b"application handler failed", + payload=b"handler queue full", ) ) break + seen.add(frame.message_id) + seen_order.append(frame.message_id) + if len(seen_order) > self.deduplication_window: + seen.discard(seen_order.popleft()) if frame.flags & Flags.ACK_REQUIRED: await peer.send_frame( Frame( @@ -242,11 +297,42 @@ async def _accept( flags=Flags.DUPLICATE if duplicate else Flags.NONE, ) ) + if not duplicate: + task = asyncio.create_task( + self._run_handler(peer, frame), + name=f"opennet-handler-{frame.message_id}", + ) + handler_tasks.add(task) + task.add_done_callback(handler_tasks.discard) + except asyncio.TimeoutError: + self.stats.timed_out_connections += 1 except (asyncio.IncompleteReadError, ConnectionError, ProtocolError): pass finally: + if handler_tasks: + await asyncio.gather(*handler_tasks, return_exceptions=True) self._peers.discard(writer) + if connection_task is not None: + self._connection_tasks.discard(connection_task) self.stats.active_connections -= 1 writer.close() with contextlib.suppress(Exception): await writer.wait_closed() + + async def _run_handler(self, peer: Peer, frame: Frame) -> None: + try: + if inspect.iscoroutinefunction(self.handler): + awaitable_result: Optional[Awaitable[None]] = self.handler(peer, frame) + else: + sync_handler = cast(Callable[[Peer, Frame], object], self.handler) + thread_result = await asyncio.wait_for( + asyncio.to_thread(sync_handler, peer, frame), + timeout=self.handler_timeout, + ) + awaitable_result = cast(Optional[Awaitable[None]], thread_result) + if awaitable_result is not None: + await asyncio.wait_for(awaitable_result, timeout=self.handler_timeout) + except asyncio.TimeoutError: + self.stats.handler_timeouts += 1 + except Exception: + self.stats.handler_errors += 1 diff --git a/python/src/opennet/stream.py b/python/src/opennet/stream.py index 31261b8..ca40b3f 100644 --- a/python/src/opennet/stream.py +++ b/python/src/opennet/stream.py @@ -10,10 +10,31 @@ async def read_frame( reader: asyncio.StreamReader, max_payload: int = DEFAULT_MAX_PAYLOAD, + *, + idle_timeout: float | None = None, + header_timeout: float | None = None, + body_timeout: float | None = None, + min_body_bytes_per_second: float | None = None, ) -> Frame: - header = await reader.readexactly(HEADER_SIZE) + first = await asyncio.wait_for(reader.readexactly(1), timeout=idle_timeout) + remainder = await asyncio.wait_for( + reader.readexactly(HEADER_SIZE - 1), + timeout=header_timeout, + ) + header = first + remainder topic_length, payload_length = inspect_header(header, max_payload) - body = await reader.readexactly(topic_length + payload_length) + body_length = topic_length + payload_length + effective_body_timeout = body_timeout + if ( + effective_body_timeout is not None + and min_body_bytes_per_second is not None + and body_length + ): + effective_body_timeout += body_length / min_body_bytes_per_second + body = await asyncio.wait_for( + reader.readexactly(body_length), + timeout=effective_body_timeout, + ) return Frame.from_parts(header, body, max_payload) diff --git a/python/tests/test_integration.py b/python/tests/test_integration.py index cca7a80..97d41c4 100644 --- a/python/tests/test_integration.py +++ b/python/tests/test_integration.py @@ -1,5 +1,6 @@ import asyncio import ssl +import time from pathlib import Path import pytest @@ -289,6 +290,168 @@ async def handler(_peer, frame): await server.close() +async def test_server_ack_and_ping_progress_before_slow_handler_finishes(): + started = asyncio.Event() + release = asyncio.Event() + + async def handler(_peer: Peer, _frame: Frame) -> None: + started.set() + await release.wait() + + server = OpenNetServer(handler, host="127.0.0.1", port=0) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + try: + async with OpenNetClient( + "127.0.0.1", + port, + ack_timeout=0.1, + ) as client: + assert await client.send("slow/work", "value") == 1 + await asyncio.wait_for(started.wait(), 1) + assert not release.is_set() + await client.ping(timeout=0.2) + release.set() + finally: + release.set() + await server.close() + + +async def test_slow_synchronous_handler_runs_off_event_loop(): + def handler(_peer: Peer, _frame: Frame) -> None: + time.sleep(0.15) + + server = OpenNetServer(handler, host="127.0.0.1", port=0) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + try: + async with OpenNetClient("127.0.0.1", port) as client: + assert await client.send("sync/work", "value") == 1 + await client.ping(timeout=0.1) + finally: + await server.close() + + +async def test_handler_side_effect_then_exception_does_not_revoke_ack(): + side_effect = asyncio.Event() + + async def handler(_peer: Peer, _frame: Frame) -> None: + side_effect.set() + raise RuntimeError("failed after side effect") + + server = OpenNetServer(handler, host="127.0.0.1", port=0) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + try: + async with OpenNetClient("127.0.0.1", port) as client: + assert await client.send("effect/work", "value") == 1 + await asyncio.wait_for(side_effect.wait(), 1) + for _ in range(20): + if server.stats.handler_errors == 1: + break + await asyncio.sleep(0.01) + assert server.stats.handler_errors == 1 + await client.ping(timeout=0.2) + finally: + await server.close() + + +async def test_server_handler_overload_sends_error_and_closes(): + release = asyncio.Event() + + async def handler(_peer: Peer, _frame: Frame) -> None: + await release.wait() + + server = OpenNetServer( + handler, + host="127.0.0.1", + port=0, + max_handler_tasks=1, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + reader, writer = await asyncio.open_connection("127.0.0.1", port) + try: + await write_frame( + writer, + Frame( + FrameKind.DATA, + topic="work/one", + message_id=1, + flags=Flags.ACK_REQUIRED, + ), + 1024, + ) + assert (await asyncio.wait_for(read_frame(reader), 1)).kind is FrameKind.ACK + await write_frame( + writer, + Frame(FrameKind.DATA, topic="work/two", message_id=2), + 1024, + ) + response = await asyncio.wait_for(read_frame(reader), 1) + assert response.kind is FrameKind.ERROR + assert response.payload == b"handler queue full" + assert server.stats.overloaded_messages == 1 + finally: + release.set() + writer.close() + await writer.wait_closed() + await server.close() + + +@pytest.mark.parametrize("partial_frame", (b"", b"O")) +async def test_server_times_out_idle_or_partial_header(partial_frame): + server = OpenNetServer( + lambda _peer, _frame: None, + host="127.0.0.1", + port=0, + idle_timeout=0.05, + header_timeout=0.05, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + reader, writer = await asyncio.open_connection("127.0.0.1", port) + try: + if partial_frame: + writer.write(partial_frame) + await writer.drain() + assert await asyncio.wait_for(reader.read(), 1) == b"" + assert server.stats.timed_out_connections == 1 + finally: + writer.close() + await writer.wait_closed() + await server.close() + + +async def test_server_times_out_partial_body(): + server = OpenNetServer( + lambda _peer, _frame: None, + host="127.0.0.1", + port=0, + body_timeout=0.05, + min_body_bytes_per_second=1_000_000, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + reader, writer = await asyncio.open_connection("127.0.0.1", port) + encoded = Frame( + FrameKind.DATA, + ValueType.UTF8, + topic="slow/body", + payload=b"value", + message_id=1, + ).to_bytes() + try: + writer.write(encoded[:24]) + await writer.drain() + assert await asyncio.wait_for(reader.read(), 1) == b"" + assert server.stats.timed_out_connections == 1 + finally: + writer.close() + await writer.wait_closed() + await server.close() + + async def test_authorizer_allows_before_handler_and_ack(): events: list[str] = [] From a9ee91481cc4b3bc042a1dcaa85436c6cfa185a8 Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:39:56 +0800 Subject: [PATCH 08/14] fix: align cross-language frame validation --- CHANGELOG.md | 2 + docs/protocol.md | 16 +- docs/releases/v0.2.0.md | 4 + python/src/opennet/protocol.py | 28 ++- python/tests/test_protocol.py | 34 ++++ src/OpenNet.cpp | 336 ++++++++++++++++++++++++++++++++- test/test_native/test_main.cpp | 27 +++ test/vectors/onp1_vectors.inc | 9 + 8 files changed, 448 insertions(+), 8 deletions(-) create mode 100644 test/vectors/onp1_vectors.inc diff --git a/CHANGELOG.md b/CHANGELOG.md index 1999322..66a9212 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,6 +41,8 @@ All notable changes are documented here. OpenNet follows ambiguity. - Python servers bound idle/header/body/TLS/write/handler waits and reject a full handler set explicitly. +- Python and Arduino now share malformed UTF-8/JSON conformance vectors, enforce + strict text/control encoding, and bound JSON nesting to 32. ### Compatibility diff --git a/docs/protocol.md b/docs/protocol.md index 50120ee..26e04cd 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -34,7 +34,7 @@ The fixed header is 24 bytes: | 8 | 4 | message ID | unsigned, non-zero for DATA | | 12 | 4 | payload length | application payload bytes | | 16 | 4 | CRC-32 | CRC of topic bytes followed by payload bytes | -| 20 | 4 | reserved | must be zero; ignore when receiving | +| 20 | 4 | reserved | senders MUST emit zero; version-1 receivers MUST ignore | The header is followed by `topic length` bytes and then `payload length` bytes. Topics must be valid UTF-8, between 1 and 1024 bytes for DATA frames, and use `/` @@ -80,7 +80,7 @@ window. |---:|---|---| | 0 | BYTES | uninterpreted bytes | | 1 | UTF8 | valid UTF-8 | -| 2 | JSON | UTF-8 JSON value | +| 2 | JSON | Strict UTF-8 JSON value; maximum nesting depth 32 | | 3 | INT64 | signed 64-bit integer | | 4 | FLOAT64 | IEEE-754 binary64 | | 5 | BOOL | exactly one byte: `0x00` or `0x01` | @@ -89,6 +89,11 @@ window. Kind-specific control payloads use the encoding described in the frame-kind table, regardless of the value-type byte. +UTF-8 validation applies to DATA topics, UTF8 and JSON values, and non-empty +CLOSE and ERROR reasons. JSON uses the RFC 8259 grammar: `NaN`, `Infinity`, +trailing commas, invalid escapes, and lone UTF-16 surrogate escapes are invalid. +Receivers reject JSON nesting deeper than 32 arrays/objects before delivery. + ## CRC CRC-32 is the IEEE polynomial used by zlib. Initialize to zero through the public @@ -117,9 +122,10 @@ backpressure to prevent ACK, PING, PONG, CLOSE, or ERROR processing indefinitely ## Compatibility -Senders must use version 1. A version-1 receiver must ignore non-zero reserved -header bytes for forward compatibility but reject reserved flag bits. New value -types or frame kinds require a protocol revision or an extension specification. +Senders must use version 1 and emit zero in the reserved 32-bit header field. A +version-1 receiver must ignore that field even when it is non-zero; it does not +alter parsing. Reserved flag bits remain invalid. New value types or frame kinds +require a protocol revision or an extension specification. OpenNet v0.2.0 adds an optional Python server authorization callback without changing the ONP/1 frame, kinds, flags, types, or connection lifecycle. A denied diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index 35e98de..2cda9ef 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -40,6 +40,10 @@ The C++ object size itself remains toolchain-dependent; the documented memory model separates bounded frame bytes from `String`, `vector`, allocator, transport, and TLS overhead. +Python and Arduino also consume one shared malformed-frame vector set for UTF-8, +JSON, control text, and typed scalar validation. JSON nesting is bounded to 32 +arrays/objects in both implementations. + ## Install ```sh diff --git a/python/src/opennet/protocol.py b/python/src/opennet/protocol.py index 16bf995..cff9adc 100644 --- a/python/src/opennet/protocol.py +++ b/python/src/opennet/protocol.py @@ -16,6 +16,7 @@ HEADER_SIZE: Final = HEADER.size MAX_TOPIC_SIZE: Final = 1024 DEFAULT_MAX_PAYLOAD: Final = 16 * 1024 * 1024 +MAX_JSON_DEPTH: Final = 32 class ProtocolError(ValueError): @@ -181,6 +182,11 @@ def from_parts( frame = cls(kind, value_type, topic, payload, message_id, Flags(raw_flags)) if kind is FrameKind.DATA: decode_value(value_type, payload) + elif kind in (FrameKind.CLOSE, FrameKind.ERROR) and payload: + try: + payload.decode("utf-8") + except UnicodeDecodeError as exc: + raise ProtocolError(str(exc)) from exc return frame @@ -280,10 +286,12 @@ def decode_value(value_type: ValueType, payload: bytes) -> Any: if value_type is ValueType.UTF8: return payload.decode("utf-8") if value_type is ValueType.JSON: - return json.loads( + value = json.loads( payload.decode("utf-8"), parse_constant=_reject_json_constant, ) + _validate_json_depth(value) + return value if value_type is ValueType.INT64: if len(payload) != 8: raise ProtocolError("INT64 payload must be 8 bytes") @@ -304,3 +312,21 @@ def decode_value(value_type: ValueType, payload: bytes) -> Any: def _reject_json_constant(value: str) -> None: raise ValueError(f"non-standard JSON constant {value}") + + +def _validate_json_depth(value: Any, depth: int = 0) -> None: + if isinstance(value, str): + if any(0xD800 <= ord(character) <= 0xDFFF for character in value): + raise ValueError("JSON string contains a lone surrogate") + return + if isinstance(value, (list, dict)): + depth += 1 + if depth > MAX_JSON_DEPTH: + raise ValueError(f"JSON nesting exceeds {MAX_JSON_DEPTH}") + if isinstance(value, dict): + for key, child in value.items(): + _validate_json_depth(key, depth) + _validate_json_depth(child, depth) + else: + for child in value: + _validate_json_depth(child, depth) diff --git a/python/tests/test_protocol.py b/python/tests/test_protocol.py index 39597df..c5971ea 100644 --- a/python/tests/test_protocol.py +++ b/python/tests/test_protocol.py @@ -1,4 +1,6 @@ +import re import struct +from pathlib import Path import pytest @@ -156,3 +158,35 @@ def test_received_data_payload_must_match_declared_type(): def test_json_rejects_non_standard_numeric_constants(constant): with pytest.raises(ProtocolError, match="non-standard JSON"): decode_value(ValueType.JSON, constant) + + +def test_json_nesting_is_bounded(): + depth_32 = b"[" * 32 + b"0" + b"]" * 32 + depth_33 = b"[" * 33 + b"0" + b"]" * 33 + assert decode_value(ValueType.JSON, depth_32) + with pytest.raises(ProtocolError, match="nesting"): + decode_value(ValueType.JSON, depth_33) + + +def test_shared_python_arduino_conformance_vectors(): + vectors = ( + Path(__file__).resolve().parents[2] + / "test" + / "vectors" + / "onp1_vectors.inc" + ).read_text(encoding="utf-8") + pattern = re.compile( + r'ONP_VECTOR\((\w+), (true|false), "([0-9a-f]+)"\)' + ) + parsed = pattern.findall(vectors) + assert parsed + for name, accepted_text, encoded_hex in parsed: + encoded = bytes.fromhex(encoded_hex) + topic_length, payload_length = inspect_header(encoded[: HEADER.size]) + body = encoded[HEADER.size : HEADER.size + topic_length + payload_length] + if accepted_text == "true": + assert Frame.from_parts(encoded[: HEADER.size], body) + else: + with pytest.raises(ProtocolError, match=".*") as result: + Frame.from_parts(encoded[: HEADER.size], body) + assert result.value, name diff --git a/src/OpenNet.cpp b/src/OpenNet.cpp index 4b7e6da..c15352b 100644 --- a/src/OpenNet.cpp +++ b/src/OpenNet.cpp @@ -4,6 +4,315 @@ #include #include +namespace { + +bool isContinuation(uint8_t byte) { return (byte & 0xC0u) == 0x80u; } + +bool validUtf8(const uint8_t* data, size_t length) { + size_t index = 0; + while (index < length) { + const uint8_t first = data[index++]; + if (first <= 0x7Fu) { + continue; + } + if (first >= 0xC2u && first <= 0xDFu) { + if (index >= length || !isContinuation(data[index++])) { + return false; + } + continue; + } + if (first >= 0xE0u && first <= 0xEFu) { + if (index + 1 >= length) { + return false; + } + const uint8_t second = data[index++]; + const uint8_t third = data[index++]; + if (!isContinuation(third) || + (first == 0xE0u && (second < 0xA0u || second > 0xBFu)) || + (first == 0xEDu && (second < 0x80u || second > 0x9Fu)) || + (first != 0xE0u && first != 0xEDu && + !isContinuation(second))) { + return false; + } + continue; + } + if (first >= 0xF0u && first <= 0xF4u) { + if (index + 2 >= length) { + return false; + } + const uint8_t second = data[index++]; + const uint8_t third = data[index++]; + const uint8_t fourth = data[index++]; + if (!isContinuation(third) || !isContinuation(fourth) || + (first == 0xF0u && (second < 0x90u || second > 0xBFu)) || + (first == 0xF4u && (second < 0x80u || second > 0x8Fu)) || + (first != 0xF0u && first != 0xF4u && + !isContinuation(second))) { + return false; + } + continue; + } + return false; + } + return true; +} + +class JsonValidator { + public: + JsonValidator(const uint8_t* data, size_t length) + : data_(data), length_(length), position_(0) {} + + bool validate() { + skipWhitespace(); + if (!parseValue(0)) { + return false; + } + skipWhitespace(); + return position_ == length_; + } + + private: + static constexpr uint8_t kMaxDepth = 32; + + const uint8_t* data_; + size_t length_; + size_t position_; + + void skipWhitespace() { + while (position_ < length_ && + (data_[position_] == ' ' || data_[position_] == '\t' || + data_[position_] == '\r' || data_[position_] == '\n')) { + ++position_; + } + } + + bool consume(char expected) { + if (position_ >= length_ || + data_[position_] != static_cast(expected)) { + return false; + } + ++position_; + return true; + } + + bool consumeLiteral(const char* literal) { + for (size_t index = 0; literal[index] != '\0'; ++index) { + if (!consume(literal[index])) { + return false; + } + } + return true; + } + + bool parseValue(uint8_t depth) { + if (position_ >= length_) { + return false; + } + switch (data_[position_]) { + case '"': + return parseString(); + case '{': + return parseObject(depth); + case '[': + return parseArray(depth); + case 't': + return consumeLiteral("true"); + case 'f': + return consumeLiteral("false"); + case 'n': + return consumeLiteral("null"); + default: + return parseNumber(); + } + } + + bool parseObject(uint8_t depth) { + if (depth >= kMaxDepth || !consume('{')) { + return false; + } + skipWhitespace(); + if (consume('}')) { + return true; + } + while (true) { + if (!parseString()) { + return false; + } + skipWhitespace(); + if (!consume(':')) { + return false; + } + skipWhitespace(); + if (!parseValue(static_cast(depth + 1))) { + return false; + } + skipWhitespace(); + if (consume('}')) { + return true; + } + if (!consume(',')) { + return false; + } + skipWhitespace(); + } + } + + bool parseArray(uint8_t depth) { + if (depth >= kMaxDepth || !consume('[')) { + return false; + } + skipWhitespace(); + if (consume(']')) { + return true; + } + while (true) { + if (!parseValue(static_cast(depth + 1))) { + return false; + } + skipWhitespace(); + if (consume(']')) { + return true; + } + if (!consume(',')) { + return false; + } + skipWhitespace(); + } + } + + static bool isHex(uint8_t byte) { + return (byte >= '0' && byte <= '9') || + (byte >= 'a' && byte <= 'f') || + (byte >= 'A' && byte <= 'F'); + } + + static uint16_t hexValue(uint8_t byte) { + if (byte >= '0' && byte <= '9') { + return static_cast(byte - '0'); + } + if (byte >= 'a' && byte <= 'f') { + return static_cast(byte - 'a' + 10); + } + return static_cast(byte - 'A' + 10); + } + + bool parseHexEscape(uint16_t& value) { + if (position_ + 4 > length_) { + return false; + } + value = 0; + for (uint8_t index = 0; index < 4; ++index) { + const uint8_t byte = data_[position_++]; + if (!isHex(byte)) { + return false; + } + value = static_cast((value << 4) | hexValue(byte)); + } + return true; + } + + bool parseString() { + if (!consume('"')) { + return false; + } + while (position_ < length_) { + const uint8_t byte = data_[position_++]; + if (byte == '"') { + return true; + } + if (byte < 0x20u) { + return false; + } + if (byte != '\\') { + continue; + } + if (position_ >= length_) { + return false; + } + const uint8_t escape = data_[position_++]; + if (escape == '"' || escape == '\\' || escape == '/' || + escape == 'b' || escape == 'f' || escape == 'n' || + escape == 'r' || escape == 't') { + continue; + } + if (escape != 'u') { + return false; + } + uint16_t code = 0; + if (!parseHexEscape(code)) { + return false; + } + if (code >= 0xD800u && code <= 0xDBFFu) { + if (!consume('\\') || !consume('u')) { + return false; + } + uint16_t low = 0; + if (!parseHexEscape(low) || low < 0xDC00u || low > 0xDFFFu) { + return false; + } + } else if (code >= 0xDC00u && code <= 0xDFFFu) { + return false; + } + } + return false; + } + + bool parseDigits() { + const size_t start = position_; + while (position_ < length_ && data_[position_] >= '0' && + data_[position_] <= '9') { + ++position_; + } + return position_ > start; + } + + bool parseNumber() { + if (position_ < length_ && data_[position_] == '-') { + ++position_; + } + if (position_ >= length_) { + return false; + } + if (data_[position_] == '0') { + ++position_; + if (position_ < length_ && data_[position_] >= '0' && + data_[position_] <= '9') { + return false; + } + } else if (data_[position_] >= '1' && data_[position_] <= '9') { + if (!parseDigits()) { + return false; + } + } else { + return false; + } + if (position_ < length_ && data_[position_] == '.') { + ++position_; + if (!parseDigits()) { + return false; + } + } + if (position_ < length_ && + (data_[position_] == 'e' || data_[position_] == 'E')) { + ++position_; + if (position_ < length_ && + (data_[position_] == '+' || data_[position_] == '-')) { + ++position_; + } + if (!parseDigits()) { + return false; + } + } + return true; + } +}; + +bool validJson(const uint8_t* data, size_t length) { + return validUtf8(data, length) && JsonValidator(data, length).validate(); +} + +} // namespace + String OpenNetMessage::text() const { if (valueType != OpenNetValueType::Utf8 && valueType != OpenNetValueType::Json) { @@ -381,7 +690,30 @@ void OpenNetClient::handleFrame() { } const auto kind = static_cast(header_[3]); + const auto valueType = static_cast(header_[5]); const uint32_t messageId = readU32(header_ + 8); + if (kind == OpenNetFrameKind::Data && + !validUtf8(topicBytes, expectedTopicLength_)) { + lastError_ = OpenNetError::InvalidFrame; + stopTransport(); + return; + } + if ((kind == OpenNetFrameKind::Close || + kind == OpenNetFrameKind::Error || + (kind == OpenNetFrameKind::Data && + valueType == OpenNetValueType::Utf8)) && + !validUtf8(payload, expectedPayloadLength_)) { + lastError_ = OpenNetError::InvalidFrame; + stopTransport(); + return; + } + if (kind == OpenNetFrameKind::Data && + valueType == OpenNetValueType::Json && + !validJson(payload, expectedPayloadLength_)) { + lastError_ = OpenNetError::InvalidFrame; + stopTransport(); + return; + } if (kind == OpenNetFrameKind::Ack) { lastAcknowledged_ = messageId; return; @@ -394,7 +726,7 @@ void OpenNetClient::handleFrame() { if (kind != OpenNetFrameKind::Data) { return; } - if (static_cast(header_[5]) == OpenNetValueType::Bool && + if (valueType == OpenNetValueType::Bool && payload[0] > 1) { lastError_ = OpenNetError::InvalidFrame; stopTransport(); @@ -403,7 +735,7 @@ void OpenNetClient::handleFrame() { OpenNetMessage message; message.kind = kind; - message.valueType = static_cast(header_[5]); + message.valueType = valueType; message.messageId = messageId; message.flags = header_[4]; message.topic.reserve(expectedTopicLength_); diff --git a/test/test_native/test_main.cpp b/test/test_native/test_main.cpp index 853d092..cce278a 100644 --- a/test/test_native/test_main.cpp +++ b/test/test_native/test_main.cpp @@ -10,6 +10,18 @@ namespace { +struct SharedVector { + const char* name; + bool accepted; + const char* hex; +}; + +const SharedVector kSharedVectors[] = { +#define ONP_VECTOR(name, accepted, hex) {#name, accepted, hex}, +#include "../vectors/onp1_vectors.inc" +#undef ONP_VECTOR +}; + std::vector fromHex(const char* hex) { std::vector result; for (std::size_t index = 0; hex[index] != '\0'; index += 2) { @@ -173,6 +185,20 @@ void testMalformedTypedPayloadClosesTransport() { assert(client.lastError() == OpenNetError::InvalidFrame); } +void testSharedConformanceVectors() { + for (const SharedVector& vector : kSharedVectors) { + FakeClient transport; + OpenNetClient client(transport); + assert(client.connect("127.0.0.1", 8765)); + bool handled = false; + client.onMessage([&handled](const OpenNetMessage&) { handled = true; }); + transport.receive(fromHex(vector.hex)); + client.poll(); + assert(client.connected() == vector.accepted); + assert(handled == vector.accepted); + } +} + } // namespace int main() { @@ -183,5 +209,6 @@ int main() { testOversizedDeclarationClosesTransport(); testTypedMessageAccessors(); testMalformedTypedPayloadClosesTransport(); + testSharedConformanceVectors(); return 0; } diff --git a/test/vectors/onp1_vectors.inc b/test/vectors/onp1_vectors.inc new file mode 100644 index 0000000..10442e2 --- /dev/null +++ b/test/vectors/onp1_vectors.inc @@ -0,0 +1,9 @@ +ONP_VECTOR(valid_utf8, true, "4f4e0101000100090000000100000005632f70680000000073656e736f722fc2b5636166c3a9") +ONP_VECTOR(invalid_topic_utf8, false, "4f4e01010001000400000001000000026275ca0e00000000626164ff6f6b") +ONP_VECTOR(invalid_utf8_payload, false, "4f4e01010001000400000001000000048b3c35530000000074657874626164ff") +ONP_VECTOR(valid_json, true, "4f4e01010002000400000001000000162fc307d0000000006a736f6e7b226f6b223a5b747275652c6e756c6c2c332e355d7d") +ONP_VECTOR(json_nan, false, "4f4e0101000200040000000100000003680145ad000000006a736f6e4e614e") +ONP_VECTOR(json_trailing_comma, false, "4f4e010100020004000000010000000832efd4c9000000006a736f6e7b2278223a312c7d") +ONP_VECTOR(json_lone_surrogate, false, "4f4e0101000200040000000100000008d855b5e5000000006a736f6e225c754438303022") +ONP_VECTOR(invalid_error_utf8, false, "4f4e0106000100000000000000000004d5ed00ad00000000626164ff") +ONP_VECTOR(invalid_bool, false, "4f4e01010005000400000001000000012258fdff00000000626f6f6c02") From 5ea655bf1ea1ac0a7aea92489e9f42ed4152e234 Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:50:33 +0800 Subject: [PATCH 09/14] feat: bound Arduino transport work --- CHANGELOG.md | 5 + docs/architecture.md | 6 + docs/embedded-memory.md | 61 +++++- docs/releases/v0.2.0.md | 2 + docs/tips.md | 8 +- src/OpenNet.cpp | 334 ++++++++++++++++++++++++++++----- src/OpenNet.h | 51 ++++- test/test_native/Arduino.h | 8 + test/test_native/test_main.cpp | 126 ++++++++++++- 9 files changed, 535 insertions(+), 66 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 66a9212..b4b3e4e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,6 +43,11 @@ All notable changes are documented here. OpenNet follows handler set explicitly. - Python and Arduino now share malformed UTF-8/JSON conformance vectors, enforce strict text/control encoding, and bound JSON nesting to 32. +- Arduino polling has a configurable byte budget, partial writes have a deadline + and cooperative yield, generic sends enforce typed invariants, and ACK tracking + uses bounded pending/history tables. +- Arduino applications can provide a fixed receive buffer and function-pointer + message view to avoid OpenNet receive-path heap allocation. ### Compatibility diff --git a/docs/architecture.md b/docs/architecture.md index a5e92ec..2aa5d17 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -23,6 +23,12 @@ server. The Arduino implementation accepts either a `Client` (`WiFiClient`, `WiFiClientSecure`) or an already-open Arduino `Stream` (`Serial`, `BluetoothSerial`). The frame bytes are identical across transports. +Arduino `poll()` has a configurable byte budget, partial writes have a total +deadline and cooperative yields, and ACK-required sends use an eight-entry +pending table. A caller-owned receive buffer plus `onMessageView` provides a +no-allocation OpenNet receive path for firmware that does not use the legacy +`String`/vector callback. + BLE GATT is packet-oriented, not a continuous byte stream. It needs an adapter that fragments and reassembles ONP frames; v0.2.0 does not claim direct BLE support. ESP32-C3/S3 boards also do not provide Bluetooth Classic SPP. diff --git a/docs/embedded-memory.md b/docs/embedded-memory.md index d0eae99..9a7aafe 100644 --- a/docs/embedded-memory.md +++ b/docs/embedded-memory.md @@ -42,17 +42,48 @@ constructor's `maxPayload`, which defaults to 4,096 bytes. At the default, the largest accepted body therefore contains 5,120 bytes. Vector capacity and allocator bookkeeping may be larger than the logical content. -After CRC and type validation, OpenNet moves that same body allocation into the -`OpenNetMessage` payload used by the callback. It shifts payload bytes over the -topic prefix but does not allocate and copy a second payload vector. After the -callback, the client takes the vector back so its capacity can be reused by the -next frame. +In the default API, after CRC and type validation OpenNet moves that same body +allocation into the `OpenNetMessage` payload used by the callback. It shifts +payload bytes over the topic prefix but does not allocate and copy a second +payload vector. After the callback, the client takes the vector back so its +capacity can be reused by the next frame. The topic is still copied into an Arduino `String` for the callback. Calling `OpenNetMessage::text()` creates another `String` containing a UTF-8 or JSON payload. The typed scalar accessors read the eight-byte or one-byte payload without creating a text copy. +## Caller-owned receive mode + +For long-running firmware that must avoid receive-path heap allocation, pass a +caller-owned body buffer and register the function-pointer view callback: + +```cpp +uint8_t receiveBody[576]; +WiFiClient transport; +OpenNetClient client(transport, receiveBody, sizeof(receiveBody), 512); + +void onView(const OpenNetMessageView& message, void*) { + // topic and payload are byte spans, not null-terminated strings. + // They remain valid only for this callback. +} + +void setup() { + client.onMessageView(onView); +} +``` + +This path stores the frame body in `receiveBody` and calls a plain function +pointer with topic/payload spans. It does not construct an Arduino `String`, +payload vector, or capturing `std::function` for delivery. The 576-byte example +is only an application choice: it can hold a 64-byte topic plus a 512-byte +payload, not the protocol's maximum 1,024-byte topic. + +If the declared topic plus payload exceeds the supplied capacity, the client +sets `ReceiveBufferTooSmall` and closes the `Client` transport. Registering the +legacy `onMessage` callback as well still creates its convenient `String` and +vector copies. + ## Send allocation Sending does not assemble a complete frame in one heap buffer. The client writes @@ -82,9 +113,23 @@ Also account separately for: - task stacks and other firmware components; - allocator fragmentation over the intended run time. -OpenNet v0.2.0 does not provide a no-heap Arduino profile, a fixed caller-owned -receive buffer, or a guarantee that an allocation failure can be recovered on -every Arduino core. Those would be separate API designs with their own tests. +OpenNet cannot guarantee that allocations elsewhere in the Arduino core, +transport, TLS stack, or application are absent or recoverable. The caller-owned +mode only removes OpenNet's dynamic allocation from the receive-body and callback +delivery path. + +## Control-loop and ACK bounds + +`poll()` processes at most 256 transport bytes per call by default. Pass a +smaller or larger byte budget explicitly when the surrounding control loop has a +different latency target. Outbound writes cooperatively yield between partial +writes and fail after the configured total write deadline; OpenNet cannot +preempt a transport whose individual `write()` call blocks internally. + +The Arduino client permits at most eight ACK-required sends to remain pending. +`TooManyPendingAcks` rejects the next one until an ACK arrives. The most recent +eight acknowledged IDs are retained for `acknowledged(id)`; this is bounded +process memory, not durable delivery state. ## Measuring a real firmware diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index 2cda9ef..9001edc 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -18,6 +18,8 @@ ONP/1 byte unchanged and adds an optional policy callback to the Python server. - `ServerStats.authorization_denials` counts fail-closed policy decisions. - Arduino receive callbacks reuse the validated frame allocation for payloads instead of copying into a second vector. +- Constrained firmware can instead use a caller-owned receive buffer and + `onMessageView`, while poll work, writes, and outstanding ACKs are bounded. - The release workflow now runs native Arduino tests and the ESP32 build before packaging a tag. diff --git a/docs/tips.md b/docs/tips.md index 59e69bf..486d6b9 100644 --- a/docs/tips.md +++ b/docs/tips.md @@ -8,8 +8,9 @@ - Remember that an Arduino receive buffer can hold `maxPayload + 1024` bytes of frame body. See the [embedded memory model](embedded-memory.md) before choosing a limit. -- Call Arduino `poll()` every loop iteration. Long `delay()` calls postpone ACKs, - pings, and incoming messages. +- Call Arduino `poll()` every loop iteration. It processes at most 256 bytes by + default; use `poll(maxBytes)` to tighten that work budget. Long `delay()` calls + still postpone ACKs, pings, and incoming messages. - Size Python `receive_queue_size` for the application's burst behavior and keep consuming it. Queue overflow deliberately sends ERROR and closes so control frames never wait forever behind DATA backpressure. @@ -19,6 +20,9 @@ bytes for already encoded data. - Prefer `asInt`, `asDouble`, and `asBool` when possible. Calling `text()` creates a new Arduino `String` containing the payload. +- Use the caller-owned receive buffer with `onMessageView` when a firmware cannot + accept receive-path heap allocation. View spans are callback-scoped and are + not null-terminated. - Do not base64-encode binary data; ONP carries bytes directly. - Use `opennet benchmark` on the deployment network and report p95/p99, not only the fastest observation. diff --git a/src/OpenNet.cpp b/src/OpenNet.cpp index c15352b..0829155 100644 --- a/src/OpenNet.cpp +++ b/src/OpenNet.cpp @@ -311,6 +311,40 @@ bool validJson(const uint8_t* data, size_t length) { return validUtf8(data, length) && JsonValidator(data, length).validate(); } +bool validTypedPayload(OpenNetValueType type, const uint8_t* payload, + size_t length) { + if (payload == nullptr && length != 0) { + return false; + } + switch (type) { + case OpenNetValueType::Bytes: + return true; + case OpenNetValueType::Utf8: + return validUtf8(payload, length); + case OpenNetValueType::Json: + return validJson(payload, length); + case OpenNetValueType::Int64: + return length == 8; + case OpenNetValueType::Float64: { + if (length != 8) { + return false; + } + uint64_t bits = 0; + for (size_t index = 0; index < 8; ++index) { + bits = (bits << 8) | payload[index]; + } + double value = 0; + memcpy(&value, &bits, sizeof(value)); + return std::isfinite(value); + } + case OpenNetValueType::Bool: + return length == 1 && payload[0] <= 1; + case OpenNetValueType::Null: + return length == 0; + } + return false; +} + } // namespace String OpenNetMessage::text() const { @@ -363,20 +397,48 @@ bool OpenNetMessage::isNull() const { return valueType == OpenNetValueType::Null && payload.empty(); } -OpenNetClient::OpenNetClient(Stream& transport, uint32_t maxPayload) +OpenNetClient::OpenNetClient(Stream& transport, uint32_t maxPayload, + uint32_t writeTimeoutMs) : transport_(transport), clientTransport_(nullptr), maxPayload_(maxPayload), + writeTimeoutMs_(writeTimeoutMs), nextMessageId_(1), lastAcknowledged_(0), + pendingAcks_{}, + pendingAckCount_(0), + acknowledgedIds_{}, + acknowledgedCount_(0), lastError_(OpenNetError::None), + viewHandler_(nullptr), + viewContext_(nullptr), headerReceived_(0), + receiveBuffer_(nullptr), + receiveCapacity_(0), bodyReceived_(0), expectedTopicLength_(0), expectedPayloadLength_(0) {} -OpenNetClient::OpenNetClient(Client& transport, uint32_t maxPayload) - : OpenNetClient(static_cast(transport), maxPayload) { +OpenNetClient::OpenNetClient(Client& transport, uint32_t maxPayload, + uint32_t writeTimeoutMs) + : OpenNetClient(static_cast(transport), maxPayload, + writeTimeoutMs) { + clientTransport_ = &transport; +} + +OpenNetClient::OpenNetClient(Stream& transport, uint8_t* receiveBuffer, + size_t receiveCapacity, uint32_t maxPayload, + uint32_t writeTimeoutMs) + : OpenNetClient(transport, maxPayload, writeTimeoutMs) { + receiveBuffer_ = receiveBuffer; + receiveCapacity_ = receiveCapacity; +} + +OpenNetClient::OpenNetClient(Client& transport, uint8_t* receiveBuffer, + size_t receiveCapacity, uint32_t maxPayload, + uint32_t writeTimeoutMs) + : OpenNetClient(static_cast(transport), receiveBuffer, + receiveCapacity, maxPayload, writeTimeoutMs) { clientTransport_ = &transport; } @@ -387,6 +449,7 @@ bool OpenNetClient::connect(const char* host, uint16_t port) { return false; } resetReceiver(); + resetAcknowledgements(); if (!clientTransport_->connect(host, port)) { lastError_ = OpenNetError::NotConnected; return false; @@ -404,23 +467,26 @@ void OpenNetClient::disconnect(const char* reason) { } stopTransport(); resetReceiver(); + resetAcknowledgements(); } bool OpenNetClient::connected() const { return clientTransport_ == nullptr || clientTransport_->connected(); } -void OpenNetClient::poll() { +void OpenNetClient::poll(size_t maxBytes) { if (!connected()) { return; } - while (transport_.available() > 0) { + size_t processed = 0; + while (transport_.available() > 0 && processed < maxBytes) { if (headerReceived_ < kHeaderSize) { const int byte = transport_.read(); if (byte < 0) { return; } + ++processed; header_[headerReceived_++] = static_cast(byte); if (headerReceived_ == kHeaderSize && !prepareBody()) { stopTransport(); @@ -430,12 +496,14 @@ void OpenNetClient::poll() { } else { const size_t total = static_cast(expectedTopicLength_) + expectedPayloadLength_; - while (bodyReceived_ < total && transport_.available() > 0) { + while (bodyReceived_ < total && transport_.available() > 0 && + processed < maxBytes) { const int byte = transport_.read(); if (byte < 0) { return; } - body_[bodyReceived_++] = static_cast(byte); + ++processed; + bodyData()[bodyReceived_++] = static_cast(byte); } if (bodyReceived_ == total) { handleFrame(); @@ -449,6 +517,11 @@ void OpenNetClient::onMessage(MessageHandler handler) { handler_ = std::move(handler); } +void OpenNetClient::onMessageView(MessageViewHandler handler, void* context) { + viewHandler_ = handler; + viewContext_ = context; +} + OpenNetError OpenNetClient::lastError() const { return lastError_; } uint32_t OpenNetClient::lastAcknowledgedMessage() const { @@ -456,7 +529,15 @@ uint32_t OpenNetClient::lastAcknowledgedMessage() const { } bool OpenNetClient::acknowledged(uint32_t messageId) const { - return messageId != 0 && lastAcknowledged_ == messageId; + if (messageId == 0) { + return false; + } + for (size_t index = 0; index < acknowledgedCount_; ++index) { + if (acknowledgedIds_[index] == messageId) { + return true; + } + } + return false; } uint32_t OpenNetClient::send(const char* topic, OpenNetValueType type, @@ -471,12 +552,23 @@ uint32_t OpenNetClient::send(const char* topic, OpenNetValueType type, lastError_ = OpenNetError::InvalidArgument; return 0; } - const uint32_t id = nextMessageId_; + if (requireAck && pendingAckCount_ >= kAckWindowSize) { + lastError_ = OpenNetError::TooManyPendingAcks; + return 0; + } + uint32_t id = nextMessageId_; + while (isPendingAck(id)) { + id = id == UINT32_MAX ? 1 : id + 1; + } nextMessageId_ = id == UINT32_MAX ? 1 : id + 1; if (!sendFrame(OpenNetFrameKind::Data, type, topic, payload, length, id, requireAck ? kAckRequired : 0)) { return 0; } + if (requireAck && !rememberPendingAck(id)) { + lastError_ = OpenNetError::TooManyPendingAcks; + return 0; + } return id; } @@ -556,10 +648,49 @@ bool OpenNetClient::sendFrame(OpenNetFrameKind kind, OpenNetValueType type, lastError_ = OpenNetError::TopicTooLong; return false; } + if (!validUtf8(reinterpret_cast(safeTopic), topicLength)) { + lastError_ = OpenNetError::InvalidArgument; + return false; + } if (length > maxPayload_ || length > UINT32_MAX) { lastError_ = OpenNetError::PayloadTooLarge; return false; } + if ((flags & ~kAllowedFlags) != 0) { + lastError_ = OpenNetError::InvalidArgument; + return false; + } + if (kind == OpenNetFrameKind::Data) { + if (topicLength == 0 || messageId == 0 || + !validTypedPayload(type, payload, length)) { + lastError_ = OpenNetError::InvalidArgument; + return false; + } + } else if (kind == OpenNetFrameKind::Ack) { + if (topicLength != 0 || messageId == 0 || length != 0 || + type != OpenNetValueType::Null) { + lastError_ = OpenNetError::InvalidArgument; + return false; + } + } else if (kind == OpenNetFrameKind::Ping || + kind == OpenNetFrameKind::Pong) { + if (topicLength != 0 || messageId != 0 || length != 0 || + type != OpenNetValueType::Null || flags != 0) { + lastError_ = OpenNetError::InvalidArgument; + return false; + } + } else if (kind == OpenNetFrameKind::Close || + kind == OpenNetFrameKind::Error) { + if (topicLength != 0 || messageId != 0 || flags != 0 || + (length != 0 && type != OpenNetValueType::Utf8) || + !validUtf8(payload, length)) { + lastError_ = OpenNetError::InvalidArgument; + return false; + } + } else { + lastError_ = OpenNetError::InvalidArgument; + return false; + } uint8_t header[kHeaderSize] = {}; header[0] = 'O'; @@ -579,7 +710,9 @@ bool OpenNetClient::sendFrame(OpenNetFrameKind kind, OpenNetValueType type, if (!writeAll(header, kHeaderSize) || !writeAll(reinterpret_cast(safeTopic), topicLength) || !writeAll(payload, length)) { - lastError_ = OpenNetError::TransportWrite; + if (lastError_ != OpenNetError::WriteTimeout) { + lastError_ = OpenNetError::TransportWrite; + } return false; } lastError_ = OpenNetError::None; @@ -588,12 +721,21 @@ bool OpenNetClient::sendFrame(OpenNetFrameKind kind, OpenNetValueType type, bool OpenNetClient::writeAll(const uint8_t* data, size_t length) { size_t offset = 0; + const uint32_t started = millis(); while (offset < length) { + if (offset > 0 && + static_cast(millis() - started) >= writeTimeoutMs_) { + lastError_ = OpenNetError::WriteTimeout; + return false; + } const size_t written = transport_.write(data + offset, length - offset); if (written == 0 || written > length - offset) { return false; } offset += written; + if (offset < length) { + yield(); + } } return true; } @@ -606,7 +748,7 @@ void OpenNetClient::stopTransport() { void OpenNetClient::resetReceiver() { headerReceived_ = 0; - body_.clear(); + clearBody(); bodyReceived_ = 0; expectedTopicLength_ = 0; expectedPayloadLength_ = 0; @@ -667,9 +809,13 @@ bool OpenNetClient::prepareBody() { lastError_ = OpenNetError::PayloadTooLarge; return false; } - body_.resize(static_cast(expectedTopicLength_) + - expectedPayloadLength_); - if (body_.empty()) { + const size_t bodyLength = + static_cast(expectedTopicLength_) + expectedPayloadLength_; + if (!reserveBody(bodyLength)) { + lastError_ = OpenNetError::ReceiveBufferTooSmall; + return false; + } + if (bodyLength == 0) { handleFrame(); resetReceiver(); } @@ -678,9 +824,11 @@ bool OpenNetClient::prepareBody() { void OpenNetClient::handleFrame() { const uint32_t expectedChecksum = readU32(header_ + 16); - const uint8_t* topicBytes = body_.empty() ? nullptr : body_.data(); + const size_t bodyLength = + static_cast(expectedTopicLength_) + expectedPayloadLength_; + const uint8_t* topicBytes = bodyLength == 0 ? nullptr : bodyData(); const uint8_t* payload = - body_.empty() ? nullptr : body_.data() + expectedTopicLength_; + bodyLength == 0 ? nullptr : bodyData() + expectedTopicLength_; uint32_t actualChecksum = crc32(topicBytes, expectedTopicLength_); actualChecksum = crc32(payload, expectedPayloadLength_, actualChecksum); if (actualChecksum != expectedChecksum) { @@ -699,23 +847,22 @@ void OpenNetClient::handleFrame() { return; } if ((kind == OpenNetFrameKind::Close || - kind == OpenNetFrameKind::Error || - (kind == OpenNetFrameKind::Data && - valueType == OpenNetValueType::Utf8)) && + kind == OpenNetFrameKind::Error) && !validUtf8(payload, expectedPayloadLength_)) { lastError_ = OpenNetError::InvalidFrame; stopTransport(); return; } if (kind == OpenNetFrameKind::Data && - valueType == OpenNetValueType::Json && - !validJson(payload, expectedPayloadLength_)) { + !validTypedPayload(valueType, payload, expectedPayloadLength_)) { lastError_ = OpenNetError::InvalidFrame; stopTransport(); return; } if (kind == OpenNetFrameKind::Ack) { - lastAcknowledged_ = messageId; + if (rememberAcknowledged(messageId)) { + lastAcknowledged_ = messageId; + } return; } if (kind == OpenNetFrameKind::Ping) { @@ -726,45 +873,122 @@ void OpenNetClient::handleFrame() { if (kind != OpenNetFrameKind::Data) { return; } - if (valueType == OpenNetValueType::Bool && - payload[0] > 1) { - lastError_ = OpenNetError::InvalidFrame; - stopTransport(); - return; + if ((header_[4] & kAckRequired) != 0) { + sendFrame(OpenNetFrameKind::Ack, OpenNetValueType::Null, "", nullptr, 0, + messageId, 0); + } + if (viewHandler_ != nullptr) { + const OpenNetMessageView view{ + kind, + valueType, + topicBytes, + expectedTopicLength_, + payload, + expectedPayloadLength_, + messageId, + header_[4], + }; + viewHandler_(view, viewContext_); } + if (handler_) { + OpenNetMessage message; + message.kind = kind; + message.valueType = valueType; + message.messageId = messageId; + message.flags = header_[4]; + message.topic.reserve(expectedTopicLength_); + for (size_t i = 0; i < expectedTopicLength_; ++i) { + message.topic += static_cast(topicBytes[i]); + } + if (receiveBuffer_ != nullptr) { + if (expectedPayloadLength_ > 0) { + message.payload.assign(payload, payload + expectedPayloadLength_); + } + } else { + if (expectedPayloadLength_ > 0) { + memmove(body_.data(), payload, expectedPayloadLength_); + body_.resize(expectedPayloadLength_); + } else { + body_.clear(); + } + message.payload = std::move(body_); + } + handler_(message); + if (receiveBuffer_ == nullptr) { + body_ = std::move(message.payload); + } + } +} - OpenNetMessage message; - message.kind = kind; - message.valueType = valueType; - message.messageId = messageId; - message.flags = header_[4]; - message.topic.reserve(expectedTopicLength_); - for (size_t i = 0; i < expectedTopicLength_; ++i) { - message.topic += static_cast(body_[i]); +uint8_t* OpenNetClient::bodyData() { + return receiveBuffer_ == nullptr ? body_.data() : receiveBuffer_; +} + +const uint8_t* OpenNetClient::bodyData() const { + return receiveBuffer_ == nullptr ? body_.data() : receiveBuffer_; +} + +bool OpenNetClient::reserveBody(size_t length) { + if (receiveBuffer_ != nullptr) { + return length <= receiveCapacity_; } - if (expectedPayloadLength_ > 0) { - memmove(body_.data(), payload, expectedPayloadLength_); - body_.resize(expectedPayloadLength_); - } else { + body_.resize(length); + return true; +} + +void OpenNetClient::clearBody() { + if (receiveBuffer_ == nullptr) { body_.clear(); } - message.payload = std::move(body_); - if (message.valueType == OpenNetValueType::Float64) { - double value = 0; - if (!message.asDouble(value)) { - lastError_ = OpenNetError::InvalidFrame; - stopTransport(); - return; +} + +bool OpenNetClient::isPendingAck(uint32_t messageId) const { + for (size_t index = 0; index < pendingAckCount_; ++index) { + if (pendingAcks_[index] == messageId) { + return true; } } - if (handler_) { - handler_(message); + return false; +} + +bool OpenNetClient::rememberPendingAck(uint32_t messageId) { + if (pendingAckCount_ >= kAckWindowSize) { + return false; } - body_ = std::move(message.payload); - if ((message.flags & kAckRequired) != 0) { - sendFrame(OpenNetFrameKind::Ack, OpenNetValueType::Null, "", nullptr, 0, - message.messageId, 0); + pendingAcks_[pendingAckCount_++] = messageId; + return true; +} + +bool OpenNetClient::rememberAcknowledged(uint32_t messageId) { + bool wasPending = false; + for (size_t index = 0; index < pendingAckCount_; ++index) { + if (pendingAcks_[index] == messageId) { + for (size_t move = index + 1; move < pendingAckCount_; ++move) { + pendingAcks_[move - 1] = pendingAcks_[move]; + } + --pendingAckCount_; + wasPending = true; + break; + } + } + if (!wasPending) { + return false; } + if (acknowledgedCount_ < kAckWindowSize) { + acknowledgedIds_[acknowledgedCount_++] = messageId; + return true; + } + for (size_t index = 1; index < kAckWindowSize; ++index) { + acknowledgedIds_[index - 1] = acknowledgedIds_[index]; + } + acknowledgedIds_[kAckWindowSize - 1] = messageId; + return true; +} + +void OpenNetClient::resetAcknowledgements() { + pendingAckCount_ = 0; + acknowledgedCount_ = 0; + lastAcknowledged_ = 0; } uint16_t OpenNetClient::readU16(const uint8_t* bytes) { @@ -821,6 +1045,12 @@ const char* openNetErrorName(OpenNetError error) { return "invalid frame"; case OpenNetError::ChecksumMismatch: return "checksum mismatch"; + case OpenNetError::WriteTimeout: + return "write timeout"; + case OpenNetError::TooManyPendingAcks: + return "too many pending acknowledgements"; + case OpenNetError::ReceiveBufferTooSmall: + return "receive buffer too small"; } return "unknown error"; } diff --git a/src/OpenNet.h b/src/OpenNet.h index fc97dd4..1829cf6 100644 --- a/src/OpenNet.h +++ b/src/OpenNet.h @@ -45,6 +45,17 @@ struct OpenNetMessage { bool isNull() const; }; +struct OpenNetMessageView { + OpenNetFrameKind kind; + OpenNetValueType valueType; + const uint8_t* topic; + size_t topicLength; + const uint8_t* payload; + size_t payloadLength; + uint32_t messageId; + uint8_t flags; +}; + enum class OpenNetError : uint8_t { None, NotConnected, @@ -54,6 +65,9 @@ enum class OpenNetError : uint8_t { TransportWrite, InvalidFrame, ChecksumMismatch, + WriteTimeout, + TooManyPendingAcks, + ReceiveBufferTooSmall, }; static_assert(sizeof(OpenNetError) == sizeof(uint8_t), @@ -62,23 +76,37 @@ static_assert(sizeof(OpenNetError) == sizeof(uint8_t), class OpenNetClient { public: using MessageHandler = std::function; + using MessageViewHandler = void (*)(const OpenNetMessageView&, void*); static constexpr uint32_t kDefaultMaxPayload = 4096; static constexpr uint16_t kMaxTopicLength = 1024; + static constexpr size_t kDefaultPollBytes = 256; + static constexpr size_t kAckWindowSize = 8; explicit OpenNetClient(Client& transport, - uint32_t maxPayload = kDefaultMaxPayload); + uint32_t maxPayload = kDefaultMaxPayload, + uint32_t writeTimeoutMs = 1000); explicit OpenNetClient(Stream& transport, - uint32_t maxPayload = kDefaultMaxPayload); + uint32_t maxPayload = kDefaultMaxPayload, + uint32_t writeTimeoutMs = 1000); + OpenNetClient(Client& transport, uint8_t* receiveBuffer, + size_t receiveCapacity, + uint32_t maxPayload = kDefaultMaxPayload, + uint32_t writeTimeoutMs = 1000); + OpenNetClient(Stream& transport, uint8_t* receiveBuffer, + size_t receiveCapacity, + uint32_t maxPayload = kDefaultMaxPayload, + uint32_t writeTimeoutMs = 1000); // Stream transports such as Serial and BluetoothSerial are opened by their // caller; connect() is available for Client transports such as WiFiClient. bool connect(const char* host, uint16_t port); void disconnect(const char* reason = nullptr); bool connected() const; - void poll(); + void poll(size_t maxBytes = kDefaultPollBytes); void onMessage(MessageHandler handler); + void onMessageView(MessageViewHandler handler, void* context = nullptr); OpenNetError lastError() const; uint32_t lastAcknowledgedMessage() const; bool acknowledged(uint32_t messageId) const; @@ -107,14 +135,23 @@ class OpenNetClient { Stream& transport_; Client* clientTransport_; uint32_t maxPayload_; + uint32_t writeTimeoutMs_; uint32_t nextMessageId_; uint32_t lastAcknowledged_; + uint32_t pendingAcks_[kAckWindowSize]; + size_t pendingAckCount_; + uint32_t acknowledgedIds_[kAckWindowSize]; + size_t acknowledgedCount_; OpenNetError lastError_; MessageHandler handler_; + MessageViewHandler viewHandler_; + void* viewContext_; uint8_t header_[kHeaderSize]; size_t headerReceived_; std::vector body_; + uint8_t* receiveBuffer_; + size_t receiveCapacity_; size_t bodyReceived_; uint16_t expectedTopicLength_; uint32_t expectedPayloadLength_; @@ -127,6 +164,14 @@ class OpenNetClient { void resetReceiver(); bool prepareBody(); void handleFrame(); + uint8_t* bodyData(); + const uint8_t* bodyData() const; + bool reserveBody(size_t length); + void clearBody(); + bool isPendingAck(uint32_t messageId) const; + bool rememberPendingAck(uint32_t messageId); + bool rememberAcknowledged(uint32_t messageId); + void resetAcknowledgements(); static uint16_t readU16(const uint8_t* bytes); static uint32_t readU32(const uint8_t* bytes); diff --git a/test/test_native/Arduino.h b/test/test_native/Arduino.h index b294fe3..2265905 100644 --- a/test/test_native/Arduino.h +++ b/test/test_native/Arduino.h @@ -4,6 +4,14 @@ #include #include +inline unsigned long& fakeArduinoMillis() { + static unsigned long value = 0; + return value; +} + +inline unsigned long millis() { return fakeArduinoMillis(); } +inline void yield() { ++fakeArduinoMillis(); } + class Stream { public: virtual ~Stream() = default; diff --git a/test/test_native/test_main.cpp b/test/test_native/test_main.cpp index cce278a..cafd4ff 100644 --- a/test/test_native/test_main.cpp +++ b/test/test_native/test_main.cpp @@ -100,15 +100,26 @@ void testPartialTransportWritesAreCompleted() { assert(std::memcmp(transport.outbound.data() + 24, "statusonline", 12) == 0); } +void testWriteBudgetStopsTinyProgress() { + FakeClient transport; + transport.maxWrite = 1; + OpenNetClient client(transport, OpenNetClient::kDefaultMaxPayload, 3); + assert(client.connect("127.0.0.1", 8765)); + assert(client.sendText("status", String("online")) == 0); + assert(client.lastError() == OpenNetError::WriteTimeout); +} + void testFragmentedAckIsRecognized() { FakeClient transport; OpenNetClient client(transport); assert(client.connect("127.0.0.1", 8765)); + assert(client.sendNull("pending") == 1); + transport.outbound.clear(); transport.receive( fromHex("4f4e01020006000000000001000000000000000000000000")); while (transport.available() > 0) { - client.poll(); + client.poll(1); } assert(client.lastAcknowledgedMessage() == 1); assert(client.lastError() == OpenNetError::None); @@ -143,6 +154,22 @@ void testIncomingDataIsDeliveredAndAcknowledged() { fromHex("4f4e01020006000000000009000000000000000000000000")); } +void testPollByteBudgetYieldsBeforeWholeFrame() { + FakeClient transport; + OpenNetClient client(transport); + assert(client.connect("127.0.0.1", 8765)); + bool handled = false; + client.onMessage([&handled](const OpenNetMessage&) { handled = true; }); + transport.receive(fromHex( + "4f4e0101010100060000000900000002cd83e74a000000007365727665726f6b")); + + client.poll(24); + assert(!handled); + assert(transport.available() == 8); + client.poll(); + assert(handled); +} + void testOversizedDeclarationClosesTransport() { FakeClient transport; OpenNetClient client(transport, 16); @@ -174,6 +201,97 @@ void testTypedMessageAccessors() { assert(message.isNull()); } +void testOutboundTypedPayloadsAreValidated() { + FakeClient transport; + OpenNetClient client(transport); + assert(client.connect("127.0.0.1", 8765)); + const uint8_t invalidBool = 2; + const uint8_t invalidNull = 1; + const uint8_t invalidUtf8[] = {0xFF}; + const uint8_t nonFinite[] = {0x7F, 0xF8, 0, 0, 0, 0, 0, 0}; + + assert(client.send("value", OpenNetValueType::Bool, &invalidBool, 1) == 0); + assert(client.send("value", OpenNetValueType::Null, &invalidNull, 1) == 0); + assert(client.send("value", OpenNetValueType::Utf8, invalidUtf8, + sizeof(invalidUtf8)) == 0); + assert(client.send("value", OpenNetValueType::Float64, nonFinite, + sizeof(nonFinite)) == 0); + assert(client.sendJson("value", String("{\"x\":1,}")) == 0); + assert(transport.outbound.empty()); +} + +void testJsonNestingIsBoundedOnSend() { + FakeClient transport; + OpenNetClient client(transport); + assert(client.connect("127.0.0.1", 8765)); + const std::string valid = + std::string(32, '[') + "0" + std::string(32, ']'); + const std::string invalid = + std::string(33, '[') + "0" + std::string(33, ']'); + assert(client.sendJson("json", String(valid.c_str()), false) == 1); + assert(client.sendJson("json", String(invalid.c_str()), false) == 0); +} + +void testBoundedPendingAndAcknowledgementHistory() { + FakeClient transport; + OpenNetClient client(transport); + assert(client.connect("127.0.0.1", 8765)); + for (uint32_t id = 1; id <= OpenNetClient::kAckWindowSize; ++id) { + assert(client.sendNull("pending") == id); + } + assert(client.sendNull("pending") == 0); + assert(client.lastError() == OpenNetError::TooManyPendingAcks); + + transport.receive( + fromHex("4f4e010200060000000003e7000000000000000000000000")); + transport.receive( + fromHex("4f4e01020006000000000001000000000000000000000000")); + transport.receive( + fromHex("4f4e01020006000000000002000000000000000000000000")); + client.poll(); + assert(!client.acknowledged(999)); + assert(client.acknowledged(1)); + assert(client.acknowledged(2)); + assert(client.sendNull("pending") == 9); +} + +struct ViewObservation { + bool handled = false; +}; + +void observeMessageView(const OpenNetMessageView& message, void* context) { + auto* observation = static_cast(context); + observation->handled = true; + assert(message.topicLength == 6); + assert(std::memcmp(message.topic, "server", 6) == 0); + assert(message.payloadLength == 2); + assert(std::memcmp(message.payload, "ok", 2) == 0); +} + +void testCallerOwnedReceiveBufferAndView() { + FakeClient transport; + uint8_t receiveBuffer[8] = {}; + OpenNetClient client(transport, receiveBuffer, sizeof(receiveBuffer), 16); + assert(client.connect("127.0.0.1", 8765)); + ViewObservation observation; + client.onMessageView(observeMessageView, &observation); + transport.receive(fromHex( + "4f4e0101010100060000000900000002cd83e74a000000007365727665726f6b")); + client.poll(); + assert(observation.handled); + + FakeClient smallTransport; + uint8_t smallBuffer[7] = {}; + OpenNetClient smallClient(smallTransport, smallBuffer, sizeof(smallBuffer), + 16); + assert(smallClient.connect("127.0.0.1", 8765)); + smallTransport.receive(fromHex( + "4f4e0101010100060000000900000002cd83e74a000000007365727665726f6b")); + smallClient.poll(); + assert(!smallClient.connected()); + assert(smallClient.lastError() == OpenNetError::ReceiveBufferTooSmall); +} + void testMalformedTypedPayloadClosesTransport() { FakeClient transport; OpenNetClient client(transport); @@ -204,10 +322,16 @@ void testSharedConformanceVectors() { int main() { testSendProducesConformantFrame(); testPartialTransportWritesAreCompleted(); + testWriteBudgetStopsTinyProgress(); testFragmentedAckIsRecognized(); testIncomingDataIsDeliveredAndAcknowledged(); + testPollByteBudgetYieldsBeforeWholeFrame(); testOversizedDeclarationClosesTransport(); testTypedMessageAccessors(); + testOutboundTypedPayloadsAreValidated(); + testJsonNestingIsBoundedOnSend(); + testBoundedPendingAndAcknowledgementHistory(); + testCallerOwnedReceiveBufferAndView(); testMalformedTypedPayloadClosesTransport(); testSharedConformanceVectors(); return 0; From ffa20f813ac314893cbee91c6ff3bdfb9b4dd582 Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:52:59 +0800 Subject: [PATCH 10/14] fix: harden CLI data handling --- CHANGELOG.md | 3 ++ docs/cli.md | 19 ++++++++--- docs/releases/v0.2.0.md | 3 ++ python/src/opennet/cli.py | 63 ++++++++++++++++++++++++++++------ python/tests/test_cli.py | 71 ++++++++++++++++++++++++++++++++++++++- 5 files changed, 143 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b4b3e4e..56be18d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,6 +48,9 @@ All notable changes are documented here. OpenNet follows uses bounded pending/history tables. - Arduino applications can provide a fixed receive buffer and function-pointer message view to avoid OpenNet receive-path heap allocation. +- The CLI rejects oversized files before reading them, logs payload metadata + rather than values by default, offers a shared topic-prefix allowlist, and + labels its benchmark as sequential acknowledged round trips. ### Compatibility diff --git a/docs/cli.md b/docs/cli.md index 5f3fe3f..51f2564 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -10,11 +10,15 @@ opennet serve [--host HOST] [--port PORT] [--echo] Useful controls include `--max-payload`, `--max-connections`, `--deduplication-window`, `--tls-cert`, `--tls-key`, and `--tls-client-ca`. -Remote plaintext listeners require `--allow-plaintext`. +Remote plaintext listeners require `--allow-plaintext`. Logs contain message +metadata and payload size by default; `--log-values` is an explicit +secret-exposure opt-in. -The general-purpose CLI server logs or echoes messages; it does not accept a -custom topic policy on the command line. Use the Python `OpenNetServer` -`authorizer` API when certificate-to-topic rules are required. +Repeat `--allow-topic-prefix PREFIX` to apply one shared allowlist to every +client. For example, `--allow-topic-prefix sensor/ --allow-topic-prefix status/` +rejects other topics before the handler and ACK. This simple CLI policy does not +map individual certificates to different topics. Use the Python +`OpenNetServer` `authorizer` API for identity-specific rules. ## Send @@ -25,6 +29,8 @@ opennet send TOPIC [VALUE] [--type TYPE] Types are `text`, `json`, `int`, `float`, `bool`, `null`, `hex`, and `file`. The command waits for an ACK by default and retries twice. Adjust with `--retries`, `--retry-delay`, `--ack-timeout`, or `--no-ack`. +For `--type file`, the CLI checks the file size against `--max-payload` before +reading it. Examples: @@ -43,7 +49,10 @@ opennet benchmark --count 1000 --warmup 25 --payload-size 1024 --json ``` Both commands measure complete ONP round trips rather than ICMP. `benchmark` -uses acknowledged binary DATA frames. +uses one sequential acknowledged binary DATA frame at a time. Its +`messages_per_second` value is the reciprocal of those sampled round trips, not +maximum concurrent throughput; JSON output labels the mode +`sequential_acknowledged`. ## Connection and TLS options diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index 9001edc..a8f3f77 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -20,6 +20,9 @@ ONP/1 byte unchanged and adds an optional policy callback to the Python server. instead of copying into a second vector. - Constrained firmware can instead use a caller-owned receive buffer and `onMessageView`, while poll work, writes, and outstanding ACKs are bounded. +- CLI file sends are size-checked before reading, server logs hide values by + default, and repeated `--allow-topic-prefix` options provide a simple shared + allowlist. - The release workflow now runs native Arduino tests and the ESP32 build before packaging a tag. diff --git a/python/src/opennet/cli.py b/python/src/opennet/cli.py index 57c883b..9aab117 100644 --- a/python/src/opennet/cli.py +++ b/python/src/opennet/cli.py @@ -13,7 +13,7 @@ import sys import time from pathlib import Path -from typing import Any, Sequence +from typing import Any, Callable, Sequence from . import __version__ from .client import OpenNetClient @@ -43,6 +43,7 @@ def _add_connection_options(parser: argparse.ArgumentParser) -> None: ) parser.add_argument("--connect-timeout", type=float, default=10.0) parser.add_argument("--ack-timeout", type=float, default=5.0) + parser.add_argument("--max-payload", type=int, default=DEFAULT_MAX_PAYLOAD) def build_parser() -> argparse.ArgumentParser: @@ -74,6 +75,18 @@ def build_parser() -> argparse.ArgumentParser: serve.add_argument( "--echo", action="store_true", help="send each accepted value back to its peer" ) + serve.add_argument( + "--log-values", + action="store_true", + help="include decoded payload values in logs (may expose sensitive data)", + ) + serve.add_argument( + "--allow-topic-prefix", + action="append", + default=[], + metavar="PREFIX", + help="accept only topics with this prefix; repeat to allow more prefixes", + ) send = commands.add_parser("send", help="send one typed value and await its ACK") _add_connection_options(send) @@ -163,6 +176,7 @@ def _client(args: argparse.Namespace) -> OpenNetClient: ssl_context=ssl_context, connect_timeout=args.connect_timeout, ack_timeout=args.ack_timeout, + max_payload=args.max_payload, ) @@ -190,10 +204,40 @@ def _value_from_args(args: argparse.Namespace) -> Any: if args.type == "hex": return bytes.fromhex(value) if args.type == "file": - return Path(value).read_bytes() + path = Path(value) + file_size = path.stat().st_size + if file_size > args.max_payload: + raise ValueError( + f"file is {file_size} bytes; --max-payload is {args.max_payload}" + ) + return path.read_bytes() raise AssertionError(f"unhandled value type {args.type}") +def _topic_prefix_authorizer( + prefixes: Sequence[str], +) -> Callable[[Peer, Frame], bool] | None: + if not prefixes: + return None + if any(not prefix for prefix in prefixes): + raise ValueError("--allow-topic-prefix cannot be empty") + allowed = tuple(prefixes) + return lambda _peer, frame: frame.topic.startswith(allowed) + + +def _frame_log_fields(frame: Frame, *, include_value: bool) -> dict[str, Any]: + fields: dict[str, Any] = { + "id": frame.message_id, + "topic": frame.topic, + "value_type": frame.value_type.name.lower(), + "payload_bytes": len(frame.payload), + } + if include_value: + value = decode_value(frame.value_type, frame.payload) + fields["value"] = value.hex() if isinstance(value, bytes) else value + return fields + + def _percentile(values: list[float], fraction: float) -> float: ordered = sorted(values) index = min(len(ordered) - 1, max(0, round((len(ordered) - 1) * fraction))) @@ -223,24 +267,19 @@ async def _serve(args: argparse.Namespace) -> None: async def log_message(peer: Peer, frame: Frame) -> None: nonlocal next_server_message_id - value = decode_value(frame.value_type, frame.payload) - summary = ( - {"bytes": len(value)} if isinstance(value, bytes) else {"value": value} - ) logging.info( "%s", json.dumps( { "peer": str(peer.address), - "id": frame.message_id, - "topic": frame.topic, - **summary, + **_frame_log_fields(frame, include_value=args.log_values), }, ensure_ascii=False, default=str, ), ) if args.echo: + value = decode_value(frame.value_type, frame.payload) await peer.send_value(frame.topic, value, next_server_message_id) next_server_message_id = ( 1 if next_server_message_id == 0xFFFFFFFF else next_server_message_id + 1 @@ -256,6 +295,7 @@ async def log_message(peer: Peer, frame: Frame) -> None: max_payload=args.max_payload, max_connections=args.max_connections, deduplication_window=args.deduplication_window, + authorizer=_topic_prefix_authorizer(args.allow_topic_prefix), ) await server.start() bound = ", ".join(str(sock.getsockname()) for sock in server.sockets) @@ -316,7 +356,10 @@ async def _benchmark(args: argparse.Namespace) -> None: "_opennet/benchmark", payload, retries=args.retries, retry_delay=0 ) samples.append((time.perf_counter() - started) * 1000) - metrics = _metrics(samples, args.payload_size) + metrics: dict[str, float | int | str] = { + "mode": "sequential_acknowledged", + **_metrics(samples, args.payload_size), + } if args.json_output: print(json.dumps(metrics, sort_keys=True)) else: diff --git a/python/tests/test_cli.py b/python/tests/test_cli.py index 9c99cb3..0b4d45d 100644 --- a/python/tests/test_cli.py +++ b/python/tests/test_cli.py @@ -2,7 +2,14 @@ import pytest -from opennet.cli import _require_secure_remote, _value_from_args, build_parser +from opennet.cli import ( + _frame_log_fields, + _require_secure_remote, + _topic_prefix_authorizer, + _value_from_args, + build_parser, +) +from opennet.protocol import Frame, FrameKind, ValueType @pytest.mark.parametrize( @@ -42,3 +49,65 @@ def test_plaintext_remote_requires_explicit_opt_in(): _require_secure_remote("192.168.1.20", tls=True, allow_plaintext=False) with pytest.raises(ValueError, match="plaintext"): _require_secure_remote("192.168.1.20", tls=False, allow_plaintext=False) + + +def test_oversized_file_is_rejected_before_read(tmp_path, monkeypatch): + path = tmp_path / "payload.bin" + path.write_bytes(b"four") + + def fail_if_read(_path): + raise AssertionError("oversized file should not be read") + + monkeypatch.setattr(type(path), "read_bytes", fail_if_read) + args = argparse.Namespace(type="file", value=str(path), max_payload=3) + with pytest.raises(ValueError, match="4 bytes"): + _value_from_args(args) + + +def test_server_logs_metadata_without_values_by_default(): + frame = Frame( + FrameKind.DATA, + ValueType.UTF8, + "secret/topic", + b"token", + message_id=1, + ) + fields = _frame_log_fields(frame, include_value=False) + assert fields == { + "id": frame.message_id, + "topic": "secret/topic", + "value_type": "utf8", + "payload_bytes": 5, + } + assert _frame_log_fields(frame, include_value=True)["value"] == "token" + + +def test_cli_topic_prefix_policy_is_explicit(): + policy = _topic_prefix_authorizer(("sensor/", "status/")) + assert policy is not None + allowed = Frame( + FrameKind.DATA, + ValueType.INT64, + "sensor/temperature", + (24).to_bytes(8, "big", signed=True), + message_id=1, + ) + denied = Frame( + FrameKind.DATA, + ValueType.BOOL, + "admin/reset", + b"\x01", + message_id=2, + ) + assert policy(None, allowed) # type: ignore[arg-type] + assert not policy(None, denied) # type: ignore[arg-type] + assert _topic_prefix_authorizer(()) is None + with pytest.raises(ValueError, match="cannot be empty"): + _topic_prefix_authorizer(("",)) + + +def test_serve_value_logging_is_opt_in(): + args = build_parser().parse_args(["serve", "--allow-topic-prefix", "sensor/"]) + assert args.log_values is False + assert args.allow_topic_prefix == ["sensor/"] + assert args.max_payload > 0 From 31f0db489e1e1ab6adf67af85b9a25c49404b91d Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:54:49 +0800 Subject: [PATCH 11/14] fix: reject server overload explicitly --- CHANGELOG.md | 2 ++ docs/architecture.md | 4 ++++ docs/releases/v0.2.0.md | 2 ++ python/src/opennet/server.py | 14 ++++++++++++++ python/tests/test_integration.py | 31 +++++++++++++++++++++++++++++++ 5 files changed, 53 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 56be18d..c67c1b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,6 +51,8 @@ All notable changes are documented here. OpenNet follows - The CLI rejects oversized files before reading them, logs payload metadata rather than values by default, offers a shared topic-prefix allowlist, and labels its benchmark as sequential acknowledged round trips. +- A Python server at its connection limit now sends a best-effort ERROR before + closing the rejected connection. ### Compatibility diff --git a/docs/architecture.md b/docs/architecture.md index 2aa5d17..3e84eb4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -85,6 +85,10 @@ Handler exceptions and timeouts increment counters without retracting an ACK that has already reported transport acceptance. If the handler set is full, the server sends a generic `handler queue full` ERROR and closes. +The connection set is bounded separately. A connection arriving at that limit +receives a best-effort ERROR before the server closes it, subject to the same +bounded write timeout as other control frames. + Server defaults also bound idle waits, incomplete headers, incomplete bodies with a payload-size allowance, TLS handshakes, writes, and handler duration. These values are configurable because a serial bridge and a local TCP service diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index a8f3f77..ace03fb 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -23,6 +23,8 @@ ONP/1 byte unchanged and adds an optional policy callback to the Python server. - CLI file sends are size-checked before reading, server logs hide values by default, and repeated `--allow-topic-prefix` options provide a simple shared allowlist. +- Connection and handler overload paths return bounded ERROR responses before + closing instead of failing silently. - The release workflow now runs native Arduino tests and the ESP32 build before packaging a tag. diff --git a/python/src/opennet/server.py b/python/src/opennet/server.py index 467b359..b650f10 100644 --- a/python/src/opennet/server.py +++ b/python/src/opennet/server.py @@ -218,6 +218,20 @@ async def _accept( ) -> None: if len(self._peers) >= self.max_connections: self.stats.rejected_connections += 1 + reason = b"server connection limit reached"[: self.max_payload] + with contextlib.suppress(Exception): + await asyncio.wait_for( + write_frame( + writer, + Frame( + FrameKind.ERROR, + ValueType.UTF8, + payload=reason, + ), + self.max_payload, + ), + timeout=self.write_timeout, + ) writer.close() with contextlib.suppress(Exception): await writer.wait_closed() diff --git a/python/tests/test_integration.py b/python/tests/test_integration.py index 97d41c4..4ac01e2 100644 --- a/python/tests/test_integration.py +++ b/python/tests/test_integration.py @@ -399,6 +399,37 @@ async def handler(_peer: Peer, _frame: Frame) -> None: await server.close() +async def test_server_connection_limit_sends_error_before_close(): + server = OpenNetServer( + lambda _peer, _frame: None, + host="127.0.0.1", + port=0, + max_connections=1, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + _first_reader, first_writer = await asyncio.open_connection("127.0.0.1", port) + for _ in range(100): + if server.stats.active_connections == 1: + break + await asyncio.sleep(0) + assert server.stats.active_connections == 1 + second_reader, second_writer = await asyncio.open_connection("127.0.0.1", port) + try: + response = await asyncio.wait_for(read_frame(second_reader), 1) + assert response.kind is FrameKind.ERROR + assert response.payload == b"server connection limit reached" + assert await asyncio.wait_for(second_reader.read(), 1) == b"" + assert server.stats.rejected_connections == 1 + assert server.stats.active_connections == 1 + finally: + first_writer.close() + second_writer.close() + await first_writer.wait_closed() + await second_writer.wait_closed() + await server.close() + + @pytest.mark.parametrize("partial_frame", (b"", b"O")) async def test_server_times_out_idle_or_partial_header(partial_frame): server = OpenNetServer( From eeeece2710da96d43003b8182bc186bac7af88ee Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:56:06 +0800 Subject: [PATCH 12/14] build: pin GitHub Actions dependencies --- .github/workflows/ci.yml | 18 +++++++++--------- .github/workflows/codeql.yml | 8 ++++---- .github/workflows/release.yml | 4 ++-- CHANGELOG.md | 2 ++ docs/releases/v0.2.0.md | 2 ++ 5 files changed, 19 insertions(+), 15 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1bef450..1cf4a80 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -22,8 +22,8 @@ jobs: matrix: python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"] steps: - - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: ${{ matrix.python-version }} cache: pip @@ -40,8 +40,8 @@ jobs: name: ESP32 Arduino examples runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.14" cache: pip @@ -54,8 +54,8 @@ jobs: name: Arduino native tests runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.14" - run: python -m pip install platformio @@ -67,8 +67,8 @@ jobs: env: SOURCE_DATE_EPOCH: "1700000000" steps: - - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.14" cache: pip @@ -79,7 +79,7 @@ jobs: - run: python scripts/build_checksums.py - run: python scripts/verify_reproducible.py - run: sh -n packaging/linux/install.sh packaging/linux/uninstall.sh - - uses: actions/upload-artifact@v7 + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: packages path: | diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index abad022..17c1114 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -18,16 +18,16 @@ jobs: matrix: language: ["python", "cpp"] steps: - - uses: actions/checkout@v7 - - uses: github/codeql-action/init@v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: github/codeql-action/init@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 with: languages: ${{ matrix.language }} - if: matrix.language == 'cpp' - uses: actions/setup-python@v7 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.14" - if: matrix.language == 'cpp' run: | python -m pip install platformio platformio run -e esp32dev - - uses: github/codeql-action/analyze@v4 + - uses: github/codeql-action/analyze@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index efa8b63..8b3e6eb 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -13,10 +13,10 @@ jobs: env: SOURCE_DATE_EPOCH: "1700000000" steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 - - uses: actions/setup-python@v7 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.14" cache: pip diff --git a/CHANGELOG.md b/CHANGELOG.md index c67c1b1..4345fca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,8 @@ All notable changes are documented here. OpenNet follows labels its benchmark as sequential acknowledged round trips. - A Python server at its connection limit now sends a best-effort ERROR before closing the rejected connection. +- GitHub Actions dependencies are pinned to immutable upstream commits with + readable release-version comments. ### Compatibility diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index ace03fb..c09a752 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -25,6 +25,8 @@ ONP/1 byte unchanged and adds an optional policy callback to the Python server. allowlist. - Connection and handler overload paths return bounded ERROR responses before closing instead of failing silently. +- CI, CodeQL, and release workflows pin third-party actions to immutable commit + IDs. - The release workflow now runs native Arduino tests and the ESP32 build before packaging a tag. From e79bd132471d2c65c62cd650e283a2109b792220 Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 18:57:52 +0800 Subject: [PATCH 13/14] test: add deterministic transport fault coverage --- CHANGELOG.md | 3 + docs/releases/v0.2.0.md | 3 + python/tests/test_faults.py | 126 ++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+) create mode 100644 python/tests/test_faults.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 4345fca..7730eae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -55,6 +55,9 @@ All notable changes are documented here. OpenNet follows closing the rejected connection. - GitHub Actions dependencies are pinned to immutable upstream commits with readable release-version comments. +- Deterministic fault tests cover concurrent sends, every truncation point of a + valid frame, fragmented ACK delivery, and a reproducible 24-byte-header + mutation corpus. ### Compatibility diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index c09a752..3bab0ff 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -27,6 +27,9 @@ ONP/1 byte unchanged and adds an optional policy callback to the Python server. closing instead of failing silently. - CI, CodeQL, and release workflows pin third-party actions to immutable commit IDs. +- Deterministic network-fault coverage now exercises concurrent sends, + byte-by-byte frame truncation, fragmented ACK delivery, and 2,240 single-bit + or seeded-random header mutations. - The release workflow now runs native Arduino tests and the ESP32 build before packaging a tag. diff --git a/python/tests/test_faults.py b/python/tests/test_faults.py new file mode 100644 index 0000000..faa892d --- /dev/null +++ b/python/tests/test_faults.py @@ -0,0 +1,126 @@ +import asyncio +import contextlib +import random + +import pytest + +from opennet import Flags, Frame, FrameKind, OpenNetClient, OpenNetServer, Peer, ValueType +from opennet.protocol import HEADER_SIZE, ProtocolError, inspect_header +from opennet.stream import read_frame + +TRUNCATION_FRAME = Frame( + FrameKind.DATA, + ValueType.UTF8, + "fault/topic", + b"payload", + message_id=17, + flags=Flags.ACK_REQUIRED, +).to_bytes() + + +async def test_concurrent_sends_keep_frames_and_message_ids_distinct(): + received: asyncio.Queue[int] = asyncio.Queue() + + async def handler(_peer: Peer, frame: Frame) -> None: + await received.put(frame.message_id) + + server = OpenNetServer( + handler, + host="127.0.0.1", + port=0, + max_handler_tasks=64, + ) + await server.start() + port = server.sockets[0].getsockname()[1] # type: ignore[union-attr] + try: + async with OpenNetClient("127.0.0.1", port) as client: + identifiers = await asyncio.gather( + *(client.send(f"concurrent/{index}", index) for index in range(32)) + ) + delivered = [ + await asyncio.wait_for(received.get(), 1) for _ in identifiers + ] + assert identifiers == list(range(1, 33)) + assert sorted(delivered) == identifiers + finally: + await server.close() + + +@pytest.mark.parametrize("cut", range(len(TRUNCATION_FRAME))) +async def test_disconnect_at_every_frame_byte_is_rejected(cut: int): + reader = asyncio.StreamReader() + reader.feed_data(TRUNCATION_FRAME[:cut]) + reader.feed_eof() + + with pytest.raises(asyncio.IncompleteReadError): + await read_frame(reader, max_payload=64) + + +async def test_complete_frame_after_all_truncation_points_still_decodes(): + reader = asyncio.StreamReader() + reader.feed_data(TRUNCATION_FRAME) + reader.feed_eof() + assert await read_frame(reader, max_payload=64) == Frame( + FrameKind.DATA, + ValueType.UTF8, + "fault/topic", + b"payload", + message_id=17, + flags=Flags.ACK_REQUIRED, + ) + + +def test_deterministic_header_mutation_corpus_stays_bounded(): + valid_header = TRUNCATION_FRAME[:HEADER_SIZE] + corpus: list[bytes] = [] + for byte_index in range(HEADER_SIZE): + for bit_index in range(8): + mutated = bytearray(valid_header) + mutated[byte_index] ^= 1 << bit_index + corpus.append(bytes(mutated)) + + generator = random.Random(0x4F4E5031) + corpus.extend( + bytes(generator.getrandbits(8) for _ in range(HEADER_SIZE)) + for _ in range(2048) + ) + + for header in corpus: + try: + topic_length, payload_length = inspect_header(header, max_payload=64) + except ProtocolError: + continue + assert 0 <= topic_length <= 1024 + assert 0 <= payload_length <= 64 + body = bytes(topic_length + payload_length) + with contextlib.suppress(ProtocolError): + Frame.from_parts(header, body, max_payload=64) + + +async def test_fragmented_ack_with_deterministic_delays_completes_send(): + async def accept(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + outbound = await read_frame(reader) + encoded_ack = Frame(FrameKind.ACK, message_id=outbound.message_id).to_bytes() + offset = 0 + for chunk_size in (1, 2, 3, 5, 8, 13): + if offset >= len(encoded_ack): + break + writer.write(encoded_ack[offset : offset + chunk_size]) + await writer.drain() + await asyncio.sleep(0) + offset += chunk_size + if offset < len(encoded_ack): + writer.write(encoded_ack[offset:]) + await writer.drain() + await read_frame(reader) + writer.close() + await writer.wait_closed() + + server = await asyncio.start_server(accept, "127.0.0.1", 0) + port = server.sockets[0].getsockname()[1] + try: + async with OpenNetClient("127.0.0.1", port) as client: + assert await client.send("fault/fragmented-ack", "ok") == 1 + finally: + server.close() + await server.wait_closed() From fdfc54f5e35696aed9cbc9929cbeb220c4969305 Mon Sep 17 00:00:00 2001 From: "@dev.mako" <95612413+devkyato@users.noreply.github.com> Date: Wed, 29 Jul 2026 19:02:09 +0800 Subject: [PATCH 14/14] docs: finalize v0.2.0 verification record --- docs/releases/v0.2.0.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/releases/v0.2.0.md b/docs/releases/v0.2.0.md index 3bab0ff..16e2251 100644 --- a/docs/releases/v0.2.0.md +++ b/docs/releases/v0.2.0.md @@ -43,9 +43,11 @@ an application result rather than a transport-delivery signal. ## Compatibility ONP/1 remains version 1. The 24-byte header, frame kinds, flags, value types, -CRC-32, ACK behavior for accepted messages, and Arduino frame bytes are -unchanged. Existing Python servers do not need to provide an authorizer, and the -Arduino API has no compatibility change in this release. +CRC-32, and Arduino/Python frame bytes are unchanged. ACK emission now follows +the existing transport-acceptance definition consistently; it still uses the +same ONP/1 ACK frame. Existing Python servers do not need to provide an +authorizer, and existing Arduino constructors and `onMessage` callbacks remain +source-compatible while the bounded alternatives are optional. The enum wire-facing types remain explicitly one byte, with compile-time checks. The C++ object size itself remains toolchain-dependent; the documented memory @@ -69,11 +71,14 @@ download against `SHA256SUMS`. ## Verification -- 46 Python tests, including allow, deny, fail-closed, plaintext, and real - loopback mutual-TLS authorization paths. +- 113 Python tests, including authorization, real loopback mutual TLS, control + backpressure, timeout, concurrency, truncation, and deterministic mutation + paths. - Strict Ruff and mypy checks. -- Native C++ protocol tests and the `esp32dev` firmware build. -- Local documentation links and reproducible Arduino/Linux bundles. +- Native C++ protocol tests, the main `esp32dev` firmware build, and all four + Arduino example compiles. +- Local documentation links, a fresh-wheel import/CLI smoke test, matching + release checksums, and reproducible Arduino/Linux bundles. These tests do not turn into a hardware field claim. The existing measured loopback numbers remain labeled as v0.1.1 results, and direct BLE GATT remains