feat: migrate away from Arduino - #438
Open
hhvrc wants to merge 127 commits into
Open
Conversation
I'd prefer to be able to distinguish between the different types easily in support chat.
…o feat/arduino-3.0
Merge develop into feat/arduino-3.0. Replace remaining tcb::span with std::span, add missing WiFi.h include. Build not yet clean — remaining issues: Serial namespace rename (SerialCmds), USBSerial API, and CaptivePortalConfig struct changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the Arduino-style one-byte getter (nextSerialByte) with an index-based scanner over the staging chunk. Serial::Read already batches the transport read; only the parser was still consuming a byte at a time. tryReadSerialLine now walks the chunk with a local index, handling control bytes individually and bulk-appending runs of printable characters via a new SerialBuffer::append. skipSerialWhitespaces scans the chunk directly as well. Behavior is byte-identical to the previous implementation (verified against the old parser across backspace/tab/control-byte/overflow cases).
# Conflicts: # .github/workflows/ci-build.yml # .github/workflows/codeql.yml # .github/workflows/cpp-linter.yml # .github/workflows/get-vars.yml
A crafted query could overflow the response buffer. The question end was
only bounded by the received length, so a 512-byte query whose question
filled the datagram left no room for the 16-byte answer: the memcpy that
appended it wrote past `resp`, and the following sendto transmitted those
bytes. Any client associated with the portal AP could trigger it.
Check that the reply fits before building it, and validate the query
before trusting it:
- reject responses (QR set), which otherwise let two responders answer
each other indefinitely and made us usable as a reflector
- reject anything that isn't a single standard query (QDCOUNT, opcode)
- reject compressed, reserved-type, and over-long names; a pointer in a
question has nothing valid to point back at, and echoing one aimed the
answer's own 0xC00C at a dangling chain
- drop datagrams that exactly fill the buffer, since recvfrom truncates
Also parse the response address with inet_pton, which rejects the
out-of-range components and trailing garbage that a scanf of four ints
silently truncated, and answer with a 60 s TTL so clients stop re-querying
on every probe. The reply is now built in place, which frees the second
512-byte buffer from the task's 3 KB stack.
Errors are handled separately from timeouts so a persistently failing
socket can't spin the core, and the socket is closed after the task has
stopped rather than before. shutdown() is gone: lwIP rejects it with
EOPNOTSUPP on anything that isn't TCP, so the receive timeout was always
the real wakeup and is now mandatory rather than best-effort.
StopTask() waited for a task to self-delete by polling eTaskGetState() on
its handle, but that is exactly what a self-deleting task's handle cannot
be used for. vTaskDelete(nullptr) only queues the task:
vListInsertEnd(&xTasksWaitingTermination, &(pxTCB->xStateListItem));
++uxDeletedTasksWaitingCleanUp;
after which the idle task frees the TCB. eTaskGetState() dereferences the
handle to read pxTCB->xStateListItem, so once that free has happened the
poll reads freed heap — and StopTask() reads it twice, in the loop
condition and again in the check after it, with the other core's idle task
free to run in between. If the freed memory doesn't happen to match
xTasksWaitingTermination or NULL, the state comes back as something other
than eDeleted and StopTask() escalates to vTaskDelete() on a freed TCB.
Only the task itself can signal its exit safely, so TaskUtils owns that
pattern now. TaskExiting() publishes a caller-owned flag and then deletes
the task; StopTask() takes the same flag and waits on it, and only touches
the handle when the flag is still clear — precisely the case where the task
provably has not deleted itself, so the force-kill path is sound.
The signature change is deliberate: it makes the old racy call impossible
to write by accident. All six tasks are migrated, each clearing its flag
before the task is created, and no raw vTaskDelete(nullptr) remains outside
TaskExiting().
The three public headers get a bare std::atomic<bool> rather than an
include of TaskUtils.h, and CommandHandler's flag is fully qualified since
it is declared above that file's using-directive.
Covers the user-visible half of the fix: the crash and memory disclosure a crafted query could cause on the setup network, the stricter rules about what the responder will answer, the non-zero answer TTL, and the stricter validation of the setup network address.
|
Important Review skippedToo many files! This PR contains 390 files, which is 290 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. Usage-priced reviews support at most 300 files. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (7)
📒 Files selected for processing (390)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Firmware publishing to the repository server is authenticated by a GitHub Actions OIDC token, and registration on the server is per (owner, repo). Neither can be exercised locally or from any other repository: the token request URL only exists inside an Actions run, and a token minted elsewhere proves nothing about this one. So the check has to live here. It writes nothing. Both probes either read a public endpoint or name an object that cannot exist, so a run leaves no releases, no storage objects and nothing to clean up. Aborting a random release id is the pass condition: the lookup misses, and reaching that miss means signature, issuer, audience, registration and scope were all accepted. The board catalog is checked at the same time. It is seeded by hand and names have to match the boards/*.defaults basenames a hub reports as its own board, so a board this repository builds but the server does not know stays invisible until a release init rejects it mid-publish. 401 is reported with both of its causes spelled out, since the server refuses an unregistered repository with the same response as a bad token and the two cannot be told apart from the wire.
The smoke test proved the server accepts a token from this repository, but it proved it with requests nothing else makes. A probe that only the probe runs tells you the probe works. What has to be known before release day is that the publish path works, so the publish path is what should be running. Three composite actions, and one of them does the real thing: repo-server-token mints the OIDC token, one definition of the audience repo-server-preflight reads only: server up, repo authorised, boards known repo-server-publish init, upload every board, then promote or discard `mode` picks the ending and nothing before it differs. A dry run parses the same changelog through the same grammar, uploads the same bytes, and passes the same server-side hash verification as a real publish; only the promote is skipped and the staged release is discarded instead. So ci-build dry-runs on every build, and release day runs code that has been exercised continuously rather than for the first time. Discarding is unconditional on the way out, including after a failure. A staged release that is never aborted holds its version against the retry meant to fix it, until the server's TTL job reaps it. The real publish stays gated on OPENSHOCK_REPO_SERVER_PUBLISH rather than on the deploy gate, because the CDN path still serves devices. Flipping that variable is the cutover; until then this proves the cutover will work without performing it.
The dry run stops before the promote, which leaves the last three links in the
chain untested until the day they matter: whether publishing makes a version
visible, whether the CDN serves the bytes, and whether a hub is told to update.
"No change on the server" was the dry run working correctly, and also the reason
none of that had ever been exercised.
repo-server-verify closes it. After a real publish it reads the release back as
an anonymous caller, through the endpoints a hub and the flashtool use, and for
every board downloads each artifact from the URL the server hands out and hashes
it against the file this run compiled. A match means the bytes a device would
flash are the bytes CI built - which the upload response cannot establish, since
that only proves the server received them, not that the CDN serves them from the
published key.
It also checks the hash the API advertises against the file behind it, because a
correct file behind a wrong advertised hash fails every OTA update while looking
healthy from outside. And that a hub reporting the version just published gets
204 rather than being offered the same build again.
Testing the server now runs on one axis, cheapest first, each row the row above
plus more:
repo-server-preflight any branch, seconds, no build - server reachable,
repo authorised, catalog complete
ci-build dry run + init, upload every board, server-side hash checks
ci-build publish + promote, read back, compare served bytes to built
repo-server-smoke was the first row already, but named as though it were the
whole thing and sitting beside a ci-build job that ran the same preflight before
doing the real work. Renamed to say what it is. It stays separate because it
answers questions about server state an admin changes underneath CI - a scope
revoked, a board not yet added - and re-checking those should not cost a
30-minute build. Everything that writes lives in ci-build, where the artifacts
are.
OPENSHOCK_REPO_SERVER_PUBLISH is the only switch: set it, dispatch ci-build on
any ref, unset it after; leaving it set is the cutover. Not a dispatch input,
because GitHub only offers inputs from the default branch's copy of a workflow,
so one would never appear on the branch developing this. Not tied to the deploy
gate either, which fails the same way from the other end - a dispatch on a
feature branch yields build=true, deploy=false, putting the full test out of
reach exactly where it is being written.
CI's contract with the repository server is: submit each blob with the hash it computed, and fail on any non-success response. Everything else was overreach, and the CDN was the largest piece of it. cdn-deploy and cdn-bump are gone, along with the cdn-upload-firmware, cdn-upload-version-info, cdn-bump-version and sftp-mirror actions and the cdn environment they read their Bunny credentials from. Where artifacts are stored, which hostname serves them and the moment a channel starts offering a version are decisions the server makes behind one API call. CI was making them from the outside, holding a storage credential to do it, and could leave objects, pointer files and the release index disagreeing about what is live. What is left is the contract: mint a token, init, upload every board with its sha256 manifest, promote or discard, and fail on any status that is not the expected one. Nothing else. repo-server-verify is gone for the same reason it always should have been. It read published releases back and compared the served bytes to the built ones, which is the server's invariant - it is the only side that knows where it wrote and which hostname fronts that storage. CI was inferring a server-side misconfiguration from the outside, badly, and reporting it as a hash mismatch. repo-server-preflight is gone too. An unregistered repository, a missing scope and an unknown board are all non-success responses from init, before a single artifact is uploaded, and each already names its own cause. Asking the same questions beforehand moved the answer earlier by about a second and gave every one of them a second place to be wrong. Publishing now follows the deploy gate rather than OPENSHOCK_REPO_SERVER_PUBLISH, which is retired - delete the repo variable. That variable existed to keep the publish decoupled from deploy while the CDN still served devices; with the CDN path gone, a release tag that built and published nothing would be the failure mode instead. A tag or a nightly/manual develop deploy publishes; every other build does the same init-upload-discard dry run as before. An unset OPENSHOCK_REPO_SERVER_URL no longer skips the job silently. That was safe when the CDN was the real deploy; now it is the difference between shipping and not, so it fails loudly on a deploy and only degrades to a quiet skip elsewhere. The staged-draft-release guard moved from cdn-deploy into the publish it was always protecting, scoped to real publishes so dry runs do not poll GitHub for a minute over nothing. The GitHub release and the API announcement still run last, in that order, each chained on the publish succeeding with no always() guard - a failed or discarded publish skips both.
The publish path was bash calling curl and jq, then Python calling urllib with a multipart body assembled by hand. Both were writing protocol code that is not this repository's problem to solve. repo_server_publish.py and repo_server_token.py now use requests, and get-vars.py parses tags with the semver package instead of a local regex and version class. The hand-rolled MultipartBody and MultipartReader are gone; a Session carries the bearer token in one place and reuses the connection across boards. Dependencies are pinned by digest in .github/scripts/requirements.txt and installed with --require-hashes. The transitive requests dependencies are listed for the same reason, and charset-normalizer carries several hashes because pip may choose any of the compiled wheels. This is the job that holds the publish credential; what runs next to it should not be whatever the index happens to serve that morning. The three scripts share .github/scripts/gha.py for workflow commands, step outputs and the interpreter floor, instead of each restating them. gha.py is deliberately conservative in syntax: a module whose job is to report that the interpreter is too old has to be importable on that interpreter, or the check fails as a SyntaxError instead of the sentence explaining what to install. setup-python now pins the interpreter via .github/scripts/.python-version rather than relying on what the runner image ships, which is also what guarantees pip is present. get-vars.py was verified against its predecessor across ten refs - three branches, a feature branch, rc/stable/beta/develop tags, the legacy v0.8.1 tag and an invalid one - and produces identical version, channel and error output for every one. Python bytecode is ignored repo-wide from the root .gitignore, replacing the single-line scripts/.gitignore that only covered one directory.
The dry run proved the publish path worked and then discarded the only thing anyone wanted out of it. OPENSHOCK_REPO_SERVER_URL points at the development instance now, so a live version there is the point rather than a hazard: the artifacts a branch just produced are fetchable by a hub aimed at the same instance, which a discarded release never was. mode is publish unconditionally. The deploy gate no longer decides whether to publish, only whether to cut a GitHub release. The staged-draft requirement drops its deploy condition. get-vars maps a push to master or beta onto a channel devices follow, tag or no tag, so with every build publishing, that gate was the only thing standing between an ordinary branch push and the fleet. An unset OPENSHOCK_REPO_SERVER_URL is now fatal rather than a quiet skip. That was survivable while a build might publish nothing; there is no such build left, so it now means a run compiled firmware with nowhere to ship it. Publishing consumes a version and only a maintainer can undo it. Branch builds carry +<short-sha>, so ordinary pushes do not collide; re-running a build on an already published commit does. The comments in this job are cut to match. They restated the code beside them, argued against a preflight that no longer exists, and named /admin/firmware/releases twice.
The comments around the publish path had grown to restate the code beside them and to justify choices nobody was asking about. A module docstring explaining that Python puts a script's own directory on sys.path is not telling anyone something they could not have assumed. What is left is what cannot be inferred from the code: that GitHub does not evaluate paths-ignore for tags, why concurrency needs no group shared across refs, why gha.py is deliberately conservative in syntax. Comments are no longer wrapped to a column. A sentence occupies one line however long it runs, and a line break means a sentence ended.
Four guard steps each carried an if: condition, putting the rules in an expression language with no types, no tests, and a failure mode of silently evaluating false. repo_server_policy.py decides what applies from the server, channel, tag and commit, so four steps and four conditions become one and none.
Kept this branch's ESP-IDF build and repository-server publish path, which supersede develop's PlatformIO actions, CDN deploy jobs and its own port of the OTA update manager. Took develop's dependency bumps, the publish-tag and pr-check-comment actions, and the frontend package updates.
hhvrc
had a problem deploying
to
repo-server-dev
August 18, 2026 13:21 — with 
GitHub Actions
Failure
GitHub Actions
Failure
The release was opened at the very end of a run, so a version already published, a concurrent run holding the same one, or an unknown board was refused after a half-hour matrix build instead of before it. A stage job now inits the release straight after get-vars and hands the id downstream, and the abort job is the finally for a lifetime that no longer fits inside one script. The prod reviewer moves to stage with it, so a release is approved before it spends the compute rather than after, which is why the build roots wait on stage. The dispatch tag input is gone and channel is threaded into get-vars instead. Both inputs previously applied only at publish time, after the build had already baked the derived version into OPENSHOCK_FW_VERSION and the filename, so a dispatch naming a version shipped firmware that reported one version and was published as another.
A thirteen-way matrix spent more on checkout, pip and artifact downloads per runner than on the work: the merge is a few seconds of esptool per board. One job now does all of them, so the merged binaries arrive as a single firmware_merged artifact rather than one per board, and resolve_artifacts tells them apart by the board in their filename.
merge_image.py took one board, which would have meant driving it from a bash loop. It now takes a list, expands {board} into --bindir and --output itself, and merges across the CPUs it has. Workers are processes rather than threads because esptool keeps module-level state and exits the interpreter instead of raising, and each board's output is captured and printed whole so parallel runs do not braid their logs together.
A failing board no longer stops the rest, which is what fail-fast: false gave the matrix: every board that failed is named at the end.
hhvrc
had a problem deploying
to
repo-server-dev
August 18, 2026 14:33 — with
GitHub Actions
Failure
GitHub Actions
Failure
The root CMakeLists force-included a single generated header into every compilation in the project through COMPILE_OPTIONS, so all of ESP-IDF saw it too. That header carried OPENSHOCK_FW_GIT_COMMIT and a version string ending in +<short-sha>, both of which change on every commit. The effect was not that ccache stopped reusing objects - it still recovered about 95% of them - but that it could only do so through its preprocessor mode. Direct-mode lookups missed on every translation unit, so each build ran the full preprocessor over the entire project before it could match anything. Measured on NodeMCU-32S with a warm cache and a fresh build directory, a commit-only change cost 345s against 142s for the identical tree, with a direct-mode hit rate of 0.11%. Split the generated header in two. openshock_board.h holds board and chip identity, build mode and log level, and is included by OpenShock.h, so only OpenShock's own components see it. openshock_version.h holds the version and commit, and is included by the five translation units that report them. Neither is force-included. The same measurement is now 173s at a 100% direct-mode hit rate. The version header is also regenerated at build time rather than only at configure time. It was previously written by execute_process during CMake configure, which meant an incremental build kept reporting whatever version its build directory was first configured with - `idf.py app` after changing the commit finished in 5s and silently produced a binary stamped with the old SHA. It now refreshes on every build and writes only when the contents differ, so a new commit rebuilds five objects and an unchanged one rebuilds nothing.
Thirteen boards map onto four chips, but no two of them shared a single compiled ESP-IDF object. The five per-board GPIO settings were Kconfig symbols, so they landed in sdkconfig.h - which 922 of ESP-IDF 6.0.2's component sources include directly, and many more transitively. Two boards that differed only by an LED pin therefore produced a different sdkconfig.h and recompiled the entire framework. They are plain generated macros in openshock_board.h now, set per board as bare OPENSHOCK_* lines in boards/<board>.defaults and stripped from the sdkconfig fragment by scripts/build.py before idf.py sees it. The settings that are identical across every board - domains, hostname, AP prefix, URI buffer - stay Kconfig, because they cost nothing in shared objects. This collapses 13 distinct sdkconfigs into 6: the five ESP32/4MB boards become one, as do the three ESP32-S3/8MB boards. Comparing the compile command for an ESP-IDF source across two boards in a group, 115 of 117 argv tokens now match.
Three separate problems in one cache key. The key contained github.sha. Cache entries are immutable, so a key that always misses always writes a full new copy: 13 fresh entries on every push, 47 live at the time of writing. That, more than anything else, is what pushed the repository past the 10 GB cap and left GitHub evicting the ESP-IDF entry every job needs. The SHA is now only a save discriminator; restore-keys takes the newest store for the group. The key was per board, when what determines object identity is the resolved sdkconfig. get-vars.py now derives a cache group per board - the chip plus a digest of every CONFIG_* line in sdkconfig.defaults and the board fragment - and puts it in the build matrix. Thirteen boards resolve to six stores. The digest is derived rather than hardcoded as chip+flash so that a board which later changes any other sdkconfig setting splits into its own group by itself, instead of silently sharing a store it can never hit. Finally, the build directory path appears on every compile command line, in -I<dir>/config and the @<dir>/toolchain/cflags response file. Building each board under build/<board> gave every board a different command line, so even with an identical sdkconfig two boards shared nothing - a same-group build measured 0% hits for exactly this reason. Each CI job builds one board, so it now sets OPENSHOCK_BUILD_DIR to a fixed path; local builds keep build/<board>. The same measurement is now 98.25% hits, 155s against 295s cold. Also sets CCACHE_COMPILERCHECK=content, since the toolchain is restored from a cache with fresh mtimes every run, bounds the store with CCACHE_MAXSIZE, prints the stats, and skips saving on pull requests - a PR restores the base branch's store in full, but writing from every PR push is what multiplies entries fastest.
…base Two independent costs paid on every run. The ESP-IDF install was cached whole, at 3.65 GB per entry, and duplicated across refs: 6.81 GB of the repository's 11.63 GB was two byte-identical copies of it. Split the cache into restore/save so the install can be trimmed in between, and prune what CI never invokes - dist/ (the download archives, already extracted into tools/), the GDBs, OpenOCD, the ULP toolchain, and esp-clang, which only clang-tidy uses and cpp-linter runs with tidy-checks '-*'. The gcc toolchains are left alone: xtensa builds 12 boards and riscv32 builds the ESP32-C3. A measurement step reports the size of the install per directory first. The figures the prune list was drawn from come from a local install polluted by other projects, and EIM may not install esp-clang on a runner at all, so the first run is what confirms them. If a warm run starts taking minutes in the install step again, the installer is re-fetching something pruned here. Separately, build-compilationdb ran a full uncached firmware build for both CodeQL and cpp-linter. CodeQL needs it - its tracer only sees the C/C++ if the compiler actually runs. cpp-linter does not: it reads compile_commands.json, which CMake writes at configure time, before a single object is compiled. It now runs `idf.py reconfigure` instead, which produces the same 8.3 MB database in 62s and compiles nothing.
Arduino is no longer a component of this project - it appears in neither dependencies.lock nor any idf_component.yml - so kconfgen reported all fourteen CONFIG_ARDUINO_SELECTIVE_* and CONFIG_AUTOSTART_ARDUINO symbols as unknown and discarded them. They had no effect on the build. CONFIG_MBEDTLS_PSK_MODES, CONFIG_MBEDTLS_KEY_EXCHANGE_PSK and CONFIG_LWIP_SO_RCVBUF were also only there for Arduino, but unlike the above they are real settings that still take effect, so they are flagged in a comment for the de-Arduino migration rather than changed as a side effect of a caching pass. OPENSHOCK_FW_BUILD_DATE was exported by build-firmware, build-compilationdb and codeql.yml, but nothing reads it - gen_env_header.py has never emitted it.
…e on PRs Three things the first CI run exposed. The prune matched the wrong paths. A runner installs the toolchains directly under the install root (/tmp/esp/esp-clang) where a local EIM install puts them under a tools/ subdirectory (~/.espressif/tools/esp-clang), and only the latter was checked. The single directory that did match was ~/.espressif/dist, so the run reported "pruned 1099 MB in total" and left esp-clang's 1.9 GB, esp-clang-libs' 335 MB, 247 MB of GDBs, OpenOCD and the ULP toolchain in the cache. Both layouts are now checked, by name rather than by glob so that xtensa-esp-elf-gdb can go without putting xtensa-esp-elf at risk. The key moves to -4 so the trimmed install is stored fresh. The measurement step did its job: the install is 7.7 GB under /tmp/esp plus 1.1 GB of archives, and esp-clang is present on a runner - which was the open question the prune list was written against. Every matrix job tried to write the ESP-IDF cache. The install is identical for all 13, so when the key changes they all tar and upload ~3.5 GB at once; one wins the reservation and the other twelve spend ~20s each to be told "another job may be creating this cache". get-vars now nominates a single writer and the rest restore read-only. Finally, ccache was never saved at all. The save was skipped on pull requests, on the assumption that a PR restores its base branch's store - but ci-build only runs `push` for develop, beta and master, so a feature branch only ever builds as a PR and the store was never written by anything. First run measured 0% hits and "Cache not found" for both keys. It now saves on every event; at 6 stores per push instead of 13, each around 30 MB for one board and capped by CCACHE_MAXSIZE, the entry count this costs is affordable in a way the old per-board-per-SHA key was not.
`stage` is skipped on every run that does not publish, and GitHub propagates that skip down the needs graph to any job that does not override it. build-frontend and build-firmware each carry their own condition and so ran; build-staticfs did not, so on a pull request it skipped, and took merge-partitions and checkpoint-build with it. A PR therefore compiled every board and then produced no flashable images. Pre-existing, and not caused by the caching work - run 32147411341, a pull request from before any of it, shows build-frontend succeeding and build-staticfs skipped in exactly the same way. It builds fine on workflow_dispatch, where nothing upstream is skipped, which is why it went unnoticed. Both jobs now carry an explicit condition, matching what build-frontend and build-firmware already do.
The installer, not the compiler, was the cost of a build. In the run that had to be cancelled, one job spent 867s in espressif/install-esp-idf-action and 128s compiling firmware. Roughly 700s of that was apt fetching cmake at about 35 KB/s, because all 13 matrix jobs were doing the same thing at the same moment. The action re-runs apt and re-prepares the IDF clone on every invocation, cache hit or not. The restored tree is self-contained, so a warm job no longer needs the action at all. EIM leaves an activate_idf_<version>.sh beside the install with absolute paths baked in, and the whole tree is cached, so those paths still resolve. The script's own `-e` mode prints KEY=VALUE lines, which map onto GITHUB_ENV and GITHUB_PATH. Sourcing it is not an option: it refuses to run unless $0 is a shell name, which it never is inside a GitHub step, and it reads $ZSH_VERSION without a default, which trips `set -u`. $IDF_PATH/tools is added separately, since idf.py lives there and the script's PATH line does not cover it. ccache is then the only build prerequisite the IDF tree does not carry - cmake, ninja and python all come from it - and it is installed only when missing: 592 KB at worst against the action's ~15 MB. Verified locally by building NodeMCU-32S with nothing but the environment that step produces: no installer, no clone, no apt. idf.py and ccache both resolve and the build completes. A warm-esp-idf job now populates the cache once before the matrix fans out. Otherwise a changed key puts all 13 boards back into the installer simultaneously, which is the stampede that made the run unusable. It calls setup-esp-idf with the new lookup-only input, so the key stays defined in exactly one place and the probe checks for the entry without downloading it - seconds when it is already there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilds the firmware on native ESP-IDF 6. The Arduino framework, the PlatformIO build and the CDN deployment are all gone; what replaces them is an
idf.pybuild over a layered component tree, a security-verified TLS path, host-runnable tests for the pure logic, and a CI that deploys through the repository server.The branch started as an Arduino 3.0 bump and turned into the removal of Arduino instead. It is large because there is no partial version of this: the framework provided WiFi, HTTP, WebSockets, RMT, the filesystem, the serial console and the logging sink, and each had to be replaced before the framework could go.
Build
scripts/build.pydrivesidf.py; per-board sdkconfig fragments live inboards/*.defaultsand layer ontosdkconfig.defaults.platformio.ini, the.envfiles and the extra_scripts are gone. flatbuffers is vendored as a submodule component.CONFIG_OPENSHOCK_*directly rather than-Dflags. This surfaced a real bug: the WS2812B R/G channel swap tested an un-prefixed macro the native build never defines, so it silently never triggered on Waveshare S3-Zero or Wemos S3-Mini.Subsystems moved off Arduino
WiFiclass,arduino_event_tesp_wifi+esp_netif+esp_eventHTTPClient/WiFiClientSecureesp_http_client, reusable keep-aliveHTTP::ClientWebSocketsClientesp_websocket_clientwith manual fragment reassemblyesp_http_server+ native DNS responder +StaticFsesp32-hal-rmtdriver/rmt_txfscomponent:LfsPartition/StaticFs/ConfigFsover rawesp_partitionSerial,log_printfserialtransport (UART0 / USB-Serial-JTAG); logging writes bytes to itmbedtls/md.hgeneric interfaceosjson(jsmn zero-copy parse + streaming generation)Structure
main/now holds onlyapp_main. Everything else is an auto-discovered component, and the interconnected tangle has been progressively pulled apart into layers that depend one way:common,crypto,logging,temporal,osjson,events,hwutil,chipset,littlefs,fs,flatbuffers,serialization,config,protocols,led_drivers,http,dns_server,rfc8908,serial,device_control,core,serial_console.The extractions that mattered: generated flatbuffer schemas out of
core(which broke the config<->serialization cycle), thenconfig, thendevice_control(estop, RF transmitter, visual state, command handler). Leaky internal headers -CaptivePortalInstance.h,GatewayClient.h, the message-handler internals - moved out ofinclude/intosrc/soesp_http_server,esp_websocket_clientand friends stopped propagating to consumers.Security
CONFIG_ESP_TLS_INSECUREandSKIP_SERVER_CERT_VERIFYwere set, leaving the gateway WebSocket and the HTTP/OTA clients open to MITM. The firmware now verifies against a compiled-in bundle, with GlobalSign Root CA R1 pinned -api.openshock.apppresents a chain topped by the cross-signed GTS Root R4, whose issuer Mozilla/curl retired, andesp_crt_bundledoes no path building. A weeklypinned-cert-auditworkflow retires pins no live chain still needs and warns before a needed one expires. Costs ~16 KB flash.inet_ptoninstead of ascanfof four ints.StopTask()polledeTaskGetState()on the handle of a task that had already calledvTaskDelete(nullptr), reading a TCB the idle task was free to have freed - and could escalate to force-killing it. Tasks now publish a caller-owned exit flag; the signature change makes the old racy call impossible to write by accident, and all six tasks are migrated.., so1.5.0-rc.7dropped the.7- different RCs compared equal, which is OpenShock's own tag format and drove OTA ordering.Tests
Host-runnable Unity suites on the linux target, for the pure-logic pockets that had no off-hardware coverage at all: the RF bit-encoders (frame layout decoded back from RMT symbols, intensity clamping, model dispatch),
osjson, the enum parsers, SemVer,Convert, string/hex helpers,Checksum,TinyVec,FnProxy,DigitCounter.CI/CD
scripts/build.pythroughinstall-esp-idf-action; the matrix is derived fromboards/*.defaultsby a Python port ofget-vars. C++ and staticfs build in parallel and merge afterward, with all 13 boards merged in one parallel job.cdn-deploy,cdn-bump, the four CDN composite actions and thecdnenvironment are gone along with the Bunny storage credential. What is left is the contract: mint an OIDC token, init the release, upload every board with its sha256 manifest, promote or discard. Where artifacts live, which host serves them and when a channel starts offering a version are the server's decisions behind one API call.if:expressions intorepo_server_policy.py; the scripts userequestsand thesemverpackage instead of hand-rolled multipart and regex, with dependencies pinned by digest and installed--require-hashes.Change files
esp-idf-6-migration.md(breaking),dns-server-hardening.md,ci-repository-server-only-deploy.md.Review
All Copilot findings from the earlier Arduino 3.0 pass are resolved or no longer applicable, each answered in thread.
Open item
-DARDUINO_USB_CDC_ON_BOOT=1was dropped for Waveshare S3-Zero and Wemos Lolin S3-Mini and has no native replacement in their.defaults- neither board setsCONFIG_ESP_CONSOLE_*, so both fall back to the IDF default rather than USB-Serial-JTAG. Worth confirming the console comes up on hardware before merge.