Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,42 @@ on:
pull_request:

jobs:
map-consumer:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Initialize a consumer fixture from approved published maps
env:
PYTHONPATH: src
run: >-
python3 -m pokesim.maps.consumer_sync update
--consumer "$GITHUB_WORKSPACE" --source "$GITHUB_WORKSPACE" --offline
--game bdsp --game frlg --revision "$(git rev-parse origin/main)"
- name: Exercise the Arduino integration action
uses: ./integrations/arduino-source/verify-map-packs
- name: Ensure a local map edit fails the consumer check
env:
PYTHONPATH: src
run: |
python3 - <<'PY'
from pathlib import Path
import subprocess
import sys
from pokesim.maps.consumer_sync import DESTINATIONS
target = next(Path(DESTINATIONS['bdsp']).glob('*.json'))
target.write_bytes(target.read_bytes() + b'\n')
result = subprocess.run([sys.executable, '-m', 'pokesim.maps.consumer_sync',
'check', '--consumer', '.', '--source', '.pokesim-map-source', '--offline'],
capture_output=True, text=True)
assert result.returncode == 1 and 'modified ' + target.name in result.stderr, result
print('Consumer correctly rejected local drift: ' + target.name)
PY

unittest:
runs-on: ubuntu-22.04
strategy:
Expand All @@ -22,6 +58,8 @@ jobs:
python-version: ${{ matrix.python-version }}
- name: Run the tests
run: PYTHONPATH=src python -m unittest discover -v -s src -t src
- name: Audit published map inventory and hashes
run: PYTHONPATH=src python -m pokesim.maps.runtime_map_pack

battle:
runs-on: ubuntu-22.04
Expand Down
9 changes: 5 additions & 4 deletions docs/autostory-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,11 @@ semantics match and keep game-specific adapters where they differ.
pinned exports, remote-manifest hash verification and map corrections here.
- [ ] Recover exact pret revisions for BDSP inputs recorded as default branch;
archive/pin inputs before refreshing them.
- [ ] Implement the pinned consumer lock/update/integrity check, configure it
as required, and adopt it in the owning Arduino PRs. Remove duplicate offline
authoring there; preserve native controller/detector tests. Design is recorded;
implementation and consumer adoption remain pending.
- [x] Implement pinned consumer lock/update/check commands, a reusable CI action
and regression coverage. See [the integration](../integrations/arduino-source/README.md).
- [ ] Adopt the check in owning Arduino PRs and configure its required status.
Resolve pending map differences first, remove duplicate offline authoring
there and preserve native controller/detector tests.
- [ ] Consolidate the older eight `pokenav-map/1` fixtures only after explicit
frame conversion and parity checks; do not overwrite them with cropped maps.

Expand Down
24 changes: 13 additions & 11 deletions docs/maps/consumer-process.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Map changes and consumer updates

Status: proposed process; implementation pending. This document does not
install an Arduino check, import PR #226, or mark the consumer migration complete.
Status: check/update tooling and a reusable CI action are implemented; adoption
in Arduino and its required status settings remain pending. See the
[integration commands and rollout](../../integrations/arduino-source/README.md).
No changes from PR #226 are imported, and consumer migration is not complete.

PokemonSimulation is the authoring repository for map data, source parsers,
builders, schemas and corrections. Arduino-Source consumes a pinned export.
Expand Down Expand Up @@ -34,10 +36,10 @@ Audit dependent CLI wrappers before removing their decoder imports. Route
drawing/planning tools are a separate migration; do not delete all of
`devtools/maps` just because builders move.

Use one consumer lock file, provisionally
Use one consumer lock file,
`devtools/maps/map-packs.lock.json`. Each selected game pins a full
PokemonSimulation commit, the manifest path and its SHA-256. This is a proposed
shape, with placeholders, not an existing working lock:
PokemonSimulation commit, the manifest path and its SHA-256. The update command
writes this shape; this example uses placeholders rather than a real lock:

```json
{
Expand Down Expand Up @@ -160,13 +162,13 @@ generator/correction change and evidence. The required consumer check would
reject all five edited JSON files under the old pin, even if Arduino's local
simulations passed. Changing the local lock hash alone would still fail against
the manifest fetched from the pinned source. None of these changes is imported
by this design PR.
by this implementation.

## Rollout and remaining milestone 1 work

1. Implement this repo's publication checks and update tool contract. Existing
`runtime_map_pack` already validates bytes/inventory/links; it does not yet
publish a consumer lock or run reproducible source generation in CI.
1. Implemented: source audit in CI, pinned lock/check/update commands,
consumer CI action and regression tests. Reproducible source generation in
CI and consumer adoption remain pending.
2. Archive the exact BDSP source inputs used for current output with their file
hashes. Recover pret commits where possible; if not recoverable, label the
snapshot's original commit unresolved and pin its archived bytes. Do not
Expand All @@ -184,5 +186,5 @@ by this design PR.
namespace after frame/parity checks; carry their measurements with them.
Do not overwrite them with runtime crops or let consumer exports include them.

Only then mark map ownership complete. This design completes the process
decision, not the implementation or adoption steps above.
Only then mark map ownership complete. The core tooling implements the process
decision; source-input pinning and Arduino adoption above remain open.
5 changes: 3 additions & 2 deletions docs/maps/runtime-maps.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@ and offline checks. Arduino remains the consumer of generated JSON. Consumer
branches have not been changed or had their generators deleted by this import.
The staged work is tracked in [the integration plan](../autostory-plan.md).
The [consumer process](consumer-process.md) specifies pinned exports and hash
checks so future map corrections are authored here. It is a design awaiting
implementation; Arduino's map directories are not yet protected by that gate.
checks so future map corrections are authored here. The
[check/update tools and CI action](../../integrations/arduino-source/README.md)
are implemented; Arduino adoption and required status settings are pending.

## Inventory and provenance

Expand Down
105 changes: 105 additions & 0 deletions integrations/arduino-source/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Arduino map consumer integration

The implementation lives in `pokesim.maps.consumer_sync` and
`pokesim.maps.consumer_source`. Python 3.8+ and Git suffice; no Python packages
or battle engine are needed. This folder supplies the CI action and workflow
template. Installing it in the owning Arduino branches is still pending;
the template is not an active Arduino workflow.

## Run the tooling

From a PokemonSimulation checkout with these modules, set `PYTHONPATH=src`.
`--source` names a Git checkout of `dayfine/PokemonSimulation`; Git object reads
use the selected commit, never uncommitted source files. `--consumer` names an
Arduino checkout or an empty directory for a trial export.

```sh
# Read-only preflight before adopting a pin; select a full publishing SHA.
PYTHONPATH=src python3 -m pokesim.maps.consumer_sync compare \
--source . --consumer /path/to/Arduino-Source \
--game bdsp --revision FULL_PUBLISHING_COMMIT

# Initialize matching/empty exports, or advance previously clean locked packs.
PYTHONPATH=src python3 -m pokesim.maps.consumer_sync update \
--source . --consumer /path/to/Arduino-Source \
--game bdsp --revision FULL_PUBLISHING_COMMIT

# Check every locked game, without changing consumer files.
PYTHONPATH=src python3 -m pokesim.maps.consumer_sync check \
--source . --consumer /path/to/Arduino-Source
```

Repeat `--game` to initialize both FRLG and BDSP together. All existing map
directories must be locked; removing a game from the lock does not exempt its
maps. Games can subsequently advance independently.

The default command refreshes source `origin/main` before verifying ancestry.
`--offline` explicitly trusts the cached `origin/main` ref and objects; it
does not establish their freshness or authenticate a manually fabricated cache.
The origin URL is restricted to PokemonSimulation. Shallow caches cannot be
refreshed by this tool; use a full-history checkout. An unmerged publishing
commit, mutable revision, missing object or unsupported format fails.

`check` fails on altered bytes, missing/extra maps, symlinks, lock/manifest
inconsistency, unlocked game directories and retired authoring modules.
`compare` diagnoses a proposed pin without creating a lock or changing exports;
it is not a substitute for the CI `check` command.

`update` validates the published source and current exports first. It will
not overwrite a pending local correction, drop an unknown file, or delete a
duplicate generator. Initial adoption accepts only empty directories or
byte-identical copies. For an existing lock, current packs must pass integrity
before an update can replace them. Files are staged as complete game directories;
ordinary swap errors roll back, and the lock is replaced last. A process kill
during swaps can require recovery; integrity fails rather than reporting success
under an inconsistent lock. Restore exports from the previous locked revision
before retrying. Review and commit exports and lock together.

## Activate in the owning consumer branches

1. Review pending map differences in PokemonSimulation first. In particular,
`compare` against the #40 publishing baseline flags #226's five modified
BDSP maps; this implementation does not approve or import those corrections.
Do not erase them to obtain a passing check.
2. Remove the duplicate authoring modules in their owning Arduino PRs, after
migrating dependent tools/tests. For BDSP these are `bdsp_maps.py` and
`decomp.py`; for FRLG, `frlg_maps.py`. The legacy `devtools/maps/__main__.py`
imports `decomp` and exposes its CLI, so update that dependency before
removing the parser. Keep unrelated drawing helpers and native readers.
3. Use `update` to create `devtools/maps/map-packs.lock.json` and the exact
selected packs. Consumer destinations are fixed to the existing CMake map
paths. No C++ reader, embedding or build-network change is necessary.
4. Add these narrow rules to the consumer's `.gitattributes` before checking
out/exporting on Windows; verify the resulting bytes with `check`:

```gitattributes
SerialPrograms/Source/PokemonBDSP/Programs/AutoStory/Maps/*.json text eol=lf
SerialPrograms/Source/VideoGames/PokemonFRLG/Overworld/Maps/*.json text eol=lf
```

5. Copy `map-pack-integrity.yml.example` to
`.github/workflows/map-pack-integrity.yml`, replacing
`IMPLEMENTATION_COMMIT` with the full merged PokemonSimulation commit that
contains this action. The workflow runs for drafts too. Default PR checkout
checks the merge result; the action obtains full approved publishing history
independently and compares it to the consumer lock.
6. Run native reader/story tests against the selected exports and configure
`map-pack-integrity` as a required check on the actual merge target branches.
Apply the same gate to owning map PRs and their stacked story branches;
merely adding a check on fork main does not protect another stack's target.
Record this ownership policy in the Arduino agents instructions. Do not
claim enforcement until the workflow and required status are both active.

The action runs the verifier from the pinned action commit and reads map data
from the independently checked-out publishing repository. Consumer-local
manifests or builder output are never its reference. Builds still use vendored
JSON and need no network access.

## Validation of the implementation

Colocated tests use real temporary Git repositories and cover drift in terrain,
points, NPCs, triggers, patches and metadata; inventory changes; forged local
manifest/hash; unmerged/mutable pins; symlinks; retired generators; independent
game updates; and rollback on a failed second directory swap. Simulation CI
also exercises the composite action with a complete real published export and
confirms that a subsequent local edit is rejected.
19 changes: 19 additions & 0 deletions integrations/arduino-source/map-pack-integrity.yml.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Install as .github/workflows/map-pack-integrity.yml in the owning Arduino PR.
# Replace IMPLEMENTATION_COMMIT with this implementation's full merged commit.
name: Map pack integrity
on:
pull_request:
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
jobs:
map-pack-integrity:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- uses: dayfine/PokemonSimulation/integrations/arduino-source/verify-map-packs@IMPLEMENTATION_COMMIT
23 changes: 23 additions & 0 deletions integrations/arduino-source/verify-map-packs/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: Verify PokemonSimulation map packs
description: Check vendored Arduino JSON against its pinned publishing commits.
runs:
using: composite
steps:
- name: Fetch approved map publishing history
uses: actions/checkout@v4
with:
repository: dayfine/PokemonSimulation
ref: main
fetch-depth: 0
path: .pokesim-map-source
persist-credentials: false
- name: Check locked map exports
shell: bash
working-directory: ${{ github.action_path }}/../../..
env:
PYTHONPATH: ${{ github.action_path }}/../../../src
run: >-
python3 -m pokesim.maps.consumer_sync check
--consumer "$GITHUB_WORKSPACE"
--source "$GITHUB_WORKSPACE/.pokesim-map-source"
--offline
Loading
Loading