diff --git a/AUDIT.md b/AUDIT.md new file mode 100644 index 0000000..529f6f6 --- /dev/null +++ b/AUDIT.md @@ -0,0 +1,176 @@ +# System Audit and Remediation — verdi + +Started 2026-11-03 on branch `ci/pypi-release` (base `4664c96`). All changes are +uncommitted working-tree edits; nothing was pushed or published. + +## Verification commands + +| Check | Command | Baseline | Final | +|---|---|---|---| +| Lint (CI gate) | `ruff check .` | clean | clean | +| Tests (CI gate) | `python -m pytest` | 672 passed, 1 skipped | 769 passed, 1 skipped | +| Types (not a CI gate) | `mypy src/verdi --ignore-missing-imports` | 73 errors | 66 errors (all triaged, see R7) | +| Packaging | `pip wheel . --no-deps` + fresh venv, `verdi --help`, `verdi init`, `verdi sweep … --dry-run` | worked | not re-run (no packaging config changed) | +| Formatting | `ruff format --check .` | 156 files would change | unchanged — not enforced by CI; mass reformat deliberately skipped | + +Secret scan of tracked files and history (`sk-…`, `AKIA…`, `Bearer …`): only +obvious test fixtures (`tests/audit/test_judge_live.py`, `tests/experiments/test_g5_close.py`). +`.env` is git-ignored and was never committed. + +## Method + +Pass 1: four independent read-only reviewers (world/governance/schemas/scheduler; +society/agents/pilot/config; analysis/measurement/audit; CLI/experiments/packaging/docs) +plus mypy triage. Every finding was re-checked against the code before fixing. +Pass 2: an independent reviewer over the diff (regressions) and a fresh reviewer over +areas pass 1 only skimmed (pilot, g5_*, paired_run, measurement, adapters, recorded). +Each fix has a regression test that was confirmed to FAIL on the pre-fix source +(`git stash push -- src/…`) and pass after. + +New tests live in `tests/*/test_audit_regressions.py` (agents, world, governance, +society, analysis, experiments). `tests/test_cli.py` was updated (see F22). + +## Fixed findings (most severe first) + +| # | Sev | Where | Defect | Fix | +|---|---|---|---|---| +| F1 | Critical | `world/reducer.py` `_transfer`, `world/validate.py` | Transfer to self minted resources (recipient read before sender debit written back): `food=10`, `transfer{to:self,units:10}` → 20. Validator accepted it, so an LLM could do it every tick. | Validator rejects with new non-sanctionable code `self_transfer`; reducer raises `InvariantViolationError` as defence in depth. | +| F2 | High | `governance/constitution.py` | Two amendments enacted in one assembly: second was built from the shared submission base, silently dropping the first's rules. | New version = delta applied onto `active_at(effective_tick)`; vote rules still from the submission base. | +| F3 | High | `governance/constitution.py` | Caller-supplied votes counted raw: no dedup per voter, wrong-proposal votes counted, ballots `cast_vote` had rejected still counted; ineligible voters decided the threshold. | Recorded ballot is the single source of truth; caller votes go through `record_vote`; tally uses the same eligibility filter as quorum. | +| F4 | High | `governance/constitution.py` | Out-of-range `quorum`/`amendment_threshold` in a payload: amendment recorded as valid, then `ConstitutionVersion` raised, leaving store inconsistent. | Validated as fractions in (0, 1] → `invalid_quorum` / `invalid_amendment_threshold`. | +| F5 | High | `governance/sanctions.py`, `appeals.py`, `society/campaign.py` | One revocation per (agent, right): expiring/overturning one of two overlapping sanctions restored the right while the other was active; campaign cleared `under_sanction` on any expiry. | `RightsView` keeps all intervals; `SanctionEngine.end()` re-revokes from remaining live sanctions; campaign keeps the flag while `has_live_sanction`. | +| F6 | High | `agents/client.py` | 200 response with non-JSON body raised raw `JSONDecodeError` (not `ModelError`), aborting whole campaign batches; `"content": null` propagated `None` → `TypeError`/`AttributeError` in policy parsing, reflection, consolidation. | Parse inside the guarded block; `None` → `""`; non-string content → `ModelError`. | +| F7 | High | `analysis/comparative_report.py` | Enactment row used `f"…".format()` after inserting rule values: dict values crashed the report (`KeyError`), `{v}` in values was substituted, values double-escaped. | Single f-string, each value escaped once. | +| F8 | High | `analysis/outcome_metrics.py` `recovery_time` | Declared due ticks zipped with outcome events in YAML order, but the campaign evaluates in due-tick order; shipped `examples/full.yaml` declares `[12, 30, 46, 30]` → outcomes mislabelled. Manifest + resolved YAML ticks were also appended twice, forcing the raw-tick fallback; YAML errors crashed. | Resolved YAML replaces manifest ticks; ticks sorted (stable) before zip; YAML errors handled. | +| F9 | High | `experiments/sweep.py` | Ctrl-C: `with ThreadPoolExecutor` drained the queue, *starting* every queued run (spending model calls), then marked finished runs "interrupted". | Manual executor; `shutdown(cancel_futures=True)` on interrupt; finished-but-unrecorded results kept. | +| F10 | High | `experiments/sweep.py` | Resume reused "complete" runs by `run_id` only (a changed `--ticks` was ignored); runs from earlier invocations leaked into results, summary and exit code. | Per-run `scenario_sha256`; mismatch reruns; `current_runs` filters `SweepResult`, `collect()` and the exit code. Legacy entries without a hash are trusted as before. | +| F11 | High | `experiments/interview_experiment.py` | Post-pass audit always crashed: `asyncio.run()` given the dict returned by synchronous `audit_from_file`. | Call it directly. | +| F12 | Medium | `interview_experiment.py` | Post-pass audit ignored the scenario `model:` block (wrong/missing endpoint). | Uses `resolve_endpoint_config(load_scenario(...))`. | +| F13 | Medium | `config.py`, `g5_pairs.py`, `society/live.py` | Leak guards only checked `MODEL_API_KEY`/`VERDI_VLLM_API_KEY`; a key read via `model.api_key_env` was never scanned for. `ModelEndpointConfig.__repr__` printed the key. | Every resolved key is tracked for `api_key_values()`; G5 artifact audit scans all keys; `api_key` is `repr=False`. | +| F14 | Medium | `config.py` | `max_retries < 0` made every call raise `AssertionError("unreachable")`; zero timeout / concurrency accepted. | Range-checked with readable `ValueError`. | +| F15 | Medium | `society/campaign.py`, `memory.py` | Consolidation prompt's "existing summary" contained every recent entry (string split on the rendered history). | `AgentMemory.consolidated_summary()`. Note: changes consolidation prompt text (see R4). | +| F16 | Medium | `society/tasks.py`, `campaign.py` | One bad field (`"amount": null`) discarded the whole orchestrated tape; invalid transfer targets produced tasks the validator must reject; failed-parse calls were never logged, so token accounting undercounted. | Per-field fallback; invalid transfer → produce; meta returned with `parse_error` and recorded. | +| F17 | Medium | `society/live.py`, `agents/client.py` | Stall watchdog: reflection/consolidation write `model_calls.jsonl` only after the whole batch, so a long healthy batch could trip `os._exit(3)`; retry window ignored pool/connect timeouts and backoff. | In-process `completed_calls` counter feeds the watchdog; window includes pool + connect + max backoff per attempt. | +| F18 | Medium | `world/branch.py` | Branching after a prefix ending in a rejected event always failed (`after_hash=None`). | Terminal hash = `after_hash or before_hash`. | +| F19 | Medium | `world/policy.py`, `validate.py` | `max_produce_units_per_tick` was a per-action cap without an economy block. | Cumulative using `produced_this_tick` when supplied. | +| F20 | Medium | `measurement/longitudinal.py` `_parse_choice` | `"Yes."`, `'"yes"'`, `"Yes, I would."` scored as the low pole; Likert took the first digit anywhere (`"0 to 5 … 4"` → 0, `"10"` → 1, `"7 out of 10"` → crash of the whole probe round). | First-word match for word options; last standalone in-range integer for Likert; anything else flagged unparsed instead of raising. | +| F21 | Medium | `experiments/paired_run.py` | Missing `output_root` → `TypeError`; `except BaseException` swallowed Ctrl-C and continued; resume ignored scenario changes; `--concurrency` help promised `VERDI_MAX_CONCURRENCY` but code hard-coded 4. | Readable error; `except Exception`; per-arm `scenario_sha256`; default honours the env var. | +| F22 | Medium | `cli.py` | `verdi run x.yaml --run-dir=out` ignored the `=` form; bare `verdi` printed a pointer to a non-existent `docs/IMPLEMENTATION_PLAN.md`. | Both forms detected; bare `verdi` prints help (exit 0 kept for compatibility). Test updated to assert the help text instead of the stale string. | +| F23 | Medium | `pilot/driver.py` | Sanction evidence could cite a quarantine rejection of the subject's proposal (false sanction); default live capture dir had 1-second resolution (two runs could share one `live_tape.jsonl`). | Evidence only from the scripted violation; dir stamp has microseconds + pid. Offline canonical path unchanged. | +| F24 | Low | `world/validate.py` | Non-string `resource` (list/dict/null from model JSON) crashed with `TypeError`. | Rejected as `missing_argument`. | +| F25 | Low | `world/snapshot.py` | `load` never re-hashed content; digest used unvalidated as a filename (`../`). | Digest must be 64 hex; content hash verified on load. | +| F26 | Low | `governance/constitution.py` | A proposal could be enacted twice; version names could collide with an existing one and be silently skipped. | `ProposalNotInVoteError` on re-enact; unique version names. | +| F27 | Low | `world/economy.py` | `role_consumption` for a resource not in `consumption_per_tick` was silently ignored. | Rejected with a clear config error (shipped examples unaffected). | +| F28 | Low | `audit/live_bridge.py` | Scorer `ValueError` recorded as score 0.0 (worst), indistinguishable from judge disagreement. | `None` + `rubric_score_error`. | +| F29 | Low | `experiments/sweep.py` | Dot-only labels (`..`) escaped the sweep folder; bad sweep YAML / `vary: null` gave tracebacks; setup I/O errors aborted the sweep. | Labels sanitised; readable `SweepError`s; setup inside the per-run guard. | +| F30 | Low | `analysis/cross_run_compare.py` | `"tick": null` in provenance crashed (`int(None)`). | Non-numeric ticks skipped. | +| F31 | Medium | `experiments/sweep.py` (regression from F10, found by pass-2 diff review; also pre-existing for failed-run retries) | Relaunching a run into its old directory appended a second ledger to `events.jsonl` / `model_calls.jsonl` (two `event_id: 0`, broken hash chain). | Previous non-empty attempt is renamed to `.prev-` (never deleted) before relaunch. | +| F32 | Low | `analysis/outcome_metrics.py` (regression from F8) | Malformed `due_tick` (e.g. `soon`) raised instead of degrading, in both the resolved-YAML and manifest paths. | Falls back to raw ledger ticks as before. | +| F33 | High | `society/campaign.py` (found pass 3) | Production cap is documented per agent per tick (`economy.py`, `examples/full.yaml`, `institutional_recovery_constrained.yaml`), but the campaign summed production across ALL agents: once one agent hit the cap, every later producer that tick was rejected with `law:production_limit`. Also completes F19 (the cumulative constitutional cap was never fed in runs without an economy block). | `produced_this_tick` keyed by actor, tracked with or without an economy block. Campaign-level test in `tests/society/test_economy_campaign.py`. **Changes outcomes of economy runs vs earlier results.** | +| F34 | High | `audit/judge_live.py` `KEY_MATERIAL_PATTERNS` (found pass 3) | Leak guard matched ordinary speech (`Bearer of bad news`, `Authorization: the assembly approved…`): one such interview answer aborted `_write_artifacts` mid-way (truncated `interviews.jsonl`, all later artifacts lost) and judge tape writes. The guard's exception also echoed the matched secret into logs/findings. | Bearer / api_key / authorization patterns require a token-shaped value (all real key shapes still flagged, see tests); messages show only a 6-char prefix + length. `tests/experiments/test_g5_close.py::test_b2_planted_key_material_fails_leakage` updated to assert the finding locates the leak and does **not** contain the key (stricter, not weaker). | +| F35 | Medium | `measurement/interview_values.py` `_nearest_event` (pass 3) | Returned the agent's EARLIEST event, and compared lifecycle ticks (0–29) with ledger transition ticks (1–373): every trajectory row linked to the agent's first action (checked on `runs/interview-12agent-seed42`). | Lifecycle tick from the event's task id (`soc-tNN-`), carried forward for events without one; latest match wins. | +| F36 | Medium | `experiments/g5_close.py` (pass 3) | One judge rationale containing "causally" made the G5-close report unbuildable (frozen causal-word guard ran over verbatim model text). | Guard runs on the report with model-authored `rationale` / `raw` fields blanked. | +| F37 | Low–Med | `audit/interview_audit.py` (pass 3) | SimpleAudit reports scenario/judge failures as `severity="ERROR"` without raising; rows had `error: None`, so summaries reported 0 errors when the endpoint was down. | `severity == ERROR` sets `error`. | +| F38 | Medium | `measurement/longitudinal.py` (regression in F20, pass 3) | First F20 version rejected `"4."`/`"4.0"` and picked the scale top from `"4/5"`, `"4 out of 5"`, `"(scale 0-5)"`. | Scale ranges and denominators removed before taking the last standalone integer. | +| F39 | Medium | `experiments/paired_run.py` (pass 3) | Re-running a failed/stale arm appended into the old run dir (same root cause as F31). | Shares `_set_aside_previous_attempt`. | +| F40 | High | `analysis/outcome_metrics.py` `exclusion_episodes` (pass 4) | Open episodes ran to the last LEDGER tick (transition counter) while sanction ticks are LIFECYCLE ticks: `runs/interview-6agent-god-orch` grace sanctioned at t19 of 20 got `excluded_ticks: 85`. Overlapping sanctions were summed (dave 7 instead of 6) and flagged as repeat episodes. | Run end from `resources.jsonl` (ledger tick only as fallback); intervals merged per agent; `repeat_episode` = more than one merged interval. Now grace 0, dave 6. | +| F41 | Medium | `society/campaign.py` `_compute_metrics`, `live.py` (pass 4) | `metrics.json` `sanctions` counted world shocks and rejected impositions; `appeals` counted filed + adjudicated rows (6 for 3 appeals). Flows into sweep `summary.csv`. | Count accepted `sanction_imposed` and `phase == filed` only. | +| F42 | Medium | `society/prompting.py` (pass 4) | Action prompt said "It is now tick 104" in a 20-tick run (ledger counter), while durations/deadlines/caps count lifecycle ticks. | Uses `task.tick_created`; `SOCIAL_PROMPT_VERSION` bumped to `/5` (prompt text changes). | +| F43 | Low | `analysis/comparative_report.py` (pass 4) | Undefined baseline (no objective due in the preregistered window) was labelled "CENSORED — never recovered". | Labelled "not evaluable"; preregistered thresholds untouched (see R14). | +| F44 | Medium | `measurement/longitudinal.py` (regression in F38, pass 4) | "Last integer wins" let trailing reasoning override the answer (`"Choice: 4 … 2 factors"` → 2); JSON `{"choice": 4}` (int) fell through to free-text parsing. | First in-range standalone integer; int `choice` accepted. | +| F45 | Low–Med | `audit/judge_live.py` (regression in F34, pass 4) | Narrowed patterns missed `Authorization: Basic …`, digit-free keys, `"api_key": "…"` (JSON) and `x-api-key`. | Optional scheme word, 20+-char digit-free values, quote before separator, `x-api-key`. Zero hits over 398 real run artifacts + the plotly report bundle; ordinary speech still passes. | +| F46 | Low | `experiments/g5_close.py` (pass 4) | Quarantined rows' `error` embeds the model's stance verbatim, still able to trip the causal guard. | `error` added to excluded model-text keys. | +| F47 | Medium | `analysis/outcome_metrics.py` `completion_rate` (pass 5) | `due_tick` was the outcome event's LEDGER tick (`runs/smoke_final`: 56/127/198 for declared 10/24/38); shown in the comparative report and sweep/pair CSVs. | Declared due tick from `scenario_resolved.yaml` by objective id, else lifecycle tick carried from task ids; raw value kept as new `ledger_tick` field (schema test updated for the added field). | +| F48 | Medium | `analysis/outcome_metrics.py` `recovery_time` (exposed by F47) | A 0.0 pre-shock baseline gave threshold 0, so a run where nothing ever completed "recovered". `test_recovery_time_recovers_on_completed_post_delivery` only passed before because it fed the ledger tick as the due tick. | Zero baseline treated like an undefined one (not evaluable → censored). | +| F49 | Medium | `society/inner_life.py` (pass 5; site F42 missed) | Inner-life prompt also said "It is now tick "; enabled in both shipped full scenarios. | Lifecycle tick passed from the worker; `INNER_LIFE_PROMPT_VERSION` → `/2`. | +| F50 | Low–Med | `society/campaign.py` `_RecordingCaller` (pass 5) | Probe rows in `model_calls.jsonl` got their subject in COMPLETION order (parallel calls): `runs/smoke_final` had 14/24 probe rows attributed to the wrong agent (tokens/latency, HTML agent filter). | Subject taken before the await (send order). | +| F51 | Low | `audit/judge_live.py` (pass 5, regression in F45) | Case-insensitive `Bearer` + 16 hyphenated chars flagged prose ("standard-bearer counter-proposals"). | Case-sensitive `Bearer` followed by a digit-bearing 16+ token. | +| F52 | Medium | `world/validate.py` (pass 6) | Two accepted `produce 1e308` pushed a balance to `inf`; next tick `int(inf)` in `society/tasks.py` raised `OverflowError` and aborted the run (any scenario without an economy, incl. `starter.yaml`). | Any action whose destination balance would become non-finite is rejected (non-sanctionable code). | +| F53 | Low | `world/validate.py` (pass 6) | `"amount": true` accepted as 1 unit. | Booleans are not quantities. | +| F54 | Medium | `world/economy.py` `parse_task_objectives` (pass 6) | Duplicate objective ids (incl. auto `obj-NN` colliding with an explicit id) overwrote another objective's baseline, flipping completed → failed. | Duplicate ids rejected at load. | +| F55 | Medium | `analysis/scenario.py` `build_t0_state` (pass 6) | Values that load fine then crash deep into a live run: `target_pool` typo (crash at the due tick, after 57 calls), negative / NaN t0 resources, non-integer shock ticks (silently skipped), NaN/inf shock deltas (NaN silently zeroed the resource), empty agent list. | Load-time `ScenarioLoadError`s. All shipped scenarios (examples/, experiments/, tests/, src/) still load. | +| F56 | Medium | `analysis/outcome_metrics.py` `completion_rate` (pass 6, refinement of F47) | Task-id carry-forward lagged a tick when every action of the due tick was quarantined. | Under an economy, lifecycle tick = end-of-tick `consumption` events so far − 1 (exact); task ids only as fallback. | +| F57 | Low | `analysis/comparative_report.py` (pass 6, completes F43/F48) | A 0.0 baseline still printed "CENSORED — never recovered". | Labelled "not evaluable — pre-shock baseline rate is 0". | + +## Remaining risks / not fixed (with justification) + +| # | Sev | Where | Issue | Why not fixed | +|---|---|---|---|---| +| R1 | High | `measurement/probes.py:128-137` | Value axis is `names[i % 5]` but template is `templates[i % 3]`: the "solidarity" item text says "This is about cooperation"; private items 3–4 repeat dilemmas under different axes. | Changes the measurement instrument. Frozen pilot tapes must replay byte-for-byte and existing runs would stop being comparable. Needs an owner decision and an `instrument_version` bump. | +| R2 | High | `measurement/probes.py:125`, `scoring.py:96` | Yes/no items always score yes=1.0, no=0.1 regardless of item polarity: "accept a secret side-deal? yes" scores integrity 1.0. | Same as R1: an instrument redesign (per-item `reverse_scored`) that changes historical scores. | +| R3 | Medium | `analysis/estimates.py:287, 377-389` | Undefined estimands become 0.0 (no actions → safety 0, no sanctions → false-punishment 0, all-zero Gini → 0); a missing pre-sanction level becomes 0.0, so the agent always "recovers". | The 0.0 convention is written into the preregistration addendum. Changing estimand definitions is a research decision; flag to the analysis owner. | +| R4 | Medium | `society/campaign.py` consolidation (F15) | Consolidation prompts now differ from before, so recorded/replayed runs that captured the old prompt text will not reproduce consolidation calls byte-for-byte. | Intended: the old prompt was wrong. Live runs are not byte-reproducible anyway. | +| R5 | Medium | `measurement/longitudinal.py` | Unparsed probe rows still enter drift calculations (they are flagged but not filtered). | Filtering changes the drift estimand; F20 makes unparsed rows much rarer. | +| R6 | Low | `world/replay.py:90-97`, `branch.py:206-220` | Generic `replay`/`branch_from_snapshot` fold every `TapeStep` as an action; campaign-only kinds (`rights_change`, `world_shock`, …) are not understood. The campaign has its own correct replay. | Only matters if someone feeds a campaign tape to the generic replay; no caller does today. A shared dispatcher is a refactor. | +| R7 | Low | mypy (66 errors) | All triaged: none found to be a runtime bug beyond those fixed. Mostly Optional narrowing and loose annotations. | mypy is not a CI gate; fixing annotations is churn with no behaviour change. | +| R8 | Low | `agents/pool.py`, `campaign.py:_run_bounded` | `asyncio.gather` without cancellation: one failing call leaves siblings running detached. | Moving to `TaskGroup` changes error semantics (ExceptionGroup); low impact since callers abort the run anyway. | +| R9 | Low | `society/memory.py` | `max_entries` is stored but never enforced (growth bounded only by consolidation, which the campaign always runs). | No path appends without consolidating today. | +| R10 | Low | `experiments/treatments.py:135-145` | `rights_restoration_treatment` reports `applied=True` with a tape identical to placebo. | Naming/semantics of a treatment definition; owner decision. | +| R11 | Low | `pilot/live.py:236` | ADR-010 call budget checked only after the run. | Pilot path is sequential and short; enforcing mid-run changes pilot semantics. | +| R12 | Low | `.github/workflows/*` | CI and release install with `pip install -e`, never testing the built wheel's `force-include` examples. | Workflow change best done by the maintainer; packaging was verified manually (wheel + fresh venv + `verdi init`). | +| R13 | Low | `ruff format` | 156 files not formatted. | Not enforced; reformatting would bury the substantive diff. | +| R14 | High | `analysis/comparative_report.py:199-205` | Preregistered recovery criterion (baseline window ticks 0–10, post-shock ≥14, sustained 3 ticks) cannot be evaluated for the shipped `institutional_recovery_constrained.yaml` (due ticks 12, 30, 30, 46): no baseline objective, and 3 consecutive objective-bearing ticks never occur. Every run is "not evaluable" (was mislabelled CENSORED, F43). | The thresholds are explicitly preregistered ("defined BEFORE reading results"); changing them is a research decision. Options: amend the preregistration, or change the scenario's due ticks. | +| R15 | Low | `examples/full.yaml:56-57`, `docs/SCENARIOS.md` | `sanction_policy.max_duration_ticks` and `appeal_policy.deadline_ticks` are inert in campaign runs (fixed 3-tick sanctions; deadline always set explicitly). | Either wire them into `campaign.py` (changes run outcomes) or document them as inert; owner decision. | +| R16 | Low | `analysis/outcome_metrics.py` `recovery_time` | Reports `censored=True` when the baseline is undefined (tests pin this). | Report layer now distinguishes it (F43); changing the metric's contract is an analysis decision. | +| R17 | Low | `analysis/scenario.py`, `campaign.py` | Shocks, assemblies and deliveries scheduled at or after `total_ticks` are silently ignored (a late delivery drops out of `completion_rate`'s denominator). | Not made an error: `verdi sweep --ticks N` deliberately shortens runs below declared ticks. A warning (or reporting dropped objectives in metrics) is the right follow-up. | +| R18 | Low | `analysis/outcome_metrics.py` `exclusion_episodes` | An episode still open at run end is counted up to, not including, the last lifecycle tick (an agent sanctioned only on the final tick shows 0). | Convention pinned by the existing `test_exclusion_repeat_episode_flag_synthetic`; owner decision. | +| R19 | Low | `measurement/longitudinal.py` `_parse_choice` | An answer given after reasoning ("Considering 2 factors, I choose 4") parses as 2. | Same as the original first-digit rule; the prompt asks for a single digit. A structured `{"choice": N}` reply is parsed exactly. | +| R20 | Low | `society/memory.py` | `AgentMemory.from_checkpoint` ignores the stored `max_entries`. | No effect today (R9; `from_checkpoint` unused, `checkpoint.json` write-only). | + +## Behaviour changes researchers should know about + +These fixes intentionally change results compared with runs made before the audit: + +- **F33** per-agent production caps: economy runs accept produces that were rejected before. +- **F42 / F49** prompt text: the action prompt is now `verdi.society-prompt/5` and the inner-life prompt is `verdi.inner-life/2`. They show the lifecycle tick. +- **F15** consolidation prompt text. +- **F16** orchestrator fallbacks. +- **F1 / F52 / F53** validator now rejects self-transfers, overflowing amounts and boolean amounts. +- **F3** ineligible ballots are no longer counted. +- **F2** stacked amendments now apply together. +- **F5** overlapping sanctions are kept separate. +- **Metrics:** F40 and F41 (`metrics.json` counts and `excluded_ticks`), and F47 and F48 (`due_tick`, zero-baseline recovery). + +Recorded tapes still replay, because replay re-applies the recorded validations. Comparisons with older runs should account for these changes. + +## Verification results (final) + +- `ruff check .`: clean. +- `python -m pytest`: **769 passed, 1 skipped** (baseline 672 passed, 1 skipped). Of the 97 new tests, almost all live in `tests/*/test_audit_regressions.py`, plus one campaign-level test in `tests/society/test_economy_campaign.py`. Each was confirmed to fail on the pre-fix source. +- **Existing tests changed (3):** + - `tests/test_cli.py`: the stale-message assertion now checks the help output. + - `tests/experiments/test_g5_close.py::test_b2_planted_key_material_fails_leakage`: now asserts the finding locates the leak **without** repeating the key. This is stricter. + - `tests/analysis/test_outcome_metrics.py::test_completion_rate_well_formed`: adds the new `ledger_tick` field to the pinned key set. + - No test was deleted or skipped, and no assertion was weakened. +- `mypy`: 66 errors (baseline 73). All were triaged as typing-only (R7). +- Every shipped scenario YAML still passes `build_t0_state`. +- The key-material patterns produce 0 hits over 398 real run artifacts plus the bundled plotly report. + +## Audit passes and convergence + +| Pass | Scope | New actionable findings | +|---|---|---| +| 1 | 4 area reviewers + mypy triage | ~30 (1 critical, 10 high) | +| 2 | Diff review + lightly covered areas | 2 regressions + 9 new (4 high, all measurement-instrument; R1/R2) | +| 3 | Diff review + full read of pattern-scanned files | 3 regressions + 4 new (1 high: F34) | +| 4 | Diff review + end-to-end data-flow sweep | 3 regressions + 6 new (2 high: F40, R14) | +| 5 | Diff review + tick-unit / metrics sweep | 2 low regressions + 3 medium | +| 6 | Diff review + edge-input / scenario-validation probing | 2 low/medium refinements + 5 medium/low (no high) | + +**The stopping criterion was not formally met.** That criterion is two consecutive independent passes with no new actionable findings, and pass 6 still found medium-severity input-validation issues. All of them were fixed in this session. + +Severity has fallen with each pass. No unresolved critical issue remains. The only unresolved high-severity items are R1, R2 and R14. Each is a measurement-instrument or preregistration decision that needs the research owner and is documented above with evidence. + +## Continuation instructions + +1. Decide R1, R2 and R14 with the research owner: the probe-bank axis and polarity, and the preregistered recovery window versus the shipped scenario. Bump `instrument_version` if the probe bank changes. +2. Run two more independent passes. Suggested angles that have not been tried yet: + - The adapters (`adapters/concordia*.py`) under real Concordia objects. + - `html_report.py` with large runs (size and performance). + - Concurrency soak (`tests/agents/test_client_soak.py`) at higher fan-out. + - The g5 pipeline end to end on a real `runs/` root. +3. Optional hygiene, with no behaviour change: + - Fix the 66 mypy annotations and add mypy to CI. + - Make the PyPI workflow test the built wheel (R12). + - Do a separate `ruff format` commit (R13). +4. All work is uncommitted on branch `ci/pypi-release` (37 files changed). Split it into reviewable commits (world/governance, society, analysis/measurement, experiments/CLI, tests) on a fresh branch before opening a PR. diff --git a/pyproject.toml b/pyproject.toml index 560c716..8305291 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "verdi-agents" -version = "0.1.1" +version = "0.2.0" description = "Auditable testbed for temporal value expression and institutional recovery in LLM agent societies" readme = "README.md" license = "MIT" diff --git a/src/verdi/__init__.py b/src/verdi/__init__.py index de1e006..4450c43 100644 --- a/src/verdi/__init__.py +++ b/src/verdi/__init__.py @@ -1,3 +1,3 @@ """Verdi — auditable testbed for value expression and institutional recovery in LLM agent societies.""" -__version__ = "0.1.1" +__version__ = "0.2.0" diff --git a/src/verdi/agents/client.py b/src/verdi/agents/client.py index 3c23e10..e7bd7f1 100644 --- a/src/verdi/agents/client.py +++ b/src/verdi/agents/client.py @@ -25,6 +25,8 @@ RETRYABLE_STATUS = (429, 502, 503, 524) _BACKOFF_BASE = 0.5 _BACKOFF_CAP = 10.0 +_CONNECT_TIMEOUT = 15.0 +_POOL_TIMEOUT = 30.0 class ModelError(RuntimeError): @@ -66,6 +68,11 @@ def __init__( ) -> None: self._config = config self._rng = rng or random.Random() + #: Successful completions so far. A liveness signal for the stall + #: watchdog: batched phases (reflection, consolidation) only write + #: model_calls.jsonl after the whole batch, so file lines alone can + #: stay flat for a long, healthy batch. + self.completed_calls = 0 self._client = httpx.AsyncClient( base_url=config.base_url, headers={"Authorization": f"Bearer {config.api_key}"}, @@ -80,8 +87,8 @@ def __init__( # socket) always trips the read timeout and becomes retryable, # rather than blocking indefinitely. timeout=httpx.Timeout( - connect=15.0, read=config.timeout_seconds, - write=config.timeout_seconds, pool=30.0, + connect=_CONNECT_TIMEOUT, read=config.timeout_seconds, + write=config.timeout_seconds, pool=_POOL_TIMEOUT, ), # Bounded pool sized to the concurrency we actually run at. Keepalive # REUSE is allowed (capped) so sockets are recycled rather than opening @@ -156,7 +163,9 @@ async def generate_structured( continue if response.status_code == 200: - return self._parse(response, trace_id) + result = self._parse(response, trace_id) + self.completed_calls += 1 + return result raise ModelError( f"request {trace_id} failed with HTTP {response.status_code}" @@ -169,15 +178,26 @@ async def generate_structured( @staticmethod def _parse(response: httpx.Response, trace_id: str) -> ModelResult: - body = json.loads(response.text) try: + body = json.loads(response.text) message = body["choices"][0]["message"] content = message["content"] - except (KeyError, IndexError, TypeError) as exc: + except (ValueError, KeyError, IndexError, TypeError) as exc: + # ValueError covers JSONDecodeError: a proxy/CDN can answer 200 with + # an HTML page. Surface it as the typed ModelError callers handle. raise ModelError( f"request {trace_id} returned a structurally invalid completion body", trace_id=trace_id, ) from exc + # Reasoning models may return ``"content": null`` (e.g. max_tokens spent + # on thinking). Normalise to "" so callers can rely on a str. + if content is None: + content = "" + elif not isinstance(content, str): + raise ModelError( + f"request {trace_id} returned non-string completion content", + trace_id=trace_id, + ) return ModelResult( content=content, request_id=str(body.get("id", "")), diff --git a/src/verdi/analysis/comparative_report.py b/src/verdi/analysis/comparative_report.py index 3887aa1..fb94b13 100644 --- a/src/verdi/analysis/comparative_report.py +++ b/src/verdi/analysis/comparative_report.py @@ -158,18 +158,19 @@ def _institution_rows(metrics: dict[str, Any]) -> list[str]: deltas = enactment.get("rule_deltas", {}) delta_bits = [] for key, added in (deltas.get("added") or {}).items(): - delta_bits.append(f"+{key}={_esc(added)}") + delta_bits.append(f"+{_esc(key)}={_esc(added)}") for key, change in (deltas.get("changed") or {}).items(): - delta_bits.append(f"~{key}:{_esc(change['from'])}→{_esc(change['to'])}") + delta_bits.append(f"~{_esc(key)}:{_esc(change['from'])}→{_esc(change['to'])}") for key in (deltas.get("removed") or {}): - delta_bits.append(f"-{key}") + delta_bits.append(f"-{_esc(key)}") + # One f-string, escaped once: a ``.format`` pass over already-inserted + # rule values would treat their braces (e.g. dict values) as fields. rows.append( - "Enacted amendment {v} @ tick {t}" + f"Enacted amendment {_esc(enactment['version'])}" + f" @ tick {_esc(enactment['tick'])}" f"{len(delta_bits)} rule change(s)" - f"{_esc(' '.join(delta_bits))} " - f"constitution_history.jsonl".format( - v=_esc(enactment["version"]), t=enactment["tick"] - ) + f"{' '.join(delta_bits)} " + f"constitution_history.jsonl" ) enacted_ids = mech.get("amendment_enacted_event_ids", []) rejected_ids = mech.get("amendment_rejected_event_ids", []) @@ -207,6 +208,13 @@ def _recovery_row(summary: dict[str, Any], arm: str, seed: Any) -> str: f"unavailable ({_esc(exc)})

") if rec["recovered"]: label = f"recovered at tick {rec['recovery_tick']}" + elif not rec["pre_shock_rate"]: + # No objective due in the preregistered baseline window (None), or a + # 0.0 baseline: the criterion is undefined -- not evidence of + # non-recovery (recovery_time still reports it censored). + label = ("not evaluable — no delivery objective is due in the " + "preregistered baseline window" if rec["pre_shock_rate"] is None + else "not evaluable — pre-shock baseline rate is 0") elif rec["censored"]: label = "CENSORED — never recovered by run end" else: diff --git a/src/verdi/analysis/cross_run_compare.py b/src/verdi/analysis/cross_run_compare.py index 99afb80..fee442e 100644 --- a/src/verdi/analysis/cross_run_compare.py +++ b/src/verdi/analysis/cross_run_compare.py @@ -126,7 +126,10 @@ def compare_runs(run_dirs: list[Path]) -> dict[str, Any]: round(sum(1 for c in consistent if not c) / len(consistent), 3) if consistent else None ), - "ticks_audited": sorted({int(r.get("tick", -1)) for r in prov}), + "ticks_audited": sorted({ + int(r["tick"]) for r in prov + if isinstance(r.get("tick"), (int, float)) and not isinstance(r.get("tick"), bool) + }), } runs.append({ "run_dir": str(rd), diff --git a/src/verdi/analysis/outcome_metrics.py b/src/verdi/analysis/outcome_metrics.py index a9cefaa..a25672c 100644 --- a/src/verdi/analysis/outcome_metrics.py +++ b/src/verdi/analysis/outcome_metrics.py @@ -38,6 +38,7 @@ import json import math import os +import re import sys from pathlib import Path from typing import Any @@ -141,6 +142,27 @@ def gini(values: list[float]) -> float | None: # 1. completion rate (objective task success) # --------------------------------------------------------------------------- # +_TASK_TICK = re.compile(r"-t(\d+)-") + + +def _declared_due_by_objective(run_dir: Path) -> dict[str, int]: + """objective_id -> declared due tick from ``scenario_resolved.yaml`` + ({} when absent or malformed).""" + try: + import yaml + + from verdi.world.economy import parse_task_objectives + + resolved = yaml.safe_load( + (run_dir / "scenario_resolved.yaml").read_text(encoding="utf-8")) + if not isinstance(resolved, dict): + return {} + return {o["objective_id"]: int(o["due_tick"]) + for o in parse_task_objectives(resolved.get("tasks"))} + except (OSError, ValueError, yaml.YAMLError): + return {} + + def completion_rate(run_dir: Path) -> dict[str, Any]: """Verified delivery-objective outcomes, straight off the event ledger. @@ -158,25 +180,49 @@ def completion_rate(run_dir: Path) -> dict[str, Any]: Returns:: {"objectives": [{objective_id, status, required, baseline, actual, - delta, due_tick, evidence_event_ids}], + delta, due_tick, ledger_tick, evidence_event_ids}], "overall": float | None, "completed": int, "failed": int, "evaluated": int} """ run_dir = _require_run_dir(run_dir) events = _read_jsonl(run_dir, "events") + declared_due = _declared_due_by_objective(run_dir) objectives: list[dict[str, Any]] = [] + # Lifecycle tick of each outcome: under an economy there is exactly one + # end-of-tick ``consumption`` event per lifecycle tick, written just + # before objectives are evaluated, so (consumption events so far) - 1 is + # exact even when a whole tick's actions were quarantined. Otherwise the + # tick is carried forward from task ids (soc-tNN-..). + has_consumption = any(e.get("kind") == "consumption" for e in events) + lifecycle = 0 + consumed = 0 for ev in events: + if has_consumption: + if ev.get("kind") == "consumption": + consumed += 1 + lifecycle = consumed - 1 + else: + m = _TASK_TICK.search(str((ev.get("payload") or {}).get("task_id") or "")) + if m: + lifecycle = int(m.group(1)) if ev.get("kind") not in ("task_completed", "task_failed"): continue payload = ev.get("payload") or {} + oid = str(payload.get("objective_id", "")) + ledger_tick = int(ev.get("tick", -1)) objectives.append({ - "objective_id": str(payload.get("objective_id", "")), + "objective_id": oid, "status": "completed" if ev["kind"] == "task_completed" else "failed", "required": payload.get("required"), "baseline": payload.get("baseline"), "actual": payload.get("actual"), "delta": payload.get("delta"), - "due_tick": int(ev.get("tick", -1)), + # Lifecycle due tick: declared in the frozen scenario when present, + # else the lifecycle tick of the actions just before the outcome + # (objectives are evaluated after that tick's actions). The + # event's own ``tick`` is the ledger transition counter. + "due_tick": declared_due.get(oid, lifecycle), + "ledger_tick": ledger_tick, "evidence_event_ids": [int(ev["event_id"])], }) objectives.sort(key=lambda o: (o["due_tick"], o["objective_id"])) @@ -325,7 +371,16 @@ def exclusion_episodes(run_dir: Path) -> dict[str, Any]: run_dir = _require_run_dir(run_dir) sanctions = _read_jsonl(run_dir, "sanctions") events = _read_jsonl(run_dir, "events") - last_ledger_tick = max((int(e.get("tick", -1)) for e in events), default=-1) + # Sanction ticks are LIFECYCLE ticks; events.jsonl ``tick`` is the + # reducer's transition counter (often 5x larger). The run's last lifecycle + # tick comes from resources.jsonl (one row per lifecycle tick); the + # ledger tick is only a fallback for runs without it. + resource_ticks = [r.get("tick") for r in _read_jsonl(run_dir, "resources")] + resource_ticks = [int(t) for t in resource_ticks if isinstance(t, (int, float))] + # Open episodes count up to (not including) the last observed tick -- + # the convention pinned by test_exclusion_repeat_episode_flag_synthetic. + run_end_tick = (max(resource_ticks) if resource_ticks + else max((int(e.get("tick", -1)) for e in events), default=-1)) agents: dict[str, dict[str, Any]] = {} totals = {"sanctions_imposed": 0, "appeals_filed": 0, @@ -360,9 +415,6 @@ def _agent(agent_id: str) -> dict[str, Any]: continue subject, episode = hit episode["restored_tick"] = int(rec.get("tick")) - entry = _agent(subject) - lo, hi = episode["effective_tick"], episode["restored_tick"] - entry["excluded_ticks"] += max(0, hi - lo) elif rtype == "appeal": # sanctions.jsonl appeal records: kind appeal_filed (phase "filed") # and adjudication events with kind appeal_upheld / appeal_dismissed @@ -373,19 +425,24 @@ def _agent(agent_id: str) -> dict[str, Any]: elif kind == "appeal_upheld": totals["appeals_upheld"] += 1 - # Open episodes: excluded until the last ledger tick (inclusive count). - for subject, episode in open_episodes.values(): - entry = _agent(subject) - hi = max(episode["effective_tick"], last_ledger_tick) - entry["excluded_ticks"] += max(0, hi - episode["effective_tick"]) del open_episodes out_agents: dict[str, Any] = {} for aid in sorted(agents): entry = agents[aid] entry["episodes"].sort(key=lambda e: (e["effective_tick"], e["sanction_id"])) - closed = [e for e in entry["episodes"] if e["restored_tick"] is not None] - entry["repeat_episode"] = len(closed) > 1 or len(entry["episodes"]) > 1 + # Overlapping sanctions are ONE exclusion: merge intervals before + # counting (open episodes run to the last lifecycle tick). + merged: list[list[int]] = [] + for e in entry["episodes"]: + lo = e["effective_tick"] + hi = e["restored_tick"] if e["restored_tick"] is not None else max(lo, run_end_tick) + if merged and lo <= merged[-1][1]: + merged[-1][1] = max(merged[-1][1], hi) + else: + merged.append([lo, hi]) + entry["excluded_ticks"] = sum(max(0, hi - lo) for lo, hi in merged) + entry["repeat_episode"] = len(merged) > 1 out_agents[aid] = entry return {"agents": out_agents, "totals": totals} @@ -457,7 +514,11 @@ def recovery_time( scenario = manifest.get("scenario") if isinstance(manifest.get("scenario"), dict) else {} for delivery in (scenario.get("tasks") or {}).get("deliveries", []) or []: if isinstance(delivery, dict) and delivery.get("due_tick") is not None: - due_ticks.append(int(delivery["due_tick"])) + try: + due_ticks.append(int(delivery["due_tick"])) + except (TypeError, ValueError): + due_ticks = [] # malformed: degrade to raw ledger ticks + break # The campaign writes the frozen scenario it actually ran next to the # artifacts; that is the authoritative source of declared due ticks. try: @@ -465,11 +526,22 @@ def recovery_time( resolved = yaml.safe_load( (run_dir / "scenario_resolved.yaml").read_text(encoding="utf-8")) - for delivery in (resolved.get("tasks") or {}).get("deliveries", []) or []: - if isinstance(delivery, dict) and delivery.get("due_tick") is not None: - due_ticks.append(int(delivery["due_tick"])) - except (OSError, ValueError): - due_ticks = [] + except (OSError, ValueError, yaml.YAMLError): + resolved = None + if isinstance(resolved, dict): + # Authoritative: replaces (never appends to) manifest-derived ticks. + try: + due_ticks = [ + int(delivery["due_tick"]) + for delivery in (resolved.get("tasks") or {}).get("deliveries", []) or [] + if isinstance(delivery, dict) and delivery.get("due_tick") is not None + ] + except (AttributeError, TypeError, ValueError): + due_ticks = [] # malformed declaration: degrade to raw ledger ticks + # The campaign evaluates objectives in due-tick order (declaration order + # within a tick), so the declared list must be put in that order before + # zipping; a YAML listing like [12, 30, 46, 30] would otherwise mislabel. + due_ticks.sort() outcomes = [ ev for ev in events if ev.get("kind") in ("task_completed", "task_failed") @@ -509,7 +581,10 @@ def _rate(tick: int) -> float | None: if r is not None: rates[tick] = r - threshold = (target_fraction * pre_shock_rate) if pre_shock_rate is not None else None + # A zero baseline gives a zero threshold that any rate (even 0.0) meets, + # i.e. "recovery" with nothing recovered; treat it like an undefined + # baseline (criterion not evaluable -> censored). + threshold = (target_fraction * pre_shock_rate) if pre_shock_rate else None recovered = False recovery_tick: int | None = None if threshold is not None: diff --git a/src/verdi/analysis/scenario.py b/src/verdi/analysis/scenario.py index a833b3b..760e718 100644 --- a/src/verdi/analysis/scenario.py +++ b/src/verdi/analysis/scenario.py @@ -10,6 +10,7 @@ import hashlib import json +import math from dataclasses import dataclass from pathlib import Path from typing import Any @@ -102,8 +103,11 @@ def build_t0_state(scenario: dict[str, Any]) -> WorldState: values), so a scenario file fully defines the simulation it drives. """ agent_ids = [a["agent_id"] for a in scenario["agents"]] + if not agent_ids: + raise ScenarioLoadError("scenario has no agents") if len(agent_ids) != len(set(agent_ids)): raise ScenarioLoadError("scenario contains duplicate agent ids") + _check_values(scenario, set(agent_ids)) _normalize_governance(scenario) # Tranche 12m: an agent entry may carry a `sanction: {duration_ticks: N}` # block (a scenario-declared t0 sanction with auto-expiry). That is campaign @@ -127,6 +131,41 @@ def build_t0_state(scenario: dict[str, Any]) -> WorldState: return state +def _finite_nonneg(value: Any) -> bool: + return (isinstance(value, (int, float)) and not isinstance(value, bool) + and math.isfinite(value) and value >= 0) + + +def _check_values(scenario: dict[str, Any], agent_ids: set[str]) -> None: + """Load-time checks for values that would otherwise load fine and then + crash (or silently misbehave) many model calls into a run.""" + for a in scenario["agents"]: + for res, val in (a.get("resources") or {}).items(): + if not _finite_nonneg(val): + raise ScenarioLoadError( + f"agent {a['agent_id']!r} resource {res!r} must be a finite number >= 0, got {val!r}") + for res, val in (scenario.get("world_resources") or {}).items(): + if not _finite_nonneg(val): + raise ScenarioLoadError(f"world_resources.{res} must be a finite number >= 0, got {val!r}") + for i, shock in enumerate(scenario.get("shocks") or []): + if not isinstance(shock, dict): + raise ScenarioLoadError(f"shocks[{i}] must be a mapping") + tick = shock.get("tick") + if isinstance(tick, bool) or not isinstance(tick, int) or tick < 0: + raise ScenarioLoadError(f"shocks[{i}].tick must be an integer >= 0, got {tick!r}") + delta = shock.get("delta") + if isinstance(delta, bool) or not isinstance(delta, (int, float)) or not math.isfinite(delta): + raise ScenarioLoadError(f"shocks[{i}].delta must be a finite number, got {delta!r}") + tasks = scenario.get("tasks") + deliveries = tasks.get("deliveries") if isinstance(tasks, dict) else tasks + for i, entry in enumerate(deliveries or []): + target = entry.get("target_pool", "world") if isinstance(entry, dict) else "world" + if target != "world" and target not in agent_ids: + # Otherwise this only raises at the due tick, deep into a live run. + raise ScenarioLoadError( + f"tasks.deliveries[{i}].target_pool {target!r} is neither 'world' nor an agent id") + + def _normalize_governance(scenario: dict[str, Any]) -> None: """Fill in the scenario's ``constitution`` and ``law`` blocks in place. diff --git a/src/verdi/audit/interview_audit.py b/src/verdi/audit/interview_audit.py index 148894e..e67770b 100644 --- a/src/verdi/audit/interview_audit.py +++ b/src/verdi/audit/interview_audit.py @@ -422,6 +422,15 @@ async def audit_interview( out["judgment"] = getattr(result, "judgment", None) out["summary"] = getattr(result, "summary", None) out["severity"] = getattr(result, "severity", None) + if str(out["severity"]).upper() == "ERROR": + # SimpleAudit converts scenario exceptions / unparseable judge + # output into severity="ERROR" instead of raising; count it as an + # error so summaries do not report a failed audit as a success. + j = getattr(result, "judgment", None) + detail = (j.get("error") if isinstance(j, dict) else None) or "; ".join( + str(i) for i in (getattr(result, "issues_found", None) or []) + ) + out["error"] = f"auditor_error: {detail or 'severity=ERROR'}" out["issues_found"] = list(getattr(result, "issues_found", []) or []) out["positive_behaviors"] = list(getattr(result, "positive_behaviors", []) or []) out["recommendations"] = list(getattr(result, "recommendations", []) or []) diff --git a/src/verdi/audit/judge_live.py b/src/verdi/audit/judge_live.py index d4f318e..7b109ef 100644 --- a/src/verdi/audit/judge_live.py +++ b/src/verdi/audit/judge_live.py @@ -169,11 +169,26 @@ def agreement_ci( # the 12-char floor stops short test tokens ("sk-testfake123" is 8 chars after # the prefix) from being flagged while real OpenAI/vLLM keys (50+ chars) are # still caught. Tests pin both sides of this tradeoff. +# +# The Bearer / api_key= / authorization: patterns require a token-shaped value +# (Bearer: 16+ token chars; key/authorization: 8+ token chars incl. a digit). +# A bare ``\S+`` matched ordinary agent speech ("Bearer of bad news", +# "Authorization: the assembly approved it") and aborted artifact writes. +# A "value" is 8+ token chars containing a digit, or any 20+ token-char run +# (long digit-free keys); prose words are neither. An optional auth scheme +# word (Basic/Bearer/Token/ApiKey) may precede it, and a closing quote may +# sit between key and separator (JSON: "api_key": "..."). +_TOKEN = r"[A-Za-z0-9._~+/=\-]" +_VALUE = rf"(?:(?={_TOKEN}*\d){_TOKEN}{{8,}}|{_TOKEN}{{20,}})" +_SEP = r"[\"']?\s*[=:]\s*[\"']?" +_SCHEME = r"(?:(?:basic|bearer|token|apikey)\s+)?" KEY_MATERIAL_PATTERNS = ( re.compile(r"(? None: for pattern in KEY_MATERIAL_PATTERNS: m = pattern.search(text) if m: - raise AssertionError(f"{what} would contain key material: {m.group(0)!r}") + # Never echo the matched secret itself into the exception/logs. + hit = m.group(0) + raise AssertionError( + f"{what} would contain key material: {hit[:6]!r}... ({len(hit)} chars)" + ) def _write_json(path: Path, obj: dict[str, Any]) -> None: diff --git a/src/verdi/audit/live_bridge.py b/src/verdi/audit/live_bridge.py index 3331f01..2828292 100644 --- a/src/verdi/audit/live_bridge.py +++ b/src/verdi/audit/live_bridge.py @@ -594,10 +594,15 @@ def audit_live_run( scoring_rubric_id=RUBRIC_ID, ) verdict = "agree" if judgment.get("correct") else "disagree" + score_error: str | None = None try: score: float | None = scorer.score(probe, verdict) - except ValueError: - score = 0.0 + except ValueError as exc: + # A scorer/rubric mismatch is an instrument error, not the worst + # possible score: record it as missing so it cannot pass for a + # judge disagreement. + score = None + score_error = str(exc) state = evidence["scenario_meta"]["decision"].get("state") or {} judge_rows.append( { @@ -613,6 +618,7 @@ def audit_live_run( "judge_severity": judgment.get("severity"), "judge_correct": judgment.get("correct"), "rubric_score": score, + "rubric_score_error": score_error, "violation_flags": state.get("violation_flags"), } ) diff --git a/src/verdi/cli.py b/src/verdi/cli.py index 76db47f..0f2d5a1 100644 --- a/src/verdi/cli.py +++ b/src/verdi/cli.py @@ -75,7 +75,7 @@ def _run_main(argv: list[str]) -> int: if argv and not argv[0].startswith("-"): scenario, rest = argv[0], argv[1:] argv = ["--scenario", scenario, *rest] - if "--run-dir" not in rest: + if not any(a == "--run-dir" or a.startswith("--run-dir=") for a in rest): argv += ["--run-dir", str(Path("runs") / Path(scenario).stem)] return live_main(argv) @@ -100,9 +100,9 @@ def main(argv: list[str] | None = None) -> int: "only what you list (verdi sweep --help)", formatter_class=argparse.RawDescriptionHelpFormatter) parser.parse_args(argv) - print("verdi CLI scaffold — see docs/IMPLEMENTATION_PLAN.md") - print("Commands: verdi init [dir] | verdi run | " - "verdi sweep (see docs/SWEEPS.md)") + # No subcommand given: show usage (the old text pointed at a doc that + # does not exist). + parser.print_help() return 0 diff --git a/src/verdi/config.py b/src/verdi/config.py index 65f6e78..be35e90 100644 --- a/src/verdi/config.py +++ b/src/verdi/config.py @@ -15,7 +15,7 @@ import os from collections.abc import Mapping -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Any API_KEY_ENV = "MODEL_API_KEY" @@ -32,7 +32,8 @@ class ModelEndpointConfig: base_url: str model: str - api_key: str + # repr=False: a printed config or a traceback must never show the key. + api_key: str = field(repr=False) max_concurrency: int timeout_seconds: float max_retries: int @@ -52,14 +53,24 @@ def _validate_model_block(block: Mapping[str, Any]) -> None: ) +# Every key value resolved in this process (including ones read through a +# scenario's ``model.api_key_env``), so leak guards check the real secrets and +# not only the default variable names. +_RESOLVED_KEYS: set[str] = set() + + def _resolve_api_key(api_key_env: str | None) -> str: if api_key_env: - return os.environ.get(api_key_env) or "EMPTY" - return ( - os.environ.get(API_KEY_ENV) - or os.environ.get(_LEGACY_API_KEY_ENV) - or "EMPTY" - ) + key = os.environ.get(api_key_env) or "EMPTY" + else: + key = ( + os.environ.get(API_KEY_ENV) + or os.environ.get(_LEGACY_API_KEY_ENV) + or "EMPTY" + ) + if key != "EMPTY": + _RESOLVED_KEYS.add(key) + return key def model_endpoint_from_env( @@ -76,27 +87,40 @@ def model_endpoint_from_env( name = block.get("name") or os.environ.get("VERDI_VLLM_MODEL") if not base_url or not name: return None - timeout = block.get("timeout_seconds") - retries = block.get("max_retries") + timeout = float( + block["timeout_seconds"] if block.get("timeout_seconds") is not None + else os.environ.get("VERDI_REQUEST_TIMEOUT", "120") + ) + retries = int( + block["max_retries"] if block.get("max_retries") is not None + else os.environ.get("VERDI_MAX_RETRIES", "5") + ) + concurrency = int(os.environ.get("VERDI_MAX_CONCURRENCY", "8")) + if timeout <= 0: + raise ValueError(f"model timeout_seconds must be > 0, got {timeout}") + if retries < 0: + raise ValueError(f"model max_retries must be >= 0, got {retries}") + if concurrency < 1: + raise ValueError(f"VERDI_MAX_CONCURRENCY must be >= 1, got {concurrency}") return ModelEndpointConfig( base_url=str(base_url), model=str(name), api_key=_resolve_api_key(block.get("api_key_env")), - max_concurrency=int(os.environ.get("VERDI_MAX_CONCURRENCY", "8")), - timeout_seconds=float( - timeout if timeout is not None - else os.environ.get("VERDI_REQUEST_TIMEOUT", "120") - ), - max_retries=int( - retries if retries is not None - else os.environ.get("VERDI_MAX_RETRIES", "5") - ), + max_concurrency=concurrency, + timeout_seconds=timeout, + max_retries=retries, ) def api_key_values() -> list[str]: - """Every configured endpoint key value, for leak guards.""" - return [ + """Every configured endpoint key value, for leak guards. + + Default-variable keys come first (stable order), followed by any key that + was resolved via ``model.api_key_env`` in this process. + """ + values = [ v for v in (os.environ.get(API_KEY_ENV), os.environ.get(_LEGACY_API_KEY_ENV)) if v ] + values.extend(sorted(k for k in _RESOLVED_KEYS if k not in values)) + return values diff --git a/src/verdi/experiments/g5_close.py b/src/verdi/experiments/g5_close.py index 5df8b2e..6c4874d 100644 --- a/src/verdi/experiments/g5_close.py +++ b/src/verdi/experiments/g5_close.py @@ -677,10 +677,27 @@ def build_g5_close_report( "manifest_sha256": sample_manifest.get("manifest_sha256"), }, ) - causal_word_guard(report) + # The guard polices the report's OWN wording; verbatim model-authored + # text (judge rationales, quarantined raw payloads) is quoted evidence, + # and one judge saying "causally" must not make the report unbuildable. + causal_word_guard(_without_model_text(report)) return report +#: Keys whose values are verbatim model output, excluded from the causal guard. +#: ("error" on quarantined rows embeds the model's invalid stance verbatim.) +_MODEL_TEXT_KEYS = frozenset({"rationale", "raw", "error"}) + + +def _without_model_text(obj: Any) -> Any: + if isinstance(obj, Mapping): + return {k: (None if k in _MODEL_TEXT_KEYS else _without_model_text(v)) + for k, v in obj.items()} + if isinstance(obj, (list, tuple)): + return [_without_model_text(v) for v in obj] + return obj + + def causal_word_guard(report: Mapping[str, Any]) -> None: """Frozen causal-word guard (ADR-014 §5). diff --git a/src/verdi/experiments/g5_pairs.py b/src/verdi/experiments/g5_pairs.py index 0ed45a6..913ec37 100644 --- a/src/verdi/experiments/g5_pairs.py +++ b/src/verdi/experiments/g5_pairs.py @@ -771,12 +771,6 @@ def arm_run_dict(run: PilotRun, arm: str) -> dict[str, Any]: # --------------------------------------------------------------------------- # -def _api_key_value() -> str | None: - from verdi.config import api_key_values - - return next(iter(api_key_values()), None) - - def audit_artifacts(run_root: str | Path) -> dict[str, Any]: """Mandatory audits over a g5_pairs run directory (ADR-013 §2). @@ -787,10 +781,12 @@ def audit_artifacts(run_root: str | Path) -> dict[str, Any]: any finding. """ run_root = Path(run_root) - key = _api_key_value() + from verdi.config import api_key_values + + keys = [k.encode() for k in api_key_values()] findings: list[str] = [] key_hits = 0 - if key: + if keys: for path in sorted(run_root.rglob("*")): if not path.is_file(): continue @@ -798,7 +794,7 @@ def audit_artifacts(run_root: str | Path) -> dict[str, Any]: text = path.read_bytes() except OSError: continue - if key.encode() in text: + if any(k in text for k in keys): findings.append(f"api key value found in {path}") key_hits += 1 for meta in sorted(run_root.rglob("*meta*.json")): @@ -840,7 +836,7 @@ def audit_artifacts(run_root: str | Path) -> dict[str, Any]: raise G5PipelineError("G5 artifact audit FAILURES: " + "; ".join(findings[:5])) return { "scanned_root": str(run_root), - "api_key_value_scanned": bool(key), + "api_key_value_scanned": bool(keys), "api_key_value_hits": key_hits, "artifacts": len(list(run_root.rglob("*"))), "clean": True, diff --git a/src/verdi/experiments/interview_experiment.py b/src/verdi/experiments/interview_experiment.py index 66438e6..90596a1 100644 --- a/src/verdi/experiments/interview_experiment.py +++ b/src/verdi/experiments/interview_experiment.py @@ -286,11 +286,12 @@ def _audit_run(run_dir: Path, *, base_url: str, api_key: str, model: str, client = ModelClient(config) agent_context_by_id = _agent_context_map(scenario_path, run_dir) try: - summary = asyncio.run(audit_from_file( + # audit_from_file is synchronous (it runs its own event loop). + summary = audit_from_file( interviews_path, run_dir, base_url=base_url, api_key=api_key, model=model, constitution_rules=rules, concurrency=concurrency, probe_turns=probe_turns, client=client, agent_context_by_id=agent_context_by_id, - )) + ) finally: if client is not None: with contextlib.suppress(Exception): @@ -349,7 +350,12 @@ def main(argv: list[str] | None = None) -> int: # Run the separate post-pass audit only when it was NOT already done inline. if not args.no_audit and not inline_audit: - cfg = _endpoint() + if args.scenario is not None: + # Same endpoint the campaign used (scenario `model:` block first). + from verdi.society.live import resolve_endpoint_config + cfg = resolve_endpoint_config(load_scenario(args.scenario)) + else: + cfg = _endpoint() _audit_run(args.run_dir, base_url=cfg.base_url, api_key=cfg.api_key, model=cfg.model, concurrency=args.concurrency or 4, probe_turns=args.probe_turns, scenario_path=args.scenario) diff --git a/src/verdi/experiments/paired_run.py b/src/verdi/experiments/paired_run.py index 8e4759d..0460b79 100644 --- a/src/verdi/experiments/paired_run.py +++ b/src/verdi/experiments/paired_run.py @@ -47,6 +47,7 @@ import argparse import asyncio import copy +import hashlib import json import os import sys @@ -58,6 +59,7 @@ from typing import Any from verdi.analysis.scenario import load_scenario +from verdi.experiments.sweep import _set_aside_previous_attempt from verdi.society.campaign import SocietyCampaign ALLOWED_CONFIG_KEYS = frozenset({ @@ -302,6 +304,11 @@ async def _run_one_arm(*, arm: str, seed: int, scenario: dict[str, Any], # the public API # --------------------------------------------------------------------------- # +def _scenario_sha256(scenario: dict[str, Any]) -> str: + blob = json.dumps(scenario, sort_keys=True, default=str, ensure_ascii=True) + return hashlib.sha256(blob.encode("utf-8")).hexdigest() + + def run_pair(config: dict[str, Any], caller_factory: Callable[[dict[str, Any], Path], Any], *, @@ -319,7 +326,10 @@ def run_pair(config: dict[str, Any], Never raises for a failed arm; returns a :class:`PairRunResult` whose ``exit_code`` is nonzero when any pair did not complete. """ - root = Path(output_root if output_root is not None else config["output_root"]) + chosen_root = output_root if output_root is not None else config.get("output_root") + if chosen_root is None: + _fail("no output folder: set output_root in the paired config or pass --output-root") + root = Path(chosen_root) root.mkdir(parents=True, exist_ok=True) manifest_path = root / MANIFEST_NAME manifest = _load_manifest(manifest_path) @@ -351,17 +361,24 @@ async def _drive() -> None: for arm in config["arms"]: run_dir = root / f"{arm}-seed{seed}" prior = entries.get((seed, arm)) - if prior is not None and prior.get("status") == "complete": + scenario_sha = _scenario_sha256(arm_scenarios[arm]) + if (prior is not None and prior.get("status") == "complete" + and prior.get("scenario_sha256", scenario_sha) == scenario_sha): + # (entries from before hashes were recorded are trusted) print(f"[skip] {arm}-seed{seed}: already complete (resume)") continue print(f"[run ] {arm}-seed{seed}: {run_dir}", flush=True) try: + # A failed/stale earlier attempt must not be appended to + # (the ledger opens in append mode); keep it, renamed. + _set_aside_previous_attempt(run_dir) outcome = await _run_one_arm( arm=arm, seed=seed, scenario=arm_scenarios[arm], run_dir=run_dir, caller_factory=caller_factory, concurrency=concurrency, ) - except BaseException as exc: # noqa: BLE001 - record, don't abort the pair loop + except Exception as exc: # noqa: BLE001 - record, don't abort the pair loop + # (Exception, not BaseException: Ctrl-C / cancellation must stop the loop.) outcome = { "status": "failed", "error": f"{type(exc).__name__}: {exc}", @@ -372,6 +389,7 @@ async def _drive() -> None: "seed": seed, "arm": arm, "run_dir": str(run_dir), + "scenario_sha256": scenario_sha, "updated_at_utc": _utc_now(), **outcome, } @@ -453,10 +471,13 @@ def main(argv: list[str] | None = None) -> int: except ValueError: _fail(f"--seeds must be comma-separated integers, got: {args.seeds!r}") - # Endpoint resolution happens up front so a misconfigured environment fails - # BEFORE any arm is launched (and resume-skip decisions stay free). + # The factory resolves the endpoint lazily, per arm, from that arm's + # scenario (a misconfigured endpoint fails that arm, recorded as failed). caller_factory = _real_caller_factory() - concurrency = args.concurrency if args.concurrency is not None else 4 + concurrency = (args.concurrency if args.concurrency is not None + else int(os.environ.get("VERDI_MAX_CONCURRENCY", "4"))) + if concurrency < 1: + _fail(f"concurrency must be >= 1, got {concurrency}") result = run_pair(config, caller_factory, seeds=seeds, output_root=args.output_root, concurrency=concurrency) diff --git a/src/verdi/experiments/sweep.py b/src/verdi/experiments/sweep.py index 6173c7a..d7311f7 100644 --- a/src/verdi/experiments/sweep.py +++ b/src/verdi/experiments/sweep.py @@ -42,6 +42,7 @@ import copy import csv import difflib +import hashlib import html import json import os @@ -52,7 +53,7 @@ import threading import time from collections.abc import Callable -from concurrent.futures import ThreadPoolExecutor, as_completed +from concurrent.futures import Future, ThreadPoolExecutor, as_completed from dataclasses import dataclass, field from datetime import datetime, timezone from itertools import product @@ -130,8 +131,11 @@ class RunSpec: def _label(value: Any) -> str: text = value if isinstance(value, str) else json.dumps(value, sort_keys=True) - text = re.sub(r"[^A-Za-z0-9._-]+", "-", str(text)).strip("-") - return text[:40] or "value" + text = re.sub(r"[^A-Za-z0-9._-]+", "-", str(text)).strip("-")[:40] + # Labels become path components: "." / ".." would escape the sweep folder. + if not text.strip("."): + return "value" + return text def _factor(path: str, raw: Any) -> Factor: @@ -423,6 +427,26 @@ def _atomic_write(path: Path, text: str) -> None: os.replace(tmp, path) +def _set_aside_previous_attempt(run_dir: Path) -> Path | None: + """Move a non-empty run dir from an earlier attempt out of the way. + + The ledger and model_calls files are opened in append mode, so relaunching + into an old directory would concatenate two runs (two event_id 0 lines, + broken hash chain). The old attempt is renamed, never deleted. + """ + if not run_dir.is_dir() or not any(run_dir.iterdir()): + return None + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S.%fZ") + aside = run_dir.with_name(f"{run_dir.name}.prev-{stamp}") + run_dir.rename(aside) + return aside + + +def _scenario_sha256(run: RunSpec) -> str: + blob = json.dumps(run.scenario, sort_keys=True, default=str, ensure_ascii=True) + return hashlib.sha256(blob.encode("utf-8")).hexdigest() + + def _write_scenario(run: RunSpec) -> Path: import yaml @@ -496,7 +520,17 @@ def run_sweep(cfg: SweepConfig, runs: list[RunSpec], def save() -> None: _atomic_write(manifest_path, json.dumps(manifest, indent=2, sort_keys=True) + "\n") - todo = [r for r in runs if entries.get(r.run_id, {}).get("status") != "complete"] + def reusable(r: RunSpec) -> bool: + e = entries.get(r.run_id, {}) + if e.get("status") != "complete": + return False + # A finished run is only reused if it ran this exact scenario; a run_id + # (condition/seed) alone does not capture e.g. a changed --ticks. + # Entries from before hashes were recorded are trusted as before. + recorded = e.get("scenario_sha256") + return recorded is None or recorded == _scenario_sha256(r) + + todo = [r for r in runs if not reusable(r)] if len(todo) < len(runs): out(f"Resuming: {len(runs) - len(todo)} run(s) already complete, {len(todo)} to go.") for r in runs: @@ -505,18 +539,26 @@ def save() -> None: "seed": r.seed, "run_dir": str(r.run_dir)}) if r in todo: entries[r.run_id]["status"] = "pending" + entries[r.run_id]["scenario_sha256"] = _scenario_sha256(r) + # Runs left over from an earlier invocation with other seeds/factors stay + # on disk but are excluded from this sweep's results and exit code. + manifest["current_runs"] = sorted(r.run_id for r in runs) save() done = len(runs) - len(todo) def one(run: RunSpec) -> tuple[RunSpec, int, float]: - run.run_dir.mkdir(parents=True, exist_ok=True) - _write_scenario(run) - with lock: - entries[run.run_id].update({"status": "running", "started_at_utc": _utc_now()}) - save() t0 = time.monotonic() try: + # Setup I/O is inside the guard too: one bad run dir must not + # abort the whole sweep. + _set_aside_previous_attempt(run.run_dir) + run.run_dir.mkdir(parents=True, exist_ok=True) + _write_scenario(run) + with lock: + entries[run.run_id].update({"status": "running", "started_at_utc": _utc_now()}) + save() + t0 = time.monotonic() code = launcher(run, cfg.agent_concurrency) except Exception as exc: # noqa: BLE001 - one run must never stop the sweep with lock: @@ -524,21 +566,38 @@ def one(run: RunSpec) -> tuple[RunSpec, int, float]: code = 1 return run, code, time.monotonic() - t0 + def record(run: RunSpec, code: int, secs: float) -> str: + status = _status(code) + with lock: + entries[run.run_id].update({"status": status, "exit_code": code, + "wall_seconds": round(secs, 1), + "finished_at_utc": _utc_now()}) + save() + return status + + pool = ThreadPoolExecutor(max_workers=cfg.parallel_runs) + futures: list[Future] = [] + recorded: set[int] = set() try: - with ThreadPoolExecutor(max_workers=cfg.parallel_runs) as pool: - futures = [pool.submit(one, r) for r in todo] - for fut in as_completed(futures): - run, code, secs = fut.result() - status = _status(code) - with lock: - done += 1 - entries[run.run_id].update({"status": status, "exit_code": code, - "wall_seconds": round(secs, 1), - "finished_at_utc": _utc_now()}) - save() - tail = "" if status == "complete" else f" (see {run.run_dir / LOG_NAME})" - out(f"[{done}/{len(runs)}] {status:<8} {run.run_id} {_fmt_secs(secs)}{tail}") + futures = [pool.submit(one, r) for r in todo] + for fut in as_completed(futures): + run, code, secs = fut.result() + recorded.add(id(fut)) + status = record(run, code, secs) + done += 1 + tail = "" if status == "complete" else f" (see {run.run_dir / LOG_NAME})" + out(f"[{done}/{len(runs)}] {status:<8} {run.run_id} {_fmt_secs(secs)}{tail}") except KeyboardInterrupt: + # Do not let the executor drain its queue (a `with` block would wait + # for, and so START, every queued run after Ctrl-C). + pool.shutdown(wait=False, cancel_futures=True) + # Keep results of runs that already finished but were not yet recorded. + for fut in futures: + if id(fut) in recorded or not fut.done() or fut.cancelled(): + continue + with contextlib.suppress(BaseException): + record(*fut.result()) + done += 1 with lock: for e in entries.values(): if e.get("status") in ("running", "pending"): @@ -546,7 +605,10 @@ def one(run: RunSpec) -> tuple[RunSpec, int, float]: save() out("Interrupted. Run the same command again to resume.") raise - return SweepResult(output=cfg.output, entries=entries) + pool.shutdown(wait=True) + current = {r.run_id for r in runs} + return SweepResult(output=cfg.output, + entries={k: v for k, v in entries.items() if k in current}) # --------------------------------------------------------------------------- # @@ -602,8 +664,11 @@ def _run_row(run_dir: Path) -> dict[str, Any]: def collect(output: Path) -> list[dict[str, Any]]: manifest = json.loads((output / MANIFEST_NAME).read_text(encoding="utf-8")) + current = manifest.get("current_runs") rows = [] for run_id, e in sorted(manifest.get("runs", {}).items()): + if current is not None and run_id not in current: + continue # left over from an earlier invocation of this sweep run_dir = Path(e["run_dir"]) row: dict[str, Any] = {"run": run_id, "condition": e.get("condition"), "seed": e.get("seed"), "status": e.get("status")} @@ -757,12 +822,20 @@ def main(argv: list[str] | None = None) -> int: parser.error("give a sweep file OR --scenario (not both)") if args.sweep_file: import yaml - raw = yaml.safe_load(Path(args.sweep_file).read_text(encoding="utf-8")) \ - if Path(args.sweep_file).is_file() else None - if raw is None: - raise SweepError(f"sweep file not found or empty: {args.sweep_file}") + path = Path(args.sweep_file) + if not path.is_file(): + raise SweepError(f"sweep file not found: {path}") + try: + raw = yaml.safe_load(path.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise SweepError(f"{path} is not valid YAML:\n{exc}") from None + if not isinstance(raw, dict): + raise SweepError(f"{path} must be a non-empty YAML mapping") if args.vary: - raw.setdefault("vary", {}).update(_parse_vary(args.vary)) + vary = raw.get("vary") or {} + if not isinstance(vary, dict): + raise SweepError(f"{path}: `vary:` must be a mapping") + raw["vary"] = {**vary, **_parse_vary(args.vary)} else: raw = {"scenario": args.scenario, "vary": _parse_vary(args.vary)} if args.name: diff --git a/src/verdi/governance/appeals.py b/src/verdi/governance/appeals.py index 41acde2..c6696d3 100644 --- a/src/verdi/governance/appeals.py +++ b/src/verdi/governance/appeals.py @@ -107,11 +107,9 @@ def adjudicate(self, appeal_id: str, upheld: bool, grounds: str, restitution={}, reason=grounds, ) - # Immediate restoration: release rights and end the sanction. - self._sanctions.rights.release(sanction.subject_id, frozenset(rights_restored)) - record = self._sanctions._active.get(appeal.sanction_id) - if record is not None: - record.expired = True + # Immediate restoration: end the sanction and release its rights + # (rights still held by another live sanction stay revoked). + self._sanctions.end(appeal.sanction_id) self._status[appeal_id] = "upheld" self.metrics.upheld += 1 self._restorations[appeal.sanction_id] = restoration diff --git a/src/verdi/governance/constitution.py b/src/verdi/governance/constitution.py index d042814..5410601 100644 --- a/src/verdi/governance/constitution.py +++ b/src/verdi/governance/constitution.py @@ -40,6 +40,13 @@ def deterministic_checksum(rules: dict[str, Any], quorum: float, threshold: floa return f"sha256:{hashlib.sha256(raw.encode('utf-8')).hexdigest()[:32]}" +def _is_fraction(value: Any) -> bool: + return ( + isinstance(value, (int, float)) and not isinstance(value, bool) + and 0 < value <= 1 + ) + + class ConstitutionStore: """Append-only registry of constitution versions. @@ -107,7 +114,12 @@ def active_at(self, tick: int) -> ConstitutionVersion: return best[2] def _next_version(self) -> str: - return f"{self.current.constitution_id}-v{len(self._order) + 1}" + n = len(self._order) + 1 + # register() ignores a version name it has seen, so never reuse one + # (e.g. an initial version that happens to be named "-v2"). + while f"{self.current.constitution_id}-v{n}" in self._versions: + n += 1 + return f"{self.current.constitution_id}-v{n}" # -- voters ----------------------------------------------------------- @@ -203,9 +215,27 @@ def enact_if_valid(self, votes: Sequence[VoteRecord], tick: int, raise ValueError("proposal_id required when no votes are given") proposal_id = votes[0].proposal_id proposal = self.get_proposal(proposal_id) - votes = self.votes_for(proposal_id) if not votes else list(votes) + if self._proposal_status[proposal_id] != "under_vote": + # Re-enacting would register a duplicate version for one proposal. + raise ProposalNotInVoteError(f"proposal {proposal_id} is not under vote") + # The recorded ballot is the single source of truth. Caller-supplied + # votes are routed through record_vote so they get the same checks + # (right proposal, one vote per voter); a vote that was already + # rejected or duplicated must never be counted. + for vote in votes: + if vote.proposal_id != proposal_id: + continue + recorded = self._votes[proposal_id].get(vote.voter_id) + if recorded is None: + self.record_vote(vote) + votes = self.votes_for(proposal_id) base = self.get(proposal.constitution_version_base) - for_votes, against_votes, abstain_votes = self._count_votes(votes) + # Only eligible, valid ballots decide the outcome (same filter as quorum). + counted = [ + v for v in votes + if v.eligibility_valid and (not self._eligible_voters or v.voter_id in self._eligible_voters) + ] + for_votes, against_votes, abstain_votes = self._count_votes(counted) invalid_reasons: list[str] = [] if not self.amendments_enabled: @@ -224,6 +254,12 @@ def enact_if_valid(self, votes: Sequence[VoteRecord], tick: int, validate_enforceable_rules(candidate_rules) except ValueError: invalid_reasons.append("invalid_enforceable_policy") + # Procedural parameters must be fractions in (0, 1]; an out-of-range + # value would otherwise fail ConstitutionVersion validation AFTER the + # amendment was recorded as valid, leaving the store inconsistent. + for key in ("quorum", "amendment_threshold"): + if key in proposal.payload and not _is_fraction(proposal.payload[key]): + invalid_reasons.append(f"invalid_{key}") # 1. Immutable law outranks the constitution. law_violations = self._law.law_overrides(proposal.payload) @@ -270,7 +306,15 @@ def enact_if_valid(self, votes: Sequence[VoteRecord], tick: int, self.amendments.append(amendment) if valid: - self._register_new_version(base, proposal.payload, new_version, effective_from_tick) + # Apply the delta onto the version that would otherwise be in force + # at the effective tick, not onto the (possibly stale) submission + # base: two amendments enacted in one assembly share a base, and + # building both from it would silently drop the first one's rules. + # Vote rules above still come from ``base`` (in force when voting). + self._register_new_version( + self.active_at(effective_from_tick), proposal.payload, + new_version, effective_from_tick, + ) self._proposal_status[proposal_id] = "enacted" else: self._proposal_status[proposal_id] = "invalid" if law_violations else "rejected" diff --git a/src/verdi/governance/sanctions.py b/src/verdi/governance/sanctions.py index d8c230d..3cff01a 100644 --- a/src/verdi/governance/sanctions.py +++ b/src/verdi/governance/sanctions.py @@ -31,18 +31,21 @@ class _Revocation: @dataclass class RightsView: - """Which agents have which rights revoked, and until when.""" + """Which agents have which rights revoked, and until when. - _revocations: dict[str, dict[str, _Revocation]] = field(default_factory=dict) + Each right keeps every revocation interval, so overlapping or disjoint + sanctions on the same right are represented exactly (a single merged + interval would either restore rights early or revoke them too soon). + """ + + _revocations: dict[str, dict[str, list[_Revocation]]] = field(default_factory=dict) def revoke(self, agent_id: str, rights: frozenset[str], from_tick: int, until: int) -> None: per_agent = self._revocations.setdefault(agent_id, {}) for right in rights: - existing = per_agent.get(right) - per_agent[right] = _Revocation( - from_tick=from_tick, - until=max(until, existing.until if existing else until), + per_agent.setdefault(right, []).append( + _Revocation(from_tick=from_tick, until=until) ) def release(self, agent_id: str, rights: frozenset[str]) -> None: @@ -54,29 +57,27 @@ def release(self, agent_id: str, rights: frozenset[str]) -> None: if not per_agent: self._revocations.pop(agent_id, None) + @staticmethod + def _active(revs: list[_Revocation], tick: int) -> bool: + return any(rev.from_tick <= tick < rev.until for rev in revs) + def revoked_rights(self, agent_id: str, tick: int) -> frozenset[str]: per_agent = self._revocations.get(agent_id, {}) return frozenset( - right - for right, rev in per_agent.items() - if rev.from_tick <= tick < rev.until + right for right, revs in per_agent.items() if self._active(revs, tick) ) def has_active_revocation(self, agent_id: str, tick: int) -> bool: return bool(self.revoked_rights(agent_id, tick)) def expires_at(self, agent_id: str, right: str) -> int | None: - rev = self._revocations.get(agent_id, {}).get(right) - return rev.until if rev else None + revs = self._revocations.get(agent_id, {}).get(right) + return max(rev.until for rev in revs) if revs else None def all_active(self, tick: int) -> dict[str, frozenset[str]]: out: dict[str, frozenset[str]] = {} - for agent_id, per_agent in self._revocations.items(): - active = frozenset( - right - for right, rev in per_agent.items() - if rev.from_tick <= tick < rev.until - ) + for agent_id in self._revocations: + active = self.revoked_rights(agent_id, tick) if active: out[agent_id] = active return out @@ -159,11 +160,40 @@ def advance_tick(self, tick: int) -> list[str]: if not record.expired and tick >= record.expires_at ] for sid in sorted(expired): - record = self._active[sid] - record.expired = True - self.rights.release(record.sanction.subject_id, frozenset(record.sanction.scope)) + self.end(sid) return sorted(expired) + def end(self, sanction_id: str) -> None: + """End one sanction and restore only the rights no other live sanction holds. + + Rights are released for the sanction's scope and then re-revoked from + every other non-expired sanction on the same subject, so expiring (or + overturning) one of two overlapping sanctions keeps the other in force. + """ + record = self._active.get(sanction_id) + if record is None: + return + record.expired = True + subject = record.sanction.subject_id + scope = frozenset(record.sanction.scope) + self.rights.release(subject, scope) + for other in sorted(self._active.values(), key=lambda r: r.sanction_id): + if other.expired or other.sanction.subject_id != subject: + continue + overlap = scope & frozenset(other.sanction.scope) + if overlap: + self.rights.revoke( + subject, overlap, + from_tick=other.sanction.effective_tick, until=other.expires_at, + ) + + def has_live_sanction(self, agent_id: str) -> bool: + """True while any non-expired sanction on ``agent_id`` remains.""" + return any( + not r.expired and r.sanction.subject_id == agent_id + for r in self._active.values() + ) + def is_active(self, sanction_id: str, tick: int) -> bool: record = self._active.get(sanction_id) if record is None or record.expired: diff --git a/src/verdi/measurement/interview_values.py b/src/verdi/measurement/interview_values.py index df45103..fd72090 100644 --- a/src/verdi/measurement/interview_values.py +++ b/src/verdi/measurement/interview_values.py @@ -20,6 +20,7 @@ from __future__ import annotations import json +import re from datetime import datetime, timezone from pathlib import Path from typing import Any @@ -69,22 +70,38 @@ def interview_axis_scores(parsed: dict[str, Any] | None) -> dict[str, float]: # --------------------------------------------------------------------------- # # Nearest-event linking # --------------------------------------------------------------------------- # +_TASK_TICK = re.compile(r"-t(\d+)-") + + def _nearest_event(agent_id: str, tick: int, events: list[dict[str, Any]]) -> dict[str, Any] | None: - """The most recent committed world event at-or-before ``tick`` that involves - this agent (or a society-level governance event if none agent-specific). - Returns {event_id, kind, tick} or None.""" + """The most recent committed world event at-or-before lifecycle ``tick`` that + involves this agent (or a society-level governance event if none + agent-specific). Returns {event_id, kind, tick, lifecycle_tick} or None. + + ``tick`` is the campaign's lifecycle tick, while a ledger event's ``tick`` + is the reducer's transition counter, so the two are not comparable. The + lifecycle tick of an event is read from its task id (``soc-tNN-..``); + events without one (governance, consumption) inherit the tick of the + preceding event, since the ledger is chronological. + """ agent_ev = None society_ev = None + current = 0 for ev in events: - et = ev.get("tick") - if et is None or et > tick: - continue + task_id = str((ev.get("payload") or {}).get("task_id") or "") + m = _TASK_TICK.search(task_id) + if m: + current = int(m.group(1)) + if current > tick: + break actor = ev.get("actor_id") or ev.get("actor") - kind = ev.get("kind", "") - if actor == agent_id and agent_ev is None or (actor == agent_id and (agent_ev is None)): - agent_ev = {"event_id": ev.get("event_id"), "kind": kind, "tick": et} - if actor in (None, "assembly", "governance") and society_ev is None: - society_ev = {"event_id": ev.get("event_id"), "kind": kind, "tick": et} + ref = {"event_id": ev.get("event_id"), "kind": ev.get("kind", ""), + "tick": ev.get("tick"), "lifecycle_tick": current} + # Later events overwrite earlier ones: we want the MOST RECENT match. + if actor == agent_id: + agent_ev = ref + elif actor in (None, "assembly", "governance"): + society_ev = ref return agent_ev or society_ev diff --git a/src/verdi/measurement/longitudinal.py b/src/verdi/measurement/longitudinal.py index 3ee2306..636c315 100644 --- a/src/verdi/measurement/longitudinal.py +++ b/src/verdi/measurement/longitudinal.py @@ -132,9 +132,24 @@ def _resolve(raw: str) -> tuple[str, bool]: if opt.lower() == r: return opt, False if all(opt.isdigit() for opt in canonical): - m = re.search(r"\d", r) - if m: - return m.group(0), False + # Drop scale descriptions ("0 to 5", "0-5") and denominators + # ("4/5", "4 out of 5" -> 4), then take the FIRST standalone + # in-range integer: the answer leads, trailing reasoning ("2 + # factors") must not override it. A trailing "." or ".0" is fine; + # "3.5", "10" and "7 out of 10" never yield an in-range digit. + r = re.sub(r"\b\d+\s*(?:-|–|to)\s*\d+\b", " ", r) + r = re.sub(r"(\d+)\s*(?:/|out of)\s*\d+", r"\1", r) + for num in re.findall(r"(? tuple[str, bool]: payload = json.loads(text) except json.JSONDecodeError: payload = None - if isinstance(payload, dict) and isinstance(payload.get("choice"), str): - return _resolve(payload["choice"]) + if isinstance(payload, dict): + choice = payload.get("choice") + if isinstance(choice, str): + return _resolve(choice) + if isinstance(choice, int) and not isinstance(choice, bool): + return _resolve(str(choice)) # {"choice": 4, ...} return _resolve(text) diff --git a/src/verdi/pilot/driver.py b/src/verdi/pilot/driver.py index f90be4c..a538c26 100644 --- a/src/verdi/pilot/driver.py +++ b/src/verdi/pilot/driver.py @@ -21,6 +21,7 @@ import hashlib import importlib.metadata import json +import os from dataclasses import dataclass, field from datetime import datetime, timezone from pathlib import Path @@ -308,8 +309,10 @@ def __init__( # including quarantined proposals. Default dir when a live model is # injected but no dir given: runs/live//. if live_capture_dir is None and model is not None: - stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") - live_capture_dir = root / "runs" / "live" / stamp + # Microseconds + pid: two live runs started in the same second must + # not append into one live_tape.jsonl. + stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S.%fZ") + live_capture_dir = root / "runs" / "live" / f"{stamp}-{os.getpid()}" self._live_capture_dir = Path(live_capture_dir) if live_capture_dir else None # ------------------------------------------------------------------ # @@ -650,7 +653,11 @@ def record_probes(lifecycle_tick: int) -> None: if ( lifecycle_tick == _SANCTION_TICK and action.actor_id == violation_subject + and _VIOLATION_CODE in validation.attempted_violation_codes ): + # Only the scripted violation may serve as sanction + # evidence -- never a quarantine rejection of the same + # agent's proposal (that would manufacture a false sanction). violation_event_id = sealed.event_id trace_id = f"trace-ev-{sealed.event_id}" source.add_span( diff --git a/src/verdi/society/campaign.py b/src/verdi/society/campaign.py index 7db0bf8..74b73e0 100644 --- a/src/verdi/society/campaign.py +++ b/src/verdi/society/campaign.py @@ -429,6 +429,10 @@ async def run(self) -> SocietyResult: if tick > 0: for ev in governance.advance_tick(tick): self._restoration_events.append({"tick": tick, **ev}) + if governance.sanctions.has_live_sanction(ev.get("actor", "")): + # Another sanction on this agent is still in force + # (e.g. a declared t0 sanction): keep the flag set. + continue # F2: duration-expired sanctions clear the world-state flag # at loop-top, before this tick's snapshot/tape, so the # restored agent is eligible again from this tick on. (An @@ -485,7 +489,12 @@ async def run(self) -> SocietyResult: # P0-B: per-tick production accounting under an active economy — # resets each tick; only consulted by the validator when the # economy block is present, so legacy runs are unaffected. - produced_this_tick = {} + # Keyed per ACTOR then resource: the cap is documented (economy.py, + # examples/*.yaml) as per-agent per-tick. A society-wide sum let the + # first producer exhaust everyone else's cap. Tracked with or + # without an economy block so the constitutional + # max_produce_units_per_tick is cumulative in both cases. + produced_this_tick: dict[str, dict[str, float]] = {} # Live-state mirror for same-tick prompt building (economy only); # kept in sync at every commit point below. self._live_state = state @@ -537,8 +546,9 @@ async def run(self) -> SocietyResult: fallback_seed=f"soc-v2:{self._run_seed}", memories=memory, ) - if used_llm: - # Record orchestrator model call for telemetry + if orch_meta: + # Record every completed orchestrator call (including ones + # whose output was unusable) so token accounting is honest. fingerprint = self._model_fingerprint() self._model_calls.append( _ModelCall(tick, "orchestrator", "orchestrator", orch_meta.get("trace_id", ""), @@ -587,7 +597,7 @@ async def run(self) -> SocietyResult: results = await TaskPool(self._concurrency).run_all(self._make_worker( snapshot, tick, memory, governance.constitution.active_at(tick).rules, - produced_this_tick if self._economy_spec else None, + {} if self._economy_spec else None, ), tape) # deterministic-order commit (single reducer, sequential). @@ -608,12 +618,17 @@ async def run(self) -> SocietyResult: state, pr.action, policy_rules=governance.constitution.active_at(tick).rules, economy_spec=self._economy_spec, - produced_this_tick=produced_this_tick if self._economy_spec else None, + produced_this_tick=produced_this_tick.get(pr.action.actor_id, {}), ) before = state state = reducer.apply(before, pr.action, validation) self._live_state = state sealed = self._commit_action(log, before, state, pr.action, validation) + if validation.accepted and pr.action.action_kind == "produce": + _mine = produced_this_tick.setdefault(pr.action.actor_id, {}) + _res = str(pr.action.arguments.get("resource", "")) + _mine[_res] = _mine.get(_res, 0.0) + float( + pr.action.arguments.get("amount", 0.0)) if ( validation.accepted and self._economy_spec is not None @@ -641,9 +656,6 @@ async def run(self) -> SocietyResult: ) state = self._deduct_production_input(state, pr.action.actor_id, _resource, _amount) self._live_state = state - produced_this_tick[_resource] = ( - produced_this_tick.get(_resource, 0.0) + _amount - ) self._tape.append(TapeStep( action=None, # type: ignore[arg-type] validation=ValidationResult(accepted=True), @@ -722,7 +734,7 @@ async def run(self) -> SocietyResult: for p in list(self._pending_appeals): if p["file_tick"] == tick: upheld = self._file_and_adjudicate_appeal(p, governance, tick) - if upheld: + if upheld and not governance.sanctions.has_live_sanction(p["subject_id"]): # F2: an upheld appeal restores rights immediately, so # clear the world-state flag this tick. state = self._commit_rights_change(state, p["subject_id"], False) @@ -1047,6 +1059,8 @@ def _build_economy_ledger_event( def _make_worker(self, snapshot: WorldState, tick: int, memory: dict[str, AgentMemory], policy_rules: dict[str, Any] | None = None, produced_this_tick: dict[str, float] | None = None) -> Callable: + # produced_this_tick here is only an "economy active" flag (prompts + # show current holdings); validation uses the per-actor tally. caller = self._caller temperature = self._temperature fingerprint = self._model_fingerprint() @@ -1063,6 +1077,7 @@ async def worker(task) -> PolicyResult: il_messages = build_inner_life_prompt_messages( actor_id, snapshot, memory[actor_id], shock_info=self._current_shock_info, + tick=task.tick_created, ) il_res = await caller.generate_structured( il_messages, @@ -1190,7 +1205,7 @@ async def _consolidate_memories( async def _consolidate_one(aid: str): agent = state.agents[aid] overflow = memory[aid].get_overflow_entries(aid) - existing_summary = memory[aid].render_history_summary(aid).split("- From earlier: ")[-1] if "From earlier:" in memory[aid].render_history_summary(aid) else "" + existing_summary = memory[aid].consolidated_summary(aid) # Include current inner state so the reflection is grounded in # "how I feel right now" — makes the summary a living document @@ -1816,11 +1831,14 @@ async def generate_structured(self, messages, **kwargs): content = messages[0].get("content", "") if messages else "" t_start = time.monotonic() ts_start = _utc_now_iso() + # Resolve a provider-supplied subject BEFORE awaiting: calls are sent + # in the known order but complete in any order under concurrency. + provided = self._subject_provider() if self._subject_provider is not None else None res = await self._inner.generate_structured(messages, **kwargs) elapsed = int((time.monotonic() - t_start) * 1000) ts_end = _utc_now_iso() - if self._subject_provider is not None: - subject = self._subject_provider() + if provided is not None: + subject = provided else: subject = self._default_subject m = re.search(r'agent "([^"]+)"', content) @@ -1920,8 +1938,14 @@ def _compute_metrics(result: SocietyResult, events: list[WorldEvent]) -> dict[st k: len(v) for k, v in sorted(result.assembly_quarantined.items()) }, "assembly_outcome": (result.assembly_outcome or {}).get("outcome"), - "sanctions": len(result.sanction_events), - "appeals": len(result.appeal_events), + # sanction_events also carries world_shock rows and rejected + # impositions, and appeal_events has a filed AND an adjudicated row + # per appeal: count only what the metric names. + "sanctions": sum( + 1 for e in result.sanction_events + if e.get("kind") == "sanction_imposed" and e.get("decision", "accepted") == "accepted" + ), + "appeals": sum(1 for e in result.appeal_events if e.get("phase") == "filed"), "restorations": len(result.restoration_events), "model_calls": len(result.model_calls), "final_state_hash": result.final_state_hash, diff --git a/src/verdi/society/inner_life.py b/src/verdi/society/inner_life.py index 7d3740a..08138bf 100644 --- a/src/verdi/society/inner_life.py +++ b/src/verdi/society/inner_life.py @@ -29,7 +29,7 @@ log = logging.getLogger(__name__) -INNER_LIFE_PROMPT_VERSION = "verdi.inner-life/1" +INNER_LIFE_PROMPT_VERSION = "verdi.inner-life/2" # --- Stable system prompt (identical across ticks for the same agent) ------- _INNER_LIFE_SYSTEM_PROMPT = """\ @@ -164,8 +164,13 @@ def build_inner_life_prompt_messages( *, prompt_version: str = INNER_LIFE_PROMPT_VERSION, shock_info: str = "", + tick: int | None = None, ) -> list[dict[str, str]]: - """Build [system, user] messages with a stable system prefix for caching.""" + """Build [system, user] messages with a stable system prefix for caching. + + ``tick`` is the campaign's lifecycle tick; ``state.tick`` (the ledger + transition counter) is only a fallback for callers that do not pass it. + """ agent = state.agents[agent_id] signals_text = "\n".join(f"- {s}" for s in agent.public_signals) or "(none)" @@ -181,7 +186,7 @@ def build_inner_life_prompt_messages( prompt_version=prompt_version, ) user = _build_inner_life_user( - tick=state.tick, + tick=state.tick if tick is None else tick, resources_json=json.dumps(agent.resources, sort_keys=True), voting=agent.rights.voting, speaking=agent.rights.speaking, diff --git a/src/verdi/society/live.py b/src/verdi/society/live.py index 64bec32..d10bac6 100644 --- a/src/verdi/society/live.py +++ b/src/verdi/society/live.py @@ -27,10 +27,11 @@ import shutil import sys import time +from collections.abc import Callable from datetime import datetime, timezone from pathlib import Path -from verdi.agents.client import ModelClient +from verdi.agents.client import _BACKOFF_CAP, _CONNECT_TIMEOUT, _POOL_TIMEOUT, ModelClient from verdi.analysis.scenario import load_scenario from verdi.audit.judge_live import _assert_no_key_material from verdi.config import ModelEndpointConfig, api_key_values, model_endpoint_from_env @@ -79,7 +80,8 @@ def _progress_count(run_dir: Path) -> int: ) -async def _stall_watchdog(run_dir: Path, stall_timeout_seconds: float) -> None: +async def _stall_watchdog(run_dir: Path, stall_timeout_seconds: float, + extra_progress: Callable[[], int] | None = None) -> None: """Independent liveness task for a live run. A hung endpoint can hold a socket open without ever responding, which @@ -92,12 +94,17 @@ async def _stall_watchdog(run_dir: Path, stall_timeout_seconds: float) -> None: partial run is preserved. It runs as a SEPARATE asyncio task so it survives a hung in-flight call. """ + def progress() -> int: + # extra_progress: in-process completed-call counter, so a long batch + # whose file lines are written only at the end still counts as alive. + return _progress_count(run_dir) + (extra_progress() if extra_progress else 0) + deadline = time.monotonic() + stall_timeout_seconds - last_count = _progress_count(run_dir) + last_count = progress() while True: await asyncio.sleep(_HEARTBEAT_INTERVAL_SECONDS) now = time.monotonic() - count = _progress_count(run_dir) + count = progress() _write_run_meta(run_dir, { "heartbeat_utc": datetime.now(timezone.utc).isoformat(), "events_logged": _file_line_count(run_dir / "events.jsonl"), @@ -212,7 +219,9 @@ async def _run(scenario_path: Path, run_dir: Path, seed: int | None, concurrency # Never let a (possibly user-supplied) stall window trip while the # client is still legitimately retrying one dead call: floor it at # timeout_seconds x (max_retries+1) + one heartbeat interval. - retry_window = config.timeout_seconds * (config.max_retries + 1) + _HEARTBEAT_INTERVAL_SECONDS + # Per attempt: pool wait + connect + read timeout + max backoff. + per_attempt = config.timeout_seconds + _POOL_TIMEOUT + _CONNECT_TIMEOUT + _BACKOFF_CAP + retry_window = per_attempt * (config.max_retries + 1) + _HEARTBEAT_INTERVAL_SECONDS stall_timeout = stall_timeout_seconds if stall_timeout_seconds is not None \ else DEFAULT_STALL_TIMEOUT_SECONDS if stall_timeout < retry_window: @@ -221,7 +230,8 @@ async def _run(scenario_path: Path, run_dir: Path, seed: int | None, concurrency f"(client retry window)", flush=True, ) stall_timeout = retry_window - watchdog = asyncio.create_task(_stall_watchdog(run_dir, stall_timeout)) + watchdog = asyncio.create_task(_stall_watchdog( + run_dir, stall_timeout, extra_progress=lambda: getattr(client, "completed_calls", 0))) try: result = await campaign.run() except Exception: @@ -252,7 +262,11 @@ async def _run(scenario_path: Path, run_dir: Path, seed: int | None, concurrency print(f" events = {len(result.events)}") print(f" probes = {len(result.probes)}") print(f" assembly = {(result.assembly_outcome or {}).get('outcome')}") - print(f" sanctions/appeals = {len(result.sanction_events)}/{len(result.appeal_events)}") + n_sanctions = sum(1 for e in result.sanction_events + if e.get("kind") == "sanction_imposed" + and e.get("decision", "accepted") == "accepted") + n_appeals = sum(1 for e in result.appeal_events if e.get("phase") == "filed") + print(f" sanctions/appeals = {n_sanctions}/{n_appeals}") print(f" replay_matches = {result.replay_matches}") print(f" replay_log_identical = {result.replay_log_identical}") print(f" wall_seconds = {result.wall_seconds:.1f}") diff --git a/src/verdi/society/memory.py b/src/verdi/society/memory.py index 14f0398..a49bdec 100644 --- a/src/verdi/society/memory.py +++ b/src/verdi/society/memory.py @@ -187,6 +187,10 @@ def version(self, agent_id: str) -> int: """Number of entries ever committed for this agent (0 if unseen).""" return self._total.get(agent_id, 0) + def consolidated_summary(self, agent_id: str) -> str: + """The LLM-produced early summary alone ("" when none yet).""" + return self._summary.get(agent_id, "") + def render_history_summary(self, agent_id: str) -> str: """Deterministic prompt-facing text of this agent's memory. diff --git a/src/verdi/society/prompting.py b/src/verdi/society/prompting.py index 8bc7fc6..4e59155 100644 --- a/src/verdi/society/prompting.py +++ b/src/verdi/society/prompting.py @@ -17,7 +17,7 @@ from .memory import AgentMemory #: Versioned prompt constant. Bump deliberately; experiments log it. -SOCIAL_PROMPT_VERSION = "verdi.society-prompt/4" +SOCIAL_PROMPT_VERSION = "verdi.society-prompt/5" #: Exact required argument field names per action kind, mirroring #: ``verdi.world.validate``. Rendered into the prompt so the model cannot @@ -183,7 +183,9 @@ def build_social_prompt( return _PROMPT_TEMPLATE.format( prompt_version=prompt_version, agent_id=agent_id, - tick=state.tick, + # The campaign's lifecycle tick (what sanction durations, deadlines and + # per-tick caps count), not WorldState.tick, the ledger transition counter. + tick=task.tick_created, role=agent.role, persona_summary=agent.persona_summary, resources_json=json.dumps(agent.resources, sort_keys=True), @@ -233,7 +235,9 @@ def build_social_prompt_messages( prompt_version=prompt_version, ) user = _build_user_prompt( - tick=state.tick, + # The campaign's lifecycle tick (what sanction durations, deadlines and + # per-tick caps count), not WorldState.tick, the ledger transition counter. + tick=task.tick_created, resources_json=json.dumps(agent.resources, sort_keys=True), voting=agent.rights.voting, speaking=agent.rights.speaking, diff --git a/src/verdi/society/tasks.py b/src/verdi/society/tasks.py index 6ec1b0d..57b9f93 100644 --- a/src/verdi/society/tasks.py +++ b/src/verdi/society/tasks.py @@ -42,7 +42,7 @@ def is_adversary_persona(agent: AgentState) -> bool: return ADVERSARY_MARKER in (agent.persona_summary or "") -def _call_rng(tape_seed: int, tick: int, rng: random.Random) -> random.Random: +def _call_rng(tape_seed: int | str, tick: int, rng: random.Random) -> random.Random: """Fresh per-call RNG seeded from (tape_seed, tick). Deterministic across processes and interpreters; the external ``rng`` @@ -129,7 +129,7 @@ def generate_tick_tape( state: WorldState, rng: random.Random, *, - tape_seed: int, + tape_seed: int | str, task_kind_bias: str = "balanced", ) -> list[Task]: """Build this tick's task tape: ~1 mutating task per agent. @@ -310,8 +310,19 @@ def _parse_orchestrator_response(content: str, state: WorldState, tick: int) -> if kind not in MUTATING_KINDS: kind = "produce" resource = spec.get("resource") or "food" - amount = int(spec.get("amount", 1)) + try: + amount = int(spec.get("amount", 1)) + except (TypeError, ValueError): + # e.g. "amount": null / "two" -- one bad field must not discard + # the whole orchestrated tape. + amount = 1 target = spec.get("to") + if kind == "transfer" and not ( + isinstance(target, str) and target in state.agents and target != aid + ): + # A transfer the validator is certain to reject (no/self/unknown + # recipient) would only inflate rejection metrics. + kind = "produce" # Sanity-clamp: never exceed what the agent holds agent = state.agents[aid] @@ -389,7 +400,9 @@ async def generate_llm_tape( Returns (tasks, used_llm, meta). ``meta`` contains telemetry for the orchestrator model call (duration_ms, prompt_tokens, completion_tokens, trace_id, ts_start, ts_end). If the LLM call fails or returns unparseable - output, falls back to the deterministic seeded tape and returns (tasks, False, {}). + output, falls back to the deterministic seeded tape and returns (tasks, False, meta), + where meta is {} if the call failed and carries ``parse_error`` if the + call succeeded but its output was unusable. """ import time as _time @@ -416,16 +429,24 @@ async def generate_llm_tape( "ts_end": ts_end, "response_chars": len(res.content), } - tasks = _parse_orchestrator_response(res.content, state, tick) + except Exception: # noqa: BLE001 - any call failure falls back deterministically + meta = {} + else: + try: + tasks = _parse_orchestrator_response(res.content, state, tick) + except Exception as exc: # noqa: BLE001 - malformed output falls back + tasks = None + meta["parse_error"] = type(exc).__name__ if tasks is not None: return tasks, True, meta - except Exception: # noqa: BLE001 - any failure falls back deterministically - pass + meta.setdefault("parse_error", "unusable_tape") # Deterministic fallback import random as _random tape = generate_tick_tape(tick, state, _random.Random(), tape_seed=fallback_seed) - return tape, False, {} + # meta is non-empty when the model was called but its output was unusable, + # so the caller can still account for the spent tokens. + return tape, False, meta __all__ = [ diff --git a/src/verdi/world/branch.py b/src/verdi/world/branch.py index 834e9de..fb147f7 100644 --- a/src/verdi/world/branch.py +++ b/src/verdi/world/branch.py @@ -186,10 +186,13 @@ def branch_from_snapshot( # terminates in. Without this, a spec can match a snapshot that is # merely consistent with the ledger length but divergent from the # ledger's terminal state (an inconsistent (snapshot, log) pair). - if log.events and actual_hash != log.events[-1].after_hash: + # A rejected event leaves state unchanged and carries no after_hash, so + # its terminal state is its before_hash. + terminal = (log.events[-1].after_hash or log.events[-1].before_hash) if log.events else None + if log.events and actual_hash != terminal: raise BranchOriginError( "snapshot is not the terminal state of the prefix ledger: " - f"snapshot={actual_hash!r} ledger_terminal={log.events[-1].after_hash!r}" + f"snapshot={actual_hash!r} ledger_terminal={terminal!r}" ) # Rebuild the prefix in a fresh ledger from the sealed JSONL bytes. diff --git a/src/verdi/world/economy.py b/src/verdi/world/economy.py index 3550150..b21dcd9 100644 --- a/src/verdi/world/economy.py +++ b/src/verdi/world/economy.py @@ -106,6 +106,13 @@ def parse_economy_spec(block: Mapping[str, Any] | None) -> EconomySpec | None: raise ValueError("role_consumption role keys must be nonempty strings") res_map = _nonneg_mapping(resources, f"role_consumption.{role}") for resource, value in res_map.items(): + if resource not in consumption: + # Overrides only apply to resources with a base rate; a + # role-only entry used to be ignored without any error. + raise ValueError( + f"role_consumption.{role}.{resource} overrides a resource missing " + "from consumption_per_tick; add a base rate there (0 is allowed)" + ) role_consumption[(role, resource)] = value return EconomySpec( version=int(version), @@ -463,8 +470,14 @@ def parse_task_objectives(block: Mapping[str, Any] | None) -> list[dict[str, Any ) if not math.isfinite(amount) or amount <= 0: raise ValueError(f"tasks.deliveries[{i}].amount must be a positive finite number") + objective_id = str(entry.get("objective_id") or f"obj-{i:02d}") + if any(o["objective_id"] == objective_id for o in objectives): + # Baselines and outcomes are keyed by id: a duplicate (including an + # auto id "obj-NN" colliding with an explicit one) would overwrite + # another objective's baseline. + raise ValueError(f"tasks.deliveries[{i}]: duplicate objective_id {objective_id!r}") objectives.append({ - "objective_id": str(entry.get("objective_id") or f"obj-{i:02d}"), + "objective_id": objective_id, "resource": resource, "amount": amount, "target_pool": target, diff --git a/src/verdi/world/policy.py b/src/verdi/world/policy.py index b91330a..6104924 100644 --- a/src/verdi/world/policy.py +++ b/src/verdi/world/policy.py @@ -42,11 +42,15 @@ def _number(value: Any, key: str, *, strictly_positive: bool) -> float: def policy_block(kind: str, arguments: Mapping[str, Any], world_resources: Mapping[str, float], - rules: Mapping[str, Any] | None) -> tuple[str, str] | None: + rules: Mapping[str, Any] | None, + *, already_produced: float = 0.0) -> tuple[str, str] | None: """Return (rule key, explanation) if a valid action violates policy. Call after invariant checks: malformed actions and overdrafts retain their original rejection codes. Policy rejection does not itself impose sanctions. + ``already_produced`` is what this actor already produced of the resource + this tick, so ``max_produce_units_per_tick`` is a per-tick (not + per-action) cap. """ if not rules: return None @@ -59,9 +63,10 @@ def policy_block(kind: str, arguments: Mapping[str, Any], if kind == "produce" and "max_produce_units_per_tick" in rules: amount = float(arguments["amount"]) cap = float(rules["max_produce_units_per_tick"]) - if amount > cap: + if already_produced + amount > cap + 1e-9: return ("max_produce_units_per_tick", - f"produce amount {amount:g} exceeds per-tick limit {cap:g}") + f"produce amount {amount:g} exceeds per-tick limit {cap:g}" + + (f" ({already_produced:g} already produced)" if already_produced else "")) if kind == "withdraw": amount = float(arguments["amount"]) if "max_withdraw_units" in rules: diff --git a/src/verdi/world/reducer.py b/src/verdi/world/reducer.py index 3eb1a42..1c67a02 100644 --- a/src/verdi/world/reducer.py +++ b/src/verdi/world/reducer.py @@ -210,6 +210,11 @@ def _apply_mutation( @staticmethod def _transfer(sender, sender_id: str, agents: dict[str, Any], arguments: dict[str, Any]) -> None: recipient_id = str(_arg(arguments, "to")) + if recipient_id == sender_id: + # Reading the recipient before writing the sender back would turn + # a self-transfer into money creation; the validator rejects it, + # so reaching here is an invariant breach. + raise InvariantViolationError(f"self-transfer by agent {sender_id!r}") recipient = agents.get(recipient_id) if recipient is None: raise InvariantViolationError(f"transfer to unknown agent: {recipient_id!r}") diff --git a/src/verdi/world/snapshot.py b/src/verdi/world/snapshot.py index d8cb2aa..411eecc 100644 --- a/src/verdi/world/snapshot.py +++ b/src/verdi/world/snapshot.py @@ -50,17 +50,26 @@ def load(self, digest: str) -> WorldState: """Materialize the snapshot stored under ``digest``.""" payload = self._find(digest) try: - return WorldState.model_validate_json(payload) + state = WorldState.model_validate_json(payload) except ValidationError as exc: raise SnapshotError(f"corrupt snapshot {digest!r}: {exc}") from exc + # Content addressing is only a guarantee if it is checked on read. + if canonical_sha256(state.model_dump(mode="json")) != digest: + raise SnapshotError(f"snapshot {digest!r} does not match its content hash") + return state def contains(self, digest: str) -> bool: """True if a snapshot with this digest is stored.""" + if not _is_digest(digest): + return False if digest in self._memory: return True return self._dir is not None and (self._dir / f"{digest}.json").exists() def _find(self, digest: str) -> bytes: + if not _is_digest(digest): + # Digests become file names: refuse anything that could traverse. + raise SnapshotError(f"invalid snapshot digest: {digest!r}") if digest in self._memory: return self._memory[digest] if self._dir is not None: @@ -70,3 +79,10 @@ def _find(self, digest: str) -> bytes: self._memory[digest] = payload return payload raise SnapshotError(f"unknown snapshot digest: {digest!r}") + + +def _is_digest(value: str) -> bool: + return ( + isinstance(value, str) and len(value) == 64 + and all(c in "0123456789abcdef" for c in value) + ) diff --git a/src/verdi/world/validate.py b/src/verdi/world/validate.py index ff9d35c..98d12ea 100644 --- a/src/verdi/world/validate.py +++ b/src/verdi/world/validate.py @@ -8,6 +8,7 @@ """ from __future__ import annotations +import math from collections.abc import Mapping from typing import Any @@ -77,13 +78,23 @@ def validate_action(state: WorldState, action: ProposedAction, *, f"transfer missing required argument {key!r}", ) recipient_id = args["to"] - if recipient_id not in state.agents: + if recipient_id == action.actor_id: + # Not a world-rule violation (no value moves), so it gets its own + # non-substantive code: format-level, never sanctionable. + return _rejected( + "law:self_transfer", + ["self_transfer"], + f"agent {action.actor_id!r} cannot transfer to itself", + ) + if not isinstance(recipient_id, str) or recipient_id not in state.agents: return _rejected( "law:unknown_recipient", ["unknown_recipient"], f"transfer recipient {recipient_id!r} is not present in world state", ) resource = args["resource"] + if not _is_resource_name(resource): + return _bad_resource(kind, resource) units = _as_float(args["units"]) if units is None: return _rejected( @@ -98,6 +109,9 @@ def validate_action(state: WorldState, action: ProposedAction, *, ["overdraft"], f"agent {action.actor_id!r} holds {held} {resource!r}, cannot transfer {units}", ) + over = _overflow(kind, state.agents[recipient_id].resources.get(resource, 0.0), units) + if over is not None: + return over return _policy_checked(kind, args, state.world_resources, policy_rules) if kind in ("deposit", "withdraw"): @@ -109,6 +123,8 @@ def validate_action(state: WorldState, action: ProposedAction, *, f"{kind} missing required argument {key!r}", ) resource = args["resource"] + if not _is_resource_name(resource): + return _bad_resource(kind, resource) amount = _as_float(args["amount"]) if amount is None: return _rejected( @@ -132,6 +148,11 @@ def validate_action(state: WorldState, action: ProposedAction, *, ["insufficient_world"], f"world pool holds {available} {resource!r}, cannot withdraw {amount}", ) + current = (state.world_resources.get(resource, 0.0) if kind == "deposit" + else actor.resources.get(resource, 0.0)) + over = _overflow(kind, current, amount) + if over is not None: + return over return _policy_checked(kind, args, state.world_resources, policy_rules) if kind == "produce": @@ -142,6 +163,8 @@ def validate_action(state: WorldState, action: ProposedAction, *, ["missing_argument"], f"produce missing required argument {key!r}", ) + if not _is_resource_name(args["resource"]): + return _bad_resource(kind, args["resource"]) amount = _as_float(args["amount"]) if amount is None: return _rejected( @@ -149,6 +172,9 @@ def validate_action(state: WorldState, action: ProposedAction, *, ["non_positive_amount"], f"produce amount {args['amount']!r} is not a positive number", ) + over = _overflow(kind, actor.resources.get(args["resource"], 0.0), amount) + if over is not None: + return over if economy_spec is not None: resource = str(args["resource"]) produced = float((produced_this_tick or {}).get(resource, 0.0)) @@ -161,7 +187,9 @@ def validate_action(state: WorldState, action: ProposedAction, *, if blocked is not None: blocked_by, codes, details = blocked return _rejected(blocked_by, list(codes), details) - return _policy_checked(kind, args, state.world_resources, policy_rules) + already = float((produced_this_tick or {}).get(args["resource"], 0.0)) + return _policy_checked(kind, args, state.world_resources, policy_rules, + already_produced=already) if kind == "emit_signal": text = args.get("text") @@ -182,8 +210,10 @@ def validate_action(state: WorldState, action: ProposedAction, *, def _policy_checked(kind: str, arguments: Mapping[str, Any], world_resources: Mapping[str, float], - policy_rules: Mapping[str, Any] | None) -> ValidationResult: - blocked = policy_block(kind, arguments, world_resources, policy_rules) + policy_rules: Mapping[str, Any] | None, + *, already_produced: float = 0.0) -> ValidationResult: + blocked = policy_block(kind, arguments, world_resources, policy_rules, + already_produced=already_produced) if blocked is not None: key, details = blocked return _rejected(f"constitution:{key}", ["policy_limit"], details) @@ -203,8 +233,36 @@ def _rejected(blocked_by: str, codes: list[str], details: str) -> ValidationResu ) +def _overflow(kind: str, current: float, amount: float) -> ValidationResult | None: + # A finite amount can still push a balance to inf (e.g. two 1e308 + # produces), which later crashes int() in prompt/tape code. + if math.isfinite(float(current) + amount): + return None + return _rejected( + "law:non_positive_amount", + ["non_positive_amount"], + f"{kind} amount {amount!r} would make a balance non-finite", + ) + + +def _is_resource_name(value: Any) -> bool: + return isinstance(value, str) and bool(value) + + +def _bad_resource(kind: str, resource: Any) -> ValidationResult: + # Format failure (e.g. a list or null from model JSON), not a world-rule + # violation; rejecting here also keeps unhashable values out of dict lookups. + return _rejected( + "law:missing_argument", + ["missing_argument"], + f"{kind} resource must be a non-empty string, got {resource!r}", + ) + + def _as_float(value: Any) -> float | None: """Positive finite float, or None if the value is not one.""" + if isinstance(value, bool): # float(True) == 1.0: not a quantity + return None try: amount = float(value) except (TypeError, ValueError): diff --git a/tests/agents/test_audit_regressions.py b/tests/agents/test_audit_regressions.py new file mode 100644 index 0000000..d72ebb8 --- /dev/null +++ b/tests/agents/test_audit_regressions.py @@ -0,0 +1,79 @@ +"""Regression tests for defects found in the 2026-11 system audit (AUDIT.md).""" +from __future__ import annotations + +import asyncio + +import httpx +import pytest + +from verdi import config as cfg_mod +from verdi.agents import ModelClient +from verdi.agents.client import ModelError + +from .fakes import completion_body, make_config + + +def _client(handler) -> ModelClient: + return ModelClient(make_config(max_retries=0), transport=httpx.MockTransport(handler)) + + +def _call(client: ModelClient): + async def run(): + try: + return await client.generate_structured([{"role": "user", "content": "hi"}]) + finally: + await client.aclose() + + return asyncio.run(run()) + + +def test_non_json_200_body_raises_typed_model_error(): + # A proxy/CDN answering 200 with an HTML page used to leak a raw + # JSONDecodeError past every `except ModelError` handler. + client = _client(lambda req: httpx.Response(200, text="gateway")) + with pytest.raises(ModelError, match="structurally invalid"): + _call(client) + + +def test_null_content_is_normalised_to_empty_string(): + body = completion_body("x") + body["choices"][0]["message"]["content"] = None + result = _call(_client(lambda req: httpx.Response(200, json=body))) + assert result.content == "" + + +def test_non_string_content_raises_model_error(): + body = completion_body("x") + body["choices"][0]["message"]["content"] = [{"type": "text"}] + with pytest.raises(ModelError): + _call(_client(lambda req: httpx.Response(200, json=body))) + + +def test_config_repr_never_shows_api_key(): + assert "SECRET-DO-NOT-LOG" not in repr(make_config()) + + +@pytest.mark.parametrize( + "block, env, message", + [ + ({"max_retries": -1}, {}, "max_retries"), + ({"timeout_seconds": 0}, {}, "timeout_seconds"), + ({}, {"VERDI_MAX_CONCURRENCY": "0"}, "VERDI_MAX_CONCURRENCY"), + ], +) +def test_endpoint_config_rejects_out_of_range_values(monkeypatch, block, env, message): + for k, v in env.items(): + monkeypatch.setenv(k, v) + with pytest.raises(ValueError, match=message): + cfg_mod.model_endpoint_from_env({"base_url": "http://x/v1", "name": "m", **block}) + + +def test_api_key_values_include_keys_read_via_api_key_env(monkeypatch): + monkeypatch.delenv("MODEL_API_KEY", raising=False) + monkeypatch.delenv("VERDI_VLLM_API_KEY", raising=False) + monkeypatch.setattr(cfg_mod, "_RESOLVED_KEYS", set()) + monkeypatch.setenv("OTHER_PROVIDER_KEY", "custom-secret-value-123") + cfg_mod.model_endpoint_from_env( + {"base_url": "http://x/v1", "name": "m", "api_key_env": "OTHER_PROVIDER_KEY"} + ) + assert "custom-secret-value-123" in cfg_mod.api_key_values() diff --git a/tests/analysis/test_audit_regressions.py b/tests/analysis/test_audit_regressions.py new file mode 100644 index 0000000..c3209fa --- /dev/null +++ b/tests/analysis/test_audit_regressions.py @@ -0,0 +1,141 @@ +"""Regression tests for defects found in the 2026-11 system audit (AUDIT.md).""" +from __future__ import annotations + +import json + +import pytest +import yaml + +from verdi.analysis.outcome_metrics import recovery_time + + +def test_recovery_time_maps_outcomes_to_due_ticks_in_evaluation_order(tmp_path): + # Declared out of order (as in examples/full.yaml); the campaign evaluates + # objectives in due-tick order, so outcomes arrive for 12, 30, 30, 46. + (tmp_path / "scenario_resolved.yaml").write_text(yaml.safe_dump({"tasks": { + "deliveries": [{"due_tick": t} for t in (12, 30, 46, 30)]}})) + (tmp_path / "manifest.json").write_text("{}") + outcomes = [(1, 13, "task_completed"), (2, 31, "task_failed"), + (3, 31, "task_failed"), (4, 47, "task_completed")] + (tmp_path / "events.jsonl").write_text("".join( + json.dumps({"event_id": eid, "tick": tick, "kind": kind}) + "\n" + for eid, tick, kind in outcomes)) + rec = recovery_time(tmp_path, pre_shock_window=(0, 12), post_shock_start=13, + target_fraction=1.0, sustained_ticks=1) + assert rec["per_tick_rates"] == {"12": 1.0, "30": 0.0, "46": 1.0} + assert rec["recovery_tick"] == 46 + + +def test_recovery_time_degrades_on_malformed_due_tick(tmp_path): + (tmp_path / "scenario_resolved.yaml").write_text(yaml.safe_dump({"tasks": { + "deliveries": [{"due_tick": "soon"}]}})) + (tmp_path / "events.jsonl").write_text( + json.dumps({"event_id": 1, "tick": 3, "kind": "task_completed"}) + "\n") + rec = recovery_time(tmp_path, pre_shock_window=(0, 3), post_shock_start=4, + target_fraction=1.0, sustained_ticks=1) + assert rec["per_tick_rates"] == {"3": 1.0} + + +def _write_run(tmp_path, sanctions, resource_ticks, ledger_ticks): + (tmp_path / "sanctions.jsonl").write_text("".join(json.dumps(r) + "\n" for r in sanctions)) + (tmp_path / "resources.jsonl").write_text( + "".join(json.dumps({"tick": t, "agents": {}}) + "\n" for t in resource_ticks)) + (tmp_path / "events.jsonl").write_text( + "".join(json.dumps({"event_id": i, "tick": t}) + "\n" for i, t in enumerate(ledger_ticks))) + + +def _imposed(sid, subject, tick): + return {"record_type": "sanction", "kind": "sanction_imposed", "decision": "accepted", + "sanction_id": sid, "subject_id": subject, "tick": tick} + + +def test_open_exclusion_ends_at_last_lifecycle_tick_not_ledger_tick(tmp_path): + from verdi.analysis.outcome_metrics import exclusion_episodes + + _write_run(tmp_path, [_imposed("s1", "grace", 19)], range(20), [1, 50, 104]) + # bounded by the last LIFECYCLE tick (19), not the ledger counter (104 -> 85) + assert exclusion_episodes(tmp_path)["agents"]["grace"]["excluded_ticks"] == 0 + + +def test_overlapping_exclusions_are_merged(tmp_path): + from verdi.analysis.outcome_metrics import exclusion_episodes + + _write_run(tmp_path, [ + _imposed("s1", "dave", 0), _imposed("s2", "dave", 2), + {"record_type": "restoration", "sanction_id": "s2", "tick": 3}, + {"record_type": "restoration", "sanction_id": "s1", "tick": 6}, + ], range(20), [1, 2]) + dave = exclusion_episodes(tmp_path)["agents"]["dave"] + assert dave["excluded_ticks"] == 6 + assert dave["repeat_episode"] is False + + +def test_comparative_report_marks_undefined_baseline_not_evaluable(tmp_path): + from verdi.analysis.comparative_report import _recovery_row + + (tmp_path / "events.jsonl").write_text( + json.dumps({"event_id": 1, "tick": 31, "kind": "task_completed"}) + "\n") + (tmp_path / "scenario_resolved.yaml").write_text(yaml.safe_dump( + {"tasks": {"deliveries": [{"due_tick": 30}]}})) + html = _recovery_row({"arms": {"a": {"1": {"run_dir": str(tmp_path)}}}}, "a", 1) + assert "not evaluable" in html and "CENSORED" not in html + + +def test_completion_due_tick_is_lifecycle_tick_not_ledger_tick(tmp_path): + from verdi.analysis.outcome_metrics import completion_rate + + rows = [ + {"event_id": 0, "tick": 55, "kind": "action_accepted", + "payload": {"task_id": "soc-t10-03"}}, + {"event_id": 1, "tick": 56, "kind": "task_completed", + "payload": {"objective_id": "pre"}}, + ] + (tmp_path / "events.jsonl").write_text("".join(json.dumps(r) + "\n" for r in rows)) + (obj,) = completion_rate(tmp_path)["objectives"] + assert (obj["due_tick"], obj["ledger_tick"]) == (10, 56) + + (tmp_path / "scenario_resolved.yaml").write_text(yaml.safe_dump({"tasks": { + "deliveries": [{"objective_id": "pre", "due_tick": 12, "resource": "food", + "amount": 1}]}})) + assert completion_rate(tmp_path)["objectives"][0]["due_tick"] == 12 + + +def _scenario(**extra): + import copy + from pathlib import Path + + from verdi.analysis.scenario import load_scenario + + sc = copy.deepcopy(load_scenario(Path(__file__).resolve().parents[2] / "examples" / "starter.yaml")) + sc.update(extra) + return sc + + +@pytest.mark.parametrize("mutate, message", [ + (lambda sc: sc["agents"][0]["resources"].update(food=-1), "finite number >= 0"), + (lambda sc: sc["agents"][0]["resources"].update(food=float("nan")), "finite number >= 0"), + (lambda sc: sc.update(shocks=[{"tick": "soon", "resource": "food", "delta": -1}]), "shocks"), + (lambda sc: sc.update(shocks=[{"tick": 2, "resource": "food", "delta": float("nan")}]), "delta"), + (lambda sc: sc.update(tasks={"deliveries": [{"due_tick": 2, "resource": "food", + "amount": 1, "target_pool": "alcie"}]}), + "target_pool"), + (lambda sc: sc.update(agents=[]), "no agents"), +]) +def test_bad_scenario_values_fail_at_load_time(mutate, message): + from verdi.analysis.scenario import ScenarioLoadError, build_t0_state + + sc = _scenario() + mutate(sc) + with pytest.raises(ScenarioLoadError, match=message): + build_t0_state(sc) + + +def test_completion_due_tick_counts_consumption_ticks_when_economy_active(tmp_path): + from verdi.analysis.outcome_metrics import completion_rate + + rows = [{"event_id": i, "tick": i + 1, "kind": "consumption"} for i in range(11)] + # every tick-10 action was quarantined: no soc-t10 task id in the ledger + rows.append({"event_id": 11, "tick": 12, "kind": "task_completed", + "payload": {"objective_id": "pre"}}) + (tmp_path / "events.jsonl").write_text("".join(json.dumps(r) + "\n" for r in rows)) + assert completion_rate(tmp_path)["objectives"][0]["due_tick"] == 10 diff --git a/tests/analysis/test_outcome_metrics.py b/tests/analysis/test_outcome_metrics.py index ecc5348..527da5a 100644 --- a/tests/analysis/test_outcome_metrics.py +++ b/tests/analysis/test_outcome_metrics.py @@ -285,7 +285,8 @@ def test_completion_rate_well_formed(pair_root): assert 0.0 <= comp["overall"] <= 1.0 for obj in comp["objectives"]: assert set(obj) == {"objective_id", "status", "required", "baseline", - "actual", "delta", "due_tick", "evidence_event_ids"} + "actual", "delta", "due_tick", "ledger_tick", + "evidence_event_ids"} assert obj["status"] in ("completed", "failed") assert obj["evidence_event_ids"], "every objective carries ledger evidence" diff --git a/tests/audit/test_audit_regressions.py b/tests/audit/test_audit_regressions.py new file mode 100644 index 0000000..ace0ca8 --- /dev/null +++ b/tests/audit/test_audit_regressions.py @@ -0,0 +1,64 @@ +"""Regression tests for defects found in the 2026-11 system audit (AUDIT.md).""" +from __future__ import annotations + +import asyncio +from types import SimpleNamespace + +import pytest + +import verdi.audit.interview_audit as ia +from verdi.audit.judge_live import _assert_no_key_material + + +@pytest.mark.parametrize("text", [ + "Bearer of bad news, I told Erin.", + "Authorization: the assembly approved my request", + "my api_key: none", + "as standard-bearer counter-proposals matter", + "Bearer responsibilities-and-obligations", +]) +def test_key_guard_ignores_ordinary_agent_speech(text): + _assert_no_key_material(text, "interviews.jsonl") # must not raise + + +@pytest.mark.parametrize("secret", [ + "Bearer eyJhbGciOiJIUzI1NiJ9abcdef", + "api_key=abc12345xyz", + "authorization: tok_9f8e7d6c5b", + "sk-abcdef1234567890xyz", +]) +def test_key_guard_still_flags_tokens_without_echoing_them(secret): + with pytest.raises(AssertionError) as exc: + _assert_no_key_material(f"x {secret} y", "f") + assert secret not in str(exc.value) + + +def test_simpleaudit_error_severity_counts_as_error(monkeypatch): + class _Auditor: + def set_target(self, target): + pass + + async def run_async(self, scenarios, **kwargs): + res = SimpleNamespace(judgment={"error": "judge returned no JSON"}, + severity="ERROR", issues_found=[], summary=None, + positive_behaviors=[], recommendations=[], conversation=[]) + return SimpleNamespace(results=[res]) + + monkeypatch.setattr(ia, "_build_auditor", lambda **kw: _Auditor()) + out = asyncio.run(ia.audit_interview( + record={"agent_id": "a1", "tick": 1, "raw_answer": "{}", "parsed": {}}, + axis_focus="integrity", constitution_rules=None, + base_url="http://x/v1", api_key="k", model="m", + )) + assert out["error"] and "judge returned no JSON" in out["error"] + + +@pytest.mark.parametrize("secret", [ + "Authorization: Basic dXNlcjpwYXNzd29yZA==", + "api_key=abcdefghijklmnopqrstuv", + '"api_key": "abcdefghijklmnopqrstuvwx"', + "x-api-key: abcdef12345678", +]) +def test_key_guard_flags_other_real_key_shapes(secret): + with pytest.raises(AssertionError): + _assert_no_key_material(secret, "f") diff --git a/tests/experiments/test_audit_regressions.py b/tests/experiments/test_audit_regressions.py new file mode 100644 index 0000000..2b1f595 --- /dev/null +++ b/tests/experiments/test_audit_regressions.py @@ -0,0 +1,212 @@ +"""Regression tests for defects found in the 2026-11 system audit (AUDIT.md).""" +from __future__ import annotations + +import threading +import time + +import pytest + +from verdi.analysis.comparative_report import _institution_rows +from verdi.experiments import sweep as sw + +from .test_sweep import STARTER, _no_real_env # noqa: F401 - autouse fixture + + +def _cfg(tmp_path, **raw): + return sw.build_config({"scenario": STARTER, "output": str(tmp_path / "out"), **raw}) + + +def test_enacted_rule_values_with_braces_render_without_crashing(): + metrics = {"institutional_mechanism": {"enactments": [{ + "version": "c-v2", "tick": 4, + "rule_deltas": {"added": {"limits": {"max": 3}}, "changed": {}, "removed": {}}, + }]}} + html = "".join(_institution_rows(metrics)) + assert "c-v2" in html and "@ tick 4" in html + assert "'max'" in html # escaped exactly once + assert "&#x27;" not in html + + +@pytest.mark.parametrize("value", ["..", ".", "..."]) +def test_dot_only_labels_cannot_escape_the_sweep_folder(value): + assert sw._label(value) == "value" + + +def test_resume_reruns_complete_runs_whose_scenario_changed(tmp_path): + calls: list[str] = [] + + def launcher(run, conc): + calls.append(run.run_id) + return 0 + + cfg1 = _cfg(tmp_path, seeds=[1, 2], ticks=1) + sw.run_sweep(cfg1, sw.expand(cfg1), launcher=launcher, out=lambda _: None) + calls.clear() + cfg2 = _cfg(tmp_path, seeds=[1], ticks=3) + result = sw.run_sweep(cfg2, sw.expand(cfg2), launcher=launcher, out=lambda _: None) + assert calls == ["baseline/seed-1"], "changed ticks must not reuse the old run" + # seed-2 from the earlier invocation is not part of this sweep's result + assert set(result.entries) == {"baseline/seed-1"} + assert [r["run"] for r in sw.collect(cfg2.output)] == ["baseline/seed-1"] + + +def test_ctrl_c_does_not_start_queued_runs(tmp_path): + cfg = _cfg(tmp_path, seeds=[1, 2, 3, 4], ticks=1, parallel_runs=1) + started: list[str] = [] + lock = threading.Lock() + + def launcher(run, conc): + with lock: + started.append(run.run_id) + time.sleep(0.05) + return 0 + + real_as_completed = sw.as_completed + + def interrupting(futures): + for i, fut in enumerate(real_as_completed(futures)): + if i == 1: + raise KeyboardInterrupt + yield fut + + sw.as_completed = interrupting + try: + with pytest.raises(KeyboardInterrupt): + sw.run_sweep(cfg, sw.expand(cfg), launcher=launcher, out=lambda _: None) + finally: + sw.as_completed = real_as_completed + assert len(started) < 4 + + +def test_interview_post_pass_audit_returns_summary(tmp_path, monkeypatch): + # audit_from_file is synchronous; wrapping its dict in asyncio.run crashed + # every post-pass audit with "a coroutine was expected". + import verdi.audit.interview_audit as ia + from verdi.experiments.interview_experiment import _audit_run + + (tmp_path / "interviews.jsonl").write_text("{}\n") + monkeypatch.setattr(ia, "audit_from_file", lambda *a, **k: {"provenance_rows": 1}) + summary = _audit_run(tmp_path, base_url="http://x/v1", api_key="k", model="m", + concurrency=1) + assert summary == {"provenance_rows": 1} + + +@pytest.mark.parametrize("reply, choice", [ + ("Yes.", "yes"), ('"no"', "no"), ("Yes, I would.", "yes"), +]) +def test_probe_parser_accepts_punctuated_yes_no(reply, choice): + from verdi.measurement.longitudinal import _parse_choice + + assert _parse_choice(reply, ["yes", "no"]) == (choice, False) + + +@pytest.mark.parametrize("reply, expected", [ + ("On a scale of 0 to 5, I would say 4", ("4", False)), + ("10", ("0", True)), + ("I rate it 7 out of 10", ("0", True)), +]) +def test_probe_parser_likert_takes_last_in_range_integer(reply, expected): + from verdi.measurement.longitudinal import _parse_choice + + assert _parse_choice(reply, [str(i) for i in range(6)]) == expected + + +def test_paired_run_without_output_root_fails_readably(tmp_path): + from verdi.experiments.paired_run import run_pair + + config = {"experiment_id": "x", "base_scenario": STARTER, "arms": ["fixed"], + "seeds": [1], "total_ticks_override": None, "output_root": None} + with pytest.raises(SystemExit, match="output_root"): + run_pair(config, lambda sc, rd: None) + + +def test_rerun_moves_previous_attempt_aside_instead_of_appending(tmp_path): + def launcher(run, conc): + log = run.run_dir / "events.jsonl" + with log.open("a") as fh: + fh.write('{"event_id": 0}\n') + return 0 + + cfg1 = _cfg(tmp_path, seeds=[1], ticks=1) + sw.run_sweep(cfg1, sw.expand(cfg1), launcher=launcher, out=lambda _: None) + cfg2 = _cfg(tmp_path, seeds=[1], ticks=2) + (run,) = sw.expand(cfg2) + sw.run_sweep(cfg2, [run], launcher=launcher, out=lambda _: None) + assert (run.run_dir / "events.jsonl").read_text().count('"event_id": 0') == 1 + assert list(run.run_dir.parent.glob(f"{run.run_dir.name}.prev-*")) + + +@pytest.mark.parametrize("reply, choice", [ + ("4.", "4"), ("4.0", "4"), ("I rate it 4/5", "4"), ("4 out of 5", "4"), + ("Answer: 3 (scale 0-5)", "3"), +]) +def test_probe_parser_likert_ignores_scale_and_denominator(reply, choice): + from verdi.measurement.longitudinal import _parse_choice + + assert _parse_choice(reply, [str(i) for i in range(6)]) == (choice, False) + + +def test_paired_run_rerun_sets_previous_attempt_aside(tmp_path, monkeypatch): + import verdi.experiments.paired_run as pr + + calls = [] + + async def fake_arm(*, arm, seed, scenario, run_dir, caller_factory, concurrency): + run_dir.mkdir(parents=True, exist_ok=True) + with (run_dir / "events.jsonl").open("a") as fh: + fh.write('{"event_id": 0}\n') + calls.append(arm) + return {"status": "failed" if len(calls) == 1 else "complete"} + + monkeypatch.setattr(pr, "_run_one_arm", fake_arm) + config = {"experiment_id": "x", "base_scenario": STARTER, "arms": ["fixed"], + "seeds": [1], "total_ticks_override": 1, "output_root": str(tmp_path / "o")} + pr.run_pair(config, lambda sc, rd: None) + pr.run_pair(config, lambda sc, rd: None) + run_dir = tmp_path / "o" / "fixed-seed1" + assert (run_dir / "events.jsonl").read_text().count('"event_id": 0') == 1 + assert list(run_dir.parent.glob("fixed-seed1.prev-*")) + + +def test_causal_guard_ignores_verbatim_model_text(): + from verdi.experiments.g5_close import _without_model_text, causal_word_guard + + report = {"judge": {"rows": [{"rationale": "The proposal is causally linked.", + "raw": "causal causal causal"}]}} + causal_word_guard(_without_model_text(report)) # must not raise + with pytest.raises(AssertionError): + causal_word_guard(_without_model_text({"summary": "The effect is causal."})) + + +def test_nearest_event_returns_latest_event_by_lifecycle_tick(): + from verdi.measurement.interview_values import _nearest_event + + events = [ + {"event_id": 0, "tick": 1, "actor_id": "e", "kind": "a", + "payload": {"task_id": "soc-t00-00"}}, + {"event_id": 1, "tick": 2, "actor_id": "e", "kind": "a", + "payload": {"task_id": "soc-t01-00"}}, + {"event_id": 2, "tick": 3, "actor_id": "e", "kind": "a", + "payload": {"task_id": "soc-t05-00"}}, + ] + assert _nearest_event("e", 3, events)["event_id"] == 1 + assert _nearest_event("e", 29, events)["event_id"] == 2 + + +@pytest.mark.parametrize("reply, choice", [ + ("Choice: 4\n\nReasoning: 2 factors", "4"), + ("I choose 3. That is because 5 people agree.", "3"), + ('{"choice": 4, "reason": "weighing 2 duties"}', "4"), + ("Answer: 4 — 10x more", "4"), +]) +def test_probe_parser_likert_answer_leads_over_trailing_reasoning(reply, choice): + from verdi.measurement.longitudinal import _parse_choice + + assert _parse_choice(reply, [str(i) for i in range(6)]) == (choice, False) + + +def test_causal_guard_ignores_model_stance_in_error_field(): + from verdi.experiments.g5_close import _without_model_text, causal_word_guard + + causal_word_guard(_without_model_text( + {"rows": [{"error": "invalid stance 'causally_compliant'"}]})) diff --git a/tests/experiments/test_g5_close.py b/tests/experiments/test_g5_close.py index 2305d2b..6a08270 100644 --- a/tests/experiments/test_g5_close.py +++ b/tests/experiments/test_g5_close.py @@ -235,7 +235,12 @@ def test_b2_planted_key_material_fails_leakage(mini_run_root): res = audit_leakage(mini_run_root) assert res["passed"] is False assert len(res["key_material_findings"]) == 1 - assert planted in res["key_material_findings"][0] + finding = res["key_material_findings"][0] + # The finding locates the leak and identifies it by prefix + length, but + # must not itself repeat the secret (findings are written into reports). + assert "pair_1 row seq=2" in finding and "sk-fak" in finding + assert f"({len(planted)} chars)" in finding + assert planted not in finding def test_b2_prompt_ordering_leak_fails_leakage(mini_run_root): diff --git a/tests/governance/test_audit_regressions.py b/tests/governance/test_audit_regressions.py new file mode 100644 index 0000000..07e06a2 --- /dev/null +++ b/tests/governance/test_audit_regressions.py @@ -0,0 +1,104 @@ +"""Regression tests for defects found in the 2026-11 system audit (AUDIT.md).""" +from __future__ import annotations + +import pytest + +from verdi.governance import ConstitutionStore, SanctionEngine, UniversalLaw +from verdi.governance.appeals import AppealEngine +from verdi.governance.constitution import ProposalNotInVoteError + +from .conftest import CONSTITUTION, LAW, make_sanction, make_vote + + +def _store(eligible=("a1", "a2", "a3", "a4")): + store = ConstitutionStore(UniversalLaw(LAW)) + store.register(CONSTITUTION) + store.set_eligible_voters(eligible) + return store + + +def _pass(store, pid, voters=("a1", "a2", "a3")): + for v in voters: + store.record_vote(make_vote(v, pid, 10, "for")) + return store.enact_if_valid([], 10, proposal_id=pid) + + +def test_two_amendments_enacted_together_both_take_effect(): + store = _store() + store.propose("p1", "a1", 10, "one", {"tax_rate": 0.2}) + store.propose("p2", "a2", 10, "two", {"max_transfer_units": 3}) + assert _pass(store, "p1").deterministically_valid + assert _pass(store, "p2").deterministically_valid + rules = store.current.rules + assert rules["tax_rate"] == 0.2 + assert rules["max_transfer_units"] == 3 + + +def test_caller_supplied_duplicate_votes_are_not_double_counted(): + store = _store(eligible=("a", "b")) + store.propose("p1", "a", 10, "x", {"tax_rate": 0.2}) + votes = [make_vote("a", "p1", 10, "for", vote_id=f"v{i}") for i in range(3)] + votes += [make_vote("b", "p1", 10, "against", vote_id=f"w{i}") for i in range(2)] + amendment = store.enact_if_valid(votes, 10, proposal_id="p1") + assert (amendment.for_votes, amendment.against_votes) == (1, 1) + + +def test_ineligible_ballots_do_not_decide_threshold(): + store = _store(eligible=("a1", "a2")) + store.propose("p1", "a1", 10, "x", {"tax_rate": 0.2}) + store.record_vote(make_vote("a1", "p1", 10, "against")) + store.record_vote(make_vote("a2", "p1", 10, "against")) + for outsider in ("z1", "z2", "z3"): + store.record_vote(make_vote(outsider, "p1", 10, "for")) + amendment = store.enact_if_valid([], 10, proposal_id="p1") + assert amendment.for_votes == 0 + assert not amendment.deterministically_valid + + +@pytest.mark.parametrize("payload", [{"quorum": 2}, {"amendment_threshold": 0}, + {"quorum": "half"}]) +def test_out_of_range_procedural_values_are_invalid_not_crashing(payload): + store = _store() + store.propose("p1", "a1", 10, "x", payload) + amendment = _pass(store, "p1") + assert not amendment.deterministically_valid + assert any(r.startswith("invalid_") for r in amendment.invalid_reasons) + assert len(store.versions()) == 1 + + +def test_proposal_cannot_be_enacted_twice(): + store = _store() + store.propose("p1", "a1", 10, "x", {"tax_rate": 0.2}) + _pass(store, "p1") + with pytest.raises(ProposalNotInVoteError): + store.enact_if_valid([], 11, proposal_id="p1") + + +def test_overlapping_sanctions_keep_rights_revoked_until_last_expires(): + engine = SanctionEngine() + engine.impose(make_sanction("s1", "a1", ["voting"], duration=10, effective_tick=0)) + engine.impose(make_sanction("s2", "a1", ["voting"], duration=20, effective_tick=0)) + assert engine.advance_tick(10) == ["s1"] + assert engine.rights.revoked_rights("a1", 12) == frozenset({"voting"}) + assert engine.has_live_sanction("a1") + assert engine.advance_tick(20) == ["s2"] + assert engine.rights.revoked_rights("a1", 20) == frozenset() + assert not engine.has_live_sanction("a1") + + +def test_disjoint_sanction_intervals_are_both_enforced(): + engine = SanctionEngine() + engine.impose(make_sanction("s1", "a1", ["voting"], duration=2, effective_tick=0)) + engine.impose(make_sanction("s2", "a1", ["voting"], duration=2, effective_tick=5)) + assert engine.rights.revoked_rights("a1", 1) == frozenset({"voting"}) + assert engine.rights.revoked_rights("a1", 3) == frozenset() + assert engine.rights.revoked_rights("a1", 6) == frozenset({"voting"}) + + +def test_upheld_appeal_keeps_rights_held_by_another_sanction(): + engine = SanctionEngine() + engine.impose(make_sanction("s1", "a1", ["voting"], duration=10, effective_tick=0)) + engine.impose(make_sanction("s2", "a1", ["voting"], duration=10, effective_tick=0)) + engine.end("s1") # the path an upheld appeal takes + assert engine.rights.revoked_rights("a1", 5) == frozenset({"voting"}) + assert AppealEngine(engine) is not None diff --git a/tests/society/test_audit_regressions.py b/tests/society/test_audit_regressions.py new file mode 100644 index 0000000..9ac222b --- /dev/null +++ b/tests/society/test_audit_regressions.py @@ -0,0 +1,152 @@ +"""Regression tests for defects found in the 2026-11 system audit (AUDIT.md).""" +from __future__ import annotations + +import asyncio +import json +from types import SimpleNamespace + +from verdi.society.memory import AgentMemory +from verdi.society.tasks import _parse_orchestrator_response, generate_llm_tape + +from .test_tasks import make_agent, make_state + + +def _state(): + return make_state({a: make_agent(a, edges=["a1", "a2"]) for a in ("a1", "a2")}) + + +def test_one_bad_amount_does_not_discard_the_orchestrated_tape(): + content = json.dumps({ + "a1": {"action_type": "produce", "amount": None}, + "a2": {"action_type": "produce", "amount": "two"}, + }) + tasks = _parse_orchestrator_response(content, _state(), 1) + assert tasks is not None and len(tasks) == 2 + + +def test_transfer_to_self_or_unknown_becomes_a_valid_task(): + content = json.dumps({ + "a1": {"action_type": "transfer", "resource": "food", "amount": 1, "to": "a1"}, + "a2": {"action_type": "transfer", "resource": "food", "amount": 1, "to": None}, + }) + tasks = _parse_orchestrator_response(content, _state(), 1) + assert tasks is not None + assert all(t.action_type != "transfer" for t in tasks) + + +class _Caller: + def __init__(self, content: str): + self._content = content + + async def generate_structured(self, messages, **kwargs): + return SimpleNamespace(content=self._content, usage={"prompt_tokens": 11, + "completion_tokens": 5}, trace_id="t-1") + + +def test_unusable_orchestrator_output_still_reports_call_meta(): + tasks, used_llm, meta = asyncio.run(generate_llm_tape(1, _state(), _Caller("not json"))) + assert not used_llm and tasks + # Tokens were spent: the campaign must be able to record the call. + assert meta["prompt_tokens"] == 11 + assert "parse_error" in meta + + +def test_consolidated_summary_excludes_recent_entries(): + mem = AgentMemory() + for i in range(30): + mem.append_reflection("a1", f"r{i}") + mem.apply_consolidation("a1", "SUM") + assert mem.consolidated_summary("a1") == "SUM" + assert mem.consolidated_summary("nobody") == "" + + +def test_watchdog_counts_in_process_calls_during_long_batches(tmp_path, monkeypatch): + # Reflection/consolidation batches write model_calls.jsonl only at the + # end; completed calls must still reset the stall deadline. + import verdi.society.live as live_module + + (tmp_path / "events.jsonl").write_text("") + monkeypatch.setattr(live_module, "_HEARTBEAT_INTERVAL_SECONDS", 0.05) + monkeypatch.setattr("os._exit", lambda code: (_ for _ in ()).throw(SystemExit(code))) + calls = {"n": 0} + + def tick_calls(): + calls["n"] += 1 + return calls["n"] + + async def scenario(): + watchdog = asyncio.create_task( + live_module._stall_watchdog(tmp_path, 0.2, extra_progress=tick_calls)) + await asyncio.sleep(0.6) + assert not watchdog.done(), "watchdog fired despite completed calls" + watchdog.cancel() + + asyncio.run(scenario()) + + +def test_prompt_shows_lifecycle_tick_not_ledger_counter(): + from verdi.schemas import Task + from verdi.society.prompting import build_social_prompt_messages + + state = make_state({"a1": make_agent("a1")}, tick=104) + task = Task(task_id="soc-t07-00", actor_id="a1", action_type="produce", + counterpart_ids=[], constraints={"resource": "food", "amount": 1}, + public_observations=["produce 1 food"], tick_created=7) + user = build_social_prompt_messages("a1", task, state, AgentMemory())[1]["content"] + assert "It is now tick 7." in user + assert "tick 104" not in user + + +def test_metrics_count_only_accepted_sanctions_and_filed_appeals(): + from verdi.society.campaign import _compute_metrics + + result = SimpleNamespace( + probes=[], constitution_versions=[], quarantined=[], assembly_quarantined={}, + assembly_outcome=None, restoration_events=[], model_calls=[], + final_state_hash="h", + sanction_events=[ + {"kind": "sanction_imposed", "decision": "accepted"}, + {"kind": "sanction_imposed"}, # t0 declaration (no decision field) + {"kind": "sanction_imposed", "decision": "rejected"}, + {"kind": "world_shock", "resource": "food"}, + ], + appeal_events=[{"phase": "filed"}, {"phase": "adjudicated"}], + ) + metrics = _compute_metrics(result, []) + assert metrics["sanctions"] == 2 + assert metrics["appeals"] == 1 + + +def test_inner_life_prompt_uses_lifecycle_tick(): + from verdi.society.inner_life import build_inner_life_prompt_messages + + state = make_state({"a1": make_agent("a1")}, tick=104) + user = build_inner_life_prompt_messages("a1", state, AgentMemory(), tick=7)[1]["content"] + assert "tick 7" in user and "tick 104" not in user + + +def test_probe_subjects_follow_send_order_not_completion_order(): + from verdi.society.campaign import _RecordingCaller + + class _Slow: + async def generate_structured(self, messages, **kw): + await asyncio.sleep(0.05 if messages[0]["content"] == "first" else 0) + return SimpleNamespace(content="yes", usage={}, trace_id=messages[0]["content"]) + + campaign = SimpleNamespace(_tick=0, _model_calls=[], + _model_fingerprint=lambda: "fp", + _write_model_calls_lines=lambda lines: None) + subjects = iter(["alice", "bob"]) + rec = _RecordingCaller(_Slow(), campaign, "probe", "probe", + subject_provider=lambda: next(subjects)) + + async def run(): + await asyncio.gather( + rec.generate_structured([{"role": "user", "content": "first"}]), + rec.generate_structured([{"role": "user", "content": "second"}]), + ) + + asyncio.run(run()) + # "second" completes first, but it was sent by bob. + by_trace = {c.request_id: c.subject for c in campaign._model_calls} + assert by_trace == {"first": "alice", "second": "bob"} diff --git a/tests/society/test_economy_campaign.py b/tests/society/test_economy_campaign.py index 21f0de9..50c7efe 100644 --- a/tests/society/test_economy_campaign.py +++ b/tests/society/test_economy_campaign.py @@ -352,3 +352,15 @@ def test_legacy_scenario_without_economy_block_unchanged(tmp_path): assert not [s for s in res.tape if s.kind == "production_cost"] assert res.replay_matches is True assert res.replay_log_identical is True + + +def test_production_cap_is_per_agent_not_society_wide(tmp_path): + """Audit F33: the cap is documented per agent per tick. Two producers each + producing exactly the cap (2.0) in the same tick must both be accepted; + the campaign used to sum production across agents, so the second + producer was rejected with law:production_limit.""" + res = _run(tmp_path, "atcap", amend=False, produce_amount=2.0) + rejected = [e for e in res.events + if e.kind == "action_rejected" and e.payload.get("blocked_by") == "law:production_limit"] + assert rejected == [] + assert res.replay_matches is True diff --git a/tests/test_cli.py b/tests/test_cli.py index c86d367..bddb3ee 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -15,4 +15,17 @@ def test_cli_runs() -> None: check=False, ) assert result.returncode == 0 - assert "verdi CLI scaffold" in result.stdout + assert "usage: verdi" in result.stdout + for command in ("init", "run", "sweep"): + assert command in result.stdout + assert "IMPLEMENTATION_PLAN" not in result.stdout + + +def test_run_respects_run_dir_equals_form(monkeypatch) -> None: + import verdi.cli as cli + import verdi.society.live as live + + seen: list[list[str]] = [] + monkeypatch.setattr(live, "main", lambda argv: seen.append(argv) or 0) + assert cli.main(["run", "examples/starter.yaml", "--run-dir=out/mine"]) == 0 + assert seen == [["--scenario", "examples/starter.yaml", "--run-dir=out/mine"]] diff --git a/tests/world/test_audit_regressions.py b/tests/world/test_audit_regressions.py new file mode 100644 index 0000000..44e1307 --- /dev/null +++ b/tests/world/test_audit_regressions.py @@ -0,0 +1,137 @@ +"""Regression tests for defects found in the 2026-11 system audit (AUDIT.md).""" +from __future__ import annotations + +import pytest + +from verdi.schemas import AgentState, ProposedAction, ValueProfile, WorldState +from verdi.world.reducer import InvariantViolationError, WorldReducer +from verdi.world.validate import validate_action + +_VALUES = ValueProfile( + integrity=0.8, fairness=0.9, cooperation=0.7, autonomy=0.5, solidarity=0.85 +) + + +def _state() -> WorldState: + def agent(aid: str) -> AgentState: + return AgentState(agent_id=aid, role="citizen", persona_summary="p", + declared_values=_VALUES, resources={"food": 10.0}) + + return WorldState(world_id="w", tick=1, event_cursor=1, law_version="law-1", + constitution_version="cons-1", + agents={"a": agent("a"), "b": agent("b")}, + world_resources={"food": 5.0}) + + +def _action(kind: str, **arguments) -> ProposedAction: + return ProposedAction(actor_id="a", action_kind=kind, arguments=arguments, task_id="t-1") + + +def test_self_transfer_is_rejected_without_a_sanctionable_code(): + result = validate_action(_state(), _action("transfer", to="a", resource="food", units=10)) + assert not result.accepted + assert result.attempted_violation_codes == ["self_transfer"] + + +def test_reducer_refuses_self_transfer_instead_of_minting_resources(): + with pytest.raises(InvariantViolationError, match="self-transfer"): + WorldReducer._transfer( + _state().agents["a"], "a", dict(_state().agents), + {"to": "a", "resource": "food", "units": 10}, + ) + + +@pytest.mark.parametrize("kind, args", [ + ("transfer", {"to": "b", "resource": ["food"], "units": 1}), + ("deposit", {"resource": None, "amount": 1}), + ("withdraw", {"resource": {"x": 1}, "amount": 1}), + ("produce", {"resource": ["food"], "amount": 1}), +]) +def test_non_string_resource_is_rejected_not_crashing(kind, args): + result = validate_action(_state(), _action(kind, **args)) + assert not result.accepted + assert result.attempted_violation_codes == ["missing_argument"] + + +def test_branch_from_a_prefix_ending_in_a_rejected_event(): + from tests.world.helpers import accepted, action, initial_world, rejected_double_spend + + from verdi.world.branch import BranchSpec, branch_from_snapshot + from verdi.world.log import EventLog + from verdi.world.replay import TapeStep, replay + + log = EventLog() + state, _ = replay(log, initial_world(), [ + TapeStep(action=action("a1", "p-1", "emit_signal", text="s0"), validation=accepted()), + TapeStep(action=action("a1", "p-2", "transfer", recipient="a2", amount=1.0), + validation=rejected_double_spend()), + ]) + assert log.events[-1].after_hash is None + spec = BranchSpec(origin_state_hash=WorldReducer.canonical_state_hash(state), + origin_event_count=len(log), origin_tick=state.tick, + branch_id="b", treatment_seed=1) + tape = [TapeStep(action=action("a2", "t-1", "emit_signal", text="x"), validation=accepted())] + result = branch_from_snapshot(state, log, tape, spec) + assert len(result.log) == len(log) + 1 + + +def test_snapshot_store_rejects_traversal_and_tampering(tmp_path): + from tests.world.helpers import initial_world + + from verdi.world.snapshot import SnapshotError, SnapshotStore + + store = SnapshotStore(tmp_path) + digest = store.save(initial_world()) + assert store.load(digest) == initial_world() + with pytest.raises(SnapshotError, match="invalid snapshot digest"): + store.load("../" + digest[3:]) + assert not store.contains("../../etc/passwd") + snap = tmp_path / "snapshots" / f"{digest}.json" + tampered = snap.read_text().replace('"a1"', '"zz"', 1) + snap.write_text(tampered) + with pytest.raises(SnapshotError, match="content hash"): + SnapshotStore(tmp_path).load(digest) + + +def test_produce_policy_cap_is_cumulative_per_tick_without_economy(): + rules = {"max_produce_units_per_tick": 5} + ok = validate_action(_state(), _action("produce", resource="food", amount=5), + policy_rules=rules, produced_this_tick={"food": 0.0}) + over = validate_action(_state(), _action("produce", resource="food", amount=5), + policy_rules=rules, produced_this_tick={"food": 5.0}) + assert ok.accepted + assert not over.accepted and over.blocked_by == "constitution:max_produce_units_per_tick" + + +def test_role_only_consumption_override_is_rejected(): + from verdi.world.economy import parse_economy_spec + + with pytest.raises(ValueError, match="consumption_per_tick"): + parse_economy_spec({"version": 1, "consumption_per_tick": {"food": 1}, + "role_consumption": {"guard": {"water": 2}}}) + + +def test_amount_that_would_overflow_a_balance_is_rejected(): + big = _state().model_copy(update={"agents": { + **_state().agents, + "a": _state().agents["a"].model_copy(update={"resources": {"food": 1e308}})}}) + result = validate_action(big, _action("produce", resource="food", amount=1e308)) + assert not result.accepted + + +@pytest.mark.parametrize("kind, args", [ + ("produce", {"resource": "food", "amount": True}), + ("deposit", {"resource": "food", "amount": True}), +]) +def test_boolean_amount_is_not_a_quantity(kind, args): + assert not validate_action(_state(), _action(kind, **args)).accepted + + +def test_duplicate_objective_ids_are_rejected(): + from verdi.world.economy import parse_task_objectives + + with pytest.raises(ValueError, match="duplicate objective_id"): + parse_task_objectives({"deliveries": [ + {"objective_id": "obj-01", "due_tick": 4, "resource": "food", "amount": 1}, + {"due_tick": 6, "start_tick": 3, "resource": "food", "amount": 1}, # auto obj-01 + ]})