Skip to content

[Bug] Admission writer: misleading merge cancellations and repeated HTTP 422 invalidation failures #203

Open
Open
[Bug] Admission writer: misleading merge cancellations and repeated HTTP 422 invalidation failures#203
Assignees
Dong1017
Labels
area:knowledgeKnowledge line: packs, skill library, knowledge admissionkind:taskBounded delivery task with a delegated slice

Description

@Dong1017
Dong1017
opened on Oct 28, 2026

Problem and scope

Corrective follow-up to #50 / #81 / #140 / #154. Recent main commits show red crosses from two different causes: cancelled duplicate declaration writers, and genuinely incomplete admission sweeps. They must not be conflated with failed code tests or hidden by unconditional success.

Investigated read-only on 2026-10-28 against main 31db51cb7851fc7d68ff2c1098ad58318315905b. No workflow dispatch, status publication experiment, settings change or paid execution was performed.

A. Merge-triggered cancellation noise: root cause established

Merging #97 triggered both:

The workflow puts eligible events in one global knowledge-policy-status-writer concurrency group with cancel-in-progress: false. That protects the running writer, not the default single pending slot. New pending work can replace pending work. GitHub concurrency documentation.

Latest five main commits sampled: all merge-lane jobs passed; push declaration runs were four cancelled and one failed. The failed one was #153 push 40615822077. #97 merge lane passed. Merge lane covers manifest/install/smoke/claim checks, not the full Ready suite.

This is normal GitHub queue behavior but an InferRSI trigger/observability problem. A successor run's success does not rewrite the cancelled run attached to the original commit.

B. Repeated HTTP 422 invalidation failures: failure path established, server reason unresolved

The successor #97 run still reports global execution unfinished: five invalidation-failed targets (#51, #55, #90, #117, #132), plus the then-noncompliant #113/old #199. Its green exit is the trigger-PR projection, NOT a globally healthy sweep. #199's declaration subsequently changed; that historical rejection is not asserted to remain current.

The failing operation is POST /repos/InferMatrixLab/InferRSI/statuses/{head}, setting knowledge-model-policy to pending with description Admission evaluation in progress. See invalidated().

Read-only pagination (100 entries/page, nine full pages then an empty tenth) returned 900 entries for this context on EACH affected head. This is an observed correlation, NOT proof of a 900-entry service limit, nor proof of exhausting the documented/configured 1000-entry budget. Ready #117/#132 fail too, so draft state is not an adequate explanation.

Current code:

  • STATUS_LIMIT=1000, STATUS_RESERVE=8; 900 entries pass the pre-write budget check.
  • GitHubHTTPError safely redacts raw response data and recognizes only one exact capacity error shape; retained logs classify these responses as validation / reason unknown.
  • Invalidation catches API errors as retryable invalidation-failed; the runner retries incomplete work three times. Repeated deterministic rejection needs an explicit bounded diagnosis/recovery path.
  • Failure to invalidate leaves existing remote status in place. At inspection knowledge: add source-grounded Lithos Metal candidate skills #117's latest status was pending and knowledge: Drama-3 decode graphs on Ascend 910B1 #132's was success. Neither is proof of a fresh evaluation. Preserve this uncertainty; do not claim a safe overwrite occurred.
Affected exact heads and final-attempt GitHub request IDs
PR Head Request ID
#51 3c1b2ff969537f3216d0125d034ae903d0743e7a F00F:348253:6C93FE:74A9F1:6AE1536B
#55 c69961bd9814beb93608f4f001a2ab3eae5cf26f F013:949C0:6B2265:733CA9:6AE15367
#90 86f003364804d116b3202c4d807cc898a3a02ade F015:177C34:6D39A4:7547D9:6AE15363
#117 c1dc3b1c235005bbbc7efc6e7ba3d3f3d2c93669 F00B:351629:6CE9F9:7501C6:6AE1535E
#132 951b218b2f708e424a5453dd01da3fbe15077d91 F006:348805:6AF27F:72FBE6:6AE1535A

Source: final attempt in 40616176059. All five logged HTTP 422 / validation / unknown; primary rate-limit remaining was positive (3277–3321). The write precedes declaration assessment; this failure is not a policy verdict. Existing tests cover locally counted limits and transient invalidation errors, but the observed below-limit validation rejection needs dedicated coverage. Do not force fresh empty commits or rename the context as a workaround before establishing the cause and migration safety.

Required outcome

Main commit health must be interpretable: code checks, per-PR declaration admission and global writer/sweep health remain distinguishable.

flowchart TD
 M[Merge event] --> P[Push trigger]
 M --> R[PR closed trigger]
 P --> Q[Single writer queue]
 R --> Q
 Q --> T[Trigger PR result]
 Q --> G[Global sweep health]
 G --> I[Status invalidation POST]
 I --> E[422: diagnose and bounded recovery]
 T --> V[Readable commit and PR evidence]
 G --> V
Loading

Acceptance criteria

  • Reproduce and document merge-event queue behavior; normal duplicate triggers no longer produce misleading failed/cancelled commit health, while relevant lifecycle/body/approval/main changes remain covered.
  • Retain a single serialized status writer and shared-head safety. Evaluate deduplication/event routing or supported queuing (e.g. queue: max) with bounded backlog/API cost; do not assume queuing alone fixes 422.
  • Establish each 422 cause using safely allowlisted diagnostics, operation, head/context, counts/completeness and request IDs. Do not log credentials or arbitrary raw response bodies; unknown reasons remain unknown.
  • Handle confirmed non-retryable/capacity conditions without blind identical retries. Preserve fail-closed semantics and provide an honest recovery path when GitHub cannot replace a stale status. No arbitrary lowering of the budget based only on the 900-count observation.
  • Add regressions for duplicate merge triggers, invalidation failure, recognized capacity response, unknown 422, retained old success/pending, and trigger-PR versus global-health reporting.
  • Verify applicable software tests and, through separately authorized safe validation, a normal merge/event sequence and affected failure/recovery paths. Link exact SHAs/runs/statuses. A green trigger job alone does not complete global recovery.

Non-goals

No model admission relaxation, blanket success, hidden failure, required-check removal, repository settings/credential change, model/hardware experiments, or retroactive relabeling of historical results. This issue records diagnosis and corrective scope; implementation ownership is not yet claimed.

Activity

  1. Dong1017
    added
    kind:taskBounded delivery task with a delegated slice
    area:knowledgeKnowledge line: packs, skill library, knowledge admission
    on Oct 28, 2026

  2. Dong1017 commented on Oct 28, 2026

    @Dong1017
    Dong1017
    on Oct 28, 2026 – with ChatGPT Codex Connector
    CollaboratorAuthor

    Claiming the bounded corrective implementation for #203 on behalf of @Dong1017, per the owner's request. I will use an isolated cloud checkout, not consume or replace the occupied local agent sessions. Scope: investigate status-write rejection, improve safe diagnostics/bounded handling, and remove duplicate merge-trigger cancellation noise while preserving serialized publication and honest per-PR/global results. Tests and exact-head PR review precede any merge. No model/hardware calls, credentials/settings changes or blanket-green workaround.

    flowchart LR
 A[203 observed failures] --> B[Isolated corrective changes]
 B --> C[Regression tests and independent review]
 C --> D[Scoped PR]
    Loading
  3. Dong1017
    self-assigned this
    on Oct 28, 2026

  4. Dong1017 commented on Oct 28, 2026

    @Dong1017
    Dong1017
    on Oct 28, 2026 – with ChatGPT Codex Connector
    CollaboratorAuthor

    Temporary mitigation delivered; historical 422 recovery remains open

    #204 merged as 0f85fa8f86b0139167d54785eeda9e038d30859e. Independent delegated review, Ready four jobs, focused 181 tests/65 subtests, full local software suite, Ruff and remote-tree verification passed.

    Live results after merge:

    flowchart LR
 A[Two merge events] --> B[Both retained and serialized]
 B --> C[One attempted write per rejected target]
 C --> D[Explicit global failure, no blind retries]
 D --> E[422 cause and remote recovery still open]
    Loading

    This verifies the temporary queue/retry mitigation, not end-to-end recovery. Keep #203 open. Next diagnostic step is to identify the response shape/rejection reason using strictly bounded safe metadata and retained request IDs, without logging raw content or guessing a new numeric cap. Historical cancelled/failed run records are not retroactively rewritten.

  5. Dong1017 commented on Oct 28, 2026

    @Dong1017
    Dong1017
    on Oct 28, 2026 – with ChatGPT Codex Connector
    CollaboratorAuthor

    Follow-up #205 merged as 2c1da9dc8ea055f4870843a847cefa9d7564f05e; new live diagnostics now narrow all five 422s to valid JSON objects with errors as a string, not the structured error-array shape expected by the existing classifier. The 8 KiB read limit was not reached. This rules out the earlier parse/truncation hypotheses for these responses; it still does not disclose or prove the string's meaning.

    flowchart LR
 A[Unknown 422] --> B[JSON parsed successfully]
 B --> C[errors is string, not array]
 C --> D[Exact allowlisted-message compatibility check next]
    Loading

    Next bounded correction will recognize the already-allowlisted capacity message in the observed string shape without accepting arbitrary text or changing capacity numbers. Keep failure semantics and #203 open until the live reason and recovery are established.

  6. Dong1017 commented on Oct 28, 2026

    @Dong1017
    Dong1017
    on Oct 28, 2026 – with ChatGPT Codex Connector
    CollaboratorAuthor

    Live verification after #206: capacity hypothesis not confirmed

    Run 40630096445, policy SHA 5bea15760982277925e33c0e8deef85035473012, completed with the same five status-rejected results and rerun=false. Responses are valid JSON objects, errors is a string, the 8 KiB read bound is not reached, but none matches the existing exact allowlisted capacity message. Therefore neither the 900 visible statuses nor string-shaped errors establish capacity exhaustion. The global failure remains truthful.

    flowchart LR
 A[Parsed string error verified] --> B[Exact capacity-message test]
 B --> C[No match on five heads]
 C --> D[Server rejection text still needed privately]
    Loading

    The queue/identical-retry mitigations are verified, and response-format compatibility is in place. The unresolved requirement is the actual rejection reason: it requires one safely obtained, redacted server error payload or GitHub-side investigation using the recorded request IDs, not more guessed message branches. Do not dump arbitrary responses into public workflow logs, lower the numerical budget based on correlation, rename contexts to bypass state, or mark these heads compliant. #203 stays open; no historical status recovery is claimed.

  7. Dong1017 commented on Oct 28, 2026

    @Dong1017
    Dong1017
    on Oct 28, 2026 – with ChatGPT Codex Connector
    CollaboratorAuthor

    Actual rejection reason established for PR #51

    Owner-authorized one-shot encrypted capture in #212, merge 6e19e680df60c798b426771d2a57a923293e5d1f, run 40652228175, recovered this generic server error after private decryption:

    Validation failed: This SHA and context has reached the maximum number of statuses.

    This is a direct errors string. The exact classifier previously missed the Validation failed: prefix. Capacity is now proven for #51's head 3c1b2ff969537f3216d0125d034ae903d0743e7a and knowledge-model-policy; the other four heads are not yet proven by this capture. The 900 visible entries alone were not the evidence.

    flowchart LR
 A[Repeated status appends] --> B[SHA/context capacity reached]
 B --> C[Existing head rejects further status writes]
 C --> D[Prevent unnecessary writes]
 C --> E[Supported recovery required]
    Loading

    The capture used the existing failed POST, with no additional status request. Only encrypted diagnostic bytes were retained; no token, request headers or raw response body were captured. Temporary instrumentation will now be removed and the observed exact message added to the safe classifier.

    GitHub documents a 1,000-status limit per repository/SHA/context. Its status API exposes no documented deletion/update operation that resets this limit. Repeating the same POST, waiting, or calling a classifier fix a recovery is incorrect. Prevention and recovery must be separated; no context rotation, synthetic empty commit or false success is being proposed as a bypass.

    #208's live close-event routing was verified (closed-impact, target 208 only); its policy-changing push correctly retained full refresh. #203 remains open pending prevention and a supported recovery path for exhausted heads.

  8. Dong1017 commented on Oct 28, 2026

    @Dong1017
    Dong1017
    on Oct 28, 2026 – with ChatGPT Codex Connector
    CollaboratorAuthor

    Post-merge verification at 57f47bea46f2048321907bd6b192683e3e636311: #214 and #216 merged; both merge lanes and all four declaration runs completed successfully.

    flowchart LR
 A[Main software checks] --> B[Passed]
 C[Reconciliation execution] --> D[Completed with findings]
 D --> E[Five old heads still saturated]
 E --> F[Genuine corrective or integration update]
 F --> G[Fresh-head evaluation and recovery receipt pending]
    Loading

    The misleading main-commit health defect is verified corrected. This is not global admission recovery: an old remote success may remain where GitHub refuses further writes. Keeping the issue open for the previously recorded affected-head recovery verification; no empty commits, context rotation, required-check removal or acceptance relaxation is proposed. A genuine fix/integration revision of an affected PR should be evaluated normally and linked here as the remaining recovery receipt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:knowledgeKnowledge line: packs, skill library, knowledge admissionkind:taskBounded delivery task with a delegated slice

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions