Repository navigation
fix(canary): testable evaluator; unresolved streaks fail instead of passing green - #336
Closed
labgadget015-dotcom wants to merge 4 commits into
Closed
labgadget015-dotcom wants to merge 4 commits into
labgadget015-dotcom wants to merge 4 commits into
Conversation
Port the intake-canary probe from inline bash to a stdlib-only script so its invariants can be tested: - fresh failure (<= ALERT_WINDOW_MINUTES, inclusive) -> exit 1, Slack - stale streak -> exit 0, ::warning::, status=stale_incident carrying the failing execution id and age; never status=healthy - missing key, non-200, transport error, bad shape, missing/unparseable timestamps, unrecognised execution status -> UNKNOWN (canary_broken), exit 1 - freshness judged per workflow; correlation id taken from the failing workflow's own response (v1 used the router's for DRC-only failures) - running/queued executions at the head no longer hide a streak behind them - one JSON evidence object per run (stdout, step output, step summary); ids, statuses, timestamps and counts only, as the repo is public 57 tests, including parity with every bash branch and the freshness boundary; mutation-checked (stale->healthy, <= -> <, dropped transport check, ignored unknown status, router-only freshness all caught). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With PR #316 a streak older than 60 minutes emitted a warning and then status=healthy / 'Pipeline healthy.'. Live on 2026-09-23 (run 35815360651) that path reported healthy while the Event Router had 20/20 failed executions and the DRC 12 consecutive failures. Switch the probe step to .github/scripts/intake_canary.py. The step id and the status/detail/slack outputs are unchanged, so Alert Slack is untouched; status gains the value stale_incident, and verdict/evidence are new. Adds a sparse, credential-free checkout of the script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An exception outside the handled transport errors (e.g. http.client.IncompleteRead, or a malformed threshold env var) exited 1 with a traceback and no status/slack/evidence outputs, so the Slack reminder and JSON evidence were lost. The bash probe routed curl failures to canary_broken; restore that behaviour. - catch http.client.HTTPException in fetch_executions - main() wraps evaluation; any crash becomes UNKNOWN/canary_broken with outputs, evidence and summary still written Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iting 0 WARN_STALE (open streak, newest failure outside the alert window, no success since) exited 0 with a warning, so the job stayed green through the 2026-09-23 n8n Cloud quota outage while every execution errored. Stale incidents now exit 1; Slack is paged only on the ~4/day reminder gate so a long outage does not spam. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017DsEH4FYupGjqJ693GEGn6
Contributor
📊 Code Complexity AnalysisSummary:
|
| File | Function | Complexity | Line |
|---|---|---|---|
core/risk_scorer.py |
score_pull_request |
35 | 141 |
autopilot/autopilot.py |
generate_summary |
24 | 195 |
.github/scripts/inline_pr_commenter.py |
_is_likely_safe_b608 |
19 | 252 |
.github/scripts/threshold_monitor.py |
_is_likely_safe_b608 |
19 | 169 |
autopilot/staleness_engine.py |
process_stale_prs |
16 | 281 |
.github/scripts/intake_canary.py |
analyse |
16 | 115 |
.github/scripts/intake_canary.py |
evaluate |
16 | 164 |
autopilot/ai_optimization/performance_monitor.py |
get_benchmark_stats |
15 | 184 |
.github/scripts/weekly_digest.py |
build_blocks |
15 | 38 |
.github/scripts/batch_scan_dependabot.py |
main |
15 | 64 |
... and 23 more
Recommendations:
- Break down large functions into smaller, focused units
- Extract complex conditional logic into separate functions
- Use early returns to reduce nesting
🔧 Low Maintainability Files
These files have low maintainability scores and may need refactoring:
| File | Score | Status |
|---|---|---|
.github/scripts/health_dashboard_generator.py |
28.14 | 🔴 |
.github/scripts/threshold_monitor.py |
32.4 | 🔴 |
.github/scripts/workflow_monitor.py |
33.73 | 🔴 |
.github/scripts/ai_code_suggestor.py |
33.76 | 🔴 |
.github/scripts/ai_workflow_optimizer.py |
35.51 | 🔴 |
.github/scripts/inline_pr_commenter.py |
35.51 | 🔴 |
.github/scripts/performance_benchmark.py |
39.46 | 🔴 |
.github/scripts/self_healing_system.py |
40.27 | 🔴 |
.github/scripts/parallel_code_analyzer_optimized.py |
41.16 | 🔴 |
autopilot/tests/test_recommendation_contract.py |
42.05 | 🔴 |
.github/scripts/intake_canary.py |
42.28 | 🔴 |
autopilot/autopilot.py |
42.45 | 🔴 |
autopilot/ai_optimization/anomaly_detector.py |
42.56 | 🔴 |
.github/scripts/refactoring_assistant.py |
43.03 | 🔴 |
autopilot/ai_optimization/intelligent_cache.py |
43.28 | 🔴 |
autopilot/ai_optimization/commit_summarizer.py |
44.05 | 🔴 |
.github/scripts/async_parallel_analyzer.py |
44.47 | 🔴 |
autopilot/ai_optimization/performance_monitor.py |
44.69 | 🔴 |
.github/scripts/badge_generator.py |
45.28 | 🔴 |
.github/scripts/copilot_integration.py |
45.37 | 🔴 |
.github/scripts/distributed_monitoring.py |
45.53 | 🔴 |
autopilot/dependency_graph.py |
45.65 | 🔴 |
.github/scripts/elite_copilot.py |
45.69 | 🔴 |
.github/scripts/issue_auto_creator.py |
46.39 | 🔴 |
.github/scripts/cost_calculator.py |
46.4 | 🔴 |
.github/scripts/complexity_reporter.py |
46.78 | 🔴 |
.github/scripts/pr_triage.py |
47.13 | 🔴 |
core/risk_scorer.py |
48.15 | 🔴 |
autopilot/ai_optimization/nlp_relevance_filter.py |
48.43 | 🔴 |
.github/scripts/pr_inline_commenter.py |
48.47 | 🔴 |
.github/scripts/dependency_audit.py |
48.7 | 🔴 |
autopilot/staleness_engine.py |
48.73 | 🔴 |
.github/scripts/dependency_updater.py |
48.91 | 🔴 |
.github/scripts/metrics_collector.py |
48.91 | 🔴 |
autopilot/ai_optimization/ml_priority_scorer.py |
49.53 | 🔴 |
.github/scripts/parallel_code_analyzer.py |
49.96 | 🔴 |
autopilot/ai_optimization/api_optimizer.py |
50.46 | 🟡 |
autopilot/recommendation_contract.py |
51.32 | 🟡 |
.github/scripts/workflow_optimizer.py |
51.67 | 🟡 |
.github/scripts/cot_selector.py |
51.73 | 🟡 |
.github/scripts/release_manager.py |
51.92 | 🟡 |
.github/scripts/check_quality.py |
52.33 | 🟡 |
.github/scripts/auto_pr.py |
52.72 | 🟡 |
.github/scripts/changelog_generator.py |
53.13 | 🟡 |
.github/scripts/notification_manager.py |
53.58 | 🟡 |
.github/scripts/prometheus_exporter.py |
54.96 | 🟡 |
.github/scripts/weekly_digest.py |
55.02 | 🟡 |
core/audit_logger.py |
55.6 | 🟡 |
.github/scripts/llm_router.py |
55.85 | 🟡 |
.github/scripts/batch_scan_dependabot.py |
56.3 | 🟡 |
core/llm_provider.py |
56.32 | 🟡 |
autopilot/decisions/ledger.py |
56.33 | 🟡 |
.github/scripts/streaming_results.py |
56.64 | 🟡 |
.github/scripts/setup_branch_protection.py |
57.0 | 🟡 |
.github/scripts/gather_context.py |
57.53 | 🟡 |
.github/scripts/optimized_github_client.py |
58.27 | 🟡 |
agents/orchestrator_agent.py |
59.02 | 🟡 |
core/incident_freeze.py |
59.67 | 🟡 |
core/github_client.py |
61.96 | 🟡 |
core/message_queue.py |
63.22 | 🟡 |
core/agent_config.py |
63.82 | 🟡 |
core/idempotency.py |
64.45 | 🟡 |
Maintainability Index Guide:
- 🟢 85-100: Excellent maintainability
- 🟡 65-84: Good maintainability
- 🟠 50-64: Moderate maintainability (consider refactoring)
- 🔴 0-49: Poor maintainability (needs refactoring)
Contributor
✅ Quality Ratchet — PASSEDNo new ruff violations introduced by this PR. 🎉 |
Contributor
🤖 Elite AI Copilot AnalysisElite AI Copilot Analysis ReportGenerated: 2026-09-30 02:23:55 🎯 Health Score: 100.0/100🚀 Top Recommendations
📊 Detailed InsightsCode Quality Baseline Established
Security Scan Initiated
Repository Structure Analyzed
Performance Baseline Captured
Documentation Structure Good
Powered by Elite AI Copilot v1.0 |
Contributor
Code Quality Analysis ❌ FAILEDDuration: 0.02s Tool Results
View detailed results{
"timestamp": "2026-09-30 02:24:03",
"elapsed_seconds": 0.02,
"summary": {
"total_issues": 10,
"critical": 0,
"high": 0,
"medium": 0,
"low": 0
},
"tools": {
"pylint": {
"status": "failed",
"output": "",
"errors": "Pylint error: [Errno 2] No such file or directory: 'pylint'"
},
"flake8": {
"status": "failed",
"output": "",
"errors": "Flake8 error: [Errno 2] No such file or directory: 'flake8'"
},
"bandit": {
"status": "failed",
"output": "",
"errors": "Bandit error: [Errno 2] No such file or directory: 'bandit'"
},
"radon_cc": {
"status": "failed",
"output": "",
"errors": "Radon error: [Errno 2] No such file or directory: 'radon'"
},
"radon_mi": {
"status": "failed",
"output": "",
"errors": "Radon MI error: [Errno 2] No such file or directory: 'radon'"
}
},
"passed": false
} |
Contributor
🔍 Pre-commit Checks✅ All pre-commit checks passed! Your code follows the project style guidelines. Pre-commit hooks help maintain code quality and consistency. |
Contributor
🔒 Security Scan Results🛡️ Bandit Security Scan
📦 Dependency Vulnerabilities
Vulnerable Dependencies:
Security scans run automatically on every PR. View detailed reports in the Actions tab. |
github-actions
Bot
added
testing
maintenance
Contributor
🟡 Risk Assessment: MEDIUM (3.5/10)Analysed 3 files, 759+ / 155− lines. Security-sensitive paths detected. Test coverage unchanged or improved. Scoring breakdown
|
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The canary’s data source has a stated cutover limitation, and the PR requires Gadget’s authorization before merging during that window.
Review effort: Balanced
Findings: 2
Open (4)
What changed in this PR
This PR moves the Intake Canary’s health checks into a testable Python evaluator and makes unresolved failure streaks fail the job instead of appearing healthy.
Changes:
- Evaluates execution freshness per workflow and emits structured evidence.
- Fails on stale, unresolved streaks while limiting Slack reminders.
- Adds unit tests for verdicts, API failures, and workflow outputs.
| File | Description |
|---|---|
tests/unit/test_intake_canary.py |
Tests evaluator decisions and output handling. |
.github/workflows/intake-canary.yml |
Runs the Python evaluator in place of shell logic. |
.github/scripts/intake_canary.py |
Implements health evaluation and evidence output. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+40
to
+41
| FAILURE_STATUSES = frozenset({"error", "crashed"}) | ||
| SKIPPED_STATUSES = frozenset({"new", "running", "waiting", "canceled"}) |
Comment on lines
+148
to
+154
| started = [ | ||
| item.get("startedAt") for item in data if item.get("startedAt") is not None | ||
| ] | ||
| if started: | ||
| ev.newest_execution_started_at = parse_ts(started[0]).isoformat() | ||
| elif data: | ||
| raise DataError(f"{label}: no execution carries a startedAt timestamp") |
Comment on lines
+135
to
+136
| status = item.get("status") | ||
| if status in SKIPPED_STATUSES: |
| ev.failure_streak = streak | ||
| if newest_failure is not None: | ||
| failed_at = parse_ts(newest_failure.get("startedAt")) | ||
| ev.newest_failure_id = str(newest_failure.get("id", "?")) |
Open
This was referenced Oct 2, 2026
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
During the n8n Cloud execution-quota outage (every execution erroring since ~2026-09-23), Intake Canary stayed green. On
main(after #316), a failure streak whose newest error is older than 60 min is downgraded to a warning, then the script writesstatus=healthyand exits 0.What
d1ef0bb–9425971(already on this branch): move the logic into.github/scripts/intake_canary.py(stdlib only), with per-workflow freshness, UNKNOWN on bad key/transport/shape/unknown status, structured evidence, and UNKNOWN outputs even if the canary crashes.05d6fbf(new):WARN_STALE(streak still open, no success since) now exits 1. Slack is paged only on the existing ~4/day reminder gate, so a long outage fails the job without spamming. The two Python files are also formatted with the pinned black (they were failing at the branch tip).Evidence
pytest tests/unit/test_intake_canary.py: 63 passed (61 existing + 2 new: stale pages only on the reminder gate, and exits 1 either way).test_stale_incident_fails_job_never_healthymirrors the live 09-23 shape (router 20/20 errors, newest 100 min old).Known limitation (not fixed here)
The canary polls the n8n Cloud executions API. After the DRC/Event Router move to the M900, the local n8n API is not reachable from GitHub-hosted runners (the Funnel exposes only
/n8n/webhook/*). This canary will need a new data source after cutover.Do not merge during the cutover window unless Gadget says so.
🤖 Generated with Claude Code
https://claude.ai/code/session_017DsEH4FYupGjqJ693GEGn6