diff --git a/README.md b/README.md index 6f95bef..b299593 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,10 @@ not a product version. Code is available under the [MIT license](LICENSE). [Architecture](#how-it-works) · [Evaluation](#what-has-been-measured) · [Roadmap](docs/roadmap.md) · [Contributing](#contributing) +The reliability benchmark is [frozen](docs/reliability-campaign.md); historical failures +remain recorded. The [taxi demonstration results and chart previews](results/taxi-demonstration-20261114/README.md) +show measured analysis and an unresolved artifact-transfer failure. + ## Capabilities today | Area | Implemented behavior | Current boundary | @@ -63,8 +67,9 @@ have been used for testing; BackIntel has not trained or fine-tuned its models o runtime support has not been qualified. Python-only CI runs on Linux. - Docker for the initial sandbox image build/export and optional database services. Prepared SQLite/file analysis runs in SmolVM without Docker execution. -- For live analysis, an authorized ChatGPT subscription connection or a configured - supported OpenRouter model. No model account is needed for the offline checks below. +- For live analysis, native OpenAI/Anthropic API credentials, an authorized ChatGPT + subscription connection, or an explicitly selected supported OpenRouter model, + depending on the entry point. No model account is needed for the offline checks below. ### 1. Check out and verify the project @@ -183,12 +188,17 @@ separate from this repository's code license; see the [benchmark specification]( ```mermaid flowchart LR - Brief[Business brief] --> Agent[One Deep Agents graph] - Agent --> Query[Read-only store tools] + Question[Question: ask] --> Analyst[Analyst graph] + Brief[Business brief: analyze] --> Specialist[Specialist graph] + Analyst --> Query[Read-only store tools] + Specialist --> Query Query --> Data[Assigned sources] - Agent --> VM[SmolVM shell and Python] + Analyst --> VM[SmolVM Python] + Specialist --> Workspace[SmolVM shell and Python workspace] VM --> Data - Agent --> Check[Structured completion and reproduced tables] + Workspace --> Data + Analyst --> Answer[Structured answer and executable evidence] + Specialist --> Check[Structured completion and reproduced tables] Check --> Render[Clean renderer VM] Render --> Files[Report, charts, CSV and XLSX] ``` @@ -207,6 +217,8 @@ original limits; it is never silently converted into success. The standard specialist profile uses **2 CPUs and 2 GiB RAM**, with separate workspace, artifact and temporary-file quotas. A larger profile exists, but is not evidence of real large-data reliability. Aggregate capacity reservation across concurrent runs is not implemented. +The benchmark analyst's Python commands use **1 CPU and 768 MiB RAM**, with a 120-second +command limit. Each entry point remains independently invoked. ### Prompts, skills and tools @@ -230,8 +242,27 @@ or causal-analysis capabilities. These are versioned development observations, not a production accuracy guarantee. +The first native 92-response reliability comparison is complete. Its candidate passed +44/46 responses versus baseline 41/46, but **reserved acceptance rejected it**: two +mandatory cases regressed. The correct profit lacked executable major-unit output, and +an unavailable-period report used `answered` with null required metrics. A revised +implementation requires fresh qualification; it has no live acceptance result yet. +See the [complete comparison and LangSmith traces](results/reliability-native-20261114.md) +and [release qualification package](docs/v0.0.1-release.md). + +The following revision passed 12/12 original and 17/17 development responses, then +16/17 fresh reserved candidate responses. Its remaining failure was an explicit report +value binding; the fresh baseline arm did not run. The latest code fixes that binding +for SQL and structured Python output and passes 277 offline tests, but has no live +qualification yet. See the [revision checkpoint](results/reliability-revision-20261114.md). + | Experiment | Observed result | Scope | | --- | --- | --- | +| Native original regressions | Baseline 9/12; candidate 12/12 | Six cases, two trials per arm; frozen improvement gate passed | +| Native controlled development | Baseline 15/17; candidate 17/17 | Two improvements, no deterministic regression | +| Native reserved acceptance | Baseline 17/17; candidate 15/17 | Candidate rejected for two mandatory regressions; full failed results retained | +| Revised original/development | Candidate 12/12 and 17/17 | Reused unchanged baselines 9/12 and 15/17; both gates passed | +| Revised fresh reserved generation 2 | Candidate 16/17; baseline not run | One evidence-binding failure; paired qualification incomplete | | Historical Olist storage matrix | 143/144 supported responses | GPT-6 Luna; 24 questions repeated across six stores on the earlier runtime | | Historical UCI storage runs | 80/80 | 20 questions on four stores; repetitions are not independent domains | | Historical standing reports and replay | 15/17 | Three standing reports plus 14 historical replay cases; no live scheduler | diff --git a/benchmarks/demonstrations/nyc-yellow-q1-2025.json b/benchmarks/demonstrations/nyc-yellow-q1-2025.json new file mode 100644 index 0000000..c5da590 --- /dev/null +++ b/benchmarks/demonstrations/nyc-yellow-q1-2025.json @@ -0,0 +1,356 @@ +{ + "retrieved_at": "2026-11-14T04:32:37.429215+00:00", + "source": "NYC TLC yellow taxi Q1 2025", + "assets": [ + { + "file": "taxi_zone_lookup.csv", + "url": "https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv", + "bytes": 12331, + "sha256": "1a99e105092230f8620f301edcca7f80d3080642ff404d28ed957d3fa222c8ed" + }, + { + "file": "yellow_tripdata_2025-01.parquet", + "url": "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-01.parquet", + "bytes": 59158238, + "sha256": "9af277e4c0d3f9deb30644da822981e1e7df6af58313170fd3aa8a474485488a", + "rows": 3475226, + "columns": [ + { + "name": "VendorID", + "type": "int32" + }, + { + "name": "tpep_pickup_datetime", + "type": "timestamp[us]" + }, + { + "name": "tpep_dropoff_datetime", + "type": "timestamp[us]" + }, + { + "name": "passenger_count", + "type": "int64" + }, + { + "name": "trip_distance", + "type": "double" + }, + { + "name": "RatecodeID", + "type": "int64" + }, + { + "name": "store_and_fwd_flag", + "type": "large_string" + }, + { + "name": "PULocationID", + "type": "int32" + }, + { + "name": "DOLocationID", + "type": "int32" + }, + { + "name": "payment_type", + "type": "int64" + }, + { + "name": "fare_amount", + "type": "double" + }, + { + "name": "extra", + "type": "double" + }, + { + "name": "mta_tax", + "type": "double" + }, + { + "name": "tip_amount", + "type": "double" + }, + { + "name": "tolls_amount", + "type": "double" + }, + { + "name": "improvement_surcharge", + "type": "double" + }, + { + "name": "total_amount", + "type": "double" + }, + { + "name": "congestion_surcharge", + "type": "double" + }, + { + "name": "Airport_fee", + "type": "double" + }, + { + "name": "cbd_congestion_fee", + "type": "double" + } + ], + "uncompressed_column_bytes": 92602414 + }, + { + "file": "yellow_tripdata_2025-02.parquet", + "url": "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-02.parquet", + "bytes": 60343086, + "sha256": "037cba555a73663f3a51a2c27816e40e3feb364769942bdf122b9da31e377bd3", + "rows": 3577543, + "columns": [ + { + "name": "VendorID", + "type": "int32" + }, + { + "name": "tpep_pickup_datetime", + "type": "timestamp[us]" + }, + { + "name": "tpep_dropoff_datetime", + "type": "timestamp[us]" + }, + { + "name": "passenger_count", + "type": "int64" + }, + { + "name": "trip_distance", + "type": "double" + }, + { + "name": "RatecodeID", + "type": "int64" + }, + { + "name": "store_and_fwd_flag", + "type": "large_string" + }, + { + "name": "PULocationID", + "type": "int32" + }, + { + "name": "DOLocationID", + "type": "int32" + }, + { + "name": "payment_type", + "type": "int64" + }, + { + "name": "fare_amount", + "type": "double" + }, + { + "name": "extra", + "type": "double" + }, + { + "name": "mta_tax", + "type": "double" + }, + { + "name": "tip_amount", + "type": "double" + }, + { + "name": "tolls_amount", + "type": "double" + }, + { + "name": "improvement_surcharge", + "type": "double" + }, + { + "name": "total_amount", + "type": "double" + }, + { + "name": "congestion_surcharge", + "type": "double" + }, + { + "name": "Airport_fee", + "type": "double" + }, + { + "name": "cbd_congestion_fee", + "type": "double" + } + ], + "uncompressed_column_bytes": 94223289 + }, + { + "file": "yellow_tripdata_2025-03.parquet", + "url": "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2025-03.parquet", + "bytes": 69964745, + "sha256": "20c4b77cce457b7cfdae77c5069ca7dc91c167f876a41467f4502ab69487a186", + "rows": 4145257, + "columns": [ + { + "name": "VendorID", + "type": "int32" + }, + { + "name": "tpep_pickup_datetime", + "type": "timestamp[us]" + }, + { + "name": "tpep_dropoff_datetime", + "type": "timestamp[us]" + }, + { + "name": "passenger_count", + "type": "int64" + }, + { + "name": "trip_distance", + "type": "double" + }, + { + "name": "RatecodeID", + "type": "int64" + }, + { + "name": "store_and_fwd_flag", + "type": "large_string" + }, + { + "name": "PULocationID", + "type": "int32" + }, + { + "name": "DOLocationID", + "type": "int32" + }, + { + "name": "payment_type", + "type": "int64" + }, + { + "name": "fare_amount", + "type": "double" + }, + { + "name": "extra", + "type": "double" + }, + { + "name": "mta_tax", + "type": "double" + }, + { + "name": "tip_amount", + "type": "double" + }, + { + "name": "tolls_amount", + "type": "double" + }, + { + "name": "improvement_surcharge", + "type": "double" + }, + { + "name": "total_amount", + "type": "double" + }, + { + "name": "congestion_surcharge", + "type": "double" + }, + { + "name": "Airport_fee", + "type": "double" + }, + { + "name": "cbd_congestion_fee", + "type": "double" + } + ], + "uncompressed_column_bytes": 109555104 + } + ], + "trip_rows": 11198026, + "download_bytes": 189478400, + "zone_lookup": { + "rows": 265, + "distinct_ids": 265, + "null_ids": 0 + }, + "terms": [ + "https://www.nyc.gov/opendata/get-started/FAQs", + "https://www.nyc.gov/main/terms-of-use" + ], + "dictionary_url": "https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf", + "dictionary_binary_download": "HTTP 403; official text inspected via web parser, binary not present", + "model_calls": 0, + "purpose": "One real-data business-analysis demonstration; not a new benchmark campaign.", + "novelty": "Yellow taxi Q1 2025 is new to this demonstration. TLC green taxi January 2023 was previously prepared; source-family and model-pretraining novelty are not claimed.", + "rights": "NYC Open Data use statement under NYC.gov terms; preserve source, snapshot, attribution and modifications. Not an assertion of CC0 or MIT coverage for upstream data.", + "profile": { + "checks": [ + { + "file": "yellow_tripdata_2025-01.parquet", + "query": "SELECT count(*) AS rows,\n min(tpep_pickup_datetime)::VARCHAR AS min_pickup, max(tpep_pickup_datetime)::VARCHAR AS max_pickup,\n count(*) FILTER(WHERE tpep_pickup_datetime IS NULL) AS null_pickup,\n count(*) FILTER(WHERE strftime(tpep_pickup_datetime,'%Y-%m') <> '2025-01') AS out_of_file_month,\n count(*) FILTER(WHERE tpep_dropoff_datetime <= tpep_pickup_datetime) AS nonpositive_duration,\n count(*) FILTER(WHERE fare_amount < 0) AS negative_fare,\n count(*) FILTER(WHERE trip_distance <= 0) AS nonpositive_distance,\n count(*) FILTER(WHERE passenger_count IS NULL) AS missing_passenger_count,\n count(*) FILTER(WHERE PULocationID IS NULL OR PULocationID NOT IN (SELECT LocationID FROM zones)) AS unmatched_pickup_zone,\n count(*) FILTER(WHERE DOLocationID IS NULL OR DOLocationID NOT IN (SELECT LocationID FROM zones)) AS unmatched_dropoff_zone\n FROM read_parquet('/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/dataset-discovery-20261114/nyc-yellow-q1-2025/yellow_tripdata_2025-01.parquet')", + "result": { + "rows": 3475226, + "min_pickup": "2024-12-31 20:47:55", + "max_pickup": "2025-02-01 00:00:44", + "null_pickup": 0, + "out_of_file_month": 22, + "nonpositive_duration": 2051, + "negative_fare": 144118, + "nonpositive_distance": 90893, + "missing_passenger_count": 540149, + "unmatched_pickup_zone": 0, + "unmatched_dropoff_zone": 0 + } + }, + { + "file": "yellow_tripdata_2025-02.parquet", + "query": "SELECT count(*) AS rows,\n min(tpep_pickup_datetime)::VARCHAR AS min_pickup, max(tpep_pickup_datetime)::VARCHAR AS max_pickup,\n count(*) FILTER(WHERE tpep_pickup_datetime IS NULL) AS null_pickup,\n count(*) FILTER(WHERE strftime(tpep_pickup_datetime,'%Y-%m') <> '2025-02') AS out_of_file_month,\n count(*) FILTER(WHERE tpep_dropoff_datetime <= tpep_pickup_datetime) AS nonpositive_duration,\n count(*) FILTER(WHERE fare_amount < 0) AS negative_fare,\n count(*) FILTER(WHERE trip_distance <= 0) AS nonpositive_distance,\n count(*) FILTER(WHERE passenger_count IS NULL) AS missing_passenger_count,\n count(*) FILTER(WHERE PULocationID IS NULL OR PULocationID NOT IN (SELECT LocationID FROM zones)) AS unmatched_pickup_zone,\n count(*) FILTER(WHERE DOLocationID IS NULL OR DOLocationID NOT IN (SELECT LocationID FROM zones)) AS unmatched_dropoff_zone\n FROM read_parquet('/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/dataset-discovery-20261114/nyc-yellow-q1-2025/yellow_tripdata_2025-02.parquet')", + "result": { + "rows": 3577543, + "min_pickup": "2025-01-31 22:22:53", + "max_pickup": "2025-03-01 00:06:32", + "null_pickup": 0, + "out_of_file_month": 31, + "nonpositive_duration": 5164, + "negative_fare": 182655, + "nonpositive_distance": 99771, + "missing_passenger_count": 806937, + "unmatched_pickup_zone": 0, + "unmatched_dropoff_zone": 0 + } + }, + { + "file": "yellow_tripdata_2025-03.parquet", + "query": "SELECT count(*) AS rows,\n min(tpep_pickup_datetime)::VARCHAR AS min_pickup, max(tpep_pickup_datetime)::VARCHAR AS max_pickup,\n count(*) FILTER(WHERE tpep_pickup_datetime IS NULL) AS null_pickup,\n count(*) FILTER(WHERE strftime(tpep_pickup_datetime,'%Y-%m') <> '2025-03') AS out_of_file_month,\n count(*) FILTER(WHERE tpep_dropoff_datetime <= tpep_pickup_datetime) AS nonpositive_duration,\n count(*) FILTER(WHERE fare_amount < 0) AS negative_fare,\n count(*) FILTER(WHERE trip_distance <= 0) AS nonpositive_distance,\n count(*) FILTER(WHERE passenger_count IS NULL) AS missing_passenger_count,\n count(*) FILTER(WHERE PULocationID IS NULL OR PULocationID NOT IN (SELECT LocationID FROM zones)) AS unmatched_pickup_zone,\n count(*) FILTER(WHERE DOLocationID IS NULL OR DOLocationID NOT IN (SELECT LocationID FROM zones)) AS unmatched_dropoff_zone\n FROM read_parquet('/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/dataset-discovery-20261114/nyc-yellow-q1-2025/yellow_tripdata_2025-03.parquet')", + "result": { + "rows": 4145257, + "min_pickup": "2007-12-05 18:45:00", + "max_pickup": "2025-04-01 00:00:17", + "null_pickup": 0, + "out_of_file_month": 33, + "nonpositive_duration": 22280, + "negative_fare": 208724, + "nonpositive_distance": 103722, + "missing_passenger_count": 916663, + "unmatched_pickup_zone": 0, + "unmatched_dropoff_zone": 0 + } + } + ], + "scope": "Source suitability only; not the business analysis, no cleaning performed." + }, + "input_boundary": "Only the four raw data assets and ordinary schema/source documentation may enter the agent sandbox. Keep this selection manifest, suitability queries/results and any later independent verification outside agent inputs." +} diff --git a/benchmarks/improvement/reliability-campaign.json b/benchmarks/improvement/reliability-campaign.json new file mode 100644 index 0000000..9fadb4d --- /dev/null +++ b/benchmarks/improvement/reliability-campaign.json @@ -0,0 +1,351 @@ +{ + "schema": 1, + "name": "analytical-reliability-92", + "preparation_status": "historical_openrouter_snapshots_execution_ineligible", + "baseline_runtime_revision": "cb963fdefc6e5d2daf9779f6726a2c199ed195ee", + "candidate_runtime": "current reviewed runtime at final freeze", + "analyst_model": "openrouter/openai/gpt-6-luna", + "judge_model": "openrouter/anthropic/claude-haiku-5.5", + "sandbox": "smolvm", + "comparison": "historical analyst logic versus current analyst, both under the same current evaluator, locked dependencies and SmolVM runtime", + "response_count": 92, + "batches": [ + { + "name": "originals", + "dataset": "backintel_original_regression_v1", + "split": "development", + "case_ids": [ + "olist-l3-07", + "olist-l6-repeat-2017-09", + "olist-l6-2018-03", + "olist-l5-01", + "olist-l5-02", + "olist-l1-01" + ], + "trials_per_arm": 2, + "response_count": 24, + "require_improvement": true, + "mandatory_cases": "all", + "data_hash": "a4ed0a7916f7d6c64b806cf1bb2a6f29ea3adf648856955f1f74e7f9f415e318", + "reference_hash": "ff8ec2d2417b412f" + }, + { + "name": "development", + "dataset": "backintel_acceptance_v1_development", + "split": "development", + "case_ids": [ + "acceptance-v1-development-revenue", + "acceptance-v1-development-profit-complete", + "acceptance-v1-development-profit-missing", + "acceptance-v1-development-profit-partial", + "acceptance-v1-development-profit-zero", + "acceptance-v1-development-revenue-duplicates", + "acceptance-v1-development-delivery-report", + "acceptance-v1-development-repeat-report", + "acceptance-v1-development-delivery-causality", + "acceptance-v1-development-replay-stable", + "acceptance-v1-development-replay-shift", + "acceptance-v1-development-replay-late-observations", + "acceptance-v1-development-replay-late-observations-followup", + "acceptance-v1-development-replay-duplicate-export", + "acceptance-v1-development-replay-unobserved-period", + "acceptance-v1-development-replay-price-mix-only", + "acceptance-v1-development-replay-price-shift" + ], + "trials_per_arm": 1, + "response_count": 34, + "require_improvement": false, + "mandatory_cases": "all", + "data_hash": "0b65012dc6217c5a77b8e8b5f64f84b6ebd53b3a85e7cd73faa17da7bc60c741", + "reference_hash": "ed3e5f6f1b001583" + }, + { + "name": "reserved", + "dataset": "backintel_acceptance_v1_acceptance", + "split": "acceptance", + "case_ids": [ + "acceptance-v1-acceptance-revenue", + "acceptance-v1-acceptance-profit-complete", + "acceptance-v1-acceptance-profit-missing", + "acceptance-v1-acceptance-profit-partial", + "acceptance-v1-acceptance-profit-zero", + "acceptance-v1-acceptance-revenue-duplicates", + "acceptance-v1-acceptance-delivery-report", + "acceptance-v1-acceptance-repeat-report", + "acceptance-v1-acceptance-delivery-causality", + "acceptance-v1-acceptance-replay-stable", + "acceptance-v1-acceptance-replay-shift", + "acceptance-v1-acceptance-replay-late-observations", + "acceptance-v1-acceptance-replay-late-observations-followup", + "acceptance-v1-acceptance-replay-duplicate-export", + "acceptance-v1-acceptance-replay-unobserved-period", + "acceptance-v1-acceptance-replay-price-mix-only", + "acceptance-v1-acceptance-replay-price-shift" + ], + "trials_per_arm": 1, + "response_count": 34, + "require_improvement": false, + "mandatory_cases": "all", + "data_hash": "1eeb1cfeab9b9180fd68d20f3bd343603bfe9b8dd24ca05f6ba2c2d8d2548bc7", + "reference_hash": "2b4faa0168659d1d" + } + ], + "gates": { + "mandatory_correctness": 1, + "mandatory_evidence_when_applicable": 1, + "mandatory_answer_refusal_decision": 1, + "material_regressions_allowed": 0, + "method_score_threshold": null, + "human_calibration": "waived; method scores remain uncalibrated", + "comparison_verdict": "accept_for_review is not release/deployment authority" + }, + "visibility": { + "analyst": [ + "question", + "assigned read-only store" + ], + "excluded": [ + "reference answers", + "reference SQL", + "selection rationale", + "reserved outcome traces" + ] + }, + "funding": "The existing $5 goal allowance is separate from usable provider credit. No purchase or provider/model change is authorized by this manifest.", + "model_routing": "first_party_only", + "candidate_source_version": "2ccaa2f068afb33b", + "judge_version": "94a36eb150da", + "scorer_version": "ba972f36ad0d9f1c", + "sandbox_identity": { + "backend": "smolvm", + "version": "1.31.1", + "binary_sha256": "ae78eda57cb3a4d88960c74077055972fc058441ed49fded26a5b2f146faf086", + "image_id": "sha256:9820b507a21f383ad0af1fac20426a6123ef718df79e35740461ab3ff3087dac", + "archive_sha256": "d12436507cf87f1eb99cc360232e9dc9b2ac0ec8c9addf4167395cdc78ba24a5", + "memory_mib": 768, + "cpus": 1, + "command_timeout_seconds": 120 + }, + "prepared_plans": [ + { + "batch": "originals", + "responses": 24, + "manifest_sha256": "2d850e7489e8776fd57ebfa081578d17a9965af15b8c923a7a56516ae3ac9f02", + "manifest_hash": "4dc608da464c4543a109812543600ed218b59a64ef7e73ed5f7daf229df50b89", + "policy_hash": "ac2e662d91f20821c4681f7e1087e35d4e13a451fc5bcdf8a790111340a1d1d5", + "baseline_source_version": "3f436698d412c8c1", + "candidate_source_version": "2ccaa2f068afb33b", + "scorer_version": "ba972f36ad0d9f1c", + "judge_version": "94a36eb150da", + "runner_sha256": "18205101cdda364a355ce1368686086256956aaaf7fcd2d22e1901b219d736c6", + "model_routing": "first_party_only" + }, + { + "batch": "development", + "responses": 34, + "manifest_sha256": "076cdc0874ea88d333e3ef8d7afeb6ba2f59190e22f3eca9710ec463a7c95e45", + "manifest_hash": "4060e6d3e886a7774d563562625763871b3db73b9f273ff08069ae9a68f2115f", + "policy_hash": "e16d275885caea4df08e7268a5f56b95a652020092002b750a984ff80bd63fb4", + "baseline_source_version": "3f436698d412c8c1", + "candidate_source_version": "2ccaa2f068afb33b", + "scorer_version": "ba972f36ad0d9f1c", + "judge_version": "94a36eb150da", + "runner_sha256": "18205101cdda364a355ce1368686086256956aaaf7fcd2d22e1901b219d736c6", + "model_routing": "first_party_only" + }, + { + "batch": "reserved", + "responses": 34, + "manifest_sha256": "bffb352e4cb4e8e6fe2a6268a185ce8b537dee2b8232c077b5d63f12a3458314", + "manifest_hash": "6be63832b4964f7689709c3e089af7c0c82daa5a85b22948721f29cab8595b5e", + "policy_hash": "521adb1979fdfdfc41e6735df0264d78d518c049640923e5ee6903ca6d95b62b", + "baseline_source_version": "3f436698d412c8c1", + "candidate_source_version": "2ccaa2f068afb33b", + "scorer_version": "ba972f36ad0d9f1c", + "judge_version": "94a36eb150da", + "runner_sha256": "18205101cdda364a355ce1368686086256956aaaf7fcd2d22e1901b219d736c6", + "model_routing": "first_party_only" + } + ], + "record_scope": "Historical OpenRouter preparation; retained for reproducibility only. None of these snapshots may execute. See native_followup for current readiness.", + "historical_execution_blockers": [ + "Latest evaluator revision has offline checks only; funded live judge validation is still required.", + "Provider funding unavailable for the full campaign: latest parent-reported credit $0.007308928; last judge call returned HTTP402 and has an unresolved individual charge.", + "Integrator must reconcile aggregate remaining $5 allowance and freeze final batch caps before any calls.", + "Recheck exact prepared image/runtime and source bytes before execution." + ], + "native_followup": { + "date": "2026-11-14", + "analyst_model": "openai/gpt-6-luna", + "judge_model": "anthropic/claude-haiku-5-5", + "accounting_basis": "native_estimate", + "accounting_version": "a0cc73ec381e4778", + "latest_judge_version": "b44929467d85", + "controls_matched": 10, + "controls_total": 10, + "native_campaign_prepared": true, + "campaign_responses_executed": 92, + "estimated_native_checks_usd": 0.24891425, + "actual_native_invoice_usd": null, + "remaining_gate": "Latest report-binding revision 9463f7cec50ccc9e needs a 63-response qualification with fresh generation 3. Shared $2.05 allocation requires requested $7 cumulative cap; pending approval.", + "campaign_execution": "All 92 responses completed; original/development accepted, reserved rejected for two mandatory regressions. Revision prepared separately; no revised inference yet.", + "candidate_commit": "b0569942267e9530f0ff77fbd80263111085dbe5", + "candidate_source_version": "1812b58dfccbd6f9", + "baseline_source_version": "368cd43ab7c6df81", + "scorer_version": "c100f51adc12873d", + "max_completion_tokens": 8192, + "judge_effort": "high", + "campaign_allocation_usd": 3, + "per_call_reservation_usd": 0.05, + "native_plans": [ + { + "batch": "originals", + "output": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-20261114/originals", + "manifest_sha256": "d2c5ec083e64993f066d772ac41db9b61f2dbcdf85e260b70aca945ec97023ef", + "manifest_hash": "a565e15131cdb235a9093b399eaac68a6c2c1218fb57fa35ec631c4c5e105f7d", + "policy_hash": "6fe8128a0e410aac0b46e3bac769e67ea285cb3c2470b0776d8311dfe247a0b7", + "responses": 24, + "limit_usd": 1, + "max_calls": 600, + "reservation_usd": 0.05 + }, + { + "batch": "development", + "output": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-20261114/development", + "manifest_sha256": "794464f42d6a61c5e8e46d96f3cc878b471abb3fc6c3c0aa54ea80103653df82", + "manifest_hash": "70894366208dc19ca609a04ae9e88da8750e98deaa3e86d70eea03c7f35ffc69", + "policy_hash": "790f4b573170e3c613a55ef1e7db79536da13f510eb38cfa210aa888d1c1271d", + "responses": 34, + "limit_usd": 1, + "max_calls": 850, + "reservation_usd": 0.05 + }, + { + "batch": "reserved", + "output": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-20261114/reserved", + "manifest_sha256": "848b4a3dd7349e6aca86c31f8580bfc6f10c05fc4779ade457b38876f1a83b21", + "manifest_hash": "2b641b3d25261802a3f05c15dd2a2e8f2d84118c7a44ac31d0291c3cc56edcc2", + "policy_hash": "5be087f7ace47a56323295f7e720455be63879294d01de51360900abd9340a53", + "responses": 34, + "limit_usd": 1, + "max_calls": 850, + "reservation_usd": 0.05 + } + ], + "control_experiment": "https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/37a4b33b-7b2c-4fe8-93f1-1585356b9b33", + "original_comparison": { + "verdict": "accept_for_review", + "baseline_supported": 9, + "candidate_supported": 12, + "responses_per_arm": 12, + "quality_regressions": 0, + "improved_case_trials": 3, + "readback_runs": 24, + "readback_feedback_records": 144, + "baseline_records_hash": "1a4835f5d226fbaf9f18e4c8c9204f4cfc421e373d3e928baa349732b3ba5fda", + "candidate_records_hash": "64db657835df8b14eb9a23722f1196911cabf19e0ae81ac8e04233d32ee52d27", + "report": "results/original-regression-native-20261114.md" + }, + "recovery": { + "preserved_completed_answers": 2, + "replaced_incomplete_attempts": 1, + "previously_unstarted_slots": 9, + "new_estimated_usd": 0.4056835, + "new_model_calls": 86, + "new_admission_cap_usd": 1, + "original_batch_limit_still_encumbered_usd": 1, + "observed_original_partial_attempt_usd": 0.029098, + "retained_unknown_remainder_usd": 0.39023725, + "goal_used_or_encumbered_usd": 1.71356956225, + "goal_remaining_before_other_batches_usd": 3.28643043775 + }, + "measured_campaign": { + "responses": 92, + "verdict": "reject", + "baseline_supported": 41, + "candidate_supported": 44, + "responses_per_arm": 46, + "originals": { + "baseline_supported": 9, + "candidate_supported": 12, + "responses_per_arm": 12, + "verdict": "accept_for_review" + }, + "development": { + "baseline_supported": 15, + "candidate_supported": 17, + "responses_per_arm": 17, + "verdict": "accept_for_review", + "estimated_cost_usd": 0.927989, + "model_calls": 245 + }, + "reserved": { + "baseline_supported": 17, + "candidate_supported": 15, + "responses_per_arm": 17, + "verdict": "reject", + "estimated_cost_usd": 0.87160325, + "model_calls": 221 + }, + "goal_used_or_encumbered_usd": 3.51316181225, + "remaining_existing_goal_allowance_usd": 1.48683818775, + "report": "results/reliability-native-20261114.md", + "machine_report": "results/reliability-native-20261114.json", + "fix_pr": "https://github.com/IntelIP/BackIntel/pull/13", + "revised_candidate_source": "d0d476e04fc7f378", + "revised_candidate_commit": "661646fe8842e67e6734c2eb887dc056843cbffb", + "revised_live_responses": 46, + "proposed_new_responses": 63, + "proposed_retained_baseline_responses": 29, + "proposed_new_allocation_usd": 2.15, + "requested_cumulative_allowance_usd": 6, + "approval_status": "prior_$6_request_superseded_by_generation3_request" + }, + "revision_checkpoint": { + "measured_candidate_source": "d0d476e04fc7f378", + "new_responses": 46, + "new_supported": 45, + "new_calls": 353, + "retained_baseline_responses": 29, + "fresh_baseline_responses": 0, + "original_supported": 12, + "development_supported": 17, + "fresh_generation2_supported": 16, + "fresh_generation2_cases": 17, + "status": "candidate_rejected_paired_reserved_incomplete", + "new_inference_estimated_cost_usd": 1.413655, + "goal_used_or_encumbered_usd": 4.92681681225, + "existing_goal_limit_usd": 5, + "remaining_existing_allowance_usd": 0.07318318775, + "report": "results/reliability-revision-20261114.md", + "machine_report": "results/reliability-revision-20261114.json" + }, + "next_revision": { + "candidate_source": "9463f7cec50ccc9e", + "candidate_commit": "e501bd30be96c0a7291e2c9eec63a32e6d8257a3", + "fix_pr": "https://github.com/IntelIP/BackIntel/pull/13", + "offline_tests_passed": 277, + "optional_tests_skipped": 15, + "hosted_ci_run": 43553743854, + "new_responses_required": 63, + "retained_baseline_responses": 29, + "reserved_generation": 3, + "proposed_shared_allocation_usd": 2.05, + "requested_cumulative_allowance_usd": 7, + "maximum_total_with_proposal_usd": 6.97681681225, + "approval_status": "pending", + "live_model_calls": 0, + "observed_failed_completion_replays_rejected": 3, + "active_local_freeze": "data/output/reliability-native-refresh-20261114-r3-final/" + } + }, + "owner_freeze": { + "status": "frozen_by_owner", + "recorded_at": "2026-11-14T04:31:33.812978+00:00", + "owner_instruction": "Okay let's freeze the benchmark and then we need to find a dataset", + "effect": "Preserve cases, criteria, references, historical results and failures. Stop further variant generation and the proposed 63-response qualification. Use the existing offline regression suite for subsequent code changes.", + "release_acceptance": "Not established; freeze does not turn failures into passes.", + "additional_spending_request": "The $7 total request for this campaign is withdrawn; no added spending is approved.", + "next_step": "Select one real dataset for a bounded business-analysis artifact demonstration." + } +} diff --git a/docs/dataset-demonstration.md b/docs/dataset-demonstration.md new file mode 100644 index 0000000..cb6e2d9 --- /dev/null +++ b/docs/dataset-demonstration.md @@ -0,0 +1,103 @@ +# One real-data business-analysis demonstration + +Execution checkpoint: the [2026-11-14 demonstration](../results/taxi-demonstration-20261114/README.md) +ran one first pass and one repair. Three numerical claims matched independent SQL, and +five chart previews are retained. The final HTML/XLSX transfer failed with unexpected +end of data; INT-498 blocks usable delivery. No third model attempt is planned. The +selection and pre-execution contract below remain the recorded context. + +## Owner decision — 2026-11-14 UTC + +Freeze the reliability benchmark and select one useful real dataset. The prior +63-response rerun is shelved and its $7 request withdrawn. Preserve old results and +failures. This demonstration takes priority over adding similar controlled cases; +it does not establish that the reliability candidate passed release acceptance. + +## Selected data + +**NYC TLC yellow taxi trip records, January–March 2025, plus taxi-zone lookup.** +Downloaded from the [official TLC distribution](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page). +It contains **11,198,026 reported trip records**, 189,478,400 bytes +(189.48 MB decimal), and 265 lookup rows. Files remain unchanged; exact URLs, SHA-256, +schema and source-suitability SQL are in the [selection manifest](../benchmarks/demonstrations/nyc-yellow-q1-2025.json). +This is a practical multi-million-row demonstration, not the separate >=10 GiB scale qualification. + +| File | Rows | Downloaded bytes | +| --- | ---: | ---: | +| taxi_zone_lookup.csv | 265 lookup rows | 12,331 | +| yellow_tripdata_2025-01.parquet | 3,475,226 | 59,158,238 | +| yellow_tripdata_2025-02.parquet | 3,577,543 | 60,343,086 | +| yellow_tripdata_2025-03.parquet | 4,145,257 | 69,964,745 | + +Grain is a reported trip row; the files do not provide a unique trip identifier. +Distance is in miles, monetary amounts in USD, and pickup/dropoff times are recorded +without timezone offsets. Treat timestamps as NYC local civil time for the demonstration, +state that assumption, and handle the March daylight-saving boundary when interpreting durations. + +The [official dictionary](https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf) +provides field/payment meanings. The dictionary text was inspected through the web parser; +its direct binary download returned HTTP 403 and the PDF is not in the local package. +The [NYC Open Data FAQ](https://www.nyc.gov/opendata/get-started/FAQs) describes permitted +use; retain [NYC terms](https://www.nyc.gov/main/terms-of-use), attribution, snapshot and +transformation notes. TLC warns that source accuracy is not guaranteed. Do not label +these upstream files as covered by the repository's MIT license. + +## Why this data + +Transportation offers a different business problem from Olist retail: temporal peaks, +zone joins, spatial patterns, airport trips, fare components and operational tradeoffs. +TLC green taxi January 2023 was previously prepared in INT-440; this selected yellow-taxi +period is a new slice, not a claim that the entire source family or training corpus is unseen. +UCI Online Retail II was considered but repeats the retail domain. Reuse the existing +source family and official distribution rather than a Kaggle mirror with additional access steps. + +Source checks found 86 rows outside their file's named month, 29,495 nonpositive durations, +535,497 negative fares, and 2,263,749 missing passenger counts across the quarter. +All pickup/dropoff IDs match the unique lookup, including its unknown-location categories. +These are suitability findings, not a cleaning policy: negative amounts may represent +adjustments, and plausible duplicates lack a guaranteed identity key. Preserve raw records +and require explicit, reconciled eligibility choices in the eventual analysis. + +## Single business brief + +> You advise a hypothetical NYC taxi operations manager using historical Q1 2025 records. +> Describe where and when recorded activity and gross fare receipts concentrate, how +> patterns vary across the quarter, and which operational hypotheses deserve a limited +> follow-up trial. Discover the schemas and data-quality issues yourself. State eligible +> populations, exclusions, units and uncertainty. Deliver a concise executive report, +> polished charts and a supporting workbook with reproducible calculations. + +Use the existing GPT-6.1 Sol specialist and SmolVM configuration. Mount only raw files +and ordinary data documentation, read-only, with network disabled. Keep source-selection +notes, profiling results, evaluator checks and expected findings outside the agent inputs. +No completed analysis, solution notebook or answer-specific prompt is supplied. + +Expected deliverables: an HTML report with 3–5 prioritized findings, approximately five +clear charts (daily trend, hour/day heatmap, zone comparison, trip/fare distribution, +and an appropriate geographic or borough view), plus an XLSX workbook of supporting +aggregates, definitions and exclusions. A chart form may change when the data supports +a better explanation. Exported figures and workbook tables must agree with the report; +inspect the rendered artifacts for readable typography, units, accessible colors and clipping. + +Recorded trips do not measure unmet demand or available fleet capacity. Passenger charges +are not operator profit; costs and idle time are absent. Cash tips are unrecorded, so tip +comparisons must respect payment coverage. No causal effect of congestion pricing is +established by a quarter of observational data. Recommendations are testable hypotheses. + +## Bounded stopping condition + +Run one first-pass artifact task, preserve its full output and LangSmith trace, and make +at most one bounded repair if a material execution or output defect warrants it. Independently +recompute three consequential displayed claims and reconcile exported aggregates. Inspect +the report/charts/workbook directly. Then deliver the artifacts and a useful/limited/failed +assessment with concrete remaining defects in Linear. Do not generate additional worlds, +repeat for a favorable score, or turn this into another qualification campaign. + +This task has selected and downloaded the data with **zero model calls**. The artifact run, +its call/time allowance and any paid judge usage must be recorded separately before execution; +the withdrawn reliability allocation is not funding for a different experiment. Existing +subscription/model credentials and sandbox are retained; no provider change is proposed. + +Local files: `data/output/dataset-discovery-20261114/nyc-yellow-q1-2025/` in the canonical +checkout. `acquisition.json` and `suitability-checks.json` preserve local evidence separately +from agent-visible raw assets. The wider three-domain and large-data roadmaps remain later work. diff --git a/docs/reliability-campaign.md b/docs/reliability-campaign.md new file mode 100644 index 0000000..18d4cd3 --- /dev/null +++ b/docs/reliability-campaign.md @@ -0,0 +1,264 @@ +# The 92-response analytical reliability campaign + +This is the execution contract for INT-432 and the INT-429 acceptance parent. It uses +the existing comparison runner; it does not introduce a new agent or evaluation framework. +The [machine-readable selection](../benchmarks/improvement/reliability-campaign.json) +contains case identities and gates without expected answers. Linear owns execution status. + +## Owner freeze and next milestone — 2026-11-14 UTC + +The owner stopped further expansion and live requalification of this campaign. Preserve +all cases, references, criteria, traces and failures at their recorded identities. The +prepared generation-3 63-response run is shelved and must not execute. The $7 funding +request for it is withdrawn; no additional spending is authorized. Existing offline +regressions remain useful for code changes. This freeze does not establish release +acceptance or change any historical result. + +The next milestone is one unfamiliar real-data business-analysis demonstration with a +report, charts and supporting workbook. See [dataset selection](dataset-demonstration.md). +The campaign plans and funding requests below are historical records. + +## Historical native follow-up — 2026-11-14 UTC + +The first revised candidate (`d0d476e04fc7f378`) completed 46 new responses within the +existing $5 allowance: 12/12 original, 17/17 development, and 16/17 generation-2 reserved +candidate responses. The 17 fresh baseline responses did not run. The stable-period +answer had correct facts but put its explicit SQL status label in the unit column, +so the independent evidence check rejected it. [Revision checkpoint and all LangSmith +links](../results/reliability-revision-20261114.md) preserve that failure separately. + +PR13 now fixes explicit SQL/structured-Python metric/value associations. Its 277 offline +tests and hosted CI pass; it has no live result. Frozen source `9463f7cec50ccc9e` at +`e501bd30be96c0a7291e2c9eec63a32e6d8257a3` has a zero-call generation-3 proposal at +`data/output/reliability-native-refresh-20261114-r3-final/`. It requires 63 new responses and +reuses 29 unchanged baselines. A shared $2.05 cap would keep cumulative usage/reserves +below $6.97681681225; the requested increase from $5 to $7 is pending and supersedes +the earlier $6 request. Current usage/reserves is $4.92681681225. No further inference +or release is authorized by this preparation. +The earlier pre-review generation-3 zero-call freeze remains retained; final preparation +preserves SQLite Boolean compatibility and uses the same unconsumed generation-3 worlds. + +The following checkpoints describe the earlier freezes and their historical allocations. + +All **92 original planned responses completed**. The original-regression comparison +passed (candidate 12/12 versus baseline 9/12), and development passed (17/17 versus +15/17). Reserved acceptance rejected the candidate (15/17 versus baseline 17/17): +the profit evidence omitted executable unit conversion, and an unavailable-period +report returned answered with null required metrics. [Complete results and LangSmith +links](../results/reliability-native-20261114.md) retain both failures and all comparisons. +The aggregate 44/46 versus 41/46 supported result does not override mandatory regressions. + +Goal usage plus retained reservations is $3.51316181225 of the original $5 allowance, +leaving $1.48683818775. PR13 implements both runtime fixes and prepares distinct reserved +worlds; its 263 passing tests are implementation evidence, not live acceptance. The +revised source is `d0d476e04fc7f378` at `661646fe8842e67e6734c2eb887dc056843cbffb`. +A separate frozen proposal originally needed 63 new responses and reused 29 unchanged +baseline records under their actual identities. Its initial $2.15/$6 funding proposal +was superseded by the staged execution and next revision described above. The +original plans and the earlier zero-call revision proposal remain retained unchanged. + +The following commands and allocation records describe the completed first native +freeze. Do not execute its already-started directories again. The current revised +proposal is separate at `data/output/reliability-native-refresh-20261114-r2/` and does +not authorize new spending. + +The owner-selected routes for the next campaign are `openai/gpt-6-luna` and +`anthropic/claude-haiku-5-5`, using native account credentials and the versioned +conservative token accounting in PR10 (`a0cc73ec381e4778`). Both native adapters have +live verification. These estimates are not invoices or account credit balances. + +The remaining partial-disclosure judge defect is fixed. Judge `b44929467d85` matched +all **10/10 frozen controls**, including the six originals and four prospectively authored +sensor contrasts. It uses native JSON output, a clarified partial-disclosure rubric and +high reasoning effort. The benchmark's existing 8192-token output allowance leaves room +for reasoning and a complete assessment. Earlier failed attempts, including one truncated +2048-token assessment, remain unchanged. [Passing experiment](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/37a4b33b-7b2c-4fe8-93f1-1585356b9b33). +The [judge evidence](https://github.com/IntelIP/BackIntel/blob/codex/analyst-judge-context/results/judge-context-validation.md) +is published in PR8 at `b0569942267e9530f0ff77fbd80263111085dbe5`, stacked on PR10. +252 local tests passed with 15 optional skips; [hosted CI passed](https://github.com/IntelIP/BackIntel/actions/runs/43531747031). +This is a targeted evaluator gate, not human calibration or an analyst release result. +The comparison policy still reports method scores without a release threshold. + +Native adapter/control checks total $0.24891425 in conservative usage estimates across 43 calls. +Historical reported goal usage is $0.04697181225, with $0.012 retained for previously +unpriced calls. These are separate accounting bases. All future allocations must remain +within the original $5 authorization; no credit purchase is permitted. These amounts +leave $4.69211393775 before campaign allocation. The qualified SmolVM benchmark profile +passed an offline check with public networking blocked. + +All three native plans were frozen before any campaign inference at +`data/output/reliability-native-20261114/` in the canonical checkout. The original-regression +batch is complete after explicit interruption recovery: candidate 12/12 fully supported, +baseline 9/12, three improvements and zero frozen quality-gate regressions. The verdict is +`accept_for_review`; [full evidence and recovery accounting](../results/original-regression-native-20261114.md). +The 34-response development and 34-response reserved batches also completed, with +the pass/rejection outcomes recorded above. Baseline +source is `368cd43ab7c6df81`, candidate `1812b58dfccbd6f9`, scorer `c100f51adc12873d`, +judge `b44929467d85`. The machine-readable `native_followup.native_plans` preserves each +manifest/policy hash and selected output path. Historical OpenRouter plans are untouched. + +The frozen native allocation is **$1 per batch, $3 total**, with a **$0.05 per-call +reservation** and call ceilings 600 / 850 / 850. The larger reservation replaces the +historical $0.01 proposal for this native configuration; it accommodates the qualified +conservative rates and 8192-token output cap. Each two-arm batch shares its allowance. +Incomplete coverage, infrastructure errors, uncertain accounting or reservation breaches +stop continuation. A valid quality rejection is retained rather than rerun for a pass. + +Execute from the PR8 checkout with explicit native overrides after Doppler injection: + +```bash +BACKINTEL_MODEL=openai/gpt-6-luna \ +BACKINTEL_JUDGE_MODEL=anthropic/claude-haiku-5-5 \ +BACKINTEL_SANDBOX=smolvm BACKINTEL_MAX_COMPLETION_TOKENS=8192 \ +uv run python scripts/run_improvement_cycle.py --execute \ + --output /absolute/path/to/frozen/batch \ + --max-cost-usd 1 --max-calls 600 --reservation-usd 0.05 +``` + +Use 850 calls for development/reserved. Never invoke an already-started output directory; +the runner deliberately refuses overwriting its execution record. The aggregate admission +record is `campaign-freeze.json`. The original interrupted batch keeps its full $1 +allowance encumbered; the separately bounded recovery used $0.4056835 of a $1 cap. +The recovery’s `recovery-plan.json` and `final-accounting.json` preserve that distinction. +Prior usage plus completed recovery and retained reservations total $1.71356956225 against +the original $5 authorization. No old manifest, policy, output or expected grade changed. Completed experiments determine analyst quality; freezing +92 responses and passing judge controls do not establish that campaign results have passed. + +The OpenRouter settings, source identities and prepared plan hashes below are **historical +preparation only, not executable instructions for the native campaign**. Keep them intact. +The fresh native plans above supersede these for execution. Do not relabel the old plans +or swap their providers in place. + +## What is being compared + +The baseline is analyst logic from `cb963fdefc6e5d2daf9779f6726a2c199ed195ee`. +The runner overlays that revision's `agent.py`, `tools.py`, `contracts.py` and runtime +skills onto the current evaluator and dependencies. The candidate is the reviewed current +analyst. Both run with the same current SmolVM runtime, data, models, grader and comparison +policy. This isolates analyst-logic changes; it does not reproduce the historical Docker +environment or compare Luna against the separate Sol reporting specialist. + +- Analyst: `openrouter/openai/gpt-6-luna`. +- Method judge: `openrouter/anthropic/claude-haiku-5.5`. +- Host/runtime: the qualified macOS ARM64 SmolVM path. +- Data access: explicitly bound local SQLite stores. This campaign does not qualify other + database deployments or perform an artifact-specialist evaluation. +- Human calibration remains waived. Method ratings are reported separately and have no + new acceptance threshold. Independent numerical/evidence/refusal checks retain their + existing mandatory gates and tolerances. + +## Fixed coverage + +| Batch | Cases | Trials per arm | Arms | Responses | Additional gate | +| --- | ---: | ---: | ---: | ---: | --- | +| Original regression | 6 | 2 | 2 | 24 | Demonstrated improvement, no material regression | +| Controlled development | 17 | 1 | 2 | 34 | All mandatory cases pass, no material regression | +| Reserved acceptance | 17 | 1 | 2 | 34 | All mandatory cases pass, no material regression | +| Total | 40 distinct cases | | | 92 | | + +The original selection preserves three observed failures: missing-cost query loops, +broken cited Python evidence, and duplicate/inconsistent report metrics. It also preserves +three healthy controls. See the existing evaluator-owned +[original manifest](../benchmarks/improvement/original-regression.json). + +All 34 controlled fixtures are separate generated worlds; they are not 34 independent +customer domains. Reserved answers and outcomes stay outside analyst and coding-worker +inputs. Freeze the candidate before running reserved cases. If reserved results inform a +revision, they become regression evidence and a fresh reserved variant is required before +claiming unseen acceptance. + +## Offline preparation + +Prepare from the final reviewed evaluator checkout after its live judge validation. A +plan prepared before that revision is final is only a dry run and must not be executed. +The same final candidate source must supply all three plans. The existing runner records +source, scorer, judge, reference, dependency and policy identities and snapshots both arms. +The campaign runner pins OpenRouter requests to the selected model vendor's own provider, +disables fallback and records `model_routing=first_party_only` for each stage. This applies +equally to both analyst arms and the judge, without changing general application defaults. + +Set `BACKINTEL_DATA_DIR` to the isolated prepared data directory. It must contain verified +copies of the three historical databases from the original manifest. The controlled +builder creates its own named fixtures there; do not point it at another active run's data. +Use a fresh output root for every candidate and retain all earlier attempts. + +```bash +export BACKINTEL_MODEL=openrouter/openai/gpt-6-luna +export BACKINTEL_JUDGE_MODEL=openrouter/anthropic/claude-haiku-5.5 +export BACKINTEL_SANDBOX=smolvm +export BACKINTEL_DATA_DIR=/absolute/path/to/isolated/data + +uv run python scripts/run_improvement_cycle.py --prepare-only --candidate-source current \ + --base-ref cb963fd --scenario-manifest benchmarks/improvement/original-regression.json \ + --split development --trials 2 --output data/improvement/reliability-92/originals +uv run python scripts/run_improvement_cycle.py --prepare-only --candidate-source current \ + --base-ref cb963fd --split development --trials 1 --no-require-improvement \ + --output data/improvement/reliability-92/development +uv run python scripts/run_improvement_cycle.py --prepare-only --candidate-source current \ + --base-ref cb963fd --split acceptance --trials 1 --no-require-improvement \ + --output data/improvement/reliability-92/reserved +``` + +The three prepared policies contain all selected cases as mandatory. The original batch +requires improvement; the development and reserved batches allow both arms to pass, but +never allow a criterion to regress. No score or tolerance is changed after seeing outputs. + +## Readiness and spending + +The integrator must reconcile prior goal spending, any retained unknown charges and live +provider credit before admission. The existing $5 total authorization is not an account +balance. No credit purchase follows from preparing this plan. A reasonable bounded allocation +for review is $1 per batch ($3 total), with per-call reservation $0.01 and call ceilings of +600 / 850 / 850 respectively. These are proposed admission limits, not predicted charges or +guarantees that every case will finish. Freeze final allocations in the execution record. + +The current runner uses a shared dollar `RunBudget` within each two-arm batch. The operator +must account for all three batches against the goal allowance; unknown charges or an +exceeded reservation stop subsequent paid calls. Calls inside an answer are counted too: +92 responses are not 92 model calls. Live provider availability and source/VM readiness must be +checked before execution. Preserve the approved vendor/provenance boundary. + +ChatGPT subscription usage has no per-request dollar charge. Replacing this campaign's +analyst with the subscription route would change its model/provider comparison and would +not satisfy the runner's dollar-completeness contract. Do not silently make that substitution. + +Execute one frozen batch at a time only after readiness is recorded. Use the existing +`--execute`, `--max-cost-usd`, `--max-calls` and `--reservation-usd` options, with its exact +prepared output path. A nonzero exit can mean a valid rejected candidate; inspect the saved +comparison rather than treating every nonzero exit as infrastructure failure. Never rerun +until a favorable result appears. + +## Evidence retained + +Keep each `manifest.json`, `policy.json`, source snapshot, stage log, raw record, LangSmith +experiment link, usage ledger and final comparison. Verify complete case/trial coverage, +immutable source fingerprints and accounting before interpreting results. Report original, +development and reserved results separately. Green CI or a passing judge-control experiment +does not complete this campaign or authorize a release. + +The initial offline preparation on 2026-11-13 verified all three historical source hashes, +rebuilt all 34 controlled fixtures, passed 26 independent SQL reference checks (the other +eight cases require refusal rather than a numerical answer), and enumerated exactly 92 +unique batch/arm/trial/case rows with zero model calls. That confirms data/selection readiness; +the corrected evaluator freeze and funded execution are separate remaining steps. + +The subsequent preparation captured candidate source `2ccaa2f068afb33b`, baseline source +`3f436698d412c8c1`, scorer `ba972f36ad0d9f1c` and judge `94a36eb150da` in all three plans. +Offline read-back validated each manifest/policy, both snapshots and all bound data hashes. +The machine-readable campaign selection retains the full plan and runner hashes. + +These snapshots are **preparation only; do not execute**. The categorical caveat evaluator +has offline checks but still needs live validation. The last integrator-reported judge +request failed with HTTP 402 and no individual charge; the account had approximately +$0.00731 remaining. Funding, charge reconciliation and final batch allocation must be +resolved before calls. Any further evaluator change requires fresh plan directories. + +The verified sandbox receipt identifies SmolVM 1.31.1 and the prepared image/archive; the +benchmark Python profile is 768 MiB, one CPU and a 120-second command limit. It is distinct +from the Sol specialist's two-CPU/two-GiB profile. Binary/archive hashes were checked without +starting a VM. Reconfirm the same identities before execution; this preparation is not a +fresh VM acceptance run. + +Local evidence is retained in the preparation worktree at +`data/reliability-preparation-20261113/final-verification.json` and `final-plans/`. +Superseded pre-record-preservation plans remain in `plans/`, marked not to execute. diff --git a/docs/roadmap.md b/docs/roadmap.md index 4b0ed0c..8eaf7d9 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,5 +1,13 @@ # Capability roadmap and next development increments +Owner priority, 2026-11-14 UTC: freeze the reliability benchmark; shelve its generation-3 +rerun and withdraw the $7 request. The immediate next milestone is the [single real-data +artifact demonstration](dataset-demonstration.md) on 11.2 million NYC yellow-taxi records. +The first demonstration has run; five chart previews and three checked claims are +[recorded](../results/taxi-demonstration-20261114/README.md), while INT-498 blocks the +full HTML/XLSX delivery. Existing failures remain open; no release acceptance is claimed. The broader domain/scale +sequence below remains future work rather than a prerequisite for this demonstration. + BackIntel starts at **v0.0.1, experimental**. Version numbers below describe acceptance milestones, not released capability or delivery dates. The internal [Linear project](https://linear.app/intelip/project/backintel-0287c22fe5a7) owns priority and @@ -25,7 +33,11 @@ targeted energy fix. A fresh eight-trial guidance comparison on 2026-11-13 passe versions, including the corrected grader. The candidate eliminated incorrect virtual-skill shell lookups in all four cases, with no measured task regression. Both arms refused missing costs without rendering a substitute report. These are small controlled trials; the broader -92-response reliability comparison remains required. See +92-response reliability comparison completed with two mandatory reserved regressions, +so the measured candidate remains rejected and a revised qualification is required. See +the [revised checkpoint](../results/reliability-revision-20261114.md) for the subsequent +45/46 candidate result and remaining report-binding failure. Fresh generation-3 +qualification of the latest implementation remains pending. See [current capabilities and measured outcomes](../README.md#what-has-been-measured). ## Next achievable increments diff --git a/docs/v0.0.1-release.md b/docs/v0.0.1-release.md new file mode 100644 index 0000000..b3e9010 --- /dev/null +++ b/docs/v0.0.1-release.md @@ -0,0 +1,102 @@ +# BackIntel v0.0.1 release qualification + +Owner decision, 2026-11-14 UTC: reliability testing is frozen and the proposed final +rerun/funding request is shelved. Historical rejections remain; release acceptance is +unestablished. Proceed next with the [real-data demonstration](dataset-demonstration.md). +The qualification plans and requests below are historical, not current execution instructions. + +This package prepares the experimental analytical foundation for a release decision. +It is not a published release or a production-service acceptance record. Linear owns +execution status in INT-429 and INT-451; the dated result reports own measured evidence. + +## Capability and architecture + +The `ask` entry point runs one LangChain Deep Agents analyst with bounded, read-only +schema/query tools and offline SmolVM Python. It returns structured results, method, +caveats and executable evidence. The `analyze` entry point independently runs one +specialist with a bounded shell/Python workspace and exports report/chart/workbook +bundles through a clean renderer VM. Automatic delegation and runtime sub-agents are off. + +The native benchmark routes are `openai/gpt-6-luna` for analysis and +`anthropic/claude-haiku-5-5` for advisory method assessment. The separate artifact +specialist defaults to `chatgpt-plan/gpt-6.1-sol`, medium reasoning. Native API credentials +and the specialist's private ChatGPT account connection are different authentication paths. +See the [README capability matrix](../README.md#capabilities-today), +[runtime boundaries](sandbox-environment.md), [native configuration](native-provider-credits.md) +and [specialist configuration](specialist-default.md). + +This milestone establishes bounded descriptive business analysis. Production ETL, +forecasting/causal validity, unfamiliar-domain transfer, real 10 GiB acceptance, hosted +artifact delivery and continuous scheduling require their own roadmap acceptance. +Historical connector tests do not qualify every database deployment. + +## Evaluation contract and evidence + +The [92-response campaign](reliability-campaign.md) compares frozen analyst implementations +under the same current models, runtime, data and evaluator. It is not a model tournament +or an artifact-specialist comparison. Its numerical references and evidence checks are +deterministic; method ratings are advisory and not human-calibrated. + +| Batch | Responses | Evidence at preparation | +| --- | ---: | --- | +| Original regressions and controls | 24 | Complete; candidate 12/12 fully supported versus baseline 9/12, three improved slots, no deterministic regression | +| Controlled development | 34 | Complete; candidate 17/17 fully supported versus baseline 15/17, two improved cases, no deterministic regression | +| Reserved acceptance | 34 | Complete; candidate 15/17 versus baseline 17/17, rejected for two mandatory regressions | + +The [complete comparison report](../results/reliability-native-20261114.md) retains +all three batches and LangSmith links. The [original comparison report](../results/original-regression-native-20261114.md) +retains interruption recovery details. The current measured candidate is not release-ready. + +The [revised checkpoint](../results/reliability-revision-20261114.md) records 46 new +responses: original 12/12 and development 17/17 passed, while fresh reserved generation 2 +passed 16/17 and exposed an explicit report value-binding failure. Its 17 baseline +responses did not run. These are different reserved worlds from the initial campaign; +their aggregate scores are not an equivalent paired dataset comparison. +Candidate implementation is `b0569942267e9530f0ff77fbd80263111085dbe5`, analytical source +`1812b58dfccbd6f9`; baseline analytical source is `368cd43ab7c6df81`, scorer +`c100f51adc12873d`, judge `b44929467d85`. These identities remain frozen during execution. +All mandatory candidate cases must pass and no material deterministic regression may +remain before analytical acceptance. Completing a valid rejected comparison does not +qualify the candidate for release. + +Completed development used an estimated $0.927989 across 245 calls; reserved used +$0.87160325 across 221 calls. Both completed within their $1 allocations. Estimated +goal usage plus retained reservations is $3.51316181225 against the existing $5 total +authorization before the revised candidate stages. Those stages subsequently used +$1.413655 within that same allowance, bringing total usage/reservations to +$4.92681681225 and leaving $0.07318318775. The next code revision needs a new 63-response +qualification using fresh generation-3 reserved worlds and 29 unchanged retained +baselines. A proposed shared $2.05 cap requires an increase to $7 cumulative; approval +is pending and supersedes the earlier $6 request. Native token estimates are not invoices, and provider credits do not +expand authorization. Interrupted, invalid and unsuccessful attempts remain retained. + +## Reproducible installation and review + +Use Python 3.12 and `uv sync --frozen` from a clean source checkout, then run the README's +offline checks. Standard tests require neither model credentials nor private datasets. +SmolVM execution is qualified on macOS ARM64; its pinned binary/image preparation and +network/resource checks are documented separately. Live inference requires its own +configured credential path and allowance. + +The dependency order is native provider/accounting PR10, evaluator PR8, runtime/report +validation PR13, then the campaign and release evidence in PR9. Review findings, required hosted CI and clean-checkout +verification must be attached to the exact publication head before merge readiness. +The [clean-checkout validation report](../results/release-validation-20261114.md) records +252 passing standard tests, matching development fixture identity, four successful +scripted SmolVM integration cases and separate credential/network probes. +The release decision must identify the final integration commit, its result reports, +supported capability matrix and remaining limitations. No merge, tag or deployment is +implied by preparing this document. + +## Release and rollback handoff + +Publish a tag/release only after the owner approves the concrete reviewed candidate. +Retain the previous main commit `51846a48e52e203eabefd7354182a1bd49ba72b6` as the +pre-adoption source identity. Existing users can continue a pinned checkout; adoption +must use the same pinned dependencies and documented runtime. Do not rewrite historical +experiments or rotate credentials during a source rollback. No hosted service or +customer migration is part of this release. + +The next capability milestone is v0.0.2 unfamiliar-domain qualification. Its datasets, +reserved domain and evaluation allowance remain distinct from this retail/synthetic +reliability campaign; see the [version roadmap](roadmap.md#version-acceptance-map). diff --git a/results/original-regression-native-20261114.md b/results/original-regression-native-20261114.md new file mode 100644 index 0000000..037f2a3 --- /dev/null +++ b/results/original-regression-native-20261114.md @@ -0,0 +1,59 @@ +# Native original-regression comparison — 2026-11-14 UTC + +The recovered original-regression comparison returned **accept for review**. The current analyst produced **12/12 fully supported responses**, versus **9/12** for the historical analyst. All three failed case/trial results improved, all mandatory cases passed, and no frozen correctness/evidence/refusal gate regressed. + +This completes **24 of the planned 92 responses**: six original scenarios, two trials per arm. It does not complete the 34-response development or 34-response reserved-acceptance batches. Those remain frozen and unstarted. No merge, release, human calibration or production-reliability claim follows from this result. + +| Measure | Baseline | Candidate | +| --- | ---: | ---: | +| Fully supported responses | 9/12 | 12/12 | +| Numerically correct / correct refusal | 11/12 | 12/12 | +| Reproducible evidence, applicable answered cases | 8/10 | 10/10 | +| Correct answer/refusal decision | 12/12 | 12/12 | +| Model calls, including judges | 107 | 105 | +| Conservative estimate for completed responses | $0.4965415 | $0.48980675 | +| Summed analyst latency, excluding judges | 644.17 s | 790.69 s | + +The candidate took longer in these measured answers. Recovery also changed job boundaries, so this is not a clean throughput comparison. Method ratings remain advisory and uncalibrated; the frozen policy contains no method-score or latency threshold. + +## Case coverage + +| Scenario | Baseline supported | Candidate supported | Candidate evidence | +| --- | ---: | ---: | --- | +| olist-l1-01 | 2/2 | 2/2 | [Trial 1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/ea92d11b-ac1f-4b47-81e7-28c6f2c460fa), [trial 2](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/b8042aa6-789f-40c7-86e2-8e0afb0e8288) | +| olist-l3-07 | 2/2 | 2/2 | [Trial 1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/5102ef6f-d9d6-4521-8c4c-88ebac5f4b23), [trial 2](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/42c6d206-d5d6-4564-97d4-0c261f7089c0) | +| olist-l5-01 | 1/2 | 2/2 | [Trial 1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/c38db789-3b02-4d85-bc9b-70f2be596078), [trial 2](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d58a4813-72d4-4701-a537-bc102d0c63ab) | +| olist-l5-02 | 0/2 | 2/2 | [Trial 1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/22a8a8d8-b7ae-4092-9513-735e4f76245a), [trial 2](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/08687b79-74dc-40c4-9760-0339be19fbef) | +| olist-l6-2018-03 | 2/2 | 2/2 | [Trial 1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/5102ef6f-d9d6-4521-8c4c-88ebac5f4b23), [trial 2](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/dec62ea4-53a7-444c-bf33-64a1670d017d) | +| olist-l6-repeat-2017-09 | 2/2 | 2/2 | [Trial 1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/f1d7c41a-36fe-4cb7-8af8-8b0b87f5b0c8), [trial 2](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/111ef755-1732-4acd-9641-f2788d349845) | + +The baseline delivery report missed two required facts in one trial and supplied syntactically broken evidence in the other. The baseline repeat-purchasing report had correct facts but broken evidence in its first trial. The candidate supplied correct, reproducible results for all three slots. The missing-cost refusal remained correct in both arms; this run does not show an improvement where the baseline already passed. + +[Baseline trial 1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/6b9e6bbd-9093-4c58-a17d-9ddbcdedb650), [baseline trial 2](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/dea9c5e6-75f4-4a38-b74f-e71e6bb89a94). Read-back verified **24 run outputs and 144 feedback values**, including source/data/evaluator identities and judge usage. + +## Recovery and spending + +The original execution stopped inside `run_python`. No process exit receipt survived, so the exact termination cause is unknown. One completed candidate record was local; LangSmith retained a second complete answer and all six grades. Both were preserved without new model calls. The third attempt had seven observed completed model calls but no final answer. Its trace remains retained and has an operator recovery annotation pointing to one explicit replacement. Nine other slots had never started. No completed unfavorable result was discarded. + +Recovery used one case per job with disk-flushed reservations before dispatch and settlements after responses. Pending or unpriced calls stop subsequent jobs. The sandbox preflight passed with public networking blocked. The recovery script reused the unchanged frozen analyst, grader and comparison functions; it added no prompts, model selection, reference answers or scoring tolerance changes. + +- New recovery usage: **$0.4056835**, 86 model calls, within its separate $1 cap; complete estimated accounting and no reservation breach. +- Observed usage for the full comparison plus the abandoned partial attempt: **$1.01544625**. +- The interrupted batch’s full original $1 allowance remains encumbered. Its unverified remainder of **$0.39023725** is retained separately rather than treated as free. +- Including prior goal usage/reserves, **$1.71356956225** is used or encumbered against the original $5 authorization. That leaves **$3.28643043775**, including the $2 allocated to the unstarted development/reserved batches. + +These are conservative native token estimates and retained reservations, not provider invoices or account credit balances. + +## Frozen identities and retained evidence + +- Analyst: native `openai/gpt-6-luna`; judge: native `anthropic/claude-haiku-5-5`, high effort, 8192-token cap; sandbox: SmolVM. +- Baseline analyst revision: `cb963fdefc6e5d2daf9779f6726a2c199ed195ee`, overlaid on the same current evaluator/runtime as the candidate. +- Candidate implementation commit: `b0569942267e9530f0ff77fbd80263111085dbe5`; source `1812b58dfccbd6f9`; baseline source `368cd43ab7c6df81`. +- Judge `b44929467d85`; scorer `c100f51adc12873d`; accounting `a0cc73ec381e4778`. +- Comparison policy `6fe8128a0e410aac0b46e3bac769e67ea285cb3c2470b0776d8311dfe247a0b7`. +- Baseline records fingerprint `1a4835f5d226fbaf9f18e4c8c9204f4cfc421e373d3e928baa349732b3ba5fda`. +- Candidate records fingerprint `64db657835df8b14eb9a23722f1196911cabf19e0ae81ac8e04233d32ee52d27`. + +Canonical local evidence: `data/output/reliability-native-20261114/originals/recovery/`. It retains the recovery plan/script, original LangSmith trees and feedback, recovered records, per-case receipts, spending journal, `candidate-records.jsonl`, `comparison.json`, `comparison-readback.json`, `reconciliation.json` and `final-accounting.json`. The original batch files and all other frozen plans remain unchanged. + +See the [campaign contract](../docs/reliability-campaign.md) for the remaining development and reserved-acceptance work. diff --git a/results/release-validation-20261114.md b/results/release-validation-20261114.md new file mode 100644 index 0000000..68b1ae5 --- /dev/null +++ b/results/release-validation-20261114.md @@ -0,0 +1,41 @@ +# Clean-checkout validation for v0.0.1 + +Validated 2026-11-14 UTC on macOS ARM64, using a fresh local Git clone checked out at +`b0569942267e9530f0ff77fbd80263111085dbe5`. No private datasets or credentials were copied +into the clone. The host's already-prepared pinned SmolVM binary/image were reused for +the separately identified VM checks. + +| Check | Observed result | +| --- | --- | +| `uv sync --frozen --python 3.12` | Installed the locked project and dependencies successfully | +| `uv run backintel --help` | CLI entry point loaded successfully | +| `uv run pytest -q` | 252 passed, 15 optional skips; no provider calls | +| Ruff lint and format | Passed; 103 source/test/script files already formatted | +| Vulture, minimum confidence 100 | Passed | +| `uv run backintel acceptance build --split development` | Rebuilt 17 cases; data hash `0b65012dc6217c5a77b8e8b5f64f84b6ebd53b3a85e7cd73faa17da7bc60c741`, matching the frozen campaign | +| Four-case scripted specialist integration | 4/4 passed through the real graph, SmolVM, artifact export and independent table/workbook checks; zero provider calls | +| Fresh benchmark VM probe | Python 3.12.16, successful exit, public TCP networking blocked | +| Native credential reuse | Read-only model retrieval returned HTTP 200 for both configured native model IDs; zero inference calls | +| Assembled review package | Runtime files match the frozen candidate; Gitleaks and diff checks passed | + +The scripted specialist cases were sales/refunds, missing cost, irregular energy and +service availability. They exercise privileged reference decisions and establish +integration behavior, not autonomous model quality or visual-design acceptance. Three +cases delivered report bundles; the missing-cost case correctly declined. Report tables, +CSV/workbook contents and completion contracts were checked by the existing evaluator. + +The native credential and VM probes were separate from the credential-free clean-clone +tests. API credentials remained in the approved host secret-manager path and were not +mounted into a VM. No login, key replacement, model fallback or credit purchase occurred. + +Retained local evidence: the clean clone's +`data/output/v0001-clean-integration/comparison.json`, per-case `run.json` files and bundles; +the operator's tool receipts retain the installation/test output and read-only probes. +The clean clone remains available for direct inspection. Hosted CI for the first assembled +package passed at `5e1360101a934ab64beb470b5be82fa42efaf970` in +[run 43544176006](https://github.com/IntelIP/BackIntel/actions/runs/43544176006). +Any later publication head requires its own applicable hosted checks. + +Analytical acceptance is determined separately by the +[frozen reliability campaign](../docs/reliability-campaign.md). These checks do not +replace live development/reserved results or authorize merge, release or deployment. diff --git a/results/reliability-native-20261114.json b/results/reliability-native-20261114.json new file mode 100644 index 0000000..870483e --- /dev/null +++ b/results/reliability-native-20261114.json @@ -0,0 +1,347 @@ +{ + "status": "complete_rejected", + "responses": 92, + "candidate_source": "1812b58dfccbd6f9", + "candidate_commit": "b0569942267e9530f0ff77fbd80263111085dbe5", + "baseline_source": "368cd43ab7c6df81", + "scorer": "c100f51adc12873d", + "judge": "b44929467d85", + "accounting": "a0cc73ec381e4778", + "aggregate": { + "baseline": { + "responses": 46, + "fully_supported": 41, + "correct": 44, + "decision_correct": 45, + "answerable_cases": 36, + "evidence_reproduces_answerable": 33, + "estimated_cost_usd": 1.41156375, + "analyst_latency_s": 1590.28, + "model_calls": 342 + }, + "candidate": { + "responses": 46, + "fully_supported": 44, + "correct": 45, + "decision_correct": 45, + "answerable_cases": 36, + "evidence_reproduces_answerable": 35, + "estimated_cost_usd": 1.37437675, + "analyst_latency_s": 1663.1300000000003, + "model_calls": 336 + } + }, + "comparisons": { + "originals": { + "verdict": "accept_for_review", + "policy_hash": "6fe8128a0e410aac0b46e3bac769e67ea285cb3c2470b0776d8311dfe247a0b7", + "cost_basis": "native_estimate", + "accounting_version": "a0cc73ec381e4778", + "baseline_records_hash": "1a4835f5d226fbaf9f18e4c8c9204f4cfc421e373d3e928baa349732b3ba5fda", + "candidate_records_hash": "64db657835df8b14eb9a23722f1196911cabf19e0ae81ac8e04233d32ee52d27", + "baseline": { + "responses": 12, + "supported": 9, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.4965415, + "estimate_complete": true, + "latency_s": 644.17, + "model_calls": 107, + "source_versions": [ + "368cd43ab7c6df81" + ], + "run_ids": [ + "01a1d760-02db-7082-afcd-d31e0b43e7bd", + "01a1d760-3d76-7c7d-81d8-a2d594d3c4cf", + "01a1d762-0522-73c9-be2e-1f19aa2430df", + "01a1d762-cd0b-7b8d-bb29-e020abb7cdbc", + "01a1d763-402a-702b-9a1e-4c5309a4b795", + "01a1d764-2038-784f-9373-01f71b8e0edc", + "01a1d765-b3cf-7256-a02b-25f8e8b9deb7", + "01a1d765-d6c7-7cc0-be52-e9604f2ddd62", + "01a1d766-e50f-7952-97f5-4537bee6bd63", + "01a1d767-bde2-7fb3-bab8-9d7020760043", + "01a1d768-2e81-70ac-abd9-ffd03cff86f9", + "01a1d769-19f4-719b-8acd-18bf498bd4be" + ] + }, + "candidate": { + "responses": 12, + "supported": 12, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.48980675, + "estimate_complete": true, + "latency_s": 790.69, + "model_calls": 105, + "source_versions": [ + "1812b58dfccbd6f9" + ], + "run_ids": [ + "01a1d76a-a9fc-712e-8cb2-39a05ad7f0dd", + "01a1d76a-dea7-787f-a62c-2050ee97ea37", + "01a1d79a-4de1-745f-9a03-4dbeb6bb6e94", + "01a1d79c-14a6-7bd4-a12b-64bd2a549962", + "01a1d79d-2e54-76b5-9edb-f361077507e4", + "01a1d79e-bbdb-7d78-b129-6cd35de86cdb", + "01a1d7a1-a70c-7ed6-ba75-72ac20a5762b", + "01a1d7a2-1322-78f9-8fa8-ccc9291ca6f5", + "01a1d7a4-195a-7e55-9f41-c221238444c9", + "01a1d7a5-be17-7c46-ba48-5e2a72507f30", + "01a1d7a6-ae62-7059-bcd0-174373af9fe6", + "01a1d7a8-52f2-73a8-93f9-5260888e3f8e" + ] + }, + "invalid_reasons": [], + "regressions": [], + "improvements": [ + { + "case": "olist-l5-01", + "trial": 0 + }, + { + "case": "olist-l5-02", + "trial": 0 + }, + { + "case": "olist-l5-02", + "trial": 1 + } + ], + "acceptance_failures": [], + "promotion": "owner review required; no merge, deployment or model change performed", + "recovery": { + "plan": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-20261114/originals/recovery/recovery-plan.json", + "new_usage": { + "limit_usd": 1.0, + "max_calls": 500, + "reservation_usd": 0.05, + "reserved_usd": 0.0, + "calls": 86, + "reservation_breached": false, + "scope": "conservative token-cost admission; not provider invoice or hard billing cap", + "billing_mode": "native_api_estimate", + "accounting_version": "a0cc73ec381e4778", + "estimated_spent_usd": 0.40568350000000003, + "reported_spent_usd": null, + "unknown_charge": true, + "accounting_uncertain": false + }, + "original_batch_encumbered_usd": 1.0, + "original_interrupted_run_retained": "01a1d76c-279f-727f-bd32-dda6a081f239", + "note": "Quality comparison uses 12 fixed case/trial slots; abandoned attempt and all expenses remain separately retained." + } + }, + "development": { + "verdict": "accept_for_review", + "policy_hash": "790f4b573170e3c613a55ef1e7db79536da13f510eb38cfa210aa888d1c1271d", + "cost_basis": "native_estimate", + "accounting_version": "a0cc73ec381e4778", + "baseline_records_hash": "13e0bf484311febbbc90182b39a54ac075ab597b45d609e6430601fcdbe3549a", + "candidate_records_hash": "9de34b7310effe3d0b8cd507142a9438b503e0cda5dbcdbe4c0ed5f043fa19d1", + "baseline": { + "responses": 17, + "supported": 15, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.47826525, + "estimate_complete": true, + "latency_s": 504.84, + "model_calls": 124, + "source_versions": [ + "368cd43ab7c6df81" + ], + "run_ids": [ + "01a1d7c8-10c9-7229-ade4-48443ec4e657", + "01a1d7c8-97db-7b05-b99a-02a02d2cdfe3", + "01a1d7c9-50cb-75f4-a005-619bb78c8033", + "01a1d7ca-0eba-7a01-9056-6b26e37a498c", + "01a1d7ca-7716-720c-ac2e-6db198930ea0", + "01a1d7ca-f702-7862-8b84-3ba5efa68e14", + "01a1d7cb-98eb-75a4-9aed-1f2a997c492a", + "01a1d7cc-16d8-73ea-ade8-e81791b52688", + "01a1d7cd-02fd-72f8-a52a-ec6f330275a4", + "01a1d7cd-c602-705f-829f-86d46e8da49d", + "01a1d7ce-d862-7d5c-a9ca-9b0371e320b5", + "01a1d7cf-81e8-74fa-a3a4-aac13d1e4f23", + "01a1d7d0-3635-7786-bd0a-816154e67cdc", + "01a1d7d0-e84e-7742-97ff-f67b807c3c27", + "01a1d7d1-e244-7afd-adf6-d96271ad59ca", + "01a1d7d2-c5e9-7fd5-b6c0-654382760707", + "01a1d7d3-27dc-7b56-8c48-02c9c8e950fb" + ] + }, + "candidate": { + "responses": 17, + "supported": 17, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.44972375000000003, + "estimate_complete": true, + "latency_s": 447.66, + "model_calls": 121, + "source_versions": [ + "1812b58dfccbd6f9" + ], + "run_ids": [ + "01a1d7d3-af41-7e7f-bc06-8b426e0b0d50", + "01a1d7d4-1994-7f26-8419-efa96718ad70", + "01a1d7d4-a7bc-7b88-a130-52625ff4bebe", + "01a1d7d5-405e-7f17-bb25-b35d81cd9580", + "01a1d7d5-94c8-79d5-a22f-7de54620a8ba", + "01a1d7d5-f696-7541-852d-6b864250cd54", + "01a1d7d6-804d-7bd8-bc80-59a7fd4c55ab", + "01a1d7d6-f940-751e-a230-1b4ab75d4498", + "01a1d7d7-c813-7ca0-aaa1-05c376de163e", + "01a1d7d8-aaf3-7592-be91-f118974e3a1e", + "01a1d7d9-74e9-79b8-aebf-98a10cd7bf2b", + "01a1d7da-25ca-7c16-9a04-4c7b3cfaabb4", + "01a1d7da-bc9b-7bc9-847f-be203db3a6a6", + "01a1d7db-b381-76bb-82dd-43221bf13f40", + "01a1d7dc-8040-79af-bec2-1ab50d6cabaa", + "01a1d7dd-5137-7d2c-9338-b467e0c1b4fb", + "01a1d7dd-d65e-7a7f-b178-0029bbcf392c" + ] + }, + "invalid_reasons": [], + "regressions": [], + "improvements": [ + { + "case": "acceptance-v1-development-replay-stable", + "trial": 0 + }, + { + "case": "acceptance-v1-development-replay-unobserved-period", + "trial": 0 + } + ], + "acceptance_failures": [], + "promotion": "owner review required; no merge, deployment or model change performed", + "cycle_estimated_cost_usd": 0.9279889999999996, + "cycle_model_calls": 245, + "operator_execution_plan": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-20261114/development/durable/execution-plan.json" + }, + "reserved": { + "verdict": "reject", + "policy_hash": "5be087f7ace47a56323295f7e720455be63879294d01de51360900abd9340a53", + "cost_basis": "native_estimate", + "accounting_version": "a0cc73ec381e4778", + "baseline_records_hash": "0dd5f5506ffa9c53522b20252ed730ed66bbc74932edd7069fa467eeb8ef202d", + "candidate_records_hash": "846f8948361af1496d3f5a9613ed8b719edfe508cdf0839c051ce121e9eb14e4", + "baseline": { + "responses": 17, + "supported": 17, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.436757, + "estimate_complete": true, + "latency_s": 441.27, + "model_calls": 111, + "source_versions": [ + "368cd43ab7c6df81" + ], + "run_ids": [ + "01a1d7df-30e3-7b3a-8171-a846c5887562", + "01a1d7df-8e60-7140-bfaa-e776f9e874bd", + "01a1d7e0-2dd6-72da-93bd-8a627d140224", + "01a1d7e0-bec9-7d24-9877-8473010d0360", + "01a1d7e1-1882-7aaf-a9a7-37c8a579c31b", + "01a1d7e1-898e-7800-85e0-b29971260fc2", + "01a1d7e2-23f0-763c-b2e4-b30397eeca90", + "01a1d7e2-942e-723f-ab19-c9a7d3faf6c9", + "01a1d7e3-6f93-7af6-ab02-184fe51cdd94", + "01a1d7e4-480f-7045-a210-1b398e6caa58", + "01a1d7e5-0488-7ee9-9037-e5ab5c46b460", + "01a1d7e5-aa40-73d4-8f05-0a0d6fd8ed62", + "01a1d7e6-737d-76aa-a890-549e6df9294c", + "01a1d7e7-4a4e-75bd-882e-27a0fb427643", + "01a1d7e8-111a-733e-b2ec-67a6004d5bf6", + "01a1d7e8-fa4f-7cc4-a20a-92cd6fc1f498", + "01a1d7e9-6ee7-75c0-86e1-e44c0219fec9" + ] + }, + "candidate": { + "responses": 17, + "supported": 15, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.43484625, + "estimate_complete": true, + "latency_s": 424.78000000000003, + "model_calls": 110, + "source_versions": [ + "1812b58dfccbd6f9" + ], + "run_ids": [ + "01a1d7e9-f270-7219-b633-59be43712d03", + "01a1d7ea-837f-7332-a57c-93cbac1c722d", + "01a1d7eb-06e2-76cf-b10d-4222997620e0", + "01a1d7eb-b53b-7547-a2c2-2f487ba43088", + "01a1d7ec-10a0-7686-810e-21a5b20987e5", + "01a1d7ec-628d-7108-a8a0-9576f8c714c4", + "01a1d7ec-e127-733b-8da7-af8aadcc2fd5", + "01a1d7ed-59cf-7cb5-9a03-5e01e8ea4e9a", + "01a1d7ee-27ca-7850-a705-f85398dac5be", + "01a1d7ef-14e8-74b0-8861-8e5a892832cc", + "01a1d7f0-067e-7f7b-a6f1-520f7e09fa08", + "01a1d7f0-aae4-7d71-bc2e-2d44ce000336", + "01a1d7f1-3e27-7b8e-aff6-61d903adf27b", + "01a1d7f2-09e3-7197-98cc-8961afc79999", + "01a1d7f2-cd99-7f4d-aee2-59d5daaa7467", + "01a1d7f3-9332-7a8b-9597-58e822f3d5ce", + "01a1d7f4-30dc-7e0f-8033-fd6b5dc78b91" + ] + }, + "invalid_reasons": [], + "regressions": [ + { + "case": "acceptance-v1-acceptance-profit-complete", + "trial": 0, + "metric": "evidence_reproduces" + }, + { + "case": "acceptance-v1-acceptance-profit-complete", + "trial": 0, + "metric": "fully_supported" + }, + { + "case": "acceptance-v1-acceptance-replay-unobserved-period", + "trial": 0, + "metric": "correct" + }, + { + "case": "acceptance-v1-acceptance-replay-unobserved-period", + "trial": 0, + "metric": "abstention_correct" + }, + { + "case": "acceptance-v1-acceptance-replay-unobserved-period", + "trial": 0, + "metric": "fully_supported" + } + ], + "improvements": [], + "acceptance_failures": [ + { + "case": "acceptance-v1-acceptance-profit-complete", + "trial": 0, + "reason": "mandatory case failed" + }, + { + "case": "acceptance-v1-acceptance-replay-unobserved-period", + "trial": 0, + "reason": "mandatory case failed" + } + ], + "promotion": "owner review required; no merge, deployment or model change performed", + "cycle_estimated_cost_usd": 0.8716032500000008, + "cycle_model_calls": 221, + "operator_execution_plan": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-20261114/reserved/durable/execution-plan.json" + } + }, + "goal_used_or_encumbered_usd": 3.513161812250001, + "goal_limit_usd": 5, + "goal_remaining_usd": 1.4868381877499992, + "revised_candidate_live_validation": "not_run; proposed allowance extension awaits owner response" +} diff --git a/results/reliability-native-20261114.md b/results/reliability-native-20261114.md new file mode 100644 index 0000000..5788773 --- /dev/null +++ b/results/reliability-native-20261114.md @@ -0,0 +1,95 @@ +# Native v0.0.1 reliability campaign — 2026-11-14 UTC + +All **92 responses completed**, with valid evaluation and complete conservative native-estimate accounting. The frozen candidate is **rejected for release acceptance**: original regressions and development passed, but two mandatory reserved cases regressed. No score, tolerance, reference, model or completed answer was changed to obtain a pass. + +| Batch | Baseline supported | Candidate supported | Frozen verdict | +| --- | ---: | ---: | --- | +| originals | 9/12 | 12/12 | accept_for_review | +| development | 15/17 | 17/17 | accept_for_review | +| reserved | 17/17 | 15/17 | reject | + +| All completed responses | Baseline | Candidate | +| --- | ---: | ---: | +| Fully supported | 41/46 | 44/46 | +| Correct value or correct refusal | 44/46 | 45/46 | +| Correct answer/refusal decision | 45/46 | 45/46 | +| Reproduced evidence, answerable cases | 33/36 | 35/36 | +| Model calls including judges | 342 | 336 | +| Summed analyst latency, seconds | 1590.28 | 1663.1300000000003 | +| Estimated USD, completed responses | 1.41156375 | 1.37437675 | + +The aggregate supported score improved, but it does not override a mandatory regression. These are 40 distinct cases, with six originals repeated twice per arm, plus 17 development and 17 reserved generated worlds. They are not 92 independent domains or a production-reliability estimate. Original recovery changed process boundaries, so the aggregate latency is not a clean throughput comparison. Method-judge ratings are advisory and not human-calibrated. + +## What failed and what improved + +- Original comparison: the candidate corrected three case/trial slots with missing delivery facts or broken cited Python; baseline 9/12 versus candidate 12/12. The missing-cost refusal passed both arms. [Original results and recovery evidence](original-regression-native-20261114.md). +- Development: baseline 15/17 versus candidate 17/17. The baseline stable-period SQL put the status label in the unit column; the unavailable-period answer acknowledged missing observations but used answered rather than cannot_answer. Both passed in the candidate. +- Reserved profit: the candidate gave the correct major-currency value, but both cited SQL outputs stayed in minor units. Runtime execution-only validation accepted it; the independent value-reproduction check correctly rejected it. +- Reserved unavailable period: the candidate acknowledged that the current-period comparison could not be computed, yet returned answered with null required metrics. The baseline correctly declined. The correct-number and refusal contract checks rejected the candidate. +- The reserved baseline passed 17/17; the candidate passed 15/17. Five metric-level regression entries refer to these two case-level failures. They remain valid measured failures, not infrastructure exclusions. + +## Development and reserved LangSmith evidence + +Each linked experiment contains one preselected response and its six feedback values. Both arms used native `openai/gpt-6-luna`; native `anthropic/claude-haiku-5-5` supplied advisory method assessments. The judge retained high reasoning effort and the 8192-token cap. + +### Development + +| Case | Baseline supported / trace | Candidate supported / trace | +| --- | --- | --- | +| delivery-causality | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/bb4e60da-3074-45ac-8efc-cfa3baccf1ff) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/1c91caf2-05ea-4d62-bb9c-2082291b8930) | +| delivery-report | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/0f2464da-ee59-4156-a6e4-33ec70f40502) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/e377b467-7594-4f55-b820-068e91cf0769) | +| profit-complete | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/b37228be-cea1-4a45-88e0-f6e952ea88ed) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/b593a72e-5302-4227-9762-6987d6052998) | +| profit-missing | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/366e7b00-5da4-4a23-9c3d-fb815383029f) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/093b536f-72aa-4d63-a432-ea6e3bcc204a) | +| profit-partial | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/8688f9e4-59e1-4710-96f1-83d2ed6f0fa8) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/ca4e6c47-cbec-40c2-b34c-82e704e11145) | +| profit-zero | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/9d07818d-e35d-44b1-bcfb-f6515c07a190) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/4c4746c6-5e90-430b-b3f7-9a247cba4646) | +| repeat-report | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/89050312-2f9b-4c2e-8914-785cdf10eeb1) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/51d4b9e6-3a0d-4cd0-aab2-6cd5a9b5d780) | +| replay-duplicate-export | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/c54fc274-d2ff-46a8-b832-f9a9d419aaf2) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/607b216d-ff15-43a7-93f4-0e2e15b8982b) | +| replay-late-observations | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/dd775030-032d-41d1-8162-16018d12ae9d) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d3fc1356-37a1-480b-997e-696deff3262e) | +| replay-late-observations-followup | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/2c31210a-35af-48e9-87f3-664c021735b0) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/f573bc1b-a5dc-4740-afce-7973704bf07a) | +| replay-price-mix-only | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/b18d8b7f-c122-4898-b6db-b2482294ec54) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/73cd8df7-0170-4d4d-97e1-1dce826f125e) | +| replay-price-shift | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/fbbfb8da-d8b5-4c87-8a67-84789837e212) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/8cf43d49-7626-4830-85d1-cf06c014b2ed) | +| replay-shift | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/13d6481a-942b-4027-ab1c-c6cf8c3ecd93) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/7f0e8ae8-98b3-4016-9763-6eb07103616d) | +| replay-stable | [0/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/9ffde7b3-ebc2-4197-a0f6-64f6d7d1fe83) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/e758f8a1-5a1b-4856-842b-58b73c15544d) | +| replay-unobserved-period | [0/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/4658ca8e-758a-49fc-9910-aa8848441f30) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/8d623838-945d-4e68-a658-2963e37b4058) | +| revenue | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/047a350d-c159-4c61-8515-c548bb90af25) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/5bb29429-880c-40cc-9a3e-e8daaa167511) | +| revenue-duplicates | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/cbb95730-9d94-4769-a167-96991874208b) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/6b3c7293-0518-420a-8164-6b03ae581087) | + +### Reserved + +| Case | Baseline supported / trace | Candidate supported / trace | +| --- | --- | --- | +| delivery-causality | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/818ef1a9-6470-4b3e-813a-1038cae1818b) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/a8d7172a-d995-4fe5-b830-6a8efc1c16ed) | +| delivery-report | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/4d0c9838-e985-4a16-9e77-efb416a01a5d) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/ecfd8241-e758-432d-805f-5b15e3a6da19) | +| profit-complete | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/577ae637-5057-404c-9877-9a71d54747d1) | [0/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/aad9c59d-17ad-431e-87b5-c7ffe1537c3f) | +| profit-missing | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/1508ac69-1113-4c46-9d6c-6a7cc794729a) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/a67b37cc-7861-44f9-af75-e4adf9abd7b4) | +| profit-partial | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/aa675a13-a093-4775-a2e4-f3e43fa43978) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/9518ca10-e4cb-4fd5-a844-6503b521ace5) | +| profit-zero | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/c7e3b509-84a4-4d0c-baef-32cb7319ac61) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/362479bd-de5a-4a97-8a59-a736b7358435) | +| repeat-report | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/dbb4fe85-d75b-4916-8147-01463fbe20c9) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/65891999-2b85-4dc2-9632-138ac833b7dd) | +| replay-duplicate-export | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/2a51b9ce-55ee-45d2-9955-e5fa15fe4459) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/52940388-152b-40e4-ad09-681195a7db81) | +| replay-late-observations | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/b5930d81-d8f2-4481-8901-89c8d9de947c) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/3a6c50bb-cf0a-443f-bb55-acc5bf0c693c) | +| replay-late-observations-followup | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/0c9aba73-b7c5-4606-bb1f-70b17e44f301) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/4c83e067-782d-4442-a426-b92195da80f8) | +| replay-price-mix-only | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/27e87773-953a-46d9-a581-681fc8c5724b) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/08596f62-78b3-4a48-ae9a-5dd0a8f08704) | +| replay-price-shift | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/123e68ea-d2a1-4c07-9964-610fb6b35518) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/b710129a-309f-4a47-8285-59c7e6bf7b51) | +| replay-shift | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/256d8eb8-55a8-40c2-9b5b-1ba3b8e2aaa2) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/db1b8603-fa85-4229-83ab-cd387c31bd2d) | +| replay-stable | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/f7601783-1aef-4409-8976-631adc65f324) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/7342cb46-9071-495b-bcc7-57c0da87a1d6) | +| replay-unobserved-period | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/f514d8ca-99ec-4991-a572-6c12eb30adb5) | [0/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/77544ebc-76dc-41b8-a1e4-c1efe42821dd) | +| revenue | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/f09c66f3-ac93-4351-9317-ada9b77e68b3) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/c5e7f1c7-379c-45f4-8c42-3fb7f258a97c) | +| revenue-duplicates | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/85c84162-7e84-450e-99f3-c55e7223f04f) | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/ed4c885e-95c9-4990-b8e3-11a0ee90753d) | + +## Cost and retained attempts + +Development used **$0.927989** across 245 calls. Reserved used **$0.87160325** across 221 calls. Each completed under its separate $1 allocation, with no unsettled usage or reservation breach. All 92 completed responses total **$2.78594050** in conservative token estimates. + +Total goal usage and retained reservations is **$3.51316181225** against the existing **$5** authorization, leaving **$1.48683818775**. This includes the original interrupted attempt and retained unknown remainder as documented in the original report. Estimates are not invoices or provider balances; remaining credits do not expand authorization. + +## Revision and release boundary + +The numeric-evidence and unavailable-report gaps have local fixes in [PR13](https://github.com/IntelIP/BackIntel/pull/13) on `codex/numeric-evidence-completion`. They preserve independent scoring and return validation feedback through the existing structured-output repair path. The old candidate and all 92 results stay intact. The revised candidate has no live acceptance result yet. + +A separate proposed comparison retains the 29 unchanged original/development baseline records under their actual source identities, runs all 46 revised candidate responses, and runs 17 baseline responses on fresh reserved worlds: 63 new responses total. The new reserved generation changes periods, amounts, report distributions and world IDs; it has 13 independently checked answerable references and four refusal cases. It remains a controlled synthetic suite, not domain-transfer evidence. + +The proposed $2.15 allocation exceeds the remaining $5 envelope; an increase to $6 cumulative has been requested. The earlier zero-call proposal was superseded before execution when the second reserved failure was found. No revised paid call, merge, release or deployment follows from this report. The release qualification goal remains unfinished. + +Frozen source: candidate `1812b58dfccbd6f9` at `b0569942267e9530f0ff77fbd80263111085dbe5`, baseline `368cd43ab7c6df81`, scorer `c100f51adc12873d`, judge `b44929467d85`, accounting `a0cc73ec381e4778`. [Machine-readable comparisons](reliability-native-20261114.json) preserve each policy and record fingerprint. + +Local original evidence remains under `data/output/reliability-native-20261114/`: immutable plans, isolated source snapshots, per-job receipts, raw records, LangSmith run IDs and durable reserve/settle journals. The revised proposal is separate; old artifacts were not overwritten. diff --git a/results/reliability-revision-20261114.json b/results/reliability-revision-20261114.json new file mode 100644 index 0000000..195742e --- /dev/null +++ b/results/reliability-revision-20261114.json @@ -0,0 +1,239 @@ +{ + "status": "candidate_rejected_paired_reserved_incomplete", + "candidate_source": "d0d476e04fc7f378", + "candidate_commit": "661646fe8842e67e6734c2eb887dc056843cbffb", + "baseline_source": "368cd43ab7c6df81", + "scorer": "c100f51adc12873d", + "judge": "b44929467d85", + "new_responses": 46, + "new_supported": 45, + "new_model_calls": 353, + "retained_baseline_responses": 29, + "covered_paired_response_slots": 75, + "required_paired_response_slots": 92, + "fresh_baseline_responses": 0, + "new_inference_estimated_cost_usd": 1.4136549999999999, + "goal_used_or_encumbered_usd": 4.926816812249999, + "existing_goal_limit_usd": 5, + "remaining_existing_allowance_usd": 0.07318318775000066, + "comparisons": { + "originals": { + "verdict": "accept_for_review", + "policy_hash": "6fe8128a0e410aac0b46e3bac769e67ea285cb3c2470b0776d8311dfe247a0b7", + "cost_basis": "native_estimate", + "accounting_version": "a0cc73ec381e4778", + "baseline_records_hash": "1a4835f5d226fbaf9f18e4c8c9204f4cfc421e373d3e928baa349732b3ba5fda", + "candidate_records_hash": "aef8a0b8dc89f3f24e330fb23c8b7cc8f36f40c4142af806a935dfba8f6bcd24", + "baseline": { + "responses": 12, + "supported": 9, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.4965415, + "estimate_complete": true, + "latency_s": 644.17, + "model_calls": 107, + "source_versions": [ + "368cd43ab7c6df81" + ], + "run_ids": [ + "01a1d760-02db-7082-afcd-d31e0b43e7bd", + "01a1d760-3d76-7c7d-81d8-a2d594d3c4cf", + "01a1d762-0522-73c9-be2e-1f19aa2430df", + "01a1d762-cd0b-7b8d-bb29-e020abb7cdbc", + "01a1d763-402a-702b-9a1e-4c5309a4b795", + "01a1d764-2038-784f-9373-01f71b8e0edc", + "01a1d765-b3cf-7256-a02b-25f8e8b9deb7", + "01a1d765-d6c7-7cc0-be52-e9604f2ddd62", + "01a1d766-e50f-7952-97f5-4537bee6bd63", + "01a1d767-bde2-7fb3-bab8-9d7020760043", + "01a1d768-2e81-70ac-abd9-ffd03cff86f9", + "01a1d769-19f4-719b-8acd-18bf498bd4be" + ] + }, + "candidate": { + "responses": 12, + "supported": 12, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.469803, + "estimate_complete": true, + "latency_s": 676.8, + "model_calls": 102, + "source_versions": [ + "d0d476e04fc7f378" + ], + "run_ids": [ + "01a1d7fd-00a0-7c41-a5c7-4bf9f56e49e0", + "01a1d7fe-01eb-7322-85b6-040f0eb143b3", + "01a1d7fe-48b3-7219-b96f-22356b4ccba2", + "01a1d7ff-dbae-7f55-b4ff-b52fee784f9b", + "01a1d801-a86b-7558-b41b-e17573a9b15c", + "01a1d802-d388-71ac-b24d-2f65b81e178c", + "01a1d804-8349-7542-84bd-8e875e3411ba", + "01a1d805-3136-748a-a441-e00631d302a2", + "01a1d805-8b71-7785-a455-21c5159c8e19", + "01a1d807-2d46-725f-89b4-8d0fffd93fb1", + "01a1d808-6c20-7207-8346-ef98938d756c", + "01a1d809-f876-7145-945c-debd01746304" + ] + }, + "invalid_reasons": [], + "regressions": [], + "improvements": [ + { + "case": "olist-l5-01", + "trial": 0 + }, + { + "case": "olist-l5-02", + "trial": 0 + }, + { + "case": "olist-l5-02", + "trial": 1 + } + ], + "acceptance_failures": [], + "promotion": "owner review required; no merge, deployment or model change performed", + "cycle_estimated_cost_usd": 0.4698029999999999, + "cycle_model_calls": 102, + "operator_execution_plan": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-refresh-20261114-r2/originals/durable/execution-plan.json", + "retained_baseline_records": { + "path": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-20261114/originals/baseline/cycle-results/records.jsonl", + "sha256": "90d829a4d207c5f12461f503a728406aad3309f1719a20bba1ffb59fcb21939f", + "rule": "Reuse exact records with unchanged baseline source, model, data, reference and evaluator; never relabel source identity." + }, + "cost_scope": "cycle_estimated_cost_usd counts only new candidate inference; baseline usage remains in its original accounting" + }, + "development": { + "verdict": "accept_for_review", + "policy_hash": "790f4b573170e3c613a55ef1e7db79536da13f510eb38cfa210aa888d1c1271d", + "cost_basis": "native_estimate", + "accounting_version": "a0cc73ec381e4778", + "baseline_records_hash": "13e0bf484311febbbc90182b39a54ac075ab597b45d609e6430601fcdbe3549a", + "candidate_records_hash": "a96a067f51f268bd8d0b79e4f997a23238746142a58fc7f4d39763f502cb3050", + "baseline": { + "responses": 17, + "supported": 15, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.47826525, + "estimate_complete": true, + "latency_s": 504.84, + "model_calls": 124, + "source_versions": [ + "368cd43ab7c6df81" + ], + "run_ids": [ + "01a1d7c8-10c9-7229-ade4-48443ec4e657", + "01a1d7c8-97db-7b05-b99a-02a02d2cdfe3", + "01a1d7c9-50cb-75f4-a005-619bb78c8033", + "01a1d7ca-0eba-7a01-9056-6b26e37a498c", + "01a1d7ca-7716-720c-ac2e-6db198930ea0", + "01a1d7ca-f702-7862-8b84-3ba5efa68e14", + "01a1d7cb-98eb-75a4-9aed-1f2a997c492a", + "01a1d7cc-16d8-73ea-ade8-e81791b52688", + "01a1d7cd-02fd-72f8-a52a-ec6f330275a4", + "01a1d7cd-c602-705f-829f-86d46e8da49d", + "01a1d7ce-d862-7d5c-a9ca-9b0371e320b5", + "01a1d7cf-81e8-74fa-a3a4-aac13d1e4f23", + "01a1d7d0-3635-7786-bd0a-816154e67cdc", + "01a1d7d0-e84e-7742-97ff-f67b807c3c27", + "01a1d7d1-e244-7afd-adf6-d96271ad59ca", + "01a1d7d2-c5e9-7fd5-b6c0-654382760707", + "01a1d7d3-27dc-7b56-8c48-02c9c8e950fb" + ] + }, + "candidate": { + "responses": 17, + "supported": 17, + "reported_cost_usd": null, + "cost_complete": false, + "estimated_cost_usd": 0.45989625, + "estimate_complete": true, + "latency_s": 449.4, + "model_calls": 123, + "source_versions": [ + "d0d476e04fc7f378" + ], + "run_ids": [ + "01a1d80b-dc14-7e9d-bb55-7c0c30846b13", + "01a1d80c-705e-7b6f-8074-6435e9844b45", + "01a1d80d-0478-76b8-ad80-749d34730804", + "01a1d80d-b534-767b-8c9e-36f0e284b2e0", + "01a1d80e-0e0a-78bb-bdc7-a8c540f49bb2", + "01a1d80e-69bc-7bb6-8751-863d9e00dfdc", + "01a1d80f-07f3-77a3-a7b3-d87672722359", + "01a1d80f-80c0-7954-b9d1-6989cb7c06d2", + "01a1d810-5af4-77c6-8b4a-a26ef937b97a", + "01a1d811-2acc-71c4-8b52-8a7d24f393e0", + "01a1d811-f7c1-7970-b7ca-0af4122c6f2b", + "01a1d812-99fb-74e7-a411-22cd43374825", + "01a1d813-28d1-785d-9b31-5e22ec425012", + "01a1d814-047f-70fa-a380-168b6c87ae51", + "01a1d814-c7a4-73e3-9433-ffa81ebd29f3", + "01a1d815-9a12-78c4-85c9-7af13dfbdd1b", + "01a1d816-134b-7a75-b985-038032f65983" + ] + }, + "invalid_reasons": [], + "regressions": [], + "improvements": [ + { + "case": "acceptance-v1-development-replay-stable", + "trial": 0 + }, + { + "case": "acceptance-v1-development-replay-unobserved-period", + "trial": 0 + } + ], + "acceptance_failures": [], + "promotion": "owner review required; no merge, deployment or model change performed", + "cycle_estimated_cost_usd": 0.4598962500000001, + "cycle_model_calls": 123, + "operator_execution_plan": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-refresh-20261114-r2/development/durable/execution-plan.json", + "retained_baseline_records": { + "path": "/Users/hudson/Documents/GitHub/Products/BackIntel/BackIntel/data/output/reliability-native-20261114/development/durable/baseline-records.jsonl", + "sha256": "a8026fb7bfc1c26b0c212649b328de3191893c74c1b641f7685ab1bcff5d5a44", + "rule": "Reuse exact records with unchanged baseline source, model, data, reference and evaluator; never relabel source identity." + }, + "cost_scope": "cycle_estimated_cost_usd counts only new candidate inference; baseline usage remains in its original accounting" + } + }, + "fresh_candidate": { + "status": "candidate_rejected", + "responses": 17, + "supported": 16, + "source_version": "d0d476e04fc7f378", + "identity": { + "dataset": "backintel_acceptance_v1_reserved_2_acceptance", + "data_version": "1de641307f7cafbdbc1a1f0a6419289dd2e5f69c3265b2c9af568e94ee112897", + "reference_version": "e2ed5ca9e7e89558", + "scorer_version": "c100f51adc12873d", + "judge_model": "anthropic/claude-haiku-5-5", + "judge_version": "b44929467d85" + }, + "policy_hash": "127acfa617878c65db741952c65f34b50f90c91afd5134c9a4206692f4bc6267", + "paired_comparison_status": "incomplete; 17 fresh baseline responses have not run", + "new_inference_estimated_cost_usd": 0.4839557499999998, + "new_model_calls": 128, + "budget": { + "limit_usd": 0.55, + "max_calls": 850, + "reservation_usd": 0.05, + "reserved_usd": 0.0, + "calls": 128, + "reservation_breached": false, + "scope": "conservative token-cost admission; not provider invoice or hard billing cap", + "billing_mode": "native_api_estimate", + "accounting_version": "a0cc73ec381e4778", + "estimated_spent_usd": 0.4839557499999998, + "reported_spent_usd": null, + "unknown_charge": true, + "accounting_uncertain": false + }, + "record_fingerprint": "d12aa764952069e9c68867b63899fdffa5d177edc58ef33584b785113b714d83" + } +} diff --git a/results/reliability-revision-20261114.md b/results/reliability-revision-20261114.md new file mode 100644 index 0000000..f3528c1 --- /dev/null +++ b/results/reliability-revision-20261114.md @@ -0,0 +1,104 @@ +# Revised candidate qualification — 2026-11-14 UTC + +The revised candidate passed its original and development gates, then failed one mandatory fresh reserved case. Its 46 new responses are valid and retained; 45 are fully supported. The 17 fresh baseline responses did not run, so the paired reserved comparison is incomplete and no release acceptance is claimed. + +| Stage | Retained baseline | Revised candidate | Outcome | +| --- | ---: | ---: | --- | +| Original regressions, two trials | 9/12 | 12/12 | Accepted for review; three improved slots, no deterministic regression | +| Development | 15/17 | 17/17 | Accepted for review; two improved cases, no deterministic regression | +| Fresh reserved generation 2 | Not run | 16/17 | Candidate rejected; paired comparison incomplete | + +The 29 reused baselines retain their actual unchanged source, data, reference, model and evaluator identities; no historical output was relabeled. This covers 75 of 92 paired response slots for this comparison. The initial [92-response campaign](reliability-native-20261114.md) remains a separate rejected experiment. Its generation-1 held-out data is different; the aggregate percentages are not a paired comparison between those generations. + +## Measured outcomes + +The numeric-unit and unavailable-required-metric cases passed in generation 2. The remaining failure was the stable-period report: all seven reported facts were correct, but SQL returned a numeric status value while placing the reported label in its unit column. The independent evidence check correctly failed that explicit metric/value association. The second, correct checking query did not repair the contradictory cited output. Its score remains zero; no retry was selected for a pass. + +The next implementation checks explicit associations in SQL and structured Python JSON/JSON-lines output. It uses no hidden reference values and changes no frozen grading rules. That implementation is separate from the measured source below and needs new qualification with generation 3. + +## LangSmith evidence + +Each linked experiment contains one revised response and its six feedback values. Reused baseline experiment links remain in the initial reports. + +### Originals + +| Case | Trial | Supported / trace | +| --- | ---: | --- | +| olist-l1-01 | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d1971880-b87c-4b8b-9201-0fb7856e5f56) | +| olist-l1-01 | 2 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/843d1f69-b75e-490b-b6ab-e940a6f4e59c) | +| olist-l3-07 | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/1b0bc6d8-dec4-4f29-859c-bbe7bb7c3c55) | +| olist-l3-07 | 2 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/6e7ebf91-e4cb-4945-811b-33171ee82706) | +| olist-l5-01 | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/e73d30c1-f9a4-4d48-8cd6-3e13a216633b) | +| olist-l5-01 | 2 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/876cfd7e-7b17-403b-8f04-2659f503ecbc) | +| olist-l5-02 | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/263a57df-32c0-43a6-b493-bfec6b620b39) | +| olist-l5-02 | 2 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/eb0c5260-e658-4b89-9749-ea9431e2bf17) | +| olist-l6-2018-03 | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/20144271-1dd3-4f52-8d40-c3c5d269d0e3) | +| olist-l6-2018-03 | 2 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/f9541a18-3d31-4093-a4a0-d0d1f6aeca86) | +| olist-l6-repeat-2017-09 | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/ca7ace60-68dd-4a65-a0f0-3d8dfdbf3c91) | +| olist-l6-repeat-2017-09 | 2 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/95821f73-4403-49d1-870d-f20c3e900b6e) | + +### Development + +| Case | Trial | Supported / trace | +| --- | ---: | --- | +| acceptance-v1-development-delivery-causality | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/9d2a3654-ddb0-4f24-a565-500220e4af22) | +| acceptance-v1-development-delivery-report | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/e8d07833-6eeb-4e33-88e4-6c5ebf34bc12) | +| acceptance-v1-development-profit-complete | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/b01a2f2a-253d-465f-907f-9f1089b6b346) | +| acceptance-v1-development-profit-missing | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/e92d07eb-1711-4240-9cbb-fd616ea3c9b9) | +| acceptance-v1-development-profit-partial | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/3510c21b-9ca3-4615-9292-c5ea6304078c) | +| acceptance-v1-development-profit-zero | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/9e16f798-f9e8-4dc8-949c-ef05ff8845ae) | +| acceptance-v1-development-repeat-report | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/2c5fcb37-7b26-43a8-9f67-a0ad0ced393a) | +| acceptance-v1-development-replay-duplicate-export | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/7e7e925a-1b85-4d2f-8e39-d232041bd384) | +| acceptance-v1-development-replay-late-observations | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/52b1e136-dbe1-43f1-85d7-f338c4819adf) | +| acceptance-v1-development-replay-late-observations-followup | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/e6807ae8-980a-40ad-91b2-45e6d3242077) | +| acceptance-v1-development-replay-price-mix-only | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/71c08b00-e521-4222-8232-9f9413578b73) | +| acceptance-v1-development-replay-price-shift | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/ce7c5cac-54ac-4ee9-8702-ccf9c41983e4) | +| acceptance-v1-development-replay-shift | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/4c916a1d-b22e-4b7a-a851-6ebafc971567) | +| acceptance-v1-development-replay-stable | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/45c710f6-d191-48f0-8e0d-a8763ac34009) | +| acceptance-v1-development-replay-unobserved-period | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/563252a7-f057-4dda-91d8-e846a9f8da6b) | +| acceptance-v1-development-revenue | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/75e4cdcb-b150-48de-84d0-c135f2ed2b11) | +| acceptance-v1-development-revenue-duplicates | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/95896104-8d71-4c61-bb19-9b378615ab86) | + +### Reserved + +| Case | Trial | Supported / trace | +| --- | ---: | --- | +| acceptance-v1-reserved-2-acceptance-delivery-causality | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/1a35363b-d1c4-493a-b2cb-18679c940a63) | +| acceptance-v1-reserved-2-acceptance-delivery-report | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/6194a5df-7aa2-4e19-8dfd-802892679c79) | +| acceptance-v1-reserved-2-acceptance-profit-complete | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/fee5d21f-89ab-459b-95f2-eb73e9eb0824) | +| acceptance-v1-reserved-2-acceptance-profit-missing | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/813c5a30-e6cc-4e0b-b646-815c1e13d4e5) | +| acceptance-v1-reserved-2-acceptance-profit-partial | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/9df0da83-58ae-4d86-a1c1-5b744c0f220c) | +| acceptance-v1-reserved-2-acceptance-profit-zero | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/a793ee35-9cc0-4245-b1d5-e6cc1d6f7aae) | +| acceptance-v1-reserved-2-acceptance-repeat-report | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/7df2523b-d183-435f-a224-8c46e4ed219c) | +| acceptance-v1-reserved-2-acceptance-replay-duplicate-export | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/cf28f9da-3448-408f-bd4b-907731b6a339) | +| acceptance-v1-reserved-2-acceptance-replay-late-observations | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/3afe7cec-f774-44b8-9a69-5ea32b42a36a) | +| acceptance-v1-reserved-2-acceptance-replay-late-observations-followup | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/81293e9b-edb8-4d45-b105-ab725d97e8cf) | +| acceptance-v1-reserved-2-acceptance-replay-price-mix-only | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/fe463e9f-58e4-46ec-8c26-bc6d078f9f92) | +| acceptance-v1-reserved-2-acceptance-replay-price-shift | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/0a2fde89-7457-432c-82f3-55de08b61e02) | +| acceptance-v1-reserved-2-acceptance-replay-shift | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/b507bc2b-3ebb-483e-b584-0c6eb0a0bfc2) | +| acceptance-v1-reserved-2-acceptance-replay-stable | 1 | [0/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d52a79fd-9a05-4a01-89fe-628375586dab) | +| acceptance-v1-reserved-2-acceptance-replay-unobserved-period | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/71bc8345-a2bb-4818-af37-8936dfecaba4) | +| acceptance-v1-reserved-2-acceptance-revenue | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/8b7b57de-900f-4ec7-b80c-51e1907091dd) | +| acceptance-v1-reserved-2-acceptance-revenue-duplicates | 1 | [1/1](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d8d2d0ca-6741-432d-8a77-e794e9024837) | + +## Accounting and limits + +New candidate usage was **$1.413655** across **353** model calls including judges: original $0.469803, development $0.45989625, fresh reserved candidate $0.48395575. Reused baseline usage is not charged again. All stage ledgers settled with no uncertainty or reservation breach. + +Cumulative goal usage and retained reservations is **$4.92681681225 of $5**, leaving **$0.07318318775**. The earlier interrupted run's unknown remainder remains reserved: both provider billing panes require sign-in, so no reconciliation or release of that reservation was assumed. These are conservative estimates and reservations, not provider invoices or balances. + +The original/development stages and the fresh candidate-only arm were separately admitted within the existing $5 allowance. Continuation was not treated as approval for a higher cap. The earlier $6 request is superseded by the new candidate revision: a proposed shared $2.05 follow-up allowance requires a $7 cumulative cap. No next-revision inference is authorized by this report. + +## Reproducibility and release boundary + +Measured candidate: `d0d476e04fc7f378` at `661646fe8842e67e6734c2eb887dc056843cbffb`; retained baseline `368cd43ab7c6df81`; scorer `c100f51adc12873d`; judge `b44929467d85`; native accounting `a0cc73ec381e4778`. Analyst: native `openai/gpt-6-luna`; advisory judge: native `anthropic/claude-haiku-5-5`, high effort, 8192-token cap. Sandbox: unchanged SmolVM profile. No human calibration, model training, merge or release is claimed. + +Local evidence is retained under `data/output/reliability-native-refresh-20261114-r2/`: immutable plans and source snapshots, staged admissions, exact reused baseline records, all new answers, per-job receipts, LangSmith IDs and disk-flushed usage journals. [Machine-readable report](reliability-revision-20261114.json) retains comparison policies and fingerprints. + +The final implementation source `9463f7cec50ccc9e` at +`e501bd30be96c0a7291e2c9eec63a32e6d8257a3` passes 277 tests (15 optional skips) and +[hosted CI 43553743854](https://github.com/IntelIP/BackIntel/actions/runs/43553743854). +An offline replay of the three real failed completions confirms that the current runtime +rejects each: numeric unit mismatch, unavailable required metrics and contradictory report +value binding. This uses zero model calls, changes no historical grades and is not new +held-out acceptance. The final generation-3 proposal remains unfunded and unexecuted. diff --git a/results/taxi-demonstration-20261114/README.md b/results/taxi-demonstration-20261114/README.md new file mode 100644 index 0000000..edd28c4 --- /dev/null +++ b/results/taxi-demonstration-20261114/README.md @@ -0,0 +1,97 @@ +# Taxi business-analysis demonstration — 2026-11-14 UTC + +**Needs revision: analysis produced, full artifact delivery failed.** INT-444 ran on +11,198,026 original NYC TLC yellow-taxi Q1 2025 rows and the official zone lookup. +This is one practical demonstration with one repair, not a resumed reliability campaign. +[Data, terms and brief](../../docs/dataset-demonstration.md). + +| Attempt | Application outcome | Model calls | Duration | VM commands | +| --- | --- | ---: | ---: | ---: | +| First pass | cannot_answer; no artifacts | 17 | 480.446 s | 40 / 40 | +| One repair | failed at final export | 33 | 527.372 s | 55 / 80 | + +Both used GPT-6.1 Sol through the owner's ChatGPT subscription, medium reasoning, +SmolVM1.31.1, 4 GiB/3 CPUs, read-only assigned data and no guest network. Source +`9463f7cec50ccc9e` at `84de9d7` remained unchanged. The repair used the +existing custom resource-profile API and ordinary tooling guidance about bounded +JSON evidence, numeric precision and command budgeting. No expected data answers +were supplied. Admission declared four source files and a dictionary; the runtime +also hashed a zero-byte macOS `Icon\r` metadata file. No curator profiling or oracle +was mounted. This source family was previously prepared in a different taxi slice; +training/source-family novelty is not claimed. + +Total subscription usage: **50 model calls**, **1,172,396 input tokens**, **22,350 output +tokens**, about **16.8 minutes** of recorded run time. No per-request dollar charge is +reported for this route; no paid API fallback or new paid judge run occurred. Tokens +and subscription capacity are still consumed. Peak sampled VM use was about 397 MiB +(first) and 337 MiB (repair), including guest overhead; this is not >=10 GiB scale acceptance. + +The first attempt recovered from a DuckDB API error, but evidence mismatch and command +exhaustion prevented rendering. The repair rendered twice, inspected five charts, +returned structured completion and performed workbook checks in the VM. The clean +publisher then failed with **`ReadError: unexpected end of data`** while exporting a +single **25-file / 24,885,509-byte** bundle. The application correctly kept failed status +and removed partial host output. HTML and XLSX have **not** been delivered or reviewed +on the host. The model's completed response does not override that application failure. +Root cause is unconfirmed; [INT-498](https://linear.app/intelip/issue/INT-498) already +tracked the same error for larger combined exports and now includes this single-report case. +Fix and verify the transfer offline using valid artifacts before any further inference. + +## Three independently recomputed claims + +Using fresh host SQL over raw Parquet and the lookup, with decimal currency sums: + +| Claim under the declared eligibility | Recomputed | Result | +| --- | ---: | --- | +| Eligible trip records | 10,455,713 | Matches final render metric | +| Gross recorded metered fare | $193,587,031.30 | Matches final render metric | +| Airport concentration | JFK $24,330,170.25 / 12.57%; LaGuardia $11,490,575.70 / 5.94% | Matches reported shares | + +Eligibility: Q1 pickup, payment codes 0/1/2, positive finite fare/charge, duration above +zero and at most 24 hours, metered fare at most $1,000. The last screen removes nine +otherwise eligible records / $58,062.58. These are reported rows, not unique vehicles +or observed unmet demand. Metered fare is not audited settlement or profit. Missing +passenger counts/distance remain; occupancy/speed claims are avoided. Local civil +time/DST, changing payment coverage and observational-policy limitations remain. +The scoped checks support these three values; they do not certify every report claim. +Portable third-check SQL is retained in [assessment.json](assessment.json); exact local +SQL/results are outside Git in `data/output/int444-taxi-20261114/`. + +## Retained chart previews + +These five PNGs were recovered **without model calls** from `inspect_chart` messages. +Each SHA-256 matches the last successful render manifest. They are actual retained +previews, not a reconstructed or delivered report bundle. Titles and primary comparisons +are readable. Internal column names, cluttered automatic date ticks and the airport-mix +heatmap's weak comparison of small shares fall short of the desired executive polish. +The active renderer needs that refinement; no additional visual repair was attempted. + +### Monthly activity + +![Monthly activity; retained agent chart preview](monthly_activity.png) + +### Weekday and hour + +![Weekday and hour; retained agent chart preview](pickup_rhythm.png) + +### Pickup-zone fares + +![Pickup-zone fares; retained agent chart preview](pickup_receipts.png) + +### Fare distribution + +![Fare distribution; retained agent chart preview](fare_distribution.png) + +### Airport fare mix + +![Airport fare mix; retained agent chart preview](monthly_mix.png) + +## Traces and stopping decision + +- [First attempt](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d975ee3d-b39c-4ea1-83a5-5007cd2e3be6/r/829419de-5a11-4c9c-98ee-164b9305798f?poll=true) +- [Single bounded repair](https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d975ee3d-b39c-4ea1-83a5-5007cd2e3be6/r/be4a2de5-a513-4e4e-bb71-23ac339581c9?poll=true) +- [Structured assessment, usage and preview hashes](assessment.json) + +No third model attempt will run for this demonstration. Preserve both failures, keep +INT-444 open for usable artifact delivery, and resolve INT-498 offline. Do not restart +the frozen 63-response campaign or claim release qualification from this run. diff --git a/results/taxi-demonstration-20261114/assessment.json b/results/taxi-demonstration-20261114/assessment.json new file mode 100644 index 0000000..8cf93ad --- /dev/null +++ b/results/taxi-demonstration-20261114/assessment.json @@ -0,0 +1,212 @@ +{ + "assessment": "needs_revision", + "source_version": "9463f7cec50ccc9e", + "git_sha": "84de9d7", + "model": "chatgpt-plan/gpt-6.1-sol", + "reasoning": "medium", + "sandbox": "smolvm", + "data_rows": 11198026, + "attempts": [ + { + "name": "first-pass", + "status": "cannot_answer", + "error": null, + "completion": { + "status": "cannot_answer", + "summary": "The analysis supports several operational hypotheses: eligible recorded activity rose from about 105,595 trips per day in January to 124,079 in March; JFK and LaGuardia accounted for approximately 6.2% of eligible trips but 22.8% of recorded card/cash metered fares; and Thursday 18:00\u201319:00 was the strongest quarter-average weekday-hour pickup cell. The eligible population contains 10,455,722 recorded trips, with $156,152,091 in positive recorded card/cash metered fares. These results favor limited, controlled schedule and airport-rotation trials rather than broad redeployment. However, the required report bundle could not be completed and verified.", + "reason": "Artifact generation did not complete. The first render rejected a supporting table\u2019s evidence match. After stabilizing derived floating-point measures and regenerating the specification, the next render stopped during evidence execution with 'Whole-run command/time budget exhausted.' Consequently, no successful render or required chart inspection occurred. A follow-up run must finish evidence validation, render the prepared analytical specification, and inspect all five charts before claiming completion.", + "caveats": [ + "Recorded trip activity is not unmet demand or fleet availability; metered fares and passenger charges are not profit.", + "Activity includes eligible Flex/card/cash records; receipt calculations include only card/cash records. Growing Flex coverage complicates comparisons across months.", + "Eligibility requires a Q1 pickup, positive finite fare and total charge, and a positive recorded duration no longer than 24 hours. Exclusions are sequential and documented in the prepared analysis.", + "Timestamps are assumed NYC local civil time without timezone offsets; daylight-saving limitations apply.", + "No report, charts, CSV tables or workbook should be represented as delivered or visually verified from this unsuccessful run." + ] + }, + "usage": { + "llm_calls": 17, + "input_tokens": 359325, + "output_tokens": 10488, + "cost_usd": null, + "unpriced_llm_calls": 17, + "usage_complete": true, + "cost_complete": false, + "scope": "ChatGPT subscription token usage; no per-request dollar charge reported", + "billing_mode": "chatgpt_plan", + "cost_applicable": false + }, + "latency_s": 480.446, + "resources": { + "backend": "smolvm", + "runtime_version": "1.31.1", + "peak_memory_bytes": 416509952, + "memory_measurement": "sampled VM used memory; includes guest runtime", + "commands": 40, + "elapsed_ms": 458335, + "image_id": "sha256:9820b507a21f383ad0af1fac20426a6123ef718df79e35740461ab3ff3087dac", + "image_archive_sha256": "d12436507cf87f1eb99cc360232e9dc9b2ac0ec8c9addf4167395cdc78ba24a5", + "profile": { + "memory_mib": 4096, + "cpus": 3.0, + "workspace_mib": 1024, + "artifacts_mib": 128, + "temporary_mib": 512, + "pids": 128, + "command_timeout_s": 300, + "run_timeout_s": 1200, + "max_commands": 40 + }, + "cancelled": true + }, + "trace": { + "trace_id": "829419de-5a11-4c9c-98ee-164b9305798f", + "url": "https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d975ee3d-b39c-4ea1-83a5-5007cd2e3be6/r/829419de-5a11-4c9c-98ee-164b9305798f?poll=true" + } + }, + { + "name": "repair", + "status": "failed", + "error": "ReadError: unexpected end of data", + "completion": { + "status": "completed", + "summary": "Q1 2025 contains 10,455,713 eligible paid-trip records with USD 193,587,031.30 in gross recorded metered fare. Eligible trips per calendar day rose from 105,595 in January to 124,079 in March; Thursday 18:00\u201318:59 was the busiest average weekday-hour. JFK supplied 12.57% of metered fare from 3.67% of records, while LaGuardia supplied 5.94% from 2.57%. Prioritize limited matched-shift trials of evening staging and airport versus core-city positioning, measuring queue/idle time, costs and net receipts per vehicle-hour before expanding. Five charts were visually reviewed; supporting aggregate workbook cells reconcile with the canonical report tables, and grouped totals and zone-join checks passed.", + "reason": null, + "caveats": [ + "Eligibility requires Q1 pickup time, payment codes 0/1/2, finite positive fare and total passenger charge, positive duration no greater than 24 hours, and metered fare no greater than USD 1,000. The final upper-fare screen excludes 9 records and USD 58,062.58.", + "Gross metered fare is a recorded receipt proxy, not audited settlement, total passenger charges or profit. Cash tips, costs, idle time, fleet availability and unmet demand are not observed.", + "Missing passenger counts and invalid distances are retained without occupancy, mileage or speed claims. No unique trip ID supports definitive deduplication; records are not deduplicated.", + "Naive timestamps are assumed to represent NYC local civil time without daylight-saving correction. Calendar normalization does not control for holidays, weather or weekday mix.", + "Historical associations do not establish congestion-pricing effects or justify current fleet expansion without fresh data and a controlled operational follow-up." + ] + }, + "usage": { + "llm_calls": 33, + "input_tokens": 813071, + "output_tokens": 11862, + "cost_usd": null, + "unpriced_llm_calls": 33, + "usage_complete": true, + "cost_complete": false, + "scope": "ChatGPT subscription token usage; no per-request dollar charge reported", + "billing_mode": "chatgpt_plan", + "cost_applicable": false + }, + "latency_s": 527.372, + "resources": { + "backend": "smolvm", + "runtime_version": "1.31.1", + "peak_memory_bytes": 353755136, + "memory_measurement": "sampled VM used memory; includes guest runtime", + "commands": 55, + "elapsed_ms": 506022, + "image_id": "sha256:9820b507a21f383ad0af1fac20426a6123ef718df79e35740461ab3ff3087dac", + "image_archive_sha256": "d12436507cf87f1eb99cc360232e9dc9b2ac0ec8c9addf4167395cdc78ba24a5", + "profile": { + "memory_mib": 4096, + "cpus": 3.0, + "workspace_mib": 1024, + "artifacts_mib": 128, + "temporary_mib": 512, + "pids": 128, + "command_timeout_s": 300, + "run_timeout_s": 1200, + "max_commands": 80 + }, + "cancelled": false + }, + "trace": { + "trace_id": "be4a2de5-a513-4e4e-bb71-23ac339581c9", + "url": "https://smith.langchain.com/o/90b24546-f3e2-4f48-af72-b06e73b27b49/projects/p/d975ee3d-b39c-4ea1-83a5-5007cd2e3be6/r/be4a2de5-a513-4e4e-bb71-23ac339581c9?poll=true" + } + } + ], + "model_calls": 50, + "input_tokens": 1172396, + "output_tokens": 22350, + "model_run_seconds": 1007.818, + "billing_mode": "chatgpt_plan", + "per_request_dollar_cost": null, + "full_html_xlsx_exported": false, + "final_render_files": 25, + "final_render_bytes": 24885509, + "independent_checks": { + "status": "three_scoped_claim_checks_passed", + "checks": [ + { + "claim": "Eligible Q1 trip records", + "observed": 10455713, + "passed": true + }, + { + "claim": "Gross recorded metered fare USD", + "observed": "193587031.3", + "passed": true + }, + { + "claim": "Airport fare concentration", + "observed": [ + { + "zone": "JFK Airport", + "trips": 383928, + "fare_usd": "24330170.25", + "fare_share_pct": "12.57" + }, + { + "zone": "LaGuardia Airport", + "trips": 269016, + "fare_usd": "11490575.70", + "fare_share_pct": "5.94" + } + ], + "passed": true, + "sql": "WITH eligible AS (\n SELECT * FROM read_parquet('ASSIGNED_DATA_DIR/yellow_tripdata_2025-*.parquet')\n WHERE tpep_pickup_datetime >= TIMESTAMP '2025-01-01' AND tpep_pickup_datetime < TIMESTAMP '2025-04-01'\n AND payment_type IN (0,1,2)\n AND isfinite(fare_amount) AND isfinite(total_amount) AND fare_amount > 0 AND total_amount > 0\n AND tpep_dropoff_datetime > tpep_pickup_datetime\n AND tpep_dropoff_datetime <= tpep_pickup_datetime + INTERVAL '24 hours'\n AND fare_amount <= 1000\n), locations AS (SELECT * FROM read_csv_auto('ASSIGNED_DATA_DIR/taxi_zone_lookup.csv'))\nSELECT z.Zone, count(*) AS trips, sum(CAST(e.fare_amount AS DECIMAL(18,2))) AS fare FROM eligible e JOIN locations z ON e.PULocationID=z.LocationID WHERE z.Zone IN ('JFK Airport','LaGuardia Airport') GROUP BY z.Zone ORDER BY z.Zone" + } + ], + "scope": "Three numerical claims only. HTML/XLSX export failed; no full report/workbook review or population reliability claim." + }, + "retained_previews": { + "origin": "PNG results returned by inspect_chart in the retained model trace; not a recovered HTML/XLSX bundle", + "previews": [ + { + "name": "monthly_activity.png", + "bytes": 71069, + "sha256": "7b15adc95abf9948e2381156157a1035d64c17ad92de4cdb92b9cb72cfde03ea", + "matches_final_render": true + }, + { + "name": "pickup_rhythm.png", + "bytes": 87041, + "sha256": "c6b811e49091d97bfa418a7e3f0d38221ec974a7f9d0562a9e5ebb0ed02b0c89", + "matches_final_render": true + }, + { + "name": "pickup_receipts.png", + "bytes": 56830, + "sha256": "73a0cb02dcc61aca191fd1cb4db5262b2a59548245ec7bd423dc1fea1c164213", + "matches_final_render": true + }, + { + "name": "fare_distribution.png", + "bytes": 40720, + "sha256": "501c61aca5091046b4d09835cc3a9da2fb9076254da146b1a3ec0735e4f923ab", + "matches_final_render": true + }, + { + "name": "monthly_mix.png", + "bytes": 68482, + "sha256": "4d83eeef5a6338653c706411922f37a7cf5a02186371a2878cb4e93ce18318c4", + "matches_final_render": true + } + ], + "successful_render_manifests": 2 + }, + "issue": "INT-498", + "further_model_attempts": 0, + "visual_findings": [ + "Five retained PNGs inspected; readable titles, units and comparisons.", + "Internal field names remain in axis labels.", + "Monthly line chart has cluttered automatic mid-month date ticks.", + "Monthly mix heatmap makes small airport differences hard to compare against the dominant other-pickups category." + ], + "review_limits": "Three numerical claims checked. Whole HTML, interactivity and host workbook review unavailable because export failed. Agent workbook checks are retained in its trace, not relabeled as host verification." +} diff --git a/results/taxi-demonstration-20261114/fare_distribution.png b/results/taxi-demonstration-20261114/fare_distribution.png new file mode 100644 index 0000000..afe4801 Binary files /dev/null and b/results/taxi-demonstration-20261114/fare_distribution.png differ diff --git a/results/taxi-demonstration-20261114/monthly_activity.png b/results/taxi-demonstration-20261114/monthly_activity.png new file mode 100644 index 0000000..39d1ce5 Binary files /dev/null and b/results/taxi-demonstration-20261114/monthly_activity.png differ diff --git a/results/taxi-demonstration-20261114/monthly_mix.png b/results/taxi-demonstration-20261114/monthly_mix.png new file mode 100644 index 0000000..7b4eb7f Binary files /dev/null and b/results/taxi-demonstration-20261114/monthly_mix.png differ diff --git a/results/taxi-demonstration-20261114/pickup_receipts.png b/results/taxi-demonstration-20261114/pickup_receipts.png new file mode 100644 index 0000000..73e23fe Binary files /dev/null and b/results/taxi-demonstration-20261114/pickup_receipts.png differ diff --git a/results/taxi-demonstration-20261114/pickup_rhythm.png b/results/taxi-demonstration-20261114/pickup_rhythm.png new file mode 100644 index 0000000..4937de7 Binary files /dev/null and b/results/taxi-demonstration-20261114/pickup_rhythm.png differ