Package unfamiliar data and grade reproduced report facts

benchmarks/unfamiliar/unfamiliar-v1/development-tasks.json

@@ -0,0 +1,104 @@
+{
+  "version": "unfamiliar-v1-task-draft-20261113",
+  "split": "development",
+  "cases": [
+    {
+      "id": "transport-01",
+      "domain": "transport",
+      "category": "population",
+      "question": "Count records with pickup in [2023-01-01, 2023-02-01) using recorded timestamps. Report excluded records outside that window separately. Retain duplicate rows unless the source supplies a record identity that proves duplication.",
+      "mandatory": true
+    },
+    {
+      "id": "transport-02",
+      "domain": "transport",
+      "category": "aggregation",
+      "question": "For January pickups, report signed recorded total_amount in USD by payment_type, including an explicit unknown group, and the grand total. Do not interpret these amounts as driver profit or complete cash-tip receipts.",
+      "mandatory": true
+    },
+    {
+      "id": "transport-03",
+      "domain": "transport",
+      "category": "join",
+      "question": "For January pickups, attach pickup Borough from the supplied zone lookup and report trip counts by borough. Preserve every input trip, expose unmatched/unknown zones, and show that the join has not multiplied records.",
+      "mandatory": true
+    },
+    {
+      "id": "transport-04",
+      "domain": "transport",
+      "category": "distribution",
+      "question": "For January pickups with drop-off later than pickup and duration at most 180 minutes, report the median and 95th percentile duration in minutes using linear-interpolated quantiles (type 7). Report the excluded-row count; do not silently clamp durations.",
+      "mandatory": true
+    },
+    {
+      "id": "transport-05",
+      "domain": "transport",
+      "category": "temporal",
+      "question": "Compare recorded trip counts for the equal pickup windows [2023-01-02, 2023-01-09) and [2023-01-09, 2023-01-16). Report each count, absolute change, and relative change from the first window. Attribute no causal explanation.",
+      "mandatory": true
+    },
+    {
+      "id": "transport-06",
+      "domain": "transport",
+      "category": "insufficient_data",
+      "question": "What was total driver net profit for the January trips after fuel, vehicle expenses, platform/lease charges, and all tips? Explain whether the supplied sources can establish that metric.",
+      "mandatory": true
+    },
+    {
+      "id": "transport-07",
+      "domain": "transport",
+      "category": "discovery",
+      "question": "Produce a concise January service report from the assigned records with at most five independently reproducible findings, linked charts, CSV tables and a supporting workbook. State scope and consequential source limitations. Fewer than five findings are acceptable.",
+      "mandatory": true
+    },
+    {
+      "id": "energy-01",
+      "domain": "building-energy",
+      "category": "coverage",
+      "question": "For [2007-01-01, 2007-02-01) in the recorded wall-clock labels, report eligible minute slots, rows present, and minutes with a numeric Global_active_power reading. Treat missing/unknown readings separately from zero. Make no unsupported timezone or DST conversion.",
+      "mandatory": true
+    },
+    {
+      "id": "energy-02",
+      "domain": "building-energy",
+      "category": "unit_conversion",
+      "question": "For numeric Global_active_power readings in January 2007, calculate observed energy in kWh from the documented minute-averaged kW measurements. Report observed-minute coverage; do not impute missing readings or present a partial-observation total as a complete meter bill.",
+      "mandatory": true
+    },
+    {
+      "id": "energy-03",
+      "domain": "building-energy",
+      "category": "reconciliation",
+      "question": "For January 2007 minutes with all required numeric measurements, report observed total active energy and Sub_metering_1, Sub_metering_2, Sub_metering_3 energies in Wh, plus the residual outside these three submeters. Retain signed residuals and report the eligible-minute population; do not count the global meter and submeters as additive independent loads.",
+      "mandatory": true
+    },
+    {
+      "id": "energy-04",
+      "domain": "building-energy",
+      "category": "ranking",
+      "question": "Rank the five January 2007 calendar-label days with greatest observed energy in kWh, retaining only days with at least 95% of their 1,440 minute slots observed. Show observed minutes and coverage for each; break exact energy ties by ascending date.",
+      "mandatory": true
+    },
+    {
+      "id": "energy-05",
+      "domain": "building-energy",
+      "category": "temporal",
+      "question": "Compare observed average minute-averaged active power in kW for [2007-01-01, 2007-01-08) and [2007-01-08, 2007-01-15). Report each observed-minute count, each mean, and the difference (second-window mean minus first-window mean) in kW. Missing readings are excluded, not zeros, and differences are descriptive.",
+      "mandatory": true
+    },
+    {
+      "id": "energy-06",
+      "domain": "building-energy",
+      "category": "insufficient_data",
+      "question": "What was the household electricity bill in euros for January 2007, including time-of-use tariffs, fixed fees and taxes? Explain whether the supplied records support an exact bill.",
+      "mandatory": true
+    },
+    {
+      "id": "energy-07",
+      "domain": "building-energy",
+      "category": "discovery",
+      "question": "Produce a concise January 2007 energy report with at most five reproducible findings, a coverage view, appropriate charts, CSV tables and a workbook. Distinguish recorded power, observed energy and missing observations, and do not claim causal explanations.",
+      "mandatory": true
+    }
+  ]
+}

benchmarks/unfamiliar/unfamiliar-v1/reference-receipt.json

@@ -0,0 +1,61 @@
+{
+  "commits": 0,
+  "comparison_difference_count": 0,
+  "development": {
+    "cases": 12,
+    "continuous_numbers": {
+      "absolute_tolerance": 1e-06,
+      "relative_tolerance": 1e-08
+    },
+    "counts_and_labels": "exact",
+    "created_at": "2026-11-13T18:31:01.013866+00:00",
+    "difference_count": 0,
+    "human_calibration": false,
+    "inputs": {
+      "comparison_implementation_sha256": "2e2dc0f3e3698f572cbd36d65db80ab3e3929a92207a39428e5d3a1e01b7029b",
+      "python_implementation_sha256": "91ad403db55db9f455858b5fe3bccf50553f7aea4cfa56f3dd10d62eb1f491a5",
+      "sources_sha256": "1c90a5986a68c1fa8192a06bf42c517ecc5e220c87a1091cd950fd9a1fe85087",
+      "sql_implementation_sha256": "f4b66fb3c4e28fd244985ff7937afab7c6d9bf7aadcdbde5c177c32f067287ba",
+      "tasks_sha256": "e6fde92929fa6762621b9d7e5ddea1ebfae2149b1a88f218d5e355d2d71adb61"
+    },
+    "insufficient_data_cases": 2,
+    "model_calls": 0,
+    "numeric_cases": 10,
+    "python_sha256": "5933ec485b0f668b1cc234e4f727a98ff3d0e9f3e81284469bb5cac978715682",
+    "sql_sha256": "6e7711bd9406524387467a3e0ff0d56f2b0f954e396d4fa528a8c687dfc5ab55"
+  },
+  "development_python_exact_repeat_confirmed": true,
+  "discovery_verification": "Partial: evaluator-mapped facts and existing artifact bindings only; generic narrative/causality and refusal semantics not integrated.",
+  "insufficient_data_cases": 3,
+  "model_calls": 0,
+  "numeric_cases": 15,
+  "private_reference_visibility": "Evaluator-only; hashes/coverage public, no reserved answers or precise questions included.",
+  "profile_decoy_excluded": true,
+  "profile_sha256": "436e74dea81ae51bca96dd507f98739e17d665dfae1be174e30e792a17e8bd48",
+  "pushes": 0,
+  "reserved": {
+    "cases": 6,
+    "continuous_numbers": {
+      "absolute_tolerance": 1e-06,
+      "relative_tolerance": 1e-08
+    },
+    "counts_and_labels": "exact",
+    "created_at": "2026-11-13T18:30:30.018284+00:00",
+    "difference_count": 0,
+    "human_calibration": false,
+    "inputs": {
+      "comparison_implementation_sha256": "2e2dc0f3e3698f572cbd36d65db80ab3e3929a92207a39428e5d3a1e01b7029b",
+      "implementation_sha256": "5235e76a05cb6e46384bfaef3d4187fe168cd24f43d2be33165a32647c2b1e0b",
+      "sources_sha256": "1c90a5986a68c1fa8192a06bf42c517ecc5e220c87a1091cd950fd9a1fe85087",
+      "task_spec_sha256": "783f3e299767f91cb8271310d36a442607dfa88db69f86dec9cc5b4bfcac9411"
+    },
+    "insufficient_data_cases": 1,
+    "model_calls": 0,
+    "numeric_cases": 5,
+    "python_sha256": "40ffb9ed168fcb8798b2cfcb625eb8810793b54bcc74d7c9bdca2205b580a2a9",
+    "sql_sha256": "41c34c536eadda0e2a8b1ece807dfefe9130299715adbd25fbeb51e0a0a9b960"
+  },
+  "source_profiler_sha256": "503af0a3a0a282a0ea6dac874afc5f3010c962ea86050258e606e7be517c8464",
+  "total_fixed_challenges": 18,
+  "version": "unfamiliar-v1-references-20261113"
+}

benchmarks/unfamiliar/unfamiliar-v1/reserved-task-commitment.json

@@ -0,0 +1,17 @@
+{
+  "version": "unfamiliar-v1-task-draft-20261113",
+  "domain": "weather",
+  "scenario_count": 7,
+  "sha256": "783f3e299767f91cb8271310d36a442607dfa88db69f86dec9cc5b4bfcac9411",
+  "case_ids": [
+    "weather-01",
+    "weather-02",
+    "weather-03",
+    "weather-04",
+    "weather-05",
+    "weather-06",
+    "weather-07"
+  ],
+  "private_relative_path": "data/unfamiliar-v1/evaluator/reserved-task-spec.json",
+  "truth_generated": false
+}

benchmarks/unfamiliar/unfamiliar-v1/source-docs/building-energy.md

@@ -0,0 +1,21 @@
+# Individual household electric power consumption
+
+Creators: Georges Hebrail and Alice Berard. Repository: University of California,
+Irvine, DOI 10.24432/C58K54. License: CC BY 4.0. The semicolon-delimited text file
+is the exact qualified ZIP member; no measurement values have been transformed.
+
+This source contains minute measurements from one household in Sceaux, France.
+`Date` uses day/month/year and `Time` uses hour:minute:second. `Global_active_power`
+and `Global_reactive_power` are minute-averaged kW and kVAR respectively. `Voltage`
+is minute-averaged volts; `Global_intensity` is minute-averaged amperes.
+`Sub_metering_1`, `Sub_metering_2`, and `Sub_metering_3` report watt-hours for the
+minute. The source describes missing measurements; inspect the supplied raw
+labels rather than assuming that every nonempty value is numeric. No timezone
+conversion has been applied.
+
+This note contains source definitions, not data-profile findings or task answers.
+Retain creator attribution and identify any further transformations.
+
+Source: https://archive.ics.uci.edu/dataset/235/individual+household+electric+power+consumption
+License: https://creativecommons.org/licenses/by/4.0/
+Source documentation consulted during qualification on 2026-11-13.

benchmarks/unfamiliar/unfamiliar-v1/source-docs/transport.md

@@ -0,0 +1,22 @@
+# NYC TLC green taxi records
+
+Publisher: New York City Taxi and Limousine Commission. This is the January 2023
+green trip file and the TLC taxi-zone lookup, retained without modification.
+
+`lpep_pickup_datetime` and `lpep_dropoff_datetime` are recorded trip timestamps.
+`trip_distance` is recorded distance in miles. `PULocationID` and `DOLocationID`
+refer to `LocationID` in the zone lookup. The lookup describes borough, zone and
+service zone. Recorded fare/payment amounts are in US dollars. `total_amount`
+is the total charged as reported; it does not capture cash tips. `tip_amount`
+captures credit-card tips. Payment types are 1 credit card, 2 cash, 3 no charge,
+4 dispute, 5 unknown, and 6 voided trip in the published dictionary.
+
+The source publisher does not warrant submitted trip completeness or accuracy.
+The current published dictionary describes newer schema fields as well; inspect
+the supplied file for the actual columns. This note supplies no measured quality
+counts, task answers, filtering recipe or reference calculations.
+
+Source: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
+Dictionary: https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_green.pdf
+Use terms: https://www.nyc.gov/opendata/get-started/FAQs and https://www.nyc.gov/main/terms-of-use
+Source documentation consulted during qualification on 2026-11-13.

benchmarks/unfamiliar/unfamiliar-v1/source-docs/weather.md

@@ -0,0 +1,19 @@
+# NOAA Global Summary of the Day
+
+Publisher: NOAA National Centers for Environmental Information, US Department of
+Commerce. The assigned files are original 2023 CSVs for three US stations. Read
+the included original `gsod_readme.txt` for variable definitions, missing-value
+sentinels, precipitation attributes, units, and observation-period qualifications.
+
+The original files have not been reinterpreted, imputed, resampled or converted
+to a different calendar/timezone. Source quality and observation periods must be
+considered before interpreting a field. Preserve NOAA/NCEI attribution and do
+not imply endorsement. The README's non-US commercial redistribution restriction
+is why this package is restricted to US stations.
+
+This note contains source context only, not private questions, observed quality
+counts or reference answers.
+
+Metadata: https://www.ncei.noaa.gov/metadata/geoportal/rest/metadata/item/gov.noaa.ncdc:C00516/html
+Original documentation: https://www.ncei.noaa.gov/data/global-summary-of-the-day/doc/readme.txt
+Source documentation consulted during qualification on 2026-11-13.