From d7514044feeba2a0364ba5b954197f44a9a5e318 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:31:41 +0100 Subject: [PATCH 01/13] feat: define the client contract and its error map Every client must give the same results and errors for the same responses of the gateway. client.md fixes them, with the map from each status to one error code that ERR-1 asks for. The calls of 2 Oct 2026 showed facts that the first interface missed: lookup has a status, suggestions have no label, an empty autocomplete query gets no answer, and a 429 carries no Retry-After. The spec moves to 0.2.0, because it adds the contract. Signed-off-by: MobileMage --- CONTEXT.md | 14 ++- README.md | 6 +- VERSION | 2 +- client.md | 158 +++++++++++++++++++++++++++ grammar.md | 4 +- scripts/contract_files.py | 18 +++ scripts/tests/test_contract_files.py | 45 ++++++++ standards/CODING_STANDARDS.md | 8 +- standards/languages/typescript.md | 12 +- 9 files changed, 256 insertions(+), 11 deletions(-) create mode 100644 client.md create mode 100644 scripts/contract_files.py create mode 100644 scripts/tests/test_contract_files.py diff --git a/CONTEXT.md b/CONTEXT.md index 64a0f59..fcc8182 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -122,9 +122,21 @@ _Avoid_: using this word for a Gatepost component ### Gatepost **Spec**: -The `spec` repo. It holds the grammar, the data, the vectors and the standards. The API fixtures and the completed OpenAPI file come later. +The `spec` repo. It holds the grammar, the client contract, the data, the vectors, the fixtures, the completed OpenAPI file and the standards. _Avoid_: schema, contract (for the repo) +**Contract scenario**: +One shared test case in `spec/contract/`. It names the calls, the mock server's responses and the outcome that every client must reach. +_Avoid_: contract test (for the file), vector, fixture + +**Attempt**: +One request that a client sends for one call. A retry is the second or third attempt. +_Avoid_: try, request (when you count them) + +**Evidence**: +How Gatepost knows the shape of a gateway response: `observed` in its own calls, `documented` by NIPOST only, or `assumed` by the mock server. +_Avoid_: source, confidence, proof + **Vector**: One shared test case in `spec/vectors/` that every SDK must pass. _Avoid_: fixture, example, golden file diff --git a/README.md b/README.md index 8efd4e6..8e1a4fa 100644 --- a/README.md +++ b/README.md @@ -9,12 +9,16 @@ The shared contract for every Gatepost SDK: the postcode grammar, the data files | Path | Holds | |---|---| | `grammar.md` | how every SDK reads and writes postcodes, and the public interface | +| `client.md` | how every client calls NIPOST's gateway: results, errors, retries and the queue | +| `openapi/` | NIPOST's gateway, completed with the responses that its own file leaves out | +| `fixtures/` | synthetic gateway responses and keys for the mock server | +| `contract/` | shared client test cases that every client must pass against the mock server | | `data/` | state codes, GPS accuracy limits and the postcode format | | `vectors/` | shared test cases that every SDK must pass | | `standards/` | the coding standards and one file for each language | | `CONTEXT.md` | the glossary | | `templates/` | files for new repos | -| `scripts/` | `check-tells`, the vector builder, and a plain reading of the grammar that checks each vector | +| `scripts/` | `check-tells`, the vector builder, a plain reading of the grammar that checks each vector, and the checks of the OpenAPI file, the fixtures and the scenarios | ## Check diff --git a/VERSION b/VERSION index 6e8bf73..0ea3a94 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.1.0 +0.2.0 diff --git a/client.md b/client.md new file mode 100644 index 0000000..9c1686e --- /dev/null +++ b/client.md @@ -0,0 +1,158 @@ +# Client + +This file defines how every Gatepost client calls NIPOST's gateway. The contract scenarios in `contract/` test each rule here, except the rules that the last section lists. If this file and a scenario disagree, this file wins. Such a disagreement is a bug in the spec. + +`openapi/gateway.completed.yaml` describes the gateway itself. Each response in it carries an evidence mark. Gatepost saw some responses in calls of its own. It knows others only from NIPOST's docs. A client reads both kinds in the same way. + +## Interface + +A client exposes the symbols in this table and the types that they need. A language file gives the idiomatic form of each name, as for the core. + +| Symbol | Takes | Returns | +|---|---|---| +| `PostcodeClient(options)` | client options | a client | +| `lookup(code, level)` | a postcode as text or as a postcode, and a lookup level, which is 1 by default | a lookup result | +| `reverse(lat, lng, maxDistanceM)` | a latitude and a longitude in degrees, and an optional radius in metres | a reverse result | +| `autocomplete(q)` | the text that a user typed | an autocomplete result | +| `clearCache()` | none | none | +| `PostcodeError` | none | the error that each call can raise | + +Each call that sends a request is asynchronous, and the caller can cancel it (API-11). The client does not wrap `nearby`, `assemble` and `disassemble`. The core does the work of the assembly endpoints offline. No call to `nearby` has returned a unit yet, so its result has no defined fields. + +### Client options + +| Option | Meaning | Default | +|---|---|---| +| `apiKey` | the key for the `X-API-Key` header | none, so the client sends no key | +| `baseUrl` | the gateway's address | `https://api.postcode.gov.ng` | +| `transport` | the HTTP transport, in the language's standard form | the platform's own | +| `timeoutMs` | the longest wait for one attempt, in milliseconds | 8000 | +| `maxRetries` | the most retries after the first attempt | 2 | +| `cacheTtlMs` | how long the client keeps a result, in milliseconds | 0, which turns the cache off | + +A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` or `cacheTtlMs`. + +### Lookup result + +| Field | Value | +|---|---| +| `postcode` | the postcode that the caller asked for, as the core parsed it | +| `valid` | true when the gateway knows the postcode | +| `status` | `valid`, `invalid`, `not_found`, `restricted`, or null | +| `levelRequested` | the lookup level that the caller asked for | +| `levelReceived` | the lookup level of the data in the response | +| `administrativeAddress` | the state name, the LGA name, the locality name and the zone, or null | +| `recentHouseAddress` | text, or null | +| `buildingUseStatus` | text, or null | + +- The client never takes `postcode` from the response. The gateway echoes the caller's text there, also for text that is not a postcode. +- `status` is null when the response has no `status`, or a value that this table does not list. +- `levelReceived` comes from the fields of the response. It is 5 when `point_geometry` is present, 4 for `other_building_info`, 3 for `building_use_status`, and 2 for either address field. Otherwise it is 1. +- `recentHouseAddress` is the text in `recent_house_address.recent`. +- A response with `valid: false` is a result, not an error. + +### Reverse result + +| Field | Value | +|---|---| +| `found` | true when the gateway found a unit or an area within the radius | +| `radiusM` | the radius that the gateway applied, in metres | +| `unit` | the nearest unit, or null | +| `area` | the area as a postcode, or null | +| `district` | the district as a postcode, or null | +| `state` | the state code, or null | + +A unit has these fields: + +| Field | Value | +|---|---| +| `postcode` | the unit's postcode, as the core parsed it | +| `distanceM` | the distance from the coordinate, in metres | +| `confidence` | `high`, `medium` or `low` | +| `stateName`, `lgaName`, `localityName` | text, or null | +| `address` | text, or null | + +- `found` can be true while `unit` is null. The gateway then found an area, but no unit within the radius. +- `confidence` is `low` for a value that this table does not list. +- The gateway sends the coordinate as `[lng, lat]`. The client does not return it. + +### Autocomplete result + +| Field | Value | +|---|---| +| `segment` | the segment that holds the last character that the user typed | +| `suggestions` | a list of suggestions, in the gateway's order | + +A suggestion has these fields: + +| Field | Value | +|---|---| +| `code` | the value of the active segment, as the gateway sent it | +| `label` | text, or null | +| `postcode` | the partial postcode of the typed segments before the active one, followed by `code`, or null | + +The gateway sends the value of one segment only. For the text `FC01Z`, the suggestion `Z99` gives the postcode `FC-01-Z99`. The client parses that text with partial postcodes allowed. When it does not parse, `postcode` is null. + +## Requests + +- The client sends `X-API-Key` only when the caller gave a key. +- `lookup` parses the code with the core first. A code that does not parse raises `invalid_input`, and no request goes out. Partial postcodes do not pass. The request sends the canonical form and the level: `code=FC-01-Z99-ZZ-01&level=1`. +- `reverse` sends `lat` and `lng`, and `max_distance_m` when the caller gave a radius. +- `autocomplete` normalises its text with the core. Empty text, text of more than 11 characters, and text with a character other than A to Z and 0 to 9 raise `invalid_input`, and no request goes out. The gateway does not answer an empty `q`. The request sends the normalised text: `q=FC01Z`. + +## Errors + +`PostcodeError` has these fields: + +| Field | Value | +|---|---| +| `code` | one of the error codes below | +| `status` | the HTTP status, or null when no response arrived | +| `apiCode` | the `error.code` of the response body, or null | +| `retryAfterMs` | the wait that `Retry-After` asked for, in milliseconds, or null | + +The error code comes from the first row that matches. + +| Condition | Error code | Retried | +|---|---|---| +| the client's own check of the input fails | `invalid_input` | no request goes out | +| status 401 | `unauthorized` | no | +| status 402 | `insufficient_credits` | no | +| status 403, with the API code `origin_not_allowed` | `origin_not_allowed` | no | +| status 403, with any other API code | `forbidden` | no | +| status 429 | `rate_limited` | see below | +| status 502, 503 or 504 | `server_error` | yes | +| any other status from 400 to 499 | `invalid_input` | no | +| any other status that is not 200 | `server_error` | no | +| status 200, with a body that is not a JSON object with a `data` field | `server_error` | no | +| no response, because the connection failed | `network_error` | yes | +| no response within `timeoutMs` | `timeout` | yes, except for `autocomplete` | + +- `apiCode` is set when the body is a JSON object whose `error.code` is text. The client never raises an error for an unexpected error body. +- A cancelled call ends with the platform's own cancellation error, not with `PostcodeError` (API-11). +- No error message holds the API key (ERR-3). + +## Retries + +- A client makes at most `1 + maxRetries` attempts for one call. +- Before retry `n`, the first being 1, the client waits `500 x 2^(n - 1)` milliseconds, plus a random wait from 0 to 250 milliseconds. The first retry waits 500 to 750 ms, and the second waits 1000 to 1250 ms. +- A 429 with `Retry-After` of 10 seconds or less: the client waits that long, with no random part, and tries again. `Retry-After` can hold seconds or an HTTP date. +- A 429 with a longer `Retry-After`: the client raises `rate_limited` at once, and `retryAfterMs` holds the wait. A caller can then tell a user when to try again. +- A 429 with no `Retry-After`: the client raises `rate_limited` at once. The gateway counts requests in each clock minute, so a retry after half a second would fail too. +- The client does not retry a timeout of `autocomplete`. The next keystroke replaces the call. + +## Sharing, queue and cache + +- Two calls of the same method with the same arguments share one request while the first one is in flight. Both callers get the same result or the same error. +- A client sends at most 4 requests at a time. Further calls wait in a queue, in the order of the calls. +- With `cacheTtlMs` above 0, the client keeps each result for that long. A call with the same method and the same arguments then gets the kept result, and no request goes out. Errors stay out of the cache. `clearCache()` removes every kept result. + +## Rules that no scenario tests + +Each client tests these rules itself. + +- The programmer errors in the client options. +- Cancellation (API-11), because each language cancels in its own way. +- That no error message holds the API key. +- The random part of each wait, beyond its bounds. +- A `Retry-After` that holds an HTTP date. A scenario cannot hold a date that is a few seconds ahead. diff --git a/grammar.md b/grammar.md index 8dabf3a..a87b6c9 100644 --- a/grammar.md +++ b/grammar.md @@ -1,6 +1,6 @@ # Postcode grammar -Spec version 0.1.0. This file defines how every Gatepost SDK reads and writes Nigeria's digital postcodes. The values live in `data/`. The vectors in `vectors/` test every rule, except the rules in the list below. If this file and a vector disagree, this file wins. Such a disagreement is a bug in the spec. +Spec version 0.2.0. This file defines how every Gatepost SDK reads and writes Nigeria's digital postcodes. The values live in `data/`. The vectors in `vectors/` test every rule, except the rules in the list below. If this file and a vector disagree, this file wins. Such a disagreement is a bug in the spec. No shared vector tests these rules. Each SDK tests them itself. @@ -95,7 +95,7 @@ Parse the corrected code with the same options. If it parses, the suggestion is ## Interface -An SDK exposes the symbols in this table and the types that they need. A language file gives the idiomatic form of each name and lists the additions that its language needs, such as an options type. The function names are the names in the vector files. This section covers the core. The interface of the client comes later, with the completed OpenAPI file. +An SDK exposes the symbols in this table and the types that they need. A language file gives the idiomatic form of each name and lists the additions that its language needs, such as an options type. The function names are the names in the vector files. This section covers the core. `client.md` covers the client. | Symbol | Takes | Returns | Rules | |---|---|---|---| diff --git a/scripts/contract_files.py b/scripts/contract_files.py new file mode 100644 index 0000000..f90830c --- /dev/null +++ b/scripts/contract_files.py @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: 2026 The Gatepost authors +# SPDX-License-Identifier: Apache-2.0 +"""Read client.md and, from the next change on, the contract scenarios.""" + +from __future__ import annotations + +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +CLIENT_DOC = ROOT / "client.md" + + +def error_codes(path: Path = CLIENT_DOC) -> list[str]: + """Return the error codes in the table of the Errors section of client.md, in order.""" + section = path.read_text(encoding="utf-8").partition("\n## Errors\n")[2].partition("\n## ")[0] + codes = re.findall(r"^\| [^|]+ \| `([a-z_]+)` \|", section, flags=re.MULTILINE) + return list(dict.fromkeys(codes)) diff --git a/scripts/tests/test_contract_files.py b/scripts/tests/test_contract_files.py new file mode 100644 index 0000000..c0af5d6 --- /dev/null +++ b/scripts/tests/test_contract_files.py @@ -0,0 +1,45 @@ +# SPDX-FileCopyrightText: 2026 The Gatepost authors +# SPDX-License-Identifier: Apache-2.0 +"""Tests for contract_files: client.md lists the error codes, and the standards point to it.""" + +import tempfile +import unittest +from pathlib import Path + +import contract_files + +CODES = contract_files.error_codes() +STANDARDS = contract_files.ROOT / "standards" / "CODING_STANDARDS.md" + + +class ClientDocTest(unittest.TestCase): + def test_lists_the_error_codes_of_the_design_in_the_order_of_the_map(self) -> None: + self.assertEqual( + CODES, + [ + "invalid_input", + "unauthorized", + "insufficient_credits", + "origin_not_allowed", + "forbidden", + "rate_limited", + "server_error", + "network_error", + "timeout", + ], + ) + + def test_the_standards_send_the_status_map_to_client_md(self) -> None: + text = STANDARDS.read_text(encoding="utf-8") + err_1 = next(line for line in text.splitlines() if "**ERR-1" in line) + self.assertIn("`spec/client.md`", err_1) + + def test_reads_the_rows_of_the_error_table_and_no_other_table(self) -> None: + text = ( + "# Client\n\n## Errors\n\n| a | `x_code` | no |\n\n" + "## Retries\n\n| b | `y_code` | yes |\n" + ) + with tempfile.TemporaryDirectory() as folder: + path = Path(folder) / "client.md" + path.write_text(text, encoding="utf-8") + self.assertEqual(contract_files.error_codes(path), ["x_code"]) diff --git a/standards/CODING_STANDARDS.md b/standards/CODING_STANDARDS.md index cf14654..dd933bd 100644 --- a/standards/CODING_STANDARDS.md +++ b/standards/CODING_STANDARDS.md @@ -1,6 +1,6 @@ # Gatepost coding standards -Version 1.0, 1 Oct 2026. +Version 1.1, 3 Oct 2026. These standards apply to every Gatepost repo, in every language. Each repo also follows its language file in `standards/languages/`. Together, the two files are the standard for that repo. @@ -11,7 +11,7 @@ These standards apply to every Gatepost repo, in every language. Each repo also - A **SHOULD** rule can be skipped only with a written reason in the pull request. - A rule marked **(tool)** is enforced by CI. Reviewers do not check it by hand. - A language file can change a SHOULD rule for its language. It cannot change a MUST rule. -- `spec/grammar.md` (with its Interface section), `spec/data/`, `spec/vectors/` and the contract scenarios in `spec/contract/` rank above this document. `spec/grammar.md` wins over a vector. This document ranks above the language files. If two of them disagree, the higher one wins. Then fix the lower one. +- `spec/grammar.md` (with its Interface section), `spec/client.md`, `spec/data/`, `spec/vectors/`, `spec/fixtures/` and the contract scenarios in `spec/contract/` rank above this document. `spec/grammar.md` wins over a vector, and `spec/client.md` wins over a scenario. This document ranks above the language files. If two of them disagree, the higher one wins. Then fix the lower one. - To change a rule, open a pull request on the `spec` repo. Increase the version at the top of this file. ## Principles @@ -28,7 +28,7 @@ The rules come from these seven principles. If no rule covers a case, apply the ## Public interface -- **API-1 MUST.** Every public symbol maps to a concept in the Interface section of `spec/grammar.md`. A language file lists the idiomatic additions that its language needs, such as options types, async variants and callback types. To add, rename or remove a concept, change the spec first. To add an idiomatic addition, change the language file first. +- **API-1 MUST.** Every public symbol maps to a concept in the Interface section of `spec/grammar.md` or `spec/client.md`. A dev tool that no SDK user imports, such as the mock server, is outside this rule. A language file lists the idiomatic additions that its language needs, such as options types, async variants and callback types. To add, rename or remove a concept, change the spec first. To add an idiomatic addition, change the language file first. - **API-2 MUST.** Names and call shapes follow the cross-language map below. - **API-3 MUST.** Parse, don't validate. A function that reads a postcode returns a typed result, not a boolean. A boolean helper can exist only as a thin wrapper over `parse`. - **API-4 MUST.** Expected failures are values. `parse` returns its errors. Only I/O failures, API failures and programmer errors raise or throw. @@ -137,7 +137,7 @@ The command exits with 0 when all files pass, with 1 when a rule is broken, and ## Errors and logging -- **ERR-1 MUST.** Each HTTP status maps to exactly one error code. The spec does not define the map yet. It will define the map with the contract scenarios, before the first client release. +- **ERR-1 MUST.** Each HTTP status maps to one error code, through the map in `spec/client.md`. Where the map gives one status two error codes, the API's own error code chooses between them, for example 403 with `origin_not_allowed`. - **ERR-2 MUST.** A library does not log. It returns or raises errors. A caller can attach a debug hook. - **ERR-3 MUST.** No error, log or exception message contains an API key, a NIN, an email address, or a full postcode next to personal data. Use `redact()` for postcodes. - **ERR-4 MUST.** An error message says what happened and what to do, in plain English. Example: "The postcode has 10 characters. A postcode has 11 characters." diff --git a/standards/languages/typescript.md b/standards/languages/typescript.md index acf58ac..1be81a8 100644 --- a/standards/languages/typescript.md +++ b/standards/languages/typescript.md @@ -1,6 +1,6 @@ # TypeScript standards -Version 1.0, 1 Oct 2026. +Version 1.1, 3 Oct 2026. This file and `standards/CODING_STANDARDS.md` together are the standard for TypeScript code. They apply to every package and app in the `js` repo: core, client, field, React wrapper, docs site and demos. Later TypeScript work follows them too, such as the MCP server, the n8n node and the spreadsheet add-ins. @@ -31,7 +31,7 @@ This file and `standards/CODING_STANDARDS.md` together are the standard for Type | publint and `@arethetypeswrong/cli` | package checks | every published package | | API Extractor | API report | `etc/.api.md`, checked in CI | | Changesets | versions and changelogs | one change file per user-visible change | -| commitlint | commit messages | `@commitlint/config-conventional`, `scope-enum` with the scopes `core`, `client`, `field`, `react`, `repo`, `deps` and `release`, `scope-empty` set to `never` so each commit needs a scope (GIT-1), and `header-max-length` set to 72 characters (TELL-18) | +| commitlint | commit messages | `@commitlint/config-conventional`, `scope-enum` with the scopes `core`, `client`, `field`, `react`, `mock`, `repo`, `deps` and `release`, `scope-empty` set to `never` so each commit needs a scope (GIT-1), and `header-max-length` set to 72 characters (TELL-18) | | REUSE | licence headers | `reuse lint` | | gitleaks | secret scan | each pull request and each push to `main` | | `spec/scripts/check-tells` | agent tells | line length 100, file names, TODO form, debug calls, emojis and non-ASCII | @@ -147,6 +147,14 @@ packages// CHANGELOG.md generated by Changesets ``` +## The mock server + +`packages/mock-server` is a private package that never goes to npm. It is a dev tool, not an SDK. These rules do not apply to it: API-1, CS-2, DOC-5, PERF-1, and the package checks with publint, attw, size-limit and API Extractor. It has no change files. The version in its `package.json` is the tag of its container image. + +- Node runs its TypeScript source directly, so it has no build. The source uses only syntax that Node can strip, which `erasableSyntaxOnly` checks, and each import names a `.ts` file. +- It can use Node's modules. It does not call `console`, and it writes to `process.stdout` only from its command. +- Its coverage floor is 90 % of branches, the floor for a client (T-7). + ## Publishing - Changesets opens the release pull request. From fdd6028dda1a02551d860dabc8225a79c41eb2b6 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:37:32 +0100 Subject: [PATCH 02/13] fix: complete the client contract for input and unknown values A review found three gaps that would let the clients differ: an unknown lookup status became null, no rule covered a bad level, coordinate or distance, and the contract did not say that a client ignores unknown fields in a response. Signed-off-by: MobileMage --- client.md | 28 ++++++++++++++++++---------- scripts/tests/test_contract_files.py | 28 ++++++++++++++++++++++++++++ standards/languages/typescript.md | 2 +- 3 files changed, 47 insertions(+), 11 deletions(-) diff --git a/client.md b/client.md index 9c1686e..afccfc2 100644 --- a/client.md +++ b/client.md @@ -28,9 +28,9 @@ Each call that sends a request is asynchronous, and the caller can cancel it (AP | `transport` | the HTTP transport, in the language's standard form | the platform's own | | `timeoutMs` | the longest wait for one attempt, in milliseconds | 8000 | | `maxRetries` | the most retries after the first attempt | 2 | -| `cacheTtlMs` | how long the client keeps a result, in milliseconds | 0, which turns the cache off | +| `cacheTtlMs` | how long the client keeps a result, in milliseconds | 0, which disables the cache | -A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` or `cacheTtlMs`. +A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` or `cacheTtlMs`. The default of 8000 is open: some `autocomplete` calls took 10 to 14 seconds, and a timed run must decide it. ### Lookup result @@ -38,15 +38,15 @@ A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` |---|---| | `postcode` | the postcode that the caller asked for, as the core parsed it | | `valid` | true when the gateway knows the postcode | -| `status` | `valid`, `invalid`, `not_found`, `restricted`, or null | +| `status` | text, or null. The known values are `valid`, `invalid`, `not_found` and `restricted` | | `levelRequested` | the lookup level that the caller asked for | | `levelReceived` | the lookup level of the data in the response | -| `administrativeAddress` | the state name, the LGA name, the locality name and the zone, or null | +| `administrativeAddress` | an object with `stateName`, `lgaName`, `localityName` and `zone`, or null | | `recentHouseAddress` | text, or null | | `buildingUseStatus` | text, or null | - The client never takes `postcode` from the response. The gateway echoes the caller's text there, also for text that is not a postcode. -- `status` is null when the response has no `status`, or a value that this table does not list. +- `status` is null when the response has no `status`. The client returns any other value of `status` as it is, also when this table does not list it, so that a new gateway status is not lost. - `levelReceived` comes from the fields of the response. It is 5 when `point_geometry` is present, 4 for `other_building_info`, 3 for `building_use_status`, and 2 for either address field. Otherwise it is 1. - `recentHouseAddress` is the text in `recent_house_address.recent`. - A response with `valid: false` is a result, not an error. @@ -93,12 +93,20 @@ A suggestion has these fields: The gateway sends the value of one segment only. For the text `FC01Z`, the suggestion `Z99` gives the postcode `FC-01-Z99`. The client parses that text with partial postcodes allowed. When it does not parse, `postcode` is null. +A client ignores a field that the tables above do not list, in a lookup, a reverse or an autocomplete response. Such a field never changes the result and never raises an error. The scenario `unknown-fields` tests this rule for each of the three calls. + ## Requests - The client sends `X-API-Key` only when the caller gave a key. -- `lookup` parses the code with the core first. A code that does not parse raises `invalid_input`, and no request goes out. Partial postcodes do not pass. The request sends the canonical form and the level: `code=FC-01-Z99-ZZ-01&level=1`. +- `lookup` parses the code with the core first. A code that does not parse raises `invalid_input`, and the client sends no request. Partial postcodes do not pass. The request sends the canonical form and the level: `code=FC-01-Z99-ZZ-01&level=1`. - `reverse` sends `lat` and `lng`, and `max_distance_m` when the caller gave a radius. -- `autocomplete` normalises its text with the core. Empty text, text of more than 11 characters, and text with a character other than A to Z and 0 to 9 raise `invalid_input`, and no request goes out. The gateway does not answer an empty `q`. The request sends the normalised text: `q=FC01Z`. +- Each of these raises `invalid_input`, and the client sends no request: + - A `level` that is not a whole number from 1 to 5. + - A `lat` that is not finite or is outside -90 to 90. + - A `lng` that is not finite or is outside -180 to 180. + - A `maxDistanceM` below 0 or above 250. The OpenAPI file gives the range 0 to 250. +- The client writes each number in the query as a plain decimal, with no exponent: `lat=9.0`, not `lat=9e0`. Languages print a number in different forms, such as `1e-7`. One plain form gives every client the same query for the same input, and the mock server can match it. +- `autocomplete` normalises its text with the core. Empty text, text of more than 11 characters, and text with a character other than A to Z and 0 to 9 raise `invalid_input`, and the client sends no request. The gateway does not answer an empty `q`. The request sends the normalised text: `q=FC01Z`. ## Errors @@ -115,7 +123,7 @@ The error code comes from the first row that matches. | Condition | Error code | Retried | |---|---|---| -| the client's own check of the input fails | `invalid_input` | no request goes out | +| the client's own check of the input fails | `invalid_input` | no request is sent | | status 401 | `unauthorized` | no | | status 402 | `insufficient_credits` | no | | status 403, with the API code `origin_not_allowed` | `origin_not_allowed` | no | @@ -136,7 +144,7 @@ The error code comes from the first row that matches. - A client makes at most `1 + maxRetries` attempts for one call. - Before retry `n`, the first being 1, the client waits `500 x 2^(n - 1)` milliseconds, plus a random wait from 0 to 250 milliseconds. The first retry waits 500 to 750 ms, and the second waits 1000 to 1250 ms. -- A 429 with `Retry-After` of 10 seconds or less: the client waits that long, with no random part, and tries again. `Retry-After` can hold seconds or an HTTP date. +- A 429 with `Retry-After` of 10 seconds or less: the client waits that long, with no random part, and retries. `Retry-After` can hold seconds or an HTTP date. - A 429 with a longer `Retry-After`: the client raises `rate_limited` at once, and `retryAfterMs` holds the wait. A caller can then tell a user when to try again. - A 429 with no `Retry-After`: the client raises `rate_limited` at once. The gateway counts requests in each clock minute, so a retry after half a second would fail too. - The client does not retry a timeout of `autocomplete`. The next keystroke replaces the call. @@ -145,7 +153,7 @@ The error code comes from the first row that matches. - Two calls of the same method with the same arguments share one request while the first one is in flight. Both callers get the same result or the same error. - A client sends at most 4 requests at a time. Further calls wait in a queue, in the order of the calls. -- With `cacheTtlMs` above 0, the client keeps each result for that long. A call with the same method and the same arguments then gets the kept result, and no request goes out. Errors stay out of the cache. `clearCache()` removes every kept result. +- With `cacheTtlMs` above 0, the client keeps each result for that long. A call with the same method and the same arguments then gets the kept result, and the client sends no request. Errors stay out of the cache. `clearCache()` removes every kept result. ## Rules that no scenario tests diff --git a/scripts/tests/test_contract_files.py b/scripts/tests/test_contract_files.py index c0af5d6..eb8abd6 100644 --- a/scripts/tests/test_contract_files.py +++ b/scripts/tests/test_contract_files.py @@ -10,6 +10,7 @@ CODES = contract_files.error_codes() STANDARDS = contract_files.ROOT / "standards" / "CODING_STANDARDS.md" +CLIENT = contract_files.CLIENT_DOC.read_text(encoding="utf-8") class ClientDocTest(unittest.TestCase): @@ -43,3 +44,30 @@ def test_reads_the_rows_of_the_error_table_and_no_other_table(self) -> None: path = Path(folder) / "client.md" path.write_text(text, encoding="utf-8") self.assertEqual(contract_files.error_codes(path), ["x_code"]) + + +class ClientInputAndUnknownValuesTest(unittest.TestCase): + def test_keeps_an_unknown_lookup_status_as_text(self) -> None: + self.assertNotIn("no `status`, or a value that this table does not list", CLIENT) + self.assertIn("any other value of `status` as it is", CLIENT) + + def test_rejects_a_bad_level_coordinate_or_distance_before_a_request(self) -> None: + for rule in ( + "A `level` that is not a whole number from 1 to 5", + "A `lat` that is not finite or is outside -90 to 90", + "A `lng` that is not finite or is outside -180 to 180", + "A `maxDistanceM` below 0 or above 250", + ): + self.assertIn(rule, CLIENT) + + def test_writes_a_number_in_the_query_as_a_plain_decimal(self) -> None: + self.assertIn("plain decimal, with no exponent", CLIENT) + + def test_ignores_unknown_fields_and_names_the_scenario(self) -> None: + self.assertIn("A client ignores a field that the tables above do not list", CLIENT) + self.assertIn("`unknown-fields`", CLIENT) + + def test_leaves_the_core_only_rule_out_of_the_mock_server_exceptions(self) -> None: + path = contract_files.ROOT / "standards" / "languages" / "typescript.md" + section = path.read_text(encoding="utf-8").partition("## The mock server")[2] + self.assertNotIn("CS-2", section.partition("## Publishing")[0]) diff --git a/standards/languages/typescript.md b/standards/languages/typescript.md index 1be81a8..49b86f3 100644 --- a/standards/languages/typescript.md +++ b/standards/languages/typescript.md @@ -149,7 +149,7 @@ packages// ## The mock server -`packages/mock-server` is a private package that never goes to npm. It is a dev tool, not an SDK. These rules do not apply to it: API-1, CS-2, DOC-5, PERF-1, and the package checks with publint, attw, size-limit and API Extractor. It has no change files. The version in its `package.json` is the tag of its container image. +`packages/mock-server` is a private package that never goes to npm. It is a dev tool, not an SDK. These rules do not apply to it: API-1, DOC-5, PERF-1, and the package checks with publint, attw, size-limit and API Extractor. It has no change files. The version in its `package.json` is the tag of its container image. - Node runs its TypeScript source directly, so it has no build. The source uses only syntax that Node can strip, which `erasableSyntaxOnly` checks, and each import names a `.ts` file. - It can use Node's modules. It does not call `console`, and it writes to `process.stdout` only from its command. From aea1d8627237cbddab32509eebc7f479d15a3b28 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:39:24 +0100 Subject: [PATCH 03/13] feat: add the completed OpenAPI file of the gateway NIPOST's OpenAPI file gives no error responses, no nearby body and no rate-limit headers. The completed file adds them, and marks each response as observed, documented or assumed, so a reader knows what a call has shown. The checks reject a path that NIPOST does not document and a response with no mark. Signed-off-by: MobileMage --- Makefile | 27 +- openapi/gateway.completed.yaml | 447 ++++++++++++++++++++++++++++ scripts/gateway_files.py | 100 +++++++ scripts/tests/test_gateway_files.py | 70 +++++ 4 files changed, 640 insertions(+), 4 deletions(-) create mode 100644 openapi/gateway.completed.yaml create mode 100644 scripts/gateway_files.py create mode 100644 scripts/tests/test_gateway_files.py diff --git a/Makefile b/Makefile index 5130d37..b721c8c 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ # SPDX-FileCopyrightText: 2026 The Gatepost authors # SPDX-License-Identifier: Apache-2.0 -.PHONY: check test vectors tells lint types licences actions +.PHONY: check test vectors contract tells lint types licences actions # Each tool has an exact version. A new release can add rules and turn CI red on a change that # has nothing to do with it. Renovate opens the pull requests that move these versions. @@ -14,18 +14,36 @@ REUSE_VERSION := 6.2.0 CHARSET_NORMALIZER_VERSION := 3.5.2 # renovate: datasource=pypi depName=zizmor ZIZMOR_VERSION := 1.30.1 +# renovate: datasource=pypi depName=pyyaml +PYYAML_VERSION := 6.0.3 +# renovate: datasource=pypi depName=jsonschema +JSONSCHEMA_VERSION := 4.26.0 +# renovate: datasource=pypi depName=types-pyyaml +TYPES_PYYAML_VERSION := 6.0.12.20260906 +# renovate: datasource=pypi depName=types-jsonschema +TYPES_JSONSCHEMA_VERSION := 4.26.0.20260518 +# renovate: datasource=pypi depName=openapi-spec-validator +OPENAPI_SPEC_VALIDATOR_VERSION := 0.9.0 # scripts/check-tells has no .py suffix, so the tools need its name. SCRIPTS := scripts scripts/check-tells -check: test vectors tells lint types licences actions +# The checks of the OpenAPI file read YAML and JSON Schema. uv adds the two packages to the +# Python on PATH, so each Python of the CI matrix runs the tests. +PYTHON := uv run --no-project --python-preference only-system \ + --with pyyaml==$(PYYAML_VERSION) --with jsonschema==$(JSONSCHEMA_VERSION) python3 + +check: test vectors contract tells lint types licences actions test: - python3 -m unittest discover -s scripts/tests -t scripts + $(PYTHON) -m unittest discover -s scripts/tests -t scripts vectors: python3 scripts/build_vectors.py --check +contract: + uvx openapi-spec-validator@$(OPENAPI_SPEC_VALIDATOR_VERSION) openapi/gateway.completed.yaml + tells: python3 scripts/check-tells @@ -34,7 +52,8 @@ lint: uvx ruff@$(RUFF_VERSION) format --check $(SCRIPTS) types: - uvx mypy@$(MYPY_VERSION) + uvx --with types-pyyaml==$(TYPES_PYYAML_VERSION) \ + --with types-jsonschema==$(TYPES_JSONSCHEMA_VERSION) mypy@$(MYPY_VERSION) # reuse needs an encoding detector, and uvx installs reuse without one. licences: diff --git a/openapi/gateway.completed.yaml b/openapi/gateway.completed.yaml new file mode 100644 index 0000000..404b996 --- /dev/null +++ b/openapi/gateway.completed.yaml @@ -0,0 +1,447 @@ +# SPDX-FileCopyrightText: 2026 The Gatepost authors +# SPDX-License-Identifier: Apache-2.0 +openapi: 3.1.0 +info: + title: NIPOST postcode gateway, completed by Gatepost + version: 0.1.0-gatepost.1 + summary: Unofficial. Not made or endorsed by NIPOST. + description: >- + Gatepost wrote this description of NIPOST's public postcode gateway. It adds the responses, + error codes and headers that NIPOST's OpenAPI file 0.1.0 leaves out. Each response carries + x-gatepost-evidence. "observed" means that Gatepost saw the shape in its own calls on + 2 Oct 2026. "documented" means that NIPOST's docs describe it, but no call showed it. + "assumed" means that neither source shows it, and Gatepost's mock server chose it. + Gatepost's calls used a key with lookup level 1 only, so every lookup body above level 1 + is documented, not observed. + license: + name: Apache-2.0 + identifier: Apache-2.0 +servers: + - url: https://api.postcode.gov.ng + description: NIPOST's gateway +security: + - ApiKey: [] +paths: + /healthz: + get: + operationId: health + summary: Tell whether the gateway runs + security: [] + responses: + "200": + description: The gateway runs. + x-gatepost-evidence: documented + /v1/lookup: + get: + operationId: lookup + summary: Look up one postcode at a lookup level + description: >- + A client sends a whole number from 1 to 5. The levels add fields. Level 1 gives + validity. Level 2 adds the administrative address + and the recent house address. Level 3 adds the building use status. Level 4 adds other + building information, and level 5 adds point geometry. A key holds at most level 3. + A request above the key's level gets 403 level_not_granted. NIPOST's docs say that the + gateway lowers the level instead, but no call showed that. + parameters: + - name: code + in: query + required: true + description: The postcode, in any form. The gateway ignores spaces, hyphens and case. + schema: { type: string } + - name: level + in: query + required: false + schema: { type: integer, minimum: 1, maximum: 5, default: 1 } + responses: + "200": + description: >- + The lookup result. An unknown or malformed postcode also gets 200, with valid set + to false. + x-gatepost-evidence: observed + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/LookupEnvelope" } + "400": { $ref: "#/components/responses/BadRequest" } + "401": { $ref: "#/components/responses/Unauthorized" } + "402": { $ref: "#/components/responses/PaymentRequired" } + "403": { $ref: "#/components/responses/Forbidden" } + "429": { $ref: "#/components/responses/TooManyRequests" } + /v1/search/reverse: + get: + operationId: reverse + summary: Find the nearest unit to a coordinate + parameters: + - name: lat + in: query + required: true + description: The latitude in degrees. A client rejects a value outside this range. + schema: { type: number, minimum: -90, maximum: 90 } + - name: lng + in: query + required: true + description: The longitude in degrees. A client rejects a value outside this range. + schema: { type: number, minimum: -180, maximum: 180 } + - name: max_distance_m + in: query + required: false + description: >- + The search radius in metres. The gateway lowers a radius above 250 m to 250 m. + A client rejects a value outside 0 to 250 and sends no request. + schema: { type: number, minimum: 0, maximum: 250, default: 25 } + responses: + "200": + description: The nearest unit or area, or found set to false. + x-gatepost-evidence: observed + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/ReverseEnvelope" } + "400": { $ref: "#/components/responses/BadRequest" } + "401": { $ref: "#/components/responses/Unauthorized" } + "403": { $ref: "#/components/responses/Forbidden" } + "429": { $ref: "#/components/responses/TooManyRequests" } + /v1/search/nearby: + get: + operationId: nearby + summary: List the units within a radius + parameters: + - name: lat + in: query + required: true + description: The latitude in degrees. A client rejects a value outside this range. + schema: { type: number, minimum: -90, maximum: 90 } + - name: lng + in: query + required: true + description: The longitude in degrees. A client rejects a value outside this range. + schema: { type: number, minimum: -180, maximum: 180 } + - name: radius + in: query + required: false + schema: { type: number, default: 300 } + responses: + "200": + description: >- + A list of units. The only call so far returned an empty list, so the fields of a + unit are not known. + x-gatepost-evidence: observed + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/NearbyEnvelope" } + "400": { $ref: "#/components/responses/BadRequest" } + "401": { $ref: "#/components/responses/Unauthorized" } + "403": { $ref: "#/components/responses/Forbidden" } + "429": { $ref: "#/components/responses/TooManyRequests" } + /v1/search/autocomplete: + get: + operationId: autocomplete + summary: Suggest values for the segment that a user types + description: >- + The gateway did not answer an empty q within 20 seconds. A client must not send one. + parameters: + - name: q + in: query + required: true + description: The start of a postcode, for example FC01Z. + schema: { type: string, minLength: 1 } + responses: + "200": + description: The values of the active segment. + x-gatepost-evidence: observed + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/AutocompleteEnvelope" } + "400": { $ref: "#/components/responses/BadRequest" } + "401": { $ref: "#/components/responses/Unauthorized" } + "403": { $ref: "#/components/responses/Forbidden" } + "429": { $ref: "#/components/responses/TooManyRequests" } + /v1/assembly/assemble: + post: + operationId: assemble + summary: Build a postcode from its segments + requestBody: + required: true + content: + application/json: + schema: { $ref: "#/components/schemas/Segments" } + responses: + "200": + description: The postcode in three forms. Letters become upper case, numbers get a zero. + x-gatepost-evidence: observed + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/AssembleEnvelope" } + "400": { $ref: "#/components/responses/BadRequest" } + "401": { $ref: "#/components/responses/Unauthorized" } + "403": { $ref: "#/components/responses/Forbidden" } + "429": { $ref: "#/components/responses/TooManyRequests" } + /v1/assembly/disassemble: + get: + operationId: disassemble + summary: Split a postcode into its segments + parameters: + - name: code + in: query + required: true + schema: { type: string } + responses: + "200": + description: The five segments. + x-gatepost-evidence: observed + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/DisassembleEnvelope" } + "400": { $ref: "#/components/responses/BadRequest" } + "401": { $ref: "#/components/responses/Unauthorized" } + "403": { $ref: "#/components/responses/Forbidden" } + "429": { $ref: "#/components/responses/TooManyRequests" } +components: + securitySchemes: + ApiKey: + type: apiKey + in: header + name: X-API-Key + description: >- + A secret key starts with nipost_test_ or nipost_live_. A publishable key starts with + nipost_pk_test_ or nipost_pk_live_, and a browser can use it only from the origins on + its list. + headers: + RateLimitLimit: + description: The requests that one key can make in each clock minute. + schema: { type: integer, examples: [600] } + RateLimitRemaining: + description: The requests that the key has left in this clock minute. + schema: { type: integer, minimum: 0 } + RetryAfter: + description: The seconds to wait before the next request. + schema: { type: integer, minimum: 0 } + responses: + BadRequest: + description: The request has a missing or malformed parameter. + x-gatepost-evidence: assumed + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } + Unauthorized: + description: >- + No key (auth_required, observed), or a key that the gateway does not know + (invalid_api_key, observed on 1 Oct 2026). Neither response has rate-limit headers. + x-gatepost-evidence: observed + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } + PaymentRequired: + description: The organisation has no credits for a lookup above level 1. + x-gatepost-evidence: documented + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } + Forbidden: + description: >- + A lookup above the key's level (level_not_granted, observed), a key without the scope + (code assumed), or a publishable key from an origin that is not on its list + (origin_not_allowed, documented). + x-gatepost-evidence: observed + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } + TooManyRequests: + description: >- + The key used up its requests for this clock minute. NIPOST's docs give the status, but + no call reached it, so the error code and Retry-After are assumed. + x-gatepost-evidence: documented + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + Retry-After: { $ref: "#/components/headers/RetryAfter" } + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } + schemas: + ErrorEnvelope: + type: object + required: [error] + properties: + error: + type: object + required: [code, message] + properties: + code: { type: string, examples: [auth_required, level_not_granted] } + message: { type: string } + Segments: + type: object + required: [state, lga, district, area, unit] + properties: + state: { type: string, examples: [FC] } + lga: { type: string, examples: ["01"] } + district: { type: string, examples: [Z99] } + area: { type: string, examples: [ZZ] } + unit: { type: string, examples: ["01"] } + LookupEnvelope: + type: object + required: [data] + properties: + data: { $ref: "#/components/schemas/Lookup" } + Lookup: + type: object + required: [postcode, valid] + properties: + postcode: + type: string + description: The caller's text in upper case. It is not always a postcode. + valid: { type: boolean } + status: + type: string + description: >- + Observed at level 1. The widget's lookup also names restricted. NIPOST's example + bodies above level 1 have no status. A client keeps any other value as text. + examples: [valid, invalid, not_found, restricted] + verified: + type: boolean + description: Observed at level 1, and false in every call. NIPOST does not define it. + administrative_address: { $ref: "#/components/schemas/AdministrativeAddress" } + recent_house_address: { $ref: "#/components/schemas/RecentHouseAddress" } + building_use_status: + type: string + description: Level 3 and above. Documented, not observed. + examples: [residential] + other_building_info: + type: object + description: Level 4 and above. NIPOST gives no fields. + point_geometry: + type: object + description: Level 5. NIPOST gives no fields. + AdministrativeAddress: + type: object + description: Level 2 and above. Documented, not observed. + required: [state_name, lga_name, locality_name, zone] + properties: + state_name: { type: string } + lga_name: { type: string } + locality_name: { type: string } + zone: { type: string } + RecentHouseAddress: + type: object + description: Level 2 and above. Documented, not observed. + required: [recent] + properties: + recent: { type: string } + ReverseEnvelope: + type: object + required: [data] + properties: + data: { $ref: "#/components/schemas/Reverse" } + Reverse: + type: object + required: [found, coordinate, radius_m] + properties: + found: { type: boolean } + coordinate: + type: array + description: The request's coordinate as [lng, lat]. + items: { type: number } + minItems: 2 + maxItems: 2 + unit: { $ref: "#/components/schemas/ReverseUnit" } + area: { type: string, examples: [FC-01-Z99-ZZ] } + district: { type: string, examples: [FC-01-Z99] } + state: { type: string, examples: [FC] } + depth: + type: string + description: The most precise segment found. Observed values are area and unit. + message: + type: string + description: Set when found is false. + radius_m: + type: number + description: The radius that the gateway applied. Calls showed whole numbers only. + ReverseUnit: + type: object + description: >- + Absent when found is false, and also when the gateway found an area but no unit within + the radius. + required: [postcode, display, distance_m, confidence] + properties: + postcode: { type: string, examples: [FC-01-Z99-ZZ-01] } + display: { type: string, examples: [FC 01 Z99 ZZ 01] } + distance_m: { type: number } + confidence: + type: string + enum: [high, medium, low] + description: Calls showed low only. NIPOST's file names all three. + state_name: { type: string, description: Level 2 and above. Documented. } + lga_name: { type: string, description: Level 2 and above. Documented. } + locality_name: { type: string, description: Level 2 and above. Documented. } + address: { type: string, description: Level 2 and above. Documented. } + NearbyEnvelope: + type: object + required: [data] + properties: + data: + type: array + description: The units in the radius. No call has returned a unit yet. + AutocompleteEnvelope: + type: object + required: [data] + properties: + data: { $ref: "#/components/schemas/Autocomplete" } + Autocomplete: + type: object + required: [segment, suggestions] + properties: + segment: + type: string + enum: [state, lga, district, area, unit] + description: The segment of the last character in q. Unit is documented only. + suggestions: + type: array + description: In ascending order. One call returned 20 items, which can be a limit. + items: + type: object + required: [code] + properties: + code: + type: string + description: The value of the active segment only, for example 01 for EK01. + label: + type: string + description: NIPOST's file names it, but no suggestion in any call had one. + AssembleEnvelope: + type: object + required: [data] + properties: + data: + type: object + required: [compact, display, postcode] + properties: + compact: { type: string, examples: [FC01Z99ZZ01] } + display: { type: string, examples: [FC 01 Z99 ZZ 01] } + postcode: { type: string, examples: [FC-01-Z99-ZZ-01] } + DisassembleEnvelope: + type: object + required: [data] + properties: + data: { $ref: "#/components/schemas/Segments" } diff --git a/scripts/gateway_files.py b/scripts/gateway_files.py new file mode 100644 index 0000000..1d621ad --- /dev/null +++ b/scripts/gateway_files.py @@ -0,0 +1,100 @@ +# SPDX-FileCopyrightText: 2026 The Gatepost authors +# SPDX-License-Identifier: Apache-2.0 +"""Read the completed OpenAPI file, and check its paths, its evidence marks and its schemas. + +Each check returns a list of problems, one line each. An empty list means that the file passes. +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterator +from pathlib import Path +from typing import Any + +import yaml +from jsonschema import Draft202012Validator +from referencing import Registry, Resource +from referencing.jsonschema import DRAFT202012 + +ROOT = Path(__file__).resolve().parents[1] +OPENAPI = ROOT / "openapi" / "gateway.completed.yaml" +EVIDENCE = ("observed", "documented", "assumed") +# NIPOST documents these gateway paths. SEC-4 lets a client call no other path. +DOCUMENTED_PATHS = ( + "/healthz", + "/v1/lookup", + "/v1/search/reverse", + "/v1/search/nearby", + "/v1/search/autocomplete", + "/v1/assembly/assemble", + "/v1/assembly/disassemble", +) +OPENAPI_URI = "urn:gatepost:openapi" + +SchemaCheck = Callable[[str, object], list[str]] + + +def load_openapi(path: Path = OPENAPI) -> dict[str, Any]: + """Return the completed OpenAPI file as a mapping.""" + document = yaml.safe_load(path.read_text(encoding="utf-8")) + if not isinstance(document, dict): + raise TypeError(f"{path.name} must hold a mapping.") + return document + + +def responses(openapi: dict[str, Any]) -> Iterator[tuple[str, dict[str, Any]]]: + """Yield a label and the response object for each response of each operation. + + A response that refers to `components/responses` yields the component that it names. + """ + shared = openapi.get("components", {}).get("responses", {}) + for path, operations in openapi["paths"].items(): + for method, operation in operations.items(): + for status, response in operation["responses"].items(): + reference = response.get("$ref", "") + target = shared.get(reference.rpartition("/")[2], response) + yield f"{method.upper()} {path} {status}", target + + +def openapi_problems(openapi: dict[str, Any]) -> list[str]: + """Return the problems of the OpenAPI file: extra paths and missing evidence marks.""" + problems = [ + f"The path {path} is not a documented gateway path." + for path in openapi["paths"] + if path not in DOCUMENTED_PATHS + ] + for label, response in responses(openapi): + if response.get("x-gatepost-evidence") not in EVIDENCE: + problems.append(f"{label} needs x-gatepost-evidence: one of {', '.join(EVIDENCE)}.") + return problems + + +def schema_check(openapi: dict[str, Any]) -> SchemaCheck: + """Return a function that checks a value against a schema of the OpenAPI file. + + The function takes a name in `components/schemas` and a value, and returns the problems. + """ + resource = Resource.from_contents(openapi, default_specification=DRAFT202012) + registry: Registry[Any] = Registry().with_resource(OPENAPI_URI, resource) + known = openapi["components"]["schemas"] + + def check(name: str, value: object) -> list[str]: + if name not in known: + return [f"The OpenAPI file has no schema {name}."] + schema = {"$ref": f"{OPENAPI_URI}#/components/schemas/{name}"} + validator = Draft202012Validator(schema, registry=registry) + return [ + f"{error.message} at {error.json_path}" + for error in sorted(validator.iter_errors(value), key=str) + ] + + return check + + +def schemas_in_responses(openapi: dict[str, Any]) -> set[str]: + """Return the names of the schemas that the responses use directly.""" + names = set() + for _, response in responses(openapi): + for media in response.get("content", {}).values(): + names.add(media["schema"]["$ref"].rpartition("/")[2]) + return names diff --git a/scripts/tests/test_gateway_files.py b/scripts/tests/test_gateway_files.py new file mode 100644 index 0000000..06a0cb9 --- /dev/null +++ b/scripts/tests/test_gateway_files.py @@ -0,0 +1,70 @@ +# SPDX-FileCopyrightText: 2026 The Gatepost authors +# SPDX-License-Identifier: Apache-2.0 +"""Tests for gateway_files: the OpenAPI file has the documented paths, evidence and schemas.""" + +import copy +import unittest + +import gateway_files + +OPENAPI = gateway_files.load_openapi() +CHECK = gateway_files.schema_check(OPENAPI) + + +class OpenApiTest(unittest.TestCase): + def test_the_file_has_no_problem(self) -> None: + self.assertEqual(gateway_files.openapi_problems(OPENAPI), []) + + def test_lists_each_documented_gateway_path_and_no_other(self) -> None: + self.assertEqual(sorted(OPENAPI["paths"]), sorted(gateway_files.DOCUMENTED_PATHS)) + + def test_rejects_a_path_that_nipost_does_not_document(self) -> None: + openapi = copy.deepcopy(OPENAPI) + openapi["paths"]["/v1/widget/lookup"] = openapi["paths"]["/v1/lookup"] + self.assertEqual( + gateway_files.openapi_problems(openapi), + ["The path /v1/widget/lookup is not a documented gateway path."], + ) + + def test_rejects_a_response_with_no_evidence_mark(self) -> None: + openapi = copy.deepcopy(OPENAPI) + del openapi["paths"]["/v1/lookup"]["get"]["responses"]["200"]["x-gatepost-evidence"] + [problem] = gateway_files.openapi_problems(openapi) + self.assertTrue(problem.startswith("GET /v1/lookup 200 needs x-gatepost-evidence")) + + def test_reads_the_evidence_of_a_shared_response_from_its_component(self) -> None: + openapi = copy.deepcopy(OPENAPI) + del openapi["components"]["responses"]["Unauthorized"]["x-gatepost-evidence"] + labels = [problem.split(" needs")[0] for problem in gateway_files.openapi_problems(openapi)] + self.assertIn("GET /v1/lookup 401", labels) + self.assertIn("POST /v1/assembly/assemble 401", labels) + + def test_marks_each_lookup_above_level_1_as_not_observed(self) -> None: + schemas = OPENAPI["components"]["schemas"] + for name in ("AdministrativeAddress", "RecentHouseAddress"): + with self.subTest(name): + self.assertIn("Documented, not observed.", schemas[name]["description"]) + building_use = schemas["Lookup"]["properties"]["building_use_status"] + self.assertIn("Documented, not observed.", building_use["description"]) + + +class SchemaCheckTest(unittest.TestCase): + def test_names_the_missing_field_and_its_place(self) -> None: + self.assertEqual( + CHECK("Segments", {"state": "FC"}), + [ + "'area' is a required property at $", + "'district' is a required property at $", + "'lga' is a required property at $", + "'unit' is a required property at $", + ], + ) + + def test_follows_references_between_schemas(self) -> None: + body = {"data": {"postcode": "FC-01-Z99-ZZ-01", "valid": "yes"}} + self.assertEqual( + CHECK("LookupEnvelope", body), ["'yes' is not of type 'boolean' at $.data.valid"] + ) + + def test_rejects_a_schema_name_that_the_file_does_not_have(self) -> None: + self.assertEqual(CHECK("Lookups", {}), ["The OpenAPI file has no schema Lookups."]) From 4b3b6448fcffbc6fd66dcb721fd2172ca9787e3e Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:40:52 +0100 Subject: [PATCH 04/13] feat: add the synthetic fixtures and the mock keys The mock server needs one body for each response that a client must handle, and NIPOST's terms allow only synthetic values. Every postcode here has the district Z99 and the area ZZ, and a test fails on any other full postcode. A test also checks each body against its schema in the OpenAPI file, so the two cannot drift apart without a red build. Signed-off-by: MobileMage --- fixtures/README.md | 43 ++++++++ fixtures/errors/auth-required.json | 13 +++ fixtures/errors/insufficient-credits.json | 13 +++ fixtures/errors/invalid-api-key.json | 13 +++ fixtures/errors/invalid-request.json | 13 +++ fixtures/errors/level-not-granted-1.json | 13 +++ fixtures/errors/level-not-granted-2.json | 13 +++ fixtures/errors/level-not-granted-3.json | 13 +++ fixtures/errors/origin-not-allowed.json | 13 +++ fixtures/errors/rate-limited.json | 13 +++ fixtures/errors/scope-not-granted.json | 13 +++ fixtures/errors/unknown-path.json | 13 +++ fixtures/keys.json | 72 ++++++++++++++ fixtures/lookup/invalid.json | 15 +++ fixtures/lookup/not-found.json | 15 +++ fixtures/lookup/valid-level-1.json | 15 +++ fixtures/lookup/valid-level-2.json | 22 +++++ fixtures/lookup/valid-level-3.json | 23 +++++ fixtures/nearby/empty.json | 10 ++ fixtures/reverse/area.json | 21 ++++ fixtures/reverse/not-found.json | 18 ++++ fixtures/reverse/unit.json | 27 +++++ scripts/gateway_files.py | 85 +++++++++++++++- scripts/tests/test_gateway_files.py | 114 +++++++++++++++++++++- 24 files changed, 621 insertions(+), 2 deletions(-) create mode 100644 fixtures/README.md create mode 100644 fixtures/errors/auth-required.json create mode 100644 fixtures/errors/insufficient-credits.json create mode 100644 fixtures/errors/invalid-api-key.json create mode 100644 fixtures/errors/invalid-request.json create mode 100644 fixtures/errors/level-not-granted-1.json create mode 100644 fixtures/errors/level-not-granted-2.json create mode 100644 fixtures/errors/level-not-granted-3.json create mode 100644 fixtures/errors/origin-not-allowed.json create mode 100644 fixtures/errors/rate-limited.json create mode 100644 fixtures/errors/scope-not-granted.json create mode 100644 fixtures/errors/unknown-path.json create mode 100644 fixtures/keys.json create mode 100644 fixtures/lookup/invalid.json create mode 100644 fixtures/lookup/not-found.json create mode 100644 fixtures/lookup/valid-level-1.json create mode 100644 fixtures/lookup/valid-level-2.json create mode 100644 fixtures/lookup/valid-level-3.json create mode 100644 fixtures/nearby/empty.json create mode 100644 fixtures/reverse/area.json create mode 100644 fixtures/reverse/not-found.json create mode 100644 fixtures/reverse/unit.json diff --git a/fixtures/README.md b/fixtures/README.md new file mode 100644 index 0000000..cb31954 --- /dev/null +++ b/fixtures/README.md @@ -0,0 +1,43 @@ +# Fixtures + +The mock server answers with these files. Each file holds one response of NIPOST's gateway. Every postcode in them is synthetic: the district is `Z99` and the area is `ZZ`. The files hold no real address and no real coordinate. + +## Format + +```json +{ + "version": 1, + "description": "The synthetic unit FC-01-Z99-ZZ-01 at lookup level 1.", + "evidence": "observed", + "schema": "LookupEnvelope", + "status": 200, + "body": { "data": { "postcode": "FC-01-Z99-ZZ-01", "valid": true } } +} +``` + +- `version` is the format version, which is 1. +- `evidence` is `observed`, `documented` or `assumed`, as in `openapi/gateway.completed.yaml`. +- `schema` names a schema in `components/schemas` of that file. The body must match it, and `make check` tests that it does. +- `status` is the HTTP status of the response. + +## How the mock server uses them + +| Request | Fixture | +|---|---| +| `lookup` of the postcode in `lookup/valid-level-1.json` | `lookup/valid-level-N.json`, where N is the level | +| `lookup` of another postcode that is well-formed | `lookup/not-found.json` | +| `lookup` of text that is not a postcode | `lookup/invalid.json` | +| `reverse` at the coordinate of `reverse/unit.json` or `reverse/area.json` | that file | +| `reverse` at any other coordinate | `reverse/not-found.json` | +| `nearby` | `nearby/empty.json` | +| an error | the file in `errors/` with the same error code | + +- The mock server puts the caller's text, in upper case, in `postcode` of a lookup body, as the gateway does. +- It puts the request's coordinate and the applied radius in a reverse body. +- `level-not-granted-N.json` answers a key that holds level N. +- `keys.json` lists the mock server's keys, and the way that each one behaves. +- `autocomplete`, `assemble` and `disassemble` have no files. The mock server builds their bodies from the postcodes above and from `data/states.json`. + +## After a new observation + +A call can show a shape that differs from a file here, for example a lookup at level 2. Then change the file and its `evidence`, and change the schema in the OpenAPI file in the same pull request. Never commit the real response. Copy its shape, and keep the synthetic values. diff --git a/fixtures/errors/auth-required.json b/fixtures/errors/auth-required.json new file mode 100644 index 0000000..deba944 --- /dev/null +++ b/fixtures/errors/auth-required.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A request with no key.", + "evidence": "observed", + "schema": "ErrorEnvelope", + "status": 401, + "body": { + "error": { + "code": "auth_required", + "message": "send an API key in the X-API-Key header" + } + } +} diff --git a/fixtures/errors/insufficient-credits.json b/fixtures/errors/insufficient-credits.json new file mode 100644 index 0000000..f7bca80 --- /dev/null +++ b/fixtures/errors/insufficient-credits.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A lookup above level 1 with no credits left.", + "evidence": "documented", + "schema": "ErrorEnvelope", + "status": 402, + "body": { + "error": { + "code": "insufficient_credits", + "message": "not enough credits; top up to continue" + } + } +} diff --git a/fixtures/errors/invalid-api-key.json b/fixtures/errors/invalid-api-key.json new file mode 100644 index 0000000..c78c74d --- /dev/null +++ b/fixtures/errors/invalid-api-key.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A key that the gateway does not know. The message is Gatepost's.", + "evidence": "observed", + "schema": "ErrorEnvelope", + "status": 401, + "body": { + "error": { + "code": "invalid_api_key", + "message": "the API key is not valid" + } + } +} diff --git a/fixtures/errors/invalid-request.json b/fixtures/errors/invalid-request.json new file mode 100644 index 0000000..6484f44 --- /dev/null +++ b/fixtures/errors/invalid-request.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A missing or malformed parameter.", + "evidence": "assumed", + "schema": "ErrorEnvelope", + "status": 400, + "body": { + "error": { + "code": "invalid_request", + "message": "a parameter is missing or malformed" + } + } +} diff --git a/fixtures/errors/level-not-granted-1.json b/fixtures/errors/level-not-granted-1.json new file mode 100644 index 0000000..7939c32 --- /dev/null +++ b/fixtures/errors/level-not-granted-1.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A lookup above level 1 with a key that holds level 1.", + "evidence": "observed", + "schema": "ErrorEnvelope", + "status": 403, + "body": { + "error": { + "code": "level_not_granted", + "message": "this key cannot read above lookup level 1" + } + } +} diff --git a/fixtures/errors/level-not-granted-2.json b/fixtures/errors/level-not-granted-2.json new file mode 100644 index 0000000..4996aef --- /dev/null +++ b/fixtures/errors/level-not-granted-2.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A lookup above level 2 with a key that holds level 2.", + "evidence": "assumed", + "schema": "ErrorEnvelope", + "status": 403, + "body": { + "error": { + "code": "level_not_granted", + "message": "this key cannot read above lookup level 2" + } + } +} diff --git a/fixtures/errors/level-not-granted-3.json b/fixtures/errors/level-not-granted-3.json new file mode 100644 index 0000000..ed0b39d --- /dev/null +++ b/fixtures/errors/level-not-granted-3.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A lookup above level 3 with a key that holds level 3.", + "evidence": "assumed", + "schema": "ErrorEnvelope", + "status": 403, + "body": { + "error": { + "code": "level_not_granted", + "message": "this key cannot read above lookup level 3" + } + } +} diff --git a/fixtures/errors/origin-not-allowed.json b/fixtures/errors/origin-not-allowed.json new file mode 100644 index 0000000..e5b7879 --- /dev/null +++ b/fixtures/errors/origin-not-allowed.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A publishable key from an origin off its list. The message is Gatepost's.", + "evidence": "documented", + "schema": "ErrorEnvelope", + "status": 403, + "body": { + "error": { + "code": "origin_not_allowed", + "message": "this origin is not allowed for this key" + } + } +} diff --git a/fixtures/errors/rate-limited.json b/fixtures/errors/rate-limited.json new file mode 100644 index 0000000..5cf8794 --- /dev/null +++ b/fixtures/errors/rate-limited.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A key that used up its requests for this clock minute.", + "evidence": "assumed", + "schema": "ErrorEnvelope", + "status": 429, + "body": { + "error": { + "code": "rate_limited", + "message": "too many requests; wait for the next minute" + } + } +} diff --git a/fixtures/errors/scope-not-granted.json b/fixtures/errors/scope-not-granted.json new file mode 100644 index 0000000..1f32a75 --- /dev/null +++ b/fixtures/errors/scope-not-granted.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A lookup with a key that has no lookup scope.", + "evidence": "assumed", + "schema": "ErrorEnvelope", + "status": 403, + "body": { + "error": { + "code": "scope_not_granted", + "message": "this key has no lookup scope" + } + } +} diff --git a/fixtures/errors/unknown-path.json b/fixtures/errors/unknown-path.json new file mode 100644 index 0000000..635dd96 --- /dev/null +++ b/fixtures/errors/unknown-path.json @@ -0,0 +1,13 @@ +{ + "version": 1, + "description": "A path that the gateway does not serve.", + "evidence": "assumed", + "schema": "ErrorEnvelope", + "status": 404, + "body": { + "error": { + "code": "not_found", + "message": "no such endpoint" + } + } +} diff --git a/fixtures/keys.json b/fixtures/keys.json new file mode 100644 index 0000000..6de0ac3 --- /dev/null +++ b/fixtures/keys.json @@ -0,0 +1,72 @@ +{ + "version": 1, + "description": "The mock server's keys. Each one makes the mock behave in one way.", + "requestsPerMinute": 600, + "keys": [ + { + "key": "nipost_test_mock_l1", + "kind": "secret", + "lookupLevel": 1, + "lookupScope": true, + "credits": true, + "rateLimited": false, + "origins": null + }, + { + "key": "nipost_test_mock_l2", + "kind": "secret", + "lookupLevel": 2, + "lookupScope": true, + "credits": true, + "rateLimited": false, + "origins": null + }, + { + "key": "nipost_test_mock_l3", + "kind": "secret", + "lookupLevel": 3, + "lookupScope": true, + "credits": true, + "rateLimited": false, + "origins": null + }, + { + "key": "nipost_test_mock_no_credits", + "kind": "secret", + "lookupLevel": 3, + "lookupScope": true, + "credits": false, + "rateLimited": false, + "origins": null + }, + { + "key": "nipost_test_mock_no_scope", + "kind": "secret", + "lookupLevel": 3, + "lookupScope": false, + "credits": true, + "rateLimited": false, + "origins": null + }, + { + "key": "nipost_test_mock_rate_limited", + "kind": "secret", + "lookupLevel": 3, + "lookupScope": true, + "credits": true, + "rateLimited": true, + "origins": null + }, + { + "key": "nipost_pk_test_mock", + "kind": "publishable", + "lookupLevel": 3, + "lookupScope": true, + "credits": true, + "rateLimited": false, + "origins": [ + "http://localhost:3000" + ] + } + ] +} diff --git a/fixtures/lookup/invalid.json b/fixtures/lookup/invalid.json new file mode 100644 index 0000000..5d318d8 --- /dev/null +++ b/fixtures/lookup/invalid.json @@ -0,0 +1,15 @@ +{ + "version": 1, + "description": "Text that is not a postcode. The gateway echoes it in upper case.", + "evidence": "observed", + "schema": "LookupEnvelope", + "status": 200, + "body": { + "data": { + "postcode": "HELLO", + "valid": false, + "status": "invalid", + "verified": false + } + } +} diff --git a/fixtures/lookup/not-found.json b/fixtures/lookup/not-found.json new file mode 100644 index 0000000..0a0569f --- /dev/null +++ b/fixtures/lookup/not-found.json @@ -0,0 +1,15 @@ +{ + "version": 1, + "description": "A well-formed postcode that the gateway does not know.", + "evidence": "observed", + "schema": "LookupEnvelope", + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-02", + "valid": false, + "status": "not_found", + "verified": false + } + } +} diff --git a/fixtures/lookup/valid-level-1.json b/fixtures/lookup/valid-level-1.json new file mode 100644 index 0000000..71e8d9a --- /dev/null +++ b/fixtures/lookup/valid-level-1.json @@ -0,0 +1,15 @@ +{ + "version": 1, + "description": "The synthetic unit FC-01-Z99-ZZ-01 at lookup level 1.", + "evidence": "observed", + "schema": "LookupEnvelope", + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "verified": false + } + } +} diff --git a/fixtures/lookup/valid-level-2.json b/fixtures/lookup/valid-level-2.json new file mode 100644 index 0000000..3a72671 --- /dev/null +++ b/fixtures/lookup/valid-level-2.json @@ -0,0 +1,22 @@ +{ + "version": 1, + "description": "The synthetic unit at lookup level 2, in the shape of NIPOST's example.", + "evidence": "documented", + "schema": "LookupEnvelope", + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "administrative_address": { + "state_name": "FEDERAL CAPITAL TERRITORY", + "lga_name": "SYNTHETIC LGA", + "locality_name": "SYNTHETIC LOCALITY", + "zone": "NORTH CENTRAL" + }, + "recent_house_address": { + "recent": "1 SYNTHETIC STREET, SYNTHETIC LOCALITY" + } + } + } +} diff --git a/fixtures/lookup/valid-level-3.json b/fixtures/lookup/valid-level-3.json new file mode 100644 index 0000000..0b71d1d --- /dev/null +++ b/fixtures/lookup/valid-level-3.json @@ -0,0 +1,23 @@ +{ + "version": 1, + "description": "The synthetic unit at lookup level 3, in the shape of NIPOST's example.", + "evidence": "documented", + "schema": "LookupEnvelope", + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "administrative_address": { + "state_name": "FEDERAL CAPITAL TERRITORY", + "lga_name": "SYNTHETIC LGA", + "locality_name": "SYNTHETIC LOCALITY", + "zone": "NORTH CENTRAL" + }, + "recent_house_address": { + "recent": "1 SYNTHETIC STREET, SYNTHETIC LOCALITY" + }, + "building_use_status": "residential" + } + } +} diff --git a/fixtures/nearby/empty.json b/fixtures/nearby/empty.json new file mode 100644 index 0000000..6ee3a3a --- /dev/null +++ b/fixtures/nearby/empty.json @@ -0,0 +1,10 @@ +{ + "version": 1, + "description": "The only nearby body seen so far.", + "evidence": "observed", + "schema": "NearbyEnvelope", + "status": 200, + "body": { + "data": [] + } +} diff --git a/fixtures/reverse/area.json b/fixtures/reverse/area.json new file mode 100644 index 0000000..a6c958a --- /dev/null +++ b/fixtures/reverse/area.json @@ -0,0 +1,21 @@ +{ + "version": 1, + "description": "A point in the synthetic area with no unit within the radius.", + "evidence": "observed", + "schema": "ReverseEnvelope", + "status": 200, + "body": { + "data": { + "found": true, + "coordinate": [ + 7.001, + 9.001 + ], + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC", + "depth": "area", + "radius_m": 25 + } + } +} diff --git a/fixtures/reverse/not-found.json b/fixtures/reverse/not-found.json new file mode 100644 index 0000000..f538004 --- /dev/null +++ b/fixtures/reverse/not-found.json @@ -0,0 +1,18 @@ +{ + "version": 1, + "description": "A point with no postcode within the radius.", + "evidence": "observed", + "schema": "ReverseEnvelope", + "status": 200, + "body": { + "data": { + "found": false, + "coordinate": [ + 0, + 0 + ], + "message": "no postcode found near this point", + "radius_m": 25 + } + } +} diff --git a/fixtures/reverse/unit.json b/fixtures/reverse/unit.json new file mode 100644 index 0000000..20127a3 --- /dev/null +++ b/fixtures/reverse/unit.json @@ -0,0 +1,27 @@ +{ + "version": 1, + "description": "A point 4.2 m from the synthetic unit.", + "evidence": "observed", + "schema": "ReverseEnvelope", + "status": 200, + "body": { + "data": { + "found": true, + "coordinate": [ + 7, + 9 + ], + "unit": { + "postcode": "FC-01-Z99-ZZ-01", + "display": "FC 01 Z99 ZZ 01", + "distance_m": 4.2, + "confidence": "high" + }, + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC", + "depth": "unit", + "radius_m": 25 + } + } +} diff --git a/scripts/gateway_files.py b/scripts/gateway_files.py index 1d621ad..274fdaf 100644 --- a/scripts/gateway_files.py +++ b/scripts/gateway_files.py @@ -1,12 +1,14 @@ # SPDX-FileCopyrightText: 2026 The Gatepost authors # SPDX-License-Identifier: Apache-2.0 -"""Read the completed OpenAPI file, and check its paths, its evidence marks and its schemas. +"""Read the completed OpenAPI file, the fixtures and the mock keys, and check that they agree. Each check returns a list of problems, one line each. An empty list means that the file passes. """ from __future__ import annotations +import json +import re from collections.abc import Callable, Iterator from pathlib import Path from typing import Any @@ -18,6 +20,7 @@ ROOT = Path(__file__).resolve().parents[1] OPENAPI = ROOT / "openapi" / "gateway.completed.yaml" +FIXTURES = ROOT / "fixtures" EVIDENCE = ("observed", "documented", "assumed") # NIPOST documents these gateway paths. SEC-4 lets a client call no other path. DOCUMENTED_PATHS = ( @@ -29,11 +32,25 @@ "/v1/assembly/assemble", "/v1/assembly/disassemble", ) +FIXTURE_FIELDS = ("version", "description", "evidence", "schema", "status", "body") +KEY_FIELDS = ("key", "kind", "lookupLevel", "lookupScope", "credits", "rateLimited", "origins") +# A mock key must never look like a real key, so a real key cannot slip into the table. +MOCK_KEY = re.compile(r"nipost_(?:pk_)?test_mock(?:_[a-z0-9]+)*") +# Five segments, with or without separators, in either letter case. +POSTCODE = re.compile( + r"(? Any: + """Return the JSON value in a file.""" + return json.loads(path.read_text(encoding="utf-8")) + + def load_openapi(path: Path = OPENAPI) -> dict[str, Any]: """Return the completed OpenAPI file as a mapping.""" document = yaml.safe_load(path.read_text(encoding="utf-8")) @@ -98,3 +115,69 @@ def schemas_in_responses(openapi: dict[str, Any]) -> set[str]: for media in response.get("content", {}).values(): names.add(media["schema"]["$ref"].rpartition("/")[2]) return names + + +def load_fixtures(root: Path = FIXTURES) -> dict[str, Any]: + """Return each fixture by its name, the path under the folder without `.json`.""" + return { + path.relative_to(root).with_suffix("").as_posix(): read_json(path) + for path in sorted(root.glob("*/*.json")) + } + + +def fixture_problems(name: str, fixture: Any, check: SchemaCheck) -> list[str]: + """Return the problems of one fixture, including a body that does not match its schema.""" + if not isinstance(fixture, dict) or tuple(fixture) != FIXTURE_FIELDS: + return [f"Fixture {name} must have the fields {', '.join(FIXTURE_FIELDS)}."] + problems = [] + if fixture["version"] != 1: + problems.append(f"Fixture {name} must have version 1.") + if fixture["evidence"] not in EVIDENCE: + problems.append(f"Fixture {name} needs evidence: one of {', '.join(EVIDENCE)}.") + if type(fixture["status"]) is not int: + problems.append(f"Fixture {name} needs an integer status.") + problems += [ + f"Fixture {name}: {message}" for message in check(fixture["schema"], fixture["body"]) + ] + return problems + + +def key_problems(document: Any) -> list[str]: + """Return the problems of `fixtures/keys.json`.""" + problems = [] + keys = document.get("keys", []) if isinstance(document, dict) else [] + if not keys or document.get("version") != 1: + problems.append("keys.json must have version 1 and a list of keys.") + seen = set() + for entry in keys: + if not isinstance(entry, dict) or tuple(entry) != KEY_FIELDS: + problems.append(f"Each key must have the fields {', '.join(KEY_FIELDS)}.") + continue + key = entry["key"] + if key in seen or not MOCK_KEY.fullmatch(key): + problems.append(f"The key {key} must be unique and start with a mock prefix.") + seen.add(key) + if entry["lookupLevel"] not in (1, 2, 3): + problems.append(f"The key {key} needs a lookupLevel from 1 to 3.") + return problems + + +def real_postcodes(text: str) -> list[str]: + """Return each full postcode in the text that is not synthetic. + + A synthetic postcode has the district Z99 and the area ZZ. + """ + return [ + match.group(0) + for match in POSTCODE.finditer(text) + if (match.group(3).upper(), match.group(4).upper()) != ("Z99", "ZZ") + ] + + +def postcode_problems(paths: list[Path]) -> list[str]: + """Return a problem for each postcode in the files that is not synthetic.""" + return [ + f"{path.relative_to(ROOT)} holds the postcode {code}. Use a synthetic code." + for path in paths + for code in real_postcodes(path.read_text(encoding="utf-8")) + ] diff --git a/scripts/tests/test_gateway_files.py b/scripts/tests/test_gateway_files.py index 06a0cb9..142088a 100644 --- a/scripts/tests/test_gateway_files.py +++ b/scripts/tests/test_gateway_files.py @@ -1,14 +1,25 @@ # SPDX-FileCopyrightText: 2026 The Gatepost authors # SPDX-License-Identifier: Apache-2.0 -"""Tests for gateway_files: the OpenAPI file has the documented paths, evidence and schemas.""" +"""Tests for gateway_files: the OpenAPI file, fixtures and keys agree and stay synthetic.""" import copy +import tempfile import unittest +from pathlib import Path +from typing import Any import gateway_files OPENAPI = gateway_files.load_openapi() CHECK = gateway_files.schema_check(OPENAPI) +FIXTURES = gateway_files.load_fixtures() +KEYS = gateway_files.read_json(gateway_files.FIXTURES / "keys.json") + + +def edited_fixture(name: str, **fields: Any) -> dict[str, Any]: + fixture: dict[str, Any] = copy.deepcopy(FIXTURES[name]) + fixture.update(fields) + return fixture class OpenApiTest(unittest.TestCase): @@ -68,3 +79,104 @@ def test_follows_references_between_schemas(self) -> None: def test_rejects_a_schema_name_that_the_file_does_not_have(self) -> None: self.assertEqual(CHECK("Lookups", {}), ["The OpenAPI file has no schema Lookups."]) + + +class FixtureTest(unittest.TestCase): + def test_each_fixture_has_no_problem(self) -> None: + for name, fixture in FIXTURES.items(): + with self.subTest(name): + self.assertEqual(gateway_files.fixture_problems(name, fixture, CHECK), []) + + def test_each_fixture_uses_a_schema_of_a_response(self) -> None: + used = gateway_files.schemas_in_responses(OPENAPI) + self.assertEqual(sorted({fixture["schema"] for fixture in FIXTURES.values()} - used), []) + + def test_has_a_body_for_each_lookup_level_that_a_mock_key_holds(self) -> None: + for level in (1, 2, 3): + with self.subTest(level=level): + self.assertIn(f"lookup/valid-level-{level}", FIXTURES) + self.assertIn(f"errors/level-not-granted-{level}", FIXTURES) + + def test_rejects_a_body_that_does_not_match_its_schema(self) -> None: + body = {"data": {"found": True, "coordinate": [7], "radius_m": 25}} + fixture = edited_fixture("reverse/unit", body=body) + self.assertEqual( + gateway_files.fixture_problems("reverse/unit", fixture, CHECK), + ["Fixture reverse/unit: [7] is too short at $.data.coordinate"], + ) + + def test_rejects_an_evidence_value_that_the_openapi_file_does_not_use(self) -> None: + fixture = edited_fixture("nearby/empty", evidence="seen") + [problem] = gateway_files.fixture_problems("nearby/empty", fixture, CHECK) + self.assertEqual( + problem, "Fixture nearby/empty needs evidence: one of observed, documented, assumed." + ) + + def test_rejects_a_status_that_is_not_an_integer(self) -> None: + fixture = edited_fixture("nearby/empty", status="200") + self.assertEqual( + gateway_files.fixture_problems("nearby/empty", fixture, CHECK), + ["Fixture nearby/empty needs an integer status."], + ) + + def test_rejects_a_fixture_with_a_field_missing(self) -> None: + fixture = edited_fixture("nearby/empty") + del fixture["evidence"] + [problem] = gateway_files.fixture_problems("nearby/empty", fixture, CHECK) + self.assertTrue(problem.startswith("Fixture nearby/empty must have the fields version,")) + + +class KeysTest(unittest.TestCase): + def test_the_key_file_has_no_problem(self) -> None: + self.assertEqual(gateway_files.key_problems(KEYS), []) + + def test_rejects_a_key_that_could_be_real(self) -> None: + keys = copy.deepcopy(KEYS) + keys["keys"][0]["key"] = "nipost_test_a1b2c3d4e5" + self.assertEqual( + gateway_files.key_problems(keys), + ["The key nipost_test_a1b2c3d4e5 must be unique and start with a mock prefix."], + ) + + def test_rejects_a_key_twice(self) -> None: + keys = copy.deepcopy(KEYS) + keys["keys"].append(copy.deepcopy(keys["keys"][0])) + self.assertEqual(len(gateway_files.key_problems(keys)), 1) + + def test_rejects_a_level_that_no_key_can_hold(self) -> None: + keys = copy.deepcopy(KEYS) + keys["keys"][0]["lookupLevel"] = 4 + self.assertEqual( + gateway_files.key_problems(keys), + ["The key nipost_test_mock_l1 needs a lookupLevel from 1 to 3."], + ) + + +class SyntheticPostcodeTest(unittest.TestCase): + def test_finds_no_real_postcode_in_the_committed_files(self) -> None: + files = sorted( + [ + *gateway_files.FIXTURES.rglob("*.json"), + *(gateway_files.ROOT / "contract").glob("*.json"), + ] + ) + self.assertEqual(gateway_files.postcode_problems(files), []) + + def test_finds_a_real_postcode_in_each_form_and_case(self) -> None: + text = "EK-01-A03-FK-01, EK 01 A03 FK 01, ek01a03fk01 and FC-01-Z99-ZZ-01" + self.assertEqual( + gateway_files.real_postcodes(text), + ["EK-01-A03-FK-01", "EK 01 A03 FK 01", "ek01a03fk01"], + ) + + def test_keeps_a_partial_postcode_and_a_longer_word(self) -> None: + self.assertEqual(gateway_files.real_postcodes("EK-01-A03 and XEK01A03FK01"), []) + + def test_names_the_file_that_holds_a_real_postcode(self) -> None: + with tempfile.TemporaryDirectory(dir=gateway_files.ROOT) as folder: + path = Path(folder) / "probe.json" + path.write_text('{"postcode": "LA-12-K23-IV-95"}', encoding="utf-8") + [problem] = gateway_files.postcode_problems([path]) + self.assertTrue( + problem.endswith("probe.json holds the postcode LA-12-K23-IV-95. Use a synthetic code.") + ) From 5ab0439da9f3dd88838b327811581107c86abdc1 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:42:23 +0100 Subject: [PATCH 05/13] feat: add the contract scenarios that every client must pass Every client must pass the same scenarios against the mock server. Each scenario names the calls, the responses and the outcome, with the attempts and the waits, so a TypeScript and a PHP client that pass them behave the same. The scenarios also cover the input rules, the plain form of a number, unknown fields and an unknown lookup status. A check fails when a scenario names a missing fixture or an error code that client.md does not list, and when an error code or a required rule has no scenario. Signed-off-by: MobileMage --- Makefile | 1 + contract/README.md | 85 +++++++++ contract/autocomplete-bad-character.json | 14 ++ contract/autocomplete-district.json | 38 ++++ contract/autocomplete-empty-input.json | 14 ++ contract/autocomplete-state.json | 27 +++ contract/autocomplete-timeout.json | 14 ++ contract/autocomplete-unknown-fields.json | 31 +++ contract/lookup-bad-request.json | 21 +++ contract/lookup-cache-hit.json | 41 ++++ contract/lookup-cache-off.json | 41 ++++ contract/lookup-error-not-cached.json | 37 ++++ contract/lookup-insufficient-credits.json | 21 +++ contract/lookup-invalid-input.json | 14 ++ contract/lookup-invalid-key.json | 21 +++ contract/lookup-invalid-level.json | 14 ++ contract/lookup-level-1.json | 31 +++ contract/lookup-level-2.json | 30 +++ contract/lookup-level-3.json | 30 +++ contract/lookup-level-not-granted.json | 27 +++ contract/lookup-malformed-body.json | 20 ++ contract/lookup-max-retries-zero.json | 23 +++ contract/lookup-network-error.json | 15 ++ contract/lookup-no-key.json | 27 +++ contract/lookup-not-found.json | 31 +++ contract/lookup-origin-not-allowed.json | 21 +++ contract/lookup-queue-limit.json | 94 ++++++++++ contract/lookup-rate-limited.json | 21 +++ contract/lookup-retries-exhausted.json | 24 +++ contract/lookup-retry-502.json | 29 +++ contract/lookup-retry-503.json | 29 +++ contract/lookup-retry-504.json | 29 +++ contract/lookup-retry-after-too-long.json | 21 +++ contract/lookup-retry-after.json | 29 +++ contract/lookup-server-error.json | 23 +++ contract/lookup-shared-request.json | 41 ++++ contract/lookup-timeout.json | 15 ++ contract/lookup-unknown-fields.json | 39 ++++ contract/lookup-unknown-status.json | 30 +++ contract/reverse-area.json | 29 +++ contract/reverse-invalid-lat.json | 14 ++ contract/reverse-invalid-lng.json | 14 ++ contract/reverse-invalid-max-distance.json | 14 ++ contract/reverse-not-found.json | 23 +++ contract/reverse-plain-decimal.json | 29 +++ contract/reverse-unit.json | 37 ++++ contract/reverse-unknown-fields.json | 54 ++++++ scripts/contract_files.py | 208 ++++++++++++++++++++- scripts/tests/test_contract_files.py | 169 ++++++++++++++--- 49 files changed, 1676 insertions(+), 28 deletions(-) create mode 100644 contract/README.md create mode 100644 contract/autocomplete-bad-character.json create mode 100644 contract/autocomplete-district.json create mode 100644 contract/autocomplete-empty-input.json create mode 100644 contract/autocomplete-state.json create mode 100644 contract/autocomplete-timeout.json create mode 100644 contract/autocomplete-unknown-fields.json create mode 100644 contract/lookup-bad-request.json create mode 100644 contract/lookup-cache-hit.json create mode 100644 contract/lookup-cache-off.json create mode 100644 contract/lookup-error-not-cached.json create mode 100644 contract/lookup-insufficient-credits.json create mode 100644 contract/lookup-invalid-input.json create mode 100644 contract/lookup-invalid-key.json create mode 100644 contract/lookup-invalid-level.json create mode 100644 contract/lookup-level-1.json create mode 100644 contract/lookup-level-2.json create mode 100644 contract/lookup-level-3.json create mode 100644 contract/lookup-level-not-granted.json create mode 100644 contract/lookup-malformed-body.json create mode 100644 contract/lookup-max-retries-zero.json create mode 100644 contract/lookup-network-error.json create mode 100644 contract/lookup-no-key.json create mode 100644 contract/lookup-not-found.json create mode 100644 contract/lookup-origin-not-allowed.json create mode 100644 contract/lookup-queue-limit.json create mode 100644 contract/lookup-rate-limited.json create mode 100644 contract/lookup-retries-exhausted.json create mode 100644 contract/lookup-retry-502.json create mode 100644 contract/lookup-retry-503.json create mode 100644 contract/lookup-retry-504.json create mode 100644 contract/lookup-retry-after-too-long.json create mode 100644 contract/lookup-retry-after.json create mode 100644 contract/lookup-server-error.json create mode 100644 contract/lookup-shared-request.json create mode 100644 contract/lookup-timeout.json create mode 100644 contract/lookup-unknown-fields.json create mode 100644 contract/lookup-unknown-status.json create mode 100644 contract/reverse-area.json create mode 100644 contract/reverse-invalid-lat.json create mode 100644 contract/reverse-invalid-lng.json create mode 100644 contract/reverse-invalid-max-distance.json create mode 100644 contract/reverse-not-found.json create mode 100644 contract/reverse-plain-decimal.json create mode 100644 contract/reverse-unit.json create mode 100644 contract/reverse-unknown-fields.json diff --git a/Makefile b/Makefile index b721c8c..c80691f 100644 --- a/Makefile +++ b/Makefile @@ -43,6 +43,7 @@ vectors: contract: uvx openapi-spec-validator@$(OPENAPI_SPEC_VALIDATOR_VERSION) openapi/gateway.completed.yaml + $(PYTHON) scripts/contract_files.py tells: python3 scripts/check-tells diff --git a/contract/README.md b/contract/README.md new file mode 100644 index 0000000..c6e187f --- /dev/null +++ b/contract/README.md @@ -0,0 +1,85 @@ +# Contract scenarios + +Each JSON file here is one contract scenario. It names the calls that a test makes, the responses that the mock server gives, and the outcome that every client must reach. `client.md` defines the behaviour. The scenarios test it in every SDK (T-3). + +## Format + +```json +{ + "version": 1, + "id": "lookup-retry-503", + "description": "A 503 is retried after the first wait, and the second attempt succeeds.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + {"status": 503, "body": {"error": {"code": "unavailable", "message": "try again"}}}, + {"fixture": "lookup/valid-level-1"} + ], + "expect": { + "attempts": 2, + "waitsMs": [{"min": 500, "max": 750}], + "outcomes": [{"result": {"postcode": "FC-01-Z99-ZZ-01", "valid": true}}] + } +} +``` + +The outcome above is short. A real scenario lists every field of the result. + +| Field | Meaning | +|---|---| +| `version` | the format version, which is 1 | +| `id` | the file name without `.json` | +| `description` | the rule that the scenario tests, in one sentence | +| `client` | the client options: `apiKey`, `timeoutMs`, `maxRetries` and `cacheTtlMs`. A missing option keeps its default. The test sets `baseUrl` to the mock server | +| `order` | `parallel` or `sequential`, when there is more than one call | +| `calls` | the calls, in order | +| `responses` | the responses of the mock server, one for each request, in order | +| `expect` | the outcome | + +### Calls + +| `method` | Other fields | +|---|---| +| `lookup` | `code` and `level` | +| `reverse` | `lat`, `lng`, and `maxDistanceM` when the call passes a radius | +| `autocomplete` | `q` | + +With `parallel`, the test starts every call before it waits for any of them. With `sequential`, it waits for each call before it starts the next one. + +### Responses + +The mock server gives the first response to the first request, the second response to the second request, and so on. When the requests outnumber the responses, it repeats the last response. Each response has exactly one of these fields: + +| Field | The mock server | +|---|---| +| `body` | sends this JSON body with `status` | +| `text` | sends this text body with `status` | +| `fixture` | sends the fixture `fixtures/.json`, with its status and body | +| `hang` | sends nothing, and keeps the connection open until the client closes it | +| `drop` | closes the connection without a response | + +- `headers` adds response headers to `body`, `text` or `fixture`. The mock server adds `Content-Type: application/json` to a JSON body, and the CORS headers to every response. +- `delayMs` makes the mock server wait before it answers. +- A scenario with no request has an empty `responses` list. + +### Expect + +| Field | Meaning | +|---|---| +| `attempts` | the number of requests that reached the transport, for all calls together | +| `waitsMs` | for one call, the bounds of each wait between two attempts, in milliseconds | +| `maxInFlight` | the most requests that were in flight at one time | +| `request` | the first request: `method`, `path`, `query` and `apiKey`, which is null when the request had no `X-API-Key` header | +| `outcomes` | one outcome for each call, in the order of the calls | + +An outcome is `{"result": ...}` or `{"error": ...}`. A result uses the field names in `client.md`. A postcode in a result is its canonical form. An error has `code`, `status`, `apiCode` and `retryAfterMs`. + +A wait runs from the end of one attempt to the start of the next one. Timers can fire late, so a test can accept a wait up to 100 ms over `max`. It must not accept a wait under `min`. + +## Running a scenario + +1. Start the mock server. +2. Build a client with the scenario's options, and a transport that wraps the real one. The wrapper adds two headers to each request: `X-Scenario-Id` with the scenario's id, and `X-Scenario-Run` with a value that is new for each run. It also counts the requests, the waits between them and the requests in flight. +3. Make the calls, and compare the outcomes and the counts with `expect`. + +The client itself never sends `X-Scenario-Id` or `X-Scenario-Run`. The mock server keeps the position in `responses` for each run, so a test can run a scenario again with a new run value. A request with `X-Scenario-Id` and no `X-Scenario-Run` gets a 400 from the mock server. diff --git a/contract/autocomplete-bad-character.json b/contract/autocomplete-bad-character.json new file mode 100644 index 0000000..5858b68 --- /dev/null +++ b/contract/autocomplete-bad-character.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "autocomplete-bad-character", + "description": "Text with a character other than A to Z and 0 to 9 sends no request.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "autocomplete", "q": "FC!"}], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + {"error": {"code": "invalid_input", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/autocomplete-district.json b/contract/autocomplete-district.json new file mode 100644 index 0000000..1a90a9b --- /dev/null +++ b/contract/autocomplete-district.json @@ -0,0 +1,38 @@ +{ + "version": 1, + "id": "autocomplete-district", + "description": "The client sends normalised text, keeps a label, and gives null for a bad code.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "autocomplete", "q": "fc 01 z"}], + "responses": [ + { + "status": 200, + "body": { + "data": { + "segment": "district", + "suggestions": [{"code": "Z99", "label": "SYNTHETIC DISTRICT"}, {"code": "ZZZZ"}] + } + } + } + ], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/search/autocomplete", + "query": {"q": "FC01Z"}, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "segment": "district", + "suggestions": [ + {"code": "Z99", "label": "SYNTHETIC DISTRICT", "postcode": "FC-01-Z99"}, + {"code": "ZZZZ", "label": null, "postcode": null} + ] + } + } + ] + } +} diff --git a/contract/autocomplete-empty-input.json b/contract/autocomplete-empty-input.json new file mode 100644 index 0000000..89d7abc --- /dev/null +++ b/contract/autocomplete-empty-input.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "autocomplete-empty-input", + "description": "Text that normalises to nothing sends no request. The gateway would not answer.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "autocomplete", "q": " - "}], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + {"error": {"code": "invalid_input", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/autocomplete-state.json b/contract/autocomplete-state.json new file mode 100644 index 0000000..7dccb11 --- /dev/null +++ b/contract/autocomplete-state.json @@ -0,0 +1,27 @@ +{ + "version": 1, + "id": "autocomplete-state", + "description": "A suggestion holds a segment value, and the client adds the partial postcode.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "autocomplete", "q": "f"}], + "responses": [ + {"status": 200, "body": {"data": {"segment": "state", "suggestions": [{"code": "FC"}]}}} + ], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/search/autocomplete", + "query": {"q": "F"}, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "segment": "state", + "suggestions": [{"code": "FC", "label": null, "postcode": "FC"}] + } + } + ] + } +} diff --git a/contract/autocomplete-timeout.json b/contract/autocomplete-timeout.json new file mode 100644 index 0000000..245fa3a --- /dev/null +++ b/contract/autocomplete-timeout.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "autocomplete-timeout", + "description": "A timeout of autocomplete is not retried. The next keystroke replaces it.", + "client": {"apiKey": "nipost_test_mock_l3", "timeoutMs": 200}, + "calls": [{"method": "autocomplete", "q": "F"}], + "responses": [{"hang": true}], + "expect": { + "attempts": 1, + "outcomes": [ + {"error": {"code": "timeout", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/autocomplete-unknown-fields.json b/contract/autocomplete-unknown-fields.json new file mode 100644 index 0000000..35319ab --- /dev/null +++ b/contract/autocomplete-unknown-fields.json @@ -0,0 +1,31 @@ +{ + "version": 1, + "id": "autocomplete-unknown-fields", + "description": "The client ignores a field that client.md does not list.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "autocomplete", "q": "f"}], + "responses": [ + { + "status": 200, + "body": { + "data": { + "segment": "state", + "suggestions": [{"code": "FC", "rank": 1}], + "total": 1 + }, + "meta": 1 + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "segment": "state", + "suggestions": [{"code": "FC", "label": null, "postcode": "FC"}] + } + } + ] + } +} diff --git a/contract/lookup-bad-request.json b/contract/lookup-bad-request.json new file mode 100644 index 0000000..07ba607 --- /dev/null +++ b/contract/lookup-bad-request.json @@ -0,0 +1,21 @@ +{ + "version": 1, + "id": "lookup-bad-request", + "description": "Another status from 400 to 499 gives invalid_input, with no retry.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"fixture": "errors/invalid-request"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "invalid_input", + "status": 400, + "apiCode": "invalid_request", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-cache-hit.json b/contract/lookup-cache-hit.json new file mode 100644 index 0000000..7819be8 --- /dev/null +++ b/contract/lookup-cache-hit.json @@ -0,0 +1,41 @@ +{ + "version": 1, + "id": "lookup-cache-hit", + "description": "With cacheTtlMs above 0, a second identical call gets the kept result.", + "client": {"apiKey": "nipost_test_mock_l3", "cacheTtlMs": 60000}, + "order": "sequential", + "calls": [ + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1} + ], + "responses": [{"fixture": "lookup/valid-level-1"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-cache-off.json b/contract/lookup-cache-off.json new file mode 100644 index 0000000..7ed794e --- /dev/null +++ b/contract/lookup-cache-off.json @@ -0,0 +1,41 @@ +{ + "version": 1, + "id": "lookup-cache-off", + "description": "With the default cacheTtlMs of 0, each call sends a request.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "order": "sequential", + "calls": [ + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1} + ], + "responses": [{"fixture": "lookup/valid-level-1"}], + "expect": { + "attempts": 2, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-error-not-cached.json b/contract/lookup-error-not-cached.json new file mode 100644 index 0000000..a68103e --- /dev/null +++ b/contract/lookup-error-not-cached.json @@ -0,0 +1,37 @@ +{ + "version": 1, + "id": "lookup-error-not-cached", + "description": "An error stays out of the cache, so the next identical call sends a request.", + "client": {"apiKey": "nipost_test_mock_l3", "cacheTtlMs": 60000}, + "order": "sequential", + "calls": [ + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1} + ], + "responses": [{"fixture": "errors/invalid-request"}, {"fixture": "lookup/valid-level-1"}], + "expect": { + "attempts": 2, + "outcomes": [ + { + "error": { + "code": "invalid_input", + "status": 400, + "apiCode": "invalid_request", + "retryAfterMs": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-insufficient-credits.json b/contract/lookup-insufficient-credits.json new file mode 100644 index 0000000..a1c4f59 --- /dev/null +++ b/contract/lookup-insufficient-credits.json @@ -0,0 +1,21 @@ +{ + "version": 1, + "id": "lookup-insufficient-credits", + "description": "A 402 gives insufficient_credits, with no retry.", + "client": {"apiKey": "nipost_test_mock_no_credits"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 2}], + "responses": [{"fixture": "errors/insufficient-credits"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "insufficient_credits", + "status": 402, + "apiCode": "insufficient_credits", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-invalid-input.json b/contract/lookup-invalid-input.json new file mode 100644 index 0000000..fcb14ad --- /dev/null +++ b/contract/lookup-invalid-input.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "lookup-invalid-input", + "description": "A partial postcode fails the client's check, so no request goes out.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC01Z99", "level": 1}], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + {"error": {"code": "invalid_input", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/lookup-invalid-key.json b/contract/lookup-invalid-key.json new file mode 100644 index 0000000..abbbc72 --- /dev/null +++ b/contract/lookup-invalid-key.json @@ -0,0 +1,21 @@ +{ + "version": 1, + "id": "lookup-invalid-key", + "description": "A key that the gateway does not know gives unauthorized, with no retry.", + "client": {"apiKey": "nipost_test_mock_unknown"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"fixture": "errors/invalid-api-key"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "unauthorized", + "status": 401, + "apiCode": "invalid_api_key", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-invalid-level.json b/contract/lookup-invalid-level.json new file mode 100644 index 0000000..d390963 --- /dev/null +++ b/contract/lookup-invalid-level.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "lookup-invalid-level", + "description": "A level that is not a whole number from 1 to 5 sends no request.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 6}], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + {"error": {"code": "invalid_input", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/lookup-level-1.json b/contract/lookup-level-1.json new file mode 100644 index 0000000..e6af0e7 --- /dev/null +++ b/contract/lookup-level-1.json @@ -0,0 +1,31 @@ +{ + "version": 1, + "id": "lookup-level-1", + "description": "A lookup at level 1 sends the canonical form and the level.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"fixture": "lookup/valid-level-1"}], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/lookup", + "query": {"code": "FC-01-Z99-ZZ-01", "level": "1"}, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-level-2.json b/contract/lookup-level-2.json new file mode 100644 index 0000000..fee5a70 --- /dev/null +++ b/contract/lookup-level-2.json @@ -0,0 +1,30 @@ +{ + "version": 1, + "id": "lookup-level-2", + "description": "A level 2 body in NIPOST's documented shape gives the address fields.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 2}], + "responses": [{"fixture": "lookup/valid-level-2"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": null, + "levelRequested": 2, + "levelReceived": 2, + "administrativeAddress": { + "stateName": "FEDERAL CAPITAL TERRITORY", + "lgaName": "SYNTHETIC LGA", + "localityName": "SYNTHETIC LOCALITY", + "zone": "NORTH CENTRAL" + }, + "recentHouseAddress": "1 SYNTHETIC STREET, SYNTHETIC LOCALITY", + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-level-3.json b/contract/lookup-level-3.json new file mode 100644 index 0000000..1d6886f --- /dev/null +++ b/contract/lookup-level-3.json @@ -0,0 +1,30 @@ +{ + "version": 1, + "id": "lookup-level-3", + "description": "A level 3 body in NIPOST's documented shape adds the building use status.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 3}], + "responses": [{"fixture": "lookup/valid-level-3"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": null, + "levelRequested": 3, + "levelReceived": 3, + "administrativeAddress": { + "stateName": "FEDERAL CAPITAL TERRITORY", + "lgaName": "SYNTHETIC LGA", + "localityName": "SYNTHETIC LOCALITY", + "zone": "NORTH CENTRAL" + }, + "recentHouseAddress": "1 SYNTHETIC STREET, SYNTHETIC LOCALITY", + "buildingUseStatus": "residential" + } + } + ] + } +} diff --git a/contract/lookup-level-not-granted.json b/contract/lookup-level-not-granted.json new file mode 100644 index 0000000..ad6c6b8 --- /dev/null +++ b/contract/lookup-level-not-granted.json @@ -0,0 +1,27 @@ +{ + "version": 1, + "id": "lookup-level-not-granted", + "description": "A level above the key's level gives forbidden. The client keeps the level.", + "client": {"apiKey": "nipost_test_mock_l1"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 2}], + "responses": [{"fixture": "errors/level-not-granted-1"}], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/lookup", + "query": {"code": "FC-01-Z99-ZZ-01", "level": "2"}, + "apiKey": "nipost_test_mock_l1" + }, + "outcomes": [ + { + "error": { + "code": "forbidden", + "status": 403, + "apiCode": "level_not_granted", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-malformed-body.json b/contract/lookup-malformed-body.json new file mode 100644 index 0000000..b6528ff --- /dev/null +++ b/contract/lookup-malformed-body.json @@ -0,0 +1,20 @@ +{ + "version": 1, + "id": "lookup-malformed-body", + "description": "A 200 whose body is not JSON gives server_error, with no retry.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + { + "status": 200, + "headers": {"Content-Type": "text/html"}, + "text": "Sign in to the network" + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + {"error": {"code": "server_error", "status": 200, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/lookup-max-retries-zero.json b/contract/lookup-max-retries-zero.json new file mode 100644 index 0000000..e142ca5 --- /dev/null +++ b/contract/lookup-max-retries-zero.json @@ -0,0 +1,23 @@ +{ + "version": 1, + "id": "lookup-max-retries-zero", + "description": "With maxRetries 0, a 503 gives server_error after one attempt.", + "client": {"apiKey": "nipost_test_mock_l3", "maxRetries": 0}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + {"status": 503, "body": {"error": {"code": "unavailable", "message": "try again"}}} + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "server_error", + "status": 503, + "apiCode": "unavailable", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-network-error.json b/contract/lookup-network-error.json new file mode 100644 index 0000000..3548398 --- /dev/null +++ b/contract/lookup-network-error.json @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "lookup-network-error", + "description": "A dropped connection is retried, and the last one gives network_error.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"drop": true}], + "expect": { + "attempts": 3, + "waitsMs": [{"min": 500, "max": 750}, {"min": 1000, "max": 1250}], + "outcomes": [ + {"error": {"code": "network_error", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/lookup-no-key.json b/contract/lookup-no-key.json new file mode 100644 index 0000000..a520d74 --- /dev/null +++ b/contract/lookup-no-key.json @@ -0,0 +1,27 @@ +{ + "version": 1, + "id": "lookup-no-key", + "description": "With no key, the client sends no X-API-Key header, and maps the 401.", + "client": {}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"fixture": "errors/auth-required"}], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/lookup", + "query": {"code": "FC-01-Z99-ZZ-01", "level": "1"}, + "apiKey": null + }, + "outcomes": [ + { + "error": { + "code": "unauthorized", + "status": 401, + "apiCode": "auth_required", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-not-found.json b/contract/lookup-not-found.json new file mode 100644 index 0000000..0ed75e8 --- /dev/null +++ b/contract/lookup-not-found.json @@ -0,0 +1,31 @@ +{ + "version": 1, + "id": "lookup-not-found", + "description": "A postcode that the gateway does not know gives valid false, not an error.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "fc01 z99 zz02", "level": 1}], + "responses": [{"fixture": "lookup/not-found"}], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/lookup", + "query": {"code": "FC-01-Z99-ZZ-02", "level": "1"}, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-02", + "valid": false, + "status": "not_found", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-origin-not-allowed.json b/contract/lookup-origin-not-allowed.json new file mode 100644 index 0000000..93ac4ba --- /dev/null +++ b/contract/lookup-origin-not-allowed.json @@ -0,0 +1,21 @@ +{ + "version": 1, + "id": "lookup-origin-not-allowed", + "description": "A 403 with the API code origin_not_allowed has its own error code.", + "client": {"apiKey": "nipost_pk_test_mock"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"fixture": "errors/origin-not-allowed"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "origin_not_allowed", + "status": 403, + "apiCode": "origin_not_allowed", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-queue-limit.json b/contract/lookup-queue-limit.json new file mode 100644 index 0000000..e5f6cb9 --- /dev/null +++ b/contract/lookup-queue-limit.json @@ -0,0 +1,94 @@ +{ + "version": 1, + "id": "lookup-queue-limit", + "description": "Six calls at once send at most four requests at a time.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "order": "parallel", + "calls": [ + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-02", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-03", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-04", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-05", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-06", "level": 1} + ], + "responses": [{"fixture": "lookup/not-found", "delayMs": 100}], + "expect": { + "attempts": 6, + "maxInFlight": 4, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": false, + "status": "not_found", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-02", + "valid": false, + "status": "not_found", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-03", + "valid": false, + "status": "not_found", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-04", + "valid": false, + "status": "not_found", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-05", + "valid": false, + "status": "not_found", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-06", + "valid": false, + "status": "not_found", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-rate-limited.json b/contract/lookup-rate-limited.json new file mode 100644 index 0000000..c7b5aed --- /dev/null +++ b/contract/lookup-rate-limited.json @@ -0,0 +1,21 @@ +{ + "version": 1, + "id": "lookup-rate-limited", + "description": "A 429 with no Retry-After gives rate_limited at once.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"fixture": "errors/rate-limited"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "rate_limited", + "status": 429, + "apiCode": "rate_limited", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-retries-exhausted.json b/contract/lookup-retries-exhausted.json new file mode 100644 index 0000000..6220d69 --- /dev/null +++ b/contract/lookup-retries-exhausted.json @@ -0,0 +1,24 @@ +{ + "version": 1, + "id": "lookup-retries-exhausted", + "description": "A 503 on every attempt gives server_error after three attempts.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + {"status": 503, "body": {"error": {"code": "unavailable", "message": "try again"}}} + ], + "expect": { + "attempts": 3, + "waitsMs": [{"min": 500, "max": 750}, {"min": 1000, "max": 1250}], + "outcomes": [ + { + "error": { + "code": "server_error", + "status": 503, + "apiCode": "unavailable", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-retry-502.json b/contract/lookup-retry-502.json new file mode 100644 index 0000000..d635fea --- /dev/null +++ b/contract/lookup-retry-502.json @@ -0,0 +1,29 @@ +{ + "version": 1, + "id": "lookup-retry-502", + "description": "A 502 is retried after the first wait, and the second attempt succeeds.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + {"status": 502, "body": {"error": {"code": "unavailable", "message": "try again"}}}, + {"fixture": "lookup/valid-level-1"} + ], + "expect": { + "attempts": 2, + "waitsMs": [{"min": 500, "max": 750}], + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-retry-503.json b/contract/lookup-retry-503.json new file mode 100644 index 0000000..5a3470d --- /dev/null +++ b/contract/lookup-retry-503.json @@ -0,0 +1,29 @@ +{ + "version": 1, + "id": "lookup-retry-503", + "description": "A 503 is retried after the first wait, and the second attempt succeeds.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + {"status": 503, "body": {"error": {"code": "unavailable", "message": "try again"}}}, + {"fixture": "lookup/valid-level-1"} + ], + "expect": { + "attempts": 2, + "waitsMs": [{"min": 500, "max": 750}], + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-retry-504.json b/contract/lookup-retry-504.json new file mode 100644 index 0000000..ca805f7 --- /dev/null +++ b/contract/lookup-retry-504.json @@ -0,0 +1,29 @@ +{ + "version": 1, + "id": "lookup-retry-504", + "description": "A 504 is retried after the first wait, and the second attempt succeeds.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + {"status": 504, "body": {"error": {"code": "unavailable", "message": "try again"}}}, + {"fixture": "lookup/valid-level-1"} + ], + "expect": { + "attempts": 2, + "waitsMs": [{"min": 500, "max": 750}], + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-retry-after-too-long.json b/contract/lookup-retry-after-too-long.json new file mode 100644 index 0000000..286816a --- /dev/null +++ b/contract/lookup-retry-after-too-long.json @@ -0,0 +1,21 @@ +{ + "version": 1, + "id": "lookup-retry-after-too-long", + "description": "A 429 that asks for over 10 seconds gives rate_limited at once, with the wait.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"fixture": "errors/rate-limited", "headers": {"Retry-After": "120"}}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "rate_limited", + "status": 429, + "apiCode": "rate_limited", + "retryAfterMs": 120000 + } + } + ] + } +} diff --git a/contract/lookup-retry-after.json b/contract/lookup-retry-after.json new file mode 100644 index 0000000..a78836a --- /dev/null +++ b/contract/lookup-retry-after.json @@ -0,0 +1,29 @@ +{ + "version": 1, + "id": "lookup-retry-after", + "description": "A 429 with Retry-After of 1 second is retried after exactly that wait.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + {"fixture": "errors/rate-limited", "headers": {"Retry-After": "1"}}, + {"fixture": "lookup/valid-level-1"} + ], + "expect": { + "attempts": 2, + "waitsMs": [{"min": 1000, "max": 1000}], + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-server-error.json b/contract/lookup-server-error.json new file mode 100644 index 0000000..7be92d8 --- /dev/null +++ b/contract/lookup-server-error.json @@ -0,0 +1,23 @@ +{ + "version": 1, + "id": "lookup-server-error", + "description": "A 500 gives server_error, and the client does not retry it.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + {"status": 500, "body": {"error": {"code": "internal", "message": "internal error"}}} + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "server_error", + "status": 500, + "apiCode": "internal", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-shared-request.json b/contract/lookup-shared-request.json new file mode 100644 index 0000000..bb58566 --- /dev/null +++ b/contract/lookup-shared-request.json @@ -0,0 +1,41 @@ +{ + "version": 1, + "id": "lookup-shared-request", + "description": "Two identical calls share one request while it is in flight.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "order": "parallel", + "calls": [ + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}, + {"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1} + ], + "responses": [{"fixture": "lookup/valid-level-1", "delayMs": 100}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-timeout.json b/contract/lookup-timeout.json new file mode 100644 index 0000000..ebcf065 --- /dev/null +++ b/contract/lookup-timeout.json @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "lookup-timeout", + "description": "A request with no answer ends at timeoutMs and is retried, then gives timeout.", + "client": {"apiKey": "nipost_test_mock_l3", "timeoutMs": 200}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [{"hang": true}], + "expect": { + "attempts": 3, + "waitsMs": [{"min": 500, "max": 750}, {"min": 1000, "max": 1250}], + "outcomes": [ + {"error": {"code": "timeout", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/lookup-unknown-fields.json b/contract/lookup-unknown-fields.json new file mode 100644 index 0000000..6134180 --- /dev/null +++ b/contract/lookup-unknown-fields.json @@ -0,0 +1,39 @@ +{ + "version": 1, + "id": "lookup-unknown-fields", + "description": "The client ignores a field that client.md does not list.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + { + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "verified": false, + "new_field": {"any": 1} + }, + "meta": 1 + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-unknown-status.json b/contract/lookup-unknown-status.json new file mode 100644 index 0000000..01c9c8b --- /dev/null +++ b/contract/lookup-unknown-status.json @@ -0,0 +1,30 @@ +{ + "version": 1, + "id": "lookup-unknown-status", + "description": "A status that client.md does not list stays text in the result.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], + "responses": [ + { + "status": 200, + "body": {"data": {"postcode": "FC-01-Z99-ZZ-01", "valid": true, "status": "pending"}} + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "pending", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/reverse-area.json b/contract/reverse-area.json new file mode 100644 index 0000000..3104c95 --- /dev/null +++ b/contract/reverse-area.json @@ -0,0 +1,29 @@ +{ + "version": 1, + "id": "reverse-area", + "description": "found can be true while unit is null, when no unit lies within the radius.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "reverse", "lat": 9.001, "lng": 7.001, "maxDistanceM": 25}], + "responses": [{"fixture": "reverse/area"}], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/search/reverse", + "query": {"lat": "9.001", "lng": "7.001", "max_distance_m": "25"}, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "found": true, + "radiusM": 25, + "unit": null, + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC" + } + } + ] + } +} diff --git a/contract/reverse-invalid-lat.json b/contract/reverse-invalid-lat.json new file mode 100644 index 0000000..ced0538 --- /dev/null +++ b/contract/reverse-invalid-lat.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "reverse-invalid-lat", + "description": "A latitude outside -90 to 90 sends no request.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "reverse", "lat": 91, "lng": 7}], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + {"error": {"code": "invalid_input", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/reverse-invalid-lng.json b/contract/reverse-invalid-lng.json new file mode 100644 index 0000000..70a001b --- /dev/null +++ b/contract/reverse-invalid-lng.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "reverse-invalid-lng", + "description": "A longitude outside -180 to 180 sends no request.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "reverse", "lat": 9, "lng": 181}], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + {"error": {"code": "invalid_input", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/reverse-invalid-max-distance.json b/contract/reverse-invalid-max-distance.json new file mode 100644 index 0000000..bcde1a3 --- /dev/null +++ b/contract/reverse-invalid-max-distance.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "reverse-invalid-max-distance", + "description": "A radius above 250 metres sends no request.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "reverse", "lat": 9, "lng": 7, "maxDistanceM": 251}], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + {"error": {"code": "invalid_input", "status": null, "apiCode": null, "retryAfterMs": null}} + ] + } +} diff --git a/contract/reverse-not-found.json b/contract/reverse-not-found.json new file mode 100644 index 0000000..cc3dec4 --- /dev/null +++ b/contract/reverse-not-found.json @@ -0,0 +1,23 @@ +{ + "version": 1, + "id": "reverse-not-found", + "description": "A point with no postcode in range is a result with found false.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "reverse", "lat": 0, "lng": 0}], + "responses": [{"fixture": "reverse/not-found"}], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "found": false, + "radiusM": 25, + "unit": null, + "area": null, + "district": null, + "state": null + } + } + ] + } +} diff --git a/contract/reverse-plain-decimal.json b/contract/reverse-plain-decimal.json new file mode 100644 index 0000000..565875e --- /dev/null +++ b/contract/reverse-plain-decimal.json @@ -0,0 +1,29 @@ +{ + "version": 1, + "id": "reverse-plain-decimal", + "description": "The client writes a small number as a plain decimal, with no exponent.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "reverse", "lat": 0.0000001, "lng": 7}], + "responses": [{"fixture": "reverse/not-found"}], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/search/reverse", + "query": {"lat": "0.0000001", "lng": "7"}, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "found": false, + "radiusM": 25, + "unit": null, + "area": null, + "district": null, + "state": null + } + } + ] + } +} diff --git a/contract/reverse-unit.json b/contract/reverse-unit.json new file mode 100644 index 0000000..f23717a --- /dev/null +++ b/contract/reverse-unit.json @@ -0,0 +1,37 @@ +{ + "version": 1, + "id": "reverse-unit", + "description": "A reverse result with a unit.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "reverse", "lat": 9, "lng": 7}], + "responses": [{"fixture": "reverse/unit"}], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/search/reverse", + "query": {"lat": "9", "lng": "7"}, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "found": true, + "radiusM": 25, + "unit": { + "postcode": "FC-01-Z99-ZZ-01", + "distanceM": 4.2, + "confidence": "high", + "stateName": null, + "lgaName": null, + "localityName": null, + "address": null + }, + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC" + } + } + ] + } +} diff --git a/contract/reverse-unknown-fields.json b/contract/reverse-unknown-fields.json new file mode 100644 index 0000000..e05ab61 --- /dev/null +++ b/contract/reverse-unknown-fields.json @@ -0,0 +1,54 @@ +{ + "version": 1, + "id": "reverse-unknown-fields", + "description": "The client ignores unknown fields, and maps an unknown confidence to low.", + "client": {"apiKey": "nipost_test_mock_l3"}, + "calls": [{"method": "reverse", "lat": 9, "lng": 7}], + "responses": [ + { + "status": 200, + "body": { + "data": { + "found": true, + "coordinate": [7, 9], + "unit": { + "postcode": "FC-01-Z99-ZZ-01", + "display": "FC 01 Z99 ZZ 01", + "distance_m": 4.2, + "confidence": "very_high", + "floor": 2 + }, + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC", + "depth": "unit", + "radius_m": 25, + "source": "new" + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "found": true, + "radiusM": 25, + "unit": { + "postcode": "FC-01-Z99-ZZ-01", + "distanceM": 4.2, + "confidence": "low", + "stateName": null, + "lgaName": null, + "localityName": null, + "address": null + }, + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC" + } + } + ] + } +} diff --git a/scripts/contract_files.py b/scripts/contract_files.py index f90830c..bce6aa0 100644 --- a/scripts/contract_files.py +++ b/scripts/contract_files.py @@ -1,14 +1,64 @@ +#!/usr/bin/env python3 # SPDX-FileCopyrightText: 2026 The Gatepost authors # SPDX-License-Identifier: Apache-2.0 -"""Read client.md and, from the next change on, the contract scenarios.""" +"""Check the contract scenarios, and run every check of the OpenAPI file and the fixtures. + +contract/README.md defines the format of a scenario, and client.md defines the error codes. +Run python3 scripts/contract_files.py. It prints one line for each problem, and it exits with 1 +when it finds one. +""" from __future__ import annotations import re +import sys from pathlib import Path +from typing import Any + +import gateway_files -ROOT = Path(__file__).resolve().parents[1] +ROOT = gateway_files.ROOT +CONTRACT = ROOT / "contract" CLIENT_DOC = ROOT / "client.md" +TOP_FIELDS = ("version", "id", "description", "client", "calls", "responses", "expect") +CLIENT_OPTIONS = ("apiKey", "timeoutMs", "maxRetries", "cacheTtlMs") +ORDERS = ("parallel", "sequential") +# For each method: the fields that a call needs, and the fields that it can add. +CALL_FIELDS = { + "lookup": ({"code", "level"}, set()), + "reverse": ({"lat", "lng"}, {"maxDistanceM"}), + "autocomplete": ({"q"}, set()), +} +RESPONSE_KINDS = ("body", "text", "fixture", "hang", "drop") +RESPONSE_EXTRAS = ("status", "headers", "delayMs") +EXPECT_FIELDS = {"attempts", "outcomes", "waitsMs", "maxInFlight", "request"} +ERROR_FIELDS = ("code", "status", "apiCode", "retryAfterMs") +REQUEST_FIELDS = ("method", "path", "query", "apiKey") +# The minimum that the design asks for, beside one scenario for each error code: each retry +# rule, Retry-After, the timeout, request sharing, the queue limit, unknown fields and values, +# each rule of input and the plain form of a number. +REQUIRED_SCENARIOS = ( + "lookup-retry-502", + "lookup-retry-503", + "lookup-retry-504", + "lookup-retries-exhausted", + "lookup-network-error", + "lookup-retry-after", + "lookup-rate-limited", + "lookup-timeout", + "autocomplete-timeout", + "lookup-shared-request", + "lookup-queue-limit", + "lookup-unknown-fields", + "reverse-unknown-fields", + "autocomplete-unknown-fields", + "lookup-unknown-status", + "lookup-invalid-level", + "reverse-invalid-lat", + "reverse-invalid-lng", + "reverse-invalid-max-distance", + "reverse-plain-decimal", +) def error_codes(path: Path = CLIENT_DOC) -> list[str]: @@ -16,3 +66,157 @@ def error_codes(path: Path = CLIENT_DOC) -> list[str]: section = path.read_text(encoding="utf-8").partition("\n## Errors\n")[2].partition("\n## ")[0] codes = re.findall(r"^\| [^|]+ \| `([a-z_]+)` \|", section, flags=re.MULTILINE) return list(dict.fromkeys(codes)) + + +def load_scenarios(root: Path = CONTRACT) -> dict[str, Any]: + """Return each scenario by its file name without `.json`.""" + return {path.stem: gateway_files.read_json(path) for path in sorted(root.glob("*.json"))} + + +def call_problems(call: Any) -> list[str]: + """Return the problems of one call.""" + method = call.get("method") if isinstance(call, dict) else None + if method not in CALL_FIELDS: + return [f"A call needs a method: one of {', '.join(CALL_FIELDS)}."] + needed, optional = CALL_FIELDS[method] + fields = set(call) - {"method"} + if not needed <= fields <= needed | optional: + return [f"A {method} call needs {', '.join(sorted(needed))}, and nothing else."] + return [] + + +def response_problems(response: Any, fixtures: set[str]) -> list[str]: + """Return the problems of one response of the mock server.""" + if not isinstance(response, dict): + return ["Each response must be an object."] + kinds = [kind for kind in RESPONSE_KINDS if kind in response] + if len(kinds) != 1 or set(response) - {*RESPONSE_KINDS, *RESPONSE_EXTRAS}: + return [f"A response needs exactly one of {', '.join(RESPONSE_KINDS)}."] + kind = kinds[0] + problems = [] + if kind in ("body", "text") and type(response.get("status")) is not int: + problems.append(f"A {kind} response needs an integer status.") + if kind not in ("body", "text") and "status" in response: + problems.append(f"A {kind} response must not have a status.") + if kind in ("hang", "drop") and (response[kind] is not True or "headers" in response): + problems.append(f"A {kind} response needs the value true, and no headers.") + if kind == "fixture" and response["fixture"] not in fixtures: + problems.append(f"No fixture is named {response['fixture']}.") + delay = response.get("delayMs", 0) + if type(delay) is not int or delay < 0: + problems.append("delayMs must be a whole number of 0 or more.") + return problems + + +def outcome_problems(outcome: Any, codes: list[str]) -> list[str]: + """Return the problems of one outcome.""" + if not isinstance(outcome, dict) or len(outcome) != 1 or set(outcome) - {"result", "error"}: + return ["An outcome needs exactly one of result and error."] + error = outcome.get("error") + if error is None: + return [] + if not isinstance(error, dict) or tuple(error) != ERROR_FIELDS: + return [f"An error needs the fields {', '.join(ERROR_FIELDS)}."] + if error["code"] not in codes: + return [f"The error code {error['code']} is not in client.md."] + return [] + + +def expect_problems(scenario: dict[str, Any], codes: list[str]) -> list[str]: + """Return the problems of the expect object of a scenario.""" + expect = scenario["expect"] + if not isinstance(expect, dict) or not {"attempts", "outcomes"} <= set(expect): + return ["expect needs attempts and outcomes."] + problems = [] + if set(expect) - EXPECT_FIELDS: + problems.append(f"expect has an unknown field: {sorted(set(expect) - EXPECT_FIELDS)}.") + if len(expect["outcomes"]) != len(scenario["calls"]): + problems.append("expect needs one outcome for each call.") + if (expect["attempts"] == 0) != (scenario["responses"] == []): + problems.append("No attempt means no response, and the other way round.") + waits = expect.get("waitsMs") + if waits is not None and (len(scenario["calls"]) != 1 or len(waits) != expect["attempts"] - 1): + problems.append("waitsMs needs one call, and one wait less than the attempts.") + if waits is not None and any(wait["min"] > wait["max"] for wait in waits): + problems.append("Each wait needs a min that is not above its max.") + if "request" in expect and tuple(expect["request"]) != REQUEST_FIELDS: + problems.append(f"request needs the fields {', '.join(REQUEST_FIELDS)}.") + for outcome in expect["outcomes"]: + problems += outcome_problems(outcome, codes) + return problems + + +def scenario_problems(name: str, scenario: Any, fixtures: set[str], codes: list[str]) -> list[str]: + """Return the problems of one scenario, each line led by its file name.""" + if not isinstance(scenario, dict) or not set(TOP_FIELDS) <= set(scenario): + return [f"{name}: a scenario needs the fields {', '.join(TOP_FIELDS)}."] + problems = [] + if scenario["version"] != 1 or scenario["id"] != name: + problems.append("The version must be 1, and the id must equal the file name.") + if set(scenario) - {*TOP_FIELDS, "order"}: + problems.append("The scenario has a field that the format does not name.") + if set(scenario["client"]) - set(CLIENT_OPTIONS): + problems.append(f"client can hold only {', '.join(CLIENT_OPTIONS)}.") + if (scenario.get("order") in ORDERS) != (len(scenario["calls"]) > 1): + problems.append(f"Two or more calls need an order: {', '.join(ORDERS)}.") + for call in scenario["calls"]: + problems += call_problems(call) + for response in scenario["responses"]: + problems += response_problems(response, fixtures) + problems += expect_problems(scenario, codes) + return [f"{name}: {problem}" for problem in problems] + + +def coverage_problems(scenarios: dict[str, Any], codes: list[str]) -> list[str]: + """Return the error codes and the required scenarios that no scenario covers.""" + reached = { + outcome["error"]["code"] + for scenario in scenarios.values() + for outcome in scenario["expect"]["outcomes"] + if "error" in outcome + } + problems = [ + f"No scenario ends with the error code {code}." for code in codes if code not in reached + ] + problems += [ + f"The scenario {name} is missing." for name in REQUIRED_SCENARIOS if name not in scenarios + ] + return problems + + +def all_problems() -> list[str]: + """Return the problems of the OpenAPI file, the fixtures, the keys and the scenarios.""" + openapi = gateway_files.load_openapi() + check = gateway_files.schema_check(openapi) + fixtures = gateway_files.load_fixtures() + scenarios = load_scenarios() + codes = error_codes() + problems = gateway_files.openapi_problems(openapi) + for name, fixture in fixtures.items(): + problems += gateway_files.fixture_problems(name, fixture, check) + used = gateway_files.schemas_in_responses(openapi) + problems += [ + f"Fixture {name} uses {fixture['schema']}, which no response uses." + for name, fixture in fixtures.items() + if fixture.get("schema") not in used + ] + problems += gateway_files.key_problems( + gateway_files.read_json(gateway_files.FIXTURES / "keys.json") + ) + for name, scenario in scenarios.items(): + problems += scenario_problems(name, scenario, set(fixtures), codes) + problems += coverage_problems(scenarios, codes) + files = sorted([*gateway_files.FIXTURES.rglob("*.json"), *CONTRACT.glob("*.json")]) + return problems + gateway_files.postcode_problems(files) + + +def main() -> int: + """Print each problem, and return 1 when there is one.""" + problems = all_problems() + for problem in problems: + sys.stderr.write(problem + "\n") + return 1 if problems else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/tests/test_contract_files.py b/scripts/tests/test_contract_files.py index eb8abd6..841c08a 100644 --- a/scripts/tests/test_contract_files.py +++ b/scripts/tests/test_contract_files.py @@ -1,16 +1,26 @@ # SPDX-FileCopyrightText: 2026 The Gatepost authors # SPDX-License-Identifier: Apache-2.0 -"""Tests for contract_files: client.md lists the error codes, and the standards point to it.""" +"""Tests for contract_files: each scenario keeps the format, and together they cover client.md.""" +import copy import tempfile import unittest from pathlib import Path +from typing import Any import contract_files +import gateway_files +SCENARIOS = contract_files.load_scenarios() +FIXTURE_NAMES = set(gateway_files.load_fixtures()) CODES = contract_files.error_codes() -STANDARDS = contract_files.ROOT / "standards" / "CODING_STANDARDS.md" -CLIENT = contract_files.CLIENT_DOC.read_text(encoding="utf-8") +STANDARDS = gateway_files.ROOT / "standards" / "CODING_STANDARDS.md" + + +def problems_of(name: str, edit: Any) -> list[str]: + scenario = copy.deepcopy(SCENARIOS[name]) + edit(scenario) + return contract_files.scenario_problems(name, scenario, FIXTURE_NAMES, CODES) class ClientDocTest(unittest.TestCase): @@ -31,8 +41,9 @@ def test_lists_the_error_codes_of_the_design_in_the_order_of_the_map(self) -> No ) def test_the_standards_send_the_status_map_to_client_md(self) -> None: - text = STANDARDS.read_text(encoding="utf-8") - err_1 = next(line for line in text.splitlines() if "**ERR-1" in line) + err_1 = next( + line for line in STANDARDS.read_text(encoding="utf-8").splitlines() if "**ERR-1" in line + ) self.assertIn("`spec/client.md`", err_1) def test_reads_the_rows_of_the_error_table_and_no_other_table(self) -> None: @@ -46,28 +57,134 @@ def test_reads_the_rows_of_the_error_table_and_no_other_table(self) -> None: self.assertEqual(contract_files.error_codes(path), ["x_code"]) -class ClientInputAndUnknownValuesTest(unittest.TestCase): - def test_keeps_an_unknown_lookup_status_as_text(self) -> None: - self.assertNotIn("no `status`, or a value that this table does not list", CLIENT) - self.assertIn("any other value of `status` as it is", CLIENT) +class ScenarioFileTest(unittest.TestCase): + def test_each_scenario_has_no_problem(self) -> None: + for name, scenario in SCENARIOS.items(): + with self.subTest(name): + self.assertEqual( + contract_files.scenario_problems(name, scenario, FIXTURE_NAMES, CODES), [] + ) + + def test_the_scenarios_cover_each_error_code_and_each_required_rule(self) -> None: + self.assertEqual(contract_files.coverage_problems(SCENARIOS, CODES), []) + + def test_the_whole_check_finds_no_problem(self) -> None: + self.assertEqual(contract_files.all_problems(), []) + + +class ScenarioProblemTest(unittest.TestCase): + def test_rejects_an_id_that_differs_from_the_file_name(self) -> None: + self.assertEqual( + problems_of("lookup-level-1", lambda scenario: scenario.update(id="lookup-level-one")), + ["lookup-level-1: The version must be 1, and the id must equal the file name."], + ) + + def test_rejects_a_fixture_that_does_not_exist(self) -> None: + def rename(scenario: dict[str, Any]) -> None: + scenario["responses"][0]["fixture"] = "lookup/valid-level-9" + + self.assertEqual( + problems_of("lookup-level-1", rename), + ["lookup-level-1: No fixture is named lookup/valid-level-9."], + ) + + def test_rejects_a_response_with_two_kinds(self) -> None: + def add_drop(scenario: dict[str, Any]) -> None: + scenario["responses"][0]["drop"] = True + + [problem] = problems_of("lookup-level-1", add_drop) + self.assertIn("needs exactly one of body, text, fixture, hang, drop", problem) - def test_rejects_a_bad_level_coordinate_or_distance_before_a_request(self) -> None: - for rule in ( - "A `level` that is not a whole number from 1 to 5", - "A `lat` that is not finite or is outside -90 to 90", - "A `lng` that is not finite or is outside -180 to 180", - "A `maxDistanceM` below 0 or above 250", - ): - self.assertIn(rule, CLIENT) + def test_rejects_a_status_beside_a_fixture(self) -> None: + def add_status(scenario: dict[str, Any]) -> None: + scenario["responses"][0]["status"] = 200 - def test_writes_a_number_in_the_query_as_a_plain_decimal(self) -> None: - self.assertIn("plain decimal, with no exponent", CLIENT) + self.assertEqual( + problems_of("lookup-level-1", add_status), + ["lookup-level-1: A fixture response must not have a status."], + ) - def test_ignores_unknown_fields_and_names_the_scenario(self) -> None: - self.assertIn("A client ignores a field that the tables above do not list", CLIENT) - self.assertIn("`unknown-fields`", CLIENT) + def test_rejects_a_body_with_no_status(self) -> None: + def drop_status(scenario: dict[str, Any]) -> None: + del scenario["responses"][0]["status"] + + self.assertEqual( + problems_of("lookup-server-error", drop_status), + ["lookup-server-error: A body response needs an integer status."], + ) - def test_leaves_the_core_only_rule_out_of_the_mock_server_exceptions(self) -> None: - path = contract_files.ROOT / "standards" / "languages" / "typescript.md" - section = path.read_text(encoding="utf-8").partition("## The mock server")[2] - self.assertNotIn("CS-2", section.partition("## Publishing")[0]) + def test_rejects_an_error_code_that_client_md_does_not_list(self) -> None: + def rename(scenario: dict[str, Any]) -> None: + scenario["expect"]["outcomes"][0]["error"]["code"] = "bad_gateway" + + self.assertEqual( + problems_of("lookup-server-error", rename), + ["lookup-server-error: The error code bad_gateway is not in client.md."], + ) + + def test_rejects_a_missing_outcome(self) -> None: + def drop_outcome(scenario: dict[str, Any]) -> None: + scenario["expect"]["outcomes"].pop() + + self.assertEqual( + problems_of("lookup-shared-request", drop_outcome), + ["lookup-shared-request: expect needs one outcome for each call."], + ) + + def test_rejects_waits_that_do_not_fit_the_attempts(self) -> None: + def add_wait(scenario: dict[str, Any]) -> None: + scenario["expect"]["waitsMs"].append({"min": 2000, "max": 2250}) + + self.assertEqual( + problems_of("lookup-retry-503", add_wait), + ["lookup-retry-503: waitsMs needs one call, and one wait less than the attempts."], + ) + + def test_rejects_two_calls_with_no_order(self) -> None: + self.assertEqual( + problems_of("lookup-shared-request", lambda scenario: scenario.pop("order")), + ["lookup-shared-request: Two or more calls need an order: parallel, sequential."], + ) + + def test_rejects_a_call_with_a_field_of_another_method(self) -> None: + def add_field(scenario: dict[str, Any]) -> None: + scenario["calls"][0]["q"] = "F" + + self.assertEqual( + problems_of("lookup-level-1", add_field), + ["lookup-level-1: A lookup call needs code, level, and nothing else."], + ) + + def test_rejects_a_client_option_that_client_md_does_not_name(self) -> None: + self.assertEqual( + problems_of("lookup-level-1", lambda scenario: scenario["client"].update(retries=1)), + ["lookup-level-1: client can hold only apiKey, timeoutMs, maxRetries, cacheTtlMs."], + ) + + def test_rejects_an_attempt_with_no_response(self) -> None: + self.assertEqual( + problems_of("lookup-level-1", lambda scenario: scenario.update(responses=[])), + ["lookup-level-1: No attempt means no response, and the other way round."], + ) + + +class CoverageTest(unittest.TestCase): + def test_reports_an_error_code_that_no_scenario_reaches(self) -> None: + scenarios = { + name: scenario + for name, scenario in SCENARIOS.items() + if name != "lookup-origin-not-allowed" + } + self.assertEqual( + contract_files.coverage_problems(scenarios, CODES), + ["No scenario ends with the error code origin_not_allowed."], + ) + + def test_reports_a_required_scenario_that_is_missing(self) -> None: + scenarios = { + name: scenario for name, scenario in SCENARIOS.items() if name != "lookup-queue-limit" + } + self.assertEqual( + contract_files.coverage_problems(scenarios, CODES), + ["The scenario lookup-queue-limit is missing."], + ) From 5726eac8381def201037e4a5c68323d61ac96369 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:46:13 +0100 Subject: [PATCH 06/13] fix: state the rules of the client contract that a review found open A review found that the contract left room for clients to differ. The unknown-fields rule was too wide, the form of a number in a query was not fixed, and the contract gave no rule for a reply with a broken part, for a Retry-After on a 5xx reply, for a total deadline, for a shared request that one caller cancels, or for the wait of autocomplete. Each rule now says what a client does, and the tests check that meaning. A reply with a broken part gets the new error code unexpected_response, so that no client returns a guessed value. Signed-off-by: MobileMage --- client.md | 28 ++-- contract/lookup-malformed-body.json | 11 +- scripts/contract_files.py | 2 + scripts/tests/test_client_doc.py | 223 +++++++++++++++++++++++++++ scripts/tests/test_contract_files.py | 1 + 5 files changed, 253 insertions(+), 12 deletions(-) create mode 100644 scripts/tests/test_client_doc.py diff --git a/client.md b/client.md index afccfc2..27e4641 100644 --- a/client.md +++ b/client.md @@ -26,11 +26,11 @@ Each call that sends a request is asynchronous, and the caller can cancel it (AP | `apiKey` | the key for the `X-API-Key` header | none, so the client sends no key | | `baseUrl` | the gateway's address | `https://api.postcode.gov.ng` | | `transport` | the HTTP transport, in the language's standard form | the platform's own | -| `timeoutMs` | the longest wait for one attempt, in milliseconds | 8000 | +| `timeoutMs` | the longest wait for one attempt, in milliseconds | 8000, and 15000 for `autocomplete` | | `maxRetries` | the most retries after the first attempt | 2 | | `cacheTtlMs` | how long the client keeps a result, in milliseconds | 0, which disables the cache | -A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` or `cacheTtlMs`. The default of 8000 is open: some `autocomplete` calls took 10 to 14 seconds, and a timed run must decide it. +A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` or `cacheTtlMs`. ### Lookup result @@ -49,6 +49,7 @@ A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` - `status` is null when the response has no `status`. The client returns any other value of `status` as it is, also when this table does not list it, so that a new gateway status is not lost. - `levelReceived` comes from the fields of the response. It is 5 when `point_geometry` is present, 4 for `other_building_info`, 3 for `building_use_status`, and 2 for either address field. Otherwise it is 1. - `recentHouseAddress` is the text in `recent_house_address.recent`. +- A `status` that is present, not null and not text raises `unexpected_response`. - A response with `valid: false` is a result, not an error. ### Reverse result @@ -93,7 +94,7 @@ A suggestion has these fields: The gateway sends the value of one segment only. For the text `FC01Z`, the suggestion `Z99` gives the postcode `FC-01-Z99`. The client parses that text with partial postcodes allowed. When it does not parse, `postcode` is null. -A client ignores a field that the tables above do not list, in a lookup, a reverse or an autocomplete response. Such a field never changes the result and never raises an error. The scenario `unknown-fields` tests this rule for each of the three calls. +A client ignores a field that this contract does not use, in a lookup, a reverse or an autocomplete response. Such a field never changes the result and never raises an error. The scenarios `lookup-unknown-fields`, `reverse-unknown-fields` and `autocomplete-unknown-fields` test this rule, one for each call. The scenario `lookup-unknown-status` tests that an unknown `status` stays text. ## Requests @@ -101,11 +102,11 @@ A client ignores a field that the tables above do not list, in a lookup, a rever - `lookup` parses the code with the core first. A code that does not parse raises `invalid_input`, and the client sends no request. Partial postcodes do not pass. The request sends the canonical form and the level: `code=FC-01-Z99-ZZ-01&level=1`. - `reverse` sends `lat` and `lng`, and `max_distance_m` when the caller gave a radius. - Each of these raises `invalid_input`, and the client sends no request: - - A `level` that is not a whole number from 1 to 5. - - A `lat` that is not finite or is outside -90 to 90. - - A `lng` that is not finite or is outside -180 to 180. - - A `maxDistanceM` below 0 or above 250. The OpenAPI file gives the range 0 to 250. -- The client writes each number in the query as a plain decimal, with no exponent: `lat=9.0`, not `lat=9e0`. Languages print a number in different forms, such as `1e-7`. One plain form gives every client the same query for the same input, and the mock server can match it. + - A `level` that is not a whole number from 1 to 5. The scenario `lookup-invalid-level` tests it. + - A `lat` that is not finite or is outside -90 to 90. The scenario `reverse-invalid-lat` tests it. + - A `lng` that is not finite or is outside -180 to 180. The scenario `reverse-invalid-lng` tests it. + - A `maxDistanceM` below 0 or above 250. The OpenAPI file gives the range 0 to 250. The gateway cuts a larger value to 250 without a sign, so the client rejects it. It does not return a result for another radius. The scenario `reverse-invalid-max-distance` tests it. +- The client writes each number in the query in its shortest decimal form, with no exponent and no trailing zero: `lat=9`, not `lat=9.0` and not `lat=9e0`. A negative zero is written `0`. A small value is written in full: `0.0000001`, not `1e-7`. Languages print numbers in different forms, so one fixed form gives every client the same query for the same input. The mock server compares numbers by value, so it also accepts `9.0`. The scenarios `reverse-plain-decimal` and `reverse-whole-number` test this rule. - `autocomplete` normalises its text with the core. Empty text, text of more than 11 characters, and text with a character other than A to Z and 0 to 9 raise `invalid_input`, and the client sends no request. The gateway does not answer an empty `q`. The request sends the normalised text: `q=FC01Z`. ## Errors @@ -132,11 +133,12 @@ The error code comes from the first row that matches. | status 502, 503 or 504 | `server_error` | yes | | any other status from 400 to 499 | `invalid_input` | no | | any other status that is not 200 | `server_error` | no | -| status 200, with a body that is not a JSON object with a `data` field | `server_error` | no | +| status 200, with a body that is not a JSON object with a `data` field, or with a part that this contract uses missing or of the wrong type | `unexpected_response` | no | | no response, because the connection failed | `network_error` | yes | | no response within `timeoutMs` | `timeout` | yes, except for `autocomplete` | - `apiCode` is set when the body is a JSON object whose `error.code` is text. The client never raises an error for an unexpected error body. +- A part that is missing or has the wrong type gives `unexpected_response`. These parts are: `valid` of a lookup, `found` of a reverse, `suggestions` of an autocomplete, a `status` that is present, not null and not text, and a postcode of a reverse result that the core cannot parse. The client never returns a guessed empty value in their place. The scenarios `lookup-malformed-body`, `lookup-missing-valid`, `lookup-status-not-text`, `reverse-missing-found` and `autocomplete-suggestions-not-list` test this rule. - A cancelled call ends with the platform's own cancellation error, not with `PostcodeError` (API-11). - No error message holds the API key (ERR-3). @@ -146,12 +148,16 @@ The error code comes from the first row that matches. - Before retry `n`, the first being 1, the client waits `500 x 2^(n - 1)` milliseconds, plus a random wait from 0 to 250 milliseconds. The first retry waits 500 to 750 ms, and the second waits 1000 to 1250 ms. - A 429 with `Retry-After` of 10 seconds or less: the client waits that long, with no random part, and retries. `Retry-After` can hold seconds or an HTTP date. - A 429 with a longer `Retry-After`: the client raises `rate_limited` at once, and `retryAfterMs` holds the wait. A caller can then tell a user when to try again. -- A 429 with no `Retry-After`: the client raises `rate_limited` at once. The gateway counts requests in each clock minute, so a retry after half a second would fail too. +- A 429 with no `Retry-After`, or with an invalid one: the client raises `rate_limited` at once. The gateway counts requests in each clock minute, so a retry after half a second would fail too. +- A 502, 503 or 504 with a valid `Retry-After` of 10 seconds or less: the client waits that long, with no random part, as for a 429. A valid `Retry-After` is a whole number of seconds from 0 up, or an HTTP date that is not in the past. With any other value, or with no header, the client uses its own wait. `retryAfterMs` holds a valid value. The scenarios `lookup-retry-after-503` and `lookup-retry-after-invalid` test this rule. +- A call has a total deadline. It starts when the client sends the first attempt. It lasts `(1 + maxRetries) x timeoutMs`, plus `500 x 2^(n - 1) + 250` milliseconds for each retry `n`. With the defaults for `lookup`, that is 26 seconds. When the next wait would end after the deadline, the client raises the error of the last attempt at once. The scenario `lookup-deadline` tests this rule. - The client does not retry a timeout of `autocomplete`. The next keystroke replaces the call. ## Sharing, queue and cache - Two calls of the same method with the same arguments share one request while the first one is in flight. Both callers get the same result or the same error. +- The key of a shared request and of a cache entry is the method with the canonical form of the arguments. So `lookup("fc01z99zz01")` and `lookup("FC-01-Z99-ZZ-01")` share one request and one cache entry. The scenario `lookup-shared-spellings` tests the request. +- A shared request stops only when every caller that shares it cancels. A cancelled call frees its place among the 4 at once. - A client sends at most 4 requests at a time. Further calls wait in a queue, in the order of the calls. - With `cacheTtlMs` above 0, the client keeps each result for that long. A call with the same method and the same arguments then gets the kept result, and the client sends no request. Errors stay out of the cache. `clearCache()` removes every kept result. @@ -162,5 +168,7 @@ Each client tests these rules itself. - The programmer errors in the client options. - Cancellation (API-11), because each language cancels in its own way. - That no error message holds the API key. +- The cancellation rules of sharing and the queue: a shared request stops only when every caller cancels, and a cancelled call frees its place among the 4. +- The default `timeoutMs` of 15 seconds for `autocomplete`, because a scenario would wait that long. - The random part of each wait, beyond its bounds. - A `Retry-After` that holds an HTTP date. A scenario cannot hold a date that is a few seconds ahead. diff --git a/contract/lookup-malformed-body.json b/contract/lookup-malformed-body.json index b6528ff..5ec1ae7 100644 --- a/contract/lookup-malformed-body.json +++ b/contract/lookup-malformed-body.json @@ -1,7 +1,7 @@ { "version": 1, "id": "lookup-malformed-body", - "description": "A 200 whose body is not JSON gives server_error, with no retry.", + "description": "A 200 whose body is not JSON gives unexpected_response, with no retry.", "client": {"apiKey": "nipost_test_mock_l3"}, "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 1}], "responses": [ @@ -14,7 +14,14 @@ "expect": { "attempts": 1, "outcomes": [ - {"error": {"code": "server_error", "status": 200, "apiCode": null, "retryAfterMs": null}} + { + "error": { + "code": "unexpected_response", + "status": 200, + "apiCode": null, + "retryAfterMs": null + } + } ] } } diff --git a/scripts/contract_files.py b/scripts/contract_files.py index bce6aa0..49e0642 100644 --- a/scripts/contract_files.py +++ b/scripts/contract_files.py @@ -64,6 +64,8 @@ def error_codes(path: Path = CLIENT_DOC) -> list[str]: """Return the error codes in the table of the Errors section of client.md, in order.""" section = path.read_text(encoding="utf-8").partition("\n## Errors\n")[2].partition("\n## ")[0] + if not section.strip(): + raise ValueError(f"{path} has no Errors section.") codes = re.findall(r"^\| [^|]+ \| `([a-z_]+)` \|", section, flags=re.MULTILINE) return list(dict.fromkeys(codes)) diff --git a/scripts/tests/test_client_doc.py b/scripts/tests/test_client_doc.py new file mode 100644 index 0000000..f826474 --- /dev/null +++ b/scripts/tests/test_client_doc.py @@ -0,0 +1,223 @@ +# SPDX-FileCopyrightText: 2026 The Gatepost authors +# SPDX-License-Identifier: Apache-2.0 +"""Tests for client.md: each rule says what a client does, and the standards point to it.""" + +import re +import tempfile +import unittest +from pathlib import Path + +import contract_files + +CLIENT = contract_files.CLIENT_DOC.read_text(encoding="utf-8") +TYPESCRIPT = contract_files.ROOT / "standards" / "languages" / "typescript.md" +STANDARDS = contract_files.ROOT / "standards" / "CODING_STANDARDS.md" + + +def section(text: str, heading: str) -> str: + """Return the text under a heading, up to the next heading of the same or a higher level.""" + level = heading.split(" ", maxsplit=1)[0] + body = text.partition(f"\n{heading}\n")[2] + return re.split(rf"\n#{{1,{len(level)}}} ", body)[0] + + +def bullets(text: str, lead: str) -> list[str]: + """Return the sub-bullets under the bullet that starts with lead.""" + lines = text.splitlines() + start = next(i for i, line in enumerate(lines) if line.startswith(f"- {lead}")) + found = [] + for line in lines[start + 1 :]: + if not line.startswith(" - "): + break + found.append(line[4:]) + return found + + +def row(text: str, first_cell: str) -> list[str]: + """Return the cells of the table row whose first cell starts with first_cell.""" + line = next(x for x in text.splitlines() if x.startswith(f"| {first_cell}")) + return [cell.strip() for cell in line.strip("|").split("|")] + + +class LookupResultTest(unittest.TestCase): + def test_a_status_is_text_and_an_unknown_value_stays_as_it_is(self) -> None: + part = section(CLIENT, "### Lookup result") + cells = row(part, "`status`") + self.assertTrue(cells[1].startswith("text, or null")) + self.assertIn("The known values are", cells[1]) + bullet = next(x for x in part.splitlines() if x.startswith("- `status` is null")) + self.assertIn("returns any other value of `status` as it is", bullet) + self.assertNotIn("is null when the response has no `status`, or", bullet) + + def test_a_status_that_is_not_text_is_an_unexpected_response(self) -> None: + part = section(CLIENT, "### Lookup result") + bullet = next(x for x in part.splitlines() if "and not text" in x) + self.assertIn("`unexpected_response`", bullet) + + +class RequestRulesTest(unittest.TestCase): + requests = section(CLIENT, "## Requests") + + def test_a_bad_input_value_raises_invalid_input_before_any_request(self) -> None: + lead = "Each of these raises `invalid_input`, and the client sends no request:" + self.assertEqual( + [re.sub(r" The .*", "", rule) for rule in bullets(self.requests, lead)], + [ + "A `level` that is not a whole number from 1 to 5.", + "A `lat` that is not finite or is outside -90 to 90.", + "A `lng` that is not finite or is outside -180 to 180.", + "A `maxDistanceM` below 0 or above 250.", + ], + ) + + def test_a_distance_over_250_is_rejected_because_the_gateway_would_clamp_it(self) -> None: + lead = "Each of these raises `invalid_input`, and the client sends no request:" + rule = bullets(self.requests, lead)[3] + self.assertIn("The gateway cuts a larger value to 250 without a sign", rule) + self.assertNotIn("is clamped", self.requests) + + def test_each_input_rule_names_its_scenario(self) -> None: + lead = "Each of these raises `invalid_input`, and the client sends no request:" + rules = bullets(self.requests, lead) + names = [re.findall(r"`([a-z-]+-invalid-[a-z-]+)`", rule) for rule in rules] + self.assertEqual( + names, + [ + ["lookup-invalid-level"], + ["reverse-invalid-lat"], + ["reverse-invalid-lng"], + ["reverse-invalid-max-distance"], + ], + ) + + def test_a_number_is_written_in_its_shortest_decimal_form(self) -> None: + bullet = next(x for x in self.requests.splitlines() if "shortest decimal form" in x) + for text in ( + "with no exponent and no trailing zero", + "`lat=9`, not `lat=9.0`", + "A negative zero is written `0`", + "`0.0000001`, not `1e-7`", + "The mock server compares numbers by value", + "`reverse-plain-decimal`", + "`reverse-whole-number`", + ): + self.assertIn(text, bullet) + + +class UnknownFieldsTest(unittest.TestCase): + def test_a_field_that_the_contract_does_not_use_never_changes_a_result(self) -> None: + paragraph = next(x for x in CLIENT.split("\n\n") if x.startswith("A client ignores")) + self.assertIn("a field that this contract does not use", paragraph) + self.assertIn("never changes the result and never raises an error", paragraph) + for call in ("lookup", "reverse", "autocomplete"): + self.assertIn(f"`{call}-unknown-fields`", paragraph) + self.assertIn("`lookup-unknown-status`", paragraph) + + +class ErrorRulesTest(unittest.TestCase): + errors = section(CLIENT, "## Errors") + + def test_each_condition_gives_its_error_code(self) -> None: + table = [ + [cell.strip() for cell in line.strip("|").split("|")] + for line in self.errors.splitlines() + if re.match(r"^\| [^|]+ \| `[a-z_]+` \|", line) + ] + self.assertEqual( + {cells[0]: cells[1] for cells in table}, + { + "the client's own check of the input fails": "`invalid_input`", + "status 401": "`unauthorized`", + "status 402": "`insufficient_credits`", + "status 403, with the API code `origin_not_allowed`": "`origin_not_allowed`", + "status 403, with any other API code": "`forbidden`", + "status 429": "`rate_limited`", + "status 502, 503 or 504": "`server_error`", + "any other status from 400 to 499": "`invalid_input`", + "any other status that is not 200": "`server_error`", + "status 200, with a body that is not a JSON object with a `data` field, " + "or with a part that this contract uses missing or of the wrong type": ( + "`unexpected_response`" + ), + "no response, because the connection failed": "`network_error`", + "no response within `timeoutMs`": "`timeout`", + }, + ) + + def test_a_broken_reply_gives_no_guessed_value(self) -> None: + text = self.errors + self.assertIn("The client never returns a guessed empty value", text) + + +class RetryRulesTest(unittest.TestCase): + retries = section(CLIENT, "## Retries") + + def test_a_call_has_a_total_deadline(self) -> None: + bullet = next(x for x in self.retries.splitlines() if "total deadline" in x) + self.assertIn("starts when the client sends the first attempt", bullet) + self.assertIn("`(1 + maxRetries) x timeoutMs`", bullet) + self.assertIn("`500 x 2^(n - 1) + 250`", bullet) + self.assertIn("raises the error of the last attempt at once", bullet) + self.assertIn("`lookup-deadline`", bullet) + + def test_a_retry_after_on_a_5xx_is_used_when_valid_and_short(self) -> None: + bullet = next(x for x in self.retries.splitlines() if x.startswith("- A 502, 503")) + self.assertIn("valid `Retry-After` of 10 seconds or less", bullet) + self.assertIn("with no random part", bullet) + self.assertIn("a whole number of seconds from 0 up, or an HTTP date", bullet) + self.assertIn("its own wait", bullet) + self.assertIn("`lookup-retry-after-503`", bullet) + self.assertIn("`lookup-retry-after-invalid`", bullet) + + def test_autocomplete_waits_15_seconds_and_the_other_calls_8(self) -> None: + options = section(CLIENT, "### Client options") + cells = row(options, "`timeoutMs`") + self.assertEqual(cells[2], "8000, and 15000 for `autocomplete`") + self.assertNotIn("is open", options) + + +class SharingRulesTest(unittest.TestCase): + sharing = section(CLIENT, "## Sharing, queue and cache") + + def test_two_spellings_of_one_postcode_share_one_request(self) -> None: + bullet = next(x for x in self.sharing.splitlines() if "canonical form" in x) + self.assertIn('`lookup("fc01z99zz01")` and `lookup("FC-01-Z99-ZZ-01")`', bullet) + self.assertIn("share one request and one cache entry", bullet) + self.assertIn("`lookup-shared-spellings`", bullet) + + def test_a_shared_request_stops_only_when_every_caller_cancels(self) -> None: + bullet = next(x for x in self.sharing.splitlines() if "cancels" in x) + self.assertIn("only when every caller that shares it cancels", bullet) + + def test_a_cancelled_call_frees_its_place_among_the_four_at_once(self) -> None: + self.assertIn("A cancelled call frees its place among the 4 at once", self.sharing) + + def test_the_rules_that_only_a_client_can_test_are_listed(self) -> None: + untested = section(CLIENT, "## Rules that no scenario tests") + for text in ("shared request", "frees its place", "15 seconds"): + self.assertIn(text, untested) + + +class StandardsTest(unittest.TestCase): + def test_the_ranking_line_and_api_1_send_the_reader_to_client_md(self) -> None: + lines = STANDARDS.read_text(encoding="utf-8").splitlines() + ranking = next(x for x in lines if "rank above this document" in x) + api_1 = next(x for x in lines if x.startswith("- **API-1")) + self.assertIn("`spec/client.md`", ranking) + self.assertIn("`spec/client.md` wins over a scenario", ranking) + self.assertIn("`spec/client.md`", api_1) + + def test_the_mock_server_section_does_not_list_the_core_only_rule(self) -> None: + text = TYPESCRIPT.read_text(encoding="utf-8") + mock = section(text, "## The mock server") + self.assertIn("These rules do not apply to it:", mock) + self.assertNotIn("CS-2", mock) + + +class ErrorTableReaderTest(unittest.TestCase): + def test_a_file_with_no_errors_section_is_an_error(self) -> None: + with tempfile.TemporaryDirectory() as folder: + path = Path(folder) / "client.md" + path.write_text("# Client\n\n## Requests\n\n| a | `x` | no |\n", encoding="utf-8") + with self.assertRaises(ValueError): + contract_files.error_codes(path) diff --git a/scripts/tests/test_contract_files.py b/scripts/tests/test_contract_files.py index 841c08a..c48eaa8 100644 --- a/scripts/tests/test_contract_files.py +++ b/scripts/tests/test_contract_files.py @@ -35,6 +35,7 @@ def test_lists_the_error_codes_of_the_design_in_the_order_of_the_map(self) -> No "forbidden", "rate_limited", "server_error", + "unexpected_response", "network_error", "timeout", ], From f926a2ba8a65593907f754092f3e7215c08ab657 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:47:15 +0100 Subject: [PATCH 07/13] fix: add scenarios for the new rules of the client contract Each rule that a scenario can test now has one: the total deadline, a Retry-After on a 503, an invalid Retry-After, two spellings of one postcode, a whole number in a query, and the four replies with a missing or broken part. A test checks that every scenario that client.md names exists, and the required list covers the new ones. Signed-off-by: MobileMage --- .../autocomplete-suggestions-not-list.json | 38 +++++++++++++ contract/lookup-deadline.json | 44 +++++++++++++++ contract/lookup-missing-valid.json | 38 +++++++++++++ contract/lookup-retry-after-503.json | 55 ++++++++++++++++++ contract/lookup-retry-after-invalid.json | 55 ++++++++++++++++++ contract/lookup-shared-spellings.json | 56 +++++++++++++++++++ contract/lookup-status-not-text.json | 40 +++++++++++++ contract/reverse-missing-found.json | 38 +++++++++++++ contract/reverse-whole-number.json | 44 +++++++++++++++ scripts/contract_files.py | 9 +++ scripts/tests/test_client_doc.py | 8 +++ 11 files changed, 425 insertions(+) create mode 100644 contract/autocomplete-suggestions-not-list.json create mode 100644 contract/lookup-deadline.json create mode 100644 contract/lookup-missing-valid.json create mode 100644 contract/lookup-retry-after-503.json create mode 100644 contract/lookup-retry-after-invalid.json create mode 100644 contract/lookup-shared-spellings.json create mode 100644 contract/lookup-status-not-text.json create mode 100644 contract/reverse-missing-found.json create mode 100644 contract/reverse-whole-number.json diff --git a/contract/autocomplete-suggestions-not-list.json b/contract/autocomplete-suggestions-not-list.json new file mode 100644 index 0000000..711fdf8 --- /dev/null +++ b/contract/autocomplete-suggestions-not-list.json @@ -0,0 +1,38 @@ +{ + "version": 1, + "id": "autocomplete-suggestions-not-list", + "description": "A 200 whose suggestions are not a list gives unexpected_response.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "autocomplete", + "q": "f" + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "segment": "state", + "suggestions": "FC" + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "unexpected_response", + "status": 200, + "apiCode": null, + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-deadline.json b/contract/lookup-deadline.json new file mode 100644 index 0000000..b3328a3 --- /dev/null +++ b/contract/lookup-deadline.json @@ -0,0 +1,44 @@ +{ + "version": 1, + "id": "lookup-deadline", + "description": "A wait past the total deadline ends the call at once, with the last error.", + "client": { + "apiKey": "nipost_test_mock_l3", + "timeoutMs": 200, + "maxRetries": 1 + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "status": 503, + "body": { + "error": { + "code": "unavailable", + "message": "try again" + } + }, + "headers": { + "Retry-After": "2" + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "server_error", + "status": 503, + "apiCode": "unavailable", + "retryAfterMs": 2000 + } + } + ] + } +} diff --git a/contract/lookup-missing-valid.json b/contract/lookup-missing-valid.json new file mode 100644 index 0000000..cbb4e61 --- /dev/null +++ b/contract/lookup-missing-valid.json @@ -0,0 +1,38 @@ +{ + "version": 1, + "id": "lookup-missing-valid", + "description": "A 200 with no valid field gives unexpected_response, with no retry.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01" + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "unexpected_response", + "status": 200, + "apiCode": null, + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-retry-after-503.json b/contract/lookup-retry-after-503.json new file mode 100644 index 0000000..003fe6e --- /dev/null +++ b/contract/lookup-retry-after-503.json @@ -0,0 +1,55 @@ +{ + "version": 1, + "id": "lookup-retry-after-503", + "description": "A 503 with Retry-After of 1 second is retried after exactly that wait.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "status": 503, + "body": { + "error": { + "code": "unavailable", + "message": "try again" + } + }, + "headers": { + "Retry-After": "1" + } + }, + { + "fixture": "lookup/valid-level-1" + } + ], + "expect": { + "attempts": 2, + "waitsMs": [ + { + "min": 1000, + "max": 1000 + } + ], + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-retry-after-invalid.json b/contract/lookup-retry-after-invalid.json new file mode 100644 index 0000000..f6959bb --- /dev/null +++ b/contract/lookup-retry-after-invalid.json @@ -0,0 +1,55 @@ +{ + "version": 1, + "id": "lookup-retry-after-invalid", + "description": "A 503 with an invalid Retry-After is retried after the client's own wait.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "status": 503, + "body": { + "error": { + "code": "unavailable", + "message": "try again" + } + }, + "headers": { + "Retry-After": "soon" + } + }, + { + "fixture": "lookup/valid-level-1" + } + ], + "expect": { + "attempts": 2, + "waitsMs": [ + { + "min": 500, + "max": 750 + } + ], + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-shared-spellings.json b/contract/lookup-shared-spellings.json new file mode 100644 index 0000000..c637225 --- /dev/null +++ b/contract/lookup-shared-spellings.json @@ -0,0 +1,56 @@ +{ + "version": 1, + "id": "lookup-shared-spellings", + "description": "Two spellings of one postcode share one request while it is in flight.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "order": "parallel", + "calls": [ + { + "method": "lookup", + "code": "fc01z99zz01", + "level": 1 + }, + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "fixture": "lookup/valid-level-1", + "delayMs": 100 + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + }, + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-status-not-text.json b/contract/lookup-status-not-text.json new file mode 100644 index 0000000..c206127 --- /dev/null +++ b/contract/lookup-status-not-text.json @@ -0,0 +1,40 @@ +{ + "version": 1, + "id": "lookup-status-not-text", + "description": "A 200 with a status that is not text gives unexpected_response.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": 5 + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "unexpected_response", + "status": 200, + "apiCode": null, + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/reverse-missing-found.json b/contract/reverse-missing-found.json new file mode 100644 index 0000000..de792ec --- /dev/null +++ b/contract/reverse-missing-found.json @@ -0,0 +1,38 @@ +{ + "version": 1, + "id": "reverse-missing-found", + "description": "A 200 with no found field gives unexpected_response.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "reverse", + "lat": 9, + "lng": 7 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "radius_m": 25 + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "unexpected_response", + "status": 200, + "apiCode": null, + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/reverse-whole-number.json b/contract/reverse-whole-number.json new file mode 100644 index 0000000..ddc93b8 --- /dev/null +++ b/contract/reverse-whole-number.json @@ -0,0 +1,44 @@ +{ + "version": 1, + "id": "reverse-whole-number", + "description": "The client writes a whole number with no trailing zero, so 9.0 becomes 9.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "reverse", + "lat": 9.0, + "lng": 7.0 + } + ], + "responses": [ + { + "fixture": "reverse/not-found" + } + ], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/search/reverse", + "query": { + "lat": "9", + "lng": "7" + }, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "found": false, + "radiusM": 25, + "unit": null, + "area": null, + "district": null, + "state": null + } + } + ] + } +} diff --git a/scripts/contract_files.py b/scripts/contract_files.py index 49e0642..364f9c5 100644 --- a/scripts/contract_files.py +++ b/scripts/contract_files.py @@ -58,6 +58,15 @@ "reverse-invalid-lng", "reverse-invalid-max-distance", "reverse-plain-decimal", + "reverse-whole-number", + "lookup-deadline", + "lookup-retry-after-503", + "lookup-retry-after-invalid", + "lookup-shared-spellings", + "lookup-missing-valid", + "lookup-status-not-text", + "reverse-missing-found", + "autocomplete-suggestions-not-list", ) diff --git a/scripts/tests/test_client_doc.py b/scripts/tests/test_client_doc.py index f826474..e678e73 100644 --- a/scripts/tests/test_client_doc.py +++ b/scripts/tests/test_client_doc.py @@ -221,3 +221,11 @@ def test_a_file_with_no_errors_section_is_an_error(self) -> None: path.write_text("# Client\n\n## Requests\n\n| a | `x` | no |\n", encoding="utf-8") with self.assertRaises(ValueError): contract_files.error_codes(path) + + +class ScenarioNamesTest(unittest.TestCase): + def test_each_scenario_that_client_md_names_exists(self) -> None: + scenarios = contract_files.load_scenarios() + names = set(re.findall(r"`((?:lookup|reverse|autocomplete)-[a-z0-9-]+)`", CLIENT)) + self.assertGreaterEqual(len(names), 19) + self.assertEqual(sorted(names - set(scenarios)), []) From 39659dfbb2ba256653ee06931bd15f862fcaf83c Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:56:34 +0100 Subject: [PATCH 08/13] fix: mark each gateway response and error code by its own evidence One mark on a shared response claimed more than the calls showed. The calls without a key reached lookup and autocomplete only, a 403 came back from lookup only, and no call sent an unknown key. Each operation now has its own mark for each status, and each error code has its own mark, so a reader can tell the observed code from the assumed ones. A check fails when a code claims more than its response, and when an error response or a header has no mark. The file now also describes the 5xx replies and the answer to an unknown path, both assumed, so that every reply the mock server can send has a mark. Signed-off-by: MobileMage --- openapi/gateway.completed.yaml | 147 +++++++++++++++++++++++----- scripts/gateway_files.py | 76 +++++++++++--- scripts/tests/test_gateway_files.py | 111 ++++++++++++++++++++- 3 files changed, 290 insertions(+), 44 deletions(-) diff --git a/openapi/gateway.completed.yaml b/openapi/gateway.completed.yaml index 404b996..614002e 100644 --- a/openapi/gateway.completed.yaml +++ b/openapi/gateway.completed.yaml @@ -7,12 +7,15 @@ info: summary: Unofficial. Not made or endorsed by NIPOST. description: >- Gatepost wrote this description of NIPOST's public postcode gateway. It adds the responses, - error codes and headers that NIPOST's OpenAPI file 0.1.0 leaves out. Each response carries - x-gatepost-evidence. "observed" means that Gatepost saw the shape in its own calls on - 2 Oct 2026. "documented" means that NIPOST's docs describe it, but no call showed it. - "assumed" means that neither source shows it, and Gatepost's mock server chose it. - Gatepost's calls used a key with lookup level 1 only, so every lookup body above level 1 - is documented, not observed. + error codes and headers that NIPOST's OpenAPI file 0.1.0 leaves out. Each response and each + header carries x-gatepost-evidence. "observed" means that Gatepost saw the shape in its own + calls on 2 Oct 2026. "documented" means that NIPOST's docs describe it, but no call showed + it. "assumed" means that neither source shows it, and Gatepost's mock server chose it. An + error response lists its error codes in x-gatepost-error-codes, each with its own mark, and + a code never claims more than its response. Gatepost's calls used a key with lookup level 1 + only, so every lookup body above level 1 is documented, not observed. The calls without a + key reached lookup and autocomplete only. x-gatepost-unknown-path describes the answer to a + path that this file does not list. license: name: Apache-2.0 identifier: Apache-2.0 @@ -46,7 +49,9 @@ paths: - name: code in: query required: true - description: The postcode, in any form. The gateway ignores spaces, hyphens and case. + description: >- + The postcode, in any form. NIPOST's docs say that the gateway ignores spaces, hyphens + and case. Calls showed hyphens and lower case. No call sent spaces. schema: { type: string } - name: level in: query @@ -65,10 +70,11 @@ paths: application/json: schema: { $ref: "#/components/schemas/LookupEnvelope" } "400": { $ref: "#/components/responses/BadRequest" } - "401": { $ref: "#/components/responses/Unauthorized" } + "401": { $ref: "#/components/responses/UnauthorizedSeen" } "402": { $ref: "#/components/responses/PaymentRequired" } - "403": { $ref: "#/components/responses/Forbidden" } + "403": { $ref: "#/components/responses/LookupForbidden" } "429": { $ref: "#/components/responses/TooManyRequests" } + "5XX": { $ref: "#/components/responses/ServerError" } /v1/search/reverse: get: operationId: reverse @@ -77,12 +83,16 @@ paths: - name: lat in: query required: true - description: The latitude in degrees. A client rejects a value outside this range. + description: >- + The latitude in degrees. The range is Gatepost's rule, not NIPOST's: a client rejects + a value outside it. No call showed what the gateway does with such a value. schema: { type: number, minimum: -90, maximum: 90 } - name: lng in: query required: true - description: The longitude in degrees. A client rejects a value outside this range. + description: >- + The longitude in degrees. The range is Gatepost's rule, not NIPOST's: a client + rejects a value outside it. No call showed what the gateway does with such a value. schema: { type: number, minimum: -180, maximum: 180 } - name: max_distance_m in: query @@ -105,6 +115,7 @@ paths: "401": { $ref: "#/components/responses/Unauthorized" } "403": { $ref: "#/components/responses/Forbidden" } "429": { $ref: "#/components/responses/TooManyRequests" } + "5XX": { $ref: "#/components/responses/ServerError" } /v1/search/nearby: get: operationId: nearby @@ -113,12 +124,16 @@ paths: - name: lat in: query required: true - description: The latitude in degrees. A client rejects a value outside this range. + description: >- + The latitude in degrees. The range is Gatepost's rule, not NIPOST's: a client rejects + a value outside it. No call showed what the gateway does with such a value. schema: { type: number, minimum: -90, maximum: 90 } - name: lng in: query required: true - description: The longitude in degrees. A client rejects a value outside this range. + description: >- + The longitude in degrees. The range is Gatepost's rule, not NIPOST's: a client + rejects a value outside it. No call showed what the gateway does with such a value. schema: { type: number, minimum: -180, maximum: 180 } - name: radius in: query @@ -140,6 +155,7 @@ paths: "401": { $ref: "#/components/responses/Unauthorized" } "403": { $ref: "#/components/responses/Forbidden" } "429": { $ref: "#/components/responses/TooManyRequests" } + "5XX": { $ref: "#/components/responses/ServerError" } /v1/search/autocomplete: get: operationId: autocomplete @@ -163,9 +179,10 @@ paths: application/json: schema: { $ref: "#/components/schemas/AutocompleteEnvelope" } "400": { $ref: "#/components/responses/BadRequest" } - "401": { $ref: "#/components/responses/Unauthorized" } + "401": { $ref: "#/components/responses/UnauthorizedSeen" } "403": { $ref: "#/components/responses/Forbidden" } "429": { $ref: "#/components/responses/TooManyRequests" } + "5XX": { $ref: "#/components/responses/ServerError" } /v1/assembly/assemble: post: operationId: assemble @@ -189,6 +206,7 @@ paths: "401": { $ref: "#/components/responses/Unauthorized" } "403": { $ref: "#/components/responses/Forbidden" } "429": { $ref: "#/components/responses/TooManyRequests" } + "5XX": { $ref: "#/components/responses/ServerError" } /v1/assembly/disassemble: get: operationId: disassemble @@ -212,6 +230,20 @@ paths: "401": { $ref: "#/components/responses/Unauthorized" } "403": { $ref: "#/components/responses/Forbidden" } "429": { $ref: "#/components/responses/TooManyRequests" } + "5XX": { $ref: "#/components/responses/ServerError" } +x-gatepost-unknown-path: + description: >- + The answer to a path that this file does not list. No call and no doc showed it. The mock + server sends it. + responses: + "404": + description: The gateway serves no such path. + x-gatepost-evidence: assumed + x-gatepost-error-codes: + not_found: assumed + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } components: securitySchemes: ApiKey: @@ -224,44 +256,92 @@ components: its list. headers: RateLimitLimit: - description: The requests that one key can make in each clock minute. + description: >- + The requests that one key can make in each window. Calls showed 600, and a counter that + started again at each clock minute. No header states the window. + x-gatepost-evidence: observed schema: { type: integer, examples: [600] } RateLimitRemaining: - description: The requests that the key has left in this clock minute. + description: The requests that the key has left in this window. A 403 also counts. + x-gatepost-evidence: observed schema: { type: integer, minimum: 0 } RetryAfter: - description: The seconds to wait before the next request. - schema: { type: integer, minimum: 0 } + description: >- + The wait before the next request, in seconds or as an HTTP date. No call showed this + header. The mock server sends seconds. + x-gatepost-evidence: assumed + schema: + oneOf: + - { type: integer, minimum: 0 } + - { type: string, description: An HTTP date. } responses: BadRequest: description: The request has a missing or malformed parameter. x-gatepost-evidence: assumed + x-gatepost-error-codes: + invalid_request: assumed content: application/json: schema: { $ref: "#/components/schemas/ErrorEnvelope" } - Unauthorized: + UnauthorizedSeen: description: >- - No key (auth_required, observed), or a key that the gateway does not know - (invalid_api_key, observed on 1 Oct 2026). Neither response has rate-limit headers. + A request with no key got auth_required in a call. NIPOST's docs list 401 for a missing + or invalid key. No call sent a key that the gateway does not know, so invalid_api_key is + assumed. The response to a call with no key had no rate-limit headers. x-gatepost-evidence: observed + x-gatepost-error-codes: + auth_required: observed + invalid_api_key: assumed + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } + Unauthorized: + description: >- + NIPOST's OpenAPI file asks for a key on this path, and its docs list 401 for a missing or + invalid key. No call without a key reached this path, so both codes are assumed. + x-gatepost-evidence: documented + x-gatepost-error-codes: + auth_required: assumed + invalid_api_key: assumed content: application/json: schema: { $ref: "#/components/schemas/ErrorEnvelope" } PaymentRequired: description: The organisation has no credits for a lookup above level 1. x-gatepost-evidence: documented + x-gatepost-error-codes: + insufficient_credits: documented headers: X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } content: application/json: schema: { $ref: "#/components/schemas/ErrorEnvelope" } - Forbidden: + LookupForbidden: description: >- - A lookup above the key's level (level_not_granted, observed), a key without the scope - (code assumed), or a publishable key from an origin that is not on its list - (origin_not_allowed, documented). + A lookup above the key's level got level_not_granted in a call. NIPOST's docs list 403 + for a key without the scope or the level, but name no code for the scope. NIPOST's docs + name origin_not_allowed only for the widget's calls to another host. x-gatepost-evidence: observed + x-gatepost-error-codes: + level_not_granted: observed + scope_not_granted: assumed + origin_not_allowed: assumed + headers: + X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } + X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } + Forbidden: + description: >- + NIPOST's docs list 403 for a key without the scope, but name no code. No call got a 403 + on this path. NIPOST's docs name origin_not_allowed only for the widget's calls to another + host. + x-gatepost-evidence: documented + x-gatepost-error-codes: + scope_not_granted: assumed + origin_not_allowed: assumed headers: X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } @@ -270,9 +350,11 @@ components: schema: { $ref: "#/components/schemas/ErrorEnvelope" } TooManyRequests: description: >- - The key used up its requests for this clock minute. NIPOST's docs give the status, but - no call reached it, so the error code and Retry-After are assumed. + The key has no requests left in this window. NIPOST's docs give the status, but no call + reached it, so the error code and Retry-After are assumed. x-gatepost-evidence: documented + x-gatepost-error-codes: + rate_limited: assumed headers: X-RateLimit-Limit: { $ref: "#/components/headers/RateLimitLimit" } X-RateLimit-Remaining: { $ref: "#/components/headers/RateLimitRemaining" } @@ -280,6 +362,17 @@ components: content: application/json: schema: { $ref: "#/components/schemas/ErrorEnvelope" } + ServerError: + description: >- + A fault of the gateway, such as 500, 502, 503 or 504. No call and no doc showed one. The + client contract gives the rule for each status. + x-gatepost-evidence: assumed + x-gatepost-error-codes: + internal: assumed + unavailable: assumed + content: + application/json: + schema: { $ref: "#/components/schemas/ErrorEnvelope" } schemas: ErrorEnvelope: type: object diff --git a/scripts/gateway_files.py b/scripts/gateway_files.py index 274fdaf..2a5dfcf 100644 --- a/scripts/gateway_files.py +++ b/scripts/gateway_files.py @@ -21,7 +21,14 @@ ROOT = Path(__file__).resolve().parents[1] OPENAPI = ROOT / "openapi" / "gateway.completed.yaml" FIXTURES = ROOT / "fixtures" +# The marks, from the strongest claim to the weakest. EVIDENCE = ("observed", "documented", "assumed") +MARKS = ", ".join(EVIDENCE) +EVIDENCE_MARK = "x-gatepost-evidence" +ERROR_CODES = "x-gatepost-error-codes" +# The gateway answers a path that it does not serve. This key of the file describes that answer. +UNKNOWN_PATH = "x-gatepost-unknown-path" +HTTP_METHODS = ("get", "put", "post", "delete", "options", "head", "patch", "trace") # NIPOST documents these gateway paths. SEC-4 lets a client call no other path. DOCUMENTED_PATHS = ( "/healthz", @@ -59,30 +66,71 @@ def load_openapi(path: Path = OPENAPI) -> dict[str, Any]: return document -def responses(openapi: dict[str, Any]) -> Iterator[tuple[str, dict[str, Any]]]: - """Yield a label and the response object for each response of each operation. +def operations(openapi: dict[str, Any]) -> Iterator[tuple[str, dict[str, Any]]]: + """Yield a label and each operation, then the answer to a path that the file does not list. + + A path item can also hold keys that are not methods, such as `parameters`. They are skipped. + """ + for path, item in openapi["paths"].items(): + for method in HTTP_METHODS: + if method in item: + yield f"{method.upper()} {path}", item[method] + yield "Any unknown path", openapi[UNKNOWN_PATH] + + +def responses(openapi: dict[str, Any]) -> Iterator[tuple[str, str, dict[str, Any]]]: + """Yield the operation's label, the status and the response object of each response. A response that refers to `components/responses` yields the component that it names. """ shared = openapi.get("components", {}).get("responses", {}) - for path, operations in openapi["paths"].items(): - for method, operation in operations.items(): - for status, response in operation["responses"].items(): - reference = response.get("$ref", "") - target = shared.get(reference.rpartition("/")[2], response) - yield f"{method.upper()} {path} {status}", target + for label, operation in operations(openapi): + for status, response in operation["responses"].items(): + reference = response.get("$ref", "") + yield label, status, shared.get(reference.rpartition("/")[2], response) + + +def code_problems(label: str, status: str, response: dict[str, Any]) -> list[str]: + """Return the problems of the error codes of one response. + + An error response names each code that it can carry, and gives each code its own mark. A + code can claim no more than its response: a code is not observed in a response that no call + returned. + """ + codes = response.get(ERROR_CODES) + if status.startswith("2"): + return [] if codes is None else [f"{label} {status} is a success and has no error codes."] + if not isinstance(codes, dict) or not codes: + return [f"{label} {status} needs {ERROR_CODES}."] + ceiling = response[EVIDENCE_MARK] + problems = [] + for code, mark in codes.items(): + if mark not in EVIDENCE: + problems.append(f"{label} {status} needs a mark for {code}: one of {MARKS}.") + elif EVIDENCE.index(mark) < EVIDENCE.index(ceiling): + problems.append( + f"{label} {status} marks {code} {mark}, above its response ({ceiling})." + ) + return problems def openapi_problems(openapi: dict[str, Any]) -> list[str]: - """Return the problems of the OpenAPI file: extra paths and missing evidence marks.""" + """Return the problems of the OpenAPI file: extra paths, and missing or excessive marks.""" problems = [ f"The path {path} is not a documented gateway path." for path in openapi["paths"] if path not in DOCUMENTED_PATHS ] - for label, response in responses(openapi): - if response.get("x-gatepost-evidence") not in EVIDENCE: - problems.append(f"{label} needs x-gatepost-evidence: one of {', '.join(EVIDENCE)}.") + problems += [ + f"The header {name} needs {EVIDENCE_MARK}: one of {MARKS}." + for name, header in openapi["components"]["headers"].items() + if header.get(EVIDENCE_MARK) not in EVIDENCE + ] + for label, status, response in responses(openapi): + if response.get(EVIDENCE_MARK) not in EVIDENCE: + problems.append(f"{label} {status} needs {EVIDENCE_MARK}: one of {MARKS}.") + continue + problems += code_problems(label, status, response) return problems @@ -111,7 +159,7 @@ def check(name: str, value: object) -> list[str]: def schemas_in_responses(openapi: dict[str, Any]) -> set[str]: """Return the names of the schemas that the responses use directly.""" names = set() - for _, response in responses(openapi): + for _, _, response in responses(openapi): for media in response.get("content", {}).values(): names.add(media["schema"]["$ref"].rpartition("/")[2]) return names @@ -133,7 +181,7 @@ def fixture_problems(name: str, fixture: Any, check: SchemaCheck) -> list[str]: if fixture["version"] != 1: problems.append(f"Fixture {name} must have version 1.") if fixture["evidence"] not in EVIDENCE: - problems.append(f"Fixture {name} needs evidence: one of {', '.join(EVIDENCE)}.") + problems.append(f"Fixture {name} needs evidence: one of {MARKS}.") if type(fixture["status"]) is not int: problems.append(f"Fixture {name} needs an integer status.") problems += [ diff --git a/scripts/tests/test_gateway_files.py b/scripts/tests/test_gateway_files.py index 142088a..763583f 100644 --- a/scripts/tests/test_gateway_files.py +++ b/scripts/tests/test_gateway_files.py @@ -16,6 +16,13 @@ KEYS = gateway_files.read_json(gateway_files.FIXTURES / "keys.json") +def marks(openapi: dict[str, Any]) -> dict[str, str]: + return { + f"{operation} {status}": response["x-gatepost-evidence"] + for operation, status, response in gateway_files.responses(openapi) + } + + def edited_fixture(name: str, **fields: Any) -> dict[str, Any]: fixture: dict[str, Any] = copy.deepcopy(FIXTURES[name]) fixture.update(fields) @@ -43,12 +50,110 @@ def test_rejects_a_response_with_no_evidence_mark(self) -> None: [problem] = gateway_files.openapi_problems(openapi) self.assertTrue(problem.startswith("GET /v1/lookup 200 needs x-gatepost-evidence")) + def test_rejects_an_evidence_value_that_is_not_a_mark(self) -> None: + openapi = copy.deepcopy(OPENAPI) + openapi["paths"]["/v1/lookup"]["get"]["responses"]["200"]["x-gatepost-evidence"] = "guess" + [problem] = gateway_files.openapi_problems(openapi) + self.assertTrue(problem.startswith("GET /v1/lookup 200 needs x-gatepost-evidence")) + def test_reads_the_evidence_of_a_shared_response_from_its_component(self) -> None: openapi = copy.deepcopy(OPENAPI) - del openapi["components"]["responses"]["Unauthorized"]["x-gatepost-evidence"] + del openapi["components"]["responses"]["BadRequest"]["x-gatepost-evidence"] labels = [problem.split(" needs")[0] for problem in gateway_files.openapi_problems(openapi)] - self.assertIn("GET /v1/lookup 401", labels) - self.assertIn("POST /v1/assembly/assemble 401", labels) + self.assertIn("GET /v1/lookup 400", labels) + self.assertIn("POST /v1/assembly/assemble 400", labels) + + def test_marks_as_observed_only_the_responses_that_a_call_returned(self) -> None: + observed = sorted(label for label, mark in marks(OPENAPI).items() if mark == "observed") + self.assertEqual( + observed, + [ + "GET /v1/assembly/disassemble 200", + "GET /v1/lookup 200", + "GET /v1/lookup 401", + "GET /v1/lookup 403", + "GET /v1/search/autocomplete 200", + "GET /v1/search/autocomplete 401", + "GET /v1/search/nearby 200", + "GET /v1/search/reverse 200", + "POST /v1/assembly/assemble 200", + ], + ) + + def test_marks_as_observed_only_the_error_codes_that_a_call_returned(self) -> None: + observed = sorted( + f"{operation} {status} {code}" + for operation, status, response in gateway_files.responses(OPENAPI) + for code, mark in response.get("x-gatepost-error-codes", {}).items() + if mark == "observed" + ) + self.assertEqual( + observed, + [ + "GET /v1/lookup 401 auth_required", + "GET /v1/lookup 403 level_not_granted", + "GET /v1/search/autocomplete 401 auth_required", + ], + ) + + def test_gives_each_error_code_of_a_response_its_own_mark(self) -> None: + lookup = OPENAPI["paths"]["/v1/lookup"]["get"]["responses"] + shared = OPENAPI["components"]["responses"] + forbidden = shared[lookup["403"]["$ref"].rpartition("/")[2]] + self.assertEqual( + forbidden["x-gatepost-error-codes"], + { + "level_not_granted": "observed", + "scope_not_granted": "assumed", + "origin_not_allowed": "assumed", + }, + ) + + def test_rejects_an_error_response_with_no_error_codes(self) -> None: + openapi = copy.deepcopy(OPENAPI) + del openapi["components"]["responses"]["BadRequest"]["x-gatepost-error-codes"] + problems = gateway_files.openapi_problems(openapi) + self.assertIn("GET /v1/lookup 400 needs x-gatepost-error-codes.", problems) + + def test_rejects_an_error_code_that_claims_more_than_its_response(self) -> None: + openapi = copy.deepcopy(OPENAPI) + codes = openapi["components"]["responses"]["BadRequest"]["x-gatepost-error-codes"] + codes["invalid_request"] = "observed" + problems = gateway_files.openapi_problems(openapi) + self.assertIn( + "GET /v1/lookup 400 marks invalid_request observed, above its response (assumed).", + problems, + ) + + def test_describes_the_answer_to_a_path_that_the_gateway_does_not_serve(self) -> None: + [(status, response)] = [ + (status, response) + for operation, status, response in gateway_files.responses(OPENAPI) + if operation == "Any unknown path" + ] + self.assertEqual(status, "404") + self.assertEqual(response["x-gatepost-error-codes"], {"not_found": "assumed"}) + + def test_rejects_a_header_with_no_evidence_mark(self) -> None: + openapi = copy.deepcopy(OPENAPI) + del openapi["components"]["headers"]["RetryAfter"]["x-gatepost-evidence"] + [problem] = gateway_files.openapi_problems(openapi) + self.assertEqual( + problem, + "The header RetryAfter needs x-gatepost-evidence: " + "one of observed, documented, assumed.", + ) + + def test_marks_the_retry_after_header_as_assumed(self) -> None: + headers = OPENAPI["components"]["headers"] + self.assertEqual(headers["RetryAfter"]["x-gatepost-evidence"], "assumed") + self.assertEqual(headers["RateLimitLimit"]["x-gatepost-evidence"], "observed") + + def test_reads_a_path_item_with_shared_parameters(self) -> None: + openapi = copy.deepcopy(OPENAPI) + openapi["paths"]["/v1/lookup"]["parameters"] = [] + openapi["paths"]["/v1/lookup"]["summary"] = "Lookups" + self.assertEqual(gateway_files.openapi_problems(openapi), []) def test_marks_each_lookup_above_level_1_as_not_observed(self) -> None: schemas = OPENAPI["components"]["schemas"] From 0706f5dbe0e2e5fc1076b2e0d080b9136205aec8 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 00:59:30 +0100 Subject: [PATCH 09/13] fix: check each fixture against the response that it stands for A fixture with a wrong status or a wrong error code passed the checks, and the mock server would then send a wrong reply. Each fixture must now have a status that a response with its schema declares, an error code of that response, and the code that its file name states. Its evidence can claim no more than the mark of that response or code, so the key that the gateway does not know and the origin error are now assumed. The scan for real postcodes now also reads the docs beside the fixtures and the scenarios, accepts only the state FC, and knows more separators. A new scan fails on text in the form of a real key. Two error messages now use Gatepost's own words. Signed-off-by: MobileMage --- fixtures/README.md | 10 +- fixtures/errors/insufficient-credits.json | 2 +- fixtures/errors/invalid-api-key.json | 4 +- fixtures/errors/origin-not-allowed.json | 2 +- fixtures/errors/rate-limited.json | 4 +- scripts/contract_files.py | 25 +++-- scripts/gateway_files.py | 99 +++++++++++++++++--- scripts/tests/test_contract_files.py | 20 ++++ scripts/tests/test_gateway_files.py | 107 +++++++++++++++++++--- 9 files changed, 230 insertions(+), 43 deletions(-) diff --git a/fixtures/README.md b/fixtures/README.md index cb31954..783bfa1 100644 --- a/fixtures/README.md +++ b/fixtures/README.md @@ -16,9 +16,12 @@ The mock server answers with these files. Each file holds one response of NIPOST ``` - `version` is the format version, which is 1. -- `evidence` is `observed`, `documented` or `assumed`, as in `openapi/gateway.completed.yaml`. -- `schema` names a schema in `components/schemas` of that file. The body must match it, and `make check` tests that it does. -- `status` is the HTTP status of the response. +- `evidence` is `observed`, `documented` or `assumed`, as in `openapi/gateway.completed.yaml`. It cannot claim more than the mark of the response in that file, or of its error code. +- `schema` names a schema in `components/schemas` of that file. The body must match it. +- `status` is the HTTP status of the response. A response of the OpenAPI file with the same schema must have this status. +- An error body carries an error code of that response. The file name of an error states its code: `rate-limited.json` holds `rate_limited`. A number at the end names a level. `unknown-path.json` names a case, not a code. + +`make check` tests each of these rules. It also fails on a postcode outside `FC`, `Z99` and `ZZ`, and on text in the form of a key that is not a mock key. ## How the mock server uses them @@ -31,6 +34,7 @@ The mock server answers with these files. Each file holds one response of NIPOST | `reverse` at any other coordinate | `reverse/not-found.json` | | `nearby` | `nearby/empty.json` | | an error | the file in `errors/` with the same error code | +| a path that the gateway does not serve | `errors/unknown-path.json` | - The mock server puts the caller's text, in upper case, in `postcode` of a lookup body, as the gateway does. - It puts the request's coordinate and the applied radius in a reverse body. diff --git a/fixtures/errors/insufficient-credits.json b/fixtures/errors/insufficient-credits.json index f7bca80..db12413 100644 --- a/fixtures/errors/insufficient-credits.json +++ b/fixtures/errors/insufficient-credits.json @@ -7,7 +7,7 @@ "body": { "error": { "code": "insufficient_credits", - "message": "not enough credits; top up to continue" + "message": "the organisation has no credits for this lookup level" } } } diff --git a/fixtures/errors/invalid-api-key.json b/fixtures/errors/invalid-api-key.json index c78c74d..df78bc2 100644 --- a/fixtures/errors/invalid-api-key.json +++ b/fixtures/errors/invalid-api-key.json @@ -1,7 +1,7 @@ { "version": 1, - "description": "A key that the gateway does not know. The message is Gatepost's.", - "evidence": "observed", + "description": "A key that the gateway does not know. No call showed the code or the message.", + "evidence": "assumed", "schema": "ErrorEnvelope", "status": 401, "body": { diff --git a/fixtures/errors/origin-not-allowed.json b/fixtures/errors/origin-not-allowed.json index e5b7879..9c40f4d 100644 --- a/fixtures/errors/origin-not-allowed.json +++ b/fixtures/errors/origin-not-allowed.json @@ -1,7 +1,7 @@ { "version": 1, "description": "A publishable key from an origin off its list. The message is Gatepost's.", - "evidence": "documented", + "evidence": "assumed", "schema": "ErrorEnvelope", "status": 403, "body": { diff --git a/fixtures/errors/rate-limited.json b/fixtures/errors/rate-limited.json index 5cf8794..f2d81c7 100644 --- a/fixtures/errors/rate-limited.json +++ b/fixtures/errors/rate-limited.json @@ -1,13 +1,13 @@ { "version": 1, - "description": "A key that used up its requests for this clock minute.", + "description": "A key with no requests left in this minute.", "evidence": "assumed", "schema": "ErrorEnvelope", "status": 429, "body": { "error": { "code": "rate_limited", - "message": "too many requests; wait for the next minute" + "message": "too many requests for this key in this minute" } } } diff --git a/scripts/contract_files.py b/scripts/contract_files.py index 364f9c5..6cbf40d 100644 --- a/scripts/contract_files.py +++ b/scripts/contract_files.py @@ -195,6 +195,16 @@ def coverage_problems(scenarios: dict[str, Any], codes: list[str]) -> list[str]: return problems +def synthetic_files() -> list[Path]: + """Return the files that may hold only synthetic postcodes and mock keys. + + These are the fixtures, the scenarios, the docs beside them, client.md and the OpenAPI file. + """ + folders = [*gateway_files.FIXTURES.rglob("*"), *CONTRACT.rglob("*")] + files = [path for path in folders if path.suffix in (".json", ".md")] + return sorted([*files, CLIENT_DOC, gateway_files.OPENAPI]) + + def all_problems() -> list[str]: """Return the problems of the OpenAPI file, the fixtures, the keys and the scenarios.""" openapi = gateway_files.load_openapi() @@ -204,21 +214,18 @@ def all_problems() -> list[str]: codes = error_codes() problems = gateway_files.openapi_problems(openapi) for name, fixture in fixtures.items(): - problems += gateway_files.fixture_problems(name, fixture, check) - used = gateway_files.schemas_in_responses(openapi) - problems += [ - f"Fixture {name} uses {fixture['schema']}, which no response uses." - for name, fixture in fixtures.items() - if fixture.get("schema") not in used - ] + shape = gateway_files.fixture_problems(name, fixture, check) + problems += shape or gateway_files.fixture_response_problems(name, fixture, openapi) problems += gateway_files.key_problems( gateway_files.read_json(gateway_files.FIXTURES / "keys.json") ) for name, scenario in scenarios.items(): problems += scenario_problems(name, scenario, set(fixtures), codes) problems += coverage_problems(scenarios, codes) - files = sorted([*gateway_files.FIXTURES.rglob("*.json"), *CONTRACT.glob("*.json")]) - return problems + gateway_files.postcode_problems(files) + files = synthetic_files() + return ( + problems + gateway_files.postcode_problems(files) + gateway_files.key_text_problems(files) + ) def main() -> int: diff --git a/scripts/gateway_files.py b/scripts/gateway_files.py index 2a5dfcf..2eefd55 100644 --- a/scripts/gateway_files.py +++ b/scripts/gateway_files.py @@ -26,6 +26,7 @@ MARKS = ", ".join(EVIDENCE) EVIDENCE_MARK = "x-gatepost-evidence" ERROR_CODES = "x-gatepost-error-codes" +ERROR_SCHEMA = "ErrorEnvelope" # The gateway answers a path that it does not serve. This key of the file describes that answer. UNKNOWN_PATH = "x-gatepost-unknown-path" HTTP_METHODS = ("get", "put", "post", "delete", "options", "head", "patch", "trace") @@ -43,11 +44,15 @@ KEY_FIELDS = ("key", "kind", "lookupLevel", "lookupScope", "credits", "rateLimited", "origins") # A mock key must never look like a real key, so a real key cannot slip into the table. MOCK_KEY = re.compile(r"nipost_(?:pk_)?test_mock(?:_[a-z0-9]+)*") +# Text in the form of a NIPOST key. Only a mock key may appear in a committed file. +KEY_TEXT = re.compile(r"nipost_(?:pk_)?(?:test|live)_[A-Za-z0-9_]+") # Five segments, with or without separators, in either letter case. POSTCODE = re.compile( - r"(? list[str]: return check -def schemas_in_responses(openapi: dict[str, Any]) -> set[str]: - """Return the names of the schemas that the responses use directly.""" - names = set() - for _, _, response in responses(openapi): - for media in response.get("content", {}).values(): - names.add(media["schema"]["$ref"].rpartition("/")[2]) - return names +def status_matches(declared: str, status: int) -> bool: + """Tell whether a status key of the OpenAPI file, such as 404 or 5XX, covers a status.""" + if declared.endswith("XX"): + return str(status).startswith(declared[0]) + return declared == str(status) + + +def schema_name(response: dict[str, Any]) -> str | None: + """Return the name of the schema of a JSON response, or None for a response with no body.""" + media = response.get("content", {}).get("application/json") + return None if media is None else str(media["schema"]["$ref"].rpartition("/")[2]) def load_fixtures(root: Path = FIXTURES) -> dict[str, Any]: - """Return each fixture by its name, the path under the folder without `.json`.""" + """Return each fixture by its name, the path under the folder without `.json`. + + A fixture lives in a folder under the root, at any depth. The root holds `keys.json`, which + is not a fixture. + """ return { path.relative_to(root).with_suffix("").as_posix(): read_json(path) - for path in sorted(root.glob("*/*.json")) + for path in sorted(root.rglob("*.json")) + if path.parent != root } @@ -190,6 +204,55 @@ def fixture_problems(name: str, fixture: Any, check: SchemaCheck) -> list[str]: return problems +def stated_code(name: str) -> str: + """Return the error code that the file name of a fixture states. + + errors/level-not-granted-2 states level_not_granted. A number at the end names a level. + """ + return re.sub(r"-[0-9]+$", "", name.rpartition("/")[2]).replace("-", "_") + + +def evidence_problems(name: str, evidence: str, marks: list[str]) -> list[str]: + """Return a problem when a fixture claims more than the strongest of the marks.""" + strongest = min(marks, key=EVIDENCE.index) + if EVIDENCE.index(evidence) >= EVIDENCE.index(strongest): + return [] + return [f"Fixture {name} claims {evidence}, but the OpenAPI file says {strongest}."] + + +def fixture_response_problems( + name: str, fixture: dict[str, Any], openapi: dict[str, Any] +) -> list[str]: + """Return the problems of a fixture against the responses of the OpenAPI file. + + A response with the fixture's schema must declare the fixture's status. An error fixture + must carry a code of such a response, and the code that its file name states. The fixture's + evidence can claim no more than the mark of its response or of its error code. + """ + status, schema = fixture["status"], fixture["schema"] + found = [ + response + for _, declared, response in responses(openapi) + if status_matches(declared, status) and schema_name(response) == schema + ] + if not found: + return [f"Fixture {name}: no response with the schema {schema} has the status {status}."] + if schema != ERROR_SCHEMA: + return evidence_problems(name, fixture["evidence"], [r[EVIDENCE_MARK] for r in found]) + code = fixture["body"]["error"]["code"] + named = stated_code(name) + listed = { + each for _, _, response in responses(openapi) for each in response.get(ERROR_CODES, {}) + } + problems = [] + if named in listed and named != code: + problems.append(f"Fixture {name}: its name asks for the error code {named}.") + marks = [r[ERROR_CODES][code] for r in found if code in r.get(ERROR_CODES, {})] + if not marks: + return [*problems, f"Fixture {name}: no {status} response has the error code {code}."] + return problems + evidence_problems(name, fixture["evidence"], marks) + + def key_problems(document: Any) -> list[str]: """Return the problems of `fixtures/keys.json`.""" problems = [] @@ -213,12 +276,12 @@ def key_problems(document: Any) -> list[str]: def real_postcodes(text: str) -> list[str]: """Return each full postcode in the text that is not synthetic. - A synthetic postcode has the district Z99 and the area ZZ. + A synthetic postcode has the state FC, the district Z99 and the area ZZ. """ return [ match.group(0) for match in POSTCODE.finditer(text) - if (match.group(3).upper(), match.group(4).upper()) != ("Z99", "ZZ") + if tuple(match.group(i).upper() for i in (1, 3, 4)) != SYNTHETIC ] @@ -229,3 +292,13 @@ def postcode_problems(paths: list[Path]) -> list[str]: for path in paths for code in real_postcodes(path.read_text(encoding="utf-8")) ] + + +def key_text_problems(paths: list[Path]) -> list[str]: + """Return a problem for each text in the files in the form of a key that is no mock key.""" + return [ + f"{path.relative_to(ROOT)} holds the key {key}. Use a mock key." + for path in paths + for key in KEY_TEXT.findall(path.read_text(encoding="utf-8")) + if not MOCK_KEY.fullmatch(key) + ] diff --git a/scripts/tests/test_contract_files.py b/scripts/tests/test_contract_files.py index c48eaa8..2dd757a 100644 --- a/scripts/tests/test_contract_files.py +++ b/scripts/tests/test_contract_files.py @@ -15,6 +15,7 @@ FIXTURE_NAMES = set(gateway_files.load_fixtures()) CODES = contract_files.error_codes() STANDARDS = gateway_files.ROOT / "standards" / "CODING_STANDARDS.md" +FILES = contract_files.synthetic_files() def problems_of(name: str, edit: Any) -> list[str]: @@ -189,3 +190,22 @@ def test_reports_a_required_scenario_that_is_missing(self) -> None: contract_files.coverage_problems(scenarios, CODES), ["The scenario lookup-queue-limit is missing."], ) + + +class SyntheticFilesTest(unittest.TestCase): + def test_scans_the_fixtures_the_scenarios_and_the_docs_beside_them(self) -> None: + names = {path.relative_to(gateway_files.ROOT).as_posix() for path in FILES} + for name in ( + "client.md", + "contract/README.md", + "contract/lookup-level-1.json", + "fixtures/README.md", + "fixtures/keys.json", + "fixtures/lookup/valid-level-1.json", + "openapi/gateway.completed.yaml", + ): + self.assertIn(name, names) + + def test_finds_no_real_postcode_and_no_real_key_in_them(self) -> None: + self.assertEqual(gateway_files.postcode_problems(FILES), []) + self.assertEqual(gateway_files.key_text_problems(FILES), []) diff --git a/scripts/tests/test_gateway_files.py b/scripts/tests/test_gateway_files.py index 763583f..15713c1 100644 --- a/scripts/tests/test_gateway_files.py +++ b/scripts/tests/test_gateway_files.py @@ -192,9 +192,69 @@ def test_each_fixture_has_no_problem(self) -> None: with self.subTest(name): self.assertEqual(gateway_files.fixture_problems(name, fixture, CHECK), []) - def test_each_fixture_uses_a_schema_of_a_response(self) -> None: - used = gateway_files.schemas_in_responses(OPENAPI) - self.assertEqual(sorted({fixture["schema"] for fixture in FIXTURES.values()} - used), []) + def test_each_fixture_matches_a_response_of_the_openapi_file(self) -> None: + for name, fixture in FIXTURES.items(): + with self.subTest(name): + self.assertEqual( + gateway_files.fixture_response_problems(name, fixture, OPENAPI), [] + ) + + def test_rejects_a_status_that_no_response_with_the_schema_has(self) -> None: + cases = { + "errors/rate-limited": (200, "ErrorEnvelope"), + "lookup/valid-level-1": (500, "LookupEnvelope"), + } + for name, (status, schema) in cases.items(): + with self.subTest(name): + fixture = edited_fixture(name, status=status) + [problem] = gateway_files.fixture_response_problems(name, fixture, OPENAPI) + self.assertEqual( + problem, + f"Fixture {name}: no response with the schema {schema} " + f"has the status {status}.", + ) + + def test_rejects_an_error_code_that_the_status_does_not_carry(self) -> None: + fixture = edited_fixture("errors/rate-limited") + fixture["body"]["error"]["code"] = "banana" + self.assertEqual( + gateway_files.fixture_response_problems("errors/rate-limited", fixture, OPENAPI), + [ + "Fixture errors/rate-limited: its name asks for the error code rate_limited.", + "Fixture errors/rate-limited: no 429 response has the error code banana.", + ], + ) + + def test_rejects_an_error_code_that_differs_from_the_file_name(self) -> None: + fixture = edited_fixture("errors/level-not-granted-2") + fixture["body"]["error"]["code"] = "scope_not_granted" + name = "errors/level-not-granted-2" + self.assertEqual( + gateway_files.fixture_response_problems(name, fixture, OPENAPI), + [f"Fixture {name}: its name asks for the error code level_not_granted."], + ) + + def test_rejects_evidence_that_claims_more_than_the_openapi_file(self) -> None: + fixture = edited_fixture("errors/invalid-api-key", evidence="observed") + self.assertEqual( + gateway_files.fixture_response_problems("errors/invalid-api-key", fixture, OPENAPI), + ["Fixture errors/invalid-api-key claims observed, but the OpenAPI file says assumed."], + ) + + def test_accepts_a_status_in_a_range_of_the_openapi_file(self) -> None: + fixture = edited_fixture("errors/rate-limited", status=503, evidence="assumed") + fixture["body"]["error"]["code"] = "unavailable" + self.assertEqual( + gateway_files.fixture_response_problems("errors/unavailable", fixture, OPENAPI), [] + ) + + def test_loads_a_fixture_in_a_deeper_folder_and_not_the_key_file(self) -> None: + with tempfile.TemporaryDirectory() as folder: + root = Path(folder) + (root / "lookup" / "extra").mkdir(parents=True) + (root / "lookup" / "extra" / "deep.json").write_text("{}", encoding="utf-8") + (root / "keys.json").write_text("{}", encoding="utf-8") + self.assertEqual(list(gateway_files.load_fixtures(root)), ["lookup/extra/deep"]) def test_has_a_body_for_each_lookup_level_that_a_mock_key_holds(self) -> None: for level in (1, 2, 3): @@ -258,15 +318,6 @@ def test_rejects_a_level_that_no_key_can_hold(self) -> None: class SyntheticPostcodeTest(unittest.TestCase): - def test_finds_no_real_postcode_in_the_committed_files(self) -> None: - files = sorted( - [ - *gateway_files.FIXTURES.rglob("*.json"), - *(gateway_files.ROOT / "contract").glob("*.json"), - ] - ) - self.assertEqual(gateway_files.postcode_problems(files), []) - def test_finds_a_real_postcode_in_each_form_and_case(self) -> None: text = "EK-01-A03-FK-01, EK 01 A03 FK 01, ek01a03fk01 and FC-01-Z99-ZZ-01" self.assertEqual( @@ -274,6 +325,16 @@ def test_finds_a_real_postcode_in_each_form_and_case(self) -> None: ["EK-01-A03-FK-01", "EK 01 A03 FK 01", "ek01a03fk01"], ) + def test_finds_a_postcode_with_other_separators(self) -> None: + text = "EK/01/A03/FK/01, EK_01_A03_FK_01 and EK.01.A03.FK.01" + self.assertEqual( + gateway_files.real_postcodes(text), + ["EK/01/A03/FK/01", "EK_01_A03_FK_01", "EK.01.A03.FK.01"], + ) + + def test_finds_a_synthetic_district_in_a_state_other_than_fc(self) -> None: + self.assertEqual(gateway_files.real_postcodes("LA-01-Z99-ZZ-01"), ["LA-01-Z99-ZZ-01"]) + def test_keeps_a_partial_postcode_and_a_longer_word(self) -> None: self.assertEqual(gateway_files.real_postcodes("EK-01-A03 and XEK01A03FK01"), []) @@ -285,3 +346,25 @@ def test_names_the_file_that_holds_a_real_postcode(self) -> None: self.assertTrue( problem.endswith("probe.json holds the postcode LA-12-K23-IV-95. Use a synthetic code.") ) + + +class KeyTextTest(unittest.TestCase): + def test_names_the_file_that_holds_text_in_the_form_of_a_real_key(self) -> None: + with tempfile.TemporaryDirectory(dir=gateway_files.ROOT) as folder: + path = Path(folder) / "probe.json" + path.write_text( + '{"message": "nipost_live_a1b2c3d4e5f6a7b8", "key": "nipost_test_mock_l1"}', + encoding="utf-8", + ) + [problem] = gateway_files.key_text_problems([path]) + self.assertTrue( + problem.endswith( + "probe.json holds the key nipost_live_a1b2c3d4e5f6a7b8. Use a mock key." + ) + ) + + def test_keeps_the_prefixes_that_the_docs_name(self) -> None: + with tempfile.TemporaryDirectory(dir=gateway_files.ROOT) as folder: + path = Path(folder) / "probe.md" + path.write_text("A key starts with nipost_test_ or nipost_pk_live_.", encoding="utf-8") + self.assertEqual(gateway_files.key_text_problems([path]), []) From 8ab969fcc34cf1ef0e54b01e0fe07bac217de8f4 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 01:05:10 +0100 Subject: [PATCH 10/13] fix: tie each scenario to its rule and to the responses that it lists The checks let a deleted scenario or a false outcome pass. client.md now names the scenarios of each rule, and its error table names them for each row. The check fails when a named scenario has no file, when a scenario has no rule, and when a scenario of a row ends with another error code. With one call, the reply to the last attempt must give the outcome: the error with its status, API code and Retry-After, or the result with its level, validity and unit. A fixture of another method and too many attempts fail too. New scenarios cover the rules that had none: levels 4 and 5, a lower level, the default level, the limits of the radius, text over 11 characters, the limits of Retry-After, a shared error, the echoed postcode, and a reverse postcode that does not parse. Signed-off-by: MobileMage --- client.md | 78 ++-- contract/README.md | 6 +- contract/autocomplete-too-long.json | 28 ++ contract/lookup-building-use-only.json | 44 +++ contract/lookup-default-level.json | 45 +++ contract/lookup-echo-ignored.json | 37 ++ contract/lookup-level-4.json | 44 +++ contract/lookup-level-5.json | 44 +++ contract/lookup-level-not-granted.json | 2 +- contract/lookup-lower-level.json | 37 ++ contract/lookup-retry-after-eleven.json | 36 ++ contract/lookup-retry-after-ten.json | 49 +++ contract/lookup-shared-error.json | 48 +++ contract/reverse-bad-postcode.json | 53 +++ .../reverse-invalid-negative-distance.json | 30 ++ contract/reverse-max-distance-250.json | 57 +++ contract/reverse-max-distance-zero.json | 57 +++ contract/reverse-unknown-confidence.json | 63 ++++ contract/reverse-unknown-fields.json | 6 +- scripts/contract_files.py | 350 +++++++++++++++--- scripts/tests/test_client_doc.py | 2 +- scripts/tests/test_contract_files.py | 166 ++++++++- 22 files changed, 1158 insertions(+), 124 deletions(-) create mode 100644 contract/autocomplete-too-long.json create mode 100644 contract/lookup-building-use-only.json create mode 100644 contract/lookup-default-level.json create mode 100644 contract/lookup-echo-ignored.json create mode 100644 contract/lookup-level-4.json create mode 100644 contract/lookup-level-5.json create mode 100644 contract/lookup-lower-level.json create mode 100644 contract/lookup-retry-after-eleven.json create mode 100644 contract/lookup-retry-after-ten.json create mode 100644 contract/lookup-shared-error.json create mode 100644 contract/reverse-bad-postcode.json create mode 100644 contract/reverse-invalid-negative-distance.json create mode 100644 contract/reverse-max-distance-250.json create mode 100644 contract/reverse-max-distance-zero.json create mode 100644 contract/reverse-unknown-confidence.json diff --git a/client.md b/client.md index 27e4641..4cfebf5 100644 --- a/client.md +++ b/client.md @@ -1,6 +1,6 @@ # Client -This file defines how every Gatepost client calls NIPOST's gateway. The contract scenarios in `contract/` test each rule here, except the rules that the last section lists. If this file and a scenario disagree, this file wins. Such a disagreement is a bug in the spec. +This file defines how every Gatepost client calls NIPOST's gateway. The contract scenarios in `contract/` test each rule here, except the rules that the last section lists. Each rule names its scenarios, and `make check` fails when a named scenario has no file or a scenario has no rule here. If this file and a scenario disagree, this file wins. Such a disagreement is a bug in the spec. `openapi/gateway.completed.yaml` describes the gateway itself. Each response in it carries an evidence mark. Gatepost saw some responses in calls of its own. It knows others only from NIPOST's docs. A client reads both kinds in the same way. @@ -45,12 +45,12 @@ A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` | `recentHouseAddress` | text, or null | | `buildingUseStatus` | text, or null | -- The client never takes `postcode` from the response. The gateway echoes the caller's text there, also for text that is not a postcode. +- The client never takes `postcode` from the response. The gateway echoes the caller's text there, also for text that is not a postcode. The scenario `lookup-echo-ignored` tests this rule. - `status` is null when the response has no `status`. The client returns any other value of `status` as it is, also when this table does not list it, so that a new gateway status is not lost. -- `levelReceived` comes from the fields of the response. It is 5 when `point_geometry` is present, 4 for `other_building_info`, 3 for `building_use_status`, and 2 for either address field. Otherwise it is 1. -- `recentHouseAddress` is the text in `recent_house_address.recent`. +- `levelReceived` comes from the fields of the response. It is 5 when `point_geometry` is present, 4 for `other_building_info`, 3 for `building_use_status`, and 2 for either address field. Otherwise it is 1. It can be lower than `levelRequested`. The scenarios `lookup-level-2`, `lookup-level-3`, `lookup-level-4`, `lookup-level-5`, `lookup-building-use-only` and `lookup-lower-level` test this rule. +- `recentHouseAddress` is the text in `recent_house_address.recent`. The scenario `lookup-level-2` tests it. - A `status` that is present, not null and not text raises `unexpected_response`. -- A response with `valid: false` is a result, not an error. +- A response with `valid: false` is a result, not an error. The scenario `lookup-not-found` tests it. ### Reverse result @@ -73,8 +73,10 @@ A unit has these fields: | `stateName`, `lgaName`, `localityName` | text, or null | | `address` | text, or null | -- `found` can be true while `unit` is null. The gateway then found an area, but no unit within the radius. -- `confidence` is `low` for a value that this table does not list. +- The scenario `reverse-unit` tests a result with a unit. +- `found` can be true while `unit` is null. The gateway then found an area, but no unit within the radius. The scenario `reverse-area` tests it. +- A response with `found: false` is a result, not an error. The scenario `reverse-not-found` tests it. +- `confidence` is `low` for a value that this table does not list. The scenario `reverse-unknown-confidence` tests it. - The gateway sends the coordinate as `[lng, lat]`. The client does not return it. ### Autocomplete result @@ -92,22 +94,22 @@ A suggestion has these fields: | `label` | text, or null | | `postcode` | the partial postcode of the typed segments before the active one, followed by `code`, or null | -The gateway sends the value of one segment only. For the text `FC01Z`, the suggestion `Z99` gives the postcode `FC-01-Z99`. The client parses that text with partial postcodes allowed. When it does not parse, `postcode` is null. +The gateway sends the value of one segment only. For the text `FC01Z`, the suggestion `Z99` gives the postcode `FC-01-Z99`. The client parses that text with partial postcodes allowed. When it does not parse, `postcode` is null. The scenarios `autocomplete-state` and `autocomplete-district` test this rule. A client ignores a field that this contract does not use, in a lookup, a reverse or an autocomplete response. Such a field never changes the result and never raises an error. The scenarios `lookup-unknown-fields`, `reverse-unknown-fields` and `autocomplete-unknown-fields` test this rule, one for each call. The scenario `lookup-unknown-status` tests that an unknown `status` stays text. ## Requests -- The client sends `X-API-Key` only when the caller gave a key. -- `lookup` parses the code with the core first. A code that does not parse raises `invalid_input`, and the client sends no request. Partial postcodes do not pass. The request sends the canonical form and the level: `code=FC-01-Z99-ZZ-01&level=1`. -- `reverse` sends `lat` and `lng`, and `max_distance_m` when the caller gave a radius. +- The client sends `X-API-Key` only when the caller gave a key. The scenario `lookup-no-key` tests it. +- `lookup` parses the code with the core first. A code that does not parse raises `invalid_input`, and the client sends no request. Partial postcodes do not pass. The request sends the canonical form and the level: `code=FC-01-Z99-ZZ-01&level=1`. The scenarios `lookup-invalid-input`, `lookup-level-1` and `lookup-default-level` test this rule. +- `reverse` sends `lat` and `lng`, and `max_distance_m` when the caller gave a radius. The scenarios `reverse-unit` and `reverse-area` test this rule. - Each of these raises `invalid_input`, and the client sends no request: - A `level` that is not a whole number from 1 to 5. The scenario `lookup-invalid-level` tests it. - A `lat` that is not finite or is outside -90 to 90. The scenario `reverse-invalid-lat` tests it. - A `lng` that is not finite or is outside -180 to 180. The scenario `reverse-invalid-lng` tests it. - - A `maxDistanceM` below 0 or above 250. The OpenAPI file gives the range 0 to 250. The gateway cuts a larger value to 250 without a sign, so the client rejects it. It does not return a result for another radius. The scenario `reverse-invalid-max-distance` tests it. + - A `maxDistanceM` below 0 or above 250. The OpenAPI file gives the range 0 to 250. The gateway cuts a larger value to 250 without a sign, so the client rejects it. It does not return a result for another radius. The scenarios `reverse-invalid-max-distance` and `reverse-invalid-negative-distance` test it. The scenarios `reverse-max-distance-zero` and `reverse-max-distance-250` test that both limits pass. - The client writes each number in the query in its shortest decimal form, with no exponent and no trailing zero: `lat=9`, not `lat=9.0` and not `lat=9e0`. A negative zero is written `0`. A small value is written in full: `0.0000001`, not `1e-7`. Languages print numbers in different forms, so one fixed form gives every client the same query for the same input. The mock server compares numbers by value, so it also accepts `9.0`. The scenarios `reverse-plain-decimal` and `reverse-whole-number` test this rule. -- `autocomplete` normalises its text with the core. Empty text, text of more than 11 characters, and text with a character other than A to Z and 0 to 9 raise `invalid_input`, and the client sends no request. The gateway does not answer an empty `q`. The request sends the normalised text: `q=FC01Z`. +- `autocomplete` normalises its text with the core. Empty text, text of more than 11 characters, and text with a character other than A to Z and 0 to 9 raise `invalid_input`, and the client sends no request. The gateway does not answer an empty `q`. The request sends the normalised text: `q=FC01Z`. The scenarios `autocomplete-empty-input`, `autocomplete-too-long`, `autocomplete-bad-character` and `autocomplete-district` test this rule. ## Errors @@ -122,44 +124,44 @@ A client ignores a field that this contract does not use, in a lookup, a reverse The error code comes from the first row that matches. -| Condition | Error code | Retried | -|---|---|---| -| the client's own check of the input fails | `invalid_input` | no request is sent | -| status 401 | `unauthorized` | no | -| status 402 | `insufficient_credits` | no | -| status 403, with the API code `origin_not_allowed` | `origin_not_allowed` | no | -| status 403, with any other API code | `forbidden` | no | -| status 429 | `rate_limited` | see below | -| status 502, 503 or 504 | `server_error` | yes | -| any other status from 400 to 499 | `invalid_input` | no | -| any other status that is not 200 | `server_error` | no | -| status 200, with a body that is not a JSON object with a `data` field, or with a part that this contract uses missing or of the wrong type | `unexpected_response` | no | -| no response, because the connection failed | `network_error` | yes | -| no response within `timeoutMs` | `timeout` | yes, except for `autocomplete` | +| Condition | Error code | Retried | Scenarios | +|---|---|---|---| +| the client's own check of the input fails | `invalid_input` | no request is sent | `lookup-invalid-input` | +| status 401 | `unauthorized` | no | `lookup-no-key`, `lookup-invalid-key` | +| status 402 | `insufficient_credits` | no | `lookup-insufficient-credits` | +| status 403, with the API code `origin_not_allowed` | `origin_not_allowed` | no | `lookup-origin-not-allowed` | +| status 403, with any other API code | `forbidden` | no | `lookup-level-not-granted` | +| status 429 | `rate_limited` | see below | `lookup-rate-limited` | +| status 502, 503 or 504 | `server_error` | yes | `lookup-retries-exhausted` | +| any other status from 400 to 499 | `invalid_input` | no | `lookup-bad-request` | +| any other status that is not 200 | `server_error` | no | `lookup-server-error` | +| status 200, with a body that is not a JSON object with a `data` field, or with a part that this contract uses missing or of the wrong type | `unexpected_response` | no | `lookup-malformed-body` | +| no response, because the connection failed | `network_error` | yes | `lookup-network-error` | +| no response within `timeoutMs` | `timeout` | yes, except for `autocomplete` | `lookup-timeout`, `autocomplete-timeout` | - `apiCode` is set when the body is a JSON object whose `error.code` is text. The client never raises an error for an unexpected error body. -- A part that is missing or has the wrong type gives `unexpected_response`. These parts are: `valid` of a lookup, `found` of a reverse, `suggestions` of an autocomplete, a `status` that is present, not null and not text, and a postcode of a reverse result that the core cannot parse. The client never returns a guessed empty value in their place. The scenarios `lookup-malformed-body`, `lookup-missing-valid`, `lookup-status-not-text`, `reverse-missing-found` and `autocomplete-suggestions-not-list` test this rule. +- A part that is missing or has the wrong type gives `unexpected_response`. These parts are: `valid` of a lookup, `found` of a reverse, `suggestions` of an autocomplete, a `status` that is present, not null and not text, and a postcode of a reverse result that the core cannot parse. The client never returns a guessed empty value in their place. The scenarios `lookup-malformed-body`, `lookup-missing-valid`, `lookup-status-not-text`, `reverse-missing-found`, `reverse-bad-postcode` and `autocomplete-suggestions-not-list` test this rule. - A cancelled call ends with the platform's own cancellation error, not with `PostcodeError` (API-11). - No error message holds the API key (ERR-3). ## Retries -- A client makes at most `1 + maxRetries` attempts for one call. -- Before retry `n`, the first being 1, the client waits `500 x 2^(n - 1)` milliseconds, plus a random wait from 0 to 250 milliseconds. The first retry waits 500 to 750 ms, and the second waits 1000 to 1250 ms. -- A 429 with `Retry-After` of 10 seconds or less: the client waits that long, with no random part, and retries. `Retry-After` can hold seconds or an HTTP date. -- A 429 with a longer `Retry-After`: the client raises `rate_limited` at once, and `retryAfterMs` holds the wait. A caller can then tell a user when to try again. -- A 429 with no `Retry-After`, or with an invalid one: the client raises `rate_limited` at once. The gateway counts requests in each clock minute, so a retry after half a second would fail too. +- A client makes at most `1 + maxRetries` attempts for one call. The scenarios `lookup-retries-exhausted` and `lookup-max-retries-zero` test it. +- Before retry `n`, the first being 1, the client waits `500 x 2^(n - 1)` milliseconds, plus a random wait from 0 to 250 milliseconds. The first retry waits 500 to 750 ms, and the second waits 1000 to 1250 ms. The scenarios `lookup-retry-502`, `lookup-retry-503`, `lookup-retry-504` and `lookup-network-error` test this rule. +- A 429 with `Retry-After` of 10 seconds or less: the client waits that long, with no random part, and retries. `Retry-After` can hold seconds or an HTTP date. The scenarios `lookup-retry-after` and `lookup-retry-after-ten` test this rule. +- A 429 with a longer `Retry-After`: the client raises `rate_limited` at once, and `retryAfterMs` holds the wait. A caller can then tell a user when to try again. The scenarios `lookup-retry-after-eleven` and `lookup-retry-after-too-long` test this rule. +- A 429 with no `Retry-After`, or with an invalid one: the client raises `rate_limited` at once. Calls suggest that the gateway counts requests in each clock minute, so a retry after half a second would probably fail too. The scenario `lookup-rate-limited` tests it. - A 502, 503 or 504 with a valid `Retry-After` of 10 seconds or less: the client waits that long, with no random part, as for a 429. A valid `Retry-After` is a whole number of seconds from 0 up, or an HTTP date that is not in the past. With any other value, or with no header, the client uses its own wait. `retryAfterMs` holds a valid value. The scenarios `lookup-retry-after-503` and `lookup-retry-after-invalid` test this rule. - A call has a total deadline. It starts when the client sends the first attempt. It lasts `(1 + maxRetries) x timeoutMs`, plus `500 x 2^(n - 1) + 250` milliseconds for each retry `n`. With the defaults for `lookup`, that is 26 seconds. When the next wait would end after the deadline, the client raises the error of the last attempt at once. The scenario `lookup-deadline` tests this rule. -- The client does not retry a timeout of `autocomplete`. The next keystroke replaces the call. +- The client does not retry a timeout of `autocomplete`. The next keystroke replaces the call. The scenario `autocomplete-timeout` tests it. ## Sharing, queue and cache -- Two calls of the same method with the same arguments share one request while the first one is in flight. Both callers get the same result or the same error. +- Two calls of the same method with the same arguments share one request while the first one is in flight. Both callers get the same result or the same error. The scenarios `lookup-shared-request` and `lookup-shared-error` test this rule. - The key of a shared request and of a cache entry is the method with the canonical form of the arguments. So `lookup("fc01z99zz01")` and `lookup("FC-01-Z99-ZZ-01")` share one request and one cache entry. The scenario `lookup-shared-spellings` tests the request. - A shared request stops only when every caller that shares it cancels. A cancelled call frees its place among the 4 at once. -- A client sends at most 4 requests at a time. Further calls wait in a queue, in the order of the calls. -- With `cacheTtlMs` above 0, the client keeps each result for that long. A call with the same method and the same arguments then gets the kept result, and the client sends no request. Errors stay out of the cache. `clearCache()` removes every kept result. +- A client sends at most 4 requests at a time. Further calls wait in a queue, in the order of the calls. The scenario `lookup-queue-limit` tests the limit. +- With `cacheTtlMs` above 0, the client keeps each result for that long. A call with the same method and the same arguments then gets the kept result, and the client sends no request. Errors stay out of the cache. `clearCache()` removes every kept result. The scenarios `lookup-cache-hit`, `lookup-cache-off` and `lookup-error-not-cached` test this rule. ## Rules that no scenario tests @@ -172,3 +174,5 @@ Each client tests these rules itself. - The default `timeoutMs` of 15 seconds for `autocomplete`, because a scenario would wait that long. - The random part of each wait, beyond its bounds. - A `Retry-After` that holds an HTTP date. A scenario cannot hold a date that is a few seconds ahead. +- The order of the queue. A scenario counts the requests in flight, but not the order in which they leave. +- `clearCache()`. The scenario format has calls of the three methods only. diff --git a/contract/README.md b/contract/README.md index c6e187f..24aace3 100644 --- a/contract/README.md +++ b/contract/README.md @@ -40,7 +40,7 @@ The outcome above is short. A real scenario lists every field of the result. | `method` | Other fields | |---|---| -| `lookup` | `code` and `level` | +| `lookup` | `code`, and `level` when the call passes a level | | `reverse` | `lat`, `lng`, and `maxDistanceM` when the call passes a radius | | `autocomplete` | `q` | @@ -69,10 +69,10 @@ The mock server gives the first response to the first request, the second respon | `attempts` | the number of requests that reached the transport, for all calls together | | `waitsMs` | for one call, the bounds of each wait between two attempts, in milliseconds | | `maxInFlight` | the most requests that were in flight at one time | -| `request` | the first request: `method`, `path`, `query` and `apiKey`, which is null when the request had no `X-API-Key` header | +| `request` | the first request: `method`, `path`, `query` and `apiKey`, which is null when the request had no `X-API-Key` header. `query` holds every parameter of the request, and no other, with each value as text | | `outcomes` | one outcome for each call, in the order of the calls | -An outcome is `{"result": ...}` or `{"error": ...}`. A result uses the field names in `client.md`. A postcode in a result is its canonical form. An error has `code`, `status`, `apiCode` and `retryAfterMs`. +An outcome is `{"result": ...}` or `{"error": ...}`. With one call, the reply to the last attempt decides the outcome, as `client.md` says, and `make check` compares the two. A result uses the field names in `client.md`. A postcode in a result is its canonical form. An error has `code`, `status`, `apiCode` and `retryAfterMs`. A wait runs from the end of one attempt to the start of the next one. Timers can fire late, so a test can accept a wait up to 100 ms over `max`. It must not accept a wait under `min`. diff --git a/contract/autocomplete-too-long.json b/contract/autocomplete-too-long.json new file mode 100644 index 0000000..0feaba9 --- /dev/null +++ b/contract/autocomplete-too-long.json @@ -0,0 +1,28 @@ +{ + "version": 1, + "id": "autocomplete-too-long", + "description": "Text of more than 11 characters sends no request.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "autocomplete", + "q": "FC01Z99ZZ012" + } + ], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + { + "error": { + "code": "invalid_input", + "status": null, + "apiCode": null, + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/lookup-building-use-only.json b/contract/lookup-building-use-only.json new file mode 100644 index 0000000..c187836 --- /dev/null +++ b/contract/lookup-building-use-only.json @@ -0,0 +1,44 @@ +{ + "version": 1, + "id": "lookup-building-use-only", + "description": "A body with building_use_status alone gives level 3, with no address.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 3 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "building_use_status": "residential" + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": null, + "levelRequested": 3, + "levelReceived": 3, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": "residential" + } + } + ] + } +} diff --git a/contract/lookup-default-level.json b/contract/lookup-default-level.json new file mode 100644 index 0000000..9efcc5c --- /dev/null +++ b/contract/lookup-default-level.json @@ -0,0 +1,45 @@ +{ + "version": 1, + "id": "lookup-default-level", + "description": "A lookup with no level sends level 1.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01" + } + ], + "responses": [ + { + "fixture": "lookup/valid-level-1" + } + ], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/lookup", + "query": { + "code": "FC-01-Z99-ZZ-01", + "level": "1" + }, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-echo-ignored.json b/contract/lookup-echo-ignored.json new file mode 100644 index 0000000..0b0a903 --- /dev/null +++ b/contract/lookup-echo-ignored.json @@ -0,0 +1,37 @@ +{ + "version": 1, + "id": "lookup-echo-ignored", + "description": "The client keeps the caller's postcode, not the text that the gateway echoes.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "fixture": "lookup/invalid" + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": false, + "status": "invalid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-level-4.json b/contract/lookup-level-4.json new file mode 100644 index 0000000..0355ee0 --- /dev/null +++ b/contract/lookup-level-4.json @@ -0,0 +1,44 @@ +{ + "version": 1, + "id": "lookup-level-4", + "description": "A body with other_building_info gives level 4.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 4 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "other_building_info": {} + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": null, + "levelRequested": 4, + "levelReceived": 4, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-level-5.json b/contract/lookup-level-5.json new file mode 100644 index 0000000..8e9dd51 --- /dev/null +++ b/contract/lookup-level-5.json @@ -0,0 +1,44 @@ +{ + "version": 1, + "id": "lookup-level-5", + "description": "A body with point_geometry gives level 5.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 5 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "point_geometry": {} + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": null, + "levelRequested": 5, + "levelReceived": 5, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-level-not-granted.json b/contract/lookup-level-not-granted.json index ad6c6b8..0db0d95 100644 --- a/contract/lookup-level-not-granted.json +++ b/contract/lookup-level-not-granted.json @@ -1,7 +1,7 @@ { "version": 1, "id": "lookup-level-not-granted", - "description": "A level above the key's level gives forbidden. The client keeps the level.", + "description": "Too high a level gives forbidden. The request holds the caller's level.", "client": {"apiKey": "nipost_test_mock_l1"}, "calls": [{"method": "lookup", "code": "FC-01-Z99-ZZ-01", "level": 2}], "responses": [{"fixture": "errors/level-not-granted-1"}], diff --git a/contract/lookup-lower-level.json b/contract/lookup-lower-level.json new file mode 100644 index 0000000..0c96697 --- /dev/null +++ b/contract/lookup-lower-level.json @@ -0,0 +1,37 @@ +{ + "version": 1, + "id": "lookup-lower-level", + "description": "A body with fewer fields than the level asked for gives the lower level.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 3 + } + ], + "responses": [ + { + "fixture": "lookup/valid-level-1" + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 3, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-retry-after-eleven.json b/contract/lookup-retry-after-eleven.json new file mode 100644 index 0000000..68c02b8 --- /dev/null +++ b/contract/lookup-retry-after-eleven.json @@ -0,0 +1,36 @@ +{ + "version": 1, + "id": "lookup-retry-after-eleven", + "description": "A 429 with Retry-After of 11 seconds gives rate_limited at once, with the wait.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "fixture": "errors/rate-limited", + "headers": { + "Retry-After": "11" + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "rate_limited", + "status": 429, + "apiCode": "rate_limited", + "retryAfterMs": 11000 + } + } + ] + } +} diff --git a/contract/lookup-retry-after-ten.json b/contract/lookup-retry-after-ten.json new file mode 100644 index 0000000..51a5977 --- /dev/null +++ b/contract/lookup-retry-after-ten.json @@ -0,0 +1,49 @@ +{ + "version": 1, + "id": "lookup-retry-after-ten", + "description": "A 429 with Retry-After of 10 seconds, the longest wait, is retried after it.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "fixture": "errors/rate-limited", + "headers": { + "Retry-After": "10" + } + }, + { + "fixture": "lookup/valid-level-1" + } + ], + "expect": { + "attempts": 2, + "waitsMs": [ + { + "min": 10000, + "max": 10000 + } + ], + "outcomes": [ + { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true, + "status": "valid", + "levelRequested": 1, + "levelReceived": 1, + "administrativeAddress": null, + "recentHouseAddress": null, + "buildingUseStatus": null + } + } + ] + } +} diff --git a/contract/lookup-shared-error.json b/contract/lookup-shared-error.json new file mode 100644 index 0000000..8adca09 --- /dev/null +++ b/contract/lookup-shared-error.json @@ -0,0 +1,48 @@ +{ + "version": 1, + "id": "lookup-shared-error", + "description": "Two identical calls that share one request both get its error.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "order": "parallel", + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + }, + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "fixture": "errors/invalid-request", + "delayMs": 100 + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "invalid_input", + "status": 400, + "apiCode": "invalid_request", + "retryAfterMs": null + } + }, + { + "error": { + "code": "invalid_input", + "status": 400, + "apiCode": "invalid_request", + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/reverse-bad-postcode.json b/contract/reverse-bad-postcode.json new file mode 100644 index 0000000..e17d642 --- /dev/null +++ b/contract/reverse-bad-postcode.json @@ -0,0 +1,53 @@ +{ + "version": 1, + "id": "reverse-bad-postcode", + "description": "A 200 whose unit postcode does not parse gives unexpected_response.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "reverse", + "lat": 9, + "lng": 7 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "found": true, + "coordinate": [ + 7, + 9 + ], + "unit": { + "postcode": "HELLO", + "display": "FC 01 Z99 ZZ 01", + "distance_m": 4.2, + "confidence": "high" + }, + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC", + "depth": "unit", + "radius_m": 25 + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "unexpected_response", + "status": 200, + "apiCode": null, + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/reverse-invalid-negative-distance.json b/contract/reverse-invalid-negative-distance.json new file mode 100644 index 0000000..fca9d05 --- /dev/null +++ b/contract/reverse-invalid-negative-distance.json @@ -0,0 +1,30 @@ +{ + "version": 1, + "id": "reverse-invalid-negative-distance", + "description": "A radius below 0 metres sends no request.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "reverse", + "lat": 9, + "lng": 7, + "maxDistanceM": -1 + } + ], + "responses": [], + "expect": { + "attempts": 0, + "outcomes": [ + { + "error": { + "code": "invalid_input", + "status": null, + "apiCode": null, + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/reverse-max-distance-250.json b/contract/reverse-max-distance-250.json new file mode 100644 index 0000000..f84a54f --- /dev/null +++ b/contract/reverse-max-distance-250.json @@ -0,0 +1,57 @@ +{ + "version": 1, + "id": "reverse-max-distance-250", + "description": "A radius of 250 metres is a limit of the range, so the client sends it.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "reverse", + "lat": 9, + "lng": 7, + "maxDistanceM": 250 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "found": false, + "coordinate": [ + 7, + 9 + ], + "message": "no postcode found near this point", + "radius_m": 250 + } + } + } + ], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/search/reverse", + "query": { + "lat": "9", + "lng": "7", + "max_distance_m": "250" + }, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "found": false, + "radiusM": 250, + "unit": null, + "area": null, + "district": null, + "state": null + } + } + ] + } +} diff --git a/contract/reverse-max-distance-zero.json b/contract/reverse-max-distance-zero.json new file mode 100644 index 0000000..3c7e5ff --- /dev/null +++ b/contract/reverse-max-distance-zero.json @@ -0,0 +1,57 @@ +{ + "version": 1, + "id": "reverse-max-distance-zero", + "description": "A radius of 0 metres is a limit of the range, so the client sends it.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "reverse", + "lat": 9, + "lng": 7, + "maxDistanceM": 0 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "found": false, + "coordinate": [ + 7, + 9 + ], + "message": "no postcode found near this point", + "radius_m": 0 + } + } + } + ], + "expect": { + "attempts": 1, + "request": { + "method": "GET", + "path": "/v1/search/reverse", + "query": { + "lat": "9", + "lng": "7", + "max_distance_m": "0" + }, + "apiKey": "nipost_test_mock_l3" + }, + "outcomes": [ + { + "result": { + "found": false, + "radiusM": 0, + "unit": null, + "area": null, + "district": null, + "state": null + } + } + ] + } +} diff --git a/contract/reverse-unknown-confidence.json b/contract/reverse-unknown-confidence.json new file mode 100644 index 0000000..8eb583b --- /dev/null +++ b/contract/reverse-unknown-confidence.json @@ -0,0 +1,63 @@ +{ + "version": 1, + "id": "reverse-unknown-confidence", + "description": "The client maps a confidence that client.md does not list to low.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "reverse", + "lat": 9, + "lng": 7 + } + ], + "responses": [ + { + "status": 200, + "body": { + "data": { + "found": true, + "coordinate": [ + 7, + 9 + ], + "unit": { + "postcode": "FC-01-Z99-ZZ-01", + "display": "FC 01 Z99 ZZ 01", + "distance_m": 4.2, + "confidence": "very_high" + }, + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC", + "depth": "unit", + "radius_m": 25 + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "result": { + "found": true, + "radiusM": 25, + "unit": { + "postcode": "FC-01-Z99-ZZ-01", + "distanceM": 4.2, + "confidence": "low", + "stateName": null, + "lgaName": null, + "localityName": null, + "address": null + }, + "area": "FC-01-Z99-ZZ", + "district": "FC-01-Z99", + "state": "FC" + } + } + ] + } +} diff --git a/contract/reverse-unknown-fields.json b/contract/reverse-unknown-fields.json index e05ab61..eb6d99c 100644 --- a/contract/reverse-unknown-fields.json +++ b/contract/reverse-unknown-fields.json @@ -1,7 +1,7 @@ { "version": 1, "id": "reverse-unknown-fields", - "description": "The client ignores unknown fields, and maps an unknown confidence to low.", + "description": "The client ignores a field that client.md does not list.", "client": {"apiKey": "nipost_test_mock_l3"}, "calls": [{"method": "reverse", "lat": 9, "lng": 7}], "responses": [ @@ -15,7 +15,7 @@ "postcode": "FC-01-Z99-ZZ-01", "display": "FC 01 Z99 ZZ 01", "distance_m": 4.2, - "confidence": "very_high", + "confidence": "high", "floor": 2 }, "area": "FC-01-Z99-ZZ", @@ -38,7 +38,7 @@ "unit": { "postcode": "FC-01-Z99-ZZ-01", "distanceM": 4.2, - "confidence": "low", + "confidence": "high", "stateName": null, "lgaName": null, "localityName": null, diff --git a/scripts/contract_files.py b/scripts/contract_files.py index 6cbf40d..727591f 100644 --- a/scripts/contract_files.py +++ b/scripts/contract_files.py @@ -3,17 +3,20 @@ # SPDX-License-Identifier: Apache-2.0 """Check the contract scenarios, and run every check of the OpenAPI file and the fixtures. -contract/README.md defines the format of a scenario, and client.md defines the error codes. -Run python3 scripts/contract_files.py. It prints one line for each problem, and it exits with 1 -when it finds one. +contract/README.md defines the format of a scenario. client.md defines the error codes, and it +names the scenario of each rule. A scenario must also agree with the responses that it lists: the +reply to the last attempt of a call decides its outcome. Run python3 scripts/contract_files.py. +It prints one line for each problem, and it exits with 1 when it finds one. """ from __future__ import annotations +import json import re import sys +from http import HTTPStatus from pathlib import Path -from typing import Any +from typing import Any, NamedTuple import gateway_files @@ -25,7 +28,7 @@ ORDERS = ("parallel", "sequential") # For each method: the fields that a call needs, and the fields that it can add. CALL_FIELDS = { - "lookup": ({"code", "level"}, set()), + "lookup": ({"code"}, {"level"}), "reverse": ({"lat", "lng"}, {"maxDistanceM"}), "autocomplete": ({"q"}, set()), } @@ -34,49 +37,89 @@ EXPECT_FIELDS = {"attempts", "outcomes", "waitsMs", "maxInFlight", "request"} ERROR_FIELDS = ("code", "status", "apiCode", "retryAfterMs") REQUEST_FIELDS = ("method", "path", "query", "apiKey") -# The minimum that the design asks for, beside one scenario for each error code: each retry -# rule, Retry-After, the timeout, request sharing, the queue limit, unknown fields and values, -# each rule of input and the plain form of a number. -REQUIRED_SCENARIOS = ( - "lookup-retry-502", - "lookup-retry-503", - "lookup-retry-504", - "lookup-retries-exhausted", - "lookup-network-error", - "lookup-retry-after", - "lookup-rate-limited", - "lookup-timeout", - "autocomplete-timeout", - "lookup-shared-request", - "lookup-queue-limit", - "lookup-unknown-fields", - "reverse-unknown-fields", - "autocomplete-unknown-fields", - "lookup-unknown-status", - "lookup-invalid-level", - "reverse-invalid-lat", - "reverse-invalid-lng", - "reverse-invalid-max-distance", - "reverse-plain-decimal", - "reverse-whole-number", - "lookup-deadline", - "lookup-retry-after-503", - "lookup-retry-after-invalid", - "lookup-shared-spellings", - "lookup-missing-valid", - "lookup-status-not-text", - "reverse-missing-found", - "autocomplete-suggestions-not-list", +SHAPES = { + "client": (dict, "an object"), + "calls": (list, "a list"), + "responses": (list, "a list"), + "expect": (dict, "an object"), +} +# client.md gives these defaults. +DEFAULT_LEVEL = 1 +DEFAULT_MAX_RETRIES = 2 +# A scenario id in client.md: the method, a hyphen and the rule, in backticks. +SCENARIO_NAME = re.compile(r"`((?:lookup|reverse|autocomplete)-[a-z0-9-]+)`") +ERROR_ROW = re.compile(r"^\| [^|]+ \| `[a-z_]+` \|") +# The error map of client.md for a response that arrived. A test compares it with the table. +STATUS_CODES = {401: "unauthorized", 402: "insufficient_credits", 429: "rate_limited"} +RETRIED_STATUSES = (502, 503, 504) +CLIENT_ERRORS = range(400, 500) +RETRY_AFTER_STATUSES = (429, *RETRIED_STATUSES) +NO_RESPONSE_CODES = {"hang": "timeout", "drop": "network_error"} +METHOD_SCHEMAS = { + "lookup": "LookupEnvelope", + "reverse": "ReverseEnvelope", + "autocomplete": "AutocompleteEnvelope", +} +# The fields of a lookup body that raise the level of the data, from the highest level down. +LEVEL_FIELDS = ( + ("point_geometry", 5), + ("other_building_info", 4), + ("building_use_status", 3), + ("administrative_address", 2), + ("recent_house_address", 2), ) +CONFIDENCES = ("high", "medium", "low") +UNIT_FIELDS = ("postcode", "distanceM", "confidence") + + +class ErrorRow(NamedTuple): + """One row of the error table in client.md.""" + + condition: str + code: str + scenarios: tuple[str, ...] + + +class Reply(NamedTuple): + """What the mock server sends for one response of a scenario.""" + + kind: str + status: int | None + body: Any + headers: dict[str, str] + + +def error_rows(text: str) -> list[ErrorRow]: + """Return the rows of the error table in the Errors section of client.md, in order.""" + section = text.partition("\n## Errors\n")[2].partition("\n## ")[0] + if not section.strip(): + raise ValueError("client.md has no Errors section.") + rows = [] + for line in section.splitlines(): + if ERROR_ROW.match(line): + cells = [cell.strip() for cell in line.strip().strip("|").split("|")] + names = SCENARIO_NAME.findall(" ".join(cells[3:])) + rows.append(ErrorRow(cells[0], cells[1].strip("`"), tuple(names))) + return rows def error_codes(path: Path = CLIENT_DOC) -> list[str]: """Return the error codes in the table of the Errors section of client.md, in order.""" - section = path.read_text(encoding="utf-8").partition("\n## Errors\n")[2].partition("\n## ")[0] - if not section.strip(): - raise ValueError(f"{path} has no Errors section.") - codes = re.findall(r"^\| [^|]+ \| `([a-z_]+)` \|", section, flags=re.MULTILINE) - return list(dict.fromkeys(codes)) + rows = error_rows(path.read_text(encoding="utf-8")) + return list(dict.fromkeys(row.code for row in rows)) + + +def error_code(status: int, api_code: str | None) -> str: + """Return the error code that the error map of client.md gives for a response.""" + if status in STATUS_CODES: + return STATUS_CODES[status] + if status == HTTPStatus.FORBIDDEN: + return "origin_not_allowed" if api_code == "origin_not_allowed" else "forbidden" + if status == HTTPStatus.OK: + return "unexpected_response" + if status in RETRIED_STATUSES or status not in CLIENT_ERRORS: + return "server_error" + return "invalid_input" def load_scenarios(root: Path = CONTRACT) -> dict[str, Any]: @@ -92,11 +135,12 @@ def call_problems(call: Any) -> list[str]: needed, optional = CALL_FIELDS[method] fields = set(call) - {"method"} if not needed <= fields <= needed | optional: - return [f"A {method} call needs {', '.join(sorted(needed))}, and nothing else."] + extra = f" and can hold {', '.join(sorted(optional))}" if optional else "" + return [f"A {method} call needs {', '.join(sorted(needed))}{extra}, and nothing else."] return [] -def response_problems(response: Any, fixtures: set[str]) -> list[str]: +def response_problems(response: Any, fixtures: dict[str, Any]) -> list[str]: """Return the problems of one response of the mock server.""" if not isinstance(response, dict): return ["Each response must be an object."] @@ -111,7 +155,9 @@ def response_problems(response: Any, fixtures: set[str]) -> list[str]: problems.append(f"A {kind} response must not have a status.") if kind in ("hang", "drop") and (response[kind] is not True or "headers" in response): problems.append(f"A {kind} response needs the value true, and no headers.") - if kind == "fixture" and response["fixture"] not in fixtures: + if kind == "fixture" and ( + type(response["fixture"]) is not str or response["fixture"] not in fixtures + ): problems.append(f"No fixture is named {response['fixture']}.") delay = response.get("delayMs", 0) if type(delay) is not int or delay < 0: @@ -136,8 +182,8 @@ def outcome_problems(outcome: Any, codes: list[str]) -> list[str]: def expect_problems(scenario: dict[str, Any], codes: list[str]) -> list[str]: """Return the problems of the expect object of a scenario.""" expect = scenario["expect"] - if not isinstance(expect, dict) or not {"attempts", "outcomes"} <= set(expect): - return ["expect needs attempts and outcomes."] + if not {"attempts", "outcomes"} <= set(expect) or not isinstance(expect["outcomes"], list): + return ["expect needs attempts and a list of outcomes."] problems = [] if set(expect) - EXPECT_FIELDS: problems.append(f"expect has an unknown field: {sorted(set(expect) - EXPECT_FIELDS)}.") @@ -157,10 +203,22 @@ def expect_problems(scenario: dict[str, Any], codes: list[str]) -> list[str]: return problems -def scenario_problems(name: str, scenario: Any, fixtures: set[str], codes: list[str]) -> list[str]: - """Return the problems of one scenario, each line led by its file name.""" +def scenario_problems( + name: str, scenario: Any, fixtures: dict[str, Any], codes: list[str] +) -> list[str]: + """Return the problems of one scenario, each line led by its file name. + + The checks of truth run only on a scenario that keeps the format. + """ if not isinstance(scenario, dict) or not set(TOP_FIELDS) <= set(scenario): return [f"{name}: a scenario needs the fields {', '.join(TOP_FIELDS)}."] + wrong = [ + f"{name}: {field} must be {word}." + for field, (kind, word) in SHAPES.items() + if not isinstance(scenario[field], kind) + ] + if wrong: + return wrong problems = [] if scenario["version"] != 1 or scenario["id"] != name: problems.append("The version must be 1, and the id must equal the file name.") @@ -175,26 +233,196 @@ def scenario_problems(name: str, scenario: Any, fixtures: set[str], codes: list[ for response in scenario["responses"]: problems += response_problems(response, fixtures) problems += expect_problems(scenario, codes) + if not problems: + problems = truth_problems(scenario, fixtures) return [f"{name}: {problem}" for problem in problems] -def coverage_problems(scenarios: dict[str, Any], codes: list[str]) -> list[str]: - """Return the error codes and the required scenarios that no scenario covers.""" - reached = { - outcome["error"]["code"] - for scenario in scenarios.values() - for outcome in scenario["expect"]["outcomes"] - if "error" in outcome +def reply_of(response: dict[str, Any], fixtures: dict[str, Any]) -> Reply: + """Return what the mock server sends for one response of a scenario.""" + kind = next(kind for kind in RESPONSE_KINDS if kind in response) + headers = response.get("headers", {}) + if kind == "fixture": + fixture = fixtures[response["fixture"]] + return Reply(kind, fixture["status"], fixture["body"], headers) + return Reply(kind, response.get("status"), response.get("body"), headers) + + +def last_reply(scenario: dict[str, Any], fixtures: dict[str, Any]) -> Reply | None: + """Return the reply to the last attempt of a scenario, or None when no request goes out. + + The mock server repeats the last response when the requests outnumber the responses. + """ + attempts = scenario["expect"]["attempts"] + if attempts == 0: + return None + responses = scenario["responses"] + return reply_of(responses[min(attempts, len(responses)) - 1], fixtures) + + +def api_code(body: Any) -> str | None: + """Return the `error.code` of a body when it is text, as client.md reads it.""" + error = body.get("error") if isinstance(body, dict) else None + code = error.get("code") if isinstance(error, dict) else None + return code if isinstance(code, str) else None + + +def retry_after_ms(reply: Reply) -> int | None: + """Return the wait of a valid Retry-After in seconds, when the status can carry one.""" + value = reply.headers.get("Retry-After", "") + if reply.status not in RETRY_AFTER_STATUSES or not re.fullmatch(r"[0-9]+", value): + return None + return int(value) * 1000 + + +def expected_error(reply: Reply | None) -> dict[str, Any]: + """Return the error that client.md gives for the reply to the last attempt.""" + if reply is None or reply.status is None: + code = "invalid_input" if reply is None else NO_RESPONSE_CODES[reply.kind] + return {"code": code, "status": None, "apiCode": None, "retryAfterMs": None} + return { + "code": error_code(reply.status, api_code(reply.body)), + "status": reply.status, + "apiCode": api_code(reply.body), + "retryAfterMs": retry_after_ms(reply), } + + +def level_received(data: dict[str, Any]) -> int: + """Return the lookup level of the data, from the fields that it holds.""" + return next((level for field, level in LEVEL_FIELDS if field in data), DEFAULT_LEVEL) + + +def body_fields(call: dict[str, Any], data: dict[str, Any]) -> dict[str, Any]: + """Return the fields of the result that client.md takes from the body, for one call.""" + if call["method"] == "lookup": + return { + "valid": data.get("valid"), + "status": data.get("status"), + "levelRequested": call.get("level", DEFAULT_LEVEL), + "levelReceived": level_received(data), + } + if call["method"] == "autocomplete": + codes = [suggestion.get("code") for suggestion in data.get("suggestions", [])] + return {"segment": data.get("segment"), "suggestions": codes} + fields = {key: data.get(key) for key in ("found", "area", "district", "state")} + fields["radiusM"] = data.get("radius_m") + unit = data.get("unit") + fields["unit"] = None + if isinstance(unit, dict): + confidence = unit.get("confidence") + fields["unit"] = { + "postcode": unit.get("postcode"), + "distanceM": unit.get("distance_m"), + "confidence": confidence if confidence in CONFIDENCES else "low", + } + return fields + + +def result_fields(result: dict[str, Any], names: list[str]) -> dict[str, Any]: + """Return the named fields of an expected result, in the form that body_fields gives.""" + fields = {name: result.get(name) for name in names} + unit, suggestions = fields.get("unit"), fields.get("suggestions") + if isinstance(unit, dict): + fields["unit"] = {name: unit.get(name) for name in UNIT_FIELDS} + if isinstance(suggestions, list): + fields["suggestions"] = [suggestion.get("code") for suggestion in suggestions] + return fields + + +def outcome_truth(call: dict[str, Any], outcome: dict[str, Any], reply: Reply | None) -> list[str]: + """Return the problems of the outcome of a call against the reply to its last attempt.""" + if "error" in outcome: + expected = expected_error(reply) + if outcome["error"] == expected: + return [] + return [ + f"The last response gives {json.dumps(expected)}, not {json.dumps(outcome['error'])}." + ] + data = reply.body.get("data") if reply and isinstance(reply.body, dict) else None + if reply is None or reply.status != HTTPStatus.OK or not isinstance(data, dict): + return ["A result needs a last response with the status 200 and an object in data."] + expected = body_fields(call, data) + stated = result_fields(outcome["result"], list(expected)) + return [ + f"The last response gives {name} {expected[name]!r}, not {stated[name]!r}." + for name in expected + if expected[name] != stated[name] + ] + + +def fixture_method_problems(scenario: dict[str, Any], fixtures: dict[str, Any]) -> list[str]: + """Return a problem for each fixture that answers another method than the calls use.""" + methods = {call["method"] for call in scenario["calls"]} + allowed = {gateway_files.ERROR_SCHEMA, *(METHOD_SCHEMAS[method] for method in methods)} + names = [response["fixture"] for response in scenario["responses"] if "fixture" in response] + return [ + f"A {' or '.join(sorted(methods))} call cannot get {name}." + for name in names + if fixtures[name]["schema"] not in allowed + ] + + +def truth_problems(scenario: dict[str, Any], fixtures: dict[str, Any]) -> list[str]: + """Return the problems of a scenario whose outcomes disagree with its responses. + + With one call, the reply to the last attempt decides the outcome, as client.md says. Calls + that share the responses, a cache or a queue get no such check, but each of their errors + must still fit its own status and API code. + """ + calls, expect = scenario["calls"], scenario["expect"] + limit = len(calls) * (1 + scenario["client"].get("maxRetries", DEFAULT_MAX_RETRIES)) + problems = fixture_method_problems(scenario, fixtures) + if expect["attempts"] > limit: + problems.append(f"{expect['attempts']} attempts are more than the calls allow ({limit}).") + if len(calls) == 1: + return problems + outcome_truth( + calls[0], expect["outcomes"][0], last_reply(scenario, fixtures) + ) + errors = [outcome["error"] for outcome in expect["outcomes"] if "error" in outcome] + return problems + [ + f"The status {error['status']} gives {error_code(error['status'], error['apiCode'])}, " + f"not {error['code']}." + for error in errors + if error["status"] is not None + and error_code(error["status"], error["apiCode"]) != error["code"] + ] + + +def coverage_problems(scenarios: dict[str, Any], client_text: str) -> list[str]: + """Return the problems of the link between client.md and the scenarios. + + client.md names the scenario of each rule. Each name needs a file, and each file needs a + name. Each row of the error table names at least one scenario, and each of them ends with the + code of its row. + """ + named = set(SCENARIO_NAME.findall(client_text)) problems = [ - f"No scenario ends with the error code {code}." for code in codes if code not in reached + f"client.md names the scenario {name}, which has no file." + for name in sorted(named - set(scenarios)) ] problems += [ - f"The scenario {name} is missing." for name in REQUIRED_SCENARIOS if name not in scenarios + f"client.md names no rule for the scenario {name}." + for name in sorted(set(scenarios) - named) ] + for row in error_rows(client_text): + if not row.scenarios: + problems.append(f"The error row {row.condition} in client.md names no scenario.") + problems += [ + f"The scenario {name} does not end with {row.code}, as its row says." + for name in row.scenarios + if name in scenarios and row.code not in final_codes(scenarios[name]) + ] return problems +def final_codes(scenario: dict[str, Any]) -> set[str]: + """Return the error codes that the calls of a scenario end with.""" + return { + outcome["error"]["code"] for outcome in scenario["expect"]["outcomes"] if "error" in outcome + } + + def synthetic_files() -> list[Path]: """Return the files that may hold only synthetic postcodes and mock keys. @@ -220,8 +448,8 @@ def all_problems() -> list[str]: gateway_files.read_json(gateway_files.FIXTURES / "keys.json") ) for name, scenario in scenarios.items(): - problems += scenario_problems(name, scenario, set(fixtures), codes) - problems += coverage_problems(scenarios, codes) + problems += scenario_problems(name, scenario, fixtures, codes) + problems += coverage_problems(scenarios, CLIENT_DOC.read_text(encoding="utf-8")) files = synthetic_files() return ( problems + gateway_files.postcode_problems(files) + gateway_files.key_text_problems(files) diff --git a/scripts/tests/test_client_doc.py b/scripts/tests/test_client_doc.py index e678e73..2434556 100644 --- a/scripts/tests/test_client_doc.py +++ b/scripts/tests/test_client_doc.py @@ -86,7 +86,7 @@ def test_each_input_rule_names_its_scenario(self) -> None: ["lookup-invalid-level"], ["reverse-invalid-lat"], ["reverse-invalid-lng"], - ["reverse-invalid-max-distance"], + ["reverse-invalid-max-distance", "reverse-invalid-negative-distance"], ], ) diff --git a/scripts/tests/test_contract_files.py b/scripts/tests/test_contract_files.py index 2dd757a..640be9e 100644 --- a/scripts/tests/test_contract_files.py +++ b/scripts/tests/test_contract_files.py @@ -12,8 +12,9 @@ import gateway_files SCENARIOS = contract_files.load_scenarios() -FIXTURE_NAMES = set(gateway_files.load_fixtures()) +FIXTURES = gateway_files.load_fixtures() CODES = contract_files.error_codes() +CLIENT = contract_files.CLIENT_DOC.read_text(encoding="utf-8") STANDARDS = gateway_files.ROOT / "standards" / "CODING_STANDARDS.md" FILES = contract_files.synthetic_files() @@ -21,7 +22,15 @@ def problems_of(name: str, edit: Any) -> list[str]: scenario = copy.deepcopy(SCENARIOS[name]) edit(scenario) - return contract_files.scenario_problems(name, scenario, FIXTURE_NAMES, CODES) + return contract_files.scenario_problems(name, scenario, FIXTURES, CODES) + + +def without(name: str) -> dict[str, Any]: + return {other: scenario for other, scenario in SCENARIOS.items() if other != name} + + +def error_row(condition: str) -> str: + return next(line for line in CLIENT.splitlines() if line.startswith(f"| {condition} |")) class ClientDocTest(unittest.TestCase): @@ -64,11 +73,11 @@ def test_each_scenario_has_no_problem(self) -> None: for name, scenario in SCENARIOS.items(): with self.subTest(name): self.assertEqual( - contract_files.scenario_problems(name, scenario, FIXTURE_NAMES, CODES), [] + contract_files.scenario_problems(name, scenario, FIXTURES, CODES), [] ) def test_the_scenarios_cover_each_error_code_and_each_required_rule(self) -> None: - self.assertEqual(contract_files.coverage_problems(SCENARIOS, CODES), []) + self.assertEqual(contract_files.coverage_problems(SCENARIOS, CLIENT), []) def test_the_whole_check_finds_no_problem(self) -> None: self.assertEqual(contract_files.all_problems(), []) @@ -154,7 +163,7 @@ def add_field(scenario: dict[str, Any]) -> None: self.assertEqual( problems_of("lookup-level-1", add_field), - ["lookup-level-1: A lookup call needs code, level, and nothing else."], + ["lookup-level-1: A lookup call needs code and can hold level, and nothing else."], ) def test_rejects_a_client_option_that_client_md_does_not_name(self) -> None: @@ -169,27 +178,148 @@ def test_rejects_an_attempt_with_no_response(self) -> None: ["lookup-level-1: No attempt means no response, and the other way round."], ) + def test_accepts_a_lookup_call_with_no_level(self) -> None: + self.assertEqual(problems_of("lookup-level-1", lambda s: s["calls"][0].pop("level")), []) -class CoverageTest(unittest.TestCase): - def test_reports_an_error_code_that_no_scenario_reaches(self) -> None: - scenarios = { - name: scenario - for name, scenario in SCENARIOS.items() - if name != "lookup-origin-not-allowed" + def test_names_a_field_of_the_wrong_type_instead_of_failing(self) -> None: + def name_a_list(scenario: dict[str, Any]) -> None: + scenario["responses"][0]["fixture"] = ["lookup/valid-level-1"] + + cases = { + "calls": (lambda s: s.update(calls=None), "calls must be a list."), + "outcomes": ( + lambda s: s["expect"].update(outcomes=5), + "expect needs attempts and a list of outcomes.", + ), + "fixture": (name_a_list, "No fixture is named ['lookup/valid-level-1']."), } + for field, (edit, problem) in cases.items(): + with self.subTest(field): + self.assertEqual( + problems_of("lookup-level-1", edit), [f"lookup-level-1: {problem}"] + ) + + +class ScenarioTruthTest(unittest.TestCase): + def test_rejects_an_error_code_that_the_last_response_does_not_give(self) -> None: + def say_forbidden(scenario: dict[str, Any]) -> None: + scenario["expect"]["outcomes"][0]["error"]["code"] = "forbidden" + + [problem] = problems_of("lookup-bad-request", say_forbidden) + self.assertTrue( + problem.startswith( + 'lookup-bad-request: The last response gives {"code": "invalid_input", ' + ) + ) + + def test_rejects_a_status_that_the_last_response_does_not_have(self) -> None: + def say_200(scenario: dict[str, Any]) -> None: + scenario["expect"]["outcomes"][0]["error"]["status"] = 200 + + [problem] = problems_of("lookup-server-error", say_200) + self.assertIn('"status": 500', problem) + + def test_rejects_a_result_that_the_fixture_does_not_give(self) -> None: + def send_level_3(scenario: dict[str, Any]) -> None: + scenario["responses"][0]["fixture"] = "lookup/valid-level-3" + + self.assertEqual( + problems_of("lookup-level-1", send_level_3), + [ + "lookup-level-1: The last response gives status None, not 'valid'.", + "lookup-level-1: The last response gives levelReceived 3, not 1.", + ], + ) + + def test_rejects_a_fixture_of_another_method(self) -> None: + def send_reverse(scenario: dict[str, Any]) -> None: + scenario["responses"][0]["fixture"] = "reverse/unit" + + problems = problems_of("lookup-level-1", send_reverse) + self.assertEqual(problems[0], "lookup-level-1: A lookup call cannot get reverse/unit.") + + def test_rejects_more_attempts_than_the_retries_allow(self) -> None: + self.assertEqual( + problems_of("lookup-level-1", lambda s: s["expect"].update(attempts=4)), + ["lookup-level-1: 4 attempts are more than the calls allow (3)."], + ) + + def test_rejects_a_code_that_does_not_fit_its_status_when_calls_share_responses(self) -> None: + def say_forbidden(scenario: dict[str, Any]) -> None: + scenario["expect"]["outcomes"][0]["error"]["code"] = "forbidden" + self.assertEqual( - contract_files.coverage_problems(scenarios, CODES), - ["No scenario ends with the error code origin_not_allowed."], + problems_of("lookup-error-not-cached", say_forbidden), + ["lookup-error-not-cached: The status 400 gives invalid_input, not forbidden."], ) - def test_reports_a_required_scenario_that_is_missing(self) -> None: - scenarios = { - name: scenario for name, scenario in SCENARIOS.items() if name != "lookup-queue-limit" + def test_reads_the_retry_after_of_the_last_response(self) -> None: + def drop_wait(scenario: dict[str, Any]) -> None: + scenario["expect"]["outcomes"][0]["error"]["retryAfterMs"] = None + + [problem] = problems_of("lookup-retry-after-too-long", drop_wait) + self.assertIn('"retryAfterMs": 120000', problem) + + +class StatusMapTest(unittest.TestCase): + def test_maps_each_status_as_the_error_table_of_client_md_does(self) -> None: + samples: dict[str, list[tuple[int, str | None]]] = { + "status 401": [(401, "auth_required")], + "status 402": [(402, "insufficient_credits")], + "status 403, with the API code `origin_not_allowed`": [(403, "origin_not_allowed")], + "status 403, with any other API code": [(403, "level_not_granted"), (403, None)], + "status 429": [(429, "rate_limited")], + "status 502, 503 or 504": [(502, None), (503, "unavailable"), (504, None)], + "any other status from 400 to 499": [(400, None), (404, "not_found"), (499, None)], + "any other status that is not 200": [(500, "internal"), (501, None), (302, None)], } + rows = [row for row in contract_files.error_rows(CLIENT) if "status" in row.condition] self.assertEqual( - contract_files.coverage_problems(scenarios, CODES), - ["The scenario lookup-queue-limit is missing."], + [row.condition for row in rows if not row.condition.startswith("status 200")], + list(samples), ) + for row in rows: + for status, api_code in samples.get(row.condition, [(200, None)]): + with self.subTest(status=status, api_code=api_code): + self.assertEqual(contract_files.error_code(status, api_code), row.code) + + +class CoverageTest(unittest.TestCase): + def test_reports_each_scenario_that_is_deleted(self) -> None: + for name in SCENARIOS: + with self.subTest(name): + self.assertIn( + f"client.md names the scenario {name}, which has no file.", + contract_files.coverage_problems(without(name), CLIENT), + ) + + def test_reports_a_scenario_that_client_md_does_not_name(self) -> None: + scenarios = {**SCENARIOS, "lookup-level-one": SCENARIOS["lookup-level-1"]} + self.assertEqual( + contract_files.coverage_problems(scenarios, CLIENT), + ["client.md names no rule for the scenario lookup-level-one."], + ) + + def test_reports_an_error_row_with_no_scenario(self) -> None: + row = error_row("status 402") + text = CLIENT.replace(row, row.replace("`lookup-insufficient-credits`", "")) + self.assertEqual( + contract_files.coverage_problems(without("lookup-insufficient-credits"), text), + ["The error row status 402 in client.md names no scenario."], + ) + + def test_reports_a_scenario_of_a_row_that_ends_with_another_code(self) -> None: + row = error_row("status 402") + text = CLIENT.replace(row, f"{row} `lookup-no-key` |") + self.assertEqual( + contract_files.coverage_problems(SCENARIOS, text), + ["The scenario lookup-no-key does not end with insufficient_credits, as its row says."], + ) + + def test_reads_the_scenarios_of_each_error_row(self) -> None: + rows = {row.condition: row for row in contract_files.error_rows(CLIENT)} + self.assertEqual(rows["status 429"].code, "rate_limited") + self.assertIn("lookup-rate-limited", rows["status 429"].scenarios) class SyntheticFilesTest(unittest.TestCase): From 16b5a5ba38fe18c0bc447f299aff37d5b7a70df6 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 01:34:51 +0100 Subject: [PATCH 11/13] fix: play each scenario as the client contract says The check of a scenario compared only the last outcome, so a scenario could keep a wrong count of attempts, a wrong wait, a wrong query or an error for a good reply. The check now plays the calls as client.md says: the retries and their waits, the deadline, shared requests, kept results and the limit of four requests in flight. It compares the attempts, the waits, the first request and each outcome with the scenario, and a good reply must give a result. The play lives in its own module, so the file of format checks stays short. Each rule of client.md now names its scenario or is left to each client, and a body above lookup level 1 cannot claim to be observed. A new scenario covers a reply with no data field. Signed-off-by: MobileMage --- client.md | 10 +- contract/README.md | 4 +- contract/lookup-no-data.json | 39 +++ contract/reverse-unit.json | 2 +- fixtures/README.md | 2 +- fixtures/reverse/unit.json | 4 +- openapi/gateway.completed.yaml | 21 +- scripts/contract_files.py | 218 +----------- scripts/gateway_files.py | 44 ++- scripts/scenario_truth.py | 481 +++++++++++++++++++++++++++ scripts/tests/test_client_doc.py | 35 ++ scripts/tests/test_contract_files.py | 25 +- scripts/tests/test_gateway_files.py | 26 ++ scripts/tests/test_scenario_truth.py | 137 ++++++++ 14 files changed, 806 insertions(+), 242 deletions(-) create mode 100644 contract/lookup-no-data.json create mode 100644 scripts/scenario_truth.py create mode 100644 scripts/tests/test_scenario_truth.py diff --git a/client.md b/client.md index 4cfebf5..7cc0ab9 100644 --- a/client.md +++ b/client.md @@ -46,10 +46,10 @@ A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` | `buildingUseStatus` | text, or null | - The client never takes `postcode` from the response. The gateway echoes the caller's text there, also for text that is not a postcode. The scenario `lookup-echo-ignored` tests this rule. -- `status` is null when the response has no `status`. The client returns any other value of `status` as it is, also when this table does not list it, so that a new gateway status is not lost. +- `status` is null when the response has no `status`. The client returns any other value of `status` as it is, also when this table does not list it, so that a new gateway status is not lost. The scenarios `lookup-level-2` and `lookup-unknown-status` test this rule. - `levelReceived` comes from the fields of the response. It is 5 when `point_geometry` is present, 4 for `other_building_info`, 3 for `building_use_status`, and 2 for either address field. Otherwise it is 1. It can be lower than `levelRequested`. The scenarios `lookup-level-2`, `lookup-level-3`, `lookup-level-4`, `lookup-level-5`, `lookup-building-use-only` and `lookup-lower-level` test this rule. - `recentHouseAddress` is the text in `recent_house_address.recent`. The scenario `lookup-level-2` tests it. -- A `status` that is present, not null and not text raises `unexpected_response`. +- A `status` that is present, not null and not text raises `unexpected_response`. The scenario `lookup-status-not-text` tests it. - A response with `valid: false` is a result, not an error. The scenario `lookup-not-found` tests it. ### Reverse result @@ -77,7 +77,7 @@ A unit has these fields: - `found` can be true while `unit` is null. The gateway then found an area, but no unit within the radius. The scenario `reverse-area` tests it. - A response with `found: false` is a result, not an error. The scenario `reverse-not-found` tests it. - `confidence` is `low` for a value that this table does not list. The scenario `reverse-unknown-confidence` tests it. -- The gateway sends the coordinate as `[lng, lat]`. The client does not return it. +- The gateway sends the coordinate as `[lng, lat]`. The client does not return it. The scenario `reverse-unit` tests it. ### Autocomplete result @@ -139,8 +139,8 @@ The error code comes from the first row that matches. | no response, because the connection failed | `network_error` | yes | `lookup-network-error` | | no response within `timeoutMs` | `timeout` | yes, except for `autocomplete` | `lookup-timeout`, `autocomplete-timeout` | -- `apiCode` is set when the body is a JSON object whose `error.code` is text. The client never raises an error for an unexpected error body. -- A part that is missing or has the wrong type gives `unexpected_response`. These parts are: `valid` of a lookup, `found` of a reverse, `suggestions` of an autocomplete, a `status` that is present, not null and not text, and a postcode of a reverse result that the core cannot parse. The client never returns a guessed empty value in their place. The scenarios `lookup-malformed-body`, `lookup-missing-valid`, `lookup-status-not-text`, `reverse-missing-found`, `reverse-bad-postcode` and `autocomplete-suggestions-not-list` test this rule. +- `apiCode` is set when the body is a JSON object whose `error.code` is text. The client never raises an error for an unexpected error body. The scenarios `lookup-bad-request` and `lookup-malformed-body` test this rule. +- A part that is missing or has the wrong type gives `unexpected_response`. These parts are: `valid` of a lookup, `found` of a reverse, `suggestions` of an autocomplete, a `status` that is present, not null and not text, and a postcode of a reverse result that the core cannot parse. The client never returns a guessed empty value in their place. The scenarios `lookup-malformed-body`, `lookup-missing-valid`, `lookup-status-not-text`, `lookup-no-data`, `reverse-missing-found`, `reverse-bad-postcode` and `autocomplete-suggestions-not-list` test this rule. - A cancelled call ends with the platform's own cancellation error, not with `PostcodeError` (API-11). - No error message holds the API key (ERR-3). diff --git a/contract/README.md b/contract/README.md index 24aace3..48a66c1 100644 --- a/contract/README.md +++ b/contract/README.md @@ -72,9 +72,9 @@ The mock server gives the first response to the first request, the second respon | `request` | the first request: `method`, `path`, `query` and `apiKey`, which is null when the request had no `X-API-Key` header. `query` holds every parameter of the request, and no other, with each value as text | | `outcomes` | one outcome for each call, in the order of the calls | -An outcome is `{"result": ...}` or `{"error": ...}`. With one call, the reply to the last attempt decides the outcome, as `client.md` says, and `make check` compares the two. A result uses the field names in `client.md`. A postcode in a result is its canonical form. An error has `code`, `status`, `apiCode` and `retryAfterMs`. +An outcome is `{"result": ...}` or `{"error": ...}`. `make check` plays each scenario as `client.md` says. It checks the attempts, the waits, the requests in flight, the first request and each outcome. The reply to the last attempt of a call decides its outcome. A result uses the field names in `client.md`. A postcode in a result is its canonical form. An error has `code`, `status`, `apiCode` and `retryAfterMs`. -A wait runs from the end of one attempt to the start of the next one. Timers can fire late, so a test can accept a wait up to 100 ms over `max`. It must not accept a wait under `min`. +A wait runs from the end of one attempt to the start of the next one. Timers can fire late, so a test can accept a wait up to 100 ms over `max`. It must not accept a wait under `min`. The scenario `lookup-retry-after-ten` waits 10 seconds, so a runner needs a time limit of at least 12 seconds for it. ## Running a scenario diff --git a/contract/lookup-no-data.json b/contract/lookup-no-data.json new file mode 100644 index 0000000..aa79063 --- /dev/null +++ b/contract/lookup-no-data.json @@ -0,0 +1,39 @@ +{ + "version": 1, + "id": "lookup-no-data", + "description": "A 200 with a JSON body but no data field gives unexpected_response.", + "client": { + "apiKey": "nipost_test_mock_l3" + }, + "calls": [ + { + "method": "lookup", + "code": "FC-01-Z99-ZZ-01", + "level": 1 + } + ], + "responses": [ + { + "status": 200, + "body": { + "result": { + "postcode": "FC-01-Z99-ZZ-01", + "valid": true + } + } + } + ], + "expect": { + "attempts": 1, + "outcomes": [ + { + "error": { + "code": "unexpected_response", + "status": 200, + "apiCode": null, + "retryAfterMs": null + } + } + ] + } +} diff --git a/contract/reverse-unit.json b/contract/reverse-unit.json index f23717a..547db0e 100644 --- a/contract/reverse-unit.json +++ b/contract/reverse-unit.json @@ -21,7 +21,7 @@ "unit": { "postcode": "FC-01-Z99-ZZ-01", "distanceM": 4.2, - "confidence": "high", + "confidence": "low", "stateName": null, "lgaName": null, "localityName": null, diff --git a/fixtures/README.md b/fixtures/README.md index 783bfa1..bb95813 100644 --- a/fixtures/README.md +++ b/fixtures/README.md @@ -34,7 +34,7 @@ The mock server answers with these files. Each file holds one response of NIPOST | `reverse` at any other coordinate | `reverse/not-found.json` | | `nearby` | `nearby/empty.json` | | an error | the file in `errors/` with the same error code | -| a path that the gateway does not serve | `errors/unknown-path.json` | +| a path that the gateway does not serve | `errors/unknown-path.json`, which `x-gatepost-unknown-path` of the OpenAPI file describes | - The mock server puts the caller's text, in upper case, in `postcode` of a lookup body, as the gateway does. - It puts the request's coordinate and the applied radius in a reverse body. diff --git a/fixtures/reverse/unit.json b/fixtures/reverse/unit.json index 20127a3..e84511a 100644 --- a/fixtures/reverse/unit.json +++ b/fixtures/reverse/unit.json @@ -1,6 +1,6 @@ { "version": 1, - "description": "A point 4.2 m from the synthetic unit.", + "description": "A point 4.2 m from the synthetic unit. Calls showed the confidence low only.", "evidence": "observed", "schema": "ReverseEnvelope", "status": 200, @@ -15,7 +15,7 @@ "postcode": "FC-01-Z99-ZZ-01", "display": "FC 01 Z99 ZZ 01", "distance_m": 4.2, - "confidence": "high" + "confidence": "low" }, "area": "FC-01-Z99-ZZ", "district": "FC-01-Z99", diff --git a/openapi/gateway.completed.yaml b/openapi/gateway.completed.yaml index 614002e..d1c4279 100644 --- a/openapi/gateway.completed.yaml +++ b/openapi/gateway.completed.yaml @@ -8,14 +8,14 @@ info: description: >- Gatepost wrote this description of NIPOST's public postcode gateway. It adds the responses, error codes and headers that NIPOST's OpenAPI file 0.1.0 leaves out. Each response and each - header carries x-gatepost-evidence. "observed" means that Gatepost saw the shape in its own - calls on 2 Oct 2026. "documented" means that NIPOST's docs describe it, but no call showed - it. "assumed" means that neither source shows it, and Gatepost's mock server chose it. An - error response lists its error codes in x-gatepost-error-codes, each with its own mark, and - a code never claims more than its response. Gatepost's calls used a key with lookup level 1 - only, so every lookup body above level 1 is documented, not observed. The calls without a - key reached lookup and autocomplete only. x-gatepost-unknown-path describes the answer to a - path that this file does not list. + header carries x-gatepost-evidence, and so does each field of a body that no call showed. + "observed" means that Gatepost saw the shape in its own calls on 2 Oct 2026. "documented" means + that NIPOST's docs describe it, but no call showed it. "assumed" means that neither source shows + it, and Gatepost's mock server chose it. An error response lists its error codes in + x-gatepost-error-codes, each with its own mark, and a code never claims more than its response. + Gatepost's calls used a key with lookup level 1 only, so every lookup body above level 1 is + documented, not observed. The calls without a key reached lookup and autocomplete only. + x-gatepost-unknown-path describes the answer to a path that this file does not list. license: name: Apache-2.0 identifier: Apache-2.0 @@ -420,16 +420,20 @@ components: building_use_status: type: string description: Level 3 and above. Documented, not observed. + x-gatepost-evidence: documented examples: [residential] other_building_info: type: object description: Level 4 and above. NIPOST gives no fields. + x-gatepost-evidence: documented point_geometry: type: object description: Level 5. NIPOST gives no fields. + x-gatepost-evidence: documented AdministrativeAddress: type: object description: Level 2 and above. Documented, not observed. + x-gatepost-evidence: documented required: [state_name, lga_name, locality_name, zone] properties: state_name: { type: string } @@ -439,6 +443,7 @@ components: RecentHouseAddress: type: object description: Level 2 and above. Documented, not observed. + x-gatepost-evidence: documented required: [recent] properties: recent: { type: string } diff --git a/scripts/contract_files.py b/scripts/contract_files.py index 727591f..3d1d2b9 100644 --- a/scripts/contract_files.py +++ b/scripts/contract_files.py @@ -11,14 +11,13 @@ from __future__ import annotations -import json import re import sys -from http import HTTPStatus from pathlib import Path from typing import Any, NamedTuple import gateway_files +import scenario_truth ROOT = gateway_files.ROOT CONTRACT = ROOT / "contract" @@ -32,7 +31,7 @@ "reverse": ({"lat", "lng"}, {"maxDistanceM"}), "autocomplete": ({"q"}, set()), } -RESPONSE_KINDS = ("body", "text", "fixture", "hang", "drop") +RESPONSE_KINDS = scenario_truth.RESPONSE_KINDS RESPONSE_EXTRAS = ("status", "headers", "delayMs") EXPECT_FIELDS = {"attempts", "outcomes", "waitsMs", "maxInFlight", "request"} ERROR_FIELDS = ("code", "status", "apiCode", "retryAfterMs") @@ -43,33 +42,9 @@ "responses": (list, "a list"), "expect": (dict, "an object"), } -# client.md gives these defaults. -DEFAULT_LEVEL = 1 -DEFAULT_MAX_RETRIES = 2 # A scenario id in client.md: the method, a hyphen and the rule, in backticks. SCENARIO_NAME = re.compile(r"`((?:lookup|reverse|autocomplete)-[a-z0-9-]+)`") ERROR_ROW = re.compile(r"^\| [^|]+ \| `[a-z_]+` \|") -# The error map of client.md for a response that arrived. A test compares it with the table. -STATUS_CODES = {401: "unauthorized", 402: "insufficient_credits", 429: "rate_limited"} -RETRIED_STATUSES = (502, 503, 504) -CLIENT_ERRORS = range(400, 500) -RETRY_AFTER_STATUSES = (429, *RETRIED_STATUSES) -NO_RESPONSE_CODES = {"hang": "timeout", "drop": "network_error"} -METHOD_SCHEMAS = { - "lookup": "LookupEnvelope", - "reverse": "ReverseEnvelope", - "autocomplete": "AutocompleteEnvelope", -} -# The fields of a lookup body that raise the level of the data, from the highest level down. -LEVEL_FIELDS = ( - ("point_geometry", 5), - ("other_building_info", 4), - ("building_use_status", 3), - ("administrative_address", 2), - ("recent_house_address", 2), -) -CONFIDENCES = ("high", "medium", "low") -UNIT_FIELDS = ("postcode", "distanceM", "confidence") class ErrorRow(NamedTuple): @@ -80,15 +55,6 @@ class ErrorRow(NamedTuple): scenarios: tuple[str, ...] -class Reply(NamedTuple): - """What the mock server sends for one response of a scenario.""" - - kind: str - status: int | None - body: Any - headers: dict[str, str] - - def error_rows(text: str) -> list[ErrorRow]: """Return the rows of the error table in the Errors section of client.md, in order.""" section = text.partition("\n## Errors\n")[2].partition("\n## ")[0] @@ -109,19 +75,6 @@ def error_codes(path: Path = CLIENT_DOC) -> list[str]: return list(dict.fromkeys(row.code for row in rows)) -def error_code(status: int, api_code: str | None) -> str: - """Return the error code that the error map of client.md gives for a response.""" - if status in STATUS_CODES: - return STATUS_CODES[status] - if status == HTTPStatus.FORBIDDEN: - return "origin_not_allowed" if api_code == "origin_not_allowed" else "forbidden" - if status == HTTPStatus.OK: - return "unexpected_response" - if status in RETRIED_STATUSES or status not in CLIENT_ERRORS: - return "server_error" - return "invalid_input" - - def load_scenarios(root: Path = CONTRACT) -> dict[str, Any]: """Return each scenario by its file name without `.json`.""" return {path.stem: gateway_files.read_json(path) for path in sorted(root.glob("*.json"))} @@ -234,161 +187,10 @@ def scenario_problems( problems += response_problems(response, fixtures) problems += expect_problems(scenario, codes) if not problems: - problems = truth_problems(scenario, fixtures) + problems = scenario_truth.truth_problems(scenario, fixtures) return [f"{name}: {problem}" for problem in problems] -def reply_of(response: dict[str, Any], fixtures: dict[str, Any]) -> Reply: - """Return what the mock server sends for one response of a scenario.""" - kind = next(kind for kind in RESPONSE_KINDS if kind in response) - headers = response.get("headers", {}) - if kind == "fixture": - fixture = fixtures[response["fixture"]] - return Reply(kind, fixture["status"], fixture["body"], headers) - return Reply(kind, response.get("status"), response.get("body"), headers) - - -def last_reply(scenario: dict[str, Any], fixtures: dict[str, Any]) -> Reply | None: - """Return the reply to the last attempt of a scenario, or None when no request goes out. - - The mock server repeats the last response when the requests outnumber the responses. - """ - attempts = scenario["expect"]["attempts"] - if attempts == 0: - return None - responses = scenario["responses"] - return reply_of(responses[min(attempts, len(responses)) - 1], fixtures) - - -def api_code(body: Any) -> str | None: - """Return the `error.code` of a body when it is text, as client.md reads it.""" - error = body.get("error") if isinstance(body, dict) else None - code = error.get("code") if isinstance(error, dict) else None - return code if isinstance(code, str) else None - - -def retry_after_ms(reply: Reply) -> int | None: - """Return the wait of a valid Retry-After in seconds, when the status can carry one.""" - value = reply.headers.get("Retry-After", "") - if reply.status not in RETRY_AFTER_STATUSES or not re.fullmatch(r"[0-9]+", value): - return None - return int(value) * 1000 - - -def expected_error(reply: Reply | None) -> dict[str, Any]: - """Return the error that client.md gives for the reply to the last attempt.""" - if reply is None or reply.status is None: - code = "invalid_input" if reply is None else NO_RESPONSE_CODES[reply.kind] - return {"code": code, "status": None, "apiCode": None, "retryAfterMs": None} - return { - "code": error_code(reply.status, api_code(reply.body)), - "status": reply.status, - "apiCode": api_code(reply.body), - "retryAfterMs": retry_after_ms(reply), - } - - -def level_received(data: dict[str, Any]) -> int: - """Return the lookup level of the data, from the fields that it holds.""" - return next((level for field, level in LEVEL_FIELDS if field in data), DEFAULT_LEVEL) - - -def body_fields(call: dict[str, Any], data: dict[str, Any]) -> dict[str, Any]: - """Return the fields of the result that client.md takes from the body, for one call.""" - if call["method"] == "lookup": - return { - "valid": data.get("valid"), - "status": data.get("status"), - "levelRequested": call.get("level", DEFAULT_LEVEL), - "levelReceived": level_received(data), - } - if call["method"] == "autocomplete": - codes = [suggestion.get("code") for suggestion in data.get("suggestions", [])] - return {"segment": data.get("segment"), "suggestions": codes} - fields = {key: data.get(key) for key in ("found", "area", "district", "state")} - fields["radiusM"] = data.get("radius_m") - unit = data.get("unit") - fields["unit"] = None - if isinstance(unit, dict): - confidence = unit.get("confidence") - fields["unit"] = { - "postcode": unit.get("postcode"), - "distanceM": unit.get("distance_m"), - "confidence": confidence if confidence in CONFIDENCES else "low", - } - return fields - - -def result_fields(result: dict[str, Any], names: list[str]) -> dict[str, Any]: - """Return the named fields of an expected result, in the form that body_fields gives.""" - fields = {name: result.get(name) for name in names} - unit, suggestions = fields.get("unit"), fields.get("suggestions") - if isinstance(unit, dict): - fields["unit"] = {name: unit.get(name) for name in UNIT_FIELDS} - if isinstance(suggestions, list): - fields["suggestions"] = [suggestion.get("code") for suggestion in suggestions] - return fields - - -def outcome_truth(call: dict[str, Any], outcome: dict[str, Any], reply: Reply | None) -> list[str]: - """Return the problems of the outcome of a call against the reply to its last attempt.""" - if "error" in outcome: - expected = expected_error(reply) - if outcome["error"] == expected: - return [] - return [ - f"The last response gives {json.dumps(expected)}, not {json.dumps(outcome['error'])}." - ] - data = reply.body.get("data") if reply and isinstance(reply.body, dict) else None - if reply is None or reply.status != HTTPStatus.OK or not isinstance(data, dict): - return ["A result needs a last response with the status 200 and an object in data."] - expected = body_fields(call, data) - stated = result_fields(outcome["result"], list(expected)) - return [ - f"The last response gives {name} {expected[name]!r}, not {stated[name]!r}." - for name in expected - if expected[name] != stated[name] - ] - - -def fixture_method_problems(scenario: dict[str, Any], fixtures: dict[str, Any]) -> list[str]: - """Return a problem for each fixture that answers another method than the calls use.""" - methods = {call["method"] for call in scenario["calls"]} - allowed = {gateway_files.ERROR_SCHEMA, *(METHOD_SCHEMAS[method] for method in methods)} - names = [response["fixture"] for response in scenario["responses"] if "fixture" in response] - return [ - f"A {' or '.join(sorted(methods))} call cannot get {name}." - for name in names - if fixtures[name]["schema"] not in allowed - ] - - -def truth_problems(scenario: dict[str, Any], fixtures: dict[str, Any]) -> list[str]: - """Return the problems of a scenario whose outcomes disagree with its responses. - - With one call, the reply to the last attempt decides the outcome, as client.md says. Calls - that share the responses, a cache or a queue get no such check, but each of their errors - must still fit its own status and API code. - """ - calls, expect = scenario["calls"], scenario["expect"] - limit = len(calls) * (1 + scenario["client"].get("maxRetries", DEFAULT_MAX_RETRIES)) - problems = fixture_method_problems(scenario, fixtures) - if expect["attempts"] > limit: - problems.append(f"{expect['attempts']} attempts are more than the calls allow ({limit}).") - if len(calls) == 1: - return problems + outcome_truth( - calls[0], expect["outcomes"][0], last_reply(scenario, fixtures) - ) - errors = [outcome["error"] for outcome in expect["outcomes"] if "error" in outcome] - return problems + [ - f"The status {error['status']} gives {error_code(error['status'], error['apiCode'])}, " - f"not {error['code']}." - for error in errors - if error["status"] is not None - and error_code(error["status"], error["apiCode"]) != error["code"] - ] - - def coverage_problems(scenarios: dict[str, Any], client_text: str) -> list[str]: """Return the problems of the link between client.md and the scenarios. @@ -416,11 +218,15 @@ def coverage_problems(scenarios: dict[str, Any], client_text: str) -> list[str]: return problems -def final_codes(scenario: dict[str, Any]) -> set[str]: - """Return the error codes that the calls of a scenario end with.""" - return { - outcome["error"]["code"] for outcome in scenario["expect"]["outcomes"] if "error" in outcome - } +def final_codes(scenario: Any) -> set[str]: + """Return the error codes that the calls of a scenario end with. + + A scenario out of the format gives no code, and the check of its file names the fault. + """ + expect = scenario.get("expect") if isinstance(scenario, dict) else None + outcomes = expect.get("outcomes") if isinstance(expect, dict) else None + errors = [item.get("error") for item in outcomes or [] if isinstance(item, dict)] + return {error["code"] for error in errors if isinstance(error, dict) and "code" in error} def synthetic_files() -> list[Path]: diff --git a/scripts/gateway_files.py b/scripts/gateway_files.py index 2eefd55..09239ca 100644 --- a/scripts/gateway_files.py +++ b/scripts/gateway_files.py @@ -212,14 +212,46 @@ def stated_code(name: str) -> str: return re.sub(r"-[0-9]+$", "", name.rpartition("/")[2]).replace("-", "_") -def evidence_problems(name: str, evidence: str, marks: list[str]) -> list[str]: - """Return a problem when a fixture claims more than the strongest of the marks.""" - strongest = min(marks, key=EVIDENCE.index) +def evidence_problems(name: str, evidence: str, marks: list[Any]) -> list[str]: + """Return a problem when a fixture claims more than the strongest of the marks. + + A mark that is not valid counts for nothing here. The check of the OpenAPI file names it. + """ + valid = [mark for mark in marks if mark in EVIDENCE] + if not valid: + return [] + strongest = min(valid, key=EVIDENCE.index) if EVIDENCE.index(evidence) >= EVIDENCE.index(strongest): return [] return [f"Fixture {name} claims {evidence}, but the OpenAPI file says {strongest}."] +def resolve(openapi: dict[str, Any], schema: dict[str, Any]) -> dict[str, Any]: + """Return the schema that a `$ref` names, or the schema itself when it has no `$ref`.""" + if "$ref" not in schema: + return schema + named: dict[str, Any] = openapi["components"]["schemas"][schema["$ref"].rpartition("/")[2]] + return named + + +def field_marks(openapi: dict[str, Any], schema: str, body: Any) -> list[str]: + """Return the marks of the fields in the `data` of a body, where the schema marks a field. + + A lookup body above level 1 holds such fields, because no call showed them. + """ + data = body.get("data") if isinstance(body, dict) else None + envelope = openapi["components"]["schemas"][schema].get("properties", {}) + fields = resolve(openapi, envelope.get("data", {})).get("properties", {}) + if not isinstance(data, dict): + return [] + marks = [ + fields[name].get(EVIDENCE_MARK) or resolve(openapi, fields[name]).get(EVIDENCE_MARK) + for name in data + if name in fields + ] + return [mark for mark in marks if mark in EVIDENCE] + + def fixture_response_problems( name: str, fixture: dict[str, Any], openapi: dict[str, Any] ) -> list[str]: @@ -238,7 +270,11 @@ def fixture_response_problems( if not found: return [f"Fixture {name}: no response with the schema {schema} has the status {status}."] if schema != ERROR_SCHEMA: - return evidence_problems(name, fixture["evidence"], [r[EVIDENCE_MARK] for r in found]) + weakest = sorted(field_marks(openapi, schema, fixture["body"]), key=EVIDENCE.index)[-1:] + response_marks = [r.get(EVIDENCE_MARK) for r in found] + return evidence_problems(name, fixture["evidence"], response_marks) or evidence_problems( + name, fixture["evidence"], weakest + ) code = fixture["body"]["error"]["code"] named = stated_code(name) listed = { diff --git a/scripts/scenario_truth.py b/scripts/scenario_truth.py new file mode 100644 index 0000000..07eb897 --- /dev/null +++ b/scripts/scenario_truth.py @@ -0,0 +1,481 @@ +# SPDX-FileCopyrightText: 2026 The Gatepost authors +# SPDX-License-Identifier: Apache-2.0 +"""Check that a contract scenario agrees with client.md and with the responses that it lists. + +The check plays the scenario as client.md says that a client does: it counts the attempts and the +waits of each call, shares a request or a kept result where client.md allows it, and takes the +outcome of each call from the reply to its last attempt. Each check returns a list of problems. +""" + +from __future__ import annotations + +import json +import math +import re +from decimal import Decimal +from http import HTTPStatus +from typing import Any, NamedTuple + +from gateway_files import ERROR_SCHEMA + +# client.md gives these defaults. +DEFAULT_LEVEL = 1 +LEVELS = range(1, 6) +DEFAULT_MAX_RETRIES = 2 +DEFAULT_TIMEOUT_MS = 8000 +AUTOCOMPLETE_TIMEOUT_MS = 15000 +MAX_IN_FLIGHT = 4 +LONGEST_RETRY_AFTER_MS = 10_000 +MAX_DISTANCE_M = 250 +AUTOCOMPLETE_LENGTH = 11 +# The error map of client.md for a response that arrived. A test compares it with the table. +STATUS_CODES = {401: "unauthorized", 402: "insufficient_credits", 429: "rate_limited"} +RETRIED_STATUSES = (502, 503, 504) +CLIENT_ERRORS = range(400, 500) +RETRY_AFTER_STATUSES = (429, *RETRIED_STATUSES) +NO_RESPONSE_CODES = {"hang": "timeout", "drop": "network_error"} +RESPONSE_KINDS = ("body", "text", "fixture", "hang", "drop") +METHOD_SCHEMAS = { + "lookup": "LookupEnvelope", + "reverse": "ReverseEnvelope", + "autocomplete": "AutocompleteEnvelope", +} +METHOD_PATHS = { + "lookup": "/v1/lookup", + "reverse": "/v1/search/reverse", + "autocomplete": "/v1/search/autocomplete", +} +# The fields of a lookup body that raise the level of the data, from the highest level down. +LEVEL_FIELDS = ( + ("point_geometry", 5), + ("other_building_info", 4), + ("building_use_status", 3), + ("administrative_address", 2), + ("recent_house_address", 2), +) +ADDRESS_FIELDS = { + "stateName": "state_name", + "lgaName": "lga_name", + "localityName": "locality_name", + "zone": "zone", +} +UNIT_NAMES = { + "stateName": "state_name", + "lgaName": "lga_name", + "localityName": "locality_name", + "address": "address", +} +CONFIDENCES = ("high", "medium", "low") +# A full postcode after the core's normalising: no spaces and no hyphens, in upper case. +COMPACT_POSTCODE = re.compile(r"[A-Z]{2}[0-9]{2}[A-Z0-9]{3}[A-Z]{2}[0-9]{2}") +AUTOCOMPLETE_TEXT = re.compile(rf"[A-Z0-9]{{1,{AUTOCOMPLETE_LENGTH}}}") + + +class Reply(NamedTuple): + """What the mock server sends for one response of a scenario.""" + + kind: str + status: int | None + body: Any + headers: dict[str, str] + delay_ms: int + + +class Played(NamedTuple): + """The outcome of one call as client.md plays it.""" + + reply: Reply | None + attempts: int + waits: list[dict[str, int]] + + +def error_code(status: int, api_code: str | None) -> str: + """Return the error code that the error map of client.md gives for a response.""" + if status in STATUS_CODES: + return STATUS_CODES[status] + if status == HTTPStatus.FORBIDDEN: + return "origin_not_allowed" if api_code == "origin_not_allowed" else "forbidden" + if status == HTTPStatus.OK: + return "unexpected_response" + if status in RETRIED_STATUSES or status not in CLIENT_ERRORS: + return "server_error" + return "invalid_input" + + +def query_number(number: float) -> str: + """Write a number as client.md asks: the shortest decimal form, with no exponent. + + A whole number has no fraction part, and a negative zero is 0. + """ + text = format(Decimal(repr(float(number))), "f") + if "." in text: + text = text.rstrip("0").rstrip(".") + return "0" if text == "-0" else text + + +def compact(text: str) -> str: + """Return text as the core normalises it: no spaces and no hyphens, in upper case.""" + return re.sub(r"[\s-]", "", text).upper() + + +def canonical(code: str) -> str: + """Return the canonical form of a compact full postcode.""" + return "-".join((code[:2], code[2:4], code[4:7], code[7:9], code[9:])) + + +def in_range(value: Any, low: float, high: float) -> bool: + """Tell whether a value is a finite number from low to high.""" + number = type(value) in (int, float) and math.isfinite(value) + return number and low <= value <= high + + +def call_key(call: dict[str, Any]) -> tuple[Any, ...] | None: + """Return the key that shares a request and a kept result, or None for input that fails. + + client.md makes the key from the method and the canonical form of the arguments. + """ + method = call["method"] + if method == "lookup": + code, level = compact(call["code"]), call.get("level", DEFAULT_LEVEL) + valid = COMPACT_POSTCODE.fullmatch(code) and type(level) is int and level in LEVELS + return (method, canonical(code), level) if valid else None + if method == "autocomplete": + text = compact(call["q"]) + return (method, text) if AUTOCOMPLETE_TEXT.fullmatch(text) else None + distance = call.get("maxDistanceM", 0) + if not ( + in_range(call["lat"], -90, 90) + and in_range(call["lng"], -180, 180) + and in_range(distance, 0, MAX_DISTANCE_M) + ): + return None + return (method, call["lat"], call["lng"], call.get("maxDistanceM")) + + +def expected_request(call: dict[str, Any], client: dict[str, Any]) -> dict[str, Any]: + """Return the request that client.md gives for a call with valid input.""" + key = call_key(call) + if call["method"] == "lookup" and key: + query = {"code": key[1], "level": str(key[2])} + elif call["method"] == "autocomplete" and key: + query = {"q": key[1]} + else: + query = {"lat": query_number(call["lat"]), "lng": query_number(call["lng"])} + if "maxDistanceM" in call: + query["max_distance_m"] = query_number(call["maxDistanceM"]) + return { + "method": "GET", + "path": METHOD_PATHS[call["method"]], + "query": query, + "apiKey": client.get("apiKey"), + } + + +def reply_of(response: dict[str, Any], fixtures: dict[str, Any]) -> Reply: + """Return what the mock server sends for one response of a scenario.""" + kind = next(kind for kind in RESPONSE_KINDS if kind in response) + headers, delay = response.get("headers", {}), response.get("delayMs", 0) + if kind == "fixture": + fixture = fixtures[response["fixture"]] + return Reply(kind, fixture["status"], fixture["body"], headers, delay) + return Reply(kind, response.get("status"), response.get("body"), headers, delay) + + +def api_code(body: Any) -> str | None: + """Return the `error.code` of a body when it is text, as client.md reads it.""" + error = body.get("error") if isinstance(body, dict) else None + code = error.get("code") if isinstance(error, dict) else None + return code if isinstance(code, str) else None + + +def retry_after_ms(reply: Reply) -> int | None: + """Return the wait of a valid Retry-After in seconds, when the status can carry one.""" + value = reply.headers.get("Retry-After", "") + if reply.status not in RETRY_AFTER_STATUSES or not re.fullmatch(r"[0-9]+", value): + return None + return int(value) * 1000 + + +def backoff(retry: int) -> dict[str, int]: + """Return the bounds of the client's own wait before a retry, the first being 1.""" + shortest = 500 * 2 ** (retry - 1) + return {"min": shortest, "max": shortest + 250} + + +def deadline_ms(timeout_ms: int, max_retries: int) -> int: + """Return the total deadline of a call: every timeout, and the longest wait of each retry.""" + waits = sum(backoff(retry)["max"] for retry in range(1, max_retries + 1)) + return (1 + max_retries) * timeout_ms + waits + + +def next_wait(reply: Reply, method: str, retry: int) -> dict[str, int] | None: + """Return the wait before the next attempt, or None when client.md does not retry.""" + if reply.kind == "hang": + return None if method == "autocomplete" else backoff(retry) + if reply.kind == "drop": + return backoff(retry) + asked = retry_after_ms(reply) + short = asked if asked is not None and asked <= LONGEST_RETRY_AFTER_MS else None + if reply.status == HTTPStatus.TOO_MANY_REQUESTS: + return None if short is None else {"min": short, "max": short} + if reply.status in RETRIED_STATUSES: + return backoff(retry) if short is None else {"min": short, "max": short} + return None + + +def play_call( + call: dict[str, Any], client: dict[str, Any], replies: list[Reply], *, sent: int +) -> Played: + """Return the attempts, the waits and the last reply of one call that sends a request. + + sent is the number of requests before this call. The mock server repeats its last reply. + Elapsed time counts each wait at its upper bound, as the deadline of client.md does. + """ + retries = client.get("maxRetries", DEFAULT_MAX_RETRIES) + default = AUTOCOMPLETE_TIMEOUT_MS if call["method"] == "autocomplete" else DEFAULT_TIMEOUT_MS + timeout = client.get("timeoutMs", default) + deadline = deadline_ms(timeout, retries) + attempt, elapsed = 1, 0 + waits: list[dict[str, int]] = [] + while True: + reply = replies[min(sent + attempt, len(replies)) - 1] + elapsed += timeout if reply.kind == "hang" else min(reply.delay_ms, timeout) + wait = next_wait(reply, call["method"], attempt) + if wait is None or attempt > retries or elapsed + wait["max"] > deadline: + return Played(reply, attempt, waits) + waits.append(wait) + elapsed += wait["max"] + attempt += 1 + + +def broken_body(method: str, body: Any) -> bool: + """Tell whether a 200 body breaks client.md, so that the call gives unexpected_response.""" + data = body.get("data") if isinstance(body, dict) else None + if not isinstance(data, dict): + return True + if method == "lookup": + status = data.get("status") + return type(data.get("valid")) is not bool or not isinstance(status, str | None) + if method == "autocomplete": + return not isinstance(data.get("suggestions"), list) + unit = data.get("unit") + if unit is None: + return type(data.get("found")) is not bool + postcode = unit.get("postcode") if isinstance(unit, dict) else None + full = isinstance(postcode, str) and COMPACT_POSTCODE.fullmatch(compact(postcode)) + return type(data.get("found")) is not bool or not full + + +def expected_error(reply: Reply | None) -> dict[str, Any]: + """Return the error that client.md gives for the reply to the last attempt.""" + if reply is None or reply.status is None: + code = "invalid_input" if reply is None else NO_RESPONSE_CODES[reply.kind] + return {"code": code, "status": None, "apiCode": None, "retryAfterMs": None} + return { + "code": error_code(reply.status, api_code(reply.body)), + "status": reply.status, + "apiCode": api_code(reply.body), + "retryAfterMs": retry_after_ms(reply), + } + + +def gives_result(method: str, reply: Reply | None) -> bool: + """Tell whether a reply gives a result, not an error.""" + if reply is None or reply.status != HTTPStatus.OK: + return False + return not broken_body(method, reply.body) + + +def lookup_fields(call: dict[str, Any], data: dict[str, Any]) -> dict[str, Any]: + """Return the lookup result that client.md builds from a call and a body.""" + address = data.get("administrative_address") + recent = data.get("recent_house_address") + return { + "postcode": canonical(compact(call["code"])), + "valid": data["valid"], + "status": data.get("status"), + "levelRequested": call.get("level", DEFAULT_LEVEL), + "levelReceived": next( + (level for field, level in LEVEL_FIELDS if field in data), DEFAULT_LEVEL + ), + "administrativeAddress": None + if not isinstance(address, dict) + else {name: address.get(field) for name, field in ADDRESS_FIELDS.items()}, + "recentHouseAddress": recent.get("recent") if isinstance(recent, dict) else None, + "buildingUseStatus": data.get("building_use_status"), + } + + +def reverse_fields(data: dict[str, Any]) -> dict[str, Any]: + """Return the reverse result that client.md builds from a body.""" + fields = {key: data.get(key) for key in ("found", "area", "district", "state")} + fields["radiusM"] = data.get("radius_m") + unit = data.get("unit") + fields["unit"] = None + if isinstance(unit, dict): + confidence = unit.get("confidence") + fields["unit"] = { + "postcode": canonical(compact(unit["postcode"])), + "distanceM": unit.get("distance_m"), + "confidence": confidence if confidence in CONFIDENCES else "low", + **{name: unit.get(field) for name, field in UNIT_NAMES.items()}, + } + return fields + + +def autocomplete_fields(data: dict[str, Any]) -> dict[str, Any]: + """Return the parts of an autocomplete result that a body decides: segment, codes, labels.""" + return { + "segment": data.get("segment"), + "suggestions": [[item.get("code"), item.get("label")] for item in data["suggestions"]], + } + + +def stated_fields(method: str, result: dict[str, Any], names: list[str]) -> dict[str, Any]: + """Return the named fields of an expected result, in the form of the fields above.""" + fields = {name: result.get(name) for name in names} + suggestions = fields.get("suggestions") + if method == "autocomplete" and isinstance(suggestions, list): + fields["suggestions"] = [[item.get("code"), item.get("label")] for item in suggestions] + return fields + + +def result_problems(call: dict[str, Any], result: Any, body: dict[str, Any]) -> list[str]: + """Return a line for each field of an expected result that the body does not give.""" + data, method = body["data"], call["method"] + if method == "lookup": + fields = lookup_fields(call, data) + elif method == "reverse": + fields = reverse_fields(data) + else: + fields = autocomplete_fields(data) + stated = stated_fields(method, result if isinstance(result, dict) else {}, list(fields)) + return [ + f"The last response gives {name} {fields[name]!r}, not {stated[name]!r}." + for name in fields + if fields[name] != stated[name] + ] + + +def outcome_truth(call: dict[str, Any], outcome: dict[str, Any], reply: Reply | None) -> list[str]: + """Return the problems of the outcome of a call against the reply to its last attempt.""" + result = gives_result(call["method"], reply) + if "error" in outcome and result: + return ["The last response is a good 200, so the call has a result."] + if "error" in outcome: + expected = expected_error(reply) + if outcome["error"] == expected: + return [] + return [ + f"The last response gives {json.dumps(expected)}, not {json.dumps(outcome['error'])}." + ] + if reply is None or not result: + if reply is not None and reply.status == HTTPStatus.OK: + return [ + "The last response breaks client.md, so the call ends with unexpected_response." + ] + return [f"The last response gives the error {expected_error(reply)['code']}, not a result."] + return result_problems(call, outcome["result"], reply.body) + + +def play(scenario: dict[str, Any], fixtures: dict[str, Any]) -> tuple[list[int], list[Played]]: + """Play the calls of a scenario. + + Returns, for each call, the call that sent its request (itself, or an earlier call that it + shares a request or a kept result with), and the played outcome of each call. + """ + client, calls = scenario["client"], scenario["calls"] + parallel = scenario.get("order") == "parallel" + cache = client.get("cacheTtlMs", 0) > 0 + replies = [reply_of(response, fixtures) for response in scenario["responses"]] + sent, leaders = 0, [] + played: list[Played] = [] + shared: dict[tuple[Any, ...], int] = {} + for index, call in enumerate(calls): + key = call_key(call) + if key in shared: + leaders.append(shared[key]) + played.append(played[shared[key]]) + continue + outcome = Played(None, 0, []) + if key is not None: + outcome = play_call(call, client, replies, sent=sent) + sent += outcome.attempts + leaders.append(index) + played.append(outcome) + if key is not None and ( + parallel or (cache and gives_result(call["method"], outcome.reply)) + ): + shared[key] = index + return leaders, played + + +def fixture_method_problems(scenario: dict[str, Any], fixtures: dict[str, Any]) -> list[str]: + """Return a problem for each fixture that answers another method than the calls use.""" + methods = {call["method"] for call in scenario["calls"]} + allowed = {ERROR_SCHEMA, *(METHOD_SCHEMAS[method] for method in methods)} + names = [response["fixture"] for response in scenario["responses"] if "fixture" in response] + return [ + f"A {' or '.join(sorted(methods))} call cannot get {name}." + for name in names + if fixtures[name]["schema"] not in allowed + ] + + +def count_problems(scenario: dict[str, Any], leaders: list[int], played: list[Played]) -> list[str]: + """Return the problems of the attempts, the waits, the requests in flight and the request.""" + expect, calls = scenario["expect"], scenario["calls"] + own = [outcome for index, outcome in enumerate(played) if leaders[index] == index] + problems = [] + attempts = sum(outcome.attempts for outcome in own) + if expect["attempts"] != attempts: + problems.append( + f"The responses and client.md give {attempts} attempts, not {expect['attempts']}." + ) + waits = played[0].waits if len(calls) == 1 else [] + if waits and "waitsMs" not in expect: + problems.append("A call that retries needs waitsMs.") + if "waitsMs" in expect and expect["waitsMs"] != waits: + problems.append( + f"The responses and client.md give the waits {waits}, not {expect['waitsMs']}." + ) + senders = [outcome for outcome in own if outcome.attempts] + in_flight = min(MAX_IN_FLIGHT, len(senders)) if scenario.get("order") == "parallel" else 1 + if expect.get("maxInFlight", in_flight) != in_flight: + problems.append( + f"client.md gives {in_flight} requests in flight at most, not {expect['maxInFlight']}." + ) + first = next((calls[i] for i, outcome in enumerate(played) if outcome.attempts), None) + if "request" in expect and first is not None: + request = expected_request(first, scenario["client"]) + if expect["request"] != request: + problems.append( + f"The first request is {json.dumps(request)}, not {json.dumps(expect['request'])}." + ) + return problems + + +def truth_problems(scenario: dict[str, Any], fixtures: dict[str, Any]) -> list[str]: + """Return the problems of a scenario whose expect part disagrees with client.md. + + The scenario must keep the format first. + """ + problems = fixture_method_problems(scenario, fixtures) + if problems: + return problems + leaders, played = play(scenario, fixtures) + problems = count_problems(scenario, leaders, played) + outcomes, calls = scenario["expect"]["outcomes"], scenario["calls"] + for index, call in enumerate(calls): + lead = leaders[index] + if lead != index: + if outcomes[index] != outcomes[lead]: + problems.append( + f"Call {index + 1} gets the outcome of call {lead + 1}, " + "so the two must be the same." + ) + continue + found = outcome_truth(call, outcomes[index], played[index].reply) + prefix = f"Call {index + 1}: " if len(calls) > 1 else "" + problems += [prefix + problem for problem in found] + return problems diff --git a/scripts/tests/test_client_doc.py b/scripts/tests/test_client_doc.py index 2434556..4b47d2a 100644 --- a/scripts/tests/test_client_doc.py +++ b/scripts/tests/test_client_doc.py @@ -8,6 +8,7 @@ from pathlib import Path import contract_files +import scenario_truth CLIENT = contract_files.CLIENT_DOC.read_text(encoding="utf-8") TYPESCRIPT = contract_files.ROOT / "standards" / "languages" / "typescript.md" @@ -74,6 +75,7 @@ def test_a_distance_over_250_is_rejected_because_the_gateway_would_clamp_it(self lead = "Each of these raises `invalid_input`, and the client sends no request:" rule = bullets(self.requests, lead)[3] self.assertIn("The gateway cuts a larger value to 250 without a sign", rule) + self.assertIn("so the client rejects it", rule) self.assertNotIn("is clamped", self.requests) def test_each_input_rule_names_its_scenario(self) -> None: @@ -158,6 +160,8 @@ def test_a_call_has_a_total_deadline(self) -> None: self.assertIn("`(1 + maxRetries) x timeoutMs`", bullet) self.assertIn("`500 x 2^(n - 1) + 250`", bullet) self.assertIn("raises the error of the last attempt at once", bullet) + seconds = scenario_truth.deadline_ms(8000, 2) // 1000 + self.assertIn(f"With the defaults for `lookup`, that is {seconds} seconds.", bullet) self.assertIn("`lookup-deadline`", bullet) def test_a_retry_after_on_a_5xx_is_used_when_valid_and_short(self) -> None: @@ -229,3 +233,34 @@ def test_each_scenario_that_client_md_names_exists(self) -> None: names = set(re.findall(r"`((?:lookup|reverse|autocomplete)-[a-z0-9-]+)`", CLIENT)) self.assertGreaterEqual(len(names), 19) self.assertEqual(sorted(names - set(scenarios)), []) + + +class RuleScenarioTest(unittest.TestCase): + def test_each_rule_names_a_scenario_or_is_left_to_each_client(self) -> None: + headings = ( + "### Lookup result", + "### Reverse result", + "### Autocomplete result", + "## Requests", + "## Errors", + "## Retries", + "## Sharing, queue and cache", + ) + rules = [ + rule + for heading in headings + for rule in re.split(r"\n(?=- )", section(CLIENT, heading)) + if rule.startswith("- ") + ] + unnamed = [rule[2:40] for rule in rules if not contract_files.SCENARIO_NAME.search(rule)] + self.assertEqual( + unnamed, + [ + "A cancelled call ends with the platfor", + "No error message holds the API key (ER", + "A shared request stops only when every", + ], + ) + untested = section(CLIENT, "## Rules that no scenario tests") + for text in ("Cancellation (API-11)", "no error message holds", "a shared request stops"): + self.assertIn(text, untested) diff --git a/scripts/tests/test_contract_files.py b/scripts/tests/test_contract_files.py index 640be9e..796a696 100644 --- a/scripts/tests/test_contract_files.py +++ b/scripts/tests/test_contract_files.py @@ -10,6 +10,7 @@ import contract_files import gateway_files +import scenario_truth SCENARIOS = contract_files.load_scenarios() FIXTURES = gateway_files.load_fixtures() @@ -223,13 +224,9 @@ def test_rejects_a_result_that_the_fixture_does_not_give(self) -> None: def send_level_3(scenario: dict[str, Any]) -> None: scenario["responses"][0]["fixture"] = "lookup/valid-level-3" - self.assertEqual( - problems_of("lookup-level-1", send_level_3), - [ - "lookup-level-1: The last response gives status None, not 'valid'.", - "lookup-level-1: The last response gives levelReceived 3, not 1.", - ], - ) + problems = problems_of("lookup-level-1", send_level_3) + self.assertIn("lookup-level-1: The last response gives status None, not 'valid'.", problems) + self.assertIn("lookup-level-1: The last response gives levelReceived 3, not 1.", problems) def test_rejects_a_fixture_of_another_method(self) -> None: def send_reverse(scenario: dict[str, Any]) -> None: @@ -238,19 +235,21 @@ def send_reverse(scenario: dict[str, Any]) -> None: problems = problems_of("lookup-level-1", send_reverse) self.assertEqual(problems[0], "lookup-level-1: A lookup call cannot get reverse/unit.") - def test_rejects_more_attempts_than_the_retries_allow(self) -> None: + def test_rejects_more_attempts_than_the_responses_give(self) -> None: self.assertEqual( problems_of("lookup-level-1", lambda s: s["expect"].update(attempts=4)), - ["lookup-level-1: 4 attempts are more than the calls allow (3)."], + ["lookup-level-1: The responses and client.md give 1 attempts, not 4."], ) def test_rejects_a_code_that_does_not_fit_its_status_when_calls_share_responses(self) -> None: def say_forbidden(scenario: dict[str, Any]) -> None: scenario["expect"]["outcomes"][0]["error"]["code"] = "forbidden" - self.assertEqual( - problems_of("lookup-error-not-cached", say_forbidden), - ["lookup-error-not-cached: The status 400 gives invalid_input, not forbidden."], + [problem] = problems_of("lookup-error-not-cached", say_forbidden) + self.assertTrue( + problem.startswith( + 'lookup-error-not-cached: Call 1: The last response gives {"code": "invalid_input"' + ) ) def test_reads_the_retry_after_of_the_last_response(self) -> None: @@ -281,7 +280,7 @@ def test_maps_each_status_as_the_error_table_of_client_md_does(self) -> None: for row in rows: for status, api_code in samples.get(row.condition, [(200, None)]): with self.subTest(status=status, api_code=api_code): - self.assertEqual(contract_files.error_code(status, api_code), row.code) + self.assertEqual(scenario_truth.error_code(status, api_code), row.code) class CoverageTest(unittest.TestCase): diff --git a/scripts/tests/test_gateway_files.py b/scripts/tests/test_gateway_files.py index 15713c1..5556ee0 100644 --- a/scripts/tests/test_gateway_files.py +++ b/scripts/tests/test_gateway_files.py @@ -109,6 +109,15 @@ def test_gives_each_error_code_of_a_response_its_own_mark(self) -> None: }, ) + def test_rejects_error_codes_on_a_success(self) -> None: + openapi = copy.deepcopy(OPENAPI) + success = openapi["paths"]["/v1/lookup"]["get"]["responses"]["200"] + success["x-gatepost-error-codes"] = {"not_found": "assumed"} + self.assertEqual( + gateway_files.openapi_problems(openapi), + ["GET /v1/lookup 200 is a success and has no error codes."], + ) + def test_rejects_an_error_response_with_no_error_codes(self) -> None: openapi = copy.deepcopy(OPENAPI) del openapi["components"]["responses"]["BadRequest"]["x-gatepost-error-codes"] @@ -241,6 +250,23 @@ def test_rejects_evidence_that_claims_more_than_the_openapi_file(self) -> None: ["Fixture errors/invalid-api-key claims observed, but the OpenAPI file says assumed."], ) + def test_rejects_a_body_above_level_1_marked_as_observed(self) -> None: + for name in ("lookup/valid-level-2", "lookup/valid-level-3"): + with self.subTest(name): + fixture = edited_fixture(name, evidence="observed") + self.assertEqual( + gateway_files.fixture_response_problems(name, fixture, OPENAPI), + [f"Fixture {name} claims observed, but the OpenAPI file says documented."], + ) + + def test_skips_a_mark_that_is_not_valid_and_lets_the_file_check_name_it(self) -> None: + openapi = copy.deepcopy(OPENAPI) + openapi["paths"]["/v1/lookup"]["get"]["responses"]["200"]["x-gatepost-evidence"] = "guess" + fixture = edited_fixture("lookup/valid-level-1") + self.assertEqual( + gateway_files.fixture_response_problems("lookup/valid-level-1", fixture, openapi), [] + ) + def test_accepts_a_status_in_a_range_of_the_openapi_file(self) -> None: fixture = edited_fixture("errors/rate-limited", status=503, evidence="assumed") fixture["body"]["error"]["code"] = "unavailable" diff --git a/scripts/tests/test_scenario_truth.py b/scripts/tests/test_scenario_truth.py new file mode 100644 index 0000000..933d71d --- /dev/null +++ b/scripts/tests/test_scenario_truth.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: 2026 The Gatepost authors +# SPDX-License-Identifier: Apache-2.0 +"""Tests for scenario_truth: a scenario's outcome, attempts, waits and request follow client.md.""" + +import copy +import unittest +from typing import Any + +import contract_files +import gateway_files +import scenario_truth + +SCENARIOS = contract_files.load_scenarios() +FIXTURES = gateway_files.load_fixtures() +CODES = contract_files.error_codes() + + +def problems_of(name: str, edit: Any) -> list[str]: + scenario = copy.deepcopy(SCENARIOS[name]) + edit(scenario) + return contract_files.scenario_problems(name, scenario, FIXTURES, CODES) + + +def set_expect(**fields: Any) -> Any: + return lambda scenario: scenario["expect"].update(fields) + + +class OutcomeTest(unittest.TestCase): + def test_rejects_an_error_for_a_good_200_body(self) -> None: + error = {"code": "unexpected_response", "status": 200, "apiCode": None} + edit = set_expect(outcomes=[{"error": {**error, "retryAfterMs": None}}]) + self.assertEqual( + problems_of("lookup-unknown-fields", edit), + ["lookup-unknown-fields: The last response is a good 200, so the call has a result."], + ) + + def test_rejects_a_result_for_a_200_body_that_breaks_client_md(self) -> None: + result = SCENARIOS["lookup-level-1"]["expect"]["outcomes"][0] + self.assertEqual( + problems_of("lookup-missing-valid", set_expect(outcomes=[result])), + [ + "lookup-missing-valid: The last response breaks client.md, " + "so the call ends with unexpected_response." + ], + ) + + def test_rejects_an_address_that_the_body_does_not_hold(self) -> None: + def rename(scenario: dict[str, Any]) -> None: + outcome = scenario["expect"]["outcomes"][0]["result"] + outcome["administrativeAddress"]["lgaName"] = "OTHER LGA" + + [problem] = problems_of("lookup-level-2", rename) + self.assertTrue(problem.startswith("lookup-level-2: The last response gives administr")) + + def test_rejects_a_postcode_that_is_not_the_callers(self) -> None: + def take_echo(scenario: dict[str, Any]) -> None: + scenario["expect"]["outcomes"][0]["result"]["postcode"] = "HELLO" + + self.assertEqual( + problems_of("lookup-echo-ignored", take_echo), + [ + "lookup-echo-ignored: The last response gives postcode " + "'FC-01-Z99-ZZ-01', not 'HELLO'." + ], + ) + + def test_rejects_a_shared_call_with_another_outcome(self) -> None: + def change_second(scenario: dict[str, Any]) -> None: + scenario["expect"]["outcomes"][1]["result"]["valid"] = False + + self.assertEqual( + problems_of("lookup-cache-hit", change_second), + ["lookup-cache-hit: Call 2 gets the outcome of call 1, so the two must be the same."], + ) + + +class AttemptTest(unittest.TestCase): + def test_counts_the_attempts_of_each_scenario_in_the_review(self) -> None: + cases = { + "lookup-cache-hit": (2, 1), + "lookup-shared-request": (2, 1), + "autocomplete-timeout": (3, 1), + "lookup-deadline": (2, 1), + } + for name, (planted, real) in cases.items(): + with self.subTest(name): + self.assertEqual( + problems_of(name, set_expect(attempts=planted)), + [f"{name}: The responses and client.md give {real} attempts, not {planted}."], + ) + + def test_rejects_too_few_attempts_with_matching_waits(self) -> None: + edit = set_expect(attempts=2, waitsMs=[{"min": 500, "max": 750}]) + self.assertIn( + "lookup-retries-exhausted: The responses and client.md give 3 attempts, not 2.", + problems_of("lookup-retries-exhausted", edit), + ) + + def test_rejects_a_wait_that_client_md_does_not_give(self) -> None: + cases = { + "lookup-retry-after": [{"min": 100, "max": 100}], + "lookup-retry-502": [{"min": 0, "max": 5}], + } + for name, waits in cases.items(): + with self.subTest(name): + [problem] = problems_of(name, set_expect(waitsMs=waits)) + self.assertTrue(problem.startswith(f"{name}: The responses and client.md give")) + + def test_rejects_a_retry_with_no_waits(self) -> None: + self.assertEqual( + problems_of("lookup-retry-503", lambda scenario: scenario["expect"].pop("waitsMs")), + ["lookup-retry-503: A call that retries needs waitsMs."], + ) + + def test_rejects_more_requests_in_flight_than_the_queue_allows(self) -> None: + self.assertEqual( + problems_of("lookup-queue-limit", set_expect(maxInFlight=9)), + ["lookup-queue-limit: client.md gives 4 requests in flight at most, not 9."], + ) + + +class RequestTest(unittest.TestCase): + def test_rejects_a_query_value_that_is_not_in_the_shortest_form(self) -> None: + def write_9_0(scenario: dict[str, Any]) -> None: + scenario["expect"]["request"]["query"]["lat"] = "9.0" + + [problem] = problems_of("reverse-unit", write_9_0) + self.assertTrue(problem.startswith('reverse-unit: The first request is {"method": "GET"')) + + def test_writes_each_number_in_its_shortest_decimal_form(self) -> None: + cases = {9.0: "9", 1e-07: "0.0000001", -0.0: "0", 9.001: "9.001", 250: "250", -7.5: "-7.5"} + for number, text in cases.items(): + with self.subTest(number=number): + self.assertEqual(scenario_truth.query_number(number), text) + + def test_the_deadline_of_a_lookup_with_the_defaults_is_26_seconds(self) -> None: + self.assertEqual(scenario_truth.deadline_ms(8000, 2), 26000) From 69510559c7710b2afd171ad53f60f9e28f0b05a6 Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 02:36:15 +0100 Subject: [PATCH 12/13] fix: check the postcode of each suggestion in the truth check The truth check compared the code and the label of a suggestion, and not its postcode. A scenario with a wrong postcode passed, and every correct client then failed it. The check now builds the postcode from the typed segments before the active one, plus the code, and a test plants the fault. Signed-off-by: MobileMage --- scripts/scenario_truth.py | 44 ++++++++++++++++++++++++---- scripts/tests/test_scenario_truth.py | 8 +++++ 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/scripts/scenario_truth.py b/scripts/scenario_truth.py index 07eb897..3bb6de7 100644 --- a/scripts/scenario_truth.py +++ b/scripts/scenario_truth.py @@ -14,6 +14,7 @@ import re from decimal import Decimal from http import HTTPStatus +from itertools import pairwise from typing import Any, NamedTuple from gateway_files import ERROR_SCHEMA @@ -53,6 +54,8 @@ ("administrative_address", 2), ("recent_house_address", 2), ) +SEGMENT_LENGTHS = (("state", 2), ("lga", 2), ("district", 3), ("area", 2), ("unit", 2)) +OFFSETS = (0, 2, 4, 7, 9, 11) ADDRESS_FIELDS = { "stateName": "state_name", "lgaName": "lga_name", @@ -118,6 +121,22 @@ def compact(text: str) -> str: return re.sub(r"[\s-]", "", text).upper() +def partial_postcode(prefix: str, segment: Any, code: Any) -> str | None: + """Return the partial postcode of the typed segments before the active one, plus its code. + + It is None when the code does not fit its segment, as client.md says. + """ + start = 0 + for name, length in SEGMENT_LENGTHS: + if name == segment: + if not isinstance(code, str) or len(code) != length: + return None + text = compact(prefix)[:start] + code + return "-".join(text[a:b] for a, b in pairwise(OFFSETS) if text[a:b]) + start += length + return None + + def canonical(code: str) -> str: """Return the canonical form of a compact full postcode.""" return "-".join((code[:2], code[2:4], code[4:7], code[7:9], code[9:])) @@ -323,11 +342,22 @@ def reverse_fields(data: dict[str, Any]) -> dict[str, Any]: return fields -def autocomplete_fields(data: dict[str, Any]) -> dict[str, Any]: - """Return the parts of an autocomplete result that a body decides: segment, codes, labels.""" +def autocomplete_fields(call: dict[str, Any], data: dict[str, Any]) -> dict[str, Any]: + """Return the parts of an autocomplete result that a body decides. + + These are the segment, and the code, label and postcode of each suggestion. + """ + segment = data.get("segment") return { - "segment": data.get("segment"), - "suggestions": [[item.get("code"), item.get("label")] for item in data["suggestions"]], + "segment": segment, + "suggestions": [ + [ + item.get("code"), + item.get("label"), + partial_postcode(call.get("q", ""), segment, item.get("code")), + ] + for item in data["suggestions"] + ], } @@ -336,7 +366,9 @@ def stated_fields(method: str, result: dict[str, Any], names: list[str]) -> dict fields = {name: result.get(name) for name in names} suggestions = fields.get("suggestions") if method == "autocomplete" and isinstance(suggestions, list): - fields["suggestions"] = [[item.get("code"), item.get("label")] for item in suggestions] + fields["suggestions"] = [ + [item.get("code"), item.get("label"), item.get("postcode")] for item in suggestions + ] return fields @@ -348,7 +380,7 @@ def result_problems(call: dict[str, Any], result: Any, body: dict[str, Any]) -> elif method == "reverse": fields = reverse_fields(data) else: - fields = autocomplete_fields(data) + fields = autocomplete_fields(call, data) stated = stated_fields(method, result if isinstance(result, dict) else {}, list(fields)) return [ f"The last response gives {name} {fields[name]!r}, not {stated[name]!r}." diff --git a/scripts/tests/test_scenario_truth.py b/scripts/tests/test_scenario_truth.py index 933d71d..be9342b 100644 --- a/scripts/tests/test_scenario_truth.py +++ b/scripts/tests/test_scenario_truth.py @@ -64,6 +64,14 @@ def take_echo(scenario: dict[str, Any]) -> None: ], ) + def test_rejects_a_suggestion_postcode_that_the_typed_text_does_not_give(self) -> None: + def move_postcode(scenario: dict[str, Any]) -> None: + suggestions = scenario["expect"]["outcomes"][0]["result"]["suggestions"] + suggestions[0]["postcode"] = "FC-01-ZZZ" + + [problem] = problems_of("autocomplete-district", move_postcode) + self.assertTrue(problem.startswith("autocomplete-district: The last response gives sugg")) + def test_rejects_a_shared_call_with_another_outcome(self) -> None: def change_second(scenario: dict[str, Any]) -> None: scenario["expect"]["outcomes"][1]["result"]["valid"] = False From a4f601d0c58240318a4cce68302574c8bb021cae Mon Sep 17 00:00:00 2001 From: MobileMage Date: Sat, 3 Oct 2026 02:36:16 +0100 Subject: [PATCH 13/13] fix: close the gaps that the final review found in the contract The deadline scenario now uses a 2 s attempt and a 7 s wait, so a slow runner cannot make it flaky while it still tests the deadline and not the 10 s limit. Five reverse and autocomplete fields get an evidence mark. Two sentences now credit NIPOST's OpenAPI file for the radius clamp, because no call of ours tested it. The contract names the null cases of radiusM, area and district, lists the negative zero and the wait between attempts as rules that no scenario tests, and gives the radius rule of the reverse fixture. The TypeScript standard says that the mock server is outside CS-2. A test uses a postcode that cannot be real. Signed-off-by: MobileMage --- client.md | 10 ++++++---- contract/lookup-deadline.json | 6 +++--- fixtures/README.md | 2 +- openapi/gateway.completed.yaml | 24 +++++++++++++++++++----- scripts/tests/test_client_doc.py | 2 +- scripts/tests/test_gateway_files.py | 4 ++-- standards/languages/typescript.md | 4 ++-- 7 files changed, 34 insertions(+), 18 deletions(-) diff --git a/client.md b/client.md index 7cc0ab9..4d47ed6 100644 --- a/client.md +++ b/client.md @@ -57,10 +57,10 @@ A `timeoutMs` of 0 or less is a programmer error. So is a negative `maxRetries` | Field | Value | |---|---| | `found` | true when the gateway found a unit or an area within the radius | -| `radiusM` | the radius that the gateway applied, in metres | +| `radiusM` | the radius that the gateway applied, in metres, or null when the body has no `radius_m` | | `unit` | the nearest unit, or null | -| `area` | the area as a postcode, or null | -| `district` | the district as a postcode, or null | +| `area` | the text that the gateway sent for the area, or null. The client does not parse it | +| `district` | the text that the gateway sent for the district, or null. The client does not parse it | | `state` | the state code, or null | A unit has these fields: @@ -107,7 +107,7 @@ A client ignores a field that this contract does not use, in a lookup, a reverse - A `level` that is not a whole number from 1 to 5. The scenario `lookup-invalid-level` tests it. - A `lat` that is not finite or is outside -90 to 90. The scenario `reverse-invalid-lat` tests it. - A `lng` that is not finite or is outside -180 to 180. The scenario `reverse-invalid-lng` tests it. - - A `maxDistanceM` below 0 or above 250. The OpenAPI file gives the range 0 to 250. The gateway cuts a larger value to 250 without a sign, so the client rejects it. It does not return a result for another radius. The scenarios `reverse-invalid-max-distance` and `reverse-invalid-negative-distance` test it. The scenarios `reverse-max-distance-zero` and `reverse-max-distance-250` test that both limits pass. + - A `maxDistanceM` below 0 or above 250. The OpenAPI file gives the range 0 to 250. NIPOST's OpenAPI file says that the gateway cuts a larger value to 250 without a sign. Gatepost did not test this, so the client rejects it. It does not return a result for another radius. The scenarios `reverse-invalid-max-distance` and `reverse-invalid-negative-distance` test it. The scenarios `reverse-max-distance-zero` and `reverse-max-distance-250` test that both limits pass. - The client writes each number in the query in its shortest decimal form, with no exponent and no trailing zero: `lat=9`, not `lat=9.0` and not `lat=9e0`. A negative zero is written `0`. A small value is written in full: `0.0000001`, not `1e-7`. Languages print numbers in different forms, so one fixed form gives every client the same query for the same input. The mock server compares numbers by value, so it also accepts `9.0`. The scenarios `reverse-plain-decimal` and `reverse-whole-number` test this rule. - `autocomplete` normalises its text with the core. Empty text, text of more than 11 characters, and text with a character other than A to Z and 0 to 9 raise `invalid_input`, and the client sends no request. The gateway does not answer an empty `q`. The request sends the normalised text: `q=FC01Z`. The scenarios `autocomplete-empty-input`, `autocomplete-too-long`, `autocomplete-bad-character` and `autocomplete-district` test this rule. @@ -176,3 +176,5 @@ Each client tests these rules itself. - A `Retry-After` that holds an HTTP date. A scenario cannot hold a date that is a few seconds ahead. - The order of the queue. A scenario counts the requests in flight, but not the order in which they leave. - `clearCache()`. The scenario format has calls of the three methods only. +- A negative zero in a query. A JSON reader may turn `-0.0` into `0`, so a scenario cannot send it in the same way to each language. The rule is in the section Requests. +- Whether a call that waits between two attempts keeps its place among the 4. This contract leaves it open, and no scenario depends on it. diff --git a/contract/lookup-deadline.json b/contract/lookup-deadline.json index b3328a3..ce8eb3d 100644 --- a/contract/lookup-deadline.json +++ b/contract/lookup-deadline.json @@ -4,7 +4,7 @@ "description": "A wait past the total deadline ends the call at once, with the last error.", "client": { "apiKey": "nipost_test_mock_l3", - "timeoutMs": 200, + "timeoutMs": 2000, "maxRetries": 1 }, "calls": [ @@ -24,7 +24,7 @@ } }, "headers": { - "Retry-After": "2" + "Retry-After": "7" } } ], @@ -36,7 +36,7 @@ "code": "server_error", "status": 503, "apiCode": "unavailable", - "retryAfterMs": 2000 + "retryAfterMs": 7000 } } ] diff --git a/fixtures/README.md b/fixtures/README.md index bb95813..18199a8 100644 --- a/fixtures/README.md +++ b/fixtures/README.md @@ -30,7 +30,7 @@ The mock server answers with these files. Each file holds one response of NIPOST | `lookup` of the postcode in `lookup/valid-level-1.json` | `lookup/valid-level-N.json`, where N is the level | | `lookup` of another postcode that is well-formed | `lookup/not-found.json` | | `lookup` of text that is not a postcode | `lookup/invalid.json` | -| `reverse` at the coordinate of `reverse/unit.json` or `reverse/area.json` | that file | +| `reverse` at the coordinate of `reverse/unit.json` or `reverse/area.json` | that file, unless the radius is below the `distance_m` of the unit, and then `reverse/not-found.json` | | `reverse` at any other coordinate | `reverse/not-found.json` | | `nearby` | `nearby/empty.json` | | an error | the file in `errors/` with the same error code | diff --git a/openapi/gateway.completed.yaml b/openapi/gateway.completed.yaml index d1c4279..3ae748b 100644 --- a/openapi/gateway.completed.yaml +++ b/openapi/gateway.completed.yaml @@ -98,7 +98,8 @@ paths: in: query required: false description: >- - The search radius in metres. The gateway lowers a radius above 250 m to 250 m. + The search radius in metres. NIPOST's OpenAPI file says that the gateway lowers a + radius above 250 m to 250 m. A client rejects a value outside 0 to 250 and sends no request. schema: { type: number, minimum: 0, maximum: 250, default: 25 } responses: @@ -490,10 +491,22 @@ components: type: string enum: [high, medium, low] description: Calls showed low only. NIPOST's file names all three. - state_name: { type: string, description: Level 2 and above. Documented. } - lga_name: { type: string, description: Level 2 and above. Documented. } - locality_name: { type: string, description: Level 2 and above. Documented. } - address: { type: string, description: Level 2 and above. Documented. } + state_name: + type: string + description: Level 2 and above. Documented. + x-gatepost-evidence: documented + lga_name: + type: string + description: Level 2 and above. Documented. + x-gatepost-evidence: documented + locality_name: + type: string + description: Level 2 and above. Documented. + x-gatepost-evidence: documented + address: + type: string + description: Level 2 and above. Documented. + x-gatepost-evidence: documented NearbyEnvelope: type: object required: [data] @@ -527,6 +540,7 @@ components: label: type: string description: NIPOST's file names it, but no suggestion in any call had one. + x-gatepost-evidence: documented AssembleEnvelope: type: object required: [data] diff --git a/scripts/tests/test_client_doc.py b/scripts/tests/test_client_doc.py index 4b47d2a..f518a3d 100644 --- a/scripts/tests/test_client_doc.py +++ b/scripts/tests/test_client_doc.py @@ -74,7 +74,7 @@ def test_a_bad_input_value_raises_invalid_input_before_any_request(self) -> None def test_a_distance_over_250_is_rejected_because_the_gateway_would_clamp_it(self) -> None: lead = "Each of these raises `invalid_input`, and the client sends no request:" rule = bullets(self.requests, lead)[3] - self.assertIn("The gateway cuts a larger value to 250 without a sign", rule) + self.assertIn("the gateway cuts a larger value to 250 without a sign", rule) self.assertIn("so the client rejects it", rule) self.assertNotIn("is clamped", self.requests) diff --git a/scripts/tests/test_gateway_files.py b/scripts/tests/test_gateway_files.py index 5556ee0..923d28d 100644 --- a/scripts/tests/test_gateway_files.py +++ b/scripts/tests/test_gateway_files.py @@ -367,10 +367,10 @@ def test_keeps_a_partial_postcode_and_a_longer_word(self) -> None: def test_names_the_file_that_holds_a_real_postcode(self) -> None: with tempfile.TemporaryDirectory(dir=gateway_files.ROOT) as folder: path = Path(folder) / "probe.json" - path.write_text('{"postcode": "LA-12-K23-IV-95"}', encoding="utf-8") + path.write_text('{"postcode": "LA-12-Z98-IV-95"}', encoding="utf-8") [problem] = gateway_files.postcode_problems([path]) self.assertTrue( - problem.endswith("probe.json holds the postcode LA-12-K23-IV-95. Use a synthetic code.") + problem.endswith("probe.json holds the postcode LA-12-Z98-IV-95. Use a synthetic code.") ) diff --git a/standards/languages/typescript.md b/standards/languages/typescript.md index 49b86f3..65ee968 100644 --- a/standards/languages/typescript.md +++ b/standards/languages/typescript.md @@ -78,8 +78,8 @@ tsdown builds the packages, so `tsc` only checks types. Each package's `tsconfig | `jsdoc/tag-lines` with `startLines: 1` | error in packages | one blank line between a description and its tags | | `@typescript-eslint/explicit-module-boundary-types` | error in `packages/*/src` | TS-8 | | `@typescript-eslint/no-non-null-assertion` | off in `packages/*/test` | TS-4 | -| `no-restricted-imports` for every Node built-in module, by bare name and as `node:*` | error in `packages/*/src` | CS-2 | -| `no-restricted-globals` for `process`, `Buffer`, `__dirname`, `__filename`, `global` and `require` | error in `packages/*/src` | CS-2 | +| `no-restricted-imports` for every Node built-in module, by bare name and as `node:*` | error in `packages/*/src`, except `packages/mock-server` | CS-2 | +| `no-restricted-globals`, and `no-restricted-properties` on `globalThis`, for `process`, `Buffer`, `__dirname`, `__filename`, `global` and `require` | error in `packages/*/src`, except `packages/mock-server` | CS-2 | | `max-depth` | 3 | TELL-2 | | `complexity` | 10 | TELL-2 | | `max-params` | 4 | TELL-3 |