Skip to content

categorisation.md

Latest commit

 

History

History
91 lines (77 loc) · 28.2 KB

categorisation.md

File metadata and controls

91 lines (77 loc) · 28.2 KB
Raw

Transaction categorisation

Durable facts that the code cannot carry in a name. Each bullet names the module it belongs to. Source: comments deleted when the no-comments rule was adopted.

The contributor-facing overview of the pipeline — the stages, their confidence values, the artifact fields, the classifier and its cache — is Transaction categorisation; the dictionary build and its sources are packages/api/data/README.md. What follows is only what those two omit.

Normalisation

  • packages/api/src/categorisation/normalise/normalise-descriptor.ts › normaliseDescriptor — the single spelling authority. The runtime (packages/api/src/lib/bank-sync.ts on write), the dictionary build and packages/api/scripts/generate-place-tokens.ts all call it. Stage 3 is an exact Map.get, so any divergence between the two sides turns every lookup into a silent miss; that is why the module imports nothing and holds no institution-specific behaviour.
  • packages/api/src/categorisation/normalise/normalise-descriptor.ts › normaliseDescriptor — the result is persisted (Transaction.normalisedDescriptor) rather than derived in a query. Postgres cannot reproduce it in an expression index: unaccent() is STABLE, not IMMUTABLE. packages/api/src/routers/budget.ts groups on the stored column in raw SQL.
  • packages/api/src/categorisation/normalise/normalise-descriptor.ts › IDENTITY_FREE_TOKENS — channel verbs, card markers, settlement-timing markers and legal forms are dropped so that a bank descriptor converges on the dictionary's normalisedName, which the build produced with the same function.
  • packages/api/src/categorisation/normalise/normalise-descriptor.ts › REMOVED_APOSTROPHES — apostrophes are deleted, not treated as separators: banks print MCDONALDS, so splitting McDonald's into mcdonald s would never match.
  • packages/api/src/categorisation/types.ts › OutgoingNegativeMinorUnits — the alias erases to number, so nothing type-checks the sign convention it documents. Two places depend on it: normalise/institutions/parse-engine.ts › partyNames picks which party is the counterparty, and deterministic.ts › readsAsRefund decides direction.
  • packages/api/src/categorisation/merchant-key.ts › IBAN_FAMILY_CODES — the ISO 20022 family codes that route to the IBAN path: RCDT received credit transfer, RDDT received direct debit, ICDT issued credit transfer, PMNT generic payment. PMNT is the generic family, which is why it carries no channel in normalise/institutions/iso20022-channel.ts.
  • packages/api/src/categorisation/merchant-key.ts › looksLikeCardDescriptor — CARD_PREFIX_LENGTH is 15 because that is the measured width of the intermediary prefixes French banks print (PAYPAL *MERCHANT NAME). An asterisk further into the string is merchant text, not a marker.
  • packages/api/src/categorisation/merchant-key.ts › withoutTrailingPlaces — French card descriptors suffix the town (CARREFOUR MARKET PARIS), so town-suffixed descriptors only converge on one key once the trailing place tokens are gone. The token list comes from packages/api/scripts/generate-place-tokens.ts, whose EUROPEAN_COUNTRIES scope is deliberately wider than SUPPORTED_COUNTRIES so cross-border descriptors normalise too.
  • packages/api/src/categorisation/place-tokens.ts › isPlaceToken — returns false for every token until bun run build:data writes data/place-tokens.json; the directory ships only a README.md. On a fresh checkout merchant-key.test.ts › "drops deferred and immediate debit markers" therefore fails on its second assertion with monoprix paris instead of monoprix. That failure is the missing artifact, not the code.
  • packages/api/src/categorisation/merchant-key.test.ts › "drops deferred and immediate debit markers" — deferred-debit cards print DEBIT DIFFERE between the card marker and the merchant name. The merchant key is an exact dictionary lookup, so one kept marker misses every merchant behind it.

Institution parsers

  • packages/api/src/categorisation/normalise/institutions/iso20022-channel.ts › CHANNEL_BY_FAMILY_CODE — the keys are ISO 20022 External Code Sets ExternalBankTransactionFamily values, delivered by Enable Banking as bank_transaction_code.code. They are standardised across SEPA banks, so the family code seeds the channel and a descriptor verb prefix only fills it in when it is still unknown.
  • packages/api/src/categorisation/normalise/institutions/parse-engine.ts › parseWithInstitution — remittance-line order is not guaranteed by providers, and a pattern match overwrites an already-found payee, so line order changes the answer. The lines are sorted before parsing to make the outcome deterministic.
  • packages/api/src/categorisation/normalise/institutions/parse-engine.ts › parseWithInstitution — the returned normalisedDescriptor is the normalised payee, not the whole remittance text. Lines the parser dropped never reach the keyword tables or the classifier, both of which read that field.
  • packages/api/src/categorisation/normalise/institutions/parse-engine.ts › applyCounterparty — for SEPA direct debits the counterparty name is the merchant, so it overrides a parsed payee. The remittance text of a direct debit carries the billing reason (Loyer, Internet fibre), not the payee.
  • packages/api/src/categorisation/normalise/types.ts › DescriptorParseResult.labelDate — a date embedded in the descriptor itself, never the bank's booking date, and ISO yyyy-mm-dd only when unambiguous. The Crédit Mutuel card rules in institutions/countries/fr.ts capture a four-digit DDMM with no year and deliberately declare no parseLabelDate, so they yield no label date at all.
  • packages/api/src/categorisation/normalise/institutions/countries/fr.ts — the French descriptor vocabulary the patterns match: PRLV/PRELEVEMENT direct debit, VIR/VIREMENT credit transfer, RETRAIT DAB ATM withdrawal, ACHAT CB/FACTURE CARTE/PAIEMENT PAR CARTE card payment, PSC contactless card payment, MOB mobile card payment, MDT SEPA mandate id, ECH instalment (échéance), ECH PRET loan instalment, AVOIR card refund, PAYWEB Crédit Mutuel virtual card, FRAIS/COTISATION/COMMISSION bank fees.
  • packages/api/src/categorisation/normalise/institutions/countries/fr.ts › CENTURY_PREFIX — French bank descriptors carry no century digit, so two-digit years are expanded with a fixed 20.
  • packages/api/src/categorisation/normalise/institutions/countries/fr.ts › BOURSORAMA_TRAILING_LOCATION_RE — Boursorama appends a backslash-delimited location after the merchant name (MONOPRIX\PARIS 15\ FR).
  • packages/api/src/categorisation/normalise/institutions/countries/index.ts — every supported country's institutions, channel verbs and trailing-noise patterns are flattened once at module load, and the flattened lists are country-blind: a descriptor is parsed against every country's verb table regardless of the transaction's country. Only InstitutionDef matching (BIC prefix, group or name substring) narrows a parse.

Dictionary

  • packages/api/src/categorisation/dictionary.ts › loadedScopeCovers — batches share one module-global map, so a narrower load must never satisfy a wider caller: two concurrent batches in different countries would otherwise inherit whichever filter loaded first. A loadedScope of null covers every request; a widening rebuild publishes a superset map, so a batch still reading the previous map stays correct.
  • packages/api/src/categorisation/dictionary.ts › ensureLoaded — the new load is assigned to state.loading before it is awaited and chains behind the in-flight one, so a concurrent caller queues instead of starting a second decompression, and widenTo re-tests the scope once the queue settles.
  • packages/api/src/categorisation/dictionary.ts › lookupDictionary — populates the module cache without incrementing openBatches, so it loads outside the reference-count protocol, and it asks for EVERY_COUNTRY (null, every country) because a lookup outside a batch has no scope and a narrower load would risk a false miss. A test that calls it leaves the real artifact resident for later files, which is why the dictionary tests call unloadDictionary() in beforeEach and afterEach.
  • packages/api/src/categorisation/dictionary.ts › DATA_DIR — the loader once resolved one directory above the package's own data/, found nothing and logged only a warning, turning every stage-3 lookup into a miss. dictionary.test.ts guards that path: the fixture must occupy the real artifact path, the build output is renamed aside for the run, and beforeAll restores an interrupted run's backup first because the artifact is gitignored and a second rename would destroy the only copy.
  • packages/api/src/categorisation/dictionary.ts › indexMerchantLine — a stored category can predate either taxonomy change, so it is decoded through resolveCategorySlug rather than rejected: the artifact outlives the code that built it, and both the pre-hierarchy and the two-level spellings still resolve. A malformed line is skipped silently and never fails the batch.
  • packages/api/src/categorisation/verify.ts › ALGORITHM_FROM_KEY — node:crypto's verify(null, …) takes the algorithm from the key itself, which is Ed25519 (packages/api/scripts/generate-signing-key.ts). Verification fails closed: with no DICTIONARY_PUBLIC_KEY, verifySignature returns false before it looks at anything.
  • packages/api/src/categorisation/merchant-scope.ts › isWorldwide — an empty countries array asserts worldwide, not "belongs nowhere", so it is the widest scope: it absorbs any narrower one on merge and survives every country filter. A source that stated no scope (Wikidata with no P17) must be omitted from the merge, never passed as [], which would erase every sibling's real scope.
  • packages/api/src/categorisation/merchant-scope.test.ts › mergeCountryScopes — the evidence behind that rule. NSI ships McDonald's as worldwide in seven categories and US-scoped in others, so a merge that narrowed would hide it from every batch but those. Adidas is worldwide in NSI and P17=DE in Wikidata: a Wikidata country of origin says where a brand comes from, not where it trades, and must never narrow a scope.
  • packages/api/src/categorisation/supported-countries.ts › SUPPORTED_COUNTRIES — a country belongs here once it has keyword heuristics and institution profiles; a per-country merchant dictionary is optional. It is not the geographic scope used for place-token generation, which is deliberately broader. Adding a country is one keyword module plus one entry in each of keywords/index.ts › registry and normalise/institutions/countries/index.ts, whose satisfies Record<SupportedCountry, …> tables fail to compile when they drift from this list.

Scoring and recurrence

  • packages/api/src/categorisation/deterministic.ts › MCC_CONFIDENCE, RULE_CONFIDENCE — an MCC outranks a keyword rule because the card network assigned the code to the merchant; it is not parsed text. A curated country rule sits one step below an exact merchant match.
  • packages/api/src/categorisation/deterministic.ts › deterministicCategory — the keyword table reads the provider's bank transaction code alone. A regular expression over a descriptor or a counterparty name guesses a merchant from free text, which is the job of the dictionary (whole-merchant match) and of the classifier (a model that reads the whole payment); the tables kept firing netflix inside an unrelated label and could not be ranked against either.
  • packages/api/src/categorisation/deterministic.test.ts › "names no category for a word that only says money moved" — Powens reports its own transaction type as the bank transaction code, so a salary credit arrives as transfer. No keyword rule names a movement any more: a transfer between the reader's own accounts is isInternalTransfer, and a transfer to anyone else is people, which a bare VIREMENT is no evidence for. The test pins the credit and the debit reading of each such word, because the old protection was the direction check and that only covered the credit.
  • packages/api/src/categorisation/resolve.ts › merchantKeyCandidates — only the tail of a merchant key is stripped, and only known service words. Matching any window of the key would read forfait mobile as the fuel brand Mobil, and t mobile is a brand where mobile alone is not.
  • packages/api/src/categorisation/types.ts › ResolutionBand — auto is displayed without hedging, suggest may show a prompt, unknown is an honest "uncategorised". An unconfident classifier yields unknown rather than a forced guess.
  • packages/api/src/categorisation/internal-transfer.ts › pairedLegIds — opposite sign, equal absolute amount, different account, same currency and dates within one day is sufficient; no IBAN is needed. Candidates deliberately include already-categorised transactions, because a transfer counted as income or expense wrecks the budget figures; only categoryOverride rows are excluded.
  • packages/api/src/categorisation/internal-transfer.ts › matchInternal — the updateMany writes isInternalTransfer and nothing else. A movement between the user's own accounts is a transaction type, not a category, so there is no internal-transfer slug to write; the boolean is what every aggregate and the pipeline's own row selection read. Nothing else clears the category a leg already carries, and the flag keeps it out of the figures either way.
  • packages/api/src/categorisation/user-override.ts, internal-transfer.ts, recurrence.ts — a categorisation-stage failure must never break a transaction sync or a read. The override write swallows its error, transfer matching reports zero matches, and recurrence detection reports empty aggregates.
  • packages/api/src/categorisation/recurrence.ts › recurringFrom — a group's reported category is the modal value of each row's category ?? resolvedCategory, so one corrected transaction cannot flip the group, and a stored slug that no longer resolves degrades to uncategorised rather than dropping the group. uncategorised is also the reader-facing Other category, so a degraded row reads as an honest catch-all rather than as a blank.

The classifier

  • packages/api/src/categorisation/classifier/types.ts › ClassificationInput — the complete allow-list of what may reach a provider: merchant spelling, counterparty, merchant category code, direction, a bucketed amount, currency, country, channel and path. No exact amount, no date, no IBAN, no raw descriptor, and no account or user identifier may ever be added to it.
  • packages/api/src/categorisation/classifier/payload.ts › classificationInputFrom — the merchant key of an IBAN-path transaction is the creditor IBAN (merchant-key.ts › merchantKeyOf), so it is dropped rather than sent: such a payment is identified to a provider by its normalised descriptor and its counterparty name alone. That also keeps the IBAN out of the signature, whose rows are shared by every user of the instance and would otherwise let anyone holding an IBAN test whether it has been seen.
  • packages/api/src/categorisation/classifier/signature.ts › classificationSignature — the amount bucket, the currency, the counterparty, the channel and the path are deliberately outside the hash: they describe one payment, while the cache answers for a merchant identity. Keying on them would ask the same merchant again for every bucket. The provider, the model string and TAXONOMY_VERSION are inside it, so renaming or re-parenting a slug invalidates every stored answer instead of decoding into a category that moved. That head is also what gives each classifier of a cascade its own row: a weak first answer stays cached while the escalation's answer is cached beside it, so a second batch calls neither.
  • packages/api/src/categorisation/classifier/signature.ts › payloadSignature — grouping a batch must not depend on which classifier is configured, so the in-memory group key hashes the merchant identity alone. It is never stored: a row keyed on it would answer for every provider at once, which is exactly what the provider-scoped cache exists to prevent.
  • packages/api/src/lib/taxonomy.ts › TAXONOMY_VERSION — bump it whenever a slug is added, removed or re-parented. Nothing else invalidates the classification cache.
  • packages/api/src/categorisation/classifier/store.ts › MerchantClassification — the row holds the hash and the answer, never the merchant key or the descriptor, which is what lets one table serve every user without leaking one account's merchants to another. A null category is an explicit abstention and is cached too, because re-asking a merchant the model already declined costs money for the same answer; ABSTENTION_RETRY_AFTER_MS in resolve.ts is what stops that from being permanent.
  • packages/api/src/categorisation/classifier/questions.ts › QUESTIONS_BY_DIRECTION — one question, not a group question and a leaf question per group. The cascade existed because 75 categories under 16 groups made a flat option list unusable; 27 categories do not, and greedy two-stage search cannot recover from a wrong group. The two sets are built at module load from categoriesForDirection, so a model is never offered a category the picker would refuse on that transaction.
  • packages/api/src/categorisation/classifier/questions.ts › CATEGORY_CRITERIA — a rubric per option rather than its label, because Other, People and Shopping carry no meaning alone. It sits beside the question rather than in lib/taxonomy, whose labels are what a reader sees. Every provider shares this one text: the system-one question sends it as the option criteria, the llm prompt prints it after each letter, and the zero-shot request sends the rubrics themselves as the candidate labels. A cross-encoder scores an entailment, and the slug people entails nothing.
  • packages/api/src/categorisation/classifier/confidence.ts › concentration — one statistic for every provider, 1 - H(p)/ln(n) over the offered options, which is what TypeSafe reports as confidence and what decider reports as certainty. A raw probability for the picked option falls as options are added, so a threshold on one is a threshold on the size of the taxonomy; a product of two such probabilities across two questions is worse again. n is always the offered count, never the number of weights the endpoint returned, or a truncated answer would look sharper than a complete one. The two vendors disagree on what the field named confidence means, which is why system-one prefers certainty when the temperature is neutral, and recomputes the statistic from probabilities only when an operator fitted a temperature. Recomputing it unconditionally would be worse, not better: an endpoint that returns the picked option alone has an entropy of zero over one weight, and would read as perfectly certain.
  • packages/api/src/categorisation/classifier/confidence.ts › ACCEPT_CONFIDENCE — 0.5 for every provider, and comparable across them precisely because they all report the same statistic. A cascade threshold that met a per-provider definition of confidence would compare two different quantities and escalate on noise.
  • packages/api/src/categorisation/classifier/providers/llm.ts › TABLE_BY_DIRECTION — the options are lettered because a slug spans several tokens, and a distribution can only be read at one position. top_logprobs is asked for at the option count, which is 20 on a debit: that is also the ceiling most OpenAI-compatible servers accept, so the debit set fits with no headroom. A wider taxonomy would need a different readout, not a bigger request.
  • packages/api/src/categorisation/classifier/confidence.ts › UNDISTRIBUTED_CONFIDENCE — an llm endpoint that returns no log probabilities still answers, at exactly ACCEPT_CONFIDENCE, so its answer can only ever suggest a category. Asking the model to state its own confidence would be cheaper and is refused: a chat model's stated number is uncorrelated with its accuracy, and at 0.85 it would write a wrong category into a budget with no reader in the loop.
  • packages/api/src/categorisation/resolve.ts › contradictsDirection — the invariant belongs to the resolver and not only to the provider, because every stage can produce a category and only the picker's own rule decides what a transaction may hold. A category the user cannot select on that row must never be written to it.
  • packages/api/src/categorisation/deterministic.ts › accepted — the merchant category code branch runs through the same refund guard as the bank code branch. A refund at a supermarket reports MCC 5411, and the unguarded branch used to file it under groceries; rejecting it inside the function, rather than at the resolver, lets the bank code table still answer.
  • packages/api/src/categorisation/classifier/providers/system-one.ts › classifyWith — every provider ends the same way: a genuine abstention resolves to null, and an unreachable endpoint, a refused request, an undecodable answer and a category outside the offered set all reject. The distinction is load-bearing rather than cosmetic, because consultClassifier caches a null as an abstention for ABSTENTION_RETRY_AFTER_MS. A provider that swallowed a 429 into null would freeze one outage into 30 days of silence, and in a cascade it would shift every merchant onto the escalation at the escalation's price.
  • packages/api/src/categorisation/resolve.ts › consultClassifier — a failure, a timeout and an answer outside the taxonomy are never written to the store: caching them would freeze a transient provider fault into a permanent abstention. A store failure is swallowed for the same reason a user-override write is: a categorisation stage must never break a sync.
  • packages/api/src/categorisation/classifier/endpoint.ts › CLASSIFIER_REQUEST_TIMEOUT_MS — 10 seconds, inside the resolver's own CLASSIFIER_TIMEOUT_MS of 15. The inner abort is what normally fires, and it arrives as a provider failure with a reason; the resolver's timeout is the outer guard for a provider that hangs without honouring its own signal. Raising the inner value above the outer one would make every slow call look like a resolver timeout and lose the reason.
  • packages/api/src/categorisation/resolve.ts › resolveGroup — the escalation bar defaults to AUTO_BAND_MIN_CONFIDENCE, so the chain asks a second model exactly when the first answer could not be written without a reader confirming it. The strongest answer wins rather than the last one: a fallback exists to rescue a weak answer, never to overrule a strong one. A cached low answer still escalates, which is what lets a chain of a cheap local model and a paid one settle a merchant permanently after one paid call.
  • packages/api/src/categorisation/classifier/registry.ts › ClassifierSlotSettings — a slot names a protocol and carries its own URL, model, key and temperature, and there is no per-vendor variable set. TypeSafe Jev, Mapika/decider-2b and Laya all answer the one /v1/systemone request, so a vendor-named provider would have been three copies of one code path with three copies of its configuration. The slot shape is also what lets one chain hold two endpoints of the same protocol, which a vendor enum cannot express: a local decider that escalates to hosted Jev is the shape this pays for.
  • packages/api/src/categorisation/classifier/registry.ts › createClassifierChain — a fallback that repeats both the protocol and the model of the first slot stops the start. The signature head is the protocol and the model, so both slots would hash to one row: the escalation would read the row it had just written and answer nothing, at twice the latency. Two endpoints of one protocol are legitimate, and each must name its own model.

Reference data

  • packages/api/src/categorisation/nsi/location-scope.ts › resolveNsiCountries — NSI's locationSet.include is a mixed bag: ISO 3166-1 alpha-2 codes, ISO 3166-2 subdivisions, UN M49 region codes (001 world, 150 Europe, 419 Latin America), NSI's own region filenames (fr-ara.geojson), inline GeoJSON geometry, free text (conus), and codes that are not ISO at all. Everything it cannot fold to a country is dropped, and an empty result means the brand is unscoped, never excluded.
  • packages/api/src/categorisation/nsi/location-scope.ts › NSI_CODE_FOR_METROPOLITAN_FRANCE — only fx is folded onto an ISO code. It carries more French brands than fr itself, and dropping it leaves the French scope at roughly a third of its real size. NSI's other non-ISO spellings (el Greece, ra Argentina, pi Philippines, kv Kosovo) are left to fail validation, and ja must fail: it means Japan in some entries and Jamaica in others, so folding it would invent a fact.
  • packages/api/src/categorisation/nsi/location-scope.ts › namesAnIsoRegion — Intl.DisplayNames.prototype.of echoes back any code it knows nothing about, so of(code) !== code is the only membership test available. It also throws RangeError on anything that is not a well-formed region subtag, which would abort the whole dictionary build, so WELL_FORMED_REGION_SUBTAG lets only two-letter tokens reach it.
  • packages/api/src/categorisation/intermediaries/detect.ts › SCHEME_PERMITTED_ACQUIRER_PREFIX_WIDTHS — Visa and Mastercard scheme rules permit an acquirer prefix of 3, 7 or 12 characters before the asterisk (Chase Paymentech and Worldpay descriptor format rules). An asterisk at one of those raw offsets corroborates a marker match and promotes medium confidence to high; hasSchemeDocumentedPrefix records the intermediaries whose own prefix convention is scheme-documented, for which a marker match alone is enough.
  • packages/api/src/categorisation/intermediaries/detect.ts › detectIntermediary — returns null rather than a weak guess, and IntermediaryConfidence has no low member: a wrong intermediary attribution is worse than none.
  • packages/api/src/categorisation/intermediaries/catalogue.ts › INTERMEDIARY_CATALOGUE — each key is the intermediaryId, and it is never persisted: the only stored field is Transaction.intermediaryName. Renaming a key is therefore safe, while renaming a name changes what stored rows mean.
  • packages/api/src/categorisation/keywords/anchor.ts › wholeTokenPattern — \b is ASCII-only, so a leading \b can never fire before överföring or impôt. The lookarounds are \p{L}/\p{N} aware instead, which the bank-code table needs both ways: a provider writes several codes in one string, and an unanchored pattern would fire on a substring of the wrong one.
  • packages/api/src/categorisation/keywords/default.ts › bankCodeKeywords, keywords/fr.ts › bankCodeKeywords — the tables shrank with the taxonomy: default.ts went from 6 rows to 3 and fr.ts from 5 to 3. Two kinds of row went. The insurance rows had no category left to name, because the flat set files a policy under the thing it insures and a bank code says which policy no more than a brand name does. The transfer rows named a movement rather than a purpose: isInternalTransfer carries a movement between the reader's own accounts, and a transfer to anyone else is people, which a bare VIREMENT is no evidence for.
  • packages/api/src/categorisation/keywords/default.ts › merchantQualifiers, keywords/fr.ts › merchantQualifiers — these are the service words banks append after a brand (free mobile, edf electricite), which is why resolve.ts › merchantKeyCandidates strips them from the tail of a missed merchant key and never from the middle.
  • packages/api/src/categorisation/keywords/index.ts › DEFAULTS_ONLY_TABLES, TABLES_BY_COUNTRY — the layers are concatenated once at module load because keywordsFor runs per transaction and must never spread on a call. The repository-wide rule behind that is in AGENTS.md: no Effect fiber per row in this package, Option, Result, Match and Predicate only.
  • packages/api/src/categorisation/keywords/index.ts — this subtree imports only lib/taxonomy, supported-countries and anchor, and nothing else may be added: deterministic.ts imports both this subtree and lib/mcc-categories, so an import from under keywords/ that reaches either would close a cycle.