DATA.md

DATA

The brief supplies no training data. This file says where the data came from, how it was made, how it was used to test, and what I expect to break on real tickets.

1. Sources

File	Rows	What it is	Where it came from
data/training/tickets.csv	1,000 tickets	Complaint, product, age, category, escalate flag and a reason	TODO (candidate): state exactly how this file was produced and who labelled it.
data/training/seed_clauses.csv	828 sentences	One symptom per sentence, 60 for each of the 11 symptom categories and 168 "no symptom" lines (questions, resolved problems, negated symptoms, service complaints, narrative background)	Drafted with an AI assistant from general knowledge of how people describe appliance faults, then reviewed. Generated by data/training/make_seed_clauses.py. See AI.md
data/testing/safety_cases.py	201 lines	Safety edge cases: 103 that must escalate under C1 (plain, typo, capitals, Hinglish, call-centre shorthand), 20 gas-smell and gas-leak cases, and 78 look-alikes that must not escalate, such as a normal burner flame	Drafted with an AI assistant, reviewed. Used only for testing, never for training
Cases inside tests/	about 100	Edge cases for error codes, rule precedence, the installation rule and odd input	Written for the tests

The ticket file is template-built: about 120 symptom sentences and about 26 filler sentences ("Please call me back"), combined with greetings, typos and a few Hindi, Spanish and German phrases. 131 tickets describe two symptoms; 163 must be escalated; 43 are not complaints.

2. Label policy

• A safety word (sparks, smoke, burning smell, fire, electric shock, tripped supply) means electrical_or_control_board and escalate, whatever else is said.
• A gas smell on an oven, cooktop or water heater escalates; the category comes from the rest of the text.
• Age 30 days or less with a fit, levelling, connection or first-use problem is installation_or_setup.
• Two symptoms: the category earlier in Appendix B.
• Component runs but is loud: noise_or_vibration. Component does not run or runs wrongly: motor_or_drive.
• Completely dead: no_power. Power that comes and goes, or a board fault: electrical_or_control_board.
• Water on the floor: water_leak. Water that will not drain: drainage_or_blockage.
• Thank-you notes, sales questions, spam, admin requests: other.

3. From ticket labels to sentence labels

The model classifies sentences, but the file labels whole tickets. fault/sentences.py recovers a label for each sentence: product names and numbers are masked to get a template; a template seen under four or more categories is filler; otherwise it takes the category it appears with. The reason column gives both categories of a two-symptom ticket. 88 sentences could not be labelled and are dropped.

4. How the data was used to test

A random split is misleading here, because the same 120 sentences appear in training and test. So there are three schemes, all in fault/evaluate.py:

Scheme	Train	Test	Macro-F1 (95% CI)	Escalation recall	False escalations
A. Random stratified 5-fold	80% of tickets + seeds	20% of tickets	0.992 (0.986 to 0.997)	1.000	0
B. Leave-template-out 5-fold	Tickets with none of the held-out wording + seeds	Tickets whose symptom wording was never seen; a fifth of filler wording also held out	0.974 (0.963 to 0.984)	1.000	0
C. Seed-only	The 828 hand-written sentences only	All 1,000 tickets	0.959 (0.946 to 0.971)	1.000	0

Scheme B, other measures: accuracy 0.976, balanced accuracy 0.978, Cohen's kappa 0.973, MCC 0.973, calibration error (ECE) 0.085, confidence AUROC 0.675.

Typo robustness on scheme B (letters swapped, dropped or doubled at random):

Letters corrupted	Macro-F1	Escalation recall	False escalations
5%	0.888	0.988	0
10%	0.796	0.890	0
15%	0.686	0.785	0

fault/evaluate.py turns these into 13 hard gates and exits non-zero if any fails. The safety rules are not trained, so the whole ticket file is a fair test of them: 163 of 163 escalations caught, 0 false alarms.

5. Audit with fresh tickets

To check the scores above I wrote tickets in new wording, each with an expected answer, and ran them once.

Probe set	When	Agreed with the expected answer
70 tickets (short text, negation, questions, Hinglish, hazards worded unusually)	Before the fixes below	47 of 70
40 new tickets, never used for tuning	After the fixes	34 of 40

The first set exposed real faults: a normal burner flame escalated as a fire; hazards such as a melted plug were missed; negated and resolved symptoms were counted; questions and service complaints got a fault category. I fixed the rules, added 324 training sentences aimed at those gaps, and stopped sentence fragments from overriding a sentence that clearly has no symptom. The first set then scored 66 of 70, but that number is tuned and proves little. The second set is the fairer figure. Its six misses included two unusual hazard wordings ("a tingle from the tap", "the plug has turned brown"), which I then added to the safety rules; the other four were left alone.

These probe sets are small and were written by the same person as the rules, so they are a sanity check, not a benchmark.

6. Test on real incident reports

The strongest check so far. A file of 19,856 real consumer incident reports (four input columns, no labels; long narratives, median 516 characters) was run through the service, and two samples were labelled by hand before looking at the model's answers. The full file is not in this repository; the two labelled samples and their scoring scripts are in data/testing/real_reports/.

Full run, 19,856 reports: no crashes or invalid answers; latency p50 7.6 ms, p95 22.8 ms, slowest 59 ms.

Sample	Escalation: caught / should escalate	False escalations	Category acceptable	Category when no safety word decides it
100 reports, first run	46 / 48	0	76 of 100	31 of 55
Same 100 after fixes (tuned on, so optimistic)	48 / 48	0	84 of 100	39 of 55
50 new reports, scored once after the fixes	20 / 21	0	36 of 50	16 of 29

What this shows:

• The safety rules transfer to real text. Every report where a hazard actually happened was escalated in the first sample (34 of 34) and all but one in the second (16 of 17), with no false escalations in 150 reports. The misses were gas leaks described without the word "smell" and a tripped breaker described as "had to switch the power back on"; both wordings were then added.
• The category model does not transfer well. When no safety word fixes the answer, it is right a little over half the time on real narratives, far below the 0.97 above. The fixes made after the first sample (a product check, ignoring a weak second symptom in long text, more "no symptom" training lines) lifted the tuned sample but not the fresh one, so they did not solve the underlying problem.
• Why: the model learned from short template sentences. Real reports are long stories with background detail, and many describe things the taxonomy has no category for (shattered oven glass, mould, sharp edges).

Limits: I wrote the labels myself and allowed several acceptable categories where the taxonomy does not fit; the samples are small; and these are safety incident reports, which differ from an ordinary service queue.

7. How much to trust these numbers

• Scheme A is memorisation and says little about new tickets.
• Schemes B and C are optimistic. The keyword list and the seed sentences were written after reading the tickets, so "unseen wording" was not unseen to their author. The lexicon alone moved scheme B from 0.79 to 0.97, while fresh tickets agreed only 67% of the time before the audit fixes.
• All three share one writing style. The hidden set will not.
• There is no blind set yet. A set written by someone who has not seen tickets.csv, scored once and never read, is the missing check.
• Confidence cannot be perfectly calibrated here. Rule C6 forces confidence below 0.3 and rule C5 lowers it, although both paths are usually right.

The real-report test in section 6 is the best guide: escalation recall near 1.0, and category accuracy somewhere between 0.55 and 0.75 on long real narratives, higher on short, plainly worded complaints.

8. What I expect to break on real data

1. Wording outside the lexicon and seeds. Unknown phrasing falls to a low-confidence guess or to other.
2. Heavy typos. Above about 5% corrupted letters, escalation recall drops; only one-letter slips are repaired.
3. Safety phrasings not on the list, for example "it zapped me" or "melted plug".
4. Symptoms joined without punctuation. Splitting on commas and "and" helps, but long run-on text is handled weakly.
5. Three or more symptoms, sarcasm, and complaints about a past repair.
6. Languages other than English and common Hinglish.
7. Brand-specific error codes. There is no code map, so a lone code is always control_panel_or_display.
8. Different filler. Sentences such as "the technician said the board is faulty" are read as symptoms.
9. Label disagreement on the confusable pairs in section 2.