Fix end-edge attribution (off-by-one), node-id collisions, and stars - #1
Open
deerdear wants to merge 3 commits into
Open
Fix end-edge attribution (off-by-one), node-id collisions, and stars#1deerdear wants to merge 3 commits into
deerdear wants to merge 3 commits into
Conversation
Corrects three data errors in the `process.py` script:
- Duplicate pattern ID 73 ('THE FAMILY') is changed to 75.
- Duplicate pattern ID 201 ('THICKENING THE OUTER WALLS') is changed to 211.
- Incorrect subsection range for 'STRUCTURAL SUBSIDIARIES' is changed from range(26, 233) to range(226, 233).
Adds a new test file `test_process.py` with tests to verify that there are no duplicate pattern IDs and no overlapping subsection ranges, preventing similar issues in the future.
The paragraph before each pattern's heading is the PREVIOUS pattern's closing links; addEdges attributed it to the current pattern, shifting every 'end' edge one pattern forward and creating 66 self-edges. Validated against two independent transcriptions of the book: as attributed, only 141/1053 end edges matched; shifted to the previous pattern, 1008/1053 match. Whole-graph agreement (undirected Jaccard vs the apl-md transcription) goes 0.34 -> 0.81. Also: fix the 'A ROOM OF ONE'S OWN' case typo (pattern 141's heading never matched, dropping its links and stars), add a self/duplicate edge guard, and repair the committed patterns.graphml (node renames 73->75, 201->211 from the fix-data-errors branch, end-edge shift, dedupe, and 64 star corrections cross-checked against the book). The committed graphml was repaired by transforming the previous file (the pipeline needs a book .txt we don't have); links for the four patterns affected by the id collisions were restored from the apl-md transcription's link metadata.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm using this dataset for a visualization project — thank you for publishing it. While validating it against two independent transcriptions of the book, I found a systematic extraction bug and some smaller ones. This PR fixes them.
The main bug:
endedges are attributed one pattern forwardIn
addEdges,lines[i - 2]is the paragraph before pattern N's heading — which is pattern N−1's closing links — but the code attributes those references to N:Evidence (validated against the apl-md hand transcription and the iwritewordsgood.com mirror):
endedges match the book's link structure. Re-attributed to the previous pattern, 1008/1053 (96%) match.The fix tracks the previous heading's node and attributes
lines[i - 2]to it, plus a guard against self-references and duplicate edges (the book declares most links in both patterns' texts).Smaller fixes
"A ROOM OF ONE's OWN"(lowercase s) — pattern 141's heading never matched, silently dropping its links and stars.fix-data-errorsbranch (merged in, credited to that commit):(73, 'THE FAMILY')→ 75,(201, 'THICKENING THE OUTER WALLS')→ 211, and therange(26, 233)subsection typo. The id collisions also silently misrouted both patterns' edges and stars via thenodesDictoverwrite..gitignorefor__pycache__(the branch had committed bytecode).The committed
patterns.graphmlI don't have a
.txtof the book, so I couldn't re-run the pipeline. Instead the committed file was mechanically repaired with an equivalent transformation: rename the duplicate nodes (73→75, 201→211 + section/subsection), shift everyorigin=endedge's source back by one, drop self-edges/duplicates, and correct 64starsvalues cross-checked against the book mirror (231/253 headings parseable; apl-md confidence tags as fallback). Edges for the four patterns affected by the id collisions were unrecoverable from the file (both patterns' links landed on one node id) and were restored from the apl-md transcription's link metadata.Result: 253 nodes with unique ids 1–253, 1,474 directed edges, zero self-edges/duplicates, Jaccard 0.81 vs apl-md.
Known remaining limitation (out of scope)
Even after the fix, edge recall is ~60% — the source text conversion missed a large share of the book's cross-references (the book has ~1,750+ links). That's a property of the input
.txt, not the script; I can share my comparison tooling if useful.Happy to split this differently (e.g. rebase onto
fix-data-errors) if you'd prefer.