Repository navigation
Conversation
Move the v2.1 notebook to legacy/v2.1 (single copy), the example corpus and framework to examples/example_project, the blank framework to template/, and the outputs committed with v2.1 to tests/fixtures/golden_v2.1. Remove the LibreOffice lock file and the generated SQLite database, whose documents table had been overwritten with results. legacy/v2.1/ctrlfindr_v21.py holds the notebook's functions unchanged; it reproduces the committed v2.1 outputs from the committed sentences and is the baseline for regression tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
- ods.py: read .ods cells as displayed text (variable 1.10 stays 1.10), standard library only. - framework.py: validate every sheet before processing and report all problems at once: empty lists, unknown names, names reused across sheets, blank or partial rows, bad distances, duplicate variable numbers, invalid aggregation queries, query loops. - query.py: tokenizer and recursive-descent parser for AND/OR/NOT with Python precedence; vectorized evaluation replaces eval(); queries may refer to other queries. - matching.py: one Aho-Corasick pass for all taxonomy terms with whole-word rules equal to (?<!\w)...(?!\w); co-occurrence distance is the number of words between non-overlapping matches. - config.py: TOML settings with paths relative to the config file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Ingest: sorted, case-insensitive .txt discovery; UTF-8 with BOM, explicit handling of other encodings; NFKC and dash/quote folding; line breaks become spaces (v2.1 glued words); paragraph-aware pySBD splitting with offsets; vaderSentiment on original-case sentences; incremental by file hash. Documents, sentences and results are separate SQLite tables. Analysis: same five TSVs (vadar_compound renamed vader_compound; sentence_id/start/end added), configurable sentiment thresholds, run_metadata.json, long-segment warning. Tests pin the reviewed v2.1 -> v3 differences and the v3 outputs of the example project. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
CI runs the tests and the example project on Ubuntu and Windows with Python 3.11 and 3.13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
docs/result_changes.md attributes every difference between the v2.1 and v3 outputs of the example project to a cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
- framework: accept decimal commas in variable numbers and distances (LibreOffice in comma-decimal locales displays 1.1 as "1,1"). - ods: expand blank row/column runs that are followed by content, so row numbers in validation messages stay exact after empty rows. - matching: linear word tokenizer (the regex was quadratic on long punctuation runs) and one tokenization per sentence for all co-occurrence rules. - ingest: hash files by streaming and read only new or changed files, one at a time, instead of holding the whole corpus in memory. - analyze: remove result files of earlier runs that the current settings no longer produce. - config: reject fractional values for whole-number settings. - remove the unused Framework.uses_sentiment field and an unused import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Replace the 2.x five-sheet framework with a self-contained format read from .ods or .xlsx: - meta (required format_version, free metadata), settings (sentiment and thresholds, aggregate_children, tests_must_pass), taxonomy in long layout (list | term | note), queries, variables, tests, instructions. - Terms: * wildcard, ~ for hyphen/space/nothing, standalone _ for one word, exclusion phrases (-phrase) and excluded lists (-@list) that drop only contained matches. - Queries: names without quotes, {term}, NEAR/BEFORE/NOT_NEAR, DOC/DOC_SHARE/PARAGRAPH/WINDOW, COUNT/COUNT_DISTINCT with comparisons, references to other queries. - Variables: expressions with V(n), explicit or numbered parents, optional aggregation of children, hierarchy warning. - Tests sheet checked on every run; check, template and sample commands; stratified validation report against hand coding; codebook.md; paragraph numbers stored at ingest. - Naming rules with suggested fixes; all problems reported together. The example framework expresses the 2.1 example rules in the new format; on the 2.1 sentences it reproduces exactly the reviewed differences, and the example outputs keep their values (new paragraph column, column order, variable names keep their case). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
README sections for every sheet, the term syntax, functions, variables, tests and validation; changelog and result report updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Remove the PARAGRAPH() function, the paragraph column in the database, in results_sentences.tsv and in the tests sheet. A blank line still ends a sentence during splitting. Results are otherwise unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
- .txt files are read as unstructured text; .md files are parsed as Markdown and every sentence is labelled header, body, list or table (Markdown syntax removed; headings and table rows kept whole). The label is stored with each sentence and exported as 'structure'. - New framework setting exclude_sentence_types leaves sentence types out of the analysis (and out of per-document totals). - VADER is fully optional: sentiment is off by default, vaderSentiment is an optional extra, scores are computed during the analysis only when a framework uses sentiment, and cached in the database (cleared for re-ingested documents or a new vaderSentiment version). - Markdown example project; tests for the parser, labels, exclusion, running without vaderSentiment and the sentiment cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Footnote texts ([^1]: ...) with their continuation lines and further paragraphs get the structure label 'footnote' and can be excluded with exclude_sentence_types. Footnote markers [^1], <sup>12</sup> and ^12^ are removed from sentences, so they no longer leave stray numbers at sentence starts. The Markdown example now has a footnote and excludes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
- Ingest commits the settings check and removals first and then each
file in its own transaction: a stopped or crashed ingest keeps every
finished file, and the next run continues with the rest. A changed
file keeps its previous version until it is processed again. Ctrl+C
exits with code 130 and a message.
- The analysis processes batches of whole documents
([analysis] batch_sentences, default 50000), streams results to the
TSV and the database, and accumulates the document tables, warnings
and validation counts. Memory stays flat; batch size never changes
results (tested down to one document per batch). Outputs are swapped
in only when the run completes; sentiment scores are committed per
batch. sample reads only the columns it needs.
- .txt cleaning adds a missing space after a full stop
("published.The minister"), leaving abbreviations such as U.S.Army,
Ph.D., St.Louis and e.g. alone. In the example corpus this splits 3
sentences in one document; reference outputs and the report updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
…INDOW - Validation: hand codes are checked against the coded sentence text; a sentence that moved after its document changed is found under its new number, one that is gone is left out and reported (before, codes were silently compared with different sentences). - Ingest drops the analysis results table when documents change or are removed, so sample() can no longer draw from outdated results. - Markdown: inside a paragraph only '1.' starts an ordered list (CommonMark), so a wrapped line starting with '2020.' stays text; superscript ordinals and exponents are kept (20th, 10^6), letter footnote markers removed. - WINDOW measures distance in sentence numbers, so sentences removed by exclude_sentence_types still count as being in between. - Hidden and temporary files (., ~) are skipped by ingest. - The results-table swap runs in one transaction. - Clear error when sentiment scores are missing; removed unused helpers (read_table, write_table, term_regex, validation_report) and a duplicate of table_exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
A framework (.xlsx and .ods) on the four FAO dimensions that uses every term marker, query function and variable feature, its own 29 tests, and a small .txt/.md corpus with odd cases: BOM and CRLF, missing spaces, ellipsis, URL, citation mark, glued heading, non-ASCII, en dash term, exclusions, .TXT extension, empty and hidden files, Markdown headings, lists, table, footnote, wrapped year and superscript ordinal. Expected outputs and a checklist README; the test suite checks that the project reproduces them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrites the v2.1 notebook as the
ctrlfindrpackage and introduces a new assessment framework format. It adds a command line (template,check,ingest,analyze,run,sample), a thin notebook, tests, and CI on Windows and Linux with Python 3.11, 3.12 and 3.13. Inputs are.txtor.mdfiles plus a framework in.odsor.xlsx; the five output TSVs are kept. 2.x frameworks are not read. The analysis unit is the sentence.Text inputs
structurelabel per sentence.txtunstructured.mdheader,body,list,tableresults_sentences.tsv.exclude_sentence_types(e.g.header, table) leaves sentence types out of the analysis, including per-document totals and percentages.examples/markdown_projectshows Markdown input with exclusions.Framework format 1
metaformat_version; title, version, author, language, description and any other keys (copied into the codebook andrun_metadata.json)settingssentiment(default off), thresholds,aggregate_children,tests_must_pass,exclude_sentence_types: the framework alone defines the codingtaxonomylist | term | note)queriessearch_strings,co_occurrencesanddoc_conditionals)variablesnumber | name | definition | parent | notetestsinstructions*stands for any letters or digits;~stands for a hyphen, a space or nothing (human~wildlife);_stands for one word (climate _ risk);-phraseand-@listare exclusions that drop only the matches lying inside them.{term}uses a term inline;NEAR,BEFORE,NOT_NEAR;DOC,DOC_SHARE,WINDOW;COUNT,COUNT_DISTINCT;V(n)to use another variable;parentcolumn, or a hierarchy taken from the numbering;ctrlfindr samplewrites a blind, stratified sample to code by hand. The analysis then reports precision, estimated recall and F1 per variable.template/(.odsand.xlsx), with dropdowns.Sentiment (optional)
sentiment = true.pip install vaderSentiment). Without it, everything else runs, andcheckexplains what is missing when a framework needs it.Engine and robustness (earlier commits)
1.10read as1.1;\bfailing on punctuated terms.eval;Effect on results
docs/result_changes.md traces every difference on the example project to its cause. The format 1 example framework reproduces the reviewed differences from v2.1 exactly. Later changes (new formats, optional VADER) left the example's values unchanged; only the column layout changed.
Performance
Testing
requirements.lock, on Python 3.12 and 3.13 (Linux).🤖 Generated with Claude Code
https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM