Skip to content

CtrlFindr 3.0: package rewrite, new framework format, tests and CI - #1

Open
CeMSc wants to merge 15 commits into
mainfrom
claude/jolly-volta-8uaqej
Open

CeMSc wants to merge 15 commits into
mainfrom
claude/jolly-volta-8uaqej

Conversation

@CeMSc

@CeMSc CeMSc commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Rewrites the v2.1 notebook as the ctrlfindr package and introduces a new assessment framework format. It adds a command line (template, check, ingest, analyze, run, sample), a thin notebook, tests, and CI on Windows and Linux with Python 3.11, 3.12 and 3.13. Inputs are .txt or .md files plus a framework in .ods or .xlsx; the five output TSVs are kept. 2.x frameworks are not read. The analysis unit is the sentence.

Text inputs

file treated as structure label per sentence
.txt unstructured text unstructured
.md Markdown: Markdown syntax is removed, and headings and table rows are kept whole header, body, list, table
  • The label is stored in the database and exported in results_sentences.tsv.
  • The framework setting exclude_sentence_types (e.g. header, table) leaves sentence types out of the analysis, including per-document totals and percentages.
  • Sentences are split with pySBD.
  • examples/markdown_project shows Markdown input with exclusions.

Framework format 1

sheet content
meta required format_version; title, version, author, language, description and any other keys (copied into the codebook and run_metadata.json)
settings sentiment (default off), thresholds, aggregate_children, tests_must_pass, exclude_sentence_types: the framework alone defines the coding
taxonomy one row per term (list | term | note)
queries named expressions (replaces search_strings, co_occurrences and doc_conditionals)
variables number | name | definition | parent | note
tests example sentences with expected results, checked on every run
instructions one rule per row, with examples
  • Terms:
    • * stands for any letters or digits;
    • ~ stands for a hyphen, a space or nothing (human~wildlife);
    • a standalone _ stands for one word (climate _ risk);
    • -phrase and -@list are exclusions that drop only the matches lying inside them.
  • Queries:
    • names can be written with or without quotes, and {term} uses a term inline;
    • proximity: NEAR, BEFORE, NOT_NEAR;
    • document scope: DOC, DOC_SHARE, WINDOW;
    • counts: COUNT, COUNT_DISTINCT;
    • comparisons, and chaining of queries.
  • Variables:
    • full expressions, plus V(n) to use another variable;
    • a parent column, or a hierarchy taken from the numbering;
    • optional aggregation of children, and a warning when a child is true where its parent is false.
  • Naming rules come with suggested fixes, and every problem is reported at once with its sheet and row.
  • Validation: ctrlfindr sample writes a blind, stratified sample to code by hand. The analysis then reports precision, estimated recall and F1 per variable.
  • Templates in template/ (.ods and .xlsx), with dropdowns.

Sentiment (optional)

  • Off by default: VADER runs only when a framework sets sentiment = true.
  • Optional install: vaderSentiment is an optional package (pip install vaderSentiment). Without it, everything else runs, and check explains what is missing when a framework needs it.
  • Scored during the analysis and cached: scores are computed during the analysis, not at ingest, and cached in the database. The cache is cleared for re-ingested documents or after a vaderSentiment upgrade.

Engine and robustness (earlier commits)

  • Bug fixes:
    • co-occurrence self-matches and distance semantics;
    • words glued together at line breaks;
    • the results table overwriting the source table;
    • truthy unresolved terms;
    • empty lists matching everything;
    • 1.10 read as 1.1;
    • names silently overwritten;
    • \b failing on punctuated terms.
  • Robustness:
    • no eval;
    • SQLite connections always closed;
    • explicit encoding handling;
    • incremental ingest by file hash.

Effect on results

docs/result_changes.md traces every difference on the example project to its cause. The format 1 example framework reproduces the reviewed differences from v2.1 exactly. Later changes (new formats, optional VADER) left the example's values unchanged; only the column layout changed.

Performance

corpus (analysis step) v2.1 v3
31,780 sentences 18.8 s 0.45 s
32 segments of ~4,800 words 10.7 s 0.32 s

Testing

  • Clean environments: 171 tests pass in clean venvs built from requirements.lock, on Python 3.12 and 3.13 (Linux).
  • Without vaderSentiment: a separate environment confirmed that everything except sentiment runs.
  • CI: Windows and Python 3.11 are covered by the CI matrix on this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM

Move the v2.1 notebook to legacy/v2.1 (single copy), the example corpus and
framework to examples/example_project, the blank framework to template/, and
the outputs committed with v2.1 to tests/fixtures/golden_v2.1. Remove the
LibreOffice lock file and the generated SQLite database, whose documents
table had been overwritten with results.

legacy/v2.1/ctrlfindr_v21.py holds the notebook's functions unchanged; it
reproduces the committed v2.1 outputs from the committed sentences and is
the baseline for regression tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
- ods.py: read .ods cells as displayed text (variable 1.10 stays 1.10),
  standard library only.
- framework.py: validate every sheet before processing and report all
  problems at once: empty lists, unknown names, names reused across
  sheets, blank or partial rows, bad distances, duplicate variable
  numbers, invalid aggregation queries, query loops.
- query.py: tokenizer and recursive-descent parser for AND/OR/NOT with
  Python precedence; vectorized evaluation replaces eval(); queries may
  refer to other queries.
- matching.py: one Aho-Corasick pass for all taxonomy terms with
  whole-word rules equal to (?<!\w)...(?!\w); co-occurrence distance is
  the number of words between non-overlapping matches.
- config.py: TOML settings with paths relative to the config file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Ingest: sorted, case-insensitive .txt discovery; UTF-8 with BOM, explicit
handling of other encodings; NFKC and dash/quote folding; line breaks
become spaces (v2.1 glued words); paragraph-aware pySBD splitting with
offsets; vaderSentiment on original-case sentences; incremental by file
hash. Documents, sentences and results are separate SQLite tables.

Analysis: same five TSVs (vadar_compound renamed vader_compound;
sentence_id/start/end added), configurable sentiment thresholds,
run_metadata.json, long-segment warning.

Tests pin the reviewed v2.1 -> v3 differences and the v3 outputs of the
example project.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
CI runs the tests and the example project on Ubuntu and Windows with
Python 3.11 and 3.13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
docs/result_changes.md attributes every difference between the v2.1 and
v3 outputs of the example project to a cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
- framework: accept decimal commas in variable numbers and distances
  (LibreOffice in comma-decimal locales displays 1.1 as "1,1").
- ods: expand blank row/column runs that are followed by content, so
  row numbers in validation messages stay exact after empty rows.
- matching: linear word tokenizer (the regex was quadratic on long
  punctuation runs) and one tokenization per sentence for all
  co-occurrence rules.
- ingest: hash files by streaming and read only new or changed files,
  one at a time, instead of holding the whole corpus in memory.
- analyze: remove result files of earlier runs that the current
  settings no longer produce.
- config: reject fractional values for whole-number settings.
- remove the unused Framework.uses_sentiment field and an unused import.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Replace the 2.x five-sheet framework with a self-contained format read
from .ods or .xlsx:

- meta (required format_version, free metadata), settings (sentiment
  and thresholds, aggregate_children, tests_must_pass), taxonomy in
  long layout (list | term | note), queries, variables, tests,
  instructions.
- Terms: * wildcard, ~ for hyphen/space/nothing, standalone _ for one
  word, exclusion phrases (-phrase) and excluded lists (-@list) that
  drop only contained matches.
- Queries: names without quotes, {term}, NEAR/BEFORE/NOT_NEAR,
  DOC/DOC_SHARE/PARAGRAPH/WINDOW, COUNT/COUNT_DISTINCT with
  comparisons, references to other queries.
- Variables: expressions with V(n), explicit or numbered parents,
  optional aggregation of children, hierarchy warning.
- Tests sheet checked on every run; check, template and sample
  commands; stratified validation report against hand coding;
  codebook.md; paragraph numbers stored at ingest.
- Naming rules with suggested fixes; all problems reported together.

The example framework expresses the 2.1 example rules in the new
format; on the 2.1 sentences it reproduces exactly the reviewed
differences, and the example outputs keep their values (new paragraph
column, column order, variable names keep their case).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
README sections for every sheet, the term syntax, functions, variables,
tests and validation; changelog and result report updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
@CeMSc CeMSc changed the title CtrlFindr 3.0: package rewrite with validation, faster matching, tests and CI CtrlFindr 3.0: package rewrite, new framework format, tests and CI Nov 4, 2026
Remove the PARAGRAPH() function, the paragraph column in the database,
in results_sentences.tsv and in the tests sheet. A blank line still
ends a sentence during splitting. Results are otherwise unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
- .txt files are read as unstructured text; .md files are parsed as
  Markdown and every sentence is labelled header, body, list or table
  (Markdown syntax removed; headings and table rows kept whole). The
  label is stored with each sentence and exported as 'structure'.
- New framework setting exclude_sentence_types leaves sentence types
  out of the analysis (and out of per-document totals).
- VADER is fully optional: sentiment is off by default, vaderSentiment
  is an optional extra, scores are computed during the analysis only
  when a framework uses sentiment, and cached in the database (cleared
  for re-ingested documents or a new vaderSentiment version).
- Markdown example project; tests for the parser, labels, exclusion,
  running without vaderSentiment and the sentiment cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Footnote texts ([^1]: ...) with their continuation lines and further
paragraphs get the structure label 'footnote' and can be excluded with
exclude_sentence_types. Footnote markers [^1], <sup>12</sup> and ^12^
are removed from sentences, so they no longer leave stray numbers at
sentence starts. The Markdown example now has a footnote and excludes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
- Ingest commits the settings check and removals first and then each
  file in its own transaction: a stopped or crashed ingest keeps every
  finished file, and the next run continues with the rest. A changed
  file keeps its previous version until it is processed again. Ctrl+C
  exits with code 130 and a message.
- The analysis processes batches of whole documents
  ([analysis] batch_sentences, default 50000), streams results to the
  TSV and the database, and accumulates the document tables, warnings
  and validation counts. Memory stays flat; batch size never changes
  results (tested down to one document per batch). Outputs are swapped
  in only when the run completes; sentiment scores are committed per
  batch. sample reads only the columns it needs.
- .txt cleaning adds a missing space after a full stop
  ("published.The minister"), leaving abbreviations such as U.S.Army,
  Ph.D., St.Louis and e.g. alone. In the example corpus this splits 3
  sentences in one document; reference outputs and the report updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
…INDOW

- Validation: hand codes are checked against the coded sentence text;
  a sentence that moved after its document changed is found under its
  new number, one that is gone is left out and reported (before, codes
  were silently compared with different sentences).
- Ingest drops the analysis results table when documents change or
  are removed, so sample() can no longer draw from outdated results.
- Markdown: inside a paragraph only '1.' starts an ordered list
  (CommonMark), so a wrapped line starting with '2020.' stays text;
  superscript ordinals and exponents are kept (20th, 10^6), letter
  footnote markers removed.
- WINDOW measures distance in sentence numbers, so sentences removed by
  exclude_sentence_types still count as being in between.
- Hidden and temporary files (., ~) are skipped by ingest.
- The results-table swap runs in one transaction.
- Clear error when sentiment scores are missing; removed unused
  helpers (read_table, write_table, term_regex, validation_report) and
  a duplicate of table_exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
A framework (.xlsx and .ods) on the four FAO dimensions that uses every
term marker, query function and variable feature, its own 29 tests, and
a small .txt/.md corpus with odd cases: BOM and CRLF, missing spaces,
ellipsis, URL, citation mark, glued heading, non-ASCII, en dash term,
exclusions, .TXT extension, empty and hidden files, Markdown headings,
lists, table, footnote, wrapped year and superscript ordinal.
Expected outputs and a checklist README; the test suite checks that the
project reproduces them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dt1Q7JGmCE5UNfQVKPSdVM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants