corpex

Read-only repository miner that builds a corpus of excerpts. Extracts reusable prompts, instructions, personas, skills, methodologies, rules, standards, guidelines, agent behavior, and other context artifacts into a reviewable corpus.

Run

corpex /path/to/repo --out repo_extraction --clean

Installed via pip install -e .. The pre-rename entry points (context_miner.py, cookbook_extractor.py, prompt_miner.py) were two-line shims and have been dropped.

What changed from the previous extractor

• Content detection is no longer extension-allowlist based. Extensionless and unusual text files are extracted when their bytes are actually text.
• Explicit support for .prompt, XML/SVG, RTF, DOCX, PPTX, XLSX, ODT/ODS/ODP, PDF, JSONL/NDJSON, CSV, Mermaid, TeX, shell files, and arbitrary textual source files.
• PDF extraction uses pypdf, then PyMuPDF, then pdftotext when available.
• Whole semantic artifacts are preserved as primary candidates instead of being replaced by heading fragments.
• Sections and embedded prompt values are still emitted as fragment candidates for search/reuse.
• Python static prompt composition resolves concatenation, f-strings, referenced constants, .format(), and dedent() templates when possible.
• Bare names like system_desc no longer count as prompt evidence just because they contain the word system.
• Exact-content deduplication retains all provenance occurrences instead of discarding duplicate source locations.
• --scope tests, --only-tests, and --production-only allow clean test/production passes.
• Repeatable --exclude-glob and --include-glob support provenance/source filtering without hardcoded vendor structure.
• The output directory is automatically excluded when it lives inside the source repository.

Useful commands

Production source only:

corpex /path/to/repo --out production_extraction --production-only --clean

Tests only:

corpex /path/to/repo --out test_extraction --only-tests --clean

Exclude a vendored copy of another library:

corpex /path/to/repo \
  --exclude-glob 'EXAMPLES/vendored_library/**' \
  --out repo_extraction --clean

Raise the per-file extraction budget:

corpex /path/to/repo --max-file-bytes 52428800

Output contract

repo_extraction/
├── README.md
├── summary.json
├── inventory.jsonl
├── candidates.csv
├── candidates.jsonl
├── candidates/
└── corpus/

corpus/ is the recall/completeness layer. candidates/ is the reusable-artifact discovery layer. Keep those responsibilities separate: a heuristic miss in candidate detection must not erase the extracted source text.

Dependencies

The miner has no mandatory third-party Python dependency for ordinary text, source, notebooks, RTF, Office Open XML, or OpenDocument extraction.

For PDF text extraction, it tries these backends in order:

1. pypdf
2. PyMuPDF (fitz)
3. the pdftotext executable

At least one is needed only if you want PDF extraction.