Skip to content

fix: read the baseline and git history the same under any locale - #5

Merged
giacope merged 1 commit into
mainfrom
claude/laughing-pascal-3vuexa
Sep 26, 2026
Merged

giacope merged 1 commit into
mainfrom
claude/laughing-pascal-3vuexa

Conversation

@giacope

@giacope giacope commented Sep 26, 2026

Copy link
Copy Markdown
Owner

When the locale isn't UTF-8 (LANG/LC_ALL unset, as in many containers), Ruby tags file and pipe input as US-ASCII. Any non-ASCII byte then made hashira stop with an internal error (exit 70).

What broke

  • --ratchet / --update-baseline with a baseline holding non-ASCII text, e.g. an em dash in an acceptance reason. kimera's hashira_baseline.json hits this:
    hashira: internal error — Encoding::InvalidByteSequenceError: "\xE2" on US-ASCII
      at .../json-3.0.2/lib/json/common.rb:302:in 'String#encode'
    
    Baseline#read did JSON.parse(File.read(@path)), and JSON could not transcode the US-ASCII-tagged text.
  • Every run, when git's core.quotepath is off and the history names a non-ASCII file. GitLog#counts split the US-ASCII-tagged git log output and got ArgumentError: invalid byte sequence in US-ASCII.

Changes

  • CI::Baseline reads and writes the baseline as UTF-8. The write was already correct under a plain non-UTF-8 locale. With default_internal also set (e.g. ruby -U), it transcoded UTF-8 to US-ASCII and raised UndefinedConversionError. Now it can't.
  • GitLog reads git log as bytes (binmode: true), and Churn#hits compares paths byte for byte. Decoding git's output as UTF-8 would not be enough: under a US-ASCII locale Dir[] returns non-ASCII paths as BINARY, and end_with? raises on UTF-8 vs BINARY. Comparing bytes matches the names under either locale.
  • spec/readme_spec.rb read README.md the same way and failed under that locale. It now reads it as UTF-8, so all of bin/ci passes with LANG unset.
  • Other inputs checked and left alone: source files go through Prism.parse_file, which reads bytes and defaults to UTF-8. The JSON and text reports print correctly under that locale. hashira has no config file.

Tests

  • A new with_default_encodings spec helper swaps Encoding.default_external/default_internal and restores both in an ensure.
  • New specs:
    • A baseline with non-ASCII text loads under US-ASCII.
    • A baseline write under US-ASCII external + UTF-8 internal writes the UTF-8 bytes.
    • A repo with core.quotepath=false and a file named café.rb gets churn 1 under both UTF-8 and US-ASCII.
  • All three fail on main with the errors above. Each change was also reverted alone (.b, the write encoding, the read encoding), and each revert fails its own spec.
  • bin/ci passes with LANG=C.UTF-8 and with LANG/LC_ALL unset: rubocop, rubocop-kata doctor, 502 examples, the gate, and Ratchet OK: 16 edges, 0 findings, unchanged.
  • kimera ci --since origin/main: 5 mutants on changed lines, all killed.
  • End to end, on a throwaway project with lib/café/menú.rb and core.quotepath=false under an unset LANG: --ratchet, --update-baseline and a plain run all exit 0 (before: 70), and churn counts café/menú.rb.

Not in this PR

With git's default core.quotepath=true, git log quotes non-ASCII paths ("lib/caf\303\251.rb"). Churn then never matches those files and reports 0 under any locale. Passing -c core.quotePath=false in GitLog::LOG would fix it. It's a separate bug, so I left it out.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PcDfzLuRgjoSnfPLrTaA4h


Generated by Claude Code

With LANG unset, Ruby tags file and pipe input US-ASCII, so any non-ASCII
byte broke hashira with an internal error (exit 70):

- `--ratchet` / `--update-baseline`: JSON.parse could not transcode a
  baseline holding e.g. an em dash. The baseline is now read and written
  as UTF-8, so a write under `ruby -U` no longer transcodes it either.
- Every run, when git's core.quotepath is off and the history names a
  non-ASCII file: splitting the git log raised. The log is read as bytes
  and churn matches paths byte for byte, since Dir tags the same names
  BINARY under that locale and UTF-8 under a UTF-8 one.

readme_spec read README.md the same way; it now passes under that locale
too, so bin/ci is green with LANG unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PcDfzLuRgjoSnfPLrTaA4h
@giacope
giacope merged commit 29c8516 into main Sep 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants