FIX: Strip UTF-8 BOM in csv converter to prevent parse errors by dnyaneshwarmane7 · Pull Request #81 · patterninc/caterpillar

Dnyaneshwar Mane (dnyaneshwarmane7) · 2026-07-28T14:29:53Z

Description

Problem

The CSV converter task failed when processing files that start with a UTF-8 byte-order mark (BOM), which is common in S3/spreadsheet exports. The BOM bytes get attached to the first field of the header row, causing Go's csv.Reader to choke:

error in convert_from_csv: parse error on line 1, column 4: bare " in non-quoted-field

Fix

Added prepareCSVLine, a small helper that strips a leading UTF-8 BOM (0xEF 0xBB 0xBF) and trims surrounding whitespace before the data is handed to csv.Reader. This is applied when initializing columns from the header row (and when parsing records), so BOM-prefixed headers no longer break parsing.
Replaced the textutil.Slugify dependency for column-name sanitization with an inline, pre-compiled regex ([^a-zA-Z0-9]+ → _, lowercased). This removes the external dependency and keeps sanitization behavior self-contained within the converter.

Impact

CSV files exported with a BOM now parse correctly instead of erroring on the first row.
No change to the converter's output format or public interface.

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

My code follows the code style of this project.
My change requires a change to the documentation and I have updated the documentation accordingly.
I have added tests to cover my changes.

Copilot

Pull request overview

This PR targets CSV ingestion reliability by stripping a UTF-8 BOM from incoming CSV data to avoid parse issues (notably on the header row) in the pipeline task converter.

Changes:

Added a prepareCSVLine helper to strip a UTF-8 BOM (and currently whitespace) from CSV input.
Applied BOM/line preparation when initializing columns from the first CSV row.
Reworked header sanitization logic to use an in-file regex-based approach.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Yash Shrivastava (alephys26)

Lets remove the regexp work here and stick to textutil.

The BOM was only stripped in initializeColumns, but convert() builds its record-parsing reader over the raw data. With skip_first: false the first row is parsed as data, so the BOM still reached that reader and either failed with `bare " in non-quoted-field` or embedded the BOM in the first value. Add fixtures for the two paths a BOM can reach: header-derived column names and first-row-as-data. Document the CSV header normalization contract, which was previously unstated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

stripUTF8BOMAndWhitespace's TrimPrefix could never match: convert() strips the BOM before initializeColumns is reached. Only the whitespace trim was still doing work, so inline it and note why it is there — it rescues a header row whose quoted first field is preceded by stray whitespace, which csv.Reader otherwise rejects the same way it rejects a BOM. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

FIX: Strip UTF-8 BOM in csv converter to prevent parse errors

b824c10

Copilot AI review requested due to automatic review settings July 28, 2026 14:29

Dnyaneshwar Mane (dnyaneshwarmane7) requested a review from a team as a code owner July 28, 2026 14:29

Copilot started reviewing on behalf of Dnyaneshwar Mane (dnyaneshwarmane7) July 28, 2026 14:30 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Comment thread internal/pkg/pipeline/task/converter/csv.go Outdated

Comment thread internal/pkg/pipeline/task/converter/csv.go

Comment thread internal/pkg/pipeline/task/converter/csv.go

Comment thread internal/pkg/pipeline/task/converter/csv.go Outdated

Updated function name and addressed comments

09af148