Skip to content

plan.py: check that a pattern's indices agree before planning across them - #46

Merged
litkhai merged 1 commit into
mainfrom
plan-mapping-conflicts
Sep 29, 2026
Merged

litkhai merged 1 commit into
mainfrom
plan-mapping-conflicts

Conversation

@litkhai

@litkhai litkhai commented Sep 29, 2026

Copy link
Copy Markdown
Owner

Closes #40.

plan.py plans across an index pattern, mapping_to_ddl.py reads one index and run.py
loads into one table. So a pattern whose indices disagree lands two shapes in one table,
or fails on the first chunk from the odd index out — and a rollover alias with months of
backing indices that each grew their own fields is the normal Elastic case, not the exotic
one.

One _field_caps request over the pattern answers it, and the two findings get different
treatment because they are different problems:

Finding Treatment
a field with two types across the pattern refused. One ClickHouse column cannot hold a keyword and a long, so there is no plan to make. Narrow the pattern, or pass --allow-mapping-conflicts once the column has been decided
a field only some indices have a warning. That column is empty for rows from the indices that lack it — usually a mapping that grew over time, occasionally the sign that this pattern is two datasets

Both lists land in plan.json either way, so the decision stays visible after the terminal
has scrolled.

mapping_to_ddl.py made the same distinction badly — a pattern produced not found in _mapping response (got keys: [...]). It now compares the resolved mappings: identical ones
are one shape and it says which index it read, different ones are refused with a pointer at
plan.py, which names the fields that disagree.

Building the fixture found the case worth having

With dynamic mapping on, indexing a string into an index that never declared the field
creates it as text there and keyword elsewhere. So "a field some indices lack" and "a
field with two types" are the same event in practice
— my first fixture produced both
findings for one field, and testing them apart needed dynamic: strict. That is not an
artifact of the test; it is how a real pattern acquires conflicts.

Verified on

Elasticsearch 8.17.0, four fixture indices with dynamic: strict.

Checked Result
agreeing indices plans silently, no warnings
a field only one index has WARNING, exit 0, recorded in plan.json as mapping_partial_fields
status as long on one and keyword on the other refused, exit 1, both types and their indices named, with the two ways forward
--allow-mapping-conflicts plans, exit 0, conflict kept in plan.json
mapping_to_ddl.py on identical mappings note: … resolved to 2 indices with identical mappings; reading mtest-a, DDL emitted
mapping_to_ddl.py on differing mappings refused, pointing at plan.py
single-index regression full pipeline: 4 chunks, 300,000 rows, parity passing

AGENTS.md's scope entry moves from "not covered" to "detected, and still does not merge
two shapes" — and the failure table gains the refusal, with the note that
--allow-mapping-conflicts is for after the decision, not for getting past the message.

🤖 Generated with Claude Code

…them

plan.py plans across an index pattern, mapping_to_ddl.py reads one index and
run.py loads into one table. So a pattern whose indices disagree lands two
shapes in one table, or fails on the first chunk from the odd index out --
and a rollover alias with months of backing indices that each grew their own
fields is the normal Elastic case, not the exotic one.

One _field_caps request over the pattern answers it, and the two findings get
different treatment because they are different problems:

  * a field with two types across the pattern is refused. One ClickHouse
    column cannot hold a keyword and a long, so there is no plan to make --
    narrow the pattern, or pass --allow-mapping-conflicts once the column has
    been decided.
  * a field only some indices have is a warning. That column is empty for
    rows from the indices that lack it, which is usually a mapping that grew
    over time and occasionally the sign that this pattern is two datasets.

Both lists land in plan.json either way, so the decision stays visible after
the terminal has scrolled.

mapping_to_ddl.py made the same distinction badly: a pattern produced "not
found in _mapping response (got keys: [...])". It now compares the resolved
mappings -- identical ones are one shape and it says which index it read,
different ones are refused with a pointer at plan.py, which names the fields
that disagree.

Verified on Elasticsearch 8.17.0 with four fixture indices: agreeing indices
plan silently; a field only one of them has warns, exits 0 and is recorded in
plan.json; `status` as a long on one and a keyword on the other is refused
with both types and their indices named; --allow-mapping-conflicts plans
anyway and keeps the conflict in plan.json; mapping_to_ddl.py notes the
identical-mapping case and refuses the differing one. The single-index
pipeline still runs end to end (300,000 rows, parity passing).

Building the fixture found the case worth having: with dynamic mapping on,
indexing a string into an index that never declared the field creates it as
`text` there and `keyword` elsewhere -- so "a field some indices lack" and "a
field with two types" are the same event in practice, and the fixture had to
use `dynamic: strict` to test them apart.

Closes #40

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@litkhai
litkhai merged commit f805c9d into main Sep 29, 2026
6 checks passed
@litkhai
litkhai deleted the plan-mapping-conflicts branch September 29, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

plan.py plans across an index pattern, but the DDL and the load assume one mapping

1 participant