plan.py: check that a pattern's indices agree before planning across them - #46
Merged
Merged
Conversation
…them
plan.py plans across an index pattern, mapping_to_ddl.py reads one index and
run.py loads into one table. So a pattern whose indices disagree lands two
shapes in one table, or fails on the first chunk from the odd index out --
and a rollover alias with months of backing indices that each grew their own
fields is the normal Elastic case, not the exotic one.
One _field_caps request over the pattern answers it, and the two findings get
different treatment because they are different problems:
* a field with two types across the pattern is refused. One ClickHouse
column cannot hold a keyword and a long, so there is no plan to make --
narrow the pattern, or pass --allow-mapping-conflicts once the column has
been decided.
* a field only some indices have is a warning. That column is empty for
rows from the indices that lack it, which is usually a mapping that grew
over time and occasionally the sign that this pattern is two datasets.
Both lists land in plan.json either way, so the decision stays visible after
the terminal has scrolled.
mapping_to_ddl.py made the same distinction badly: a pattern produced "not
found in _mapping response (got keys: [...])". It now compares the resolved
mappings -- identical ones are one shape and it says which index it read,
different ones are refused with a pointer at plan.py, which names the fields
that disagree.
Verified on Elasticsearch 8.17.0 with four fixture indices: agreeing indices
plan silently; a field only one of them has warns, exits 0 and is recorded in
plan.json; `status` as a long on one and a keyword on the other is refused
with both types and their indices named; --allow-mapping-conflicts plans
anyway and keeps the conflict in plan.json; mapping_to_ddl.py notes the
identical-mapping case and refuses the differing one. The single-index
pipeline still runs end to end (300,000 rows, parity passing).
Building the fixture found the case worth having: with dynamic mapping on,
indexing a string into an index that never declared the field creates it as
`text` there and `keyword` elsewhere -- so "a field some indices lack" and "a
field with two types" are the same event in practice, and the fixture had to
use `dynamic: strict` to test them apart.
Closes #40
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #40.
plan.pyplans across an index pattern,mapping_to_ddl.pyreads one index andrun.pyloads into one table. So a pattern whose indices disagree lands two shapes in one table,
or fails on the first chunk from the odd index out — and a rollover alias with months of
backing indices that each grew their own fields is the normal Elastic case, not the exotic
one.
One
_field_capsrequest over the pattern answers it, and the two findings get differenttreatment because they are different problems:
keywordand along, so there is no plan to make. Narrow the pattern, or pass--allow-mapping-conflictsonce the column has been decidedBoth lists land in
plan.jsoneither way, so the decision stays visible after the terminalhas scrolled.
mapping_to_ddl.pymade the same distinction badly — a pattern producednot found in _mapping response (got keys: [...]). It now compares the resolved mappings: identical onesare one shape and it says which index it read, different ones are refused with a pointer at
plan.py, which names the fields that disagree.Building the fixture found the case worth having
With dynamic mapping on, indexing a string into an index that never declared the field
creates it as
textthere andkeywordelsewhere. So "a field some indices lack" and "afield with two types" are the same event in practice — my first fixture produced both
findings for one field, and testing them apart needed
dynamic: strict. That is not anartifact of the test; it is how a real pattern acquires conflicts.
Verified on
Elasticsearch 8.17.0, four fixture indices with
dynamic: strict.WARNING, exit 0, recorded inplan.jsonasmapping_partial_fieldsstatusaslongon one andkeywordon the other--allow-mapping-conflictsplan.jsonmapping_to_ddl.pyon identical mappingsnote: … resolved to 2 indices with identical mappings; reading mtest-a, DDL emittedmapping_to_ddl.pyon differing mappingsplan.pyAGENTS.md's scope entry moves from "not covered" to "detected, and still does not mergetwo shapes" — and the failure table gains the refusal, with the note that
--allow-mapping-conflictsis for after the decision, not for getting past the message.🤖 Generated with Claude Code