Repository navigation
labs/elastic-migration/AGENTS.md: state what the lab does not do - #44
Merged
Merged
Conversation
Every Elasticsearch request in the lab was unauthenticated. The only two
Authorization headers in data/ were both on the ClickHouse side, and the
throwaway cluster in _base/ runs with xpack.security.enabled=false -- so the
tools worked there and could not connect to any real 8.x cluster, where
security is on by default. Found while answering "can another agent just
follow this?", which is the honest answer to that question.
es_client.py is shared rather than copied into four scripts, because the
credential handling is the part that is either right or wrong in one place:
* basic auth (ES_USER/ES_PASSWORD) or an API key (ES_API_KEY), and both at
once is an error rather than a precedence rule -- a tool that silently
picks one gets debugged against the wrong identity
* TLS against a private CA (ES_CA_CERT), since 8.x generates its own on
first start, plus --es-insecure that warns on every call. The alternative
to that flag is not "everyone configures a CA properly", it is "someone
disables security on the source cluster"
* credentials come from the environment and are refused in a URL, which
ends up in shell history, ps and error messages. run.py passes them to
the export.py it spawns in the child's environment, never in its argv
* 401 and 403 are translated into the thing to change, and a wrong password
prints a sentence instead of a traceback
The minimum privileges, found by narrowing an API key until each tool broke:
cluster [monitor] and index [read, view_index_metadata, monitor]. The
index-level monitor is the one that gets left out -- _cat/indices needs both
cluster:monitor/state and indices:monitor/stats, and the 403 names the action
rather than the privilege, which is why the hint says it outright.
_base/ gains an `elastic-secure` profile: the same Elasticsearch with security
on, on 9201, so the authenticated path has a reproducible home instead of
being tested once by hand. HTTP TLS is off there on purpose -- authentication
was the gap, and a self-signed CA on top would mean copying a certificate out
of a container before anything runs at all. The CA path is verified against a
default-configuration container instead.
Verified on Elasticsearch 8.17.0 with security enabled: the whole pipeline
with basic auth, then again with an API key scoped to exactly those
privileges (20,000 documents, 3 chunks, 20,000 distinct _id in ClickHouse
26.6.8.7); https with ES_CA_CERT connects, without it fails with one line of
cause and one of remedy, and --es-insecure runs while warning. Both
credentials together and credentials in a URL are refused. The unauthenticated
path still runs end to end -- 300,000 documents, 4 chunks, parity passing --
so the quick path did not pay for this.
Closes #42
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guide had twelve invariants, a failure table and a definition of done, and no scope boundary -- so an agent reading it had no way to tell where the verified path ends. That is the same failure the whole lab is built to prevent, applied to the documentation instead of the data: emitting something plausible for a case that did not convert. Ten entries, each with what it means for the reader and the issue to say so in: schema design (the sort key is a default, #39), the scale it was actually verified at (300,000 documents against a ten-million-row premise), the Cloud-specific path (#41), an index pattern whose mappings disagree (#40), ingest (#21), dashboards (#5), ID translation not being a tracked stage (#33), a live cutover, the reverse direction, and everything that needs a business decision. It goes first, before "where to look", because a reader deciding whether this covers their migration should not have to finish the document to find out. Follows #35: the guide shipped there, and this is the part of it that was missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #43. Follows #35 — the agent guide shipped there, and this is the part of it
that was missing.
The guide had twelve invariants, a symptom→action table and a definition of done, and no
scope boundary. An agent reading it had no way to tell where the verified path ends — which
is the same failure the whole lab is built to prevent, applied to the documentation instead
of the data: emitting something plausible for a case that did not convert.
Ten entries, each with what it means for the reader and where to say they hit it:
ORDER BY (<time>)and nothing else — no low-cardinality prefix, no TTL, no codecs. Correct, possibly far more expensive to query, and the one decision that is expensive to change after the data lands (#39)s3(),SSD_CACHE(PATH …)and the memory ceiling were measured on a local single node (#41)0meansIt goes first, before "where to look": a reader deciding whether this covers their
migration should not have to finish the document to find out. And the closing line matters
as much as the table — hitting one of these is not a bug in your run, it is the edge of what
has been built and verified, and the issue number is where to say you hit it.
The lab README's pointer now says the guide opens with its own limits, in both languages.
🤖 Generated with Claude Code