Repository navigation
fix(stack): seed acme and apply the code style - #447
Merged
Merged
Conversation
Each comment is one line, the apt packages are sorted and the Confluent repo line no longer wraps. Two wrong comments go: BINARY has no default, and the base image digest does not fix apt package versions. The build steps are unchanged.
Kafbat reads DFE_KAFKA_BOOTSTRAP_SERVERS instead of its own copy of kafka:9092, so it shows the broker the apps use. The in-stack Apache Kafka's KRaft settings, Kafbat's dynamic-config flag and cluster name, and the engine's container paths are now literals in docker-compose.yml and leave .env.example and docs/configuration.md. Published ports, the advertised host and Redpanda sizing stay in .env. The resolved compose config is unchanged for every service.
The engine renders the receiver's routing and rules over the base on every profile with core, so only the HTTP org binding took effect, for the detection tests and make post alone. The detection e2e tests now mount tests/e2e/configs/receiver-e2e-org.yaml for it and make post runs its hunt on the engine's default org.
HYPERDX_POSTGRES_USER and HYPERDX_POSTGRES_PASSWORD become HYPERDX_DOCUMENTDB_USER and HYPERDX_DOCUMENTDB_PASSWORD, since the store is DocumentDB. Each DocumentDB comment is one line.
make keeps the whitespace before an inline comment as part of a value, so every annotated pin it exported reached docker compose as a reference Docker refuses. make stack now strips the render's comments and writes repinned images and DFE_STACK_VERSION without their notes, which it still prints.
usage_cores.py and show_limits.py take their placeholder environment and compose model from one helper pair in _common.py. make infra now hands the collector its CPU limits. Comments, docstrings and calls follow the house style.
share_checkout gave every file group read and write, which turned certs/console.key from 0600 into 0660. It now leaves the cert directory alone. The rest is house style: one-line docstrings, main returns its exit code, one-line comments and no comma before and.
…ps their own section
The transform-elastic and HyperDX version comments are one sentence each, without the doubled hyphen. docker-compose.override.yml points at IMAGE_CONSUMERS instead of repeating its service list.
The five comments on the usage rollup and host CPU pipeline say what each block does in one short sentence.
`make demo` reads `resources/demo/sources/`, one full POST /sources body per source, and the sample lines under `resources/demo/events/<source>/`, replacing `demo.yaml` and `samples/`. Two fetchers that need no credentials pull crates.io metadata and the latest GitHub release of each DFE component beside the receiver's syslog push. Each run brings every source to its file: create, update by PUT when the engine's copy differs, deploy, then set the file's state. A vendor file that drifts from the engine's catalogue entry warns. A fetcher that cannot be set up warns and is skipped. A generator that exits fails the run with its output.
Component settings and secrets move from .env to env/<component>.env, and dfe-ui's INTERNAL_API_URL becomes a compose literal. An existing .env is not migrated: rerun `make init`. `make orgs` and `make accounts` create the resources/seed orgs and accounts, minting each new account's password into env/engine.env for `make creds`. Comments and banners in env files stay within 64 characters.
The env templates and .env.example drop what nothing reads: AWS_REGION and transform-elastic's Kafka note. DFE_API_METRICS_BACKEND and NEXT_PUBLIC_THEME are commented, since both equal their component's default. .env.example keeps every key under one-line comments, and the passwords `make init` mints get placeholders it fills in place. dfe-loader's compose block takes one-line comments and sorted keys.
Env templates carry only keys, every key described in docs/configuration.md. `make e2e-posture` writes DFE_E2E_SERVER and the admin password to env/engine.env, where the engine reads them. The lines main's latest commits added follow the branch style, and demo.py finds catalogue entries at any depth.
…ose/ Remove docker-compose.live.yml and LIVE=1, whose mounts name dfe-engine directories that no longer exist. The committed override goes: make dev chains the overlay its build writes to .tmp/, and make writes one unpublish fragment from the exposure dials in place of six files. live-ui, storage, tls and container-logs move to compose/ with one-line comments.
Add `DFE_STACK_VERSION` and `DFE_ENGINE_ENV` to `.env.example` and a `DFE_STACK_VERSION` row to `docs/configuration.md`. The `make hunts` help text and its Makefile section comment now say it creates orgs as well, matching `scripts/seed.py`.
`generate_compose_override` mounted a test's config on the service only. Each `config-check-<service>` one-shot extends the service's compose entry, so on a gRPC stack it checked `config/receiver/kafka.yaml` with no brokers set and the stack failed to start. The override gives the one-shot the same mount and, for an engine-rendered config, the same `--config` path.
DFE_STACK_REPIN_IMAGES=1 on the update service updates the box on each DFE image release, not only on a stack release. Each tick compares the newest stack and a digest of the image pins against .dfe-stack-applied. A failed make ci records nothing, so the next tick retries. GitHub releases are read anonymously where gh is not installed, so the box needs no credential. When DFE_STACK_VERSION is latest or rc, make proof records a pass against the pins, so each new image set is proven once.
Over plain http the suite signed in at `http://localhost:<port>` while the console's own links used `DFE_EXTERNAL_ORIGIN`. The session cookie does not cross hosts, so on a box with a real hostname Create Rule landed on the login page. `console_url` follows `DFE_EXTERNAL_ORIGIN` on plain http as it does under TLS and as `test_source` does, and falls back to localhost where none is set.
The AWS seed sources set no poll interval. CloudTrail and Security Hub now set `interval_secs` to 30. GuardDuty sets 300, because every poll stores all its findings again. docs/configuration.md documents the intervals beside the region.
Container logs reached ClickHouse with no severity, so Search showed none. `transform/container_logs` now reads the level from the engine's JSON lines and from the word after a Rust line's timestamp, which also moves that timestamp into the record's time. It maps the level to an OTel severity number, WARNING to WARN and CRITICAL to FATAL. Console lines carry no level and stay blank.
HyperDX serves `/metrics` in scalo's format on port 8000 but sends no OTLP metrics, so its Components tab read "not reporting". The collector's prometheus receiver now scrapes it every 30 seconds as job `hyperdx`, the service name the engine reads it under.
- post.py and _search_rule.py bracket each `or` and `not` operand and pass arguments by keyword - `_routing_sources` drops its local variable annotation - The engine route comment in both proxy configs drops its full stop and fits 64 characters - `Route` has a one-line docstring and its keywords in alphabetical order - The route test's comment is shorter, with no full stop
The engine's e2e schema seeds refuse to run unless `DFE_ENGINE_E2E_SCHEMA_FIXTURES_DIR` names a directory outside the schema tree. `make e2e-posture` never set it, so the dfe-ui suite's schema search specs failed at their seed. `make e2e-posture` now writes it to env/engine.env as `/tmp/dfe-e2e-schema-fixtures`, a path inside the engine container. The engine creates the directory on its first write.
# Conflicts: # README.md # config/otel-collector/config.yaml # docker-compose.yml # docs/configuration.md # docs/operating.md # ops/daemon-update/install.sh # scripts/demo.py # scripts/quickstart.py # scripts/test_e2e.py # scripts/tests/test_demo.py # scripts/tests/test_install_sh.py # scripts/tests/test_quickstart.py
HyperDX now sends its container logs through the collector like DFE's own services. Each line's `[API]`-style prefix moves to a `process` attribute, the pino level number becomes a severity and the `msg` field becomes the body.
The demo created its organisation through post.py's hunt helper, a second way to make an org beside seed.py. It now lives in resources/demo/orgs/demo.yaml, and seed's org kind creates it for both `make demo` and the published demo. Its display name is "Demo", like the seed set's "Default". The account stays in code, because the published demo has nowhere to record a password. The hunt runner's secrets mount comment is now one line under 64 characters.
installer.py, publish_compose.py, quickstart_json.py, secrets_init.py and demo_seed.py now put their helpers first and in alphabetical order, with keyword-only parameters and keyword calls. Each `or` and `not` operand is bracketed, lambdas become named functions, comments are one line of at most 64 characters and docstrings are one line. The lines main added to demo.py, _common.py, creds.py, quickstart.py and the tests follow the same rules. Behaviour is unchanged.
Comments in the dfe-docker image's Dockerfile, its .dockerignore and the image workflow are now one line of at most 64 characters with no full stop. The comments main added to ci.yml, the Makefile and docker-compose.yml follow the same rule. Workflow keys are sorted, and both workflows parse to the same structure as before. README.md, docs/evaluating.md and docs/observability.md lose the comma before "and" or "or" and a doubled hyphen.
# Conflicts: # scripts/publish_compose.py
Comments about the shared fetcher in _common.py, demo.py, service_profiles.yaml and three test files are now one line of at most 64 characters. Calls into instances.py from check_compose.py and test_instances.py pass keywords. docs/profiles.md loses two commas before "and". Behaviour is unchanged.
# Conflicts: # .github/workflows/ci.yml
Helpers come first and take an underscore prefix. Parameters are keyword-only and calls pass keywords in alphabetical order. `or` and `not` operands are bracketed, lambdas are named functions, docstrings are one line and comments are one line of at most 64 characters. Docs drop the comma before "and" or "or" and the doubled hyphen. New docs keep each paragraph on one line. The only output change is that error messages naming a value now quote it.
The seed org is now `acme` (Acme Corp) in place of `default`. `make orgs` names it as the deployment's own through the engine's `/system/organisation` route, so the setup wizard skips its organisation step. The AWS fetchers stamp their records with it and the seeded hunt reads it. `make events`, `make post` and `make test-e2e` send receiver records with the token of the ingest credential `dfe-docker`, bound to `acme`. The first run creates it, records the token in `env/engine.env` as `DFE_INGEST_TOKEN` and restarts dfe-receiver if its HTTP listener still takes any token. Without an engine login, `make post` sends the recorded token. `test_e2e.py` drops its own `http_post` for the one in `_pipeline.py`, which takes the token.
The Redpanda image declares `VOLUME /var/lib/redpanda/data`. `kafka-redpanda-config` mounted nothing there. Every `make down` left an anonymous volume that `make clean` cannot remove. The one-shot mounts the named volume `kafka-redpanda-config-data`, which stays empty. `check_compose.py` lists the one-shot in `_IMAGE_VOLUMES`. The check only fails where the daemon holds the Redpanda image. The CI runner does not hold that image.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
make orgsseeds the orgacme(Acme Corp) in place ofdefaultand names it the deployment's own through the engine's/system/organisationroute. The setup wizard skips its organisation step. The seeded AWS fetchers stamp their records with it and the seeded hunt reads it. This needs dfe-engine#1224 on the engine.make demoreads its org fromresources/demo/orgs/demo.yaml.kafka-redpanda-configmounts the named volumekafka-redpanda-config-dataover the Redpanda image'sVOLUME, somake downno longer leaves an anonymous volume thatmake cleancannot remove.Scripts, tests and the demo generators follow the code style below. Helpers come first with an underscore prefix, parameters are keyword-only and calls pass keywords in alphabetical order.
orandnotoperands are bracketed, lambdas are named functions, docstrings are one line and comments are single lines of 64 characters or fewer. Docs drop the comma before "and" or "or" and the doubled hyphen.A
test_post.pytest that ranpost.mainwithout thedotenvfixture now holds it, so a developer's.envcannot change its result.docs/broker.mdgains a missing paragraph break.docs/checks.mdandci.ymlname the zero_ok tile test by its new name.make checkpasses locally: 2690 tests, 1 skipped, every gate exits 0. Not run against a live stack.