Repository navigation
feat(container): publish a Cassandra-backed ExtendDB image - #725
Conversation
95f2762 to
dfaa726
Compare
dfaa726 to
95e24f0
Compare
LeeroyHannigan
left a comment
There was a problem hiding this comment.
Two findings below need a change before merge. The others are Minor or Optional.
This PR adds extenddb/extenddb-cassandra, with Dockerfile.cassandra, a reference Compose stack with password authentication and CassandraAuthorizer, a smoke test and three docs pages. The release, promote, mirror, sign and Docker Hub description workflows now include the image.
Dockerfile.cassandra differs from Dockerfile.mongodb only in --features cassandra, the notices file and the labels. So it keeps the digest-pinned bases, UID 10001, the setuid strip, tini and the health check. Every image allowlist in the five image workflows names extenddb-cassandra, and licenses.yml now runs on changes to the Cassandra notices file. No action reference or permission changed. The publish, sign and propagate jobs take the repository from the gate job's output. So the workflows sign and mirror the image under its own name. git log shows Dockerfile.mongodb and the generator's --backend flag arriving together in #331. So the new gate gives postgres and mongodb tags the result they had before.
The Compose stack publishes ExtendDB on 127.0.0.1 only, and Cassandra publishes no port. The three passwords are ${VAR:?} references, so Compose stops with an error when one is missing. shellcheck 0.11.0 passes on the three smoke scripts and on the three programs extracted from the compose file. actionlint 1.7.12 reports only the custom runner label.
The docs match the code on these points:
- the grants (
crates/storage-cassandra/src/migrations/mod.rs:600) - the character set of the application password and its
extenddb-local-devdefault initwriting only the first contact point andreplication_factor = 1- the admin password printed when only it is set
- the migrate exit codes 3 and 6, with the 60 s lock timeout
- the instance id taken from the bind address (
crates/server/src/serve.rs:349)
I ran the compose file's cassandra and cassandra-auth-init programs against a Cassandra 4.1.12 node on an internal network:
- The first run replaced the stock password, and a second run logged in at once. After that, the stock password got
Bad credentials. - The programs refused
cassandraand a password with a trailing space before any change. - A different password failed after 65 s with the
down -vmessage. - A node left on
AllowAllAuthenticatorfailed with "Cassandra accepted a random password".
Findings
Major .github/workflows/release-image.yml:326. No workflow runs ci/smoke-test-cassandra-container.sh or the Compose stack before a release dispatch. The PR says the arm64 leg never ran. The PostgreSQL and MongoDB smoke tests also run only at release, but they ran at earlier releases, and this script never ran in CI. The release builds and smoke-tests the tagged commit. So if the script or the stack fails on ubuntu-24.04-arm at v1.0.0, that tag cannot publish a Cassandra image. The fix then needs a new commit, a version bump and a new tag. Add a workflow on pull_request and workflow_dispatch, filtered to Dockerfile.cassandra, docker-compose.cassandra.yml and the smoke script. Run the script there in build mode on ubuntu-latest and ubuntu-24.04-arm, including on this PR.
Major docs/cassandra-setup.md:24. The local setup runs docker run -d --name cassandra -p 9042:9042. It publishes the native port of Cassandra on every host interface. Step 4 keeps the stock cassandra/cassandra superuser, so anyone who can reach the host can log in as superuser. Line 60 says the node listens on 127.0.0.1:9042, which the command does not do. These lines predate this PR. But the PR edits this file, and its new "Container Image" section says the requirements in it apply to the image. Change line 24 to -p 127.0.0.1:9042:9042. Optional in the same edit: at line 116, pass the password through EXTENDDB_CASSANDRA_PASSWORD instead of --cassandra-pass cassandra, as the image docs do.
Minor ci/smoke-test-cassandra-container.sh:12. Errtrace is off, so functions do not inherit the ERR trap at line 102. A failure inside check_item, cql or set_endpoint exits without the "error: line" message and the serving container's log lines that the PR description promises. I checked this in bash: a failing command in a function exits 1 and the trap does not run. Use set -Eeuo pipefail.
Minor docker/README-cassandra.md:88 and :133. The README says to add contact points and set replication_factor in extenddb.toml, but the file is in a named volume and the image has no editor. Give the command, for example docker run --rm -v extenddb-state:/v --entrypoint sed extenddb/extenddb-cassandra:X.Y.Z -i 's/^replication_factor = 1 .*/replication_factor = 3/' /v/extenddb.toml, and the same form for contact_points. It runs as UID 10001, which owns the file, and GNU sed keeps the 0600 mode.
Optional docker-compose.cassandra.yml:55 and :56. When a guard grep -qx fails, the node exits 1 with nothing in its log, and up reports only that a dependency failed. Print the reason first, for example grep -qx 'authenticator: PasswordAuthenticator' /etc/cassandra/cassandra.yaml || { echo "cassandra.yaml has no top-level authenticator line" >&2; exit 1; }, and the same for the authorizer.
Optional docker/README-cassandra.md:212. Every Compose command, down, ps and logs included, needs the three passwords. Suggest an untracked .env file, as docker/README.md:89 does for the PostgreSQL stack. .gitignore already lists .env.
Optional ci/smoke-test-cassandra-container.sh. Nothing checks that the stack refuses to start without each password. CASSANDRA_SUPERUSER_PASSWORD= docker compose -f docker-compose.cassandra.yml config must fail, and the same for the other two, so a default added to the compose file later fails the smoke test.
Before merge (maintainer step, as the PR description says): create the Docker Hub repository extenddb/extenddb-cassandra. The merge push runs dockerhub-description.yml for the new overview. Its validate job reads the repository from Docker Hub (line 242), so main goes red until the repository exists.
Not checked
- I did not build the image or run
ci/smoke-test-cassandra-container.sh: the review host has no Docker Compose, and a cold image build would load a shared host. Thecassandra-production-buildcheck compiles the same feature set withcargo check --release. - The arm64 entry of the pinned Cassandra index digest.
- The node guard against a
cassandra.yamlwith another layout. - Any run of the release, promote, mirror, sign or Docker Hub description workflows.
Adds extenddb/extenddb-cassandra to the container release pipeline for the 1.0 release, built and released the same way as extenddb-mongodb. - Dockerfile.cassandra: the MongoDB image's Dockerfile with the cassandra feature, its notices file and labels. - docker-compose.cassandra.yml: local reference stack and what the release smoke test runs. One Cassandra 4.1 node with PasswordAuthenticator and CassandraAuthorizer, not published on the host; the node's entrypoint checks that both edits of cassandra.yaml took, since sed exits 0 whether or not a line matched. cassandra-auth-init replaces the stock superuser password on first start through 0600 cqlsh credentials files, fails closed if password authentication is off, and gives up after five minutes. All three passwords are required variables. Later starts run the migrate preview and run migrate --yes only when migrations are pending. - ci/smoke-test-cassandra-container.sh: image identity and hardening, Cassandra authentication and exposure, the application role's credentials and grants, a DynamoDB round trip, persistence across restart, SIGTERM, and, after recreating the stack, the bootstrap's migrate preview and the replaced superuser password. In build mode the image is built with docker build before the bounded compose up. A failed assertion names its line. - smoke-cassandra-image.yml runs that script in build mode on amd64 and arm64 runners for every pull request that changes the image, the compose file or the script, so a release dispatch meets a script that has passed on both. - release-image, promote-image, ghcr-mirror, sign-image and dockerhub-description gain extenddb-cassandra. release-image's gate now refuses a tag that has no Dockerfile or smoke script for the chosen image. The licences push trigger watches SOFTWARE-LICENSE-NOTICES-CASSANDRA.html. promote-image and sign-image read the image input from the environment instead of interpolating it into a script. - The two existing smoke scripts' "tool is absent" checks used `! command -v`, which never fails under `sh -e`; they now exit explicitly. - Docs: docker/README-cassandra.md, the Docker Hub overview, README.md, and docs/cassandra-setup.md (container image section; keyspace replication procedure including system_auth; the local node published on loopback only). Signed-off-by: Scott Robinson <robinnsc@amazon.com>
95e24f0 to
3c174a5
Compare
LeeroyHannigan
left a comment
There was a problem hiding this comment.
3c174a5 resolves every finding from my review.
- Major, a run of the smoke test before a release: resolved.
smoke-cassandra-image.ymlruns the script in build mode onubuntu-latestandubuntu-24.04-armfor each pull request that changes the Dockerfile, the compose file, the script or the workflow. Both jobs passed on this PR. - Major,
docs/cassandra-setup.md:24: resolved. The setup now publishes the local node on 127.0.0.1 only, with a note that says why, andinittakes the superuser password fromEXTENDDB_CASSANDRA_PASSWORD. - Minor, the ERR trap in functions: resolved with
set -Eeuo pipefail. - Minor, how to edit
extenddb.tomlin the volume: resolved with the twosedcommands. - Optional, the guard messages in the compose file: done.
- Optional, an untracked
.envfile: done. - Optional, a check that the stack refuses to start without each password: done. The script renders the compose file with each password empty and expects a failure.
The Docker Hub repository still has to exist before the merge, as the PR description says.
|
Thanks. Everything is in the new head (3c174a5, one commit on main); the body is updated to match. Major, no CI run of the smoke test. New Major, Minor, errtrace. Minor, editing Optional, all three taken. The two Also in the head: rebased onto main after #666, #681, #744, #752, #785 and #424 merged (the cache-scope change is gone, since #785 removed the cache), and the body's maintainer step now names the Secrets Manager token from #744 rather than |
What
Adds
extenddb/extenddb-cassandrato the container release pipeline, built,smoke-tested, scanned, signed, mirrored and promoted the same way as
extenddb-mongodb.Dockerfile.cassandra:Dockerfile.mongodbwith--features cassandra,SOFTWARE-LICENSE-NOTICES-CASSANDRA.htmland its own labels. Same pinnedbases,
apt-get upgrade, UID 10001, setuid bits stripped.docker-compose.cassandra.yml: local reference stack, and what the releasesmoke test runs.
cassandra:4.1node (pinned digest, from the ECR Public mirror) withPasswordAuthenticatorandCassandraAuthorizerenabled incassandra.yaml, soinitgrants the application role only itspermissions (feat(migrations): catalog migrations with a ledger in every store, for Cassandra and MongoDB #681). No host port.
cassandra-auth-initreplaces the stockcassandra/cassandrapassword onfirst start. It hands passwords to cqlsh through 0600 credentials files on
a tmpfs, never on a command line; refuses
cassandraand any passwordcqlsh's configparser would not read back exactly; fails closed if a random
password is accepted (authenticator not active); and reports a password
that differs from the one the volume was first started with.
${VAR:?}variables. ExtendDB is published on127.0.0.1only.migratepreview and runsmigrate --yesonly when it exits 3, soupon a running stack whosecatalog is current succeeds instead of waiting for the running server.
The Cassandra node's entrypoint checks that its two
sededits ofcassandra.yamltook, sincesedexits 0 whether or not a line matched;a
CASSANDRA_IMAGEwhose yaml is laid out differently fails to start,naming the missing line, rather than running with
AllowAllAuthorizer. The auth-init wait isbounded by the clock (5 minutes) rather than by a pass count.
ci/smoke-test-cassandra-container.sh: random passwords per run; imagelabels and
--version; hardening; no Cassandra, cqlsh or JVM in the image;the stock and a random password refused with
Bad credentials; theapplication role logs in with its configured password, can read the catalog
and is refused
ALTER ROLEon the superuser (Unauthorized);extenddb.tomlholds the application password and not the superuser's; a DynamoDB round
trip; persistence across restart; graceful SIGTERM; and, after recreating
the stack on its volumes, the bootstrap's existing-config branch: the
migrate preview, run as the application role, reports the catalog up to
date (the log line is asserted), and the replaced superuser password still
holds. Nothing is pending at this commit, so
migrate --yesis notreached. Every
compose upis bounded byCOMPOSE_UP_TIMEOUT(600 s); inbuild mode the image is built first with
docker build(notcompose build, which needs buildx 0.17) and is not under that bound. Afailed assertion names its line and prints the serving container's last
log lines before the stack is torn down (
set -E, so a failure inside ahelper function is reported too).
timeoutis a checked prerequisite. Thescript first renders the compose file with each password variable empty in
turn and requires each render to fail naming that variable, so a default
added to the compose file later fails the test.
smoke-cassandra-image.yml: on a pull request that changesDockerfile.cassandra, the compose file or the smoke script (and ondispatch), builds the image and runs the smoke script in build mode on
ubuntu-latestandubuntu-24.04-arm. The release workflow runs the samescript against the tagged commit, so without this a script that failed on
the arm64 runner would first fail at a release tag. It runs on this PR.
release-image,promote-image,ghcr-mirror,sign-imageanddockerhub-descriptiongainextenddb-cassandra.release-image's gate now refuses a tag that has no Dockerfile or smokescript for the chosen image, with a message naming the missing file,
before cargo-about is installed. This replaces the MongoDB-only
--backendcheck. For postgres and mongodb every tag gets the same gateresult as on main. The licence step also takes
--backendfromenv:instead of interpolating it into the script.
casethat failsclosed.
promote-imagevalidates the image fromenv:instead of interpolatingthe input into the script;
sign-image's summary step does the same.licenses.yml's push trigger also watchesSOFTWARE-LICENSE-NOTICES-CASSANDRA.html(the PR check already ran it).ci/smoke-test-container.shandci/smoke-test-mongodb-container.shused! command -v, which never failsunder
sh -e. They now exit explicitly.docker/README-cassandra.md(init, serve, certificate, thesedcommands that edit
contact_pointsandreplication_factorin the volume,replication factor, production, upgrade, reference stack with an
.envfile for the three passwords),
docker/dockerhub-overview-cassandra.md,README.md(which now also saysSQLite has no production image and links
docs/dev-image.md), anddocs/cassandra-setup.md(a container image section, and a keyspacereplication procedure that covers the keyspaces
initcreates at RF 1 andsystem_auth; the local node is now published on127.0.0.1:9042only, asthe text already said, and
inittakes the superuser password fromEXTENDDB_CASSANDRA_PASSWORD). Both image pages carry the backend's experimental notice,recommend
CassandraAuthorizer, and say to use a cluster for ExtendDBalone under
AllowAllAuthorizer, where the application role can read, dropand alter everything in the cluster.
Why
1.0 is the first release with a published Cassandra-backed artifact. Until
now the backend could only be built from source, and
release-imagehad noCassandra option.
Testing done
On an x86_64 host, at this commit (rebased onto main after #666, #681, #744,
#752 and #785 merged; amd64 only):
ci/smoke-test-cassandra-container.shpasses end to end in build mode(the image built by the script) and in prebuilt mode against that image,
including the recreate step's migrate preview with the rotated superuser
password under
CassandraAuthorizer. Before this revision, build modecould not finish a cold build: the 600 s bound covered the image build.
docker compose upon a running, current stack exits 0 after 5 s(the preview reports up to date).
authorizer:renamed in the image'scassandra.yaml, the entrypoint exits 1 before handing over; unchanged, itreaches the hand-over.
extracted from the compose file (
docker compose config,$$undone).During earlier review rounds, on this branch before #681 merged (same image
content apart from #681's code, earlier compose file without the authorizer
and the preview step):
cassandra-auth-initagainst a scratch node: the stock password isrefused; the first start rotates the password; a restart with a different
password fails with the
down -vmessage; a trailing non-breaking space andan embedded carriage return are refused before rotation; a password with
' " \ % $ # ; = [ ]and non-ASCII characters rotates and logs in; nocredentials file is left on the container's writable layer;
downleavesno anonymous volume.
init,serveand certificate commands (only the image tag,the contact point and
--networksubstituted) against a scratchpassword-authenticated node: init succeeds on a named volume, serve
is healthy under
--read-only --cap-drop ALL.ALTER KEYSPACE system_auth ...from the replication procedure runs onCassandra 4.1.12. On a single node it then makes the default superuser's
logins fail with
Cannot achieve consistency level QUORUM, which is why theguide caps the factor at the node count.
high or critical findings.
ci/smoke-test-container.shandci/smoke-test-mongodb-container.shpassagainst
extenddb-postgres:0.1.14andextenddb-mongodb:0.1.14; their newabsence checks exit 1 when the tool is planted on
PATH, where the oldones passed.
The arm64 build and smoke run is the
smoke (arm64)check on this PR. Nottested: any release workflow run, since no tag contains
Dockerfile.cassandra.Not closed here
readable after propagation. Today a private GHCR package would pass the
pipeline.
about 18 hand-kept copies).
cassandra-auth-initprogram lives inline in the compose file, with$doubled, so it is linted only by extracting it. Moving it to a fileunder
docker/and bind-mounting it would make it lintable in place at thecost of the compose file no longer standing on its own.
migrate --yes(nothing is pending at anycommit that has the image), nor the preview's exit-3 branch.
docker/dockerhub-overview-mongodb.mdanddocker/README-mongodb.mdhavethe same host-directory ownership problem the Cassandra pages had, and
docker/README-mongodb.mdpins a0.1.8tag that was never published (notag before v0.1.12 has
Dockerfile.mongodb).Maintainer setup
Before merge:
extenddb/extenddb-cassandra, andconfirm the access token in Secrets Manager
extenddb/dockerhub-publish(read through the
ExtendDBDockerHubPublisherrole since ci: read the Docker Hub credential from Secrets Manager via OIDC #744) can pushto it and edit its description. The merge push runs the Docker Hub
description workflow for the new overview, and its validate job fails
while the repository does not exist.
Before the first
release-imagedispatch forextenddb-cassandra:extenddb/extenddb-cassandra, and confirmthe
ExtendDBEcrPublicGitHubPublisherpolicy covers it.ghcr.io/extenddb/extenddb-cassandratothe repository and make it public before promoting.
guide and both image pages carry it.
Checklist
cargo test --workspace) — not run: no Rust source changedci/smoke-test-cassandra-container.shADR / RFC: n/a — packaging and release workflow only.
Breaking changes
None.
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.