Skip to content

feat(container): publish a Cassandra-backed ExtendDB image - #725

Merged
LeeroyHannigan merged 1 commit into
mainfrom
feat/cassandra-image
Nov 4, 2026
Merged

LeeroyHannigan merged 1 commit into
mainfrom
feat/cassandra-image

Conversation

@robinnsc

@robinnsc robinnsc commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds extenddb/extenddb-cassandra to the container release pipeline, built,
smoke-tested, scanned, signed, mirrored and promoted the same way as
extenddb-mongodb.

  • Dockerfile.cassandra: Dockerfile.mongodb with --features cassandra,
    SOFTWARE-LICENSE-NOTICES-CASSANDRA.html and its own labels. Same pinned
    bases, apt-get upgrade, UID 10001, setuid bits stripped.
  • docker-compose.cassandra.yml: local reference stack, and what the release
    smoke test runs.
    • One cassandra:4.1 node (pinned digest, from the ECR Public mirror) with
      PasswordAuthenticator and CassandraAuthorizer enabled in
      cassandra.yaml, so init grants the application role only its
      permissions (feat(migrations): catalog migrations with a ledger in every store, for Cassandra and MongoDB #681). No host port.
    • cassandra-auth-init replaces the stock cassandra/cassandra password on
      first start. It hands passwords to cqlsh through 0600 credentials files on
      a tmpfs, never on a command line; refuses cassandra and any password
      cqlsh's configparser would not read back exactly; fails closed if a random
      password is accepted (authenticator not active); and reports a password
      that differs from the one the volume was first started with.
    • The superuser, application-role and admin passwords are required
      ${VAR:?} variables. ExtendDB is published on 127.0.0.1 only.
    • On later starts the bootstrap runs the migrate preview and runs
      migrate --yes only when it exits 3, so up on a running stack whose
      catalog is current succeeds instead of waiting for the running server.
      The Cassandra node's entrypoint checks that its two sed edits of
      cassandra.yaml took, since sed exits 0 whether or not a line matched;
      a CASSANDRA_IMAGE whose yaml is laid out differently fails to start,
      naming the missing line, rather than running with AllowAllAuthorizer. The auth-init wait is
      bounded by the clock (5 minutes) rather than by a pass count.
  • ci/smoke-test-cassandra-container.sh: random passwords per run; image
    labels and --version; hardening; no Cassandra, cqlsh or JVM in the image;
    the stock and a random password refused with Bad credentials; the
    application role logs in with its configured password, can read the catalog
    and is refused ALTER ROLE on the superuser (Unauthorized); extenddb.toml
    holds the application password and not the superuser's; a DynamoDB round
    trip; persistence across restart; graceful SIGTERM; and, after recreating
    the stack on its volumes, the bootstrap's existing-config branch: the
    migrate preview, run as the application role, reports the catalog up to
    date (the log line is asserted), and the replaced superuser password still
    holds. Nothing is pending at this commit, so migrate --yes is not
    reached. Every compose up is bounded by COMPOSE_UP_TIMEOUT (600 s); in
    build mode the image is built first with docker build (not
    compose build, which needs buildx 0.17) and is not under that bound. A
    failed assertion names its line and prints the serving container's last
    log lines before the stack is torn down (set -E, so a failure inside a
    helper function is reported too). timeout is a checked prerequisite. The
    script first renders the compose file with each password variable empty in
    turn and requires each render to fail naming that variable, so a default
    added to the compose file later fails the test.
  • New workflow smoke-cassandra-image.yml: on a pull request that changes
    Dockerfile.cassandra, the compose file or the smoke script (and on
    dispatch), builds the image and runs the smoke script in build mode on
    ubuntu-latest and ubuntu-24.04-arm. The release workflow runs the same
    script against the tagged commit, so without this a script that failed on
    the arm64 runner would first fail at a release tag. It runs on this PR.
  • Workflows: release-image, promote-image, ghcr-mirror, sign-image and
    dockerhub-description gain extenddb-cassandra.
    • release-image's gate now refuses a tag that has no Dockerfile or smoke
      script for the chosen image, with a message naming the missing file,
      before cargo-about is installed. This replaces the MongoDB-only
      --backend check. For postgres and mongodb every tag gets the same gate
      result as on main. The licence step also takes --backend from env:
      instead of interpolating it into the script.
    • The smoke step dispatches on the image's variable with a case that fails
      closed.
    • promote-image validates the image from env: instead of interpolating
      the input into the script; sign-image's summary step does the same.
    • licenses.yml's push trigger also watches
      SOFTWARE-LICENSE-NOTICES-CASSANDRA.html (the PR check already ran it).
  • The "tool is absent" checks in ci/smoke-test-container.sh and
    ci/smoke-test-mongodb-container.sh used ! command -v, which never fails
    under sh -e. They now exit explicitly.
  • Docs: docker/README-cassandra.md (init, serve, certificate, the sed
    commands that edit contact_points and replication_factor in the volume,
    replication factor, production, upgrade, reference stack with an .env
    file for the three passwords),
    docker/dockerhub-overview-cassandra.md, README.md (which now also says
    SQLite has no production image and links docs/dev-image.md), and
    docs/cassandra-setup.md (a container image section, and a keyspace
    replication procedure that covers the keyspaces init creates at RF 1 and
    system_auth; the local node is now published on 127.0.0.1:9042 only, as
    the text already said, and init takes the superuser password from
    EXTENDDB_CASSANDRA_PASSWORD). Both image pages carry the backend's experimental notice,
    recommend CassandraAuthorizer, and say to use a cluster for ExtendDB
    alone under AllowAllAuthorizer, where the application role can read, drop
    and alter everything in the cluster.

Why

1.0 is the first release with a published Cassandra-backed artifact. Until
now the backend could only be built from source, and release-image had no
Cassandra option.

Testing done

On an x86_64 host, at this commit (rebased onto main after #666, #681, #744,
#752 and #785 merged; amd64 only):

  • ci/smoke-test-cassandra-container.sh passes end to end in build mode
    (the image built by the script) and in prebuilt mode against that image,
    including the recreate step's migrate preview with the rotated superuser
    password under CassandraAuthorizer. Before this revision, build mode
    could not finish a cold build: the 600 s bound covered the image build.
  • A second docker compose up on a running, current stack exits 0 after 5 s
    (the preview reports up to date).
  • The Cassandra entrypoint's guard: with authorizer: renamed in the image's
    cassandra.yaml, the entrypoint exits 1 before handing over; unchanged, it
    reaches the hand-over.
  • actionlint 1.7.7 on the new workflow and the four touched ones: clean.
  • shellcheck clean on the three smoke scripts and on the three scripts
    extracted from the compose file (docker compose config, $$ undone).

During earlier review rounds, on this branch before #681 merged (same image
content apart from #681's code, earlier compose file without the authorizer
and the preview step):

  • cassandra-auth-init against a scratch node: the stock password is
    refused; the first start rotates the password; a restart with a different
    password fails with the down -v message; a trailing non-breaking space and
    an embedded carriage return are refused before rotation; a password with
    ' " \ % $ # ; = [ ] and non-ASCII characters rotates and logs in; no
    credentials file is left on the container's writable layer; down leaves
    no anonymous volume.
  • The README's init, serve and certificate commands (only the image tag,
    the contact point and --network substituted) against a scratch
    password-authenticated node: init succeeds on a named volume, serve
    is healthy under --read-only --cap-drop ALL.
  • ALTER KEYSPACE system_auth ... from the replication procedure runs on
    Cassandra 4.1.12. On a single node it then makes the default superuser's
    logins fail with Cannot achieve consistency level QUORUM, which is why the
    guide caps the factor at the node count.
  • Trivy 0.76.0 with the release gate's flags on the saved image: no fixable
    high or critical findings.
  • ci/smoke-test-container.sh and ci/smoke-test-mongodb-container.sh pass
    against extenddb-postgres:0.1.14 and extenddb-mongodb:0.1.14; their new
    absence checks exit 1 when the tool is planted on PATH, where the old
    ones passed.

The arm64 build and smoke run is the smoke (arm64) check on this PR. Not
tested: any release workflow run, since no tag contains Dockerfile.cassandra.

Not closed here

  • An anonymous check that the GHCR and ECR Public copies are publicly
    readable after propagation. Today a private GHCR package would pass the
    pipeline.
  • A test that the image list agrees across the workflows (the list now has
    about 18 hand-kept copies).
  • The cassandra-auth-init program lives inline in the compose file, with
    $ doubled, so it is linted only by extracting it. Moving it to a file
    under docker/ and bind-mounting it would make it lintable in place at the
    cost of the compose file no longer standing on its own.
  • The smoke test does not reach migrate --yes (nothing is pending at any
    commit that has the image), nor the preview's exit-3 branch.
  • docker/dockerhub-overview-mongodb.md and docker/README-mongodb.md have
    the same host-directory ownership problem the Cassandra pages had, and
    docker/README-mongodb.md pins a 0.1.8 tag that was never published (no
    tag before v0.1.12 has Dockerfile.mongodb).

Maintainer setup

Before merge:

  1. Create the public Docker Hub repository extenddb/extenddb-cassandra, and
    confirm the access token in Secrets Manager extenddb/dockerhub-publish
    (read through the ExtendDBDockerHubPublisher role since ci: read the Docker Hub credential from Secrets Manager via OIDC #744) can push
    to it and edit its description. The merge push runs the Docker Hub
    description workflow for the new overview, and its validate job fails
    while the repository does not exist.

Before the first release-image dispatch for extenddb-cassandra:

  1. Create the ECR Public repository extenddb/extenddb-cassandra, and confirm
    the ExtendDBEcrPublicGitHubPublisher policy covers it.
  2. After the first propagate, link ghcr.io/extenddb/extenddb-cassandra to
    the repository and make it public before promoting.
  3. Decide whether 1.0 changes the backend's "experimental" status; the setup
    guide and both image pages carry it.

Checklist

  • All tests pass (cargo test --workspace) — not run: no Rust source changed
  • Code is formatted / clippy — not applicable to Rust; shellcheck and actionlint as above
  • I have added or updated tests for new functionality — ci/smoke-test-cassandra-container.sh
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)

ADR / RFC: n/a — packaging and release workflow only.

Breaking changes

None.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

@robinnsc
robinnsc force-pushed the feat/cassandra-image branch from 95f2762 to dfaa726 Compare November 1, 2026 09:44
@robinnsc
robinnsc changed the base branch from main to feat/catalog-migrations-cassandra-mongodb November 1, 2026 09:45
Base automatically changed from feat/catalog-migrations-cassandra-mongodb to main November 2, 2026 22:45
@robinnsc robinnsc added this to the 1.0 milestone Nov 3, 2026
@robinnsc robinnsc added the release-blocker Must ship for its milestone's release label Nov 3, 2026
@robinnsc
robinnsc force-pushed the feat/cassandra-image branch from dfaa726 to 95e24f0 Compare November 3, 2026 22:46
@robinnsc
robinnsc marked this pull request as ready for review November 3, 2026 22:47

@LeeroyHannigan LeeroyHannigan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two findings below need a change before merge. The others are Minor or Optional.

This PR adds extenddb/extenddb-cassandra, with Dockerfile.cassandra, a reference Compose stack with password authentication and CassandraAuthorizer, a smoke test and three docs pages. The release, promote, mirror, sign and Docker Hub description workflows now include the image.

Dockerfile.cassandra differs from Dockerfile.mongodb only in --features cassandra, the notices file and the labels. So it keeps the digest-pinned bases, UID 10001, the setuid strip, tini and the health check. Every image allowlist in the five image workflows names extenddb-cassandra, and licenses.yml now runs on changes to the Cassandra notices file. No action reference or permission changed. The publish, sign and propagate jobs take the repository from the gate job's output. So the workflows sign and mirror the image under its own name. git log shows Dockerfile.mongodb and the generator's --backend flag arriving together in #331. So the new gate gives postgres and mongodb tags the result they had before.

The Compose stack publishes ExtendDB on 127.0.0.1 only, and Cassandra publishes no port. The three passwords are ${VAR:?} references, so Compose stops with an error when one is missing. shellcheck 0.11.0 passes on the three smoke scripts and on the three programs extracted from the compose file. actionlint 1.7.12 reports only the custom runner label.

The docs match the code on these points:

  • the grants (crates/storage-cassandra/src/migrations/mod.rs:600)
  • the character set of the application password and its extenddb-local-dev default
  • init writing only the first contact point and replication_factor = 1
  • the admin password printed when only it is set
  • the migrate exit codes 3 and 6, with the 60 s lock timeout
  • the instance id taken from the bind address (crates/server/src/serve.rs:349)

I ran the compose file's cassandra and cassandra-auth-init programs against a Cassandra 4.1.12 node on an internal network:

  • The first run replaced the stock password, and a second run logged in at once. After that, the stock password got Bad credentials.
  • The programs refused cassandra and a password with a trailing space before any change.
  • A different password failed after 65 s with the down -v message.
  • A node left on AllowAllAuthenticator failed with "Cassandra accepted a random password".

Findings

Major .github/workflows/release-image.yml:326. No workflow runs ci/smoke-test-cassandra-container.sh or the Compose stack before a release dispatch. The PR says the arm64 leg never ran. The PostgreSQL and MongoDB smoke tests also run only at release, but they ran at earlier releases, and this script never ran in CI. The release builds and smoke-tests the tagged commit. So if the script or the stack fails on ubuntu-24.04-arm at v1.0.0, that tag cannot publish a Cassandra image. The fix then needs a new commit, a version bump and a new tag. Add a workflow on pull_request and workflow_dispatch, filtered to Dockerfile.cassandra, docker-compose.cassandra.yml and the smoke script. Run the script there in build mode on ubuntu-latest and ubuntu-24.04-arm, including on this PR.

Major docs/cassandra-setup.md:24. The local setup runs docker run -d --name cassandra -p 9042:9042. It publishes the native port of Cassandra on every host interface. Step 4 keeps the stock cassandra/cassandra superuser, so anyone who can reach the host can log in as superuser. Line 60 says the node listens on 127.0.0.1:9042, which the command does not do. These lines predate this PR. But the PR edits this file, and its new "Container Image" section says the requirements in it apply to the image. Change line 24 to -p 127.0.0.1:9042:9042. Optional in the same edit: at line 116, pass the password through EXTENDDB_CASSANDRA_PASSWORD instead of --cassandra-pass cassandra, as the image docs do.

Minor ci/smoke-test-cassandra-container.sh:12. Errtrace is off, so functions do not inherit the ERR trap at line 102. A failure inside check_item, cql or set_endpoint exits without the "error: line" message and the serving container's log lines that the PR description promises. I checked this in bash: a failing command in a function exits 1 and the trap does not run. Use set -Eeuo pipefail.

Minor docker/README-cassandra.md:88 and :133. The README says to add contact points and set replication_factor in extenddb.toml, but the file is in a named volume and the image has no editor. Give the command, for example docker run --rm -v extenddb-state:/v --entrypoint sed extenddb/extenddb-cassandra:X.Y.Z -i 's/^replication_factor = 1 .*/replication_factor = 3/' /v/extenddb.toml, and the same form for contact_points. It runs as UID 10001, which owns the file, and GNU sed keeps the 0600 mode.

Optional docker-compose.cassandra.yml:55 and :56. When a guard grep -qx fails, the node exits 1 with nothing in its log, and up reports only that a dependency failed. Print the reason first, for example grep -qx 'authenticator: PasswordAuthenticator' /etc/cassandra/cassandra.yaml || { echo "cassandra.yaml has no top-level authenticator line" >&2; exit 1; }, and the same for the authorizer.

Optional docker/README-cassandra.md:212. Every Compose command, down, ps and logs included, needs the three passwords. Suggest an untracked .env file, as docker/README.md:89 does for the PostgreSQL stack. .gitignore already lists .env.

Optional ci/smoke-test-cassandra-container.sh. Nothing checks that the stack refuses to start without each password. CASSANDRA_SUPERUSER_PASSWORD= docker compose -f docker-compose.cassandra.yml config must fail, and the same for the other two, so a default added to the compose file later fails the smoke test.

Before merge (maintainer step, as the PR description says): create the Docker Hub repository extenddb/extenddb-cassandra. The merge push runs dockerhub-description.yml for the new overview. Its validate job reads the repository from Docker Hub (line 242), so main goes red until the repository exists.

Not checked

  • I did not build the image or run ci/smoke-test-cassandra-container.sh: the review host has no Docker Compose, and a cold image build would load a shared host. The cassandra-production-build check compiles the same feature set with cargo check --release.
  • The arm64 entry of the pinned Cassandra index digest.
  • The node guard against a cassandra.yaml with another layout.
  • Any run of the release, promote, mirror, sign or Docker Hub description workflows.

Adds extenddb/extenddb-cassandra to the container release pipeline for the
1.0 release, built and released the same way as extenddb-mongodb.

- Dockerfile.cassandra: the MongoDB image's Dockerfile with the cassandra
  feature, its notices file and labels.
- docker-compose.cassandra.yml: local reference stack and what the release
  smoke test runs. One Cassandra 4.1 node with PasswordAuthenticator and
  CassandraAuthorizer, not published on the host; the node's entrypoint
  checks that both edits of cassandra.yaml took, since sed exits 0 whether or
  not a line matched. cassandra-auth-init replaces the stock superuser
  password on first start through 0600 cqlsh credentials files, fails closed
  if password authentication is off, and gives up after five minutes. All
  three passwords are required variables. Later starts run the migrate
  preview and run migrate --yes only when migrations are pending.
- ci/smoke-test-cassandra-container.sh: image identity and hardening,
  Cassandra authentication and exposure, the application role's credentials
  and grants, a DynamoDB round trip, persistence across restart, SIGTERM,
  and, after recreating the stack, the bootstrap's migrate preview and the
  replaced superuser password. In build mode the image is built with docker
  build before the bounded compose up. A failed assertion names its line.
- smoke-cassandra-image.yml runs that script in build mode on amd64 and
  arm64 runners for every pull request that changes the image, the compose
  file or the script, so a release dispatch meets a script that has passed
  on both.
- release-image, promote-image, ghcr-mirror, sign-image and
  dockerhub-description gain extenddb-cassandra. release-image's gate now
  refuses a tag that has no Dockerfile or smoke script for the chosen image.
  The licences push trigger watches SOFTWARE-LICENSE-NOTICES-CASSANDRA.html.
  promote-image and sign-image read the image input from the environment
  instead of interpolating it into a script.
- The two existing smoke scripts' "tool is absent" checks used `! command -v`,
  which never fails under `sh -e`; they now exit explicitly.
- Docs: docker/README-cassandra.md, the Docker Hub overview, README.md, and
  docs/cassandra-setup.md (container image section; keyspace replication
  procedure including system_auth; the local node published on loopback
  only).

Signed-off-by: Scott Robinson <robinnsc@amazon.com>
@robinnsc
robinnsc force-pushed the feat/cassandra-image branch from 95e24f0 to 3c174a5 Compare November 4, 2026 00:23

@LeeroyHannigan LeeroyHannigan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3c174a5 resolves every finding from my review.

  • Major, a run of the smoke test before a release: resolved. smoke-cassandra-image.yml runs the script in build mode on ubuntu-latest and ubuntu-24.04-arm for each pull request that changes the Dockerfile, the compose file, the script or the workflow. Both jobs passed on this PR.
  • Major, docs/cassandra-setup.md:24: resolved. The setup now publishes the local node on 127.0.0.1 only, with a note that says why, and init takes the superuser password from EXTENDDB_CASSANDRA_PASSWORD.
  • Minor, the ERR trap in functions: resolved with set -Eeuo pipefail.
  • Minor, how to edit extenddb.toml in the volume: resolved with the two sed commands.
  • Optional, the guard messages in the compose file: done.
  • Optional, an untracked .env file: done.
  • Optional, a check that the stack refuses to start without each password: done. The script renders the compose file with each password empty and expects a failure.

The Docker Hub repository still has to exist before the merge, as the PR description says.

@robinnsc

robinnsc commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks. Everything is in the new head (3c174a5, one commit on main); the body is updated to match.

Major, no CI run of the smoke test. New .github/workflows/smoke-cassandra-image.yml: on pull_request filtered to Dockerfile.cassandra, docker-compose.cassandra.yml, the smoke script and the workflow file, and on workflow_dispatch, it runs ci/smoke-test-cassandra-container.sh in build mode on ubuntu-latest and ubuntu-24.04-arm. Both legs passed on this PR (smoke (amd64) 6m28s, smoke (arm64) 6m40s), which is the first arm64 run of the script. The AWS CLI install block is the pinned one from release-image.yml; no new third-party action.

Major, docs/cassandra-setup.md:24. -p 127.0.0.1:9042:9042, with a sentence saying why, so line 60 is now true. The init example takes the password through EXTENDDB_CASSANDRA_PASSWORD, as the image docs do.

Minor, errtrace. set -Eeuo pipefail. Checked: a failing command inside a function now reports its line and dumps the serving container's log, and a function called inside an if condition (expect_bad_credentials) still does not trigger the trap.

Minor, editing extenddb.toml in the volume. The README now gives the sed command for contact_points and for replication_factor, in the form you suggested, with docker restart after the second.

Optional, all three taken. The two grep -qx guards print which line is missing before exiting 1. The README recommends an untracked .env file for the three passwords, as docker/README.md does. The smoke script starts by rendering the compose file with each password variable empty in turn and requires each render to fail naming that variable.

Also in the head: rebased onto main after #666, #681, #744, #752, #785 and #424 merged (the cache-scope change is gone, since #785 removed the cache), and the body's maintainer step now names the Secrets Manager token from #744 rather than DOCKERHUB_TOKEN.

@LeeroyHannigan
LeeroyHannigan added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 7c8a776 Nov 4, 2026
34 checks passed
@LeeroyHannigan
LeeroyHannigan deleted the feat/cassandra-image branch November 4, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-blocker Must ship for its milestone's release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants