Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
684a26c
style: tidy the Rust builder Dockerfile
kazmosahebi Oct 23, 2026
02fd771
fix: Kafbat follows the broker address
kazmosahebi Oct 26, 2026
74f97bf
fix: keep the receiver configs to transport only
kazmosahebi Oct 26, 2026
2d5dd62
style: cut the bearer-token mount comment to one line
kazmosahebi Oct 26, 2026
91773eb
fix: give bearer-tokens-init the engine's literal config path
kazmosahebi Oct 26, 2026
67c28d6
fix: name the HyperDX DocumentDB login for what it is
kazmosahebi Oct 26, 2026
8f1260d
fix: say the DocumentDB pin defaults in compose
kazmosahebi Oct 26, 2026
c9c4617
fix: write no inline comments into .env pins
kazmosahebi Oct 26, 2026
1f62e9b
fix: share the compose helpers and tidy the cores rollup
kazmosahebi Oct 26, 2026
a74d982
fix: one helper for the source runner's checkouts, tidy the proof
kazmosahebi Oct 26, 2026
6c7ba64
fix: keep the console key owner-only on a shared checkout
kazmosahebi Oct 26, 2026
b2b074c
style: tidy the identifier-quote helper
kazmosahebi Oct 26, 2026
8ab7bff
style: tidy the log line helper and its tests
kazmosahebi Oct 27, 2026
187a7d6
style: make init's argv keyword-only
kazmosahebi Oct 27, 2026
9297b91
style: derive the proof's source cases from their apps, tidy the prof…
kazmosahebi Oct 27, 2026
4771e64
fix: read the e2e poll interval from one constant, its test included
kazmosahebi Oct 27, 2026
52b0a99
style: one-line engine transport docs, its test on a case table
kazmosahebi Oct 27, 2026
fff8abb
chore: give the gRPC archiver reason its own paragraph, per-source ap…
kazmosahebi Oct 27, 2026
e26fde2
style: sort the receiver's record cap among its keys
kazmosahebi Oct 27, 2026
8f2f497
fix: keep make demo's re-runs honest and its vendor sources dormant
kazmosahebi Oct 27, 2026
14e2bba
fix: regenerate the FortiGate demo sample from its generator, lint th…
kazmosahebi Oct 27, 2026
3bd7b44
fix: say the transforms push to the loader on gRPC
kazmosahebi Oct 27, 2026
2549d7f
style: conform make demo to the cleanup, note a receiver restart ends…
kazmosahebi Oct 27, 2026
790dbd2
style: one-sentence comments in compose and .env.example
kazmosahebi Oct 27, 2026
849777b
style: trim the collector's usage comments
kazmosahebi Oct 27, 2026
6e8077a
fix: full source files and fetchers for make demo
kazmosahebi Oct 28, 2026
f6efcf3
fix: per-component env files and seed targets
kazmosahebi Oct 28, 2026
0fe5659
fix: trim env templates to keys in use
kazmosahebi Oct 29, 2026
ca59c42
fix: env files without comments, main's commits in branch style
kazmosahebi Oct 29, 2026
f288410
fix: generate the dev and unpublish overlays, move the rest into comp…
kazmosahebi Oct 29, 2026
2626958
fix: .env.example by component, real defaults
kazmosahebi Oct 30, 2026
d44c775
fix: compose templates, one-shots and checks
kazmosahebi Oct 30, 2026
2de28d8
fix: compose comment facts into the docs
kazmosahebi Oct 30, 2026
218dc1a
fix: apache log dirs, shared app env, config paths
kazmosahebi Nov 2, 2026
ef35866
fix: bash probes and stop grace on the templates
kazmosahebi Nov 2, 2026
08d96bb
fix: loader and receiver ports, compose comments to docs
kazmosahebi Nov 2, 2026
6ba7737
fix: drop the transforms' dead ports and keys
kazmosahebi Nov 2, 2026
bc6fab8
fix: drop unread engine vars, admin name to engine.env
kazmosahebi Nov 2, 2026
5062b1c
fix: keep the engine's logins out of the hunt runner
kazmosahebi Nov 2, 2026
b650e04
fix: ui, proxy and collector comments to docs
kazmosahebi Nov 2, 2026
1579f98
fix: set the archiver's DFE_ARCHIVER_ env names
kazmosahebi Nov 2, 2026
c8426c8
fix: HyperDX comments to docs, transform env names
kazmosahebi Nov 3, 2026
2f95f3b
fix: merge main, make demo on bare python3
kazmosahebi Nov 3, 2026
a6ee674
fix: read the engine's DFE_ENGINE_ names
kazmosahebi Nov 3, 2026
5d26c61
fix: merge main, demo Sources stay YAML
kazmosahebi Nov 3, 2026
1c99b18
fix: move compose comments to docs
kazmosahebi Nov 4, 2026
6fe1f4a
fix: merge main, keep the engine's env names
kazmosahebi Nov 4, 2026
5b0c6aa
fix: mount dfe-engine-config read-only in hunt runner
kazmosahebi Nov 4, 2026
4add9c2
fix: limit hunt runner to the config it reads
kazmosahebi Nov 4, 2026
3297356
fix: simplify the self-update daemon, tick hourly
kazmosahebi Nov 4, 2026
ac65f2b
fix: set single-kafka as the active profile
kazmosahebi Nov 4, 2026
3dc7c26
fix: replace acme seed org with default
kazmosahebi Nov 4, 2026
e2f494a
fix: expect single-kafka links in default test
kazmosahebi Nov 4, 2026
1af9b37
fix: bring down all compose profiles in e2e stack_down
kazmosahebi Nov 4, 2026
69a79b6
fix: leave the Culvert pin out of make stack
kazmosahebi Nov 5, 2026
4619318
fix: document DFE_STACK_VERSION and the orgs make hunts creates
kazmosahebi Nov 5, 2026
10b71f5
fix: give e2e config checks the test's config
kazmosahebi Nov 5, 2026
f0efb2e
fix: let the daemon follow DFE image releases
kazmosahebi Nov 5, 2026
1d9c831
fix: drive the e2e console at the external origin
kazmosahebi Nov 5, 2026
93d5bf3
fix(seed): set poll intervals on the AWS seeds
kazmosahebi Nov 9, 2026
001bc39
fix(otel): give container logs a level
kazmosahebi Nov 9, 2026
0b45964
fix(otel): scrape HyperDX's metrics
kazmosahebi Nov 9, 2026
7a44b1e
Merge remote-tracking branch 'origin/main' into chore/cleanup-main
kazmosahebi Nov 10, 2026
afeefcd
style: apply the code style rules to the script and proxy lines
kazmosahebi Nov 10, 2026
6ccb804
fix(e2e): give the engine a schema fixtures dir in e2e posture
kazmosahebi Nov 10, 2026
8c7280b
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 12, 2026
d3fef9a
fix(otel): log HyperDX under dfe-hyperdx
kazmosahebi Nov 11, 2026
bc1d0fd
fix(demo): read the demo org from a seed file
kazmosahebi Nov 12, 2026
fa223ea
style(scripts): restyle what main added
kazmosahebi Nov 12, 2026
a42fa87
style: restyle main's comments, workflows and docs
kazmosahebi Nov 12, 2026
ef13bf8
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 13, 2026
9a9946f
style: restyle the shared fetcher lines
kazmosahebi Nov 13, 2026
3cb0b8c
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 16, 2026
504ad7b
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 16, 2026
f7fb968
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 17, 2026
0ec6bcc
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 19, 2026
a5d0abf
style: apply code style to scripts, tests and demo generators
kazmosahebi Nov 20, 2026
ef8d704
fix(orgs): seed acme as the deployment's own org
kazmosahebi Nov 20, 2026
6bff940
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 21, 2026
637241e
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 21, 2026
4f2c03f
fix(compose): mount a volume on the Redpanda config one-shot
kazmosahebi Nov 21, 2026
3819703
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 21, 2026
d773e96
Merge remote-tracking branch 'origin/main' into chore/cleanup
kazmosahebi Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 5 additions & 9 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,11 @@
#
# License: BUSL-1.1
# Copyright: (c) 2026 HYPERI PTY LIMITED
#
# The dfe-docker image copies its whole context into the folder `init` sets up,
# so this names what that folder holds and nothing more. An allowlist, because
# a file nobody listed then stays out: .env, env/, certs/ and access-summary.md
# hold secrets and can never reach an image layer.
#
# scripts/build_dev_images.py stages each component's source into its own temp
# context, so this file does not reach those builds.

# An allowlist, as init's folder holds only what it names
*

# Secrets stay out: .env, env/, certs/, access-summary.md
!.env.example
!Makefile
!VERSION
Expand All @@ -31,7 +25,9 @@
!scripts/
!service_profiles.yaml

# Inside the kept directories: tests and bytecode never run in a deployment
# Inside kept directories, tests and bytecode never run
scripts/tests/
**/__pycache__/
**/*.pyc

# build_dev_images.py stages its own contexts, so this skips it
88 changes: 44 additions & 44 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,7 @@
# License: BUSL-1.1 - HYPERI PTY LIMITED
# Copyright: (c) 2026 HYPERI PTY LIMITED
#
# Hand-written: hyperi-ci has no reusable workflow for a compose-packaging repo.
# ubuntu-26.04 is named, not ubuntu-latest, so a label move cannot swap the tools and the arm64 image build, which has no -latest label, runs on the same OS.
# Renovate automerges patch and digest updates once this workflow passes.
# Hand-written (no hyperi-ci compose workflow), runner pinned

name: CI

Expand All @@ -16,26 +14,26 @@ on:
push:
branches:
- main
# The inputs `hyperi-ci release` dispatches with. A bare dispatch only validates.
# `hyperi-ci release` sets these, a bare dispatch only validates
workflow_dispatch:
inputs:
tag:
type: string
bump:
default: "auto"
description: "Version for from-head: auto | patch | minor (forced) | X.Y.Z (exact)."
required: false
default: ""
description: "Existing tag to publish a GitHub release for when it has none."
from-head:
type: string
required: false
from-head:
default: ""
description: "'true' to release the current HEAD. The CI creates the tag."
bump:
required: false
type: string
tag:
default: ""
description: "Existing tag to publish a GitHub release for when it has none."
required: false
default: "auto"
description: "Version for from-head: auto | patch | minor (forced) | X.Y.Z (exact)."
type: string

# A release run has its own group and is never cancelled, because a cancel after the tag push leaves a tag with no GitHub release.
# Release runs are never cancelled, so no tag loses its release
concurrency:
group: "${{ github.workflow }}-${{ (inputs.tag != '' || inputs.from-head == 'true' || github.event_name == 'push' && (contains(github.event.head_commit.message, 'Release: true') || contains(github.event.head_commit.message, 'Release:true') || contains(github.event.head_commit.message, 'Publish: true') || contains(github.event.head_commit.message, 'Publish:true'))) && 'release-' || '' }}${{ github.ref }}"
cancel-in-progress: "${{ !(inputs.tag != '' || inputs.from-head == 'true' || github.event_name == 'push' && (contains(github.event.head_commit.message, 'Release: true') || contains(github.event.head_commit.message, 'Release:true') || contains(github.event.head_commit.message, 'Publish: true') || contains(github.event.head_commit.message, 'Publish:true'))) }}"
Expand All @@ -47,7 +45,7 @@ permissions:
# The house Python floor. The runner's own python3 is older.
env:
PYTHON_VERSION: "3.14.7"
# The hyperi-ci CLI the Release job runs. The HYPERCI_INSTALL_OVERRIDE variable replaces it, as in hyperi-ci's own workflows.
# The Release job's CLI, replaced by HYPERCI_INSTALL_OVERRIDE
HYPERCI_INSTALL: ${{ vars.HYPERCI_INSTALL_OVERRIDE || 'uvx --python 3.14 --no-cache --refresh hyperi-ci' }}

jobs:
Expand Down Expand Up @@ -99,7 +97,7 @@ jobs:
pipx install --python "${{ steps.python.outputs.python-path }}" "pytest==$(make -s print-pytest-version)"
pipx inject pytest "pyyaml==$(make -s print-pyyaml-version)"

# The churn tile test reads the stack's ClickHouse pin through scripts/stack.py, which needs oras, and runs the tiles on that image
# The zero_ok tile test reads the ClickHouse pin through oras
- name: Install oras
uses: oras-project/setup-oras@5e2f2482b30b0cb3e14e67f89a80c88c659a7c63 # v2.0.3

Expand Down Expand Up @@ -146,8 +144,7 @@ jobs:
- name: Proxy configs
run: make check-proxy

# Only Docker on PATH: each profile is set up, and the default one starts, serves
# the console and runs the demo. The Windows job runs the default install.
# Only Docker on PATH: every profile set up, the default run
image:
name: Image
runs-on: ubuntu-26.04
Expand All @@ -165,11 +162,11 @@ jobs:
with:
python-version: ${{ env.PYTHON_VERSION }}

# The stack deployment.example.yaml pins, as image.yml bakes into the :<version> image
# The stack deployment.example.yaml pins, as image.yml bakes in
- name: Build the image
run: docker build --file docker/dfe-docker.Dockerfile --build-arg "IMAGE=$IMAGE" --build-arg "STACK=$(python3 scripts/quickstart_json.py --print-stack)" --tag "$IMAGE" .

# docker, curl for the console check and the few tools the checks below use. No make, no python
# docker, curl and a few check tools. No make, no python
- name: Put only Docker on PATH
run: |
bin="$RUNNER_TEMP/docker-only"
Expand Down Expand Up @@ -208,7 +205,7 @@ jobs:
docker compose config -q
docker compose config --services > "$WORK/default.services"

# A second init, a bad profile and an upgrade, each against what it must leave alone
# Re-init, a bad profile and an upgrade must leave state alone
- name: Refuse and upgrade
run: |
export PATH="$DOCKER_ONLY"
Expand Down Expand Up @@ -238,17 +235,17 @@ jobs:
exit 1
fi

# A throwaway install whose stack is torn down below, so its minted secrets protect nothing
# A throwaway install, so its minted secrets protect nothing
- name: Hand the default install to the Windows job
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
if-no-files-found: error
include-hidden-files: true
name: default-install
path: |
${{ env.WORK }}/default
${{ env.WORK }}/default.services
!${{ env.WORK }}/default/access-summary.md
include-hidden-files: true
if-no-files-found: error
retention-days: 1

- name: Start the default profile and reach the console
Expand All @@ -274,7 +271,7 @@ jobs:
run: |
docker compose ps --all
docker compose logs --no-color --tail 100
# Low runner disk stops Redpanda and ClickHouse taking writes, so a demo that lands nothing shows it here
# A full runner disk stops writes, so a rowless demo shows it
df -h "$(docker info --format '{{.DockerRootDir}}')" || true
docker compose exec -T kafka-redpanda rpk cluster health || true

Expand All @@ -286,7 +283,7 @@ jobs:
docker compose --profile '*' down --volumes --remove-orphans
fi

# One Published compose leg per demo pack: resources/demo, then each resources/demo-<pack> directory in byte order, as scripts/_common.py demo_pack_names lists them
# One Published leg per pack, in demo_pack_names order
demo-packs:
name: Demo Packs
runs-on: ubuntu-26.04
Expand All @@ -305,7 +302,7 @@ jobs:
dirs="$({ echo demo; find resources -mindepth 1 -maxdepth 1 -type d -name 'demo-*' -printf '%f\n' | LC_ALL=C sort; } | jq -Rnc '[inputs]')"
echo "dirs=$dirs" | tee -a "$GITHUB_OUTPUT"

# The files a release publishes as OCI artifacts, generated from this commit and run from a folder holding nothing else, as `-f oci://` runs them. Each demo file holds the whole stack plus its pack, so it starts and has to land the data its pack replays. The demo leg also writes the stack file, which is read and checked
# Release OCI files from this commit, run from an empty folder
published:
name: Published compose (${{ matrix.pack }})
needs: demo-packs
Expand Down Expand Up @@ -337,7 +334,7 @@ jobs:
- name: Build the image
run: docker build --file docker/dfe-docker.Dockerfile --build-arg "IMAGE=$IMAGE" --tag "$IMAGE" .

# The image is local here, so its digest is the one finding allowed
# The image is local, so its digest is the one finding allowed
- name: Generate and check the published files
env:
PACK: ${{ matrix.pack }}
Expand All @@ -350,14 +347,15 @@ jobs:
python3 scripts/publish_compose.py --image "$IMAGE" --out-dir "$RUNNER_TEMP/compose" "${pack[@]}"
echo "COMPOSE_DIR=$RUNNER_TEMP/compose" >> "$GITHUB_ENV"

# The demo file holds the whole stack plus the demo
- name: Start the demo and reach the console
timeout-minutes: 25
working-directory: ${{ env.COMPOSE_DIR }}
run: |
docker compose -f "$FILE" up -d --wait --wait-timeout 1200
curl -fsSL --retry 5 --retry-all-errors -o /dev/null -w 'console answered %{http_code} at %{url_effective}\n' http://localhost:3000/

# The secrets are this runner's own: no service holds one in its environment, ClickHouse refuses its empty default password, and the login the file prints is held for a change
# Secrets minted here: none in an env, no defaults accepted
- name: Run on secrets minted here
working-directory: ${{ env.COMPOSE_DIR }}
run: |
Expand Down Expand Up @@ -393,7 +391,7 @@ jobs:

# Runs after the secrets step because the console login check replaces the login's issued password, which that step expects held. One checker proves the pack worked: its replayed sources land parsed rows, the intake and default tables stay empty, every source is deployed, each hunt fires under its ceiling, each dashboard and saved search returns rows, the console login can search, replayed fetcher polls land and the dead-letter queue stays empty. What a pack expects is data in resources/<pack>/verify.json
- name: Verify the pack worked
# Above the checker's summed waits (land, hunts, console and fetchers: 38 min), so a slow failure still prints its results
# Over the checker's 38 min of summed waits, so failures print
timeout-minutes: 40
working-directory: ${{ env.COMPOSE_DIR }}
env:
Expand All @@ -413,7 +411,7 @@ jobs:
esac
done
docker compose -f "$FILE" logs --no-color --tail 100 "${services[@]}"
# A broker under its free-space floor takes no writes, so a pack that lands nothing shows it here
# A broker under its disk floor takes no writes, shown here
docker compose -f "$FILE" exec -T kafka-redpanda rpk cluster health || true
docker compose -f "$FILE" exec -T kafka-redpanda df -h /var/lib/redpanda/data || true
df -h "$(docker info --format '{{.DockerRootDir}}')" || true
Expand All @@ -425,7 +423,7 @@ jobs:
docker compose -f "$COMPOSE_DIR/$FILE" down --volumes --remove-orphans
fi

# Compose on Windows splits COMPOSE_FILE on `;` unless COMPOSE_PATH_SEPARATOR in .env says otherwise. The one runner exception: only a Windows runner runs that path.
# Windows splits COMPOSE_FILE unless .env sets a separator
image-windows:
name: Image (Windows compose)
needs: image
Expand Down Expand Up @@ -453,54 +451,56 @@ jobs:
}
Write-Output "Windows compose starts the same services: $got"

# Tags and publishes a release on `Release: true` or a `hyperi-ci release` dispatch,
# after the image jobs. The App's token publishes it so image.yml starts on it.
# Tags and publishes on `Release: true` or a dispatch
release:
name: Release
# An image that fails CI never gets a tag
needs: [compose, dockerfile, image, image-windows, published]
if: github.event_name != 'pull_request'
runs-on: ubuntu-26.04
timeout-minutes: 20
# semantic-release's dry run checks push access, which needs write
# A dry run checks push access, which needs write
permissions:
contents: write
# Shared actions: hyperi-ci's release takes four languages
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.tag || github.sha }}
# semantic-release reads every commit since the last tag
fetch-depth: 0
persist-credentials: false
ref: ${{ inputs.tag || github.sha }}

# The trailer gate, the classification check and the version, as every hyperi-ci pipeline decides them
# Trailer gate, classification and version, as hyperi-ci does
- name: Plan the release
id: plan
uses: hyperi-io/hyperi-ci/.github/actions/predict-version@main
with:
bump: ${{ inputs.bump || 'auto' }}
from-head: ${{ inputs.from-head }}
github-token: ${{ secrets.GITHUB_TOKEN }}
tag: ${{ inputs.tag }}
from-head: ${{ inputs.from-head }}
bump: ${{ inputs.bump || 'auto' }}

- name: Install uv
if: steps.plan.outputs.will-release == 'true'
uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
enable-cache: false

# A GITHUB_TOKEN release starts no image.yml run
- name: Mint the release token
id: bot
if: steps.plan.outputs.will-release == 'true'
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
with:
client-id: ${{ vars.GH_APP_CLIENT_ID }}
private-key: ${{ secrets.GH_APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: ${{ github.event.repository.name }}
permission-contents: write
private-key: ${{ secrets.GH_APP_PRIVATE_KEY }}
repositories: ${{ github.event.repository.name }}

# Plan installed semantic-release and the central config in this job
# Plan installed semantic-release and its config in this job
- name: Tag (semantic-release)
if: steps.plan.outputs.will-release == 'true' && (github.event_name == 'push' || inputs.from-head == 'true' && inputs.bump == 'auto')
env:
Expand All @@ -514,7 +514,7 @@ jobs:
RELEASE_VERSION: ${{ steps.plan.outputs.version }}
run: ${{ env.HYPERCI_INSTALL }} tag-head --bump "$RELEASE_VERSION"

# semantic-release tags nothing when this checkout is behind main, and gh release create would then tag main's tip
# Behind main nothing is tagged, so gh would tag main's tip
- name: Check the tag is on this commit
if: steps.plan.outputs.will-release == 'true'
env:
Expand Down Expand Up @@ -554,7 +554,7 @@ jobs:
fi
gh release create "$tag" --repo "$GITHUB_REPOSITORY" --verify-tag --title "$tag" --generate-notes "${kind[@]}"

# Non-fatal: the release is already published. A tag dispatch checks out an old tag, so it commits nothing back
# Non-fatal: the release is out, and an old tag commits nothing
- name: Commit VERSION and CHANGELOG.md back
if: steps.plan.outputs.will-release == 'true' && inputs.tag == ''
continue-on-error: true
Expand Down
Loading