Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ on:

permissions:
contents: write
# packages lets the job push the release images to ghcr.io with GITHUB_TOKEN.
packages: write
# id-token lets cosign mint a short-lived Fulcio certificate from the
# workflow's OIDC identity for keyless signing; attestations lets the build
# publish SLSA provenance for the released artifacts.
Expand All @@ -22,15 +24,24 @@ jobs:
- uses: actions/setup-go@v6
with:
go-version-file: go.mod
# QEMU and buildx build the arm64 image on the amd64 runner.
- uses: docker/setup-qemu-action@v4
- uses: docker/setup-buildx-action@v4
- name: Log in to ghcr.io
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: anchore/sbom-action/download-syft@v0
- uses: sigstore/cosign-installer@v3
- uses: goreleaser/goreleaser-action@v7
with:
version: "~> v2"
# Container images are deferred (ghcr publishing not set up yet), so
# docker is skipped; the checksums are cosign-signed keyless via the
# workflow OIDC identity.
args: release --clean --skip=docker
# Builds and pushes the amd64 and arm64 images and their manifest to
# ghcr.io. The checksums and the image manifests are cosign-signed
# keyless via the workflow OIDC identity.
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Attest build provenance
Expand Down
2 changes: 2 additions & 0 deletions .goreleaser.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ dockers:
goarch: arm64
build_flag_templates:
- "--platform=linux/arm64"
- "--label=org.opencontainers.image.source=https://github.com/dcadolph/fleetsweeper"
- "--label=org.opencontainers.image.licenses=MIT"

docker_manifests:
- name_template: "ghcr.io/dcadolph/fleetsweeper:{{ .Version }}"
Expand Down
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,14 +149,20 @@ part. The outlier detection under it is the real part.
## Install for real

```
helm install fleetsweeper deploy/helm/fleetsweeper \
kubectl create namespace fleetsweeper
kubectl --namespace fleetsweeper create secret generic fleet-kubeconfig \
--from-file=config=$HOME/.kube/config
helm install fleetsweeper deploy/helm/fleetsweeper --namespace fleetsweeper \
--set auth.token=$(openssl rand -hex 32) \
--set kubeconfig.existingSecret=fleet-kubeconfig \
--set controller.enabled=true
kubectl apply -f deploy/examples/clusterscan-prod.yaml
```

The controller reconciles `ClusterScan` resources and writes outcomes back
to `.status`. Full installation paths in
The chart pulls the signed `ghcr.io/dcadolph/fleetsweeper` image and scans
every context in the kubeconfig Secret. Without the Secret it scans only the
cluster it runs in. The controller reconciles `ClusterScan` resources and
writes outcomes back to `.status`. Full installation paths in
[`docs/operator/helm.md`](docs/operator/helm.md). Scoped API keys for
pipelines in [`docs/operator/rbac.md`](docs/operator/rbac.md).

Expand Down
24 changes: 24 additions & 0 deletions UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,30 @@ stability contract and the GitHub release notes for the full feature list.
5. Verify `kubectl logs deploy/fleetsweeper | grep migrate` shows no errors
and `/readyz` returns 200.

## v0.15.0

No schema migration.

- **Reads need a credential.** A server that holds any credential (a bootstrap
`--auth-token`, single sign-on, or at least one API key) now answers 401 to
an `/api` read that carries neither a bearer token nor a session cookie.
Before, such a read was served as an anonymous viewer. Give every API client
and scraper of `/api` a viewer key. `/healthz`, `/readyz`, `/openapi.yaml`,
`/oidc/*`, signed `/api/webhooks/*` calls, and the dashboard's static files
stay open. The dashboard asks for a token on first load. `--insecure`,
`--demo`, and a server with no credential at all keep anonymous reads.

- **The container image is published.** Each release pushes
`ghcr.io/dcadolph/fleetsweeper:<version>` and `:latest` for amd64 and
arm64, signed keyless with cosign. The chart's default image now pulls.

- **The chart can scan a fleet.** Set `kubeconfig.existingSecret` to a Secret
holding a kubeconfig and the server scans every context in it. Without one,
scheduled scans see the host cluster through the ServiceAccount, reported as
the context `in-cluster`. Before, a pod with no kubeconfig found no contexts
and scheduled scans did nothing. The ClusterScan controller stays on the
host cluster unless `--controller-context` names a context.

## v0.14.0

No schema migration. Reports, scores, and a few API shapes change on the
Expand Down
14 changes: 12 additions & 2 deletions cmd/cmd_serve.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ func init() {
serveCmd.Flags().String("seal-key", "", "HMAC-SHA256 key used to sign saved scan reports. Empty disables sealing.")
serveCmd.Flags().Bool("controller", false, "Enable the ClusterScan controller: watch ClusterScan CRs in the home cluster and reconcile them by triggering scans on their declared interval.")
serveCmd.Flags().String("controller-namespace", "", "Namespace the controller watches. Empty watches all namespaces (requires cluster-wide RBAC).")
serveCmd.Flags().String("controller-context", "", "Kubeconfig context the controller uses to reach the home cluster. Empty uses the default context or in-cluster config.")
serveCmd.Flags().String("controller-context", "", "Kubeconfig context the controller uses to reach the home cluster. Empty uses the in-cluster config inside a pod, else the kubeconfig's current context.")
serveCmd.Flags().Duration("controller-poll", 15*time.Second, "How often the controller re-evaluates ClusterScan resources.")
serveCmd.Flags().Bool("leader-election", true, "Enable leader election for the scheduler and controller. Honored only when running inside Kubernetes. Multiple replicas without leader election will double-fire scheduled work.")
serveCmd.Flags().String("leader-namespace", "", "Namespace for the leader-election Lease object. Defaults to $POD_NAMESPACE.")
Expand Down Expand Up @@ -218,7 +218,7 @@ func runServe(cmd *cobra.Command, _ []string) error {
ctrlNS, _ := cmd.Flags().GetString("controller-namespace")
ctrlContext, _ := cmd.Flags().GetString("controller-context")
ctrlPoll, _ := cmd.Flags().GetDuration("controller-poll")
dyn, events, err := controller.Clients(kubeconfigPath, ctrlContext)
dyn, events, err := controller.Clients(controllerKubeconfig(kubeconfigPath, ctrlContext, kube.InCluster()), ctrlContext)
if err != nil {
log.Warn("controller: client setup failed", zap.Error(err))
return
Expand Down Expand Up @@ -391,6 +391,16 @@ func resolveScheduleContexts(kubeconfigPath string, explicit []string, all bool)
return explicit, nil
}

// controllerKubeconfig returns the kubeconfig path the controller reaches its home cluster with. A
// pod with no --controller-context uses its ServiceAccount, so a mounted fleet kubeconfig never
// moves the controller off the cluster that holds its ClusterScan objects.
func controllerKubeconfig(kubeconfigPath, controllerContext string, inCluster bool) string {
if controllerContext == "" && inCluster {
return ""
}
return kubeconfigPath
}

// openServeStore opens whichever backend --db / --db-driver point at. --demo
// runs on an in-memory SQLite database the server seeds with the demo fleet,
// so it refuses a --db rather than write the demo into a real database.
Expand Down
29 changes: 29 additions & 0 deletions cmd/cmd_serve_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -245,3 +245,32 @@ func TestLeaderElectionPlan(t *testing.T) {
})
}
}

// TestControllerKubeconfig verifies the controller stays on the pod's own cluster when a fleet
// kubeconfig is mounted, and follows the kubeconfig when a context is named or it runs outside a
// pod.
func TestControllerKubeconfig(t *testing.T) {
t.Parallel()
tests := []struct {
Path string
Context string
InCluster bool
WantResult string
}{{ // Test 0: Pod with a fleet kubeconfig and no context uses its ServiceAccount.
Path: "/var/run/fleetsweeper/kubeconfig", InCluster: true, WantResult: "",
}, { // Test 1: Pod with a named context reads the kubeconfig.
Path: "/var/run/fleetsweeper/kubeconfig", Context: "home", InCluster: true,
WantResult: "/var/run/fleetsweeper/kubeconfig",
}, { // Test 2: Outside a pod the kubeconfig is used.
Path: "/home/u/.kube/config", WantResult: "/home/u/.kube/config",
}}
for testNum, test := range tests {
t.Run(fmt.Sprintf("test %d", testNum), func(t *testing.T) {
t.Parallel()
got := controllerKubeconfig(test.Path, test.Context, test.InCluster)
if diff := cmp.Diff(test.WantResult, got); diff != "" {
t.Errorf("mismatch (-want +got):\n%s", diff)
}
})
}
}
4 changes: 2 additions & 2 deletions deploy/helm/fleetsweeper/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ apiVersion: v2
name: fleetsweeper
description: Multi-cluster Kubernetes fleet comparison and audit
type: application
version: 0.14.0
appVersion: "0.14.0"
version: 0.15.0
appVersion: "0.15.0"
home: https://github.com/dcadolph/fleetsweeper
sources:
- https://github.com/dcadolph/fleetsweeper
Expand Down
12 changes: 12 additions & 0 deletions deploy/helm/fleetsweeper/templates/NOTES.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,18 @@ Fleetsweeper {{ .Chart.AppVersion }} is installed.
kubectl --namespace {{ .Release.Namespace }} port-forward svc/{{ .Release.Name }} 8080:{{ .Values.service.port }}

Then open http://localhost:8080.
{{- if .Values.kubeconfig.existingSecret }}

Scans read every context in the kubeconfig stored in Secret
{{ .Values.kubeconfig.existingSecret }}, key {{ .Values.kubeconfig.key }}.
{{- else }}

Scans see only this cluster, through the release's ServiceAccount. To scan
a fleet, store a kubeconfig in a Secret and set kubeconfig.existingSecret:

kubectl --namespace {{ .Release.Namespace }} create secret generic fleet-kubeconfig --from-file=config=$HOME/.kube/config
helm upgrade {{ .Release.Name }} <chart> --reuse-values --set kubeconfig.existingSecret=fleet-kubeconfig
{{- end }}

{{- if .Values.auth.token }}

Expand Down
16 changes: 16 additions & 0 deletions deploy/helm/fleetsweeper/templates/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ spec:
{{- if .Values.scheduler.allContexts }}
- --all-contexts
{{- end }}
{{- if .Values.kubeconfig.existingSecret }}
- --kubeconfig=/var/run/fleetsweeper/kubeconfig
{{- end }}
{{- range .Values.cors.origins }}
- --cors-origin={{ . }}
{{- end }}
Expand Down Expand Up @@ -163,6 +166,11 @@ spec:
mountPath: /data
- name: tmp
mountPath: /tmp
{{- if .Values.kubeconfig.existingSecret }}
- name: kubeconfig
mountPath: /var/run/fleetsweeper
readOnly: true
{{- end }}
volumes:
- name: data
{{- if .Values.persistence.enabled }}
Expand All @@ -173,3 +181,11 @@ spec:
{{- end }}
- name: tmp
emptyDir: {}
{{- if .Values.kubeconfig.existingSecret }}
- name: kubeconfig
secret:
secretName: {{ .Values.kubeconfig.existingSecret }}
items:
- key: {{ .Values.kubeconfig.key }}
path: kubeconfig
{{- end }}
9 changes: 9 additions & 0 deletions deploy/helm/fleetsweeper/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,15 @@ scheduler:
allContexts: true
contexts: []

# Kubeconfig the server scans with. Without one the server sees only the cluster it runs in,
# through its ServiceAccount. To scan a fleet, store a kubeconfig in a Secret and name it here:
# kubectl create secret generic fleet-kubeconfig --from-file=config=$HOME/.kube/config
kubeconfig:
# Name of an existing Secret holding the kubeconfig. Empty scans the host cluster only.
existingSecret: ""
# Key in that Secret whose value is the kubeconfig file.
key: config

persistence:
enabled: true
size: 5Gi
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,6 @@ See [outliers](concepts/outliers.md) for the full statistical treatment.
behind norm-based detection.
- [Server mode](operator/server-mode.md). Run as a service with dashboard and API.
- [ClusterScan CRD](operator/clusterscan.md). Full spec reference.
- [RBAC and API keys](operator/rbac.md). Multi-tenant access control.
- [RBAC and API keys](operator/rbac.md). Role and cluster-scoped API keys for writes.
- Go SDK. A standard-library client for the HTTP API lives in the `client`
package, with contract tests that run against a live server.
12 changes: 8 additions & 4 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,20 +36,24 @@ Run both together.
## Three-line install

```shell
kubectl apply -f https://raw.githubusercontent.com/dcadolph/fleetsweeper/main/deploy/crds/clusterscan.yaml
helm install fleetsweeper deploy/helm/fleetsweeper \
kubectl --namespace fleetsweeper create secret generic fleet-kubeconfig --from-file=config=$HOME/.kube/config
helm install fleetsweeper deploy/helm/fleetsweeper --namespace fleetsweeper \
--set auth.token="$(openssl rand -hex 32)" \
--set kubeconfig.existingSecret=fleet-kubeconfig \
--set controller.enabled=true
kubectl apply -f deploy/examples/clusterscan-prod.yaml
```

Then visit the dashboard at port 8080 of the fleetsweeper service.
Create the `fleetsweeper` namespace first. The chart installs the CRDs and
pulls the signed `ghcr.io/dcadolph/fleetsweeper` image. Without the
kubeconfig Secret it scans only the cluster it runs in. Then visit the
dashboard at port 8080 of the fleetsweeper service and paste the token.

## Where to go next

- [Getting Started](getting-started.md). First scan in five minutes.
- [Operator overview](operator/overview.md). Declarative scans with
ClusterScan CRs and a reconciler.
- [RBAC and API keys](operator/rbac.md). Multi-tenant access control.
- [RBAC and API keys](operator/rbac.md). Role and cluster-scoped API keys for writes.
- [The fleet is the policy](concepts/fleet-is-policy.md). Why norm-based
detection beats rule authoring at scale.
27 changes: 26 additions & 1 deletion docs/operator/helm.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ helm install fleetsweeper deploy/helm/fleetsweeper \
| `cors.origins` | `[]` | CORS allowlist. Empty refuses cross-origin requests. |
| `scheduler.interval` | `""` | Auto-scan interval (e.g. `30m`). Empty disables. |
| `scheduler.allContexts` | `true` | Use every kubeconfig context for scheduled scans. |
| `kubeconfig.existingSecret` | `""` | Secret holding the kubeconfig to scan with. Empty scans the host cluster only. |
| `kubeconfig.key` | `config` | Key in that Secret whose value is the kubeconfig file. |
| `persistence.enabled` | `true` | PVC for `/data/fleet.db`. |
| `persistence.size` | `5Gi` | Disk size. |
| `rbac.create` | `true` | Install least-privilege ClusterRole + binding. |
Expand All @@ -27,10 +29,33 @@ helm install fleetsweeper deploy/helm/fleetsweeper \
| `controller.namespace` | `""` | Restrict the controller to one namespace. |
| `controller.pollInterval` | `15s` | How often the reconciler re-evaluates. |

## Scanning a fleet

Without a kubeconfig the server sees one cluster: the one it runs in, reached
through the release's ServiceAccount and named `in-cluster` in reports. To
scan a fleet, store a kubeconfig in a Secret and point the chart at it:

```shell
kubectl --namespace fleetsweeper create secret generic fleet-kubeconfig \
--from-file=config=$HOME/.kube/config
helm upgrade fleetsweeper deploy/helm/fleetsweeper --namespace fleetsweeper \
--reuse-values --set kubeconfig.existingSecret=fleet-kubeconfig
```

The Secret is mounted read-only at `/var/run/fleetsweeper/kubeconfig` and
passed as `--kubeconfig`. Scheduled and triggered scans then read every context
in it. Each context needs read access on its own cluster, the same access the
chart's ClusterRole grants on the host. A kubeconfig whose users run an exec
plugin (`aws`, `gcloud`, `kubelogin`) needs that binary in the image, which the
distroless image does not carry, so use static tokens or client certificates
for in-cluster scans. The ClusterScan controller keeps using the
ServiceAccount for its own cluster unless `--controller-context` names a
context.

## CRD lifecycle

`crds.install` defaults to `true` so an out-of-the-box `helm install` is
self-contained. In multi-tenant or GitOps environments where CRDs are
self-contained. In shared or GitOps clusters where CRDs are
managed separately, set it to `false` and apply the CRDs from
`deploy/crds/` independently.

Expand Down
7 changes: 5 additions & 2 deletions docs/operator/rbac.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ Fleetsweeper has two authentication paths, designed to coexist:
| `operator` | Trigger scans, acknowledge findings, and manage tags and locations within scope. Cannot manage groups or keys. |
| `viewer` | Read-only. Cannot reach `/admin/*`. |

Read-only endpoints (GET) are open to viewers and above (or to anyone when
`--insecure` is set). Mutating endpoints require `operator` or `admin`.
Read-only endpoints (GET) need a viewer key or above, the bootstrap token, or
a single sign-on session. A request with no credential gets 401. Only
`--insecure`, `--demo`, or a server with no token, no single sign-on, and no
keys at all serves reads to anyone. Mutating endpoints require `operator` or
`admin`.

## Cluster scope

Expand Down
2 changes: 1 addition & 1 deletion docs/operator/server-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ By default, mutating endpoints (`POST /api/scans`, `POST /api/groups`,
`DELETE /api/groups/...`) return 403 because `serve` will scan whatever
kubeconfig the process holds. Two ways to allow mutations:

1. **Recommended**: set `--auth-token` to a long random string. Clients must send `Authorization: Bearer <token>` on mutating requests. Comparison is constant-time.
1. **Recommended**: set `--auth-token` to a long random string. Clients must then send `Authorization: Bearer <token>` on every API request, reads included. Comparison is constant-time. The dashboard asks for the token on first load.
2. **Explicit opt-out**: pass `--insecure` to disable authentication entirely. A loud warning is logged at startup.

`--demo` turns authentication off on its own. The demo serves a synthetic fleet from an
Expand Down
4 changes: 2 additions & 2 deletions docs/production-readiness.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ What Fleetsweeper ships ready for the box.
| ---------------------- | -------------- |
| HA backend | SQLite default, PostgreSQL via `--db-driver=postgres` |
| Multi-replica safe | Kubernetes Lease-based leader election (`coordination.k8s.io/v1`) |
| Multi-tenant RBAC | Scoped API keys (`admin`, `operator`, `viewer`) with cluster scope (`*`, names, or `group:<n>`) on writes. Reads are not scoped |
| Access control | Every API read and write needs a credential. Role and cluster-scoped API keys (`admin`, `operator`, `viewer`; `*`, names, or `group:<n>`) for writes. Reads are not scoped by cluster |
| Audit log | Every mutating request captured. Queryable at `GET /api/admin/audit`. Retention via `--audit-retention` |
| Declarative ops | `ClusterScan` CRD with an in-process reconciler |
| GitOps integrations | `FleetDriftReport` CR plus `PolicyReport` (wgpolicyk8s.io) |
Expand All @@ -27,4 +27,4 @@ What Fleetsweeper ships ready for the box.
- [Backends](operator/backends.md). SQLite vs Postgres trade-offs.
- [Leader election](operator/leader-election.md). Multi-replica safety details.
- [Audit log](operator/audit.md). What is captured and how to query it.
- [RBAC and API keys](operator/rbac.md). Scoped multi-tenant access.
- [RBAC and API keys](operator/rbac.md). Role and cluster-scoped API keys for writes.
Loading
Loading