Switch to using goose for migrations

[iplay88keys]

Warning

This is an intentional clean slate for 1.0:

• Existing 0.10.x databases require a new PostgreSQL database; there is no migration bridge.
• Downgrade from 1.0 to 0.10.x is unsupported.
• Starting with 1.0, migrations are append-only and future minor releases retain compatibility with the previous release line.

Summary

Replace golang-migrate with Goose and establish a clean PostgreSQL migration baseline for Kagent 1.0.

• Capture the current core and optional vector schemas as direct version 1 Goose baselines.
• Make each schema change and its migration record commit atomically.
• Simplify the database migration CLI around Goose.
• Reject pre-1.0 database layouts instead of attempting an in-place bridge.
• Preserve the Helm controls for vector migrations and out-of-band migration management in controller-v2.
• Prepare normal and rolling upgrade CI to enforce compatibility automatically after 1.0 is published.

Why

golang-migrate records a dirty version separately from the migration transaction.
A failure between the schema commit and the final version update can leave a committed schema marked dirty, requiring an operator to inspect the database and force a version.

Goose records the schema change and migration version in the same PostgreSQL transaction.
A failed migration rolls both back, and startup can safely retry it.

Substrate PR #1196 made the same change; this review thread describes the failure mode.

What changed

Goose runner and migration files

• Replace github.com/golang-migrate/migrate with github.com/pressly/goose/v3.
• Replace the split core migration history with a direct final-schema definition in core/000001_initial.sql.
• Replace the split vector migration history with a direct final-schema definition in vector/000001_initial.sql.
• Require both -- +goose Up and -- +goose Down sections for Kagent migrations.
• Reject -- +goose NO TRANSACTION so schema changes and migration records remain atomic.
• Validate all sources and run all prechecks before applying the first source.
• Reject legacy golang-migrate ledgers with a clear fresh-database error.
• Allow non-destructive startup when the database is ahead of the binary, preserving rolling compatibility.

Each source still commits independently.
If a later source fails, previously completed sources remain committed and the next startup resumes from their recorded versions.

Migration immutability CI permits deletion of the legacy split files for this cutover, then protects merged Goose migrations from modification, rename, or deletion.

Database CLI

The migration CLI now operates on Goose providers:

• up applies all pending sources.
• down N and goto V move one selected source within its embedded Goose sequence.
• status and version report Goose state.
• force and the dirty-state output are removed.

Destructive commands reject a database version newer than the CLI's embedded migrations.
The CLI continues to resolve vector-source enablement from its environment, the controller ConfigMap, or its standalone fallback.

Controller startup and Helm settings

Controller startup now honors the existing Helm settings:

• DATABASE_VECTOR_ENABLED=true includes the vector migration source and registers pgvector types on database connections.
• SKIP_MIGRATIONS=true performs read-only migration verification instead of applying migrations.

This matches the 0.10.x behavior of database.postgres.vectorEnabled and database.postgres.skipMigrations.

Upgrade tests and CI

The upgrade suites now understand Goose state and use the API v2 AgentInstance gRPC interaction test.
They cover data survival, previous-release behavior after target migrations, clean-install schema equivalence, application rollback, schema rollback, and the old-controller/new-schema window during a rolling deployment.

Previous-release tests run from a temporary worktree at the matching tag.
The previous chart installation also reads its bundled-PostgreSQL overrides from that tag's Makefile so the baseline matches the release as shipped.

Both normal and rolling jobs include these matrix legs:

• adjacent tests patch and prerelease upgrades on release branches.
• prev-stable tests the previous release line for minor-version compatibility.

The shared setup skips 0.10.x because it predates Goose.
There will be no 0.11.x release.
Once a 1.0 prerelease or final artifact is published, prev-stable automatically begins testing 1.0 against the next development line.

Verification

• Applied the migrations from current origin/main and the new Goose baselines to separate fresh databases; their schema-only pg_dump output matched exactly after excluding the intentionally different migration-ledger tables.
• go test ./core/pkg/migrations ./core/pkg/cli/db/migrate ./core/test/upgrade ./core/internal/database
• go vet ./core/pkg/migrations ./core/pkg/cli/db/migrate ./core/test/upgrade ./core/internal/database
• make lint
• make sqlc-generate
• go test -short ./core/pkg/migrations ./core/pkg/cli/db/migrate ./core/cli/internal/commands/db
• go test -race ./core/pkg/migrations ./core/pkg/cli/db/migrate
• go test ./core/pkg/app ./core/cmd/controller-v2 ./core/pkg/migrations ./core/internal/database
• bash scripts/version-resolution_test.sh
• git diff --check

[EItanya]

Can we define these env vars as consts somewhere so they're properly documented, we used to have a env var package somewhere

[EItanya]

Why this change, you're explicitly running it twice?