feat: upgrade default PostgreSQL to 18

[zdrapela]

Summary

• update default PostgreSQL images from 15 to 18
• add PostgreSQL dump, volume replacement, and restore coverage to Operator upgrade E2E tests
• run the migration flow in nightly main upgrades and a dedicated pull-request check

This upgrade path was tested with the full E2E test suite in redhat-developer/rhdh#5145

Verification

• make lint
• make test OPERATOR_DP_PROCESSING=false
• focused restore validation tests
• actionlint for the new pull-request workflow

Jira: https://redhat.atlassian.net/browse/RHIDP-13959

[rhdh-qodo-merge]

PR Summary by Qodo

Upgrade default PostgreSQL to 18 with migration E2E coverage

✨ Enhancement 🧪 Tests ⚙️ Configuration changes 📝 Documentation 🕐 40+ Minutes

AI Description

• Makes PostgreSQL 18 the default local database across profiles, bundles, and restricted mirrors.
• Validates 15-to-18 upgrades through dump, volume replacement, restore, and data checks.
• Runs migration coverage in nightly main upgrades and affected pull requests.

Diagram

sequenceDiagram
  participant CI as Upgrade CI
  participant Suite as E2E Suite
  participant App as Backstage App
  participant Old as PostgreSQL 15
  participant Op as RHDH Operator
  participant Vol as DB Volume
  participant New as PostgreSQL 18
  CI->>Suite: Run flagged upgrade
  Suite->>Old: Seed test data
  Suite->>App: Stop workloads
  Suite->>Old: Dump databases
  Suite->>Op: Apply target manifest
  Suite->>Op: Stop reconciliation
  Suite->>Vol: Replace old volume
  Suite->>Op: Resume reconciliation
  Op->>New: Provision database
  Suite->>New: Restore dump
  Suite->>App: Restart workloads
  Suite->>New: Verify restored data

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Use pg_upgrade in place

• ➕ Avoids logical export and import for large databases
• ➕ Can reduce migration downtime
• ➖ Requires compatible PostgreSQL 15 and 18 binaries in the migration environment
• ➖ Adds filesystem and extension compatibility concerns
• ➖ Would require more operator-owned migration and rollback logic

2. Automate migration in the operator

• ➕ Provides a more seamless user upgrade experience
• ➕ Centralizes backup, replacement, restore, and status reporting
• ➖ Makes reconciliation destructive and significantly increases upgrade risk
• ➖ Requires durable checkpoints, rollback behavior, storage capacity checks, and failure recovery
• ➖ Expands operator responsibility beyond the documented manual migration procedure

Recommendation: Keep the PR's logical dump, fresh-volume provisioning, and restore validation. PostgreSQL major versions cannot safely share the existing data directory, and this approach accurately exercises the documented manual procedure without embedding destructive migration behavior into normal operator reconciliation. An automated operator migration should only be considered as a separately designed feature with explicit opt-in, rollback, and recovery semantics.

Files changed (22) +603 / -32

Tests (7) +434 / -11

db-statefulset_test.goAssert the PostgreSQL 18 model default +2/-2

Assert the PostgreSQL 18 model default

• Adds an explicit assertion for the new default database image. It also uses 't.Setenv' so the override test automatically restores environment state.

pkg/model/db-statefulset_test.go

db-statefulset.yamlUpdate default database model fixture +2/-2

Update default database model fixture

• Moves the default StatefulSet test fixture to PostgreSQL 18.

pkg/model/testdata/default-config/db-statefulset.yaml

ips-db-statefulset.yamlUpdate IPS database model fixture +1/-2

Update IPS database model fixture

• Changes the IPS database fixture's PostgreSQL image expectation from version 15 to 18.

pkg/model/testdata/ips-db-statefulset.yaml

rhdh-db-statefulset.yamlUpdate RHDH database model fixture +2/-2

Update RHDH database model fixture

• Moves the RHDH StatefulSet test fixture to PostgreSQL 18.

pkg/model/testdata/rhdh-db-statefulset.yaml

e2e_upgrade_test.goIntegrate database migration into operator upgrades +40/-3

Integrate database migration into operator upgrades

• Conditionally prepares and completes the PostgreSQL migration around the operator upgrade. It also enforces serial execution, waits for namespace deletion and operator rollout, and verifies the deployed operator image.

tests/e2e/e2e_upgrade_test.go

postgresql_upgrade_restore_test.goTest PostgreSQL restore error validation +51/-0

Test PostgreSQL restore error validation

• Covers acceptance of the expected bootstrap role conflict and rejection of additional SQL, fatal, connection, or missing-conflict outcomes.

tests/e2e/postgresql_upgrade_restore_test.go

postgresql_upgrade_test.goImplement PostgreSQL 15-to-18 migration validation +336/-0

Implement PostgreSQL 15-to-18 migration validation

• Adds helpers that seed PostgreSQL 15, dump all roles and databases, stop workloads, replace the StatefulSet and PVC, restore into PostgreSQL 18, refresh collations, and verify preserved data. The test also confirms expected resource replacement, Secret preservation, image versions, and Backstage recovery.

tests/e2e/postgresql_upgrade_test.go

Documentation (1) +19 / -0

README.mdDocument the PostgreSQL migration E2E scenario +19/-0

Document the PostgreSQL migration E2E scenario

• Documents the opt-in upgrade flag, PostgreSQL dump-and-restore lifecycle, example command, required manifests, and destructive disposable-cluster warning.

tests/e2e/README.md

Other (14) +150 / -21

nightly-upgrade-test.yamlEnable PostgreSQL migration coverage for main upgrades +1/-0

Enable PostgreSQL migration coverage for main upgrades

• Enables the PostgreSQL upgrade scenario when the nightly matrix upgrades to the main branch. Other upgrade destinations retain the existing test behavior.

.github/workflows/nightly-upgrade-test.yaml

pr-postgresql-upgrade-test.yamlAdd dedicated PostgreSQL upgrade pull-request workflow +119/-0

Add dedicated PostgreSQL upgrade pull-request workflow

• Creates a Kind-based pull-request job that builds the candidate operator, installs the RHDH 1.10 source version, and runs the PostgreSQL 15-to-18 migration test. Path filtering, concurrency cancellation, disk cleanup, and pinned actions constrain execution and resource usage.

.github/workflows/pr-postgresql-upgrade-test.yaml

prepare-restricted-environment.shMirror the PostgreSQL 18 restricted-environment image +2/-2

Mirror the PostgreSQL 18 restricted-environment image

• Replaces the RHEL 9 PostgreSQL 15 mirror mapping with the RHEL 10 PostgreSQL 18 image mapping for restricted environments.

.rhdh/scripts/prepare-restricted-environment.sh

backstage-default-config_v1_configmap.yamlSet PostgreSQL 18 in the Backstage bundle defaults +2/-2

Set PostgreSQL 18 in the Backstage bundle defaults

• Updates the embedded database StatefulSet to use PostgreSQL 18. The YAML block scalar is also normalized to preserve the generated content format.

bundle/backstage.io/manifests/backstage-default-config_v1_configmap.yaml

backstage-operator.clusterserviceversion.yamlPublish PostgreSQL 18 as the related database image +2/-2

Publish PostgreSQL 18 as the related database image

• Updates both the operator environment variable and CSV related-images declaration from PostgreSQL 15 to PostgreSQL 18.

bundle/rhdh/manifests/backstage-operator.clusterserviceversion.yaml

rhdh-default-config_v1_configmap.yamlSet PostgreSQL 18 in the RHDH bundle defaults +2/-2

Set PostgreSQL 18 in the RHDH bundle defaults

• Updates the embedded RHDH database StatefulSet image to PostgreSQL 18 and normalizes its YAML block scalar.

bundle/rhdh/manifests/rhdh-default-config_v1_configmap.yaml

rhdh-plugin-deps_v1_configmap.yamlUpgrade the plugin dependency psql image +1/-1

Upgrade the plugin dependency psql image

• Moves the SonataFlow plugin dependency container from PostgreSQL 15 to PostgreSQL 18.

bundle/rhdh/manifests/rhdh-plugin-deps_v1_configmap.yaml

db-statefulset.yamlUpgrade the Backstage profile database image +2/-2

Upgrade the Backstage profile database image

• Changes the default local database image for the Backstage profile to PostgreSQL 18.

config/profile/backstage.io/default-config/db-statefulset.yaml

db-statefulset.yamlUpgrade the RHDH profile database image +2/-2

Upgrade the RHDH profile database image

• Changes the default local database image for the RHDH profile to PostgreSQL 18.

config/profile/rhdh/default-config/db-statefulset.yaml

deployment-patch.yamlInject PostgreSQL 18 as the related image +1/-1

Inject PostgreSQL 18 as the related image

• Updates the RHDH operator deployment patch so reconciled local databases resolve to PostgreSQL 18.

config/profile/rhdh/patches/deployment-patch.yaml

sonataflow.yamlUpgrade the SonataFlow psql container +1/-1

Upgrade the SonataFlow psql container

• Uses PostgreSQL 18 for the psql utility container in the SonataFlow plugin dependency configuration.

config/profile/rhdh/plugin-deps/sonataflow.yaml

install.yamlRegenerate the Backstage installation manifest +2/-2

Regenerate the Backstage installation manifest

• Propagates the PostgreSQL 18 database default and normalized embedded YAML formatting into the generated Backstage installation manifest.

dist/backstage.io/install.yaml

install.yamlRegenerate the RHDH installation manifest +4/-4

Regenerate the RHDH installation manifest

• Propagates PostgreSQL 18 into the default database, plugin dependency, and operator related-image settings in the generated RHDH installer.

dist/rhdh/install.yaml

e2e_suite_test.goRetry manifest installation after CRD establishment +9/-0

Retry manifest installation after CRD establishment

• Handles initial manifest-application ordering failures by waiting for all CRDs to become established and retrying the apply operation.

tests/e2e/e2e_suite_test.go

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Existing local databases fail to start 🐞 Bug ≡ Correctness

Description

The generated StatefulSet’s default image changes from PostgreSQL 15 to 18, while reconciliation
retains the existing data PVC and provides no major-version migration path. When an Operator
managing an existing PostgreSQL 15 local database is upgraded, PostgreSQL 18 starts against the
PostgreSQL 15 data directory, disrupting database startup and Backstage availability until an
administrator manually dumps, replaces the volume, and restores the database.

Code

config/profile/rhdh/default-config/db-statefulset.yaml[36]

+          image: quay.io/fedora/postgresql-18:latest # will be replaced with the actual image

Relevance

●●● Strong

Accepted upgrade-path precedent requires explicitly addressing incompatible existing resources;
documented manual migration does not prevent reconciliation failure.

PR-#2036

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The profile manifest selects PostgreSQL 18 while continuing to mount the persistent data claim at
PostgreSQL’s data path, and the controller force-applies the generated StatefulSet during
reconciliation. The model only updates and validates that StatefulSet without migration handling,
while the added E2E procedure demonstrates that a 15-to-18 migration requires dumping the old
database, deleting the StatefulSet and PVC, provisioning PostgreSQL 18, and restoring the dump—none
of which ordinary reconciliation performs.

config/profile/rhdh/default-config/db-statefulset.yaml[33-36]
config/profile/rhdh/default-config/db-statefulset.yaml[91-111]
pkg/model/db-statefulset.go[42-67]
internal/controller/backstage_controller.go[143-176]
tests/e2e/postgresql_upgrade_test.go[53-60]
tests/e2e/postgresql_upgrade_test.go[83-109]
config/profile/rhdh/default-config/db-statefulset.yaml[84-111]
pkg/model/db-statefulset.go[76-101]
tests/e2e/postgresql_upgrade_test.go[53-59]
tests/e2e/postgresql_upgrade_test.go[83-106]
tests/e2e/README.md[113-129]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Changing the default local PostgreSQL image from 15 to 18 causes existing generated StatefulSets to roll onto PostgreSQL 18 while retaining their PostgreSQL 15 PVC, even though the tested migration procedure requires a logical dump, volume replacement, and restore. Add an explicit safe migration path or prevent this unsupported in-place major upgrade, while keeping PostgreSQL 18 as the default for newly created instances and documenting the administrator procedure.

## Fix Focus Areas
- config/profile/rhdh/default-config/db-statefulset.yaml[33-36]
- pkg/model/db-statefulset.go[42-67]
- pkg/model/db-statefulset.go[76-101]
- internal/controller/backstage_controller.go[143-176]
- tests/e2e/postgresql_upgrade_test.go[83-106]

## Recommended Fix
Before applying a major-version image change to an existing local PostgreSQL StatefulSet, detect its existing database version or image and prevent reconciliation from applying PostgreSQL 18 while it remains backed by a PostgreSQL 15 volume. Either retain PostgreSQL 15 for existing instances until an explicit migration action confirms that the volume has been replaced, or implement an opt-in operator-managed flow that exports the PostgreSQL 15 data, deletes and replaces the StatefulSet and PVC, provisions PostgreSQL 18 with a fresh volume, restores the dump, and refreshes collations. Document this supported migration procedure in the product documentation, and continue using PostgreSQL 18 by default only for newly created instances.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 18 rules

✅ Cross-repo context — repo relationships

  Explored: repo: redhat-developer/rhdh (sha: b8140abe)

Review mode: 🧠 Deep: This PR combines broad PostgreSQL image/config changes with substantial new CI and multi-stage E2E migration logic across many independent paths, creating a high density of subtle, hard-to-detect defects.

Tip of the day

💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[rhdh-qodo-merge]

1. Existing local databases fail to start 🐞 Bug ≡ Correctness

The generated StatefulSet’s default image changes from PostgreSQL 15 to 18, while reconciliation
retains the existing data PVC and provides no major-version migration path. When an Operator
managing an existing PostgreSQL 15 local database is upgraded, PostgreSQL 18 starts against the
PostgreSQL 15 data directory, disrupting database startup and Backstage availability until an
administrator manually dumps, replaces the volume, and restores the database.

Agent Prompt

## Issue description
Changing the default local PostgreSQL image from 15 to 18 causes existing generated StatefulSets to roll onto PostgreSQL 18 while retaining their PostgreSQL 15 PVC, even though the tested migration procedure requires a logical dump, volume replacement, and restore. Add an explicit safe migration path or prevent this unsupported in-place major upgrade, while keeping PostgreSQL 18 as the default for newly created instances and documenting the administrator procedure.

## Fix Focus Areas
- config/profile/rhdh/default-config/db-statefulset.yaml[33-36]
- pkg/model/db-statefulset.go[42-67]
- pkg/model/db-statefulset.go[76-101]
- internal/controller/backstage_controller.go[143-176]
- tests/e2e/postgresql_upgrade_test.go[83-106]

## Recommended Fix
Before applying a major-version image change to an existing local PostgreSQL StatefulSet, detect its existing database version or image and prevent reconciliation from applying PostgreSQL 18 while it remains backed by a PostgreSQL 15 volume. Either retain PostgreSQL 15 for existing instances until an explicit migration action confirms that the volume has been replaced, or implement an opt-in operator-managed flow that exports the PostgreSQL 15 data, deletes and replaces the StatefulSet and PVC, provisions PostgreSQL 18 with a fresh volume, restores the dump, and refreshes collations. Document this supported migration procedure in the product documentation, and continue using PostgreSQL 18 by default only for newly created instances.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[rhdh-qodo-merge]

Important

The /generate_labels command by Qodo is sunsetting on the 1st of October 2026 and will no longer be available. We recommend switching to the latest Qodo review capabilities. Learn more

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rm3l]

Do we really need this unit test testing a helper defined in the E2E suite? captureOperatorLogsBeforeCleanup is a 3 lines function..

[rm3l]

Same for this btw. This unit test file doesn't seem necessary IMO.

[rm3l]

Do we really need this on every PR? Maybe just the nightly upgrade workflow above is enough? We could maybe just limit it for now to the release-1.10 => main upgrade path today (and later include 1.10 => release-2.1 too)?