Support CPD 5.3.1 - Helm-based install for WML, WSL, Spark, Cognos Analytics and SPSS - #555
Support CPD 5.3.1 - Helm-based install for WML, WSL, Spark, Cognos Analytics and SPSS#555Sakshi-Singhroha1 wants to merge 111 commits into
Conversation
…Jobs for WML, Spark, SPSS
…stead of lt(semver,semver)
…fore namespace-scoped helm install
… for Helm 5.3.1+ installs
…I groups for Helm install
- Gate OperandRegistry/OperandConfig to < 5.3.1 to prevent OLM resource drift when services are installed via Helm - Gate spark-hb-control-plane patch job to < 5.3.1; OLM-managed deployment does not exist in Helm install model - Add CRD establishment wait inside WSL install_component() between cluster-scoped apply and namespace-scoped helm upgrade to prevent race condition - Extend Helm Job --wait timeout from 30m to 60m for WML, SPSS and Spark to match WSL and avoid premature timeout on loaded clusters
* Added CPD 5.3.1 gitops support * [minor] Fix CatalogSources template - add CPD 5.3.1 block (MASCORE-15507) * For CPD 5.3.1 services are installed via Helm * MASCORE-15507: Fix WSL Helm Job - install deps in sequence; add Helm Jobs for WML, Spark, SPSS * MASCORE-15507: Pass cpd_product_version to WSL, Spark, SPSS ArgoCD Applications * MASCORE-15507: Fix heredoc in WSL Helm Job - use printf to avoid YAML/shell conflict * Fix semver comparison - use toString to handle unquoted cpd_product_version value * MASCORE-15507: skip OLM CCS/OpenSearch post-sync patching for CPD 5.3.1+ * MASCORE-15507: bump job version to v2 to force new job name after script change * minor update * changes undo * MASCORE-15507: gate CCS/datarefinery/ws-runtimes OLM subscriptions behind < 5.3.1 * MASCORE-15507: fix semver comparison syntax - use .Compare pattern instead of lt(semver,semver) * MASCORE-15507: install cluster-scoped chart first to register CRDs before namespace-scoped helm install * MASCORE-15507: grant cpd-sa cluster-level CRD/ClusterRole permissions for Helm 5.3.1+ installs * MASCORE-15507: expand cpd-sa ClusterRole with escalate, roles, CPD API groups for Helm install * MASCORE-15507: pin cli image to digest instead of :latest in Helm install jobs * adding temp changes for facilities * undone postbackflow.sh script * MASCORE-15507: fix CPD 5.3.1 Helm service install correctness - Gate OperandRegistry/OperandConfig to < 5.3.1 to prevent OLM resource drift when services are installed via Helm - Gate spark-hb-control-plane patch job to < 5.3.1; OLM-managed deployment does not exist in Helm install model - Add CRD establishment wait inside WSL install_component() between cluster-scoped apply and namespace-scoped helm upgrade to prevent race condition - Extend Helm Job --wait timeout from 30m to 60m for WML, SPSS and Spark to match WSL and avoid premature timeout on loaded clusters --------- Co-authored-by: Sakshi Singhroha <Sakshi.Singhroha@ibm.com> Co-authored-by: Nikita Kumble <nikitakumble@gmail.com>
… Jobs Static-named ArgoCD sync hook Jobs (install-wsl-helm, install-wml-helm, install-spss-helm, install-spark-helm) were failing with: 'spec.template: Invalid value ... field is immutable' This happens when an old Job with the same name exists from a prior sync and ArgoCD tries to patch it. Since Job spec.template is immutable, the patch is rejected. Fix: add HookFailed to hook-delete-policy on all four Jobs so ArgoCD deletes the Job on both success AND failure, ensuring no stale Job exists when the next sync runs.
The wait loop was polling for: opensearchclusters.opensearch.cloudpackopen.ibm.com (does not exist) The actual CRD name registered by the IBM OpenSearch Helm chart is: clusters.opensearch.cloudpackopen.ibm.com This caused the WSL Helm Job to always timeout after 5 minutes at the OpenSearch step, blocking CCS, DataRefinery, WS-Runtimes and WSL from ever installing.
…f oc wait oc wait exits immediately with error if the resource does not exist. The CCS CR (ccs-cr) is created by the CCS operator after Helm installs it, so it is not immediately present. Replace oc wait with a polling loop using --ignore-not-found, same pattern used for all other CRD waits in this script. Polls every 30s up to 60 minutes.
…fter CCS Helm install
…us for CCS CR readiness
For CPD >= 5.3.1 the native ArgoCD Helm App (wave 122) installs the WML Helm chart but there is no mechanism to confirm the WmlBase CR reaches Completed status and to surface WML credentials to AWS Secrets Manager. This commit adds: - instance-applications/120-ibm-wml/templates/03-ibm-wml-post-verify.yaml A Job (gated >= 5.3.1) that polls WmlBase wml-cr until wmlStatus=Completed, then writes url/username/password to SM path <account>/<cluster>/<instance>/wml-cp4d. RBAC (SA, Role, RoleBinding) is scoped to cpd_operators_namespace and cpd_instance_namespace following the WSL post-verify pattern. - root-applications/ibm-mas-instance-root/templates/120-ibm-wml-app.yaml Wave-123 ArgoCD Application (wml-post-verify.<cluster>.<instance>) using the AVP plugin to inject SM credentials into the post-verify Job. Renders only for CPD >= 5.3.1 (inside the existing $is531 block). - instance-applications/120-ibm-wml/values.yaml Added account_id, region_id, cluster_id, instance_id, sm_aws_access_key_id, sm_aws_secret_access_key, cli_image_repo keys consumed by the post-verify job.
The cognos-analytics Helm chart renders a CAService CR. The operator immediately mutates spec.license (adding license: Enterprise) and adds spec.enableInstanaMetricCollection: false. ArgoCD sees these as drift and keeps the app OutOfSync. Add ignoreDifferences for both fields plus ServiceAccount/imagePullSecrets (standard pattern across all service apps). Add RespectIgnoreDifferences to syncOptions so ArgoCD applies the exclusions during sync.
…o MASCORE-16031
… SPSS (wave 122) SPSS operator at wave 122 tries to create a WS dependency CR of kind ws.cpd.ibm.com/v1beta1.WS. If ws-cluster-scoped runs at wave 123 (after SPSS), the CRD doesn't exist yet and SPSS fails with: 'Failed to find exact match for ws.cpd.ibm.com/v1beta1.WS' Move ws-cluster-scoped from wave 123 to wave 119 so it runs alongside the other cluster-scoped CRD installs, well before SPSS at wave 122.
…124) SPSS operator creates a WS CR and waits for Watson Studio as a dependency. The WSL operator (ws chart) deploys at wave 124, so SPSS namespace-scoped app must be at wave 125 or later to avoid a race condition where SPSS tries to create a WS CR before the WS operator is running. Also update wave comment from 123 to 119 on wsl-cluster section (cosmetic).
The instance-applications/120-ibm-wsl post-verify job (cpd-wsl-post-verify) was only deployed via the CPD < 5.3.1 AVP app path. For CPD >= 5.3.1 the native Helm apps are used for the WS operator/CR, but the post-verify job still needs to run to write the wsl-cp4d and wsl-project-id secrets to AWS SM. Without these secrets, inst02-watsonstudio-system AVP render fails. Wave 126 ensures it runs after wsl (124) and spss (125).
mnivedithaa
left a comment
mnivedithaa
left a comment
There was a problem hiding this comment.
Testing of this should include below scenarios
- Testing of cpd 5.3.1 - new instance
- Testing of previous cpd version deployment unaffected
- Upgrade from 5.2 OLM based deployment to 5.3.1 helm based deployment (Is there any conflict that happens since CRDs exist from OLM ones during upgrade? Do we need to ensure the older CRs are cleaned up after which Helm should kick in? Probably need to consider if we move CRD installation to cluster level)
| kind: Subscription | ||
| metadata: | ||
| name: "cpd-spss-operator" | ||
| name: spss-modeler-subscription |
There was a problem hiding this comment.
Can you please revert the name change and sync wave update?
There was a problem hiding this comment.
Reverted — name back to cpd-spss-operator and sync-wave back to 091.
| kind: Spss | ||
| metadata: | ||
| name: "spss-sample" | ||
| name: "spssmodeler-cr" |
There was a problem hiding this comment.
Same here. Also why are few values removed from this?
There was a problem hiding this comment.
That was from an older draft — the file only has name: "spssmodeler-cr" which is the correct name expected by the SPSS operator.
| @@ -1,62 +0,0 @@ | |||
| SPSS Modeler | |||
| =============================================================================== | |||
There was a problem hiding this comment.
Can you revert back this readme update if deleted by mistake?
There was a problem hiding this comment.
Restored — was accidentally deleted
| @@ -1,3 +1,24 @@ | |||
| # --- | |||
| # apiVersion: operators.coreos.com/v1alpha1 | |||
There was a problem hiding this comment.
remove the commented lines here and in other places.
There was a problem hiding this comment.
Removed from 00-ibm-wsl_Subscription.yaml and 01-ibm-wsl-cr.yaml. These were left over from development when the OLM content was temporarily commented out while adding the < 5.3.1 gate. Checked WML, Spark, SPSS — no other files had the same issue.
| 2. olm-utils-cm ConfigMap lookup (deployed by wave 110-ibm-cp4d) | ||
| 3. Hardcoded default (safety net for helm template dry-runs) | ||
| */}} | ||
| {{- $olmUtils := lookup "v1" "ConfigMap" $cpd_instance_ns "olm-utils-cm" }} |
There was a problem hiding this comment.
Does this lookup plugin work with Argocd? Argocd helm templates are generated even before connecting to cluster as far as I know. So can you verify if it is picking the values? If not, an alternative way to access it?
There was a problem hiding this comment.
Yes, lookup works here because ArgoCD renders Helm templates server-side via the argocd-vault-plugin-helm CMP sidecar running inside the cluster — so it has full Kubernetes API access at render time. Verified on noble10: every CPD service app's targetRevision matches exactly what's in olm-utils-cm.
| retry: | ||
| limit: 20 | ||
| syncOptions: | ||
| - CreateNamespace=true |
There was a problem hiding this comment.
This should be false here? As operator namespace is created prior to this
There was a problem hiding this comment.
Fixed — both namespaces exist before wave 119. CreateNamespace=false applied across all five service app templates.
| # Must run AFTER WSL at wave 124 — SPSS operator creates a WS CR and waits for it, | ||
| # so the WSL operator must already be running when SPSS reconciles. | ||
| apiVersion: argoproj.io/v1alpha1 | ||
| kind: Application |
There was a problem hiding this comment.
There are multiple application definition in single file. Maybe split them up for better readability? Same for other services as well
| retry: | ||
| limit: 20 | ||
| syncOptions: | ||
| - CreateNamespace=true |
There was a problem hiding this comment.
Here as well. Verify in other places as well
There was a problem hiding this comment.
Fixed — both namespaces exist before wave 119. CreateNamespace=false applied across all five service app templates.
| - group: ca.cpd.ibm.com | ||
| kind: CAService | ||
| jsonPointers: | ||
| - /spec/license/license | ||
| - /spec/enableInstanaMetricCollection |
There was a problem hiding this comment.
Is there any particular reason for ignoring this?
There was a problem hiding this comment.
The IBM Helm chart sets spec.license.license: "Enterprise" in the CR but the Cognos operator removes that field after reconciliation (live CR only has spec.license.accept: true). For enableInstanaMetricCollection, the operator sets it based on cluster state. Without ignoreDifferences, ArgoCD detects these as drift and loops. Verified on noble10 — with ignoreDifferences the app stays Synced Healthy.
There was a problem hiding this comment.
Hi Sakshi - I was actually requested that these items be move to namespace or instance scoped, rather than being at cluster scope. I don't believe the resources should be managed at the cluster scope precisely because 2 or more instances would then be trying to manage the same entity. Therefore we need to move these 100% to the namespace/instance scope, rather than cluster scope.
The problem is that we have customers deployed in clusters where multiple CPD installs are present. I expressed a concern that these were being deployed at the cluster level. I believe this is severe enough that this needs to be resolved now.
8 participants
Issue
https://jsw.ibm.com/browse/MASCORE-15507
https://jsw.ibm.com/browse/MASCORE-15752
https://jsw.ibm.com/browse/MASCORE-16163
Description
Updates GitOps automation to support CPD 5.3.1 in addition to existing CPD 5.2 and 5.3 support. For CPD 5.3.1+, OLM-based installation is replaced with Helm-based installation using IBM Helm charts.
Summary of changes
CPD 5.3.1 OLM bootstrap (
101-ibm-sync-jobs-cp4d, 110-ibm-cp4d)WML (
120-ibm-wml)WSL (
120-ibm-wsl)Spark (
120-ibm-spark)Cognos Analytics (
120-ibm-cognos-analytics)SPSS Modeler (
120-ibm-spss) — re-added with CPD 5.3.1 supportRe-added chart with OLM Subscription + SPSSModeler CR for CPD < 5.3.1
Added Helm-based install job for CPD 5.3.1+ installing canvasbase (v12.1.0, dependency) then spss (v12.1.0)
Added ArgoCD Application template 120-ibm-spss-app.yaml
Post-sync job (
121-ibm-post-sync-job-cp4d-services)Skip OLM-specific CCS/OpenSearch post-sync patching for CPD 5.3.1+
Bumped $_job_version to v2
RBAC
Test Results
Tested on noble8 cluster with CPD 5.3.1 (inst02):
wml.noble8.inst02 → WmlBase/wml-cr Completed 100%
wsl.noble8.inst02 → Synced Healthy
spark.noble8.inst02 → Synced Healthy
cognos.noble8.inst02 → CAService/ca-addon-cr Completed 100%
spss.noble8.inst02 → canvasbase + spss Helm charts installed, canvasbase Completed 100%