Add CI job for lightspeed operator deployment - #65
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: malingatembo The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Advanced Run ID: Comment |
|
/recheck |
|
Build failed (check pipeline). Post Warning: |
|
/recheck |
|
recheck |
1 similar comment
|
recheck |
0c3b371 to
cecc1eb
Compare
|
Unable to freeze job graph: Job lightspeed-operator-deployment-crc does not specify a run playbook |
|
Build failed (check pipeline). Post ❌ lightspeed-operator-deployment-crc FAILURE in 29m 14s |
| # CI job for deploying OpenStack Lightspeed operator | ||
| - job: | ||
| name: lightspeed-operator-deployment-crc | ||
| parent: cifmw-base-crc |
There was a problem hiding this comment.
question (non-blocking): Does this job deploy OpenStack as well? I think it does not but I am not 100 % certain about it. In a long term it would be good if the job deployed OpenStack as well because then we can do some basic checks that involve MCP.
There was a problem hiding this comment.
No no, this one only deploys OpenStack Lightspeed operator using the CRC ( so no full deployment of OpenStack control plane). for full OpenStack deployment + MCP integration testing, i think we'd need to add that to a seperate job or enhance this one - maybe we can keep track and have a seperate ticket for it
| - job: | ||
| name: lightspeed-operator-deployment-crc | ||
| parent: cifmw-base-crc | ||
| attempts: 1 |
There was a problem hiding this comment.
suggestion (blocking): I think you want to add openstack-k8s-operators-content-provider job as a dependency [1]. This job builds the operator and then provides it to the "child" jobs that depend on it. This allows you to test the operator from a content in a PR.
There was a problem hiding this comment.
Thanks for the pointer to this feature.. Yeah, I agree we should require a build from pr. I have make the change in ebf2560
| --- | ||
| # CI job for deploying OpenStack Lightspeed operator | ||
| - job: | ||
| name: lightspeed-operator-deployment-crc |
There was a problem hiding this comment.
suggestion (non-blocking): If I were working on this I would be using autohold to investigate the job from the inside.
|
Build failed (check pipeline). Post ❌ lightspeed-operator-deployment-crc FAILURE in 20m 52s |
|
Unable to freeze job graph: Job lightspeed-operator-deployment-crc depends on openstack-k8s-operators-content-provider which was not run. |
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 7m 04s |
| @@ -0,0 +1,17 @@ | |||
| --- | |||
There was a problem hiding this comment.
@malingatembo just FYI, you want to set the CONTAINER_TOOL to podman in Makefile. I've encountered it too:).
Lines 81 to 85 in 8eded89
Because of the failure in the job:
2026-08-26 17:01:59.168382 | controller | TASK [operator_build : lightspeed-operator - Call docker-build dry_run={{ cifmw_operator_build_dryrun|bool }}, chdir={{ operator.src }}, output_dir={{ cifmw_operator_build_basedir }}/artifacts, script=make docker-build, extra_args={'IMG': '{{ operator_img }}'}] ***
2026-08-26 17:01:59.168387 | controller | Wednesday 26 August 2026 17:01:59 +0000 (0:00:00.025) 0:02:31.959 ******
2026-08-26 17:01:59.168391 | controller | Wednesday 26 August 2026 17:01:59 +0000 (0:00:00.025) 0:02:31.958 ******
2026-08-26 17:01:59.315000 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_002_lightspeed_operator_call_docke.log
2026-08-26 17:01:59.315047 | controller | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NoneType: None
2026-08-26 17:01:59.315054 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/operator_build/tasks/build.yml:163
2026-08-26 17:01:59.315059 | controller | fatal: [localhost]: FAILED! =>
2026-08-26 17:01:59.315066 | controller | changed: true
2026-08-26 17:01:59.315072 | controller | msg: non-zero return code
2026-08-26 17:01:59.315077 | controller | rc: 2
2026-08-26 17:01:59.315082 | controller | stderr: ''
2026-08-26 17:01:59.315088 | controller | stderr_lines: []
2026-08-26 17:01:59.315093 | controller | stdout: |
2026-08-26 17:01:59.315098 | controller | ~/src/github.com/openstack-k8s-operators/lightspeed-operator ~/ci-framework-data/artifacts
2026-08-26 17:01:59.315104 | controller | docker build --build-arg GOMAXPROCS= -t 38.129.56.96:5001/openstack-k8s-operators/lightspeed-operator:011ef9691d76dd9e4f494b1c25294ed98df09a39 .
2026-08-26 17:01:59.315110 | controller | bash: line 1: docker: command not found
2026-08-26 17:01:59.315116 | controller | make: *** [Makefile:184: docker-build] Error 127
2026-08-26 17:01:59.315122 | controller | stdout_lines:
2026-08-26 17:01:59.315126 | controller | - ~/src/github.com/openstack-k8s-operators/lightspeed-operator ~/ci-framework-data/artifacts
2026-08-26 17:01:59.315131 | controller | - docker build --build-arg GOMAXPROCS= -t 38.129.56.96:5001/openstack-k8s-operators/lightspeed-operator:011ef9691d76dd9e4f494b1c25294ed98df09a39
2026-08-26 17:01:59.315137 | controller | .
2026-08-26 17:01:59.315142 | controller | - 'bash: line 1: docker: command not found'
2026-08-26 17:01:59.315146 | controller | - 'make: *** [Makefile:184: docker-build] Error 127'
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 50m 09s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 55m 18s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 57m 53s |
63b4ae6 to
b30b91e
Compare
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
b30b91e to
fadb45e
Compare
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
Creates lightspeed-operator-deployment-crc job that: - Deploys operator using ci-framework playbook - Uses example LLM endpoint (no real credentials) - Expected to fail on auth verification - Inherits from cifmw-base-crc for CRC cluster provisioning JIRA: OSPRH-33344
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 58m 44s |
Zuul generates its own ansible.cfg, ignoring repo's version. Use host-vars to set ansible_collections_paths environment variable. Changes: - Remove ansible.cfg (unused by Zuul) - Add host-vars.controller.ansible_collections_paths - Points to both symlink location and ci-framework original Addresses: cifmw.general.ci_script module resolution across ansible-playbook process boundaries
Per feedback: Ensures Zuul clones ci-framework into workspace before running playbooks, making collection available at standard path. Changes: - Add openstack-k8s-operators/ci-framework to required-projects - Remove broken host-vars.ansible_collections_paths (Ansible expects ANSIBLE_COLLECTIONS_PATH env var, not inventory var) Symlink in pre-run still needed to expose collection at ~/.ansible/collections
|
@malingatembo you are missing |
| # CI job for deploying OpenStack Lightspeed operator | ||
| - job: | ||
| name: lightspeed-operator-deployment-crc | ||
| parent: cifmw-base-crc-openstack |
There was a problem hiding this comment.
@malingatembo I'm still wondering whether the cifmw-base-crc-openstack is the right base job for us. If I was working on this I would probably try to use this parent job [1]. And then I would try to use the pre_tests_*: to execute your playbook. It has
- the benefit of OpenStack already being deployed in that job (simplifies stuff for us later when we want to do the MCP testing) +
- we have already working template we can follow [1] +
- we can simply configure the test_operator to run the tests using config from Jan later (until we do that we can simply set
cifmw_run_tests: false).
But I'm assuming you picked the job for a reason. This is just my thinking.
There was a problem hiding this comment.
You are right, [1] use a parent that already has open stack, that way we can just deploy lightspeed operator in pre_test hook. On it!
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 21m 01s |
Follow pattern from test-operator: use parent job that provides fully deployed OpenStack environment, then deploy our operator on top via pre_tests hook. Depends-On: openstack-k8s-operators/ci-framework#4108
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 22m 18s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 52m 08s |
| @@ -0,0 +1,28 @@ | |||
| --- | |||
There was a problem hiding this comment.
@malingatembo nice progress, it got the point of execution of the playbook! 🎉 Here are the logs from the hook with the exact failure if that helps [1]. I would take a look in the autohold whether the used image is really the correct one:
cifmw_openstack_lightspeed_catalog_image: "{{ content_provider_registry_ip }}:5001/openstack-k8s-operators/openstack-lightspeed-operator-catalog:{{ zuul.patchset }}"
According to [2] I would try this with a little bit of investigation:
# 38.102.83.44:5001/openstack-k8s-operators/lightspeed-operator-index:8cb8895710d6b31d03fcbb1879e062184412e93a
cifmw_openstack_lightspeed_catalog_image: "{{ content_provider_registry_ip }}:5001/openstack-k8s-operators/lightspeed-operator-index:{{ zuul.patchset }}"
Btw you are still missing the Depends-On: 🙈
[1] https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/logs//ff6/rdoproject.org/ff663b9e238d4cec984446cb5ec639ec/controller/ci-framework-data/logs/ci_script_023_run_hook_without_retry_deploy.log
[2] https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/logs//89b/rdoproject.org/89b0b1e9bcf24502ac23dd8a14c4b52d/ci-framework-data/logs/ci_script_007_lightspeed_operator_call_catal.log
There was a problem hiding this comment.
Thank you for you assistance and guidance in debugging this @lpiwowar ++
Test install-openstack-lightspeed.yml playbook to complete deployment without requiring real LLM credentials.
8cb8895 to
1cb0027
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 58m 58s |
|
recheck |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 30m 28s |
|
recheck |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 26m 22s |
|
recheck |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 29m 48s |
|
Nice @malingatembo, according to the logs it looks like the OpenStack Lightspeed operator got deployed in the job! 🎉 Two notes:
[1] https://github.com/openstack-k8s-operators/ci-framework/pull/4108/changes#diff-cc1f203fd237d3ffe937880e51fb432bfbc5c3d753cbf5816e3b73271c0d7215R185 |
Testing the CR readiness verification added to ci-framework. Expected: Job will fail at assert step (proving validation works) since we're using dummy credentials." Depends-On: openstack-k8s-operators/ci-framework#4108
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 10m 30s |
| name: lightspeed-operator-deployment-crc | ||
| parent: podified-multinode-hci-deployment-crc-1comp-backends | ||
| attempts: 1 | ||
| pre-run: ci/playbooks/lightspeed/pre-run.yaml |
There was a problem hiding this comment.
This is not typical in a ci-framework-based job. Is this a temporary thing for debugging with autohold?
| - openstack-k8s-operators/lightspeed-operator | ||
| - openstack-k8s-operators/ci-framework | ||
| vars: | ||
| zuul_work_dir: "{{ ansible_user_dir }}/{{ zuul.projects['github.com/openstack-k8s-operators/lightspeed-operator'].src_dir }}" |
There was a problem hiding this comment.
This might mess up other things that depend on zuul_work_dir.
You can set the path in the source for the playbook on line 18
| - openstack-k8s-operators/ci-framework | ||
| vars: | ||
| zuul_work_dir: "{{ ansible_user_dir }}/{{ zuul.projects['github.com/openstack-k8s-operators/lightspeed-operator'].src_dir }}" | ||
| pre_tests: |
There was a problem hiding this comment.
Note: this overrides all other pre_tests hooks. The preferred way to define is to define a single hook in its own parameter [1]. The format of the var switches to a dict instead of a list of dicts, and ciframework takes care of identifying the hooks and running them at the appropriate time.
The hook name format is <hook stage>_<hook name> e.g. pre_tests_install_openstack_lightspeed, and the hooks are run in alphabetical order, so the name is often prefixed with a number.
There are some examples from telemetry-operator in [2]
[1] https://github.com/openstack-k8s-operators/ci-framework/tree/main/roles/run_hook#single-hook-in-its-own-parameter
[2] https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/ci/vars-cloudkitty-tempest.yml
[3] https://github.com/openstack-k8s-operators/telemetry-operator/blob/main/ci/vars-audit-logging.yml
[4] https://github.com/infrawatch/feature-verification-tests/blob/master/ci/vars-cloudkitty-fvt.yml
Changed from list format that overrides all pre_tests hooks to single hook parameter format that merges with other hooks. chages: - Changed from list format `- name:` to dict format - Removed name field (derived from parameter name) - Added 01 prefix to ensure deployment runs before other pre_tests hooksZZ
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 38m 24s |
4 participants
Creates lightspeed-operator-deployment-crc job that:
JIRA: OSPRH-33344
Depends-On: openstack-k8s-operators/ci-framework#4108