Skip to content

[disconnected] Add option to perform image signature verification - #4141

Open
drosenfe wants to merge 1 commit into
openstack-k8s-operators:mainfrom
drosenfe:imagesignatureverify
Open

[disconnected] Add option to perform image signature verification#4141
drosenfe wants to merge 1 commit into
openstack-k8s-operators:mainfrom
drosenfe:imagesignatureverify

Conversation

@drosenfe

Copy link
Copy Markdown
Contributor

Add option to the existing configure openshift cluster for disconnected deployment hook to perform image signature verification. This may be used when oc mirror v2 has mirrored both container images and their cryptographic signatures.

jira: https://redhat.atlassian.net/browse/OSPRH-35167

Signed-off-by: David Rosenfeld drosenfe@redhat.com

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@openshift-ci

openshift-ci Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nemarjan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@drosenfe
drosenfe marked this pull request as draft August 25, 2026 13:17
@drosenfe drosenfe self-assigned this Aug 25, 2026
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/aabb940afc6449f493a9d1f7f9d5e571

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 53m 03s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 29m 37s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 48m 12s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 19m 45s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 3h 52m 30s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 2h 05m 09s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 29s
cifmw-pod-pre-commit FAILURE in 8m 20s

@drosenfe
drosenfe force-pushed the imagesignatureverify branch from cc9445e to 3372477 Compare August 25, 2026 17:22
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/8961d52bf79041b98e6e95724dc82073

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 59m 18s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 55s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 42m 01s
cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000181612 failed in 0s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 39m 33s
cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000181625 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 10m 56s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 02s

@drosenfe

Copy link
Copy Markdown
Contributor Author

recheck

@evallesp evallesp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm in general.

Comment thread hooks/playbooks/config_cluster_for_disconnected_deployment.yml Outdated
Comment thread hooks/playbooks/config_cluster_for_disconnected_deployment.yml Outdated
Comment thread hooks/playbooks/config_cluster_for_disconnected_deployment.yml Outdated
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/585684f705c14bb3a311e2dfaa65a02e

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 46m 57s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 43m 01s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 54m 36s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 34m 16s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 1h 14m 38s
cifmw-crc-podified-edpm-baremetal-bootc FAILURE in 36m 27s
✔️ noop SUCCESS in 0s
cifmw-pod-ansible-test FAILURE in 6m 30s
cifmw-pod-pre-commit FAILURE in 9m 20s

@drosenfe
drosenfe force-pushed the imagesignatureverify branch from e1740a1 to 565cc7d Compare September 3, 2026 17:47
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/d4340425e410426fa1b9f225b51fe820

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 10m 08s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 38s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 57m 07s
cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 44m 45s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 57m 47s
cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000191673 failed in 0s
✔️ noop SUCCESS in 0s
cifmw-pod-ansible-test FAILURE in 6m 11s
cifmw-pod-pre-commit FAILURE in 8m 44s

@drosenfe
drosenfe force-pushed the imagesignatureverify branch from 565cc7d to baf984e Compare September 4, 2026 13:19
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/3d758de16fc64af99832ec350152f80b

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 41m 01s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 27m 30s
cifmw-crc-podified-edpm-baremetal FAILURE in 2h 27m 54s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 27m 49s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 15m 44s
cifmw-crc-podified-edpm-baremetal-bootc FAILURE in 1h 40m 44s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 10m 16s
cifmw-pod-pre-commit FAILURE in 10m 15s

@drosenfe
drosenfe force-pushed the imagesignatureverify branch from baf984e to 99839c9 Compare September 4, 2026 17:07
@drosenfe
drosenfe marked this pull request as ready for review September 7, 2026 18:14
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@rabi rabi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A question on the default otherwise good for me.

mirror_location: "{{ disconnect_working_dir }}/mirror_location"
local_registry: "{{ disconnect_working_dir }}/local_registry"
oc_mirror_cert_manager_catalog_url: "{{ cifmw_cert_manager_catalog_url | default('registry.redhat.io/redhat/redhat-operator-index:v4.18') }}"
verify_image_signatures: "{{ cifmw_disconnected_verify_image_signatures | default(false) }}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image signature verification is true by default downstream right? Otherwise one has to use insecureAcceptAnything: true in /etc/containers/policy.json for default/registry.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image signature verification is not true by default downstream. This is from /etc/containers/policy.json:

"default": [ { "type": "insecureAcceptAnything" } ]

What I've found is that the test operator that runs tempest tests doesn't support image signature verification. I left the default at false to be consistent with current downstream jobs.

@evallesp evallesp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM in general!

delay: 30

- name: Generate ClusterImagePolicies from mirror data
cifmw.general.generate_cluster_image_policies:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: Great! Thanks !

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.

output_file = module.params.get("output_file")

# Red Hat public key: security.access.redhat.com/data/63405576.txt (between BEGIN and END, passed through | base64 -w0)
sigstore_key = """LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUEwQVN5dUgyVExXdkJVcVBIWjRJcAo3NWc3RW5jQmtnUUhkSm5qenhBVzVLUVRNaC9zaUJvQi9Cb1NydGlQTXduQ2hiVENuUU9JUWVadURpRm5odUo3Ck0vRDNiN0pvWDBtMTIzTmNDU242N21BZGpCYTZCZzZrdWtaZ0NQNFpVWmVFU2FqV1gvRWp5bEZjUkZPWFc1N3AKUkRDRU40MkovallsVnF0K2c5K0dya2VyOFN6ODZIM2wwdGJxT2RqYnovVnhIWWh3RjBjdFVNSHN5VlJEcTJRUAp0cXpOWGxtbE1oUy9Qb0ZyNlI0dS83SENuL0srTGVnY08yZkFGT2I0MEt2S1NLS1ZENmxld1VaRXJob3AxQ2dKClhqRHRHbW1POWRHTUY3MW1mNkhFZmFLU2R5K0VFNmlTRjJBMlZ2OVFoQmF3TWlxMmtPekVpTGc0bkFkSlQ4d2cKWnJNQW1QQ3FHSXNYTkdaNC9RK1lUd3dsY2UzZ2xxYjVMOXRmTm96RWRTUjlOODVERVNmUUxRRWRZM0NhbHdLTQpCVDFPRWhFWDF3SFJDVTRkck1PZWo2Qk5XMFZ0c2NHdEhtQ3JzNzRqUGV6aHdOVDh5cGt5UytUMHpUNFRzeTZmClZYa0o4WVNIeWVuU3pNQjJPcDJidnNFM2dyWStzNzRXaEc5VUlBNkRCeGNUaWUxNU5Tekt3Znphb05XT0RjTEYKcDdCWThhYUhFMk1xRnhZRlgrSWJqcGtRUmZhZVFRc291REZkQ2tYRUZWZlBwYkQyZGs2RmxlYU1UUHV5eHRJVApnalZFdEdRSzJxR0NGR2lRSEZkNGhmVitlQ0E2M0pybzF6MHpvQk01QmJJSVEzK2VWRnd0M0FsWnA1VVZ3cjZkCnNlY3FraS95cm12M1kwZHFaOVZPbjNVQ0F3RUFBUT09Ci0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQ=="""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(non-blocking) suggestion: Let's move this below NO_POLICIES in upper case as a constant.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved and capitalized in latest version.

for mirror, source in unique_pairs:
name = mirror.split("/")[-1].replace(".", "-").replace(":", "-")
policy = {
"apiVersion": "config.openshift.io/v1",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(non-blocking) question: DO we need to check openshift deployed version? I think in > 4.18 this is correct. If not this is not applied and we're silently not running this.

We can comment this that requires specific ocp version to run, or retrieving the info by "oc explain clusterimagepolicy --recursive | head -1" to parametrice this line, or maybe just not running the python code at all.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added that openshift 4.19 or later is required to the Documentation description.

"---\n" + "\n---\n".join(yaml.dump(p, sort_keys=False) for p in policies)
)

print(f"Generated {len(policies)} ClusterImagePolicy objects in {output_file}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(blocking) let's move this to module.log or put in a result field. This might bring runtime error of not able to parse JSON object.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to use logging module in latest version.

cifmw.general.generate_cluster_image_policies:
input_dir: "{{ mirror_location }}/working-dir/cluster-resources"
output_file: "{{ disconnect_working_dir }}/ClusterImagePolicies.yaml"
"""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(non-blocking) suggestion: Let's add the RETURN explanation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RETURN explanation has been added.

# Collect mirror-source pairs
pairs = []

for fname in os.listdir(input_dir):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(blocking) suggestion: let's check this exists first so we can fail_json in case of not existing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking for existence of input_dir has been added.

else "imageTagMirrors"
)
for entry in doc["spec"].get(key, []):
source = entry["source"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(non-blocking) suggestion: I'd check if both keys exists before trying to access them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check for existence of source and mirrors has been added.


result = {
"success": False,
"changed": False,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(blocking) suggestion: We need to change this as True somewhere. Might be good L134?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting changed to True has been added to suggested line.

type: str
output_file:
description:
- Absolute path to directory when ClusterImagePolicy file is created

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(blocking) typo: I think this should be "to file" instead "to directory"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to file instead of directory has been made.


print(f"Generated {len(policies)} ClusterImagePolicy objects in {output_file}")

# Ensure some cluster image policies were created

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(blocking) suggestion: let's move this to L 121, so we don't create a file the header.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested move has been made.

@evallesp

evallesp commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

(blocking) suggestion: Also I think this new python code suitable to have some tests located at: tests/unit/modules/test_generate_cluster_image_policies.py.
This would be taken automatically by molecule.

Add option to the existing configure openshift cluster for disconnected
deployment hook to perform image signature verification. This may be used
when oc mirror v2 has mirrored both container images and their cryptographic
signatures.

jira: https://redhat.atlassian.net/browse/OSPRH-35167

Signed-off-by: David Rosenfeld drosenfe@redhat.com
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/26d34a47985e489f856396d26a392fce

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 19m 06s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 56s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 35m 03s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 56m 50s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 06m 11s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 30m 31s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 32s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 46s

@drosenfe

Copy link
Copy Markdown
Contributor Author

(blocking) suggestion: Also I think this new python code suitable to have some tests located at: tests/unit/modules/test_generate_cluster_image_policies.py. This would be taken automatically by molecule.

Not sure how to make molecule tests. Can you re-review the rest while I try to figure out molecule tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants