Skip to content

Retry transient oc exec errors during shiftstack pod polling - #4150

Open
tusharjadhav3302 wants to merge 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-exec-transient-retry
Open

Retry transient oc exec errors during shiftstack pod polling#4150
tusharjadhav3302 wants to merge 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-exec-transient-retry

Conversation

@tusharjadhav3302

Copy link
Copy Markdown
Contributor

Summary

  • Treat known transient oc exec stderr patterns as retryable while polling long-running shiftstack tests in shiftstackclient
  • Prevents false post-deployment failures when the API/admission webhook returns transient errors (for example SCCExecRestrictions timeouts) while the background command is still running
  • Improves the unreachable-pod failure message with the polling attempt limit

Context

shiftstack-rhel9-rhoso18.0-ocp4.22 on serval70 failed in post-deployment after ~32 minutes even though:

  • RHOSO deploy and openstack-edpm were healthy
  • The nested OCP 4.22 install inside shiftstackclient continued and completed successfully afterward

Failure signature:

admission plugin "security.openshift.io/SCCExecRestrictions" failed to complete validation in 13s

Related Jira: https://redhat.atlassian.net/browse/OSPRH-36061

Test plan

  • Re-run shiftstack-rhel9-rhoso18.0-ocp4.22 on serval70 and confirm post-deployment survives transient oc exec polling errors during osp_verification_4.22_nightly.yaml
  • Confirm non-transient failures (for example pod NotFound) still fail promptly

Made with Cursor

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@openshift-ci

openshift-ci Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Hi @tusharjadhav3302. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/b4f2059d92234a2898c1b7c94a2ac513

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 33m 28s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 31m 09s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 51m 37s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 20m 17s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 00s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 17m 08s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 41m 41s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 03s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 19s
cifmw-molecule-shiftstack FAILURE in 21m 20s

@eshulman2

Copy link
Copy Markdown
Contributor

/ok-to-test

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

recheck

@bshewale

Copy link
Copy Markdown
Contributor

/ok-to-test

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from e75f403 to fb296a0 Compare August 31, 2026 07:32
@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

2 similar comments
@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from fb296a0 to 6df275c Compare August 31, 2026 12:31
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/2260089efffe45f2ac95d8615f6248ac

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 11m 00s
podified-multinode-edpm-deployment-crc NODE_FAILURE Node(set) request 099-0000186871 failed in 0s
cifmw-crc-podified-edpm-baremetal FAILURE in 49m 00s
cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 50m 24s
✔️ cifmw-pod-zuul-files SUCCESS in 6m 16s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 22m 25s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 44m 37s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 22s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 48s
✔️ cifmw-molecule-shiftstack SUCCESS in 25m 16s

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 6df275c to b18c45e Compare September 1, 2026 06:52
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from b18c45e to 2857eaa Compare September 1, 2026 19:56
Comment thread roles/shiftstack/tasks/exec_command_in_pod.yml Outdated
Comment thread roles/shiftstack/tasks/exec_command_in_pod.yml Outdated
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 2857eaa to 5aedad7 Compare September 2, 2026 09:55
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from a06e258 to 4ac0715 Compare September 2, 2026 10:13
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/f6ebbb51f37a4d78a54474aca81d3978

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 25m 23s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 37m 53s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 2h 08m 38s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 12m 24s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 02s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 43m 49s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 2h 02m 57s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 45s
✔️ cifmw-pod-pre-commit SUCCESS in 10m 30s
cifmw-molecule-shiftstack FAILURE in 19m 05s

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 4ac0715 to a418566 Compare September 3, 2026 05:46
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/fb9c14f1c3ae4ac6bed2b13a4a3cc21a

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 34m 41s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 33m 51s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 57m 40s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 17m 51s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 57s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 50m 19s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 2h 03m 29s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 51s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 48s
cifmw-molecule-shiftstack FAILURE in 19m 35s

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from a418566 to ecb6282 Compare September 7, 2026 06:42
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/d95dc30870044ae0ab7dea2df1043216

✔️ openstack-k8s-operators-content-provider SUCCESS in 14m 30s
podified-multinode-edpm-deployment-crc NODE_FAILURE Node(set) request 099-0000193806 failed in 0s
cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000193807 failed in 0s
cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000193808 failed in 0s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 47s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 38m 26s
cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000193827 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 27s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 27s
✔️ cifmw-molecule-shiftstack SUCCESS in 24m 41s

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/29f97dfa51eb491f8aa872010b0b0786

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 44m 14s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 26m 24s
cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000194337 failed in 0s
cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000194338 failed in 0s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 39s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 39m 10s
cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000194358 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 20s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 19s
cifmw-molecule-shiftstack FAILURE in 21m 03s

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch 3 times, most recently from 9c443a8 to 0937831 Compare September 8, 2026 13:23
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/204c8f12856d44f28e445662b6afb151

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 29m 33s
podified-multinode-edpm-deployment-crc FAILURE in 1h 20m 56s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 33m 51s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 08m 06s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 35s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 04m 25s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 24m 11s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 48s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 18s
cifmw-molecule-shiftstack FAILURE in 20m 12s

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

@brjackma

brjackma commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/8673f7b84aa340bfa2ee69b4e3caf381

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 47m 32s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 25m 46s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 42m 38s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 03m 22s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 37s
openstack-k8s-operators-content-provider-bootc FAILURE in 34m 26s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 23s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 26s
✔️ cifmw-molecule-shiftstack SUCCESS in 26m 34s

@brjackma

brjackma commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/d3c93093ab794aa08e3aa524f87b224a

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 10m 48s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 26m 07s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 32m 49s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 56m 10s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 36s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 33m 32s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 33m 13s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 37s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 03s
cifmw-molecule-shiftstack FAILURE in 23m 03s

@evallesp

Copy link
Copy Markdown
Contributor

/override rdoproject.org/github-check

@openshift-ci

openshift-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@evallesp: Overrode contexts on behalf of evallesp: rdoproject.org/github-check

Details

In response to this:

/override rdoproject.org/github-check

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@evallesp evallesp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Sep 10, 2026
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 0937831 to 5e06ffc Compare September 10, 2026 08:30
@openshift-ci openshift-ci Bot removed the lgtm label Sep 10, 2026
@openshift-ci

openshift-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from evallesp. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/54155a0ae4dc4b6a86118ac29dbeca54

✔️ openstack-k8s-operators-content-provider SUCCESS in 4h 58m 04s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 29s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 33m 54s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 02m 14s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 34s
openstack-k8s-operators-content-provider-bootc FAILURE in 10m 05s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 18s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 30s
✔️ cifmw-molecule-shiftstack SUCCESS in 25m 44s

…ling

Long-running osp_verification tests poll shiftstackclient via oc exec.
Transient API/admission failures (for example SCCExecRestrictions webhook
timeouts) were treated as pod loss and aborted post-deployment early.

Keep polling on configurable transient stderr patterns so the test can
finish when the background command is still running. Patterns live in
defaults; failure messages report actual attempt count.

Related: OSPRH-36061
Signed-off-by: Tushar Jadhav <tjadhav@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-transient-retry branch from 5e06ffc to 9de78d2 Compare September 11, 2026 03:56
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/efd70722592a48c991b8b0d9614bb301

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 24s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 12s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 29m 18s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 58m 30s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 59s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 07m 28s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 32m 15s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 35s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 59s
✔️ cifmw-molecule-shiftstack SUCCESS in 27m 45s

@evallesp evallesp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Sep 11, 2026
@brjackma

Copy link
Copy Markdown
Contributor

/approved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants