Skip to content

[Test] Stabilize test_build_image_no_internet. - #7600

Draft
gmarciani wants to merge 4 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3170/stabilize-tests-0902-1
Draft

[Test] Stabilize test_build_image_no_internet.#7600
gmarciani wants to merge 4 commits into
aws:developfrom
gmarciani:wip/mgiacomo/3170/stabilize-tests-0902-1

Conversation

@gmarciani

@gmarciani gmarciani commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Description of changes

Stabilize test_build_image_no_internet: align the build image config to the one used in test_build_image (with internet), where lustre installation is disabled on ubuntu2404. The test was not honoring this configuration because it missed the parameter to disable lustre. With this change we can drive the test to conclusion at the cost of not covering the case of ubu24 with lustre enabled. Now the test without internet covers the same of build-image with internet barring the proxied env.

Future improvements for test_build_image_no_internet and test_build_image: we must increase the coverage so that we cover all the combinations of update OS and Lustre instalation.

Tests

ONGOING:

test-suites:
  createami:
    test_createami.py::test_build_image_no_internet:
      dimensions:
        # RHEL family (rhel8, rhel9) -> Red Hat Enterprise Linux reservation platform
        - regions: [{{ g4dn_2xlarge_CAPACITY_RESERVATION_2_INSTANCES_3_HOURS_NOPG_rhel9 }}]
          instances: ["g4dn.2xlarge"]
          oss: ["rhel8", "rhel9"]
        # Non-RHEL family (alinux2023, ubuntu2204, ubuntu2404, rocky8, rocky9) -> Linux/UNIX reservation platform
        - regions: [{{ g4dn_2xlarge_CAPACITY_RESERVATION_5_INSTANCES_3_HOURS_NOPG_ubuntu2404 }}]
          instances: ["g4dn.2xlarge"]
          oss: ["alinux2023", "ubuntu2204", "ubuntu2404", "rocky8", "rocky9"]

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…e OS updates and Lustre installation.

The test is now aligned to what the test build_image does: for ubuntu2204, ubuntu2404 and rhel9 the OS updates are enabled and the lustre client installation is disabled.
When ubuntu2204 is used, the test uses the DLAMI Ubu2204.
@gmarciani
gmarciani requested review from a team as code owners September 2, 2026 20:45
@gmarciani gmarciani added skip-changelog-update Disables the check that enforces changelog updates in PRs 3.x Test labels Sep 2, 2026
@gmarciani
gmarciani marked this pull request as draft September 2, 2026 20:45
# Use official AMIs. First stage AMIs must not be used as parent image in this test.
base_ami = retrieve_latest_ami(region, os, ami_type="official", architecture=architecture)
update_os_packages = True
if os in ["ubuntu2204", "rhel9", "ubuntu2404"]:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes for the reviewer

Removing ubuntu2204 here is safe and correct because that case is covered by the if condition above. This is to cleanup the condition and avoid misudnerstandings.

if os in ["ubuntu2204"]:
        # Test Deep Learning AMIs

Iam:
AdditionalIamPolicies:
- Policy: arn:{{ partition }}:iam::aws:policy/AmazonS3ReadOnlyAccess
UpdateOsPackages:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes for the reviewer

The build config w/o internet is now aligned to the build config w/ internet where we can control the UpdateOs and LustreClient from the test code.

echo "Capping kernel to the max FSx Lustre-supported version: ${cap}"
sudo dnf install -y python3-dnf-plugin-versionlock
sudo dnf versionlock add "kernel-${cap}" "kernel-core-${cap}" "kernel-modules-${cap}"
sudo dnf versionlock add "kernel-${cap}" "kernel-core-${cap}" "kernel-modules-${cap}" "kernel-modules-core-${cap}" "kernel-tools-${cap}"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes for the reviewer

Experimentally proven that with these additional pinning, no kernel package can escape the pinning.

[root@ip-27-6-28-180 ~]# sudo dnf versionlock list
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use "rhc" or "subscription-manager" to register.

Last metadata expiration check: 0:10:13 ago on Wed 02 Sep 2026 08:04:52 PM UTC.
kernel-0:5.14.0-570.123.1.el9_6.*
kernel-core-0:5.14.0-570.123.1.el9_6.*
kernel-modules-0:5.14.0-570.123.1.el9_6.*
kernel-modules-core-0:5.14.0-570.123.1.el9_6.*
kernel-tools-0:5.14.0-570.123.1.el9_6.*
[root@ip-27-6-28-180 ~]# sudo dnf update --assumeno 2>&1 | grep -E '^ (kernel)' | awk '{print $1, $3}'
[root@ip-27-6-28-180 ~]#

LOGIN_NODE: ["tls"],
},
# Kernel modules that must be loaded on every node type, both before and after patching.
COMMON_MANDATORY_KERNEL_MODULES = [

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes for the reviewer

With this change we do not need to add new kernel modules to the list of LAZY_KERNEL_MODULES whenever we observe a false alarm on those. Instead we verify the things we are actually interested on: the mandatory kernel modules we are in control of.

@gmarciani
gmarciani marked this pull request as ready for review September 2, 2026 21:10
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3170/stabilize-tests-0902-1 branch 2 times, most recently from 7f95f80 to 09e0297 Compare September 3, 2026 02:35
@gmarciani gmarciani changed the title [Test] Fix and stabilize test_patching_cluster and test_build_image_no_internet. [Test] Stabilize test_build_image_no_internet. Sep 3, 2026
…ilure message to facilitate troubleshooting.
@gmarciani
gmarciani force-pushed the wip/mgiacomo/3170/stabilize-tests-0902-1 branch from 09e0297 to fbfe4dc Compare September 3, 2026 13:35
…and disable unattended upgrades.

 This is meant to prevent failures in Proxy and ProxyClient bootstrap caused by boot-time upgrades holding the dpkg lock.
@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.07%. Comparing base (1c49f73) to head (3c5cd27).
⚠️ Report is 5 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #7600   +/-   ##
========================================
  Coverage    90.07%   90.07%           
========================================
  Files          180      180           
  Lines        16410    16410           
========================================
  Hits         14782    14782           
  Misses        1628     1628           
Flag Coverage Δ
unittests 90.07% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.x skip-changelog-update Disables the check that enforces changelog updates in PRs Test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant