Skip to content

K8s/XCalibur Support - #579

Open
sjayaram-nv wants to merge 12 commits into
NVIDIA-NeMo:mainfrom
sjayaram-nv:sjayaram/k8s-excalibur
Open

K8s/XCalibur Support#579
sjayaram-nv wants to merge 12 commits into
NVIDIA-NeMo:mainfrom
sjayaram-nv:sjayaram/k8s-excalibur

Conversation

@sjayaram-nv

@sjayaram-nv sjayaram-nv commented Aug 3, 2026

Copy link
Copy Markdown

This branch contains changes for adding the XCalibur executor backend to enable llm benchmarking runs in K8s.

@sjayaram-nv
K8s/XCalibur Support
51b1502 
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@sjayaram-nv
sjayaram-nv marked this pull request as draft August 3, 2026 06:16
@sudostock sudostock mentioned this pull request Aug 4, 2026
Open
sudostock
sudostock reviewed Aug 5, 2026
View reviewed changes
Comment thread nemo_run/run/torchx_backend/schedulers/nvcre.py
sudostock
sudostock reviewed Aug 5, 2026
View reviewed changes
Comment thread nemo_run/core/execution/xcalibur.py Outdated
sudostock
sudostock reviewed Aug 5, 2026
View reviewed changes
Comment thread nemo_run/run/torchx_backend/schedulers/nvcre.py
sudostock
sudostock reviewed Aug 5, 2026
View reviewed changes
Comment thread nemo_run/run/torchx_backend/schedulers/xcalibur.py Outdated
github-advanced-security[bot]
github-advanced-security Bot found potential problems Aug 7, 2026
View reviewed changes
Comment thread nemo_run/core/execution/xcalibur.py Fixed
Comment thread nemo_run/core/execution/xcalibur.py Fixed
sjayaram-nv and others added 9 commits August 11, 2026 17:40
@sjayaram-nv
Support for KAI scheduler in XCalibur executor
03043fd 
Address review comments

Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@sjayaram-nv
Fix detached runs for XCalibur jobs
7d43922 
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@sjayaram-nv
Merge branch 'NVIDIA-NeMo:main' into sjayaram/k8s-excalibur
d22d483 
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@sjayaram-nv
Merge branch 'NVIDIA-NeMo:main' into sjayaram/k8s-excalibur
38010c7 
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@sjayaram-nv
Added tests to improve test coverage
03831d3 
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@balasaajay @sjayaram-nv
Add Renovate configuration file (NVIDIA-NeMo#593)
6725d3d 
Signed-off-by: Ajay <abalasa@nvidia.com>

Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@balasaajay @sjayaram-nv
chore: add build constraint for setuptools version in pyproject.toml (N…
a8c4314 
…VIDIA-NeMo#595)

* chore: add build constraint for setuptools version in pyproject.toml

Added a build constraint to specify that setuptools must be less than version 82. This change ensures compatibility with existing dependencies.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

* chore: add build constraint for setuptools version in uv.lock

Included a build constraint to specify that setuptools must be less than version 82, ensuring compatibility with existing dependencies.

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

---------

Signed-off-by: Ajay Balasa <abalasa@nvidia.com>

Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@sjayaram-nv
Rename XCalibur to Nvcre
fe5eab5 
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@sjayaram-nv
Merge branch 'main' into sjayaram/k8s-excalibur
c3fe84c 
Signed-off-by: sjayaram-nv <sjayaram@nvidia.com>
@sjayaram-nv
sjayaram-nv force-pushed the sjayaram/k8s-excalibur branch from 0a4a6b3 to c3fe84c Compare September 8, 2026 15:17
@sjayaram-nv
sjayaram-nv marked this pull request as ready for review September 8, 2026 15:17
sudostock
sudostock reviewed Sep 9, 2026
View reviewed changes
Comment thread nemo_run/run/torchx_backend/schedulers/nvcre.py Outdated
sudostock
sudostock reviewed Sep 9, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
sudostock
sudostock reviewed Sep 9, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
sudostock
sudostock reviewed Sep 9, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
sudostock
sudostock reviewed Sep 9, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
sudostock
sudostock reviewed Sep 9, 2026
View reviewed changes
Comment thread nemo_run/run/torchx_backend/schedulers/nvcre.py Outdated
sudostock
sudostock reviewed Sep 10, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
sudostock
sudostock reviewed Sep 10, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
@sjayaram-nv
Addressed review comments and cleaned up redundant code
75387b8 
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
sudostock
sudostock reviewed Sep 15, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
sudostock
sudostock reviewed Sep 15, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
sudostock
sudostock reviewed Sep 15, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
@sudostock

sudostock commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Test suite is currently failing

sudostock
sudostock reviewed Sep 15, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py Outdated
sudostock
sudostock reviewed Sep 15, 2026
View reviewed changes
Comment thread nemo_run/run/torchx_backend/schedulers/nvcre.py Outdated
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 16, 2026
@sjayaram-nv
Job name is derived from execution-id, torch wrapper simplified
b6d260e 
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
@sjayaram-nv

sjayaram-nv commented Sep 17, 2026

Copy link
Copy Markdown
Author

Test suite is currently failing

Ran the nvcre tests manually; all are passing. Some of the other tests are failing for other dependenies:
`uv run pytest test/core/execution/test_nvcre.py test/run/torchx_backend/schedulers/test_nvcre.py
Test session starts (platform: linux, Python 3.11.9, pytest 8.3.5, pytest-sugar 1.0.0)
rootdir: /home/sjayaram/REPO/Run
configfile: pyproject.toml
plugins: anyio-4.8.0, cov-6.1.0, mock-3.14.0, sugar-1.0.0
collected 82 items

test/core/execution/test_nvcre.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 68% ██████▉
test/run/torchx_backend/schedulers/test_nvcre.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 100% ███████
....
Results (1.24s):
82 passed`

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-customer Waiting on the original author to respond labels Sep 17, 2026
github-advanced-security[bot]
github-advanced-security Bot found potential problems Sep 22, 2026
View reviewed changes
Comment thread nemo_run/core/execution/nvcre.py
launch_path = os.path.join(self.job_dir, "launch.sh")
with open(launch_path, "w") as f:
f.write(script)
os.chmod(launch_path, 0o555)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sudostock sudostock sudostock left review comments
+1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sjayaram-nv @sudostock @github-advanced-security @svcnvidia-nemo-ci @balasaajay