Release transient SYCL queues at the end of each test file to avoid OOM on low-memory GPUs#3001
Merged
Merged
Conversation
Every distinct SyclQueue a test creates is retained forever as a key in dpctl's SequentialOrderManager (keyed by queue identity), which pins its host-task events and the backing USM memory for the whole session. Over a full run this accumulates hundreds of queues and steadily drains device memory, occasionally triggering out-of-resources failures on low-memory GPU devices. Add a pytest_runtest_teardown hook that drains and drops the order-manager entries at each test-file boundary, keeping the footprint flat without paying the cost after every individual test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
View rendered docs @ https://intelpython.github.io/dpnp/index.html |
antonwolfy
marked this pull request as ready for review
July 15, 2026 12:03
antonwolfy
requested review from
ndgrigorian and
vlad-perevezentsev
as code owners
July 15, 2026 12:03
Collaborator
|
coverage: 78.118% (-0.009%) from 78.127% — fix-huge-queue-leak-in-tests into master |
Contributor
|
Array API standard conformance tests for dpnp=0.21.0dev3=py313h509198e_6 ran successfully. |
7 tasks
vlad-perevezentsev
approved these changes
Jul 16, 2026
vlad-perevezentsev
left a comment
vlad-perevezentsev
left a comment
Contributor
There was a problem hiding this comment.
LGTM
Thank you @antonwolfy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running the full test suite in a single process steadily drains device memory and can end in a
RuntimeError: ... UR_RESULT_ERROR_OUT_OF_RESOURCES(seen intermittently on a low-memory iGPU, where the kernel logsVM worker error: -12/exec queue reset detected).Root cause: every distinct
dpctl.SyclQueuea test creates is retained for the whole session as a key in dpctl'sSequentialOrderManager(the map is keyed by queue identity and only ever grows). Each retained entry holds a host-task event that pins the backing USM allocation, so the memory is never released. Over a full run the map accumulates ~300 queuesand free device memory bleeds from several GB down to ~1 GB, occasionally crossing the device limit.
This is amplified on integrated GPUs, where "device memory" is system RAM, so the leaked USM competes directly with the host and the container's memory budget.