Resolve native symbols for containerized python applications#329
Resolve native symbols for containerized python applications#329pramodk wants to merge 1 commit into
Conversation
- Add a libdwfl fallback that opens absolute module paths through /proc/<pid>/root so remote Pyxis/Enroot targets can resolve libraries that are only visible inside the container filesystem. - Associate the analyzed PID with each DWFL module before attach so the ELF lookup callback can find the target process root during native unwinding. - Preserve the existing build-id and linux-proc lookup paths first; the process-root fallback is used only when normal host lookup fails. - This fixes the behavior seen with the ImageNet Pyxis NCCL run where PyStack 1.6 reported insufficient native information or lost libtorch/libtorch_cuda/NCCL frames from container targets.
pramodk
force-pushed
the
pramodk/container-native-symbols
branch
2 times, most recently
from
9558b20 to
695ff34
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #329 +/- ##
==========================================
- Coverage 77.93% 77.81% -0.13%
==========================================
Files 56 56
Lines 5988 6005 +17
Branches 631 637 +6
==========================================
+ Hits 4667 4673 +6
- Misses 1321 1332 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| if (ret < 0) { | ||
| ret = find_elf_through_proc_pid_root(userdata, modname, file_name); | ||
| } |
There was a problem hiding this comment.
This only falls back to checking through /proc/$pid/root if the lookup on the host filesystem failed, but that seems wrong to me - isn't it possible that we've found entirely the wrong file if we looked on the host filesystem first? I would think that we'd want to always go through /proc/$pid/root whenever the mount namespace doesn't match. Or always, really? I can't think of any reason we wouldn't want to always resolve relative to it...
There was a problem hiding this comment.
In fact, I wonder whether this is something we should be implementing in pystack at all. Perhaps libdwfl should automatically be using /proc/$pid/root whenever it resolves files from /proc/$pid/map...
|
This will definitely need an automated integration test added to the test suite, also! |
pramodk
force-pushed
the
pramodk/container-native-symbols
branch
2 times, most recently
from
2e21b43 to
32d6854
Compare
3 participants
Describe your changes
This change fixes native symbol lookup for Python processes running inside container runtimes such as Enroot.
When the libdwfl lookup cannot open an absolute module path from
/proc/<pid>/maps, this PR now retries that same path through the target process root:The fallback is only used after build-id lookup and normal host path lookup fail, so existing host behavior remains unchanged.
Fixes #327.
Testing performed
Additional context
The issue is specific to native C/C++ symbol lookup. Python frames can still be visible because PyStack can inspect the target process, but native library paths from
/proc/<pid>/mapsmay exist only inside the container filesystem. As containers are used extensively in certain domains like AI/ML/Data Science, it would be really helpful to fix this.