Lapras gpu1 drop #129

Activity

1. jhgoh

   added

   incidentSomething isn't workingSomething isn't working

   on Aug 25, 2026
2. jhgoh

   added the

   Bug

   issue type on Aug 25, 2026
3. jhgoh commented on Aug 25, 2026

   

   jhgoh

   on Aug 25, 2026

   MemberAuthor

   More actions

   리붓 후 재 테스트중에도 GPU1이 drop됨.
4. slowmoyang commented on Aug 31, 2026

   

   slowmoyang

   on Aug 31, 2026

   More actions

   ❯ hostname
   lapras.lo
   
   ❯ date
   Mon Aug 31 10:06:39 AM KST 2026
   
   ❯ nvidia-smi -i 1
   Unable to determine the device handle for GPU1: 0000:21:00.0: Unknown Error
   No devices were found
   
   ❯ uv run python -c "import torch;print(torch.__version__);print(torch.torch.cuda.device_count())"
   2.13.0+cu130
   /users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1112: UserWarning: Can't initialize NVML
     raw_cnt = _raw_device_count_nvml()
   /users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1170: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /__w/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:119.)
     r = torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count
   0
   
   ❯ CUDA_VISIBLE_DEVICES=0,2,3 uv run python -c "import torch;print(torch.__version__);print(torch.torch.cuda.device_count())"
   2.13.0+cu130
   /users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1112: UserWarning: Can't initialize NVML
     raw_cnt = _raw_device_count_nvml()
   /users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1170: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /__w/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:119.)
     r = torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count
   0
   
   ❯ CUDA_VISIBLE_DEVICES=0 uv run python -c "import torch;print(torch.__version__);print(torch.torch.cuda.device_count())"
   2.13.0+cu130
   /users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1112: UserWarning: Can't initialize NVML
     raw_cnt = _raw_device_count_nvml()
   /users/hep/slowmoyang/work/test-gpu/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:1170: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /__w/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:119.)
     r = torch._C._cuda_getDeviceCount() if nvml_count < 0 else nvml_count
   0
5. slowmoyang commented on Aug 31, 2026

   

   slowmoyang

   on Aug 31, 2026

   More actions

   reboot 해볼까요?

   👍React with 👍1Reacted by Junghwan John Goh
6. slowmoyang commented on Aug 31, 2026

   

   slowmoyang

   on Aug 31, 2026

   More actions

   @jhgoh 직접 하신건가요?
7. jhgoh commented on Aug 31, 2026

   

   jhgoh

   on Aug 31, 2026

   MemberAuthor

   More actions

   리붓완료

   👍React with 👍1Reacted by slowmoyang
8. jhgoh commented on Sep 19, 2026

   

   jhgoh

   on Sep 19, 2026

   Last edited by jhgoh

   MemberAuthor

   More actions

   다시 빨간불 들어와서 재부팅 함
9. jhgoh commented on Oct 10, 2026

   

   jhgoh

   on Oct 10, 2026

   MemberAuthor

   More actions

   

   리부팅 시작.
10. jhgoh commented on Oct 10, 2026

    

    jhgoh

    on Oct 10, 2026

    MemberAuthor

    More actions

    리붓 완료.