0. 문제 발견
GPU Passthrough를 통해 인스턴스에서 GPU를 사용하려고 한 처음에 문제가 생겼다.
nvidia driver, cuda를 설치 후 nvidia-smi로 gpu는 정상적으로 인식이 되었다.
하지만, gpu를 사용하는 순간 현재 터미널이 멈추는 문제가 발생했다.
다른 터미널에서 확인해보니 gpu에 부하도 걸리지 않고 사용이 되지 않았다.
이를 해결하기 위해 많은 구글링,,을 했지만 찾지 못했다.
그래서 로그와 설정들을 엄청나게.. 확인해 해결했다.
1. 원인 분석
원인 분석을 위해 많은 과정에 시간을 보냈지만, 결론만 정리해 보았다.
libvirt 로그 확인
root@com02:/home/nscl# journalctl -u libvirtd -f
Jun 25 17:32:08 com02 dnsmasq[1450]: read /etc/hosts - 10 names
Jun 25 17:32:08 com02 dnsmasq[1450]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 names
Jun 25 17:32:08 com02 dnsmasq-dhcp[1450]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jun 25 17:54:59 com02 libvirtd[1241]: libvirt version: 8.0.0, package: 1ubuntu7.10 (Marc Deslauriers <marc.deslauriers@ubuntu.com> Fri, 12 Apr 2024 13:48:21 -0400)
Jun 25 17:54:59 com02 libvirtd[1241]: hostname: com02
Jun 25 17:54:59 com02 libvirtd[1241]: Unable to read from monitor: Connection reset by peer
Jun 25 17:54:59 com02 libvirtd[1241]: internal error: qemu unexpectedly closed the monitor: 2025-06-25T08:54:59.349607Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio 0000:01:00.0: failed to setup container for group 13: Failed to set iommu for container: Operation not permitted
Jun 25 18:02:59 com02 libvirtd[1241]: End of file while reading data: Input/output error
Jun 25 18:04:13 com02 libvirtd[1241]: Unable to read from monitor: Connection reset by peer
Jun 25 18:04:13 com02 libvirtd[1241]: internal error: qemu unexpectedly closed the monitor: 2025-06-25T09:04:13.075894Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio 0000:01:00.0: failed to setup container for group 13: Failed to set iommu for container: Operation not permitted
==> /var/log/libvirt/qemu/instance-00000047.log <==
2025-06-25T09:46:10.594526Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x170ff8, 0x0,1) failed: Device or resource busy
2025-06-25T09:46:10.594529Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x170ff9, 0x0,1) failed: Device or resource busy
2025-06-25T09:46:10.594531Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x170ffa, 0x0,1) failed: Device or resource busy
2025-06-25T09:46:10.594534Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x170ffb, 0x0,1) failed: Device or resource busy
2025-06-25T09:46:10.594537Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x170ffc, 0x0,1) failed: Device or resource busy
2025-06-25T09:46:10.594539Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x170ffd, 0x0,1) failed: Device or resource busy
2025-06-25T09:46:10.594542Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x170ffe, 0x0,1) failed: Device or resource busy
2025-06-25T09:46:10.594544Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x170fff, 0x0,1) failed: Device or resource busy
2025-06-25T09:46:32.923622Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x190000, 0x280100000000,8) failed: Device or resource busy
2025-06-25T09:46:32.923655Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x1a008c, 0x1,4) failed: Device or resource busy
[ 4448.778460] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778462] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778465] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778467] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778470] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778472] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778475] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778482] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778485] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778487] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778490] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778493] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778495] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778498] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778500] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778503] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778505] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778508] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778510] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778513] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778515] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778518] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778520] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778523] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778525] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778528] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778530] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778533] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778536] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778538] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778541] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778543] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4448.778546] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4471.107364] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
[ 4471.107404] vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
VFIO 로그 확인
root@com02:/home/nscl# journalctl -f | grep -i vfio
Jun 25 18:46:10 com02 kernel: vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
Jun 25 18:46:10 com02 kernel: vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
Jun 25 18:46:32 com02 kernel: vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
Jun 25 18:46:32 com02 kernel: vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
문제 파악
로그 확인 결과, GPU BAR1 메모리 예약이 실패하였다.
실패 원인을 찾기 위해 메모리 매핑 상태를 확인하였다.
BAR 1: can't reserve [mem 0xf800000000-0xf80fffffff 64bit pref]
vfio_region_write(...) failed: Device or resource busy
메모리 매핑 상태 확인
root@com02:/home/nscl# cat /proc/iomem | grep -A 5 -B 5 f8000000
9e85a9000-9e97fffff : Kernel bss
200de80000-200eebffff : RAM buffer
200eec0000-f7ffffffff : Reserved
2010000000-208fffffff : pnp 00:02
2090000000-f7ffffffff : PNP0A08:00
f800000000-f817ffffff : PCI Bus 0000:01
f800000000-f80fffffff : 0000:01:00.0
f800000000-f8002fffff : efifb
f810000000-f811ffffff : 0000:01:00.0
f810000000-f811ffffff : vfio-pci
f818000000-f82fffffff : PCI Bus 0000:79
f818000000-f8181fffff : 0000:79:00.0
f820000000-f82fffffff : 0000:79:00.0
PCI 메모리 리소스 확인
root@com02:/home/nscl# cat /proc/iomem | grep -E "(pci|PCI)"
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
f0000000-f17fffff : PCI Bus 0000:01
f0000000-f0ffffff : vfio-pci
f1800000-f18fffff : PCI Bus 0000:02
f1900000-f27fffff : PCI Bus 0000:03
f1900000-f27fffff : PCI Bus 0000:04
f1900000-f19fffff : PCI Bus 0000:05
f1a00000-f25fffff : PCI Bus 0000:0a
f1a00000-f25fffff : PCI Bus 0000:0b
f1a00000-f1bfffff : PCI Bus 0000:0d
f1c00000-f1cfffff : PCI Bus 0000:0e
f1d00000-f23fffff : PCI Bus 0000:0f
f2400000-f24fffff : PCI Bus 0000:11
f2500000-f25fffff : PCI Bus 0000:12
f2600000-f26fffff : PCI Bus 0000:13
f2700000-f27fffff : PCI Bus 0000:14
f2800000-f2dfffff : PCI Bus 0000:15
f2800000-f2dfffff : PCI Bus 0000:16
f2800000-f29fffff : PCI Bus 0000:17
f2a00000-f2bfffff : PCI Bus 0000:47
f2c00000-f2cfffff : PCI Bus 0000:77
f2d00000-f2dfffff : PCI Bus 0000:78
f2e00000-f31fffff : PCI Bus 0000:79
f3200000-f32fffff : PCI Bus 0000:7a
f800000000-f817ffffff : PCI Bus 0000:01
f810000000-f811ffffff : vfio-pci
f818000000-f82fffffff : PCI Bus 0000:79
f830000000-f8300fffff : PCI Bus 0000:03
f830000000-f8300fffff : PCI Bus 0000:04
f830000000-f8300fffff : PCI Bus 0000:0a
f830000000-f8300fffff : PCI Bus 0000:0b
f830000000-f8300fffff : PCI Bus 0000:0f
f830100000-f8304fffff : PCI Bus 0000:15
f830100000-f8304fffff : PCI Bus 0000:16
f830100000-f8302fffff : PCI Bus 0000:17
f830300000-f8304fffff : PCI Bus 0000:47
원인 발견
EFI 프레임버퍼(efifb)가 GPU BAR 1의 시작 부분을 점유하고 있어서 VFIO가 전체 영역을 예약할 수 없어 발생했다.
f800000000-f80fffffff : 0000:01:00.0
f800000000-f8002fffff : efifb ← EFI 프레임버퍼가 GPU 메모리와 충돌!
2. 문제 해결
EFI 프레임버퍼를 비활성화하여 메모리 충돌을 해결한다.
SSH 접속이 가능한 환경이기 때문에 비활성화를 진행한 것이다.
만약 직접 서버에서 작업을 하는 경우, 비활성화 시 화면이 나오지 않으니 주의가 필요하다.
GRUB 설정 수정
video=efifb:off- EFI 프레임버퍼 비활성화
root@com02:/home/nscl# vi /etc/default/grub
....
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_aspm=off pci=noacpi video=efifb:off pcie_ports=native pcie_port_pm=off"
GRUB_CMDLINE_LINUX=""
....
root@com02:/home/nscl# update-grub
root@com02:/home/nscl# reboot
3. 정상 동작 확인
인스턴스에서 gpu 사용을 테스트한다.
git clone https://github.com/wilicc/gpu-burn
cd gpu-burn/
vi Makefile
UDAPATH ?= /usr
ifneq ("$(wildcard /usr/bin/nvcc)", "")
...
make
./gpu_burn 30
nvidia-smil
Wed Jun 25 12:28:01 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.230.02 Driver Version: 535.230.02 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA RTX A6000 Off | 00000000:00:05.0 Off | Off |
| 32% 66C P2 298W / 300W | 43798MiB / 49140MiB | 100% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2885 C ./gpu_burn 43792MiB |
+---------------------------------------------------------------------------------------+
4. downgraded 관련
문제를 해결하며 호스트 서버에서 PCIe 속도가 자꾸
downgraded로 바뀌어 이것이 원인이라고 생각했다.
하지만 내 지식이 부족해 오해했던 것이며, GPU를 사용하게 되면ok, 즉 정상으로 바뀐다.
일종의절전 상태와 같은 것 같다.
GPU 사용 X
root@com02:/etc/libvirt/qemu# lspci -vv -s 01:00.0 | grep "LnkSta"
LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
GPU 사용 O
root@com02:/etc/libvirt/qemu# lspci -vv -s 01:00.0 | grep "LnkSta"
LnkSta: Speed 16GT/s (ok), Width x16 (ok)
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
root@com02:/etc/libvirt/qemu# lspci -vv -s 01:00.0 | grep "LnkSta"
LnkSta: Speed 16GT/s (ok), Width x16 (ok)
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
root@com02:/etc/libvirt/qemu# lspci -vv -s 01:00.0 | grep "LnkSta"
LnkSta: Speed 16GT/s (ok), Width x16 (ok)
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
root@com02:/etc/libvirt/qemu# lspci -vv -s 01:00.0 | grep "LnkSta"
LnkSta: Speed 16GT/s (ok), Width x16 (ok)
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
root@com02:/etc/libvirt/qemu# lspci -vv -s 01:00.0 | grep "LnkSta"
LnkSta: Speed 2.5GT/s (downgraded), Width x16 (ok)
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
결론
OpenStack에서 GPU를 사용하기 위해 설정하며 많은 것을 배우고 있다.
과정에서 느끼는 것은 문제의 원인은 정말 생각지도 못한 부분일 수도 있으며, 구글링에는 한계가 있을 수 있다는 것이다.
결국 직접 로그와 설정을 하나씩 확인하며 해결하는 것이 필요하며, 이 과정을 너무 힘들지 않게, 즐기며 하는 것이 필요하다고 생각한다.
이 문제를 해결하면서 배운다는 자세로 나름 즐겁게(?) 임했던 것 같다.
끝!
Cloud/DevOps & Network Virtualization에 관심 있는 Engineer입니다. 🐳⚓️👨✈️