동일한 cpu를 가졌음에도 live-migration이 되지못하는 경우가 있습니다.
이 경우, 대부분은 cpu의 flag가 달라서 발생하는 문제로 보입니다.
nova-compute의 log를 보게되면 아래와 같이 나타나고있습니다.
2024-08-09 00:52:16.054 3332011 ERROR nova.virt.libvirt.driver [req-85a295db-d2d5-457d-801d-8872473f422e dfbe729bd3bd4022ad03434692166412 08a94bc1846a4226b63bed76cf4e7c6a - default default] CPU doesn't have compatibility.보시는 바와 같이 cpu의 호환이 되지못하여 live migration이 실패하였습니다.
(해당 환경의 경우, nova.conf에 적용된 host_mode는 host-model입니다.)
lscpu를 통하여 cpu를 확인해보면 다음과 같습니다.
compute-01
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 72
On-line CPU(s) list: 0-71
Thread(s) per core: 2
Core(s) per socket: 18
Socket(s): 2
NUMA node(s): 4
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
Stepping: 4
CPU MHz: 2300.000
BogoMIPS: 4600.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-8,36-44
NUMA node1 CPU(s): 9-17,45-53
NUMA node2 CPU(s): 18-26,54-62
NUMA node3 CPU(s): 27-35,63-71
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs 다른 compute들과 차이가 나는 점은 cpu flag중 "arch_capabilities" 의 차이입니다.\
그럼 해당 flag는 무엇일까요?
arch_capabilities flag는 무엇일까요?
해당 flag는 보안 취약점을 조치한 CPU의 기능입니다.
즉, 해당 flag가 있는 compute-01은 어디로도 migration이 불가능하다는 점을 확인하였습니다.
# openstack server show 27b40cb6-4a42-41f9-bcac-10a24c5edeac
+-------------------------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | public-zone |
| OS-EXT-SRV-ATTR:host | compute-01 |
| OS-EXT-SRV-ATTR:hostname | test-vm-01-0808 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-01 |
| OS-EXT-SRV-ATTR:instance_name | instance-00001d19 |# openstack server migrate --live-migration --host compute-10 27b40cb6-4a42-41f9-bcac-10a24c5edeac
# openstack server list
+--------------------------------------+-----------------+--------+--------------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------+--------------------------+---------+
| 27b40cb6-4a42-41f9-bcac-10a24c5edeac | test-vm-01-0808 | ACTIVE | vlan-203=192.168.203.232 | N/A (booted from volume) | m1.tiny |보시는 바와 같이 migration이 수행되지 않는 모습을 보이고 있습니다.
nova-compute log를 보아도 동일하게 cpu호환이 되지 않아 수행이 불가능한 내용이 발생하고있습니다.
2024-08-12 15:43:54.384 3332011 ERROR nova.virt.libvirt.driver [req-d9f60f1e-4088-4ec2-938d-47c99277a850 dfbe729bd3bd4022ad03434692166412 7cfc00a8700447d4ab28e058d9e3ef48 - default default] CPU doesn't have compatibility.그렇다면 해당 cpu flag는 어떻게 예외하고 생성할 수 있을까?
nova.conf docs :: (https://docs.openstack.org/nova/yoga/configuration/sample-config.html)
example :: ( https://opendev.org/openstack/nova/commit/bcd6b42)
openstack yoga 버전의 nova.conf 문서를 확인해보면 아래와 같습니다.
Enable or disable guest CPU flags. For more information, refer to the
# documentation. (list value)
#cpu_model_extra_flags =nova 옵션 중 CPU 플래그를 사용하거나 사용하지 않도록 설정하는 "cpu_model_extra_flags" 를 사용하여 이슈를 해결하는 방안도 있습니다.
다만 이는 환경에 차이가있어 정상적으로 동작하지 않을 수 있습니다.
아래 환경에서 테스트를 해보았습니다.
#dev-compute-01
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
#dev-compute-02
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d
두 환경의 cpu flag차이는 ssbd, ibrs, ibpb, stibp, md_clear, flush_l1d 로 모두 보안기능입니다.
Ref:: ( https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/cpuid-enumeration-and-architectural-msrs.html )
nova의 libvirt부분에 아래와 같이 기입하여 배포하였습니다.
libvirt:
connection_uri: "qemu+tcp://127.0.0.1/system"
images_type: qcow2
+ cpu_model_extra_flags:"-ssbd -ibrs -ibpb -stibp -md-clear"live migration 테스트를 해보겠습니다.
# openstack server show 6c686d7c-c649-4701-9a59-76966e939222
+-------------------------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | public-zone |
| OS-EXT-SRV-ATTR:host | compute-01 |
| OS-EXT-SRV-ATTR:hostname | test-vm-0813 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | dev-compute-01 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000394 |해당 테스트 VM은 위와 같습니다. 현재 dev-compute-01에 위치해있습니다.
# openstack server migrate --live-migration --host dev-compute-02 6c686d7c-c649-4701-9a59-76966e939222
# openstack server list
+--------------------------------------+--------------+-----------+---------------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------+-----------+---------------------------+--------------------------+---------+
| 6c686d7c-c649-4701-9a59-76966e939222 | test-vm-0813 | MIGRATING | private-net=172.30.1.129 | N/A (booted from volume) | m1.tiny |
+--------------------------------------+--------------+-----------+---------------------------+--------------------------+---------+
# openstack server show 6c686d7c-c649-4701-9a59-76966e939222
+-------------------------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | public-zone |
| OS-EXT-SRV-ATTR:host | dev-compute-02 |
| OS-EXT-SRV-ATTR:hostname | test-vm-0813 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | dev-compute-02 |정상적으로 live migration 성공
반대로 dev-compute-02에서 dev-compute-01은 어떨까?
# openstack server migrate --live-migration --host dev-compute-01 6c686d7c-c649-4701-9a59-76966e939222
# openstack server list
+--------------------------------------+--------------+-----------+---------------------------+--------------------------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------+-----------+---------------------------+--------------------------+---------+
| 6c686d7c-c649-4701-9a59-76966e939222 | test-vm-0813 | MIGRATING | private-net=172.30.1.129 | N/A (booted from volume) | m1.tiny |
+--------------------------------------+--------------+-----------+---------------------------+--------------------------+---------+
# openstack server show 6c686d7c-c649-4701-9a59-76966e939222
+-------------------------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | public-zone |
| OS-EXT-SRV-ATTR:host | dev-compute-01 |
| OS-EXT-SRV-ATTR:hostname | test-vm-0813 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | dev-compute-01 |출발/도착지를 변경하여도 정상적으로 동작한다.
즉, cpu_model_extra_flags" 를 이용하여 같은 model의 서로 다른 flag가 있어도 live migration이 가능하다 라고 표현할 수 있습니다.
추가로, cpu_model_extra_flags는 nova 15.0.0 버전부터 도입되어 nova 버전이 낮다면 다른 방안을 고려해야 합니다.
다른 방안으로는 libvirt의 cpu map에서 예외하려는 flag를 삭제하여 libvirt에서 사용하지못하게 막아두는 방법도 있습니다.
(Ref :: https://stackoverflow.com/questions/41498261/libvirt-cpu-mode-host-model-confuses-while-mapping-cpu-models)
$ sudo kubectl exec -ti -n openstack libvirt-libvirt-default-cnsnm -- bash
Defaulted container "libvirt" out of: libvirt, init (init)
()[root@dev-compute-02 /]# cd /usr/share/libvirt/cpu_map
()[root@dev-compute-02 cpu_map]# ls -al
total 336
drwxr-xr-x 2 root root 4096 Aug 12 16:56 .
drwxr-xr-x 7 root root 106 Aug 12 16:57 ..
-rw-r--r-- 1 root root 104 Apr 23 09:30 arm_cortex-a53.xml
-rw-r--r-- 1 root root 104 Apr 23 09:30 arm_cortex-a57.xml
-rw-r--r-- 1 root root 104 Apr 23 09:30 arm_cortex-a72.xml
-rw-r--r-- 1 root root 105 Apr 23 09:30 arm_Falkor.xml
-rw-r--r-- 1 root root 512 Apr 23 09:30 arm_features.xml
-rw-r--r-- 1 root root 106 Apr 23 09:30 arm_FT-2000plus.xml
-rw-r--r-- 1 root root 111 Apr 23 09:30 arm_Kunpeng-920.xml
-rw-r--r-- 1 root root 111 Apr 23 09:30 arm_Tengyun-S2500.xml
-rw-r--r-- 1 root root 111 Apr 23 09:30 arm_ThunderX299xx.xml
-rw-r--r-- 1 root root 539 Apr 23 09:30 arm_vendors.xml
-rw-r--r-- 1 root root 4106 Apr 23 09:30 index.xml
-rw-r--r-- 1 root root 123 Apr 23 09:30 ppc64_POWER6.xml
-rw-r--r-- 1 root root 171 Apr 23 09:30 ppc64_POWER7.xml
-rw-r--r-- 1 root root 219 Apr 23 09:30 ppc64_POWER8.xml
-rw-r--r-- 1 root root 123 Apr 23 09:30 ppc64_POWER9.xml
-rw-r--r-- 1 root root 136 Apr 23 09:30 ppc64_POWERPC_e5500.xml
-rw-r--r-- 1 root root 138 Apr 23 09:30 ppc64_POWERPC_e6500.xml
-rw-r--r-- 1 root root 67 Apr 23 09:30 ppc64_vendors.xml
-rw-r--r-- 1 root root 160 Apr 23 09:30 x86_486.xml
-rw-r--r-- 1 root root 724 Apr 23 09:30 x86_athlon.xml
-rw-r--r-- 1 root root 1857 Apr 23 09:30 x86_Broadwell-IBRS.xml
-rw-r--r-- 1 root root 1811 Apr 23 09:30 x86_Broadwell-noTSX-IBRS.xml
-rw-r--r-- 1 root root 1774 Apr 23 09:30 x86_Broadwell-noTSX.xml
-rw-r--r-- 1 root root 1820 Apr 23 09:30 x86_Broadwell.xml
-rw-r--r-- 1 root root 2221 Apr 23 09:30 x86_Cascadelake-Server-noTSX.xml
-rw-r--r-- 1 root root 2266 Apr 23 09:30 x86_Cascadelake-Server.xml
-rw-r--r-- 1 root root 942 Apr 23 09:30 x86_Conroe.xml
-rw-r--r-- 1 root root 2560 Apr 23 09:30 x86_Cooperlake.xml
-rw-r--r-- 1 root root 860 Apr 23 09:30 x86_core2duo.xml
-rw-r--r-- 1 root root 748 Apr 23 09:30 x86_coreduo.xml
-rw-r--r-- 1 root root 752 Apr 23 09:30 x86_cpu64-rhel5.xml
-rw-r--r-- 1 root root 809 Apr 23 09:30 x86_cpu64-rhel6.xml
-rw-r--r-- 1 root root 1961 Apr 23 09:30 x86_Dhyana.xml
-rw-r--r-- 1 root root 2048 Apr 23 09:30 x86_EPYC-IBPB.xml
-rw-r--r-- 1 root root 2549 Apr 23 09:30 x86_EPYC-Milan.xml
-rw-r--r-- 1 root root 2289 Apr 23 09:30 x86_EPYC-Rome.xml
-rw-r--r-- 1 root root 2016 Apr 23 09:30 x86_EPYC.xml
-rw-r--r-- 1 root root 17998 Apr 23 09:30 x86_features.xml
-rw-r--r-- 1 root root 1737 Apr 23 09:30 x86_Haswell-IBRS.xml
-rw-r--r-- 1 root root 1691 Apr 23 09:30 x86_Haswell-noTSX-IBRS.xml
-rw-r--r-- 1 root root 1654 Apr 23 09:30 x86_Haswell-noTSX.xml
-rw-r--r-- 1 root root 1700 Apr 23 09:30 x86_Haswell.xml
-rw-r--r-- 1 root root 2316 Apr 23 09:30 x86_Icelake-Client-noTSX.xml
-rw-r--r-- 1 root root 2361 Apr 23 09:30 x86_Icelake-Client.xml
-rw-r--r-- 1 root root 2587 Apr 23 09:30 x86_Icelake-Server-noTSX.xml
-rw-r--r-- 1 root root 2632 Apr 23 09:30 x86_Icelake-Server.xml
-rw-r--r-- 1 root root 1467 Apr 23 09:30 x86_IvyBridge-IBRS.xml
-rw-r--r-- 1 root root 1430 Apr 23 09:30 x86_IvyBridge.xml
-rw-r--r-- 1 root root 666 Apr 23 09:30 x86_kvm32.xml
-rw-r--r-- 1 root root 773 Apr 23 09:30 x86_kvm64.xml
-rw-r--r-- 1 root root 773 Apr 23 09:30 x86_n270.xml
-rw-r--r-- 1 root root 1204 Apr 23 09:30 x86_Nehalem-IBRS.xml
-rw-r--r-- 1 root root 1167 Apr 23 09:30 x86_Nehalem.xml
-rw-r--r-- 1 root root 831 Apr 23 09:30 x86_Opteron_G1.xml
-rw-r--r-- 1 root root 943 Apr 23 09:30 x86_Opteron_G2.xml
-rw-r--r-- 1 root root 1104 Apr 23 09:30 x86_Opteron_G3.xml
-rw-r--r-- 1 root root 1376 Apr 23 09:30 x86_Opteron_G4.xml
-rw-r--r-- 1 root root 1455 Apr 23 09:30 x86_Opteron_G5.xml
-rw-r--r-- 1 root root 998 Apr 23 09:30 x86_Penryn.xml
-rw-r--r-- 1 root root 559 Apr 23 09:30 x86_pentium2.xml
-rw-r--r-- 1 root root 585 Apr 23 09:30 x86_pentium3.xml
-rw-r--r-- 1 root root 534 Apr 23 09:30 x86_pentiumpro.xml
-rw-r--r-- 1 root root 319 Apr 23 09:30 x86_pentium.xml
-rw-r--r-- 1 root root 947 Apr 23 09:30 x86_phenom.xml
-rw-r--r-- 1 root root 556 Apr 23 09:30 x86_qemu32.xml
-rw-r--r-- 1 root root 1038 Apr 23 09:30 x86_qemu64.xml
-rw-r--r-- 1 root root 1302 Apr 23 09:30 x86_SandyBridge-IBRS.xml
-rw-r--r-- 1 root root 1265 Apr 23 09:30 x86_SandyBridge.xml
-rw-r--r-- 1 root root 2240 Apr 23 09:30 x86_Skylake-Client-IBRS.xml
-rw-r--r-- 1 root root 2195 Apr 23 09:30 x86_Skylake-Client-noTSX-IBRS.xml
-rw-r--r-- 1 root root 2203 Apr 23 09:30 x86_Skylake-Client.xml
-rw-r--r-- 1 root root 2174 Apr 23 09:30 x86_Skylake-Server-IBRS.xml
-rw-r--r-- 1 root root 2129 Apr 23 09:30 x86_Skylake-Server-noTSX-IBRS.xml
-rw-r--r-- 1 root root 2137 Apr 23 09:30 x86_Skylake-Server.xml
-rw-r--r-- 1 root root 2003 Apr 23 09:30 x86_Snowridge.xml
-rw-r--r-- 1 root root 154 Apr 23 09:30 x86_vendors.xml
-rw-r--r-- 1 root root 1066 Apr 23 09:30 x86_Westmere-IBRS.xml
-rw-r--r-- 1 root root 1139 Apr 23 09:30 x86_Westmere.xmlcpu_maps에서 확인해보면 libvirt가 현재 가용가능한 cpu 맵을 확인할 수 있습니다.
여기서 예외할 cpu flag를 모두 삭제하면 될 것같습니다.
저의 환경의 경우 container형태로 이루어진 환경으로 image자체를 수정해야합니다.
아래 dockerfile로 구성하여 이미지를 수정해보겠습니다.
$ cat Dockerfile.libvirt
FROM 10.3.1.150:5000/burrito/centos-source-nova-libvirt:2.0.7
LABEL maintainer="Test-livemigration"
USER root
RUN sed -i '/<feature name='\''md-clear'\''>/,+3d' /usr/share/libvirt/cpu_map/x86_features.xml && \
sed -i '/<feature name='\''ssbd'\''>/,+3d' /usr/share/libvirt/cpu_map/x86_features.xml && \
sed -i '/<feature name='\''ibrs'\''>/,+3d' /usr/share/libvirt/cpu_map/x86_features.xml && \
sed -i '/<feature name='\''ibpb'\''>/,+3d' /usr/share/libvirt/cpu_map/x86_features.xml && \
sed -i '/<feature name='\''stibp'\''>/,+3d' /usr/share/libvirt/cpu_map/x86_features.xml && \
sed -i '/<feature name='\''ssbd'\''\/>/d' /usr/share/libvirt/cpu_map/*.xml && \
sed -i '/<feature name='\''ibrs'\''\/>/d' /usr/share/libvirt/cpu_map/*.xml && \
sed -i '/<feature name='\''ibpb'\''\/>/d' /usr/share/libvirt/cpu_map/*.xml && \
sed -i '/<feature name='\''stibp'\''\/>/d' /usr/share/libvirt/cpu_map/*.xml
$ sudo docker build --network host --no-cache -t 10.3.1.150:5000/burrito/centos-source-nova-libvirt:2.0.7-3 -f Dockerfile.libvirt ./usr/share/libvirt/cpu_map/ 에서 dev-compute 간 차이나는 flag를 모두 삭제하는 내용으로 dockerfile을 작성하였습니다.
다만, 이렇게 하는 것 보다는 cpu를 custom mode로 돌려서 진행하는 것이 제일 간편합니다.
