To configure QoS Over SR-IOV, we have changed our NIC to Mellanox ConnectX-3
First, I had to reinstall ubuntu to downgrade to Ubuntu 18.04
Then, I had to set the boot menu using ctrl+b as below:
Install the MLNX_OFED driver for Linux that supports SR-IOV:
Mellanox OFED Spec found here
Download & Install driver here
# Download Driver
wget https://content.mellanox.com/ofed/MLNX_OFED-4.9-6.0.6.0/MLNX_OFED_LINUX-4.9-6.0.6.0-ubuntu18.04-x86_64.iso
#Mount Driver
sudo mkdir /mnt/iso
mount -o loop /share/MLNX_OFED_LINUX-4.2-1.0.0.0-ubuntu16.04-x86_64.iso /mnt/iso
cd /mnt/iso
#Install Driver (srptools not working on x86_64/amd64)
sudo ./mlnxofedinstall --force --without-neohost-backend
#reboot
/etc/init.d/openibd restart
reboot
mlxconfig -d <mst_device> s SRIOV_EN=1
Check and start mst status
#Download mstflint for MST(Mellanox firmware burning application)
sudo apt-get install -y mstflint
mst start
mst status
SR-IOV can be enabled and managed by running the mlxconfig tool and setting the SRIOV_EN parameter to "1" without re-burning the firmware.
mlxconfig -d /dev/mst/mt3102_pciconf0 s SRIOV_EN=1
reboot
After reboot via
mstflint -dev 65:00.0 dc
we can see:
wrap up settings:
ifconfig enp101s0 20.0.1.3 netmask 255.255.255.0
# add at /etc/network/interfaces
auto enp101s0
iface enp101s0 inet static
address 20.0.1.3
netmask 255.255.255.0
#Check Changes
ifconfig -a
Note: At this point, the VFs are not seen via the lspci. Only when SR-IOV is enabled on the MLNX_OFED driver, you will be able to see them.
Lastly, insert options in /etc/modprobe.d/mlx4_core.conf file (add if nonexist)
vim /etc/modprobe.d/mlx4_core.conf
### add
options mlx4_core num_vfs=4 port_type_array=2,2 probe_vf=0
/etc/init.d/openibd restart
reboot
Then, we can see that VFs has been successfully made.
check if KVM is available via:
sudo apt install cpu-checker
sudo kvm-ok
Download needed packages:
sudo apt install -y qemu qemu-kvm libvirt-daemon libvirt-clients bridge-utils virt-manager
Check Successfully installed:
lsmod | grep -i kvm
Download Ubuntu ISO
wget https://releases.ubuntu.com/bionic/ubuntu-18.04.6-desktop-amd64.iso
And install Ubuntu at virt-manager
First Add PCI device to VM.
Then after reboot, will be set as below:
Yet after reboot, boot was unavailable due IOMMU was not on.
change IOMMU to on via:
vim /etc/default/grub
#add
GRUB_CMDLINE_LINUX="intel_iommu=on"
Then, We can see:
Lastly, I tried a ping test from the VM on b1 to JY to see if things worked correctly.
via
ping -I ens9 20.0.1.4
and was successfully recieved from
tcpdump -i enp3s0d1
Then resulted Diagram made is as followed: