After successfully configuring SR-IOV on VMs, I had to set DPDK for the VMs for full performance.
For VMs to use DPDK, driver updates were needed:
At host:
#Assume Driver Already downloaded and installed beforehand
#Install Driver (neohost-backend not working on x86_64/amd64)
sudo ./mlnxofedinstall --dpdk --upstream-libs --enable-sriov --hypervisor --force --without-neohost-backend
#reboot
sudo /etc/init.d/openibd restart
reboot
At VMs:
# Download Driver
wget https://content.mellanox.com/ofed/MLNX_OFED-4.9-6.0.6.0/MLNX_OFED_LINUX-4.9-6.0.6.0-ubuntu18.04-x86_64.iso
#Mount Driver
sudo mkdir /mnt/iso
mount -o loop /share/MLNX_OFED_LINUX-4.2-1.0.0.0-ubuntu16.04-x86_64.iso /mnt/iso
cd /mnt/iso
#Install Driver (neohost-backend not working on x86_64/amd64)
sudo ./mlnxofedinstall --dpdk --upstream-libs --guest --force --without-neohost-backend
#reboot
sudo /etc/init.d/openibd restart
reboot
*Note that compatability with DPDK <-> Pktgen is fragile. The below Steps were done using
DPDK - v.20.08
Pktgen - v.20.09
First we need to download DPDK Pktgen and libbpf codes:
# Download and unzip DPDK
wget https://codeload.github.com/DPDK/dpdk/tar.gz/refs/tags/v20.08
tar xJf dpdk-<version>.tar.xz
cd dpdk-<version>
#Download & Install Meson/Ninja
sudo apt-get -y install python3-pip
pip3 install meson
pip3 install ninja
pip3 install pyelftools
reboot
#DPDK Config & Install
meson setup -Denable_kmods=true build
cd ./build/
sudo ninja && sudo ninja install && sudo ldconfig
#Change mlx4 setting
vim config/common_base
# Change CONFIG_RTE_LIBRTE_MLX4_PMD=y
#Hugepage setting
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepage
or
sudo sysctl -w vm.nr_hugepages=102400
#Get Pktgen
wget https://codeload.github.com/pktgen/Pktgen-DPDK/tar.gz/refs/tags/pktgen-20.09.0
tar -xvf pktgen-21.11.0.tar.gz
cd Pktgen-DPDK-pktgen-21.11.0/
#Download Dependencies
sudo apt install cmake
sudo apt-get install -y pkg-config
sudo apt-get install -y libdpdk-dev
sudo apt-get install -y libnuma-dev
sudo apt-get install git libpcap-dev
#Install Pktgen
meson setup build
ninja -C build
sudo ninja -C build install
# Run Pktgen
Due to unknown issues, pktgen was unavailable to find the ports. Thus troubleshooting was needed.
*This part has a lot of troubleshooting.
Hope that I could find a way that allows DPDK on SR-IOV VMs.
< Referred Links >
1. DPDK Linux Driver
MLX4 poll mode driver library
First, after having the link down, I need to bind the interface to DPDK.
It is said:
It is recommended that vfio-pci be used as the kernel module for DPDK-bound ports in all cases. If an IOMMU is unavailable, the vfio-pci can be used in no-iommu mode. If, for some reason, vfio is unavailable, then UIO-based modules, igb_uio and uio_pci_generic may be used. See section UIO for details.
Thus, I tried to add this driver via:
sudo modprobe vfio-pci
However, it was added, yet when trying to bind it stated:
Then I found,
For some devices which lack support for legacy interrupts, e.g. virtual function (VF) devices, the igb_uio module may be needed in place of uio_pci_generic.
Because we're dealing with VFs with SR-IOV, I believed this was the right shot. To use this driver, I had to download and make:
# Install driver
git clone git://dpdk.org/dpdk-kmods
make dpdk-kmods/linux/igb_uio
# Set driver
sudo modprobe uio
sudo insmod igb_uio.ko
# Bind driver (at DPDK usertool dir)
./dpdk-devbind.py -b igb_uio 00:09.0
And with ./dpdk-devbing.py -s
I was able to see it was successfully binded:
However, after trying the helloworld example, it aborted as below:
Then, it worked using the original mlx4_core without bind/unbinding anything.
Then, with the command as below, we can use the DPDK-Pktgen:
sudo /usr/local/bin/pktgen -l 0-8 -n 2 -- -T -P -m "[1-4:5-8].0"