[Concept] Revisit Concept: SR-IOV

1. Overview of Single Root I/O Virtualization (SR-IOV)

SR-IOV is a specification that allows a PCIe device to appear as multiple separate physical PCIe devices. The SR-IOV interface is standardized by the PCI SIG. SR-IOV works by introducing the concept of physical functions (PFs) and virtual functions (VFs). PFs are full-featured PCIe functions; VFs are "lightweight" functions that lack configuration resources. SR-IOV requires support from the BIOS, the operating system, and the network adapter. It's used in virtualization scenarios where it can provide increased I/O performance and reduced CPU utilization.

1.1 SR-IOV Architecture and Components:

SR-IOV Architecture:
SR-IOV architecture includes a Physical Function (PF) and one or more Virtual Functions (VFs). The PF is a full PCIe function that includes the SR-IOV Extended Capability. The VFs are lightweight PCIe functions that share one or more physical resources with the PF.

SR-IOV Physical Function (PF):
The PF is a full-featured PCIe function that includes the SR-IOV capabilities. It's used for managing the SR-IOV functionality. It includes the SR-IOV Extended Capability which is used to configure and manage the SR-IOV functionality.

SR-IOV Virtual Functions (VFs):
VFs are lightweight PCIe functions that share one or more physical resources with the PF. They are used for data movement. They include resources necessary for data movement but lack configuration resources.

NIC Switches:
The NIC Switch is a logical entity that routes packets between VFs and the external network. It can have one or more ports, including the default port (associated with the PF), and VF ports (associated with VFs).

Virtual Ports (VPorts):
VPorts are logical entities associated with a port on the NIC switch. They represent the network connectivity of a PF or a VF.

1.2 SR-IOV Data Paths and Failover:

SR-IOV Data Paths:
The SR-IOV data path is the path that network traffic follows when SR-IOV is enabled. It includes the VF data path and the synthetic data path. The VF data path is used when a VF is attached to a Hyper-V child partition. The synthetic data path is used when a VF is not available or not attached to a Hyper-V child partition.

Overview of SR-IOV Data Paths:
When a Hyper-V child partition is started, the Network Virtual Service Client (NetVSC) starts and exposes a VM network adapter. If the physical network adapter supports SR-IOV, it can enable one or more PCIe VFs. Each VF can be attached to a Hyper-V child partition. When this happens, network traffic occurs over the VF data path.

SR-IOV VF Data Path:
If the physical network adapter supports SR-IOV, it can enable one or more PCIe VFs. Each VF can be attached to a Hyper-V child partition. When this happens, network traffic in the guest operating system occurs over the VF data path. Packets are sent or received over the underlying VF of the network adapter instead of the synthetic data path.

SR-IOV Synthetic Data Path:
When a Hyper-V child partition is started, the virtualization stack starts the Network Virtual Service Client (NetVSC). NetVSC exposes a VM network adapter that provides a miniport driver edge to the protocol stacks that run in the guest operating system. NetVSC also communicates with the Hyper-V extensible switch that runs in the management operating system of the Hyper-V parent partition. The extensible switch component operates as a Network Virtual Service Provider (NetVSP). The interface between the NetVSC and NetVSP provides a software data path that is known as the synthetic data path.

SR-IOV VF Failover and Live Migration Support:
After the Hyper-V child partition is started, network traffic flows over the synthetic data path. If the physical network adapter supports SR-IOV, it can enable one or more PCIe VFs. Each VF can be attached to a Hyper-V child partition. When this happens, network traffic flows over the hardware-optimized SR-IOV VF Data Path. After the VF data path is established, network traffic can revert to the synthetic data path if any of the following conditions is true: A VF was attached to a Hyper-V child partition but becomes detached; The Hyper-V child partition is being live migrated to a different host. The transition between the VF and synthetic data paths occurs with minimum loss of packets and prevents the loss of TCP connections.

2. PF Driver Initialization

PF Miniport Driver Initialization:
The PF Miniport Driver Initialization process involves the creation of a NIC switch and the allocation of resources for the Physical Function (PF). The PF miniport driver must support the static creation of a NIC switch during initialization. The driver should also enable SR-IOV on the network adapter by calling the NdisMEnableVirtualization function with the EnableVirtualization parameter set to TRUE.

NIC Switch Management:
The NIC switch is a virtual switch that resides in the network adapter. It's responsible for routing network traffic between the PF and the Virtual Functions (VFs). The PF miniport driver is responsible for managing the NIC switch. It should support the creation, configuration, and deletion of the NIC switch.

Virtual Port Management:
Virtual ports (VPorts) are associated with the NIC switch. They are used to route network traffic to and from the VFs and the PF. The PF miniport driver should support the creation, configuration, and deletion of VPorts.

Managing Virtual Functions:
The PF miniport driver is responsible for managing the VFs. This includes allocating resources for a VF, initializing and tearing down a VF, and freeing resources for a VF. The driver should also support the enumeration of VFs on a network adapter and querying the parameters of a VF.

Virtual Function Initialization and Teardown:
The PF miniport driver is responsible for initializing and tearing down a VF. The driver should call the NdisMAllocateSharedMemory function to allocate shared memory for the VF. To tear down a VF, the driver should free all resources that were allocated for the VF.

Virtual Function Resource Allocation and Freeing:
The PF miniport driver should allocate resources for a VF by calling the NdisMAllocateSharedMemory function. To free resources for a VF, the driver should call the NdisMFreeSharedMemory function.

Enumerating Virtual Functions on a Network Adapter:
The PF miniport driver should support the enumeration of VFs on a network adapter. The driver should return a list of all VFs that have been allocated on the network adapter.

Querying the Parameters of a Virtual Function:
The PF miniport driver should support querying the parameters of a VF. The driver should return the current parameters for a VF.

Setting the Power State of a Virtual Function:
The PF miniport driver should support setting the power state of a VF. The driver should change the power state of a VF by issuing an OID set request of OID_SRIOV_SET_VF_POWER_STATE.

Resetting a Virtual Function:
The PF miniport driver should support resetting a VF. The driver should reset a VF by issuing an OID set request of OID_SRIOV_RESET_VF.

Halting a PF Miniport Driver:
The PF miniport driver should support halting the miniport driver for a PF. The driver should disable virtualization on the adapter by calling the NdisMEnableVirtualization function with the EnableVirtualization parameter set to FALSE and the NumVFs parameter set to zero.

3. VF Driver Initialization

Initializing a VF Miniport Driver:
The VF miniport driver follows the same steps as any NDIS miniport driver when its MiniportInitializeEx function is called. In addition to these steps, the VF miniport driver must verify that it is running in the Hyper-V child partition by calling the NdisGetHypervisorInfo function. If the partition type is reported as NdisHypervisorPartitionMsHvChild, the miniport driver is running in a Hyper-V child partition that is attached to the PF on the adapter. The VF miniport driver must not be installed with the SR-IOV standardized keywords and must not attempt to read these keywords. The VF miniport driver reports the SR-IOV hardware capabilities of the underlying virtual network adapter through an NDIS_SRIOV_CAPABILITIES structure. The VF miniport driver must not advertise virtual machine queue (VMQ) capabilities. However, the driver can advertise support for other NDIS technologies, such as power management and receive side scaling (RSS).

Halting a VF Miniport Driver:
The VF miniport driver follows the same steps as any NDIS miniport driver when its MiniportHaltEx function is called.

INF Requirements for VF Miniport Drivers:
The INF file for the miniport driver of a PCI Express (PCIe) Virtual Function (VF) does not specify any standardized INF keywords for single root I/O virtualization (SR-IOV). Only the INF file of a PCIe Physical Function (PF) specifies standardized SR-IOV keywords. The INF for a VF miniport driver follows (with one exception) the same requirements as other INF files for network adapters. The only exception is that the INF file for the VF miniport driver must define the binding relationships to the services that manage the SR-IOV data paths. This is needed to ensure that network access can fail over to the synthetic data path if the VF data path is torn down for any reason. To bind to the services that manage these data paths, the INF file for the VF miniport driver must specify the following settings for the UpperRange and LowerRange entries: HKR, Ndi\Interfaces, UpperRange, 0, "ndisvf" and HKR, Ndi\Interfaces, LowerRange, 0, "iovvf".