[dpdk-users] How to configue XL710 to communicate between virtual functions in KVM?

[dpdk-users] How to configue XL710 to communicate between virtual functions in KVM?

[Hao Li]

Hi all,

I have an XL710 (2 ports 40Gbe) and configure it to have 8 virtual
functions (VFs).
I then install two KVM guest OSes (both Ubuntu 20.04) and assign one VF for
each of them.

In host OS, ifconfig shows the following:

enp6s0f0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
> enp6s0f1: flags=4098<BROADCAST,MULTICAST>  mtu 1500
> enp6s0f0v0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
> enp6s0f0v3: flags=4098<BROADCAST,MULTICAST>  mtu 1500
> enp6s0f0v4: flags=4098<BROADCAST,MULTICAST>  mtu 1500
> enp6s0f0v5: flags=4098<BROADCAST,MULTICAST>  mtu 1500
> enp6s0f0v6: flags=4098<BROADCAST,MULTICAST>  mtu 1500
> enp6s0f0v7: flags=4098<BROADCAST,MULTICAST>  mtu 1500


where enp6s0f0 and enp6s0f1 are two PF ports, and others are VFs from
enp6s0f0.
enp6s0f0v1 and enp6s0f0v2 disappear since they are assigned to the two KVM
VMs.

In guest OS, ifconfig shows the following:

enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
> enp6s0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
> lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536


where enp6s0 is the VF assigned to it.

I then bind the uio_pci_generic for this VF

Network devices using DPDK-compatible driver
> ============================================
> 0000:06:00.0 'Ethernet Virtual Function 700 Series 154c'
> drv=uio_pci_generic unused=iavf,vfio-pci
> Network devices using kernel driver
> ===================================
> 0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci
> unused=vfio-pci,uio_pci_generic *Active*

I can then use "-a 06:00.0,driver=i40avf" to properly start this VF inside
the VM. (at least I can send packets with it)

Now my question is, how to forward packets between two VFs that have
already been assigned to VMs?
Or, putting it in a simpler way, how to determine which VFs should be sent
to when a packet comes into the physical port?

I used to think that I just need to configure the dst mac address of the
incoming packet, but it seems not to work.
Then I learn that rte_flow can be used to direct the flows between VFs, but
since VFs are already assigned to VMs,
I don't think the host DPDK application can configure them using rte_flow.

I'm not sure I'm doing everything right, please indicate if I misunderstood
anything.

Thanks in advance!

Best,
Hao