Hi Long Li,
Thanks for your feedback . I updated the diagram.
I am still not clear about the different kernel module interactions in the packet path.
The VFs appear as PCI devices in the guest kernel and are managed by MLNX drivers.
The synthetic interfaces are being managed by the uio_hv_generic driver in the kernel.
The netvsc PMD and mlnx PMD in dpdk app are instrumental in sending packets from app to NIC bypassing the kernel.

Questions about packet path:
1. Is the MLNX driver in the kernel used when packets are sent from NIC to the kernel without VF ?
2. Do the MLNX driver in kernel and uio_hv_genric driver interact?
3. Do the netvsc and mlnx PMDs interact  with uio_hv_generic driver?

Regards,
Nandini

On Fri, Jul 26, 2024 at 6:14 PM Long Li <longli@microsoft.com> wrote:

The VF data path is separate from the synthetic data path. The chart doesn’t show clearly that all the VF packets go directly to DPDK.

 

From the physical NIC, the packet data can either go through VF, or through netvsc, but not at the same time.

 

In practice, the synthetic path carries small number of packets (<1%). The majority of packets go through the VF path.

 

From: Nandini Rangaswamy <nandini.rangaswamy@broadcom.com>
Sent: Friday, July 26, 2024 3:46 PM
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: users@dpdk.org; Long Li <longli@microsoft.com>
Subject: Re: Query regarding netvsc PMD : VF removal

 

Hi Stephen and Long,

I am trying to understand the complete packet path and I am confused here.  The synthetic device is being created and managed by a driver in guest kernel space.

Shouldn't the packets be directly DMAed from NIC to VFs in guest. If all packets go through synthetic interface to the VF, then wouldn't kernel get involved here and defeat the purpose of DPDK ?

Please find attached a block diagram that i came up with highlights the devices and drivers and my understanding of their interactions.

I have shown in the diagram that for every VNIC , a synthetic and VF is present . Synthetic interfaces are managed by uio_hv_generic and VFs that appear as PCI devices in the guest kernel are managed by MLNX.

I have also shown that netvsc pmd depends on uio_hv_generic module. The red arrow represents synthetic data path and green arrows represent the accelerated data path.

I am still unclear about pkt flow from NIC to VF through synthetic interface and vice-versa involving netvsc pmd, mlnx driver/PMD.

Can you please clarify this?

Regards,

Nandini 

 

 

On Thu, Jul 25, 2024 at 2:52 PM Stephen Hemminger <stephen@networkplumber.org> wrote:

On Thu, 25 Jul 2024 11:41:59 -0700
Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> wrote:

> Hi Stephen,
> I am using the DPDK 22.11.5 LTS. Should the DPDK code take care of
> unbinding synthetic interface from uio_hv_generic or should it be done by
> the app when it gets hotplug notification?
> Regards,
> Nandini

The synthetic device (netvsc) controlled by uio_hv_generic is never hot plugged.
Only the VF which is mlx device can be removed and added.
The management of the VF for traffic is handled completely internally to
the netvsc device (same as the Linux netdevice). What happens is the
host notifys the guest over vmbus when VF changes. For devices handled
by the DPDK (netvsc PMD), this notification causes it to add/remove use
of the VF.

The VF should not be used directly. The synthetic device (netvsc) routes
traffic over the VF and reads from it when it is present and up.


This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.


This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.