Query regarding netvsc PMD : VF removal

Query regarding netvsc PMD : VF removal

[Nandini Rangaswamy]

[-- Attachment #1: Type: text/plain, Size: 5228 bytes --]

Hi,
I am integrating my DPDK app (using DPDK version 22.11) with Netvsc PMD. I
was testing the use case of VF removal with the DPDK app.
I registered for hotplug notification in the DPDK app and enabled DPDK
logs. I see the APIs being invoked for VF removal from the logs below:

2024-07-25T17:28:13.317 |13590| MSG        [NET] dpdk_log_write:112
hn_eth_rmv_event_callback(): Removing VF portid 0

2024-07-25T17:28:13.318 |13590| MSG        [NET] dpdk_log_write:112 EAL:
Ignoring uevent 'remove@
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/80033a99-bc5e-43c7-aeea-e5773d4414d2/pcibc5e:00/bc5e:00:02.0/infiniband/mlx5_0'

2024-07-25T17:28:13.318 |13590| MSG        [NET] dpdk_log_write:112
hn_nvs_set_datapath(): set datapath Synthetic

2024-07-25T17:28:13.318 |13590| MSG        [NET] dpdk_log_write:112
hn_remove_delayed(): Start to remove port 0

2024-07-25T17:28:13.319 |13590| MSG        [NET] dpdk_log_write:112
mlx5_net: port 0 stopping device

2024-07-25T17:28:13.319 |13590| MSG        [NET] dpdk_log_write:112
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x10c9e6300
removed.

2024-07-25T17:28:13.319 |13590| MSG        [NET] dpdk_log_write:112
mlx5_common: mlx5 list hrxq entry 0x10c942f00 removed.

2024-07-25T17:28:13.319 |13590| MSG        [NET] dpdk_log_write:112
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x10c9e6a40
removed.

2024-07-25T17:28:13.319 |13590| MSG        [NET] dpdk_log_write:112
mlx5_common: mlx5 list mlx5_0_flow_table entry 0x17fdc9040 removed.

2024-07-25T17:28:13.319 |13590| MSG        [NET] dpdk_log_write:112
mlx5_net: port 0: 0 flows flushed before stopping

2024-07-25T17:28:13.320 |13590| MSG        [NET] dpdk_log_write:112
mlx5_net: port 0 Tx queue 0 freeing WRs

2024-07-25T17:28:13.321 |13590| MSG        [NET] dpdk_log_write:112
mlx5_net: port 0 Rx queue 0 freeing 8192 WRs

2024-07-25T17:28:13.321 |13590| MSG        [NET] dpdk_log_write:112
mlx5_net: port 0 closing device "mlx5_0"

2024-07-25T17:28:13.321 |13590| MSG        [NET] dpdk_log_write:112
mlx5_net: port 0: 0 flows flushed before stopping

2024-07-25T17:28:13.322 |13590| MSG        [NET] dpdk_log_write:112
mlx5_common: freeing B-tree 0x10cb88134 with table 0x10cb86bc0

2024-07-25T17:28:13.324 |13590| MSG        [NET] dpdk_log_write:112
mlx5_net: port 0 Tx queue 0 freeing WRs

2024-07-25T17:28:13.324 |13590| MSG        [NET] dpdk_log_write:112
mlx5_common: freeing B-tree 0x10ca2dfe8 with table 0x10ca2c980

2024-07-25T17:28:13.334 |13590| MSG        [NET] dpdk_log_write:112 EAL:
request: eal_dev_mp_request

2024-07-25T17:28:13.334 |13590| MSG        [NET] dpdk_log_write:112 EAL:
PCI device bc5e:00:02.0 on NUMA socket -1

2024-07-25T17:28:13.334 |13590| MSG        [NET] dpdk_log_write:112 EAL:
remove driver: 15b3:1016 mlx5_pci

2024-07-25T17:28:13.336 |13590| MSG        [NET] dpdk_log_write:112
mlx5_common: freeing B-tree 0x1801d9dc0 with table 0x1801d7500









2024-07-25T17:28:13.828 |13590| MSG        [NET]
dpdk_dev_event_callback:295 HOTPLUG: event callback 1 device bc5e:00:02.0

2024-07-25T17:28:13.828 |13590| MSG        [NET] dpdk_log_write:112
netvsc_hotadd_callback(): Device notification type=1
device_name=bc5e:00:02.0

2024-07-25T17:28:13.828 |13590| MSG        [NET] dpdk_log_write:112
netvsc_hotadd_callback(): Device notification type=1
device_name=bc5e:00:02.0

2024-07-25T17:28:13.829 |13590| MSG        [NET] dpdk_log_write:112 EAL:
receive uevent(name:(null), type:1, subsystem:0)

2024-07-25T17:28:13.829 |13590| MSG        [NET] dpdk_log_write:112 EAL:
Ignoring uevent 'unbind@
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/80033a99-bc5e-43c7-aeea-e5773d4414d2'

2024-07-25T17:28:13.829 |13590| MSG        [NET] dpdk_log_write:112 EAL:
Ignoring uevent 'remove@
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/device:07/VMBUS:01/80033a99-bc5e-43c7-aeea-e5773d4414d2'

2024-07-25T17:28:13.829 |13590| MSG        [NET] dpdk_log_write:112 EAL:
Ignoring uevent 'remove@/channels/41'

2024-07-25T17:28:13.829 |13597| MSG        [NET] dpdk_log_write:112
hn_nvs_handle_vfassoc(): VF serial 3 remove from port 2

2024-07-25T17:28:13.829 |13597| MSG        [NET] dpdk_log_write:112
hn_vf_remove(): VF path not active.


However, I see the synthetic interface not being unbound from the
uio_hv_generic driver.

Should it be done as part of hotplug notification by my DPDK app or should
DPDK code already take care of it ?

Regards,

Nandini

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #2: Type: text/html, Size: 8183 bytes --]

Re: Query regarding netvsc PMD : VF removal

[Stephen Hemminger]

On Thu, 25 Jul 2024 10:52:18 -0700
Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> wrote:

> Hi,
> I am integrating my DPDK app (using DPDK version 22.11) with Netvsc PMD. I
> was testing the use case of VF removal with the DPDK app.
> I registered for hotplug notification in the DPDK app and enabled DPDK
> logs. I see the APIs being invoked for VF removal from the logs below:

Make sure you are using the updated 22.11 LTS release.

Re: Query regarding netvsc PMD : VF removal

[Nandini Rangaswamy]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

Hi Stephen,
I am using the DPDK 22.11.5 LTS. Should the DPDK code take care of
unbinding synthetic interface from uio_hv_generic or should it be done by
the app when it gets hotplug notification?
Regards,
Nandini

On Thu, Jul 25, 2024 at 11:11 AM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Thu, 25 Jul 2024 10:52:18 -0700
> Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> wrote:
>
> > Hi,
> > I am integrating my DPDK app (using DPDK version 22.11) with Netvsc PMD.
> I
> > was testing the use case of VF removal with the DPDK app.
> > I registered for hotplug notification in the DPDK app and enabled DPDK
> > logs. I see the APIs being invoked for VF removal from the logs below:
>
> Make sure you are using the updated 22.11 LTS release.
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #2: Type: text/html, Size: 2059 bytes --]

Re: Query regarding netvsc PMD : VF removal

[Stephen Hemminger]

On Thu, 25 Jul 2024 11:41:59 -0700
Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> wrote:

> Hi Stephen,
> I am using the DPDK 22.11.5 LTS. Should the DPDK code take care of
> unbinding synthetic interface from uio_hv_generic or should it be done by
> the app when it gets hotplug notification?
> Regards,
> Nandini

The synthetic device (netvsc) controlled by uio_hv_generic is never hot plugged.
Only the VF which is mlx device can be removed and added.
The management of the VF for traffic is handled completely internally to
the netvsc device (same as the Linux netdevice). What happens is the
host notifys the guest over vmbus when VF changes. For devices handled
by the DPDK (netvsc PMD), this notification causes it to add/remove use
of the VF.

The VF should not be used directly. The synthetic device (netvsc) routes
traffic over the VF and reads from it when it is present and up.

Re: Query regarding netvsc PMD : VF removal

[Nandini Rangaswamy]

[-- Attachment #1.1: Type: text/plain, Size: 2940 bytes --]

Hi Stephen and Long,
I am trying to understand the complete packet path and I am confused here.
The synthetic device is being created and managed by a driver in guest
kernel space.
Shouldn't the packets be directly DMAed from NIC to VFs in guest. If all
packets go through synthetic interface to the VF, then wouldn't kernel get
involved here and defeat the purpose of DPDK ?
Please find attached a block diagram that i came up with highlights the
devices and drivers and my understanding of their interactions.
I have shown in the diagram that for every VNIC , a synthetic and VF is
present . Synthetic interfaces are managed by uio_hv_generic and VFs that
appear as PCI devices in the guest kernel are managed by MLNX.
I have also shown that netvsc pmd depends on uio_hv_generic module. The red
arrow represents synthetic data path and green arrows represent the
accelerated data path.
I am still unclear about pkt flow from NIC to VF through synthetic
interface and vice-versa involving netvsc pmd, mlnx driver/PMD.
Can you please clarify this?
Regards,
Nandini


On Thu, Jul 25, 2024 at 2:52 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Thu, 25 Jul 2024 11:41:59 -0700
> Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> wrote:
>
> > Hi Stephen,
> > I am using the DPDK 22.11.5 LTS. Should the DPDK code take care of
> > unbinding synthetic interface from uio_hv_generic or should it be done by
> > the app when it gets hotplug notification?
> > Regards,
> > Nandini
>
> The synthetic device (netvsc) controlled by uio_hv_generic is never hot
> plugged.
> Only the VF which is mlx device can be removed and added.
> The management of the VF for traffic is handled completely internally to
> the netvsc device (same as the Linux netdevice). What happens is the
> host notifys the guest over vmbus when VF changes. For devices handled
> by the DPDK (netvsc PMD), this notification causes it to add/remove use
> of the VF.
>
> The VF should not be used directly. The synthetic device (netvsc) routes
> traffic over the VF and reads from it when it is present and up.
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 3525 bytes --]

[-- Attachment #2: Screenshot 2024-07-26 at 3.40.44 PM.png --]
[-- Type: image/png, Size: 278264 bytes --]

RE: Query regarding netvsc PMD : VF removal

[Long Li]

[-- Attachment #1: Type: text/plain, Size: 3566 bytes --]

The VF data path is separate from the synthetic data path. The chart doesn’t show clearly that all the VF packets go directly to DPDK.

From the physical NIC, the packet data can either go through VF, or through netvsc, but not at the same time.

In practice, the synthetic path carries small number of packets (<1%). The majority of packets go through the VF path.

From: Nandini Rangaswamy <nandini.rangaswamy@broadcom.com>
Sent: Friday, July 26, 2024 3:46 PM
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: users@dpdk.org; Long Li <longli@microsoft.com>
Subject: Re: Query regarding netvsc PMD : VF removal

Hi Stephen and Long,
I am trying to understand the complete packet path and I am confused here.  The synthetic device is being created and managed by a driver in guest kernel space.
Shouldn't the packets be directly DMAed from NIC to VFs in guest. If all packets go through synthetic interface to the VF, then wouldn't kernel get involved here and defeat the purpose of DPDK ?
Please find attached a block diagram that i came up with highlights the devices and drivers and my understanding of their interactions.
I have shown in the diagram that for every VNIC , a synthetic and VF is present . Synthetic interfaces are managed by uio_hv_generic and VFs that appear as PCI devices in the guest kernel are managed by MLNX.
I have also shown that netvsc pmd depends on uio_hv_generic module. The red arrow represents synthetic data path and green arrows represent the accelerated data path.
I am still unclear about pkt flow from NIC to VF through synthetic interface and vice-versa involving netvsc pmd, mlnx driver/PMD.
Can you please clarify this?
Regards,
Nandini


On Thu, Jul 25, 2024 at 2:52 PM Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>> wrote:
On Thu, 25 Jul 2024 11:41:59 -0700
Nandini Rangaswamy <nandini.rangaswamy@broadcom.com<mailto:nandini.rangaswamy@broadcom.com>> wrote:

> Hi Stephen,
> I am using the DPDK 22.11.5 LTS. Should the DPDK code take care of
> unbinding synthetic interface from uio_hv_generic or should it be done by
> the app when it gets hotplug notification?
> Regards,
> Nandini

The synthetic device (netvsc) controlled by uio_hv_generic is never hot plugged.
Only the VF which is mlx device can be removed and added.
The management of the VF for traffic is handled completely internally to
the netvsc device (same as the Linux netdevice). What happens is the
host notifys the guest over vmbus when VF changes. For devices handled
by the DPDK (netvsc PMD), this notification causes it to add/remove use
of the VF.

The VF should not be used directly. The synthetic device (netvsc) routes
traffic over the VF and reads from it when it is present and up.

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

[-- Attachment #2: Type: text/html, Size: 7287 bytes --]

Re: Query regarding netvsc PMD : VF removal

[Nandini Rangaswamy]

[-- Attachment #1.1: Type: text/plain, Size: 5342 bytes --]

Hi Long Li,
Thanks for your feedback . I updated the diagram.
I am still not clear about the different kernel module interactions in the
packet path.
The VFs appear as PCI devices in the guest kernel and are managed by MLNX
drivers.
The synthetic interfaces are being managed by the uio_hv_generic driver in
the kernel.
The netvsc PMD and mlnx PMD in dpdk app are instrumental in sending packets
from app to NIC bypassing the kernel.

Questions about packet path:
1. Is the MLNX driver in the kernel used when packets are sent from NIC to
the kernel without VF ?
2. Do the MLNX driver in kernel and uio_hv_genric driver interact?
3. Do the netvsc and mlnx PMDs interact  with uio_hv_generic driver?

Regards,
Nandini

On Fri, Jul 26, 2024 at 6:14 PM Long Li <longli@microsoft.com> wrote:

> The VF data path is separate from the synthetic data path. The chart
> doesn’t show clearly that all the VF packets go directly to DPDK.
>
>
>
> From the physical NIC, the packet data can either go through VF, or
> through netvsc, but not at the same time.
>
>
>
> In practice, the synthetic path carries small number of packets (<1%). The
> majority of packets go through the VF path.
>
>
>
> *From:* Nandini Rangaswamy <nandini.rangaswamy@broadcom.com>
> *Sent:* Friday, July 26, 2024 3:46 PM
> *To:* Stephen Hemminger <stephen@networkplumber.org>
> *Cc:* users@dpdk.org; Long Li <longli@microsoft.com>
> *Subject:* Re: Query regarding netvsc PMD : VF removal
>
>
>
> Hi Stephen and Long,
>
> I am trying to understand the complete packet path and I am confused
> here.  The synthetic device is being created and managed by a driver in
> guest kernel space.
>
> Shouldn't the packets be directly DMAed from NIC to VFs in guest. If all
> packets go through synthetic interface to the VF, then wouldn't kernel get
> involved here and defeat the purpose of DPDK ?
>
> Please find attached a block diagram that i came up with highlights the
> devices and drivers and my understanding of their interactions.
>
> I have shown in the diagram that for every VNIC , a synthetic and VF is
> present . Synthetic interfaces are managed by uio_hv_generic and VFs that
> appear as PCI devices in the guest kernel are managed by MLNX.
>
> I have also shown that netvsc pmd depends on uio_hv_generic module. The
> red arrow represents synthetic data path and green arrows represent the
> accelerated data path.
>
> I am still unclear about pkt flow from NIC to VF through synthetic
> interface and vice-versa involving netvsc pmd, mlnx driver/PMD.
>
> Can you please clarify this?
>
> Regards,
>
> Nandini
>
>
>
>
>
> On Thu, Jul 25, 2024 at 2:52 PM Stephen Hemminger <
> stephen@networkplumber.org> wrote:
>
> On Thu, 25 Jul 2024 11:41:59 -0700
> Nandini Rangaswamy <nandini.rangaswamy@broadcom.com> wrote:
>
> > Hi Stephen,
> > I am using the DPDK 22.11.5 LTS. Should the DPDK code take care of
> > unbinding synthetic interface from uio_hv_generic or should it be done by
> > the app when it gets hotplug notification?
> > Regards,
> > Nandini
>
> The synthetic device (netvsc) controlled by uio_hv_generic is never hot
> plugged.
> Only the VF which is mlx device can be removed and added.
> The management of the VF for traffic is handled completely internally to
> the netvsc device (same as the Linux netdevice). What happens is the
> host notifys the guest over vmbus when VF changes. For devices handled
> by the DPDK (netvsc PMD), this notification causes it to add/remove use
> of the VF.
>
> The VF should not be used directly. The synthetic device (netvsc) routes
> traffic over the VF and reads from it when it is present and up.
>
>
> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 8261 bytes --]

[-- Attachment #2: Screenshot 2024-08-05 at 2.49.17 PM.png --]
[-- Type: image/png, Size: 176103 bytes --]