Does the mlx5 NIC support reloading

Does the mlx5 NIC support reloading

[jinag]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

Hi 
I am verifying the reload function of the mlx5 nic based on dpdk 19.11:
echo 1 > /sys/bus/pci/devices/$pci_address/remove
echo 1 > /sys/bus/pci/rescan
rte_bus_probe();  
rte_eth_dev_stop(); 
rte_eth_dev_start(); 
    net_mlx5: port 0 TX queue 0 CQ creation failure
    net_mlx5: port 0 TX queue allocation failed: cannot allocate memory
The nic cannot be reinitialized.


I am not sure if the above operation is correct. Could you please tell me if the mlx5 nic supports reloading(for example, the network card is down during normal  operation) and which dpdk functions need to be called.


Thanks!






[-- Attachment #2: Type: text/html, Size: 2112 bytes --]

RE: Does the mlx5 NIC support reloading

[Matan Azrad]

[-- Attachment #1: Type: text/plain, Size: 1356 bytes --]

Hi Jinag

After plugging out the device from the bus, you need to get the event RTE_ETH_EVENT_INTR_RMV.
You need to listen to this event and close the port when you see it.

After plugging in the device, you need to scan the bus again and attach the mlx5 device, so it will be probed again and a new ethdev port will be created.

Then, you need to reconfigure the port, as regular, and reuse the device.

Thanks,
Matan

From: jinag <15720603159@163.com>
Sent: Monday, 14 August 2023 6:07
To: users@dpdk.org; Matan Azrad <matan@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>
Subject: Does the mlx5 NIC support reloading

External email: Use caution opening links or attachments

Hi
I am verifying the reload function of the mlx5 nic based on dpdk 19.11:
echo 1 > /sys/bus/pci/devices/$pci_address/remove
echo 1 > /sys/bus/pci/rescan
rte_bus_probe();
rte_eth_dev_stop();
rte_eth_dev_start();
    net_mlx5: port 0 TX queue 0 CQ creation failure
    net_mlx5: port 0 TX queue allocation failed: cannot allocate memory
The nic cannot be reinitialized.

I am not sure if the above operation is correct. Could you please tell me if the mlx5 nic supports reloading(for example, the network card is down during normal  operation) and which dpdk functions need to be called.

Thanks!




[-- Attachment #2: Type: text/html, Size: 8903 bytes --]

Re:RE: Does the mlx5 NIC support reloading

[jinag]

[-- Attachment #1: Type: text/plain, Size: 1504 bytes --]

Hi Matan:
Could you please provide me a example?
Thanks a lot



At 2023-08-14 14:41:22, "Matan Azrad" <matan@nvidia.com> wrote:

Hi Jinag

 

After plugging out the device from the bus, you need to get the event RTE_ETH_EVENT_INTR_RMV.

You need to listen to this event and close the port when you see it.

 

After plugging in the device, you need to scan the bus again and attach the mlx5 device, so it will be probed again and a new ethdev port will be created.

 

Then, you need to reconfigure the port, as regular, and reuse the device.

 

Thanks,

Matan

 

From: jinag <15720603159@163.com>
Sent: Monday, 14 August 2023 6:07
To: users@dpdk.org; Matan Azrad <matan@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>
Subject: Does the mlx5 NIC support reloading

 

|

External email: Use caution opening links or attachments

|

 

Hi 

I am verifying the reload function of the mlx5 nic based on dpdk 19.11:

echo 1 > /sys/bus/pci/devices/$pci_address/remove

echo 1 > /sys/bus/pci/rescan

rte_bus_probe();  

rte_eth_dev_stop(); 

rte_eth_dev_start(); 

    net_mlx5: port 0 TX queue 0 CQ creation failure

    net_mlx5: port 0 TX queue allocation failed: cannot allocate memory

The nic cannot be reinitialized.

 

I am not sure if the above operation is correct. Could you please tell me if the mlx5 nic supports reloading(for example, the network card is down during normal  operation) and which dpdk functions need to be called.

 

Thanks!

 

 

 

[-- Attachment #2: Type: text/html, Size: 7982 bytes --]

RE: Re:RE: Does the mlx5 NIC support reloading

[Matan Azrad]

[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]

You can see the failsafe driver.
https://git.dpdk.org/dpdk/tree/drivers/net/failsafe?h=v19.11

Generally, it gets 2 devices.
Once the primary device gets the RMV event it closes it and switch the control to a secondary device.
Meanwhile the drive periodically checks if the primary device is back in the bus and if so, it probe the device again and reconfigure it to be able to switch the control back to the primary device.

Thanks,
Matan


From: jinag <15720603159@163.com>
Sent: Monday, 14 August 2023 13:06
To: Matan Azrad <matan@nvidia.com>
Cc: users@dpdk.org; Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>
Subject: Re:RE: Does the mlx5 NIC support reloading

External email: Use caution opening links or attachments

Hi Matan:
Could you please provide me a example?
Thanks a lot


At 2023-08-14 14:41:22, "Matan Azrad" <matan@nvidia.com<mailto:matan@nvidia.com>> wrote:
Hi Jinag

After plugging out the device from the bus, you need to get the event RTE_ETH_EVENT_INTR_RMV.
You need to listen to this event and close the port when you see it.

After plugging in the device, you need to scan the bus again and attach the mlx5 device, so it will be probed again and a new ethdev port will be created.

Then, you need to reconfigure the port, as regular, and reuse the device.

Thanks,
Matan

From: jinag <15720603159@163.com<mailto:15720603159@163.com>>
Sent: Monday, 14 August 2023 6:07
To: users@dpdk.org<mailto:users@dpdk.org>; Matan Azrad <matan@nvidia.com<mailto:matan@nvidia.com>>; Shahaf Shuler <shahafs@nvidia.com<mailto:shahafs@nvidia.com>>; Slava Ovsiienko <viacheslavo@nvidia.com<mailto:viacheslavo@nvidia.com>>
Subject: Does the mlx5 NIC support reloading

External email: Use caution opening links or attachments

Hi
I am verifying the reload function of the mlx5 nic based on dpdk 19.11:
echo 1 > /sys/bus/pci/devices/$pci_address/remove
echo 1 > /sys/bus/pci/rescan
rte_bus_probe();
rte_eth_dev_stop();
rte_eth_dev_start();
    net_mlx5: port 0 TX queue 0 CQ creation failure
    net_mlx5: port 0 TX queue allocation failed: cannot allocate memory
The nic cannot be reinitialized.

I am not sure if the above operation is correct. Could you please tell me if the mlx5 nic supports reloading(for example, the network card is down during normal  operation) and which dpdk functions need to be called.

Thanks!




[-- Attachment #2: Type: text/html, Size: 17557 bytes --]

RE: Does the mlx5 NIC support reloading

[Slava Ovsiienko]

[-- Attachment #1: Type: text/plain, Size: 2120 bytes --]

Hi, Jinag

5 cents to add: please see how FailSafe PMD handles device removal events and probes the devices periodically.
This is good example to understand how device hotplug works (Failsafe uses rte_eal_hotplug_add())
Also,  to add the device the previous instance should closed (RTE_ETH_EVENT_INTR_RMV) shoud be handled accordingly.

With best regards,
Slava

From: Matan Azrad <matan@nvidia.com>
Sent: Monday, August 14, 2023 9:41 AM
To: jinag <15720603159@163.com>; users@dpdk.org; Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>
Subject: RE: Does the mlx5 NIC support reloading

Hi Jinag

After plugging out the device from the bus, you need to get the event RTE_ETH_EVENT_INTR_RMV.
You need to listen to this event and close the port when you see it.

After plugging in the device, you need to scan the bus again and attach the mlx5 device, so it will be probed again and a new ethdev port will be created.

Then, you need to reconfigure the port, as regular, and reuse the device.

Thanks,
Matan


From: jinag <15720603159@163.com<mailto:15720603159@163.com>>
Sent: Monday, 14 August 2023 6:07
To: users@dpdk.org<mailto:users@dpdk.org>; Matan Azrad <matan@nvidia.com<mailto:matan@nvidia.com>>; Shahaf Shuler <shahafs@nvidia.com<mailto:shahafs@nvidia.com>>; Slava Ovsiienko <viacheslavo@nvidia.com<mailto:viacheslavo@nvidia.com>>
Subject: Does the mlx5 NIC support reloading

External email: Use caution opening links or attachments

Hi
I am verifying the reload function of the mlx5 nic based on dpdk 19.11:
echo 1 > /sys/bus/pci/devices/$pci_address/remove
echo 1 > /sys/bus/pci/rescan
rte_bus_probe();
rte_eth_dev_stop();
rte_eth_dev_start();
    net_mlx5: port 0 TX queue 0 CQ creation failure
    net_mlx5: port 0 TX queue allocation failed: cannot allocate memory
The nic cannot be reinitialized.

I am not sure if the above operation is correct. Could you please tell me if the mlx5 nic supports reloading(for example, the network card is down during normal  operation) and which dpdk functions need to be called.

Thanks!




[-- Attachment #2: Type: text/html, Size: 11311 bytes --]

Re:RE: Does the mlx5 NIC support reloading

[jinag]

[-- Attachment #1: Type: text/plain, Size: 2792 bytes --]

Hi Slava, matan:
Thanks for the info! 


I read the failsafe code briefly. here are my superficial undestanding. if there is anything worng, please help correct
failsafe seems to be a virtual device, which can be created by using --vdev args when dpdk app is started:
./app --vdev 'net_failsafe0,mac=xx:xx, dev(84:00:00)'
the above command will create a net_failsafe0 device, its sub device is 84:00:00, this can be a mlx5 physical NIC. 
By adding the mlx5 physical NIC as a failsafe subdevice, the mlx5 NIC can hotplug without requiring me to modify my app? 
I also found the dpdk community documentation:
53. Fail-safe poll mode driver library — Data Plane Development Kit 20.02.1 documentation (dpdk.org)




At 2023-08-17 13:13:18, "Slava Ovsiienko" <viacheslavo@nvidia.com> wrote:

Hi, Jinag

 

5 cents to add: please see how FailSafe PMD handles device removal events and probes the devices periodically.

This is good example to understand how device hotplug works (Failsafe uses rte_eal_hotplug_add())

Also,  to add the device the previous instance should closed (RTE_ETH_EVENT_INTR_RMV) shoud be handled accordingly.

 

With best regards,
Slava

 

From: Matan Azrad <matan@nvidia.com>
Sent: Monday, August 14, 2023 9:41 AM
To: jinag <15720603159@163.com>; users@dpdk.org; Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>
Subject: RE: Does the mlx5 NIC support reloading

 

Hi Jinag

 

After plugging out the device from the bus, you need to get the event RTE_ETH_EVENT_INTR_RMV.

You need to listen to this event and close the port when you see it.

 

After plugging in the device, you need to scan the bus again and attach the mlx5 device, so it will be probed again and a new ethdev port will be created.

 

Then, you need to reconfigure the port, as regular, and reuse the device.

 

Thanks,

Matan

 

 

From: jinag <15720603159@163.com>
Sent: Monday, 14 August 2023 6:07
To:users@dpdk.org; Matan Azrad <matan@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>
Subject: Does the mlx5 NIC support reloading

 

|

External email: Use caution opening links or attachments

|

 

Hi 

I am verifying the reload function of the mlx5 nic based on dpdk 19.11:

echo 1 > /sys/bus/pci/devices/$pci_address/remove

echo 1 > /sys/bus/pci/rescan

rte_bus_probe();  

rte_eth_dev_stop(); 

rte_eth_dev_start(); 

    net_mlx5: port 0 TX queue 0 CQ creation failure

    net_mlx5: port 0 TX queue allocation failed: cannot allocate memory

The nic cannot be reinitialized.

 

I am not sure if the above operation is correct. Could you please tell me if the mlx5 nic supports reloading(for example, the network card is down during normal  operation) and which dpdk functions need to be called.

 

Thanks!

 

 

 

[-- Attachment #2: Type: text/html, Size: 11232 bytes --]

Does the mlx5 NIC support reloading

[jinag]

[-- Attachment #1: Type: text/plain, Size: 275 bytes --]

Hi 
I am verifying the reload function of the mlx5 nic based on dpdk 19.11:
echo 1 > /sys/bus/pci/devices/$pci_address/remove
echo 1 > /sys/bus/pci/rescan
rte_bus_probe();   //
rte_eth_dev_stop(); //
rte_eth_dev_start(); 
    net_mlx5: port 0 TX queue 0 CQ creation failure


[-- Attachment #2: Type: text/html, Size: 702 bytes --]