[PATCH] eal: fix bus cleanup in secondary process

[PATCH] eal: fix bus cleanup in secondary process

[myang]

eal_bus_cleanup has been added in rte_eal_cleanup. But for
secondary process, eal_bus_cleanup will trigger vdev_cleanup
which trigger rte_vdev_driver to remove. Then our crypto devices
will execute ipsec_mb_remove to rte_cryptodev_pmd_destroy.

Finally error logs occur as below:
CRYPTODEV: rte_cryptodev_close() line 1453: Device 0 must be
stopped before closing
EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad
file descriptor
EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
USER1: Create MR request to primary process failed.

Function call trace: rte_eal_cleanup->eal_bus_cleanup->
vdev_cleanup->rte_vdev_driver->ipsec_mb_remove->
1. ipsec_mb_remove->rte_cryptodev_pmd_destroy->
rte_cryptodev_pmd_release_device->rte_cryptodev_close
2. ipsec_mb_remove->ipsec_mb_qp_release->ipsec_mb_secondary_qp_op
->rte_mp_request_async->mp_request_async

Fixes: 1cab1a40ea9b ("bus: cleanup devices on shutdown")
Cc: stable@dpdk.org

Signed-off-by: myang <ming.1.yang@nokia-sbell.com>
---
 lib/eal/linux/eal.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/eal/linux/eal.c b/lib/eal/linux/eal.c
index a6220524a4..eec791ce1e 100644
--- a/lib/eal/linux/eal.c
+++ b/lib/eal/linux/eal.c
@@ -1320,7 +1320,8 @@ rte_eal_cleanup(void)
 	vfio_mp_sync_cleanup();
 #endif
 	rte_mp_channel_cleanup();
-	eal_bus_cleanup();
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		eal_bus_cleanup();
 	rte_trace_save();
 	eal_trace_fini();
 	eal_mp_dev_hotplug_cleanup();
-- 
2.34.1

Re: [PATCH] eal: fix bus cleanup in secondary process

[Stephen Hemminger]

On Thu, 28 Nov 2024 13:48:29 +0800
myang <ming.1.yang@nokia-sbell.com> wrote:

> eal_bus_cleanup has been added in rte_eal_cleanup. But for
> secondary process, eal_bus_cleanup will trigger vdev_cleanup
> which trigger rte_vdev_driver to remove. Then our crypto devices
> will execute ipsec_mb_remove to rte_cryptodev_pmd_destroy.
> 
> Finally error logs occur as below:
> CRYPTODEV: rte_cryptodev_close() line 1453: Device 0 must be
> stopped before closing
> EAL: failed to send to (/tmp/dpdk/l2hicu/mp_socket) due to Bad
> file descriptor
> EAL: Fail to send request /tmp/dpdk/l2hicu/mp_socket:ipsec_mb_mp_msg
> USER1: Create MR request to primary process failed.
> 
> Function call trace: rte_eal_cleanup->eal_bus_cleanup->
> vdev_cleanup->rte_vdev_driver->ipsec_mb_remove->
> 1. ipsec_mb_remove->rte_cryptodev_pmd_destroy->
> rte_cryptodev_pmd_release_device->rte_cryptodev_close
> 2. ipsec_mb_remove->ipsec_mb_qp_release->ipsec_mb_secondary_qp_op
> ->rte_mp_request_async->mp_request_async  
> 
> Fixes: 1cab1a40ea9b ("bus: cleanup devices on shutdown")
> Cc: stable@dpdk.org
> 
> Signed-off-by: myang <ming.1.yang@nokia-sbell.com>

There is was a reason for calling cleanup on shutdown.
It looks like more of a bug in the crypto device not the EAL.


Freebsd and Windows also calls eal_bus_cleanup.