Hi everyone,
I'm trying to use a secondary process to rx/tx packets through a NIC and running into segfaults due to some data structures not being populated on the secondary. I've figured out a solution, but it involves the use of internal APIs and I wanted to check with the community if there's a better way to do this.

I'm using DPDK 24.11 LTS for this test. The test code is at https://gist.github.com/praetorian20/0c1b69abbc7843d958da72fdf611a5d7. I'll describe the steps I'm following here.

Primary process:
  1. Command line: sudo LD_LIBRARY_PATH=/path/to/dpdk/shared/libs/ ./test-ethdev --main-lcore 0 --file-prefix=mpdemo --socket-mem=1024 -d /path/to/dpdk/shared/libs/dpdk/pmds-25.0 --proc-type primary
  2. Calls rte_eal_init
  3. Registers message handler using  rte_mp_action_register
  4. Waits indefinitely for CTRL+C
Secondary process:
  1. Command line: Same as primary, except --proc-type=secondary
  2. Calls rte_eal_init
  3. Registers message handler using  rte_mp_action_register
  4. Creates a mempool using rte_pktmbuf_pool_create
  5. Sends request message to primary using  rte_mp_request_async
Primary process (upon receiving the async request):
  1. Look up the mempool created by the secondary
  2. Get port id for a particular ethdev using PCI address
  3. Call rte_eth_dev_configure,  rte_eth_rx_queue_setup,  rte_eth_tx_queue_setup and  rte_eth_dev_start to start the ethdev
  4. Send response to secondary
Secondary process (upon receiving response):
  1. Call  rte_eth_dev_get_port_by_name to fetch port id
  2. Call  rte_eth_dev_attach_secondary and  rte_eth_dev_probing_finish to initialize data structures. Without this the next step has segv because rte_eth_fp_ops.rxq.data is nullptr
  3. Call rte_eth_rx_burst to receive packets

The above works but I have a few questions whether this process can be improved. I'm using an Nvidia ConnectX6 NIC (mlx5 driver) in case that matters.

  • Are there public APIs I can call instead of  rte_eth_dev_attach_secondary and  rte_eth_dev_probing_finish?
  • Is it possible to perform rte_eth_dev_configure, tx/rx queue setup and  rte_eth_dev_start from the secondary process? These functions return various error codes when I try this.
  • When I call rte_eth_dev_start on the primary, it's sending a message to the secondary that goes unhandled. Is this a problem, and possibly the reason for the unpopulated rte_eth_fp_ops on the secondary? This is the error message I see from the primary:
EAL: Fail to recv reply for request /var/run/dpdk/mpdemo/mp_socket_1094695_b7080eff23acd4:common_mlx5_mp
mlx5_net: port 2 failed to request stop/start Rx/Tx (5)

 Thanks for all your help,
Ashish