ethdev use from secondary process

ethdev use from secondary process

[Ashish Sadanandan]

[-- Attachment #1: Type: text/plain, Size: 2734 bytes --]

Hi everyone,
I'm trying to use a secondary process to rx/tx packets through a NIC and
running into segfaults due to some data structures not being populated on
the secondary. I've figured out a solution, but it involves the use of
internal APIs and I wanted to check with the community if there's a better
way to do this.

I'm using DPDK 24.11 LTS for this test. The test code is at
https://gist.github.com/praetorian20/0c1b69abbc7843d958da72fdf611a5d7. I'll
describe the steps I'm following here.

Primary process:

   1. Command line: sudo LD_LIBRARY_PATH=/path/to/dpdk/shared/libs/
   ./test-ethdev --main-lcore 0 --file-prefix=mpdemo --socket-mem=1024 -d
   /path/to/dpdk/shared/libs/dpdk/pmds-25.0 --proc-type primary
   2. Calls rte_eal_init
   3. Registers message handler using  rte_mp_action_register
   4. Waits indefinitely for CTRL+C

Secondary process:

   1. Command line: Same as primary, except --proc-type=secondary
   2. Calls rte_eal_init
   3. Registers message handler using  rte_mp_action_register
   4. Creates a mempool using rte_pktmbuf_pool_create
   5. Sends request message to primary using  rte_mp_request_async

Primary process (upon receiving the async request):

   1. Look up the mempool created by the secondary
   2. Get port id for a particular ethdev using PCI address
   3. Call rte_eth_dev_configure,  rte_eth_rx_queue_setup,
rte_eth_tx_queue_setup
   and  rte_eth_dev_start to start the ethdev
   4. Send response to secondary

Secondary process (upon receiving response):

   1. Call  rte_eth_dev_get_port_by_name to fetch port id
   2. Call  rte_eth_dev_attach_secondary and  rte_eth_dev_probing_finish to
   initialize data structures. Without this the next step has segv
   because rte_eth_fp_ops.rxq.data is nullptr
   3. Call rte_eth_rx_burst to receive packets


The above works but I have a few questions whether this process can be
improved. I'm using an Nvidia ConnectX6 NIC (mlx5 driver) in case that
matters.


   - Are there public APIs I can call instead of
   rte_eth_dev_attach_secondary and  rte_eth_dev_probing_finish?
   - Is it possible to perform rte_eth_dev_configure, tx/rx queue setup and
   rte_eth_dev_start from the secondary process? These functions return
   various error codes when I try this.
   - When I call rte_eth_dev_start on the primary, it's sending a message
   to the secondary that goes unhandled. Is this a problem, and possibly the
   reason for the unpopulated rte_eth_fp_ops on the secondary? This is the
   error message I see from the primary:

EAL: Fail to recv reply for request
> /var/run/dpdk/mpdemo/mp_socket_1094695_b7080eff23acd4:common_mlx5_mp
> mlx5_net: port 2 failed to request stop/start Rx/Tx (5)


 Thanks for all your help,
Ashish

[-- Attachment #2: Type: text/html, Size: 3911 bytes --]

Re: ethdev use from secondary process

[Stephen Hemminger]

On Mon, 2 Jun 2025 14:35:04 -0600
Ashish Sadanandan <ashish.sadanandan@gmail.com> wrote:

> Hi everyone,
> I'm trying to use a secondary process to rx/tx packets through a NIC and
> running into segfaults due to some data structures not being populated on
> the secondary. I've figured out a solution, but it involves the use of
> internal APIs and I wanted to check with the community if there's a better
> way to do this.
> 
> I'm using DPDK 24.11 LTS for this test. The test code is at
> https://gist.github.com/praetorian20/0c1b69abbc7843d958da72fdf611a5d7. I'll
> describe the steps I'm following here.
> 
> Primary process:
> 
>    1. Command line: sudo LD_LIBRARY_PATH=/path/to/dpdk/shared/libs/
>    ./test-ethdev --main-lcore 0 --file-prefix=mpdemo --socket-mem=1024 -d
>    /path/to/dpdk/shared/libs/dpdk/pmds-25.0 --proc-type primary
>    2. Calls rte_eal_init
>    3. Registers message handler using  rte_mp_action_register
>    4. Waits indefinitely for CTRL+C
> 
> Secondary process:
> 
>    1. Command line: Same as primary, except --proc-type=secondary
>    2. Calls rte_eal_init
>    3. Registers message handler using  rte_mp_action_register
>    4. Creates a mempool using rte_pktmbuf_pool_create
>    5. Sends request message to primary using  rte_mp_request_async
> 
> Primary process (upon receiving the async request):
> 
>    1. Look up the mempool created by the secondary
>    2. Get port id for a particular ethdev using PCI address
>    3. Call rte_eth_dev_configure,  rte_eth_rx_queue_setup,
> rte_eth_tx_queue_setup
>    and  rte_eth_dev_start to start the ethdev
>    4. Send response to secondary
> 
> Secondary process (upon receiving response):
> 
>    1. Call  rte_eth_dev_get_port_by_name to fetch port id
>    2. Call  rte_eth_dev_attach_secondary and  rte_eth_dev_probing_finish to
>    initialize data structures. Without this the next step has segv
>    because rte_eth_fp_ops.rxq.data is nullptr
>    3. Call rte_eth_rx_burst to receive packets
> 
> 
> The above works but I have a few questions whether this process can be
> improved. I'm using an Nvidia ConnectX6 NIC (mlx5 driver) in case that
> matters.
> 
> 
>    - Are there public APIs I can call instead of
>    rte_eth_dev_attach_secondary and  rte_eth_dev_probing_finish?
>    - Is it possible to perform rte_eth_dev_configure, tx/rx queue setup and
>    rte_eth_dev_start from the secondary process? These functions return
>    various error codes when I try this.
>    - When I call rte_eth_dev_start on the primary, it's sending a message
>    to the secondary that goes unhandled. Is this a problem, and possibly the
>    reason for the unpopulated rte_eth_fp_ops on the secondary? This is the
>    error message I see from the primary:
> 
> EAL: Fail to recv reply for request
> > /var/run/dpdk/mpdemo/mp_socket_1094695_b7080eff23acd4:common_mlx5_mp
> > mlx5_net: port 2 failed to request stop/start Rx/Tx (5)  
> 
> 
>  Thanks for all your help,
> Ashish

Look at the mp examples.
You should not be calling attach and probing finish in the applications.
The primary must start first and has to always be running.


Also, some device do not support primary/secondary process check the documentation.