Re: Multiprocess App Problems with tx_burst

[Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>]

2025-01-04 14:16 (UTC-0500), Alan Beadle:
> I am setting up one tx queue and one rx queue via the primary process
> init code. The code here resembles the "basic forwarding" sample
> application (in the skeleton/ subdir). Please let me know whether it
> would be possible for each process to use entirely separate queues and
> still pass mbuf pointers around.

It is possible. Mbufs are in memory shared by all processes.
For example, the primary process can setup 2 Rx queues:
to use queue 0 by itself, and to let the secondary process use queue 1.
Steering incoming packets to the right queue (of set of queues)
would be another question, however.

> I will explain the processes and lcores below. First of all though, my
> application uses several types of packets. There are handshake packets
> (acknacks and the like) and data packets. The data packets are
> addressed to specific subsets of my secondary processes (for now just
> 1 or 2 secondary processes exist per machine, but support for even
> more is in principle part of my design). Sometimes the data should
> also be read by other peer processes on the same machine (including
> the daemon/primary process) so I chose to make the mbufs readable
> instead of allocating a duplicate local buffer. It is important that
> mbuf pointers from one process will work in the others. Otherwise all
> of my data would need to be duplicated into non-dpdk shared buffers
> too.
> 
> The first process is the "daemon". This is the primary process. It
> uses DPDK through my shared library (which uses DPDK internally, as
> explained above). The daemon just polls the NIC and periodically
> cleans up my non-DPDK data structures in shared memory. The intent is
> to rely on the daemon to watch for packets during periods of low
> activity and avoid unnecessary CPU usage. When a packet arrives it can
> wake the correct secondary process by finding a futex in shared memory
> for that process. On both machines the daemon is mapped to core 2 with
> the parameter "-l 2".
> 
> The second process is the "server". It uses separate app code from the
> daemon but calls into the same library. Like the daemon, it receives
> and parses packets. The server can originate new data packets, and can
> also reply to inbound data packets with more data packets to be sent
> back to processes on the other machine. It sleeps on a shared futex
> during periods of inactivity. If there were additional secondary
> processes (as is the case on the other machine) it could wake them
> when packets arrive for those other processes, again using futexes in
> shared memory. On both machines this second process is mapped to core
> 4 with the parameter "-l 4".
> 
> The other machine has another secondary process (a third process)
> which is on core 6 with "-l 6". For the purposes of this discussion,
> it behaves similarly to the server process above (sends, receives, and
> sometime sleeps).

So, "deamon" and "server" may try using the same queue sometimes, correct?
Synchronizing all access to the single queue should work in this case.

BTW, rte_eth_tx_burst() returning >0 does not mean the packets have been sent.
It only means they have been enqueued for sending.
At some point the NIC will complete sending,
only then the PMD can free the mbuf (or decrement its reference count).
For most PMDs, this happens on a subsequent call to rte_eth_tx_burst().
Which PMD and HW is it?
Have you tried to print as many stats as possible when rte_eth_tx_burst()
can't consume all packets (rte_eth_stats_get(), rte_eth_xstats_get())?