Re: Multiprocess App Problems with tx_burst

[Alan Beadle <ab.beadle@gmail.com>]

> Thread-unsafe are queues, not calls to rte_eth_rx/tx_burst().
> You can call rte_eth_rx/tx_burst() concurrently without synchronization
> if they operate on different queues.
> Typically you assign each lcore to operate on one or more queues,
> but no queue to be operated by multiple lcores.
> Otherwise you need to synchronize access, which obviously hurts scaling.
> Does this hold in your case?

> Lcore is a thread to which DPDK can dispatch work.
> By default it is pinned to one physical core unless --lcores is used.
> What is lcore-to-CPU mapping in your case?
>
> What is the design of your app regarding processes, lcores, and queues?
> That is: which process runs which lcores and which queues to the latter serve?
>
> P.S. Please don't top-post.

Hi Dmitry,

On one machine I have two processes and on the other there are three
processes. For simplicity we can focus on the former but the only
difference is the number of secondary processes.

It is also worth noting that I am developing a shared library which
uses DPDK internally, rather than the app code directly using DPDK.
Therefore, all of my DPDK command line args for rte_eal_init() are
actually kept in char arrays inside of the library code.

I am setting up one tx queue and one rx queue via the primary process
init code. The code here resembles the "basic forwarding" sample
application (in the skeleton/ subdir). Please let me know whether it
would be possible for each process to use entirely separate queues and
still pass mbuf pointers around.

Before I worry about scaling though, I want correctness first. My
application is more likely to be CPU-bound than network-bound but for
other reasons (this is a research project) I must use user-mode
networking, which is why I am using DPDK.

I will explain the processes and lcores below. First of all though, my
application uses several types of packets. There are handshake packets
(acknacks and the like) and data packets. The data packets are
addressed to specific subsets of my secondary processes (for now just
1 or 2 secondary processes exist per machine, but support for even
more is in principle part of my design). Sometimes the data should
also be read by other peer processes on the same machine (including
the daemon/primary process) so I chose to make the mbufs readable
instead of allocating a duplicate local buffer. It is important that
mbuf pointers from one process will work in the others. Otherwise all
of my data would need to be duplicated into non-dpdk shared buffers
too.

The first process is the "daemon". This is the primary process. It
uses DPDK through my shared library (which uses DPDK internally, as
explained above). The daemon just polls the NIC and periodically
cleans up my non-DPDK data structures in shared memory. The intent is
to rely on the daemon to watch for packets during periods of low
activity and avoid unnecessary CPU usage. When a packet arrives it can
wake the correct secondary process by finding a futex in shared memory
for that process. On both machines the daemon is mapped to core 2 with
the parameter "-l 2".

The second process is the "server". It uses separate app code from the
daemon but calls into the same library. Like the daemon, it receives
and parses packets. The server can originate new data packets, and can
also reply to inbound data packets with more data packets to be sent
back to processes on the other machine. It sleeps on a shared futex
during periods of inactivity. If there were additional secondary
processes (as is the case on the other machine) it could wake them
when packets arrive for those other processes, again using futexes in
shared memory. On both machines this second process is mapped to core
4 with the parameter "-l 4".

The other machine has another secondary process (a third process)
which is on core 6 with "-l 6". For the purposes of this discussion,
it behaves similarly to the server process above (sends, receives, and
sometime sleeps).

Thank you,
-Alan