Re: Multiprocess App Problems with tx_burst

[Alan Beadle <ab.beadle@gmail.com>]

Hi everyone,

I'm still stuck on this. Most likely I am doing something wrong in the
initialization phase. I am trying to follow the standard code example
for symmetric multi-process, but since my code is doing very different
things from this example I cannot even begin to guess where I am going
wrong. I do not even know if what I am trying to do is permissible in
the DPDK API.

It would be very helpful if someone could provide an initialization
checklist for my use case (below).

As explained previously, I have several separately launched processes.
These processes already share a memory region for local communication.
I want all of these processes to have equal ability to read incoming
packets, place pointers to the mbufs in shared memory, and wake each
other up when packets destined for a particular one of these processes
arrives. I have one X550-T2 NIC and I am only using one one of the
physical ports. It connects to a second machine which is doing
essentially the same thing, running the same DPDK code.

In summary, each of my multiple processes should all be able to
equally receive packets of behalf of each other, and leave pointers to
rx'ed mbufs for each other in shared memory according to which process
the mbuf was destined for. Outbound packets may also be shared with
local peer processes for reading. In order to do this I am also
bumping the mbuf refcount until the peer process has read the mbuf.

I already thought I had all of this working fine, but it turns out
that they were all taking turns on the same physical core, and
everything breaks when they are run concurrently on separate cores. I
have seen conflicting information in online threads about the thread
safety of the various DPDK functions that I am using. I tried adding
synchronization around DPDK allocation and tx/rx bursts to no avail.
My code detects weird errors where either mbufs contain unexpected
things (invalid reuse?) or tx bursts start to fail in one of the
processes.

Frankly I also feel very confused about how ports, queues, mempools,
etc work and I suspect that a lot of what I have been reading is
outdated or faulty information.

Any guidance at all would be greatly appreciated!
-Alan

On Tue, Dec 31, 2024 at 12:49 PM Alan Beadle <ab.beadle@gmail.com> wrote:
>
> Hi everyone,
>
> I am working on a multi-process DPDK application. It uses one NIC, one
> port, and both separate processes send as well as receive, and they
> share memory for synchronization and IPC.
>
> I had previously made a mistake in setting up the lcores, and all of
> the processes were assigned to the same physical core. This seems to
> have concealed some DPDK thread safety issues which I am now dealing
> with.
>
> I understand that rte_eth_tx_burst() and rte_eth_rx_burst() are not
> thread safe. Previously I did not have any synchronization around
> these functions. Now that I am successfully using separate cores, I
> have added a shared spinlock around all invocations of these
> functions, as well as around all mbuf frees and allocations.
>
> However, when my code sends a packet, it checks the return value of
> rte_eth_tx_burst() to ensure that the packet was actually sent. If it
> fails to send, my app exits with an error. This was not previously
> happening, but now it happens every time I run it. I thought this was
> due to the lack of synchronization but it is still happening after I
> added the lock. Why would rte_eth_tx_burst() be failing now?
>
> Thank you,
> -Alan