Re: Multiprocess App Problems with tx_burst

[Alan Beadle <ab.beadle@gmail.com>]

> Note that I am also seeing another error. Sometimes, rather than tx
> failing, my app detects incorrect/corrupted mbuf contents and exits
> immediately. It appears that mbufs are being re-allocated when they
> should not be. I thought I had finally solved this (see my earlier
> threads) but with multi-core concurrency this problem has returned. It
> is very possible that this error is somewhere in my own library code,
> as it looks like the accompanying non-DPDK structures are also being
> corrupted (probably first).
>
> For background, I maintain a hash table of header structs to track
> individual mbufs. The sequence numbers in the headers should match
> those contained in the mbuf's payload. This check is failing after a
> few hundred successful data messages have been exchanged between the
> hosts. The sequence number in the mbuf shows that it is in the wrong
> hash bucket, and the sequence number in the header is a large
> corrupted value which is out of range for my sequence numbers (and
> also not matching the bucket).
>

There is definitely something going wrong with the mbuf allocator.
Each run results in such different errors that it is difficult to add
instrumentation for a specific one, but one frequent error is that a
newly allocated mbuf already has a refcnt of 2, and contains data that
I am still using elsewhere. At each call to rte_pktmbuf_alloc() (with
locks around it) I immediately do a rte_mbuf_refcnt_read() and ensure
that it is 1. Sometimes it is 2. This should never occur and I believe
it proves that DPDK is not working as expected here for some reason.

-Alan