Re: Multiprocess App Problems with tx_burst

[Alan Beadle <ab.beadle@gmail.com>]

> So, "deamon" and "server" may try using the same queue sometimes, correct?
> Synchronizing all access to the single queue should work in this case.

That is correct.

> BTW, rte_eth_tx_burst() returning >0 does not mean the packets have been sent.
> It only means they have been enqueued for sending.
> At some point the NIC will complete sending,
> only then the PMD can free the mbuf (or decrement its reference count).
> For most PMDs, this happens on a subsequent call to rte_eth_tx_burst().
> Which PMD and HW is it?

Here is the output of 'dpdk-devbind.py --status':

Network devices using DPDK-compatible driver
============================================
0000:65:00.1 'Ethernet Controller 10G X550T 1563' drv=vfio-pci
unused=uio_pci_generic


> Have you tried to print as many stats as possible when rte_eth_tx_burst()
> can't consume all packets (rte_eth_stats_get(), rte_eth_xstats_get())?

In setting this up, I discovered that this error only occurs when the
primary process on the other host exits (due to an error) or is not
initially running (the NIC is "down" in this case?). It happens
consistently when I only launch the processes on one of the two
machines. ***But*** counterintuitively, it looks like packets are
successfully "sent" by the daemon until the other process begins to
run. In case it is useful, I summarize the stats for this case below.

Note that I am also seeing another error. Sometimes, rather than tx
failing, my app detects incorrect/corrupted mbuf contents and exits
immediately. It appears that mbufs are being re-allocated when they
should not be. I thought I had finally solved this (see my earlier
threads) but with multi-core concurrency this problem has returned. It
is very possible that this error is somewhere in my own library code,
as it looks like the accompanying non-DPDK structures are also being
corrupted (probably first).

For background, I maintain a hash table of header structs to track
individual mbufs. The sequence numbers in the headers should match
those contained in the mbuf's payload. This check is failing after a
few hundred successful data messages have been exchanged between the
hosts. The sequence number in the mbuf shows that it is in the wrong
hash bucket, and the sequence number in the header is a large
corrupted value which is out of range for my sequence numbers (and
also not matching the bucket).

Back to the issue of failed tx bursts: Here are the stats I observed
after a packet failed to send from the daemon (after only launching
the primary+secondary processes on one of the machines). This failure
occurred after the daemon had successfully "sent" hundreds of
handshake packets (to nowhere, presumably?), and the failure occurred
as soon as the second process had finished initialization:

ipackets:0, opackets:0, ibytes:0, obytes:0, ierrors:0, oerrors:0
Got 146 xstats
Port:0, tx_q0_packets:1138
Port:0, tx_q0_bytes:125180
Port:0, mac_local_errors:2
Port:0, out_pkts_untagged:5
(All other stats had a value of 0 and are omitted).

I will continue investigating the corruption bug in the (likely) case
that it is in my library code. In the meantime please let me know if I
am using DPDK incorrectly. Thank you again!
-Alan