Concurrent invocations of the dpdk-dumpcap tool

Concurrent invocations of the dpdk-dumpcap tool

[Isaac Boukris]

Hello Stephen, all,

Currently the dumpcap tool fails if another instance of it captures on
the same interface, and the primary process will log:
"pdump_register_rx_callbacks(): rx callback for port=0 queue=0,
already exists"

However if each instance runs on a different port (-i), then they seem
to both work fine and capture packets until you stop the second one,
at which point the first one breaks with:

Packets captured: 196094 Primary process is no longer active, exiting...
EAL: failed to send to (/tmp/dpdk/rte/mp_socket) due to Connection refused
EAL: Fail to send request /tmp/dpdk/rte/mp_socket:mp_pdump
pdump_prepare_client_request(): client request for pdump enable/disable failed
RING: Cannot free ring, not created with rte_ring_create()

At the same time, the primary process crashes with (which is likely
the cause of the above):

#3  <signal handler called>
#4  0x000000000168437d in bucket_dequeue ()
#5  0x0000000002269c26 in rte_pktmbuf_copy ()
#6  0x00000000021a4939 in rte_pcapng_copy ()
#7  0x000000000209d890 in pdump_copy ()
#8  0x000000000209e350 in pdump_rx ()
#9  0x0000000002240bb5 in rte_eth_call_rx_callbacks ()
#10 0x00000000014b04bf in rte_eth_rx_burst (port_id=0, queue_id=0,
rx_pkts=0x7f00c3fb9c80, nb_pkts=128) at
/usr/local/include/rte_ethdev.h:5906

I'm testing with the 22.11.3 version code (latest LTS) but there don't
seem to be significant changes in master.

Given the above I had the hunch the ring names collide in shared
memory, so I changed both the ring and the pool names in the dumpcap
tool to include the pid, and with that the above scenario works fine
as expected, is that a proper fix?

btw, looking at related code i wondered why do we require the ring to
be MC, while we probably need MP since the app might have several
producers, is the dumpcap tool the only consumer?

In general the fact that if the tool dies abnormally or crashes,
requires restarting the primary process isn't great. While we can try
to catch more signals in the tool to avoid it, perhaps we can also
consider a -F option which disconnects stalled clients (by sending a
"disable" message). Another improvement could be to detect on the
primary process that the secondary one died and remove the
rxtx_callback so another client could connect.

Thanks!

Re: Concurrent invocations of the dpdk-dumpcap tool

[Stephen Hemminger]

On Sun, 17 Sep 2023 00:11:31 +0300
Isaac Boukris <iboukris@gmail.com> wrote:

> I'm testing with the 22.11.3 version code (latest LTS) but there don't
> seem to be significant changes in master.
> 
> Given the above I had the hunch the ring names collide in shared
> memory, so I changed both the ring and the pool names in the dumpcap
> tool to include the pid, and with that the above scenario works fine
> as expected, is that a proper fix?

If you have two processes trying to capture on same interface it
isn't going to work because both will be fighting over the same
ring of packets. To make it work would require more extensive changes
where multiple copies of packet are made.

Re: Concurrent invocations of the dpdk-dumpcap tool

[Isaac Boukris]

On Sun, Sep 17, 2023 at 5:01 AM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Sun, 17 Sep 2023 00:11:31 +0300
> Isaac Boukris <iboukris@gmail.com> wrote:
>
> > I'm testing with the 22.11.3 version code (latest LTS) but there don't
> > seem to be significant changes in master.
> >
> > Given the above I had the hunch the ring names collide in shared
> > memory, so I changed both the ring and the pool names in the dumpcap
> > tool to include the pid, and with that the above scenario works fine
> > as expected, is that a proper fix?
>
> If you have two processes trying to capture on same interface it
> isn't going to work because both will be fighting over the same
> ring of packets. To make it work would require more extensive changes
> where multiple copies of packet are made.

Specifically, the scenario of the crash is when trying to capture on
different interfaces. By running more tests it seems that what fixes
the crash is actually adding the pid to the name of the mem-pool (not
the ring). Should I submit a patch with that?