RE: mbuf refcnt issue

["Lombardo, Ed" <Ed.Lombardo@netscout.com>]

Hi,
I added to my source code:
#define RTE_LITRTE_MBUF_DEBUG 1
#define RTE_LIBRTE_MEMPOOL_DEBUG 1
#define RTE_ENABLE_ASSERT 1

When I send traffic a core file is generated.  It failed on the rte_mbuf_free() when the __rte_mbuf_raw_santy_check() is called.  

Below it shows RTE_ASSERT(rte_mbuf_refcnt_read(m) == 1) that fails because this mbuf has refcnt=2.

rte_mbuf.h shows the RTE_ASSERT check on refcnt.
565 static __rte_always_inline void
 566 __rte_mbuf_raw_sanity_check(__rte_unused const struct rte_mbuf *m)
 567 {
 568     RTE_ASSERT(rte_mbuf_refcnt_read(m) == 1);
 569     RTE_ASSERT(m->next == NULL);
 570     RTE_ASSERT(m->nb_segs == 1);
 571     __rte_mbuf_sanity_check(m, 0);
 572 }

In gdb I inspected the mbuf and it shows refcnt=2.  Which is possible in our multi-threaded design.  So, my understanding was that the refcnt is used to decide when the mbuf(s) should be returned to the mempool.  In our design if the refcnt is 2 then the tx thread is also targeted for this mbuf which may not have been transmitted and freed yet by the Tx thread.  What if I wanted to transmit this mbuf to different tx threads, is even worse for my issue.



(gdb) bt
#0  0x00007fc90548baac in __pthread_kill_implementation () from /lib/libc.so.6
#1  0x00007fc90543e686 in raise () from /lib/libc.so.6
#2  0x00007fc905428833 in abort () from /lib/libc.so.6
#3  0x000000000042e2ba in __rte_panic (funcname=<optimized out>, format=0x338a848 "line %d\tassert \"%s\" failed\n%.0s")
    at ../lib/eal/common/eal_common_debug.c:26
#4  0x0000000001cfa713 in __rte_mbuf_raw_sanity_check (m=0x190ef93c0)
    at /users/lombardoe/70_ISNG_AED_Merge_new/nsagent/lib/dpdk-2411.1_gen_dbg/include/dpdk/rte_mbuf.h:568
#5  rte_mbuf_raw_free (m=0x190ef93c0)
    at /users/lombardoe/70_ISNG_AED_Merge_new/nsagent/lib/dpdk-2411.1_gen_dbg/include/dpdk/rte_mbuf.h:628
#6  rte_pktmbuf_free_seg (m=0x190ef93c0)
    at /users/lombardoe/70_ISNG_AED_Merge_new/nsagent/lib/dpdk-2411.1_gen_dbg/include/dpdk/rte_mbuf.h:1403
#7  rte_pktmbuf_free (m=0x190ef93c0)
    at /users/lombardoe/70_ISNG_AED_Merge_new/nsagent/lib/dpdk-2411.1_gen_dbg/include/dpdk/rte_mbuf.h:1424

I enabled in meson the Atomic guard to mbuf refcnt.
mbuf_refcnt_atomic  true                  [true, false]         Atomically access the mbuf refcnt.
And the rte_build_config.h file shows "#define RTE_MBUF_REFCNT_ATOMIC"

This appears to me as a collision with same mbuf and refcnt in multi-threaded design.  And the rte_pktmbuf_free() does not decrement refcnt if not equal to 1, it tries to free the mbuf.  I don't see how the mbuf(s) can be handled by multiple threads (i.e. tx path and free those not tx).



Thanks,
Ed

-----Original Message-----
From: Lombardo, Ed 
Sent: Tuesday, April 8, 2025 6:34 PM
To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; users@dpdk.org
Subject: RE: mbuf refcnt issue

Hi Dmitry,
I added mbuf refcnt check at the point where we get the packets from DPDK rte_eth_rx_burst() and I am finding many mbufs refcnt set to 2 when I expect them to be 1.  This is more evidence that something has gone wrong within DPDK and the mempool management, perhaps, and I don't know how my setup went into this state.  I changed the traffic type from Telnet to Enterprise traffic and rte_eth_tx_burst() no longer accepts mbufs. 

However, the membuf pool is not depleted after I hit the point the Tx stops.

*** DPDK MEMORY STATS **
Reserved mbufs      :               786432
Available mbufs     :               786432
Allocated mbufs     :                    0
Is Mempool Full     :                    1
Is Mempool Empty    :                0

The receiving of mbufs from rte_eth_rx_burst() that have refcnt == 2 is leading me to believe that our application pipeline stage sets the refcnt to 2 but when the Tx thread is blocked, the mempool still allows the rte_eth_rx_burst() to reuse these mbufs and does not touch the mbuf refcnt?  Or does any mbufs provided by rte_eth_rx_burst() guarantee the mbuf refcnt will be initialized to 1?

I also tried DPDK 23.11.2 and see same issue.

This morning, I found that once the Tx operation stops, I can set the breakpoint just before rte_eth_tx_burst() and change the tx_queue_id to different value then the tx of packets pickup again and then stops shortly.  Once the Tx stops on a tx_queue_id then tx packets fail and never clears up again.

Any thoughts to my issue?

Thanks,
Ed

-----Original Message-----
From: Lombardo, Ed
Sent: Monday, April 7, 2025 3:54 PM
To: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>; users@dpdk.org
Subject: RE: mbuf refcnt issue

Hi Dmitry,
I see issues where spotty mbufs show incorrect refcnt, i.e. should be 1 but I see 2 or 3, or should be 2 but I see 3.  Like the mbuf is returned to mempool and reused by DPDK receive (and appears that mbuf refcnt is not initialized to 1, as I would expect).  I tried to pay attention in the software to eliminate any asynchronous events by multiple threads that perform either rte_mbuf_refcnt_update() and rte_pktmbuf_free() that could cause a weird behavior.  Meaning, on receive of packets the application first calls to request packets is where I increment the mbuf refcnt, then I return the list of packets to the application which will then process the packets and determine which ones to drop (should be very few drops).  In my testing, I am transmitting all the packets received so is much simplified.  From this point on there are multiple threads touching the mbuf refcnt (either DPDK rte_eth_tx_burst()or rte_pktmbuf_free()).

I am testing against DPDK 24.11.1 as generic platform build.

The Pipeline with many threads are explained somewhat below.
1. Rx thread: (two threads 1,2)
    Free mbufs processed by application in Ack ring.
    Receive mbufs from NIC via rte_ring_dequeue_burst()
    Put mbufs into Worker ring.

2. App request for packets: (two threads, 3, 4)
    Process packets from worker ring, and mark the packets to tx (via internal array of pkt info)
    Increment mbuf refcnt by 1 (assume will transmit all packets)
    Add mbufs to Ack ring for freeing later.
    At this point, no other thread should touch these mbufs.

3. App process packets and update internal array of pkt info. (two or more threads, x)
     (here I simulate that the App is processing packets, I mark them all for tx.)

4. App calls to request to send burst of packets: (2 threads, x+2)
    Process packets marked for transmit and put in rte_mbuf pointer arrays.
    Those packets to drop call  rte_pktmbuf_free() (this step is not executed because all packets are marked for transmit)
    When done then add mbufs to tx rings.  Two Tx threads will process packets from their tx ring, one per NIC port.

5. App calls an API to free mbufs in the Ack ring.

* Tx thread (two threads)
    Monitor tx rings and transmit with rte_eth_tx_burst()
    If DPDK fails to send any mbufs, perform rte_pktmbuf_free() on those mbufs that could not be transmitted.  This event is prevented because Tx rate is low.  Wanted to rule this out.

The order of pipeline #4 and #5 cannot be controlled, but counting on the mbuf refcnt to hold correct values so at the end the mbuf is freed to the mempool for DPDK reuse on receive.

I added debug and I see mbuf refcnts are not correct.  I don't believe the mempool is a first in first out so mbuf addresses are showing in random, maybe cached.

Also worth mentioning if the DPDK Tx burst fails to transmit all the supplied mbufs I am planning to issue the rte_pktmbuf_free() for those not transmitted.  Another thread touching mbufs.

Is this multi-thread approach a problem in DPDK?

Thanks,
Ed

    

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
Sent: Friday, April 4, 2025 6:29 PM
To: Lombardo, Ed <Ed.Lombardo@netscout.com>; users@dpdk.org
Subject: Re: mbuf refcnt issue

External Email: This message originated outside of NETSCOUT. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi Ed,

On 05.04.2025 01:00, Lombardo, Ed wrote:
>
> Hi,
>
> I have an application where we receive packets and transmit them.  The 
> packet data is inspected and later mbuf is freed to mempool.
>
> The pipeline is such that the Rx packet mbuf is saved to rx worker 
> ring, then the application threads process the packets and decides if 
> to transmit the packet and if true then increments the mbuf to a value 
> of 2.
>
Do I understand the pipeline correctly?

Rx thread:

     receive mbuf
     put mbuf into the ring
     inspect mbuf
     free mbuf

Worker thread:

     take mbuf from the ring
     if decided to transmit it,
         increment refcnt
         transmit mbuf

If so, there's a problem that after Rx thread puts mbuf into the ring, mbuf is owned by Rx thread and the ring, so its refcnt must be 2 when it enters the ring:

Rx thread:

     receive mbuf
     increment refcnt
     put mbuf into the ring
     inspect mbuf
     free mbuf (just decrements refcnt if > 1)

Worker thread:

     take mbuf from the ring
     if decided to transmit it,
         transmit (or put into the bulk transmitted later)
     else
         free mbuf (just decrements refcnt if > 1)

> The batch of mbufs to transmit are put in a Tx ring queue for the Tx 
> thread to pull from and call the DPDK rte_eth_tx_burst() with the 
> batch of mbufs (limited to 400 mbufs).  In theory the transmit 
> operation will decrement the mbuf refcnt.  In our application we could 
> see the tx of the mbuf followed by another application thread that 
> calls to free the mbufs, or vice versa.  We have no way to synchronize 
> these threads.
>
> Is the mbuf refcnt updates thread safe to allow un-deterministic 
> handling of the mbufs among multiple threads?  The decision to 
> transmit the mbuf and increment the mbuf refcnt and load in the tx 
> ring is completed before the application says it is finished and frees 
> the mbufs.
>
Have you validated this assumption?
If my understanding above is correct, there's no synchronization and thus no guarantees.
>
> I am seeing in my error checking code the mbuf refcnt contains large 
> values like 65520, 65529, 65530, 65534, 65535 in the early pipeline 
> stage refcnt checks.
>
> I read online and in the DPDK code that the mbuf refcnt update is 
> atomic, and is thread safe; so, this is good.
>
> Now this part is unclear to me and that is when the rte_eth_tx_burst() 
> is called and returns the number of packets transmitted , does this
>  mean that transmit of the packets are completed and mbuf refcnt is 
> decremented by 1 on return, or maybe the Tx engine queue is populated 
> and mbuf refcnt is not decremented until it is actually transmitted, 
> or much worse later in time.
>
> Is the DPDK Tx operation intended to be the last stage of any pipeline 
> that will free the mbuf if successfully transmitted?
>
Return from rte_eth_tx_burst() means that mbufs are queued for transmission.
Hardware completes transmission asynchronously.
The next call to rte_eth_tx_burst() will poll HW, learn status of mbufs *previously* queued, and calls rte_pktmbuf_free() for those that are transmitted.
The latter will free mbufs to mempool if and only if refcnt == 1.