Force driver to free sent buffers?

Force driver to free sent buffers?

[Alan Beadle]

Hi everyone,

Based on my thread from last week, I am using rte_mbuf_refcnt_update()
to increment the mbuf refcnts so that the driver will not free buffers
before my local readers are done with them.
I am also maintaining a queue of mbufs that local readers are finished
with, and periodically using rte_mbuf_refcnt_read() to see if the
refcnt is 1 yet, so that they can be freed or reused.

An aside: Is there any problem with reuse of mbufs? Without actually
freeing and re-allocating it, that is? I plan to do this in order to
avoid having to link new mbufs to other reused data structures in my
application.

The current problem is as follows: My application has a separate
unrelated heap (special purpose and bounded in size) and this heap
runs out of memory after a brief time because of having to track so
many unfreed mbufs. It appears that none of the mbufs are being freed
by the driver after being sent, and it looks like it won't free them
until it runs out of descriptors, which is too long for my situation.

I have confirmed that the packets are being sent (received on other PC
with wireshark) and should be freeable. Is there a way to force the
driver to free sent mbufs earlier than when it runs out of
descriptors?

Thanks,
-Alan

Re: Force driver to free sent buffers?

[Stephen Hemminger]

On Sun, 6 Oct 2024 12:38:56 -0400
Alan Beadle <ab.beadle@gmail.com> wrote:

> Hi everyone,
> 
> Based on my thread from last week, I am using rte_mbuf_refcnt_update()
> to increment the mbuf refcnts so that the driver will not free buffers
> before my local readers are done with them.
> I am also maintaining a queue of mbufs that local readers are finished
> with, and periodically using rte_mbuf_refcnt_read() to see if the
> refcnt is 1 yet, so that they can be freed or reused.
> 
> An aside: Is there any problem with reuse of mbufs? Without actually
> freeing and re-allocating it, that is? I plan to do this in order to
> avoid having to link new mbufs to other reused data structures in my
> application.
> 
> The current problem is as follows: My application has a separate
> unrelated heap (special purpose and bounded in size) and this heap
> runs out of memory after a brief time because of having to track so
> many unfreed mbufs. It appears that none of the mbufs are being freed
> by the driver after being sent, and it looks like it won't free them
> until it runs out of descriptors, which is too long for my situation.
> 
> I have confirmed that the packets are being sent (received on other PC
> with wireshark) and should be freeable. Is there a way to force the
> driver to free sent mbufs earlier than when it runs out of
> descriptors?
> 
> Thanks,
> -Alan

Drivers are free to hold onto transmitted mbuf's indefinitely.
Since DPDK does not use interrupts for normal Tx, the driver has to defer
doing cleaning up either on next Tx or sometimes Rx. There is a Tx threshold
that may impact some drivers, but not all.

Re: Force driver to free sent buffers?

[Alan Beadle]

On Sun, Oct 6, 2024 at 6:18 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> Drivers are free to hold onto transmitted mbuf's indefinitely.
> Since DPDK does not use interrupts for normal Tx, the driver has to defer
> doing cleaning up either on next Tx or sometimes Rx. There is a Tx threshold
> that may impact some drivers, but not all.

Stephen,

How would I attempt to set this threshold to see if it impacts the
driver that I am using? I cannot find any EAL parameters that seem to
correspond to this.

Having to re-allocate a new mbuf every time will hurt performance,
especially since most of the packet header values will not be changing
across packets but will need to be reinitialized in every newly
allocated mbuf. I had really hoped to recycle a set of <100 or so
mbufs.

-Alan

Re: Force driver to free sent buffers?

[Alan Beadle]

On Thu, Oct 10, 2024 at 10:12 AM Alan Beadle <ab.beadle@gmail.com> wrote:
>
> On Sun, Oct 6, 2024 at 6:18 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > Drivers are free to hold onto transmitted mbuf's indefinitely.
> > Since DPDK does not use interrupts for normal Tx, the driver has to defer
> > doing cleaning up either on next Tx or sometimes Rx. There is a Tx threshold
> > that may impact some drivers, but not all.
>
> Stephen,
>
> How would I attempt to set this threshold to see if it impacts the
> driver that I am using? I cannot find any EAL parameters that seem to
> correspond to this.
>
> Having to re-allocate a new mbuf every time will hurt performance,
> especially since most of the packet header values will not be changing
> across packets but will need to be reinitialized in every newly
> allocated mbuf. I had really hoped to recycle a set of <100 or so
> mbufs.


(Stephen, sorry for the double email, forgot to reply-all).

I believe I have found a solution. I'll have to investigate the impact
on performance later but advice/analysis is appreciated. I initialized
the pool with a limit of 128 tx descriptors and set the tx ring buffer
size to match. Once the ring is full, the NIC seems to free things as
it goes, resulting in having no sent-but-unfreed mbufs most of the
time.

Please let me know if there's any major issue with this approach and
if there's anything else I can do. At least this behaves in a way that
I can design my app like I planned and I'm hoping it results in
non-bursty latencies and good performance.

-Alan

Re: Force driver to free sent buffers?

[David Marchand]

On Sun, Oct 6, 2024 at 6:39 PM Alan Beadle <ab.beadle@gmail.com> wrote:
>
> Hi everyone,
>
> Based on my thread from last week, I am using rte_mbuf_refcnt_update()
> to increment the mbuf refcnts so that the driver will not free buffers
> before my local readers are done with them.
> I am also maintaining a queue of mbufs that local readers are finished
> with, and periodically using rte_mbuf_refcnt_read() to see if the
> refcnt is 1 yet, so that they can be freed or reused.
>
> An aside: Is there any problem with reuse of mbufs? Without actually
> freeing and re-allocating it, that is? I plan to do this in order to
> avoid having to link new mbufs to other reused data structures in my
> application.
>
> The current problem is as follows: My application has a separate
> unrelated heap (special purpose and bounded in size) and this heap
> runs out of memory after a brief time because of having to track so
> many unfreed mbufs. It appears that none of the mbufs are being freed

Would it help to store this per-mbuf tracking context into a private area?

You can ask for a per-mbuf private area at the mempool creation, see priv_size.
https://doc.dpdk.org/api/rte__mbuf_8h.html#a593921f13307803b94bbb4e0932db962


> by the driver after being sent, and it looks like it won't free them
> until it runs out of descriptors, which is too long for my situation.
>
> I have confirmed that the packets are being sent (received on other PC
> with wireshark) and should be freeable. Is there a way to force the
> driver to free sent mbufs earlier than when it runs out of
> descriptors?


-- 
David Marchand