[dpdk-dev] Getting started - sanity check

[dpdk-dev] Getting started - sanity check

[Clark, Gilbert]

Hi folks:

I'm brand new to DPDK.  Read about it off and on occasionally, but never had the chance to sit down and play with things until now.  It's been fun so far: just been working on a few toy applications to get myself started.

I have run into a question, though: when calling rte_eth_tx_burst with a ring-backed PMD I've set up, the mbufs I've sent never seem to be freed.  This seems to make some degree of sense, but ... since I'm new, and because the documentation says rte_eth_tx_burst should eventually free mbufs that are sent [1], I wanted to make sure I'm on track and not just misunderstanding the way something works [2].

Thanks,
Gilbert Clark

[1] From http://dpdk.org/doc/api/rte__ethdev_8h.html :

It is the responsibility of the rte_eth_tx_burst() function to transparently free the memory buffers of packets previously sent

[2] From lib/librte_pmd_ring.c:

static uint16_t
eth_ring_tx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
{
    void **ptrs = (void *)&bufs[0];
    struct ring_queue *r = q;
    const uint16_t nb_tx = (uint16_t)rte_ring_enqueue_burst(r->rng,
            ptrs, nb_bufs);
    if (r->rng->flags & RING_F_SP_ENQ) {
        r->tx_pkts.cnt += nb_tx;
        r->err_pkts.cnt += nb_bufs - nb_tx;
    } else {
        rte_atomic64_add(&(r->tx_pkts), nb_tx);
        rte_atomic64_add(&(r->err_pkts), nb_bufs - nb_tx);
    }
    return nb_tx;
}

This doesn't ever appear to free a transmitted mbuf ... unless there's code to do that somewhere else that I'm missing?

Re: [dpdk-dev] Getting started - sanity check

[Wiles, Keith]

Sent from my iPhone

> On May 9, 2015, at 9:27 AM, Clark, Gilbert <gc355804@ohio.edu> wrote:
> 
> 
> Hi folks:
> 
> I'm brand new to DPDK.  Read about it off and on occasionally, but never had the chance to sit down and play with things until now.  It's been fun so far: just been working on a few toy applications to get myself started.
> 
> I have run into a question, though: when calling rte_eth_tx_burst with a ring-backed PMD I've set up, the mbufs I've sent never seem to be freed.  This seems to make some degree of sense, but ... since I'm new, and because the documentation says rte_eth_tx_burst should eventually free mbufs that are sent [1], I wanted to make sure I'm on track and not just misunderstanding the way something works [2].

The mbufs are free as needed or when a watermark is hit in the driver. One other thing I found is you need to send enough packets to hit the water mark for the tx mbufs start to get freed. Also you have to have enough mbufs allocated to hit these watermarks. I just pick two times the ring size just to be safe. 

Hope that helps. 

> 
> Thanks,
> Gilbert Clark
> 
> [1] From http://dpdk.org/doc/api/rte__ethdev_8h.html :
> 
> It is the responsibility of the rte_eth_tx_burst() function to transparently free the memory buffers of packets previously sent
> 
> [2] From lib/librte_pmd_ring.c:
> 
> static uint16_t
> eth_ring_tx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
> {
>    void **ptrs = (void *)&bufs[0];
>    struct ring_queue *r = q;
>    const uint16_t nb_tx = (uint16_t)rte_ring_enqueue_burst(r->rng,
>            ptrs, nb_bufs);
>    if (r->rng->flags & RING_F_SP_ENQ) {
>        r->tx_pkts.cnt += nb_tx;
>        r->err_pkts.cnt += nb_bufs - nb_tx;
>    } else {
>        rte_atomic64_add(&(r->tx_pkts), nb_tx);
>        rte_atomic64_add(&(r->err_pkts), nb_bufs - nb_tx);
>    }
>    return nb_tx;
> }
> 
> This doesn't ever appear to free a transmitted mbuf ... unless there's code to do that somewhere else that I'm missing?

Re: [dpdk-dev] Getting started - sanity check

[Clark, Gilbert]

Thanks for the note!  I'll take another look at it.

Cheers,
Gilbert

________________________________________
From: Wiles, Keith <keith.wiles@intel.com>
Sent: Saturday, May 9, 2015 12:41 PM
To: Clark, Gilbert
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Getting started - sanity check

Sent from my iPhone

> On May 9, 2015, at 9:27 AM, Clark, Gilbert <gc355804@ohio.edu> wrote:
>
>
> Hi folks:
>
> I'm brand new to DPDK.  Read about it off and on occasionally, but never had the chance to sit down and play with things until now.  It's been fun so far: just been working on a few toy applications to get myself started.
>
> I have run into a question, though: when calling rte_eth_tx_burst with a ring-backed PMD I've set up, the mbufs I've sent never seem to be freed.  This seems to make some degree of sense, but ... since I'm new, and because the documentation says rte_eth_tx_burst should eventually free mbufs that are sent [1], I wanted to make sure I'm on track and not just misunderstanding the way something works [2].

The mbufs are free as needed or when a watermark is hit in the driver. One other thing I found is you need to send enough packets to hit the water mark for the tx mbufs start to get freed. Also you have to have enough mbufs allocated to hit these watermarks. I just pick two times the ring size just to be safe.

Hope that helps.

>
> Thanks,
> Gilbert Clark
>
> [1] From http://dpdk.org/doc/api/rte__ethdev_8h.html :
>
> It is the responsibility of the rte_eth_tx_burst() function to transparently free the memory buffers of packets previously sent
>
> [2] From lib/librte_pmd_ring.c:
>
> static uint16_t
> eth_ring_tx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
> {
>    void **ptrs = (void *)&bufs[0];
>    struct ring_queue *r = q;
>    const uint16_t nb_tx = (uint16_t)rte_ring_enqueue_burst(r->rng,
>            ptrs, nb_bufs);
>    if (r->rng->flags & RING_F_SP_ENQ) {
>        r->tx_pkts.cnt += nb_tx;
>        r->err_pkts.cnt += nb_bufs - nb_tx;
>    } else {
>        rte_atomic64_add(&(r->tx_pkts), nb_tx);
>        rte_atomic64_add(&(r->err_pkts), nb_bufs - nb_tx);
>    }
>    return nb_tx;
> }
>
> This doesn't ever appear to free a transmitted mbuf ... unless there's code to do that somewhere else that I'm missing?

Re: [dpdk-dev] Getting started - sanity check

[Bruce Richardson]

On Sat, May 09, 2015 at 04:27:12PM +0000, Clark, Gilbert wrote:
> 
> Hi folks:
> 
> I'm brand new to DPDK.  Read about it off and on occasionally, but never had the chance to sit down and play with things until now.  It's been fun so far: just been working on a few toy applications to get myself started.
> 
> I have run into a question, though: when calling rte_eth_tx_burst with a ring-backed PMD I've set up, the mbufs I've sent never seem to be freed.  This seems to make some degree of sense, but ... since I'm new, and because the documentation says rte_eth_tx_burst should eventually free mbufs that are sent [1], I wanted to make sure I'm on track and not just misunderstanding the way something works [2].
> 
> Thanks,
> Gilbert Clark
> 
> [1] From http://dpdk.org/doc/api/rte__ethdev_8h.html :
> 
> It is the responsibility of the rte_eth_tx_burst() function to transparently free the memory buffers of packets previously sent
> 
> [2] From lib/librte_pmd_ring.c:
> 
> static uint16_t
> eth_ring_tx(void *q, struct rte_mbuf **bufs, uint16_t nb_bufs)
> {
>     void **ptrs = (void *)&bufs[0];
>     struct ring_queue *r = q;
>     const uint16_t nb_tx = (uint16_t)rte_ring_enqueue_burst(r->rng,
>             ptrs, nb_bufs);
>     if (r->rng->flags & RING_F_SP_ENQ) {
>         r->tx_pkts.cnt += nb_tx;
>         r->err_pkts.cnt += nb_bufs - nb_tx;
>     } else {
>         rte_atomic64_add(&(r->tx_pkts), nb_tx);
>         rte_atomic64_add(&(r->err_pkts), nb_bufs - nb_tx);
>     }
>     return nb_tx;
> }
> 
> This doesn't ever appear to free a transmitted mbuf ... unless there's code to do that somewhere else that I'm missing?

Indeed it doesn't free the mbufs, because this is not a PMD backed by real hardware
so the packets are never actually transmitted anywhere, just passed to the other
end of the ring. To behave strictly like a physical PMD, we would copy the sent
packet to a new buffer on RX, and free the old one. 
However, in this case, we take a shortcut and just pass the same mbuf on RX as
was passed on TX, which saves the cycles for buffer management.

To see buffer freeing on TX occur, I suggest you look at some of the other PMDs,
perhaps the e1000/igb PMD?

/Bruce