[dpdk-dev] suggested patch: testpmd txonly mbuf_alloc does not increment refcnt

[dpdk-dev] suggested patch: testpmd txonly mbuf_alloc does not increment refcnt

[Han, Dongsu]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

Hi dpdk-dev,
  test-pmd txonly leaks mbuf from the pool.
tx_mbuf_alloc does not change the refcnt and the refcnt is 0 when it is
first allocated.

 However, rte_pktmbuf_free_seg called by the driver's xmit code decrements
reference count to -1. So mbuf never goes back to the pool.

 As a result, txonly can't send packets after it exhausts the mempool.
There might be multiple ways to fix this. I decided not to touch
memory management.

Thanks,
-Dongsu Han

---

diff --git a/app/test-pmd/txonly.c b/app/test-pmd/txonly.c
index d7c8c31..ee673e5 100644
--- a/app/test-pmd/txonly.c
+++ b/app/test-pmd/txonly.c
@@ -92,14 +92,7 @@ static struct udp_hdr pkt_udp_hdr; /**< UDP header of
transmitted packets. */
 static inline struct rte_mbuf *
 tx_mbuf_alloc(struct rte_mempool *mp)
 {
-       struct rte_mbuf *m;
-       void *mb;
-
-       if (rte_mempool_get(mp, &mb) < 0)
-               return NULL;
-       m = (struct rte_mbuf *)mb;
-       __rte_mbuf_sanity_check(m, RTE_MBUF_PKT, 1);
-       return m;
+       return rte_pktmbuf_alloc(mp);
 }

[-- Attachment #2: Type: text/html, Size: 1550 bytes --]

[dpdk-dev] [PATCH] app: fix refcnt in mbuf allocation

[Thomas Monjalon]

Hi Dongsu Han,

I think your fix is right.
I've just removed tx_mbuf_alloc() and directly called rte_pktmbuf_alloc() instead.
Is it OK for you ?
Could you also review this (modified) description ?

Thank you

---

From: Dongsu Han <dongsuh at cs.cmu.edu>

test-pmd txonly leaks mbuf from the pool.
The function tx_mbuf_alloc() does not change the refcnt
and the refcnt is 0 when it is first allocated.
However, rte_pktmbuf_free_seg called by the driver's xmit code decrements
reference count to -1. So mbuf never goes back to the pool.
As a result, txonly can't send packets after it exhausts the mempool.

The function tx_mbuf_alloc() was getting mbuf directly from mempool and so
was bypassing mbuf API.
By using the right API, refcnt is correctly handled among other
initializations.

Signed-off-by: Dongsu Han <dongsuh@cs.cmu.edu>
Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 app/test-pmd/txonly.c |   17 ++---------------
 1 file changed, 2 insertions(+), 15 deletions(-)

diff --git a/app/test-pmd/txonly.c b/app/test-pmd/txonly.c
index d7c8c31..53f7138 100644
--- a/app/test-pmd/txonly.c
+++ b/app/test-pmd/txonly.c
@@ -89,19 +89,6 @@
 static struct ipv4_hdr  pkt_ip_hdr;  /**< IP header of transmitted packets. */
 static struct udp_hdr pkt_udp_hdr; /**< UDP header of transmitted packets. */
 
-static inline struct rte_mbuf *
-tx_mbuf_alloc(struct rte_mempool *mp)
-{
-	struct rte_mbuf *m;
-	void *mb;
-
-	if (rte_mempool_get(mp, &mb) < 0)
-		return NULL;
-	m = (struct rte_mbuf *)mb;
-	__rte_mbuf_sanity_check(m, RTE_MBUF_PKT, 1);
-	return m;
-}
-
 static void
 copy_buf_to_pkt_segs(void* buf, unsigned len, struct rte_mbuf *pkt,
 		     unsigned offset)
@@ -223,7 +210,7 @@ pkt_burst_transmit(struct fwd_stream *fs)
 	vlan_tci = ports[fs->tx_port].tx_vlan_id;
 	ol_flags = ports[fs->tx_port].tx_ol_flags;
 	for (nb_pkt = 0; nb_pkt < nb_pkt_per_burst; nb_pkt++) {
-		pkt = tx_mbuf_alloc(mbp);
+		pkt = rte_pktmbuf_alloc(mbp);
 		if (pkt == NULL) {
 		nomore_mbuf:
 			if (nb_pkt == 0)
@@ -233,7 +220,7 @@ pkt_burst_transmit(struct fwd_stream *fs)
 		pkt->pkt.data_len = tx_pkt_seg_lengths[0];
 		pkt_seg = pkt;
 		for (i = 1; i < tx_pkt_nb_segs; i++) {
-			pkt_seg->pkt.next = tx_mbuf_alloc(mbp);
+			pkt_seg->pkt.next = rte_pktmbuf_alloc(mbp);
 			if (pkt_seg->pkt.next == NULL) {
 				pkt->pkt.nb_segs = i;
 				rte_pktmbuf_free(pkt);
-- 
1.7.10.4

Re: [dpdk-dev] [PATCH] app: fix refcnt in mbuf allocation

[Han, Dongsu]

[-- Attachment #1: Type: text/plain, Size: 3121 bytes --]

Sounds good. With the two bug fixes I submitted txonly now runs perfectly.
Thanks!
-Dongsu
On May 6, 2013 8:47 AM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:

> Hi Dongsu Han,
>
> I think your fix is right.
> I've just removed tx_mbuf_alloc() and directly called rte_pktmbuf_alloc()
> instead.
> Is it OK for you ?
> Could you also review this (modified) description ?
>
> Thank you
>
> ---
>
> From: Dongsu Han <dongsuh at cs.cmu.edu>
>
> test-pmd txonly leaks mbuf from the pool.
> The function tx_mbuf_alloc() does not change the refcnt
> and the refcnt is 0 when it is first allocated.
> However, rte_pktmbuf_free_seg called by the driver's xmit code decrements
> reference count to -1. So mbuf never goes back to the pool.
> As a result, txonly can't send packets after it exhausts the mempool.
>
> The function tx_mbuf_alloc() was getting mbuf directly from mempool and so
> was bypassing mbuf API.
> By using the right API, refcnt is correctly handled among other
> initializations.
>
> Signed-off-by: Dongsu Han <dongsuh@cs.cmu.edu>
> Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
> ---
>  app/test-pmd/txonly.c |   17 ++---------------
>  1 file changed, 2 insertions(+), 15 deletions(-)
>
> diff --git a/app/test-pmd/txonly.c b/app/test-pmd/txonly.c
> index d7c8c31..53f7138 100644
> --- a/app/test-pmd/txonly.c
> +++ b/app/test-pmd/txonly.c
> @@ -89,19 +89,6 @@
>  static struct ipv4_hdr  pkt_ip_hdr;  /**< IP header of transmitted
> packets. */
>  static struct udp_hdr pkt_udp_hdr; /**< UDP header of transmitted
> packets. */
>
> -static inline struct rte_mbuf *
> -tx_mbuf_alloc(struct rte_mempool *mp)
> -{
> -       struct rte_mbuf *m;
> -       void *mb;
> -
> -       if (rte_mempool_get(mp, &mb) < 0)
> -               return NULL;
> -       m = (struct rte_mbuf *)mb;
> -       __rte_mbuf_sanity_check(m, RTE_MBUF_PKT, 1);
> -       return m;
> -}
> -
>  static void
>  copy_buf_to_pkt_segs(void* buf, unsigned len, struct rte_mbuf *pkt,
>                      unsigned offset)
> @@ -223,7 +210,7 @@ pkt_burst_transmit(struct fwd_stream *fs)
>         vlan_tci = ports[fs->tx_port].tx_vlan_id;
>         ol_flags = ports[fs->tx_port].tx_ol_flags;
>         for (nb_pkt = 0; nb_pkt < nb_pkt_per_burst; nb_pkt++) {
> -               pkt = tx_mbuf_alloc(mbp);
> +               pkt = rte_pktmbuf_alloc(mbp);
>                 if (pkt == NULL) {
>                 nomore_mbuf:
>                         if (nb_pkt == 0)
> @@ -233,7 +220,7 @@ pkt_burst_transmit(struct fwd_stream *fs)
>                 pkt->pkt.data_len = tx_pkt_seg_lengths[0];
>                 pkt_seg = pkt;
>                 for (i = 1; i < tx_pkt_nb_segs; i++) {
> -                       pkt_seg->pkt.next = tx_mbuf_alloc(mbp);
> +                       pkt_seg->pkt.next = rte_pktmbuf_alloc(mbp);
>                         if (pkt_seg->pkt.next == NULL) {
>                                 pkt->pkt.nb_segs = i;
>                                 rte_pktmbuf_free(pkt);
> --
> 1.7.10.4
>
> _______________________________________________
> dev mailing list
> dev@dpdk.org
> http://dpdk.org/ml/listinfo/dev
>

[-- Attachment #2: Type: text/html, Size: 3927 bytes --]

Re: [dpdk-dev] [PATCH] app: fix refcnt in mbuf allocation

[Adrien Mazarguil]

Hi all (replying below)

On Mon, May 06, 2013 at 09:28:33AM -0400, Han, Dongsu wrote:
> Sounds good. With the two bug fixes I submitted txonly now runs perfectly.
> Thanks!
> -Dongsu
> On May 6, 2013 8:47 AM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:
> 
> > Hi Dongsu Han,
> >
> > I think your fix is right.
> > I've just removed tx_mbuf_alloc() and directly called rte_pktmbuf_alloc()
> > instead.
> > Is it OK for you ?
> > Could you also review this (modified) description ?
[...]

While using rte_pktmbuf_alloc() is the correct fix, it's much slower than
__rte_mbuf_raw_alloc() due to the unnecessary call to rte_pktmbuf_reset().

Since testpmd is often used for performance testing, we should consider a
wrapper function calling __rte_mbuf_raw_alloc() directly instead, as in
rte_rxmbuf_alloc() implemented in igb and ixgbe PMDs.

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-dev] [PATCH] app: fix refcnt in mbuf allocation

[Han, Dongsu]

[-- Attachment #1: Type: text/plain, Size: 1313 bytes --]

In addition to this, we should probably clone pkt mbuf using
rte_pktmbuf_clone()
and transmit the cloned packet. This will avoid having to copy the mac and
IP headers.

-Dongsu

On Mon, May 6, 2013 at 9:56 AM, Adrien Mazarguil <adrien.mazarguil@6wind.com
> wrote:

> Hi all (replying below)
>
> On Mon, May 06, 2013 at 09:28:33AM -0400, Han, Dongsu wrote:
> > Sounds good. With the two bug fixes I submitted txonly now runs
> perfectly.
> > Thanks!
> > -Dongsu
> > On May 6, 2013 8:47 AM, "Thomas Monjalon" <thomas.monjalon@6wind.com>
> wrote:
> >
> > > Hi Dongsu Han,
> > >
> > > I think your fix is right.
> > > I've just removed tx_mbuf_alloc() and directly called
> rte_pktmbuf_alloc()
> > > instead.
> > > Is it OK for you ?
> > > Could you also review this (modified) description ?
> [...]
>
> While using rte_pktmbuf_alloc() is the correct fix, it's much slower than
> __rte_mbuf_raw_alloc() due to the unnecessary call to rte_pktmbuf_reset().
>
> Since testpmd is often used for performance testing, we should consider a
> wrapper function calling __rte_mbuf_raw_alloc() directly instead, as in
> rte_rxmbuf_alloc() implemented in igb and ixgbe PMDs.
>
> --
> Adrien Mazarguil
> 6WIND
> _______________________________________________
> dev mailing list
> dev@dpdk.org
> http://dpdk.org/ml/listinfo/dev
>

[-- Attachment #2: Type: text/html, Size: 2183 bytes --]

[dpdk-dev] [PATCH] app: fix refcnt in mbuf allocation

[Thomas Monjalon]

From: Dongsu Han <dongsuh@cs.cmu.edu>

test-pmd txonly leaks mbuf from the pool.
The function tx_mbuf_alloc() does not change the refcnt
and the refcnt is 0 when it is first allocated.
However, rte_pktmbuf_free_seg called by the driver's xmit code decrements
reference count to -1. So mbuf never goes back to the pool.
As a result, txonly can't send packets after it exhausts the mempool.

The function tx_mbuf_alloc() was getting mbuf directly from mempool and so
was bypassing mbuf API.
The dedicated function is rte_pktmbuf_alloc() but it is much slower because
it does unnecessary initializations in rte_pktmbuf_reset().
By using the internal API __rte_mbuf_raw_alloc(), refcnt is correctly handled
without adding too much overload.

Signed-off-by: Dongsu Han <dongsuh@cs.cmu.edu>
Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
---
 app/test-pmd/txonly.c |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/app/test-pmd/txonly.c b/app/test-pmd/txonly.c
index d7c8c31..e7c9a1a 100644
--- a/app/test-pmd/txonly.c
+++ b/app/test-pmd/txonly.c
@@ -93,11 +93,8 @@ static inline struct rte_mbuf *
 tx_mbuf_alloc(struct rte_mempool *mp)
 {
 	struct rte_mbuf *m;
-	void *mb;
 
-	if (rte_mempool_get(mp, &mb) < 0)
-		return NULL;
-	m = (struct rte_mbuf *)mb;
+	m = __rte_mbuf_raw_alloc(mp);
 	__rte_mbuf_sanity_check(m, RTE_MBUF_PKT, 1);
 	return m;
 }
-- 
1.7.10.4

Re: [dpdk-dev] [PATCH] app: fix refcnt in mbuf allocation

[Adrien Mazarguil]

On Mon, May 06, 2013 at 06:00:01PM +0200, Thomas Monjalon wrote:
> From: Dongsu Han <dongsuh@cs.cmu.edu>
> 
> test-pmd txonly leaks mbuf from the pool.
> The function tx_mbuf_alloc() does not change the refcnt
> and the refcnt is 0 when it is first allocated.
> However, rte_pktmbuf_free_seg called by the driver's xmit code decrements
> reference count to -1. So mbuf never goes back to the pool.
> As a result, txonly can't send packets after it exhausts the mempool.
> 
> The function tx_mbuf_alloc() was getting mbuf directly from mempool and so
> was bypassing mbuf API.
> The dedicated function is rte_pktmbuf_alloc() but it is much slower because
> it does unnecessary initializations in rte_pktmbuf_reset().
> By using the internal API __rte_mbuf_raw_alloc(), refcnt is correctly handled
> without adding too much overload.
> 
> Signed-off-by: Dongsu Han <dongsuh@cs.cmu.edu>
> Signed-off-by: Thomas Monjalon <thomas.monjalon@6wind.com>
> ---
>  app/test-pmd/txonly.c |    5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/app/test-pmd/txonly.c b/app/test-pmd/txonly.c
> index d7c8c31..e7c9a1a 100644
> --- a/app/test-pmd/txonly.c
> +++ b/app/test-pmd/txonly.c
> @@ -93,11 +93,8 @@ static inline struct rte_mbuf *
>  tx_mbuf_alloc(struct rte_mempool *mp)
>  {
>  	struct rte_mbuf *m;
> -	void *mb;
>  
> -	if (rte_mempool_get(mp, &mb) < 0)
> -		return NULL;
> -	m = (struct rte_mbuf *)mb;
> +	m = __rte_mbuf_raw_alloc(mp);
>  	__rte_mbuf_sanity_check(m, RTE_MBUF_PKT, 1);
>  	return m;
>  }
> -- 
> 1.7.10.4

Looks good.

Acked-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>

-- 
Adrien Mazarguil
6WIND

Re: [dpdk-dev] app: fix refcnt in mbuf allocation

[Thomas Monjalon]

06/05/2013 17:47, Han, Dongsu :
> In addition to this, we should probably clone pkt mbuf using
> rte_pktmbuf_clone()
> and transmit the cloned packet. This will avoid having to copy the mac and
> IP headers.

Yes, it could be faster but it is not always possible because 
rte_pktmbuf_clone() is available only if the feature RTE_MBUF_SCATTER_GATHER 
is enabled.

-- 
Thomas

Re: [dpdk-dev] [PATCH] app: fix refcnt in mbuf allocation

[Thomas Monjalon]

pushed

thank you all
-- 
Thomas