Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Wu, Jingjing]

Hi, qi

Is your patch duplicated with this http://www.dpdk.org/dev/patchwork/patch/29814/ ?

Thanks
Jingjing

> -----Original Message-----
> From: Zhang, Qi Z
> Sent: Monday, October 9, 2017 4:54 PM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>
> Cc: dev@dpdk.org; Zhang, Qi Z <qi.z.zhang@intel.com>; stable@dpdk.org
> Subject: [PATCH] net/i40e: fix unexpected mbuf free in vPMD
> 
> The patch reset tx queue sw_ring's mbuf to NULL after it is free in
> i40_tx_free_bufs, this prevent same mbuf be free again in
> i40e_dev_tx_queue_release. This fix follow the same implemenation of non-
> vPMD.
> 
> Fixes: b4669bb95038 ("i40e: add vector Tx")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>  drivers/net/i40e/i40e_rxtx_vec_common.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/net/i40e/i40e_rxtx_vec_common.h
> b/drivers/net/i40e/i40e_rxtx_vec_common.h
> index 39a6da0..ed51b4d 100644
> --- a/drivers/net/i40e/i40e_rxtx_vec_common.h
> +++ b/drivers/net/i40e/i40e_rxtx_vec_common.h
> @@ -124,11 +124,13 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
>  	  */
>  	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
>  	m = rte_pktmbuf_prefree_seg(txep[0].mbuf);
> +	txep[0].mbuf = NULL;
>  	if (likely(m != NULL)) {
>  		free[0] = m;
>  		nb_free = 1;
>  		for (i = 1; i < n; i++) {
>  			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
> +			txep[i].mbuf = NULL;
>  			if (likely(m != NULL)) {
>  				if (likely(m->pool == free[0]->pool)) {
>  					free[nb_free++] = m;
> @@ -145,6 +147,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
>  	} else {
>  		for (i = 1; i < n; i++) {
>  			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
> +			txep[i].mbuf = NULL;
>  			if (m != NULL)
>  				rte_mempool_put(m->pool, m);
>  		}
> --
> 2.9.5

[dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Qi Zhang]

The patch reset tx queue sw_ring's mbuf to NULL after it is free in
i40_tx_free_bufs, this prevent same mbuf be free again in
i40e_dev_tx_queue_release. This fix follow the same implemenation of
non-vPMD.

Fixes: b4669bb95038 ("i40e: add vector Tx")
Cc: stable@dpdk.org

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
 drivers/net/i40e/i40e_rxtx_vec_common.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/i40e/i40e_rxtx_vec_common.h b/drivers/net/i40e/i40e_rxtx_vec_common.h
index 39a6da0..ed51b4d 100644
--- a/drivers/net/i40e/i40e_rxtx_vec_common.h
+++ b/drivers/net/i40e/i40e_rxtx_vec_common.h
@@ -124,11 +124,13 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 	  */
 	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
 	m = rte_pktmbuf_prefree_seg(txep[0].mbuf);
+	txep[0].mbuf = NULL;
 	if (likely(m != NULL)) {
 		free[0] = m;
 		nb_free = 1;
 		for (i = 1; i < n; i++) {
 			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
+			txep[i].mbuf = NULL;
 			if (likely(m != NULL)) {
 				if (likely(m->pool == free[0]->pool)) {
 					free[nb_free++] = m;
@@ -145,6 +147,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
 	} else {
 		for (i = 1; i < n; i++) {
 			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
+			txep[i].mbuf = NULL;
 			if (m != NULL)
 				rte_mempool_put(m->pool, m);
 		}
-- 
2.9.5

Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Ananyev, Konstantin]

Hi Zhang,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Qi Zhang
> Sent: Monday, October 9, 2017 9:54 AM
> To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>
> Cc: dev@dpdk.org; Zhang, Qi Z <qi.z.zhang@intel.com>; stable@dpdk.org
> Subject: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD
> 
> The patch reset tx queue sw_ring's mbuf to NULL after it is free in
> i40_tx_free_bufs, this prevent same mbuf be free again in
> i40e_dev_tx_queue_release. This fix follow the same implemenation of
> non-vPMD.

Wonder why we can't change i40e_dev_tx_queue_release() instead, 
so it will only go through the TXDs that were really armed?
Let say from txq->tx_next_dd - tx_rs_thresh - 1 till txq->tx_tail?
Let say _ixgbe_tx_queue_release_mbufs_vec() works that way.
Then we probably can keep our runtime code intact.
Konstantin

> 
> Fixes: b4669bb95038 ("i40e: add vector Tx")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
>  drivers/net/i40e/i40e_rxtx_vec_common.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/net/i40e/i40e_rxtx_vec_common.h b/drivers/net/i40e/i40e_rxtx_vec_common.h
> index 39a6da0..ed51b4d 100644
> --- a/drivers/net/i40e/i40e_rxtx_vec_common.h
> +++ b/drivers/net/i40e/i40e_rxtx_vec_common.h
> @@ -124,11 +124,13 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
>  	  */
>  	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
>  	m = rte_pktmbuf_prefree_seg(txep[0].mbuf);
> +	txep[0].mbuf = NULL;
>  	if (likely(m != NULL)) {
>  		free[0] = m;
>  		nb_free = 1;
>  		for (i = 1; i < n; i++) {
>  			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
> +			txep[i].mbuf = NULL;
>  			if (likely(m != NULL)) {
>  				if (likely(m->pool == free[0]->pool)) {
>  					free[nb_free++] = m;
> @@ -145,6 +147,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
>  	} else {
>  		for (i = 1; i < n; i++) {
>  			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
> +			txep[i].mbuf = NULL;
>  			if (m != NULL)
>  				rte_mempool_put(m->pool, m);
>  		}
> --
> 2.9.5

Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Bruce Richardson]

On Mon, Oct 09, 2017 at 12:44:25PM +0000, Ananyev, Konstantin wrote:
> Hi Zhang,
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Qi Zhang
> > Sent: Monday, October 9, 2017 9:54 AM
> > To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei <beilei.xing@intel.com>
> > Cc: dev@dpdk.org; Zhang, Qi Z <qi.z.zhang@intel.com>; stable@dpdk.org
> > Subject: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD
> > 
> > The patch reset tx queue sw_ring's mbuf to NULL after it is free in
> > i40_tx_free_bufs, this prevent same mbuf be free again in
> > i40e_dev_tx_queue_release. This fix follow the same implemenation of
> > non-vPMD.
> 
> Wonder why we can't change i40e_dev_tx_queue_release() instead, 
> so it will only go through the TXDs that were really armed?
> Let say from txq->tx_next_dd - tx_rs_thresh - 1 till txq->tx_tail?
> Let say _ixgbe_tx_queue_release_mbufs_vec() works that way.
> Then we probably can keep our runtime code intact.
> Konstantin
> 
+1 to this. We should avoid adding writes to the TX path. Instead we
should ensure that other routines properly track the buffers to be freed
using the appropriate indexes.

/Bruce

Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Zhang, Qi Z]

Hi Konstantin:

> -----Original Message-----
> From: Ananyev, Konstantin
> Sent: Monday, October 9, 2017 8:44 PM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; Wu, Jingjing <jingjing.wu@intel.com>;
> Xing, Beilei <beilei.xing@intel.com>
> Cc: dev@dpdk.org; Zhang, Qi Z <qi.z.zhang@intel.com>; stable@dpdk.org
> Subject: RE: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD
> 
> Hi Zhang,
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Qi Zhang
> > Sent: Monday, October 9, 2017 9:54 AM
> > To: Wu, Jingjing <jingjing.wu@intel.com>; Xing, Beilei
> > <beilei.xing@intel.com>
> > Cc: dev@dpdk.org; Zhang, Qi Z <qi.z.zhang@intel.com>; stable@dpdk.org
> > Subject: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD
> >
> > The patch reset tx queue sw_ring's mbuf to NULL after it is free in
> > i40_tx_free_bufs, this prevent same mbuf be free again in
> > i40e_dev_tx_queue_release. This fix follow the same implemenation of
> > non-vPMD.
> 
> Wonder why we can't change i40e_dev_tx_queue_release() instead, so it will
> only go through the TXDs that were really armed?
Yes we can, I've thought about this before the patch, but after I measure the performance impact, I saw less than 0.6% downgrade for single core performance.
So I think maybe a simple fix could be good enough and low risk and an improvement can be delivered later after carefully validate.

> Let say from txq->tx_next_dd - tx_rs_thresh - 1 till txq->tx_tail?
> Let say _ixgbe_tx_queue_release_mbufs_vec() works that way.

Yes, That's the same way I figured out, thanks for giving the detail instruction that confirm my thought :)
Now, I think my v2 can upgrade to this directly.

Thanks
Qi

> Then we probably can keep our runtime code intact.

> Konstantin
> 
> >
> > Fixes: b4669bb95038 ("i40e: add vector Tx")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > ---
> >  drivers/net/i40e/i40e_rxtx_vec_common.h | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/net/i40e/i40e_rxtx_vec_common.h
> > b/drivers/net/i40e/i40e_rxtx_vec_common.h
> > index 39a6da0..ed51b4d 100644
> > --- a/drivers/net/i40e/i40e_rxtx_vec_common.h
> > +++ b/drivers/net/i40e/i40e_rxtx_vec_common.h
> > @@ -124,11 +124,13 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
> >  	  */
> >  	txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)];
> >  	m = rte_pktmbuf_prefree_seg(txep[0].mbuf);
> > +	txep[0].mbuf = NULL;
> >  	if (likely(m != NULL)) {
> >  		free[0] = m;
> >  		nb_free = 1;
> >  		for (i = 1; i < n; i++) {
> >  			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
> > +			txep[i].mbuf = NULL;
> >  			if (likely(m != NULL)) {
> >  				if (likely(m->pool == free[0]->pool)) {
> >  					free[nb_free++] = m;
> > @@ -145,6 +147,7 @@ i40e_tx_free_bufs(struct i40e_tx_queue *txq)
> >  	} else {
> >  		for (i = 1; i < n; i++) {
> >  			m = rte_pktmbuf_prefree_seg(txep[i].mbuf);
> > +			txep[i].mbuf = NULL;
> >  			if (m != NULL)
> >  				rte_mempool_put(m->pool, m);
> >  		}
> > --
> > 2.9.5

[dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Qi Zhang]

vPMD tx does not set sw_ring's mbuf to NULL after free it.
So to prevent same mbuf be free again, we need more carefully
check in i40e_tx_queue_release_mbufs.

Fixes: b4669bb95038 ("i40e: add vector Tx")
Cc: stable@dpdk.org

Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
---
v2:
- fix at i40e_tx_queue_release_mbufs so no performance impact.

 drivers/net/i40e/i40e_rxtx.c | 26 ++++++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c
index 3a8d32c..f21c1c5 100644
--- a/drivers/net/i40e/i40e_rxtx.c
+++ b/drivers/net/i40e/i40e_rxtx.c
@@ -2305,18 +2305,40 @@ i40e_reset_rx_queue(struct i40e_rx_queue *rxq)
 void
 i40e_tx_queue_release_mbufs(struct i40e_tx_queue *txq)
 {
+	struct rte_eth_dev *dev;
 	uint16_t i;
 
+	dev = &rte_eth_devices[txq->port_id];
+
 	if (!txq || !txq->sw_ring) {
 		PMD_DRV_LOG(DEBUG, "Pointer to rxq or sw_ring is NULL");
 		return;
 	}
 
-	for (i = 0; i < txq->nb_tx_desc; i++) {
-		if (txq->sw_ring[i].mbuf) {
+	/**
+	 *  vPMD tx will not set sw_ring's mbuf to NULL after free,
+	 *  so need to free remains more carefully.
+	 */
+	if (dev->tx_pkt_burst == i40e_xmit_pkts_vec) {
+		i = txq->tx_next_dd - txq->tx_rs_thresh + 1;
+		if (txq->tx_tail < i) {
+			for (; i < txq->nb_tx_desc; i++) {
+				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+				txq->sw_ring[i].mbuf = NULL;
+			}
+			i = 0;
+		}
+		for (; i < txq->tx_tail; i++) {
 			rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
 			txq->sw_ring[i].mbuf = NULL;
 		}
+	} else {
+		for (i = 0; i < txq->nb_tx_desc; i++) {
+			if (txq->sw_ring[i].mbuf) {
+				rte_pktmbuf_free_seg(txq->sw_ring[i].mbuf);
+				txq->sw_ring[i].mbuf = NULL;
+			}
+		}
 	}
 }
 
-- 
2.9.5

Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Roger B. Melton]

On 10/10/17 7:39 AM, Bruce Richardson wrote:
> On Tue, Oct 10, 2017 at 07:05:33AM -0400, Roger B. Melton wrote:
>> Hi Bruce,
>>
>> I can.  It will take a day or 2 to get the results.
>>
>> Regards,
>> Roger
>>
> Thanks. Keep us posted. We want to ensure we have the best fix possible
> for the issue in this release.

Thanks for finding a better solution ;-).  I'll let you know the results 
when I have them, just wanted you to know it may take a couple of days 
to turn around.

-Roger

>
> /Bruce
>
>> On 10/10/17 4:48 AM, Bruce Richardson wrote:
>>> +Roger Melton
>>>
>>> On Tue, Oct 10, 2017 at 09:22:05AM -0400, Qi Zhang wrote:
>>>> vPMD tx does not set sw_ring's mbuf to NULL after free it.
>>>> So to prevent same mbuf be free again, we need more carefully
>>>> check in i40e_tx_queue_release_mbufs.
>>>>
>>>> Fixes: b4669bb95038 ("i40e: add vector Tx")
>>>> Cc: stable@dpdk.org
>>>>
>>>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>>>> ---
>>>> v2:
>>>> - fix at i40e_tx_queue_release_mbufs so no performance impact.
>>>>
>>> This patch also supercedes this one, http://dpdk.org/dev/patchwork/patch/29814/
>>> right?
>>> Roger, can you please confirm that this alternative fix works in your
>>> testing and that your patch is no longer necessary too.
>>>
>>> Thanks,
>>> /Bruce
>>>
>>> .
>>>
> .
>

Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Bruce Richardson]

On Tue, Oct 10, 2017 at 07:05:33AM -0400, Roger B. Melton wrote:
> Hi Bruce,
> 
> I can.  It will take a day or 2 to get the results.
> 
> Regards,
> Roger
> 

Thanks. Keep us posted. We want to ensure we have the best fix possible
for the issue in this release.

/Bruce

> 
> On 10/10/17 4:48 AM, Bruce Richardson wrote:
> > +Roger Melton
> > 
> > On Tue, Oct 10, 2017 at 09:22:05AM -0400, Qi Zhang wrote:
> > > vPMD tx does not set sw_ring's mbuf to NULL after free it.
> > > So to prevent same mbuf be free again, we need more carefully
> > > check in i40e_tx_queue_release_mbufs.
> > > 
> > > Fixes: b4669bb95038 ("i40e: add vector Tx")
> > > Cc: stable@dpdk.org
> > > 
> > > Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> > > ---
> > > v2:
> > > - fix at i40e_tx_queue_release_mbufs so no performance impact.
> > > 
> > This patch also supercedes this one, http://dpdk.org/dev/patchwork/patch/29814/
> > right?
> > Roger, can you please confirm that this alternative fix works in your
> > testing and that your patch is no longer necessary too.
> > 
> > Thanks,
> > /Bruce
> > 
> > .
> > 
> 

Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Roger B. Melton]

Hi Bruce,

I can.  It will take a day or 2 to get the results.

Regards,
Roger


On 10/10/17 4:48 AM, Bruce Richardson wrote:
> +Roger Melton
>
> On Tue, Oct 10, 2017 at 09:22:05AM -0400, Qi Zhang wrote:
>> vPMD tx does not set sw_ring's mbuf to NULL after free it.
>> So to prevent same mbuf be free again, we need more carefully
>> check in i40e_tx_queue_release_mbufs.
>>
>> Fixes: b4669bb95038 ("i40e: add vector Tx")
>> Cc: stable@dpdk.org
>>
>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>> ---
>> v2:
>> - fix at i40e_tx_queue_release_mbufs so no performance impact.
>>
> This patch also supercedes this one, http://dpdk.org/dev/patchwork/patch/29814/
> right?
> Roger, can you please confirm that this alternative fix works in your
> testing and that your patch is no longer necessary too.
>
> Thanks,
> /Bruce
>
> .
>

Re: [dpdk-dev] [PATCH] net/i40e: fix unexpected mbuf free in vPMD

[Bruce Richardson]

+Roger Melton

On Tue, Oct 10, 2017 at 09:22:05AM -0400, Qi Zhang wrote:
> vPMD tx does not set sw_ring's mbuf to NULL after free it.
> So to prevent same mbuf be free again, we need more carefully
> check in i40e_tx_queue_release_mbufs.
> 
> Fixes: b4669bb95038 ("i40e: add vector Tx")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
> ---
> v2:
> - fix at i40e_tx_queue_release_mbufs so no performance impact.
> 
This patch also supercedes this one, http://dpdk.org/dev/patchwork/patch/29814/
right?
Roger, can you please confirm that this alternative fix works in your
testing and that your patch is no longer necessary too.

Thanks,
/Bruce