[dpdk-dev] issue with memory sync on P9

[dpdk-dev] issue with memory sync on P9

[Dekel Peled]

Hi Chao Zhu,

We have recently encountered runtime issues when running DPDK 18.11 (and later) MLX5 PMD on P9 setup.
Using rte_wmb() was found to be insufficient in specific location, memory sync problem occurred regardless.
Using rte_mb() in this specific location fixed the issue.

--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -706,7 +706,7 @@ mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volatile struct mlx5_wqe *wqe,
        rte_cio_wmb();
        *txq->qp_db = rte_cpu_to_be_32(txq->wqe_ci);
        /* Ensure ordering between DB record and BF copy. */
-       rte_wmb();
+       rte_mb();
        mlx5_uar_write64_relaxed(*src, dst, txq->uar_lock);
        if (cond)
                rte_wmb();

I see the change in rte_wmb() implementation comes from your patch:
commit d23a6bd04d728226c99d6995c8bd65d49d7e61d1)
Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
Date:   Fri Jul 15 10:30:19 2016 +0800
eal/ppc: fix memory barrier for IBM POWER

I ran some performance tests and got the same results with and without this patch.
Do you have different findings? Will you approve removing this patch?

Regards,
Dekel

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Chao Zhu
> Sent: Wednesday, February 20, 2019 10:07 AM
> To: Thomas Monjalon <thomas@monjalon.net>; 'David Wilder'
> <dwilder@us.ibm.com>
> Cc: dev@dpdk.org; pradeep@us.ibm.com
> Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH v3] mem: Fix anonymous
> mapping on Power9.
> 
> +Pradeep
> 
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Monday, February 18, 2019 11:47 PM
> > To: David Wilder <dwilder@us.ibm.com>
> > Cc: dev@dpdk.org; chaozhu@linux.vnet.ibm.com
> > Subject: Re: [dpdk-stable] [PATCH v3] mem: Fix anonymous mapping on
> > Power9.
> >
> > Hi,
> >
> > Long time no talk about the IBM Power support status.
> >
> > 16/11/2018 18:35, David Wilder:
> > > --- a/doc/guides/rel_notes/release_18_11.rst
> > > +++ b/doc/guides/rel_notes/release_18_11.rst
> > > +* IBM Power8 is not supported by this release of DPDK. IBM Power9
> > > +is
> > supported.
> >
> > The config file is defconfig_ppc_64-power8-linuxapp-gcc.
> > Given that only Power9 is supported, should we rename it?
> >
> > Is there more changes required to make Power9 working with the latest
> DPDK?
> >
> 
> 

[dpdk-dev] issue with memory sync on P9

[Dekel Peled]

Hi Chao Zhu,

We have recently encountered runtime issues when running DPDK 18.11 (and later) MLX5 PMD on P9 setup.
Using rte_wmb() was found to be insufficient in specific location, memory sync problem occurred regardless.
Using rte_mb() in this specific location fixed the issue.

--- a/drivers/net/mlx5/mlx5_rxtx.h
+++ b/drivers/net/mlx5/mlx5_rxtx.h
@@ -706,7 +706,7 @@ mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volatile struct mlx5_wqe *wqe,
        rte_cio_wmb();
        *txq->qp_db = rte_cpu_to_be_32(txq->wqe_ci);
        /* Ensure ordering between DB record and BF copy. */
-       rte_wmb();
+       rte_mb();
        mlx5_uar_write64_relaxed(*src, dst, txq->uar_lock);
        if (cond)
                rte_wmb();

I see the change in rte_wmb() implementation comes from your patch:
commit d23a6bd04d728226c99d6995c8bd65d49d7e61d1)
Author: Chao Zhu <chaozhu@linux.vnet.ibm.com>
Date:   Fri Jul 15 10:30:19 2016 +0800
eal/ppc: fix memory barrier for IBM POWER

I ran some performance tests and got the same results with and without this patch.
Do you have different findings? Will you approve removing this patch?

Regards,
Dekel

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Chao Zhu
> Sent: Wednesday, February 20, 2019 10:07 AM
> To: Thomas Monjalon <thomas@monjalon.net>; 'David Wilder'
> <dwilder@us.ibm.com>
> Cc: dev@dpdk.org; pradeep@us.ibm.com
> Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH v3] mem: Fix anonymous
> mapping on Power9.
> 
> +Pradeep
> 
> > -----Original Message-----
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Monday, February 18, 2019 11:47 PM
> > To: David Wilder <dwilder@us.ibm.com>
> > Cc: dev@dpdk.org; chaozhu@linux.vnet.ibm.com
> > Subject: Re: [dpdk-stable] [PATCH v3] mem: Fix anonymous mapping on
> > Power9.
> >
> > Hi,
> >
> > Long time no talk about the IBM Power support status.
> >
> > 16/11/2018 18:35, David Wilder:
> > > --- a/doc/guides/rel_notes/release_18_11.rst
> > > +++ b/doc/guides/rel_notes/release_18_11.rst
> > > +* IBM Power8 is not supported by this release of DPDK. IBM Power9
> > > +is
> > supported.
> >
> > The config file is defconfig_ppc_64-power8-linuxapp-gcc.
> > Given that only Power9 is supported, should we rename it?
> >
> > Is there more changes required to make Power9 working with the latest
> DPDK?
> >
> 
>