From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by dpdk.org (Postfix) with ESMTP id CD73E37B0 for ; Fri, 22 Mar 2019 09:49:08 +0100 (CET) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 27A0C236CA; Fri, 22 Mar 2019 04:49:08 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 22 Mar 2019 04:49:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=pguyjRnaLB1k2QUcM5EnvEc+OkVrNpDYNMTorH8v27Q=; b=S67bG1RroWMa /1wZyYw/tf2uJNi3sDLjEpAaI8lUmMQqyRI6YikEortQC0dipWinWdB6m4Boa8Kp uEUzh7eeVCxziTH7yiSUlpPOcg/aKbvOGQmObUMiVzk/ocGFjZSFCAOV/NfAnb0J 82Q3mmsnjgeiZyXTYt3Cp85CibBxjlo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=pguyjRnaLB1k2QUcM5EnvEc+OkVrNpDYNMTorH8v2 7Q=; b=a8HjFp0ANE4uNBbhGsEhT5g3jy2f3qbP55FdW6S1t57v0ABwM9pB/qPG+ YMmE9/s/+ExU75rKeNhmLolJOrpUk5awDesAVDLSBmiv8HZKdPEELUrBfBcJGrUI m2aPlB1XIMioTS69tf6vPlgjA9NOznlnIfxZM6q4xx0xitxIIcEcEYKetmCHsLsy UovzpIWPGhzwzzOtLIGCq2hzQlbfyJs1OKGncAB+c48B0hI6Y/gvl9rnFy4fyGw5 S7ty94apUAezjx9397HABu8Cp8M9OSBMrYX7nN1I6KUy7ST+7dAn4Ej9QapCHDEa bRz0BQiB1fxk76RTkLhA6V+lnz9Sg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrjedtgdduvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhgggfgtsehtqhertddttdejnecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecuff homhgrihhnpeguphgukhdrohhrghenucfkphepjeejrddufeegrddvtdefrddukeegnecu rfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtne cuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 4A34BE4176; Fri, 22 Mar 2019 04:49:05 -0400 (EDT) From: Thomas Monjalon To: Pradeep Satyanarayana , David Wilder Cc: Shahaf Shuler , Chao Zhu , Dekel Peled , dev@dpdk.org, David Christensen , Ori Kam , Yongseok Koh , konstantin.ananyev@intel.com, ola.liljedahl@arm.com, honnappa.nagarahalli@arm.com, bruce.richardson@intel.com Date: Fri, 22 Mar 2019 09:49:03 +0100 Message-ID: <11283309.AIL3tCH6tf@xps> In-Reply-To: References: <1552913893-43407-1-git-send-email-dekelp@mellanox.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2019 08:49:09 -0000 We need to agree on the definitions. Please see below, 22/03/2019 02:40, Pradeep Satyanarayana: > Shahaf Shuler wrote on 03/21/2019 01:49:39 AM: > > Pradeep Satyanarayana wrote on Thu 3/21/2019 12:41 > AM: > > >> > So far, when not running on power, we used the rte_wmb for that. > > >> On x86 and ARM systems it provided the needed guarantees. > > >> > It is also mentioned in the barrier doxygen on ARM arch: > > >> > " > > >> > Write memory barrier. > > >> > > > >> > Guarantees that the STORE operations generated before the barrier > > >> > occur before the STORE operations generated after. > > >> > " > > >> > > > >> > It doesn't restrict to store to system memory only. > > >> > w/ power is on somewhat different and in fact rte_mb is required. > > >> It obviously miss the point of those barrier if we will need to use > > >> a different barrier based on the system arch. > > >> > > > >> > We need to align the definition of the different barriers in DPDK: > > >> > 1. need a clear documentation of each. this should be global and > > >> not part of the specific implementation on each arch. > > > > > >A single approach may not work for all architectures. Power is differe= nt > > >from others, so we need to be able to accommodate that. More comments > below. > > > > it don't get this claim. > > It is ok to have some differences between the different arch, but > > here you implement a well-defined barrier - rte_wmb. > > if you see a need we can discuss to define a **new** barrier which > > sync STORE only to system memory, and will be able to utilize the > > lwsync command. > > > > > > > >> The global definition is in lib/librte_eal/common/include/ > > generic/rte_atomic.h > > >> > > >> There are some copy/paste in Arm32 and PPC that I will remove. > > >> > > >> > 2. either modify ppc rte_wmb to match ARM and x86 ones or to > > >> define a new type of barrier which will sync between both I/O and > > >> stores to systems memory. > > >> > > >> The basic memory barrier of DPDK does not mention > > >> a difference between I/O and system memory. > > > > > >In the case of Power, sync will cater to both I/O and system > > memory. However, that > > >is too big a hammer in all cases. > > > > rte_wmb requires such sync. you propose to have the wrong barrier in > > favor of performance. > > to mitigate this you can take my suggestion above and define a new, > > more lightweight one. > > > > > > > >> It is not explicit (yet) but I assume it is protecting both. > > >> So, in my opinion, we need to make it explicit in the doc, > > >> and fix the PPC implementation to comply with this definition. > > >> > > >> Anyway, I don't see any significant effort from IBM to move from > > >> the alpha support stage to a real Open Source support. > > >> PS: sending a mail every two months, to promise improvements, is > > not enough! > > > > > > > [=E2=80=A6] > > > > > > > >We should retain lwsync, should not be removed as discussed in here: > > > > > >http://mails.dpdk.org/archives/dev/2019-March/126746.html > > > > i don't agree. > > it is very clear the rte_wmb implementation in power is broken and > > we need to fix this right away before other customers will hit the > > same issue. >=20 >=20 > In the DPDK source I see a couple of different classes of memory barriers. > I am > not clear on the usage of these in the drivers, but I would think the > guidelines > to be as shown below (for Power): >=20 > - rte_[rw]mb (general memory barrier) --> should be lwsync This is what may be discussed. The assumption is that the general memory barrier should cover all cases (CPU caches, SMP and I/O). That's why we think it should "sync" for Power. > - rte_smp_[rw]mb (SMP memory barrier) -->should be lwsync > - rte_io_[rw]mb (I/O memory barrier) --> should be sync > - rte_cio_[rw]mb (coherent I/O memory barrier) -->should be sync >=20 > lwsync is appropriate for cases where CPUs are accessing cacheable memory > (i.e. Memory Coherence Required) while the sync instruction should be used > in all other cases. >=20 > With the patch in: > http://mails.dpdk.org/archives/dev/2019-March/126746.html >=20 > It converts even the rte_smp_[rw]mb into sync. That is not what the > rte_smp*() should > be implementing as per the guidelines above. >=20 > static __rte_always_inline void > mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volatile struct mlx5_wqe > *wqe, > int cond) > { > uint64_t *dst =3D (uint64_t *)((uintptr_t)txq->bf_reg); > volatile uint64_t *src =3D ((volatile uint64_t *)wqe); >=20 > rte_cio_wmb(); --> would rte_cio_rmb() be more appropriate? > *txq->qp_db =3D rte_cpu_to_be_32(txq->wqe_ci); > /* Ensure ordering between DB record and BF copy. */ > rte_wmb(); --> what has been established is that for Power we need > "sync" instead of lwsync > We are dealing with device memory -should we be using an > rte_io_wmb() here? >=20 > mlx5_uar_write64_relaxed(*src, dst, txq->uar_lock); > if (cond) > rte_wmb(); --> what about here? Are there conditions when > we need sync? > } From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id F1668A00E6 for ; Fri, 22 Mar 2019 09:49:10 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5D98A1B55C; Fri, 22 Mar 2019 09:49:10 +0100 (CET) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by dpdk.org (Postfix) with ESMTP id CD73E37B0 for ; Fri, 22 Mar 2019 09:49:08 +0100 (CET) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 27A0C236CA; Fri, 22 Mar 2019 04:49:08 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 22 Mar 2019 04:49:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=pguyjRnaLB1k2QUcM5EnvEc+OkVrNpDYNMTorH8v27Q=; b=S67bG1RroWMa /1wZyYw/tf2uJNi3sDLjEpAaI8lUmMQqyRI6YikEortQC0dipWinWdB6m4Boa8Kp uEUzh7eeVCxziTH7yiSUlpPOcg/aKbvOGQmObUMiVzk/ocGFjZSFCAOV/NfAnb0J 82Q3mmsnjgeiZyXTYt3Cp85CibBxjlo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=pguyjRnaLB1k2QUcM5EnvEc+OkVrNpDYNMTorH8v2 7Q=; b=a8HjFp0ANE4uNBbhGsEhT5g3jy2f3qbP55FdW6S1t57v0ABwM9pB/qPG+ YMmE9/s/+ExU75rKeNhmLolJOrpUk5awDesAVDLSBmiv8HZKdPEELUrBfBcJGrUI m2aPlB1XIMioTS69tf6vPlgjA9NOznlnIfxZM6q4xx0xitxIIcEcEYKetmCHsLsy UovzpIWPGhzwzzOtLIGCq2hzQlbfyJs1OKGncAB+c48B0hI6Y/gvl9rnFy4fyGw5 S7ty94apUAezjx9397HABu8Cp8M9OSBMrYX7nN1I6KUy7ST+7dAn4Ej9QapCHDEa bRz0BQiB1fxk76RTkLhA6V+lnz9Sg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrjedtgdduvdehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkfgjfhgggfgtsehtqhertddttdejnecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecuff homhgrihhnpeguphgukhdrohhrghenucfkphepjeejrddufeegrddvtdefrddukeegnecu rfgrrhgrmhepmhgrihhlfhhrohhmpehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtne cuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 4A34BE4176; Fri, 22 Mar 2019 04:49:05 -0400 (EDT) From: Thomas Monjalon To: Pradeep Satyanarayana , David Wilder Cc: Shahaf Shuler , Chao Zhu , Dekel Peled , dev@dpdk.org, David Christensen , Ori Kam , Yongseok Koh , konstantin.ananyev@intel.com, ola.liljedahl@arm.com, honnappa.nagarahalli@arm.com, bruce.richardson@intel.com Date: Fri, 22 Mar 2019 09:49:03 +0100 Message-ID: <11283309.AIL3tCH6tf@xps> In-Reply-To: References: <1552913893-43407-1-git-send-email-dekelp@mellanox.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190322084903.WVYzJMDCQKrqYJIpEiJP_yv1gyihQD9LUpDFCQd4dnI@z> We need to agree on the definitions. Please see below, 22/03/2019 02:40, Pradeep Satyanarayana: > Shahaf Shuler wrote on 03/21/2019 01:49:39 AM: > > Pradeep Satyanarayana wrote on Thu 3/21/2019 12:41 > AM: > > >> > So far, when not running on power, we used the rte_wmb for that. > > >> On x86 and ARM systems it provided the needed guarantees. > > >> > It is also mentioned in the barrier doxygen on ARM arch: > > >> > " > > >> > Write memory barrier. > > >> > > > >> > Guarantees that the STORE operations generated before the barrier > > >> > occur before the STORE operations generated after. > > >> > " > > >> > > > >> > It doesn't restrict to store to system memory only. > > >> > w/ power is on somewhat different and in fact rte_mb is required. > > >> It obviously miss the point of those barrier if we will need to use > > >> a different barrier based on the system arch. > > >> > > > >> > We need to align the definition of the different barriers in DPDK: > > >> > 1. need a clear documentation of each. this should be global and > > >> not part of the specific implementation on each arch. > > > > > >A single approach may not work for all architectures. Power is differe= nt > > >from others, so we need to be able to accommodate that. More comments > below. > > > > it don't get this claim. > > It is ok to have some differences between the different arch, but > > here you implement a well-defined barrier - rte_wmb. > > if you see a need we can discuss to define a **new** barrier which > > sync STORE only to system memory, and will be able to utilize the > > lwsync command. > > > > > > > >> The global definition is in lib/librte_eal/common/include/ > > generic/rte_atomic.h > > >> > > >> There are some copy/paste in Arm32 and PPC that I will remove. > > >> > > >> > 2. either modify ppc rte_wmb to match ARM and x86 ones or to > > >> define a new type of barrier which will sync between both I/O and > > >> stores to systems memory. > > >> > > >> The basic memory barrier of DPDK does not mention > > >> a difference between I/O and system memory. > > > > > >In the case of Power, sync will cater to both I/O and system > > memory. However, that > > >is too big a hammer in all cases. > > > > rte_wmb requires such sync. you propose to have the wrong barrier in > > favor of performance. > > to mitigate this you can take my suggestion above and define a new, > > more lightweight one. > > > > > > > >> It is not explicit (yet) but I assume it is protecting both. > > >> So, in my opinion, we need to make it explicit in the doc, > > >> and fix the PPC implementation to comply with this definition. > > >> > > >> Anyway, I don't see any significant effort from IBM to move from > > >> the alpha support stage to a real Open Source support. > > >> PS: sending a mail every two months, to promise improvements, is > > not enough! > > > > > > > [=E2=80=A6] > > > > > > > >We should retain lwsync, should not be removed as discussed in here: > > > > > >http://mails.dpdk.org/archives/dev/2019-March/126746.html > > > > i don't agree. > > it is very clear the rte_wmb implementation in power is broken and > > we need to fix this right away before other customers will hit the > > same issue. >=20 >=20 > In the DPDK source I see a couple of different classes of memory barriers. > I am > not clear on the usage of these in the drivers, but I would think the > guidelines > to be as shown below (for Power): >=20 > - rte_[rw]mb (general memory barrier) --> should be lwsync This is what may be discussed. The assumption is that the general memory barrier should cover all cases (CPU caches, SMP and I/O). That's why we think it should "sync" for Power. > - rte_smp_[rw]mb (SMP memory barrier) -->should be lwsync > - rte_io_[rw]mb (I/O memory barrier) --> should be sync > - rte_cio_[rw]mb (coherent I/O memory barrier) -->should be sync >=20 > lwsync is appropriate for cases where CPUs are accessing cacheable memory > (i.e. Memory Coherence Required) while the sync instruction should be used > in all other cases. >=20 > With the patch in: > http://mails.dpdk.org/archives/dev/2019-March/126746.html >=20 > It converts even the rte_smp_[rw]mb into sync. That is not what the > rte_smp*() should > be implementing as per the guidelines above. >=20 > static __rte_always_inline void > mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volatile struct mlx5_wqe > *wqe, > int cond) > { > uint64_t *dst =3D (uint64_t *)((uintptr_t)txq->bf_reg); > volatile uint64_t *src =3D ((volatile uint64_t *)wqe); >=20 > rte_cio_wmb(); --> would rte_cio_rmb() be more appropriate? > *txq->qp_db =3D rte_cpu_to_be_32(txq->wqe_ci); > /* Ensure ordering between DB record and BF copy. */ > rte_wmb(); --> what has been established is that for Power we need > "sync" instead of lwsync > We are dealing with device memory -should we be using an > rte_io_wmb() here? >=20 > mlx5_uar_write64_relaxed(*src, dst, txq->uar_lock); > if (cond) > rte_wmb(); --> what about here? Are there conditions when > we need sync? > }