From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by dpdk.org (Postfix) with ESMTP id 655D61B277 for ; Wed, 20 Mar 2019 23:40:46 +0100 (CET) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x2KMcjDX095432 for ; Wed, 20 Mar 2019 18:40:45 -0400 Received: from smtp.notes.na.collabserv.com (smtp.notes.na.collabserv.com [192.155.248.91]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rbxdgr87r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 20 Mar 2019 18:40:45 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Wed, 20 Mar 2019 22:40:44 -0000 Received: from us1a3-smtp07.a3.dal06.isc4sb.com (10.146.103.14) by smtp.notes.na.collabserv.com (10.106.227.143) with smtp.notes.na.collabserv.com ESMTP; Wed, 20 Mar 2019 22:40:38 -0000 Received: from us1a3-mail95.a3.dal06.isc4sb.com ([10.146.21.14]) by us1a3-smtp07.a3.dal06.isc4sb.com with ESMTP id 2019032022403797-1113975 ; Wed, 20 Mar 2019 22:40:37 +0000 MIME-Version: 1.0 In-Reply-To: <12440555.pBuX5BjkXC@xps> To: Thomas Monjalon Cc: Chao Zhu , Dekel Peled , "dev@dpdk.org" , Ori Kam , Shahaf Shuler , "stable@dpdk.org" , Yongseok Koh , "David Christensen" , "David Wilder" From: "Pradeep Satyanarayana" Date: Wed, 20 Mar 2019 14:40:36 -0800 References: <1552913893-43407-1-git-send-email-dekelp@mellanox.com> <1789153.zrlSK8XYcq@xps> <12440555.pBuX5BjkXC@xps> X-KeepSent: 129065AB:1B264FB9-882583C3:00792656; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1EXT SHF993 September 20, 2018 X-LLNOutbound: False X-Disclaimed: 15151 X-TNEFEvaluated: 1 x-cbid: 19032022-9951-0000-0000-00000BC44F17 X-IBM-SpamModules-Scores: BY=0.171593; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.441973; ST=0; TS=0; UL=0; ISC=; MB=0.253333 X-IBM-SpamModules-Versions: BY=3.00010789; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000281; SDB=6.01177266; UDB=6.00615839; IPR=6.00957961; BA=6.00006264; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00026081; XFM=3.00000015; UTC=2019-03-20 22:40:43 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2019-03-20 13:12:49 - 6.00009718 x-cbparentid: 19032022-9952-0000-0000-00001C0EE063 Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-03-20_14:, , signatures=0 X-Proofpoint-Spam-Reason: safe X-Mailman-Approved-At: Thu, 21 Mar 2019 10:29:35 +0100 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Mar 2019 22:40:46 -0000 Thomas Monjalon wrote on 03/19/2019 01:45:01 PM: > From: Thomas Monjalon > To: Shahaf Shuler > Cc: Dekel Peled , Chao Zhu > , Yongseok Koh , > "dev@dpdk.org" , Ori Kam , > "stable@dpdk.org" , pradeep@us.ibm.com > Date: 03/19/2019 01:45 PM > Subject: Re: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER > > 19/03/2019 20:42, Shahaf Shuler: > > Tuesday, March 19, 2019 1:15 PM, Thomas Monjalon: > > > Subject: Re: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER > > > > > > Guys, please let's avoid top-post. > > > > > > You are both not replying to each other: > > > > > > 1/ Dekel mentioned the IBM doc but Chao did not argue about the lack of IO > > > protection with lwsync. > > > We assume that rte=5Fmb should protect any access including IO. > > > > > > 2/ Chao asked about the semantic of the barrier used in mlx5 > code, but Dekel > > > did not reply about the semantic: are we protecting IO or general memory > > > access? > > > > In mlx5 code we want to sync between two different writes: > > 1. write to system memory (RAM) > > 2. write to MMIO memory (device) > > > > We need #1 to be visible on host memory before #2 is committed to NIC. > > We want to have a single type of barrier which will translate to > the correct assembly based on the system arch, and in addition we > want it light-weight as possible. > > > > So far, when not running on power, we used the rte=5Fwmb for that. > On x86 and ARM systems it provided the needed guarantees. > > It is also mentioned in the barrier doxygen on ARM arch: > > " > > Write memory barrier. > > > > Guarantees that the STORE operations generated before the barrier > > occur before the STORE operations generated after. > > " > > > > It doesn't restrict to store to system memory only. > > w/ power is on somewhat different and in fact rte=5Fmb is required. > It obviously miss the point of those barrier if we will need to use > a different barrier based on the system arch. > > > > We need to align the definition of the different barriers in DPDK: > > 1. need a clear documentation of each. this should be global and > not part of the specific implementation on each arch. A single approach may not work for all architectures. Power is different from others, so we need to be able to accommodate that. More comments below. > > The global definition is in lib/librte=5Feal/common/include/generic/rte=5Fatomic.h > > There are some copy/paste in Arm32 and PPC that I will remove. > > > 2. either modify ppc rte=5Fwmb to match ARM and x86 ones or to > define a new type of barrier which will sync between both I/O and > stores to systems memory. > > The basic memory barrier of DPDK does not mention > a difference between I/O and system memory. In the case of Power, sync will cater to both I/O and system memory. However, that is too big a hammer in all cases. > It is not explicit (yet) but I assume it is protecting both. > So, in my opinion, we need to make it explicit in the doc, > and fix the PPC implementation to comply with this definition. > > Anyway, I don't see any significant effort from IBM to move from > the alpha support stage to a real Open Source support. > PS: sending a mail every two months, to promise improvements, is not enough! We have been working to set up CI across several branches and during the course of this testing we discovered a segfault with mlx5 PMD and reported it to Mellanox. We have worked with them to validate several iterations of the patches. Unfortunately none of this has been visible to the community. We should be submitting some patches for DTS to address issues seen on Power after next week (vacation plans). It is taking time and effort and is not visible externally since we are in catch-up mode. We discussed the patches and believe: http://mails.dpdk.org/archives/dev/2019-March/126712.html looks right and changes only the Power platform. Dave Christensen had previously validated the following patch from Mellanox: http://mails.dpdk.org/archives/dev/2019-March/126726.html We should retain lwsync, should not be removed as discussed in here: http://mails.dpdk.org/archives/dev/2019-March/126746.html > > ----------------- > > > > 19/03/2019 11:05, Dekel Peled: > > > > Hi, > > > > > > > > For ppc, rte=5Fio=5Fmb() is defined as rte=5Fmb(), which is defined > as asm sync. > > > > According to comments in arch/ppc=5F64/rte=5Fatomic.h, rte=5Fwmb() = and > > > rte=5Frmb() are the same as rte=5Fmb(), for store and load respective= ly. > > > > My patch propose to define rte=5Fwmb() and rte=5Frmb() as asm sync, like > > > rte=5Fmb(), since using lwsync is incorrect for them. > > > > > > > > Regards, > > > > Dekel > > > > > > > > From: Chao Zhu > > > > > Dekel=A3=AC > > > > > > > > > > To control the memory order for device memory, I think you should > > > > > use > > > > > rte=5Fio=5Fmb() instead of rte=5Fmb(). This will generate correct result. > > > > > rte=5Fwmb() is used for system memory. > > > > > > > > > > From: Dekel Peled > > > > > > > > > > > > From previous patch description: "to improve performance on PPC64, > > > > > > use light weight sync instruction instead of sync instruction." > > > > > > > > > > > > Excerpt from IBM doc [1], section "Memory barrier instructions": > > > > > > "The second form of the sync instruction is light-weight > sync, or lwsync. > > > > > > This form is used to control ordering for storage accesses to > > > > > > system memory only. It does not create a memory barrier for > > > > > > accesses to device > > > > > memory." > > > > > > > > > > > > This patch removes the use of lwsync, so calls to rte=5Fwmb() a= nd > > > > > > rte=5Frmb() will provide correct memory barrier to ensure order of > > > > > > accesses to system memory and device memory. Thanks Pradeep pradeep@us.ibm.com From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 35034A00E6 for ; Thu, 21 Mar 2019 10:29:44 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D44AF1B493; Thu, 21 Mar 2019 10:29:38 +0100 (CET) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by dpdk.org (Postfix) with ESMTP id 655D61B277 for ; Wed, 20 Mar 2019 23:40:46 +0100 (CET) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x2KMcjDX095432 for ; Wed, 20 Mar 2019 18:40:45 -0400 Received: from smtp.notes.na.collabserv.com (smtp.notes.na.collabserv.com [192.155.248.91]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rbxdgr87r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 20 Mar 2019 18:40:45 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Wed, 20 Mar 2019 22:40:44 -0000 Received: from us1a3-smtp07.a3.dal06.isc4sb.com (10.146.103.14) by smtp.notes.na.collabserv.com (10.106.227.143) with smtp.notes.na.collabserv.com ESMTP; Wed, 20 Mar 2019 22:40:38 -0000 Received: from us1a3-mail95.a3.dal06.isc4sb.com ([10.146.21.14]) by us1a3-smtp07.a3.dal06.isc4sb.com with ESMTP id 2019032022403797-1113975 ; Wed, 20 Mar 2019 22:40:37 +0000 MIME-Version: 1.0 In-Reply-To: <12440555.pBuX5BjkXC@xps> To: Thomas Monjalon Cc: Chao Zhu , Dekel Peled , "dev@dpdk.org" , Ori Kam , Shahaf Shuler , "stable@dpdk.org" , Yongseok Koh , "David Christensen" , "David Wilder" From: "Pradeep Satyanarayana" Date: Wed, 20 Mar 2019 14:40:36 -0800 References: <1552913893-43407-1-git-send-email-dekelp@mellanox.com> <1789153.zrlSK8XYcq@xps> <12440555.pBuX5BjkXC@xps> X-KeepSent: 129065AB:1B264FB9-882583C3:00792656; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1EXT SHF993 September 20, 2018 X-LLNOutbound: False X-Disclaimed: 15151 X-TNEFEvaluated: 1 x-cbid: 19032022-9951-0000-0000-00000BC44F17 X-IBM-SpamModules-Scores: BY=0.171593; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.441973; ST=0; TS=0; UL=0; ISC=; MB=0.253333 X-IBM-SpamModules-Versions: BY=3.00010789; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000281; SDB=6.01177266; UDB=6.00615839; IPR=6.00957961; BA=6.00006264; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00026081; XFM=3.00000015; UTC=2019-03-20 22:40:43 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2019-03-20 13:12:49 - 6.00009718 x-cbparentid: 19032022-9952-0000-0000-00001C0EE063 Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-03-20_14:, , signatures=0 X-Proofpoint-Spam-Reason: safe X-Mailman-Approved-At: Thu, 21 Mar 2019 10:29:35 +0100 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190320224036.LWgBiKYIQXeWq4WPw9-ECaCTMq40ADaid8g6UtoisTk@z> Thomas Monjalon wrote on 03/19/2019 01:45:01 PM: > From: Thomas Monjalon > To: Shahaf Shuler > Cc: Dekel Peled , Chao Zhu > , Yongseok Koh , > "dev@dpdk.org" , Ori Kam , > "stable@dpdk.org" , pradeep@us.ibm.com > Date: 03/19/2019 01:45 PM > Subject: Re: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER > > 19/03/2019 20:42, Shahaf Shuler: > > Tuesday, March 19, 2019 1:15 PM, Thomas Monjalon: > > > Subject: Re: [PATCH] eal/ppc: remove fix of memory barrier for IBM POWER > > > > > > Guys, please let's avoid top-post. > > > > > > You are both not replying to each other: > > > > > > 1/ Dekel mentioned the IBM doc but Chao did not argue about the lack of IO > > > protection with lwsync. > > > We assume that rte=5Fmb should protect any access including IO. > > > > > > 2/ Chao asked about the semantic of the barrier used in mlx5 > code, but Dekel > > > did not reply about the semantic: are we protecting IO or general memory > > > access? > > > > In mlx5 code we want to sync between two different writes: > > 1. write to system memory (RAM) > > 2. write to MMIO memory (device) > > > > We need #1 to be visible on host memory before #2 is committed to NIC. > > We want to have a single type of barrier which will translate to > the correct assembly based on the system arch, and in addition we > want it light-weight as possible. > > > > So far, when not running on power, we used the rte=5Fwmb for that. > On x86 and ARM systems it provided the needed guarantees. > > It is also mentioned in the barrier doxygen on ARM arch: > > " > > Write memory barrier. > > > > Guarantees that the STORE operations generated before the barrier > > occur before the STORE operations generated after. > > " > > > > It doesn't restrict to store to system memory only. > > w/ power is on somewhat different and in fact rte=5Fmb is required. > It obviously miss the point of those barrier if we will need to use > a different barrier based on the system arch. > > > > We need to align the definition of the different barriers in DPDK: > > 1. need a clear documentation of each. this should be global and > not part of the specific implementation on each arch. A single approach may not work for all architectures. Power is different from others, so we need to be able to accommodate that. More comments below. > > The global definition is in lib/librte=5Feal/common/include/generic/rte=5Fatomic.h > > There are some copy/paste in Arm32 and PPC that I will remove. > > > 2. either modify ppc rte=5Fwmb to match ARM and x86 ones or to > define a new type of barrier which will sync between both I/O and > stores to systems memory. > > The basic memory barrier of DPDK does not mention > a difference between I/O and system memory. In the case of Power, sync will cater to both I/O and system memory. However, that is too big a hammer in all cases. > It is not explicit (yet) but I assume it is protecting both. > So, in my opinion, we need to make it explicit in the doc, > and fix the PPC implementation to comply with this definition. > > Anyway, I don't see any significant effort from IBM to move from > the alpha support stage to a real Open Source support. > PS: sending a mail every two months, to promise improvements, is not enough! We have been working to set up CI across several branches and during the course of this testing we discovered a segfault with mlx5 PMD and reported it to Mellanox. We have worked with them to validate several iterations of the patches. Unfortunately none of this has been visible to the community. We should be submitting some patches for DTS to address issues seen on Power after next week (vacation plans). It is taking time and effort and is not visible externally since we are in catch-up mode. We discussed the patches and believe: http://mails.dpdk.org/archives/dev/2019-March/126712.html looks right and changes only the Power platform. Dave Christensen had previously validated the following patch from Mellanox: http://mails.dpdk.org/archives/dev/2019-March/126726.html We should retain lwsync, should not be removed as discussed in here: http://mails.dpdk.org/archives/dev/2019-March/126746.html > > ----------------- > > > > 19/03/2019 11:05, Dekel Peled: > > > > Hi, > > > > > > > > For ppc, rte=5Fio=5Fmb() is defined as rte=5Fmb(), which is defined > as asm sync. > > > > According to comments in arch/ppc=5F64/rte=5Fatomic.h, rte=5Fwmb() = and > > > rte=5Frmb() are the same as rte=5Fmb(), for store and load respective= ly. > > > > My patch propose to define rte=5Fwmb() and rte=5Frmb() as asm sync, like > > > rte=5Fmb(), since using lwsync is incorrect for them. > > > > > > > > Regards, > > > > Dekel > > > > > > > > From: Chao Zhu > > > > > Dekel=A3=AC > > > > > > > > > > To control the memory order for device memory, I think you should > > > > > use > > > > > rte=5Fio=5Fmb() instead of rte=5Fmb(). This will generate correct result. > > > > > rte=5Fwmb() is used for system memory. > > > > > > > > > > From: Dekel Peled > > > > > > > > > > > > From previous patch description: "to improve performance on PPC64, > > > > > > use light weight sync instruction instead of sync instruction." > > > > > > > > > > > > Excerpt from IBM doc [1], section "Memory barrier instructions": > > > > > > "The second form of the sync instruction is light-weight > sync, or lwsync. > > > > > > This form is used to control ordering for storage accesses to > > > > > > system memory only. It does not create a memory barrier for > > > > > > accesses to device > > > > > memory." > > > > > > > > > > > > This patch removes the use of lwsync, so calls to rte=5Fwmb() a= nd > > > > > > rte=5Frmb() will provide correct memory barrier to ensure order of > > > > > > accesses to system memory and device memory. Thanks Pradeep pradeep@us.ibm.com