From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0065.outbound.protection.outlook.com [104.47.38.65]) by dpdk.org (Postfix) with ESMTP id 0BF865681 for ; Wed, 30 Nov 2016 21:55:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=GIgT9kqKt+U26foaW3o6XhFLj+siHPfPASmDHyeEpDg=; b=lWvhiwmiiMhNQE5pYL0n+Vhuh5nuxlAkwBWhKnIWQ1jUwaemH5cagSTvgUwUO0UoYQnkV+vq+u2pepv4SRqdBmsG/Sbtk5obk2NKD26DOmARV/kuSw7TbdXb4phXXl3F2Su+26vOV3CMf3sCp6mnlcsQXFHIyC90/iftcyAw+kE= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@cavium.com; Received: from svelivela-lt.caveonetworks.com (50.233.148.156) by BLUPR0701MB1714.namprd07.prod.outlook.com (10.163.85.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.707.6; Wed, 30 Nov 2016 20:55:00 +0000 Date: Thu, 1 Dec 2016 02:24:56 +0530 From: Jerin Jacob To: "Ananyev, Konstantin" CC: "Richardson, Bruce" , Satha Rao , "Zhang, Helin" , "Wu, Jingjing" , "jianbo.liu@linaro.org" , "dev@dpdk.org" Message-ID: <20161130205455.GA5535@svelivela-lt.caveonetworks.com> References: <1479473533-9393-1-git-send-email-skoteshwar@caviumnetworks.com> <2601191342CEEE43887BDE71AB9772583F0DE265@irsmsx105.ger.corp.intel.com> <20161121221638.GA17380@svelivela-lt.caveonetworks.com> <20161122134654.GA58712@bricha3-MOBL3.ger.corp.intel.com> <20161122184930.GA11048@svelivela-lt.caveonetworks.com> <2601191342CEEE43887BDE71AB9772583F0E2477@irsmsx105.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <2601191342CEEE43887BDE71AB9772583F0E2477@irsmsx105.ger.corp.intel.com> User-Agent: Mutt/1.7.1 (2016-10-04) X-Originating-IP: [50.233.148.156] X-ClientProxiedBy: CO2PR05CA042.namprd05.prod.outlook.com (10.141.241.170) To BLUPR0701MB1714.namprd07.prod.outlook.com (10.163.85.140) X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 2:IsNmvfz5xiCjmdAlSLxLODmjgrw7HKUCag33Q3Aegh1fHmoQ++40ZVIpH5TfATcx58x3orK1KWGh/ro+KbkziNtFNsVggkpvfR6qrqm5dY0UNu5tcl7LNGfwRxNS2xjdrIWUFRHoFFMfFLuNO+Q/jeTCHTtmtts4yFb23QPcmks=; 3:R2764NO9nZgwdtKeSy9p+z8Zt+Z/ojWrdbnqe8nkQY57K7oMmJoIUx/0ZHiNjctIidkLWoW2m82ZKkbJ+vebv0MYv68mCUO3Hr/v2aPdL+pJHZCrOAP/tI5mW24tr/XI1F+I2wgL7G+xx5PCMSLunTiivfW0XswvgalK1UVkrwk= X-MS-Office365-Filtering-Correlation-Id: d100e6a0-d62d-47f9-6c80-08d41963273c X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001); SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 25:BfnkBbnmy/yxkBbeyzH3LMC9DE/W+HEFp2CEn3Tula/fnDV0Qd6Y6ae+CvsCW+wtv0CpRWvtD7OgTucEUR0Nw5+kxr0eZ1/SiW2AnamU6Q9uALkHdk11FeqrQwtqHjXWN020ty2Qwg496QKSXzUh2nMR/XPDHnhZssleMTJ2PVeCrCuPeOMe/YtHd0v9HDHJEd5lh4jRFbWFXUSvqf11Tm48tHnkF1meSse4r+thzlh7SnAz5V+CYqiIlPAJZdfMryqffSgDdRskSY4J6DaUJg5aXwFHhBXc40c1iJ6hgj2O4VxVl37Y++T2X36cbp+MFnc2YvNUfpu+u4Syt5MoiI2q8+9HTe0y6c47O+D5YmVHKfMXyroKqVa1lsYIpFfvesU8OvSUPyymL0iR8m3cxX4OhtyKS3r73nvMrAPsC+YM5gwq2ehtQKl3TOFn/+lkn6tstQcuZ9STLE6mhWNu4fysPEOhalzzrA+BDEWWvoNtmUB2mYqBY44d4S0yCMRn3C2AejEI3gUHtRz/2FFi2lagvfhTMuddXr2SKYUMFckbs+49EZakcuKl85Yw/hfYAwOQMjRNCR6gCzQGjgCFrpf+5FXqyFUWGApOK01I0kSXYPDoIB1WEU0m+yVL1ruxAFx+ipo+A1exnyCMqW1ZXbFUbhL9T4V19dI/wwANGFik+kdIw3Thl9Byt7TpqaXFeV7l/PbIqD8rMNbktTPhrw8kwGYdy6CdjwwWE6saB95BoFZp216NplbbsD7EG2DVMaqOMF5MmFPr/VuYzx/ctZqoWlNri+C1VZokbi9sKrR+D2gn3lXPw0luITupg2grw/iOa8DPUmvSrGfoawGoWqk86CP04baRIcLJX3UKazNeGYfZyhJAwO9zNRHik7rQ X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 31:A75uKXF7jbkmgQa1pMftphVCZIzaUDI3S+Ewd8MhSRd6tYIoqCgvTuNQAP9X0WaosyILuIubZ9o3rCPgRoTUwNplr+rj3+qes7NU3DasOynhIN5T902fKUCBu0xX5kz+Vzd7NU1s7+tkxZblc9hVXzYXLYhOfXStcOvRRbZG/VqNKQwnc+5OxRkoj7qnUVfC0rzIe7pH0vEAnllbiUpX3qdjCKMCRMT/rtcLqFyT+/AUiGNZE4HOEdnMN5i9hyGaxrX6baleoBOZSyJrCzY5iIxdIZ/rEXDz7uCd7KIr7gwrDLVziARpByA+59AADlgM; 20:lc/KNc0kbrmysZTX36HyrGW4sP72sTW6NIl4OgsYeN5vqqQvQvUAA0l5+Tuydj7fS5g9XmuCaCf/g1ss8cJbkk6T5w2mWtjO3Pxs4BWanA7QLj+UM4iMMVQbsqBkNHO5HR+oAIdVswy5FG8CSrhyxNK62TWxNUpdCasg8rWmH/IZtp7ErIPyO9p7TZ5hj+cctqhhxJb7RN3NV0qhTTr/AaOduAjYcLzH+vRSIkMw6S2fflg430cC1QFqDx+UfrFsgU1fCVlAL/w3HAV3O5X2Fdf9fI1YkGWMTKo0o286pa4uz5aeZFlQbAWVGhMy4re6OzL3TfOD/vkmu62XcPbqrIMqeQE5pyjjsZo+G1kLc1IE5EcRHskkDuYZJpj9sDvBXP+77VWtsE2DD/AP9DWnwtQGdWXJ3HKSUAQS39Fv06xcYxZW6O24PI14QRoyvFn+ZI3OkwPbzGTnClX/DdgClOK9q1w3W64ZHHO0U5JpCl679vMXeVcd4yjE2Y2kmddV13rHCbiECdUb3kDUFmRUMz3TrtbepOYzflGQZb+lb2V57kdLcIwaFEAUuqSzvrIeo+d8PKX3ZAS049kqSAljlowTDtwbA3JRUt3hG9AxIYg= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(180628864354917)(11711142671493)(58145275503218); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041248)(20161123562025)(20161123560025)(20161123564025)(20161123555025)(6072148); SRVR:BLUPR0701MB1714; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 4:AeTwJRciSQN+kkwwNADFd5wyVbgCvrqD82cBp4g6/SmTxbCR0sKJrQKH5thvleDx65UFYxo+EJj5EYMp/chNSmwQI5pVP1iYpCyFVHKoRSrzwv2LyiyQ0ShQpIHtgiOcThgXf7auRQBAqHyXTc6xHaSBhmC2mR3t38KGWlFpYwwG4+EUAE+5zTqLg36XEeZKpl1yWkKFMm+EE+D1F4gGDk8iXT3w1yCGgFbrLmGiaObXEH6onKuxARUprxWe3oerHmy3+7/59oyiJoY5/w3scuIeK9YNo6ByAKr7Y5reXblk/cN0gwUgRSAqF51aW4lK3cXjGzCHE80knhlCFX73UqDLvCYOV3/8Z/NVb7L04hTdCCNtc51FK1gM275wLAr9EqbSEeuALRFeCZ8xkjGY02G0QFTkLOwTJw0jTgxfQHXyW9720tXPalYXkx4wgeQ3MaJN8ZF7jowRcRdp9qEpJsfnS1TST7QBnM/nUBOVazCryTYxH2KvViuow9oF0DOeiwhtHrqbQ/0QaYMzbocrFwonxZ4YwEWcOpg/sMAYLxqw7WhGaaZaTq+gYkRuvK1VUBxmlFKqMRLBNT1OOH0NmoSlXb1uQmeCVnaa9thvU0BZaUXg6jfD8FiDcbX6MMpYJ+ZcCqlUg8Dd40+FrvX/iXoxnWMvkE9F2vjcKuVXqkogDY2d64AXFtz4tzCrjpqH X-Forefront-PRVS: 0142F22657 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(7916002)(199003)(24454002)(189002)(39450400002)(81156014)(8676002)(54356999)(6916009)(229853002)(3846002)(189998001)(15380165006)(42882006)(69596002)(46406003)(50466002)(97736004)(101416001)(305945005)(92566002)(6666003)(83506001)(81166006)(105586002)(9686002)(66066001)(2950100002)(39410400001)(23726003)(4326007)(50986999)(106356001)(47776003)(93886004)(53416004)(1076002)(76176999)(42186005)(2906002)(33656002)(733004)(4001350100001)(6116002)(7846002)(38730400001)(68736007)(110136003)(97756001)(7736002)(5660300001)(18370500001)(15760500001); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0701MB1714; H:svelivela-lt.caveonetworks.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0701MB1714; 23:9Urd8aPAw3DpwzmS+UzJbyGCu+vKR/KCRN1TiSt?= =?us-ascii?Q?niFjB0zUWGCnGBLXH4LJehnzPRwxJT7iaT9QO2ujbnQUz6P7A33RXxiNSkcu?= =?us-ascii?Q?PyBB/Yg6tWGbo9VHjhFNZWmXtyyfvNQL6LaTAiJSB51jipnfXsrUpy8lqY2E?= =?us-ascii?Q?FaWK8+zOrjDY6oWuwKXKoVgrx5U8kUdb4PgSzwaAnUIsLwFdtZ1GZ0TiFBI1?= =?us-ascii?Q?SKUowgQ29mEtNJzMhxNke6R4097/CSjMuKyWPoIGkWPIEmqLmVFYlEAi5cvd?= =?us-ascii?Q?xfR4lxrI3HRx0WTxn4dtCj1dUh6Klvixf7p3OuKAbPh0/kSF2/vzeYMaRLIO?= =?us-ascii?Q?oIg3bix/eegenqFo31ZLjFwid/hLSN2lKIFEjJ8cs/Dxvs1yKxr4cOP9yijH?= =?us-ascii?Q?HMGeUJw3GCkpPhBbjiP/u6H+DnuWRs5gx8t/gW/HSbuoJGwnAgt2BVIU/wQ6?= =?us-ascii?Q?CJIdM4nYuOWIDqXMVuDTlzgGq4QshXnblhi7+Y4ohhpgvqd9/7tgbCIemkQV?= =?us-ascii?Q?7XyoFiAQAa+7MVeKXwIX0AHWx4nAhNbjfWMEKnWbjewyX5zt5k4zVIltFWm3?= =?us-ascii?Q?paGoUGgOPKjlA8gvBqIQmS5/VvTW6eUmWLAuVnvkmLbOz2jLudb3Lm3S8xiS?= =?us-ascii?Q?0TaXixjXSPShBbrzzSNIAproMjMAldQ9PPDNG3ioFc5n9WIQt9Z0dEsEpCMK?= =?us-ascii?Q?w/aF5749ceCDJ4FNvOsVCnB1l+aPJ6pjrYVOt+RswObe2724+96hgUpneZIx?= =?us-ascii?Q?yIrfYA+KrQ3sc59jPLoYQ9uBgFOlZJWC4CeQUcCx6KrQunjuHPMPVWcQNB/z?= =?us-ascii?Q?q51QyazJayHs8CfNtG5iae/MpelqDSGwvO58qA064JaCPiUJ3qRqGarWmS1i?= =?us-ascii?Q?Zq6kwWAOvAcujiAZZamv51Q0N7XY6RemDGLWOANjpA/Gdw7BqNeGOETNKNOz?= =?us-ascii?Q?B/DYuJjB2R13awkmxPAuLrambXbfDcKmhtVkObG126+xN3deAwHvM4t/0E5F?= =?us-ascii?Q?cF8cfAVT1cDuMJY+p8cfEz4nBVOgnTdAiqZ1bwMDCaQGVUBGIecNbFbRXwFa?= =?us-ascii?Q?NFzEkHcGoMhypxSUKlzmatRCi34nM7o1wRsu95YSrsbi9laOJPwBzuRChh5q?= =?us-ascii?Q?59rQHXZqWc8XfFqzlXpsd1y3peGIWAXQ+yW2lsxzER/lWBaWY579Q9Hq8OME?= =?us-ascii?Q?tWft+rhIhEwn3DazJUQxdn5yr4UfJ1gBB3MDT164eNKtgjkFjAOsp32GI+5M?= =?us-ascii?Q?6do4S9AAoAjFZmCFq3nHv3up1uN2glaPD24vDaZfcCv7zEeEilmIDhfNeUsQ?= =?us-ascii?Q?MoCAxzlXQy7Yyl0VCd2BeCkWDgZ34W1mLewkMBNgSLU1edXEVNg1QbKQQCl9?= =?us-ascii?Q?lUIwuTovaZkz2I8AH+rN2LMg2xZg=3D?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 6:2FCSIhJs1NltUzDi7a/u4y9m+KRvI+vk7MPjfABoGBRctX5bOm4yGG0SgMzWmAUJ6KqV+Y3I9szFPaD5ThadM1TgYY1sp9QOiPBKziUMcnhC/6ZJMiNGtg5zs1H8hVK59xgyoGmXBptW7PM7ySeMMhTwH8h7q9pqMyn5rKN1SQOHJjp0NDbdrocizLDcpQ6rZwwLIokPFTL+GlnsWPMYr8/ve9CMi4zivER8UR6sqJBoAfUajYqsij8LYDnpZf4nirSNVbxPE8pFSCGXRBY59JSJb3+aXAkgfxNI6MhL0EyrMjuSKrc0ZzPje6qz914a3U6DO23MAMAhO8oCrGarKUHpeFXshBlB5qJkaCH416I=; 5:DlIwPlWTT4jI2UIbLx3NTHGR4Q9hbo2wI35GxLx7a0umpOTLWUl/z+bT0zho1sPDs4HyJgbRgQzx9NIMejy0Hn4m74NArB/eW+frGT+lcI++rlTqFjBCl4OUf3T408AW9lDHO0cYrpRiX3RRADUsww==; 24:jc4385Ew7HP3ftjud3VXX8D30/nX4hAexyMa7NmdQlwCHTJ3604sAfK9GFmpp+qnQtJDtq4VaAZKbNccofhk/NxXRKg8i+xeHjSSC11SM7s= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 7:3iDPNLzwtddCJLiu7BKZF3UHzddak288z6ONcyDrZhZClaINIRBUE4c4WRCCPmSY31NINcW7TLbsBVfnwYH/xDvX7q9EFlzjwtOLXBygmAODKysadOkrX19n410yIzW1eSeX9XPhrij0v/uZSbUf1ff2AKeYc/Ld1zrekInEpHjGfZ9VgkGLUCUDjj9Mvaanabm/AOEGRazxs8ntPT5+vMOzMrbAslItCGrbSkV4C1t4TFgLvH2S5pRwv6tXinjvQg2R3OL+yzH8C4gBIcvJhacV9WLFmJlXC+UqmSdiyFgU5SB9/dbFIx/iVVRnqG+/2i72r2F5i6CfXKuspNIBpqZqyysrU+h1oB+cGylxUA8= X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2016 20:55:00.7752 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0701MB1714 Subject: Re: [dpdk-dev] [PATCH v2] i40e: Fix eth_i40e_dev_init sequence on ThunderX X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Nov 2016 20:55:05 -0000 On Wed, Nov 30, 2016 at 05:52:02PM +0000, Ananyev, Konstantin wrote: > Hi Jerin, Hi Konstantin, > > > > > On Tue, Nov 22, 2016 at 01:46:54PM +0000, Bruce Richardson wrote: > > > On Tue, Nov 22, 2016 at 03:46:38AM +0530, Jerin Jacob wrote: > > > > On Sun, Nov 20, 2016 at 11:21:43PM +0000, Ananyev, Konstantin wrote: > > > > Hi > > > > > > > > > > > > i40e_asq_send_command: rd32 & wr32 under ThunderX gives unpredictable > > > > > > results. To solve this include rte memory barriers > > > > > > > > > > > > Signed-off-by: Satha Rao > > > > > > --- > > > > > > drivers/net/i40e/base/i40e_osdep.h | 14 ++++++++++++++ > > > > > > 1 file changed, 14 insertions(+) > > > > > > > > > > > > diff --git a/drivers/net/i40e/base/i40e_osdep.h b/drivers/net/i40e/base/i40e_osdep.h > > > > > > index 38e7ba5..ffa3160 100644 > > > > > > --- a/drivers/net/i40e/base/i40e_osdep.h > > > > > > +++ b/drivers/net/i40e/base/i40e_osdep.h > > > > > > @@ -158,7 +158,13 @@ do { \ > > > > > > ((volatile uint32_t *)((char *)(a)->hw_addr + (reg))) > > > > > > static inline uint32_t i40e_read_addr(volatile void *addr) > > > > > > { > > > > > > +#if defined(RTE_ARCH_ARM64) > > > > > > + uint32_t val = rte_le_to_cpu_32(I40E_PCI_REG(addr)); > > > > > > + rte_rmb(); > > > > > > + return val; > > > > > > > > > > If you really need an rmb/wmb with MMIO read/writes on ARM, > > > > > I think you can avoid #ifdefs here and use rte_smp_rmb/rte_smp_wmb. > > > > > BTW, I suppose if you need it for i40e, you would need it for other devices too. > > > > > > > > Yes. ARM would need for all devices(typically, the devices on external PCI bus). > > > > I guess rte_smp_rmb may not be the correct abstraction. So we need more of > > > > rte_rmb() as we need only non smp variant on IO side. I guess then it make sense to > > > > create new abstraction in eal with following variants so that each arch > > > > gets opportunity to make what it makes sense that specific platform > > > > > > > > rte_readb_relaxed > > > > rte_readw_relaxed > > > > rte_readl_relaxed > > > > rte_readq_relaxed > > > > rte_writeb_relaxed > > > > rte_writew_relaxed > > > > rte_writel_relaxed > > > > rte_writeq_relaxed > > > > rte_readb > > > > rte_readw > > > > rte_readl > > > > rte_readq > > > > rte_writeb > > > > rte_writew > > > > rte_writel > > > > rte_writeq > > > > > > > > Thoughts ? > > > > > > > > > > That seems like a lot of API calls! > > > Perhaps you can clarify - why would the rte_smp_rmb() not work for you? > > > > Currently arm64 mapped DMB as rte_smp_rmb() for smp case. > > > > Ideally for io barrier and non smp case, we need to map it as DSB and it is > > bit heavier than DMB > > Ok, so you need some new macro, like rte_io_(r|w)mb or so, that would expand into dmb > for ARM, correct? The io barrier expands to dsb. http://lxr.free-electrons.com/source/arch/arm64/include/asm/io.h#L110 > > > > > The linux kernel arm64 mappings > > http://lxr.free-electrons.com/source/arch/arm64/include/asm/io.h#L142 > > > > DMB vs DSB > > https://community.arm.com/thread/3833 > > > > The relaxed one are without any barriers.(the use case like accessing on > > chip peripherals may need only relaxed versions) > > > > Thoughts on new rte EAL abstraction? > > Looks like a lot of macros but if you guys think that would help - NP with that :) I don't have strong opinion here. If there is concern on a lot of macros then, I can introduce only "rte_io_(r|w)mb" instead of read[b|w|l|q]/write[b|w|l|q]/relaxed. let me know? > Again, in that case we probably can get rid of driver specific pci reg read/write defines. Yes. But, That's going to have a lot of change :-( If there is no objection then I will introduce "read[b|w|l|q]/write[b|w|l|q]/relaxed" and then change all external pcie drivers with new macros. > > Konstantin > > > > > > > > > /Bruce