From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7ECFAA0514; Wed, 15 Jan 2020 19:35:27 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8AB7F1C2A9; Wed, 15 Jan 2020 19:35:25 +0100 (CET) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id AB2231C1E9; Wed, 15 Jan 2020 19:35:22 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Jan 2020 10:35:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,323,1574150400"; d="scan'208";a="219365036" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.35]) ([10.237.221.35]) by fmsmga007.fm.intel.com with ESMTP; 15 Jan 2020 10:35:18 -0800 To: =?UTF-8?B?5pa557uf5rWpNTA0NTA=?= Cc: thomas@monjalon.net, arybchenko@solarflare.com, dev@dpdk.org, stable@dpdk.org, jia.guo@intel.com, cunming.liang@intel.com, qi.z.zhang@intel.com, jungle845943968@outlook.com, Jerin Jacob Kollanukkaran References: From: Ferruh Yigit Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= mQINBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABtCVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+iQJUBBMBCgA+AhsDAh4BAheABQsJCAcDBRUK CQgLBRYCAwEAFiEE0jZTh0IuwoTjmYHH+TPrQ98TYR8FAl1meboFCQlupOoACgkQ+TPrQ98T YR9ACBAAv2tomhyxY0Tp9Up7mNGLfEdBu/7joB/vIdqMRv63ojkwr9orQq5V16V/25+JEAD0 60cKodBDM6HdUvqLHatS8fooWRueSXHKYwJ3vxyB2tWDyZrLzLI1jxEvunGodoIzUOtum0Ce gPynnfQCelXBja0BwLXJMplM6TY1wXX22ap0ZViC0m714U5U4LQpzjabtFtjT8qOUR6L7hfy YQ72PBuktGb00UR/N5UrR6GqB0x4W41aZBHXfUQnvWIMmmCrRUJX36hOTYBzh+x86ULgg7H2 1499tA4o6rvE13FiGccplBNWCAIroAe/G11rdoN5NBgYVXu++38gTa/MBmIt6zRi6ch15oLA Ln2vHOdqhrgDuxjhMpG2bpNE36DG/V9WWyWdIRlz3NYPCDM/S3anbHlhjStXHOz1uHOnerXM 1jEjcsvmj1vSyYoQMyRcRJmBZLrekvgZeh7nJzbPHxtth8M7AoqiZ/o/BpYU+0xZ+J5/szWZ aYxxmIRu5ejFf+Wn9s5eXNHmyqxBidpCWvcbKYDBnkw2+Y9E5YTpL0mS0dCCOlrO7gca27ux ybtbj84aaW1g0CfIlUnOtHgMCmz6zPXThb+A8H8j3O6qmPoVqT3qnq3Uhy6GOoH8Fdu2Vchh TWiF5yo+pvUagQP6LpslffufSnu+RKAagkj7/RSuZV25Ag0EV9ZMvgEQAKc0Db17xNqtSwEv mfp4tkddwW9XA0tWWKtY4KUdd/jijYqc3fDD54ESYpV8QWj0xK4YM0dLxnDU2IYxjEshSB1T qAatVWz9WtBYvzalsyTqMKP3w34FciuL7orXP4AibPtrHuIXWQOBECcVZTTOdZYGAzaYzxiA ONzF9eTiwIqe9/oaOjTwTLnOarHt16QApTYQSnxDUQljeNvKYt1lZE/gAUUxNLWsYyTT+22/ vU0GDUahsJxs1+f1yEr+OGrFiEAmqrzpF0lCS3f/3HVTU6rS9cK3glVUeaTF4+1SK5ZNO35p iVQCwphmxa+dwTG/DvvHYCtgOZorTJ+OHfvCnSVjsM4kcXGjJPy3JZmUtyL9UxEbYlrffGPQ I3gLXIGD5AN5XdAXFCjjaID/KR1c9RHd7Oaw0Pdcq9UtMLgM1vdX8RlDuMGPrj5sQrRVbgYH fVU/TQCk1C9KhzOwg4Ap2T3tE1umY/DqrXQgsgH71PXFucVjOyHMYXXugLT8YQ0gcBPHy9mZ qw5mgOI5lCl6d4uCcUT0l/OEtPG/rA1lxz8ctdFBVOQOxCvwRG2QCgcJ/UTn5vlivul+cThi 6ERPvjqjblLncQtRg8izj2qgmwQkvfj+h7Ex88bI8iWtu5+I3K3LmNz/UxHBSWEmUnkg4fJl Rr7oItHsZ0ia6wWQ8lQnABEBAAGJAjwEGAEKACYCGwwWIQTSNlOHQi7ChOOZgcf5M+tD3xNh HwUCXWZ5wAUJB3FgggAKCRD5M+tD3xNhH2O+D/9OEz62YuJQLuIuOfL67eFTIB5/1+0j8Tsu o2psca1PUQ61SZJZOMl6VwNxpdvEaolVdrpnSxUF31kPEvR0Igy8HysQ11pj8AcgH0a9FrvU /8k2Roccd2ZIdpNLkirGFZR7LtRw41Kt1Jg+lafI0efkiHKMT/6D/P1EUp1RxOBNtWGV2hrd 0Yg9ds+VMphHHU69fDH02SwgpvXwG8Qm14Zi5WQ66R4CtTkHuYtA63sS17vMl8fDuTCtvfPF HzvdJLIhDYN3Mm1oMjKLlq4PUdYh68Fiwm+boJoBUFGuregJFlO3hM7uHBDhSEnXQr5mqpPM 6R/7Q5BjAxrwVBisH0yQGjsWlnysRWNfExAE2sRePSl0or9q19ddkRYltl6X4FDUXy2DTXa9 a+Fw4e1EvmcF3PjmTYs9IE3Vc64CRQXkhujcN4ZZh5lvOpU8WgyDxFq7bavFnSS6kx7Tk29/ wNJBp+cf9qsQxLbqhW5kfORuZGecus0TLcmpZEFKKjTJBK9gELRBB/zoN3j41hlEl7uTUXTI JQFLhpsFlEdKLujyvT/aCwP3XWT+B2uZDKrMAElF6ltpTxI53JYi22WO7NH7MR16Fhi4R6vh FHNBOkiAhUpoXRZXaCR6+X4qwA8CwHGqHRBfYFSU/Ulq1ZLR+S3hNj2mbnSx0lBs1eEqe2vh cA== Message-ID: <32a5b3ff-afe2-75eb-f2cb-b9437a5a8d86@intel.com> Date: Wed, 15 Jan 2020 18:35:18 +0000 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH v2] Fixes: ethdev: secondary process change shared memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 1/15/2020 6:49 AM, 方统浩50450 wrote: > Hi Ferruh, thanks for your message. > > > We developed a ethtool-dpdk which is secondary process based dpdk 17.08 version. Our device > support hotplug detach, but hotplug deatch is failed when we use ethtool-dpdk.We found the > secondary process will change the shared memory when initializing.Secondary process calls > "rte_eth_dev_pci_allocate" function and enters "rte_eth_copy_pci_info" function. > (rte_eth_dev_pci_generic_probe -> rte_eth_dev_pci_allocate -> rte_eth_copy_pci_info) > Then it sets the value of struct "rte_eth_dev_data.dev_flags" to zero.In our platform, this value > is equal to 0x0003.(RTE_ETH_DEV_DETACHABLE | RTE_ETH_DEV_INTR_LSC),but after reset > the "dev_flags", the value changed to 0x0002.(RTE_ETH_DEV_DETACHABLE).So, our device hotplug > detach is failed.I found the similar problem in other dpdk version, include dpdk 19.11.Even though > the deivce hotplug detach is discarded,but i think the shared memory changed is unexpected by primary > process. I agree this is the problem. In the driver code, 'rte_eth_copy_pci_info' is called only by primary process, but the generic code is faulty. And in 19.11 additionally 'eth_dev_pci_specific_init' also seems has same problem. > > > Our driver is ixgbe, i think this problem has a little relationship with driver, Secondary process > enters "rte_eth_copy_pci_info" by "rte_eth_dev_pci_allocate".And I agree your opinion, the helper > function should simple on what it does.I have two ways to fix this problem, one is add an if-statement > > in "rte_eth_dev_pci_allocate" function to forbid secondary process enters "rte_eth_copy_pci_info" function, > another way is add an if-statement in "rte_eth_copy_pci_info" function to forbid secondary process change > shared memory.And First way need to ensure the "rte_eth_copy_pci_info" function won't be called anywhere else. > I think the second way is simple and lower risk. Yes these are the two options. I agree adding check in the 'rte_eth_copy_pci_info' covers all cases and safer. BUT my concern was adding decision making to simple/leaf function and make it harder to debug/use, instead of giving what primary/secondary process should call decision in higher level. But I just recognized that some PMDs are calling 'rte_eth_copy_pci_info' on secondary process, like mlx4 or szedata2, and most probably this is not their intention. And 'eth_dev->intr_handle' set in 'rte_eth_copy_pci_info', not calling this function may have side affect of 'eth_dev->intr_handle' not set in secondary. With above considerations I am OK to your proposal to cover all cases, Thomas, Andrew, any concern? @Fang, only can you please make a new version to update the 'rte_eth_copy_pci_info' function comment to document shared data is not updated for the secondary process? Thanks, ferruh > > > Please forgive me because my poor english.... > > > > 发件人：Ferruh Yigit > 发送日期：2020-01-14 22:45:33 > 收件人：Fang TongHao ,thomas@monjalon.net,arybchenko@solarflare.com > 抄送人：dev@dpdk.org,stable@dpdk.org,jia.guo@intel.com,cunming.liang@intel.com,qi.z.zhang@intel.com,jungle845943968@outlook.com > 主题：Re: [dpdk-dev] [PATCH v2] Fixes: ethdev: secondary process change shared memory>On 1/13/2020 5:03 AM, Fang TongHao wrote: >>> Secondary process calls “rte_eth_dev_pci_allocate” >>> function and enters rte_eth_copy_pci_info function >>> when initializing.Then it sets the value of struct >>> "rte_eth_dev_data.dev_flags" to zero and reset it, >>> but this struct is shared by primary process and >>> secondary process.To fix this bug,by adding an >>> if-statement to forbid the secondaryprocess changing >>> the above-mentioned value. >> >> Hi Fang, >> >> Thanks for the fix, I agree with the problem statement, but not sure if this >> should be handled in the helper function or in the place where the function is >> called. Helper function is simple on what it does, do we need to put the primary >> process logic in it. >> >> Can you please give more details of the bug you have encounter, is it seen by a >> specific PMD? >> >> Thanks, >> ferruh >> >>> >>> Signed-off-by: Fang TongHao >>> --- >>> lib/librte_ethdev/rte_ethdev_pci.h | 18 ++++++++++-------- >>> 1 file changed, 10 insertions(+), 8 deletions(-) >>> >>> diff --git a/lib/librte_ethdev/rte_ethdev_pci.h b/lib/librte_ethdev/rte_ethdev_pci.h >>> index ccdbb46ec..e7dae0545 100644 >>> --- a/lib/librte_ethdev/rte_ethdev_pci.h >>> +++ b/lib/librte_ethdev/rte_ethdev_pci.h >>> @@ -60,14 +60,16 @@ rte_eth_copy_pci_info(struct rte_eth_dev *eth_dev, >>> >>> eth_dev->intr_handle = &pci_dev->intr_handle; >>> >>> - eth_dev->data->dev_flags = 0; >>> - if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_LSC) >>> - eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_LSC; >>> - if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_RMV) >>> - eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_RMV; >>> - >>> - eth_dev->data->kdrv = pci_dev->kdrv; >>> - eth_dev->data->numa_node = pci_dev->device.numa_node; >>> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) { >>> + eth_dev->data->dev_flags = 0; >>> + if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_LSC) >>> + eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_LSC; >>> + if (pci_dev->driver->drv_flags & RTE_PCI_DRV_INTR_RMV) >>> + eth_dev->data->dev_flags |= RTE_ETH_DEV_INTR_RMV; >>> + >>> + eth_dev->data->kdrv = pci_dev->kdrv; >>> + eth_dev->data->numa_node = pci_dev->device.numa_node; >>> + } >>> } >>> >>> static inline int >>> >> > > > >