From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 5A1FF1B489 for ; Wed, 10 Oct 2018 19:25:22 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Oct 2018 10:25:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,365,1534834800"; d="scan'208";a="94037206" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.49]) ([10.237.221.49]) by fmsmga002.fm.intel.com with ESMTP; 10 Oct 2018 10:24:37 -0700 To: Dan Gora Cc: dev@dpdk.org References: <20180628224513.18391-1-dg@adax.com> <20180629015508.26599-1-dg@adax.com> <20180629015508.26599-3-dg@adax.com> <611163de-bef7-488b-a77b-0e1ff190f1fb@intel.com> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= xsFNBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABzSVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+wsGVBBMBAgA/AhsDBgsJCAcDAgYVCAIJCgsE FgIDAQIeAQIXgBYhBNI2U4dCLsKE45mBx/kz60PfE2EfBQJbughWBQkHwjOGAAoJEPkz60Pf E2Eft84QAIbKWqhgqRfoiw/BbXbA1+qm2o4UgkCRQ0yJgt9QsnbpOmPKydHH0ixCliNz1J8e mRXCkMini1bTpnzp7spOjQGLeAFkNFz6BMq8YF2mVWbGEDE9WgnAxZdi0eLY7ZQnHbE6AxKL SXmpe9INb6z3ztseFt7mqje/W/6DWYIMnH3Yz9KzxujFWDcq8UCAvPkxVQXLTMpauhFgYeEx Nub5HbvhxTfUkapLwRQsSd/HbywzqZ3s/bbYMjj5JO3tgMiM9g9HOjv1G2f1dQjHi5YQiTZl 1eIIqQ3pTic6ROaiZqNmQFXPsoOOFfXF8nN2zg8kl/sSdoXWHhama5hbwwtl1vdaygQYlmdK H2ueiFh/UvT3WG3waNv2eZiEbHV8Rk52Xyn2w1G90lV0fYC6Ket1Xjoch7kjwbx793Kz/RfQ rmBY8/S4DTGn3oq3dMdQY+b6+7VMUeLMMh2CXYO9ErkOq+qNTD1IY+cBAkXnaDbQfz0zbste ZGWH74FAZ9nCpDOqbRTrBL42aMGhfOWEyeA1x7+hl6JZfabBWAuf4nnCXuorKHzBXTrf7u7p fXsKQClWRW77PF1VmzrtKNVSytQAmlCWApQIw20AarFipXmVdIjHmJPU611WoyxZPb4JTOxx 5cv9B+nr/RIB+v5dcStyHCCwO1be7nBDdCgd4F6kTQPLzsFNBFfWTL4BEACnNA29e8TarUsB L5n6eLZHXcFvVwNLVlirWOClHXf44o2KnN3ww+eBEmKVfEFo9MSuGDNHS8Zw1NiGMYxLIUgd U6gGrVVs/VrQWL82pbMk6jCj98N+BXIri+6K1z+AImz7ax7iF1kDgRAnFWU0znWWBgM2mM8Y gDjcxfXk4sCKnvf6Gjo08Ey5zmqx7dekAKU2EEp8Q1EJY3jbymLdZWRP4AFFMTS1rGMk0/tt v71NBg1GobCcbNfn9chK/jhqxYhAJqq86RdJQkt3/9x1U1Oq0vXCt4JVVHmkxePtUiuWTTt+ aYlUAsKYZsWvncExvw77x2ArYDmaK0yfjh37wp0lY7DOJHFxoyT8tyWZlLci/VMRG2Ja33xj 0CN4C1yBg+QDeV3QFxQo42iA/ykdXPUR3ezmsND3XKvVLTC4DNb3V/EZQ7jBj64+bEK0VW4G B31VP00ApNQvSoczsIOAKdk97RNbpmPw6q10ILIB+9T1xbnFYzshzGF17oC0/GENIHATx8vZ masOZoDiOZQpeneLgnFE9JfzhLTxv6wNZcc/HLXRQVTkDsQr8ERtkAoHCf1E5+b5Yr7pfnE4 YuhET746o25S53ELUYPIs49qoJsEJL34/oexMfPGyPIlrbufiNyty5jc/1MRwUlhJlJ5IOHy ZUa+6CLR7GdImusFkPJUJwARAQABwsF8BBgBAgAmAhsMFiEE0jZTh0IuwoTjmYHH+TPrQ98T YR8FAlu6CHAFCQXE7zIACgkQ+TPrQ98TYR9nXxAAqNBgkYNyGuWUuy0GwDQCbu3iiMyH1+D7 llafPcK4NYy1Z4AYuVwC9nmLaoj+ozdqS3ncRo57ncRsKEJC46nDJJZYZ5LSJVn63Y3NBF86 lxQAgjj2oyZEwaLKtKbAFsXL43jv1pUGgSvWwYtDwHITXXFQto9rZEuUDRFSx4sg9OR+Q6/6 LY+nQQ3OdHlBkflzYMPcWgDcvcTAO6yasLEUf7UcYoSWTyMYjLB4QuNlXzTswzGVMssJF/vo V8lD1eqqaSUWG3STF6GVLQOr1NLvN5+kUBiEStHFxBpgSCvYY9sNV8FS6N24CAWMBl+10W+D 2h1yiiP5dOdPcBDYKsgqDD91/sP0WdyMJkwdQJtD49f9f+lYloxHnSAxMleOpyscg1pldw+i mPaUY1bmIknLhhkqfMmjywQOXpac5LRMibAAYkcB8v7y3kwELnt8mhqqZy6LUsqcWygNbH/W K3GGt5tRpeIXeJ25x8gg5EBQ0Jnvp/IbBYQfPLtXH0Myq2QuAhk/1q2yEIbVjS+7iowEZNyE 56K63WBJxsJPB2mvmLgn98GqB4G6GufP1ndS0XDti/2K0o8rep9xoY/JDGi0n0L0tk9BHyoP Y7kaEpu7UyY3nVdRLe5H1/MnFG8hdJ97WqnPS0buYZlrbTV0nRFL/NI2VABl18vEEXvNQiO+ vM8= Message-ID: <0596f426-0fc5-a222-ea68-4407669f9b22@intel.com> Date: Wed, 10 Oct 2018 18:24:35 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH v2 02/10] kni: separate releasing netdev from freeing KNI interface X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Oct 2018 17:25:22 -0000 On 9/4/2018 1:36 AM, Dan Gora wrote: > Hi Ferruh, > > I remembered now the motivation behind separating rte_kni_release() > and rte_kni_free(). > > The problem is that the DPDK thread which calls rte_kni_release() > _cannot_ be the same thread which handles callbacks from the KNI > driver via rte_kni_handle_request(). This is because the thread which > calls rte_kni_release() will be stuck down in > ioctl(RTE_KNI_IOCTL_RELEASE) when the kernel calls the > RTE_KNI_REQ_CFG_NETWORK_IF callback to the DPDK application. Since > that thread cannot call rte_kni_handle_request(), the callback would > then just timeout unless some other thread calls > rte_kni_handle_request(). > > So then you are in a bit of a chicken and egg situation. You _have_ > to have a separate thread calling rte_kni_handle_request periodically, > but that thread also _cannot_ run after rte_kni_release returns > (actually it's worse than that because it's actually after the > ioctl(RTE_KNI_IOCTL_RELEASE) returns and the fifos are freed). I see, so we have problem in both end, -userspace side and kernel side. Agreed that separating release & free may help, but I am not sure about adding a new API for KNI. Very simply, what about prevent kni_net_release() send callback to userspace? This is already not working and removing it resolves the issues you mentioned. Sample application calls rte_eth_dev_stop() after release itself, so behavior will be same. But the issues in kernel you mentioned, using `dev` after free_netdev() called should be addressed. > > So in order to resolve this, I separated the release from the freeing > stages. This allows the DPDK application to keep the > rte_kni_handle_request() thread running while rte_kni_release() is > called so that it can handle the interface state callback, then kill > that thread so that it cannot touch any 'struct rte_kni' resources, > then free the struct rte_kni resources. > > > thanks > dan > > On Wed, Aug 29, 2018 at 7:59 AM, Ferruh Yigit wrote: > >>> When the kernel network interface is removed with unregister_netdev(), >>> if the interface is up, it will generate a callback to mark the >>> interface down, which calls kni_net_release(). kni_net_release() will >>> block waiting for the DPDK application to call rte_kni_handle_request() >>> to handle the callback, but it also needs the thread in the KNI driver >>> (either the per-dev thread for multi-thread or the per-driver thread) >>> to call kni_net_poll_resp() in order to wake the thread sleeping in >>> kni_net_release (actually kni_net_process_request()). >>> >>> So now, KNI interfaces should be removed as such: >>> >>> 1) The user calls rte_kni_release(). This only unregisters the >>> netdev in the kernel, but touches nothing else. This allows all the >>> threads to run which are necessary to handle the callback into the >>> DPDK application to mark the interface down. >>> >>> 2) The user stops the thread running rte_kni_handle_request(). >>> After rte_kni_release() has been called, there will be no more >>> callbacks for that interface so it is not necessary. It cannot be >>> running at the same time that rte_kni_free() frees all of the FIFOs >>> and DPDK memory for that KNI interface. >>> >>> 3) The user calls rte_kni_free(). This performs the RTE_KNI_IOCTL_FREE >>> ioctl which calls kni_ioctl_free(). This function removes the struct >>> kni_dev from the list of interfaces to poll (and kills the per-dev >>> kthread, if configured for multi-thread), then frees the memory in >>> the FIFOs.