From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 03499A04AF; Fri, 18 Sep 2020 16:38:51 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id AD7D41D9B1; Fri, 18 Sep 2020 16:38:50 +0200 (CEST) Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by dpdk.org (Postfix) with ESMTP id 7EEA01D8DC for ; Fri, 18 Sep 2020 16:38:49 +0200 (CEST) IronPort-SDR: YlkA+1JEeAy3ccE6PMEoBY4g6HNEpo/XkAo4+WIkQ+j7FRYGWpXvTJzGjiECApXO33FOPFXa3A Zw8L3gA5lLdw== X-IronPort-AV: E=McAfee;i="6000,8403,9747"; a="139951203" X-IronPort-AV: E=Sophos;i="5.77,274,1596524400"; d="scan'208";a="139951203" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Sep 2020 07:38:48 -0700 IronPort-SDR: Uj8xT9p51cnqLhHfJGsJNUc7Rja1Iy7xwtjv9jBHipifH1Uod/88akXo1J5xE6i+87eCD/o5T9 OUNhDtgen3OA== X-IronPort-AV: E=Sophos;i="5.77,274,1596524400"; d="scan'208";a="484232315" Received: from fyigit-mobl1.ger.corp.intel.com (HELO [10.213.227.248]) ([10.213.227.248]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Sep 2020 07:38:47 -0700 To: Rahul Lakkireddy Cc: dev@dpdk.org, kaara.satwik@chelsio.com References: <8fb4bbbf-9bbd-b3e0-b96b-aae6647fe211@intel.com> <20200918131818.GA18801@chelsio.com> From: Ferruh Yigit Message-ID: Date: Fri, 18 Sep 2020 15:38:44 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.2.2 MIME-Version: 1.0 In-Reply-To: <20200918131818.GA18801@chelsio.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH 2/2] net/cxgbe: release port resources during port close X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 9/18/2020 2:18 PM, Rahul Lakkireddy wrote: > On Thursday, September 09/17/20, 2020 at 13:46:24 +0100, Ferruh Yigit wrote: >> On 9/1/2020 6:16 PM, Rahul Lakkireddy wrote: >>> Enable RTE_ETH_DEV_CLOSE_REMOVE during PCI probe for all ports >>> enumerated under the PF. Free up the underlying port Virtual >>> Identifier (VI) and associated resources during port close. >>> Once all the ports under the PF are closed, free up the PF-wide >>> shared resources. Invoke port close function of all ports under >>> the PF, in PCI remove too. >>> >>> Signed-off-by: Rahul Lakkireddy >> >> <...> >> >>> @@ -1204,11 +1222,13 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev) >>> static int eth_cxgbe_dev_uninit(struct rte_eth_dev *eth_dev) >>> { >>> - struct port_info *pi = eth_dev->data->dev_private; >>> - struct adapter *adap = pi->adapter; >>> + struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev); >>> + uint16_t port_id; >>> /* Free up other ports and all resources */ >>> - cxgbe_close(adap); >>> + RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device) >>> + rte_eth_dev_close(port_id); >> >> Is there a reason to call the ethdev wrapper API here? Why not calling >> 'cxgbe_dev_close()' directly? > > > Chelsio NICs enumerate all physical ports under a single Physical > Function 4 (PF4). When PCI remove is invoked without > rte_eth_dev_close() first, the rte_eth_dev_release_port() is only > invoked for the first port instantiated on PF4. So, to ensure > rte_eth_dev_release_port() is invoked for the remaining ports, > we're explicitly calling rte_eth_dev_close() for all ports here. > > Here's a testpmd example that leads to the problem. > > testpmd> port stop all > Stopping ports... > Checking link statuses... > Done > testpmd> device detach 0000:03:00.4 > Removing a device... > Port 0 is now closed > Port 1 is now closed > Device 0000:03:00.4 is detached > Now total ports is 1 > Done > testpmd> > > The Chelsio NIC has 2 ports enumerated under 0000:03:00.4. When PCI > remove is invoked, rte_eth_dev_release_port() is only invoked once > for the first port. You can see that the other port is still active, > as printed by "Now total ports is 1". > > The sequence of calls is as follows from driver's PCI remove: > > eth_cxgbe_pci_remove() > rte_eth_dev_pci_generic_remove() > rte_eth_dev_pci_release() > rte_eth_dev_release_port() <--- Only invoked once > > If I explicitly invoke rte_eth_dev_close() in PCI remove for all > ports, as done in this patch, rte_eth_dev_release_port() is invoked > for both the ports. > > testpmd> port stop all > Stopping ports... > Checking link statuses... > Done > testpmd> device detach 0000:03:00.4 > Removing a device... > Port 0 is now closed > Port 1 is now closed > Device 0000:03:00.4 is detached > Now total ports is 0 > Done > > Now, both ports are released properly as printed by "Now total ports > is 0". > Hi Rahul, Thanks for detailed explanation.