From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
by inbox.dpdk.org (Postfix) with ESMTP id 03499A04AF;
Fri, 18 Sep 2020 16:38:51 +0200 (CEST)
Received: from [92.243.14.124] (localhost [127.0.0.1])
by dpdk.org (Postfix) with ESMTP id AD7D41D9B1;
Fri, 18 Sep 2020 16:38:50 +0200 (CEST)
Received: from mga17.intel.com (mga17.intel.com [192.55.52.151])
by dpdk.org (Postfix) with ESMTP id 7EEA01D8DC
for <dev@dpdk.org>; Fri, 18 Sep 2020 16:38:49 +0200 (CEST)
IronPort-SDR: YlkA+1JEeAy3ccE6PMEoBY4g6HNEpo/XkAo4+WIkQ+j7FRYGWpXvTJzGjiECApXO33FOPFXa3A
Zw8L3gA5lLdw==
X-IronPort-AV: E=McAfee;i="6000,8403,9747"; a="139951203"
X-IronPort-AV: E=Sophos;i="5.77,274,1596524400"; d="scan'208";a="139951203"
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from orsmga005.jf.intel.com ([10.7.209.41])
by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
18 Sep 2020 07:38:48 -0700
IronPort-SDR: Uj8xT9p51cnqLhHfJGsJNUc7Rja1Iy7xwtjv9jBHipifH1Uod/88akXo1J5xE6i+87eCD/o5T9
OUNhDtgen3OA==
X-IronPort-AV: E=Sophos;i="5.77,274,1596524400"; d="scan'208";a="484232315"
Received: from fyigit-mobl1.ger.corp.intel.com (HELO [10.213.227.248])
([10.213.227.248])
by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
18 Sep 2020 07:38:47 -0700
To: Rahul Lakkireddy <rahul.lakkireddy@chelsio.com>
Cc: dev@dpdk.org, kaara.satwik@chelsio.com
References: <cover.1598979347.git.rahul.lakkireddy@chelsio.com>
<ceb95db6242808b5f5e568a0da60c91a3bc2d9a6.1598979347.git.rahul.lakkireddy@chelsio.com>
<8fb4bbbf-9bbd-b3e0-b96b-aae6647fe211@intel.com>
<20200918131818.GA18801@chelsio.com>
From: Ferruh Yigit <ferruh.yigit@intel.com>
Message-ID: <c8b156cf-40cf-1a04-b480-391a2456d9f5@intel.com>
Date: Fri, 18 Sep 2020 15:38:44 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101
Thunderbird/78.2.2
MIME-Version: 1.0
In-Reply-To: <20200918131818.GA18801@chelsio.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Subject: Re: [dpdk-dev] [PATCH 2/2] net/cxgbe: release port resources during
port close
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
On 9/18/2020 2:18 PM, Rahul Lakkireddy wrote:
> On Thursday, September 09/17/20, 2020 at 13:46:24 +0100, Ferruh Yigit wrote:
>> On 9/1/2020 6:16 PM, Rahul Lakkireddy wrote:
>>> Enable RTE_ETH_DEV_CLOSE_REMOVE during PCI probe for all ports
>>> enumerated under the PF. Free up the underlying port Virtual
>>> Identifier (VI) and associated resources during port close.
>>> Once all the ports under the PF are closed, free up the PF-wide
>>> shared resources. Invoke port close function of all ports under
>>> the PF, in PCI remove too.
>>>
>>> Signed-off-by: Rahul Lakkireddy <rahul.lakkireddy@chelsio.com>
>>
>> <...>
>>
>>> @@ -1204,11 +1222,13 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)
>>> static int eth_cxgbe_dev_uninit(struct rte_eth_dev *eth_dev)
>>> {
>>> - struct port_info *pi = eth_dev->data->dev_private;
>>> - struct adapter *adap = pi->adapter;
>>> + struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(eth_dev);
>>> + uint16_t port_id;
>>> /* Free up other ports and all resources */
>>> - cxgbe_close(adap);
>>> + RTE_ETH_FOREACH_DEV_OF(port_id, &pci_dev->device)
>>> + rte_eth_dev_close(port_id);
>>
>> Is there a reason to call the ethdev wrapper API here? Why not calling
>> 'cxgbe_dev_close()' directly?
>
>
> Chelsio NICs enumerate all physical ports under a single Physical
> Function 4 (PF4). When PCI remove is invoked without
> rte_eth_dev_close() first, the rte_eth_dev_release_port() is only
> invoked for the first port instantiated on PF4. So, to ensure
> rte_eth_dev_release_port() is invoked for the remaining ports,
> we're explicitly calling rte_eth_dev_close() for all ports here.
>
> Here's a testpmd example that leads to the problem.
>
> testpmd> port stop all
> Stopping ports...
> Checking link statuses...
> Done
> testpmd> device detach 0000:03:00.4
> Removing a device...
> Port 0 is now closed
> Port 1 is now closed
> Device 0000:03:00.4 is detached
> Now total ports is 1
> Done
> testpmd>
>
> The Chelsio NIC has 2 ports enumerated under 0000:03:00.4. When PCI
> remove is invoked, rte_eth_dev_release_port() is only invoked once
> for the first port. You can see that the other port is still active,
> as printed by "Now total ports is 1".
>
> The sequence of calls is as follows from driver's PCI remove:
>
> eth_cxgbe_pci_remove()
> rte_eth_dev_pci_generic_remove()
> rte_eth_dev_pci_release()
> rte_eth_dev_release_port() <--- Only invoked once
>
> If I explicitly invoke rte_eth_dev_close() in PCI remove for all
> ports, as done in this patch, rte_eth_dev_release_port() is invoked
> for both the ports.
>
> testpmd> port stop all
> Stopping ports...
> Checking link statuses...
> Done
> testpmd> device detach 0000:03:00.4
> Removing a device...
> Port 0 is now closed
> Port 1 is now closed
> Device 0000:03:00.4 is detached
> Now total ports is 0
> Done
>
> Now, both ports are released properly as printed by "Now total ports
> is 0".
>
Hi Rahul,
Thanks for detailed explanation.