From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E5DB8A0503; Fri, 20 May 2022 17:05:11 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D5282427EB; Fri, 20 May 2022 17:05:11 +0200 (CEST) Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam08on2040.outbound.protection.outlook.com [40.107.101.40]) by mails.dpdk.org (Postfix) with ESMTP id 8842D42684 for ; Fri, 20 May 2022 17:05:09 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iOlg4sO/GtpiB3k3qRMqzc1rSIFV0Cl48R/kWAcDvPQiqtMyLwfkGb5nR08phxWlfpVTB0H4/uDJ1BENvSL3XFNBvtKL1EOVA+LUK52rhZiGvVqWmWqH9uXyogHZe2Ynd074TWLsiAKyRh84rvyl4qf2bePVM7u7jlUaH9BCFBlDandi/tHETlM+XfDGn7cajnGIkV66ybKFoRJ4vpSfGH87sKFQ37GtDShDEyUAjVDLoXHiZ7JiogFPmiLXuZUA2y08l1NjMW5hqy0RpMbnQCIEoKVkQnR84MLpW7YpQ2eKXagtgraYNM+l0frrWw92xaVQ329Ew6pb5iPjlL3xxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zgolKmNvllouwp5NRrHzcbtPF6oYF10TLI7IKgYydSs=; b=iDrDQNyS7YOmYkxH4RZ03E4befIrS64FQ/NXRxmyyhowh53vK5rqdaJrxOSq+7V/uN3EfZjI4ZmBb1AOFcKJxWyzc5HWJon06eUgCvUlVzSxn1E0oVljZ95D+o8ZS1k7bdKVTLzCW65Kl/m1bVAmF1l/d1H/NWx6KA5nu9EaWcQ+XomEfFcIddTdbcajaQEPQUtQ7wHt0X6GgXNqAjGCSJ5zBXqxuGiqk6h/be6C+ALGlhdt/HT9eBBxhOtu7tKtlmxhGroQsRnryD0+xFr41XMSvQFOufqNwLXT5eEuaLDwdzYqfMkE2f36riUHCS7gVX1Gya4Uvz8P7clIeS81JQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.80.198) smtp.rcpttodomain=huawei.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xilinx.onmicrosoft.com; s=selector2-xilinx-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zgolKmNvllouwp5NRrHzcbtPF6oYF10TLI7IKgYydSs=; b=dAoSTtbt3FAAE4mPo9QDjJpEHJHoDVZxzt8v2Bd4ZqvweScNiKwqqNVWCuQVRNnIXRvkwUtMTtFruS/0JYSwS1lepDh/MTOssRht8rUdvaVQJQGJinUM20qG55dMQ1Xs6D2Lj49UxBaSVtAKxC9D/WunKGWER/sCqo8O20Jk/bU= Received: from BN6PR19CA0066.namprd19.prod.outlook.com (2603:10b6:404:e3::28) by DM8PR02MB8293.namprd02.prod.outlook.com (2603:10b6:8:f::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.15; Fri, 20 May 2022 15:05:07 +0000 Received: from BN1NAM02FT006.eop-nam02.prod.protection.outlook.com (2603:10b6:404:e3:cafe::fa) by BN6PR19CA0066.outlook.office365.com (2603:10b6:404:e3::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.17 via Frontend Transport; Fri, 20 May 2022 15:05:07 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.80.198) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.80.198 as permitted sender) Received: from xir-pvapexch01.xlnx.xilinx.com (149.199.80.198) by BN1NAM02FT006.mail.protection.outlook.com (10.13.2.125) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5273.14 via Frontend Transport; Fri, 20 May 2022 15:05:06 +0000 Received: from xir-pvapexch01.xlnx.xilinx.com (172.21.17.15) by xir-pvapexch01.xlnx.xilinx.com (172.21.17.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Fri, 20 May 2022 16:05:05 +0100 Received: from smtp.xilinx.com (172.21.105.198) by xir-pvapexch01.xlnx.xilinx.com (172.21.17.15) with Microsoft SMTP Server id 15.1.2176.14 via Frontend Transport; Fri, 20 May 2022 16:05:05 +0100 Envelope-to: fengchengwen@huawei.com, thomas@monjalon.net, dev@dpdk.org, xiaoyun.li@intel.com, aman.deep.singh@intel.com, yuying.zhang@intel.com, andrew.rybchenko@oktetlabs.ru Received: from [10.71.119.221] (port=64964) by smtp.xilinx.com with esmtp (Exim 4.90) (envelope-from ) id 1ns4Bc-0001xa-Vv; Fri, 20 May 2022 16:05:05 +0100 Message-ID: <80bdb6d3-8259-80b1-3733-ef19b6b4f840@amd.com> Date: Fri, 20 May 2022 16:05:04 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [PATCH] app/testpmd: remove invalid ports when other process detach Content-Language: en-US To: fengchengwen , Thomas Monjalon CC: , , , , Andrew Rybchenko References: <20220302023326.16509-1-fengchengwen@huawei.com> <5569707.V25eIC5XRa@thomas> <31263d7d-b880-ff29-70d9-2a8f75b0bb45@huawei.com> From: Ferruh Yigit In-Reply-To: <31263d7d-b880-ff29-70d9-2a8f75b0bb45@huawei.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a92a44f9-d179-41ed-efaa-08da3a722038 X-MS-TrafficTypeDiagnostic: DM8PR02MB8293:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 2 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: edDLgiJmnn6hNmns2H/bd3DBM9CAniAjHBn0dPyVfzlcgA27kBAZBs9amqcA7QzEQAeCiocyCzOSzrTF5VrZu/GkOmHXWKShrmhcTHjaYVl9k6PYmTUtguTJLUk8HR2UtIE2hom3Uv/HgNIZyMmMlptmItFK4PCXU7rn+017pTBmUvpdFQCMCkFIbgfF3JmAyr2eFT8wPuaVmn71Q81jiJWtncrGbt2XnVotJAkl4f286nAnQYXfA9AJaewOE+NMbhqz7thU67S1t/sIyV5glfT5TeWwL24F2x2J6hKZA55tIKEDJY1ntlM67Rb93JhBja5YeRxsVUf8yewJ9zkftdQsqMzYsK5o/dcp2lZr4a47BsqGrlD+Z2+sp3XlD8qIVmbfz8+MRpHZyGRnRyYxwgb3CE+4N7MOcKJnsRvl9wvhcoQSKhAG4SW1vIwaKkC+BE8V1XBRE+gmduq5KNvEG6tnfaTP1krcfXwwQcjZ6EU6ljKtdNodFmAo3HG4WBZFRDJS30as7g6HzlRWHuRNuwRjUVmkkPI7Qn3G6oFvTNUWR97aSKx8FAK6yoTCahHN5AGnAtVPQK1Sqg2JxhAR22f5fZDiaMLy3GEAja5n6mLpUTv0Jp1zX3TgsDdGQtYbukhXGqwZk5ZMZW1JEJesMwIMs3gCr23W6lksSy4/tSRokTZOBURE2hu81KpTuU50E0j/DQ2/4vexxWUwQAz3H2ETJfdxLVs1DAlKKXVnmXw= X-Forefront-Antispam-Report: CIP:149.199.80.198; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:xir-pvapexch01.xlnx.xilinx.com; PTR:unknown-80-198.xilinx.com; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(83380400001)(35950700001)(2616005)(356005)(2906002)(336012)(26005)(31696002)(53546011)(40460700003)(47076005)(86362001)(82310400005)(44832011)(5660300002)(9786002)(7636003)(31686004)(36756003)(8936002)(4326008)(110136005)(8676002)(508600001)(70206006)(316002)(70586007)(54906003)(50156003)(43740500002); DIR:OUT; SFP:1101; X-OriginatorOrg: xilinx.onmicrosoft.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 May 2022 15:05:06.5912 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a92a44f9-d179-41ed-efaa-08da3a722038 X-MS-Exchange-CrossTenant-Id: 657af505-d5df-48d0-8300-c31994686c5c X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=657af505-d5df-48d0-8300-c31994686c5c; Ip=[149.199.80.198]; Helo=[xir-pvapexch01.xlnx.xilinx.com] X-MS-Exchange-CrossTenant-AuthSource: BN1NAM02FT006.eop-nam02.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR02MB8293 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 3/2/2022 8:36 AM, fengchengwen wrote: > On 2022/3/2 16:26, Thomas Monjalon wrote: >> 02/03/2022 03:33, Chengwen Feng: >>> Start main and secondary process: >>> ./dpdk-testpmd -a BDF0 -a BDF1 --proc-type=auto -- -i --rxq=8 --txq=8 >>> --num-procs=2 --proc-id=0 >>> ./dpdk-testpmd -a BDF0 -a BDF1 --proc-type=auto -- -i --rxq=8 --txq=8 >>> --num-procs=2 --proc-id=1 >>> Execute following command in main process: >>> port stop 0 >>> port detach 0 >>> Execute following command in secondary process: >>> set fwd mac >>> start >>> The secondary process will display: >>> Invalid port_id=0 >>> telcore 19 called rx_pkt_burst for not ready port 0 >>> stpmd> 8: [/lib64/libc.so.6(+0xdf600) [0xffff9e1dc600]] >>> 7: [/lib64/libpthread.so.0(+0x7c48) [0xffff9e28ac48]] >>> 6: [/usr/app/testpmd(eal_thread_loop+0x2c4) [0xb23574]] >>> 5: [/usr/app/testpmd() [0x9c21d8]] >>> 4: [/usr/app/testpmd() [0x9c2108]] >>> 3: [/usr/app/testpmd() [0x9b6cf0]] >>> 2: [/usr/app/testpmd() [0xad8620]] >>> 1: [/usr/app/testpmd(rte_dump_stack+0x20) [0xb1a130]] >>> >>> The root cause it that the secondary process has not removed invalid >>> ports when it processes RTE_ETH_EVENT_DESTROY event. >> >> Why the ports are not removed? > It is referring to application (testpmd) level state, ethdev port is removed. Above mentioned problem is valid since testpmd secondary process support is added. When primary hot remove a device, it updates relevant application state too, but for secondary process ethdev removes device without secondary process updating its state. I agree that using ethdev events is appropriate for this case, since action originated from ethdev for secondary process, need a way to notify application about it. Another option can be checking and removing invalid ports in secondary before each forwarding start, but since we already have an event for destroy, using it looks good to me. > Testpmd register function eth_event_callback to deal with DESTROY event, > currently it only assign ports[port_id].port_status with RTE_PORT_CLOSED, it > doesn't update other global variables like nb_ports. > >> >>> This patch adds a delay remove invalid ports invoke when process the >>> RTE_ETH_EVENT_DESTROY event. >> >> Why do we need this delay? > > The remove_invalid_ports will scan rte_eth_devices[], when process the DESTROY > event, the rte_eth_devices[x] still valid, so here we should a delay logic. > There is a dependency problem in the DESTROY event. We need some ethdev information to be able to deliver the event, so ethdev can't be completely destroyed until DESTROY event is processed. Which means when application received DESTROY event, ethdev is not fully destroyed yet. Except from above, 'remove_invalid_ports()' is called twice for primary process (as mentioned in commit log), this timer can be helping these two calls not conflict, but I am not sure if we can rely on a time for this. Since the problem is mainly for secondary process, assuming control commands like detaching a device will be called by primary process, so @Feng what do you think about adding a secondary process check to in the 'remove_invalid_ports_callback()' call? So 'remove_invalid_ports()' is called only once for primary process. >> >> [...] >>> +static void >>> +remove_invalid_ports_callback(void *arg) >>> +{ >>> + RTE_SET_USED(arg); >>> + remove_invalid_ports(); >>> +} >> [...] >>> case RTE_ETH_EVENT_DESTROY: >>> ports[port_id].port_status = RTE_PORT_CLOSED; >>> printf("Port %u is closed\n", port_id); >>> + if (rte_eal_alarm_set(100000, remove_invalid_ports_callback, >>> + (void *)(intptr_t)port_id)) >>> + fprintf(stderr, >>> + "Could not set up deferred device released\n"); >>> break; >> >> >> >> >> . >> >