From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6CC4442D71; Tue, 27 Jun 2023 13:06:08 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id F17FB40F18; Tue, 27 Jun 2023 13:06:07 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2087.outbound.protection.outlook.com [40.107.243.87]) by mails.dpdk.org (Postfix) with ESMTP id E89C240EE1 for ; Tue, 27 Jun 2023 13:06:06 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TevW+NYiQvyo3YS86Vu1J9wQLPYy0FllWTi966j+lWx4ceK3WLicfQM8ovkqHdnRW/ClLOdDrB7NXoAKsB3jVFrjSRJB5ntXDI5WUeBSP+Jr972U77723sm9Jx5HSG9/ul5tkUsQKlW18wkyoQ4xrAPYFiQXdCzunDaI68fOQRNk0H5JBkWJEEz+MYCueEIE6roSCea2jL7iqRtDbqWO92WO4/rsTZp5OT8F8bJHa/7RIe4hizOxZbDiZgyDz27QOPE69P/2qlRkttalmvrITLxOsCSCa9UVBgRW+9UkP4IDkpcooiiiSTSgo7WYiH4dZlqQrLmi4vmSgWScmrnjvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=w1Zc04xVwJMouVO8fYj7ZfAPiadp3Pb1SzuoLfiQXqU=; b=n16aLfGTiM6UkqB5EBa/5cF4hKwWlZm0AK1UJqyOokb+ElKYzAcS/BCnZQyvOMd4mEVL5quo37AeLZLpxG2qE1tyJI73Cbni7yHHg81mSBOFQ+DX2U/VR+rfBCdADGqjzqctRCS9vU76eAv96ck09ySTqGPFkFRcsfDkCkh4sVlAb7eKRP2Bo1uUxYu35uwne2/91/PaFGS8/roxXj9m6c7Bh7Q7S0otx84ALO2l3wghEdDGia78BpwxU0q57M8KfwOU6b4B7sucYO/27cK1kiaa78zOc8gBhGSCsnMtbpVSKSRAh+CeP+l6tdat/SlioBueJKKN0BEW5gPzjyqzbg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=w1Zc04xVwJMouVO8fYj7ZfAPiadp3Pb1SzuoLfiQXqU=; b=JDKxK88wWa25cbkIdtQa/9QOov7MzZM2gZ/u3W1KRGlFrdEXcGDir170Vtz32EHxbHUQVASWP0uI2CCr05d0m0Naj1lvTuL3v/Nt6hWXEjJUTvhukVz+GF9qp8cgEtfPtGlq1XGKF3rkdahZ54ShUfu4pFP41ChakmHyKQOzx3k= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from MN2PR12MB4301.namprd12.prod.outlook.com (2603:10b6:208:1d4::22) by SJ2PR12MB9241.namprd12.prod.outlook.com (2603:10b6:a03:57b::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.24; Tue, 27 Jun 2023 11:06:04 +0000 Received: from MN2PR12MB4301.namprd12.prod.outlook.com ([fe80::3b9d:9ac6:f3ce:e106]) by MN2PR12MB4301.namprd12.prod.outlook.com ([fe80::3b9d:9ac6:f3ce:e106%3]) with mapi id 15.20.6544.012; Tue, 27 Jun 2023 11:06:03 +0000 Message-ID: <50a80b03-301a-b9ae-6a13-916562848f19@amd.com> Date: Tue, 27 Jun 2023 12:05:07 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v4] app/testpmd: fix primary process not polling all queues Content-Language: en-US To: Jie Hai , Ali Alnubani , Aman Singh , Yuying Zhang , Anatoly Burakov , Matan Azrad , Dmitry Kozlyuk Cc: "dev@dpdk.org" , "liudongdong3@huawei.com" , "shiyangx.he@intel.com" , Raslan Darawsheh , Thomas Monjalon References: <20230529022649.51425-1-haijie1@huawei.com> <20230609090340.3942-1-haijie1@huawei.com> From: Ferruh Yigit In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO4P123CA0465.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1aa::20) To MN2PR12MB4301.namprd12.prod.outlook.com (2603:10b6:208:1d4::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR12MB4301:EE_|SJ2PR12MB9241:EE_ X-MS-Office365-Filtering-Correlation-Id: 23fdd3d6-6b53-49d5-fe16-08db76fe7f95 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IEVOkLdi4m2kWOv6ezflDpYcpMpMQlJk9AxzeljJKS5Pwrfy5qJAn8U1Ol1uhJTZE/E8zZUcR3KG7Pz7kwNdvf8KAjuCQ++uhluv/V9qOEGT2AEZMnX440yn/r8Fs6hxZNctHO9uxljykVhAepqcY1EcEwez4iPH5Z9bFU8tV5imfGneAoY7C2tjZSjlw0IDdLWaP57lVGCumoBnmKIyAAHKEPHls+y4XmpgE4kcykXl9R53UcPA+S0CFq0XMG7tcZjFfWXEXs5o8OfFVJcroezgSw5CB+1+VG8rbwKcsQZ4C8siaZxFYdMpJnW24cS7HEgy6EZCHUn5nNZ1baTPtoOSjpP3jDrMrhAS0U4zcLXXlO3zR99F180pbrDsiFDEfLvnwV91KzvOI/3GEfIGsDgztIwAougMBGT8+vziugcrdaciNhR0QyYtDlUn2+nzLIe6v/+CSysBPYdHJOBHW9yac+PIaYaDex+uPhTXM6FMoCpnTHPyKxFWB/1aM/LxJpS1+qvI4NknoKfgvuFtG2g3lzGf1gDc8S0KHAQhmYjfNogzf/pm8DBePxpQ99RGPp0Isc35t+qI1fX1zBZZ4S8ErAesKQvFMnJcT0GfrpTYXhGiA9MBskyWVWLavuQR8kqvmJ7bxOgj8ZcDeCT1IQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR12MB4301.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(39860400002)(366004)(376002)(346002)(136003)(396003)(451199021)(2906002)(53546011)(6486002)(38100700002)(26005)(83380400001)(2616005)(186003)(6506007)(6512007)(41300700001)(110136005)(31696002)(86362001)(54906003)(4326008)(478600001)(316002)(66556008)(66946007)(66476007)(8936002)(8676002)(5660300002)(36756003)(31686004)(44832011)(7416002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QWRrV1ZvYWsvclRYdGRqVWQrUVlOTFFmYVV1Zm43emhEL3U0SmdUSzJmYzFs?= =?utf-8?B?Zms4NGtCSyt6MmJpcFN3dVBsVGpPQ0RnNnI3aGlYNGJ5Mjk2dVFxMk55alRz?= =?utf-8?B?OVdZaXoyTW4wMm1DTmxzcXJQREFxeXVBWkI4cE9BSVBwUHdMaXNXRTkwQ2Fv?= =?utf-8?B?NTFaQkFobXo3VkJLY3JvMThlLy9BSnUyUmJoMStCTkhBMHExWlBHazk4WjZr?= =?utf-8?B?NE1KWWt6cVgrZ3p5ZnZSZzJGYnh0RUNXVXh2a09JRDdaZXBINDVIZ1Irb0Rs?= =?utf-8?B?SUNlVVpxb01yUXJsUCtEK3o2aWZtbXJLVUpac3BCK01pdFRlczJVaVo5cGdl?= =?utf-8?B?WDF3VVYwbXhCOEc4RytPQVQ4Q1pYUmVmb2xWNktpTzZ2YVZISElrYzhuM3ph?= =?utf-8?B?em1ONkFUN2c5bVFjQlE1REM5YUJhd1BWRDAwZTZqKzI0VkdOaEZoaE9RZjA3?= =?utf-8?B?dEhlUVl4dUl0WTRES1VyM2lhbjIybFBJaXBBeTJwYXBYaW5YSHg4b2ZwUkZQ?= =?utf-8?B?RGk3U0JLeGRubHZDcGk3SXFCMWtkKzVIN3U5MURSWWM2bGtTZ1YwTkZqVXBL?= =?utf-8?B?dmpRbEE4KzVFWWwrNGhoVHY2YnIzcDVrOVRsemYzYVpNSDcrYlorRy9oR0NT?= =?utf-8?B?WWt3YkVmcmNOdEdmbTBjTzExdWxZS3hWK2lpMDkzd3VNdCtCY3RHbmFlNnN5?= =?utf-8?B?aWU3T0VkdE9iZHBhbEtMK2lLcTRMNFJaSXhEcnZPUEx1QVBmcmRXVERWNjF6?= =?utf-8?B?aHhhWllqY25QSjIxcENpSTZqbzdacm9oYUtFcVNRWHdGaHk1YitibFZvYzVa?= =?utf-8?B?ZXpqemkxK1BrSkx0NHRWYnRGM1NRMSs4emsvbCtMWmZyMXhtZzM5OHNkeThk?= =?utf-8?B?TWJhK3BQM1pnZjN2RG5VYkhCc2U0Wi9HR2RndVZ2OVRRQ1B4V2wwQkhkT1J6?= =?utf-8?B?UnFxbjR4RzNtU0hkT1hkVVpzY2hIYm9NVnEvSld0TnUxOG1XamhqTk9EZnk4?= =?utf-8?B?RFgxNUFTMi9FQmVEREJjYlFsNndiV0d5YnNoTnpYYm9yZjBzZWx4ZGJ1WDdy?= =?utf-8?B?dTJST2YyMU82ZWw2WFU4cC9kZWgyQmtJZ29ERXllZG5ITmV4R1RSOWpuM0t0?= =?utf-8?B?WjNsZVV1ZmhGQStRZDlBelZJYmlPY2tJSDJQVjRHd05FcjJJS1dXQTRmQS92?= =?utf-8?B?NUlSMlE5alZIUm1XdlljMjBNemRqVG90ZFlIQ3Y4Qk1lQmNkZDZVZzFSYlpJ?= =?utf-8?B?ejRkT1pkMkU4bDJSckRaOHBSazVTRG4rbVBtR1I4QnB1ZGg0NTBVZlJNc01z?= =?utf-8?B?R08xa1Z6bTBOL0RXMmpPbHk0L3hvS3lmbGhzcEFKVkkxWkVWTDJtZXRXZ2Vx?= =?utf-8?B?d3VWRW1JLzdLLzExNmVqMnNEMUdQQTd1SWF0WWFqdHFrcFBPMURnWWQvdXBJ?= =?utf-8?B?K0l6YjFsNnNVVjVTODhhNmREL3puYUNPYUluNlQyaFFxTTJ3cktLcXpmTkhF?= =?utf-8?B?NnVMYTdhK3BFL21pNWlvRjlZWVRzemlCQnRSeUJDWjVjbk1xYjFLNE9sWlRl?= =?utf-8?B?Mm5wR0UzRmxIVHRuUy9sTzFVeW4zbGxuY1VudEhlempDSk15Q0RmbDBqZTEw?= =?utf-8?B?YmlGRExZZzFOWkJzUVFMMDhVOWduUXhXNG5NSHRDenJJdkZxNTlFQVN1TzRa?= =?utf-8?B?cXhJVEkvS096cUZwZUZtOVR2cmRDTUowWmxYaVU2T2FQSVg2eEJIZ3lRWHlk?= =?utf-8?B?Z1c0enNpakNPNTBHajFGS2I1Z0R0VXd6N08ySDNSYWh1WE5nOURKbGhESEUz?= =?utf-8?B?azI4eElMZGJqcUZaVVNsZitYbU9DRTJwRkdZQ1BUWXlkU2F4ZXBPOERna2ky?= =?utf-8?B?QU9LUFo4MncxKzdtZEZSaTVoZmJtaEJqNE1jY0dNSG1hbGlwVWRrTzEvaWZ4?= =?utf-8?B?NkVlNzhhSi81MTNIOVhJbFhUNjNBanhKL3Y1OTFTNjB3cVpwaVNEUTcxcUJS?= =?utf-8?B?RlF5ck9ZYUtKSHZneXZWVk5tS3JJK2VqYkNCL21MUnBybm1aaFRQZVZseDNY?= =?utf-8?B?QWxlK3dNdHVVRU5uNXMxemJhNXBkbkl3UU52VmovSU9pOVFsRkh2WFczTmJ1?= =?utf-8?Q?ivrgOfN/MJC1L2HUKlXlNX8aT?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 23fdd3d6-6b53-49d5-fe16-08db76fe7f95 X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4301.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2023 11:06:03.9490 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VfAnJkRPKRuPq8KoBw2ZXN1uFsp9CZXmfj0dAt5MU8pSLo4kFP68HDmyWoxP/bMg X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB9241 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 6/26/2023 10:30 AM, Jie Hai wrote: > On 2023/6/23 0:40, Ali Alnubani wrote: >>> -----Original Message----- >>> From: Jie Hai >>> Sent: Friday, June 9, 2023 12:04 PM >>> To: Aman Singh ; Yuying Zhang >>> ; Anatoly Burakov ; >>> Matan Azrad ; Dmitry Kozlyuk >>> >>> Cc: dev@dpdk.org; liudongdong3@huawei.com; shiyangx.he@intel.com; >>> ferruh.yigit@amd.com >>> Subject: [PATCH v4] app/testpmd: fix primary process not polling all >>> queues >>> >>> Here's how the problem arises. >>> step1: Start the app. >>>      dpdk-testpmd -a 0000:35:00.0 -l 0-3 -- -i --rxq=10 --txq=10 >>> >>> step2: Perform the following steps and send traffic. As expected, >>> queue 7 does not send or receive packets, and other queues do. >>>      port 0 rxq 7 stop >>>      port 0 txq 7 stop >>>      set fwd mac >>>      start >>> >>> step3: Perform the following steps and send traffic. All queues >>> are expected to send and receive packets normally, but that's not >>> the case for queue 7. >>>      stop >>>      port stop all >>>      port start all >>>      start >>>      show port xstats all >>> >>> In fact, only the value of rx_q7_packets for queue 7 is not zero, >>> which means queue 7 is enabled for the driver but is not involved >>> in packet receiving and forwarding by software. If we check queue >>> state by command 'show rxq info 0 7' and 'show txq info 0 7', >>> we see queue 7 is started as other queues are. >>>      Rx queue state: started >>>      Tx queue state: started >>> The queue 7 is started but cannot forward. That's the problem. >>> >>> We know that each stream has a read-only "disabled" field that >>> control if this stream should be used to forward. This field >>> depends on testpmd local queue state, please see >>> commit 3c4426db54fc ("app/testpmd: do not poll stopped queues"). >>> DPDK framework maintains ethdev queue state that drivers reported, >>> which indicates the real state of queues. >>> >>> There are commands that update these two kind queue state such as >>> 'port X rxq|txq start|stop'. But these operations take effect only >>> in one stop-start round. In the following stop-start round, the >>> preceding operations do not take effect anymore. However, only >>> the ethdev queue state is updated, causing the testpmd and ethdev >>> state information to diverge and causing unexpected side effects >>> as above problem. >>> >>> There was a similar problem for the secondary process, please see >>> commit 5028f207a4fa ("app/testpmd: fix secondary process packet >>> forwarding"). >>> >>> This patch applies its workaround with some difference to the >>> primary process. Not all PMDs implement rte_eth_rx_queue_info_get and >>> rte_eth_tx_queue_info_get, however they may support deferred_start >>> with primary process. To not break their behavior, retain the original >>> testpmd local queue state for those PMDs. >>> >>> Fixes: 3c4426db54fc ("app/testpmd: do not poll stopped queues") >>> Cc: stable@dpdk.org >>> >>> Signed-off-by: Jie Hai >>> --- >> >> Hi Jie, >> >> I see the error below when starting a representor port after >> reattaching it with this patch, is it expected? >> >> $ sudo ./build /app/dpdk-testpmd -n 4  -a >> 0000:08:00.0,dv_esw_en=1,representor=vf0-1  -a auxiliary: -a 00:00.0 >> --iova-mode="va" -- -i >> [..] >> testpmd> port stop all >> testpmd> port close 0 >> testpmd> device detach 0000:08:00.0 >> testpmd> port attach 0000:08:00.0,dv_esw_en=1,representor=0-1 >> testpmd> port start 1 >> Configuring Port 1 (socket 0) >> Port 1: FA:9E:D8:5F:D7:D8 >> Invalid Rx queue_id=0 >> testpmd: Failed to get rx queue info >> Invalid Tx queue_id=0 >> testpmd: Failed to get tx queue info >> >> Regards, >> Ali > Hi Ali, > Thanks for your feedback. > > When update_queue_state is called, the status of all queues on all ports > are updated. > The number of queues is nb_rxq|nb_txq which is stored locally by testpmd > process. > All ports on the same process shares the same nb_rxq|nb_txq. > > After detached and attached, the number of queues of port 0 is 0. > And it changes only when the port is reconfigured by testpmd, > which is when port 0 is started. > > If we start port 1 first, update_queue_state will update nb_rxq|nb_txq > queues state of port 0, and that's invalid because there's zero queues. > > If this patch is not applied, the same problem occurs when the secondary > process detaches and attaches the port, and then starts the port in the > multi-process scenario. > > I will submit a patch to fix this problem. When port starts, update > queue state based on the number of queues reported by the driver. > Hi Ali, How big a blocker is this issue, should the fix be part of -rc2?