From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9D5D4A0032 for ; Fri, 18 Feb 2022 14:40:02 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9490F40150; Fri, 18 Feb 2022 14:40:02 +0100 (CET) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2086.outbound.protection.outlook.com [40.107.243.86]) by mails.dpdk.org (Postfix) with ESMTP id 4FD8F40141 for ; Fri, 18 Feb 2022 14:40:01 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kfEYL9wgGfVDjnb8ClQChv8iWkG+e/SaxVjXxqK4qzXp3FZSrwFhgp2ZU/ICksH00jb2U1mMUUvIp6+es/nAKSGZxu5wDkmp+0Tnq+mTZusv1oFhmDPzM/cWCDMPheRxIjsiDfa38wgUUlL545CmLL6Wj4gAF0zQEDZYk9OvbkRSYfjb5wtxRDLZskxV53Ri76GZ2xQOrXZPIVbv0ZbHfYzPEToKjVQsR0b6TZffwEZzSpTcKqkPaL+hQvZWon8fYE05LGXJj84rQFGeSKXkLyfRe/c3tUFe9Ycu4eOVDtDcbmuU3GeAH34r2sNuOyhWjJmHOJBItk6xG98ljOaGvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kgf2IPY6D0QrYsIngjLv0fl7BWA2f3S+426msEeA4u4=; b=QgFqYMxl9fWUlAeKhRxIlftlm8ARP7oSCvR4lrCHdeTTcEAb31b6OonuUto5O/GXX37+I//dtlapgx60cRKufaUNop5rCE4Bd2QEYwO/Cwv4Qc5opPkdTdsBT9h6y1AX6qzgDDTdaMk4MijQAEFmrZOzLqBRvORCLNQBu8+dEPshjyi+oa6Sx8AcM7ijZnGB5GquAYMsZpjg05q+0yDFZ0PrCjpfCnW7Wts4ZNdbbl0ni3ZvztIp4hQqK7rpT+SE8CabwW0MC1CE2iPIMmFdYHUB+ywegNQwlFTOyK/RKY0e6MoyNwPOXmr4UFk7JaEOBLrDyzdvyecPrJmRW+SLiQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kgf2IPY6D0QrYsIngjLv0fl7BWA2f3S+426msEeA4u4=; b=Nz5pk75FG5r1pZ4le6ufv14BLtvtI/zLuOU9rv2V4umb7+Zuzeszh7EBtspptqGWkGrtj+8Wdg6k92FzxS9vwAlgVR2mURdY8xGRJW4yRCOMzUzEoo1qWwATI0U1NAO18b7vyevW9aA/66Pd4x0XUBl+JpPWLz09YxjMlHUz1TsbOrlcS5ATV0PZvJPV/PL4QVYcfcVpK6+PUzZBapkxIpCgOCxBmTyOOx6IBsOh0ANP1G4fKDty4lpBf/YNbxEZQfPfvP6VSJeGhna/vXkMdoC2IvsyWgtOeVXxnQHlr9A8bJ4dO0DIqDLVEDZY5eTB6yqwuZQ4tRqB18F6wY/6Zw== Received: from BN6PR21CA0010.namprd21.prod.outlook.com (2603:10b6:404:8e::20) by BY5PR12MB4147.namprd12.prod.outlook.com (2603:10b6:a03:205::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4995.15; Fri, 18 Feb 2022 13:39:59 +0000 Received: from BN8NAM11FT068.eop-nam11.prod.protection.outlook.com (2603:10b6:404:8e:cafe::da) by BN6PR21CA0010.outlook.office365.com (2603:10b6:404:8e::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.13 via Frontend Transport; Fri, 18 Feb 2022 13:39:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.236) by BN8NAM11FT068.mail.protection.outlook.com (10.13.177.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4995.15 via Frontend Transport; Fri, 18 Feb 2022 13:39:58 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 18 Feb 2022 13:39:57 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Fri, 18 Feb 2022 05:39:57 -0800 Received: from nvidia.com (10.127.8.11) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.9 via Frontend Transport; Fri, 18 Feb 2022 05:39:56 -0800 From: Dmitry Kozlyuk To: =?UTF-8?q?=D0=94=D0=BC=D0=B8=D1=82=D1=80=D0=B8=D0=B9=20=D0=A1=D1=82=D0=B5=D0=BF=D0=B0=D0=BD=D0=BE=D0=B2?= CC: Subject: RE: Mellanox performance degradation with more than 12 lcores Date: Fri, 18 Feb 2022 15:39:52 +0200 Message-ID: <20220218133952.3084134-1-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 34866a18-da51-4981-2181-08d9f2e4280a X-MS-TrafficTypeDiagnostic: BY5PR12MB4147:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eUlAfOwDRhZOii/vG3iZXatyh0xk2oInBb/Gy20zeek3gUVVV1G1Yonnf5NYYabflu/muyib3MUgeewpkZNodkPftA/pisPvcw2Q3j3jFU/oaNfrOLbGeMctKbxuhqlDSwRg/6SUTCsb+I5/Q3ZWk7JYmD6xNrogTlhhI9YfijR7AkSc3pTKcJhVrvHd9e0nyAuj+LUAeHjM20PCq3hFLunkKGWAJ3KSnbkmdbxw7q1VR6wBG7juG1V/iSbZzvz/2hIkVR7exTVxvxVCPgDFjs712edhLZ9e3OTrshQZM5QXFzyAaGY4vV+oYO9ee7DtHIs6vqKHY2PO/TVK3Hu+K2GTU+AM2dUsFC+i+fmsMGUK+F1r8snhAGE8eGqN4U8Bg4+BbxZxxGjYhFT61Laojj7qrTHpWqZKUziCPNokzicR7ZwbvgrGHsGeYgf5OH7PoheZ4mlTIyFIY2bQRmvtRaj2YGZKYLD2RLouOd6ps2Z186HJG7BHZH+D9WVlXWcZYQKVwsG+Vc/JaBh8UJAse61kH3uW63e0Cub5lC1+CRunB2FcQBzRV4zaRHmxavdzchEtW7LjOGZmv5kas3hSXgOpe5H/hHut3NardJXVVg2gPpo67S0Kep9jtpGmBb6vdg2m63EiYb9sYmabz9KocATPjqeAjvNvI7JxKd4U/ANmep713jgdZeH/VvHSm3Fkp35rakcYFvS65+tsOnX6ow== X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(336012)(186003)(426003)(83380400001)(40460700003)(7696005)(70206006)(36756003)(2906002)(70586007)(8936002)(1076003)(6666004)(26005)(47076005)(2616005)(5660300002)(8676002)(86362001)(6916009)(36860700001)(316002)(356005)(82310400004)(81166007)(508600001)(6286002)(55016003)(4326008)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Feb 2022 13:39:58.6135 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 34866a18-da51-4981-2181-08d9f2e4280a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT068.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4147 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, > With more than 12 lcores overall receive performance reduces. > With 16-32 lcores I get 100-110 Mpps, It is more about the number of queues than the number of cores: 12 queues are the threshold when Multi-Packet Receive Queue (MPRQ) is automatically enabled in mlx5 PMD. Try increasing --rxd and check out mprq_en device argument. Please see mlx5 PMD user guide for details about MPRQ. You should be able to get full 148 Mpps with your HW. > and I get a significant performance fall with 33 lcores - 84Mpps. > With 63 cores I get even 35Mpps overall receive performance. > > Are there any limitations on the total number of receive queues (total > lcores) that can handle a single port on a given NIC? This is a hardware limitation. The limit on the number of queues you can create is very high (16M), but performance can perfectly scale only up to 32 queues at high packet rates (as opposed to bit rates). Using more queues can even degrade it, just as you observe. One way to overcome this (not specific to mlx5) is to use a ring buffer for incoming packets, from which any number of processing cores can take packets.