From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8086E43F06 for ; Thu, 25 Apr 2024 18:38:17 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 369DD438DA; Thu, 25 Apr 2024 18:38:17 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2066.outbound.protection.outlook.com [40.107.223.66]) by mails.dpdk.org (Postfix) with ESMTP id A8D09438CF for ; Thu, 25 Apr 2024 18:38:15 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GgwPG4HP5vUi067Sy5rHDG9LDBHM+4Ri9OyyZWq1YbxK2tptP9gG/hnDlBOl6EtBFOGVAEI6QIqrt/5smA68GNVPjq02soFDAyPeRTYoaoYa7cbRElNQ0muwYdTLDV5nADWRUxM7FEyXOUTomoPN/i0UN7eolhKhu7iGJL7S3wvc5to9WYFn1HGzLBZwp5/W82Qrmpk5VX4V9G9JU6Ohl+AOPsLPC3jEHSCcmYYmtuQIN2BeCjCQQ2qCareK6yHUMmehmmP/p+DOwh5jtu5Sm7o9d1a6L7MmQTkBvOqIYvibYyS1U3CzRmmstKqEKPpEH7FImQRmkivpGEg0v2p7bg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QV8SkCVILKoVOyDGgMu+/Cma3Dcx13bdXK8P+Mrnx40=; b=ZFoFY4KIMwnJkAt4bzGIA4ohBaM7ciO8HyjjzmAdpoO66PD1eX+HMdOrZxijmcmHfVIKsWL+UgYKXcOouINaEJj8oVoPqbqkTZLDOchyEsRj3l53HzZ5P12/9U8zwXhpgL4mPLWwySiNaAXovvRosV2nD1qZLhWNE671YmtzSUJxe4g7dt66QrFqIWQjYFxnFufl5h9gSZw7Q0TlABVeaht0U7myRooNX8PY1ePHIqb8mewgDsJq+eSdRMF11q3+D/6/7YAXlzJzJ8lHtH2bi/ycHydGZ6HNumAzFT8wmrzmb1MGOvk38P1vok6B0QSjMH90P73SV9RhO4Dc+NXt4w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QV8SkCVILKoVOyDGgMu+/Cma3Dcx13bdXK8P+Mrnx40=; b=lcYrTW0Ypx+fZ+hcntKOMlsyPRS1sflCeOqOyxIC0J6WtPnGIv649Yfpk+zwzdB4eENoV70LO/9jPnxsch06AHf92usDCqb1nixP4P2mOGAzm1epI6yD/cjKKBoDECdaXcAGQxGxNFN57uZtS+S5+TOmL1wb/mvg0lwIIhkbDHoMcK9pyvLkgdGBLDzVTvM3gXvUH1VHspl70TANsuzyhWwNq/n/XAuyKMkxyWIYWHbkI9BKTXZZtz3q2V+6iKUKEKp30/gyoGZX4AzDHsO+jMxsULsQYvjhH1dlrcjS6qmzHnVWP0pwyY2rI48Ka9X9Q6xUfTpT0nYZPQCMS5Xatw== Received: from PH0PR12MB8800.namprd12.prod.outlook.com (2603:10b6:510:26f::12) by SA3PR12MB7921.namprd12.prod.outlook.com (2603:10b6:806:320::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.44; Thu, 25 Apr 2024 16:38:12 +0000 Received: from PH0PR12MB8800.namprd12.prod.outlook.com ([fe80::cbb1:eefb:c4e0:f710]) by PH0PR12MB8800.namprd12.prod.outlook.com ([fe80::cbb1:eefb:c4e0:f710%7]) with mapi id 15.20.7519.021; Thu, 25 Apr 2024 16:38:12 +0000 From: Dariusz Sosnowski To: Tanmay Pandey CC: "users@dpdk.org" Subject: RE: Performance Bottleneck at NIC with Openshift Thread-Topic: Performance Bottleneck at NIC with Openshift Thread-Index: AQHah03DlPAe8p1Dfk2rH49uJzq5I7F5IRmA Date: Thu, 25 Apr 2024 16:38:12 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR12MB8800:EE_|SA3PR12MB7921:EE_ x-ms-office365-filtering-correlation-id: e7686532-5bf1-4360-603b-08dc65461935 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: =?us-ascii?Q?Jtxk5pMfbeGciOs+kgwpLlt0YAByjdOb/lOavSiu3RvyVN30wr6EtKPYNMYG?= =?us-ascii?Q?xp9Uy42dERXt0krA1RRv9GJGnHmXtFkNeIGuNbvGH1+DudOyJDs7wdQg+4gZ?= =?us-ascii?Q?jBY9Ts9gZ5fVCmMlSOwuMW2bJjYQ3eXYiFuEAQZvcKHBXdIzu4/wvNSDmwPp?= =?us-ascii?Q?FpLdCJorLJFGHSit0zT9ZexpDFSwE8NH+3OnyCVmZkIVNNbdVMj8Vq38m832?= =?us-ascii?Q?l+99C9I6S1ITHJdAb4U7a33Xsc+14m4kY25fFyzaX0RhDOG7TK6vXIGtYA9e?= =?us-ascii?Q?Pamnr8AyTKM6xGtokyfyHsMCiM91SYT8SeTdILYYmNZmfu78rRATr5Q5wKQU?= =?us-ascii?Q?VK2y7x2R0l74eV5a1H/TKzEddzmOKAu2EoealrkrspUVtSmS1BSyXUA82JPW?= =?us-ascii?Q?uzGg6BmoQOHrzc3EtLEjulPuHpg3qa9UUxL7JtYIQrXA0VVwkz/vmpHkYIJh?= =?us-ascii?Q?ogJyUFFgG6T52rrKlnBzkVGOQC80VK4AV1ulFa/pPSbGwAdszsLPng2N8X/M?= =?us-ascii?Q?aaxBfg9NgyAbdJAO7Yy/LVsvKRq8s3CVMSnvYuJ5eoIO/3TIo5AKnttl5yFi?= =?us-ascii?Q?lWgPyU3tv0eA3M9g8elLru2GoBmLkC+p+sFrL2gL5KPYHXg7dTIfUX7VRFUM?= =?us-ascii?Q?2wKdGRH+wxHogiaYvPgowX81mn09caLhSYwsN4YO1UAScwQ0aQU+blHID6C5?= =?us-ascii?Q?IEC+qsQ1b2+mY4T0w2XIg/6pKQ62oWxZ9rKKP5BYEVfRJfvN/nJN1MfcbT9m?= =?us-ascii?Q?YJOwYQW2jegjz4RhKGyB9IBEZeF9tnl+W6EiogHPV05nUAvQ0+j+xiiI6hw3?= =?us-ascii?Q?gHaFNoL3X69R7O8/m9bouq2+Qu8RbrMUQaUQHXrCXMULKyjZoSfZSBsNQXWD?= =?us-ascii?Q?qLC8ggRI5wlCYDLnsvw/jaxKtnRW3C1Si7OXWoocxkRiOQ0JgcH/qJj1wwhG?= =?us-ascii?Q?MFSoJHEZ4CVlr4H6H4OcBEsrUrBEmUJofPno26RWNQwzMY17j/qF6ZjNTSJb?= =?us-ascii?Q?pBcgef8mNB3ypLVL5ddBcTPEfe4xsuuX1CJIHvIEjSNhh4vIJpDzvwm9giq2?= =?us-ascii?Q?EVwWJFKRYiGF+bd3wZWsHMx224p4CPwxQIWp9tBWvKDLwb5HdABllbJvugsU?= =?us-ascii?Q?v1VS2weW9UEnoNTR7Hl+TvmZiMiDv3cs8ZiO5MdGEnOvBn1mhqZ/QXB6wTpv?= =?us-ascii?Q?kMETxCh2nNQd5B1l6ADc2wVJq1C6TGmdkYoF9scNvrnhnrs8GUISBdA48bow?= =?us-ascii?Q?61G38rtj6Xpb35mO2FKDdQkRKPe9GHogEOioTKqBbOcRiTiVmZ7PyzhXy7R9?= =?us-ascii?Q?ArQyq6OeeVG4yARtYevFeS921TK0Awb6BLvl21+ZoVJvKQ=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR12MB8800.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(1800799015)(376005)(366007)(38070700009); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?cpCy09n5a5uSqW3XU0p2+9ol976TinpJCqdQ8UgybejAM1DFaMfhyIK+6z/L?= =?us-ascii?Q?cPMrVb7KsYCsX3D1Qp3I4J9HCDPdZmzuNwhV81WwXWHikbmTW7ybM4jyh256?= =?us-ascii?Q?taP8WOHoGpAxqx6EAIcYxSdOzMscZsiIEjty/bFeGJrlt+lCDJUQVHV9o3UI?= =?us-ascii?Q?aLvWaZenYGtKLpvSOg8ezhHK7wyW5JYEb9i4i9IhBsx1zbg3PGHk/Iwwlj0w?= =?us-ascii?Q?1BihONFXvo2nreFSPByH8sVaFJ/KiRHEvwB8MCDUm21yRnCIfz6B1gVbksKH?= =?us-ascii?Q?jssmrlI9iFipBw1H4eQYYZp+VH0RWPKmtAiJJTC10HY5tkD5HJOOzIV02aYF?= =?us-ascii?Q?r6vDlacq/2sX1LWKD5dtgpy4YTuYzBAH6PJOngHbVxLWbKFm9m05BUpsCspa?= =?us-ascii?Q?y3z78HmJSnZc6pqUdVLW0qyxgatpKOmHUpCK8auu3hmy/bFtx0+vCS5N6vSx?= =?us-ascii?Q?gNFeZh78S/AyL4bsB438g+zdboOjcrWfPGgsfCoHFCFAesRsPKTFGPqrCiCX?= =?us-ascii?Q?5F8nXtGk2K/cYglYQQTL0YUURlYpXrUMDHWVhCnhvZVEokNu3Siu+3gcFXYX?= =?us-ascii?Q?SUzpK7BqSmFTcuOdF7JTO7ePX63/4ZVmHNbnaPTOh4STL8WqnaJxTa9G33nO?= =?us-ascii?Q?zkZM6gnrX0Ad1aie3z3u5VsBO5ZSYQprLPcYdyce1ydsS+6wz/o6kVCzc9eA?= =?us-ascii?Q?JUwx5nAT1dl5QHy775MEhyPKStJ9lTABUHU6iR0y5yW45LTy6HLZkrPLY/kY?= =?us-ascii?Q?wR+H3GyOWlcwoTJ1jdF419NHAZ6aPiDnfyC98wau8nmsbk0+bdPLMRAfo5CX?= =?us-ascii?Q?ImTa/suEjcLfavXicxk3yHCZzxN40MPzeEOAbUaRZAWRcuLOKnzjGjYKs7eq?= =?us-ascii?Q?+M3YnmL7tKaAdhCTSEpqz7TZjQ67r+gM9+OQb/GYyVh77HVtXkNkl29giyaP?= =?us-ascii?Q?xUamR+mpnAMRChFghkTLUgQg96dD7fSMxNlGTUBtJAT9iqdOgOMY4JsfjsxI?= =?us-ascii?Q?lf3/K4qidtC8M2ltnvzyFjWrpoVKZMYy/WtsNN8JTmtz2zhQI73pg0/9Zyul?= =?us-ascii?Q?JOBi8pmzzBJuOQSIg8eOEqoi4bnAO+r8jXjH1E1H6W5+N9gDY+3tbtVIAfdM?= =?us-ascii?Q?1TiTFjENB6mA45W9Cm8ql1bJegZCklttpHhuvvSNhd8S8+WXs7FETRS/zzXL?= =?us-ascii?Q?jYZLfuE9tNsDK2XiUHf5uRJ92baR6hs35Rz7OPwEAHmc/zxnp0TQ6DM8m6X0?= =?us-ascii?Q?inELHi6iyu/OZUH4lOXSUeUzAey5jkz7U2SRBDI0s+aN2+FcGrPhWTmgRIlG?= =?us-ascii?Q?lCCQfD1PhSzjJU3MBV3NYpLH4uYl1m4Cl3fi+xGOYXX9JmwVqR5LuIgyZwCs?= =?us-ascii?Q?1byWMtyLwQHE8jLPZpen0AdfsZsRg/GnSOIRmNzVDNcZeq+mP0hL+HLpxIKo?= =?us-ascii?Q?seGFLiIISIsUD1aH+KlSzb2IqFI99qzSn4W0rXyMZghAEBncC7Y6HuKPj/r0?= =?us-ascii?Q?wwNo/iwkjVJ+PUQBBM38hu3PXf9DcNhfJTCHxKFwf6nRSz8CAQw8MEK2i/M/?= =?us-ascii?Q?b2EMYohU++HhYvecyGLdjbAauTO9hnD10pK4309s?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR12MB8800.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e7686532-5bf1-4360-603b-08dc65461935 X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Apr 2024 16:38:12.4691 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +S8H9yTPyR9ituIz/7ZVuONkpB7CEfKKTJmVglk3gPrJkKuMkkSMPWnjomVXWwXXWRXsa9WX9FqWhiEVkT7idw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7921 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, Since as you mentioned, the similar HW, the same DPDK and PRoX versions wer= e able to achieve much better performance, I'd guess that the problem might be related to how processes in pods are be= ing scheduled with OpenShift. Specifically, I would: - Check if both pods are not scheduled on the same cores. - Verify if cores on which these pods are running, are isolated. Anything interrupting threads responsible for generating traffic will hurt = the performance. Best regards, Dariusz Sosnowski > From: Tanmay Pandey =20 > Sent: Friday, April 5, 2024 14:41 > To: users@dpdk.org > Subject: Performance Bottleneck at NIC with Openshift >=20 > External email: Use caution opening links or attachments=20 >=20 > Hi, >=20 > I am using DPDK version 22.11 for performance evaluation running PRoX on = an Openshift Cluster where I have created two pods - I am sending traffic f= rom one and receiving on the other and I've found that I'm unable to utiliz= e more than 6GB of bandwidth in the server at the packet generation level. = I have tested with a 64-byte frame size and achieved a maximum of 6.99 MPPS= . > I've attempted to address this issue by adhering to the recommendations o= utlined in the DPDK 22.11 NVIDIA Mellanox NIC performance report available = at https://fast.dpdk.org/doc/perf/DPDK_22_11_NVIDIA_Mellanox_NIC_performanc= e_report.pdf . However, the problem persists. > Additionally, I've investigated packet loss at the NIC interface level an= d found no anomalies. The bottleneck appears to be related to packet genera= tion, but I'm uncertain about the underlying cause. > I am very new to DPDK so don't really know how to debug this issue. I bel= ieve there is something happening between the NIC layer and Openshift.=20 > Additionally, I used the same hardware running kubeadm where I was using = the same DPDK and PRoX version with a similar setup and was able to achieve= much better performance(at least for the packet generation part - where my= current bottleneck occurs). > Can someone point me in the right direction?=20 > I would be happy to provide any other required information > Below are the SUT details: > Nic Model: Ethernet controller: Mellanox Technologies MT2892 Family [Conn= ectX-6 Dx] > uname -r > 5.14.0-284.54.1.rt14.339.el9_2.x86_64 >=20 > ethtool -i enp216s0f0np0 > driver: mlx5_core > version: 5.14.0-284.54.1.rt14.339.el9_2. > firmware-version: 22.35.2000 (MT_0000000359) > expansion-rom-version: > bus-info: 0000:d8:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: no > supports-priv-flags: yes >=20 > ## CPU > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Address sizes: 46 bits physical, 48 bits virtual > Byte Order: Little Endian > CPU(s): 104 > On-line CPU(s) list: 0-103 > Vendor ID: GenuineIntel > BIOS Vendor ID: Intel > Model name: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz > BIOS Model name: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz > Operating System: > cat /etc/os-release > NAME=3D"Red Hat Enterprise Linux CoreOS" > ID=3D"rhcos" > ID_LIKE=3D"rhel fedora" > VERSION=3D"415.92.202402201450-0" > VERSION_ID=3D"4.15" > VARIANT=3D"CoreOS" > VARIANT_ID=3Dcoreos > PLATFORM_ID=3D"platform:el9" > PRETTY_NAME=3D"Red Hat Enterprise Linux CoreOS 415.92.202402201450-0 (Plo= w)" > ANSI_COLOR=3D"0;31" > CPE_NAME=3D"cpe:/o:redhat:enterprise_linux:9::coreos" > HOME_URL=3D"https://www.redhat.com/" > DOCUMENTATION_URL=3D"https://docs.openshift.com/container-platform/4.15/" > BUG_REPORT_URL=3D"https://bugzilla.redhat.com/" > REDHAT_BUGZILLA_PRODUCT=3D"OpenShift Container Platform" > REDHAT_BUGZILLA_PRODUCT_VERSION=3D"4.15" > REDHAT_SUPPORT_PRODUCT=3D"OpenShift Container Platform" > REDHAT_SUPPORT_PRODUCT_VERSION=3D"4.15" > OPENSHIFT_VERSION=3D"4.15" > RHEL_VERSION=3D"9.2" > OSTREE_VERSION=3D"415.92.202402201450-0" > OCP Cluster > oc version > Client Version: 4.15.0-202402070507.p0.g48dcf59.assembly.stream-48dcf59 > Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3