From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D132C43E01 for ; Fri, 5 Apr 2024 14:40:38 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8F1BA402D4; Fri, 5 Apr 2024 14:40:38 +0200 (CEST) Received: from EUR03-AM7-obe.outbound.protection.outlook.com (mail-am7eur03on2102.outbound.protection.outlook.com [40.107.105.102]) by mails.dpdk.org (Postfix) with ESMTP id DB0C3402CE for ; Fri, 5 Apr 2024 14:40:36 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UMuQFCdbQZlfDYwuDBu0jX2eHBOAm8yIei94iIdYY0VWxi6YrMS0wr7fd9LlF75RP6Jow0iwxPZlJAQfOrQayy6EKc1s1TDkhD3hJIpkRTnYPp1ULpCzlSD2AL44CK6GrZCR1otJEQo2myka+tnFR2zb6AyS1JJUn8eUqx/28lHQWA5k7FnSEr/dE+JZbKbbqDKFomclPnfo744B+IB0QM5G4gjBRKvNTwfA8oyIhBYM8nu7go6WvFrxb9SPaZZ1jUEP/ScjHWqFOlbp7YLHNgigIBoRzEdCF+YAEXeoaO6l1sMuEolRZ91WdaEgfoC9zPwbFFUcTByp/9YRCdlJ7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VvlnxILjAkvQDUfHczknntCvChP7Cztfv5fdglaEDkE=; b=Tiz01d9R2OTpaPmhbPFmXOfg5+sajTjCzf/gl21AddiY3vIg1auCaP9lIviS8E0a8U4eNT1K4IjDOX7/olM5N9gf731j8TjNHbsD29y17ZbhjZTSCJYJtrc3D/ya6u8xADb3WBHn65c2MMo4NGc2Pw6ewBJNaLcKFDJiD+C+t+i67jap5BRpT1OIIngtV2BJqQPlumhWoNfpkPym5COMA9zgPYLkbNrQA67APJXZ/yJ4H7+ZIVlPUHGcGzjHVp4tBBnDq7PhxFetdlVaIqKaYbDlNkfC1i+TXQb6f6LTrqHxlLjsO+jStQMI1QIIJE1xLDyNWaw5t6Sy9RSyN+C5vA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=voereir.com; dmarc=pass action=none header.from=voereir.com; dkim=pass header.d=voereir.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=voereir.onmicrosoft.com; s=selector2-voereir-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VvlnxILjAkvQDUfHczknntCvChP7Cztfv5fdglaEDkE=; b=W9sl0rCV6qYV69VJJ3sOUy/CyQE7gHLPQLVRRYCZVq4SEP6Yv0HCquLtR6UqNXgbqOfZdcOVEnokK8u0vg40hDruNENDi12cHlOXDTFKOS9zKc5qa49b+iwFDzURKlV0F8bWdx9Z+KULdpVroZOVEeQEyg/WSRFecX77FSO8Lrc= Received: from AS8PR02MB8390.eurprd02.prod.outlook.com (2603:10a6:20b:528::14) by PAVPR02MB8967.eurprd02.prod.outlook.com (2603:10a6:102:325::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Fri, 5 Apr 2024 12:40:34 +0000 Received: from AS8PR02MB8390.eurprd02.prod.outlook.com ([fe80::5e43:3701:2d27:dcab]) by AS8PR02MB8390.eurprd02.prod.outlook.com ([fe80::5e43:3701:2d27:dcab%6]) with mapi id 15.20.7409.042; Fri, 5 Apr 2024 12:40:34 +0000 From: Tanmay Pandey To: "users@dpdk.org" Subject: Performance Bottleneck at NIC with Openshift Thread-Topic: Performance Bottleneck at NIC with Openshift Thread-Index: AQHah03DlPAe8p1Dfk2rH49uJzq5Iw== Date: Fri, 5 Apr 2024 12:40:34 +0000 Message-ID: Accept-Language: en-IN, en-US Content-Language: en-IN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: AS8PR02MB8390:EE_|PAVPR02MB8967:EE_ x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: mBaN7aut0qIrCvZwH+jajkw4ZVY8+QmU4gBBcDQo34IAHk1oN0FoyU1HxTRTTFV12VHJKykPCTHCw8DYUBqOIFAuJWuXBE2PhniMANLEtIwYtEuJ5gLRL3h2SJhWUexWlaL10oXncoKlqPanIpWuRrxohyKDIoDOBDfBwyK2MCqHF0iedZ0wYlO8xWKymBA7jL+NLyhrziZe5X+1OeuVeva8NsARNfJXD0td1YKNQ/2MEF+mKIce0cqEJi8tIJn6ByX5mez80s8QFmJChErOBSPosjG2/08GbBeShUWVKmDnhBtpw1pzirpKE8OAOnXnqU3j2SE/2oYClQDkm5GWq7qH3vF3QnLJRPOge0cqAxKoxwGEHxZRtJ5N+3D7YA4Pxk8ZQkNrMoJTqOYYso9hNOJf8qYzubln7/Xa55TG3CybOdGcqSORWp8s0Uy2wUqRLGwsh4/TweoL+Cutj0pczfz8wpsY+hnLTrZtjAEBpXcXUJFo4p8XqWRQX04ilhJ08W5fS59k1703kGzBJW8HQ7LsauklSlRWGzza6OQup0aBCxnRIDH8ZzuTBLXJRebcimpfxoMwDtYhfQgIQioTiMLcz10LD5YS63U7s72P9CI= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AS8PR02MB8390.eurprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(1800799015)(376005)(366007); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?wVNytXk0/gDJO0I9gIPYLyB7zh+mtUqRAfjk5Gq17RpCvHot4gNB1SZD?= =?Windows-1252?Q?YVb/RlHjcjcPajL6gHVuNWpGfgtgMCNXyG0Siu8cPTDyAaIAnVZMyoUV?= =?Windows-1252?Q?CNN2yeqf/TtXGaseB+zxA/Gp7XJQYarW5WxWqYwZIUoGrxRudHyZMCj5?= =?Windows-1252?Q?EB3khD9mkbgC4oeZsyfA+9kC47UjikVZl7Oys/wI9bnLtV9yp8rhYlwy?= =?Windows-1252?Q?NsQ3XOCeLzQQ9CevDVLwPtIldw9ropPNByk3gdixGrp7/3Su+o2VwDIv?= =?Windows-1252?Q?SVjxZ6edzHO3IadfB/TUSfBjvjUYNSDCyfNsK7mx21hjlxp7g8TtF7wn?= =?Windows-1252?Q?oprxSPMyUfq6sXu6lUtg29zf0slF3ADh6wd54R0/6fnbg1v1T1xDNLQw?= =?Windows-1252?Q?4EnZ3Gr3ljsAm1kpG79h0vHjxmtWtwvyms1caNyZK9fzGX2f+/70nhMg?= =?Windows-1252?Q?se982Z91UatQ6A54rq/jJ+Mz0wx0SSOuPUfnclmUiaK9RphNEA+ZSgvg?= =?Windows-1252?Q?s/ZoX/28JS1r0WI1mQspnUZTbwgarU/2115o+ckWFKSO7z6mJp/or96T?= =?Windows-1252?Q?rgwJBvzAwMkbeRbsLQA5REhBpMCcEc/lQa7ODQsVPzLFLEYtqfKjuH9e?= =?Windows-1252?Q?Pv2uW7Fq+pvu1Ti879nam2Ar/gscE8aCbfHxLL4fbDffIQH/W7zOLldl?= =?Windows-1252?Q?lhUtGoQWLOTEFpJWZw8IdoVkJ5Qq8QwfZ0bhbOBvASpaCSS8s1R8oYu5?= =?Windows-1252?Q?kNNi4N+tqrhK7MGkdNrmwXiUPWLDp0+COkY0COryOCMom/D7A1P1zbDR?= =?Windows-1252?Q?qZWssXw8hSCOu8hU6y7k5dviDSd/PPTCMQxGHCfTAtiTk9cm4dKF7/kI?= =?Windows-1252?Q?EuX0fgri/eBBXqLIbjxydvtK+6cHMAZ/Am4JStMJPRSJzBkzD2KPwQ2B?= =?Windows-1252?Q?yviG4DWul+9evgEltAZ/ySkkUA9l7f7ln5z6sGkVSD9JqRjDOvDz50b9?= =?Windows-1252?Q?LGpJ3jpyNJziu3ZdFingAoB9zkgzh/CreomShbZhCByQbZce0DvQVALb?= =?Windows-1252?Q?Il3eAMUIXL9daJJZXX5EHKbZ/f/mMAHpzlhef77ALZYpqSbRwVMsTwCH?= =?Windows-1252?Q?GfIh3dgdEgqIZO+sRXlY7UuxC/p0UEluUfhUel9v0oBvEupnK3g1sb8R?= =?Windows-1252?Q?/IXxjPqQIhaFiHt5yEGrtmvtolCjqRsplBqQ2Z48T5eWCkRoylZY3Uvq?= =?Windows-1252?Q?pnAvtvisAYmOGbzPsvQIQ4yM3pIJI616Gg6E073r28pHmzST6beH3BMm?= =?Windows-1252?Q?3+Dee+1NQvdkc6TarOxGB9Q9JUk8jaQcmRVzEJabNpcrpjMGw8wui/aL?= =?Windows-1252?Q?l5ciOzpKfg50cmISb+LMA3E+1QLkE8kxq2ydS6c4pjD+lBMK6Q9LXOeB?= =?Windows-1252?Q?QsspOGLlmk0A9KBQN/oV/vyeygsKg0gu6p3H7ghk98rREPskqhy7VRXC?= =?Windows-1252?Q?Jqmf256f2cXtGwDl783IGDMAiCzuHWS7+UIMzFhUYYVzYYKj1Q+XeDOE?= =?Windows-1252?Q?pQaWpjnMyVmxMNPu3yAemvQRaK47q/B5a2ITBOghR0KB+0PZGe3tS1O2?= =?Windows-1252?Q?XjERxrL+B5fIO+X0SBtEy8p3+Yv7glDKlfibfC7zArVu1xa6Hv8ZXf1D?= =?Windows-1252?Q?SOAedA/yFFAYGomru6jNQjHNoOgSUrEyddGm1smqIJkNgN7TKei1MQ?= =?Windows-1252?Q?=3D=3D?= Content-Type: multipart/alternative; boundary="_000_AS8PR02MB8390C7A89C3F057D5C7FEAE1CB032AS8PR02MB8390eurp_" MIME-Version: 1.0 X-OriginatorOrg: voereir.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AS8PR02MB8390.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 48678a1a-23ea-47cd-df67-08dc556d9692 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Apr 2024 12:40:34.5686 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: c0332603-0945-429b-ba2e-ef89b4a710a3 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: N3AxnnmY+2vzI0a4VRaY6zpiICA6iDdXZ99s4SkhAk/unOrsIYtQAhcTGWIEZvFLluq31y3QXiGNZdQVvONXZw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAVPR02MB8967 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_AS8PR02MB8390C7A89C3F057D5C7FEAE1CB032AS8PR02MB8390eurp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi, I am using DPDK version 22.11 for performance evaluation running PRoX on an= Openshift Cluster where I have created two pods =96 I am sending traffic f= rom one and receiving on the other and I=92ve found that I=92m unable to ut= ilize more than 6GB of bandwidth in the server at the packet generation lev= el. I have tested with a 64-byte frame size and achieved a maximum of 6.99 = MPPS. I=92ve attempted to address this issue by adhering to the recommendations o= utlined in the DPDK 22.11 NVIDIA Mellanox NIC performance report available = at https://fast.dpdk.org/doc/perf/DPDK_22_11_NVIDIA_Mellanox_NIC_performanc= e_report.pdf . However, the problem persists. Additionally, I=92ve investigated packet loss at the NIC interface level an= d found no anomalies. The bottleneck appears to be related to packet genera= tion, but I=92m uncertain about the underlying cause. I am very new to DPDK so don=92t really know how to debug this issue. I bel= ieve there is something happening between the NIC layer and Openshift. Additionally, I used the same hardware running kubeadm where I was using th= e same DPDK and PRoX version with a similar setup and was able to achieve m= uch better performance(at least for the packet generation part =96 where my= current bottleneck occurs). Can someone point me in the right direction? I would be happy to provide any other required information Below are the SUT details: Nic Model: Ethernet controller: Mellanox Technologies MT2892 Family [Connec= tX-6 Dx] uname -r 5.14.0-284.54.1.rt14.339.el9_2.x86_64 ethtool -i enp216s0f0np0 driver: mlx5_core version: 5.14.0-284.54.1.rt14.339.el9_2. firmware-version: 22.35.2000 (MT_0000000359) expansion-rom-version: bus-info: 0000:d8:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes ## CPU Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 104 On-line CPU(s) list: 0-103 Vendor ID: GenuineIntel BIOS Vendor ID: Intel Model name: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz BIOS Model name: Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz Operating System: cat /etc/os-release NAME=3D"Red Hat Enterprise Linux CoreOS" ID=3D"rhcos" ID_LIKE=3D"rhel fedora" VERSION=3D"415.92.202402201450-0" VERSION_ID=3D"4.15" VARIANT=3D"CoreOS" VARIANT_ID=3Dcoreos PLATFORM_ID=3D"platform:el9" PRETTY_NAME=3D"Red Hat Enterprise Linux CoreOS 415.92.202402201450-0 (Plow)= " ANSI_COLOR=3D"0;31" CPE_NAME=3D"cpe:/o:redhat:enterprise_linux:9::coreos" HOME_URL=3D"https://www.redhat.com/" DOCUMENTATION_URL=3D"https://docs.openshift.com/container-platform/4.15/" BUG_REPORT_URL=3D"https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT=3D"OpenShift Container Platform" REDHAT_BUGZILLA_PRODUCT_VERSION=3D"4.15" REDHAT_SUPPORT_PRODUCT=3D"OpenShift Container Platform" REDHAT_SUPPORT_PRODUCT_VERSION=3D"4.15" OPENSHIFT_VERSION=3D"4.15" RHEL_VERSION=3D"9.2" OSTREE_VERSION=3D"415.92.202402201450-0" OCP Cluster oc version Client Version: 4.15.0-202402070507.p0.g48dcf59.assembly.stream-48dcf59 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 --_000_AS8PR02MB8390C7A89C3F057D5C7FEAE1CB032AS8PR02MB8390eurp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Hi,

 

I am using DPDK = version 22.11 for performance evaluation running PRoX on an Openshift Clust= er where I have created two pods =96 I am sending traffic from one and receiving on the other and I=92ve found th= at I=92m unable to utilize more than 6GB of bandwidth in the server at the = packet generation level. I have tested with a 64-byte frame size and achiev= ed a maximum of 6.99 MPPS.

I=92ve attempted to address thi= s issue by adhering to the recommendations outlined in the DPDK 22.11 NVIDI= A Mellanox NIC performance report available at https://fast.dpdk.org/doc/perf/DPDK_22_11_NV= IDIA_Mellanox_NIC_performance_report.pdf . However, the problem persist= s.

Additionally, I=92ve investigated packet loss at the NIC inte= rface level and found no anomalies. The bottleneck appears to be related to= packet generation, but I=92m uncertain about the underlying cause.

I am very new to DPDK so don=92t really know how to debug thi= s issue. I believe there is something happening between the NIC layer and O= penshift.

Additionally, I = used the same hardware running kubeadm where I was using the same DPDK and = PRoX version with a similar setup and was able to achieve much better performance(at least for the packet genera= tion part =96 where my current bottleneck occurs).

Can someone point me in the rig= ht direction?

I would be happy to provide any= other required information

Below are the SUT details:

Nic Model: Ethernet controller: Mellanox Technologies MT2892 = Family [ConnectX-6 Dx]

uname -r
5.14.0-284.54.1=
.rt14.339.el9_2.x86_64
 
ethtool -i enp216s0f0np0
driver: mlx5_core<=
/span>
version: 5.14.0-28=
4.54.1.rt14.339.el9_2.
firmware-version:<=
span style=3D"font-size:11.5pt;font-family:"var(--d-font-family--monos=
pace)",serif;color:#333333;letter-spacing:.1pt"> =
22.35.2000 (MT_0000000359)

expansion-rom-version:


bus-info: 0000:d8:00.0


supports-statistics: yes


supports-test: yes


supports-eeprom-access:<=
code> <=
span class=3D"hljs-literal">no


supports-register-dump:<=
code> <=
span class=3D"hljs-literal">no


supports-priv-flags: yes


 


## CPU


Architecture:    &n=
bsp;       x86_64=



  CPU op-m=
ode(s):        32-bit, 64-bit


  Address sizes:   &=
nbsp;     46 b=
itsphysical, =
48  virtual


  Byte Order:<=
span style=3D"font-size:11.5pt;font-family:"var(--d-font-family--monos=
pace)",serif;color:#333333;letter-spacing:.1pt">   &nbs=
p;        Little Endian


CPU(s):     =
             104


  On-line CP=
U(s) list:   0-103


Vendor ID:     =
;          


  BIOS Vendor ID:   =
     Intel=



  Model name:<=
span style=3D"font-size:11.5pt;font-family:"var(--d-font-family--monos=
pace)",serif;color:#333333;letter-spacing:.1pt">   &nbs=
p;        Intel(R)<=
/span> <=
/span>Xeon(R) Gold 62=
30RCPU @ 2.10GHz


    BIOS Model name: =
;    Intel(R) Xeo=
n(R) Gold 6230R=
 CPU @ 2.10GH=
z=




Operating System:


cat /et=
c/os-release


NAME=3D"Red Hat Enterprise Linux=
 CoreOS"


ID=3D"rhcos"<=
code>


ID_LIKE=3D"rhel fedora"=



VERSION=3D"415.92.202402201450-0=
"


VERSION_ID=3D"4.15"<=
/span>


VARIANT=3D"CoreOS"


VARIANT_ID=3Dcoreos


PLATFORM_ID=3D"platform:el9"=
;=



PRETTY_NAME=3D"Red Hat Enterpris=
e Linux CoreOS 415.92.202402201450-0 (Plow)"

ANSI_COLOR=3D"0;31"<=
/span>


CPE_NAME=3D"cpe:/o:redhat:enterp=
rise_linux:9::coreos"


HOME_URL=3D"https://www.redhat.c=
om/"


DOCUMENTATION_URL=3D"https://doc=
s.openshift.com/container-platform/4.15/"

BUG_REPORT_URL=3D"https://bugzil=
la.redhat.com/"


REDHAT_BUGZILLA_PRODUCT=3D"OpenS=
hift Container Platform"


REDHAT_BUGZILLA_PRODUCT_VERSION=3D&qu=
ot;4.15"


REDHAT_SUPPORT_PRODUCT=3D"OpenSh=
ift Container Platform"


REDHAT_SUPPORT_PRODUCT_VERSION=3D&quo=
t;4.15"


OPENSHIFT_VERSION=3D"4.15"<=
/span>


RHEL_VERSION=3D<=
span style=3D"font-size:11.5pt;font-family:"var(--d-font-family--monos=
pace)",serif;color:#333333;letter-spacing:.1pt">"9.2"=



OSTREE_VERSION=3D"415.92.2024022=
01450-0"



OCP Cluster


oc version


Client Version: 4.=
15.0-202402070507.p0.g48dcf59.assembly.stream-48dcf59<=
/span>


Kustomize Version:=
 v5.0.4-0.20230601165947-6ce0bf390ce3


 
--_000_AS8PR02MB8390C7A89C3F057D5C7FEAE1CB032AS8PR02MB8390eurp_--