From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5622C43F55 for ; Tue, 30 Apr 2024 16:30:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C2E5A4025C; Tue, 30 Apr 2024 16:30:57 +0200 (CEST) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05olkn2061.outbound.protection.outlook.com [40.92.89.61]) by mails.dpdk.org (Postfix) with ESMTP id 5934A400EF for ; Tue, 30 Apr 2024 16:30:56 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xtf+tQJqnnzsfonOu1rzfu5QPKcsRPhDaWNJ/mjBX07962jyO2LzUqK1dHp4m9GxPObVfYyti33CrMI3qUq321Zp75VZlbPIUAxuUtti6e4YeDY1WLX+UQEAPVjPTaZXBA8X3OJdqVqF9F4yTvezr4uT82NhEWVoPjNGjXeZLBHxdceZ7/ku6o5wr7JGKFbqhtAaZ2f6ggOiFTt0HOy5EulAlECLl3I2wpuhkiIQ1bt2qPap5s2LtjpXgdx/Qi4PQpi+YqMbpuWdxg2ASE1zFsMdwt0HdUFB41o8fGXcNiusD97L74kaf4bljE8jJpysAgEk97SJharaItM/jgrWqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=o+XtEiAQDypoqpHjuD8H4AkgZegDBj+bYYQ/ryvRx7g=; b=RBRRl85bixObJ2EgHboLprScDq1gVpp4gvIWCwPjE1vr1ez/AjBFXghSomRIMPVFP9Y2wGOWLXqtN/qm3c2U74aKRimobYlqconMcmzxk4IezkIczFMnDbpYAXA2jRMskHbCexQ1qFx3CjrAFNecsmm261sHFgJWj6hLl/Y1Ebj/SEoWq+Z0mGZMqY4/JJpdr3wjqiZ9r7UcxingVBhEUpSeDJE3m5KUjIybFha34fL7ULyRtx1TNL05+QHBhmjKB8zBlt4bQy+3JXeSsoTXEFtfEO9M292hMCaQm2zUVGcebHnx33/VLfeFrbE2X0yZVwrDnaCDg3Uksb/c+EBLQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o+XtEiAQDypoqpHjuD8H4AkgZegDBj+bYYQ/ryvRx7g=; b=F2NBrnmEu2Zb+QJiMakHmWIQ4vKF6q9S+L1X/vpkizvURkKubMmS6ZTOXHyOZo3Z+1V9zdOu6aVtBmmqHIJN7CE2HSKySFel31pKeudWfT/+okSAr2LW7J89tB+gwmtmu14wqftBejfipJiE0PfzFeRoPq9+gF5NimLHK/gdmUlcJpx2meLcVqA7EZ9jOfeq78evaUzs2PbBYsgN6XsqKkAjcSdHw7/JvFsiLLmT9O4OjpxVSYQkJichycV99cU2RrUGURm082w1bQdLo1kii7kAA3Hm0O50zJyvnkQ1msRm7b8wHjVjpung6VNRVUjX7nnpe9zLrfrnuRT1SPIzCw== Received: from DB9PR09MB4906.eurprd09.prod.outlook.com (2603:10a6:10:26e::13) by PRAPR09MB5508.eurprd09.prod.outlook.com (2603:10a6:102:27a::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7519.36; Tue, 30 Apr 2024 14:30:54 +0000 Received: from DB9PR09MB4906.eurprd09.prod.outlook.com ([fe80::d54f:23db:dc57:6ea0]) by DB9PR09MB4906.eurprd09.prod.outlook.com ([fe80::d54f:23db:dc57:6ea0%6]) with mapi id 15.20.7519.031; Tue, 30 Apr 2024 14:30:54 +0000 From: Tao Li To: Dariusz Sosnowski , "users@dpdk.org" CC: "tao.li06@sap.com" Subject: Re: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver) Thread-Topic: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver) Thread-Index: AQHaix4jMREohSPxmk6Gj3lNXdj4v7Fv1ptAgBEc8oM= Date: Tue, 30 Apr 2024 14:30:54 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: x-tmn: [z0kfZECtJ+xVVjZXj1ywv3vJPIaAO8iK] x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DB9PR09MB4906:EE_|PRAPR09MB5508:EE_ x-ms-office365-filtering-correlation-id: d5c64df0-2922-4827-6396-08dc692224a1 x-microsoft-antispam: BCL:0; ARA:14566002|9400799015|461199019|102099023|440099019|3412199016|1602099003; x-microsoft-antispam-message-info: RbUsa6jbGhDMm+zB1iuZp8CccmjGfH2aMk6beStHoisk201nBRa0Rx0yftrMbpZafSBtrRx+tQFQME5aoksnj4w6ZiR+Z/iTXjJgKybQF8m51Mn4qS/ghX2F8fyhzWBH4FFcFCllGuWhBLMZ0pHIa92hyvpj4BffuaasgEK8HqMK5TWfmmkF25LBIwPA76wwxGIihMfRMIuS9og9w/1IY7CGyul5yhyVQ9KxiW5QwXlJZfyBno0RI9OQxVHX+RcYsebndBFO6d1F+WizCph67tpH1KF2pcD85Wd6DMG3TI9X2c/fjuVtcZPiMXPz0iUpsih7m8cB1cfm6CHVd+OF7/ibgpteSaew0DD8Kdauvm03HbU32aMErYxxsfGH/906x2zKHA2KfiwJeWWCZotzddCuuCPuNJ9aWphQCoZ/+lcUFwPLjX1IOkvhXSNA2U4V6Ym25+DtjENBuOWsLyxtHVQ2S4FMvQR9jBlGTMFWKzavm2iLxPM5Ucnv5Uf0/Q7IY/Qjd3d2cnXzRsytNaPVrWOFC/T7PytLxiXMzUsOdqUgR+63EGVHnpV7PPjLAJR/zEEGK4I+ZLxXecCckyiUE1UTThwTAYBrXEt1GkN42ux0ZLScvqGmVsl+qoenC/ECUPaTDfNjEoFRnXEx7j4wO1Vgo5IjlaOOsQ9vv+Yk00wgCKhmql9XFFN5Q10agPp1 x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?HarOoZXua3C2D+h5oGLwKMn73bDtQxO+UP/UwVuLsKjHr7JYo2lafMdlBvjD?= =?us-ascii?Q?r4CZjb1O786cEGjPWw7KPw7UcguzB0oe7x2wOMWBscdYbjVa2CoXKoRIQSXN?= =?us-ascii?Q?+AY9Rqv3b5OWkPPvE7GjHS2sfSt/5ANdYN/XjNapH22sO/uBc2O0SrK20Z04?= =?us-ascii?Q?uDK+OX1LyYjIHffr4QG+RxmfNbrF9J2czZBNBEo8yEZlB5UuNNx4Kk/6a6WF?= =?us-ascii?Q?O0m6s+rdmViybmxGWpswn2UyF+7+mtjtMBwo5dvApFqXm3QzvDQucmJphybm?= =?us-ascii?Q?TQIBlx4uZac+g+suyprq9RH9FjCaFTD83HZBdW3SKZED+96oiTgne4pX9d2q?= =?us-ascii?Q?e9nikndJfSsgmZC2SLztvur6SiTPL/u+WgkgaH8bpSmhW9Dzxs8VDRywSebC?= =?us-ascii?Q?t+YNsefxx5qLHgsETJvuOwX00JMCdUgxThLQe0D42p3uRBCxCSXwexXnULc2?= =?us-ascii?Q?n17vSkzCwfql1PscFRwFaU6lWK3wPbdMqxyNHI2iYImfipfUgQhxSRXk/2xz?= =?us-ascii?Q?zKWzA3h6LTsNm2mXG4KsMRJzzyJbef620oNj/sDayuDKu+GbPaXjcvVcQ/ny?= =?us-ascii?Q?4txAtGY7zE+08HDAHvUvXCmfBM8OSN78inBFBX9HwCDERDHSS/rI7fn26Im4?= =?us-ascii?Q?OhiVRB8v3jIFYkLKO0pp0qZw5D8TocGobGoBd/TntaEHt6tKkcGDkjAjmM6q?= =?us-ascii?Q?05K4VQZsCNI4tBAQuAJa8KMOnqs0I/rVu4GNNBMqy6TKtHRwUnMM9dN1lrwB?= =?us-ascii?Q?B4DwDYxIQpI+MbzV867StA7bOojlPybolPO0GNZ5p8alEbLEFhdqvg3nbLdU?= =?us-ascii?Q?SpXm5kyoDoX2Ki+n+hBv4LVEYiehC6e4cDl2BWMkWzcwUMe1elsDXf/Pp8Aa?= =?us-ascii?Q?vPkSUlhuDksE3tFCN5BUATWxgTeuB5TylzOmeaOSDXMuOpKPk0r+lOHdsSjv?= =?us-ascii?Q?j7kHnc2q7SlGfM929ZB1fNcWGRkJA6HLK5FIBl3Bl8fdAf4ccJMEs6kX08fz?= =?us-ascii?Q?7QiFffyk7HMOKJfOUom5IcSQBfZkJGdWD8qtOxbeAF8+NLL4PYHtDlYn0naC?= =?us-ascii?Q?jtidWIKQ/rO21sp4IAOGy7yjWh2d+14y/DersGFIJYMdYpi1NcCat3/BhkYV?= =?us-ascii?Q?0EiBftkRDPEAQcNXUZZUNQst6OhrJGvr4Tl6QXbGLexuXAYb15aDG2vJMCNS?= =?us-ascii?Q?yiEPliiPL44grsSqTLRyLC8k1xROs6CqgULwC03tmvFZ01nKm4AyQxRKW5YA?= =?us-ascii?Q?MqIqA5ob/n8acfgBH6CL?= Content-Type: multipart/alternative; boundary="_000_DB9PR09MB490689CAF2E46ABCD64B290BA31A2DB9PR09MB4906eurp_" MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-4755-11-msonline-outlook-64da6.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DB9PR09MB4906.eurprd09.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: d5c64df0-2922-4827-6396-08dc692224a1 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Apr 2024 14:30:54.4036 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PRAPR09MB5508 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --_000_DB9PR09MB490689CAF2E46ABCD64B290BA31A2DB9PR09MB4906eurp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Dariusz, It is very appreciated that you took a look at the issue and provided sugge= stions. This time, we again performed tests using two directly connected ma= chines and focused on ICMP (IPv4) packets in addition to ICMPv6 packets men= tioned in the original problem description. The issue remains the same. I w= ould like to highlight two points in our setup: 1. ICMP packets immediately cannot be captured on PF1 right after settin= g the nic into the multiport eswitch mode. And if I switch off the multipor= t eswitch mode by using following two commands, ICMP communication is resum= ed immediately, which shall prove that configs, such as firewall, on the sy= stem are correct. I would also assume it has little to do with a running DP= DK application, as communication is already broken before starting an appli= cation like testpmd. sudo devlink dev param set pci/0000:3b:00.0 name esw_multiport value false = cmode runtime sudo devlink dev param set pci/0000:3b:00.1 name esw_multiport value false = cmode runtime 1. In this setup, we do not use MLNX_OFED drivers but rely on the upstre= am Mellanox drivers from Linux kernel 6.5.0 (which is greater than the sugg= ested kernel version 6.3). Would that make a difference? Could you share so= me more detailed information regarding the environment setup on your side? = The firmware version we are using for Mellanox ConnectX-6 is 22.39.1002. Looking forward to your further reply. Thanks in advance. Best regards, Tao Li From: Dariusz Sosnowski Date: Friday, 19. April 2024 at 19:30 To: Tao Li , users@dpdk.org Cc: tao.li06@sap.com Subject: RE: Packets cannot reach host's kernel in multiport e-switch mode = (mlx5 driver) Hi, I could not reproduce the issue locally with testpmd, with flow isolation e= nabled. I can see ICMP packets passing both ways to kernel interfaces of PF= 0 and PF1. Without flow isolation, it is expected that traffic coming to the host will= be hijacked by DPDK (depending on the MAC address, multicast config and pr= omiscuous mode). Could you please run testpmd with the following command line parameters and= execute the following commands? Testpmd command line: dpdk-testpmd -a 3b:00.0,dv_flow_en=3D2,representor=3Dpf0-1vf0 -- --= flow-isolate-all -i Testpmd commands: port stop all flow configure 0 queues_number 4 queues_size 64 flow configure 1 queues_number 4 queues_size 64 flow configure 2 queues_number 4 queues_size 64 flow configure 3 queues_number 4 queues_size 64 port start 0 port start 1 port start 2 port start 3 set verbose 1 set fwd rxonly start With this testpmd running, could you please test if both PF0 and PF1 kernel= interfaces are reachable and all packets pass? Best regards, Dariusz Sosnowski > From: Tao Li > Sent: Wednesday, April 10, 2024 10:18 > To: users@dpdk.org > Cc: tao.li06@sap.com > Subject: Packets cannot reach host's kernel in multiport e-switch mode (m= lx5 driver) > > External email: Use caution opening links or attachments > > Hi All, > > I am currently experimenting with a feature newly supported by DPDK 23.11= , known as "https://doc.dpdk.org/guides/nics/mlx5.html#multiport-e-switch" = to improve communication reliability on the server side. During the trials,= I encountered an issue in which activating multiport e-switch mode on the = NIC disrupts the hypervisor's software running on the second PF interface (= PF1). More specifically, packets coming from the second PF (PF1) cannot be = delivered to hypervisor's kernel network stack, right after setting the mul= tiport e-switch mode for the NIC as guided in documentation. A snapshot of = the packet trace comparison on the second PF (PF1, ens2f1np1) before and af= ter setting the multiport e-switch mode is attached here. Packets marked w= ith the gray color/italic in the second trace are missing under the multipo= rt e-switch mode. > > --------- > ConnectX-6 Dx with firmware version 22.39.1002 > Linux kernel version: 6.6.16 > DPDK: 23.11 > ---------- > > ---------- > 14:37:24.835716 04:3f:72:e8:cf:cb > 33:33:00:00:00:01, ethertype IPv6 (0x= 86dd), length 78: fe80::63f:72ff:fee8:cfcb > ff02::1: ICMP6, router adverti= sement, length 24 > > 14:37:28.527829 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x= 86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advert= isement, length 24 > > 14:37:28.528359 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83f= b.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS va= l 1610632473 ecr 0,nop,wscale 7], length 0 // link-local addresses are used > > 14:37:29.559918 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83f= b.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS va= l 1610633505 ecr 0,nop,wscale 7], length 0 > > 14:37:30.583925 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83f= b.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS va= l 1610634529 ecr 0,nop,wscale 7], length 0 > ---------- > > ---- ------ > 16:09:40.375865 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x= 86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advert= isement, length 24 > > 16:09:40.376473 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 94: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83= fb.179: Flags [S], seq 3409227589, win 33120, options [mss 1440,sackOK,TS v= al 2302010436 ecr 0,nop,wscale 7], length 0 > > 16:09:40.376692 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x= 86dd), length 94: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9= .36168: Flags [S.], seq 3495571820, ack 3409227590, win 63196, options [mss= 9040,sackOK,TS val 1054058675 ecr 2302010436,nop,wscale 9], length 0 > > 16:09:40.376711 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 86: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83= fb.179: Flags [.], ack 1, win 259, options [nop,nop,TS val 2302010436 ecr 1= 054058675], length 0 > > 16:09:40.376865 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 193: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:8= 3fb.179: Flags [P.], seq 1:108, ack 1, win 259, options [nop,nop,TS val 230= 2010436 ecr 1054058675], length 107: BGP > > 16:09:40.376986 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x= 86dd), length 86: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9= .36168: Flags [.], ack 108, win 124, options [nop,nop,TS val 1054058676 ecr= 2302010436], length 0 > ---- ------ > > Attempts to ping from another directly connected host to this hypervisor = also resulted in incoming ICMP packets not being captured, which is reprodu= cible in another testing environment setup. In the end, I was able to resto= re communication on the second PF by using a vdev TAP device and performing= packet forwarding between the TAP device and PF1, as shown in our public e= xamplary https://github.com/byteocean/multiport-eswitch-example. > > Enabling the isolation mode on PF1 by starting testpmd or programmably us= ing `rte_flow_isolate()` leads to no change from the behavior as described = above, but only affects whether packets can be captured and processed by th= e DPDK application. > ---- ------ > sudo ./dpdk-testpmd -a 3b:00.0,dv_flow_en=3D2,dv_esw_en=3D1,fdb_def_rule_= en=3D1,representor=3Dpf0-1vf0 -- -i --rxq=3D1 --txq=3D1 --flow-isolate-all > ---- ------ > > Any experience sharing or comment on the above described issue is very ap= preciated. Thanks a lot in advance. > > Best regards, > Tao Li --_000_DB9PR09MB490689CAF2E46ABCD64B290BA31A2DB9PR09MB4906eurp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi Dariusz,

 

It is very apprecia= ted that you took a look at the issue and provided suggestions. This time, = we again performed tests using two directly connected machines and focused on ICMP (IPv4) packets in ad= dition to ICMPv6 packets mentioned in the original problem description. The= issue remains the same. I would like to highlight two points in our setup:=

 

  1. ICMP packets immediately cannot be c= aptured on PF1 right after setting the nic into the multiport eswitch mode.= And if I switch off the multiport eswitch mode by using following two commands, ICMP communication is resumed immedi= ately, which shall prove that configs, such as firewall, on the system are = correct. I would also assume it has little to do with a running DPDK applic= ation, as communication is already broken before starting an application like testpmd.

    sudo devlink dev param set pci/0000= :3b:00.0 name esw_multiport value false cmode runtime

sudo devlink= dev param set pci/0000:3b:00.1 name esw_multiport value false cmode runtime

  1. In this setup, we do not use MLNX_OF= ED drivers but rely on the upstream Mellanox drivers from Linux kernel 6.5.= 0 (which is greater than the suggested kernel version 6.3). Would that make a difference? Could you share some mo= re detailed information regarding the environment setup on your side? The f= irmware version we are using for Mellanox ConnectX-6 is 22.39.1002.

 

Looking forward to = your further reply. Thanks in advance.

 

Best regards,<= /o:p>

Tao Li

 

From: Dariusz Sosnowski &= lt;dsosnowski@nvidia.com>
Date: Friday, 19. April 2024 at 19:30
To: Tao Li <byteocean@hotmail.com>, users@dpdk.org <users@d= pdk.org>
Cc: tao.li06@sap.com <tao.li06@sap.com>
Subject: RE: Packets cannot reach host's kernel in multiport e-switc= h mode (mlx5 driver)

Hi,

I could not reproduce the issue locally with testpmd, with flow isolation e= nabled. I can see ICMP packets passing both ways to kernel interfaces of PF= 0 and PF1.
Without flow isolation, it is expected that traffic coming to the host will= be hijacked by DPDK (depending on the MAC address, multicast config and pr= omiscuous mode).

Could you please run testpmd with the following command line parameters and= execute the following commands?

Testpmd command line:
        dpdk-testpmd -a 3b:00.0,dv_flow_= en=3D2,representor=3Dpf0-1vf0 -- --flow-isolate-all -i

Testpmd commands:
        port stop all
        flow configure 0 queues_number 4= queues_size 64
        flow configure 1 queues_number 4= queues_size 64
        flow configure 2 queues_number 4= queues_size 64
        flow configure 3 queues_number 4= queues_size 64
        port start 0
        port start 1
        port start 2
        port start 3
        set verbose 1
        set fwd rxonly
        start

With this testpmd running, could you please test if both PF0 and PF1 kernel= interfaces are reachable and all packets pass?

Best regards,
Dariusz Sosnowski

> From: Tao Li <byteocean@hotmail.com>
> Sent: Wednesday, April 10, 2024 10:18
> To: users@dpdk.org
> Cc: tao.li06@sap.com
> Subject: Packets cannot reach host's kernel in multiport e-switch mode= (mlx5 driver)
>
> External email: Use caution opening links or attachments
>
> Hi All,
> 
> I am currently experimenting with a feature newly supported by DPDK 23= .11, known as "https://doc.dpdk= .org/guides/nics/mlx5.html#multiport-e-switch" to improve communication reliability on the server side. During the trials= , I encountered an issue in which activating multiport e-switch mode on the= NIC disrupts the hypervisor's software running on the second PF interface = (PF1). More specifically, packets coming from the second PF (PF1) cannot be delivered to hypervisor's kernel= network stack, right after setting the multiport e-switch mode for the NIC= as guided in documentation. A snapshot of the packet trace comparison on t= he second PF (PF1, ens2f1np1) before and after setting the multiport e-switch mode is attached here.  Pack= ets marked with the gray color/italic in the second trace are missing under= the multiport e-switch mode.
> 
> ----<test environment>-----
> ConnectX-6 Dx with firmware version 22.39.1002
> Linux kernel version: 6.6.16
> DPDK: 23.11
> ----</test environment>------
> 
> ----<packet trace after setting multiport e-switch mode>------ > 14:37:24.835716 04:3f:72:e8:cf:cb > 33:33:00:00:00:01, ethertype IP= v6 (0x86dd), length 78: fe80::63f:72ff:fee8:cfcb > ff02::1: ICMP6, route= r advertisement, length 24
> 
> 14:37:28.527829 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IP= v6 (0x86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, rout= er advertisement, length 24
> 
> 14:37:28.528359 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IP= v6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff= :fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sac= kOK,TS val 1610632473 ecr 0,nop,wscale 7], length 0 // link-local addresses are used
> 
> 14:37:29.559918 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IP= v6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff= :fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sac= kOK,TS val 1610633505 ecr 0,nop,wscale 7], length 0
> 
> 14:37:30.583925 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IP= v6 (0x86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff= :fe33:83fb.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sac= kOK,TS val 1610634529 ecr 0,nop,wscale 7], length 0
> ----</packet trace after setting multiport e-switch mode>------<= br> > 
> ----<packet trace before setting multiport e-switch mode> ------=
> 16:09:40.375865 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IP= v6 (0x86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, rout= er advertisement, length 24
> 
> 16:09:40.376473 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IP= v6 (0x86dd), length 94: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3f= f:fe33:83fb.179: Flags [S], seq 3409227589, win 33120, options [mss 1440,sa= ckOK,TS val 2302010436 ecr 0,nop,wscale 7], length 0
> 
> 16:09:40.376692 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IP= v6 (0x86dd), length 94: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:= fe2d:11b9.36168: Flags [S.], seq 3495571820, ack 3409227590, win 63196, opt= ions [mss 9040,sackOK,TS val 1054058675 ecr 2302010436,nop,wscale 9], length 0
> 
> 16:09:40.376711 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IP= v6 (0x86dd), length 86: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3f= f:fe33:83fb.179: Flags [.], ack 1, win 259, options [nop,nop,TS val 2302010= 436 ecr 1054058675], length 0
> 
> 16:09:40.376865 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IP= v6 (0x86dd), length 193: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3= ff:fe33:83fb.179: Flags [P.], seq 1:108, ack 1, win 259, options [nop,nop,T= S val 2302010436 ecr 1054058675], length 107: BGP
> 
> 16:09:40.376986 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IP= v6 (0x86dd), length 86: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:= fe2d:11b9.36168: Flags [.], ack 108, win 124, options [nop,nop,TS val 10540= 58676 ecr 2302010436], length 0
> ----</packet trace before setting multiport e-switch mode> -----= -
> 
> Attempts to ping from another directly connected host to this hypervis= or also resulted in incoming ICMP packets not being captured, which is repr= oducible in another testing environment setup. In the end, I was able to re= store communication on the second PF by using a vdev TAP device and performing packet forwarding between the TA= P device and PF1, as shown in our public examplary <= span style=3D"font-size:11.0pt">https://github.com/byteocean/multiport-eswi= tch-example.
> 
> Enabling the isolation mode on PF1 by starting testpmd or programmably= using `rte_flow_isolate()` leads to no change from the behavior as describ= ed above, but only affects whether packets can be captured and processed by= the DPDK application.
> ----<command to start testpmd> ------
> sudo ./dpdk-testpmd -a 3b:00.0,dv_flow_en=3D2,dv_esw_en=3D1,fdb_def_ru= le_en=3D1,representor=3Dpf0-1vf0 -- -i --rxq=3D1 --txq=3D1 --flow-isolate-a= ll
> ----</command to start testpmd> ------
> 
> Any experience sharing or comment on the above described issue is very= appreciated. Thanks a lot in advance.
> 
> Best regards,
> Tao Li

--_000_DB9PR09MB490689CAF2E46ABCD64B290BA31A2DB9PR09MB4906eurp_--