From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8BB6843EB2 for ; Fri, 19 Apr 2024 19:30:44 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 80F0040691; Fri, 19 Apr 2024 19:30:44 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2075.outbound.protection.outlook.com [40.107.243.75]) by mails.dpdk.org (Postfix) with ESMTP id 9E5DB40689 for ; Fri, 19 Apr 2024 19:30:43 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CVXSKan08mb6ENJeTEgbbGFZRHJgeOuloExeB4EnaNDUX2zAAOxxzjJSuMOK2vHI4ciTj1kph/u04auwZLC41B3a1IZ4gzGtZv1HbeQbCPofMK4PTrRReWWza1U63SN7nsHZy10U6jxP+FOLs20MKUzQ5pY4dvlmibnK3K061/ucw+tRExrOeaxwRhxqFbHyRQd9WRlw/eHJW2mHI1Gi0O9s1tGisPrsJMWJulOQUvpVgfUyP5fyYixcNZ44bhuT9UuomoX4OwL6mcTgBtMI73yk9j9pb4JAus5G5NhhWiCXI0o8ZjSAVPC9Ehjp/218v1zDU1muf8iM0TIKrw/9aQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=57zaxN38MpGxsNHTny8H6N7DNKhxruj0RpX9di2/WjI=; b=MifroSPgck2d5bGf6qV6F0WZS0eVP4WpnInhZDAl7HLUPmy0pesaMvU0jFI7JvN6DYX2NaarE6EtrCw20acxcJ27ptBUOoO7wuCMFmKqpk2nQcqtvYpaCvRppAPannu243xvpjHyCbRkgoIxpAGVyFRGqWIn7Ex6PTy52rjhvUWOuwyK2DgkaZzhtqsaWgiatr7szMZ1+gijdZm0fHz64+pS01/ljS26ID1w7bT7nNbygTsdIHLBtoavNdEARxOilsLkSi03mUC5ympMTb8H16DjpLB6RGX7ITDGvH4HdOAR9WnNge/T6nMK06Vn0+CpfaY9qv9i0hcTnlhtuJGzfg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=57zaxN38MpGxsNHTny8H6N7DNKhxruj0RpX9di2/WjI=; b=jWF01gN1Hdp5enTeird2bSLKn/yYP3vseujJIdUEOvC9t0gETjl/k7kLE2z/aBl+DCTnqFSPOEr8Dmp0JODCutKDMOohiqFEntfTO4f2Kxd5FtIJB8O64UzRy0lOabxV83YzjgYunBTFeGO+q4MdcxbcCa05tPF6JIVCTME+ZXs+y7LRkz32/7sLk+YPDXPtmL9uGGq5SIrQR14vYiWijjKarKSBGIYuaFGIUz4s/0rxTBMcZDjbpJMY3Vm/rtGXe/lggyNikBZg4zdGvcYKkAr/xgvp6AoZGu/5wqayccrluHeXucaVpKfUyYiqljsnzrqS3vmeXzKWfahUV0/GBw== Received: from PH0PR12MB8800.namprd12.prod.outlook.com (2603:10b6:510:26f::12) by SA1PR12MB6799.namprd12.prod.outlook.com (2603:10b6:806:25b::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7472.39; Fri, 19 Apr 2024 17:30:38 +0000 Received: from PH0PR12MB8800.namprd12.prod.outlook.com ([fe80::cbb1:eefb:c4e0:f710]) by PH0PR12MB8800.namprd12.prod.outlook.com ([fe80::cbb1:eefb:c4e0:f710%7]) with mapi id 15.20.7472.037; Fri, 19 Apr 2024 17:30:38 +0000 From: Dariusz Sosnowski To: Tao Li , "users@dpdk.org" CC: "tao.li06@sap.com" Subject: RE: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver) Thread-Topic: Packets cannot reach host's kernel in multiport e-switch mode (mlx5 driver) Thread-Index: AQHaix4jMREohSPxmk6Gj3lNXdj4v7Fv1ptA Date: Fri, 19 Apr 2024 17:30:38 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR12MB8800:EE_|SA1PR12MB6799:EE_ x-ms-office365-filtering-correlation-id: 274c69f2-7b7b-4427-ed40-08dc60966df7 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: =?us-ascii?Q?Q/htoFMokX9T/Vn/FOo76HBwgMjmIStkMQxCRfRIVNkmecOsJ0s6UTJ4Wf/i?= =?us-ascii?Q?eWcaeWLXS4ChsAHmCE5m9KKPRKXaBh1k/EZed75iUNHSJp4jDkYpEh3KBN/s?= =?us-ascii?Q?kduHXuz7B8fE7UDFZMvbckVXKvqkijXPWcKZLvBZZuEwWIHHc7H1KnTJzUZl?= =?us-ascii?Q?pbxNFWRW8BMD63hqdwhB7YxsnHZgPrk9zLMvHrz6VYopiq/ibmW0xq9I5A9E?= =?us-ascii?Q?KdICqCqxM9rdgfukkjvB8gayYYvMyvnb4zu+0wLUkEkGqs1T/FBCcISSXyc5?= =?us-ascii?Q?kNBbuHVZITsFX9BqIB87lItufbAB/FJawN4PEZSQdvuUu5L3ux/xwhwflRRL?= =?us-ascii?Q?MI62ODun+v3CWg2A3Wn4kvFeVYf8+2QYWyBaesuG5skvmK/cpZ42DnLBF7GD?= =?us-ascii?Q?C2dp7OerVVB2GMvtKEs6WJt3TD+UkvvrMxqHIMRHUmdJ0i+jjbF7RIRfNhNT?= =?us-ascii?Q?3Sa7HNPrDnY7YWlhyRT2aGjM4vkNACiFDx9JE4vp6eTsW9ERLKiuucTFGjj9?= =?us-ascii?Q?pTh8OkuBlgJi+FZC6+0nN67drsKDJV+hKARWCk17lBXtQ7MbQl8tHW4LJPV5?= =?us-ascii?Q?SUr+MVEAdCcL9agk54gUhLMv5Brflz57qidUMc+zl1CAYB2To4PH0oSInIiM?= =?us-ascii?Q?PSZp+pDK3Zlt757wnkA38aHN2ItDDOkI1w6uKYYaSYMhDx0Tq5fGvtlPfhmN?= =?us-ascii?Q?HHrCMeE8kQsRRtLYs11tSr6EnOmg1QmklAtQPo2I+4t8lwIiAwv0Rpv8mAKn?= =?us-ascii?Q?kjYiTJE8pN/V+m7THwAL8IZe+dishshOK28smqK2zKSOXOFuDj8F5p4F5iAn?= =?us-ascii?Q?nQdY6el1f4ujDF2QyZNrM28mh9IIbXG/fELcthTFjcunFXt0WjkGrQJJ+gGy?= =?us-ascii?Q?5EF/ohuWriairbhvf86b7vu5t/0iA219NcyWP7XRF5zB4c5PIDf1lJB9G/R/?= =?us-ascii?Q?S6xRj3fynV/QbhItzJy8R2i4twJB51HzD1Y4/aQNvruG/FOYuwiiDVYNTjV1?= =?us-ascii?Q?kVZZSjrnkBwjmhf6PdzOaRmv+izIEf6lA6MstnUSnlKjDhW2qC9gWnfRYb9D?= =?us-ascii?Q?mlFQ8CXoL/mJg7pKccAga3YW+HNwCIOG/OysprIPXOpw4yqys8PWWYd0m4BX?= =?us-ascii?Q?nOTPOGCIpnF+73lmLnsUxkpFvx/fFIPPewjz+oEzzcI+4K+Ai7gXav3WqiKr?= =?us-ascii?Q?LIeu4m2qcjxLDQl5Y4pvITGrpnAEx0/pUeCK51Oc/ksGUtIiYX5HUfCUpwdM?= =?us-ascii?Q?J83a2t3BUP9riEkqL2AclZ6ln6nQltP3nMJChTd5C196hJldVmQg/9csO/fD?= =?us-ascii?Q?0yWsM8ofas6lpnCSo6px4mnBH8qilDacP12YgYW+lgcumg=3D=3D?= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR12MB8800.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366007)(376005)(1800799015)(38070700009); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?hksPNJ5ZHM/EsJOJff84fY7vgaANJKeDG6RPyrZj0hcAc+ig50ZcO2XTJ2DK?= =?us-ascii?Q?nHxcRydABXQUcB7g8BEnsp8Fsu421TGRGnvtGdVJCOZR0qmaQNBIUHBrdX8Q?= =?us-ascii?Q?cmFTYRa2ADG3cAN5Kty6M/6vUEvuyzZPZN4XUh6RoqGmJwf3T2JZf6hCo7Zy?= =?us-ascii?Q?c2XRNypdnMGqftQgsslXL8r/vuSPFW9JkTyEKy9z07JPL2T5wV4PkIxBFH+O?= =?us-ascii?Q?KvMgaJuo2j6U8Wa6qcxIG7ElNDeCR+gbXYcAVH6ptizNEQY9iJ3HcHCxHtww?= =?us-ascii?Q?NWAI8vH8Cj0J1fJgBEFfFAsgEjAUc1IlkVfbuDkAGc2154FwpCbrMJNK7b8a?= =?us-ascii?Q?RtSIVZpx5DZQExpFmoGoOyCRuelj68XAVAAEJyezuG1keLxCcxttJ8jNRAUk?= =?us-ascii?Q?v18LPzEDPFUK0X7ldgboHs1s5GFJBPZv9r+7sAwVeKIhThFukPKL73kXAq78?= =?us-ascii?Q?qRQD4XJSZ6zIKOF2x4XFgstZqZkPWN9Pgo0VzZinMOp882tbit/uOjGifFbT?= =?us-ascii?Q?qt93+bt0r7V2I45Yt0lppLh7nv1OOfAZNR3g6DdmIM/cjQXqiAa+l+v4jWE0?= =?us-ascii?Q?CXBvpd/74K4DDkzraHPo/U51mhBBUhvVIrmndxlu4amB/1DmbWblpMKk1EXD?= =?us-ascii?Q?R8wvPeANCWyqbJBql8wuRwteVZ5gE/LZMJLe/lPV/kt86ahnxV35hXEzAJjI?= =?us-ascii?Q?XsSYKMPcpEoe+SzzKfDRdDKWs9MukyTfIestjwjjBhxH3g6NSOnsRBe+Z+VJ?= =?us-ascii?Q?qX5wdMEGToNPM4mhp+yeAaEmheuOEAU4Fl2qFtRKUMwBVELQCCPVBJXyceDi?= =?us-ascii?Q?gHu9GtPdgyvdqekS92ico+JJ8eZsqVbcCOpuzJTwOTgsbeYFVbabXp9Bn6sI?= =?us-ascii?Q?vq8RPaB2sZZPrCcJtTQtn05BgkzhpK733wxWQLjYf4fXOIEIXAVsprLO3Se3?= =?us-ascii?Q?kASqkp67Xj2wG9779c6gAWNyIFYzLMRr1s57iSpZtNLsV3QCMjfoBOIUhNi9?= =?us-ascii?Q?ZdXC0KwrT9JTpWDAB05lR2KKqjQU1A2RdEY6ntzWnAlkDwzaDCw6xbYCi4fM?= =?us-ascii?Q?eCtU0SKO2+ouRIyMovXkAxBG7G8i9AjslKmGP5AbJ79f3bNqFodNYY4G2gWE?= =?us-ascii?Q?/WRopP+mdJcAD4p5vZwEQ3zD9zvn2/oL0ji6EYXn+/CMtDlfNpUlEvg3Zp72?= =?us-ascii?Q?/6whrZdR1D/sliWmBw+cUg0dV83yaKQO+oIjcnfsPqL9pe9wWadIEBBAM4VH?= =?us-ascii?Q?qcZSLeZX3faczpXMMVgi5KN+TUXVQRnOwSne9NLBFmLVhQxPU36BT3YTltQ0?= =?us-ascii?Q?Bm44tcp/Be6Zd093FUPp5w04O1garSRa0fOG3DcEkWtL55gM7dJ7pjts3Hq2?= =?us-ascii?Q?L61BAgbO/YMSkRczKN6Vc+vuc1ivc1P1cjWh/oL/kv+fBklY/DeMx263lMZg?= =?us-ascii?Q?e07/3WF4j+gQI4GQreCmdM6S2ilNG8fXbKvQCpLHY3nX/ELbyV+J5cdA3EG3?= =?us-ascii?Q?ToW2zo3KVkwXljTBGesEW37ZYjO5YmauT1/S4wbWJKLmlXh8iHTtdmPXWjyK?= =?us-ascii?Q?HXhk8fpkMkMVzuV4OzGJeSCUx8kbFwhWkJ6ZfELf?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR12MB8800.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 274c69f2-7b7b-4427-ed40-08dc60966df7 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Apr 2024 17:30:38.5684 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Rds0LWo1NjBPADm9CGqcpQ9pmHrahhyJKbBFd1lONkNn0B3AgHv2v/8eAF7uNC5AdYZx2hAAs9p6qAudc/L93Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6799 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, I could not reproduce the issue locally with testpmd, with flow isolation e= nabled. I can see ICMP packets passing both ways to kernel interfaces of PF= 0 and PF1. Without flow isolation, it is expected that traffic coming to the host will= be hijacked by DPDK (depending on the MAC address, multicast config and pr= omiscuous mode). Could you please run testpmd with the following command line parameters and= execute the following commands? Testpmd command line: dpdk-testpmd -a 3b:00.0,dv_flow_en=3D2,representor=3Dpf0-1vf0 -- --flow-is= olate-all -i Testpmd commands: port stop all flow configure 0 queues_number 4 queues_size 64 flow configure 1 queues_number 4 queues_size 64 flow configure 2 queues_number 4 queues_size 64 flow configure 3 queues_number 4 queues_size 64 port start 0 port start 1 port start 2 port start 3 set verbose 1 set fwd rxonly start With this testpmd running, could you please test if both PF0 and PF1 kernel= interfaces are reachable and all packets pass? Best regards, Dariusz Sosnowski > From: Tao Li =20 > Sent: Wednesday, April 10, 2024 10:18 > To: users@dpdk.org > Cc: tao.li06@sap.com > Subject: Packets cannot reach host's kernel in multiport e-switch mode (m= lx5 driver) >=20 > External email: Use caution opening links or attachments=20 >=20 > Hi All, > =20 > I am currently experimenting with a feature newly supported by DPDK 23.11= , known as "https://doc.dpdk.org/guides/nics/mlx5.html#multiport-e-switch" = to improve communication reliability on the server side. During the trials,= I encountered an issue in which activating multiport e-switch mode on the = NIC disrupts the hypervisor's software running on the second PF interface (= PF1). More specifically, packets coming from the second PF (PF1) cannot be = delivered to hypervisor's kernel network stack, right after setting the mul= tiport e-switch mode for the NIC as guided in documentation. A snapshot of = the packet trace comparison on the second PF (PF1, ens2f1np1) before and af= ter setting the multiport e-switch mode is attached here. Packets marked w= ith the gray color/italic in the second trace are missing under the multipo= rt e-switch mode. > =20 > --------- > ConnectX-6 Dx with firmware version 22.39.1002 > Linux kernel version: 6.6.16 > DPDK: 23.11 > ---------- > =20 > ---------- > 14:37:24.835716 04:3f:72:e8:cf:cb > 33:33:00:00:00:01, ethertype IPv6 (0x= 86dd), length 78: fe80::63f:72ff:fee8:cfcb > ff02::1: ICMP6, router adverti= sement, length 24 > =20 > 14:37:28.527829 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x= 86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advert= isement, length 24 > =20 > 14:37:28.528359 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83f= b.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS va= l 1610632473 ecr 0,nop,wscale 7], length 0 // link-local addresses are used > =20 > 14:37:29.559918 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83f= b.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS va= l 1610633505 ecr 0,nop,wscale 7], length 0 > =20 > 14:37:30.583925 04:3f:72:e8:cf:cb > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 94: fe80::63f:72ff:fee8:cfcb.54096 > fe80::923c:b3ff:fe33:83f= b.179: Flags [S], seq 2779843599, win 33120, options [mss 1440,sackOK,TS va= l 1610634529 ecr 0,nop,wscale 7], length 0 > ---------- > =20 > ---- ------ > 16:09:40.375865 90:3c:b3:33:83:fb > 33:33:00:00:00:01, ethertype IPv6 (0x= 86dd), length 78: fe80::923c:b3ff:fe33:83fb > ff02::1: ICMP6, router advert= isement, length 24 > =20 > 16:09:40.376473 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 94: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83= fb.179: Flags [S], seq 3409227589, win 33120, options [mss 1440,sackOK,TS v= al 2302010436 ecr 0,nop,wscale 7], length 0 > =20 > 16:09:40.376692 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x= 86dd), length 94: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9= .36168: Flags [S.], seq 3495571820, ack 3409227590, win 63196, options [mss= 9040,sackOK,TS val 1054058675 ecr 2302010436,nop,wscale 9], length 0 > =20 > 16:09:40.376711 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 86: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:83= fb.179: Flags [.], ack 1, win 259, options [nop,nop,TS val 2302010436 ecr 1= 054058675], length 0 > =20 > 16:09:40.376865 fa:e4:cf:2d:11:b9 > 90:3c:b3:33:83:fb, ethertype IPv6 (0x= 86dd), length 193: fe80::f8e4:cfff:fe2d:11b9.36168 > fe80::923c:b3ff:fe33:8= 3fb.179: Flags [P.], seq 1:108, ack 1, win 259, options [nop,nop,TS val 230= 2010436 ecr 1054058675], length 107: BGP > =20 > 16:09:40.376986 90:3c:b3:33:83:fb > fa:e4:cf:2d:11:b9, ethertype IPv6 (0x= 86dd), length 86: fe80::923c:b3ff:fe33:83fb.179 > fe80::f8e4:cfff:fe2d:11b9= .36168: Flags [.], ack 108, win 124, options [nop,nop,TS val 1054058676 ecr= 2302010436], length 0 > ---- ------ > =20 > Attempts to ping from another directly connected host to this hypervisor = also resulted in incoming ICMP packets not being captured, which is reprodu= cible in another testing environment setup. In the end, I was able to resto= re communication on the second PF by using a vdev TAP device and performing= packet forwarding between the TAP device and PF1, as shown in our public e= xamplary https://github.com/byteocean/multiport-eswitch-example. > =20 > Enabling the isolation mode on PF1 by starting testpmd or programmably us= ing `rte_flow_isolate()` leads to no change from the behavior as described = above, but only affects whether packets can be captured and processed by th= e DPDK application. > ---- ------ > sudo ./dpdk-testpmd -a 3b:00.0,dv_flow_en=3D2,dv_esw_en=3D1,fdb_def_rule_= en=3D1,representor=3Dpf0-1vf0 -- -i --rxq=3D1 --txq=3D1 --flow-isolate-all > ---- ------ > =20 > Any experience sharing or comment on the above described issue is very ap= preciated. Thanks a lot in advance. > =20 > Best regards, > Tao Li