From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 68F93A054F; Mon, 15 Mar 2021 20:52:49 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E70E82427C3; Mon, 15 Mar 2021 20:52:48 +0100 (CET) Received: from NAM04-BN3-obe.outbound.protection.outlook.com (mail-eopbgr680070.outbound.protection.outlook.com [40.107.68.70]) by mails.dpdk.org (Postfix) with ESMTP id 93C0C4069F for ; Mon, 15 Mar 2021 20:52:47 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ze0y8a+SZzKxsSEMNst44cDOgcaK2c+iddDb91xqy6t0ORFwjd+B9TInAn+3dtpcJE/0hs4IutFx/XaA2Nrfc4aSznVuTj8VbCOmtbmFi2QLU/E+oFQ499S0E4dnn3HwE8FxiS9W/iidko5Ld5EDTX1Z1011VKtPBe3l1N6mG3JIyegGW2Em1kdjwyuXxeSGRBVp+UHIThcadF4GwXzNI3dYpBEeoOlXWUHXTKRWPh6Wso646AK4lAXEny6ozf153IF1Ase6ZpIlpefP+97nf7NZQVbeVdqHHHTgBU4OpH1CLLCNBUas55YllUEOGrHkGAjf+znrLPb/RS+I7aA68w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Tje5OUZPFAL7hq5BCDKwk1fNokj+qIPxZwTacJf3p3k=; b=CnOcSsfyAPWpC1GQ7MVkJFVRzMx9o7GjuKuTZoDEVXsH7erJx+0gTuQWOMX4vAOV9W9MuCWR7MDl8qJM44RBOnfO7x03buRXDnO7IRinv3hrtGn+D5Weuebm2v8H/eFJWYQ8JCCY+kOVaTC3pqp67GYPnctaOVOOWGYokAWd3r5PpcTYRy8nK1BbGFqLk3Ky0/En0WHV+liNjebNL7uTlFXJfHj1NovqUSa1VBvHWisYhG3IrP0c1LiCm5T+LY7tz+StianqK2O3nMmupN5CgXewLuq5G3GJp45/YmKqHWLCY2ROUAAWJKYoub5o3c6DILP4E7reY/h0K/R7XSvZog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Tje5OUZPFAL7hq5BCDKwk1fNokj+qIPxZwTacJf3p3k=; b=lqkfEknz9kQcood1n6uJcVw1GouNk+Qr+gImzT8Vav7/C+1mtil+x1WdhJjuPsKPYiI/QazAqbTQKpPQhd//WM3RbNgXB0s1R0AOr8bsWV1mlrJWgVb0LSZhnjmV/IFe9BmPGfDOcaAeYtlqoCjJXvwq/pIP+31ooAjLm52jwtEd3PaGFDP7sdV1VlfzWNyMON3ivnXgesz3xUsASlx9xf8C4ogixDYugee+czVymMwgla/18E4ykV7rkQi34P5V4mhJeCjj8zsKOKNjtx4dFEWV0nh7wG0g8V8bmDfH3e7GQP3G1/D4dLn88FahAQNBCXuoDalShVQJLvFYSuopDg== Received: from DM6PR12MB3753.namprd12.prod.outlook.com (2603:10b6:5:1c7::18) by DM6PR12MB3417.namprd12.prod.outlook.com (2603:10b6:5:118::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.32; Mon, 15 Mar 2021 19:52:46 +0000 Received: from DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::20f7:11fc:9d46:9258]) by DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::20f7:11fc:9d46:9258%6]) with mapi id 15.20.3933.032; Mon, 15 Mar 2021 19:52:46 +0000 From: Slava Ovsiienko To: Jan Viktorin CC: "dev@dpdk.org" , Asaf Penso , Shahaf Shuler , Matan Azrad Thread-Topic: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Thread-Index: AQHXGbCpbh43bNULVUOsJef9QEMNqKqFMkHggAAGBwCAADYQMIAABr4AgAAAg7A= Date: Mon, 15 Mar 2021 19:52:46 +0000 Message-ID: References: <1615822438-10082-1-git-send-email-iviktorin@fit.vutbr.cz> <1615822438-10082-2-git-send-email-iviktorin@fit.vutbr.cz> <20210315171039.16b3965e@tanguero.localdomain> <20210315204856.615637a5@tanguero.localdomain> In-Reply-To: <20210315204856.615637a5@tanguero.localdomain> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: cesnet.cz; dkim=none (message not signed) header.d=none;cesnet.cz; dmarc=none action=none header.from=nvidia.com; x-originating-ip: [95.164.10.10] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f71155ec-b905-4d80-fee0-08d8e7ebe7b6 x-ms-traffictypediagnostic: DM6PR12MB3417: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:478; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: QfKQYi3odyRIfIEbaWKremxxC+lc3jwSdbQh6tHlaBX7PwcRQvCH0MpJd3w7Ja4fw6Kxyh0peOT2wCEUXedz4veDnSFEn8bx8ZLgFrjZGOe3BoaceukM4qOkF5GqIWap2SH0PaS7KX1vgIgCQyBcKVErC3PUaECgmpC1YeWaIDpTSvfVYY8wms9Zh6eQQsppt+VGdA/WlrT9wnQd5FqYB0DB+rrOJdESFm4z502huGTzpiHIqUvv7P0EsfMDXAgX+dwylQKCZDQEAroi6lSztxTznOww80pWOsC00+VALdQQ2ju11BSdij1ZvIdkmIrJqMfa3kViN3sRazJEaElkC8Q3fGXkttrj2HXOErsQpFPJFkD9nO24CkkB+7ZlI5KBkuz7/bE6GH48n9bbfWF4ZmVe6zqv9s2BI9SkBcFs89YFgsUqFi78sWam+UI+wx6q2RNSpUsPoEtb6mmySmvHuUDH7anAHsCgLfaoYlzPguRq+c7C6JSuRV2xCnjqchMnj9ADtCOhSH/sYlA3p5wVf+eokIzbCveBspCVZogbQk6q+T/C8bdUTo4v14j7u2wnrysJHXvSqfeWKBNzjffGBgSEY73gwVzzb1AvbpHhd01bnDbS9ufu7j9u93oWF+HTmiB/6fwzbX1r8kJ7an1CmQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB3753.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(136003)(39860400002)(346002)(396003)(376002)(66476007)(66446008)(64756008)(66556008)(66946007)(76116006)(83380400001)(26005)(6916009)(71200400001)(53546011)(107886003)(4326008)(316002)(6506007)(186003)(86362001)(54906003)(5660300002)(7696005)(8936002)(52536014)(2906002)(55016002)(8676002)(966005)(33656002)(9686003)(478600001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?ImMgd8jC7vOxI/BUVB5NImLDlkAYPzreniNMQhNJfWFQaO1MwvsnHAmpVyDR?= =?us-ascii?Q?fz8i9jGHApiI/WAHA34T59jJyfz8k3E0nsrml6kXVRQePAlj/lK2WIdbot+L?= =?us-ascii?Q?Jcp1xqOSGV8//t2rdHDR+9dI8vcxdzfw7n6yQRuJZMuAUQfNoDAyBbINkiaF?= =?us-ascii?Q?6QPQx3vWyOVQzNoqBh/bcGUeziGtvU9qnDxTLrDxAgoQTtJ7DDMEoov8eQjY?= =?us-ascii?Q?/+G3avA24aoTTX31e0ujRhqrse4W9PdRjAcxQGYchRK1bglyf++Zgd5rRQgo?= =?us-ascii?Q?fvi/kCzG2I6LTQEQwuLFQ77HO8YIxRaItPOmQBMn6r4itCJqmNihnsuXTLOn?= =?us-ascii?Q?RpT0eZ6tXmVid+Jqe2wPYfpJ9A/zw5Qfpp6JIiyfylhou8SYySnTzQtG7Z7h?= =?us-ascii?Q?mADEp4Gna9kaE5rvFcmZy2I+4coQ5W43NzpG582krc8d8jDairYyIgojjziF?= =?us-ascii?Q?mXSREXn25euPtkrz5gJ2dc/jb8kKFDjTGZ9PhuvMk5ot0TaAdGVFop95Njij?= =?us-ascii?Q?Zr425UPUQTxmo/oIq9vF3GFi9zFU+hkupp/xirkOhN7eYv705+4i9AUSrxXP?= =?us-ascii?Q?eK9/g2baubqmQNuZs3Jght1PnN6RQkF7e358w7nY2Aih1CLzEntfPXIRUK++?= =?us-ascii?Q?LQxvymqFOs/UlC5SRNqyOooA620RvZgcQ3o0tTwFRmdeb5m16grIgbSi8SKF?= =?us-ascii?Q?FDrPxIv406pwbtaoOGZDE9VJEf9Y21P8xn2jgTugcNJTC2vBCvrw4Y5tfNJk?= =?us-ascii?Q?z3FIdfHArCi9LgCen84r8/7Y7vqo2lo0HNDPmD7xahv/MDSt8q6GzmkyiafJ?= =?us-ascii?Q?kbZJMEvjt1FqMSm5ioJVrr1dq60Vjc8/5gLeTpZlEqeWpOtggGGK51qkrWaj?= =?us-ascii?Q?5lJVNDWNutP1LlbrRrxOZ6lkv5QCoW7/f3pnzaKAQhb2NaXgKpuDuX0hDOls?= =?us-ascii?Q?P386PzJj+CffMzwYGWu//rjiL8+kmtUqaUwmq5AB4bx+cKP+k6FLDyB3BOEt?= =?us-ascii?Q?Tn4bQQ5sJoZZ18u5hCo+gs7rnqJYk/P3spFk+auDCs/cUICW4JWpwsVkzIfH?= =?us-ascii?Q?sa4S60FX/6rZAia240qkXJys3fubZOsUMSnJEc49jfFQpHdTSfAkTnbq2kst?= =?us-ascii?Q?Mx7hQ5wHlLXRX48koU3sx9u0dsUnlrxPK7zrSTz47D2Yin0iaU0nqtjsYl++?= =?us-ascii?Q?/idds18EkGfmYol1b6RLis3mbU2/0DE/60oMxRaMZyVFwcvrDAK9Sddx4zdX?= =?us-ascii?Q?pEg1ZvH0KOzgJ/DHlItBXxee4jqAQSJbBO00+PS5XSizOOiSiXAvEmCt99qd?= =?us-ascii?Q?T+w=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3753.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f71155ec-b905-4d80-fee0-08d8e7ebe7b6 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Mar 2021 19:52:46.2126 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: cb1EL4wZ4/NntAFVw9waEo65Y3HfqoTz8yp0ENPB3z1PKi78Yq867EwDN7YA68GISvWOHm5uddQET4/XDmKr2A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3417 Subject: Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, Jan > -----Original Message----- > From: Jan Viktorin > Sent: Monday, March 15, 2021 21:49 > To: Slava Ovsiienko > Cc: dev@dpdk.org; Asaf Penso ; Shahaf Shuler > ; Matan Azrad > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev > mode >=20 > On Mon, 15 Mar 2021 19:31:25 +0000 > Slava Ovsiienko wrote: >=20 > > Hi, Jan >=20 > Hello Salva, >=20 > I am sorry, but I didn't get the point of your response. For me, there wa= s 1 PF > 0000:05:00.1, SRIOV disabled. So, I followed the guide [1]. >=20 > 1. Enable SRIOV via mlxconfig. Done. >=20 > 2. Configure max number of VFs. Done. >=20 > 3. Reset firmware. Done. >=20 > In fact, I had to reboot the machine here so I am not convinced the reset= of > firmware is the thing. But I can image that there is some other use case = for it. > After reboot, I configured 2 VFs: 0000:05:00.4, 0000:05:00.5. >=20 > 4. Unbind the PF 0000:05:00.1. Done. No, it is the second physical port. You should unbind your VFs instead - 00= 00:05:00.4, 0000:05:00.5. Please, see: > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. With best regards, Slava >=20 > 5. Fail, no way how to configure switchdev because of previous step. > Thus, I strongly believe that the guide is misleading here. >=20 > When I bound the PF 0000:05:00.1 back, I could configure the switchdev > mode and everything was just OK and seemed to be working. So **after** > the bind I configured the switchdev mode properly. >=20 > [1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode >=20 > Jan >=20 > > > > 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV > > enabled, 0 VFs is configured, netdev name is ens1f0 (depends on distro > and kernel version). > > > > 2. Then, we configure 3 VFs with SR-IOV, we get : > > - 08:00.0 - PF, ens1f0 > > - 08:00.1 - VF0, ens1f1 > > - 08:00.2 - VF1, ens1f2 > > - 08:00.3 - VF2, ens1f3 > > > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. > > - 08:00.0 - PF, ens1f0 > > - 08:00.1 - VF0, - > > - 08:00.2 - VF1, - > > - 08:00.3 - VF2, - > > > > 4. Then, configure switchdev mode > > - 08:00.0 - PF, ens1f0 > > - 08:00.1 - VF0, - > > - 08:00.2 - VF1, - > > - 08:00.3 - VF2, - > > + we get 3 representor netdevs: > > ens1f0_0 (no dedicated PCI device) > > ens1f0_1 (no dedicated PCI device) > > ens1f0_2 (no dedicated PCI device) > > > > 5. Then, bind VFs back (assume we are no going to map them to VMs) > > - 08:00.0 - PF, ens1f0 > > - 08:00.1 - VF0, ens1f1 > > - 08:00.2 - VF1, ens1f2 > > - 08:00.3 - VF2, ens1f3 > > - ens1f0_0 (no dedicated PCI device) > > - ens1f0_1 (no dedicated PCI device) > > - ens1f0_2 (no dedicated PCI device) > > > > 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 > > VFs, 3 representors (no PCI device, pure netdev) > > > > With best regards, Slava > > > > > -----Original Message----- > > > From: Jan Viktorin > > > Sent: Monday, March 15, 2021 18:11 > > > To: Slava Ovsiienko > > > Cc: dev@dpdk.org; Asaf Penso ; Shahaf Shuler > > > ; Matan Azrad > > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable > > > switchdev mode > > > > > > Hello Salva, > > > > > > On Mon, 15 Mar 2021 15:53:51 +0000 > > > Slava Ovsiienko wrote: > > > > > > > Hi, Jan > > > > > > > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the > > > > netdevs > > > from the mlx5_core driver. > > > > Otherwise, kernel driver refuses to be configured with switchdev > > > > mode in > > > [5]. So, [4] can't be skipped. > > > > After setting swithdev mode, VFs can be bound back (if it is > > > > needed, and > > > these ones are not mapped to VMs): > > > > > > OK, but I believe that it is **not possible** to follow the rule [5]. > > > The guide explicitly says in [4] "can be rebind **after** the > > > switchdev mode". > > > Just, if you unbind the device, there is no way how to configure the > > > switchdev mode, this is the contradiction I mentioned in the commit. > > > You cannot configure switchdev mode because the interface is gone > > > and the path /sys/class/net//compat/devlink/mode no > longer exists. > > > > > > So, maybe, just the formulation is wrong. So, what is the **exact > > > right** way how to do it? I would change the commit accordingly. > > > Just, let's make it right. Would it work this way? > > > > > > # echo -n "" > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > # echo -n "" > > > > /sys/bus/pci/drivers/mlx5_core/bind > > > # echo switchdev > /sys/class/net//compat/devlink/mode > > > > > > It is good to mention that after the rebind, the can cha= nge. > > > > > > Regards, > > > Jan > > > > > > > > > > > echo -n "" > > > > > > /sys/bus/pci/drivers/mlx5_core/bind > > > > > > > > With best regards, > > > > Slava > > > > > > > > > -----Original Message----- > > > > > From: Jan Viktorin > > > > > Sent: Monday, March 15, 2021 17:34 > > > > > To: dev@dpdk.org > > > > > Cc: Jan Viktorin ; Asaf Penso > > > > > ; Shahaf Shuler ; Slava > > > > > Ovsiienko ; Matan Azrad > > > > > > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable > > > > > switchdev mode > > > > > > > > > > From: Jan Viktorin > > > > > > > > > > The step 4 is a contradiction. It advices to unbind the device > > > > > from the mlx5_core which removes the associated system network > > > > > interface > > > (e.g. > > > > > eth0). In the step 5, the same system network interface (e.g. > > > > > eth0) is required to exist. > > > > > > > > > > Signed-off-by: Jan Viktorin > > > > > --- > > > > > doc/guides/nics/mlx5.rst | 6 +----- > > > > > 1 file changed, 1 insertion(+), 5 deletions(-) > > > > > > > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst > > > > > index 0a2dc3dee..122d8e0fc 100644 > > > > > --- a/doc/guides/nics/mlx5.rst > > > > > +++ b/doc/guides/nics/mlx5.rst > > > > > @@ -1370,11 +1370,7 @@ the DPDK application. > > > > > > > > > > echo /sys/class/net/ > > > > device>/device/sriov_numvfs > > > > > > > > > > -4. Unbind the device (can be rebind after the switchdev mode):: > > > > > - > > > > > - echo -n "" > > > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > > > - > > > > > -5. Enable switchdev mode:: > > > > > +4. Enable switchdev mode:: > > > > > > > > > > echo switchdev > /sys/class/net/ > > > > device>/compat/devlink/mode > > > > > > > > > > -- > > > > > 2.30.1 > > > > > >