From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E334EA054F; Mon, 15 Mar 2021 21:30:50 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A9F792427D3; Mon, 15 Mar 2021 21:30:50 +0100 (CET) Received: from office2.cesnet.cz (office2.cesnet.cz [195.113.144.244]) by mails.dpdk.org (Postfix) with ESMTP id 1EF8E4069F for ; Mon, 15 Mar 2021 21:30:49 +0100 (CET) Received: from tanguero.localdomain (unknown [95.82.133.55]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by office2.cesnet.cz (Postfix) with ESMTPSA id CE1DD400052; Mon, 15 Mar 2021 21:30:48 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cesnet.cz; s=office2-2020; t=1615840248; bh=tGN1RQqmuGWCUcJgvAAOn0Fgfr3JCLOlQuD8UpQkGz8=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=CRZloVGLr3Tvg+GFa0E2+vF0cwbg6swoSV2WYxNmyfxLfgl/TKV0grWk/O1diO1ww FhwAL3WmM/4bfxijCBvCweJBHDHN5C1VsYUI29U8IvFH3lzeWaJRvWy0XBV+hJXjUv QrdwzgZT/GMk3gbDdfGEMgmzK59hZ95LC+JY3qSzD/y9xOuZeFjP8yJ6g1T3hLoUco x0nllJ8/4WDZnHrN9vpH/M5I1frCHEa2Z8J4rHNOWr+Amx10byCHmIiKPwe1gX1Hk9 m1InFf/vQynatNyq30ujl6YVllSh5Tqjo9zbdzrHnAJmtS+CFI7xcqbVImoYf4DPte FfoOmxBp7MaIw== Date: Mon, 15 Mar 2021 21:30:47 +0100 From: Jan Viktorin To: Slava Ovsiienko Cc: "dev@dpdk.org" , Asaf Penso , Shahaf Shuler , Matan Azrad Message-ID: <20210315213047.67c29783@tanguero.localdomain> In-Reply-To: References: <1615822438-10082-1-git-send-email-iviktorin@fit.vutbr.cz> <1615822438-10082-2-git-send-email-iviktorin@fit.vutbr.cz> <20210315171039.16b3965e@tanguero.localdomain> <20210315204856.615637a5@tanguero.localdomain> Organization: CESNET MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, 15 Mar 2021 19:52:46 +0000 Slava Ovsiienko wrote: > Hi, Jan > > > -----Original Message----- > > From: Jan Viktorin > > Sent: Monday, March 15, 2021 21:49 > > To: Slava Ovsiienko > > Cc: dev@dpdk.org; Asaf Penso ; Shahaf Shuler > > ; Matan Azrad > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev > > mode > > > > On Mon, 15 Mar 2021 19:31:25 +0000 > > Slava Ovsiienko wrote: > > > > > Hi, Jan > > > > Hello Salva, > > > > I am sorry, but I didn't get the point of your response. For me, there was 1 PF > > 0000:05:00.1, SRIOV disabled. So, I followed the guide [1]. > > > > 1. Enable SRIOV via mlxconfig. Done. > > > > 2. Configure max number of VFs. Done. > > > > 3. Reset firmware. Done. > > > > In fact, I had to reboot the machine here so I am not convinced the reset of > > firmware is the thing. But I can image that there is some other use case for it. > > After reboot, I configured 2 VFs: 0000:05:00.4, 0000:05:00.5. > > > > 4. Unbind the PF 0000:05:00.1. Done. > No, it is the second physical port. You should unbind your VFs instead - 0000:05:00.4, 0000:05:00.5. > > Please, see: > > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. OK. Now I understand. Thus, the guide should read "Unbind the VF devices (can be rebind after the switchdev mode)". I will post v2 with this change. Thanks for clarification. Jan > > With best regards, > Slava > > > > > 5. Fail, no way how to configure switchdev because of previous step. > > Thus, I strongly believe that the guide is misleading here. > > > > When I bound the PF 0000:05:00.1 back, I could configure the switchdev > > mode and everything was just OK and seemed to be working. So **after** > > the bind I configured the switchdev mode properly. > > > > [1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode > > > > Jan > > > > > > > > 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV > > > enabled, 0 VFs is configured, netdev name is ens1f0 (depends on distro > > and kernel version). > > > > > > 2. Then, we configure 3 VFs with SR-IOV, we get : > > > - 08:00.0 - PF, ens1f0 > > > - 08:00.1 - VF0, ens1f1 > > > - 08:00.2 - VF1, ens1f2 > > > - 08:00.3 - VF2, ens1f3 > > > > > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. > > > - 08:00.0 - PF, ens1f0 > > > - 08:00.1 - VF0, - > > > - 08:00.2 - VF1, - > > > - 08:00.3 - VF2, - > > > > > > 4. Then, configure switchdev mode > > > - 08:00.0 - PF, ens1f0 > > > - 08:00.1 - VF0, - > > > - 08:00.2 - VF1, - > > > - 08:00.3 - VF2, - > > > + we get 3 representor netdevs: > > > ens1f0_0 (no dedicated PCI device) > > > ens1f0_1 (no dedicated PCI device) > > > ens1f0_2 (no dedicated PCI device) > > > > > > 5. Then, bind VFs back (assume we are no going to map them to VMs) > > > - 08:00.0 - PF, ens1f0 > > > - 08:00.1 - VF0, ens1f1 > > > - 08:00.2 - VF1, ens1f2 > > > - 08:00.3 - VF2, ens1f3 > > > - ens1f0_0 (no dedicated PCI device) > > > - ens1f0_1 (no dedicated PCI device) > > > - ens1f0_2 (no dedicated PCI device) > > > > > > 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 > > > VFs, 3 representors (no PCI device, pure netdev) > > > > > > With best regards, Slava > > > > > > > -----Original Message----- > > > > From: Jan Viktorin > > > > Sent: Monday, March 15, 2021 18:11 > > > > To: Slava Ovsiienko > > > > Cc: dev@dpdk.org; Asaf Penso ; Shahaf Shuler > > > > ; Matan Azrad > > > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable > > > > switchdev mode > > > > > > > > Hello Salva, > > > > > > > > On Mon, 15 Mar 2021 15:53:51 +0000 > > > > Slava Ovsiienko wrote: > > > > > > > > > Hi, Jan > > > > > > > > > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the > > > > > netdevs > > > > from the mlx5_core driver. > > > > > Otherwise, kernel driver refuses to be configured with switchdev > > > > > mode in > > > > [5]. So, [4] can't be skipped. > > > > > After setting swithdev mode, VFs can be bound back (if it is > > > > > needed, and > > > > these ones are not mapped to VMs): > > > > > > > > OK, but I believe that it is **not possible** to follow the rule [5]. > > > > The guide explicitly says in [4] "can be rebind **after** the > > > > switchdev mode". > > > > Just, if you unbind the device, there is no way how to configure the > > > > switchdev mode, this is the contradiction I mentioned in the commit. > > > > You cannot configure switchdev mode because the interface is gone > > > > and the path /sys/class/net//compat/devlink/mode no > > longer exists. > > > > > > > > So, maybe, just the formulation is wrong. So, what is the **exact > > > > right** way how to do it? I would change the commit accordingly. > > > > Just, let's make it right. Would it work this way? > > > > > > > > # echo -n "" > > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > > # echo -n "" > > > > > /sys/bus/pci/drivers/mlx5_core/bind > > > > # echo switchdev > /sys/class/net//compat/devlink/mode > > > > > > > > It is good to mention that after the rebind, the can change. > > > > > > > > Regards, > > > > Jan > > > > > > > > > > > > > > echo -n "" > > > > > > > /sys/bus/pci/drivers/mlx5_core/bind > > > > > > > > > > With best regards, > > > > > Slava > > > > > > > > > > > -----Original Message----- > > > > > > From: Jan Viktorin > > > > > > Sent: Monday, March 15, 2021 17:34 > > > > > > To: dev@dpdk.org > > > > > > Cc: Jan Viktorin ; Asaf Penso > > > > > > ; Shahaf Shuler ; Slava > > > > > > Ovsiienko ; Matan Azrad > > > > > > > > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable > > > > > > switchdev mode > > > > > > > > > > > > From: Jan Viktorin > > > > > > > > > > > > The step 4 is a contradiction. It advices to unbind the device > > > > > > from the mlx5_core which removes the associated system network > > > > > > interface > > > > (e.g. > > > > > > eth0). In the step 5, the same system network interface (e.g. > > > > > > eth0) is required to exist. > > > > > > > > > > > > Signed-off-by: Jan Viktorin > > > > > > --- > > > > > > doc/guides/nics/mlx5.rst | 6 +----- > > > > > > 1 file changed, 1 insertion(+), 5 deletions(-) > > > > > > > > > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst > > > > > > index 0a2dc3dee..122d8e0fc 100644 > > > > > > --- a/doc/guides/nics/mlx5.rst > > > > > > +++ b/doc/guides/nics/mlx5.rst > > > > > > @@ -1370,11 +1370,7 @@ the DPDK application. > > > > > > > > > > > > echo /sys/class/net/ > > > > > device>/device/sriov_numvfs > > > > > > > > > > > > -4. Unbind the device (can be rebind after the switchdev mode):: > > > > > > - > > > > > > - echo -n "" > > > > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > > > > - > > > > > > -5. Enable switchdev mode:: > > > > > > +4. Enable switchdev mode:: > > > > > > > > > > > > echo switchdev > /sys/class/net/ > > > > > device>/compat/devlink/mode > > > > > > > > > > > > -- > > > > > > 2.30.1 > > > > > > > > >