* [dpdk-dev] [PATCH 1/2] net/mlx5: fix typos @ 2021-03-15 15:33 Jan Viktorin 2021-03-15 15:33 ` [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Jan Viktorin 0 siblings, 1 reply; 9+ messages in thread From: Jan Viktorin @ 2021-03-15 15:33 UTC (permalink / raw) To: dev Cc: Jan Viktorin, Asaf Penso, Shahaf Shuler, Viacheslav Ovsiienko, Matan Azrad From: Jan Viktorin <viktorin@cesnet.cz> Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> --- doc/guides/nics/mlx5.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 7c50497fb..0a2dc3dee 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1372,9 +1372,9 @@ the DPDK application. 4. Unbind the device (can be rebind after the switchdev mode):: - echo -n "<device pci address" > /sys/bus/pci/drivers/mlx5_core/unbind + echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind -5. Enbale switchdev mode:: +5. Enable switchdev mode:: echo switchdev > /sys/class/net/<net device>/compat/devlink/mode -- 2.30.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode 2021-03-15 15:33 [dpdk-dev] [PATCH 1/2] net/mlx5: fix typos Jan Viktorin @ 2021-03-15 15:33 ` Jan Viktorin 2021-03-15 15:53 ` Slava Ovsiienko 2021-07-15 14:51 ` Raslan Darawsheh 0 siblings, 2 replies; 9+ messages in thread From: Jan Viktorin @ 2021-03-15 15:33 UTC (permalink / raw) To: dev Cc: Jan Viktorin, Asaf Penso, Shahaf Shuler, Viacheslav Ovsiienko, Matan Azrad From: Jan Viktorin <viktorin@cesnet.cz> The step 4 is a contradiction. It advices to unbind the device from the mlx5_core which removes the associated system network interface (e.g. eth0). In the step 5, the same system network interface (e.g. eth0) is required to exist. Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> --- doc/guides/nics/mlx5.rst | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 0a2dc3dee..122d8e0fc 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1370,11 +1370,7 @@ the DPDK application. echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs -4. Unbind the device (can be rebind after the switchdev mode):: - - echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind - -5. Enable switchdev mode:: +4. Enable switchdev mode:: echo switchdev > /sys/class/net/<net device>/compat/devlink/mode -- 2.30.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode 2021-03-15 15:33 ` [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Jan Viktorin @ 2021-03-15 15:53 ` Slava Ovsiienko 2021-03-15 16:11 ` Jan Viktorin 2021-07-15 14:51 ` Raslan Darawsheh 1 sibling, 1 reply; 9+ messages in thread From: Slava Ovsiienko @ 2021-03-15 15:53 UTC (permalink / raw) To: Jan Viktorin, dev; +Cc: Jan Viktorin, Asaf Penso, Shahaf Shuler, Matan Azrad Hi, Jan Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs from the mlx5_core driver. Otherwise, kernel driver refuses to be configured with switchdev mode in [5]. So, [4] can't be skipped. After setting swithdev mode, VFs can be bound back (if it is needed, and these ones are not mapped to VMs): echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind With best regards, Slava > -----Original Message----- > From: Jan Viktorin <iviktorin@fit.vutbr.cz> > Sent: Monday, March 15, 2021 17:34 > To: dev@dpdk.org > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso <asafp@nvidia.com>; > Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko > <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com> > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode > > From: Jan Viktorin <viktorin@cesnet.cz> > > The step 4 is a contradiction. It advices to unbind the device from the > mlx5_core which removes the associated system network interface (e.g. > eth0). In the step 5, the same system network interface (e.g. eth0) is > required to exist. > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> > --- > doc/guides/nics/mlx5.rst | 6 +----- > 1 file changed, 1 insertion(+), 5 deletions(-) > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index > 0a2dc3dee..122d8e0fc 100644 > --- a/doc/guides/nics/mlx5.rst > +++ b/doc/guides/nics/mlx5.rst > @@ -1370,11 +1370,7 @@ the DPDK application. > > echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs > > -4. Unbind the device (can be rebind after the switchdev mode):: > - > - echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/unbind > - > -5. Enable switchdev mode:: > +4. Enable switchdev mode:: > > echo switchdev > /sys/class/net/<net device>/compat/devlink/mode > > -- > 2.30.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode 2021-03-15 15:53 ` Slava Ovsiienko @ 2021-03-15 16:11 ` Jan Viktorin 2021-03-15 19:31 ` Slava Ovsiienko 0 siblings, 1 reply; 9+ messages in thread From: Jan Viktorin @ 2021-03-15 16:11 UTC (permalink / raw) To: Slava Ovsiienko; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad Hello Salva, On Mon, 15 Mar 2021 15:53:51 +0000 Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > Hi, Jan > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs from the mlx5_core driver. > Otherwise, kernel driver refuses to be configured with switchdev mode in [5]. So, [4] can't be skipped. > After setting swithdev mode, VFs can be bound back (if it is needed, and these ones are not mapped to VMs): OK, but I believe that it is **not possible** to follow the rule [5]. The guide explicitly says in [4] "can be rebind **after** the switchdev mode". Just, if you unbind the device, there is no way how to configure the switchdev mode, this is the contradiction I mentioned in the commit. You cannot configure switchdev mode because the interface is gone and the path /sys/class/net/<net device>/compat/devlink/mode no longer exists. So, maybe, just the formulation is wrong. So, what is the **exact right** way how to do it? I would change the commit accordingly. Just, let's make it right. Would it work this way? # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/bind # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode It is good to mention that after the rebind, the <net device> can change. Regards, Jan > > echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind > > With best regards, > Slava > > > -----Original Message----- > > From: Jan Viktorin <iviktorin@fit.vutbr.cz> > > Sent: Monday, March 15, 2021 17:34 > > To: dev@dpdk.org > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso <asafp@nvidia.com>; > > Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko > > <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com> > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode > > > > From: Jan Viktorin <viktorin@cesnet.cz> > > > > The step 4 is a contradiction. It advices to unbind the device from the > > mlx5_core which removes the associated system network interface (e.g. > > eth0). In the step 5, the same system network interface (e.g. eth0) is > > required to exist. > > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> > > --- > > doc/guides/nics/mlx5.rst | 6 +----- > > 1 file changed, 1 insertion(+), 5 deletions(-) > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index > > 0a2dc3dee..122d8e0fc 100644 > > --- a/doc/guides/nics/mlx5.rst > > +++ b/doc/guides/nics/mlx5.rst > > @@ -1370,11 +1370,7 @@ the DPDK application. > > > > echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs > > > > -4. Unbind the device (can be rebind after the switchdev mode):: > > - > > - echo -n "<device pci address>" > > > /sys/bus/pci/drivers/mlx5_core/unbind > > - > > -5. Enable switchdev mode:: > > +4. Enable switchdev mode:: > > > > echo switchdev > /sys/class/net/<net device>/compat/devlink/mode > > > > -- > > 2.30.1 > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode 2021-03-15 16:11 ` Jan Viktorin @ 2021-03-15 19:31 ` Slava Ovsiienko 2021-03-15 19:48 ` Jan Viktorin 0 siblings, 1 reply; 9+ messages in thread From: Slava Ovsiienko @ 2021-03-15 19:31 UTC (permalink / raw) To: Jan Viktorin; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad Hi, Jan 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV enabled, 0 VFs is configured, netdev name is ens1f0 (depends on distro and kernel version). 2. Then, we configure 3 VFs with SR-IOV, we get : - 08:00.0 - PF, ens1f0 - 08:00.1 - VF0, ens1f1 - 08:00.2 - VF1, ens1f2 - 08:00.3 - VF2, ens1f3 3. Then, we should unbind VFs ONLY, PF should be kept untouched. - 08:00.0 - PF, ens1f0 - 08:00.1 - VF0, - - 08:00.2 - VF1, - - 08:00.3 - VF2, - 4. Then, configure switchdev mode - 08:00.0 - PF, ens1f0 - 08:00.1 - VF0, - - 08:00.2 - VF1, - - 08:00.3 - VF2, - + we get 3 representor netdevs: ens1f0_0 (no dedicated PCI device) ens1f0_1 (no dedicated PCI device) ens1f0_2 (no dedicated PCI device) 5. Then, bind VFs back (assume we are no going to map them to VMs) - 08:00.0 - PF, ens1f0 - 08:00.1 - VF0, ens1f1 - 08:00.2 - VF1, ens1f2 - 08:00.3 - VF2, ens1f3 - ens1f0_0 (no dedicated PCI device) - ens1f0_1 (no dedicated PCI device) - ens1f0_2 (no dedicated PCI device) 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 VFs, 3 representors (no PCI device, pure netdev) With best regards, Slava > -----Original Message----- > From: Jan Viktorin <viktorin@cesnet.cz> > Sent: Monday, March 15, 2021 18:11 > To: Slava Ovsiienko <viacheslavo@nvidia.com> > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com> > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev > mode > > Hello Salva, > > On Mon, 15 Mar 2021 15:53:51 +0000 > Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > > > Hi, Jan > > > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs > from the mlx5_core driver. > > Otherwise, kernel driver refuses to be configured with switchdev mode in > [5]. So, [4] can't be skipped. > > After setting swithdev mode, VFs can be bound back (if it is needed, and > these ones are not mapped to VMs): > > OK, but I believe that it is **not possible** to follow the rule [5]. > The guide explicitly says in [4] "can be rebind **after** the switchdev > mode". > Just, if you unbind the device, there is no way how to configure the > switchdev mode, this is the contradiction I mentioned in the commit. You > cannot configure switchdev mode because the interface is gone and the path > /sys/class/net/<net device>/compat/devlink/mode no longer exists. > > So, maybe, just the formulation is wrong. So, what is the **exact > right** way how to do it? I would change the commit accordingly. Just, let's > make it right. Would it work this way? > > # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind > # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/bind > # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode > > It is good to mention that after the rebind, the <net device> can change. > > Regards, > Jan > > > > > echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind > > > > With best regards, > > Slava > > > > > -----Original Message----- > > > From: Jan Viktorin <iviktorin@fit.vutbr.cz> > > > Sent: Monday, March 15, 2021 17:34 > > > To: dev@dpdk.org > > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso > > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava > > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad > <matan@nvidia.com> > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev > > > mode > > > > > > From: Jan Viktorin <viktorin@cesnet.cz> > > > > > > The step 4 is a contradiction. It advices to unbind the device from > > > the mlx5_core which removes the associated system network interface > (e.g. > > > eth0). In the step 5, the same system network interface (e.g. eth0) > > > is required to exist. > > > > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> > > > --- > > > doc/guides/nics/mlx5.rst | 6 +----- > > > 1 file changed, 1 insertion(+), 5 deletions(-) > > > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst > > > index 0a2dc3dee..122d8e0fc 100644 > > > --- a/doc/guides/nics/mlx5.rst > > > +++ b/doc/guides/nics/mlx5.rst > > > @@ -1370,11 +1370,7 @@ the DPDK application. > > > > > > echo <num of vfs > /sys/class/net/<net > > > device>/device/sriov_numvfs > > > > > > -4. Unbind the device (can be rebind after the switchdev mode):: > > > - > > > - echo -n "<device pci address>" > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > - > > > -5. Enable switchdev mode:: > > > +4. Enable switchdev mode:: > > > > > > echo switchdev > /sys/class/net/<net > > > device>/compat/devlink/mode > > > > > > -- > > > 2.30.1 > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode 2021-03-15 19:31 ` Slava Ovsiienko @ 2021-03-15 19:48 ` Jan Viktorin 2021-03-15 19:52 ` Slava Ovsiienko 0 siblings, 1 reply; 9+ messages in thread From: Jan Viktorin @ 2021-03-15 19:48 UTC (permalink / raw) To: Slava Ovsiienko; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad On Mon, 15 Mar 2021 19:31:25 +0000 Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > Hi, Jan Hello Salva, I am sorry, but I didn't get the point of your response. For me, there was 1 PF 0000:05:00.1, SRIOV disabled. So, I followed the guide [1]. 1. Enable SRIOV via mlxconfig. Done. 2. Configure max number of VFs. Done. 3. Reset firmware. Done. In fact, I had to reboot the machine here so I am not convinced the reset of firmware is the thing. But I can image that there is some other use case for it. After reboot, I configured 2 VFs: 0000:05:00.4, 0000:05:00.5. 4. Unbind the PF 0000:05:00.1. Done. 5. Fail, no way how to configure switchdev because of previous step. Thus, I strongly believe that the guide is misleading here. When I bound the PF 0000:05:00.1 back, I could configure the switchdev mode and everything was just OK and seemed to be working. So **after** the bind I configured the switchdev mode properly. [1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode Jan > > 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV enabled, 0 VFs is configured, > netdev name is ens1f0 (depends on distro and kernel version). > > 2. Then, we configure 3 VFs with SR-IOV, we get : > - 08:00.0 - PF, ens1f0 > - 08:00.1 - VF0, ens1f1 > - 08:00.2 - VF1, ens1f2 > - 08:00.3 - VF2, ens1f3 > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. > - 08:00.0 - PF, ens1f0 > - 08:00.1 - VF0, - > - 08:00.2 - VF1, - > - 08:00.3 - VF2, - > > 4. Then, configure switchdev mode > - 08:00.0 - PF, ens1f0 > - 08:00.1 - VF0, - > - 08:00.2 - VF1, - > - 08:00.3 - VF2, - > + we get 3 representor netdevs: > ens1f0_0 (no dedicated PCI device) > ens1f0_1 (no dedicated PCI device) > ens1f0_2 (no dedicated PCI device) > > 5. Then, bind VFs back (assume we are no going to map them to VMs) > - 08:00.0 - PF, ens1f0 > - 08:00.1 - VF0, ens1f1 > - 08:00.2 - VF1, ens1f2 > - 08:00.3 - VF2, ens1f3 > - ens1f0_0 (no dedicated PCI device) > - ens1f0_1 (no dedicated PCI device) > - ens1f0_2 (no dedicated PCI device) > > 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 VFs, 3 representors (no PCI device, pure netdev) > > With best regards, Slava > > > -----Original Message----- > > From: Jan Viktorin <viktorin@cesnet.cz> > > Sent: Monday, March 15, 2021 18:11 > > To: Slava Ovsiienko <viacheslavo@nvidia.com> > > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler > > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com> > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev > > mode > > > > Hello Salva, > > > > On Mon, 15 Mar 2021 15:53:51 +0000 > > Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > > > > > Hi, Jan > > > > > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs > > from the mlx5_core driver. > > > Otherwise, kernel driver refuses to be configured with switchdev mode in > > [5]. So, [4] can't be skipped. > > > After setting swithdev mode, VFs can be bound back (if it is needed, and > > these ones are not mapped to VMs): > > > > OK, but I believe that it is **not possible** to follow the rule [5]. > > The guide explicitly says in [4] "can be rebind **after** the switchdev > > mode". > > Just, if you unbind the device, there is no way how to configure the > > switchdev mode, this is the contradiction I mentioned in the commit. You > > cannot configure switchdev mode because the interface is gone and the path > > /sys/class/net/<net device>/compat/devlink/mode no longer exists. > > > > So, maybe, just the formulation is wrong. So, what is the **exact > > right** way how to do it? I would change the commit accordingly. Just, let's > > make it right. Would it work this way? > > > > # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind > > # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/bind > > # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode > > > > It is good to mention that after the rebind, the <net device> can change. > > > > Regards, > > Jan > > > > > > > > echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind > > > > > > With best regards, > > > Slava > > > > > > > -----Original Message----- > > > > From: Jan Viktorin <iviktorin@fit.vutbr.cz> > > > > Sent: Monday, March 15, 2021 17:34 > > > > To: dev@dpdk.org > > > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso > > > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava > > > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad > > <matan@nvidia.com> > > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev > > > > mode > > > > > > > > From: Jan Viktorin <viktorin@cesnet.cz> > > > > > > > > The step 4 is a contradiction. It advices to unbind the device from > > > > the mlx5_core which removes the associated system network interface > > (e.g. > > > > eth0). In the step 5, the same system network interface (e.g. eth0) > > > > is required to exist. > > > > > > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> > > > > --- > > > > doc/guides/nics/mlx5.rst | 6 +----- > > > > 1 file changed, 1 insertion(+), 5 deletions(-) > > > > > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst > > > > index 0a2dc3dee..122d8e0fc 100644 > > > > --- a/doc/guides/nics/mlx5.rst > > > > +++ b/doc/guides/nics/mlx5.rst > > > > @@ -1370,11 +1370,7 @@ the DPDK application. > > > > > > > > echo <num of vfs > /sys/class/net/<net > > > > device>/device/sriov_numvfs > > > > > > > > -4. Unbind the device (can be rebind after the switchdev mode):: > > > > - > > > > - echo -n "<device pci address>" > > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > > - > > > > -5. Enable switchdev mode:: > > > > +4. Enable switchdev mode:: > > > > > > > > echo switchdev > /sys/class/net/<net > > > > device>/compat/devlink/mode > > > > > > > > -- > > > > 2.30.1 > > > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode 2021-03-15 19:48 ` Jan Viktorin @ 2021-03-15 19:52 ` Slava Ovsiienko 2021-03-15 20:30 ` Jan Viktorin 0 siblings, 1 reply; 9+ messages in thread From: Slava Ovsiienko @ 2021-03-15 19:52 UTC (permalink / raw) To: Jan Viktorin; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad Hi, Jan > -----Original Message----- > From: Jan Viktorin <viktorin@cesnet.cz> > Sent: Monday, March 15, 2021 21:49 > To: Slava Ovsiienko <viacheslavo@nvidia.com> > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com> > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev > mode > > On Mon, 15 Mar 2021 19:31:25 +0000 > Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > > > Hi, Jan > > Hello Salva, > > I am sorry, but I didn't get the point of your response. For me, there was 1 PF > 0000:05:00.1, SRIOV disabled. So, I followed the guide [1]. > > 1. Enable SRIOV via mlxconfig. Done. > > 2. Configure max number of VFs. Done. > > 3. Reset firmware. Done. > > In fact, I had to reboot the machine here so I am not convinced the reset of > firmware is the thing. But I can image that there is some other use case for it. > After reboot, I configured 2 VFs: 0000:05:00.4, 0000:05:00.5. > > 4. Unbind the PF 0000:05:00.1. Done. No, it is the second physical port. You should unbind your VFs instead - 0000:05:00.4, 0000:05:00.5. Please, see: > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. With best regards, Slava > > 5. Fail, no way how to configure switchdev because of previous step. > Thus, I strongly believe that the guide is misleading here. > > When I bound the PF 0000:05:00.1 back, I could configure the switchdev > mode and everything was just OK and seemed to be working. So **after** > the bind I configured the switchdev mode properly. > > [1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode > > Jan > > > > > 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV > > enabled, 0 VFs is configured, netdev name is ens1f0 (depends on distro > and kernel version). > > > > 2. Then, we configure 3 VFs with SR-IOV, we get : > > - 08:00.0 - PF, ens1f0 > > - 08:00.1 - VF0, ens1f1 > > - 08:00.2 - VF1, ens1f2 > > - 08:00.3 - VF2, ens1f3 > > > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. > > - 08:00.0 - PF, ens1f0 > > - 08:00.1 - VF0, - > > - 08:00.2 - VF1, - > > - 08:00.3 - VF2, - > > > > 4. Then, configure switchdev mode > > - 08:00.0 - PF, ens1f0 > > - 08:00.1 - VF0, - > > - 08:00.2 - VF1, - > > - 08:00.3 - VF2, - > > + we get 3 representor netdevs: > > ens1f0_0 (no dedicated PCI device) > > ens1f0_1 (no dedicated PCI device) > > ens1f0_2 (no dedicated PCI device) > > > > 5. Then, bind VFs back (assume we are no going to map them to VMs) > > - 08:00.0 - PF, ens1f0 > > - 08:00.1 - VF0, ens1f1 > > - 08:00.2 - VF1, ens1f2 > > - 08:00.3 - VF2, ens1f3 > > - ens1f0_0 (no dedicated PCI device) > > - ens1f0_1 (no dedicated PCI device) > > - ens1f0_2 (no dedicated PCI device) > > > > 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 > > VFs, 3 representors (no PCI device, pure netdev) > > > > With best regards, Slava > > > > > -----Original Message----- > > > From: Jan Viktorin <viktorin@cesnet.cz> > > > Sent: Monday, March 15, 2021 18:11 > > > To: Slava Ovsiienko <viacheslavo@nvidia.com> > > > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler > > > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com> > > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable > > > switchdev mode > > > > > > Hello Salva, > > > > > > On Mon, 15 Mar 2021 15:53:51 +0000 > > > Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > > > > > > > Hi, Jan > > > > > > > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the > > > > netdevs > > > from the mlx5_core driver. > > > > Otherwise, kernel driver refuses to be configured with switchdev > > > > mode in > > > [5]. So, [4] can't be skipped. > > > > After setting swithdev mode, VFs can be bound back (if it is > > > > needed, and > > > these ones are not mapped to VMs): > > > > > > OK, but I believe that it is **not possible** to follow the rule [5]. > > > The guide explicitly says in [4] "can be rebind **after** the > > > switchdev mode". > > > Just, if you unbind the device, there is no way how to configure the > > > switchdev mode, this is the contradiction I mentioned in the commit. > > > You cannot configure switchdev mode because the interface is gone > > > and the path /sys/class/net/<net device>/compat/devlink/mode no > longer exists. > > > > > > So, maybe, just the formulation is wrong. So, what is the **exact > > > right** way how to do it? I would change the commit accordingly. > > > Just, let's make it right. Would it work this way? > > > > > > # echo -n "<device pci address>" > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > # echo -n "<device pci address>" > > > > /sys/bus/pci/drivers/mlx5_core/bind > > > # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode > > > > > > It is good to mention that after the rebind, the <net device> can change. > > > > > > Regards, > > > Jan > > > > > > > > > > > echo -n "<device pci address>" > > > > > > /sys/bus/pci/drivers/mlx5_core/bind > > > > > > > > With best regards, > > > > Slava > > > > > > > > > -----Original Message----- > > > > > From: Jan Viktorin <iviktorin@fit.vutbr.cz> > > > > > Sent: Monday, March 15, 2021 17:34 > > > > > To: dev@dpdk.org > > > > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso > > > > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava > > > > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad > > > <matan@nvidia.com> > > > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable > > > > > switchdev mode > > > > > > > > > > From: Jan Viktorin <viktorin@cesnet.cz> > > > > > > > > > > The step 4 is a contradiction. It advices to unbind the device > > > > > from the mlx5_core which removes the associated system network > > > > > interface > > > (e.g. > > > > > eth0). In the step 5, the same system network interface (e.g. > > > > > eth0) is required to exist. > > > > > > > > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> > > > > > --- > > > > > doc/guides/nics/mlx5.rst | 6 +----- > > > > > 1 file changed, 1 insertion(+), 5 deletions(-) > > > > > > > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst > > > > > index 0a2dc3dee..122d8e0fc 100644 > > > > > --- a/doc/guides/nics/mlx5.rst > > > > > +++ b/doc/guides/nics/mlx5.rst > > > > > @@ -1370,11 +1370,7 @@ the DPDK application. > > > > > > > > > > echo <num of vfs > /sys/class/net/<net > > > > > device>/device/sriov_numvfs > > > > > > > > > > -4. Unbind the device (can be rebind after the switchdev mode):: > > > > > - > > > > > - echo -n "<device pci address>" > > > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > > > - > > > > > -5. Enable switchdev mode:: > > > > > +4. Enable switchdev mode:: > > > > > > > > > > echo switchdev > /sys/class/net/<net > > > > > device>/compat/devlink/mode > > > > > > > > > > -- > > > > > 2.30.1 > > > > > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode 2021-03-15 19:52 ` Slava Ovsiienko @ 2021-03-15 20:30 ` Jan Viktorin 0 siblings, 0 replies; 9+ messages in thread From: Jan Viktorin @ 2021-03-15 20:30 UTC (permalink / raw) To: Slava Ovsiienko; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad On Mon, 15 Mar 2021 19:52:46 +0000 Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > Hi, Jan > > > -----Original Message----- > > From: Jan Viktorin <viktorin@cesnet.cz> > > Sent: Monday, March 15, 2021 21:49 > > To: Slava Ovsiienko <viacheslavo@nvidia.com> > > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler > > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com> > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev > > mode > > > > On Mon, 15 Mar 2021 19:31:25 +0000 > > Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > > > > > Hi, Jan > > > > Hello Salva, > > > > I am sorry, but I didn't get the point of your response. For me, there was 1 PF > > 0000:05:00.1, SRIOV disabled. So, I followed the guide [1]. > > > > 1. Enable SRIOV via mlxconfig. Done. > > > > 2. Configure max number of VFs. Done. > > > > 3. Reset firmware. Done. > > > > In fact, I had to reboot the machine here so I am not convinced the reset of > > firmware is the thing. But I can image that there is some other use case for it. > > After reboot, I configured 2 VFs: 0000:05:00.4, 0000:05:00.5. > > > > 4. Unbind the PF 0000:05:00.1. Done. > No, it is the second physical port. You should unbind your VFs instead - 0000:05:00.4, 0000:05:00.5. > > Please, see: > > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. OK. Now I understand. Thus, the guide should read "Unbind the VF devices (can be rebind after the switchdev mode)". I will post v2 with this change. Thanks for clarification. Jan > > With best regards, > Slava > > > > > 5. Fail, no way how to configure switchdev because of previous step. > > Thus, I strongly believe that the guide is misleading here. > > > > When I bound the PF 0000:05:00.1 back, I could configure the switchdev > > mode and everything was just OK and seemed to be working. So **after** > > the bind I configured the switchdev mode properly. > > > > [1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode > > > > Jan > > > > > > > > 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV > > > enabled, 0 VFs is configured, netdev name is ens1f0 (depends on distro > > and kernel version). > > > > > > 2. Then, we configure 3 VFs with SR-IOV, we get : > > > - 08:00.0 - PF, ens1f0 > > > - 08:00.1 - VF0, ens1f1 > > > - 08:00.2 - VF1, ens1f2 > > > - 08:00.3 - VF2, ens1f3 > > > > > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched. > > > - 08:00.0 - PF, ens1f0 > > > - 08:00.1 - VF0, - > > > - 08:00.2 - VF1, - > > > - 08:00.3 - VF2, - > > > > > > 4. Then, configure switchdev mode > > > - 08:00.0 - PF, ens1f0 > > > - 08:00.1 - VF0, - > > > - 08:00.2 - VF1, - > > > - 08:00.3 - VF2, - > > > + we get 3 representor netdevs: > > > ens1f0_0 (no dedicated PCI device) > > > ens1f0_1 (no dedicated PCI device) > > > ens1f0_2 (no dedicated PCI device) > > > > > > 5. Then, bind VFs back (assume we are no going to map them to VMs) > > > - 08:00.0 - PF, ens1f0 > > > - 08:00.1 - VF0, ens1f1 > > > - 08:00.2 - VF1, ens1f2 > > > - 08:00.3 - VF2, ens1f3 > > > - ens1f0_0 (no dedicated PCI device) > > > - ens1f0_1 (no dedicated PCI device) > > > - ens1f0_2 (no dedicated PCI device) > > > > > > 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 > > > VFs, 3 representors (no PCI device, pure netdev) > > > > > > With best regards, Slava > > > > > > > -----Original Message----- > > > > From: Jan Viktorin <viktorin@cesnet.cz> > > > > Sent: Monday, March 15, 2021 18:11 > > > > To: Slava Ovsiienko <viacheslavo@nvidia.com> > > > > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler > > > > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com> > > > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable > > > > switchdev mode > > > > > > > > Hello Salva, > > > > > > > > On Mon, 15 Mar 2021 15:53:51 +0000 > > > > Slava Ovsiienko <viacheslavo@nvidia.com> wrote: > > > > > > > > > Hi, Jan > > > > > > > > > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the > > > > > netdevs > > > > from the mlx5_core driver. > > > > > Otherwise, kernel driver refuses to be configured with switchdev > > > > > mode in > > > > [5]. So, [4] can't be skipped. > > > > > After setting swithdev mode, VFs can be bound back (if it is > > > > > needed, and > > > > these ones are not mapped to VMs): > > > > > > > > OK, but I believe that it is **not possible** to follow the rule [5]. > > > > The guide explicitly says in [4] "can be rebind **after** the > > > > switchdev mode". > > > > Just, if you unbind the device, there is no way how to configure the > > > > switchdev mode, this is the contradiction I mentioned in the commit. > > > > You cannot configure switchdev mode because the interface is gone > > > > and the path /sys/class/net/<net device>/compat/devlink/mode no > > longer exists. > > > > > > > > So, maybe, just the formulation is wrong. So, what is the **exact > > > > right** way how to do it? I would change the commit accordingly. > > > > Just, let's make it right. Would it work this way? > > > > > > > > # echo -n "<device pci address>" > > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > > # echo -n "<device pci address>" > > > > > /sys/bus/pci/drivers/mlx5_core/bind > > > > # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode > > > > > > > > It is good to mention that after the rebind, the <net device> can change. > > > > > > > > Regards, > > > > Jan > > > > > > > > > > > > > > echo -n "<device pci address>" > > > > > > > /sys/bus/pci/drivers/mlx5_core/bind > > > > > > > > > > With best regards, > > > > > Slava > > > > > > > > > > > -----Original Message----- > > > > > > From: Jan Viktorin <iviktorin@fit.vutbr.cz> > > > > > > Sent: Monday, March 15, 2021 17:34 > > > > > > To: dev@dpdk.org > > > > > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso > > > > > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava > > > > > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad > > > > <matan@nvidia.com> > > > > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable > > > > > > switchdev mode > > > > > > > > > > > > From: Jan Viktorin <viktorin@cesnet.cz> > > > > > > > > > > > > The step 4 is a contradiction. It advices to unbind the device > > > > > > from the mlx5_core which removes the associated system network > > > > > > interface > > > > (e.g. > > > > > > eth0). In the step 5, the same system network interface (e.g. > > > > > > eth0) is required to exist. > > > > > > > > > > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> > > > > > > --- > > > > > > doc/guides/nics/mlx5.rst | 6 +----- > > > > > > 1 file changed, 1 insertion(+), 5 deletions(-) > > > > > > > > > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst > > > > > > index 0a2dc3dee..122d8e0fc 100644 > > > > > > --- a/doc/guides/nics/mlx5.rst > > > > > > +++ b/doc/guides/nics/mlx5.rst > > > > > > @@ -1370,11 +1370,7 @@ the DPDK application. > > > > > > > > > > > > echo <num of vfs > /sys/class/net/<net > > > > > > device>/device/sriov_numvfs > > > > > > > > > > > > -4. Unbind the device (can be rebind after the switchdev mode):: > > > > > > - > > > > > > - echo -n "<device pci address>" > > > > > > > /sys/bus/pci/drivers/mlx5_core/unbind > > > > > > - > > > > > > -5. Enable switchdev mode:: > > > > > > +4. Enable switchdev mode:: > > > > > > > > > > > > echo switchdev > /sys/class/net/<net > > > > > > device>/compat/devlink/mode > > > > > > > > > > > > -- > > > > > > 2.30.1 > > > > > > > > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode 2021-03-15 15:33 ` [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Jan Viktorin 2021-03-15 15:53 ` Slava Ovsiienko @ 2021-07-15 14:51 ` Raslan Darawsheh 1 sibling, 0 replies; 9+ messages in thread From: Raslan Darawsheh @ 2021-07-15 14:51 UTC (permalink / raw) To: Jan Viktorin, dev Cc: Jan Viktorin, Asaf Penso, Shahaf Shuler, Slava Ovsiienko, Matan Azrad Hi, Can you kindly update on the status of this series? Kindest regards, Raslan Darawsheh > -----Original Message----- > From: dev <dev-bounces@dpdk.org> On Behalf Of Jan Viktorin > Sent: Monday, March 15, 2021 5:34 PM > To: dev@dpdk.org > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso <asafp@nvidia.com>; > Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko > <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com> > Subject: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable > switchdev mode > > From: Jan Viktorin <viktorin@cesnet.cz> > > The step 4 is a contradiction. It advices to unbind the device from the > mlx5_core which removes the associated system network interface (e.g. > eth0). In the step 5, the same system network interface (e.g. eth0) is > required to exist. > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz> > --- > doc/guides/nics/mlx5.rst | 6 +----- > 1 file changed, 1 insertion(+), 5 deletions(-) > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index > 0a2dc3dee..122d8e0fc 100644 > --- a/doc/guides/nics/mlx5.rst > +++ b/doc/guides/nics/mlx5.rst > @@ -1370,11 +1370,7 @@ the DPDK application. > > echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs > > -4. Unbind the device (can be rebind after the switchdev mode):: > - > - echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/unbind > - > -5. Enable switchdev mode:: > +4. Enable switchdev mode:: > > echo switchdev > /sys/class/net/<net device>/compat/devlink/mode > > -- > 2.30.1 ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-07-15 14:51 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-15 15:33 [dpdk-dev] [PATCH 1/2] net/mlx5: fix typos Jan Viktorin 2021-03-15 15:33 ` [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Jan Viktorin 2021-03-15 15:53 ` Slava Ovsiienko 2021-03-15 16:11 ` Jan Viktorin 2021-03-15 19:31 ` Slava Ovsiienko 2021-03-15 19:48 ` Jan Viktorin 2021-03-15 19:52 ` Slava Ovsiienko 2021-03-15 20:30 ` Jan Viktorin 2021-07-15 14:51 ` Raslan Darawsheh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).