* [dpdk-dev] [PATCH 1/2] net/mlx5: fix typos
@ 2021-03-15 15:33 Jan Viktorin
2021-03-15 15:33 ` [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Jan Viktorin
0 siblings, 1 reply; 9+ messages in thread
From: Jan Viktorin @ 2021-03-15 15:33 UTC (permalink / raw)
To: dev
Cc: Jan Viktorin, Asaf Penso, Shahaf Shuler, Viacheslav Ovsiienko,
Matan Azrad
From: Jan Viktorin <viktorin@cesnet.cz>
Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
---
doc/guides/nics/mlx5.rst | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 7c50497fb..0a2dc3dee 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -1372,9 +1372,9 @@ the DPDK application.
4. Unbind the device (can be rebind after the switchdev mode)::
- echo -n "<device pci address" > /sys/bus/pci/drivers/mlx5_core/unbind
+ echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind
-5. Enbale switchdev mode::
+5. Enable switchdev mode::
echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
--
2.30.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
2021-03-15 15:33 [dpdk-dev] [PATCH 1/2] net/mlx5: fix typos Jan Viktorin
@ 2021-03-15 15:33 ` Jan Viktorin
2021-03-15 15:53 ` Slava Ovsiienko
2021-07-15 14:51 ` Raslan Darawsheh
0 siblings, 2 replies; 9+ messages in thread
From: Jan Viktorin @ 2021-03-15 15:33 UTC (permalink / raw)
To: dev
Cc: Jan Viktorin, Asaf Penso, Shahaf Shuler, Viacheslav Ovsiienko,
Matan Azrad
From: Jan Viktorin <viktorin@cesnet.cz>
The step 4 is a contradiction. It advices to unbind the device from the
mlx5_core which removes the associated system network interface (e.g.
eth0). In the step 5, the same system network interface (e.g. eth0) is
required to exist.
Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
---
doc/guides/nics/mlx5.rst | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 0a2dc3dee..122d8e0fc 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -1370,11 +1370,7 @@ the DPDK application.
echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs
-4. Unbind the device (can be rebind after the switchdev mode)::
-
- echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind
-
-5. Enable switchdev mode::
+4. Enable switchdev mode::
echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
--
2.30.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
2021-03-15 15:33 ` [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Jan Viktorin
@ 2021-03-15 15:53 ` Slava Ovsiienko
2021-03-15 16:11 ` Jan Viktorin
2021-07-15 14:51 ` Raslan Darawsheh
1 sibling, 1 reply; 9+ messages in thread
From: Slava Ovsiienko @ 2021-03-15 15:53 UTC (permalink / raw)
To: Jan Viktorin, dev; +Cc: Jan Viktorin, Asaf Penso, Shahaf Shuler, Matan Azrad
Hi, Jan
Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs from the mlx5_core driver.
Otherwise, kernel driver refuses to be configured with switchdev mode in [5]. So, [4] can't be skipped.
After setting swithdev mode, VFs can be bound back (if it is needed, and these ones are not mapped to VMs):
echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind
With best regards,
Slava
> -----Original Message-----
> From: Jan Viktorin <iviktorin@fit.vutbr.cz>
> Sent: Monday, March 15, 2021 17:34
> To: dev@dpdk.org
> Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso <asafp@nvidia.com>;
> Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
>
> From: Jan Viktorin <viktorin@cesnet.cz>
>
> The step 4 is a contradiction. It advices to unbind the device from the
> mlx5_core which removes the associated system network interface (e.g.
> eth0). In the step 5, the same system network interface (e.g. eth0) is
> required to exist.
>
> Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
> ---
> doc/guides/nics/mlx5.rst | 6 +-----
> 1 file changed, 1 insertion(+), 5 deletions(-)
>
> diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index
> 0a2dc3dee..122d8e0fc 100644
> --- a/doc/guides/nics/mlx5.rst
> +++ b/doc/guides/nics/mlx5.rst
> @@ -1370,11 +1370,7 @@ the DPDK application.
>
> echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs
>
> -4. Unbind the device (can be rebind after the switchdev mode)::
> -
> - echo -n "<device pci address>" >
> /sys/bus/pci/drivers/mlx5_core/unbind
> -
> -5. Enable switchdev mode::
> +4. Enable switchdev mode::
>
> echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
>
> --
> 2.30.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
2021-03-15 15:53 ` Slava Ovsiienko
@ 2021-03-15 16:11 ` Jan Viktorin
2021-03-15 19:31 ` Slava Ovsiienko
0 siblings, 1 reply; 9+ messages in thread
From: Jan Viktorin @ 2021-03-15 16:11 UTC (permalink / raw)
To: Slava Ovsiienko; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad
Hello Salva,
On Mon, 15 Mar 2021 15:53:51 +0000
Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> Hi, Jan
>
> Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs from the mlx5_core driver.
> Otherwise, kernel driver refuses to be configured with switchdev mode in [5]. So, [4] can't be skipped.
> After setting swithdev mode, VFs can be bound back (if it is needed, and these ones are not mapped to VMs):
OK, but I believe that it is **not possible** to follow the rule [5].
The guide explicitly says in [4] "can be rebind **after** the switchdev mode".
Just, if you unbind the device, there is no way how to configure the switchdev mode,
this is the contradiction I mentioned in the commit. You cannot configure switchdev
mode because the interface is gone and the path /sys/class/net/<net device>/compat/devlink/mode
no longer exists.
So, maybe, just the formulation is wrong. So, what is the **exact
right** way how to do it? I would change the commit accordingly. Just,
let's make it right. Would it work this way?
# echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind
# echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/bind
# echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
It is good to mention that after the rebind, the <net device> can
change.
Regards,
Jan
>
> echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind
>
> With best regards,
> Slava
>
> > -----Original Message-----
> > From: Jan Viktorin <iviktorin@fit.vutbr.cz>
> > Sent: Monday, March 15, 2021 17:34
> > To: dev@dpdk.org
> > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso <asafp@nvidia.com>;
> > Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko
> > <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com>
> > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
> >
> > From: Jan Viktorin <viktorin@cesnet.cz>
> >
> > The step 4 is a contradiction. It advices to unbind the device from the
> > mlx5_core which removes the associated system network interface (e.g.
> > eth0). In the step 5, the same system network interface (e.g. eth0) is
> > required to exist.
> >
> > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
> > ---
> > doc/guides/nics/mlx5.rst | 6 +-----
> > 1 file changed, 1 insertion(+), 5 deletions(-)
> >
> > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index
> > 0a2dc3dee..122d8e0fc 100644
> > --- a/doc/guides/nics/mlx5.rst
> > +++ b/doc/guides/nics/mlx5.rst
> > @@ -1370,11 +1370,7 @@ the DPDK application.
> >
> > echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs
> >
> > -4. Unbind the device (can be rebind after the switchdev mode)::
> > -
> > - echo -n "<device pci address>" >
> > /sys/bus/pci/drivers/mlx5_core/unbind
> > -
> > -5. Enable switchdev mode::
> > +4. Enable switchdev mode::
> >
> > echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
> >
> > --
> > 2.30.1
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
2021-03-15 16:11 ` Jan Viktorin
@ 2021-03-15 19:31 ` Slava Ovsiienko
2021-03-15 19:48 ` Jan Viktorin
0 siblings, 1 reply; 9+ messages in thread
From: Slava Ovsiienko @ 2021-03-15 19:31 UTC (permalink / raw)
To: Jan Viktorin; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad
Hi, Jan
1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV enabled, 0 VFs is configured,
netdev name is ens1f0 (depends on distro and kernel version).
2. Then, we configure 3 VFs with SR-IOV, we get :
- 08:00.0 - PF, ens1f0
- 08:00.1 - VF0, ens1f1
- 08:00.2 - VF1, ens1f2
- 08:00.3 - VF2, ens1f3
3. Then, we should unbind VFs ONLY, PF should be kept untouched.
- 08:00.0 - PF, ens1f0
- 08:00.1 - VF0, -
- 08:00.2 - VF1, -
- 08:00.3 - VF2, -
4. Then, configure switchdev mode
- 08:00.0 - PF, ens1f0
- 08:00.1 - VF0, -
- 08:00.2 - VF1, -
- 08:00.3 - VF2, -
+ we get 3 representor netdevs:
ens1f0_0 (no dedicated PCI device)
ens1f0_1 (no dedicated PCI device)
ens1f0_2 (no dedicated PCI device)
5. Then, bind VFs back (assume we are no going to map them to VMs)
- 08:00.0 - PF, ens1f0
- 08:00.1 - VF0, ens1f1
- 08:00.2 - VF1, ens1f2
- 08:00.3 - VF2, ens1f3
- ens1f0_0 (no dedicated PCI device)
- ens1f0_1 (no dedicated PCI device)
- ens1f0_2 (no dedicated PCI device)
6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 VFs, 3 representors (no PCI device, pure netdev)
With best regards, Slava
> -----Original Message-----
> From: Jan Viktorin <viktorin@cesnet.cz>
> Sent: Monday, March 15, 2021 18:11
> To: Slava Ovsiienko <viacheslavo@nvidia.com>
> Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler
> <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev
> mode
>
> Hello Salva,
>
> On Mon, 15 Mar 2021 15:53:51 +0000
> Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
>
> > Hi, Jan
> >
> > Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs
> from the mlx5_core driver.
> > Otherwise, kernel driver refuses to be configured with switchdev mode in
> [5]. So, [4] can't be skipped.
> > After setting swithdev mode, VFs can be bound back (if it is needed, and
> these ones are not mapped to VMs):
>
> OK, but I believe that it is **not possible** to follow the rule [5].
> The guide explicitly says in [4] "can be rebind **after** the switchdev
> mode".
> Just, if you unbind the device, there is no way how to configure the
> switchdev mode, this is the contradiction I mentioned in the commit. You
> cannot configure switchdev mode because the interface is gone and the path
> /sys/class/net/<net device>/compat/devlink/mode no longer exists.
>
> So, maybe, just the formulation is wrong. So, what is the **exact
> right** way how to do it? I would change the commit accordingly. Just, let's
> make it right. Would it work this way?
>
> # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind
> # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/bind
> # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
>
> It is good to mention that after the rebind, the <net device> can change.
>
> Regards,
> Jan
>
> >
> > echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind
> >
> > With best regards,
> > Slava
> >
> > > -----Original Message-----
> > > From: Jan Viktorin <iviktorin@fit.vutbr.cz>
> > > Sent: Monday, March 15, 2021 17:34
> > > To: dev@dpdk.org
> > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso
> > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava
> > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad
> <matan@nvidia.com>
> > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev
> > > mode
> > >
> > > From: Jan Viktorin <viktorin@cesnet.cz>
> > >
> > > The step 4 is a contradiction. It advices to unbind the device from
> > > the mlx5_core which removes the associated system network interface
> (e.g.
> > > eth0). In the step 5, the same system network interface (e.g. eth0)
> > > is required to exist.
> > >
> > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
> > > ---
> > > doc/guides/nics/mlx5.rst | 6 +-----
> > > 1 file changed, 1 insertion(+), 5 deletions(-)
> > >
> > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
> > > index 0a2dc3dee..122d8e0fc 100644
> > > --- a/doc/guides/nics/mlx5.rst
> > > +++ b/doc/guides/nics/mlx5.rst
> > > @@ -1370,11 +1370,7 @@ the DPDK application.
> > >
> > > echo <num of vfs > /sys/class/net/<net
> > > device>/device/sriov_numvfs
> > >
> > > -4. Unbind the device (can be rebind after the switchdev mode)::
> > > -
> > > - echo -n "<device pci address>" >
> > > /sys/bus/pci/drivers/mlx5_core/unbind
> > > -
> > > -5. Enable switchdev mode::
> > > +4. Enable switchdev mode::
> > >
> > > echo switchdev > /sys/class/net/<net
> > > device>/compat/devlink/mode
> > >
> > > --
> > > 2.30.1
> >
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
2021-03-15 19:31 ` Slava Ovsiienko
@ 2021-03-15 19:48 ` Jan Viktorin
2021-03-15 19:52 ` Slava Ovsiienko
0 siblings, 1 reply; 9+ messages in thread
From: Jan Viktorin @ 2021-03-15 19:48 UTC (permalink / raw)
To: Slava Ovsiienko; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad
On Mon, 15 Mar 2021 19:31:25 +0000
Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> Hi, Jan
Hello Salva,
I am sorry, but I didn't get the point of your response. For me, there
was 1 PF 0000:05:00.1, SRIOV disabled. So, I followed the guide [1].
1. Enable SRIOV via mlxconfig. Done.
2. Configure max number of VFs. Done.
3. Reset firmware. Done.
In fact, I had to reboot the machine here so I am not convinced the
reset of firmware is the thing. But I can image that there is some
other use case for it. After reboot, I configured 2 VFs: 0000:05:00.4,
0000:05:00.5.
4. Unbind the PF 0000:05:00.1. Done.
5. Fail, no way how to configure switchdev because of previous step.
Thus, I strongly believe that the guide is misleading here.
When I bound the PF 0000:05:00.1 back, I could configure the switchdev
mode and everything was just OK and seemed to be working. So **after**
the bind I configured the switchdev mode properly.
[1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode
Jan
>
> 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV enabled, 0 VFs is configured,
> netdev name is ens1f0 (depends on distro and kernel version).
>
> 2. Then, we configure 3 VFs with SR-IOV, we get :
> - 08:00.0 - PF, ens1f0
> - 08:00.1 - VF0, ens1f1
> - 08:00.2 - VF1, ens1f2
> - 08:00.3 - VF2, ens1f3
>
> 3. Then, we should unbind VFs ONLY, PF should be kept untouched.
> - 08:00.0 - PF, ens1f0
> - 08:00.1 - VF0, -
> - 08:00.2 - VF1, -
> - 08:00.3 - VF2, -
>
> 4. Then, configure switchdev mode
> - 08:00.0 - PF, ens1f0
> - 08:00.1 - VF0, -
> - 08:00.2 - VF1, -
> - 08:00.3 - VF2, -
> + we get 3 representor netdevs:
> ens1f0_0 (no dedicated PCI device)
> ens1f0_1 (no dedicated PCI device)
> ens1f0_2 (no dedicated PCI device)
>
> 5. Then, bind VFs back (assume we are no going to map them to VMs)
> - 08:00.0 - PF, ens1f0
> - 08:00.1 - VF0, ens1f1
> - 08:00.2 - VF1, ens1f2
> - 08:00.3 - VF2, ens1f3
> - ens1f0_0 (no dedicated PCI device)
> - ens1f0_1 (no dedicated PCI device)
> - ens1f0_2 (no dedicated PCI device)
>
> 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 VFs, 3 representors (no PCI device, pure netdev)
>
> With best regards, Slava
>
> > -----Original Message-----
> > From: Jan Viktorin <viktorin@cesnet.cz>
> > Sent: Monday, March 15, 2021 18:11
> > To: Slava Ovsiienko <viacheslavo@nvidia.com>
> > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler
> > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com>
> > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev
> > mode
> >
> > Hello Salva,
> >
> > On Mon, 15 Mar 2021 15:53:51 +0000
> > Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> >
> > > Hi, Jan
> > >
> > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs
> > from the mlx5_core driver.
> > > Otherwise, kernel driver refuses to be configured with switchdev mode in
> > [5]. So, [4] can't be skipped.
> > > After setting swithdev mode, VFs can be bound back (if it is needed, and
> > these ones are not mapped to VMs):
> >
> > OK, but I believe that it is **not possible** to follow the rule [5].
> > The guide explicitly says in [4] "can be rebind **after** the switchdev
> > mode".
> > Just, if you unbind the device, there is no way how to configure the
> > switchdev mode, this is the contradiction I mentioned in the commit. You
> > cannot configure switchdev mode because the interface is gone and the path
> > /sys/class/net/<net device>/compat/devlink/mode no longer exists.
> >
> > So, maybe, just the formulation is wrong. So, what is the **exact
> > right** way how to do it? I would change the commit accordingly. Just, let's
> > make it right. Would it work this way?
> >
> > # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind
> > # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/bind
> > # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
> >
> > It is good to mention that after the rebind, the <net device> can change.
> >
> > Regards,
> > Jan
> >
> > >
> > > echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind
> > >
> > > With best regards,
> > > Slava
> > >
> > > > -----Original Message-----
> > > > From: Jan Viktorin <iviktorin@fit.vutbr.cz>
> > > > Sent: Monday, March 15, 2021 17:34
> > > > To: dev@dpdk.org
> > > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso
> > > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava
> > > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad
> > <matan@nvidia.com>
> > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev
> > > > mode
> > > >
> > > > From: Jan Viktorin <viktorin@cesnet.cz>
> > > >
> > > > The step 4 is a contradiction. It advices to unbind the device from
> > > > the mlx5_core which removes the associated system network interface
> > (e.g.
> > > > eth0). In the step 5, the same system network interface (e.g. eth0)
> > > > is required to exist.
> > > >
> > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
> > > > ---
> > > > doc/guides/nics/mlx5.rst | 6 +-----
> > > > 1 file changed, 1 insertion(+), 5 deletions(-)
> > > >
> > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
> > > > index 0a2dc3dee..122d8e0fc 100644
> > > > --- a/doc/guides/nics/mlx5.rst
> > > > +++ b/doc/guides/nics/mlx5.rst
> > > > @@ -1370,11 +1370,7 @@ the DPDK application.
> > > >
> > > > echo <num of vfs > /sys/class/net/<net
> > > > device>/device/sriov_numvfs
> > > >
> > > > -4. Unbind the device (can be rebind after the switchdev mode)::
> > > > -
> > > > - echo -n "<device pci address>" >
> > > > /sys/bus/pci/drivers/mlx5_core/unbind
> > > > -
> > > > -5. Enable switchdev mode::
> > > > +4. Enable switchdev mode::
> > > >
> > > > echo switchdev > /sys/class/net/<net
> > > > device>/compat/devlink/mode
> > > >
> > > > --
> > > > 2.30.1
> > >
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
2021-03-15 19:48 ` Jan Viktorin
@ 2021-03-15 19:52 ` Slava Ovsiienko
2021-03-15 20:30 ` Jan Viktorin
0 siblings, 1 reply; 9+ messages in thread
From: Slava Ovsiienko @ 2021-03-15 19:52 UTC (permalink / raw)
To: Jan Viktorin; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad
Hi, Jan
> -----Original Message-----
> From: Jan Viktorin <viktorin@cesnet.cz>
> Sent: Monday, March 15, 2021 21:49
> To: Slava Ovsiienko <viacheslavo@nvidia.com>
> Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler
> <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev
> mode
>
> On Mon, 15 Mar 2021 19:31:25 +0000
> Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
>
> > Hi, Jan
>
> Hello Salva,
>
> I am sorry, but I didn't get the point of your response. For me, there was 1 PF
> 0000:05:00.1, SRIOV disabled. So, I followed the guide [1].
>
> 1. Enable SRIOV via mlxconfig. Done.
>
> 2. Configure max number of VFs. Done.
>
> 3. Reset firmware. Done.
>
> In fact, I had to reboot the machine here so I am not convinced the reset of
> firmware is the thing. But I can image that there is some other use case for it.
> After reboot, I configured 2 VFs: 0000:05:00.4, 0000:05:00.5.
>
> 4. Unbind the PF 0000:05:00.1. Done.
No, it is the second physical port. You should unbind your VFs instead - 0000:05:00.4, 0000:05:00.5.
Please, see:
> > 3. Then, we should unbind VFs ONLY, PF should be kept untouched.
With best regards,
Slava
>
> 5. Fail, no way how to configure switchdev because of previous step.
> Thus, I strongly believe that the guide is misleading here.
>
> When I bound the PF 0000:05:00.1 back, I could configure the switchdev
> mode and everything was just OK and seemed to be working. So **after**
> the bind I configured the switchdev mode properly.
>
> [1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode
>
> Jan
>
> >
> > 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV
> > enabled, 0 VFs is configured, netdev name is ens1f0 (depends on distro
> and kernel version).
> >
> > 2. Then, we configure 3 VFs with SR-IOV, we get :
> > - 08:00.0 - PF, ens1f0
> > - 08:00.1 - VF0, ens1f1
> > - 08:00.2 - VF1, ens1f2
> > - 08:00.3 - VF2, ens1f3
> >
> > 3. Then, we should unbind VFs ONLY, PF should be kept untouched.
> > - 08:00.0 - PF, ens1f0
> > - 08:00.1 - VF0, -
> > - 08:00.2 - VF1, -
> > - 08:00.3 - VF2, -
> >
> > 4. Then, configure switchdev mode
> > - 08:00.0 - PF, ens1f0
> > - 08:00.1 - VF0, -
> > - 08:00.2 - VF1, -
> > - 08:00.3 - VF2, -
> > + we get 3 representor netdevs:
> > ens1f0_0 (no dedicated PCI device)
> > ens1f0_1 (no dedicated PCI device)
> > ens1f0_2 (no dedicated PCI device)
> >
> > 5. Then, bind VFs back (assume we are no going to map them to VMs)
> > - 08:00.0 - PF, ens1f0
> > - 08:00.1 - VF0, ens1f1
> > - 08:00.2 - VF1, ens1f2
> > - 08:00.3 - VF2, ens1f3
> > - ens1f0_0 (no dedicated PCI device)
> > - ens1f0_1 (no dedicated PCI device)
> > - ens1f0_2 (no dedicated PCI device)
> >
> > 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3
> > VFs, 3 representors (no PCI device, pure netdev)
> >
> > With best regards, Slava
> >
> > > -----Original Message-----
> > > From: Jan Viktorin <viktorin@cesnet.cz>
> > > Sent: Monday, March 15, 2021 18:11
> > > To: Slava Ovsiienko <viacheslavo@nvidia.com>
> > > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler
> > > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com>
> > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable
> > > switchdev mode
> > >
> > > Hello Salva,
> > >
> > > On Mon, 15 Mar 2021 15:53:51 +0000
> > > Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> > >
> > > > Hi, Jan
> > > >
> > > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the
> > > > netdevs
> > > from the mlx5_core driver.
> > > > Otherwise, kernel driver refuses to be configured with switchdev
> > > > mode in
> > > [5]. So, [4] can't be skipped.
> > > > After setting swithdev mode, VFs can be bound back (if it is
> > > > needed, and
> > > these ones are not mapped to VMs):
> > >
> > > OK, but I believe that it is **not possible** to follow the rule [5].
> > > The guide explicitly says in [4] "can be rebind **after** the
> > > switchdev mode".
> > > Just, if you unbind the device, there is no way how to configure the
> > > switchdev mode, this is the contradiction I mentioned in the commit.
> > > You cannot configure switchdev mode because the interface is gone
> > > and the path /sys/class/net/<net device>/compat/devlink/mode no
> longer exists.
> > >
> > > So, maybe, just the formulation is wrong. So, what is the **exact
> > > right** way how to do it? I would change the commit accordingly.
> > > Just, let's make it right. Would it work this way?
> > >
> > > # echo -n "<device pci address>" >
> > > /sys/bus/pci/drivers/mlx5_core/unbind
> > > # echo -n "<device pci address>" >
> > > /sys/bus/pci/drivers/mlx5_core/bind
> > > # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
> > >
> > > It is good to mention that after the rebind, the <net device> can change.
> > >
> > > Regards,
> > > Jan
> > >
> > > >
> > > > echo -n "<device pci address>" > >
> > > > /sys/bus/pci/drivers/mlx5_core/bind
> > > >
> > > > With best regards,
> > > > Slava
> > > >
> > > > > -----Original Message-----
> > > > > From: Jan Viktorin <iviktorin@fit.vutbr.cz>
> > > > > Sent: Monday, March 15, 2021 17:34
> > > > > To: dev@dpdk.org
> > > > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso
> > > > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava
> > > > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad
> > > <matan@nvidia.com>
> > > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable
> > > > > switchdev mode
> > > > >
> > > > > From: Jan Viktorin <viktorin@cesnet.cz>
> > > > >
> > > > > The step 4 is a contradiction. It advices to unbind the device
> > > > > from the mlx5_core which removes the associated system network
> > > > > interface
> > > (e.g.
> > > > > eth0). In the step 5, the same system network interface (e.g.
> > > > > eth0) is required to exist.
> > > > >
> > > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
> > > > > ---
> > > > > doc/guides/nics/mlx5.rst | 6 +-----
> > > > > 1 file changed, 1 insertion(+), 5 deletions(-)
> > > > >
> > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
> > > > > index 0a2dc3dee..122d8e0fc 100644
> > > > > --- a/doc/guides/nics/mlx5.rst
> > > > > +++ b/doc/guides/nics/mlx5.rst
> > > > > @@ -1370,11 +1370,7 @@ the DPDK application.
> > > > >
> > > > > echo <num of vfs > /sys/class/net/<net
> > > > > device>/device/sriov_numvfs
> > > > >
> > > > > -4. Unbind the device (can be rebind after the switchdev mode)::
> > > > > -
> > > > > - echo -n "<device pci address>" >
> > > > > /sys/bus/pci/drivers/mlx5_core/unbind
> > > > > -
> > > > > -5. Enable switchdev mode::
> > > > > +4. Enable switchdev mode::
> > > > >
> > > > > echo switchdev > /sys/class/net/<net
> > > > > device>/compat/devlink/mode
> > > > >
> > > > > --
> > > > > 2.30.1
> > > >
> >
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
2021-03-15 19:52 ` Slava Ovsiienko
@ 2021-03-15 20:30 ` Jan Viktorin
0 siblings, 0 replies; 9+ messages in thread
From: Jan Viktorin @ 2021-03-15 20:30 UTC (permalink / raw)
To: Slava Ovsiienko; +Cc: dev, Asaf Penso, Shahaf Shuler, Matan Azrad
On Mon, 15 Mar 2021 19:52:46 +0000
Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> Hi, Jan
>
> > -----Original Message-----
> > From: Jan Viktorin <viktorin@cesnet.cz>
> > Sent: Monday, March 15, 2021 21:49
> > To: Slava Ovsiienko <viacheslavo@nvidia.com>
> > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler
> > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com>
> > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev
> > mode
> >
> > On Mon, 15 Mar 2021 19:31:25 +0000
> > Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> >
> > > Hi, Jan
> >
> > Hello Salva,
> >
> > I am sorry, but I didn't get the point of your response. For me, there was 1 PF
> > 0000:05:00.1, SRIOV disabled. So, I followed the guide [1].
> >
> > 1. Enable SRIOV via mlxconfig. Done.
> >
> > 2. Configure max number of VFs. Done.
> >
> > 3. Reset firmware. Done.
> >
> > In fact, I had to reboot the machine here so I am not convinced the reset of
> > firmware is the thing. But I can image that there is some other use case for it.
> > After reboot, I configured 2 VFs: 0000:05:00.4, 0000:05:00.5.
> >
> > 4. Unbind the PF 0000:05:00.1. Done.
> No, it is the second physical port. You should unbind your VFs instead - 0000:05:00.4, 0000:05:00.5.
>
> Please, see:
> > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched.
OK. Now I understand. Thus, the guide should read "Unbind the VF
devices (can be rebind after the switchdev mode)". I will post v2 with
this change.
Thanks for clarification.
Jan
>
> With best regards,
> Slava
>
> >
> > 5. Fail, no way how to configure switchdev because of previous step.
> > Thus, I strongly believe that the guide is misleading here.
> >
> > When I bound the PF 0000:05:00.1 back, I could configure the switchdev
> > mode and everything was just OK and seemed to be working. So **after**
> > the bind I configured the switchdev mode properly.
> >
> > [1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode
> >
> > Jan
> >
> > >
> > > 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV
> > > enabled, 0 VFs is configured, netdev name is ens1f0 (depends on distro
> > and kernel version).
> > >
> > > 2. Then, we configure 3 VFs with SR-IOV, we get :
> > > - 08:00.0 - PF, ens1f0
> > > - 08:00.1 - VF0, ens1f1
> > > - 08:00.2 - VF1, ens1f2
> > > - 08:00.3 - VF2, ens1f3
> > >
> > > 3. Then, we should unbind VFs ONLY, PF should be kept untouched.
> > > - 08:00.0 - PF, ens1f0
> > > - 08:00.1 - VF0, -
> > > - 08:00.2 - VF1, -
> > > - 08:00.3 - VF2, -
> > >
> > > 4. Then, configure switchdev mode
> > > - 08:00.0 - PF, ens1f0
> > > - 08:00.1 - VF0, -
> > > - 08:00.2 - VF1, -
> > > - 08:00.3 - VF2, -
> > > + we get 3 representor netdevs:
> > > ens1f0_0 (no dedicated PCI device)
> > > ens1f0_1 (no dedicated PCI device)
> > > ens1f0_2 (no dedicated PCI device)
> > >
> > > 5. Then, bind VFs back (assume we are no going to map them to VMs)
> > > - 08:00.0 - PF, ens1f0
> > > - 08:00.1 - VF0, ens1f1
> > > - 08:00.2 - VF1, ens1f2
> > > - 08:00.3 - VF2, ens1f3
> > > - ens1f0_0 (no dedicated PCI device)
> > > - ens1f0_1 (no dedicated PCI device)
> > > - ens1f0_2 (no dedicated PCI device)
> > >
> > > 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3
> > > VFs, 3 representors (no PCI device, pure netdev)
> > >
> > > With best regards, Slava
> > >
> > > > -----Original Message-----
> > > > From: Jan Viktorin <viktorin@cesnet.cz>
> > > > Sent: Monday, March 15, 2021 18:11
> > > > To: Slava Ovsiienko <viacheslavo@nvidia.com>
> > > > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler
> > > > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com>
> > > > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable
> > > > switchdev mode
> > > >
> > > > Hello Salva,
> > > >
> > > > On Mon, 15 Mar 2021 15:53:51 +0000
> > > > Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> > > >
> > > > > Hi, Jan
> > > > >
> > > > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the
> > > > > netdevs
> > > > from the mlx5_core driver.
> > > > > Otherwise, kernel driver refuses to be configured with switchdev
> > > > > mode in
> > > > [5]. So, [4] can't be skipped.
> > > > > After setting swithdev mode, VFs can be bound back (if it is
> > > > > needed, and
> > > > these ones are not mapped to VMs):
> > > >
> > > > OK, but I believe that it is **not possible** to follow the rule [5].
> > > > The guide explicitly says in [4] "can be rebind **after** the
> > > > switchdev mode".
> > > > Just, if you unbind the device, there is no way how to configure the
> > > > switchdev mode, this is the contradiction I mentioned in the commit.
> > > > You cannot configure switchdev mode because the interface is gone
> > > > and the path /sys/class/net/<net device>/compat/devlink/mode no
> > longer exists.
> > > >
> > > > So, maybe, just the formulation is wrong. So, what is the **exact
> > > > right** way how to do it? I would change the commit accordingly.
> > > > Just, let's make it right. Would it work this way?
> > > >
> > > > # echo -n "<device pci address>" >
> > > > /sys/bus/pci/drivers/mlx5_core/unbind
> > > > # echo -n "<device pci address>" >
> > > > /sys/bus/pci/drivers/mlx5_core/bind
> > > > # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
> > > >
> > > > It is good to mention that after the rebind, the <net device> can change.
> > > >
> > > > Regards,
> > > > Jan
> > > >
> > > > >
> > > > > echo -n "<device pci address>" > >
> > > > > /sys/bus/pci/drivers/mlx5_core/bind
> > > > >
> > > > > With best regards,
> > > > > Slava
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jan Viktorin <iviktorin@fit.vutbr.cz>
> > > > > > Sent: Monday, March 15, 2021 17:34
> > > > > > To: dev@dpdk.org
> > > > > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso
> > > > > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava
> > > > > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad
> > > > <matan@nvidia.com>
> > > > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable
> > > > > > switchdev mode
> > > > > >
> > > > > > From: Jan Viktorin <viktorin@cesnet.cz>
> > > > > >
> > > > > > The step 4 is a contradiction. It advices to unbind the device
> > > > > > from the mlx5_core which removes the associated system network
> > > > > > interface
> > > > (e.g.
> > > > > > eth0). In the step 5, the same system network interface (e.g.
> > > > > > eth0) is required to exist.
> > > > > >
> > > > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
> > > > > > ---
> > > > > > doc/guides/nics/mlx5.rst | 6 +-----
> > > > > > 1 file changed, 1 insertion(+), 5 deletions(-)
> > > > > >
> > > > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
> > > > > > index 0a2dc3dee..122d8e0fc 100644
> > > > > > --- a/doc/guides/nics/mlx5.rst
> > > > > > +++ b/doc/guides/nics/mlx5.rst
> > > > > > @@ -1370,11 +1370,7 @@ the DPDK application.
> > > > > >
> > > > > > echo <num of vfs > /sys/class/net/<net
> > > > > > device>/device/sriov_numvfs
> > > > > >
> > > > > > -4. Unbind the device (can be rebind after the switchdev mode)::
> > > > > > -
> > > > > > - echo -n "<device pci address>" >
> > > > > > /sys/bus/pci/drivers/mlx5_core/unbind
> > > > > > -
> > > > > > -5. Enable switchdev mode::
> > > > > > +4. Enable switchdev mode::
> > > > > >
> > > > > > echo switchdev > /sys/class/net/<net
> > > > > > device>/compat/devlink/mode
> > > > > >
> > > > > > --
> > > > > > 2.30.1
> > > > >
> > >
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode
2021-03-15 15:33 ` [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Jan Viktorin
2021-03-15 15:53 ` Slava Ovsiienko
@ 2021-07-15 14:51 ` Raslan Darawsheh
1 sibling, 0 replies; 9+ messages in thread
From: Raslan Darawsheh @ 2021-07-15 14:51 UTC (permalink / raw)
To: Jan Viktorin, dev
Cc: Jan Viktorin, Asaf Penso, Shahaf Shuler, Slava Ovsiienko, Matan Azrad
Hi,
Can you kindly update on the status of this series?
Kindest regards,
Raslan Darawsheh
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Jan Viktorin
> Sent: Monday, March 15, 2021 5:34 PM
> To: dev@dpdk.org
> Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso <asafp@nvidia.com>;
> Shahaf Shuler <shahafs@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com>
> Subject: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable
> switchdev mode
>
> From: Jan Viktorin <viktorin@cesnet.cz>
>
> The step 4 is a contradiction. It advices to unbind the device from the
> mlx5_core which removes the associated system network interface (e.g.
> eth0). In the step 5, the same system network interface (e.g. eth0) is
> required to exist.
>
> Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
> ---
> doc/guides/nics/mlx5.rst | 6 +-----
> 1 file changed, 1 insertion(+), 5 deletions(-)
>
> diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index
> 0a2dc3dee..122d8e0fc 100644
> --- a/doc/guides/nics/mlx5.rst
> +++ b/doc/guides/nics/mlx5.rst
> @@ -1370,11 +1370,7 @@ the DPDK application.
>
> echo <num of vfs > /sys/class/net/<net device>/device/sriov_numvfs
>
> -4. Unbind the device (can be rebind after the switchdev mode)::
> -
> - echo -n "<device pci address>" >
> /sys/bus/pci/drivers/mlx5_core/unbind
> -
> -5. Enable switchdev mode::
> +4. Enable switchdev mode::
>
> echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
>
> --
> 2.30.1
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-07-15 14:51 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-15 15:33 [dpdk-dev] [PATCH 1/2] net/mlx5: fix typos Jan Viktorin
2021-03-15 15:33 ` [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev mode Jan Viktorin
2021-03-15 15:53 ` Slava Ovsiienko
2021-03-15 16:11 ` Jan Viktorin
2021-03-15 19:31 ` Slava Ovsiienko
2021-03-15 19:48 ` Jan Viktorin
2021-03-15 19:52 ` Slava Ovsiienko
2021-03-15 20:30 ` Jan Viktorin
2021-07-15 14:51 ` Raslan Darawsheh
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).