From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 6BF23A054F;
	Mon, 15 Mar 2021 20:49:00 +0100 (CET)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 5691F2427C3;
	Mon, 15 Mar 2021 20:49:00 +0100 (CET)
Received: from office2.cesnet.cz (office2.cesnet.cz [195.113.144.244])
 by mails.dpdk.org (Postfix) with ESMTP id 6D0D74069F
 for <dev@dpdk.org>; Mon, 15 Mar 2021 20:48:58 +0100 (CET)
Received: from tanguero.localdomain (unknown [95.82.133.55])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest
 SHA256) (No client certificate requested)
 by office2.cesnet.cz (Postfix) with ESMTPSA id 2ECAF400052;
 Mon, 15 Mar 2021 20:48:58 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cesnet.cz;
 s=office2-2020; t=1615837738;
 bh=ygwYPTMAN80PwGFQZxcH0gsGkPXEQfzjsitC8pu8s3k=;
 h=Date:From:To:Cc:Subject:In-Reply-To:References;
 b=CktUrZUAhlmb89/1EQR6QRaqwRTVz9uAJTjRRd39XgTwINXflOrHJTkeuaRG8VQJB
 28YujVQubchjewB0MYlVMJ5xJbrOFrQDve/U5w3VRCcorycLjp9Lbp2dJvj/eUStAP
 ZZT8tnsKTRK7iZBwFwiVj082rLCALoIlsZs7cf/jPNnjGYFmVHCaMMsgD2kbPmoBTN
 iRJyZhIJgZfX5qJhl3P3VpunqDQ2hnu3inGh5eavrhih3r+25rfswcoMPe7P8bdgwC
 Sq07obg90PakggKQB6x/igGcrBZGeETyaS31NC2wSz42q6mTiIpchGxTYmrGld4kHh
 NwRIXrouprpVA==
Date: Mon, 15 Mar 2021 20:48:56 +0100
From: Jan Viktorin <viktorin@cesnet.cz>
To: Slava Ovsiienko <viacheslavo@nvidia.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>, Asaf Penso <asafp@nvidia.com>, Shahaf
 Shuler <shahafs@nvidia.com>, Matan Azrad <matan@nvidia.com>
Message-ID: <20210315204856.615637a5@tanguero.localdomain>
In-Reply-To: <DM6PR12MB3753ECA40CD54808E32EBF98DF6C9@DM6PR12MB3753.namprd12.prod.outlook.com>
References: <1615822438-10082-1-git-send-email-iviktorin@fit.vutbr.cz>
 <1615822438-10082-2-git-send-email-iviktorin@fit.vutbr.cz>
 <DM6PR12MB3753C3CAF7C3840D296DEA19DF6C9@DM6PR12MB3753.namprd12.prod.outlook.com>
 <20210315171039.16b3965e@tanguero.localdomain>
 <DM6PR12MB3753ECA40CD54808E32EBF98DF6C9@DM6PR12MB3753.namprd12.prod.outlook.com>
Organization: CESNET
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Subject: Re: [dpdk-dev] [PATCH 2/2] net/mlx5: avoid unbind step to enable
 switchdev mode
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>

On Mon, 15 Mar 2021 19:31:25 +0000
Slava Ovsiienko <viacheslavo@nvidia.com> wrote:

> Hi, Jan

Hello Salva,

I am sorry, but I didn't get the point of your response. For me, there
was 1 PF 0000:05:00.1, SRIOV disabled. So, I followed the guide [1].

1. Enable SRIOV via mlxconfig. Done.

2. Configure max number of VFs. Done.

3. Reset firmware. Done.

In fact, I had to reboot the machine here so I am not convinced the
reset of firmware is the thing. But I can image that there is some
other use case for it. After reboot, I configured 2 VFs: 0000:05:00.4,
0000:05:00.5.

4. Unbind the PF 0000:05:00.1. Done.

5. Fail, no way how to configure switchdev because of previous step.
Thus, I strongly believe that the guide is misleading here.

When I bound the PF 0000:05:00.1 back, I could configure the switchdev
mode and everything was just OK and seemed to be working. So **after**
the bind I configured the switchdev mode properly.

[1] https://doc.dpdk.org/guides/nics/mlx5.html#enable-switchdev-mode

Jan

> 
> 1. Let's suppose we have the one port NIC, PCI BDF 08:00.0, SR-IOV enabled, 0 VFs is configured,
> netdev name is ens1f0 (depends on distro and kernel version). 
> 
> 2. Then, we configure 3 VFs with SR-IOV, we get :
> - 08:00.0 - PF, ens1f0
> - 08:00.1 - VF0, ens1f1
> - 08:00.2 - VF1, ens1f2
> - 08:00.3 - VF2, ens1f3
> 
> 3. Then, we should unbind VFs ONLY, PF should be kept untouched.
> - 08:00.0 - PF, ens1f0
> - 08:00.1 - VF0, -
> - 08:00.2 - VF1, -
> - 08:00.3 - VF2, -
> 
> 4. Then, configure switchdev mode
> - 08:00.0 - PF, ens1f0
> - 08:00.1 - VF0, -
> - 08:00.2 - VF1, -
> - 08:00.3 - VF2, -
> + we get 3 representor netdevs:
> ens1f0_0 (no dedicated PCI device)
> ens1f0_1 (no dedicated PCI device)
> ens1f0_2 (no dedicated PCI device)
> 
> 5. Then, bind VFs back (assume we are no going to map them to VMs)
> - 08:00.0 - PF, ens1f0
> - 08:00.1 - VF0, ens1f1
> - 08:00.2 - VF1, ens1f2
> - 08:00.3 - VF2, ens1f3
> - ens1f0_0 (no dedicated PCI device)
> - ens1f0_1 (no dedicated PCI device)
> - ens1f0_2 (no dedicated PCI device)
> 
> 6. As result we should get 7 netdevs - 1 PF (Uplink representor), 3 VFs, 3 representors (no PCI device, pure netdev)
> 
> With best regards, Slava
> 
> > -----Original Message-----
> > From: Jan Viktorin <viktorin@cesnet.cz>
> > Sent: Monday, March 15, 2021 18:11
> > To: Slava Ovsiienko <viacheslavo@nvidia.com>
> > Cc: dev@dpdk.org; Asaf Penso <asafp@nvidia.com>; Shahaf Shuler
> > <shahafs@nvidia.com>; Matan Azrad <matan@nvidia.com>
> > Subject: Re: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev
> > mode
> > 
> > Hello Salva,
> > 
> > On Mon, 15 Mar 2021 15:53:51 +0000
> > Slava Ovsiienko <viacheslavo@nvidia.com> wrote:
> >   
> > > Hi, Jan
> > >
> > > Yes, bullet [4] explicitly requires to unbind VFs, and detach the netdevs  
> > from the mlx5_core driver.  
> > > Otherwise, kernel driver refuses to be configured with switchdev mode in  
> > [5]. So, [4] can't be skipped.  
> > > After setting swithdev mode, VFs can be bound back (if it is needed, and  
> > these ones are not mapped to VMs):
> > 
> > OK, but I believe that it is **not possible** to follow the rule [5].
> > The guide explicitly says in [4] "can be rebind **after** the switchdev
> > mode".
> > Just, if you unbind the device, there is no way how to configure the
> > switchdev mode, this is the contradiction I mentioned in the commit. You
> > cannot configure switchdev mode because the interface is gone and the path
> > /sys/class/net/<net device>/compat/devlink/mode no longer exists.
> > 
> > So, maybe, just the formulation is wrong. So, what is the **exact
> > right** way how to do it? I would change the commit accordingly. Just, let's
> > make it right. Would it work this way?
> > 
> >  # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/unbind
> >  # echo -n "<device pci address>" > /sys/bus/pci/drivers/mlx5_core/bind
> >  # echo switchdev > /sys/class/net/<net device>/compat/devlink/mode
> > 
> > It is good to mention that after the rebind, the <net device> can change.
> > 
> > Regards,
> > Jan
> >   
> > >
> > > echo -n "<device pci address>" > > /sys/bus/pci/drivers/mlx5_core/bind
> > >
> > > With best regards,
> > > Slava
> > >  
> > > > -----Original Message-----
> > > > From: Jan Viktorin <iviktorin@fit.vutbr.cz>
> > > > Sent: Monday, March 15, 2021 17:34
> > > > To: dev@dpdk.org
> > > > Cc: Jan Viktorin <viktorin@cesnet.cz>; Asaf Penso
> > > > <asafp@nvidia.com>; Shahaf Shuler <shahafs@nvidia.com>; Slava
> > > > Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad  
> > <matan@nvidia.com>  
> > > > Subject: [PATCH 2/2] net/mlx5: avoid unbind step to enable switchdev
> > > > mode
> > > >
> > > > From: Jan Viktorin <viktorin@cesnet.cz>
> > > >
> > > > The step 4 is a contradiction. It advices to unbind the device from
> > > > the mlx5_core which removes the associated system network interface  
> > (e.g.  
> > > > eth0). In the step 5, the same system network interface (e.g. eth0)
> > > > is required to exist.
> > > >
> > > > Signed-off-by: Jan Viktorin <viktorin@cesnet.cz>
> > > > ---
> > > >  doc/guides/nics/mlx5.rst | 6 +-----
> > > >  1 file changed, 1 insertion(+), 5 deletions(-)
> > > >
> > > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
> > > > index 0a2dc3dee..122d8e0fc 100644
> > > > --- a/doc/guides/nics/mlx5.rst
> > > > +++ b/doc/guides/nics/mlx5.rst
> > > > @@ -1370,11 +1370,7 @@ the DPDK application.
> > > >
> > > >          echo <num of vfs > /sys/class/net/<net  
> > > > device>/device/sriov_numvfs  
> > > >
> > > > -4. Unbind the device (can be rebind after the switchdev mode)::
> > > > -
> > > > -        echo -n "<device pci address>" >
> > > > /sys/bus/pci/drivers/mlx5_core/unbind
> > > > -
> > > > -5. Enable switchdev mode::
> > > > +4. Enable switchdev mode::
> > > >
> > > >          echo switchdev > /sys/class/net/<net  
> > > > device>/compat/devlink/mode  
> > > >
> > > > --
> > > > 2.30.1  
> > >  
>