From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <stephen@networkplumber.org>
Received: from mail-pl0-f41.google.com (mail-pl0-f41.google.com
 [209.85.160.41]) by dpdk.org (Postfix) with ESMTP id 3B88B7D19
 for <users@dpdk.org>; Wed, 13 Dec 2017 18:09:55 +0100 (CET)
Received: by mail-pl0-f41.google.com with SMTP id n13so1356919plp.11
 for <users@dpdk.org>; Wed, 13 Dec 2017 09:09:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=networkplumber-org.20150623.gappssmtp.com; s=20150623;
 h=date:from:to:cc:subject:message-id:in-reply-to:references
 :mime-version:content-transfer-encoding;
 bh=E3YSBTPyPNglTW4w2Qi4SYrkgQFBTbxEvAbrIv1z638=;
 b=Lh1UuqXF0aL1kdTlDOU4QJ1xYUTedXcAFq5J580CJDB8ga3Zflht4JCqopRKwd6huK
 R5KiKNc7yKuMqy4LqHh/5sH+jcbpGzfypVWAXLEEPTZogL9UtLejm4MvkD8zFxS4kK/j
 XM4XpwdsckgTAg+GsgYPfbno1Xd2SMV8cEbHBAd1PwLrLG7agzv+6THQzdN95ZqcIF+L
 vcVaKdK6+C2/RWH+Izbca7TVbuWxKNt5ZykJ7BQucaL33EFChIbXZNFQZTt2gCQaTIpE
 QIqPAsJlwc83AaAJfhBwS88UmXmUf308UggjeNhqindRRFUmQd7G+wZOQSK5JgxbPVux
 MG9w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to
 :references:mime-version:content-transfer-encoding;
 bh=E3YSBTPyPNglTW4w2Qi4SYrkgQFBTbxEvAbrIv1z638=;
 b=b0U3sW47S15PedgKsrdhkuCawFLUKgqq6D2Ibl+OoWcH1Mfq/ggUQ0xyBX7Jc9icfF
 SYzwbwx/HTbabeUz+EwMzbET8xrRzWmCrqQ/zam2DQvKjJ+ElE4+n8HMXTbjZT//oPAh
 CtCb9cnCKd8Wp40hU07UCB/JjcweAedhwSxzGU3DNZcCcOqi7TPRWbwim2etyNYlQgry
 HWGu26HLmkNfjOxI6rHIYoN+yy4ZrHE3vL8MmYcNLuQSOZS9MIBBArA44OJN4e3U2rff
 ULXDpbuO0RM5Wcl168lIus6sQiSJyAuzsZ20Ziw2AR14SaOf1NFp3fGt6hNbHnnUU1Va
 bquQ==
X-Gm-Message-State: AKGB3mLohSaDRgdP0AiEaOcqPcFsqVnhmf/fcPsbwWfytnSHyQ78aA6Y
 oXpLzUhkm/eujA6skgyrif4HjiPUvnA=
X-Google-Smtp-Source: ACJfBovcgoI2dAM/FNq1D2HTaowFS6vbfRclayIfk+tiTUWwSbNlzPqKXS/8SItMecvol1x8mzxQtQ==
X-Received: by 10.84.132.2 with SMTP id 2mr1932921ple.360.1513184994854;
 Wed, 13 Dec 2017 09:09:54 -0800 (PST)
Received: from xeon-e3 (76-14-207-240.or.wavecable.com. [76.14.207.240])
 by smtp.gmail.com with ESMTPSA id j13sm3940623pff.131.2017.12.13.09.09.54
 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
 Wed, 13 Dec 2017 09:09:54 -0800 (PST)
Date: Wed, 13 Dec 2017 09:09:52 -0800
From: Stephen Hemminger <stephen@networkplumber.org>
To: Ricardo Roldan <rroldan@bequant.com>
Cc: users@dpdk.org
Message-ID: <20171213090952.495bfb1b@xeon-e3>
In-Reply-To: <7a3a2174-831f-caa8-ed33-0f06133c96a2@bequant.com>
References: <fb117eef-ddad-67f6-8e0f-6febdbc0aeed@bequant.com>
 <7a3a2174-831f-caa8-ed33-0f06133c96a2@bequant.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Re: [dpdk-users] attach/detach on secondary process
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/users>,
 <mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/users>,
 <mailto:users-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Dec 2017 17:09:56 -0000

On Wed, 13 Dec 2017 17:58:53 +0100
Ricardo Roldan <rroldan@bequant.com> wrote:

> Hi,
>=20
> We have a multi-process application and we need to support=20
> attaching/detaching of ports. We are using the 17.11 version with the=20
> Intel x520 (ixgbe driver) and virtio.
>=20
> At the time we initialize our processes there are not any devices binded=
=20
> with the DPDK drivers, so we initialize all processes (primary and=20
> secondaries) with 0 ports.
>=20
> This seems to work fine only on the primary processes, but on the=20
> secondary processes we see some problems. In the following paragraphs I=20
> describe the procedure used to attach/detach interfaces with DPDK.
>=20
> For the attach procedure (all processes initially have no devices=20
> attached):
>=20
> - Bind the devices we want to attach to the DPDK driver (with the script=
=20
> dpdk-devbind, from external process)
>=20
> - Primary process: Call rte_eth_dev_attach
>=20
> - Primary process: Configure ports using ...
>=20
> - Secondary processes: Call to rte_eth_dev_attach
>=20
>=20
> Start to send/receive packets from all processes.
>=20
>=20
> For the detach procedure:
>=20
> - Secondary processes: For each port, call rte_eth_dev_stop(port),=20
> rte_eth_dev_close(port) and rte_eth_dev_detach(port, dev).
>=20
> - Primary process: After the secondary processes have detach all their=20
> ports, for each port call rte_eth_dev_stop(port),=20
> rte_eth_dev_close(port) and rte_eth_dev_detach(port, dev).
>=20
> - Bind the device to the original Linux driver (with the script=20
> dpdk-devbind, from external process)
>=20
>=20
> With this approach we have noticed that when the secondary processes=20
> call rte_dev_detach there is an error, because it calls the remove=20
> operation, which ends up calling eth_ixgbe_dev_uninit that returns=20
> -EPERM (because it does not allow a non primary process to uninitialize=20
> the driver).
>=20
>=20
> Therefore, the port attach never works again on the secondary processes=20
> as the function rte_eal_hotplug_add fails because it cannot find the=20
> device.
>=20
>=20
> dev =3D bus->find_device(NULL, cmp_detached_dev_name, devname);
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (dev =3D=3D NULL) {
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 RTE_LOG(ERR, EAL, "Cannot find unplugged device (%s)\n",
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 devname);
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 ret =3D -ENODEV;
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 goto err_devarg;
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }
>=20
>=20
> This happens because in order to check unplugged devices, the function=20
> cmp_detached_dev_name=C2=A0 checks if there is a pointer to the driver an=
d=20
> fails if it is already set, because the detach procedure never set the=20
> driver variable to NULL.
>=20
> static int cmp_detached_dev_name(const struct rte_device *dev,
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 const void *_name)
> {
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 const char *name =3D _name;
>=20
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* skip attached devices */
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 RTE_LOG(ERR, EAL, "cmp_detach=
ed_dev_name dev %p name %s driver %p"
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 " search %s\n",
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dev, dev->name, dev->driver, =
name);
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (dev->driver !=3D NULL)
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 return 1;
>=20
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return strcmp(dev->name, name=
);
> }
>=20
> To fix this behavior we have done the following changes on the DPDK code.
>=20
> First, in order to prevent cmp_detached_dev_name from failing,=20
> rte_eal_dev_detach sets driver to NULL.
>=20
> diff --git a/lib/librte_eal/common/eal_common_dev.c=20
> b/lib/librte_eal/common/eal_common_dev.c
>=20
> index dda8f5835..9a363dcf7 100644
> --- a/lib/librte_eal/common/eal_common_dev.c
> +++ b/lib/librte_eal/common/eal_common_dev.c
> @@ -114,6 +114,7 @@ int rte_eal_dev_detach(struct rte_device *dev)
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (ret)
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 RTE_LOG(ERR, EAL, "Driver cannot detach the device (%s)\=
n",
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dev->nam=
e);
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dev->driver =3D NULL;
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return ret;
>  =C2=A0}/*
> */
>=20
>=20
> Then, in the rte_eth_dev_pci_generic_remove function, the call to=20
> dev_uninit does not consider -EPERM an error, because when detaching a=20
> port, some drivers return 0 and other drivers return -EPERM to indicate=20
> that it is called from a secondary process.
>=20
> diff --git a/lib/librte_ether/rte_ethdev_pci.h=20
> b/lib/librte_ether/rte_ethdev_pci.h
> @@ -184,7 +184,7 @@ rte_eth_dev_pci_generic_remove(struct rte_pci_device=
=20
> *pci_dev,
>=20
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (dev_uninit) {
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 ret =3D dev_uninit(eth_dev);
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 if (ret)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 if (ret && ret !=3D -EPERM)
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return r=
et;
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }
>=20
> Finally, in the rte_eth_dev_pci_release function, only the fields in the=
=20
> shared memory region are reset if called from a primary process.
>=20
> /**/diff --git a/lib/librte_ether/rte_ethdev_pci.h=20
> b/lib/librte_ether/rte_ethdev_pci.h
> index 722075e09..a79188fbf 100644
> --- a/lib/librte_ether/rte_ethdev_pci.h
> +++ b/lib/librte_ether/rte_ethdev_pci.h
> @@ -125,16 +125,16 @@ rte_eth_dev_pci_release(struct rte_eth_dev *eth_dev)
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* free ether device */
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rte_eth_dev_release_port(eth_=
dev);
>=20
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (rte_eal_process_type() =3D=3D R=
TE_PROC_PRIMARY)
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (rte_eal_process_type() =3D=3D R=
TE_PROC_PRIMARY) {
> rte_free(eth_dev->data->dev_private);
> + eth_dev->data->dev_private =3D NULL;
>=20
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth_dev->data->dev_private =3D NULL;
> -
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /*
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Secondary process will chec=
k the name to attach.
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Clear this field to avoid a=
ttaching a released ports.
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */
> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth_dev->data->name[0] =3D '\0';
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 /*
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 * Secondary process will check the name to attach.
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 * Clear this field to avoid attaching a released ports.
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 */
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 eth_dev->data->name[0] =3D '\0';
> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }
>=20
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth_dev->device =3D NULL;
>  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth_dev->intr_handle =3D NULL;
>=20
>=20
> After applying these changes, it seems like attaching and detaching=20
> ports multiple times works without problems, at least with the ixgbe and=
=20
> virtio drivers.
>=20
> In order to generate a patch, It would be very helpful if anyone can=20
> confirm that this approach is correct and that we do not break any other=
=20
> parts.
>=20
> Best regards,
> Ricardo
>=20
>=20
> On 12/13/2017 03:31 PM, Ricardo Roldan wrote:
> >
> >
> > Hi,
> >
> > We have a multi-process application and we need to support=20
> > attaching/detaching of ports.
> >
> > At the time we initialize our process they aren't any devices binded=20
> > with the dpdk drivers, so we initialize in all processes with 0 ports.
> >
> > We manage to work fine only on the primary processes, on the secondary=
=20
> > process we had some problems. Here is the way call the dpdk
> > interface to attach/detach from our process
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Attach - At this moment all process (pri=
mary and secondaries)=20
> > has not devices attached
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - Bind the devices we want to attach to =
the DPDK driver (with=20
> > the script dpdk-devbind, from external process)
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - Call rte_eth_dev_attach from primary p=
rocess
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - Configure ports from primary process
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - From secondary process call to rte_eth=
_dev_attach
> >
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0 Began to send/receive packets from all process
> >
> >
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0 Detach
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - From secondary process call to:
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 rte_eth_dev_stop(port);
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 rte_eth_dev_close(port);
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 rte_eth_dev_detach(port, dev);
> >
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - From primary process call to:
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 rte_eth_dev_stop(port);
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 rte_eth_dev_close(port);
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 rte_eth_dev_detach(port, dev);
> >
> >
> > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - Bind the device to the original =
Linux driver (with the script=20
> > dpdk-devbind, from external process)
> >
> >
> > =C2=A0=C2=A0 With this second approach we notice that the detach from t=
he=20
> > secondary process returns with an error. This is because the function=20
> > eth_ixgbe_dev_uninit has a check to not uninitialize the driver from a=
=20
> > not primary process.
> >
> > =C2=A0=C2=A0 So that the detach can't finish all its task.
> >
> > =C2=A0=C2=A0 After this, the attach never works again on the secondary =
process=20
> > as the function rte_eal_hotplug_add began to fail due that it cant not=
=20
> > find the device.
> >
> > */=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dev =3D bus->find_device(N=
ULL, cmp_detached_dev_name,=20
> > devname); /**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (dev =3D=3D NULL) { /=
**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 RTE_LOG(ERR, EAL, "Cannot find unplugged device=20
> > (%s)\n", /**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 de=
vname); /**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 ret =3D -ENODEV; /**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 goto err_devarg; /**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } /**/
> > /**//*
> > =C2=A0=C2=A0 This is due that to check unplugged devices, what the comp=
aration=20
> > function cmp_detached_dev_name do, is to check if there is a pointer=20
> > to the driver.
> >
> > */static int cmp_detached_dev_name(const struct rte_device *dev, /**/
> > /**/=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=
=A0 const void *_name) /**/
> > /**/{ /**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 const char *name =3D _na=
me; /**/
> > /**//**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* skip attached devices=
 */ /**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (dev->driver !=3D NUL=
L) /**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 return 1; /**/
> > /**//**/
> > /**/=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return strcmp(dev->name,=
 name); /**/
> > /**/} /*/
> > /
> >
> > And as detach has not finish correctly on the secondary process, the=20
> > device continues with the pointer to the driver setted. So the attach=20
> > fails as the device is not found.
> >
> >
> > To overcome this behavior we had done this changes on the DPDK code:
> >
> > We have modify the dpdk to clean the pointer to the driver on the=20
> > detach.We had modify also the function rte_eth_dev_pci_generic_remove=20
> > so even if the uninit
> > of the driver return with -EPERM the function continue executing the=20
> > rest of the code. We had done this as we had seen that the check on=20
> > the uninit testing to
> > see if the process is not a primary=C2=A0 is donein all drivers, but so=
me=20
> > drivers return with no error ( 0) and others with (-EPERM). So on=20
> > rte_eth_dev_pci_generic if the call
> > to uninit returns with -EPERM we continue executing calling=20
> > rte_eth_dev_pci_release. To that last function we had also done some=20
> > changes, as only primary process
> > should be able to uninitialized some common values, that a detach on a=
=20
> > secondary process should never do.
> >
> > These are the changes:
> >
> > /*diff --git a/lib/librte_eal/common/eal_common_dev.c=20
> > b/lib/librte_eal/common/eal_common_dev.c*//*
> > *//*index dda8f5835..9a363dcf7 100644*//*
> > *//*--- a/lib/librte_eal/common/eal_common_dev.c*//*
> > *//*+++ b/lib/librte_eal/common/eal_common_dev.c*//*
> > *//*@@ -114,6 +114,7 @@ int rte_eal_dev_detach(struct rte_device *dev)*=
//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (ret)*//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 RTE_LOG(ERR, EAL, "Driver cannot detach the device=
=20
> > (%s)\n",*//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 de=
v->name);*//*
> > *//*+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dev->driver =3D NULL;*//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return ret;*//*
> > *//*=C2=A0}*//*
> > *//**//*
> > *//*diff --git a/lib/librte_ether/rte_ethdev_pci.h=20
> > b/lib/librte_ether/rte_ethdev_pci.h*//*
> > *//*index 722075e09..a79188fbf 100644*//*
> > *//*--- a/lib/librte_ether/rte_ethdev_pci.h*//*
> > *//*+++ b/lib/librte_ether/rte_ethdev_pci.h*//*
> > *//*@@ -125,16 +125,16 @@ rte_eth_dev_pci_release(struct rte_eth_dev=20
> > *eth_dev)*//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* free ether device */*=
//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rte_eth_dev_release_port=
(eth_dev);*//*
> > *//**//*
> > *//*-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (rte_eal_process_type() =
=3D=3D RTE_PROC_PRIMARY)*//*
> > *//*+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (rte_eal_process_type() =
=3D=3D RTE_PROC_PRIMARY) {*//*
> > *//*rte_free(eth_dev->data->dev_private);*//*
> > *//*+ eth_dev->data->dev_private =3D NULL;*//*
> > *//**//*
> > *//*-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth_dev->data->dev_private =
=3D NULL;*//*
> > *//*-*//*
> > *//*-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /**//*
> > *//*-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Secondary process wil=
l check the name to attach.*//*
> > *//*-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Clear this field to a=
void attaching a released ports.*//*
> > *//*-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */*//*
> > *//*-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth_dev->data->name[0] =3D '\=
0';*//*
> > *//*+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 /**//*
> > *//*+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 * Secondary process will check the name to=20
> > attach.*//*
> > *//*+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 * Clear this field to avoid attaching a released=20
> > ports.*//*
> > *//*+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 */*//*
> > *//*+ eth_dev->data->name[0] =3D '\0';*//*
> > *//*+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }*//*
> > *//**//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth_dev->device =3D NULL=
;*//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eth_dev->intr_handle =3D=
 NULL;*//*
> > *//*@@ -184,7 +184,7 @@ rte_eth_dev_pci_generic_remove(struct=20
> > rte_pci_device *pci_dev,*//*
> > *//**//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (dev_uninit) {*//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 ret =3D dev_uninit(eth_dev);*//*
> > *//*-=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 if (ret)*//*
> > *//*+=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 if (ret && ret !=3D -EPERM)*//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 re=
turn ret;*//*
> > *//*=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }*//*
> > *//**//*
> > */
> >
> >
> > And now seems to work. Is this a correct way to proceed?
> >
> > Could some one that have work with this functionalities can advise us?
> >
> > Best regards,
> >
> > Ricardo
> >
> > =20
>=20

Many DPDK drivers require that setup and initialization be done by
the primary process. This is mostly to avoid dealing with concurrency since
there can be multiple secondary processes.