[dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

[dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

[vattunuru]

From: Vamsi Attunuru <vattunuru@marvell.com>

Current pci bus driver scans pci devices in the order that
it read from sysfs. Accordingly all or whitelisted devices
are getting probed.

Patch modifies the probing order of whitelisted pci devices
in a sequence the devices are whitelisted(using EAL flags).

It ensures the eth devices that application uses are probed
in device whitelisted sequence, in turn it facilitates the
packet forwarding applications to work without any packet
loss or performance drop when the underneath network ports
have different bandwidths. By altering the whitelist order
applications like testpmd, l2fwd can forward the ingress
traffic to egress port that has of equivalent bandwidth.

Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
---
 drivers/bus/pci/pci_common.c | 67 ++++++++++++++++++++++++++++++--------------
 1 file changed, 46 insertions(+), 21 deletions(-)

diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
index 6b46b4f..c27a0e9 100644
--- a/drivers/bus/pci/pci_common.c
+++ b/drivers/bus/pci/pci_common.c
@@ -293,32 +293,57 @@ rte_pci_probe(void)
 	struct rte_pci_device *dev = NULL;
 	size_t probed = 0, failed = 0;
 	struct rte_devargs *devargs;
-	int probe_all = 0;
+	struct rte_pci_addr addr;
 	int ret = 0;
 
-	if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST)
-		probe_all = 1;
-
-	FOREACH_DEVICE_ON_PCIBUS(dev) {
-		probed++;
+	if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST) {
+		/* Probe all devices */
+		FOREACH_DEVICE_ON_PCIBUS(dev) {
+			probed++;
 
-		devargs = dev->device.devargs;
-		/* probe all or only whitelisted devices */
-		if (probe_all)
 			ret = pci_probe_all_drivers(dev);
-		else if (devargs != NULL &&
-			devargs->policy == RTE_DEV_WHITELISTED)
-			ret = pci_probe_all_drivers(dev);
-		if (ret < 0) {
-			if (ret != -EEXIST) {
-				RTE_LOG(ERR, EAL, "Requested device "
-					PCI_PRI_FMT " cannot be used\n",
-					dev->addr.domain, dev->addr.bus,
-					dev->addr.devid, dev->addr.function);
-				rte_errno = errno;
-				failed++;
+			if (ret < 0) {
+				if (ret != -EEXIST) {
+					RTE_LOG(ERR, EAL, "Requested device "
+						PCI_PRI_FMT " cannot be used\n",
+						dev->addr.domain, dev->addr.bus,
+						dev->addr.devid,
+						dev->addr.function);
+					rte_errno = errno;
+					failed++;
+				}
+				ret = 0;
+			}
+		}
+	} else {
+		/* Probe only whitelisted devices */
+		RTE_EAL_DEVARGS_FOREACH("pci", devargs) {
+			if (devargs->policy != RTE_DEV_WHITELISTED)
+				continue;
+
+			devargs->bus->parse(devargs->name, &addr);
+
+			FOREACH_DEVICE_ON_PCIBUS(dev) {
+				if (rte_pci_addr_cmp(&dev->addr, &addr))
+					continue;
+				probed++;
+				ret = pci_probe_all_drivers(dev);
+				if (ret < 0) {
+					if (ret != -EEXIST) {
+						RTE_LOG(ERR, EAL,
+							"Requested device "
+							 PCI_PRI_FMT
+							"cannot be used\n",
+							dev->addr.domain,
+							dev->addr.bus,
+							dev->addr.devid,
+							dev->addr.function);
+						rte_errno = errno;
+						failed++;
+					}
+					ret = 0;
+				}
 			}
-			ret = 0;
 		}
 	}
 
-- 
2.8.4

Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

[Slava Ovsiienko]

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of vattunuru@marvell.com
> Sent: Monday, September 23, 2019 14:57
> To: dev@dpdk.org
> Cc: gaetan.rivet@6wind.com; ferruh.yigit@intel.com;
> anatoly.burakov@intel.com; Thomas Monjalon <thomas@monjalon.net>;
> jerinj@marvell.com; Vamsi Attunuru <vattunuru@marvell.com>
> Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted
> order
> 
> From: Vamsi Attunuru <vattunuru@marvell.com>
> 
> Current pci bus driver scans pci devices in the order that it read from sysfs.
> Accordingly all or whitelisted devices are getting probed.
> 
> Patch modifies the probing order of whitelisted pci devices in a sequence the
> devices are whitelisted(using EAL flags).

Thanks, it would be nice to have opportunity to control probing order,
it might be useful for bonded devices and representors either.

Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>

> 
> It ensures the eth devices that application uses are probed in device
> whitelisted sequence, in turn it facilitates the packet forwarding applications
> to work without any packet loss or performance drop when the underneath
> network ports have different bandwidths. By altering the whitelist order
> applications like testpmd, l2fwd can forward the ingress traffic to egress port
> that has of equivalent bandwidth.
> 
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
>  drivers/bus/pci/pci_common.c | 67 ++++++++++++++++++++++++++++++----
> ----------
>  1 file changed, 46 insertions(+), 21 deletions(-)
> 
> diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c
> index 6b46b4f..c27a0e9 100644
> --- a/drivers/bus/pci/pci_common.c
> +++ b/drivers/bus/pci/pci_common.c
> @@ -293,32 +293,57 @@ rte_pci_probe(void)
>  	struct rte_pci_device *dev = NULL;
>  	size_t probed = 0, failed = 0;
>  	struct rte_devargs *devargs;
> -	int probe_all = 0;
> +	struct rte_pci_addr addr;
>  	int ret = 0;
> 
> -	if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST)
> -		probe_all = 1;
> -
> -	FOREACH_DEVICE_ON_PCIBUS(dev) {
> -		probed++;
> +	if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST) {
> +		/* Probe all devices */
> +		FOREACH_DEVICE_ON_PCIBUS(dev) {
> +			probed++;
> 
> -		devargs = dev->device.devargs;
> -		/* probe all or only whitelisted devices */
> -		if (probe_all)
>  			ret = pci_probe_all_drivers(dev);
> -		else if (devargs != NULL &&
> -			devargs->policy == RTE_DEV_WHITELISTED)
> -			ret = pci_probe_all_drivers(dev);
> -		if (ret < 0) {
> -			if (ret != -EEXIST) {
> -				RTE_LOG(ERR, EAL, "Requested device "
> -					PCI_PRI_FMT " cannot be used\n",
> -					dev->addr.domain, dev->addr.bus,
> -					dev->addr.devid, dev-
> >addr.function);
> -				rte_errno = errno;
> -				failed++;
> +			if (ret < 0) {
> +				if (ret != -EEXIST) {
> +					RTE_LOG(ERR, EAL, "Requested
> device "
> +						PCI_PRI_FMT " cannot be
> used\n",
> +						dev->addr.domain, dev-
> >addr.bus,
> +						dev->addr.devid,
> +						dev->addr.function);
> +					rte_errno = errno;
> +					failed++;
> +				}
> +				ret = 0;
> +			}
> +		}
> +	} else {
> +		/* Probe only whitelisted devices */
> +		RTE_EAL_DEVARGS_FOREACH("pci", devargs) {
> +			if (devargs->policy != RTE_DEV_WHITELISTED)
> +				continue;
> +
> +			devargs->bus->parse(devargs->name, &addr);
> +
> +			FOREACH_DEVICE_ON_PCIBUS(dev) {
> +				if (rte_pci_addr_cmp(&dev->addr, &addr))
> +					continue;
> +				probed++;
> +				ret = pci_probe_all_drivers(dev);
> +				if (ret < 0) {
> +					if (ret != -EEXIST) {
> +						RTE_LOG(ERR, EAL,
> +							"Requested device "
> +							 PCI_PRI_FMT
> +							"cannot be used\n",
> +							dev->addr.domain,
> +							dev->addr.bus,
> +							dev->addr.devid,
> +							dev->addr.function);
> +						rte_errno = errno;
> +						failed++;
> +					}
> +					ret = 0;
> +				}
>  			}
> -			ret = 0;
>  		}
>  	}
> 
> --
> 2.8.4

Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

[Gaëtan Rivet]

On Wed, Sep 25, 2019 at 06:41:36AM +0000, Slava Ovsiienko wrote:
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of vattunuru@marvell.com
> > Sent: Monday, September 23, 2019 14:57
> > To: dev@dpdk.org
> > Cc: gaetan.rivet@6wind.com; ferruh.yigit@intel.com;
> > anatoly.burakov@intel.com; Thomas Monjalon <thomas@monjalon.net>;
> > jerinj@marvell.com; Vamsi Attunuru <vattunuru@marvell.com>
> > Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted
> > order
> > 
> > From: Vamsi Attunuru <vattunuru@marvell.com>
> > 
> > Current pci bus driver scans pci devices in the order that it read from sysfs.
> > Accordingly all or whitelisted devices are getting probed.
> > 
> > Patch modifies the probing order of whitelisted pci devices in a sequence the
> > devices are whitelisted(using EAL flags).
> 
> Thanks, it would be nice to have opportunity to control probing order,
> it might be useful for bonded devices and representors either.
> 
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> 
> > 
> > It ensures the eth devices that application uses are probed in device
> > whitelisted sequence, in turn it facilitates the packet forwarding applications
> > to work without any packet loss or performance drop when the underneath
> > network ports have different bandwidths. By altering the whitelist order
> > applications like testpmd, l2fwd can forward the ingress traffic to egress port
> > that has of equivalent bandwidth.
> > 
> > Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>

Hello Vamsi, Viacheslav,

This is a nice patch. I agree that port dependency could be better
handled. The port-mapping part however should be managed at the app
level.

Vamsi, you gave the example of l2fwd and testpmd, being able to
properly configure forwarding directions implicitly. I think the better
approach here is to add these configurations items within the apps
directly. Configuring the mapping at the port level is not precise
enough. The proper control is about cores, port and queues, not only ports.
This patch only solves a limited part of this issue with testpmd.

I wrote a command to do this, that collided with some stream
rework from Intel at the time (3, 4 years back?), so I did not take the
time to force it through. If there is a need we could discuss about
adding this back. I had needed it to write a PMD, that could be useful
to others.

As you say Viacheslav, there are use-cases that will rely on
fine-grained probe order. However, this patch solves this issue only
regarding PCI devices, depending on other PCI devices. We have in EAL
an improper hack about it, forcing the vdev probe last, because
usually ports depending on others are virtual ones. As this patch shows,
the hack is not sufficient, and as the hack shows, this patch does not
cover everything.

A solution, would be an EAL parameter (I propose --no-dev), that
disable probing for all buses. Applications and devices requiring a
fine-grained probe order, are then free to start in this mode (and maybe
force it through EAL conf), then hotplug ports as they see fit.

This will keep the existing behavior stable for current apps, while allowing
flexibility for the more advanced ones.

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

[Vamsi Krishna Attunuru]

-----Original Message-----
From: dev <dev-bounces@dpdk.org> On Behalf Of Gaëtan Rivet
Sent: Wednesday, September 25, 2019 2:37 PM
To: Slava Ovsiienko <viacheslavo@mellanox.com>
Cc: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org; ferruh.yigit@intel.com; anatoly.burakov@intel.com; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>
Subject: Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

On Wed, Sep 25, 2019 at 06:41:36AM +0000, Slava Ovsiienko wrote:
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of vattunuru@marvell.com
> > Sent: Monday, September 23, 2019 14:57
> > To: dev@dpdk.org
> > Cc: gaetan.rivet@6wind.com; ferruh.yigit@intel.com; 
> > anatoly.burakov@intel.com; Thomas Monjalon <thomas@monjalon.net>; 
> > jerinj@marvell.com; Vamsi Attunuru <vattunuru@marvell.com>
> > Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in 
> > whitelisted order
> > 
> > From: Vamsi Attunuru <vattunuru@marvell.com>
> > 
> > Current pci bus driver scans pci devices in the order that it read from sysfs.
> > Accordingly all or whitelisted devices are getting probed.
> > 
> > Patch modifies the probing order of whitelisted pci devices in a 
> > sequence the devices are whitelisted(using EAL flags).
> 
> Thanks, it would be nice to have opportunity to control probing order, 
> it might be useful for bonded devices and representors either.
> 
> Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> 
> > 
> > It ensures the eth devices that application uses are probed in 
> > device whitelisted sequence, in turn it facilitates the packet 
> > forwarding applications to work without any packet loss or 
> > performance drop when the underneath network ports have different 
> > bandwidths. By altering the whitelist order applications like 
> > testpmd, l2fwd can forward the ingress traffic to egress port that has of equivalent bandwidth.
> > 
> > Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>

Hello Vamsi, Viacheslav,

This is a nice patch. I agree that port dependency could be better handled. The port-mapping part however should be managed at the app level.

Vamsi, you gave the example of l2fwd and testpmd, being able to properly configure forwarding directions implicitly. I think the better approach here is to add these configurations items within the apps directly. Configuring the mapping at the port level is not precise enough. The proper control is about cores, port and queues, not only ports.
This patch only solves a limited part of this issue with testpmd.

I wrote a command to do this, that collided with some stream rework from Intel at the time (3, 4 years back?), so I did not take the time to force it through. If there is a need we could discuss about adding this back. I had needed it to write a PMD, that could be useful to others.

As you say Viacheslav, there are use-cases that will rely on fine-grained probe order. However, this patch solves this issue only regarding PCI devices, depending on other PCI devices. We have in EAL an improper hack about it, forcing the vdev probe last, because usually ports depending on others are virtual ones. As this patch shows, the hack is not sufficient, and as the hack shows, this patch does not cover everything.

A solution, would be an EAL parameter (I propose --no-dev), that disable probing for all buses. Applications and devices requiring a fine-grained probe order, are then free to start in this mode (and maybe force it through EAL conf), then hotplug ports as they see fit.

This will keep the existing behavior stable for current apps, while allowing flexibility for the more advanced ones.


Hi Gaetan,

Thanks, vdev part was not taken care in this patch. Rather than imposing hotplug for every application which requires port mapping, If vdev probing order is also handled same as pdevs(in whitelist order),  existing whitelisting feature will serve the port mapping requirement, right. Also the existing applications get benefited instead of overloading them with more configuration options.  If these probing order is not needed by default, it can be triggered using an EAL parameter(not added yet).

Regards,
A Vamsi

--
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

[Gaëtan Rivet]

On Thu, Sep 26, 2019 at 04:15:49AM +0000, Vamsi Krishna Attunuru wrote:
> 
> 
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Gaëtan Rivet
> Sent: Wednesday, September 25, 2019 2:37 PM
> To: Slava Ovsiienko <viacheslavo@mellanox.com>
> Cc: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org; ferruh.yigit@intel.com; anatoly.burakov@intel.com; Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Subject: Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order
> 
> On Wed, Sep 25, 2019 at 06:41:36AM +0000, Slava Ovsiienko wrote:
> > > -----Original Message-----
> > > From: dev <dev-bounces@dpdk.org> On Behalf Of vattunuru@marvell.com
> > > Sent: Monday, September 23, 2019 14:57
> > > To: dev@dpdk.org
> > > Cc: gaetan.rivet@6wind.com; ferruh.yigit@intel.com; 
> > > anatoly.burakov@intel.com; Thomas Monjalon <thomas@monjalon.net>; 
> > > jerinj@marvell.com; Vamsi Attunuru <vattunuru@marvell.com>
> > > Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in 
> > > whitelisted order
> > > 
> > > From: Vamsi Attunuru <vattunuru@marvell.com>
> > > 
> > > Current pci bus driver scans pci devices in the order that it read from sysfs.
> > > Accordingly all or whitelisted devices are getting probed.
> > > 
> > > Patch modifies the probing order of whitelisted pci devices in a 
> > > sequence the devices are whitelisted(using EAL flags).
> > 
> > Thanks, it would be nice to have opportunity to control probing order, 
> > it might be useful for bonded devices and representors either.
> > 
> > Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> > 
> > > 
> > > It ensures the eth devices that application uses are probed in 
> > > device whitelisted sequence, in turn it facilitates the packet 
> > > forwarding applications to work without any packet loss or 
> > > performance drop when the underneath network ports have different 
> > > bandwidths. By altering the whitelist order applications like 
> > > testpmd, l2fwd can forward the ingress traffic to egress port that has of equivalent bandwidth.
> > > 
> > > Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> 
> Hello Vamsi, Viacheslav,
> 
> This is a nice patch. I agree that port dependency could be better handled. The port-mapping part however should be managed at the app level.
> 
> Vamsi, you gave the example of l2fwd and testpmd, being able to properly configure forwarding directions implicitly. I think the better approach here is to add these configurations items within the apps directly. Configuring the mapping at the port level is not precise enough. The proper control is about cores, port and queues, not only ports.
> This patch only solves a limited part of this issue with testpmd.
> 
> I wrote a command to do this, that collided with some stream rework from Intel at the time (3, 4 years back?), so I did not take the time to force it through. If there is a need we could discuss about adding this back. I had needed it to write a PMD, that could be useful to others.
> 
> As you say Viacheslav, there are use-cases that will rely on fine-grained probe order. However, this patch solves this issue only regarding PCI devices, depending on other PCI devices. We have in EAL an improper hack about it, forcing the vdev probe last, because usually ports depending on others are virtual ones. As this patch shows, the hack is not sufficient, and as the hack shows, this patch does not cover everything.
> 
> A solution, would be an EAL parameter (I propose --no-dev), that disable probing for all buses. Applications and devices requiring a fine-grained probe order, are then free to start in this mode (and maybe force it through EAL conf), then hotplug ports as they see fit.
> 
> This will keep the existing behavior stable for current apps, while allowing flexibility for the more advanced ones.
> 
> 
> Hi Gaetan,
> 
> Thanks, vdev part was not taken care in this patch. Rather than imposing hotplug for every application which requires port mapping, If vdev probing order is also handled same as pdevs(in whitelist order),  existing whitelisting feature will serve the port mapping requirement, right. Also the existing applications get benefited instead of overloading them with more configuration options.  If these probing order is not needed by default, it can be triggered using an EAL parameter(not added yet).
> 
> Regards,
> A Vamsi

Hi,

The way buses are written right now, they will each do a whole scan, then
they each probe all their devices.

You cannot intersperse probes across several buses, i.e. probe a PCI
device, then a vdev, then another PCI device.

Changing this structure could be difficult. A possible way to do what
you want without breaking everything would be to do what the app would
have done in my solution above, but from within the EAL: block all
probes, then go over a mixed list of (-w) and (--vdev) parameters and
hotplug them in order. This would require the --no-dev (or --no-probe,
or --no-auto-probe) flag anyway (or as a conf item, or something at
least telling the EAL to behave this way).

Would this way of doing it work for you?

In any case, controlling the probe order should be fixed properly for
all buses and the general use-case if possible, instead of limiting the
patch to the PCI bus.

Kind regards,
-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [EXT] Re: [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

[Vamsi Krishna Attunuru]

> -----Original Message-----
> From: Gaëtan Rivet <gaetan.rivet@6wind.com>
> Sent: Thursday, September 26, 2019 1:34 PM
> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>
> Cc: Slava Ovsiienko <viacheslavo@mellanox.com>; dev@dpdk.org;
> ferruh.yigit@intel.com; anatoly.burakov@intel.com; Thomas Monjalon
> <thomas@monjalon.net>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> Subject: [EXT] Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in
> whitelisted order
> 
> External Email
> 
> ----------------------------------------------------------------------
> On Thu, Sep 26, 2019 at 04:15:49AM +0000, Vamsi Krishna Attunuru wrote:
> >
> >
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Gaëtan Rivet
> > Sent: Wednesday, September 25, 2019 2:37 PM
> > To: Slava Ovsiienko <viacheslavo@mellanox.com>
> > Cc: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org;
> > ferruh.yigit@intel.com; anatoly.burakov@intel.com; Thomas Monjalon
> > <thomas@monjalon.net>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>
> > Subject: Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in
> > whitelisted order
> >
> > On Wed, Sep 25, 2019 at 06:41:36AM +0000, Slava Ovsiienko wrote:
> > > > -----Original Message-----
> > > > From: dev <dev-bounces@dpdk.org> On Behalf Of
> > > > vattunuru@marvell.com
> > > > Sent: Monday, September 23, 2019 14:57
> > > > To: dev@dpdk.org
> > > > Cc: gaetan.rivet@6wind.com; ferruh.yigit@intel.com;
> > > > anatoly.burakov@intel.com; Thomas Monjalon <thomas@monjalon.net>;
> > > > jerinj@marvell.com; Vamsi Attunuru <vattunuru@marvell.com>
> > > > Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in
> > > > whitelisted order
> > > >
> > > > From: Vamsi Attunuru <vattunuru@marvell.com>
> > > >
> > > > Current pci bus driver scans pci devices in the order that it read from sysfs.
> > > > Accordingly all or whitelisted devices are getting probed.
> > > >
> > > > Patch modifies the probing order of whitelisted pci devices in a
> > > > sequence the devices are whitelisted(using EAL flags).
> > >
> > > Thanks, it would be nice to have opportunity to control probing
> > > order, it might be useful for bonded devices and representors either.
> > >
> > > Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
> > >
> > > >
> > > > It ensures the eth devices that application uses are probed in
> > > > device whitelisted sequence, in turn it facilitates the packet
> > > > forwarding applications to work without any packet loss or
> > > > performance drop when the underneath network ports have different
> > > > bandwidths. By altering the whitelist order applications like
> > > > testpmd, l2fwd can forward the ingress traffic to egress port that has of
> equivalent bandwidth.
> > > >
> > > > Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> >
> > Hello Vamsi, Viacheslav,
> >
> > This is a nice patch. I agree that port dependency could be better handled. The
> port-mapping part however should be managed at the app level.
> >
> > Vamsi, you gave the example of l2fwd and testpmd, being able to properly
> configure forwarding directions implicitly. I think the better approach here is to
> add these configurations items within the apps directly. Configuring the mapping
> at the port level is not precise enough. The proper control is about cores, port
> and queues, not only ports.
> > This patch only solves a limited part of this issue with testpmd.
> >
> > I wrote a command to do this, that collided with some stream rework from
> Intel at the time (3, 4 years back?), so I did not take the time to force it through.
> If there is a need we could discuss about adding this back. I had needed it to
> write a PMD, that could be useful to others.
> >
> > As you say Viacheslav, there are use-cases that will rely on fine-grained probe
> order. However, this patch solves this issue only regarding PCI devices,
> depending on other PCI devices. We have in EAL an improper hack about it,
> forcing the vdev probe last, because usually ports depending on others are
> virtual ones. As this patch shows, the hack is not sufficient, and as the hack
> shows, this patch does not cover everything.
> >
> > A solution, would be an EAL parameter (I propose --no-dev), that disable
> probing for all buses. Applications and devices requiring a fine-grained probe
> order, are then free to start in this mode (and maybe force it through EAL conf),
> then hotplug ports as they see fit.
> >
> > This will keep the existing behavior stable for current apps, while allowing
> flexibility for the more advanced ones.
> >
> >
> > Hi Gaetan,
> >
> > Thanks, vdev part was not taken care in this patch. Rather than imposing
> hotplug for every application which requires port mapping, If vdev probing order
> is also handled same as pdevs(in whitelist order),  existing whitelisting feature
> will serve the port mapping requirement, right. Also the existing applications get
> benefited instead of overloading them with more configuration options.  If
> these probing order is not needed by default, it can be triggered using an EAL
> parameter(not added yet).
> >
> > Regards,
> > A Vamsi
> 
> Hi,
> 
> The way buses are written right now, they will each do a whole scan, then they
> each probe all their devices.
> 
> You cannot intersperse probes across several buses, i.e. probe a PCI device, then
> a vdev, then another PCI device.
> 
> Changing this structure could be difficult. A possible way to do what you want
> without breaking everything would be to do what the app would have done in
> my solution above, but from within the EAL: block all probes, then go over a
> mixed list of (-w) and (--vdev) parameters and hotplug them in order. This would
> require the --no-dev (or --no-probe, or --no-auto-probe) flag anyway (or as a
> conf item, or something at least telling the EAL to behave this way).
> 
> Would this way of doing it work for you?
Yes, above approach sounds fine to me and it works without breaking everything.

> 
> In any case, controlling the probe order should be fixed properly for all buses
> and the general use-case if possible, instead of limiting the patch to the PCI bus.

 Ack
> 
> Kind regards,
> --
> Gaëtan Rivet
> 6WIND

[dpdk-dev] [PATCH v1] eal: add manual probing option

[Gaetan Rivet]

Add a new EAL option enabling manual probing in the EAL.
This command line option will configure the EAL so that buses
will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order
of their devices, for drivers requiring it.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---

I haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
        and that it works as intended for basic cases.
        Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
 lib/librte_eal/linux/eal/eal.c             |  5 ++++
 lib/librte_eal/rte_eal_version.map         |  7 +++++
 11 files changed, 110 insertions(+)

diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index 27cfbd9e3..3996e429d 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL wil now propose to enforce manual probing of devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
index baa5b532a..145a96812 100644
--- a/lib/librte_eal/common/eal_common_bus.c
+++ b/lib/librte_eal/common/eal_common_bus.c
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 9e4f09d83..3b7769a3c 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
 }
 
 int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0 || !rte_dev_is_probed(dev)) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
 rte_eal_hotplug_add(const char *busname, const char *devname,
 		    const char *drvargs)
 {
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 05cae5f75..66c232b14 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index a42f34923..0006f903f 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index 9855429e5..588fa32a6 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 798ede553..fd7ac8e37 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index b7cf91214..bd9087bcc 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option.
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ */
+int rte_eal_manual_probe(void);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal/eal.c
index d53f0fe69..2daa63b23 100644
--- a/lib/librte_eal/freebsd/eal/eal.c
+++ b/lib/librte_eal/freebsd/eal/eal.c
@@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
 	return !internal_config.no_pci;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 int rte_eal_create_uio_dev(void)
 {
 	return internal_config.create_uio_dev;
diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
index 946222ccd..476fc2e4a 100644
--- a/lib/librte_eal/linux/eal/eal.c
+++ b/lib/librte_eal/linux/eal/eal.c
@@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
 	return internal_config.create_uio_dev;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 enum rte_intr_mode
 rte_eal_vfio_intr_mode(void)
 {
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index 7cbf82d37..fe5395c24 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -312,6 +312,13 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.05 {
+	global:
+
+	rte_eal_manual_probe
+
+} DPDK_19.11;
+
 EXPERIMENTAL {
 	global:
 
-- 
2.11.0

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

[Aaron Conole]

Gaetan Rivet <gaetan.rivet@6wind.com> writes:

> Add a new EAL option enabling manual probing in the EAL.
> This command line option will configure the EAL so that buses
> will not trigger their probe step on their own.
>
> Applications are then expected to hotplug devices as they see fit.
>
> Devices declared on the command line by the user (using -w and --vdev),
> will be probed using the hotplug API, in the order they are declared.
>
> This has the effect of offering a way for users to control probe order
> of their devices, for drivers requiring it.
>
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> ---
>
> I haven't heard many opinions on the matter, please shout if you see an issue
> with this approach.
>
> @Slava: I have tested rather quickly that it does not break anything,
>         and that it works as intended for basic cases.
>         Can you test it further for your use-case and tell me if it works fine?
>
> Beyond the obvious difference between both probe mode, something to keep in mind:
> while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
> probing in its track. All devices need to exist and be valid device IDs.
>
>  doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
>  lib/librte_eal/common/eal_common_bus.c     |  6 +++++
>  lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
>  lib/librte_eal/common/eal_common_options.c |  8 ++++++
>  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
>  lib/librte_eal/common/eal_options.h        |  2 ++
>  lib/librte_eal/common/eal_private.h        |  9 +++++++
>  lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
>  lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
>  lib/librte_eal/linux/eal/eal.c             |  5 ++++
>  lib/librte_eal/rte_eal_version.map         |  7 +++++
>  11 files changed, 110 insertions(+)
>
> diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> index 27cfbd9e3..3996e429d 100644
> --- a/doc/guides/rel_notes/release_19_11.rst
> +++ b/doc/guides/rel_notes/release_19_11.rst
> @@ -56,6 +56,15 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =========================================================
>  
> +* **EAL wil now propose to enforce manual probing of devices.**
> +
> +  Previously, a user could not force an order when probing declared devices.
> +  This could cause issues for drivers depending on another device being present.
> +  A new option ``--manual-probe`` is now available to do just that.
> +  This new option relies on the device bus supporting hotplug. It can
> +  also be used to disable automatic probing from the ``PCI`` bus without
> +  having to disable the whole bus.
> +
>  
>  Removed Items
>  -------------
> diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
> index baa5b532a..145a96812 100644
> --- a/lib/librte_eal/common/eal_common_bus.c
> +++ b/lib/librte_eal/common/eal_common_bus.c
> @@ -6,6 +6,7 @@
>  #include <string.h>
>  #include <sys/queue.h>
>  
> +#include <rte_eal.h>
>  #include <rte_bus.h>
>  #include <rte_debug.h>
>  #include <rte_string_fns.h>
> @@ -63,6 +64,11 @@ rte_bus_probe(void)
>  	int ret;
>  	struct rte_bus *bus, *vbus = NULL;
>  
> +	if (rte_eal_manual_probe()) {
> +		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
> +		return rte_dev_probe_devargs_list();
> +	}
> +
>  	TAILQ_FOREACH(bus, &rte_bus_list, next) {
>  		if (!strcmp(bus->name, "vdev")) {
>  			vbus = bus;
> diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
> index 9e4f09d83..3b7769a3c 100644
> --- a/lib/librte_eal/common/eal_common_dev.c
> +++ b/lib/librte_eal/common/eal_common_dev.c
> @@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
>  }
>  
>  int
> +rte_dev_probe_devargs_list(void)
> +{
> +	struct rte_device *dev;
> +	struct rte_devargs *da;
> +	int ret;
> +
> +	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
> +		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
> +		if (dev == NULL) {
> +			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
> +				da->name, da->bus->name);
> +			continue;
> +		}
> +
> +		if (rte_dev_is_probed(dev))
> +			continue;
> +
> +		if (dev->bus->plug == NULL) {
> +			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
> +					  "required by device %s\n",
> +				dev->bus->name, dev->name);
> +			continue;
> +		}
> +
> +		ret = dev->bus->plug(dev);
> +		/* Ignore positive return values, they are possibly
> +		 * triggered by blacklisted devices on the PCI bus. Probing
> +		 * should then continue.
> +		 */
> +		if (ret < 0 || !rte_dev_is_probed(dev)) {
> +			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
> +				dev->name);
> +			/* Fail on first real probe error. */
> +			return ret;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +int
>  rte_eal_hotplug_add(const char *busname, const char *devname,
>  		    const char *drvargs)
>  {
> diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
> index 05cae5f75..66c232b14 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -81,6 +81,7 @@ eal_long_options[] = {
>  	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
>  	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
>  	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
> +	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
>  	{0,                     0, NULL, 0                        }
>  };
>  
> @@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
>  			return -1;
>  		}
>  		break;
> +	case OPT_MANUAL_PROBE_NUM:
> +		conf->manual_probe = 1;
> +		break;
>  
>  	/* don't know what to do, leave this to caller */
>  	default:
> @@ -1634,6 +1638,10 @@ eal_common_usage(void)
>  	       "  --"OPT_VDEV"              Add a virtual device.\n"
>  	       "                      The argument format is <driver><id>[,key=val,...]\n"
>  	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
> +	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
> +	       "                      Disable probe step for all buses.\n"
> +	       "                      Devices will need to be probed using the hotplug API.\n"
> +	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
>  	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
>  	       "                      'va' for IOVA_VA\n"
>  	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
> diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
> index a42f34923..0006f903f 100644
> --- a/lib/librte_eal/common/eal_internal_cfg.h
> +++ b/lib/librte_eal/common/eal_internal_cfg.h
> @@ -44,6 +44,7 @@ struct internal_config {
>  	unsigned hugepage_unlink;         /**< true to unlink backing files */
>  	volatile unsigned no_pci;         /**< true to disable PCI */
>  	volatile unsigned no_hpet;        /**< true to disable HPET */
> +	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
>  	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
>  										* instead of native TSC */
>  	volatile unsigned no_shconf;      /**< true if there is no shared config */
> diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
> index 9855429e5..588fa32a6 100644
> --- a/lib/librte_eal/common/eal_options.h
> +++ b/lib/librte_eal/common/eal_options.h
> @@ -69,6 +69,8 @@ enum {
>  	OPT_IOVA_MODE_NUM,
>  #define OPT_MATCH_ALLOCATIONS  "match-allocations"
>  	OPT_MATCH_ALLOCATIONS_NUM,
> +#define OPT_MANUAL_PROBE "manual-probe"
> +	OPT_MANUAL_PROBE_NUM,
>  	OPT_LONG_MAX_NUM
>  };
>  
> diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
> index 798ede553..fd7ac8e37 100644
> --- a/lib/librte_eal/common/eal_private.h
> +++ b/lib/librte_eal/common/eal_private.h
> @@ -381,4 +381,13 @@ rte_option_init(void);
>  void
>  rte_option_usage(void);
>  
> +/**
> + * Go through the devargs list and probe everything in order.
> + *
> + * @return
> + *   0 on success, negative on error.
> + */
> +int
> +rte_dev_probe_devargs_list(void);
> +
>  #endif /* _EAL_PRIVATE_H_ */
> diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
> index b7cf91214..bd9087bcc 100644
> --- a/lib/librte_eal/common/include/rte_eal.h
> +++ b/lib/librte_eal/common/include/rte_eal.h
> @@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
>  int rte_eal_has_pci(void);
>  
>  /**
> + * Whether EAL probe is manual.
> + * Enabled by the --manual-probe option.
> + *
> + * When manual probing is enabled, batched bus probe of
> + * their devices is disabled. All devices need to be probed
> + * using the proper rte_dev API.
> + *
> + * In this mode, devices declared on the command line will
> + * be probed using the bus hotplug API. It is used to enforce
> + * a specific probe order.
> + *
> + * @return
> + *   Nonzero if manual device probing is enabled.
> + */
> +int rte_eal_manual_probe(void);
> +
> +/**
>   * Whether the EAL was asked to create UIO device.
>   *
>   * @return
> diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal/eal.c
> index d53f0fe69..2daa63b23 100644
> --- a/lib/librte_eal/freebsd/eal/eal.c
> +++ b/lib/librte_eal/freebsd/eal/eal.c
> @@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
>  	return !internal_config.no_pci;
>  }
>  
> +int rte_eal_manual_probe(void)
> +{
> +	return internal_config.manual_probe;
> +}
> +
>  int rte_eal_create_uio_dev(void)
>  {
>  	return internal_config.create_uio_dev;
> diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
> index 946222ccd..476fc2e4a 100644
> --- a/lib/librte_eal/linux/eal/eal.c
> +++ b/lib/librte_eal/linux/eal/eal.c
> @@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
>  	return internal_config.create_uio_dev;
>  }
>  
> +int rte_eal_manual_probe(void)
> +{
> +	return internal_config.manual_probe;
> +}
> +
>  enum rte_intr_mode
>  rte_eal_vfio_intr_mode(void)
>  {
> diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> index 7cbf82d37..fe5395c24 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -312,6 +312,13 @@ DPDK_19.08 {
>  
>  } DPDK_19.05;
>  
> +DPDK_19.05 {
> +	global:
> +
> +	rte_eal_manual_probe

Syntax error - needs a ';' at the end of this symbol.

> +
> +} DPDK_19.11;
> +
>  EXPERIMENTAL {
>  	global:

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

[Stephen Hemminger]

On Mon, 30 Sep 2019 14:51:03 +0200
Gaetan Rivet <gaetan.rivet@6wind.com> wrote:

> Add a new EAL option enabling manual probing in the EAL.
> This command line option will configure the EAL so that buses
> will not trigger their probe step on their own.
> 
> Applications are then expected to hotplug devices as they see fit.
> 
> Devices declared on the command line by the user (using -w and --vdev),
> will be probed using the hotplug API, in the order they are declared.
> 
> This has the effect of offering a way for users to control probe order
> of their devices, for drivers requiring it.
> 
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>

I have no problems with the patch, but it would help if there was better
way to handle device naming policy in DPDK. Applications that depend on
particular port number are prone to get broken by changes in surrounding
OS or hardware environment. Just like Linux applications that are built
to depend on "eth0"; which is unfortunately all too common.

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

[Gaëtan Rivet]

On Mon, Sep 30, 2019 at 01:51:35PM -0400, Aaron Conole wrote:
> Gaetan Rivet <gaetan.rivet@6wind.com> writes:
> 
> > Add a new EAL option enabling manual probing in the EAL.
> > This command line option will configure the EAL so that buses
> > will not trigger their probe step on their own.
> >
> > Applications are then expected to hotplug devices as they see fit.
> >
> > Devices declared on the command line by the user (using -w and --vdev),
> > will be probed using the hotplug API, in the order they are declared.
> >
> > This has the effect of offering a way for users to control probe order
> > of their devices, for drivers requiring it.
> >
> > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> > ---
> >
> > I haven't heard many opinions on the matter, please shout if you see an issue
> > with this approach.
> >
> > @Slava: I have tested rather quickly that it does not break anything,
> >         and that it works as intended for basic cases.
> >         Can you test it further for your use-case and tell me if it works fine?
> >
> > Beyond the obvious difference between both probe mode, something to keep in mind:
> > while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
> > probing in its track. All devices need to exist and be valid device IDs.
> >
> >  doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
> >  lib/librte_eal/common/eal_common_bus.c     |  6 +++++
> >  lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
> >  lib/librte_eal/common/eal_common_options.c |  8 ++++++
> >  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
> >  lib/librte_eal/common/eal_options.h        |  2 ++
> >  lib/librte_eal/common/eal_private.h        |  9 +++++++
> >  lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
> >  lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
> >  lib/librte_eal/linux/eal/eal.c             |  5 ++++
> >  lib/librte_eal/rte_eal_version.map         |  7 +++++
> >  11 files changed, 110 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> > index 27cfbd9e3..3996e429d 100644
> > --- a/doc/guides/rel_notes/release_19_11.rst
> > +++ b/doc/guides/rel_notes/release_19_11.rst
> > @@ -56,6 +56,15 @@ New Features
> >       Also, make sure to start the actual text at the margin.
> >       =========================================================
> >  
> > +* **EAL wil now propose to enforce manual probing of devices.**
> > +
> > +  Previously, a user could not force an order when probing declared devices.
> > +  This could cause issues for drivers depending on another device being present.
> > +  A new option ``--manual-probe`` is now available to do just that.
> > +  This new option relies on the device bus supporting hotplug. It can
> > +  also be used to disable automatic probing from the ``PCI`` bus without
> > +  having to disable the whole bus.
> > +
> >  
> >  Removed Items
> >  -------------
> > diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
> > index baa5b532a..145a96812 100644
> > --- a/lib/librte_eal/common/eal_common_bus.c
> > +++ b/lib/librte_eal/common/eal_common_bus.c
> > @@ -6,6 +6,7 @@
> >  #include <string.h>
> >  #include <sys/queue.h>
> >  
> > +#include <rte_eal.h>
> >  #include <rte_bus.h>
> >  #include <rte_debug.h>
> >  #include <rte_string_fns.h>
> > @@ -63,6 +64,11 @@ rte_bus_probe(void)
> >  	int ret;
> >  	struct rte_bus *bus, *vbus = NULL;
> >  
> > +	if (rte_eal_manual_probe()) {
> > +		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
> > +		return rte_dev_probe_devargs_list();
> > +	}
> > +
> >  	TAILQ_FOREACH(bus, &rte_bus_list, next) {
> >  		if (!strcmp(bus->name, "vdev")) {
> >  			vbus = bus;
> > diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
> > index 9e4f09d83..3b7769a3c 100644
> > --- a/lib/librte_eal/common/eal_common_dev.c
> > +++ b/lib/librte_eal/common/eal_common_dev.c
> > @@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
> >  }
> >  
> >  int
> > +rte_dev_probe_devargs_list(void)
> > +{
> > +	struct rte_device *dev;
> > +	struct rte_devargs *da;
> > +	int ret;
> > +
> > +	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
> > +		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
> > +		if (dev == NULL) {
> > +			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
> > +				da->name, da->bus->name);
> > +			continue;
> > +		}
> > +
> > +		if (rte_dev_is_probed(dev))
> > +			continue;
> > +
> > +		if (dev->bus->plug == NULL) {
> > +			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
> > +					  "required by device %s\n",
> > +				dev->bus->name, dev->name);
> > +			continue;
> > +		}
> > +
> > +		ret = dev->bus->plug(dev);
> > +		/* Ignore positive return values, they are possibly
> > +		 * triggered by blacklisted devices on the PCI bus. Probing
> > +		 * should then continue.
> > +		 */
> > +		if (ret < 0 || !rte_dev_is_probed(dev)) {
> > +			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
> > +				dev->name);
> > +			/* Fail on first real probe error. */
> > +			return ret;
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> > +int
> >  rte_eal_hotplug_add(const char *busname, const char *devname,
> >  		    const char *drvargs)
> >  {
> > diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
> > index 05cae5f75..66c232b14 100644
> > --- a/lib/librte_eal/common/eal_common_options.c
> > +++ b/lib/librte_eal/common/eal_common_options.c
> > @@ -81,6 +81,7 @@ eal_long_options[] = {
> >  	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
> >  	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
> >  	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
> > +	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
> >  	{0,                     0, NULL, 0                        }
> >  };
> >  
> > @@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
> >  			return -1;
> >  		}
> >  		break;
> > +	case OPT_MANUAL_PROBE_NUM:
> > +		conf->manual_probe = 1;
> > +		break;
> >  
> >  	/* don't know what to do, leave this to caller */
> >  	default:
> > @@ -1634,6 +1638,10 @@ eal_common_usage(void)
> >  	       "  --"OPT_VDEV"              Add a virtual device.\n"
> >  	       "                      The argument format is <driver><id>[,key=val,...]\n"
> >  	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
> > +	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
> > +	       "                      Disable probe step for all buses.\n"
> > +	       "                      Devices will need to be probed using the hotplug API.\n"
> > +	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
> >  	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
> >  	       "                      'va' for IOVA_VA\n"
> >  	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
> > diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
> > index a42f34923..0006f903f 100644
> > --- a/lib/librte_eal/common/eal_internal_cfg.h
> > +++ b/lib/librte_eal/common/eal_internal_cfg.h
> > @@ -44,6 +44,7 @@ struct internal_config {
> >  	unsigned hugepage_unlink;         /**< true to unlink backing files */
> >  	volatile unsigned no_pci;         /**< true to disable PCI */
> >  	volatile unsigned no_hpet;        /**< true to disable HPET */
> > +	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
> >  	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
> >  										* instead of native TSC */
> >  	volatile unsigned no_shconf;      /**< true if there is no shared config */
> > diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
> > index 9855429e5..588fa32a6 100644
> > --- a/lib/librte_eal/common/eal_options.h
> > +++ b/lib/librte_eal/common/eal_options.h
> > @@ -69,6 +69,8 @@ enum {
> >  	OPT_IOVA_MODE_NUM,
> >  #define OPT_MATCH_ALLOCATIONS  "match-allocations"
> >  	OPT_MATCH_ALLOCATIONS_NUM,
> > +#define OPT_MANUAL_PROBE "manual-probe"
> > +	OPT_MANUAL_PROBE_NUM,
> >  	OPT_LONG_MAX_NUM
> >  };
> >  
> > diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
> > index 798ede553..fd7ac8e37 100644
> > --- a/lib/librte_eal/common/eal_private.h
> > +++ b/lib/librte_eal/common/eal_private.h
> > @@ -381,4 +381,13 @@ rte_option_init(void);
> >  void
> >  rte_option_usage(void);
> >  
> > +/**
> > + * Go through the devargs list and probe everything in order.
> > + *
> > + * @return
> > + *   0 on success, negative on error.
> > + */
> > +int
> > +rte_dev_probe_devargs_list(void);
> > +
> >  #endif /* _EAL_PRIVATE_H_ */
> > diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
> > index b7cf91214..bd9087bcc 100644
> > --- a/lib/librte_eal/common/include/rte_eal.h
> > +++ b/lib/librte_eal/common/include/rte_eal.h
> > @@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
> >  int rte_eal_has_pci(void);
> >  
> >  /**
> > + * Whether EAL probe is manual.
> > + * Enabled by the --manual-probe option.
> > + *
> > + * When manual probing is enabled, batched bus probe of
> > + * their devices is disabled. All devices need to be probed
> > + * using the proper rte_dev API.
> > + *
> > + * In this mode, devices declared on the command line will
> > + * be probed using the bus hotplug API. It is used to enforce
> > + * a specific probe order.
> > + *
> > + * @return
> > + *   Nonzero if manual device probing is enabled.
> > + */
> > +int rte_eal_manual_probe(void);
> > +
> > +/**
> >   * Whether the EAL was asked to create UIO device.
> >   *
> >   * @return
> > diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal/eal.c
> > index d53f0fe69..2daa63b23 100644
> > --- a/lib/librte_eal/freebsd/eal/eal.c
> > +++ b/lib/librte_eal/freebsd/eal/eal.c
> > @@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
> >  	return !internal_config.no_pci;
> >  }
> >  
> > +int rte_eal_manual_probe(void)
> > +{
> > +	return internal_config.manual_probe;
> > +}
> > +
> >  int rte_eal_create_uio_dev(void)
> >  {
> >  	return internal_config.create_uio_dev;
> > diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
> > index 946222ccd..476fc2e4a 100644
> > --- a/lib/librte_eal/linux/eal/eal.c
> > +++ b/lib/librte_eal/linux/eal/eal.c
> > @@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
> >  	return internal_config.create_uio_dev;
> >  }
> >  
> > +int rte_eal_manual_probe(void)
> > +{
> > +	return internal_config.manual_probe;
> > +}
> > +
> >  enum rte_intr_mode
> >  rte_eal_vfio_intr_mode(void)
> >  {
> > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> > index 7cbf82d37..fe5395c24 100644
> > --- a/lib/librte_eal/rte_eal_version.map
> > +++ b/lib/librte_eal/rte_eal_version.map
> > @@ -312,6 +312,13 @@ DPDK_19.08 {
> >  
> >  } DPDK_19.05;
> >  
> > +DPDK_19.05 {
> > +	global:
> > +
> > +	rte_eal_manual_probe
> 
> Syntax error - needs a ';' at the end of this symbol.
> 

Ah yes, sorry.
I'm thinking there is a script or a validation tool using this, I should try it.

> > +
> > +} DPDK_19.11;
> > +
> >  EXPERIMENTAL {
> >  	global:

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

[Gaëtan Rivet]

On Mon, Sep 30, 2019 at 11:53:33AM -0700, Stephen Hemminger wrote:
> On Mon, 30 Sep 2019 14:51:03 +0200
> Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
> 
> > Add a new EAL option enabling manual probing in the EAL.
> > This command line option will configure the EAL so that buses
> > will not trigger their probe step on their own.
> > 
> > Applications are then expected to hotplug devices as they see fit.
> > 
> > Devices declared on the command line by the user (using -w and --vdev),
> > will be probed using the hotplug API, in the order they are declared.
> > 
> > This has the effect of offering a way for users to control probe order
> > of their devices, for drivers requiring it.
> > 
> > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> 
> I have no problems with the patch, but it would help if there was better
> way to handle device naming policy in DPDK. Applications that depend on
> particular port number are prone to get broken by changes in surrounding
> OS or hardware environment. Just like Linux applications that are built
> to depend on "eth0"; which is unfortunately all too common.

Hello Stephen,

This patch is a way to avoid having the PCI bus defining the probe order
with the current hardware environment. It seems to be a step in the
right direction for the issue you identify.

There is a tight coupling between device names and driver matches for
the vdev bus, but that seems difficult to avoid.

Do you see other EAL APIs fostering an over reliance of downstream
systems on device names?

I pushed a few months back a way to iterate / match devices by their
properties. If you identify other pain points, this could certainly be
improved as well.

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

[Jerin Jacob]

On Tue, Oct 1, 2019 at 2:40 PM Gaëtan Rivet <gaetan.rivet@6wind.com> wrote:
>
> On Mon, Sep 30, 2019 at 11:53:33AM -0700, Stephen Hemminger wrote:
> > On Mon, 30 Sep 2019 14:51:03 +0200
> > Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
> >
> > > Add a new EAL option enabling manual probing in the EAL.
> > > This command line option will configure the EAL so that buses
> > > will not trigger their probe step on their own.
> > >
> > > Applications are then expected to hotplug devices as they see fit.
> > >
> > > Devices declared on the command line by the user (using -w and --vdev),
> > > will be probed using the hotplug API, in the order they are declared.
> > >
> > > This has the effect of offering a way for users to control probe order
> > > of their devices, for drivers requiring it.
> > >
> > > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> >
> > I have no problems with the patch, but it would help if there was better
> > way to handle device naming policy in DPDK. Applications that depend on
> > particular port number are prone to get broken by changes in surrounding
> > OS or hardware environment. Just like Linux applications that are built
> > to depend on "eth0"; which is unfortunately all too common.
>
> Hello Stephen,
>
> This patch is a way to avoid having the PCI bus defining the probe order
> with the current hardware environment. It seems to be a step in the
> right direction for the issue you identify.
>
> There is a tight coupling between device names and driver matches for
> the vdev bus, but that seems difficult to avoid.
>
> Do you see other EAL APIs fostering an over reliance of downstream
> systems on device names?
>
> I pushed a few months back a way to iterate / match devices by their
> properties. If you identify other pain points, this could certainly be
> improved as well.


And this mode will be kicked in only when "--manual-probe" selected on
eal arguments.
So it won't change the behavior of the existing applications.


>
> --
> Gaėtan Rivet
> 6WIND

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

[Aaron Conole]

Gaëtan Rivet <gaetan.rivet@6wind.com> writes:

> On Mon, Sep 30, 2019 at 01:51:35PM -0400, Aaron Conole wrote:
>> Gaetan Rivet <gaetan.rivet@6wind.com> writes:
>> 
>> > Add a new EAL option enabling manual probing in the EAL.
>> > This command line option will configure the EAL so that buses
>> > will not trigger their probe step on their own.
>> >
>> > Applications are then expected to hotplug devices as they see fit.
>> >
>> > Devices declared on the command line by the user (using -w and --vdev),
>> > will be probed using the hotplug API, in the order they are declared.
>> >
>> > This has the effect of offering a way for users to control probe order
>> > of their devices, for drivers requiring it.
>> >
>> > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
>> > ---
>> >
>> > I haven't heard many opinions on the matter, please shout if you see an issue
>> > with this approach.
>> >
>> > @Slava: I have tested rather quickly that it does not break anything,
>> >         and that it works as intended for basic cases.
>> >         Can you test it further for your use-case and tell me if it works fine?
>> >
>> > Beyond the obvious difference between both probe mode, something to keep in mind:
>> > while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
>> > probing in its track. All devices need to exist and be valid device IDs.
>> >
>> >  doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
>> >  lib/librte_eal/common/eal_common_bus.c     |  6 +++++
>> >  lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
>> >  lib/librte_eal/common/eal_common_options.c |  8 ++++++
>> >  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
>> >  lib/librte_eal/common/eal_options.h        |  2 ++
>> >  lib/librte_eal/common/eal_private.h        |  9 +++++++
>> >  lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
>> >  lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
>> >  lib/librte_eal/linux/eal/eal.c             |  5 ++++
>> >  lib/librte_eal/rte_eal_version.map         |  7 +++++
>> >  11 files changed, 110 insertions(+)
>> >
>> > diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
>> > index 27cfbd9e3..3996e429d 100644
>> > --- a/doc/guides/rel_notes/release_19_11.rst
>> > +++ b/doc/guides/rel_notes/release_19_11.rst
>> > @@ -56,6 +56,15 @@ New Features
>> >       Also, make sure to start the actual text at the margin.
>> >       =========================================================
>> >  
>> > +* **EAL wil now propose to enforce manual probing of devices.**
>> > +
>> > +  Previously, a user could not force an order when probing declared devices.
>> > +  This could cause issues for drivers depending on another device being present.
>> > +  A new option ``--manual-probe`` is now available to do just that.
>> > +  This new option relies on the device bus supporting hotplug. It can
>> > +  also be used to disable automatic probing from the ``PCI`` bus without
>> > +  having to disable the whole bus.
>> > +
>> >  
>> >  Removed Items
>> >  -------------
>> > diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
>> > index baa5b532a..145a96812 100644
>> > --- a/lib/librte_eal/common/eal_common_bus.c
>> > +++ b/lib/librte_eal/common/eal_common_bus.c
>> > @@ -6,6 +6,7 @@
>> >  #include <string.h>
>> >  #include <sys/queue.h>
>> >  
>> > +#include <rte_eal.h>
>> >  #include <rte_bus.h>
>> >  #include <rte_debug.h>
>> >  #include <rte_string_fns.h>
>> > @@ -63,6 +64,11 @@ rte_bus_probe(void)
>> >  	int ret;
>> >  	struct rte_bus *bus, *vbus = NULL;
>> >  
>> > +	if (rte_eal_manual_probe()) {
>> > +		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
>> > +		return rte_dev_probe_devargs_list();
>> > +	}
>> > +
>> >  	TAILQ_FOREACH(bus, &rte_bus_list, next) {
>> >  		if (!strcmp(bus->name, "vdev")) {
>> >  			vbus = bus;
>> > diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
>> > index 9e4f09d83..3b7769a3c 100644
>> > --- a/lib/librte_eal/common/eal_common_dev.c
>> > +++ b/lib/librte_eal/common/eal_common_dev.c
>> > @@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
>> >  }
>> >  
>> >  int
>> > +rte_dev_probe_devargs_list(void)
>> > +{
>> > +	struct rte_device *dev;
>> > +	struct rte_devargs *da;
>> > +	int ret;
>> > +
>> > +	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
>> > +		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
>> > +		if (dev == NULL) {
>> > +			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
>> > +				da->name, da->bus->name);
>> > +			continue;
>> > +		}
>> > +
>> > +		if (rte_dev_is_probed(dev))
>> > +			continue;
>> > +
>> > +		if (dev->bus->plug == NULL) {
>> > +			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
>> > +					  "required by device %s\n",
>> > +				dev->bus->name, dev->name);
>> > +			continue;
>> > +		}
>> > +
>> > +		ret = dev->bus->plug(dev);
>> > +		/* Ignore positive return values, they are possibly
>> > +		 * triggered by blacklisted devices on the PCI bus. Probing
>> > +		 * should then continue.
>> > +		 */
>> > +		if (ret < 0 || !rte_dev_is_probed(dev)) {
>> > +			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
>> > +				dev->name);
>> > +			/* Fail on first real probe error. */
>> > +			return ret;
>> > +		}
>> > +	}
>> > +
>> > +	return 0;
>> > +}
>> > +
>> > +int
>> >  rte_eal_hotplug_add(const char *busname, const char *devname,
>> >  		    const char *drvargs)
>> >  {
>> > diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
>> > index 05cae5f75..66c232b14 100644
>> > --- a/lib/librte_eal/common/eal_common_options.c
>> > +++ b/lib/librte_eal/common/eal_common_options.c
>> > @@ -81,6 +81,7 @@ eal_long_options[] = {
>> >  	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
>> >  	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
>> >  	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
>> > +	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
>> >  	{0,                     0, NULL, 0                        }
>> >  };
>> >  
>> > @@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
>> >  			return -1;
>> >  		}
>> >  		break;
>> > +	case OPT_MANUAL_PROBE_NUM:
>> > +		conf->manual_probe = 1;
>> > +		break;
>> >  
>> >  	/* don't know what to do, leave this to caller */
>> >  	default:
>> > @@ -1634,6 +1638,10 @@ eal_common_usage(void)
>> >  	       "  --"OPT_VDEV"              Add a virtual device.\n"
>> >  	       "                      The argument format is <driver><id>[,key=val,...]\n"
>> >  	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
>> > +	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
>> > +	       "                      Disable probe step for all buses.\n"
>> > +	       "                      Devices will need to be probed using the hotplug API.\n"
>> > +	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
>> >  	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
>> >  	       "                      'va' for IOVA_VA\n"
>> >  	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
>> > diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
>> > index a42f34923..0006f903f 100644
>> > --- a/lib/librte_eal/common/eal_internal_cfg.h
>> > +++ b/lib/librte_eal/common/eal_internal_cfg.h
>> > @@ -44,6 +44,7 @@ struct internal_config {
>> >  	unsigned hugepage_unlink;         /**< true to unlink backing files */
>> >  	volatile unsigned no_pci;         /**< true to disable PCI */
>> >  	volatile unsigned no_hpet;        /**< true to disable HPET */
>> > +	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
>> >  	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
>> >  										* instead of native TSC */
>> >  	volatile unsigned no_shconf;      /**< true if there is no shared config */
>> > diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
>> > index 9855429e5..588fa32a6 100644
>> > --- a/lib/librte_eal/common/eal_options.h
>> > +++ b/lib/librte_eal/common/eal_options.h
>> > @@ -69,6 +69,8 @@ enum {
>> >  	OPT_IOVA_MODE_NUM,
>> >  #define OPT_MATCH_ALLOCATIONS  "match-allocations"
>> >  	OPT_MATCH_ALLOCATIONS_NUM,
>> > +#define OPT_MANUAL_PROBE "manual-probe"
>> > +	OPT_MANUAL_PROBE_NUM,
>> >  	OPT_LONG_MAX_NUM
>> >  };
>> >  
>> > diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
>> > index 798ede553..fd7ac8e37 100644
>> > --- a/lib/librte_eal/common/eal_private.h
>> > +++ b/lib/librte_eal/common/eal_private.h
>> > @@ -381,4 +381,13 @@ rte_option_init(void);
>> >  void
>> >  rte_option_usage(void);
>> >  
>> > +/**
>> > + * Go through the devargs list and probe everything in order.
>> > + *
>> > + * @return
>> > + *   0 on success, negative on error.
>> > + */
>> > +int
>> > +rte_dev_probe_devargs_list(void);
>> > +
>> >  #endif /* _EAL_PRIVATE_H_ */
>> > diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
>> > index b7cf91214..bd9087bcc 100644
>> > --- a/lib/librte_eal/common/include/rte_eal.h
>> > +++ b/lib/librte_eal/common/include/rte_eal.h
>> > @@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
>> >  int rte_eal_has_pci(void);
>> >  
>> >  /**
>> > + * Whether EAL probe is manual.
>> > + * Enabled by the --manual-probe option.
>> > + *
>> > + * When manual probing is enabled, batched bus probe of
>> > + * their devices is disabled. All devices need to be probed
>> > + * using the proper rte_dev API.
>> > + *
>> > + * In this mode, devices declared on the command line will
>> > + * be probed using the bus hotplug API. It is used to enforce
>> > + * a specific probe order.
>> > + *
>> > + * @return
>> > + *   Nonzero if manual device probing is enabled.
>> > + */
>> > +int rte_eal_manual_probe(void);
>> > +
>> > +/**
>> >   * Whether the EAL was asked to create UIO device.
>> >   *
>> >   * @return
>> > diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal/eal.c
>> > index d53f0fe69..2daa63b23 100644
>> > --- a/lib/librte_eal/freebsd/eal/eal.c
>> > +++ b/lib/librte_eal/freebsd/eal/eal.c
>> > @@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
>> >  	return !internal_config.no_pci;
>> >  }
>> >  
>> > +int rte_eal_manual_probe(void)
>> > +{
>> > +	return internal_config.manual_probe;
>> > +}
>> > +
>> >  int rte_eal_create_uio_dev(void)
>> >  {
>> >  	return internal_config.create_uio_dev;
>> > diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
>> > index 946222ccd..476fc2e4a 100644
>> > --- a/lib/librte_eal/linux/eal/eal.c
>> > +++ b/lib/librte_eal/linux/eal/eal.c
>> > @@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
>> >  	return internal_config.create_uio_dev;
>> >  }
>> >  
>> > +int rte_eal_manual_probe(void)
>> > +{
>> > +	return internal_config.manual_probe;
>> > +}
>> > +
>> >  enum rte_intr_mode
>> >  rte_eal_vfio_intr_mode(void)
>> >  {
>> > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
>> > index 7cbf82d37..fe5395c24 100644
>> > --- a/lib/librte_eal/rte_eal_version.map
>> > +++ b/lib/librte_eal/rte_eal_version.map
>> > @@ -312,6 +312,13 @@ DPDK_19.08 {
>> >  
>> >  } DPDK_19.05;
>> >  
>> > +DPDK_19.05 {
>> > +	global:
>> > +
>> > +	rte_eal_manual_probe
>> 
>> Syntax error - needs a ';' at the end of this symbol.
>> 
>
> Ah yes, sorry.
> I'm thinking there is a script or a validation tool using this, I should try it.

One method I use is to push to my personal github account, linked to
Travis-CI - then I get emails when it fails ;)

>> > +
>> > +} DPDK_19.11;
>> > +
>> >  EXPERIMENTAL {
>> >  	global:

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

[Gaëtan Rivet]

On Tue, Oct 01, 2019 at 03:19:42PM +0530, Jerin Jacob wrote:
> On Tue, Oct 1, 2019 at 2:40 PM Gaëtan Rivet <gaetan.rivet@6wind.com> wrote:
> >
> > On Mon, Sep 30, 2019 at 11:53:33AM -0700, Stephen Hemminger wrote:
> > > On Mon, 30 Sep 2019 14:51:03 +0200
> > > Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
> > >
> > > > Add a new EAL option enabling manual probing in the EAL.
> > > > This command line option will configure the EAL so that buses
> > > > will not trigger their probe step on their own.
> > > >
> > > > Applications are then expected to hotplug devices as they see fit.
> > > >
> > > > Devices declared on the command line by the user (using -w and --vdev),
> > > > will be probed using the hotplug API, in the order they are declared.
> > > >
> > > > This has the effect of offering a way for users to control probe order
> > > > of their devices, for drivers requiring it.
> > > >
> > > > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> > >
> > > I have no problems with the patch, but it would help if there was better
> > > way to handle device naming policy in DPDK. Applications that depend on
> > > particular port number are prone to get broken by changes in surrounding
> > > OS or hardware environment. Just like Linux applications that are built
> > > to depend on "eth0"; which is unfortunately all too common.
> >
> > Hello Stephen,
> >
> > This patch is a way to avoid having the PCI bus defining the probe order
> > with the current hardware environment. It seems to be a step in the
> > right direction for the issue you identify.
> >
> > There is a tight coupling between device names and driver matches for
> > the vdev bus, but that seems difficult to avoid.
> >
> > Do you see other EAL APIs fostering an over reliance of downstream
> > systems on device names?
> >
> > I pushed a few months back a way to iterate / match devices by their
> > properties. If you identify other pain points, this could certainly be
> > improved as well.
> 
> 
> And this mode will be kicked in only when "--manual-probe" selected on
> eal arguments.
> So it won't change the behavior of the existing applications.

If I read you correctly, if hardware independence is the proper way to function,
we should switch entirely to it.

I agree, but that means rewriting entirely the probe step of rte_bus.
This patch is a incremental step, I preferred to keep risks low.

It is not clear from your remark whether you are considering this
limitation a good or a bad thing however :)
Can you be a bit more explicit?

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

[Jerin Jacob]

On Tue, Oct 1, 2019 at 7:39 PM Gaëtan Rivet <gaetan.rivet@6wind.com> wrote:
>
> On Tue, Oct 01, 2019 at 03:19:42PM +0530, Jerin Jacob wrote:
> > On Tue, Oct 1, 2019 at 2:40 PM Gaėtan Rivet <gaetan.rivet@6wind.com> wrote:
> > >
> > > On Mon, Sep 30, 2019 at 11:53:33AM -0700, Stephen Hemminger wrote:
> > > > On Mon, 30 Sep 2019 14:51:03 +0200
> > > > Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
> > > >
> > > > > Add a new EAL option enabling manual probing in the EAL.
> > > > > This command line option will configure the EAL so that buses
> > > > > will not trigger their probe step on their own.
> > > > >
> > > > > Applications are then expected to hotplug devices as they see fit.
> > > > >
> > > > > Devices declared on the command line by the user (using -w and --vdev),
> > > > > will be probed using the hotplug API, in the order they are declared.
> > > > >
> > > > > This has the effect of offering a way for users to control probe order
> > > > > of their devices, for drivers requiring it.
> > > > >
> > > > > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> > > >
> > > > I have no problems with the patch, but it would help if there was better
> > > > way to handle device naming policy in DPDK. Applications that depend on
> > > > particular port number are prone to get broken by changes in surrounding
> > > > OS or hardware environment. Just like Linux applications that are built
> > > > to depend on "eth0"; which is unfortunately all too common.
> > >
> > > Hello Stephen,
> > >
> > > This patch is a way to avoid having the PCI bus defining the probe order
> > > with the current hardware environment. It seems to be a step in the
> > > right direction for the issue you identify.
> > >
> > > There is a tight coupling between device names and driver matches for
> > > the vdev bus, but that seems difficult to avoid.
> > >
> > > Do you see other EAL APIs fostering an over reliance of downstream
> > > systems on device names?
> > >
> > > I pushed a few months back a way to iterate / match devices by their
> > > properties. If you identify other pain points, this could certainly be
> > > improved as well.
> >
> >
> > And this mode will be kicked in only when "--manual-probe" selected on
> > eal arguments.
> > So it won't change the behavior of the existing applications.
>
> If I read you correctly, if hardware independence is the proper way to function,
> we should switch entirely to it.
>
> I agree, but that means rewriting entirely the probe step of rte_bus.
> This patch is a incremental step, I preferred to keep risks low.

+1

>
> It is not clear from your remark whether you are considering this
> limitation a good or a bad thing however :)

I am considering it as a good step.

> Can you be a bit more explicit?
>
> --
> Gaėtan Rivet
> 6WIND

[dpdk-dev] [PATCH v2] eal: add manual probing option

[Gaetan Rivet]

Add a new EAL option enabling manual probing in the EAL.
This command line option will configure the EAL so that buses
will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order
of their devices, for drivers requiring it.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---

I haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
        and that it works as intended for basic cases.
        Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

    Slava, are you able to test this patch?

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
 lib/librte_eal/linux/eal/eal.c             |  5 ++++
 lib/librte_eal/rte_eal_version.map         |  7 +++++
 11 files changed, 110 insertions(+)

diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index 27cfbd9e3..88d06e82e 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now propose to enforce manual probing of devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
index baa5b532a..145a96812 100644
--- a/lib/librte_eal/common/eal_common_bus.c
+++ b/lib/librte_eal/common/eal_common_bus.c
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 9e4f09d83..f4ce1c56c 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
 }
 
 int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
 rte_eal_hotplug_add(const char *busname, const char *devname,
 		    const char *drvargs)
 {
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 05cae5f75..66c232b14 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index a42f34923..0006f903f 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index 9855429e5..588fa32a6 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 798ede553..fd7ac8e37 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index b7cf91214..bd9087bcc 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option.
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ */
+int rte_eal_manual_probe(void);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal/eal.c
index d53f0fe69..2daa63b23 100644
--- a/lib/librte_eal/freebsd/eal/eal.c
+++ b/lib/librte_eal/freebsd/eal/eal.c
@@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
 	return !internal_config.no_pci;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 int rte_eal_create_uio_dev(void)
 {
 	return internal_config.create_uio_dev;
diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
index 946222ccd..476fc2e4a 100644
--- a/lib/librte_eal/linux/eal/eal.c
+++ b/lib/librte_eal/linux/eal/eal.c
@@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
 	return internal_config.create_uio_dev;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 enum rte_intr_mode
 rte_eal_vfio_intr_mode(void)
 {
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index 7cbf82d37..77f34542b 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -312,6 +312,13 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.11 {
+	global:
+
+	rte_eal_manual_probe;
+
+} DPDK_19.05;
+
 EXPERIMENTAL {
 	global:
 
-- 
2.11.0

[dpdk-dev] [PATCH v3] eal: add manual probing option

[Gaetan Rivet]

Add a new EAL option enabling manual probing in the EAL.
This command line option will configure the EAL so that buses
will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order
of their devices, for drivers requiring it.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---

I haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
        and that it works as intended for basic cases.
        Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

    Slava, are you able to test this patch?

v3: properly fixed the map file (herited 19.08 instead of 19.05).

    Added a function to set the probe manual from the application,
    without having the user do it from the command line.

    Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

  Currently manual-probe will cut off probing from all buses.
  It could be interesting to be able to only cut buses supporting hotplug,
  given that they are the one able to probe devices afterward.

  No real use-case for this right now, so leaving as-is. Might be worth
  considering in the future.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 34 +++++++++++++++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           | 10 ++++++++
 lib/librte_eal/linux/eal/eal.c             | 10 ++++++++
 lib/librte_eal/rte_eal_version.map         |  8 ++++++
 11 files changed, 138 insertions(+)

diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index 27cfbd9e3..700f9a726 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow manual probing devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
index baa5b532a..145a96812 100644
--- a/lib/librte_eal/common/eal_common_bus.c
+++ b/lib/librte_eal/common/eal_common_bus.c
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 9e4f09d83..f4ce1c56c 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
 }
 
 int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
 rte_eal_hotplug_add(const char *busname, const char *devname,
 		    const char *drvargs)
 {
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 05cae5f75..66c232b14 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index a42f34923..0006f903f 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index 9855429e5..588fa32a6 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 798ede553..fd7ac8e37 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index b7cf91214..da86fdb6b 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -465,6 +465,40 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+void rte_eal_manual_probe_set(int enabled);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal/eal.c
index d53f0fe69..cc6d846bd 100644
--- a/lib/librte_eal/freebsd/eal/eal.c
+++ b/lib/librte_eal/freebsd/eal/eal.c
@@ -972,6 +972,16 @@ rte_eal_vfio_intr_mode(void)
 	return RTE_INTR_MODE_NONE;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
 int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
 		      __rte_unused const char *dev_addr,
 		      __rte_unused int *vfio_dev_fd,
diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
index 946222ccd..da00eb14d 100644
--- a/lib/librte_eal/linux/eal/eal.c
+++ b/lib/librte_eal/linux/eal/eal.c
@@ -1382,6 +1382,16 @@ rte_eal_vfio_intr_mode(void)
 	return internal_config.vfio_intr_mode;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
 int
 rte_eal_check_module(const char *module_name)
 {
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index 7cbf82d37..ccc4ffb21 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -312,6 +312,14 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.11 {
+	global:
+
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
+
+} DPDK_19.08;
+
 EXPERIMENTAL {
 	global:
 
-- 
2.11.0

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

[Vamsi Krishna Attunuru]

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Gaetan Rivet
> Sent: Friday, October 4, 2019 6:25 PM
> To: dev@dpdk.org; Vamsi Krishna Attunuru <vattunuru@marvell.com>
> Cc: Gaetan Rivet <gaetan.rivet@6wind.com>; Ferruh Yigit
> <ferruh.yigit@intel.com>; Anatoly Burakov <anatoly.burakov@intel.com>;
> Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; David Marchand <david.marchand@redhat.com>
> Subject: [dpdk-dev] [PATCH v3] eal: add manual probing option
> 
> Add a new EAL option enabling manual probing in the EAL.
> This command line option will configure the EAL so that buses will not trigger
> their probe step on their own.
> 
> Applications are then expected to hotplug devices as they see fit.
> 
> Devices declared on the command line by the user (using -w and --vdev), will
> be probed using the hotplug API, in the order they are declared.
> 
> This has the effect of offering a way for users to control probe order of their
> devices, for drivers requiring it.
> 
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> ---
> 
> I haven't heard many opinions on the matter, please shout if you see an
> issue with this approach.
> 
> @Slava: I have tested rather quickly that it does not break anything,
>         and that it works as intended for basic cases.
>         Can you test it further for your use-case and tell me if it works fine?
> 
> Beyond the obvious difference between both probe mode, something to
> keep in mind:
> while using -w on invalid devices would not block (PCI) bus probing, it will
> stop manual probing in its track. All devices need to exist and be valid device
> IDs.
> 
> v2: fixed a few typos, map file (and used Travis to validate).
> 
>     Slava, are you able to test this patch?
> 
> v3: properly fixed the map file (herited 19.08 instead of 19.05).
> 
>     Added a function to set the probe manual from the application,
>     without having the user do it from the command line.
> 
>     Stopped spamming Slava about it, Vamsi was actually the one interested in
> it!

@Gaetan: Thanks a lot for providing this patch. I have tested it in our use cases
and it works as expected, also verified multiple combinations by passing
whitelisted pdevs, vdevs and mix of both. Probing sequence happens as required.

Acked-by : Vamsi Attunuru < vattunuru@marvell.com >
Tested-by: Vamsi Attunuru < vattunuru@marvell.com >

> 
> Standing issue worth chiming in:
> 
>   Currently manual-probe will cut off probing from all buses.
>   It could be interesting to be able to only cut buses supporting hotplug,
>   given that they are the one able to probe devices afterward.
> 
>   No real use-case for this right now, so leaving as-is. Might be worth
>   considering in the future.
> 
>  doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
>  lib/librte_eal/common/eal_common_bus.c     |  6 +++++
>  lib/librte_eal/common/eal_common_dev.c     | 41
> ++++++++++++++++++++++++++++++
>  lib/librte_eal/common/eal_common_options.c |  8 ++++++
>  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
>  lib/librte_eal/common/eal_options.h        |  2 ++
>  lib/librte_eal/common/eal_private.h        |  9 +++++++
>  lib/librte_eal/common/include/rte_eal.h    | 34
> +++++++++++++++++++++++++
>  lib/librte_eal/freebsd/eal/eal.c           | 10 ++++++++
>  lib/librte_eal/linux/eal/eal.c             | 10 ++++++++
>  lib/librte_eal/rte_eal_version.map         |  8 ++++++
>  11 files changed, 138 insertions(+)
> 
> diff --git a/doc/guides/rel_notes/release_19_11.rst
> b/doc/guides/rel_notes/release_19_11.rst
> index 27cfbd9e3..700f9a726 100644
> --- a/doc/guides/rel_notes/release_19_11.rst
> +++ b/doc/guides/rel_notes/release_19_11.rst
> @@ -56,6 +56,15 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =========================================================
> 
> +* **EAL will now allow manual probing devices.**
> +
> +  Previously, a user could not force an order when probing declared devices.
> +  This could cause issues for drivers depending on another device being
> present.
> +  A new option ``--manual-probe`` is now available to do just that.
> +  This new option relies on the device bus supporting hotplug. It can
> + also be used to disable automatic probing from the ``PCI`` bus without
> + having to disable the whole bus.
> +
> 
>  Removed Items
>  -------------
> diff --git a/lib/librte_eal/common/eal_common_bus.c
> b/lib/librte_eal/common/eal_common_bus.c
> index baa5b532a..145a96812 100644
> --- a/lib/librte_eal/common/eal_common_bus.c
> +++ b/lib/librte_eal/common/eal_common_bus.c
> @@ -6,6 +6,7 @@
>  #include <string.h>
>  #include <sys/queue.h>
> 
> +#include <rte_eal.h>
>  #include <rte_bus.h>
>  #include <rte_debug.h>
>  #include <rte_string_fns.h>
> @@ -63,6 +64,11 @@ rte_bus_probe(void)
>  	int ret;
>  	struct rte_bus *bus, *vbus = NULL;
> 
> +	if (rte_eal_manual_probe()) {
> +		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
> +		return rte_dev_probe_devargs_list();
> +	}
> +
>  	TAILQ_FOREACH(bus, &rte_bus_list, next) {
>  		if (!strcmp(bus->name, "vdev")) {
>  			vbus = bus;
> diff --git a/lib/librte_eal/common/eal_common_dev.c
> b/lib/librte_eal/common/eal_common_dev.c
> index 9e4f09d83..f4ce1c56c 100644
> --- a/lib/librte_eal/common/eal_common_dev.c
> +++ b/lib/librte_eal/common/eal_common_dev.c
> @@ -109,6 +109,47 @@ build_devargs(const char *busname, const char
> *devname,  }
> 
>  int
> +rte_dev_probe_devargs_list(void)
> +{
> +	struct rte_device *dev;
> +	struct rte_devargs *da;
> +	int ret;
> +
> +	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
> +		dev = da->bus->find_device(NULL, cmp_dev_name, da-
> >name);
> +		if (dev == NULL) {
> +			RTE_LOG(ERR, EAL, "Unable to find device %s on bus
> %s\n",
> +				da->name, da->bus->name);
> +			continue;
> +		}
> +
> +		if (rte_dev_is_probed(dev))
> +			continue;
> +
> +		if (dev->bus->plug == NULL) {
> +			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not
> supported by bus %s, "
> +					  "required by device %s\n",
> +				dev->bus->name, dev->name);
> +			continue;
> +		}
> +
> +		ret = dev->bus->plug(dev);
> +		/* Ignore positive return values, they are possibly
> +		 * triggered by blacklisted devices on the PCI bus. Probing
> +		 * should then continue.
> +		 */
> +		if (ret < 0) {
> +			RTE_LOG(ERR, EAL, "Driver cannot attach device
> %s\n",
> +				dev->name);
> +			/* Fail on first real probe error. */
> +			return ret;
> +		}
> +	}
> +
> +	return 0;
> +}
> +
> +int
>  rte_eal_hotplug_add(const char *busname, const char *devname,
>  		    const char *drvargs)
>  {
> diff --git a/lib/librte_eal/common/eal_common_options.c
> b/lib/librte_eal/common/eal_common_options.c
> index 05cae5f75..66c232b14 100644
> --- a/lib/librte_eal/common/eal_common_options.c
> +++ b/lib/librte_eal/common/eal_common_options.c
> @@ -81,6 +81,7 @@ eal_long_options[] = {
>  	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
>  	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL,
> OPT_SINGLE_FILE_SEGMENTS_NUM},
>  	{OPT_MATCH_ALLOCATIONS, 0, NULL,
> OPT_MATCH_ALLOCATIONS_NUM},
> +	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
>  	{0,                     0, NULL, 0                        }
>  };
> 
> @@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char
> *optarg,
>  			return -1;
>  		}
>  		break;
> +	case OPT_MANUAL_PROBE_NUM:
> +		conf->manual_probe = 1;
> +		break;
> 
>  	/* don't know what to do, leave this to caller */
>  	default:
> @@ -1634,6 +1638,10 @@ eal_common_usage(void)
>  	       "  --"OPT_VDEV"              Add a virtual device.\n"
>  	       "                      The argument format is <driver><id>[,key=val,...]\n"
>  	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
> +	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
> +	       "                      Disable probe step for all buses.\n"
> +	       "                      Devices will need to be probed using the hotplug
> API.\n"
> +	       "                      PCI and vdev declarations will be treated in order as
> hotplug commands.\n"
>  	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
>  	       "                      'va' for IOVA_VA\n"
>  	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
> diff --git a/lib/librte_eal/common/eal_internal_cfg.h
> b/lib/librte_eal/common/eal_internal_cfg.h
> index a42f34923..0006f903f 100644
> --- a/lib/librte_eal/common/eal_internal_cfg.h
> +++ b/lib/librte_eal/common/eal_internal_cfg.h
> @@ -44,6 +44,7 @@ struct internal_config {
>  	unsigned hugepage_unlink;         /**< true to unlink backing files */
>  	volatile unsigned no_pci;         /**< true to disable PCI */
>  	volatile unsigned no_hpet;        /**< true to disable HPET */
> +	volatile unsigned manual_probe;   /**< true to enable manual device
> probing. */
>  	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC
> mapping
> 
> 	* instead of native TSC */
>  	volatile unsigned no_shconf;      /**< true if there is no shared config
> */
> diff --git a/lib/librte_eal/common/eal_options.h
> b/lib/librte_eal/common/eal_options.h
> index 9855429e5..588fa32a6 100644
> --- a/lib/librte_eal/common/eal_options.h
> +++ b/lib/librte_eal/common/eal_options.h
> @@ -69,6 +69,8 @@ enum {
>  	OPT_IOVA_MODE_NUM,
>  #define OPT_MATCH_ALLOCATIONS  "match-allocations"
>  	OPT_MATCH_ALLOCATIONS_NUM,
> +#define OPT_MANUAL_PROBE "manual-probe"
> +	OPT_MANUAL_PROBE_NUM,
>  	OPT_LONG_MAX_NUM
>  };
> 
> diff --git a/lib/librte_eal/common/eal_private.h
> b/lib/librte_eal/common/eal_private.h
> index 798ede553..fd7ac8e37 100644
> --- a/lib/librte_eal/common/eal_private.h
> +++ b/lib/librte_eal/common/eal_private.h
> @@ -381,4 +381,13 @@ rte_option_init(void);  void  rte_option_usage(void);
> 
> +/**
> + * Go through the devargs list and probe everything in order.
> + *
> + * @return
> + *   0 on success, negative on error.
> + */
> +int
> +rte_dev_probe_devargs_list(void);
> +
>  #endif /* _EAL_PRIVATE_H_ */
> diff --git a/lib/librte_eal/common/include/rte_eal.h
> b/lib/librte_eal/common/include/rte_eal.h
> index b7cf91214..da86fdb6b 100644
> --- a/lib/librte_eal/common/include/rte_eal.h
> +++ b/lib/librte_eal/common/include/rte_eal.h
> @@ -465,6 +465,40 @@ int rte_eal_has_hugepages(void);  int
> rte_eal_has_pci(void);
> 
>  /**
> + * Whether EAL probe is manual.
> + * Enabled by the --manual-probe option or by
> + * using rte_eal_manual_probe_set().
> + *
> + * When manual probing is enabled, batched bus probe of
> + * their devices is disabled. All devices need to be probed
> + * using the proper rte_dev API.
> + *
> + * In this mode, devices declared on the command line will
> + * be probed using the bus hotplug API. It is used to enforce
> + * a specific probe order.
> + *
> + * @return
> + *   Nonzero if manual device probing is enabled.
> + *
> + * @see rte_eal_manual_probe_set
> + */
> +int rte_eal_manual_probe(void);
> +
> +/**
> + * Configure EAL probe mode -- manual or automatic.
> + *
> + * Enable or disable manual probe mode in EAL.
> + * This function can be called at any time, but must be used
> + * before calling rte_eal_init() to have any effect.
> + *
> + * @param enabled
> + *   zero to disable manual probe, non-zero to enable it.
> + *
> + * @see rte_eal_manual_probe
> + */
> +void rte_eal_manual_probe_set(int enabled);
> +
> +/**
>   * Whether the EAL was asked to create UIO device.
>   *
>   * @return
> diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal/eal.c
> index d53f0fe69..cc6d846bd 100644
> --- a/lib/librte_eal/freebsd/eal/eal.c
> +++ b/lib/librte_eal/freebsd/eal/eal.c
> @@ -972,6 +972,16 @@ rte_eal_vfio_intr_mode(void)
>  	return RTE_INTR_MODE_NONE;
>  }
> 
> +int rte_eal_manual_probe(void)
> +{
> +	return internal_config.manual_probe;
> +}
> +
> +void rte_eal_manual_probe_set(int enabled) {
> +	internal_config.manual_probe = !!enabled; }
> +
>  int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
>  		      __rte_unused const char *dev_addr,
>  		      __rte_unused int *vfio_dev_fd,
> diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c index
> 946222ccd..da00eb14d 100644
> --- a/lib/librte_eal/linux/eal/eal.c
> +++ b/lib/librte_eal/linux/eal/eal.c
> @@ -1382,6 +1382,16 @@ rte_eal_vfio_intr_mode(void)
>  	return internal_config.vfio_intr_mode;  }
> 
> +int rte_eal_manual_probe(void)
> +{
> +	return internal_config.manual_probe;
> +}
> +
> +void rte_eal_manual_probe_set(int enabled) {
> +	internal_config.manual_probe = !!enabled; }
> +
>  int
>  rte_eal_check_module(const char *module_name)  { diff --git
> a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> index 7cbf82d37..ccc4ffb21 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -312,6 +312,14 @@ DPDK_19.08 {
> 
>  } DPDK_19.05;
> 
> +DPDK_19.11 {
> +	global:
> +
> +	rte_eal_manual_probe;
> +	rte_eal_manual_probe_set;
> +
> +} DPDK_19.08;
> +
>  EXPERIMENTAL {
>  	global:
> 
> --
> 2.11.0

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

[Gaëtan Rivet]

Hi,

On Mon, Oct 07, 2019 at 01:27:48AM +0000, Vamsi Krishna Attunuru wrote:
> 
> 
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org> On Behalf Of Gaetan Rivet
> > Sent: Friday, October 4, 2019 6:25 PM
> > To: dev@dpdk.org; Vamsi Krishna Attunuru <vattunuru@marvell.com>
> > Cc: Gaetan Rivet <gaetan.rivet@6wind.com>; Ferruh Yigit
> > <ferruh.yigit@intel.com>; Anatoly Burakov <anatoly.burakov@intel.com>;
> > Thomas Monjalon <thomas@monjalon.net>; Jerin Jacob Kollanukkaran
> > <jerinj@marvell.com>; David Marchand <david.marchand@redhat.com>
> > Subject: [dpdk-dev] [PATCH v3] eal: add manual probing option
> > 
> > Add a new EAL option enabling manual probing in the EAL.
> > This command line option will configure the EAL so that buses will not trigger
> > their probe step on their own.
> > 
> > Applications are then expected to hotplug devices as they see fit.
> > 
> > Devices declared on the command line by the user (using -w and --vdev), will
> > be probed using the hotplug API, in the order they are declared.
> > 
> > This has the effect of offering a way for users to control probe order of their
> > devices, for drivers requiring it.
> > 
> > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> > ---
> > 
> > I haven't heard many opinions on the matter, please shout if you see an
> > issue with this approach.
> > 
> > @Slava: I have tested rather quickly that it does not break anything,
> >         and that it works as intended for basic cases.
> >         Can you test it further for your use-case and tell me if it works fine?
> > 
> > Beyond the obvious difference between both probe mode, something to
> > keep in mind:
> > while using -w on invalid devices would not block (PCI) bus probing, it will
> > stop manual probing in its track. All devices need to exist and be valid device
> > IDs.
> > 
> > v2: fixed a few typos, map file (and used Travis to validate).
> > 
> >     Slava, are you able to test this patch?
> > 
> > v3: properly fixed the map file (herited 19.08 instead of 19.05).
> > 
> >     Added a function to set the probe manual from the application,
> >     without having the user do it from the command line.
> > 
> >     Stopped spamming Slava about it, Vamsi was actually the one interested in
> > it!
> 
> @Gaetan: Thanks a lot for providing this patch. I have tested it in our use cases
> and it works as expected, also verified multiple combinations by passing
> whitelisted pdevs, vdevs and mix of both. Probing sequence happens as required.
> 
> Acked-by : Vamsi Attunuru < vattunuru@marvell.com >
> Tested-by: Vamsi Attunuru < vattunuru@marvell.com >
> 
> > 
> > Standing issue worth chiming in:
> > 
> >   Currently manual-probe will cut off probing from all buses.
> >   It could be interesting to be able to only cut buses supporting hotplug,
> >   given that they are the one able to probe devices afterward.
> > 
> >   No real use-case for this right now, so leaving as-is. Might be worth
> >   considering in the future.
> > 

Any further comment for this patch? Can someone review it?

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

[Jerin Jacob]

On Fri, Oct 4, 2019 at 6:25 PM Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
>
> Add a new EAL option enabling manual probing in the EAL.
> This command line option will configure the EAL so that buses
> will not trigger their probe step on their own.
>
> Applications are then expected to hotplug devices as they see fit.
>
> Devices declared on the command line by the user (using -w and --vdev),
> will be probed using the hotplug API, in the order they are declared.
>
> This has the effect of offering a way for users to control probe order
> of their devices, for drivers requiring it.
>
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> ---
>
> I haven't heard many opinions on the matter, please shout if you see an issue
> with this approach.
>
> @Slava: I have tested rather quickly that it does not break anything,
>         and that it works as intended for basic cases.
>         Can you test it further for your use-case and tell me if it works fine?
>
> Beyond the obvious difference between both probe mode, something to keep in mind:
> while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
> probing in its track. All devices need to exist and be valid device IDs.
>
> v2: fixed a few typos, map file (and used Travis to validate).
>
>     Slava, are you able to test this patch?
>
> v3: properly fixed the map file (herited 19.08 instead of 19.05).
>
>     Added a function to set the probe manual from the application,
>     without having the user do it from the command line.
>
>     Stopped spamming Slava about it, Vamsi was actually the one interested in it!
>
> Standing issue worth chiming in:
>
>   Currently manual-probe will cut off probing from all buses.
>   It could be interesting to be able to only cut buses supporting hotplug,
>   given that they are the one able to probe devices afterward.
>
>   No real use-case for this right now, so leaving as-is. Might be worth
>   considering in the future.
>
>  doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
>  lib/librte_eal/common/eal_common_bus.c     |  6 +++++
>  lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
>  lib/librte_eal/common/eal_common_options.c |  8 ++++++
>  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
>  lib/librte_eal/common/eal_options.h        |  2 ++
>  lib/librte_eal/common/eal_private.h        |  9 +++++++
>  lib/librte_eal/common/include/rte_eal.h    | 34 +++++++++++++++++++++++++
>  lib/librte_eal/freebsd/eal/eal.c           | 10 ++++++++
>  lib/librte_eal/linux/eal/eal.c             | 10 ++++++++
>  lib/librte_eal/rte_eal_version.map         |  8 ++++++
>  11 files changed, 138 insertions(+)
>
> diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> index 27cfbd9e3..700f9a726 100644
> --- a/doc/guides/rel_notes/release_19_11.rst
> +++ b/doc/guides/rel_notes/release_19_11.rst
> @@ -56,6 +56,15 @@ New Features
>       Also, make sure to start the actual text at the margin.
>       =========================================================
>
> +* **EAL will now allow manual probing devices.**
> +
> +  Previously, a user could not force an order when probing declared devices.
> +  This could cause issues for drivers depending on another device being present.
> +  A new option ``--manual-probe`` is now available to do just that.
> +  This new option relies on the device bus supporting hotplug. It can
> +  also be used to disable automatic probing from the ``PCI`` bus without
> +  having to disable the whole bus.
> +
>
>  Removed Items
>  -------------
> diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
> index baa5b532a..145a96812 100644
> --- a/lib/librte_eal/common/eal_common_bus.c
> +++ b/lib/librte_eal/common/eal_common_bus.c
> @@ -6,6 +6,7 @@
>  #include <string.h>
>  #include <sys/queue.h>
>
> +#include <rte_eal.h>
>  #include <rte_bus.h>
>  #include <rte_debug.h>
>  #include <rte_string_fns.h>
> @@ -63,6 +64,11 @@ rte_bus_probe(void)
>         int ret;
>         struct rte_bus *bus, *vbus = NULL;
>
> +       if (rte_eal_manual_probe()) {

See below,



>  int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
>                       __rte_unused const char *dev_addr,
>                       __rte_unused int *vfio_dev_fd,
> diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
> index 946222ccd..da00eb14d 100644
> --- a/lib/librte_eal/linux/eal/eal.c
> +++ b/lib/librte_eal/linux/eal/eal.c
> @@ -1382,6 +1382,16 @@ rte_eal_vfio_intr_mode(void)
>         return internal_config.vfio_intr_mode;
>  }
>
> +int rte_eal_manual_probe(void)
> +{
> +       return internal_config.manual_probe;
> +}
> +
> +void rte_eal_manual_probe_set(int enabled)
> +{
> +       internal_config.manual_probe = !!enabled;
> +}

I don't think FreeBSD and Linux specific implementation is NOT required
as internal_config accessible in eal/common. example diff.

[master][dpdk.org] $ git diff
diff --git a/lib/librte_eal/common/eal_common_bus.c
b/lib/librte_eal/common/eal_common_bus.c
index 145a96812..db4257cf1 100644
--- a/lib/librte_eal/common/eal_common_bus.c
+++ b/lib/librte_eal/common/eal_common_bus.c
@@ -13,6 +13,7 @@
 #include <rte_errno.h>

 #include "eal_private.h"
+#include "eal_internal_cfg.h"

 static struct rte_bus_list rte_bus_list =
        TAILQ_HEAD_INITIALIZER(rte_bus_list);
@@ -64,7 +65,7 @@ rte_bus_probe(void)
        int ret;
        struct rte_bus *bus, *vbus = NULL;

-       if (rte_eal_manual_probe()) {
+       if (internal_config.manual_probe) {
                RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
                return rte_dev_probe_devargs_list();
        }

>  int
>  rte_eal_check_module(const char *module_name)
>  {
> diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> index 7cbf82d37..ccc4ffb21 100644
> --- a/lib/librte_eal/rte_eal_version.map
> +++ b/lib/librte_eal/rte_eal_version.map
> @@ -312,6 +312,14 @@ DPDK_19.08 {
>
>  } DPDK_19.05;
>
> +DPDK_19.11 {
> +       global:
> +
> +       rte_eal_manual_probe;
> +       rte_eal_manual_probe_set;

Do we need public API for this? it is only used by eal lib.

checkpatch complains the following too.
ERROR: symbol rte_eal_manual_probe_set is added in the DPDK_19.11
section, but is expected to be added in the EXPERIMENTAL section of
the version map

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

[Gaëtan Rivet]

On Fri, Oct 25, 2019 at 05:29:12PM +0530, Jerin Jacob wrote:
> On Fri, Oct 4, 2019 at 6:25 PM Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
> >
> > Add a new EAL option enabling manual probing in the EAL.
> > This command line option will configure the EAL so that buses
> > will not trigger their probe step on their own.
> >
> > Applications are then expected to hotplug devices as they see fit.
> >
> > Devices declared on the command line by the user (using -w and --vdev),
> > will be probed using the hotplug API, in the order they are declared.
> >
> > This has the effect of offering a way for users to control probe order
> > of their devices, for drivers requiring it.
> >
> > Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> > ---
> >
> > I haven't heard many opinions on the matter, please shout if you see an issue
> > with this approach.
> >
> > @Slava: I have tested rather quickly that it does not break anything,
> >         and that it works as intended for basic cases.
> >         Can you test it further for your use-case and tell me if it works fine?
> >
> > Beyond the obvious difference between both probe mode, something to keep in mind:
> > while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
> > probing in its track. All devices need to exist and be valid device IDs.
> >
> > v2: fixed a few typos, map file (and used Travis to validate).
> >
> >     Slava, are you able to test this patch?
> >
> > v3: properly fixed the map file (herited 19.08 instead of 19.05).
> >
> >     Added a function to set the probe manual from the application,
> >     without having the user do it from the command line.
> >
> >     Stopped spamming Slava about it, Vamsi was actually the one interested in it!
> >
> > Standing issue worth chiming in:
> >
> >   Currently manual-probe will cut off probing from all buses.
> >   It could be interesting to be able to only cut buses supporting hotplug,
> >   given that they are the one able to probe devices afterward.
> >
> >   No real use-case for this right now, so leaving as-is. Might be worth
> >   considering in the future.
> >
> >  doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
> >  lib/librte_eal/common/eal_common_bus.c     |  6 +++++
> >  lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
> >  lib/librte_eal/common/eal_common_options.c |  8 ++++++
> >  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
> >  lib/librte_eal/common/eal_options.h        |  2 ++
> >  lib/librte_eal/common/eal_private.h        |  9 +++++++
> >  lib/librte_eal/common/include/rte_eal.h    | 34 +++++++++++++++++++++++++
> >  lib/librte_eal/freebsd/eal/eal.c           | 10 ++++++++
> >  lib/librte_eal/linux/eal/eal.c             | 10 ++++++++
> >  lib/librte_eal/rte_eal_version.map         |  8 ++++++
> >  11 files changed, 138 insertions(+)
> >
> > diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
> > index 27cfbd9e3..700f9a726 100644
> > --- a/doc/guides/rel_notes/release_19_11.rst
> > +++ b/doc/guides/rel_notes/release_19_11.rst
> > @@ -56,6 +56,15 @@ New Features
> >       Also, make sure to start the actual text at the margin.
> >       =========================================================
> >
> > +* **EAL will now allow manual probing devices.**
> > +
> > +  Previously, a user could not force an order when probing declared devices.
> > +  This could cause issues for drivers depending on another device being present.
> > +  A new option ``--manual-probe`` is now available to do just that.
> > +  This new option relies on the device bus supporting hotplug. It can
> > +  also be used to disable automatic probing from the ``PCI`` bus without
> > +  having to disable the whole bus.
> > +
> >
> >  Removed Items
> >  -------------
> > diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
> > index baa5b532a..145a96812 100644
> > --- a/lib/librte_eal/common/eal_common_bus.c
> > +++ b/lib/librte_eal/common/eal_common_bus.c
> > @@ -6,6 +6,7 @@
> >  #include <string.h>
> >  #include <sys/queue.h>
> >
> > +#include <rte_eal.h>
> >  #include <rte_bus.h>
> >  #include <rte_debug.h>
> >  #include <rte_string_fns.h>
> > @@ -63,6 +64,11 @@ rte_bus_probe(void)
> >         int ret;
> >         struct rte_bus *bus, *vbus = NULL;
> >
> > +       if (rte_eal_manual_probe()) {
> 
> See below,
> 
> 
> 
> >  int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
> >                       __rte_unused const char *dev_addr,
> >                       __rte_unused int *vfio_dev_fd,
> > diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c
> > index 946222ccd..da00eb14d 100644
> > --- a/lib/librte_eal/linux/eal/eal.c
> > +++ b/lib/librte_eal/linux/eal/eal.c
> > @@ -1382,6 +1382,16 @@ rte_eal_vfio_intr_mode(void)
> >         return internal_config.vfio_intr_mode;
> >  }
> >
> > +int rte_eal_manual_probe(void)
> > +{
> > +       return internal_config.manual_probe;
> > +}
> > +
> > +void rte_eal_manual_probe_set(int enabled)
> > +{
> > +       internal_config.manual_probe = !!enabled;
> > +}
> 
> I don't think FreeBSD and Linux specific implementation is NOT required
> as internal_config accessible in eal/common. example diff.
> 
> [master][dpdk.org] $ git diff
> diff --git a/lib/librte_eal/common/eal_common_bus.c
> b/lib/librte_eal/common/eal_common_bus.c
> index 145a96812..db4257cf1 100644
> --- a/lib/librte_eal/common/eal_common_bus.c
> +++ b/lib/librte_eal/common/eal_common_bus.c
> @@ -13,6 +13,7 @@
>  #include <rte_errno.h>
> 
>  #include "eal_private.h"
> +#include "eal_internal_cfg.h"
> 
>  static struct rte_bus_list rte_bus_list =
>         TAILQ_HEAD_INITIALIZER(rte_bus_list);
> @@ -64,7 +65,7 @@ rte_bus_probe(void)
>         int ret;
>         struct rte_bus *bus, *vbus = NULL;
> 
> -       if (rte_eal_manual_probe()) {
> +       if (internal_config.manual_probe) {
>                 RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
>                 return rte_dev_probe_devargs_list();
>         }
> 
> >  int
> >  rte_eal_check_module(const char *module_name)
> >  {
> > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> > index 7cbf82d37..ccc4ffb21 100644
> > --- a/lib/librte_eal/rte_eal_version.map
> > +++ b/lib/librte_eal/rte_eal_version.map
> > @@ -312,6 +312,14 @@ DPDK_19.08 {
> >
> >  } DPDK_19.05;
> >
> > +DPDK_19.11 {
> > +       global:
> > +
> > +       rte_eal_manual_probe;
> > +       rte_eal_manual_probe_set;
> 
> Do we need public API for this? it is only used by eal lib.
> 

I think some PMDs and some applications would need to issue warnings to
the user if manual probing is not enabled (i.e. a port representor
parameter was used in a PMD, but probing is automatic --> on some
platform it would work and others it would fail with the same config).

Some applications would also prefer forcing it enabled without having to
use the option on the command line.

This motivates the public API. Given that there is a public API, I think
it's better to use it even from within the EAL.

However I agree that it does not need two separate implementations. I
could move it to some eal_common_* part probably, maybe bus or dev?

> checkpatch complains the following too.
> ERROR: symbol rte_eal_manual_probe_set is added in the DPDK_19.11
> section, but is expected to be added in the EXPERIMENTAL section of
> the version map

The API is simple enough IMO to make it immediately stable: it will
allow applications forbidding the use of experimental API to make use of
it.

This is only for the sake of simplicity for some users. I'm not
opposed to making it experimental first if it's expected for all new
APIs.

-- 
Gaëtan Rivet
6WIND

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

[Jerin Jacob]

> > >  rte_eal_check_module(const char *module_name)
> > >  {
> > > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
> > > index 7cbf82d37..ccc4ffb21 100644
> > > --- a/lib/librte_eal/rte_eal_version.map
> > > +++ b/lib/librte_eal/rte_eal_version.map
> > > @@ -312,6 +312,14 @@ DPDK_19.08 {
> > >
> > >  } DPDK_19.05;
> > >
> > > +DPDK_19.11 {
> > > +       global:
> > > +
> > > +       rte_eal_manual_probe;
> > > +       rte_eal_manual_probe_set;
> >
> > Do we need public API for this? it is only used by eal lib.
> >
>
> I think some PMDs and some applications would need to issue warnings to
> the user if manual probing is not enabled (i.e. a port representor
> parameter was used in a PMD, but probing is automatic --> on some
> platform it would work and others it would fail with the same config).
>
> Some applications would also prefer forcing it enabled without having to
> use the option on the command line.
>
> This motivates the public API. Given that there is a public API, I think
 it's better to use it even from within the EAL.

No issue from my side to expose through public API as well.
Please rte_eal_manual_probe_set() when setting in flags in eal as well.
Currently rte_eal_manual_probe_set() not used anywhere.

>
> However I agree that it does not need two separate implementations. I
> could move it to some eal_common_* part probably, maybe bus or dev?

eal should be fine. No strong opinion on the place.

>
> > checkpatch complains the following too.
> > ERROR: symbol rte_eal_manual_probe_set is added in the DPDK_19.11
> > section, but is expected to be added in the EXPERIMENTAL section of
> > the version map
>
> The API is simple enough IMO to make it immediately stable: it will
> allow applications forbidding the use of experimental API to make use of
> it.
>
> This is only for the sake of simplicity for some users. I'm not
> opposed to making it experimental first if it's expected for all new
> APIs.

All new APIs are treated as experimental. I don't think, we need to
make an exception here.

Other these nits, This patch looks good.

>
> --
> Gaėtan Rivet
> 6WIND

[dpdk-dev] [PATCH v4] eal: add manual probing option

[Gaetan Rivet]

Add a new EAL option enabling manual probing in the EAL.
This command line option will configure the EAL so that buses
will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order
of their devices, for drivers requiring it.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---

I haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
        and that it works as intended for basic cases.
        Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

    Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

    Added a function to set the probe manual from the application,
    without having the user do it from the command line.

    Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

  Currently manual-probe will cut off probing from all buses.
  It could be interesting to be able to only cut buses supporting hotplug,
  given that they are the one able to probe devices afterward.

  No real use-case for this right now, so leaving as-is. Might be worth
  considering in the future.

v4: Rebased on master,
    Moved implementation in common EAL,
    Used public API within the EAL to set the option,
    Made the API experimental

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++
 lib/librte_eal/common/eal_common_bus.c     |  6 ++++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 +++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++
 lib/librte_eal/common/include/rte_eal.h    | 36 ++++++++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 +++
 9 files changed, 129 insertions(+)

diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index e77d226b5..7251a17b2 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 * **FreeBSD now supports `--base-virtaddr` EAL option.**
 
   FreeBSD version now also supports setting base virtual address for mapping
diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
index baa5b532a..145a96812 100644
--- a/lib/librte_eal/common/eal_common_bus.c
+++ b/lib/librte_eal/common/eal_common_bus.c
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 9e4f09d83..368afa273 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -21,6 +21,7 @@
 #include <rte_malloc.h>
 #include <rte_string_fns.h>
 
+#include "eal_internal_cfg.h"
 #include "eal_private.h"
 #include "hotplug_mp.h"
 
@@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
 	return dev->driver != NULL;
 }
 
+int
+rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void
+rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
+int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
 /* helper function to build devargs, caller should free the memory */
 static int
 build_devargs(const char *busname, const char *devname,
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index a7f9c5f9b..de73970fe 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1446,6 +1447,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1672,6 +1676,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index a42f34923..0006f903f 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index 9855429e5..588fa32a6 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 798ede553..fd7ac8e37 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index b7cf91214..71d9b4ca6 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -465,6 +465,42 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index 7cbf82d37..7d7ff6531 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -419,4 +419,8 @@ EXPERIMENTAL {
 	rte_mcfg_timer_lock;
 	rte_mcfg_timer_unlock;
 	rte_rand_max;
+
+	# added in 19.11
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
-- 
2.11.0

Re: [dpdk-dev] [PATCH v4] eal: add manual probing option

[Jerin Jacob]

On Fri, Oct 25, 2019 at 8:11 PM Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
>
> Add a new EAL option enabling manual probing in the EAL.
> This command line option will configure the EAL so that buses
> will not trigger their probe step on their own.
>
> Applications are then expected to hotplug devices as they see fit.
>
> Devices declared on the command line by the user (using -w and --vdev),
> will be probed using the hotplug API, in the order they are declared.
>
> This has the effect of offering a way for users to control probe order
> of their devices, for drivers requiring it.
>
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
> ---
>
>  doc/guides/rel_notes/release_19_11.rst     |  9 +++++
>  lib/librte_eal/common/eal_common_bus.c     |  6 ++++
>  lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++++++++++
>  lib/librte_eal/common/eal_common_options.c |  8 +++++
>  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
>  lib/librte_eal/common/eal_options.h        |  2 ++
>  lib/librte_eal/common/eal_private.h        |  9 +++++
>  lib/librte_eal/common/include/rte_eal.h    | 36 ++++++++++++++++++++
>  lib/librte_eal/rte_eal_version.map         |  4 +++

Missed to update new eal argument in
doc/guides/linux_gsg/eal_args.include.rst file.

With the above change:

Reviewed-by: Jerin Jacob <jerinj@marvell.com>

[dpdk-dev] [PATCH v5] eal: add manual probing option

[Gaetan Rivet]

Add a new EAL option enabling manual probing in the EAL.
This command line option will configure the EAL so that buses
will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order
of their devices, for drivers requiring it.

Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
---

I haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
        and that it works as intended for basic cases.
        Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

    Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

    Added a function to set the probe manual from the application,
    without having the user do it from the command line.

    Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

  Currently manual-probe will cut off probing from all buses.
  It could be interesting to be able to only cut buses supporting hotplug,
  given that they are the one able to probe devices afterward.

  No real use-case for this right now, so leaving as-is. Might be worth
  considering in the future.

v4: Rebased on master,
    Moved implementation in common EAL,
    Used public API within the EAL to set the option,
    Made the API experimental

v5: added note in the Getting Started Guide.

 doc/guides/linux_gsg/eal_args.include.rst  | 13 +++++++
 doc/guides/rel_notes/release_19_11.rst     |  9 +++++
 lib/librte_eal/common/eal_common_bus.c     |  6 ++++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 +++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++
 lib/librte_eal/common/include/rte_eal.h    | 36 ++++++++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 +++
 10 files changed, 142 insertions(+)

diff --git a/doc/guides/linux_gsg/eal_args.include.rst b/doc/guides/linux_gsg/eal_args.include.rst
index ed8b0e35b..d0717d4a0 100644
--- a/doc/guides/linux_gsg/eal_args.include.rst
+++ b/doc/guides/linux_gsg/eal_args.include.rst
@@ -69,6 +69,19 @@ Device-related options
 
        --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
 
+*   ``--manual-probe``
+
+    Switch the ``EAL`` probe mode to manual. The main bus probe step
+    is disabled and applications are expected to manually probe
+    devices using ``rte_dev_probe()``.
+
+    Devices declared on the command-line using ``-w`` and ``-vdev``
+    are interpreted as hotplug commands. They are thus probed in the
+    order they are declared.
+
+    This makes this option useful to enforce a specific device probe
+    order, instead of relying on each bus scan implementation details.
+
 *   ``-d <path to shared object or directory>``
 
     Load external drivers. An argument can be a single shared object file, or a
diff --git a/doc/guides/rel_notes/release_19_11.rst b/doc/guides/rel_notes/release_19_11.rst
index e77d226b5..7251a17b2 100644
--- a/doc/guides/rel_notes/release_19_11.rst
+++ b/doc/guides/rel_notes/release_19_11.rst
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 * **FreeBSD now supports `--base-virtaddr` EAL option.**
 
   FreeBSD version now also supports setting base virtual address for mapping
diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
index baa5b532a..145a96812 100644
--- a/lib/librte_eal/common/eal_common_bus.c
+++ b/lib/librte_eal/common/eal_common_bus.c
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 9e4f09d83..368afa273 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -21,6 +21,7 @@
 #include <rte_malloc.h>
 #include <rte_string_fns.h>
 
+#include "eal_internal_cfg.h"
 #include "eal_private.h"
 #include "hotplug_mp.h"
 
@@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
 	return dev->driver != NULL;
 }
 
+int
+rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void
+rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
+int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
 /* helper function to build devargs, caller should free the memory */
 static int
 build_devargs(const char *busname, const char *devname,
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index a7f9c5f9b..de73970fe 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1446,6 +1447,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1672,6 +1676,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index a42f34923..0006f903f 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index 9855429e5..588fa32a6 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index 798ede553..fd7ac8e37 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index b7cf91214..71d9b4ca6 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -465,6 +465,42 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index 7cbf82d37..7d7ff6531 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -419,4 +419,8 @@ EXPERIMENTAL {
 	rte_mcfg_timer_lock;
 	rte_mcfg_timer_unlock;
 	rte_rand_max;
+
+	# added in 19.11
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
-- 
2.11.0

Re: [dpdk-dev] [PATCH v5] eal: add manual probing option

[Jerin Jacob]

On Fri, Oct 25, 2019 at 9:16 PM Gaetan Rivet <gaetan.rivet@6wind.com> wrote:
>
> Add a new EAL option enabling manual probing in the EAL.
> This command line option will configure the EAL so that buses
> will not trigger their probe step on their own.
>
> Applications are then expected to hotplug devices as they see fit.
>
> Devices declared on the command line by the user (using -w and --vdev),
> will be probed using the hotplug API, in the order they are declared.
>
> This has the effect of offering a way for users to control probe order
> of their devices, for drivers requiring it.
>
> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>

Reviewed-by: Jerin Jacob <jerinj@marvell.com>

Re: [dpdk-dev] [PATCH v5] eal: add manual probing option

[Pavan Nikhilesh Bhagavatula]

Ping
@grive@u256.net

Since, you weren't reachable on regular email I rebased the patch on ToT and sent a v6
http://patches.dpdk.org/patch/64235/

>-----Original Message-----
>From: dev <dev-bounces@dpdk.org> On Behalf Of Gaetan Rivet
>Sent: Friday, October 25, 2019 9:17 PM
>To: dev@dpdk.org
>Cc: Gaetan Rivet <gaetan.rivet@6wind.com>; Vamsi Krishna Attunuru
><vattunuru@marvell.com>; David Marchand
><david.marchand@redhat.com>; Jerin Jacob Kollanukkaran
><jerinj@marvell.com>
>Subject: [dpdk-dev] [PATCH v5] eal: add manual probing option
>
>Add a new EAL option enabling manual probing in the EAL.
>This command line option will configure the EAL so that buses
>will not trigger their probe step on their own.
>
>Applications are then expected to hotplug devices as they see fit.
>
>Devices declared on the command line by the user (using -w and --
>vdev),
>will be probed using the hotplug API, in the order they are declared.
>
>This has the effect of offering a way for users to control probe order
>of their devices, for drivers requiring it.
>
>Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
>---
>
>I haven't heard many opinions on the matter, please shout if you see an
>issue
>with this approach.
>
>@Slava: I have tested rather quickly that it does not break anything,
>        and that it works as intended for basic cases.
>        Can you test it further for your use-case and tell me if it works fine?
>
>Beyond the obvious difference between both probe mode, something
>to keep in mind:
>while using -w on invalid devices would not block (PCI) bus probing, it
>will stop manual
>probing in its track. All devices need to exist and be valid device IDs.
>
>v2: fixed a few typos, map file (and used Travis to validate).
>
>    Slava, are you able to test this patch?
>
>v3: properly fixed the map file (inherited 19.08 instead of 19.05).
>
>    Added a function to set the probe manual from the application,
>    without having the user do it from the command line.
>
>    Stopped spamming Slava about it, Vamsi was actually the one
>interested in it!
>
>Standing issue worth chiming in:
>
>  Currently manual-probe will cut off probing from all buses.
>  It could be interesting to be able to only cut buses supporting hotplug,
>  given that they are the one able to probe devices afterward.
>
>  No real use-case for this right now, so leaving as-is. Might be worth
>  considering in the future.
>
>v4: Rebased on master,
>    Moved implementation in common EAL,
>    Used public API within the EAL to set the option,
>    Made the API experimental
>
>v5: added note in the Getting Started Guide.
>
> doc/guides/linux_gsg/eal_args.include.rst  | 13 +++++++
> doc/guides/rel_notes/release_19_11.rst     |  9 +++++
> lib/librte_eal/common/eal_common_bus.c     |  6 ++++
> lib/librte_eal/common/eal_common_dev.c     | 54
>++++++++++++++++++++++++++++++
> lib/librte_eal/common/eal_common_options.c |  8 +++++
> lib/librte_eal/common/eal_internal_cfg.h   |  1 +
> lib/librte_eal/common/eal_options.h        |  2 ++
> lib/librte_eal/common/eal_private.h        |  9 +++++
> lib/librte_eal/common/include/rte_eal.h    | 36
>++++++++++++++++++++
> lib/librte_eal/rte_eal_version.map         |  4 +++
> 10 files changed, 142 insertions(+)
>
>diff --git a/doc/guides/linux_gsg/eal_args.include.rst
>b/doc/guides/linux_gsg/eal_args.include.rst
>index ed8b0e35b..d0717d4a0 100644
>--- a/doc/guides/linux_gsg/eal_args.include.rst
>+++ b/doc/guides/linux_gsg/eal_args.include.rst
>@@ -69,6 +69,19 @@ Device-related options
>
>        --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
>
>+*   ``--manual-probe``
>+
>+    Switch the ``EAL`` probe mode to manual. The main bus probe step
>+    is disabled and applications are expected to manually probe
>+    devices using ``rte_dev_probe()``.
>+
>+    Devices declared on the command-line using ``-w`` and ``-vdev``
>+    are interpreted as hotplug commands. They are thus probed in the
>+    order they are declared.
>+
>+    This makes this option useful to enforce a specific device probe
>+    order, instead of relying on each bus scan implementation details.
>+
> *   ``-d <path to shared object or directory>``
>
>     Load external drivers. An argument can be a single shared object file,
>or a
>diff --git a/doc/guides/rel_notes/release_19_11.rst
>b/doc/guides/rel_notes/release_19_11.rst
>index e77d226b5..7251a17b2 100644
>--- a/doc/guides/rel_notes/release_19_11.rst
>+++ b/doc/guides/rel_notes/release_19_11.rst
>@@ -56,6 +56,15 @@ New Features
>      Also, make sure to start the actual text at the margin.
>
>======================================================
>===
>
>+* **EAL will now allow probing devices manually.**
>+
>+  Previously, a user could not force an order when probing declared
>devices.
>+  This could cause issues for drivers depending on another device being
>present.
>+  A new option ``--manual-probe`` is now available to do just that.
>+  This new option relies on the device bus supporting hotplug. It can
>+  also be used to disable automatic probing from the ``PCI`` bus without
>+  having to disable the whole bus.
>+
> * **FreeBSD now supports `--base-virtaddr` EAL option.**
>
>   FreeBSD version now also supports setting base virtual address for
>mapping
>diff --git a/lib/librte_eal/common/eal_common_bus.c
>b/lib/librte_eal/common/eal_common_bus.c
>index baa5b532a..145a96812 100644
>--- a/lib/librte_eal/common/eal_common_bus.c
>+++ b/lib/librte_eal/common/eal_common_bus.c
>@@ -6,6 +6,7 @@
> #include <string.h>
> #include <sys/queue.h>
>
>+#include <rte_eal.h>
> #include <rte_bus.h>
> #include <rte_debug.h>
> #include <rte_string_fns.h>
>@@ -63,6 +64,11 @@ rte_bus_probe(void)
> 	int ret;
> 	struct rte_bus *bus, *vbus = NULL;
>
>+	if (rte_eal_manual_probe()) {
>+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
>+		return rte_dev_probe_devargs_list();
>+	}
>+
> 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
> 		if (!strcmp(bus->name, "vdev")) {
> 			vbus = bus;
>diff --git a/lib/librte_eal/common/eal_common_dev.c
>b/lib/librte_eal/common/eal_common_dev.c
>index 9e4f09d83..368afa273 100644
>--- a/lib/librte_eal/common/eal_common_dev.c
>+++ b/lib/librte_eal/common/eal_common_dev.c
>@@ -21,6 +21,7 @@
> #include <rte_malloc.h>
> #include <rte_string_fns.h>
>
>+#include "eal_internal_cfg.h"
> #include "eal_private.h"
> #include "hotplug_mp.h"
>
>@@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
> 	return dev->driver != NULL;
> }
>
>+int
>+rte_eal_manual_probe(void)
>+{
>+	return internal_config.manual_probe;
>+}
>+
>+void
>+rte_eal_manual_probe_set(int enabled)
>+{
>+	internal_config.manual_probe = !!enabled;
>+}
>+
>+int
>+rte_dev_probe_devargs_list(void)
>+{
>+	struct rte_device *dev;
>+	struct rte_devargs *da;
>+	int ret;
>+
>+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
>+		dev = da->bus->find_device(NULL, cmp_dev_name,
>da->name);
>+		if (dev == NULL) {
>+			RTE_LOG(ERR, EAL, "Unable to find device %s
>on bus %s\n",
>+				da->name, da->bus->name);
>+			continue;
>+		}
>+
>+		if (rte_dev_is_probed(dev))
>+			continue;
>+
>+		if (dev->bus->plug == NULL) {
>+			RTE_LOG(ERR, EAL, "Manual probing (hotplug)
>not supported by bus %s, "
>+					  "required by device %s\n",
>+				dev->bus->name, dev->name);
>+			continue;
>+		}
>+
>+		ret = dev->bus->plug(dev);
>+		/* Ignore positive return values, they are possibly
>+		 * triggered by blacklisted devices on the PCI bus.
>Probing
>+		 * should then continue.
>+		 */
>+		if (ret < 0) {
>+			RTE_LOG(ERR, EAL, "Driver cannot attach device
>%s\n",
>+				dev->name);
>+			/* Fail on first real probe error. */
>+			return ret;
>+		}
>+	}
>+
>+	return 0;
>+}
>+
> /* helper function to build devargs, caller should free the memory */
> static int
> build_devargs(const char *busname, const char *devname,
>diff --git a/lib/librte_eal/common/eal_common_options.c
>b/lib/librte_eal/common/eal_common_options.c
>index a7f9c5f9b..de73970fe 100644
>--- a/lib/librte_eal/common/eal_common_options.c
>+++ b/lib/librte_eal/common/eal_common_options.c
>@@ -82,6 +82,7 @@ eal_long_options[] = {
> 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM
>},
> 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL,
>OPT_SINGLE_FILE_SEGMENTS_NUM},
> 	{OPT_MATCH_ALLOCATIONS, 0, NULL,
>OPT_MATCH_ALLOCATIONS_NUM},
>+	{OPT_MANUAL_PROBE,      0, NULL,
>OPT_MANUAL_PROBE_NUM     },
> 	{0,                     0, NULL, 0                        }
> };
>
>@@ -1446,6 +1447,9 @@ eal_parse_common_option(int opt, const
>char *optarg,
> 			return -1;
> 		}
> 		break;
>+	case OPT_MANUAL_PROBE_NUM:
>+		rte_eal_manual_probe_set(1);
>+		break;
>
> 	/* don't know what to do, leave this to caller */
> 	default:
>@@ -1672,6 +1676,10 @@ eal_common_usage(void)
> 	       "  --"OPT_VDEV"              Add a virtual device.\n"
> 	       "                      The argument format is
><driver><id>[,key=val,...]\n"
> 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
>+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
>+	       "                      Disable probe step for all buses.\n"
>+	       "                      Devices will need to be probed using the hotplug
>API.\n"
>+	       "                      PCI and vdev declarations will be treated in
>order as hotplug commands.\n"
> 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for
>IOVA_PA\n"
> 	       "                      'va' for IOVA_VA\n"
> 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
>diff --git a/lib/librte_eal/common/eal_internal_cfg.h
>b/lib/librte_eal/common/eal_internal_cfg.h
>index a42f34923..0006f903f 100644
>--- a/lib/librte_eal/common/eal_internal_cfg.h
>+++ b/lib/librte_eal/common/eal_internal_cfg.h
>@@ -44,6 +44,7 @@ struct internal_config {
> 	unsigned hugepage_unlink;         /**< true to unlink backing files
>*/
> 	volatile unsigned no_pci;         /**< true to disable PCI */
> 	volatile unsigned no_hpet;        /**< true to disable HPET */
>+	volatile unsigned manual_probe;   /**< true to enable manual
>device probing. */
> 	volatile unsigned vmware_tsc_map; /**< true to use VMware
>TSC mapping
>
>	* instead of native TSC */
> 	volatile unsigned no_shconf;      /**< true if there is no shared
>config */
>diff --git a/lib/librte_eal/common/eal_options.h
>b/lib/librte_eal/common/eal_options.h
>index 9855429e5..588fa32a6 100644
>--- a/lib/librte_eal/common/eal_options.h
>+++ b/lib/librte_eal/common/eal_options.h
>@@ -69,6 +69,8 @@ enum {
> 	OPT_IOVA_MODE_NUM,
> #define OPT_MATCH_ALLOCATIONS  "match-allocations"
> 	OPT_MATCH_ALLOCATIONS_NUM,
>+#define OPT_MANUAL_PROBE "manual-probe"
>+	OPT_MANUAL_PROBE_NUM,
> 	OPT_LONG_MAX_NUM
> };
>
>diff --git a/lib/librte_eal/common/eal_private.h
>b/lib/librte_eal/common/eal_private.h
>index 798ede553..fd7ac8e37 100644
>--- a/lib/librte_eal/common/eal_private.h
>+++ b/lib/librte_eal/common/eal_private.h
>@@ -381,4 +381,13 @@ rte_option_init(void);
> void
> rte_option_usage(void);
>
>+/**
>+ * Go through the devargs list and probe everything in order.
>+ *
>+ * @return
>+ *   0 on success, negative on error.
>+ */
>+int
>+rte_dev_probe_devargs_list(void);
>+
> #endif /* _EAL_PRIVATE_H_ */
>diff --git a/lib/librte_eal/common/include/rte_eal.h
>b/lib/librte_eal/common/include/rte_eal.h
>index b7cf91214..71d9b4ca6 100644
>--- a/lib/librte_eal/common/include/rte_eal.h
>+++ b/lib/librte_eal/common/include/rte_eal.h
>@@ -465,6 +465,42 @@ int rte_eal_has_hugepages(void);
> int rte_eal_has_pci(void);
>
> /**
>+ * Whether EAL probe is manual.
>+ * Enabled by the --manual-probe option or by
>+ * using rte_eal_manual_probe_set().
>+ *
>+ * When manual probing is enabled, batched bus probe of
>+ * their devices is disabled. All devices need to be probed
>+ * using the proper rte_dev API.
>+ *
>+ * In this mode, devices declared on the command line will
>+ * be probed using the bus hotplug API. It is used to enforce
>+ * a specific probe order.
>+ *
>+ * @return
>+ *   Nonzero if manual device probing is enabled.
>+ *
>+ * @see rte_eal_manual_probe_set
>+ */
>+__rte_experimental
>+int rte_eal_manual_probe(void);
>+
>+/**
>+ * Configure EAL probe mode -- manual or automatic.
>+ *
>+ * Enable or disable manual probe mode in EAL.
>+ * This function can be called at any time, but must be used
>+ * before calling rte_eal_init() to have any effect.
>+ *
>+ * @param enabled
>+ *   zero to disable manual probe, non-zero to enable it.
>+ *
>+ * @see rte_eal_manual_probe
>+ */
>+__rte_experimental
>+void rte_eal_manual_probe_set(int enabled);
>+
>+/**
>  * Whether the EAL was asked to create UIO device.
>  *
>  * @return
>diff --git a/lib/librte_eal/rte_eal_version.map
>b/lib/librte_eal/rte_eal_version.map
>index 7cbf82d37..7d7ff6531 100644
>--- a/lib/librte_eal/rte_eal_version.map
>+++ b/lib/librte_eal/rte_eal_version.map
>@@ -419,4 +419,8 @@ EXPERIMENTAL {
> 	rte_mcfg_timer_lock;
> 	rte_mcfg_timer_unlock;
> 	rte_rand_max;
>+
>+	# added in 19.11
>+	rte_eal_manual_probe;
>+	rte_eal_manual_probe_set;
> };
>--
>2.11.0

Re: [dpdk-dev] [PATCH v5] eal: add manual probing option

[Gaetan Rivet]

On 22/01/2020 17:51, Pavan Nikhilesh Bhagavatula wrote:
> Ping
> @grive@u256.net
> 
> Since, you weren't reachable on regular email I rebased the patch on ToT and sent a v6
> http://patches.dpdk.org/patch/64235/
> 

Hello Pavan, indeed sorry about this.
I have not heard remarks about the patchset anyway in the meantime, not 
sure whether there is anything to change.

Thanks for the rebase, I will follow-up if there are remarks blocking 
the integration.

>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Gaetan Rivet
>> Sent: Friday, October 25, 2019 9:17 PM
>> To: dev@dpdk.org
>> Cc: Gaetan Rivet <gaetan.rivet@6wind.com>; Vamsi Krishna Attunuru
>> <vattunuru@marvell.com>; David Marchand
>> <david.marchand@redhat.com>; Jerin Jacob Kollanukkaran
>> <jerinj@marvell.com>
>> Subject: [dpdk-dev] [PATCH v5] eal: add manual probing option
>>
>> Add a new EAL option enabling manual probing in the EAL.
>> This command line option will configure the EAL so that buses
>> will not trigger their probe step on their own.
>>
>> Applications are then expected to hotplug devices as they see fit.
>>
>> Devices declared on the command line by the user (using -w and --
>> vdev),
>> will be probed using the hotplug API, in the order they are declared.
>>
>> This has the effect of offering a way for users to control probe order
>> of their devices, for drivers requiring it.
>>
>> Signed-off-by: Gaetan Rivet <gaetan.rivet@6wind.com>
>> ---
>>
>> I haven't heard many opinions on the matter, please shout if you see an
>> issue
>> with this approach.
>>
>> @Slava: I have tested rather quickly that it does not break anything,
>>         and that it works as intended for basic cases.
>>         Can you test it further for your use-case and tell me if it works fine?
>>
>> Beyond the obvious difference between both probe mode, something
>> to keep in mind:
>> while using -w on invalid devices would not block (PCI) bus probing, it
>> will stop manual
>> probing in its track. All devices need to exist and be valid device IDs.
>>
>> v2: fixed a few typos, map file (and used Travis to validate).
>>
>>     Slava, are you able to test this patch?
>>
>> v3: properly fixed the map file (inherited 19.08 instead of 19.05).
>>
>>     Added a function to set the probe manual from the application,
>>     without having the user do it from the command line.
>>
>>     Stopped spamming Slava about it, Vamsi was actually the one
>> interested in it!
>>
>> Standing issue worth chiming in:
>>
>>   Currently manual-probe will cut off probing from all buses.
>>   It could be interesting to be able to only cut buses supporting hotplug,
>>   given that they are the one able to probe devices afterward.
>>
>>   No real use-case for this right now, so leaving as-is. Might be worth
>>   considering in the future.
>>
>> v4: Rebased on master,
>>     Moved implementation in common EAL,
>>     Used public API within the EAL to set the option,
>>     Made the API experimental
>>
>> v5: added note in the Getting Started Guide.
>>
>> doc/guides/linux_gsg/eal_args.include.rst  | 13 +++++++
>> doc/guides/rel_notes/release_19_11.rst     |  9 +++++
>> lib/librte_eal/common/eal_common_bus.c     |  6 ++++
>> lib/librte_eal/common/eal_common_dev.c     | 54
>> ++++++++++++++++++++++++++++++
>> lib/librte_eal/common/eal_common_options.c |  8 +++++
>> lib/librte_eal/common/eal_internal_cfg.h   |  1 +
>> lib/librte_eal/common/eal_options.h        |  2 ++
>> lib/librte_eal/common/eal_private.h        |  9 +++++
>> lib/librte_eal/common/include/rte_eal.h    | 36
>> ++++++++++++++++++++
>> lib/librte_eal/rte_eal_version.map         |  4 +++
>> 10 files changed, 142 insertions(+)
>>
>> diff --git a/doc/guides/linux_gsg/eal_args.include.rst
>> b/doc/guides/linux_gsg/eal_args.include.rst
>> index ed8b0e35b..d0717d4a0 100644
>> --- a/doc/guides/linux_gsg/eal_args.include.rst
>> +++ b/doc/guides/linux_gsg/eal_args.include.rst
>> @@ -69,6 +69,19 @@ Device-related options
>>
>>         --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
>>
>> +*   ``--manual-probe``
>> +
>> +    Switch the ``EAL`` probe mode to manual. The main bus probe step
>> +    is disabled and applications are expected to manually probe
>> +    devices using ``rte_dev_probe()``.
>> +
>> +    Devices declared on the command-line using ``-w`` and ``-vdev``
>> +    are interpreted as hotplug commands. They are thus probed in the
>> +    order they are declared.
>> +
>> +    This makes this option useful to enforce a specific device probe
>> +    order, instead of relying on each bus scan implementation details.
>> +
>> *   ``-d <path to shared object or directory>``
>>
>>      Load external drivers. An argument can be a single shared object file,
>> or a
>> diff --git a/doc/guides/rel_notes/release_19_11.rst
>> b/doc/guides/rel_notes/release_19_11.rst
>> index e77d226b5..7251a17b2 100644
>> --- a/doc/guides/rel_notes/release_19_11.rst
>> +++ b/doc/guides/rel_notes/release_19_11.rst
>> @@ -56,6 +56,15 @@ New Features
>>       Also, make sure to start the actual text at the margin.
>>
>> ======================================================
>> ===
>>
>> +* **EAL will now allow probing devices manually.**
>> +
>> +  Previously, a user could not force an order when probing declared
>> devices.
>> +  This could cause issues for drivers depending on another device being
>> present.
>> +  A new option ``--manual-probe`` is now available to do just that.
>> +  This new option relies on the device bus supporting hotplug. It can
>> +  also be used to disable automatic probing from the ``PCI`` bus without
>> +  having to disable the whole bus.
>> +
>> * **FreeBSD now supports `--base-virtaddr` EAL option.**
>>
>>    FreeBSD version now also supports setting base virtual address for
>> mapping
>> diff --git a/lib/librte_eal/common/eal_common_bus.c
>> b/lib/librte_eal/common/eal_common_bus.c
>> index baa5b532a..145a96812 100644
>> --- a/lib/librte_eal/common/eal_common_bus.c
>> +++ b/lib/librte_eal/common/eal_common_bus.c
>> @@ -6,6 +6,7 @@
>> #include <string.h>
>> #include <sys/queue.h>
>>
>> +#include <rte_eal.h>
>> #include <rte_bus.h>
>> #include <rte_debug.h>
>> #include <rte_string_fns.h>
>> @@ -63,6 +64,11 @@ rte_bus_probe(void)
>> 	int ret;
>> 	struct rte_bus *bus, *vbus = NULL;
>>
>> +	if (rte_eal_manual_probe()) {
>> +		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
>> +		return rte_dev_probe_devargs_list();
>> +	}
>> +
>> 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
>> 		if (!strcmp(bus->name, "vdev")) {
>> 			vbus = bus;
>> diff --git a/lib/librte_eal/common/eal_common_dev.c
>> b/lib/librte_eal/common/eal_common_dev.c
>> index 9e4f09d83..368afa273 100644
>> --- a/lib/librte_eal/common/eal_common_dev.c
>> +++ b/lib/librte_eal/common/eal_common_dev.c
>> @@ -21,6 +21,7 @@
>> #include <rte_malloc.h>
>> #include <rte_string_fns.h>
>>
>> +#include "eal_internal_cfg.h"
>> #include "eal_private.h"
>> #include "hotplug_mp.h"
>>
>> @@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
>> 	return dev->driver != NULL;
>> }
>>
>> +int
>> +rte_eal_manual_probe(void)
>> +{
>> +	return internal_config.manual_probe;
>> +}
>> +
>> +void
>> +rte_eal_manual_probe_set(int enabled)
>> +{
>> +	internal_config.manual_probe = !!enabled;
>> +}
>> +
>> +int
>> +rte_dev_probe_devargs_list(void)
>> +{
>> +	struct rte_device *dev;
>> +	struct rte_devargs *da;
>> +	int ret;
>> +
>> +	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
>> +		dev = da->bus->find_device(NULL, cmp_dev_name,
>> da->name);
>> +		if (dev == NULL) {
>> +			RTE_LOG(ERR, EAL, "Unable to find device %s
>> on bus %s\n",
>> +				da->name, da->bus->name);
>> +			continue;
>> +		}
>> +
>> +		if (rte_dev_is_probed(dev))
>> +			continue;
>> +
>> +		if (dev->bus->plug == NULL) {
>> +			RTE_LOG(ERR, EAL, "Manual probing (hotplug)
>> not supported by bus %s, "
>> +					  "required by device %s\n",
>> +				dev->bus->name, dev->name);
>> +			continue;
>> +		}
>> +
>> +		ret = dev->bus->plug(dev);
>> +		/* Ignore positive return values, they are possibly
>> +		 * triggered by blacklisted devices on the PCI bus.
>> Probing
>> +		 * should then continue.
>> +		 */
>> +		if (ret < 0) {
>> +			RTE_LOG(ERR, EAL, "Driver cannot attach device
>> %s\n",
>> +				dev->name);
>> +			/* Fail on first real probe error. */
>> +			return ret;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> /* helper function to build devargs, caller should free the memory */
>> static int
>> build_devargs(const char *busname, const char *devname,
>> diff --git a/lib/librte_eal/common/eal_common_options.c
>> b/lib/librte_eal/common/eal_common_options.c
>> index a7f9c5f9b..de73970fe 100644
>> --- a/lib/librte_eal/common/eal_common_options.c
>> +++ b/lib/librte_eal/common/eal_common_options.c
>> @@ -82,6 +82,7 @@ eal_long_options[] = {
>> 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM
>> },
>> 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL,
>> OPT_SINGLE_FILE_SEGMENTS_NUM},
>> 	{OPT_MATCH_ALLOCATIONS, 0, NULL,
>> OPT_MATCH_ALLOCATIONS_NUM},
>> +	{OPT_MANUAL_PROBE,      0, NULL,
>> OPT_MANUAL_PROBE_NUM     },
>> 	{0,                     0, NULL, 0                        }
>> };
>>
>> @@ -1446,6 +1447,9 @@ eal_parse_common_option(int opt, const
>> char *optarg,
>> 			return -1;
>> 		}
>> 		break;
>> +	case OPT_MANUAL_PROBE_NUM:
>> +		rte_eal_manual_probe_set(1);
>> +		break;
>>
>> 	/* don't know what to do, leave this to caller */
>> 	default:
>> @@ -1672,6 +1676,10 @@ eal_common_usage(void)
>> 	       "  --"OPT_VDEV"              Add a virtual device.\n"
>> 	       "                      The argument format is
>> <driver><id>[,key=val,...]\n"
>> 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
>> +	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
>> +	       "                      Disable probe step for all buses.\n"
>> +	       "                      Devices will need to be probed using the hotplug
>> API.\n"
>> +	       "                      PCI and vdev declarations will be treated in
>> order as hotplug commands.\n"
>> 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for
>> IOVA_PA\n"
>> 	       "                      'va' for IOVA_VA\n"
>> 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
>> diff --git a/lib/librte_eal/common/eal_internal_cfg.h
>> b/lib/librte_eal/common/eal_internal_cfg.h
>> index a42f34923..0006f903f 100644
>> --- a/lib/librte_eal/common/eal_internal_cfg.h
>> +++ b/lib/librte_eal/common/eal_internal_cfg.h
>> @@ -44,6 +44,7 @@ struct internal_config {
>> 	unsigned hugepage_unlink;         /**< true to unlink backing files
>> */
>> 	volatile unsigned no_pci;         /**< true to disable PCI */
>> 	volatile unsigned no_hpet;        /**< true to disable HPET */
>> +	volatile unsigned manual_probe;   /**< true to enable manual
>> device probing. */
>> 	volatile unsigned vmware_tsc_map; /**< true to use VMware
>> TSC mapping
>>
>> 	* instead of native TSC */
>> 	volatile unsigned no_shconf;      /**< true if there is no shared
>> config */
>> diff --git a/lib/librte_eal/common/eal_options.h
>> b/lib/librte_eal/common/eal_options.h
>> index 9855429e5..588fa32a6 100644
>> --- a/lib/librte_eal/common/eal_options.h
>> +++ b/lib/librte_eal/common/eal_options.h
>> @@ -69,6 +69,8 @@ enum {
>> 	OPT_IOVA_MODE_NUM,
>> #define OPT_MATCH_ALLOCATIONS  "match-allocations"
>> 	OPT_MATCH_ALLOCATIONS_NUM,
>> +#define OPT_MANUAL_PROBE "manual-probe"
>> +	OPT_MANUAL_PROBE_NUM,
>> 	OPT_LONG_MAX_NUM
>> };
>>
>> diff --git a/lib/librte_eal/common/eal_private.h
>> b/lib/librte_eal/common/eal_private.h
>> index 798ede553..fd7ac8e37 100644
>> --- a/lib/librte_eal/common/eal_private.h
>> +++ b/lib/librte_eal/common/eal_private.h
>> @@ -381,4 +381,13 @@ rte_option_init(void);
>> void
>> rte_option_usage(void);
>>
>> +/**
>> + * Go through the devargs list and probe everything in order.
>> + *
>> + * @return
>> + *   0 on success, negative on error.
>> + */
>> +int
>> +rte_dev_probe_devargs_list(void);
>> +
>> #endif /* _EAL_PRIVATE_H_ */
>> diff --git a/lib/librte_eal/common/include/rte_eal.h
>> b/lib/librte_eal/common/include/rte_eal.h
>> index b7cf91214..71d9b4ca6 100644
>> --- a/lib/librte_eal/common/include/rte_eal.h
>> +++ b/lib/librte_eal/common/include/rte_eal.h
>> @@ -465,6 +465,42 @@ int rte_eal_has_hugepages(void);
>> int rte_eal_has_pci(void);
>>
>> /**
>> + * Whether EAL probe is manual.
>> + * Enabled by the --manual-probe option or by
>> + * using rte_eal_manual_probe_set().
>> + *
>> + * When manual probing is enabled, batched bus probe of
>> + * their devices is disabled. All devices need to be probed
>> + * using the proper rte_dev API.
>> + *
>> + * In this mode, devices declared on the command line will
>> + * be probed using the bus hotplug API. It is used to enforce
>> + * a specific probe order.
>> + *
>> + * @return
>> + *   Nonzero if manual device probing is enabled.
>> + *
>> + * @see rte_eal_manual_probe_set
>> + */
>> +__rte_experimental
>> +int rte_eal_manual_probe(void);
>> +
>> +/**
>> + * Configure EAL probe mode -- manual or automatic.
>> + *
>> + * Enable or disable manual probe mode in EAL.
>> + * This function can be called at any time, but must be used
>> + * before calling rte_eal_init() to have any effect.
>> + *
>> + * @param enabled
>> + *   zero to disable manual probe, non-zero to enable it.
>> + *
>> + * @see rte_eal_manual_probe
>> + */
>> +__rte_experimental
>> +void rte_eal_manual_probe_set(int enabled);
>> +
>> +/**
>>   * Whether the EAL was asked to create UIO device.
>>   *
>>   * @return
>> diff --git a/lib/librte_eal/rte_eal_version.map
>> b/lib/librte_eal/rte_eal_version.map
>> index 7cbf82d37..7d7ff6531 100644
>> --- a/lib/librte_eal/rte_eal_version.map
>> +++ b/lib/librte_eal/rte_eal_version.map
>> @@ -419,4 +419,8 @@ EXPERIMENTAL {
>> 	rte_mcfg_timer_lock;
>> 	rte_mcfg_timer_unlock;
>> 	rte_rand_max;
>> +
>> +	# added in 19.11
>> +	rte_eal_manual_probe;
>> +	rte_eal_manual_probe_set;
>> };
>> --
>> 2.11.0
> 

[dpdk-dev] [PATCH v7] eal: add manual probing option

[Gaetan Rivet]

Add a new EAL option enabling manual probing in the EAL.
This command line option will configure the EAL so that buses
will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order
of their devices, for drivers requiring it.

Signed-off-by: Gaetan Rivet <grive@u256.net>
Acked-by : Vamsi Attunuru <vattunuru@marvell.com>
Tested-by: Vamsi Attunuru <vattunuru@marvell.com>
Reviewed-by: Jerin Jacob <jerinj@marvell.com>
---

 haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
        and that it works as intended for basic cases.
        Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

    Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

    Added a function to set the probe manual from the application,
    without having the user do it from the command line.

    Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

  Currently manual-probe will cut off probing from all buses.
  It could be interesting to be able to only cut buses supporting hotplug,
  given that they are the one able to probe devices afterward.

  No real use-case for this right now, so leaving as-is. Might be worth
  considering in the future.

v4: Rebased on master,
    Moved implementation in common EAL,
    Used public API within the EAL to set the option,
    Made the API experimental

v5: added note in the Getting Started Guide.

v6: Rebased on master

    see http://mails.dpdk.org/archives/dev/2020-January/154178.html
    for reference to this version, linking v7 to v5 thread.

v7: Updated author and SoB.

 doc/guides/linux_gsg/eal_args.include.rst  | 13 ++++++
 doc/guides/rel_notes/release_20_02.rst     |  9 ++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 +
 lib/librte_eal/common/eal_private.h        |  9 ++++
 lib/librte_eal/common/include/rte_eal.h    | 36 +++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 ++
 10 files changed, 142 insertions(+)

diff --git a/doc/guides/linux_gsg/eal_args.include.rst b/doc/guides/linux_gsg/eal_args.include.rst
index ed8b0e35b..d0717d4a0 100644
--- a/doc/guides/linux_gsg/eal_args.include.rst
+++ b/doc/guides/linux_gsg/eal_args.include.rst
@@ -69,6 +69,19 @@ Device-related options
 
        --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
 
+*   ``--manual-probe``
+
+    Switch the ``EAL`` probe mode to manual. The main bus probe step
+    is disabled and applications are expected to manually probe
+    devices using ``rte_dev_probe()``.
+
+    Devices declared on the command-line using ``-w`` and ``-vdev``
+    are interpreted as hotplug commands. They are thus probed in the
+    order they are declared.
+
+    This makes this option useful to enforce a specific device probe
+    order, instead of relying on each bus scan implementation details.
+
 *   ``-d <path to shared object or directory>``
 
     Load external drivers. An argument can be a single shared object file, or a
diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
index 50e2c1484..f6b3b3def 100644
--- a/doc/guides/rel_notes/release_20_02.rst
+++ b/doc/guides/rel_notes/release_20_02.rst
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 * **Added Wait Until Equal API.**
 
   A new API has been added to wait for a memory location to be updated with a
diff --git a/lib/librte_eal/common/eal_common_bus.c b/lib/librte_eal/common/eal_common_bus.c
index baa5b532a..145a96812 100644
--- a/lib/librte_eal/common/eal_common_bus.c
+++ b/lib/librte_eal/common/eal_common_bus.c
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
diff --git a/lib/librte_eal/common/eal_common_dev.c b/lib/librte_eal/common/eal_common_dev.c
index 9e4f09d83..368afa273 100644
--- a/lib/librte_eal/common/eal_common_dev.c
+++ b/lib/librte_eal/common/eal_common_dev.c
@@ -21,6 +21,7 @@
 #include <rte_malloc.h>
 #include <rte_string_fns.h>
 
+#include "eal_internal_cfg.h"
 #include "eal_private.h"
 #include "hotplug_mp.h"
 
@@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
 	return dev->driver != NULL;
 }
 
+int
+rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void
+rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
+int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
 /* helper function to build devargs, caller should free the memory */
 static int
 build_devargs(const char *busname, const char *devname,
diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c
index 5920233bc..f899eea4d 100644
--- a/lib/librte_eal/common/eal_common_options.c
+++ b/lib/librte_eal/common/eal_common_options.c
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1443,6 +1444,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1669,6 +1673,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h
index a42f34923..0006f903f 100644
--- a/lib/librte_eal/common/eal_internal_cfg.h
+++ b/lib/librte_eal/common/eal_internal_cfg.h
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
diff --git a/lib/librte_eal/common/eal_options.h b/lib/librte_eal/common/eal_options.h
index 9855429e5..588fa32a6 100644
--- a/lib/librte_eal/common/eal_options.h
+++ b/lib/librte_eal/common/eal_options.h
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h
index ddcfbe2e4..680c7db88 100644
--- a/lib/librte_eal/common/eal_private.h
+++ b/lib/librte_eal/common/eal_private.h
@@ -443,4 +443,13 @@ rte_option_usage(void);
 uint64_t
 eal_get_baseaddr(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
diff --git a/lib/librte_eal/common/include/rte_eal.h b/lib/librte_eal/common/include/rte_eal.h
index 2f9ed298d..7195f6859 100644
--- a/lib/librte_eal/common/include/rte_eal.h
+++ b/lib/librte_eal/common/include/rte_eal.h
@@ -421,6 +421,42 @@ int rte_eal_has_hugepages(void);
  */
 int rte_eal_has_pci(void);
 
+/**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
 /**
  * Whether the EAL was asked to create UIO device.
  *
diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map
index e38d02530..13d04a8bc 100644
--- a/lib/librte_eal/rte_eal_version.map
+++ b/lib/librte_eal/rte_eal_version.map
@@ -332,4 +332,8 @@ EXPERIMENTAL {
 	# added in 19.11
 	rte_log_get_stream;
 	rte_mcfg_get_single_file_segments;
+
+	# added in 20.02
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
-- 
2.25.0

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Pavan Nikhilesh Bhagavatula]

@David Marchand @thomas@monjalon.net

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

>-----Original Message-----
>From: dev <dev-bounces@dpdk.org> On Behalf Of Gaetan Rivet
>Sent: Thursday, January 23, 2020 3:28 PM
>To: dev@dpdk.org
>Cc: Vamsi Krishna Attunuru <vattunuru@marvell.com>; Jerin Jacob
>Kollanukkaran <jerinj@marvell.com>
>Subject: [dpdk-dev] [PATCH v7] eal: add manual probing option
>
>Add a new EAL option enabling manual probing in the EAL.
>This command line option will configure the EAL so that buses
>will not trigger their probe step on their own.
>
>Applications are then expected to hotplug devices as they see fit.
>
>Devices declared on the command line by the user (using -w and --
>vdev),
>will be probed using the hotplug API, in the order they are declared.
>
>This has the effect of offering a way for users to control probe order
>of their devices, for drivers requiring it.
>
>Signed-off-by: Gaetan Rivet <grive@u256.net>
>Acked-by : Vamsi Attunuru <vattunuru@marvell.com>
>Tested-by: Vamsi Attunuru <vattunuru@marvell.com>
>Reviewed-by: Jerin Jacob <jerinj@marvell.com>
>---
>
> haven't heard many opinions on the matter, please shout if you see an
>issue
>with this approach.
>
>@Slava: I have tested rather quickly that it does not break anything,
>        and that it works as intended for basic cases.
>        Can you test it further for your use-case and tell me if it works fine?
>
>Beyond the obvious difference between both probe mode, something
>to keep in mind:
>while using -w on invalid devices would not block (PCI) bus probing, it
>will stop manual
>probing in its track. All devices need to exist and be valid device IDs.
>
>v2: fixed a few typos, map file (and used Travis to validate).
>
>    Slava, are you able to test this patch?
>
>v3: properly fixed the map file (inherited 19.08 instead of 19.05).
>
>    Added a function to set the probe manual from the application,
>    without having the user do it from the command line.
>
>    Stopped spamming Slava about it, Vamsi was actually the one
>interested in it!
>
>Standing issue worth chiming in:
>
>  Currently manual-probe will cut off probing from all buses.
>  It could be interesting to be able to only cut buses supporting hotplug,
>  given that they are the one able to probe devices afterward.
>
>  No real use-case for this right now, so leaving as-is. Might be worth
>  considering in the future.
>
>v4: Rebased on master,
>    Moved implementation in common EAL,
>    Used public API within the EAL to set the option,
>    Made the API experimental
>
>v5: added note in the Getting Started Guide.
>
>v6: Rebased on master
>
>    see https://urldefense.proofpoint.com/v2/url?u=http-
>3A__mails.dpdk.org_archives_dev_2020-
>2DJanuary_154178.html&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=
>1cjuAHrGh745jHNmj2fD85sUMIJ2IPIDsIJzo6FN6Z0&m=BsQe7kO_de-
>Kb6YFvVZPpZgIhGnWWpW8Pou2KkH_Cjk&s=oFGrj6beplYlYJWodLDGY
>GCUExSbvb1iJEERsA18QnA&e=
>    for reference to this version, linking v7 to v5 thread.
>
>v7: Updated author and SoB.
>
> doc/guides/linux_gsg/eal_args.include.rst  | 13 ++++++
> doc/guides/rel_notes/release_20_02.rst     |  9 ++++
> lib/librte_eal/common/eal_common_bus.c     |  6 +++
> lib/librte_eal/common/eal_common_dev.c     | 54
>++++++++++++++++++++++
> lib/librte_eal/common/eal_common_options.c |  8 ++++
> lib/librte_eal/common/eal_internal_cfg.h   |  1 +
> lib/librte_eal/common/eal_options.h        |  2 +
> lib/librte_eal/common/eal_private.h        |  9 ++++
> lib/librte_eal/common/include/rte_eal.h    | 36 +++++++++++++++
> lib/librte_eal/rte_eal_version.map         |  4 ++
> 10 files changed, 142 insertions(+)
>
>diff --git a/doc/guides/linux_gsg/eal_args.include.rst
>b/doc/guides/linux_gsg/eal_args.include.rst
>index ed8b0e35b..d0717d4a0 100644
>--- a/doc/guides/linux_gsg/eal_args.include.rst
>+++ b/doc/guides/linux_gsg/eal_args.include.rst
>@@ -69,6 +69,19 @@ Device-related options
>
>        --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
>
>+*   ``--manual-probe``
>+
>+    Switch the ``EAL`` probe mode to manual. The main bus probe step
>+    is disabled and applications are expected to manually probe
>+    devices using ``rte_dev_probe()``.
>+
>+    Devices declared on the command-line using ``-w`` and ``-vdev``
>+    are interpreted as hotplug commands. They are thus probed in the
>+    order they are declared.
>+
>+    This makes this option useful to enforce a specific device probe
>+    order, instead of relying on each bus scan implementation details.
>+
> *   ``-d <path to shared object or directory>``
>
>     Load external drivers. An argument can be a single shared object file,
>or a
>diff --git a/doc/guides/rel_notes/release_20_02.rst
>b/doc/guides/rel_notes/release_20_02.rst
>index 50e2c1484..f6b3b3def 100644
>--- a/doc/guides/rel_notes/release_20_02.rst
>+++ b/doc/guides/rel_notes/release_20_02.rst
>@@ -56,6 +56,15 @@ New Features
>      Also, make sure to start the actual text at the margin.
>
>======================================================
>===
>
>+* **EAL will now allow probing devices manually.**
>+
>+  Previously, a user could not force an order when probing declared
>devices.
>+  This could cause issues for drivers depending on another device being
>present.
>+  A new option ``--manual-probe`` is now available to do just that.
>+  This new option relies on the device bus supporting hotplug. It can
>+  also be used to disable automatic probing from the ``PCI`` bus without
>+  having to disable the whole bus.
>+
> * **Added Wait Until Equal API.**
>
>   A new API has been added to wait for a memory location to be
>updated with a
>diff --git a/lib/librte_eal/common/eal_common_bus.c
>b/lib/librte_eal/common/eal_common_bus.c
>index baa5b532a..145a96812 100644
>--- a/lib/librte_eal/common/eal_common_bus.c
>+++ b/lib/librte_eal/common/eal_common_bus.c
>@@ -6,6 +6,7 @@
> #include <string.h>
> #include <sys/queue.h>
>
>+#include <rte_eal.h>
> #include <rte_bus.h>
> #include <rte_debug.h>
> #include <rte_string_fns.h>
>@@ -63,6 +64,11 @@ rte_bus_probe(void)
> 	int ret;
> 	struct rte_bus *bus, *vbus = NULL;
>
>+	if (rte_eal_manual_probe()) {
>+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
>+		return rte_dev_probe_devargs_list();
>+	}
>+
> 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
> 		if (!strcmp(bus->name, "vdev")) {
> 			vbus = bus;
>diff --git a/lib/librte_eal/common/eal_common_dev.c
>b/lib/librte_eal/common/eal_common_dev.c
>index 9e4f09d83..368afa273 100644
>--- a/lib/librte_eal/common/eal_common_dev.c
>+++ b/lib/librte_eal/common/eal_common_dev.c
>@@ -21,6 +21,7 @@
> #include <rte_malloc.h>
> #include <rte_string_fns.h>
>
>+#include "eal_internal_cfg.h"
> #include "eal_private.h"
> #include "hotplug_mp.h"
>
>@@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
> 	return dev->driver != NULL;
> }
>
>+int
>+rte_eal_manual_probe(void)
>+{
>+	return internal_config.manual_probe;
>+}
>+
>+void
>+rte_eal_manual_probe_set(int enabled)
>+{
>+	internal_config.manual_probe = !!enabled;
>+}
>+
>+int
>+rte_dev_probe_devargs_list(void)
>+{
>+	struct rte_device *dev;
>+	struct rte_devargs *da;
>+	int ret;
>+
>+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
>+		dev = da->bus->find_device(NULL, cmp_dev_name,
>da->name);
>+		if (dev == NULL) {
>+			RTE_LOG(ERR, EAL, "Unable to find device %s
>on bus %s\n",
>+				da->name, da->bus->name);
>+			continue;
>+		}
>+
>+		if (rte_dev_is_probed(dev))
>+			continue;
>+
>+		if (dev->bus->plug == NULL) {
>+			RTE_LOG(ERR, EAL, "Manual probing (hotplug)
>not supported by bus %s, "
>+					  "required by device %s\n",
>+				dev->bus->name, dev->name);
>+			continue;
>+		}
>+
>+		ret = dev->bus->plug(dev);
>+		/* Ignore positive return values, they are possibly
>+		 * triggered by blacklisted devices on the PCI bus.
>Probing
>+		 * should then continue.
>+		 */
>+		if (ret < 0) {
>+			RTE_LOG(ERR, EAL, "Driver cannot attach device
>%s\n",
>+				dev->name);
>+			/* Fail on first real probe error. */
>+			return ret;
>+		}
>+	}
>+
>+	return 0;
>+}
>+
> /* helper function to build devargs, caller should free the memory */
> static int
> build_devargs(const char *busname, const char *devname,
>diff --git a/lib/librte_eal/common/eal_common_options.c
>b/lib/librte_eal/common/eal_common_options.c
>index 5920233bc..f899eea4d 100644
>--- a/lib/librte_eal/common/eal_common_options.c
>+++ b/lib/librte_eal/common/eal_common_options.c
>@@ -82,6 +82,7 @@ eal_long_options[] = {
> 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM
>},
> 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL,
>OPT_SINGLE_FILE_SEGMENTS_NUM},
> 	{OPT_MATCH_ALLOCATIONS, 0, NULL,
>OPT_MATCH_ALLOCATIONS_NUM},
>+	{OPT_MANUAL_PROBE,      0, NULL,
>OPT_MANUAL_PROBE_NUM     },
> 	{0,                     0, NULL, 0                        }
> };
>
>@@ -1443,6 +1444,9 @@ eal_parse_common_option(int opt, const
>char *optarg,
> 			return -1;
> 		}
> 		break;
>+	case OPT_MANUAL_PROBE_NUM:
>+		rte_eal_manual_probe_set(1);
>+		break;
>
> 	/* don't know what to do, leave this to caller */
> 	default:
>@@ -1669,6 +1673,10 @@ eal_common_usage(void)
> 	       "  --"OPT_VDEV"              Add a virtual device.\n"
> 	       "                      The argument format is
><driver><id>[,key=val,...]\n"
> 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
>+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
>+	       "                      Disable probe step for all buses.\n"
>+	       "                      Devices will need to be probed using the hotplug
>API.\n"
>+	       "                      PCI and vdev declarations will be treated in
>order as hotplug commands.\n"
> 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for
>IOVA_PA\n"
> 	       "                      'va' for IOVA_VA\n"
> 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
>diff --git a/lib/librte_eal/common/eal_internal_cfg.h
>b/lib/librte_eal/common/eal_internal_cfg.h
>index a42f34923..0006f903f 100644
>--- a/lib/librte_eal/common/eal_internal_cfg.h
>+++ b/lib/librte_eal/common/eal_internal_cfg.h
>@@ -44,6 +44,7 @@ struct internal_config {
> 	unsigned hugepage_unlink;         /**< true to unlink backing files
>*/
> 	volatile unsigned no_pci;         /**< true to disable PCI */
> 	volatile unsigned no_hpet;        /**< true to disable HPET */
>+	volatile unsigned manual_probe;   /**< true to enable manual
>device probing. */
> 	volatile unsigned vmware_tsc_map; /**< true to use VMware
>TSC mapping
>
>	* instead of native TSC */
> 	volatile unsigned no_shconf;      /**< true if there is no shared
>config */
>diff --git a/lib/librte_eal/common/eal_options.h
>b/lib/librte_eal/common/eal_options.h
>index 9855429e5..588fa32a6 100644
>--- a/lib/librte_eal/common/eal_options.h
>+++ b/lib/librte_eal/common/eal_options.h
>@@ -69,6 +69,8 @@ enum {
> 	OPT_IOVA_MODE_NUM,
> #define OPT_MATCH_ALLOCATIONS  "match-allocations"
> 	OPT_MATCH_ALLOCATIONS_NUM,
>+#define OPT_MANUAL_PROBE "manual-probe"
>+	OPT_MANUAL_PROBE_NUM,
> 	OPT_LONG_MAX_NUM
> };
>
>diff --git a/lib/librte_eal/common/eal_private.h
>b/lib/librte_eal/common/eal_private.h
>index ddcfbe2e4..680c7db88 100644
>--- a/lib/librte_eal/common/eal_private.h
>+++ b/lib/librte_eal/common/eal_private.h
>@@ -443,4 +443,13 @@ rte_option_usage(void);
> uint64_t
> eal_get_baseaddr(void);
>
>+/**
>+ * Go through the devargs list and probe everything in order.
>+ *
>+ * @return
>+ *   0 on success, negative on error.
>+ */
>+int
>+rte_dev_probe_devargs_list(void);
>+
> #endif /* _EAL_PRIVATE_H_ */
>diff --git a/lib/librte_eal/common/include/rte_eal.h
>b/lib/librte_eal/common/include/rte_eal.h
>index 2f9ed298d..7195f6859 100644
>--- a/lib/librte_eal/common/include/rte_eal.h
>+++ b/lib/librte_eal/common/include/rte_eal.h
>@@ -421,6 +421,42 @@ int rte_eal_has_hugepages(void);
>  */
> int rte_eal_has_pci(void);
>
>+/**
>+ * Whether EAL probe is manual.
>+ * Enabled by the --manual-probe option or by
>+ * using rte_eal_manual_probe_set().
>+ *
>+ * When manual probing is enabled, batched bus probe of
>+ * their devices is disabled. All devices need to be probed
>+ * using the proper rte_dev API.
>+ *
>+ * In this mode, devices declared on the command line will
>+ * be probed using the bus hotplug API. It is used to enforce
>+ * a specific probe order.
>+ *
>+ * @return
>+ *   Nonzero if manual device probing is enabled.
>+ *
>+ * @see rte_eal_manual_probe_set
>+ */
>+__rte_experimental
>+int rte_eal_manual_probe(void);
>+
>+/**
>+ * Configure EAL probe mode -- manual or automatic.
>+ *
>+ * Enable or disable manual probe mode in EAL.
>+ * This function can be called at any time, but must be used
>+ * before calling rte_eal_init() to have any effect.
>+ *
>+ * @param enabled
>+ *   zero to disable manual probe, non-zero to enable it.
>+ *
>+ * @see rte_eal_manual_probe
>+ */
>+__rte_experimental
>+void rte_eal_manual_probe_set(int enabled);
>+
> /**
>  * Whether the EAL was asked to create UIO device.
>  *
>diff --git a/lib/librte_eal/rte_eal_version.map
>b/lib/librte_eal/rte_eal_version.map
>index e38d02530..13d04a8bc 100644
>--- a/lib/librte_eal/rte_eal_version.map
>+++ b/lib/librte_eal/rte_eal_version.map
>@@ -332,4 +332,8 @@ EXPERIMENTAL {
> 	# added in 19.11
> 	rte_log_get_stream;
> 	rte_mcfg_get_single_file_segments;
>+
>+	# added in 20.02
>+	rte_eal_manual_probe;
>+	rte_eal_manual_probe_set;
> };
>--
>2.25.0

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Thomas Monjalon]

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> @David Marchand @thomas@monjalon.net
> 
> Ping?
> 
> Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

All changes related to probing are very sensitive,
and we know a big refactoring would be better than stacking
more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate
enough time to properly think about this feature.
Please accept our apologies, and let's consider it as
a high priority for 20.05 cycle.

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Gaetan Rivet]

On 03/02/2020 23:21, Thomas Monjalon wrote:
> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
>> @David Marchand @thomas@monjalon.net
>>
>> Ping?
>>
>> Are there any more changes required for this patch? It's been in queue since last October.
> 
> Sorry we have not decided whether it is a good idea or not.
> 
> All changes related to probing are very sensitive,
> and we know a big refactoring would be better than stacking
> more and more options and corner cases.
> 
> As we are busy with ABI stability stuff, we did not allocate
> enough time to properly think about this feature.
> Please accept our apologies, and let's consider it as
> a high priority for 20.05 cycle.
> 
> 
> 

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Thomas Monjalon]

04/02/2020 11:03, Gaetan Rivet:
> On 03/02/2020 23:21, Thomas Monjalon wrote:
> > 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> >> @David Marchand @thomas@monjalon.net
> >>
> >> Ping?
> >>
> >> Are there any more changes required for this patch? It's been in queue since last October.
> > 
> > Sorry we have not decided whether it is a good idea or not.
> > 
> > All changes related to probing are very sensitive,
> > and we know a big refactoring would be better than stacking
> > more and more options and corner cases.
> > 
> > As we are busy with ABI stability stuff, we did not allocate
> > enough time to properly think about this feature.
> > Please accept our apologies, and let's consider it as
> > a high priority for 20.05 cycle.
> > 
> 
> Hello Thomas,
> 
> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> 
> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.


Yes, life is full of bad decisions and consequences.

I still think there is a risk in adding new user expectations,
and maintaining some code to workaround unknown issues.

The real question here is to know why this patch?
Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Gaetan Rivet]

On 04/02/2020 12:07, Thomas Monjalon wrote:
> 04/02/2020 11:03, Gaetan Rivet:
>> On 03/02/2020 23:21, Thomas Monjalon wrote:
>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
>>>> @David Marchand @thomas@monjalon.net
>>>>
>>>> Ping?
>>>>
>>>> Are there any more changes required for this patch? It's been in queue since last October.
>>>
>>> Sorry we have not decided whether it is a good idea or not.
>>>
>>> All changes related to probing are very sensitive,
>>> and we know a big refactoring would be better than stacking
>>> more and more options and corner cases.
>>>
>>> As we are busy with ABI stability stuff, we did not allocate
>>> enough time to properly think about this feature.
>>> Please accept our apologies, and let's consider it as
>>> a high priority for 20.05 cycle.
>>>
>>
>> Hello Thomas,
>>
>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
>>
>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> 
> 
> Yes, life is full of bad decisions and consequences.


Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.

> 
> I still think there is a risk in adding new user expectations,
> and maintaining some code to workaround unknown issues.
> 
> The real question here is to know why this patch?
> Is it to workaround a broken driver?
> Or to workaround a broken design in EAL and bus drivers?
> 
> 

Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).

I'm not sure having a dependent-probe by default is good, and that would be a big change.

If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.

[1]: First proposal:
        http://mails.dpdk.org/archives/dev/2019-September/144166.html
      My arguments:
        http://mails.dpdk.org/archives/dev/2019-September/144564.html

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Thomas Monjalon]

04/02/2020 13:43, Gaetan Rivet:
> On 04/02/2020 12:07, Thomas Monjalon wrote:
> > 04/02/2020 11:03, Gaetan Rivet:
> >> On 03/02/2020 23:21, Thomas Monjalon wrote:
> >>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> >>>> @David Marchand @thomas@monjalon.net
> >>>>
> >>>> Ping?
> >>>>
> >>>> Are there any more changes required for this patch? It's been in queue since last October.
> >>>
> >>> Sorry we have not decided whether it is a good idea or not.
> >>>
> >>> All changes related to probing are very sensitive,
> >>> and we know a big refactoring would be better than stacking
> >>> more and more options and corner cases.
> >>>
> >>> As we are busy with ABI stability stuff, we did not allocate
> >>> enough time to properly think about this feature.
> >>> Please accept our apologies, and let's consider it as
> >>> a high priority for 20.05 cycle.
> >>>
> >>
> >> Hello Thomas,
> >>
> >> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> >>
> >> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> > 
> > 
> > Yes, life is full of bad decisions and consequences.
> 
> 
> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> 
> > 
> > I still think there is a risk in adding new user expectations,
> > and maintaining some code to workaround unknown issues.
> > 
> > The real question here is to know why this patch?
> > Is it to workaround a broken driver?
> > Or to workaround a broken design in EAL and bus drivers?
> 
> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> 
> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> 
> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> 
> [1]: First proposal:
>         http://mails.dpdk.org/archives/dev/2019-September/144166.html
>       My arguments:
>         http://mails.dpdk.org/archives/dev/2019-September/144564.html


OK so there are two needs:

1/ Better control whitelist/blacklist mode.
We already know that a rework is needed here.
Unfortunately neither you or me had time to work on it,
and others who were interested disappeared.

2/ Associate ports with equivalent properties in applications.
This must be done in applications.
Tweaking the probe order is a hack.

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Gaetan Rivet]

On 04/02/2020 16:06, Thomas Monjalon wrote:
> 04/02/2020 13:43, Gaetan Rivet:
>> On 04/02/2020 12:07, Thomas Monjalon wrote:
>>> 04/02/2020 11:03, Gaetan Rivet:
>>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
>>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
>>>>>> @David Marchand @thomas@monjalon.net
>>>>>>
>>>>>> Ping?
>>>>>>
>>>>>> Are there any more changes required for this patch? It's been in queue since last October.
>>>>>
>>>>> Sorry we have not decided whether it is a good idea or not.
>>>>>
>>>>> All changes related to probing are very sensitive,
>>>>> and we know a big refactoring would be better than stacking
>>>>> more and more options and corner cases.
>>>>>
>>>>> As we are busy with ABI stability stuff, we did not allocate
>>>>> enough time to properly think about this feature.
>>>>> Please accept our apologies, and let's consider it as
>>>>> a high priority for 20.05 cycle.
>>>>>
>>>>
>>>> Hello Thomas,
>>>>
>>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
>>>>
>>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
>>>
>>>
>>> Yes, life is full of bad decisions and consequences.
>>
>>
>> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
>>
>>>
>>> I still think there is a risk in adding new user expectations,
>>> and maintaining some code to workaround unknown issues.
>>>
>>> The real question here is to know why this patch?
>>> Is it to workaround a broken driver?
>>> Or to workaround a broken design in EAL and bus drivers?
>>
>> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
>>
>> I'm not sure having a dependent-probe by default is good, and that would be a big change.
>>
>> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
>>
>> [1]: First proposal:
>>          http://mails.dpdk.org/archives/dev/2019-September/144166.html
>>        My arguments:
>>          http://mails.dpdk.org/archives/dev/2019-September/144564.html
> 
> 
> OK so there are two needs:
> 
> 1/ Better control whitelist/blacklist mode.
> We already know that a rework is needed here.
> Unfortunately neither you or me had time to work on it,
> and others who were interested disappeared.
> 
> 2/ Associate ports with equivalent properties in applications.
> This must be done in applications.
> Tweaking the probe order is a hack.
> 
> 

An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.

Instead, with this patch this app can do

   rte_eal_manual_probe_set(1);
   rte_eal_init();

to have the same behavior and be able to hotplug ports as it sees fit.

You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.

This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Jerin Jacob]

On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
>
> On 04/02/2020 16:06, Thomas Monjalon wrote:
> > 04/02/2020 13:43, Gaetan Rivet:
> >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> >>> 04/02/2020 11:03, Gaetan Rivet:
> >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> >>>>>> @David Marchand @thomas@monjalon.net
> >>>>>>
> >>>>>> Ping?
> >>>>>>
> >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> >>>>>
> >>>>> Sorry we have not decided whether it is a good idea or not.
> >>>>>
> >>>>> All changes related to probing are very sensitive,
> >>>>> and we know a big refactoring would be better than stacking
> >>>>> more and more options and corner cases.
> >>>>>
> >>>>> As we are busy with ABI stability stuff, we did not allocate
> >>>>> enough time to properly think about this feature.
> >>>>> Please accept our apologies, and let's consider it as
> >>>>> a high priority for 20.05 cycle.
> >>>>>
> >>>>
> >>>> Hello Thomas,
> >>>>
> >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> >>>>
> >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> >>>
> >>>
> >>> Yes, life is full of bad decisions and consequences.
> >>
> >>
> >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> >>
> >>>
> >>> I still think there is a risk in adding new user expectations,
> >>> and maintaining some code to workaround unknown issues.
> >>>
> >>> The real question here is to know why this patch?
> >>> Is it to workaround a broken driver?
> >>> Or to workaround a broken design in EAL and bus drivers?
> >>
> >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> >>
> >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> >>
> >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> >>
> >> [1]: First proposal:
> >>          http://mails.dpdk.org/archives/dev/2019-September/144166.html
> >>        My arguments:
> >>          http://mails.dpdk.org/archives/dev/2019-September/144564.html
> >
> >
> > OK so there are two needs:
> >
> > 1/ Better control whitelist/blacklist mode.
> > We already know that a rework is needed here.
> > Unfortunately neither you or me had time to work on it,
> > and others who were interested disappeared.
> >
> > 2/ Associate ports with equivalent properties in applications.
> > This must be done in applications.
> > Tweaking the probe order is a hack.
> >
> >
>
> An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
>
> Instead, with this patch this app can do
>
>    rte_eal_manual_probe_set(1);
>    rte_eal_init();
>
> to have the same behavior and be able to hotplug ports as it sees fit.
>
> You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
>
> This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.

Thomas,

What would be the resolution of this? What is your recommendation to
fix the issue as you have the concern of this patch?

Issue:
1) When l2fwd does the forwarding for simplicity and performance
reason it just xor the port to find the destination port to forward.
2) If the adjacent ports are not symmetrical(example: one is 40G and
other 10G) then forwarding will drop the packets.

So, either
a) We need to control the probing order

b) Or Application need
1) To track the symmetrical ports and maintain the forwarding table  OR
2) Have the command-line option to specify destination port like l3fwd.

We can fix it in the application, but do we need to complicate l2fwd?
I am fine with that, if that is consensus.

Thoughts? If you think, there is a rework needed in eal then could you
enumerate the items for the rework.

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Thomas Monjalon]

10/02/2020 15:51, Jerin Jacob:
> On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
> >
> > On 04/02/2020 16:06, Thomas Monjalon wrote:
> > > 04/02/2020 13:43, Gaetan Rivet:
> > >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> > >>> 04/02/2020 11:03, Gaetan Rivet:
> > >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> > >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> > >>>>>> @David Marchand @thomas@monjalon.net
> > >>>>>>
> > >>>>>> Ping?
> > >>>>>>
> > >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> > >>>>>
> > >>>>> Sorry we have not decided whether it is a good idea or not.
> > >>>>>
> > >>>>> All changes related to probing are very sensitive,
> > >>>>> and we know a big refactoring would be better than stacking
> > >>>>> more and more options and corner cases.
> > >>>>>
> > >>>>> As we are busy with ABI stability stuff, we did not allocate
> > >>>>> enough time to properly think about this feature.
> > >>>>> Please accept our apologies, and let's consider it as
> > >>>>> a high priority for 20.05 cycle.
> > >>>>>
> > >>>>
> > >>>> Hello Thomas,
> > >>>>
> > >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> > >>>>
> > >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> > >>>
> > >>>
> > >>> Yes, life is full of bad decisions and consequences.
> > >>
> > >>
> > >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> > >>
> > >>>
> > >>> I still think there is a risk in adding new user expectations,
> > >>> and maintaining some code to workaround unknown issues.
> > >>>
> > >>> The real question here is to know why this patch?
> > >>> Is it to workaround a broken driver?
> > >>> Or to workaround a broken design in EAL and bus drivers?
> > >>
> > >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> > >>
> > >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> > >>
> > >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> > >>
> > >> [1]: First proposal:
> > >>          http://mails.dpdk.org/archives/dev/2019-September/144166.html
> > >>        My arguments:
> > >>          http://mails.dpdk.org/archives/dev/2019-September/144564.html
> > >
> > >
> > > OK so there are two needs:
> > >
> > > 1/ Better control whitelist/blacklist mode.
> > > We already know that a rework is needed here.
> > > Unfortunately neither you or me had time to work on it,
> > > and others who were interested disappeared.
> > >
> > > 2/ Associate ports with equivalent properties in applications.
> > > This must be done in applications.
> > > Tweaking the probe order is a hack.
> > >
> > >
> >
> > An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
> >
> > Instead, with this patch this app can do
> >
> >    rte_eal_manual_probe_set(1);
> >    rte_eal_init();
> >
> > to have the same behavior and be able to hotplug ports as it sees fit.
> >
> > You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
> >
> > This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
> 
> Thomas,
> 
> What would be the resolution of this? What is your recommendation to
> fix the issue as you have the concern of this patch?
> 
> Issue:
> 1) When l2fwd does the forwarding for simplicity and performance
> reason it just xor the port to find the destination port to forward.
> 2) If the adjacent ports are not symmetrical(example: one is 40G and
> other 10G) then forwarding will drop the packets.
> 
> So, either
> a) We need to control the probing order
> 
> b) Or Application need
> 1) To track the symmetrical ports and maintain the forwarding table  OR
> 2) Have the command-line option to specify destination port like l3fwd.
> 
> We can fix it in the application, but do we need to complicate l2fwd?
> I am fine with that, if that is consensus.

You are describing an application issue,
that's why I believe it should be fixed in the application.

Should we have a command line option to configure the forwarding rules
in the application (2)? I think yes.
Should we implement an application logic to automatically create
the best forwarding rules (1)? It would be nice, but anyway,
I think we need manual config (2) as a fallback.


> Thoughts? If you think, there is a rework needed in eal then could you
> enumerate the items for the rework.

Sorry I don't have time to describe dive into EAL probing and
enumerate the items to rework.
The most important issues I remind are:
	- white/blacklist policy is a mess and should be done in a higher layer
	- devargs syntax should allow generic matching (thanks to class awareness and generic syntax)

Starting from these 2 items, we could imagine a generic path to
disable automatic probing, but I think the l2fwd logic should not
rely on probing order anyway.

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Jerin Jacob]

On Mon, Feb 10, 2020 at 8:57 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 10/02/2020 15:51, Jerin Jacob:
> > On Tue, Feb 4, 2020 at 9:32 PM Gaetan Rivet <grive@u256.net> wrote:
> > >
> > > On 04/02/2020 16:06, Thomas Monjalon wrote:
> > > > 04/02/2020 13:43, Gaetan Rivet:
> > > >> On 04/02/2020 12:07, Thomas Monjalon wrote:
> > > >>> 04/02/2020 11:03, Gaetan Rivet:
> > > >>>> On 03/02/2020 23:21, Thomas Monjalon wrote:
> > > >>>>> 03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:
> > > >>>>>> @David Marchand @thomas@monjalon.net
> > > >>>>>>
> > > >>>>>> Ping?
> > > >>>>>>
> > > >>>>>> Are there any more changes required for this patch? It's been in queue since last October.
> > > >>>>>
> > > >>>>> Sorry we have not decided whether it is a good idea or not.
> > > >>>>>
> > > >>>>> All changes related to probing are very sensitive,
> > > >>>>> and we know a big refactoring would be better than stacking
> > > >>>>> more and more options and corner cases.
> > > >>>>>
> > > >>>>> As we are busy with ABI stability stuff, we did not allocate
> > > >>>>> enough time to properly think about this feature.
> > > >>>>> Please accept our apologies, and let's consider it as
> > > >>>>> a high priority for 20.05 cycle.
> > > >>>>>
> > > >>>>
> > > >>>> Hello Thomas,
> > > >>>>
> > > >>>> This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.
> > > >>>>
> > > >>>> The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.
> > > >>>
> > > >>>
> > > >>> Yes, life is full of bad decisions and consequences.
> > > >>
> > > >>
> > > >> Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.
> > > >>
> > > >>>
> > > >>> I still think there is a risk in adding new user expectations,
> > > >>> and maintaining some code to workaround unknown issues.
> > > >>>
> > > >>> The real question here is to know why this patch?
> > > >>> Is it to workaround a broken driver?
> > > >>> Or to workaround a broken design in EAL and bus drivers?
> > > >>
> > > >> Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).
> > > >>
> > > >> I'm not sure having a dependent-probe by default is good, and that would be a big change.
> > > >>
> > > >> If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.
> > > >>
> > > >> [1]: First proposal:
> > > >>          http://mails.dpdk.org/archives/dev/2019-September/144166.html
> > > >>        My arguments:
> > > >>          http://mails.dpdk.org/archives/dev/2019-September/144564.html
> > > >
> > > >
> > > > OK so there are two needs:
> > > >
> > > > 1/ Better control whitelist/blacklist mode.
> > > > We already know that a rework is needed here.
> > > > Unfortunately neither you or me had time to work on it,
> > > > and others who were interested disappeared.
> > > >
> > > > 2/ Associate ports with equivalent properties in applications.
> > > > This must be done in applications.
> > > > Tweaking the probe order is a hack.
> > > >
> > > >
> > >
> > > An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.
> > >
> > > Instead, with this patch this app can do
> > >
> > >    rte_eal_manual_probe_set(1);
> > >    rte_eal_init();
> > >
> > > to have the same behavior and be able to hotplug ports as it sees fit.
> > >
> > > You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.
> > >
> > > This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.
> >
> > Thomas,
> >
> > What would be the resolution of this? What is your recommendation to
> > fix the issue as you have the concern of this patch?
> >
> > Issue:
> > 1) When l2fwd does the forwarding for simplicity and performance
> > reason it just xor the port to find the destination port to forward.
> > 2) If the adjacent ports are not symmetrical(example: one is 40G and
> > other 10G) then forwarding will drop the packets.
> >
> > So, either
> > a) We need to control the probing order
> >
> > b) Or Application need
> > 1) To track the symmetrical ports and maintain the forwarding table  OR
> > 2) Have the command-line option to specify destination port like l3fwd.
> >
> > We can fix it in the application, but do we need to complicate l2fwd?
> > I am fine with that, if that is consensus.
>
> You are describing an application issue,
> that's why I believe it should be fixed in the application.

Thanks for the quick reply and I agree.


>
> Should we have a command line option to configure the forwarding rules
> in the application (2)? I think yes.
> Should we implement an application logic to automatically create
> the best forwarding rules (1)? It would be nice, but anyway,
> I think we need manual config (2) as a fallback.
>
>
> > Thoughts? If you think, there is a rework needed in eal then could you
> > enumerate the items for the rework.
>
> Sorry I don't have time to describe dive into EAL probing and
> enumerate the items to rework.
> The most important issues I remind are:
>         - white/blacklist policy is a mess and should be done in a higher layer
>         - devargs syntax should allow generic matching (thanks to class awareness and generic syntax)
>
> Starting from these 2 items, we could imagine a generic path to
> disable automatic probing, but I think the l2fwd logic should not
> rely on probing order anyway.

+ Bruce as l2fwd maintainer.

Since in any case, l2fwd needs to be updated, We will focus on l2fwd
change for v20.05 and leaving the fate of this patch to EAL
maintainers.
Let us know, Are we are OK with below change in l2fwd as
- Introduce an array-based port lookup table instead of hardcoding to
xor based lookup.
- if no argument specified fill dest port as xor of source
- If argument is specified override the lookup table with a
user-specified destination port.



>
>

[dpdk-dev] [PATCH] [v1 1/1] examples/l2fwd: add cmdline option for forwarding port info

[vattunuru]

From: Vamsi Attunuru <vattunuru@marvell.com>

Current l2fwd application configures adjacent ports as destination
ports for forwarding the traffic which is a kind of static mapping
that can not be altered by the command line options.

Patch adds a config option to pass the forwarding port pair mapping
as a command line parameter which allows the user to pass required
forwarding port mapping.

If no config argument is specified, destination port map is not
changed and traffic gets forwarded with existing mapping.

When port pair mapping is passed in config option, destination port map
is configured and traffic gets forwarded accordingly.

Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

With above config option, traffic received from portid = 0 gets forwarded
to port = 3 and vice versa, similarly traffic gets forwarded on other port
pairs (1,4) and (2,5).

Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
---
 doc/guides/rel_notes/release_20_05.rst             |   6 +
 .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
 examples/l2fwd/main.c                              | 174 ++++++++++++++++++---
 3 files changed, 177 insertions(+), 21 deletions(-)

diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf5..645400b 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,12 @@ New Features
 
   * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
 
+* **Added --config command line parameter to l2fwd example.**
+
+  Added new command line option ``--config(port, port)[,(port, port)]`` to
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
 
 Removed Items
 -------------
diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
index 39d6b00..e708f88 100644
--- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
+++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
@@ -91,7 +91,10 @@ The application requires a number of command line options:
 
 .. code-block:: console
 
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --config(port, port)[,(port, port)]
 
 where,
 
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --config(port,port)[,(port,port)]: Determines which ports are mapped to
+    which ports for packet forwarding.
 
 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore and MAC address updating enabled, to forward RX traffic of ports 0 & 1
+on ports 2 & 3 respectively and vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --config="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.
 
diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
index 09257aa..a8b98b9 100644
--- a/examples/l2fwd/main.c
+++ b/examples/l2fwd/main.c
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>
 
 static volatile bool force_quit;
 
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;
+
+static struct port_pair_params port_pair_params_array[RTE_MAX_ETHPORTS];
+static struct port_pair_params *port_pair_params;
+static uint16_t nb_port_pair_params = 1;
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;
 
 #define MAX_RX_QUEUE_PER_LCORE 16
@@ -319,6 +329,60 @@ l2fwd_parse_portmask(const char *portmask)
 	return pm;
 }
 
+static int
+l2fwd_parse_port_pair_config(const char *q_arg)
+{
+	enum fieldnames {
+		FLD_PORT1 = 0,
+		FLD_PORT2,
+		_NUM_FLD
+	};
+	unsigned long int_fld[_NUM_FLD];
+	const char *p, *p0 = q_arg;
+	char *str_fld[_NUM_FLD];
+	unsigned int size;
+	char s[256];
+	char *end;
+	int i;
+
+	nb_port_pair_params = 0;
+
+	while ((p = strchr(p0, '(')) != NULL) {
+		++p;
+		p0 = strchr(p, ')');
+		if (p0 == NULL)
+			return -1;
+
+		size = p0 - p;
+		if (size >= sizeof(s))
+			return -1;
+
+		snprintf(s, sizeof(s), "%.*s", size, p);
+		if (rte_strsplit(s, sizeof(s), str_fld,
+				 _NUM_FLD, ',') != _NUM_FLD)
+			return -1;
+		for (i = 0; i < _NUM_FLD; i++) {
+			errno = 0;
+			int_fld[i] = strtoul(str_fld[i], &end, 0);
+			if (errno != 0 || end == str_fld[i] ||
+			    int_fld[i] > RTE_MAX_ETHPORTS)
+				return -1;
+		}
+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
+			printf("exceeded max number of port pair params: %hu\n",
+				nb_port_pair_params);
+			return -1;
+		}
+		port_pair_params_array[nb_port_pair_params].port[0] =
+				(uint16_t)int_fld[FLD_PORT1];
+		port_pair_params_array[nb_port_pair_params].port[1] =
+				(uint16_t)int_fld[FLD_PORT2];
+		++nb_port_pair_params;
+	}
+	port_pair_params = port_pair_params_array;
+	return 0;
+}
+
 static unsigned int
 l2fwd_parse_nqueue(const char *q_arg)
 {
@@ -361,6 +425,7 @@ static const char short_options[] =
 
 #define CMD_LINE_OPT_MAC_UPDATING "mac-updating"
 #define CMD_LINE_OPT_NO_MAC_UPDATING "no-mac-updating"
+#define CMD_LINE_OPT_CONFIG "config"
 
 enum {
 	/* long options mapped to a short option */
@@ -368,11 +433,13 @@ enum {
 	/* first long only option value must be >= 256, so that we won't
 	 * conflict with short options */
 	CMD_LINE_OPT_MIN_NUM = 256,
+	CMD_LINE_OPT_CONFIG_NUM,
 };
 
 static const struct option lgopts[] = {
 	{ CMD_LINE_OPT_MAC_UPDATING, no_argument, &mac_updating, 1},
 	{ CMD_LINE_OPT_NO_MAC_UPDATING, no_argument, &mac_updating, 0},
+	{ CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
 	{NULL, 0, 0, 0}
 };
 
@@ -386,6 +453,7 @@ l2fwd_parse_args(int argc, char **argv)
 	char *prgname = argv[0];
 
 	argvopt = argv;
+	port_pair_params = NULL;
 
 	while ((opt = getopt_long(argc, argvopt, short_options,
 				  lgopts, &option_index)) != EOF) {
@@ -423,7 +491,13 @@ l2fwd_parse_args(int argc, char **argv)
 			break;
 
 		/* long options */
-		case 0:
+		case CMD_LINE_OPT_CONFIG_NUM:
+			ret = l2fwd_parse_port_pair_config(optarg);
+			if (ret) {
+				fprintf(stderr, "Invalid config\n");
+				l2fwd_usage(prgname);
+				return -1;
+			}
 			break;
 
 		default:
@@ -440,6 +514,48 @@ l2fwd_parse_args(int argc, char **argv)
 	return ret;
 }
 
+/*
+ * Check port pair config with enabled port mask,
+ * and for valid port pair combinations.
+ */
+static int
+check_port_pair_config(void)
+{
+	uint32_t port_pair_config_mask = 0;
+	uint32_t port_pair_mask = 0;
+	uint16_t index, i, portid;
+
+	for (index = 0; index < nb_port_pair_params; index++) {
+		port_pair_mask = 0;
+
+		for (i = 0; i < NUM_PORTS; i++)  {
+			portid = port_pair_params[index].port[i];
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
+				printf("port %u is not enabled in port mask\n",
+				       portid);
+				return -1;
+			}
+			if (!rte_eth_dev_is_valid_port(portid)) {
+				printf("port %u is not present on the board\n",
+				       portid);
+				return -1;
+			}
+
+			port_pair_mask |= 1 << portid;
+		}
+
+		if (port_pair_config_mask & port_pair_mask) {
+			printf("port %u is used in other port pairs\n", portid);
+			return -1;
+		}
+		port_pair_config_mask |= port_pair_mask;
+	}
+
+	l2fwd_enabled_port_mask &= port_pair_config_mask;
+
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -555,6 +671,11 @@ main(int argc, char **argv)
 	if (nb_ports == 0)
 		rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
 
+	if (port_pair_params != NULL) {
+		if (check_port_pair_config() < 0)
+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
+	}
+
 	/* check port mask to possible port mask */
 	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
 		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
@@ -565,26 +686,40 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;
 
-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx;
+
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			if (idx % 2 == 0) {
+				portid = port_pair_params[idx >> 1].port[0];
+				l2fwd_dst_ports[portid] =
+					port_pair_params[idx >> 1].port[1];
+			} else {
+				portid = port_pair_params[idx >> 1].port[1];
+				l2fwd_dst_ports[portid] =
+					port_pair_params[idx >> 1].port[0];
+			}
+		}
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;
+
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {
+				last_port = portid;
+			}
 
+			nb_ports_in_mask++;
+		}
 		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+			printf("Notice: odd number of ports in portmask.\n");
+			l2fwd_dst_ports[last_port] = last_port;
 		}
-		else
-			last_port = portid;
-
-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
 	}
 
 	rx_lcore_id = 0;
@@ -613,7 +748,8 @@ main(int argc, char **argv)
 
 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}
 
 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd + MAX_PKT_BURST +
-- 
2.8.4

Re: [dpdk-dev] [PATCH] [v1 1/1] examples/l2fwd: add cmdline option for forwarding port info

[Andrzej Ostruszka [C]]

On 4/3/20 5:30 AM, vattunuru@marvell.com wrote:
> From: Vamsi Attunuru <vattunuru@marvell.com>
> 
> Current l2fwd application configures adjacent ports as destination
> ports for forwarding the traffic which is a kind of static mapping
> that can not be altered by the command line options.
> 
> Patch adds a config option to pass the forwarding port pair mapping
> as a command line parameter which allows the user to pass required
> forwarding port mapping.
> 
> If no config argument is specified, destination port map is not
> changed and traffic gets forwarded with existing mapping.
> 
> When port pair mapping is passed in config option, destination port map
> is configured and traffic gets forwarded accordingly.
> 
> Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"
> 
> With above config option, traffic received from portid = 0 gets forwarded
> to port = 3 and vice versa, similarly traffic gets forwarded on other port
> pairs (1,4) and (2,5).
> 
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
>  doc/guides/rel_notes/release_20_05.rst             |   6 +
>  .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
>  examples/l2fwd/main.c                              | 174 ++++++++++++++++++---
>  3 files changed, 177 insertions(+), 21 deletions(-)
> 
> diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
> index 000bbf5..645400b 100644
> --- a/doc/guides/rel_notes/release_20_05.rst
> +++ b/doc/guides/rel_notes/release_20_05.rst
> @@ -62,6 +62,12 @@ New Features
>  
>    * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
>  
> +* **Added --config command line parameter to l2fwd example.**
> +
> +  Added new command line option ``--config(port, port)[,(port, port)]`` to

Maybe use the actual syntax with '=' or space after 'config'?

> +  pass forwarding port details.
> +  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
> +  details of this parameter usage.
>  
>  Removed Items
>  -------------
> diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> index 39d6b00..e708f88 100644
> --- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> +++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> @@ -91,7 +91,10 @@ The application requires a number of command line options:
>  
>  .. code-block:: console
>  
> -    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
> +    ./build/l2fwd [EAL options] -- -p PORTMASK
> +                                   [-q NQ]
> +                                   --[no-]mac-updating
> +                                   --config(port, port)[,(port, port)]

Ditto

>  
>  where,
>  
> @@ -99,7 +102,10 @@ where,
>  
>  *   q NQ: A number of queues (=ports) per lcore (default is 1)
>  
> -*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
> +*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
> +
> +*   --config(port,port)[,(port,port)]: Determines which ports are mapped to
> +    which ports for packet forwarding.

Ditto

>  
>  To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
>  updating enabled, issue the command:
> @@ -108,6 +114,14 @@ updating enabled, issue the command:
>  
>      $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
>  
> +To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
> +per lcore and MAC address updating enabled, to forward RX traffic of ports 0 & 1
> +on ports 2 & 3 respectively and vice versa, issue the command:
> +
> +.. code-block:: console
> +
> +    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --config="(0,2)(1,3)"
> +
>  Refer to the *DPDK Getting Started Guide* for general information on running applications
>  and the Environment Abstraction Layer (EAL) options.
>  
> diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
> index 09257aa..a8b98b9 100644
> --- a/examples/l2fwd/main.c
> +++ b/examples/l2fwd/main.c
> @@ -38,6 +38,7 @@
>  #include <rte_ethdev.h>
>  #include <rte_mempool.h>
>  #include <rte_mbuf.h>
> +#include <rte_string_fns.h>
>  
>  static volatile bool force_quit;
>  
> @@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
>  /* list of enabled ports */
>  static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
>  
> +struct port_pair_params {
> +#define NUM_PORTS	2
> +	uint16_t port[NUM_PORTS];
> +} __rte_cache_aligned;
> +
> +static struct port_pair_params port_pair_params_array[RTE_MAX_ETHPORTS];
> +static struct port_pair_params *port_pair_params;
> +static uint16_t nb_port_pair_params = 1;

Why this initialization to 1?  Below (during parsing) it is initialized
to 0, so I would leave this default initialized.

> +
>  static unsigned int l2fwd_rx_queue_per_lcore = 1;
>  
>  #define MAX_RX_QUEUE_PER_LCORE 16
> @@ -319,6 +329,60 @@ l2fwd_parse_portmask(const char *portmask)
>  	return pm;
>  }
>  
> +static int
> +l2fwd_parse_port_pair_config(const char *q_arg)
> +{
> +	enum fieldnames {
> +		FLD_PORT1 = 0,
> +		FLD_PORT2,
> +		_NUM_FLD
> +	};
> +	unsigned long int_fld[_NUM_FLD];
> +	const char *p, *p0 = q_arg;
> +	char *str_fld[_NUM_FLD];
> +	unsigned int size;
> +	char s[256];
> +	char *end;
> +	int i;
> +
> +	nb_port_pair_params = 0;
> +
> +	while ((p = strchr(p0, '(')) != NULL) {
> +		++p;
> +		p0 = strchr(p, ')');
> +		if (p0 == NULL)
> +			return -1;
> +
> +		size = p0 - p;
> +		if (size >= sizeof(s))
> +			return -1;
> +
> +		snprintf(s, sizeof(s), "%.*s", size, p);

Same comment as for recent l2fwd-event - this can be simple memcpy.

> +		if (rte_strsplit(s, sizeof(s), str_fld,
> +				 _NUM_FLD, ',') != _NUM_FLD)
> +			return -1;
> +		for (i = 0; i < _NUM_FLD; i++) {
> +			errno = 0;
> +			int_fld[i] = strtoul(str_fld[i], &end, 0);
> +			if (errno != 0 || end == str_fld[i] ||
> +			    int_fld[i] > RTE_MAX_ETHPORTS)

I think this check should be >=, RTE_MAX_ETHPORTS is an invalid port id.

> +				return -1;
> +		}
> +		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
> +			printf("exceeded max number of port pair params: %hu\n",
> +				nb_port_pair_params);
> +			return -1;
> +		}
> +		port_pair_params_array[nb_port_pair_params].port[0] =
> +				(uint16_t)int_fld[FLD_PORT1];
> +		port_pair_params_array[nb_port_pair_params].port[1] =
> +				(uint16_t)int_fld[FLD_PORT2];
> +		++nb_port_pair_params;
> +	}
> +	port_pair_params = port_pair_params_array;
> +	return 0;
> +}

[...]

> @@ -565,26 +686,40 @@ main(int argc, char **argv)
>  		l2fwd_dst_ports[portid] = 0;
>  	last_port = 0;
>  
> -	/*
> -	 * Each logical core is assigned a dedicated TX queue on each port.
> -	 */
> -	RTE_ETH_FOREACH_DEV(portid) {
> -		/* skip ports that are not enabled */
> -		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
> -			continue;
> +	/* populate destination port details */
> +	if (port_pair_params != NULL) {
> +		uint16_t idx;
> +
> +		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
> +			if (idx % 2 == 0) {
> +				portid = port_pair_params[idx >> 1].port[0];
> +				l2fwd_dst_ports[portid] =
> +					port_pair_params[idx >> 1].port[1];
> +			} else {
> +				portid = port_pair_params[idx >> 1].port[1];
> +				l2fwd_dst_ports[portid] =
> +					port_pair_params[idx >> 1].port[0];
> +			}

I think this conditional can be rephrased like:

	uint16_t idx, p;
...
	p = idx & 1;
	port_id = port_pair_params[idx >> 1].port[p];
	l2fwd_dst_ports[port_id] =
		port_pair_params[idx >> 1].port[p ^ 1];

but I'm not sure which one is more readable - I leave you the decision,
 feel free to skip this comment altogether :).

> +		}
> +	} else {
> +		RTE_ETH_FOREACH_DEV(portid) {
> +			/* skip ports that are not enabled */
> +			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
> +				continue;
> +
> +			if (nb_ports_in_mask % 2) {
> +				l2fwd_dst_ports[portid] = last_port;
> +				l2fwd_dst_ports[last_port] = portid;
> +			} else {
> +				last_port = portid;
> +			}
>  
> +			nb_ports_in_mask++;
> +		}
>  		if (nb_ports_in_mask % 2) {
> -			l2fwd_dst_ports[portid] = last_port;
> -			l2fwd_dst_ports[last_port] = portid;
> +			printf("Notice: odd number of ports in portmask.\n");
> +			l2fwd_dst_ports[last_port] = last_port;
>  		}
> -		else
> -			last_port = portid;
> -
> -		nb_ports_in_mask++;
> -	}
> -	if (nb_ports_in_mask % 2) {
> -		printf("Notice: odd number of ports in portmask.\n");
> -		l2fwd_dst_ports[last_port] = last_port;
>  	}
>  
>  	rx_lcore_id = 0;
> @@ -613,7 +748,8 @@ main(int argc, char **argv)
>  
>  		qconf->rx_port_list[qconf->n_rx_port] = portid;
>  		qconf->n_rx_port++;
> -		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
> +		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
> +		       portid, l2fwd_dst_ports[portid]);
>  	}
>  
>  	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd + MAX_PKT_BURST +
> 

Apart from these comments:

Reviewed-by: Andrzej Ostruszka <aostruszka@marvell.com>

With regards
Andrzej Ostruszka

Re: [dpdk-dev] [EXT] Re: [PATCH v7] eal: add manual probing option

[Jerin Jacob Kollanukkaran]

> >
> > Should we have a command line option to configure the forwarding rules
> > in the application (2)? I think yes.
> > Should we implement an application logic to automatically create the
> > best forwarding rules (1)? It would be nice, but anyway, I think we
> > need manual config (2) as a fallback.
> >
> >
> > > Thoughts? If you think, there is a rework needed in eal then could
> > > you enumerate the items for the rework.
> >
> > Sorry I don't have time to describe dive into EAL probing and
> > enumerate the items to rework.
> > The most important issues I remind are:
> >         - white/blacklist policy is a mess and should be done in a higher layer
> >         - devargs syntax should allow generic matching (thanks to
> > class awareness and generic syntax)
> >
> > Starting from these 2 items, we could imagine a generic path to
> > disable automatic probing, but I think the l2fwd logic should not rely
> > on probing order anyway.
> 
> + Bruce as l2fwd maintainer.
> 
> Since in any case, l2fwd needs to be updated, We will focus on l2fwd change
> for v20.05 and leaving the fate of this patch to EAL maintainers.
> Let us know, Are we are OK with below change in l2fwd as
> - Introduce an array-based port lookup table instead of hardcoding to xor based
> lookup.
> - if no argument specified fill dest port as xor of source
> - If argument is specified override the lookup table with a user-specified
> destination port.

I think, this thread can be closed here.

L2fwd change sent as different patch.

Please review if it is for interest. 

http://patches.dpdk.org/patch/67722/


> 
> 
> 
> >
> >

Re: [dpdk-dev] [PATCH] [v1 1/1] examples/l2fwd: add cmdline option for forwarding port info

[Vamsi Krishna Attunuru]

> -----Original Message-----
> From: Andrzej Ostruszka [C] <aostruszka@marvell.com>
> Sent: Friday, April 3, 2020 6:22 PM
> To: Vamsi Krishna Attunuru <vattunuru@marvell.com>; dev@dpdk.org
> Cc: jerinjacobk@gmail.com; bruce.richardson@intel.com;
> david.marchand@redhat.com; grive@u256.net; Jerin Jacob Kollanukkaran
> <jerinj@marvell.com>; Pavan Nikhilesh Bhagavatula
> <pbhagavatula@marvell.com>; thomas@monjalon.net
> Subject: Re: [PATCH] [v1 1/1] examples/l2fwd: add cmdline option for
> forwarding port info
> 
> On 4/3/20 5:30 AM, vattunuru@marvell.com wrote:
> > From: Vamsi Attunuru <vattunuru@marvell.com>
> >
> > Current l2fwd application configures adjacent ports as destination
> > ports for forwarding the traffic which is a kind of static mapping
> > that can not be altered by the command line options.
> >
> > Patch adds a config option to pass the forwarding port pair mapping as
> > a command line parameter which allows the user to pass required
> > forwarding port mapping.
> >
> > If no config argument is specified, destination port map is not
> > changed and traffic gets forwarded with existing mapping.
> >
> > When port pair mapping is passed in config option, destination port
> > map is configured and traffic gets forwarded accordingly.
> >
> > Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"
> >
> > With above config option, traffic received from portid = 0 gets
> > forwarded to port = 3 and vice versa, similarly traffic gets forwarded
> > on other port pairs (1,4) and (2,5).
> >
> > Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> > ---
> >  doc/guides/rel_notes/release_20_05.rst             |   6 +
> >  .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
> >  examples/l2fwd/main.c                              | 174 ++++++++++++++++++---
> >  3 files changed, 177 insertions(+), 21 deletions(-)
> >
> > diff --git a/doc/guides/rel_notes/release_20_05.rst
> > b/doc/guides/rel_notes/release_20_05.rst
> > index 000bbf5..645400b 100644
> > --- a/doc/guides/rel_notes/release_20_05.rst
> > +++ b/doc/guides/rel_notes/release_20_05.rst
> > @@ -62,6 +62,12 @@ New Features
> >
> >    * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
> >
> > +* **Added --config command line parameter to l2fwd example.**
> > +
> > +  Added new command line option ``--config(port, port)[,(port,
> > + port)]`` to
> 
> Maybe use the actual syntax with '=' or space after 'config'?
> 
> > +  pass forwarding port details.
> > +  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for
> > + more  details of this parameter usage.
> >
> >  Removed Items
> >  -------------
> > diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> > b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> > index 39d6b00..e708f88 100644
> > --- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> > +++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> > @@ -91,7 +91,10 @@ The application requires a number of command line
> options:
> >
> >  .. code-block:: console
> >
> > -    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
> > +    ./build/l2fwd [EAL options] -- -p PORTMASK
> > +                                   [-q NQ]
> > +                                   --[no-]mac-updating
> > +                                   --config(port, port)[,(port,
> > + port)]
> 
> Ditto
> 
> >
> >  where,
> >
> > @@ -99,7 +102,10 @@ where,
> >
> >  *   q NQ: A number of queues (=ports) per lcore (default is 1)
> >
> > -*   --[no-]mac-updating: Enable or disable MAC addresses updating
> (enabled by default).
> > +*   --[no-]mac-updating: Enable or disable MAC addresses updating
> (enabled by default)
> > +
> > +*   --config(port,port)[,(port,port)]: Determines which ports are mapped
> to
> > +    which ports for packet forwarding.
> 
> Ditto
> 
> >
> >  To run the application in linux environment with 4 lcores, 16 ports
> > and 8 RX queues per lcore and MAC address  updating enabled, issue the
> command:
> > @@ -108,6 +114,14 @@ updating enabled, issue the command:
> >
> >      $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
> >
> > +To run the application in linux environment with 4 lcores, 4 ports, 8
> > +RX queues per lcore and MAC address updating enabled, to forward RX
> > +traffic of ports 0 & 1 on ports 2 & 3 respectively and vice versa, issue the
> command:
> > +
> > +.. code-block:: console
> > +
> > +    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --config="(0,2)(1,3)"
> > +
> >  Refer to the *DPDK Getting Started Guide* for general information on
> > running applications  and the Environment Abstraction Layer (EAL) options.
> >
> > diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c index
> > 09257aa..a8b98b9 100644
> > --- a/examples/l2fwd/main.c
> > +++ b/examples/l2fwd/main.c
> > @@ -38,6 +38,7 @@
> >  #include <rte_ethdev.h>
> >  #include <rte_mempool.h>
> >  #include <rte_mbuf.h>
> > +#include <rte_string_fns.h>
> >
> >  static volatile bool force_quit;
> >
> > @@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
> >  /* list of enabled ports */
> >  static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
> >
> > +struct port_pair_params {
> > +#define NUM_PORTS	2
> > +	uint16_t port[NUM_PORTS];
> > +} __rte_cache_aligned;
> > +
> > +static struct port_pair_params
> > +port_pair_params_array[RTE_MAX_ETHPORTS];
> > +static struct port_pair_params *port_pair_params; static uint16_t
> > +nb_port_pair_params = 1;
> 
> Why this initialization to 1?  Below (during parsing) it is initialized to 0, so I
> would leave this default initialized.
> 
> > +
> >  static unsigned int l2fwd_rx_queue_per_lcore = 1;
> >
> >  #define MAX_RX_QUEUE_PER_LCORE 16
> > @@ -319,6 +329,60 @@ l2fwd_parse_portmask(const char *portmask)
> >  	return pm;
> >  }
> >
> > +static int
> > +l2fwd_parse_port_pair_config(const char *q_arg) {
> > +	enum fieldnames {
> > +		FLD_PORT1 = 0,
> > +		FLD_PORT2,
> > +		_NUM_FLD
> > +	};
> > +	unsigned long int_fld[_NUM_FLD];
> > +	const char *p, *p0 = q_arg;
> > +	char *str_fld[_NUM_FLD];
> > +	unsigned int size;
> > +	char s[256];
> > +	char *end;
> > +	int i;
> > +
> > +	nb_port_pair_params = 0;
> > +
> > +	while ((p = strchr(p0, '(')) != NULL) {
> > +		++p;
> > +		p0 = strchr(p, ')');
> > +		if (p0 == NULL)
> > +			return -1;
> > +
> > +		size = p0 - p;
> > +		if (size >= sizeof(s))
> > +			return -1;
> > +
> > +		snprintf(s, sizeof(s), "%.*s", size, p);
> 
> Same comment as for recent l2fwd-event - this can be simple memcpy.
> 
> > +		if (rte_strsplit(s, sizeof(s), str_fld,
> > +				 _NUM_FLD, ',') != _NUM_FLD)
> > +			return -1;
> > +		for (i = 0; i < _NUM_FLD; i++) {
> > +			errno = 0;
> > +			int_fld[i] = strtoul(str_fld[i], &end, 0);
> > +			if (errno != 0 || end == str_fld[i] ||
> > +			    int_fld[i] > RTE_MAX_ETHPORTS)
> 
> I think this check should be >=, RTE_MAX_ETHPORTS is an invalid port id.
> 
> > +				return -1;
> > +		}
> > +		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
> > +			printf("exceeded max number of port pair params:
> %hu\n",
> > +				nb_port_pair_params);
> > +			return -1;
> > +		}
> > +		port_pair_params_array[nb_port_pair_params].port[0] =
> > +				(uint16_t)int_fld[FLD_PORT1];
> > +		port_pair_params_array[nb_port_pair_params].port[1] =
> > +				(uint16_t)int_fld[FLD_PORT2];
> > +		++nb_port_pair_params;
> > +	}
> > +	port_pair_params = port_pair_params_array;
> > +	return 0;
> > +}
> 
> [...]
> 
> > @@ -565,26 +686,40 @@ main(int argc, char **argv)
> >  		l2fwd_dst_ports[portid] = 0;
> >  	last_port = 0;
> >
> > -	/*
> > -	 * Each logical core is assigned a dedicated TX queue on each port.
> > -	 */
> > -	RTE_ETH_FOREACH_DEV(portid) {
> > -		/* skip ports that are not enabled */
> > -		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
> > -			continue;
> > +	/* populate destination port details */
> > +	if (port_pair_params != NULL) {
> > +		uint16_t idx;
> > +
> > +		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
> > +			if (idx % 2 == 0) {
> > +				portid = port_pair_params[idx >> 1].port[0];
> > +				l2fwd_dst_ports[portid] =
> > +					port_pair_params[idx >> 1].port[1];
> > +			} else {
> > +				portid = port_pair_params[idx >> 1].port[1];
> > +				l2fwd_dst_ports[portid] =
> > +					port_pair_params[idx >> 1].port[0];
> > +			}
> 
> I think this conditional can be rephrased like:
> 
> 	uint16_t idx, p;
> ...
> 	p = idx & 1;
> 	port_id = port_pair_params[idx >> 1].port[p];
> 	l2fwd_dst_ports[port_id] =
> 		port_pair_params[idx >> 1].port[p ^ 1];
> 
> but I'm not sure which one is more readable - I leave you the decision,  feel
> free to skip this comment altogether :).
> 
> > +		}
> > +	} else {
> > +		RTE_ETH_FOREACH_DEV(portid) {
> > +			/* skip ports that are not enabled */
> > +			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
> > +				continue;
> > +
> > +			if (nb_ports_in_mask % 2) {
> > +				l2fwd_dst_ports[portid] = last_port;
> > +				l2fwd_dst_ports[last_port] = portid;
> > +			} else {
> > +				last_port = portid;
> > +			}
> >
> > +			nb_ports_in_mask++;
> > +		}
> >  		if (nb_ports_in_mask % 2) {
> > -			l2fwd_dst_ports[portid] = last_port;
> > -			l2fwd_dst_ports[last_port] = portid;
> > +			printf("Notice: odd number of ports in
> portmask.\n");
> > +			l2fwd_dst_ports[last_port] = last_port;
> >  		}
> > -		else
> > -			last_port = portid;
> > -
> > -		nb_ports_in_mask++;
> > -	}
> > -	if (nb_ports_in_mask % 2) {
> > -		printf("Notice: odd number of ports in portmask.\n");
> > -		l2fwd_dst_ports[last_port] = last_port;
> >  	}
> >
> >  	rx_lcore_id = 0;
> > @@ -613,7 +748,8 @@ main(int argc, char **argv)
> >
> >  		qconf->rx_port_list[qconf->n_rx_port] = portid;
> >  		qconf->n_rx_port++;
> > -		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
> > +		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
> > +		       portid, l2fwd_dst_ports[portid]);
> >  	}
> >
> >  	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +
> MAX_PKT_BURST +
> >
> 
> Apart from these comments:
> 
> Reviewed-by: Andrzej Ostruszka <aostruszka@marvell.com>
> 
> With regards
> Andrzej Ostruszka

Thanks Andrzej,
Acked all the comments in V2.

[dpdk-dev] [PATCH] [v2 1/1] examples/l2fwd: add cmdline option for forwarding port info

[vattunuru]

From: Vamsi Attunuru <vattunuru@marvell.com>

Current l2fwd application configures adjacent ports as destination
ports for forwarding the traffic which is a kind of static mapping
that can not be altered by the command line options.

Patch adds a config option to pass the forwarding port pair mapping
as a command line parameter which allows the user to pass required
forwarding port mapping.

If no config argument is specified, destination port map is not
changed and traffic gets forwarded with existing mapping.

When port pair mapping is passed in config option, destination port map
is configured and traffic gets forwarded accordingly.

Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

With above config option, traffic received from portid = 0 gets forwarded
to port = 3 and vice versa, similarly traffic gets forwarded on other port
pairs (1,4) and (2,5).

Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
---
V2 Changes:

* Fix command option format in docs.
* Use memcpy instead of snprintf.
* Rephrase if-else condition.

 doc/guides/rel_notes/release_20_05.rst             |   6 +
 .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
 examples/l2fwd/main.c                              | 169 ++++++++++++++++++---
 3 files changed, 172 insertions(+), 21 deletions(-)

diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index 000bbf5..69e1798 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -62,6 +62,12 @@ New Features
 
   * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
 
+* **Added --config command line parameter to l2fwd example.**
+
+  Added new command line option ``--config="(port, port)[,(port, port)]"`` to
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
 
 Removed Items
 -------------
diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
index 39d6b00..5c0ee05 100644
--- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
+++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
@@ -91,7 +91,10 @@ The application requires a number of command line options:
 
 .. code-block:: console
 
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --config="(port, port)[,(port, port)]"
 
 where,
 
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --config="(port,port)[,(port,port)]": Determines which ports are mapped to
+    which ports for packet forwarding.
 
 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore and MAC address updating enabled, to forward RX traffic of ports 0 & 1
+on ports 2 & 3 respectively and vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --config="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.
 
diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
index 09257aa..98ce164 100644
--- a/examples/l2fwd/main.c
+++ b/examples/l2fwd/main.c
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>
 
 static volatile bool force_quit;
 
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;
+
+static struct port_pair_params port_pair_params_array[RTE_MAX_ETHPORTS];
+static struct port_pair_params *port_pair_params;
+static uint16_t nb_port_pair_params;
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;
 
 #define MAX_RX_QUEUE_PER_LCORE 16
@@ -319,6 +329,60 @@ l2fwd_parse_portmask(const char *portmask)
 	return pm;
 }
 
+static int
+l2fwd_parse_port_pair_config(const char *q_arg)
+{
+	enum fieldnames {
+		FLD_PORT1 = 0,
+		FLD_PORT2,
+		_NUM_FLD
+	};
+	unsigned long int_fld[_NUM_FLD];
+	const char *p, *p0 = q_arg;
+	char *str_fld[_NUM_FLD];
+	unsigned int size;
+	char s[256];
+	char *end;
+	int i;
+
+	nb_port_pair_params = 0;
+
+	while ((p = strchr(p0, '(')) != NULL) {
+		++p;
+		p0 = strchr(p, ')');
+		if (p0 == NULL)
+			return -1;
+
+		size = p0 - p;
+		if (size >= sizeof(s))
+			return -1;
+
+		memcpy(s, p, size);
+		if (rte_strsplit(s, sizeof(s), str_fld,
+				 _NUM_FLD, ',') != _NUM_FLD)
+			return -1;
+		for (i = 0; i < _NUM_FLD; i++) {
+			errno = 0;
+			int_fld[i] = strtoul(str_fld[i], &end, 0);
+			if (errno != 0 || end == str_fld[i] ||
+			    int_fld[i] >= RTE_MAX_ETHPORTS)
+				return -1;
+		}
+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
+			printf("exceeded max number of port pair params: %hu\n",
+				nb_port_pair_params);
+			return -1;
+		}
+		port_pair_params_array[nb_port_pair_params].port[0] =
+				(uint16_t)int_fld[FLD_PORT1];
+		port_pair_params_array[nb_port_pair_params].port[1] =
+				(uint16_t)int_fld[FLD_PORT2];
+		++nb_port_pair_params;
+	}
+	port_pair_params = port_pair_params_array;
+	return 0;
+}
+
 static unsigned int
 l2fwd_parse_nqueue(const char *q_arg)
 {
@@ -361,6 +425,7 @@ static const char short_options[] =
 
 #define CMD_LINE_OPT_MAC_UPDATING "mac-updating"
 #define CMD_LINE_OPT_NO_MAC_UPDATING "no-mac-updating"
+#define CMD_LINE_OPT_CONFIG "config"
 
 enum {
 	/* long options mapped to a short option */
@@ -368,11 +433,13 @@ enum {
 	/* first long only option value must be >= 256, so that we won't
 	 * conflict with short options */
 	CMD_LINE_OPT_MIN_NUM = 256,
+	CMD_LINE_OPT_CONFIG_NUM,
 };
 
 static const struct option lgopts[] = {
 	{ CMD_LINE_OPT_MAC_UPDATING, no_argument, &mac_updating, 1},
 	{ CMD_LINE_OPT_NO_MAC_UPDATING, no_argument, &mac_updating, 0},
+	{ CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
 	{NULL, 0, 0, 0}
 };
 
@@ -386,6 +453,7 @@ l2fwd_parse_args(int argc, char **argv)
 	char *prgname = argv[0];
 
 	argvopt = argv;
+	port_pair_params = NULL;
 
 	while ((opt = getopt_long(argc, argvopt, short_options,
 				  lgopts, &option_index)) != EOF) {
@@ -423,7 +491,13 @@ l2fwd_parse_args(int argc, char **argv)
 			break;
 
 		/* long options */
-		case 0:
+		case CMD_LINE_OPT_CONFIG_NUM:
+			ret = l2fwd_parse_port_pair_config(optarg);
+			if (ret) {
+				fprintf(stderr, "Invalid config\n");
+				l2fwd_usage(prgname);
+				return -1;
+			}
 			break;
 
 		default:
@@ -440,6 +514,48 @@ l2fwd_parse_args(int argc, char **argv)
 	return ret;
 }
 
+/*
+ * Check port pair config with enabled port mask,
+ * and for valid port pair combinations.
+ */
+static int
+check_port_pair_config(void)
+{
+	uint32_t port_pair_config_mask = 0;
+	uint32_t port_pair_mask = 0;
+	uint16_t index, i, portid;
+
+	for (index = 0; index < nb_port_pair_params; index++) {
+		port_pair_mask = 0;
+
+		for (i = 0; i < NUM_PORTS; i++)  {
+			portid = port_pair_params[index].port[i];
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
+				printf("port %u is not enabled in port mask\n",
+				       portid);
+				return -1;
+			}
+			if (!rte_eth_dev_is_valid_port(portid)) {
+				printf("port %u is not present on the board\n",
+				       portid);
+				return -1;
+			}
+
+			port_pair_mask |= 1 << portid;
+		}
+
+		if (port_pair_config_mask & port_pair_mask) {
+			printf("port %u is used in other port pairs\n", portid);
+			return -1;
+		}
+		port_pair_config_mask |= port_pair_mask;
+	}
+
+	l2fwd_enabled_port_mask &= port_pair_config_mask;
+
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -555,6 +671,11 @@ main(int argc, char **argv)
 	if (nb_ports == 0)
 		rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
 
+	if (port_pair_params != NULL) {
+		if (check_port_pair_config() < 0)
+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
+	}
+
 	/* check port mask to possible port mask */
 	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
 		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
@@ -565,26 +686,35 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;
 
-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx, p;
 
-		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			p = idx & 1;
+			portid = port_pair_params[idx >> 1].port[p];
+			l2fwd_dst_ports[portid] =
+				port_pair_params[idx >> 1].port[p ^ 1];
 		}
-		else
-			last_port = portid;
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;
 
-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {
+				last_port = portid;
+			}
+
+			nb_ports_in_mask++;
+		}
+		if (nb_ports_in_mask % 2) {
+			printf("Notice: odd number of ports in portmask.\n");
+			l2fwd_dst_ports[last_port] = last_port;
+		}
 	}
 
 	rx_lcore_id = 0;
@@ -613,7 +743,8 @@ main(int argc, char **argv)
 
 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}
 
 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd + MAX_PKT_BURST +
-- 
2.8.4

Re: [dpdk-dev] [PATCH] [v2 1/1] examples/l2fwd: add cmdline option for forwarding port info

[Andrzej Ostruszka [C]]

On 4/5/20 5:52 AM, vattunuru@marvell.com wrote:
> From: Vamsi Attunuru <vattunuru@marvell.com>
> 
> Current l2fwd application configures adjacent ports as destination
> ports for forwarding the traffic which is a kind of static mapping
> that can not be altered by the command line options.
> 
> Patch adds a config option to pass the forwarding port pair mapping
> as a command line parameter which allows the user to pass required
> forwarding port mapping.
> 
> If no config argument is specified, destination port map is not
> changed and traffic gets forwarded with existing mapping.
> 
> When port pair mapping is passed in config option, destination port map
> is configured and traffic gets forwarded accordingly.
> 
> Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"
> 
> With above config option, traffic received from portid = 0 gets forwarded
> to port = 3 and vice versa, similarly traffic gets forwarded on other port
> pairs (1,4) and (2,5).
> 
> Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
> ---
> V2 Changes:
> 
> * Fix command option format in docs.
> * Use memcpy instead of snprintf.
> * Rephrase if-else condition.
> 
>  doc/guides/rel_notes/release_20_05.rst             |   6 +
>  .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
>  examples/l2fwd/main.c                              | 169 ++++++++++++++++++---
>  3 files changed, 172 insertions(+), 21 deletions(-)

Acked-by: Andrzej Ostruszka <aostruszka@marvell.com>

Re: [dpdk-dev] [PATCH] [v2 1/1] examples/l2fwd: add cmdline option for forwarding port info

[Thomas Monjalon]

05/04/2020 05:52, vattunuru@marvell.com:
> From: Vamsi Attunuru <vattunuru@marvell.com>
> 
> Current l2fwd application configures adjacent ports as destination
> ports for forwarding the traffic which is a kind of static mapping
> that can not be altered by the command line options.
> 
> Patch adds a config option to pass the forwarding port pair mapping
> as a command line parameter which allows the user to pass required
> forwarding port mapping.
> 
> If no config argument is specified, destination port map is not
> changed and traffic gets forwarded with existing mapping.
> 
> When port pair mapping is passed in config option, destination port map
> is configured and traffic gets forwarded accordingly.
> 
> Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

The option name "--config" is vague.
Could we call it "--portmap", or something similar?

[dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

[pbhagavatula]

From: Vamsi Attunuru <vattunuru@marvell.com>

Current l2fwd application statically configures adjacent ports as
destination ports for forwarding the traffic.

Add a portmap option to pass the forwarding port pair mapping which allows
the user to configure forwarding port mapping.

If no portmap argument is specified, destination port map is not
changed and traffic gets forwarded with existing mapping.

To align port/queue configuration of each lcore with destination port
map, port/queue configuration of each lcore gets modified when portmap
option is specified.

Ex: ./l2fwd -c 0xff -- -p 0x3f -q 2 --portmap="(0,3)(1,4)(2,5)"

With above portmap option, traffic received from portid = 0 gets forwarded
to port = 3 and vice versa, similarly traffic gets forwarded on other port
pairs (1,4) and (2,5)

Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Andrzej Ostruszka <aostruszka@marvell.com>
---
 v3 Changes:
 * s/config/portmap/
 * rebase on master

 v2 Changes:
 * Fix command option format in docs.
 * Use memcpy instead of snprintf.
 * Rephrase if-else condition.

 doc/guides/rel_notes/release_20_05.rst        |   7 +
 .../sample_app_ug/l2_forward_real_virtual.rst |  18 +-
 examples/l2fwd/main.c                         | 182 +++++++++++++++---
 3 files changed, 181 insertions(+), 26 deletions(-)

diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index b124c3f28..019bec5d7 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -212,6 +212,13 @@ New Features
   * Added IPsec inbound load-distribution support for ipsec-secgw application
     using NIC load distribution feature(Flow Director).

+* **Added --portmap command line parameter to l2fwd example.**
+
+  Added new command line option ``--portmap="(port, port)[,(port, port)]"`` to
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
+

 Removed Items
 -------------
diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
index 39d6b0067..b54321b5b 100644
--- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
+++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
@@ -91,7 +91,10 @@ The application requires a number of command line options:

 .. code-block:: console

-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --portmap="(port, port)[,(port, port)]"

 where,

@@ -99,7 +102,10 @@ where,

 *   q NQ: A number of queues (=ports) per lcore (default is 1)

-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --portmap="(port,port)[,(port,port)]": Determines which ports are mapped to
+    which ports for packet forwarding.

 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +114,14 @@ updating enabled, issue the command:

     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff

+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore and MAC address updating enabled, to forward RX traffic of ports 0 & 1
+on ports 2 & 3 respectively and vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.

diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
index 88ddfe589..81861a22a 100644
--- a/examples/l2fwd/main.c
+++ b/examples/l2fwd/main.c
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>

 static volatile bool force_quit;

@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];

+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;
+
+static struct port_pair_params port_pair_params_array[RTE_MAX_ETHPORTS];
+static struct port_pair_params *port_pair_params;
+static uint16_t nb_port_pair_params;
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;

 #define MAX_RX_QUEUE_PER_LCORE 16
@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
 	printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
 	       "  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 	       "  -q NQ: number of queue (=ports) per lcore (default is 1)\n"
-		   "  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default, 86400 maximum)\n"
-		   "  --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)\n"
-		   "      When enabled:\n"
-		   "       - The source MAC address is replaced by the TX port MAC address\n"
-		   "       - The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID\n",
+	       "  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default, 86400 maximum)\n"
+	       "  --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)\n"
+	       "      When enabled:\n"
+	       "       - The source MAC address is replaced by the TX port MAC address\n"
+	       "       - The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID\n"
+	       "  --portmap: Configure forwarding port pair mapping\n"
+	       "	      Default: alternate port pairs\n\n",
 	       prgname);
 }

@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char *portmask)
 	return pm;
 }

+static int
+l2fwd_parse_port_pair_config(const char *q_arg)
+{
+	enum fieldnames {
+		FLD_PORT1 = 0,
+		FLD_PORT2,
+		_NUM_FLD
+	};
+	unsigned long int_fld[_NUM_FLD];
+	const char *p, *p0 = q_arg;
+	char *str_fld[_NUM_FLD];
+	unsigned int size;
+	char s[256];
+	char *end;
+	int i;
+
+	nb_port_pair_params = 0;
+
+	while ((p = strchr(p0, '(')) != NULL) {
+		++p;
+		p0 = strchr(p, ')');
+		if (p0 == NULL)
+			return -1;
+
+		size = p0 - p;
+		if (size >= sizeof(s))
+			return -1;
+
+		memcpy(s, p, size);
+		s[size] = '\0';
+		if (rte_strsplit(s, sizeof(s), str_fld,
+				 _NUM_FLD, ',') != _NUM_FLD)
+			return -1;
+		for (i = 0; i < _NUM_FLD; i++) {
+			errno = 0;
+			int_fld[i] = strtoul(str_fld[i], &end, 0);
+			if (errno != 0 || end == str_fld[i] ||
+			    int_fld[i] >= RTE_MAX_ETHPORTS)
+				return -1;
+		}
+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
+			printf("exceeded max number of port pair params: %hu\n",
+				nb_port_pair_params);
+			return -1;
+		}
+		port_pair_params_array[nb_port_pair_params].port[0] =
+				(uint16_t)int_fld[FLD_PORT1];
+		port_pair_params_array[nb_port_pair_params].port[1] =
+				(uint16_t)int_fld[FLD_PORT2];
+		++nb_port_pair_params;
+	}
+	port_pair_params = port_pair_params_array;
+	return 0;
+}
+
 static unsigned int
 l2fwd_parse_nqueue(const char *q_arg)
 {
@@ -361,6 +428,7 @@ static const char short_options[] =

 #define CMD_LINE_OPT_MAC_UPDATING "mac-updating"
 #define CMD_LINE_OPT_NO_MAC_UPDATING "no-mac-updating"
+#define CMD_LINE_OPT_PORTMAP_CONFIG "portmap"

 enum {
 	/* long options mapped to a short option */
@@ -368,11 +436,13 @@ enum {
 	/* first long only option value must be >= 256, so that we won't
 	 * conflict with short options */
 	CMD_LINE_OPT_MIN_NUM = 256,
+	CMD_LINE_OPT_PORTMAP_NUM,
 };

 static const struct option lgopts[] = {
 	{ CMD_LINE_OPT_MAC_UPDATING, no_argument, &mac_updating, 1},
 	{ CMD_LINE_OPT_NO_MAC_UPDATING, no_argument, &mac_updating, 0},
+	{ CMD_LINE_OPT_PORTMAP_CONFIG, 1, 0, CMD_LINE_OPT_PORTMAP_NUM},
 	{NULL, 0, 0, 0}
 };

@@ -386,6 +456,7 @@ l2fwd_parse_args(int argc, char **argv)
 	char *prgname = argv[0];

 	argvopt = argv;
+	port_pair_params = NULL;

 	while ((opt = getopt_long(argc, argvopt, short_options,
 				  lgopts, &option_index)) != EOF) {
@@ -423,7 +494,13 @@ l2fwd_parse_args(int argc, char **argv)
 			break;

 		/* long options */
-		case 0:
+		case CMD_LINE_OPT_PORTMAP_NUM:
+			ret = l2fwd_parse_port_pair_config(optarg);
+			if (ret) {
+				fprintf(stderr, "Invalid config\n");
+				l2fwd_usage(prgname);
+				return -1;
+			}
 			break;

 		default:
@@ -440,6 +517,48 @@ l2fwd_parse_args(int argc, char **argv)
 	return ret;
 }

+/*
+ * Check port pair config with enabled port mask,
+ * and for valid port pair combinations.
+ */
+static int
+check_port_pair_config(void)
+{
+	uint32_t port_pair_config_mask = 0;
+	uint32_t port_pair_mask = 0;
+	uint16_t index, i, portid;
+
+	for (index = 0; index < nb_port_pair_params; index++) {
+		port_pair_mask = 0;
+
+		for (i = 0; i < NUM_PORTS; i++)  {
+			portid = port_pair_params[index].port[i];
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
+				printf("port %u is not enabled in port mask\n",
+				       portid);
+				return -1;
+			}
+			if (!rte_eth_dev_is_valid_port(portid)) {
+				printf("port %u is not present on the board\n",
+				       portid);
+				return -1;
+			}
+
+			port_pair_mask |= 1 << portid;
+		}
+
+		if (port_pair_config_mask & port_pair_mask) {
+			printf("port %u is used in other port pairs\n", portid);
+			return -1;
+		}
+		port_pair_config_mask |= port_pair_mask;
+	}
+
+	l2fwd_enabled_port_mask &= port_pair_config_mask;
+
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -555,6 +674,11 @@ main(int argc, char **argv)
 	if (nb_ports == 0)
 		rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");

+	if (port_pair_params != NULL) {
+		if (check_port_pair_config() < 0)
+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
+	}
+
 	/* check port mask to possible port mask */
 	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
 		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
@@ -565,26 +689,35 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;

-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx, p;

-		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			p = idx & 1;
+			portid = port_pair_params[idx >> 1].port[p];
+			l2fwd_dst_ports[portid] =
+				port_pair_params[idx >> 1].port[p ^ 1];
 		}
-		else
-			last_port = portid;
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;

-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {
+				last_port = portid;
+			}
+
+			nb_ports_in_mask++;
+		}
+		if (nb_ports_in_mask % 2) {
+			printf("Notice: odd number of ports in portmask.\n");
+			l2fwd_dst_ports[last_port] = last_port;
+		}
 	}

 	rx_lcore_id = 0;
@@ -613,7 +746,8 @@ main(int argc, char **argv)

 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}

 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd + MAX_PKT_BURST +
--
2.17.1

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

[Sunil Kumar Kori]

>-----Original Message-----
>From: pbhagavatula@marvell.com <pbhagavatula@marvell.com>
>Sent: Monday, April 27, 2020 1:30 PM
>To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; thomas@monjalon.net;
>John McNamara <john.mcnamara@intel.com>; Marko Kovacevic
><marko.kovacevic@intel.com>; Ori Kam <orika@mellanox.com>; Bruce
>Richardson <bruce.richardson@intel.com>; Radu Nicolau
><radu.nicolau@intel.com>; Akhil Goyal <akhil.goyal@nxp.com>; Tomasz
>Kantecki <tomasz.kantecki@intel.com>; Sunil Kumar Kori
><skori@marvell.com>; Pavan Nikhilesh Bhagavatula
><pbhagavatula@marvell.com>
>Cc: Andrzej Ostruszka [C] <aostruszka@marvell.com>; dev@dpdk.org; Vamsi
>Krishna Attunuru <vattunuru@marvell.com>
>Subject: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for
>forwarding port info
>
>From: Vamsi Attunuru <vattunuru@marvell.com>
>
>Current l2fwd application statically configures adjacent ports as destination
>ports for forwarding the traffic.
>
>Add a portmap option to pass the forwarding port pair mapping which allows
>the user to configure forwarding port mapping.
>
>If no portmap argument is specified, destination port map is not changed and
>traffic gets forwarded with existing mapping.
>
>To align port/queue configuration of each lcore with destination port map,
>port/queue configuration of each lcore gets modified when portmap option is
>specified.
>
>Ex: ./l2fwd -c 0xff -- -p 0x3f -q 2 --portmap="(0,3)(1,4)(2,5)"
>
>With above portmap option, traffic received from portid = 0 gets forwarded to
>port = 3 and vice versa, similarly traffic gets forwarded on other port pairs
>(1,4) and (2,5)
>
>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>Acked-by: Andrzej Ostruszka <aostruszka@marvell.com>
>---
> v3 Changes:
> * s/config/portmap/
> * rebase on master
>
> v2 Changes:
> * Fix command option format in docs.
> * Use memcpy instead of snprintf.
> * Rephrase if-else condition.
>
> doc/guides/rel_notes/release_20_05.rst        |   7 +
> .../sample_app_ug/l2_forward_real_virtual.rst |  18 +-
> examples/l2fwd/main.c                         | 182 +++++++++++++++---
> 3 files changed, 181 insertions(+), 26 deletions(-)
>
>diff --git a/doc/guides/rel_notes/release_20_05.rst
>b/doc/guides/rel_notes/release_20_05.rst
>index b124c3f28..019bec5d7 100644
>--- a/doc/guides/rel_notes/release_20_05.rst
>+++ b/doc/guides/rel_notes/release_20_05.rst
>@@ -212,6 +212,13 @@ New Features
>   * Added IPsec inbound load-distribution support for ipsec-secgw application
>     using NIC load distribution feature(Flow Director).
>
>+* **Added --portmap command line parameter to l2fwd example.**
>+
>+  Added new command line option ``--portmap="(port, port)[,(port,
>+ port)]"`` to  pass forwarding port details.
>+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for
>+ more  details of this parameter usage.
>+
>
> Removed Items
> -------------
>diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>index 39d6b0067..b54321b5b 100644
>--- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>+++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>@@ -91,7 +91,10 @@ The application requires a number of command line
>options:
>
> .. code-block:: console
>
>-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
>+    ./build/l2fwd [EAL options] -- -p PORTMASK
>+                                   [-q NQ]
>+                                   --[no-]mac-updating
>+                                   --portmap="(port, port)[,(port, port)]"

Will it be better to represent as [--portmap="(port, port)[,(port, port)]] as it is a optional parameter ?

>
> where,
>
>@@ -99,7 +102,10 @@ where,
>
> *   q NQ: A number of queues (=ports) per lcore (default is 1)
>
>-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled
>by default).
>+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled
>by default)
>+
>+*   --portmap="(port,port)[,(port,port)]": Determines which ports are
>mapped to
>+    which ports for packet forwarding.

May be rephrased a bit shorter as "Determines forwarding ports mapping". 
>
> To run the application in linux environment with 4 lcores, 16 ports and 8 RX
>queues per lcore and MAC address  updating enabled, issue the command:
>@@ -108,6 +114,14 @@ updating enabled, issue the command:
>
>     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
>
>+To run the application in linux environment with 4 lcores, 4 ports, 8
>+RX queues per lcore and MAC address updating enabled, to forward RX
>+traffic of ports 0 & 1 on ports 2 & 3 respectively and vice versa, issue the
>command:

IMO, No need to mentioned information about MAC address update enabled because it
is enabled by default and no param is passed in below command. So information is not
relevant here.

>+
>+.. code-block:: console
>+
>+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
It looks like comma is missed in between two port map information.
>+
> Refer to the *DPDK Getting Started Guide* for general information on running
>applications  and the Environment Abstraction Layer (EAL) options.
>
>diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c index
>88ddfe589..81861a22a 100644
>--- a/examples/l2fwd/main.c
>+++ b/examples/l2fwd/main.c
>@@ -38,6 +38,7 @@
> #include <rte_ethdev.h>
> #include <rte_mempool.h>
> #include <rte_mbuf.h>
>+#include <rte_string_fns.h>
>
> static volatile bool force_quit;
>
>@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
> /* list of enabled ports */
> static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
>
>+struct port_pair_params {
>+#define NUM_PORTS	2
>+	uint16_t port[NUM_PORTS];
>+} __rte_cache_aligned;

Is there any specific reason to use this syntax to declare two ports instead of following 
struct port_pair_params {
     uint16_t port1;
     uint16_t port2;
}; 

>+
>+static struct port_pair_params
>+port_pair_params_array[RTE_MAX_ETHPORTS];
Should not be RTE_MAX_ETHPORTS/2 if only 1:1 mapping is allowed  ?
As I understood it is used to store CLI port mapping, I think it is better to
Use any other MACRO which define maximum number of port map parameter.
If you consider it, then it can also be updated in documentation.

>+static struct port_pair_params *port_pair_params; static uint16_t
>+nb_port_pair_params;
>+
> static unsigned int l2fwd_rx_queue_per_lcore = 1;
>
> #define MAX_RX_QUEUE_PER_LCORE 16
>@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
> 	printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
> 	       "  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
> 	       "  -q NQ: number of queue (=ports) per lcore (default is 1)\n"
>-		   "  -T PERIOD: statistics will be refreshed each PERIOD
>seconds (0 to disable, 10 default, 86400 maximum)\n"
>-		   "  --[no-]mac-updating: Enable or disable MAC addresses
>updating (enabled by default)\n"
>-		   "      When enabled:\n"
>-		   "       - The source MAC address is replaced by the TX port
>MAC address\n"
>-		   "       - The destination MAC address is replaced by
>02:00:00:00:00:TX_PORT_ID\n",
>+	       "  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to
>disable, 10 default, 86400 maximum)\n"
>+	       "  --[no-]mac-updating: Enable or disable MAC addresses updating
>(enabled by default)\n"
>+	       "      When enabled:\n"
>+	       "       - The source MAC address is replaced by the TX port MAC
>address\n"
>+	       "       - The destination MAC address is replaced by
>02:00:00:00:00:TX_PORT_ID\n"
>+	       "  --portmap: Configure forwarding port pair mapping\n"
>+	       "	      Default: alternate port pairs\n\n",

IMO, Indentation changes for other parameters should in separate patch. 
> 	       prgname);
> }
>
>@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char *portmask)
> 	return pm;
> }
>
>+static int
>+l2fwd_parse_port_pair_config(const char *q_arg) {
>+	enum fieldnames {
>+		FLD_PORT1 = 0,
>+		FLD_PORT2,
>+		_NUM_FLD
>+	};
>+	unsigned long int_fld[_NUM_FLD];
>+	const char *p, *p0 = q_arg;
>+	char *str_fld[_NUM_FLD];
>+	unsigned int size;
>+	char s[256];
>+	char *end;
>+	int i;
>+
>+	nb_port_pair_params = 0;
>+
>+	while ((p = strchr(p0, '(')) != NULL) {
>+		++p;
>+		p0 = strchr(p, ')');
>+		if (p0 == NULL)
>+			return -1;
>+
>+		size = p0 - p;
>+		if (size >= sizeof(s))
>+			return -1;
>+
>+		memcpy(s, p, size);
>+		s[size] = '\0';
>+		if (rte_strsplit(s, sizeof(s), str_fld,
>+				 _NUM_FLD, ',') != _NUM_FLD)
>+			return -1;
>+		for (i = 0; i < _NUM_FLD; i++) {
>+			errno = 0;
>+			int_fld[i] = strtoul(str_fld[i], &end, 0);
>+			if (errno != 0 || end == str_fld[i] ||
>+			    int_fld[i] >= RTE_MAX_ETHPORTS)
>+				return -1;
>+		}
>+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
>+			printf("exceeded max number of port pair params:
>%hu\n",
>+				nb_port_pair_params);
>+			return -1;
>+		}
>+		port_pair_params_array[nb_port_pair_params].port[0] =
>+				(uint16_t)int_fld[FLD_PORT1];
>+		port_pair_params_array[nb_port_pair_params].port[1] =
>+				(uint16_t)int_fld[FLD_PORT2];
>+		++nb_port_pair_params;
>+	}
>+	port_pair_params = port_pair_params_array;
>+	return 0;
>+}
>+
> static unsigned int
> l2fwd_parse_nqueue(const char *q_arg)
> {
>@@ -361,6 +428,7 @@ static const char short_options[] =
>
> #define CMD_LINE_OPT_MAC_UPDATING "mac-updating"
> #define CMD_LINE_OPT_NO_MAC_UPDATING "no-mac-updating"
>+#define CMD_LINE_OPT_PORTMAP_CONFIG "portmap"
>
> enum {
> 	/* long options mapped to a short option */ @@ -368,11 +436,13 @@
>enum {
> 	/* first long only option value must be >= 256, so that we won't
> 	 * conflict with short options */
> 	CMD_LINE_OPT_MIN_NUM = 256,
>+	CMD_LINE_OPT_PORTMAP_NUM,
> };
>
> static const struct option lgopts[] = {
> 	{ CMD_LINE_OPT_MAC_UPDATING, no_argument, &mac_updating,
>1},
> 	{ CMD_LINE_OPT_NO_MAC_UPDATING, no_argument,
>&mac_updating, 0},
>+	{ CMD_LINE_OPT_PORTMAP_CONFIG, 1, 0,
>CMD_LINE_OPT_PORTMAP_NUM},
> 	{NULL, 0, 0, 0}
> };
>
>@@ -386,6 +456,7 @@ l2fwd_parse_args(int argc, char **argv)
> 	char *prgname = argv[0];
>
> 	argvopt = argv;
>+	port_pair_params = NULL;
>
> 	while ((opt = getopt_long(argc, argvopt, short_options,
> 				  lgopts, &option_index)) != EOF) { @@ -423,7
>+494,13 @@ l2fwd_parse_args(int argc, char **argv)
> 			break;
>
> 		/* long options */
>-		case 0:
>+		case CMD_LINE_OPT_PORTMAP_NUM:
>+			ret = l2fwd_parse_port_pair_config(optarg);
>+			if (ret) {
>+				fprintf(stderr, "Invalid config\n");
>+				l2fwd_usage(prgname);
>+				return -1;
>+			}
> 			break;
>
> 		default:
>@@ -440,6 +517,48 @@ l2fwd_parse_args(int argc, char **argv)
> 	return ret;
> }
>
>+/*
>+ * Check port pair config with enabled port mask,
>+ * and for valid port pair combinations.
>+ */
>+static int
>+check_port_pair_config(void)
>+{
>+	uint32_t port_pair_config_mask = 0;
>+	uint32_t port_pair_mask = 0;
>+	uint16_t index, i, portid;
>+
>+	for (index = 0; index < nb_port_pair_params; index++) {
>+		port_pair_mask = 0;
>+
>+		for (i = 0; i < NUM_PORTS; i++)  {
>+			portid = port_pair_params[index].port[i];
>+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
>+				printf("port %u is not enabled in port
>mask\n",
>+				       portid);
>+				return -1;
>+			}
>+			if (!rte_eth_dev_is_valid_port(portid)) {
>+				printf("port %u is not present on the
>board\n",
>+				       portid);
>+				return -1;
>+			}
>+
>+			port_pair_mask |= 1 << portid;
>+		}
>+
>+		if (port_pair_config_mask & port_pair_mask) {
>+			printf("port %u is used in other port pairs\n", portid);
>+			return -1;
>+		}
>+		port_pair_config_mask |= port_pair_mask;
>+	}
>+
>+	l2fwd_enabled_port_mask &= port_pair_config_mask;
>+
>+	return 0;
>+}
>+
> /* Check the link status of all ports in up to 9s, and print them finally */  static
>void  check_all_ports_link_status(uint32_t port_mask) @@ -555,6 +674,11
>@@ main(int argc, char **argv)
> 	if (nb_ports == 0)
> 		rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
>
>+	if (port_pair_params != NULL) {
>+		if (check_port_pair_config() < 0)
>+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
>+	}
>+
> 	/* check port mask to possible port mask */
> 	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
> 		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
>@@ -565,26 +689,35 @@ main(int argc, char **argv)
> 		l2fwd_dst_ports[portid] = 0;
> 	last_port = 0;
>
>-	/*
>-	 * Each logical core is assigned a dedicated TX queue on each port.
>-	 */
>-	RTE_ETH_FOREACH_DEV(portid) {
>-		/* skip ports that are not enabled */
>-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
>-			continue;
>+	/* populate destination port details */
>+	if (port_pair_params != NULL) {
>+		uint16_t idx, p;
>
>-		if (nb_ports_in_mask % 2) {
>-			l2fwd_dst_ports[portid] = last_port;
>-			l2fwd_dst_ports[last_port] = portid;
>+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
>+			p = idx & 1;
>+			portid = port_pair_params[idx >> 1].port[p];
>+			l2fwd_dst_ports[portid] =
>+				port_pair_params[idx >> 1].port[p ^ 1];
> 		}
>-		else
>-			last_port = portid;
>+	} else {
>+		RTE_ETH_FOREACH_DEV(portid) {
>+			/* skip ports that are not enabled */
>+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
>+				continue;
>
>-		nb_ports_in_mask++;
>-	}
>-	if (nb_ports_in_mask % 2) {
>-		printf("Notice: odd number of ports in portmask.\n");
>-		l2fwd_dst_ports[last_port] = last_port;
>+			if (nb_ports_in_mask % 2) {
>+				l2fwd_dst_ports[portid] = last_port;
>+				l2fwd_dst_ports[last_port] = portid;
>+			} else {
>+				last_port = portid;
>+			}
>+
>+			nb_ports_in_mask++;
>+		}
>+		if (nb_ports_in_mask % 2) {
>+			printf("Notice: odd number of ports in portmask.\n");
>+			l2fwd_dst_ports[last_port] = last_port;
>+		}
> 	}
>
> 	rx_lcore_id = 0;
>@@ -613,7 +746,8 @@ main(int argc, char **argv)
>
> 		qconf->rx_port_list[qconf->n_rx_port] = portid;
> 		qconf->n_rx_port++;
>-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
>+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
>+		       portid, l2fwd_dst_ports[portid]);
> 	}
>
> 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +
>MAX_PKT_BURST +
>--
>2.17.1

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

[Andrzej Ostruszka [C]]

On 27/04/2020 11:19, Sunil Kumar Kori wrote:
>> -----Original Message-----
>> From: pbhagavatula@marvell.com <pbhagavatula@marvell.com>
[...]
>> @@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
>> /* list of enabled ports */
>> static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
>>
>> +struct port_pair_params {
>> +#define NUM_PORTS	2
>> +	uint16_t port[NUM_PORTS];
>> +} __rte_cache_aligned;
> 
> Is there any specific reason to use this syntax to declare two ports instead of following
> struct port_pair_params {
>       uint16_t port1;
>       uint16_t port2;
> };

Initially it was so, but I made a comment that this leads to code 
duplication in check_port_pair_config() (same checks for port1 and port2 
- now handled via loop).  I still stand by this comment :)

With regards
Andrzej Ostruszka

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

[Sunil Kumar Kori]

>-----Original Message-----
>From: Andrzej Ostruszka [C] <aostruszka@marvell.com>
>Sent: Monday, April 27, 2020 3:06 PM
>To: Sunil Kumar Kori <skori@marvell.com>; Pavan Nikhilesh Bhagavatula
><pbhagavatula@marvell.com>; Jerin Jacob Kollanukkaran
><jerinj@marvell.com>; thomas@monjalon.net; John McNamara
><john.mcnamara@intel.com>; Marko Kovacevic
><marko.kovacevic@intel.com>; Ori Kam <orika@mellanox.com>; Bruce
>Richardson <bruce.richardson@intel.com>; Radu Nicolau
><radu.nicolau@intel.com>; Akhil Goyal <akhil.goyal@nxp.com>; Tomasz
>Kantecki <tomasz.kantecki@intel.com>
>Cc: dev@dpdk.org; Vamsi Krishna Attunuru <vattunuru@marvell.com>
>Subject: Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for
>forwarding port info
>
>On 27/04/2020 11:19, Sunil Kumar Kori wrote:
>>> -----Original Message-----
>>> From: pbhagavatula@marvell.com <pbhagavatula@marvell.com>
>[...]
>>> @@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
>>> /* list of enabled ports */
>>> static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
>>>
>>> +struct port_pair_params {
>>> +#define NUM_PORTS	2
>>> +	uint16_t port[NUM_PORTS];
>>> +} __rte_cache_aligned;
>>
>> Is there any specific reason to use this syntax to declare two ports
>> instead of following struct port_pair_params {
>>       uint16_t port1;
>>       uint16_t port2;
>> };
>
>Initially it was so, but I made a comment that this leads to code duplication in
>check_port_pair_config() (same checks for port1 and port2
>- now handled via loop).  I still stand by this comment :)
>
Okay, I think having for two variables only is not a big deal. if it is some higher number then it look fine.
IMO, it is more readable when using suggested way. You can take on this.  

>With regards
>Andrzej Ostruszka

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

[Pavan Nikhilesh Bhagavatula]

>>diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>>b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>>index 39d6b0067..b54321b5b 100644
>>--- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>>+++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>>@@ -91,7 +91,10 @@ The application requires a number of command
>line
>>options:
>>
>> .. code-block:: console
>>
>>-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-
>updating
>>+    ./build/l2fwd [EAL options] -- -p PORTMASK
>>+                                   [-q NQ]
>>+                                   --[no-]mac-updating
>>+                                   --portmap="(port, port)[,(port, port)]"
>
>Will it be better to represent as [--portmap="(port, port)[,(port, port)]]
>as it is a optional parameter ?
>

Will change in v4.

>>
>> where,
>>
>>@@ -99,7 +102,10 @@ where,
>>
>> *   q NQ: A number of queues (=ports) per lcore (default is 1)
>>
>>-*   --[no-]mac-updating: Enable or disable MAC addresses updating
>(enabled
>>by default).
>>+*   --[no-]mac-updating: Enable or disable MAC addresses updating
>(enabled
>>by default)
>>+
>>+*   --portmap="(port,port)[,(port,port)]": Determines which ports are
>>mapped to
>>+    which ports for packet forwarding.
>
>May be rephrased a bit shorter as "Determines forwarding ports
>mapping".

Will change in v4.

>>
>> To run the application in linux environment with 4 lcores, 16 ports and
>8 RX
>>queues per lcore and MAC address  updating enabled, issue the
>command:
>>@@ -108,6 +114,14 @@ updating enabled, issue the command:
>>
>>     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
>>
>>+To run the application in linux environment with 4 lcores, 4 ports, 8
>>+RX queues per lcore and MAC address updating enabled, to forward
>RX
>>+traffic of ports 0 & 1 on ports 2 & 3 respectively and vice versa, issue
>the
>>command:
>
>IMO, No need to mentioned information about MAC address update
>enabled because it
>is enabled by default and no param is passed in below command. So
>information is not
>relevant here.
>

Will change in v4.

>>+
>>+.. code-block:: console
>>+
>>+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
>It looks like comma is missed in between two port map information.
>>+
>> Refer to the *DPDK Getting Started Guide* for general information
>on running
>>applications  and the Environment Abstraction Layer (EAL) options.
>>
>>diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c index
>>88ddfe589..81861a22a 100644
>>--- a/examples/l2fwd/main.c
>>+++ b/examples/l2fwd/main.c
>>@@ -38,6 +38,7 @@
>> #include <rte_ethdev.h>
>> #include <rte_mempool.h>
>> #include <rte_mbuf.h>
>>+#include <rte_string_fns.h>
>>
>> static volatile bool force_quit;
>>
>>@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
>> /* list of enabled ports */
>> static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
>>
>>+struct port_pair_params {
>>+#define NUM_PORTS	2
>>+	uint16_t port[NUM_PORTS];
>>+} __rte_cache_aligned;
>
>Is there any specific reason to use this syntax to declare two ports
>instead of following
>struct port_pair_params {
>     uint16_t port1;
>     uint16_t port2;
>};
>

It was done to reduce code duplication. Will leave as is.

>>+
>>+static struct port_pair_params
>>+port_pair_params_array[RTE_MAX_ETHPORTS];
>Should not be RTE_MAX_ETHPORTS/2 if only 1:1 mapping is allowed  ?
>As I understood it is used to store CLI port mapping, 

RTE_MAX_ETHPORTS/2 will be sufficient.

>I think it is better to
>Use any other MACRO which define maximum number of port map
>parameter.
>If you consider it, then it can also be updated in documentation.
>
>>+static struct port_pair_params *port_pair_params; static uint16_t
>>+nb_port_pair_params;
>>+
>> static unsigned int l2fwd_rx_queue_per_lcore = 1;
>>
>> #define MAX_RX_QUEUE_PER_LCORE 16
>>@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
>> 	printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
>> 	       "  -p PORTMASK: hexadecimal bitmask of ports to
>configure\n"
>> 	       "  -q NQ: number of queue (=ports) per lcore (default is
>1)\n"
>>-		   "  -T PERIOD: statistics will be refreshed each PERIOD
>>seconds (0 to disable, 10 default, 86400 maximum)\n"
>>-		   "  --[no-]mac-updating: Enable or disable MAC
>addresses
>>updating (enabled by default)\n"
>>-		   "      When enabled:\n"
>>-		   "       - The source MAC address is replaced by the TX
>port
>>MAC address\n"
>>-		   "       - The destination MAC address is replaced by
>>02:00:00:00:00:TX_PORT_ID\n",
>>+	       "  -T PERIOD: statistics will be refreshed each PERIOD
>seconds (0 to
>>disable, 10 default, 86400 maximum)\n"
>>+	       "  --[no-]mac-updating: Enable or disable MAC addresses
>updating
>>(enabled by default)\n"
>>+	       "      When enabled:\n"
>>+	       "       - The source MAC address is replaced by the TX port
>MAC
>>address\n"
>>+	       "       - The destination MAC address is replaced by
>>02:00:00:00:00:TX_PORT_ID\n"
>>+	       "  --portmap: Configure forwarding port pair mapping\n"
>>+	       "	      Default: alternate port pairs\n\n",
>
>IMO, Indentation changes for other parameters should in separate
>patch.

It's not a functionality change it just minor code formatting change and doesn't 
deserve a separate patch.

>> 	       prgname);
>> }
>>
>>@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char
>*portmask)
>> 	return pm;
>> }
>>

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

[Sunil Kumar Kori]

Sent from Workspace ONE Boxer
On 27-Apr-2020 10:08 PM, Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com> wrote:
>
> >>diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> >>b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> >>index 39d6b0067..b54321b5b 100644
> >>--- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> >>+++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
> >>@@ -91,7 +91,10 @@ The application requires a number of command
> >line
> >>options:
> >>
> >> .. code-block:: console
> >>
> >>-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-
> >updating
> >>+    ./build/l2fwd [EAL options] -- -p PORTMASK
> >>+                                   [-q NQ]
> >>+                                   --[no-]mac-updating
> >>+                                   --portmap="(port, port)[,(port, port)]"
> >
> >Will it be better to represent as [--portmap="(port, port)[,(port, port)]]
> >as it is a optional parameter ?
> >
>
> Will change in v4.
>
> >>
> >> where,
> >>
> >>@@ -99,7 +102,10 @@ where,
> >>
> >> *   q NQ: A number of queues (=ports) per lcore (default is 1)
> >>
> >>-*   --[no-]mac-updating: Enable or disable MAC addresses updating
> >(enabled
> >>by default).
> >>+*   --[no-]mac-updating: Enable or disable MAC addresses updating
> >(enabled
> >>by default)
> >>+
> >>+*   --portmap="(port,port)[,(port,port)]": Determines which ports are
> >>mapped to
> >>+    which ports for packet forwarding.
> >
> >May be rephrased a bit shorter as "Determines forwarding ports
> >mapping".
>
> Will change in v4.
>
> >>
> >> To run the application in linux environment with 4 lcores, 16 ports and
> >8 RX
> >>queues per lcore and MAC address  updating enabled, issue the
> >command:
> >>@@ -108,6 +114,14 @@ updating enabled, issue the command:
> >>
> >>     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
> >>
> >>+To run the application in linux environment with 4 lcores, 4 ports, 8
> >>+RX queues per lcore and MAC address updating enabled, to forward
> >RX
> >>+traffic of ports 0 & 1 on ports 2 & 3 respectively and vice versa, issue
> >the
> >>command:
> >
> >IMO, No need to mentioned information about MAC address update
> >enabled because it
> >is enabled by default and no param is passed in below command. So
> >information is not
> >relevant here.
> >
>
> Will change in v4.
>
> >>+
> >>+.. code-block:: console
> >>+
> >>+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
> >It looks like comma is missed in between two port map information.
> >>+
> >> Refer to the *DPDK Getting Started Guide* for general information
> >on running
> >>applications  and the Environment Abstraction Layer (EAL) options.
> >>
> >>diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c index
> >>88ddfe589..81861a22a 100644
> >>--- a/examples/l2fwd/main.c
> >>+++ b/examples/l2fwd/main.c
> >>@@ -38,6 +38,7 @@
> >> #include <rte_ethdev.h>
> >> #include <rte_mempool.h>
> >> #include <rte_mbuf.h>
> >>+#include <rte_string_fns.h>
> >>
> >> static volatile bool force_quit;
> >>
> >>@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
> >> /* list of enabled ports */
> >> static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
> >>
> >>+struct port_pair_params {
> >>+#define NUM_PORTS    2
> >>+     uint16_t port[NUM_PORTS];
> >>+} __rte_cache_aligned;
> >
> >Is there any specific reason to use this syntax to declare two ports
> >instead of following
> >struct port_pair_params {
> >     uint16_t port1;
> >     uint16_t port2;
> >};
> >
>
> It was done to reduce code duplication. Will leave as is.
Ack.
>
> >>+
> >>+static struct port_pair_params
> >>+port_pair_params_array[RTE_MAX_ETHPORTS];
> >Should not be RTE_MAX_ETHPORTS/2 if only 1:1 mapping is allowed  ?
> >As I understood it is used to store CLI port mapping,
>
> RTE_MAX_ETHPORTS/2 will be sufficient.
Ack.
>
> >I think it is better to
> >Use any other MACRO which define maximum number of port map
> >parameter.
> >If you consider it, then it can also be updated in documentation.
> >
> >>+static struct port_pair_params *port_pair_params; static uint16_t
> >>+nb_port_pair_params;
> >>+
> >> static unsigned int l2fwd_rx_queue_per_lcore = 1;
> >>
> >> #define MAX_RX_QUEUE_PER_LCORE 16
> >>@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
> >>       printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
> >>              "  -p PORTMASK: hexadecimal bitmask of ports to
> >configure\n"
> >>              "  -q NQ: number of queue (=ports) per lcore (default is
> >1)\n"
> >>-                "  -T PERIOD: statistics will be refreshed each PERIOD
> >>seconds (0 to disable, 10 default, 86400 maximum)\n"
> >>-                "  --[no-]mac-updating: Enable or disable MAC
> >addresses
> >>updating (enabled by default)\n"
> >>-                "      When enabled:\n"
> >>-                "       - The source MAC address is replaced by the TX
> >port
> >>MAC address\n"
> >>-                "       - The destination MAC address is replaced by
> >>02:00:00:00:00:TX_PORT_ID\n",
> >>+            "  -T PERIOD: statistics will be refreshed each PERIOD
> >seconds (0 to
> >>disable, 10 default, 86400 maximum)\n"
> >>+            "  --[no-]mac-updating: Enable or disable MAC addresses
> >updating
> >>(enabled by default)\n"
> >>+            "      When enabled:\n"
> >>+            "       - The source MAC address is replaced by the TX port
> >MAC
> >>address\n"
> >>+            "       - The destination MAC address is replaced by
> >>02:00:00:00:00:TX_PORT_ID\n"
> >>+            "  --portmap: Configure forwarding port pair mapping\n"
> >>+            "       Default: alternate port pairs\n\n",
> >
> >IMO, Indentation changes for other parameters should in separate
> >patch.
>
> It's not a functionality change it just minor code formatting change and doesn’t
> deserve a separate patch.
Ack.
>
> >>              prgname);
> >> }
> >>
> >>@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char
> >*portmask)
> >>       return pm;
> >> }
> >>

[dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[pbhagavatula]

From: Vamsi Attunuru <vattunuru@marvell.com>

Current l2fwd application statically configures adjacent ports as
destination ports for forwarding the traffic.

Add a portmap option to pass the forwarding port pair mapping which allows
the user to configure forwarding port mapping.

If no portmap argument is specified, destination port map is not
changed and traffic gets forwarded with existing mapping.

To align port/queue configuration of each lcore with destination port
map, port/queue configuration of each lcore gets modified when portmap
option is specified.

Ex: ./l2fwd -c 0xff -- -p 0x3f -q 2 --portmap="(0,3)(1,4)(2,5)"

With above portmap option, traffic received from portid = 0 gets forwarded
to port = 3 and vice versa, similarly traffic gets forwarded on other port
pairs (1,4) and (2,5)

Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Acked-by: Andrzej Ostruszka <aostruszka@marvell.com>
---
 v4 Changes:
 * Documentation changes. (Sunil)
 * reduce port_pair_params_array size. (Sunil)

 v3 Changes:
 * s/config/portmap/
 * rebase on master

 v2 Changes:
 * Fix command option format in docs.
 * Use memcpy instead of snprintf.
 * Rephrase if-else condition.

 doc/guides/rel_notes/release_20_05.rst        |   7 +
 .../sample_app_ug/l2_forward_real_virtual.rst |  17 +-
 examples/l2fwd/main.c                         | 182 +++++++++++++++---
 3 files changed, 180 insertions(+), 26 deletions(-)

diff --git a/doc/guides/rel_notes/release_20_05.rst b/doc/guides/rel_notes/release_20_05.rst
index b124c3f28..019bec5d7 100644
--- a/doc/guides/rel_notes/release_20_05.rst
+++ b/doc/guides/rel_notes/release_20_05.rst
@@ -212,6 +212,13 @@ New Features
   * Added IPsec inbound load-distribution support for ipsec-secgw application
     using NIC load distribution feature(Flow Director).

+* **Added --portmap command line parameter to l2fwd example.**
+
+  Added new command line option ``--portmap="(port, port)[,(port, port)]"`` to
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
+

 Removed Items
 -------------
diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
index 39d6b0067..90ca609d6 100644
--- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
+++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
@@ -91,7 +91,10 @@ The application requires a number of command line options:

 .. code-block:: console

-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   [--portmap="(port, port)[,(port, port)]"]

 where,

@@ -99,7 +102,9 @@ where,

 *   q NQ: A number of queues (=ports) per lcore (default is 1)

-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --portmap="(port,port)[,(port,port)]": Determines forwarding ports mapping.

 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +113,14 @@ updating enabled, issue the command:

     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff

+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore, to forward RX traffic of ports 0 & 1 on ports 2 & 3 respectively and
+vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.

diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
index 88ddfe589..1d84cd789 100644
--- a/examples/l2fwd/main.c
+++ b/examples/l2fwd/main.c
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>

 static volatile bool force_quit;

@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];

+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;
+
+static struct port_pair_params port_pair_params_array[RTE_MAX_ETHPORTS / 2];
+static struct port_pair_params *port_pair_params;
+static uint16_t nb_port_pair_params;
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;

 #define MAX_RX_QUEUE_PER_LCORE 16
@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
 	printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
 	       "  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 	       "  -q NQ: number of queue (=ports) per lcore (default is 1)\n"
-		   "  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default, 86400 maximum)\n"
-		   "  --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)\n"
-		   "      When enabled:\n"
-		   "       - The source MAC address is replaced by the TX port MAC address\n"
-		   "       - The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID\n",
+	       "  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to disable, 10 default, 86400 maximum)\n"
+	       "  --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)\n"
+	       "      When enabled:\n"
+	       "       - The source MAC address is replaced by the TX port MAC address\n"
+	       "       - The destination MAC address is replaced by 02:00:00:00:00:TX_PORT_ID\n"
+	       "  --portmap: Configure forwarding port pair mapping\n"
+	       "	      Default: alternate port pairs\n\n",
 	       prgname);
 }

@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char *portmask)
 	return pm;
 }

+static int
+l2fwd_parse_port_pair_config(const char *q_arg)
+{
+	enum fieldnames {
+		FLD_PORT1 = 0,
+		FLD_PORT2,
+		_NUM_FLD
+	};
+	unsigned long int_fld[_NUM_FLD];
+	const char *p, *p0 = q_arg;
+	char *str_fld[_NUM_FLD];
+	unsigned int size;
+	char s[256];
+	char *end;
+	int i;
+
+	nb_port_pair_params = 0;
+
+	while ((p = strchr(p0, '(')) != NULL) {
+		++p;
+		p0 = strchr(p, ')');
+		if (p0 == NULL)
+			return -1;
+
+		size = p0 - p;
+		if (size >= sizeof(s))
+			return -1;
+
+		memcpy(s, p, size);
+		s[size] = '\0';
+		if (rte_strsplit(s, sizeof(s), str_fld,
+				 _NUM_FLD, ',') != _NUM_FLD)
+			return -1;
+		for (i = 0; i < _NUM_FLD; i++) {
+			errno = 0;
+			int_fld[i] = strtoul(str_fld[i], &end, 0);
+			if (errno != 0 || end == str_fld[i] ||
+			    int_fld[i] >= RTE_MAX_ETHPORTS)
+				return -1;
+		}
+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
+			printf("exceeded max number of port pair params: %hu\n",
+				nb_port_pair_params);
+			return -1;
+		}
+		port_pair_params_array[nb_port_pair_params].port[0] =
+				(uint16_t)int_fld[FLD_PORT1];
+		port_pair_params_array[nb_port_pair_params].port[1] =
+				(uint16_t)int_fld[FLD_PORT2];
+		++nb_port_pair_params;
+	}
+	port_pair_params = port_pair_params_array;
+	return 0;
+}
+
 static unsigned int
 l2fwd_parse_nqueue(const char *q_arg)
 {
@@ -361,6 +428,7 @@ static const char short_options[] =

 #define CMD_LINE_OPT_MAC_UPDATING "mac-updating"
 #define CMD_LINE_OPT_NO_MAC_UPDATING "no-mac-updating"
+#define CMD_LINE_OPT_PORTMAP_CONFIG "portmap"

 enum {
 	/* long options mapped to a short option */
@@ -368,11 +436,13 @@ enum {
 	/* first long only option value must be >= 256, so that we won't
 	 * conflict with short options */
 	CMD_LINE_OPT_MIN_NUM = 256,
+	CMD_LINE_OPT_PORTMAP_NUM,
 };

 static const struct option lgopts[] = {
 	{ CMD_LINE_OPT_MAC_UPDATING, no_argument, &mac_updating, 1},
 	{ CMD_LINE_OPT_NO_MAC_UPDATING, no_argument, &mac_updating, 0},
+	{ CMD_LINE_OPT_PORTMAP_CONFIG, 1, 0, CMD_LINE_OPT_PORTMAP_NUM},
 	{NULL, 0, 0, 0}
 };

@@ -386,6 +456,7 @@ l2fwd_parse_args(int argc, char **argv)
 	char *prgname = argv[0];

 	argvopt = argv;
+	port_pair_params = NULL;

 	while ((opt = getopt_long(argc, argvopt, short_options,
 				  lgopts, &option_index)) != EOF) {
@@ -423,7 +494,13 @@ l2fwd_parse_args(int argc, char **argv)
 			break;

 		/* long options */
-		case 0:
+		case CMD_LINE_OPT_PORTMAP_NUM:
+			ret = l2fwd_parse_port_pair_config(optarg);
+			if (ret) {
+				fprintf(stderr, "Invalid config\n");
+				l2fwd_usage(prgname);
+				return -1;
+			}
 			break;

 		default:
@@ -440,6 +517,48 @@ l2fwd_parse_args(int argc, char **argv)
 	return ret;
 }

+/*
+ * Check port pair config with enabled port mask,
+ * and for valid port pair combinations.
+ */
+static int
+check_port_pair_config(void)
+{
+	uint32_t port_pair_config_mask = 0;
+	uint32_t port_pair_mask = 0;
+	uint16_t index, i, portid;
+
+	for (index = 0; index < nb_port_pair_params; index++) {
+		port_pair_mask = 0;
+
+		for (i = 0; i < NUM_PORTS; i++)  {
+			portid = port_pair_params[index].port[i];
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
+				printf("port %u is not enabled in port mask\n",
+				       portid);
+				return -1;
+			}
+			if (!rte_eth_dev_is_valid_port(portid)) {
+				printf("port %u is not present on the board\n",
+				       portid);
+				return -1;
+			}
+
+			port_pair_mask |= 1 << portid;
+		}
+
+		if (port_pair_config_mask & port_pair_mask) {
+			printf("port %u is used in other port pairs\n", portid);
+			return -1;
+		}
+		port_pair_config_mask |= port_pair_mask;
+	}
+
+	l2fwd_enabled_port_mask &= port_pair_config_mask;
+
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */
 static void
 check_all_ports_link_status(uint32_t port_mask)
@@ -555,6 +674,11 @@ main(int argc, char **argv)
 	if (nb_ports == 0)
 		rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");

+	if (port_pair_params != NULL) {
+		if (check_port_pair_config() < 0)
+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
+	}
+
 	/* check port mask to possible port mask */
 	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
 		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
@@ -565,26 +689,35 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;

-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx, p;

-		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			p = idx & 1;
+			portid = port_pair_params[idx >> 1].port[p];
+			l2fwd_dst_ports[portid] =
+				port_pair_params[idx >> 1].port[p ^ 1];
 		}
-		else
-			last_port = portid;
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;

-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {
+				last_port = portid;
+			}
+
+			nb_ports_in_mask++;
+		}
+		if (nb_ports_in_mask % 2) {
+			printf("Notice: odd number of ports in portmask.\n");
+			l2fwd_dst_ports[last_port] = last_port;
+		}
 	}

 	rx_lcore_id = 0;
@@ -613,7 +746,8 @@ main(int argc, char **argv)

 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}

 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd + MAX_PKT_BURST +
--
2.17.1

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Sunil Kumar Kori]

Looks okay. 

Regards
Sunil Kumar Kori

>-----Original Message-----
>From: pbhagavatula@marvell.com <pbhagavatula@marvell.com>
>Sent: Tuesday, April 28, 2020 12:01 AM
>To: Jerin Jacob Kollanukkaran <jerinj@marvell.com>; thomas@monjalon.net;
>John McNamara <john.mcnamara@intel.com>; Marko Kovacevic
><marko.kovacevic@intel.com>; Ori Kam <orika@mellanox.com>; Bruce
>Richardson <bruce.richardson@intel.com>; Radu Nicolau
><radu.nicolau@intel.com>; Akhil Goyal <akhil.goyal@nxp.com>; Tomasz
>Kantecki <tomasz.kantecki@intel.com>; Sunil Kumar Kori
><skori@marvell.com>; Pavan Nikhilesh Bhagavatula
><pbhagavatula@marvell.com>
>Cc: Andrzej Ostruszka [C] <aostruszka@marvell.com>; dev@dpdk.org; Vamsi
>Krishna Attunuru <vattunuru@marvell.com>
>Subject: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for
>forwarding port info
>
>From: Vamsi Attunuru <vattunuru@marvell.com>
>
>Current l2fwd application statically configures adjacent ports as destination
>ports for forwarding the traffic.
>
>Add a portmap option to pass the forwarding port pair mapping which allows
>the user to configure forwarding port mapping.
>
>If no portmap argument is specified, destination port map is not changed and
>traffic gets forwarded with existing mapping.
>
>To align port/queue configuration of each lcore with destination port map,
>port/queue configuration of each lcore gets modified when portmap option is
>specified.
>
>Ex: ./l2fwd -c 0xff -- -p 0x3f -q 2 --portmap="(0,3)(1,4)(2,5)"
>
>With above portmap option, traffic received from portid = 0 gets forwarded to
>port = 3 and vice versa, similarly traffic gets forwarded on other port pairs
>(1,4) and (2,5)
>
>Signed-off-by: Vamsi Attunuru <vattunuru@marvell.com>
>Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>Acked-by: Andrzej Ostruszka <aostruszka@marvell.com>
Acked-by: Sunil Kumar Kori <skori@marvell.com>

>---
> v4 Changes:
> * Documentation changes. (Sunil)
> * reduce port_pair_params_array size. (Sunil)
>
> v3 Changes:
> * s/config/portmap/
> * rebase on master
>
> v2 Changes:
> * Fix command option format in docs.
> * Use memcpy instead of snprintf.
> * Rephrase if-else condition.
>
> doc/guides/rel_notes/release_20_05.rst        |   7 +
> .../sample_app_ug/l2_forward_real_virtual.rst |  17 +-
> examples/l2fwd/main.c                         | 182 +++++++++++++++---
> 3 files changed, 180 insertions(+), 26 deletions(-)
>
>diff --git a/doc/guides/rel_notes/release_20_05.rst
>b/doc/guides/rel_notes/release_20_05.rst
>index b124c3f28..019bec5d7 100644
>--- a/doc/guides/rel_notes/release_20_05.rst
>+++ b/doc/guides/rel_notes/release_20_05.rst
>@@ -212,6 +212,13 @@ New Features
>   * Added IPsec inbound load-distribution support for ipsec-secgw application
>     using NIC load distribution feature(Flow Director).
>
>+* **Added --portmap command line parameter to l2fwd example.**
>+
>+  Added new command line option ``--portmap="(port, port)[,(port,
>+ port)]"`` to  pass forwarding port details.
>+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for
>+ more  details of this parameter usage.
>+
>
> Removed Items
> -------------
>diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>index 39d6b0067..90ca609d6 100644
>--- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>+++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst
>@@ -91,7 +91,10 @@ The application requires a number of command line
>options:
>
> .. code-block:: console
>
>-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
>+    ./build/l2fwd [EAL options] -- -p PORTMASK
>+                                   [-q NQ]
>+                                   --[no-]mac-updating
>+                                   [--portmap="(port, port)[,(port,
>+ port)]"]
>
> where,
>
>@@ -99,7 +102,9 @@ where,
>
> *   q NQ: A number of queues (=ports) per lcore (default is 1)
>
>-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled
>by default).
>+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled
>by default)
>+
>+*   --portmap="(port,port)[,(port,port)]": Determines forwarding ports
>mapping.
>
> To run the application in linux environment with 4 lcores, 16 ports and 8 RX
>queues per lcore and MAC address  updating enabled, issue the command:
>@@ -108,6 +113,14 @@ updating enabled, issue the command:
>
>     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
>
>+To run the application in linux environment with 4 lcores, 4 ports, 8
>+RX queues per lcore, to forward RX traffic of ports 0 & 1 on ports 2 &
>+3 respectively and vice versa, issue the command:
>+
>+.. code-block:: console
>+
>+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
>+
> Refer to the *DPDK Getting Started Guide* for general information on running
>applications  and the Environment Abstraction Layer (EAL) options.
>
>diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c index
>88ddfe589..1d84cd789 100644
>--- a/examples/l2fwd/main.c
>+++ b/examples/l2fwd/main.c
>@@ -38,6 +38,7 @@
> #include <rte_ethdev.h>
> #include <rte_mempool.h>
> #include <rte_mbuf.h>
>+#include <rte_string_fns.h>
>
> static volatile bool force_quit;
>
>@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
> /* list of enabled ports */
> static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
>
>+struct port_pair_params {
>+#define NUM_PORTS	2
>+	uint16_t port[NUM_PORTS];
>+} __rte_cache_aligned;
>+
>+static struct port_pair_params
>port_pair_params_array[RTE_MAX_ETHPORTS
>+/ 2]; static struct port_pair_params *port_pair_params; static uint16_t
>+nb_port_pair_params;
>+
> static unsigned int l2fwd_rx_queue_per_lcore = 1;
>
> #define MAX_RX_QUEUE_PER_LCORE 16
>@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
> 	printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
> 	       "  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
> 	       "  -q NQ: number of queue (=ports) per lcore (default is 1)\n"
>-		   "  -T PERIOD: statistics will be refreshed each PERIOD
>seconds (0 to disable, 10 default, 86400 maximum)\n"
>-		   "  --[no-]mac-updating: Enable or disable MAC addresses
>updating (enabled by default)\n"
>-		   "      When enabled:\n"
>-		   "       - The source MAC address is replaced by the TX port
>MAC address\n"
>-		   "       - The destination MAC address is replaced by
>02:00:00:00:00:TX_PORT_ID\n",
>+	       "  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to
>disable, 10 default, 86400 maximum)\n"
>+	       "  --[no-]mac-updating: Enable or disable MAC addresses updating
>(enabled by default)\n"
>+	       "      When enabled:\n"
>+	       "       - The source MAC address is replaced by the TX port MAC
>address\n"
>+	       "       - The destination MAC address is replaced by
>02:00:00:00:00:TX_PORT_ID\n"
>+	       "  --portmap: Configure forwarding port pair mapping\n"
>+	       "	      Default: alternate port pairs\n\n",
> 	       prgname);
> }
>
>@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char *portmask)
> 	return pm;
> }
>
>+static int
>+l2fwd_parse_port_pair_config(const char *q_arg) {
>+	enum fieldnames {
>+		FLD_PORT1 = 0,
>+		FLD_PORT2,
>+		_NUM_FLD
>+	};
>+	unsigned long int_fld[_NUM_FLD];
>+	const char *p, *p0 = q_arg;
>+	char *str_fld[_NUM_FLD];
>+	unsigned int size;
>+	char s[256];
>+	char *end;
>+	int i;
>+
>+	nb_port_pair_params = 0;
>+
>+	while ((p = strchr(p0, '(')) != NULL) {
>+		++p;
>+		p0 = strchr(p, ')');
>+		if (p0 == NULL)
>+			return -1;
>+
>+		size = p0 - p;
>+		if (size >= sizeof(s))
>+			return -1;
>+
>+		memcpy(s, p, size);
>+		s[size] = '\0';
>+		if (rte_strsplit(s, sizeof(s), str_fld,
>+				 _NUM_FLD, ',') != _NUM_FLD)
>+			return -1;
>+		for (i = 0; i < _NUM_FLD; i++) {
>+			errno = 0;
>+			int_fld[i] = strtoul(str_fld[i], &end, 0);
>+			if (errno != 0 || end == str_fld[i] ||
>+			    int_fld[i] >= RTE_MAX_ETHPORTS)
>+				return -1;
>+		}
>+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
>+			printf("exceeded max number of port pair params:
>%hu\n",
>+				nb_port_pair_params);
>+			return -1;
>+		}
>+		port_pair_params_array[nb_port_pair_params].port[0] =
>+				(uint16_t)int_fld[FLD_PORT1];
>+		port_pair_params_array[nb_port_pair_params].port[1] =
>+				(uint16_t)int_fld[FLD_PORT2];
>+		++nb_port_pair_params;
>+	}
>+	port_pair_params = port_pair_params_array;
>+	return 0;
>+}
>+
> static unsigned int
> l2fwd_parse_nqueue(const char *q_arg)
> {
>@@ -361,6 +428,7 @@ static const char short_options[] =
>
> #define CMD_LINE_OPT_MAC_UPDATING "mac-updating"
> #define CMD_LINE_OPT_NO_MAC_UPDATING "no-mac-updating"
>+#define CMD_LINE_OPT_PORTMAP_CONFIG "portmap"
>
> enum {
> 	/* long options mapped to a short option */ @@ -368,11 +436,13 @@
>enum {
> 	/* first long only option value must be >= 256, so that we won't
> 	 * conflict with short options */
> 	CMD_LINE_OPT_MIN_NUM = 256,
>+	CMD_LINE_OPT_PORTMAP_NUM,
> };
>
> static const struct option lgopts[] = {
> 	{ CMD_LINE_OPT_MAC_UPDATING, no_argument, &mac_updating,
>1},
> 	{ CMD_LINE_OPT_NO_MAC_UPDATING, no_argument,
>&mac_updating, 0},
>+	{ CMD_LINE_OPT_PORTMAP_CONFIG, 1, 0,
>CMD_LINE_OPT_PORTMAP_NUM},
> 	{NULL, 0, 0, 0}
> };
>
>@@ -386,6 +456,7 @@ l2fwd_parse_args(int argc, char **argv)
> 	char *prgname = argv[0];
>
> 	argvopt = argv;
>+	port_pair_params = NULL;
>
> 	while ((opt = getopt_long(argc, argvopt, short_options,
> 				  lgopts, &option_index)) != EOF) { @@ -423,7
>+494,13 @@ l2fwd_parse_args(int argc, char **argv)
> 			break;
>
> 		/* long options */
>-		case 0:
>+		case CMD_LINE_OPT_PORTMAP_NUM:
>+			ret = l2fwd_parse_port_pair_config(optarg);
>+			if (ret) {
>+				fprintf(stderr, "Invalid config\n");
>+				l2fwd_usage(prgname);
>+				return -1;
>+			}
> 			break;
>
> 		default:
>@@ -440,6 +517,48 @@ l2fwd_parse_args(int argc, char **argv)
> 	return ret;
> }
>
>+/*
>+ * Check port pair config with enabled port mask,
>+ * and for valid port pair combinations.
>+ */
>+static int
>+check_port_pair_config(void)
>+{
>+	uint32_t port_pair_config_mask = 0;
>+	uint32_t port_pair_mask = 0;
>+	uint16_t index, i, portid;
>+
>+	for (index = 0; index < nb_port_pair_params; index++) {
>+		port_pair_mask = 0;
>+
>+		for (i = 0; i < NUM_PORTS; i++)  {
>+			portid = port_pair_params[index].port[i];
>+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
>+				printf("port %u is not enabled in port
>mask\n",
>+				       portid);
>+				return -1;
>+			}
>+			if (!rte_eth_dev_is_valid_port(portid)) {
>+				printf("port %u is not present on the
>board\n",
>+				       portid);
>+				return -1;
>+			}
>+
>+			port_pair_mask |= 1 << portid;
>+		}
>+
>+		if (port_pair_config_mask & port_pair_mask) {
>+			printf("port %u is used in other port pairs\n", portid);
>+			return -1;
>+		}
>+		port_pair_config_mask |= port_pair_mask;
>+	}
>+
>+	l2fwd_enabled_port_mask &= port_pair_config_mask;
>+
>+	return 0;
>+}
>+
> /* Check the link status of all ports in up to 9s, and print them finally */  static
>void  check_all_ports_link_status(uint32_t port_mask) @@ -555,6 +674,11
>@@ main(int argc, char **argv)
> 	if (nb_ports == 0)
> 		rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
>
>+	if (port_pair_params != NULL) {
>+		if (check_port_pair_config() < 0)
>+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
>+	}
>+
> 	/* check port mask to possible port mask */
> 	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
> 		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
>@@ -565,26 +689,35 @@ main(int argc, char **argv)
> 		l2fwd_dst_ports[portid] = 0;
> 	last_port = 0;
>
>-	/*
>-	 * Each logical core is assigned a dedicated TX queue on each port.
>-	 */
>-	RTE_ETH_FOREACH_DEV(portid) {
>-		/* skip ports that are not enabled */
>-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
>-			continue;
>+	/* populate destination port details */
>+	if (port_pair_params != NULL) {
>+		uint16_t idx, p;
>
>-		if (nb_ports_in_mask % 2) {
>-			l2fwd_dst_ports[portid] = last_port;
>-			l2fwd_dst_ports[last_port] = portid;
>+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
>+			p = idx & 1;
>+			portid = port_pair_params[idx >> 1].port[p];
>+			l2fwd_dst_ports[portid] =
>+				port_pair_params[idx >> 1].port[p ^ 1];
> 		}
>-		else
>-			last_port = portid;
>+	} else {
>+		RTE_ETH_FOREACH_DEV(portid) {
>+			/* skip ports that are not enabled */
>+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
>+				continue;
>
>-		nb_ports_in_mask++;
>-	}
>-	if (nb_ports_in_mask % 2) {
>-		printf("Notice: odd number of ports in portmask.\n");
>-		l2fwd_dst_ports[last_port] = last_port;
>+			if (nb_ports_in_mask % 2) {
>+				l2fwd_dst_ports[portid] = last_port;
>+				l2fwd_dst_ports[last_port] = portid;
>+			} else {
>+				last_port = portid;
>+			}
>+
>+			nb_ports_in_mask++;
>+		}
>+		if (nb_ports_in_mask % 2) {
>+			printf("Notice: odd number of ports in portmask.\n");
>+			l2fwd_dst_ports[last_port] = last_port;
>+		}
> 	}
>
> 	rx_lcore_id = 0;
>@@ -613,7 +746,8 @@ main(int argc, char **argv)
>
> 		qconf->rx_port_list[qconf->n_rx_port] = portid;
> 		qconf->n_rx_port++;
>-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
>+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
>+		       portid, l2fwd_dst_ports[portid]);
> 	}
>
> 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +
>MAX_PKT_BURST +
>--
>2.17.1

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Varghese, Vipin]

Hi Vamsi & Pavan,

I like this idea, couple of queries

snipped
> +static int
> +check_port_pair_config(void)
> +{
> +	uint32_t port_pair_config_mask = 0;
> +	uint32_t port_pair_mask = 0;
> +	uint16_t index, i, portid;
> +
> +	for (index = 0; index < nb_port_pair_params; index++) {
> +		port_pair_mask = 0;
> +
> +		for (i = 0; i < NUM_PORTS; i++)  {
> +			portid = port_pair_params[index].port[i];
> +			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
> +				printf("port %u is not enabled in port
> mask\n",
> +				       portid);
> +				return -1;
> +			}
> +			if (!rte_eth_dev_is_valid_port(portid)) {
> +				printf("port %u is not present on the
> board\n",
> +				       portid);
> +				return -1;
> +			}
> +

Should we check & warn the user if 
1. port speed mismatch
2. on different NUMA
3. port pairs are physical and vdev like tap, and KNI (performance).

> +			port_pair_mask |= 1 << portid;
> +		}
> +

snipped


> 
> +	if (port_pair_params != NULL) {
> +		if (check_port_pair_config() < 0)
> +			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
> +	}
> +
>  	/* check port mask to possible port mask */
>  	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
>  		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
> @@ -565,26 +689,35 @@ main(int argc, char **argv)
>  		l2fwd_dst_ports[portid] = 0;
>  	last_port = 0;
> 

Should not the check_port_pair be after this? If the port is not enabled in port_mask will you skip that pair? or skip RX-TX from that port?

> -	/*
> -	 * Each logical core is assigned a dedicated TX queue on each port.
> -	 */
> -	RTE_ETH_FOREACH_DEV(portid) {
> -		/* skip ports that are not enabled */
> -		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
> -			continue;
> +	/* populate destination port details */
> +	if (port_pair_params != NULL) {
> +		uint16_t idx, p;
> 
> -		if (nb_ports_in_mask % 2) {
> -			l2fwd_dst_ports[portid] = last_port;
> -			l2fwd_dst_ports[last_port] = portid;
> +		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
> +			p = idx & 1;
> +			portid = port_pair_params[idx >> 1].port[p];
> +			l2fwd_dst_ports[portid] =
> +				port_pair_params[idx >> 1].port[p ^ 1];
>  		}
> -		else
> -			last_port = portid;
> +	} else {
> +		RTE_ETH_FOREACH_DEV(portid) {
> +			/* skip ports that are not enabled */
> +			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
> +				continue;
> 
> -		nb_ports_in_mask++;
> -	}
> -	if (nb_ports_in_mask % 2) {
> -		printf("Notice: odd number of ports in portmask.\n");
> -		l2fwd_dst_ports[last_port] = last_port;
> +			if (nb_ports_in_mask % 2) {
> +				l2fwd_dst_ports[portid] = last_port;
> +				l2fwd_dst_ports[last_port] = portid;
> +			} else {
> +				last_port = portid;
> +			}
> +
> +			nb_ports_in_mask++;
> +		}
> +		if (nb_ports_in_mask % 2) {
> +			printf("Notice: odd number of ports in portmask.\n");
> +			l2fwd_dst_ports[last_port] = last_port;
> +		}
>  	}

As mentioned above there can ports in mask which might be disabled for port pair. Should not that be skipped rather than setting last port rx-tx loopback?

> 
>  	rx_lcore_id = 0;
> @@ -613,7 +746,8 @@ main(int argc, char **argv)
> 
>  		qconf->rx_port_list[qconf->n_rx_port] = portid;
>  		qconf->n_rx_port++;
> -		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
> +		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
> +		       portid, l2fwd_dst_ports[portid]);
>  	}
> 
>  	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +
> MAX_PKT_BURST +
> --
> 2.17.1

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Pavan Nikhilesh Bhagavatula]

>Hi Vamsi & Pavan,
>
>I like this idea, couple of queries
>
>snipped
>> +static int
>> +check_port_pair_config(void)
>> +{
>> +	uint32_t port_pair_config_mask = 0;
>> +	uint32_t port_pair_mask = 0;
>> +	uint16_t index, i, portid;
>> +
>> +	for (index = 0; index < nb_port_pair_params; index++) {
>> +		port_pair_mask = 0;
>> +
>> +		for (i = 0; i < NUM_PORTS; i++)  {
>> +			portid = port_pair_params[index].port[i];
>> +			if ((l2fwd_enabled_port_mask & (1 << portid))
>== 0) {
>> +				printf("port %u is not enabled in port
>> mask\n",
>> +				       portid);
>> +				return -1;
>> +			}
>> +			if (!rte_eth_dev_is_valid_port(portid)) {
>> +				printf("port %u is not present on the
>> board\n",
>> +				       portid);
>> +				return -1;
>> +			}
>> +
>
>Should we check & warn the user if
>1. port speed mismatch
>2. on different NUMA
>3. port pairs are physical and vdev like tap, and KNI (performance).
>

Sure, it can be a separate patch as it will be applicable for multiple examples.

>> +			port_pair_mask |= 1 << portid;
>> +		}
>> +
>
>snipped
>
>
>>
>> +	if (port_pair_params != NULL) {
>> +		if (check_port_pair_config() < 0)
>> +			rte_exit(EXIT_FAILURE, "Invalid port pair
>config\n");
>> +	}
>> +
>>  	/* check port mask to possible port mask */
>>  	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
>>  		rte_exit(EXIT_FAILURE, "Invalid portmask; possible
>(0x%x)\n",
>> @@ -565,26 +689,35 @@ main(int argc, char **argv)
>>  		l2fwd_dst_ports[portid] = 0;
>>  	last_port = 0;
>>
>
>Should not the check_port_pair be after this? If the port is not enabled
>in port_mask will you skip that pair? or skip RX-TX from that port?

We check every port pair against l2fwd_enabled_port_mask in 
check_port_pair_config()

>
>> -	/*
>> -	 * Each logical core is assigned a dedicated TX queue on each
>port.
>> -	 */
>> -	RTE_ETH_FOREACH_DEV(portid) {
>> -		/* skip ports that are not enabled */
>> -		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
>> -			continue;
>> +	/* populate destination port details */
>> +	if (port_pair_params != NULL) {
>> +		uint16_t idx, p;
>>
>> -		if (nb_ports_in_mask % 2) {
>> -			l2fwd_dst_ports[portid] = last_port;
>> -			l2fwd_dst_ports[last_port] = portid;
>> +		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
>> +			p = idx & 1;
>> +			portid = port_pair_params[idx >> 1].port[p];
>> +			l2fwd_dst_ports[portid] =
>> +				port_pair_params[idx >> 1].port[p ^ 1];
>>  		}
>> -		else
>> -			last_port = portid;
>> +	} else {
>> +		RTE_ETH_FOREACH_DEV(portid) {
>> +			/* skip ports that are not enabled */
>> +			if ((l2fwd_enabled_port_mask & (1 << portid))
>== 0)
>> +				continue;
>>
>> -		nb_ports_in_mask++;
>> -	}
>> -	if (nb_ports_in_mask % 2) {
>> -		printf("Notice: odd number of ports in portmask.\n");
>> -		l2fwd_dst_ports[last_port] = last_port;
>> +			if (nb_ports_in_mask % 2) {
>> +				l2fwd_dst_ports[portid] = last_port;
>> +				l2fwd_dst_ports[last_port] = portid;
>> +			} else {
>> +				last_port = portid;
>> +			}
>> +
>> +			nb_ports_in_mask++;
>> +		}
>> +		if (nb_ports_in_mask % 2) {
>> +			printf("Notice: odd number of ports in
>portmask.\n");
>> +			l2fwd_dst_ports[last_port] = last_port;
>> +		}
>>  	}
>
>As mentioned above there can ports in mask which might be disabled
>for port pair. Should not that be skipped rather than setting last port rx-
>tx loopback?

There could be scenarios where user might want to test 2x10G and 1x40G 
Why force the user to explicitly mention 1x40G as port pair of itself in the 
portpair config?

>
>>
>>  	rx_lcore_id = 0;
>> @@ -613,7 +746,8 @@ main(int argc, char **argv)
>>
>>  		qconf->rx_port_list[qconf->n_rx_port] = portid;
>>  		qconf->n_rx_port++;
>> -		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
>> +		printf("Lcore %u: RX port %u TX port %u\n",
>rx_lcore_id,
>> +		       portid, l2fwd_dst_ports[portid]);
>>  	}
>>
>>  	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +
>> MAX_PKT_BURST +
>> --
>> 2.17.1

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Varghese, Vipin]

Hi Pavan,

snipped
> >
> >Should we check & warn the user if
> >1. port speed mismatch
> >2. on different NUMA
> >3. port pairs are physical and vdev like tap, and KNI (performance).
> >
> 
> Sure, it can be a separate patch as it will be applicable for multiple examples.
I believe this patch is for example `l2fwd`. But you would like to have to updated for all `example`. I am ok for this.

snipped
> >
> >Should not the check_port_pair be after this? If the port is not
> >enabled in port_mask will you skip that pair? or skip RX-TX from that port?
> 
> We check every port pair against l2fwd_enabled_port_mask in
> check_port_pair_config()


> 
snipped
> >
> >As mentioned above there can ports in mask which might be disabled for
> >port pair. Should not that be skipped rather than setting last port rx-
> >tx loopback?
> 
> There could be scenarios where user might want to test 2x10G and 1x40G Why
> force the user to explicitly mention 1x40G as port pair of itself in the portpair
> config?
I am not sure if I follow your thought, as your current port map only allows `1:1` mapping by `struct port_pair_params`. This can be to self like `(port0:port0),(port1:port1)` or `(port-0:port-1)`.

1. But current `l2fwd_parse_port_pair_config` does not consider the same port mapping as we have hard check for `if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2)`. 

2. `l2fwd_enabled_port_mask` is global variable of user port mask. This can contain both valid and invalid mask. Hence we check `l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1)`.

3. can these scenarios are true if we invoke `check_port_pair_config` before actual port_mask check.
 a. there are only 4 ports, hence possible mask is `0xf`.
 b. user passes port argument as `0xe`
 c. `check_port_pair_config` gets masks for `(1,3)` as input and populates `port_pair_config_mask`.
 d.  As per the code, port 2 which is valid port and part of user port mask will have lastport (which is port 3)? May be I did understand the logic correct. Can you help me?

So my concerns are 1) there is no same port mapping, 2) my understanding on lastport logic is not clear and 3) as per the code there is 1:N but 1:1.

Hence there should be sufficient warning to user if port are of wrong speed and NUMA.

Note: current speed can be fetched only if the port are started too (in Fortville). 

snipped

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Pavan Nikhilesh Bhagavatula]

Hi Vipin,

>Hi Pavan,
>
>snipped
>> >
>> >Should we check & warn the user if
>> >1. port speed mismatch
>> >2. on different NUMA
>> >3. port pairs are physical and vdev like tap, and KNI (performance).
>> >
>>
>> Sure, it can be a separate patch as it will be applicable for multiple
>examples.
>I believe this patch is for example `l2fwd`. But you would like to have to
>updated for all `example`. I am ok for this.
>
>snipped
>> >
>> >Should not the check_port_pair be after this? If the port is not
>> >enabled in port_mask will you skip that pair? or skip RX-TX from that
>port?
>>
>> We check every port pair against l2fwd_enabled_port_mask in
>> check_port_pair_config()
>
>
>>
>snipped
>> >
>> >As mentioned above there can ports in mask which might be
>disabled for
>> >port pair. Should not that be skipped rather than setting last port rx-
>> >tx loopback?
>>
>> There could be scenarios where user might want to test 2x10G and
>1x40G Why
>> force the user to explicitly mention 1x40G as port pair of itself in the
>portpair
>> config?
>I am not sure if I follow your thought, as your current port map only
>allows `1:1` mapping by `struct port_pair_params`. This can be to self
>like `(port0:port0),(port1:port1)` or `(port-0:port-1)`.
>
>1. But current `l2fwd_parse_port_pair_config` does not consider the
>same port mapping as we have hard check for `if (nb_port_pair_params
>>= RTE_MAX_ETHPORTS/2)`.
>
>2. `l2fwd_enabled_port_mask` is global variable of user port mask. This
>can contain both valid and invalid mask. Hence we check
>`l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1)`.
>
>3. can these scenarios are true if we invoke `check_port_pair_config`
>before actual port_mask check.
> a. there are only 4 ports, hence possible mask is `0xf`.
> b. user passes port argument as `0xe`
> c. `check_port_pair_config` gets masks for `(1,3)` as input and
>populates `port_pair_config_mask`.
> d.  As per the code, port 2 which is valid port and part of user port mask
>will have lastport (which is port 3)? May be I did understand the logic
>correct. Can you help me?

Here user needs to explicitly mention (2,2) for port 2 to be setup else it 
will be skipped. 
If you see `check_port_pair_config` below we disable the ports that are not 
Mentioned in portmap.

"
check_port_pair_config(void)
{

<snip>
		port_pair_config_mask |= port_pair_mask;
	}

	l2fwd_enabled_port_mask &= port_pair_config_mask;

	return 0;
}
"


>
>So my concerns are 1) there is no same port mapping, 2) my
>understanding on lastport logic is not clear and 3) as per the code there
>is 1:N but 1:1.
>
>Hence there should be sufficient warning to user if port are of wrong
>speed and NUMA.

Unless the user disables stats using -T 0 option all the prints will be skipped.

>
>Note: current speed can be fetched only if the port are started too (in
>Fortville).
>
>snipped

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Thomas Monjalon]

Bruce, as maintainer of l2fwd example, any opinion about this change?


11/05/2020 02:23, Pavan Nikhilesh Bhagavatula:
> Hi Vipin,
> 
> >Hi Pavan,
> >
> >snipped
> >> >
> >> >Should we check & warn the user if
> >> >1. port speed mismatch
> >> >2. on different NUMA
> >> >3. port pairs are physical and vdev like tap, and KNI (performance).
> >> >
> >>
> >> Sure, it can be a separate patch as it will be applicable for multiple
> >examples.
> >I believe this patch is for example `l2fwd`. But you would like to have to
> >updated for all `example`. I am ok for this.
> >
> >snipped
> >> >
> >> >Should not the check_port_pair be after this? If the port is not
> >> >enabled in port_mask will you skip that pair? or skip RX-TX from that
> >port?
> >>
> >> We check every port pair against l2fwd_enabled_port_mask in
> >> check_port_pair_config()
> >
> >
> >>
> >snipped
> >> >
> >> >As mentioned above there can ports in mask which might be
> >disabled for
> >> >port pair. Should not that be skipped rather than setting last port rx-
> >> >tx loopback?
> >>
> >> There could be scenarios where user might want to test 2x10G and
> >1x40G Why
> >> force the user to explicitly mention 1x40G as port pair of itself in the
> >portpair
> >> config?
> >I am not sure if I follow your thought, as your current port map only
> >allows `1:1` mapping by `struct port_pair_params`. This can be to self
> >like `(port0:port0),(port1:port1)` or `(port-0:port-1)`.
> >
> >1. But current `l2fwd_parse_port_pair_config` does not consider the
> >same port mapping as we have hard check for `if (nb_port_pair_params
> >>= RTE_MAX_ETHPORTS/2)`.
> >
> >2. `l2fwd_enabled_port_mask` is global variable of user port mask. This
> >can contain both valid and invalid mask. Hence we check
> >`l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1)`.
> >
> >3. can these scenarios are true if we invoke `check_port_pair_config`
> >before actual port_mask check.
> > a. there are only 4 ports, hence possible mask is `0xf`.
> > b. user passes port argument as `0xe`
> > c. `check_port_pair_config` gets masks for `(1,3)` as input and
> >populates `port_pair_config_mask`.
> > d.  As per the code, port 2 which is valid port and part of user port mask
> >will have lastport (which is port 3)? May be I did understand the logic
> >correct. Can you help me?
> 
> Here user needs to explicitly mention (2,2) for port 2 to be setup else it 
> will be skipped. 
> If you see `check_port_pair_config` below we disable the ports that are not 
> Mentioned in portmap.
> 
> "
> check_port_pair_config(void)
> {
> 
> <snip>
> 		port_pair_config_mask |= port_pair_mask;
> 	}
> 
> 	l2fwd_enabled_port_mask &= port_pair_config_mask;
> 
> 	return 0;
> }
> "
> 
> 
> >
> >So my concerns are 1) there is no same port mapping, 2) my
> >understanding on lastport logic is not clear and 3) as per the code there
> >is 1:N but 1:1.
> >
> >Hence there should be sufficient warning to user if port are of wrong
> >speed and NUMA.
> 
> Unless the user disables stats using -T 0 option all the prints will be skipped.
> 
> >
> >Note: current speed can be fetched only if the port are started too (in
> >Fortville).
> >
> >snipped

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Bruce Richardson]

On Sun, May 24, 2020 at 06:13:22PM +0200, Thomas Monjalon wrote:
> Bruce, as maintainer of l2fwd example, any opinion about this change?
> 
Assuming all previous discussion on it is resolved, I'm fine with this
patch, though I suspect it will only make 20.08 now.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

> 
> 11/05/2020 02:23, Pavan Nikhilesh Bhagavatula:
> > Hi Vipin,
> > 
> > >Hi Pavan,
> > >
> > >snipped
> > >> >
> > >> >Should we check & warn the user if
> > >> >1. port speed mismatch
> > >> >2. on different NUMA
> > >> >3. port pairs are physical and vdev like tap, and KNI (performance).
> > >> >
> > >>
> > >> Sure, it can be a separate patch as it will be applicable for multiple
> > >examples.
> > >I believe this patch is for example `l2fwd`. But you would like to have to
> > >updated for all `example`. I am ok for this.
> > >
> > >snipped
> > >> >
> > >> >Should not the check_port_pair be after this? If the port is not
> > >> >enabled in port_mask will you skip that pair? or skip RX-TX from that
> > >port?
> > >>
> > >> We check every port pair against l2fwd_enabled_port_mask in
> > >> check_port_pair_config()
> > >
> > >
> > >>
> > >snipped
> > >> >
> > >> >As mentioned above there can ports in mask which might be
> > >disabled for
> > >> >port pair. Should not that be skipped rather than setting last port rx-
> > >> >tx loopback?
> > >>
> > >> There could be scenarios where user might want to test 2x10G and
> > >1x40G Why
> > >> force the user to explicitly mention 1x40G as port pair of itself in the
> > >portpair
> > >> config?
> > >I am not sure if I follow your thought, as your current port map only
> > >allows `1:1` mapping by `struct port_pair_params`. This can be to self
> > >like `(port0:port0),(port1:port1)` or `(port-0:port-1)`.
> > >
> > >1. But current `l2fwd_parse_port_pair_config` does not consider the
> > >same port mapping as we have hard check for `if (nb_port_pair_params
> > >>= RTE_MAX_ETHPORTS/2)`.
> > >
> > >2. `l2fwd_enabled_port_mask` is global variable of user port mask. This
> > >can contain both valid and invalid mask. Hence we check
> > >`l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1)`.
> > >
> > >3. can these scenarios are true if we invoke `check_port_pair_config`
> > >before actual port_mask check.
> > > a. there are only 4 ports, hence possible mask is `0xf`.
> > > b. user passes port argument as `0xe`
> > > c. `check_port_pair_config` gets masks for `(1,3)` as input and
> > >populates `port_pair_config_mask`.
> > > d.  As per the code, port 2 which is valid port and part of user port mask
> > >will have lastport (which is port 3)? May be I did understand the logic
> > >correct. Can you help me?
> > 
> > Here user needs to explicitly mention (2,2) for port 2 to be setup else it 
> > will be skipped. 
> > If you see `check_port_pair_config` below we disable the ports that are not 
> > Mentioned in portmap.
> > 
> > "
> > check_port_pair_config(void)
> > {
> > 
> > <snip>
> > 		port_pair_config_mask |= port_pair_mask;
> > 	}
> > 
> > 	l2fwd_enabled_port_mask &= port_pair_config_mask;
> > 
> > 	return 0;
> > }
> > "
> > 
> > 
> > >
> > >So my concerns are 1) there is no same port mapping, 2) my
> > >understanding on lastport logic is not clear and 3) as per the code there
> > >is 1:N but 1:1.
> > >
> > >Hence there should be sufficient warning to user if port are of wrong
> > >speed and NUMA.
> > 
> > Unless the user disables stats using -T 0 option all the prints will be skipped.
> > 
> > >
> > >Note: current speed can be fetched only if the port are started too (in
> > >Fortville).
> > >
> > >snipped
> 
> 
> 
> 

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Jerin Jacob]

On Mon, May 25, 2020 at 2:59 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Sun, May 24, 2020 at 06:13:22PM +0200, Thomas Monjalon wrote:
> > Bruce, as maintainer of l2fwd example, any opinion about this change?
> >
> Assuming all previous discussion on it is resolved, I'm fine with this
> patch, though I suspect it will only make 20.08 now.
>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Ping for merge.


>
> >
> > 11/05/2020 02:23, Pavan Nikhilesh Bhagavatula:
> > > Hi Vipin,
> > >
> > > >Hi Pavan,
> > > >
> > > >snipped
> > > >> >
> > > >> >Should we check & warn the user if
> > > >> >1. port speed mismatch
> > > >> >2. on different NUMA
> > > >> >3. port pairs are physical and vdev like tap, and KNI (performance).
> > > >> >
> > > >>
> > > >> Sure, it can be a separate patch as it will be applicable for multiple
> > > >examples.
> > > >I believe this patch is for example `l2fwd`. But you would like to have to
> > > >updated for all `example`. I am ok for this.
> > > >
> > > >snipped
> > > >> >
> > > >> >Should not the check_port_pair be after this? If the port is not
> > > >> >enabled in port_mask will you skip that pair? or skip RX-TX from that
> > > >port?
> > > >>
> > > >> We check every port pair against l2fwd_enabled_port_mask in
> > > >> check_port_pair_config()
> > > >
> > > >
> > > >>
> > > >snipped
> > > >> >
> > > >> >As mentioned above there can ports in mask which might be
> > > >disabled for
> > > >> >port pair. Should not that be skipped rather than setting last port rx-
> > > >> >tx loopback?
> > > >>
> > > >> There could be scenarios where user might want to test 2x10G and
> > > >1x40G Why
> > > >> force the user to explicitly mention 1x40G as port pair of itself in the
> > > >portpair
> > > >> config?
> > > >I am not sure if I follow your thought, as your current port map only
> > > >allows `1:1` mapping by `struct port_pair_params`. This can be to self
> > > >like `(port0:port0),(port1:port1)` or `(port-0:port-1)`.
> > > >
> > > >1. But current `l2fwd_parse_port_pair_config` does not consider the
> > > >same port mapping as we have hard check for `if (nb_port_pair_params
> > > >>= RTE_MAX_ETHPORTS/2)`.
> > > >
> > > >2. `l2fwd_enabled_port_mask` is global variable of user port mask. This
> > > >can contain both valid and invalid mask. Hence we check
> > > >`l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1)`.
> > > >
> > > >3. can these scenarios are true if we invoke `check_port_pair_config`
> > > >before actual port_mask check.
> > > > a. there are only 4 ports, hence possible mask is `0xf`.
> > > > b. user passes port argument as `0xe`
> > > > c. `check_port_pair_config` gets masks for `(1,3)` as input and
> > > >populates `port_pair_config_mask`.
> > > > d.  As per the code, port 2 which is valid port and part of user port mask
> > > >will have lastport (which is port 3)? May be I did understand the logic
> > > >correct. Can you help me?
> > >
> > > Here user needs to explicitly mention (2,2) for port 2 to be setup else it
> > > will be skipped.
> > > If you see `check_port_pair_config` below we disable the ports that are not
> > > Mentioned in portmap.
> > >
> > > "
> > > check_port_pair_config(void)
> > > {
> > >
> > > <snip>
> > >             port_pair_config_mask |= port_pair_mask;
> > >     }
> > >
> > >     l2fwd_enabled_port_mask &= port_pair_config_mask;
> > >
> > >     return 0;
> > > }
> > > "
> > >
> > >
> > > >
> > > >So my concerns are 1) there is no same port mapping, 2) my
> > > >understanding on lastport logic is not clear and 3) as per the code there
> > > >is 1:N but 1:1.
> > > >
> > > >Hence there should be sufficient warning to user if port are of wrong
> > > >speed and NUMA.
> > >
> > > Unless the user disables stats using -T 0 option all the prints will be skipped.
> > >
> > > >
> > > >Note: current speed can be fetched only if the port are started too (in
> > > >Fortville).
> > > >
> > > >snipped
> >
> >
> >
> >

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

[Thomas Monjalon]

04/07/2020 15:36, Jerin Jacob:
> On Mon, May 25, 2020 at 2:59 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Sun, May 24, 2020 at 06:13:22PM +0200, Thomas Monjalon wrote:
> > > Bruce, as maintainer of l2fwd example, any opinion about this change?
> > >
> > Assuming all previous discussion on it is resolved, I'm fine with this
> > patch, though I suspect it will only make 20.08 now.
> >
> > Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> Ping for merge.

Applied, thanks

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Stephen Hemminger]

On Thu, 23 Jan 2020 10:58:13 +0100
Gaetan Rivet <grive@u256.net> wrote:

> Add a new EAL option enabling manual probing in the EAL.
> This command line option will configure the EAL so that buses
> will not trigger their probe step on their own.
> 
> Applications are then expected to hotplug devices as they see fit.
> 
> Devices declared on the command line by the user (using -w and --vdev),
> will be probed using the hotplug API, in the order they are declared.
> 
> This has the effect of offering a way for users to control probe order
> of their devices, for drivers requiring it.
> 
> Signed-off-by: Gaetan Rivet <grive@u256.net>
> Acked-by : Vamsi Attunuru <vattunuru@marvell.com>
> Tested-by: Vamsi Attunuru <vattunuru@marvell.com>
> Reviewed-by: Jerin Jacob <jerinj@marvell.com>
> ---
> 
>  haven't heard many opinions on the matter, please shout if you see an issue
> with this approach.
> 
> @Slava: I have tested rather quickly that it does not break anything,
>         and that it works as intended for basic cases.
>         Can you test it further for your use-case and tell me if it works fine?
> 
> Beyond the obvious difference between both probe mode, something to keep in mind:
> while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
> probing in its track. All devices need to exist and be valid device IDs.
> 
> v2: fixed a few typos, map file (and used Travis to validate).
> 
>     Slava, are you able to test this patch?
> 
> v3: properly fixed the map file (inherited 19.08 instead of 19.05).
> 
>     Added a function to set the probe manual from the application,
>     without having the user do it from the command line.
> 
>     Stopped spamming Slava about it, Vamsi was actually the one interested in it!
> 
> Standing issue worth chiming in:
> 
>   Currently manual-probe will cut off probing from all buses.
>   It could be interesting to be able to only cut buses supporting hotplug,
>   given that they are the one able to probe devices afterward.
> 
>   No real use-case for this right now, so leaving as-is. Might be worth
>   considering in the future.
> 
> v4: Rebased on master,
>     Moved implementation in common EAL,
>     Used public API within the EAL to set the option,
>     Made the API experimental
> 
> v5: added note in the Getting Started Guide.
> 
> v6: Rebased on master
> 
>     see http://mails.dpdk.org/archives/dev/2020-January/154178.html
>     for reference to this version, linking v7 to v5 thread.
> 
> v7: Updated author and SoB.
> 
>  doc/guides/linux_gsg/eal_args.include.rst  | 13 ++++++
>  doc/guides/rel_notes/release_20_02.rst     |  9 ++++
>  lib/librte_eal/common/eal_common_bus.c     |  6 +++
>  lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++
>  lib/librte_eal/common/eal_common_options.c |  8 ++++
>  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
>  lib/librte_eal/common/eal_options.h        |  2 +
>  lib/librte_eal/common/eal_private.h        |  9 ++++
>  lib/librte_eal/common/include/rte_eal.h    | 36 +++++++++++++++
>  lib/librte_eal/rte_eal_version.map         |  4 ++
>  10 files changed, 142 insertions(+)

This patch seems to have been held in limbo for 3 years.

For me, it is ok, but concerned that it opens up a whole scenario of possible
usages that may not be tested, and probably don't work. Testing all the possible
combinations of probe ordering is a geometric problem.

So if user submits bug then the response would have to be:
  Manual probing is an experimental option which may not work.

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

[Gaëtan Rivet]

On Wed, Jun 14, 2023, at 21:33, Stephen Hemminger wrote:
> On Thu, 23 Jan 2020 10:58:13 +0100
> Gaetan Rivet <grive@u256.net> wrote:
>
>> Add a new EAL option enabling manual probing in the EAL.
>> This command line option will configure the EAL so that buses
>> will not trigger their probe step on their own.
>> 
>> Applications are then expected to hotplug devices as they see fit.
>> 
>> Devices declared on the command line by the user (using -w and --vdev),
>> will be probed using the hotplug API, in the order they are declared.
>> 
>> This has the effect of offering a way for users to control probe order
>> of their devices, for drivers requiring it.
>> 
>> Signed-off-by: Gaetan Rivet <grive@u256.net>
>> Acked-by : Vamsi Attunuru <vattunuru@marvell.com>
>> Tested-by: Vamsi Attunuru <vattunuru@marvell.com>
>> Reviewed-by: Jerin Jacob <jerinj@marvell.com>
>> ---
>> 
>>  haven't heard many opinions on the matter, please shout if you see an issue
>> with this approach.
>> 
>> @Slava: I have tested rather quickly that it does not break anything,
>>         and that it works as intended for basic cases.
>>         Can you test it further for your use-case and tell me if it works fine?
>> 
>> Beyond the obvious difference between both probe mode, something to keep in mind:
>> while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
>> probing in its track. All devices need to exist and be valid device IDs.
>> 
>> v2: fixed a few typos, map file (and used Travis to validate).
>> 
>>     Slava, are you able to test this patch?
>> 
>> v3: properly fixed the map file (inherited 19.08 instead of 19.05).
>> 
>>     Added a function to set the probe manual from the application,
>>     without having the user do it from the command line.
>> 
>>     Stopped spamming Slava about it, Vamsi was actually the one interested in it!
>> 
>> Standing issue worth chiming in:
>> 
>>   Currently manual-probe will cut off probing from all buses.
>>   It could be interesting to be able to only cut buses supporting hotplug,
>>   given that they are the one able to probe devices afterward.
>> 
>>   No real use-case for this right now, so leaving as-is. Might be worth
>>   considering in the future.
>> 
>> v4: Rebased on master,
>>     Moved implementation in common EAL,
>>     Used public API within the EAL to set the option,
>>     Made the API experimental
>> 
>> v5: added note in the Getting Started Guide.
>> 
>> v6: Rebased on master
>> 
>>     see http://mails.dpdk.org/archives/dev/2020-January/154178.html
>>     for reference to this version, linking v7 to v5 thread.
>> 
>> v7: Updated author and SoB.
>> 
>>  doc/guides/linux_gsg/eal_args.include.rst  | 13 ++++++
>>  doc/guides/rel_notes/release_20_02.rst     |  9 ++++
>>  lib/librte_eal/common/eal_common_bus.c     |  6 +++
>>  lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++
>>  lib/librte_eal/common/eal_common_options.c |  8 ++++
>>  lib/librte_eal/common/eal_internal_cfg.h   |  1 +
>>  lib/librte_eal/common/eal_options.h        |  2 +
>>  lib/librte_eal/common/eal_private.h        |  9 ++++
>>  lib/librte_eal/common/include/rte_eal.h    | 36 +++++++++++++++
>>  lib/librte_eal/rte_eal_version.map         |  4 ++
>>  10 files changed, 142 insertions(+)
>
> This patch seems to have been held in limbo for 3 years.
>
> For me, it is ok, but concerned that it opens up a whole scenario of possible
> usages that may not be tested, and probably don't work. Testing all the possible
> combinations of probe ordering is a geometric problem.
>
> So if user submits bug then the response would have to be:
>   Manual probing is an experimental option which may not work.

Hello Stephen,

I am not pushing for this series anymore.
I wrote it to help other people, I guess they used another way since.
If someone needs it, I can take a moment to reanimate it.

I'm still using the PCI bus hack to force manual probing, as well as port hotplug to control strict ordering. I guess at this point this is the stable way of working with DPDK, instead of a proper documented option.

Best regards,
-- 
Gaetan Rivet