[dpdk-dev] [RFC PATCH 0/6] Improve mlx5 PMD common driver framework for multiple classes

[dpdk-dev] [RFC PATCH 0/6] Improve mlx5 PMD common driver framework for multiple classes

[Parav Pandit]

This commit introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
a new mlx5 PCI bus is created. This bus allows binding multiple
class drivers (such as net, vdpa, regex(future)) to bind to the
mlx5 PCI bus driver.

Change description
------------------
Patch-1 prepares the code to have RTE_BIT() macro defined in a
common header.
Patch-2 Prepares the code to check for enabled class
Patch-3 Changes class value to a bit field
Patch-4 Exposes mlx5_pci class driver registration APIs
PAtch-5 Implements mlx5 PCI bus
Patch-6 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
API instead of rte PCI bus API

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       -------------       /
         \______|   mlx5    |_____ /
                |   pci bus |
                -------------   
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD defines its
  specific PCI ID table and mlx5 bus driver probes matching
  class drivers.
- mlx5 pci bus driver is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI bus driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci bus, a common driver is
implemented which exposes class registration API.
However, bus model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common bus driver.
It also eliminates peer blind zone problem as bottom pci
bus layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

In future,
./testpmd -w <PCI BDF>,class=net:regex


Parav Pandit (6):
  eal: introduce macros for getting value for bit
  common/mlx5: use class enable check helper function
  common/mlx5: change mlx5 class enum values as bits
  bus/mlx5_pci: add mlx5 PCI bus
  bus/mlx5_pci: register a PCI driver
  bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

 config/common_base                            |   6 +
 config/defconfig_arm64-bluefield-linuxapp-gcc |   6 +
 drivers/bus/Makefile                          |   3 +
 drivers/bus/meson.build                       |   2 +-
 drivers/bus/mlx5_pci/Makefile                 |  49 +++
 drivers/bus/mlx5_pci/meson.build              |   6 +
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 327 ++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  66 ++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
 drivers/common/mlx5/mlx5_common.c             |  12 +-
 drivers/common/mlx5/mlx5_common.h             |   8 +-
 .../common/mlx5/rte_common_mlx5_version.map   |   2 +-
 drivers/net/mlx5/Makefile                     |   3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   4 +-
 drivers/net/mlx5/linux/mlx5_os.h              |   3 -
 drivers/net/mlx5/meson.build                  |   2 +-
 drivers/net/mlx5/mlx5.c                       |   7 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  12 +-
 lib/librte_eal/include/rte_bits.h             |  10 +
 mk/rte.app.mk                                 |   1 +
 23 files changed, 511 insertions(+), 29 deletions(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
 create mode 100644 lib/librte_eal/include/rte_bits.h

-- 
2.25.4

[dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 lib/librte_eal/include/rte_bits.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
 create mode 100644 lib/librte_eal/include/rte_bits.h

diff --git a/lib/librte_eal/include/rte_bits.h b/lib/librte_eal/include/rte_bits.h
new file mode 100644
index 000000000..37f284971
--- /dev/null
+++ b/lib/librte_eal/include/rte_bits.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd.
+ */
+
+#ifndef _RTE_BITS_H_
+#define _RTE_BITS_H_
+
+#define RTE_BIT(bit_num)	(1UL << (bit_num))
+
+#endif
-- 
2.25.4

Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit

[Gaëtan Rivet]

Hello Parav,

On 10/06/20 17:17 +0000, Parav Pandit wrote:
> There are several drivers which duplicate bit generation macro.
> Introduce a generic bit macros so that such drivers avoid redefining
> same in multiple drivers.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  lib/librte_eal/include/rte_bits.h | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>  create mode 100644 lib/librte_eal/include/rte_bits.h
> 
> diff --git a/lib/librte_eal/include/rte_bits.h b/lib/librte_eal/include/rte_bits.h
> new file mode 100644
> index 000000000..37f284971
> --- /dev/null
> +++ b/lib/librte_eal/include/rte_bits.h
> @@ -0,0 +1,10 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2020 Mellanox Technologies, Ltd.
> + */
> +
> +#ifndef _RTE_BITS_H_
> +#define _RTE_BITS_H_
> +
> +#define RTE_BIT(bit_num)	(1UL << (bit_num))
                           ^ The tab here should be replaced by a space.
> +
> +#endif
> -- 
> 2.25.4
> 

I'm not sure this kind of macro is needed, but if multiple
drivers are using the patterns let's say ok.

However I don't think it needs its own header. Would it be ok in
lib/librte_eal/include/rte_common.h for example?

-- 
Gaëtan

Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit

[Thomas Monjalon]

15/06/2020 21:33, Gaëtan Rivet:
> On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > There are several drivers which duplicate bit generation macro.
> > Introduce a generic bit macros so that such drivers avoid redefining
> > same in multiple drivers.
> > 
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > ---
[...]
> > --- /dev/null
> > +++ b/lib/librte_eal/include/rte_bits.h
> > @@ -0,0 +1,10 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright 2020 Mellanox Technologies, Ltd.
> > + */
> > +
> > +#ifndef _RTE_BITS_H_
> > +#define _RTE_BITS_H_
> > +
> > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
>                            ^ The tab here should be replaced by a space.
> > +
> > +#endif
> 
> I'm not sure this kind of macro is needed, but if multiple
> drivers are using the patterns let's say ok.
> 
> However I don't think it needs its own header. Would it be ok in
> lib/librte_eal/include/rte_common.h for example?

If we want to reuse an existing file, it could be
lib/librte_eal/include/rte_bitops.h

Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit

[Parav Pandit]

> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, June 17, 2020 1:35 PM
> 
> 15/06/2020 21:33, Gaëtan Rivet:
> > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > There are several drivers which duplicate bit generation macro.
> > > Introduce a generic bit macros so that such drivers avoid redefining
> > > same in multiple drivers.
> > >
> > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > ---
> [...]
> > > --- /dev/null
> > > +++ b/lib/librte_eal/include/rte_bits.h
> > > @@ -0,0 +1,10 @@
> > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > + * Copyright 2020 Mellanox Technologies, Ltd.
> > > + */
> > > +
> > > +#ifndef _RTE_BITS_H_
> > > +#define _RTE_BITS_H_
> > > +
> > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> >                            ^ The tab here should be replaced by a space.
> > > +
> > > +#endif
> >
> > I'm not sure this kind of macro is needed, but if multiple drivers are
> > using the patterns let's say ok.
> >
Yes. we certainly need it. Currently BIT() macro is used at 3000+ locations and defined in 10+ drivers.
Once we have this macro, it can gradually be replaced.

> > However I don't think it needs its own header. Would it be ok in
> > lib/librte_eal/include/rte_common.h for example?
> 
> If we want to reuse an existing file, it could be
> lib/librte_eal/include/rte_bitops.h
> 
o.k.
I will rename file from rte_bits.h to rte_bitops.h
More such macros will be available here to avoid redefinitions in drivers.

Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit

[Parav Pandit]

Hi Thomas,

> From: dev <dev-bounces@dpdk.org> On Behalf Of Parav Pandit
> Sent: Thursday, June 18, 2020 2:55 PM
> 
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Wednesday, June 17, 2020 1:35 PM
> >
> > 15/06/2020 21:33, Gaëtan Rivet:
> > > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > > There are several drivers which duplicate bit generation macro.
> > > > Introduce a generic bit macros so that such drivers avoid
> > > > redefining same in multiple drivers.
> > > >
> > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > ---
> > [...]
> > > > --- /dev/null
> > > > +++ b/lib/librte_eal/include/rte_bits.h
> > > > @@ -0,0 +1,10 @@
> > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > + * Copyright 2020 Mellanox Technologies, Ltd.
> > > > + */
> > > > +
> > > > +#ifndef _RTE_BITS_H_
> > > > +#define _RTE_BITS_H_
> > > > +
> > > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> > >                            ^ The tab here should be replaced by a space.
> > > > +
> > > > +#endif
> > >
> > > I'm not sure this kind of macro is needed, but if multiple drivers
> > > are using the patterns let's say ok.
> > >
> Yes. we certainly need it. Currently BIT() macro is used at 3000+ locations and
> defined in 10+ drivers.
> Once we have this macro, it can gradually be replaced.
> 
> > > However I don't think it needs its own header. Would it be ok in
> > > lib/librte_eal/include/rte_common.h for example?
> >
> > If we want to reuse an existing file, it could be
> > lib/librte_eal/include/rte_bitops.h
> >
> o.k.
> I will rename file from rte_bits.h to rte_bitops.h More such macros will be
> available here to avoid redefinitions in drivers.

I see that rte_bitops.h already exist and this new definition doesn't fit well in the rte_bitops.h.
The intent of the rte_bitops.h is to have generic mostly macros to replace current duplication of:

Such as
BIT()
BIT_ULL()
BITS_PER_BYTE()
BITS_TO_LONGS()

They do not fit well in rte_bitops.h which works on the 'bitmap'.

Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit

[Thomas Monjalon]

18/06/2020 14:16, Parav Pandit:
> From: Parav Pandit
> > From: Thomas Monjalon <thomas@monjalon.net>
> > > 15/06/2020 21:33, Gaëtan Rivet:
> > > > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > > > There are several drivers which duplicate bit generation macro.
> > > > > Introduce a generic bit macros so that such drivers avoid
> > > > > redefining same in multiple drivers.
> > > > >
> > > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > > ---
> > > [...]
> > > > > --- /dev/null
> > > > > +++ b/lib/librte_eal/include/rte_bits.h
> > > > > @@ -0,0 +1,10 @@
> > > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > > + * Copyright 2020 Mellanox Technologies, Ltd.
> > > > > + */
> > > > > +
> > > > > +#ifndef _RTE_BITS_H_
> > > > > +#define _RTE_BITS_H_
> > > > > +
> > > > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> > > >                            ^ The tab here should be replaced by a space.
> > > > > +
> > > > > +#endif
> > > >
> > > > I'm not sure this kind of macro is needed, but if multiple drivers
> > > > are using the patterns let's say ok.
> > > >
> > Yes. we certainly need it. Currently BIT() macro is used at 3000+ locations and
> > defined in 10+ drivers.
> > Once we have this macro, it can gradually be replaced.
> > 
> > > > However I don't think it needs its own header. Would it be ok in
> > > > lib/librte_eal/include/rte_common.h for example?
> > >
> > > If we want to reuse an existing file, it could be
> > > lib/librte_eal/include/rte_bitops.h
> > >
> > o.k.
> > I will rename file from rte_bits.h to rte_bitops.h More such macros will be
> > available here to avoid redefinitions in drivers.
> 
> I see that rte_bitops.h already exist and this new definition doesn't fit well in the rte_bitops.h.
> The intent of the rte_bitops.h is to have generic mostly macros to replace current duplication of:
> 
> Such as
> BIT()
> BIT_ULL()
> BITS_PER_BYTE()
> BITS_TO_LONGS()
> 
> They do not fit well in rte_bitops.h which works on the 'bitmap'.

Current bitops functions are getting and setting a bit.
I don't undertand why you say "works on the bitmap".
In my opinion, bit definition fits in this include file.

Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit

[Parav Pandit]

> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, June 18, 2020 5:52 PM
> 
> 18/06/2020 14:16, Parav Pandit:
> > From: Parav Pandit
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > 15/06/2020 21:33, Gaëtan Rivet:
> > > > > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > > > > There are several drivers which duplicate bit generation macro.
> > > > > > Introduce a generic bit macros so that such drivers avoid
> > > > > > redefining same in multiple drivers.
> > > > > >
> > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > > > ---
> > > > [...]
> > > > > > --- /dev/null
> > > > > > +++ b/lib/librte_eal/include/rte_bits.h
> > > > > > @@ -0,0 +1,10 @@
> > > > > > +/* SPDX-License-Identifier: BSD-3-Clause
> > > > > > + * Copyright 2020 Mellanox Technologies, Ltd.
> > > > > > + */
> > > > > > +
> > > > > > +#ifndef _RTE_BITS_H_
> > > > > > +#define _RTE_BITS_H_
> > > > > > +
> > > > > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> > > > >                            ^ The tab here should be replaced by a space.
> > > > > > +
> > > > > > +#endif
> > > > >
> > > > > I'm not sure this kind of macro is needed, but if multiple
> > > > > drivers are using the patterns let's say ok.
> > > > >
> > > Yes. we certainly need it. Currently BIT() macro is used at 3000+
> > > locations and defined in 10+ drivers.
> > > Once we have this macro, it can gradually be replaced.
> > >
> > > > > However I don't think it needs its own header. Would it be ok in
> > > > > lib/librte_eal/include/rte_common.h for example?
> > > >
> > > > If we want to reuse an existing file, it could be
> > > > lib/librte_eal/include/rte_bitops.h
> > > >
> > > o.k.
> > > I will rename file from rte_bits.h to rte_bitops.h More such macros
> > > will be available here to avoid redefinitions in drivers.
> >
> > I see that rte_bitops.h already exist and this new definition doesn't fit well
> in the rte_bitops.h.
> > The intent of the rte_bitops.h is to have generic mostly macros to replace
> current duplication of:
> >
> > Such as
> > BIT()
> > BIT_ULL()
> > BITS_PER_BYTE()
> > BITS_TO_LONGS()
> >
> > They do not fit well in rte_bitops.h which works on the 'bitmap'.
> 
> Current bitops functions are getting and setting a bit.
> I don't undertand why you say "works on the bitmap".
It sets in 32-bit and 64-bit addr.
Eventually it will be possibly extended to more than 64-bit. It will be a bitmap as generic API.

> In my opinion, bit definition fits in this include file.
> 
Ok. I am fine. I don't have strong opinion to create a new file.
I will place it here.

[dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes

[Parav Pandit]

This commit introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
a new mlx5 PCI bus is created. This bus allows binding multiple
class drivers (such as net, vdpa, regex(future)) to bind to the
mlx5 PCI bus driver.

Change description
------------------
Patch-1 prepares the code to have RTE_BIT() macro defined in a
common header.
Patch-2 Changes class value to a bit field
Patch-3 Exposes mlx5_pci class driver registration APIs
PAtch-4 Implements mlx5 PCI bus
Patch-5 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
API instead of rte PCI bus API
Patch-6 Removed class check code as its already part of the bus now

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       -------------       /
         \______|   mlx5    |_____ /
                |   pci bus |
                -------------   
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD defines its
  specific PCI ID table and mlx5 bus driver probes matching
  class drivers.
- mlx5 pci bus driver is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI bus driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci bus, a common driver is
implemented which exposes class registration API.
However, bus model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common bus driver.
It also eliminates peer blind zone problem as bottom pci
bus layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

In future,
./testpmd -w <PCI BDF>,class=net:regex

Changelog:
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - USe uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (6):
  eal: introduce macros for getting value for bit
  common/mlx5: change mlx5 class enum values as bits
  bus/mlx5_pci: add mlx5 PCI bus
  bus/mlx5_pci: register a PCI driver
  bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
  common/mlx5: Remove class checks from individual driver

 config/common_base                            |   6 +
 config/defconfig_arm64-bluefield-linuxapp-gcc |   6 +
 drivers/bus/Makefile                          |   3 +
 drivers/bus/meson.build                       |   2 +-
 drivers/bus/mlx5_pci/Makefile                 |  49 +++
 drivers/bus/mlx5_pci/meson.build              |   6 +
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 364 ++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  85 ++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
 drivers/common/mlx5/mlx5_common.c             |  37 --
 drivers/common/mlx5/mlx5_common.h             |   7 +-
 .../common/mlx5/rte_common_mlx5_version.map   |   2 -
 drivers/net/mlx5/Makefile                     |   3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/linux/mlx5_os.h              |   1 +
 drivers/net/mlx5/meson.build                  |   2 +-
 drivers/net/mlx5/mlx5.c                       |  24 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  28 +-
 lib/librte_eal/include/rte_bitops.h           |   2 +
 mk/rte.app.mk                                 |   1 +
 23 files changed, 565 insertions(+), 80 deletions(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

-- 
2.25.4

[dpdk-dev] [PATCH v2 1/6] eal: introduce macros for getting value for bit

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..d72c7cd93 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,8 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+#define RTE_BIT(bit_num)	(1UL << (bit_num))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.25.4

[dpdk-dev] [PATCH v2 2/6] common/mlx5: change mlx5 class enum values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v1->v2:
 - Rebasd due to removal previous patch
---
 drivers/common/mlx5/mlx5_common.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 77f10e676..6cc961e99 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 
@@ -202,9 +203,9 @@ int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT(0),
+	MLX5_CLASS_VDPA = RTE_BIT(1),
 };
 
 __rte_internal
-- 
2.25.4

[dpdk-dev] [PATCH v2 3/6] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
pci device.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v1->v2:
 - Address comments from Thomas and Gaetan
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
---
 config/common_base                            |  6 ++
 config/defconfig_arm64-bluefield-linuxapp-gcc |  6 ++
 drivers/bus/meson.build                       |  2 +-
 drivers/bus/mlx5_pci/Makefile                 | 47 +++++++++++
 drivers/bus/mlx5_pci/meson.build              |  6 ++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 84 +++++++++++++++++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
 8 files changed, 169 insertions(+), 1 deletion(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

diff --git a/config/common_base b/config/common_base
index c7d5c7321..f75b333f9 100644
--- a/config/common_base
+++ b/config/common_base
@@ -366,6 +366,12 @@ CONFIG_RTE_LIBRTE_IGC_DEBUG_TX=n
 CONFIG_RTE_LIBRTE_MLX4_PMD=n
 CONFIG_RTE_LIBRTE_MLX4_DEBUG=n
 
+#
+# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5,
+# ConnectX-6 & BlueField (MLX5) PMD
+#
+CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
+
 #
 # Compile burst-oriented Mellanox ConnectX-4, ConnectX-5,
 # ConnectX-6 & BlueField (MLX5) PMD
diff --git a/config/defconfig_arm64-bluefield-linuxapp-gcc b/config/defconfig_arm64-bluefield-linuxapp-gcc
index b49653881..15ade7ebc 100644
--- a/config/defconfig_arm64-bluefield-linuxapp-gcc
+++ b/config/defconfig_arm64-bluefield-linuxapp-gcc
@@ -14,5 +14,11 @@ CONFIG_RTE_CACHE_LINE_SIZE=64
 CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
 CONFIG_RTE_LIBRTE_VHOST_NUMA=n
 
+#
+# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5,
+# ConnectX-6 & BlueField (MLX5) PMD
+#
+CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
+
 # PMD for ConnectX-5
 CONFIG_RTE_LIBRTE_MLX5_PMD=y
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 80de2d91d..b1381838d 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
+drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
 std_deps = ['eal']
 config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
 driver_name_fmt = 'rte_bus_@0@'
diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
new file mode 100644
index 000000000..7db977ba8
--- /dev/null
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -0,0 +1,47 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Mellanox Technologies, Ltd
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_bus_mlx5_pci.a
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
+CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+LDLIBS += -lrte_eal
+LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# versioning export map
+EXPORT_MAP := rte_bus_mlx5_pci_version.map
+
+SRCS-y += mlx5_pci_bus.c
+
+# DEBUG which is usually provided on the command-line may enable
+# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
+ifeq ($(DEBUG),1)
+CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
+endif
+
+# User-defined CFLAGS.
+ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y)
+CFLAGS += -pedantic
+ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y)
+CFLAGS += -DPEDANTIC
+endif
+AUTO_CONFIG_CFLAGS += -Wno-pedantic
+else
+CFLAGS += -UPEDANTIC
+endif
+
+#
+# Export include files
+#
+SYMLINK-y-include += rte_bus_mlx5_pci.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
new file mode 100644
index 000000000..cc4a84e23
--- /dev/null
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Mellanox Technologies Ltd
+
+deps += ['pci', 'bus_pci', 'common_mlx5']
+install_headers('rte_bus_mlx5_pci.h')
+sources = files('mlx5_pci_bus.c')
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
new file mode 100644
index 000000000..66db3c7b0
--- /dev/null
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include "rte_bus_mlx5_pci.h"
+
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
+{
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
new file mode 100644
index 000000000..571f7dfd6
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _RTE_BUS_MLX5_PCI_H_
+#define _RTE_BUS_MLX5_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Bus Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This bus enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci bus driver.
+ *
+ * -----------    ------------    -----------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5        |
+ * | net pmd |    | vdpa pmd |    | new class pmd |
+ * -----------    ------------    -----------------
+ *      \              |                /
+ *       \             |               /
+ *        \       -------------       /
+ *         \______|   mlx5    |_____ /
+ *                |   pci bus |
+ *                -------------   
+ *                     |
+ *                 ----------- 
+ *                 |   mlx5  | 
+ *                 | pci dev | 
+ *                 ----------- 
+ *
+ * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct rte_mlx5_pci_driver {
+	enum mlx5_class dev_class;		/**< Class of this driver */
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a rte_mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
+
+#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv)	\
+static const char *mlx5_pci_drvinit_fn_ ## nm;	\
+RTE_INIT(mlx5_pci_drvinit_fn_ ##drv)	\
+{	\
+	(drv).driver.name = RTE_STR(nm);	\
+	rte_mlx5_pci_driver_register(&drv);	\
+}	\
+RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _RTE_BUS_MLX5_PCI_H_ */
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
new file mode 100644
index 000000000..4cfd3db10
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
@@ -0,0 +1,5 @@
+INTERNAL {
+	global:
+
+	rte_mlx5_pci_driver_register;
+};
-- 
2.25.4

Re: [dpdk-dev] [PATCH v2 3/6] bus/mlx5_pci: add mlx5 PCI bus

[Gaëtan Rivet]

Hi Parav,

On 21/06/20 19:11 +0000, Parav Pandit wrote:
> Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
> pci device.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
> Changelog:
> v1->v2:
>  - Address comments from Thomas and Gaetan
>  - Inheriting ret_pci_driver instead of rte_driver
>  - Added design and description of the mlx5_pci bus
> ---
>  config/common_base                            |  6 ++
>  config/defconfig_arm64-bluefield-linuxapp-gcc |  6 ++
>  drivers/bus/meson.build                       |  2 +-
>  drivers/bus/mlx5_pci/Makefile                 | 47 +++++++++++
>  drivers/bus/mlx5_pci/meson.build              |  6 ++
>  drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
>  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 84 +++++++++++++++++++
>  .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
>  8 files changed, 169 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/bus/mlx5_pci/Makefile
>  create mode 100644 drivers/bus/mlx5_pci/meson.build
>  create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
>  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
>  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> 
> diff --git a/config/common_base b/config/common_base
> index c7d5c7321..f75b333f9 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -366,6 +366,12 @@ CONFIG_RTE_LIBRTE_IGC_DEBUG_TX=n
>  CONFIG_RTE_LIBRTE_MLX4_PMD=n
>  CONFIG_RTE_LIBRTE_MLX4_DEBUG=n
>  
> +#
> +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5,
> +# ConnectX-6 & BlueField (MLX5) PMD
> +#
> +CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
> +

I told you that you were already using a construct in Makefiles that was
making this config useless, but you insisted to add it.

Meson does not need it and you can write your makefiles in such a way,
that this bus will be enabled automatically if any of its dependents is
enabled. This is the way to go, please remove these config lines.

>  #
>  # Compile burst-oriented Mellanox ConnectX-4, ConnectX-5,
>  # ConnectX-6 & BlueField (MLX5) PMD
> diff --git a/config/defconfig_arm64-bluefield-linuxapp-gcc b/config/defconfig_arm64-bluefield-linuxapp-gcc
> index b49653881..15ade7ebc 100644
> --- a/config/defconfig_arm64-bluefield-linuxapp-gcc
> +++ b/config/defconfig_arm64-bluefield-linuxapp-gcc
> @@ -14,5 +14,11 @@ CONFIG_RTE_CACHE_LINE_SIZE=64
>  CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
>  CONFIG_RTE_LIBRTE_VHOST_NUMA=n
>  
> +#
> +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5,
> +# ConnectX-6 & BlueField (MLX5) PMD
> +#
> +CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
> +
>  # PMD for ConnectX-5
>  CONFIG_RTE_LIBRTE_MLX5_PMD=y
> diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
> index 80de2d91d..b1381838d 100644
> --- a/drivers/bus/meson.build
> +++ b/drivers/bus/meson.build
> @@ -1,7 +1,7 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright(c) 2017 Intel Corporation
>  
> -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
> +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
>  std_deps = ['eal']
>  config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
>  driver_name_fmt = 'rte_bus_@0@'

It seems you only added your bus to Meson.
I see no modification to drivers/bus/Makefile

> diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
> new file mode 100644
> index 000000000..7db977ba8
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/Makefile
> @@ -0,0 +1,47 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright 2020 Mellanox Technologies, Ltd
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +#
> +# library name
> +#
> +LIB = librte_bus_mlx5_pci.a
> +
> +CFLAGS += -O3
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
> +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
> +CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
> +LDLIBS += -lrte_eal
> +LDLIBS += -lrte_common_mlx5
> +LDLIBS += -lrte_pci -lrte_bus_pci
> +
> +# versioning export map
> +EXPORT_MAP := rte_bus_mlx5_pci_version.map
> +
> +SRCS-y += mlx5_pci_bus.c
> +
> +# DEBUG which is usually provided on the command-line may enable
> +# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
> +ifeq ($(DEBUG),1)
> +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
> +endif
> +
> +# User-defined CFLAGS.
> +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y)
> +CFLAGS += -pedantic
> +ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y)
> +CFLAGS += -DPEDANTIC
> +endif
> +AUTO_CONFIG_CFLAGS += -Wno-pedantic
> +else
> +CFLAGS += -UPEDANTIC
> +endif
> +
> +#
> +# Export include files
> +#
> +SYMLINK-y-include += rte_bus_mlx5_pci.h
> +
> +include $(RTE_SDK)/mk/rte.lib.mk
> diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
> new file mode 100644
> index 000000000..cc4a84e23
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/meson.build
> @@ -0,0 +1,6 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2020 Mellanox Technologies Ltd
> +
> +deps += ['pci', 'bus_pci', 'common_mlx5']
> +install_headers('rte_bus_mlx5_pci.h')
> +sources = files('mlx5_pci_bus.c')
> diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> new file mode 100644
> index 000000000..66db3c7b0
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> @@ -0,0 +1,14 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#include "rte_bus_mlx5_pci.h"
> +
> +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
> +				TAILQ_HEAD_INITIALIZER(drv_list);
> +
> +void
> +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
> +{
> +	TAILQ_INSERT_TAIL(&drv_list, driver, next);
> +}
> diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> new file mode 100644
> index 000000000..571f7dfd6
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> @@ -0,0 +1,84 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#ifndef _RTE_BUS_MLX5_PCI_H_
> +#define _RTE_BUS_MLX5_PCI_H_
> +
> +/**
> + * @file
> + *
> + * RTE Mellanox PCI Bus Interface
> + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
                                                                     ^
                                                                     this should only appear in doc once supported
> + * devices. This bus enables creating such multiple class of devices on a
> + * single PCI device by allowing to bind multiple class specific device
> + * driver to attach to mlx5_pci bus driver.

Formulation is not great here. Here's a fix:

   Mellanox ConnectX PCI devices can be used in several ways, each
   represented by a device class (net, VDPA).  This bus allows using
   multiple device classes with a single PCI device.

   Currently only net and VDPA classes exists and no combination of the two
   is allowed.  New classes are expected to be introduced, this bus is the
   groundwork for their support.

This last paragraph is to be removed once regex dev are supported,
when combinations becomes valid and it's not future anymore. It should
then be replaced by a description of the rules to combine several classes.

> + *
> + * -----------    ------------    -----------------
> + * |   mlx5  |    |   mlx5   |    |   mlx5        |
> + * | net pmd |    | vdpa pmd |    | new class pmd |
> + * -----------    ------------    -----------------
> + *      \              |                /
> + *       \             |               /
> + *        \       -------------       /
> + *         \______|   mlx5    |_____ /
> + *                |   pci bus |
> + *                -------------   
                                  ^^^-- trailing spaces here
> + *                     |
> + *                 -----------  <
> + *                 |   mlx5  |  <
> + *                 | pci dev |  <
> + *                 -----------  <--- trailing spaces here
> + *
> + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
> + *   ID table of all related mlx5 PCI devices.
> + * - mlx5 class driver such as net, vdpa, regex PMD defines its
> + *   specific PCI ID table and mlx5 bus driver probes matching
> + *   class drivers.
> + * - mlx5 pci bus driver is cental place that validates supported
> + *   class combinations.

Multiple formulation issues, simpler for me to rewrite:
(please leave the empty newlines between each points.)

 - The mlx5 PCI bus driver binds to each PCI devices matching a PCI ID
   within the mlx5 table of supported PCI devices.

 - mlx5 class drivers such as net or VDPA PMD define their supported PCI
   ID table. The mlx5 bus driver will probe each matching class driver.

 - The mlx5 PCI bus driver is the central place validating the rules
   to combine and match multiple device classes.

> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif /* __cplusplus */
> +
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +
> +#include <mlx5_common.h>
> +
> +/**
> + * A structure describing a mlx5 pci driver.
> + */
> +struct rte_mlx5_pci_driver {
> +	enum mlx5_class dev_class;		/**< Class of this driver */
        ^^^^^^^^^^^^^^^           ^                                      ^ missing period here.
                \                  \
                 \                  Please do not align doc elements in your structs,
                  \                 they will be awkward to edit later when adding long-name fields.
                   \
                    enums width is implementation defined, I thinks it's
                    better to use uint32_t instead. Especially considering you use
                    offsetof-based stuff after (TAILQ).

> +	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
> +	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
> +};
> +
> +/**
> + * Register a mlx5_pci device driver.
> + *
> + * @param driver
> + *   A pointer to a rte_mlx5_pci_driver structure describing the driver
> + *   to be registered.
> + */
> +__rte_internal
> +void
> +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
> +
> +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv)	\
> +static const char *mlx5_pci_drvinit_fn_ ## nm;	\
> +RTE_INIT(mlx5_pci_drvinit_fn_ ##drv)	\
> +{	\
> +	(drv).driver.name = RTE_STR(nm);	\
> +	rte_mlx5_pci_driver_register(&drv);	\
> +}	\
> +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> +
> +#ifdef __cplusplus
> +}
> +#endif /* __cplusplus */
> +
> +#endif /* _RTE_BUS_MLX5_PCI_H_ */
> diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> new file mode 100644
> index 000000000..4cfd3db10
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> @@ -0,0 +1,5 @@
> +INTERNAL {
> +	global:
> +
> +	rte_mlx5_pci_driver_register;
> +};
> -- 
> 2.25.4
> 

-- 
Gaëtan

[dpdk-dev] [PATCH v2 4/6] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

Create a mlx5 bus driver framework for invoking drivers of
multiple classes who have registered with the mlx5_pci bus
driver.

Validate user class arguments for supported class combinations.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v1->v2:
 - Address comments from Thomas and Gaetan
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - USe uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
---
 drivers/bus/mlx5_pci/Makefile           |   2 +
 drivers/bus/mlx5_pci/meson.build        |   2 +-
 drivers/bus/mlx5_pci/mlx5_pci_bus.c     | 290 ++++++++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h |   1 +
 4 files changed, 294 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
index 7db977ba8..e53ed8856 100644
--- a/drivers/bus/mlx5_pci/Makefile
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -13,7 +13,9 @@ CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+CFLAGS += -D_DEFAULT_SOURCE
 LDLIBS += -lrte_eal
+LDLIBS += -lrte_kvargs
 LDLIBS += -lrte_common_mlx5
 LDLIBS += -lrte_pci -lrte_bus_pci
 
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
index cc4a84e23..5111baa4e 100644
--- a/drivers/bus/mlx5_pci/meson.build
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Mellanox Technologies Ltd
 
-deps += ['pci', 'bus_pci', 'common_mlx5']
+deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
 install_headers('rte_bus_mlx5_pci.h')
 sources = files('mlx5_pci_bus.c')
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index 66db3c7b0..e8f1649a3 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -3,12 +3,302 @@
  */
 
 #include "rte_bus_mlx5_pci.h"
+#include <mlx5_common_utils.h>
 
 static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
 				TAILQ_HEAD_INITIALIZER(drv_list);
 
+static const struct {
+	const char *name;
+	unsigned int dev_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .dev_class = MLX5_CLASS_NET },
+};
+
+static const unsigned int mlx5_valid_class_combo[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	/* New class combination should be added here */
+};
+
+static int class_name_to_val(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].dev_class;
+
+	}
+	return -EINVAL;
+}
+
+static int
+mlx5_bus_opt_handler(__rte_unused const char *key, const char *class_names,
+		     void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+
+		/* Check if its a valid class */
+		class_val = class_name_to_val(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s. Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, mlx5_bus_opt_handler, &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
 void
 rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
 {
 	TAILQ_INSERT_TAIL(&drv_list, driver, next);
 }
+
+static bool
+mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* check if device's ids match the class driver's ones */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+				id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+				id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+				id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+
+		return true;
+	}
+	return false;
+}
+
+static int is_valid_class_combo(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination */
+	for (i = 0; i < RTE_DIM(mlx5_valid_class_combo); i++) {
+		if (mlx5_valid_class_combo[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination */
+	return -EINVAL;
+}
+
+static int validate_single_class_dma_ops(void)
+{
+	struct rte_mlx5_pci_driver *class;
+	int dma_map_classes = 0;
+
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (class->pci_driver.dma_map)
+			dma_map_classes++;
+	}
+	if (dma_map_classes > 1) {
+		DRV_LOG(ERR, "Multiple classes with DMA ops is unsupported");
+		return -EINVAL;
+	}
+	return 0;
+}
+
+/**
+ * DPDK callback to register to probe multiple PCI class devices.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, 1 to skip this driver, a negative errno value otherwise
+ *   and rte_errno is set.
+ */
+static int
+mlx5_bus_pci_probe(struct rte_pci_driver *drv __rte_unused,
+		   struct rte_pci_device *dev)
+{
+	struct rte_mlx5_pci_driver *class;
+	uint32_t user_classes = 0;
+	int ret;
+
+	ret = validate_single_class_dma_ops();
+	if (ret)
+		return ret;
+
+	ret = parse_class_options(dev->device.devargs);
+	if (ret < 0)
+		return ret;
+
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here */
+		ret = is_valid_class_combo(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied");
+			return ret;
+		}
+	}
+
+	/* Default to net class */
+	if (user_classes == 0)
+		user_classes = MLX5_CLASS_NET;
+
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (!mlx5_bus_match(class, dev))
+			continue;
+
+		if ((class->dev_class & user_classes) == 0)
+			continue;
+
+		ret = -EINVAL;
+		if (class->loaded) {
+			/* If already loaded and class driver can handle
+			 * reprobe, probe such class driver again.
+			 */
+			if (class->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)
+				ret = class->pci_driver.probe(drv, dev);
+		} else {
+			ret = class->pci_driver.probe(drv, dev);
+		}
+		if (!ret)
+			class->loaded = true;
+	}
+	return 0;
+}
+
+/**
+ * DPDK callback to remove one or more class devices for a PCI device.
+ *
+ * This function removes all class devices belong to a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_bus_pci_remove(struct rte_pci_device *dev)
+{
+	struct rte_mlx5_pci_driver *class;
+
+	/* Remove each class driver in reverse order */
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (class->loaded)
+			class->pci_driver.remove(dev);
+	}
+	return 0;
+}
+
+static int
+mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr,
+		     uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret = -EINVAL;
+
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (!class->pci_driver.dma_map)
+			continue;
+
+		return class->pci_driver.dma_map(dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
+		       uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret = -EINVAL;
+
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (!class->pci_driver.dma_unmap)
+			continue;
+
+		return class->pci_driver.dma_unmap(dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static const struct rte_pci_id mlx5_bus_pci_id_map[] = {
+	{
+		.vendor_id = 0
+	}
+};
+
+static struct rte_pci_driver mlx5_bus_driver = {
+	.driver = {
+		.name = "mlx5_bus_pci",
+	},
+	.id_table = mlx5_bus_pci_id_map,
+	.probe = mlx5_bus_pci_probe,
+	.remove = mlx5_bus_pci_remove,
+	.dma_map = mlx5_bus_pci_dma_map,
+	.dma_unmap = mlx5_bus_pci_dma_unmap,
+	.drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV |
+		     RTE_PCI_DRV_PROBE_AGAIN,
+};
+
+RTE_PMD_REGISTER_PCI(mlx5_bus, mlx5_bus_driver);
+RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
index 571f7dfd6..c8cd7187b 100644
--- a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -55,6 +55,7 @@ struct rte_mlx5_pci_driver {
 	enum mlx5_class dev_class;		/**< Class of this driver */
 	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
 	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
+	bool loaded;
 };
 
 /**
-- 
2.25.4

Re: [dpdk-dev] [PATCH v2 4/6] bus/mlx5_pci: register a PCI driver

[Gaëtan Rivet]

On 21/06/20 19:11 +0000, Parav Pandit wrote:
> Create a mlx5 bus driver framework for invoking drivers of
> multiple classes who have registered with the mlx5_pci bus
> driver.
> 
> Validate user class arguments for supported class combinations.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
> Changelog:
> v1->v2:
>  - Address comments from Thomas and Gaetan
>  - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
>  - Use anonymous structure for class search and code changes around it
>  - Define static for class comination array
>  - Use RTE_DIM to find array size
>  - Added OOM check for strdup()
>  - Renamed copy variable to nstr_orig
>  - Returning negagive error code
>  - Returning directly if match entry found
>  - Use compat condition check
>  - Avoided cutting error message string
>  - USe uint32_t datatype instead of enum mlx5_class
>  - Changed logic to parse device arguments only once during probe()
>  - Added check to fail driver probe if multiple classes register with
>    DMA ops
>  - Renamed function to parse_class_options
> ---
>  drivers/bus/mlx5_pci/Makefile           |   2 +
>  drivers/bus/mlx5_pci/meson.build        |   2 +-
>  drivers/bus/mlx5_pci/mlx5_pci_bus.c     | 290 ++++++++++++++++++++++++
>  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h |   1 +
>  4 files changed, 294 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
> index 7db977ba8..e53ed8856 100644
> --- a/drivers/bus/mlx5_pci/Makefile
> +++ b/drivers/bus/mlx5_pci/Makefile
> @@ -13,7 +13,9 @@ CFLAGS += $(WERROR_FLAGS)
>  CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
>  CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
>  CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
> +CFLAGS += -D_DEFAULT_SOURCE
>  LDLIBS += -lrte_eal
> +LDLIBS += -lrte_kvargs
>  LDLIBS += -lrte_common_mlx5
>  LDLIBS += -lrte_pci -lrte_bus_pci
>  
> diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
> index cc4a84e23..5111baa4e 100644
> --- a/drivers/bus/mlx5_pci/meson.build
> +++ b/drivers/bus/mlx5_pci/meson.build
> @@ -1,6 +1,6 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright(c) 2020 Mellanox Technologies Ltd
>  
> -deps += ['pci', 'bus_pci', 'common_mlx5']
> +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
>  install_headers('rte_bus_mlx5_pci.h')
>  sources = files('mlx5_pci_bus.c')
> diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> index 66db3c7b0..e8f1649a3 100644
> --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> @@ -3,12 +3,302 @@
>   */
>  
>  #include "rte_bus_mlx5_pci.h"
> +#include <mlx5_common_utils.h>
>  
>  static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
>  				TAILQ_HEAD_INITIALIZER(drv_list);
>  
> +static const struct {
> +	const char *name;
> +	unsigned int dev_class;

Let me quote you when you refused to follow my comment:

>> Yes, I acked to changed to define, but I forgot that I use the enum here.
>> So I am going to keep the enum as code reads more clear with enum.

You refused to use a fixed-width integer type as per my past comments,
for readability reasons, but changed the type to "unsigned int" instead.

I insisted in the previous commit on uint32_t for exposed ABI (even if
between internal libs). Here I accept some leeway given the
compilation-unit scope of the definition. But at this point, your choice
is certainly *NOT* to use a vague type instead.

> +} mlx5_classes[] = {

mlx5_class_names.

> +	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
> +	{ .name = "net", .dev_class = MLX5_CLASS_NET },
> +};
> +
> +static const unsigned int mlx5_valid_class_combo[] = {
> +	MLX5_CLASS_NET,
> +	MLX5_CLASS_VDPA,
> +	/* New class combination should be added here */

This comment seems redundant, new class combo will be added wherever
appropriate, leave it to future dev.

> +};
> +
> +static int class_name_to_val(const char *class_name)

I think mlx5_class_from_name() is better.
(with mlx5_ namespace.)

> +{
> +	unsigned int i;

In general, size_t is the type of array iterators in C.

> +
> +	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
> +		if (strcmp(class_name, mlx5_classes[i].name) == 0)
> +			return mlx5_classes[i].dev_class;
> +
> +	}
> +	return -EINVAL;

You're mixing signed int and enum mlx5_class as return type.
Please find another way of signaling error that will make you keep the enum.

You have a sentinel value describing explicitly an invalid class, it seems perfectly
suited instead of -EINVAL. Use it instead.

> +}
> +
> +static int
> +mlx5_bus_opt_handler(__rte_unused const char *key, const char *class_names,
> +		     void *opaque)
> +{
> +	int *ret = opaque;
> +	char *nstr_org;
> +	int class_val;
> +	char *found;
> +	char *nstr;
> +
> +	*ret = 0;
> +	nstr = strdup(class_names);
> +	if (!nstr) {

Please be explicit and use (nstr == NULL).

> +		*ret = -ENOMEM;
> +		return *ret;
> +	}
> +
> +	nstr_org = nstr;

nstr_orig is more readable.

> +	while (nstr) {
        while (nstr != NULL) {

> +		/* Extract each individual class name */
> +		found = strsep(&nstr, ":");
> +		if (!found)

        ditto

> +			continue;
> +
> +		/* Check if its a valid class */
> +		class_val = class_name_to_val(found);
> +		if (class_val < 0) {

if (class_val == MLX5_CLASS_INVALID),
with the proper API change.

> +			*ret = -EINVAL;
> +			goto err;
> +		}
> +
> +		*ret |= class_val;

Once again, mixing ints and enum mlx5_class.
You don't *have* to set *ret on error.

* Change your opaque out_arg to uint32_t, stop using variable width types for bitmaps.

* Do not set it on error, use a tmp u32 for parsing and only set it once everything is ok.

* rte_kvargs_process() will mask your error values anyway, so instead set rte_errno and return -1.
  On negative return, it will itself return -1. Check for < 0 in bus_options_valid()

> +	}
> +err:
> +	free(nstr_org);
> +	if (*ret < 0)
> +		DRV_LOG(ERR, "Invalid mlx5 class options %s. Maybe typo in device class argument setting?",

Find a way to give the exact source of error. If it is an invalid name, show which token failed to be parsed
(meaning move your error code before nstr_orig is freed). Remove the "Maybe" formulation.

By the way, Thomas' comment was correct instead of mine, you should just cut your format string after
the "%s.".

> +			class_names);
> +	return *ret;
> +}
> +
> +static int
> +parse_class_options(const struct rte_devargs *devargs)
> +{
> +	const char *key = MLX5_CLASS_ARG_NAME;
> +	struct rte_kvargs *kvlist;
> +	int ret = 0;
> +
> +	if (devargs == NULL)
> +		return 0;
> +	kvlist = rte_kvargs_parse(devargs->args, NULL);
> +	if (kvlist == NULL)
> +		return 0;
> +	if (rte_kvargs_count(kvlist, key))
> +		rte_kvargs_process(kvlist, key, mlx5_bus_opt_handler, &ret);

Set ret to rte_kvargs_process() return value instead, define a specific u32 for bitmap.
Find a way to output the bitmap *separately* from the error code, or
set MLX5_CLASS_INVALID in the bitmap before returning it as sole return value for this function.
(meaning having a proper bit value for MLX5_CLASS_INVALID, if you go this way.)

I already said it in previous review, I will reformulate: stop overloading your types,
relying on implicit casts between correct and incorrect values, and merging your returned values
and the error channel.

Please be proactive into cleaning up your APIs.

> +	rte_kvargs_free(kvlist);
> +	return ret;
> +}
> +
>  void
>  rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
>  {
>  	TAILQ_INSERT_TAIL(&drv_list, driver, next);
>  }
> +
> +static bool
> +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
> +	       const struct rte_pci_device *pci_dev)
> +{
> +	const struct rte_pci_id *id_table;
> +
> +	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
> +	     id_table++) {
> +		/* check if device's ids match the class driver's ones */
> +		if (id_table->vendor_id != pci_dev->id.vendor_id &&
> +				id_table->vendor_id != PCI_ANY_ID)
> +			continue;
> +		if (id_table->device_id != pci_dev->id.device_id &&
> +				id_table->device_id != PCI_ANY_ID)
> +			continue;
> +		if (id_table->subsystem_vendor_id !=
> +		    pci_dev->id.subsystem_vendor_id &&
> +		    id_table->subsystem_vendor_id != PCI_ANY_ID)
> +			continue;
> +		if (id_table->subsystem_device_id !=
> +		    pci_dev->id.subsystem_device_id &&
> +		    id_table->subsystem_device_id != PCI_ANY_ID)
> +			continue;
> +		if (id_table->class_id != pci_dev->id.class_id &&
> +				id_table->class_id != RTE_CLASS_ANY_ID)
> +			continue;
> +
> +		return true;
> +	}
> +	return false;
> +}
> +
> +static int is_valid_class_combo(uint32_t user_classes)
> +{
> +	unsigned int i;

size_t

> +
> +	/* Verify if user specified valid supported combination */
                                    a valid combination.
> +	for (i = 0; i < RTE_DIM(mlx5_valid_class_combo); i++) {
> +		if (mlx5_valid_class_combo[i] == user_classes)

You simplified the scope of this function, which is good.
However, given the more limited scope, now it becomes a boolean
yes/no.

You are returning (0 | false) for yes, which is not ok.

reading if (is_valid_class_combo(combo)) { handle_error(combo); } is pretty
awkward.

While you're at it, you might want to use a proper bool instead.

> +			return 0;
> +	}
> +	/* Not found any valid class combination */
> +	return -EINVAL;
> +}
> +
> +static int validate_single_class_dma_ops(void)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +	int dma_map_classes = 0;
> +
> +	TAILQ_FOREACH(class, &drv_list, next) {
> +		if (class->pci_driver.dma_map)
> +			dma_map_classes++;
> +	}
> +	if (dma_map_classes > 1) {
> +		DRV_LOG(ERR, "Multiple classes with DMA ops is unsupported");
> +		return -EINVAL;
> +	}
> +	return 0;
> +}
> +
> +/**
> + * DPDK callback to register to probe multiple PCI class devices.
> + *
> + * @param[in] pci_drv
> + *   PCI driver structure.
> + * @param[in] dev
> + *   PCI device information.
> + *
> + * @return
> + *   0 on success, 1 to skip this driver, a negative errno value otherwise
> + *   and rte_errno is set.
> + */
> +static int
> +mlx5_bus_pci_probe(struct rte_pci_driver *drv __rte_unused,

drv is not unused, you are passing it to sub-drivers below.

> +		   struct rte_pci_device *dev)
> +{
> +	struct rte_mlx5_pci_driver *class;

This compilation unit targets a C compiler. I think only
headers should ensure compat with C++, but this name is not great still.

driver seems more appropriate anyway to designate a driver.

> +	uint32_t user_classes = 0;
> +	int ret;
> +

Mixing ret and user_classes as you do afterward is the result of the above API issues
already outlined. I won't go over them again, please fix everything to have proper
type discipline.

> +	ret = validate_single_class_dma_ops();
> +	if (ret)
> +		return ret;
> +
> +	ret = parse_class_options(dev->device.devargs);
> +	if (ret < 0)
> +		return ret;
> +
> +	user_classes = ret;
> +	if (user_classes) {
> +		/* Validate combination here */
> +		ret = is_valid_class_combo(user_classes);
> +		if (ret) {
> +			DRV_LOG(ERR, "Unsupported mlx5 classes supplied");
> +			return ret;
> +		}
> +	}
> +
> +	/* Default to net class */
> +	if (user_classes == 0)
> +		user_classes = MLX5_CLASS_NET;
> +
> +	TAILQ_FOREACH(class, &drv_list, next) {
> +		if (!mlx5_bus_match(class, dev))
> +			continue;
> +
> +		if ((class->dev_class & user_classes) == 0)
> +			continue;
> +
> +		ret = -EINVAL;
> +		if (class->loaded) {
> +			/* If already loaded and class driver can handle
> +			 * reprobe, probe such class driver again.
> +			 */
> +			if (class->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)
> +				ret = class->pci_driver.probe(drv, dev);

Using "drv" here instead of "class" means you are overriding the DRV_FLAG set by the
sub-driver.

Why not use "class" instead? dev->driver is setup by the upper layer, so will be correctly set
to drv instead of class.

> +		} else {
> +			ret = class->pci_driver.probe(drv, dev);
> +		}

You are ignoring probe() < 0 here, seems wrong.

> +		if (!ret)
> +			class->loaded = true;

loaded flag is not properly set.
You will set it on first successful probe, even on further errors.

Instead, use a u32 to mark each properly probed classes, then set loaded outside of this loop,
only if this "probed" bitmap matches exactly the "user_classes" bitmap.

This means also not silently ignoring dev and class mismatch. If this is the behavior you
explicitly want, then you will need to unset the mismatched class in the user_classes, so that the
exact match on probed is correct. Otherwise, logging an error is more appropriate.

> +	}
> +	return 0;
> +}
> +
> +/**
> + * DPDK callback to remove one or more class devices for a PCI device.
> + *
> + * This function removes all class devices belong to a given PCI device.
> + *
> + * @param[in] pci_dev
> + *   Pointer to the PCI device.
> + *
> + * @return
> + *   0 on success, the function cannot fail.
> + */
> +static int
> +mlx5_bus_pci_remove(struct rte_pci_device *dev)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +
> +	/* Remove each class driver in reverse order */
> +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
> +		if (class->loaded)
> +			class->pci_driver.remove(dev);
> +	}
> +	return 0;
> +}
> +
> +static int
> +mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr,
> +		     uint64_t iova, size_t len)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +	int ret = -EINVAL;
> +
> +	TAILQ_FOREACH(class, &drv_list, next) {
> +		if (!class->pci_driver.dma_map)
> +			continue;
> +
> +		return class->pci_driver.dma_map(dev, addr, iova, len);
> +	}
> +	return ret;
> +}
> +
> +static int
> +mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
> +		       uint64_t iova, size_t len)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +	int ret = -EINVAL;
> +
> +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
> +		if (!class->pci_driver.dma_unmap)
> +			continue;
> +

I see no additional logging about edge-cases that were discussed previously.
You can add them to the register function.

> +		return class->pci_driver.dma_unmap(dev, addr, iova, len);
> +	}
> +	return ret;
> +}
> +
> +static const struct rte_pci_id mlx5_bus_pci_id_map[] = {
> +	{
> +		.vendor_id = 0
> +	}
> +};
> +
> +static struct rte_pci_driver mlx5_bus_driver = {
> +	.driver = {
> +		.name = "mlx5_bus_pci",
> +	},
> +	.id_table = mlx5_bus_pci_id_map,
> +	.probe = mlx5_bus_pci_probe,
> +	.remove = mlx5_bus_pci_remove,
> +	.dma_map = mlx5_bus_pci_dma_map,
> +	.dma_unmap = mlx5_bus_pci_dma_unmap,
> +	.drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV |
> +		     RTE_PCI_DRV_PROBE_AGAIN,
> +};
> +
> +RTE_PMD_REGISTER_PCI(mlx5_bus, mlx5_bus_driver);
> +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
> diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> index 571f7dfd6..c8cd7187b 100644
> --- a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> @@ -55,6 +55,7 @@ struct rte_mlx5_pci_driver {
>  	enum mlx5_class dev_class;		/**< Class of this driver */
>  	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
>  	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
> +	bool loaded;
>  };
>  
>  /**
> -- 
> 2.25.4
> 

-- 
Gaëtan

[dpdk-dev] [PATCH v2 5/6] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Parav Pandit]

Enable class driver to match with the mlx5 pci devices.
Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v1->v2:
 - Migreate API from rte_driver to rte_pci_driver
---
 drivers/bus/Makefile                |  3 ++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c | 60 +++++++++++++++++++++++++++++
 drivers/net/mlx5/Makefile           |  3 +-
 drivers/net/mlx5/linux/mlx5_os.c    |  1 -
 drivers/net/mlx5/linux/mlx5_os.h    |  1 +
 drivers/net/mlx5/meson.build        |  2 +-
 drivers/net/mlx5/mlx5.c             | 24 +++++++-----
 drivers/net/mlx5/mlx5.h             |  1 -
 drivers/vdpa/mlx5/Makefile          |  3 +-
 drivers/vdpa/mlx5/meson.build       |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 23 ++++++-----
 mk/rte.app.mk                       |  1 +
 12 files changed, 98 insertions(+), 26 deletions(-)

diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile
index cea3b55e6..3840ac51c 100644
--- a/drivers/bus/Makefile
+++ b/drivers/bus/Makefile
@@ -8,6 +8,9 @@ ifeq ($(CONFIG_RTE_EAL_VFIO),y)
 DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
+DIRS-y += mlx5_pci
+endif
 DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
 DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
 DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index e8f1649a3..6844c1961 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -282,6 +282,66 @@ mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
 }
 
 static const struct rte_pci_id mlx5_bus_pci_id_map[] = {
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX4)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX4VF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX4LX)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5VF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5EX)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6VF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6DX)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6DXVF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6DXBF)
+	},
 	{
 		.vendor_id = 0
 	}
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index a458402dc..1cea7cd07 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_bus_mlx5_pci
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 3792371c3..60f42c5d1 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1394,7 +1394,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index f310f1773..3479c6428 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,4 +15,5 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
+
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index e95ce0267..26699a79b 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -8,7 +8,7 @@ if not (is_linux or is_windows)
 	subdir_done()
 endif
 
-deps += ['hash', 'common_mlx5']
+deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
 sources = files(
 	'mlx5.c',
 	'mlx5_ethdev.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 5c86f6fb9..22eaa64d3 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -34,6 +34,7 @@
 #include <rte_spinlock.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
+#include <rte_bus_mlx5_pci.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
@@ -1987,16 +1988,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct rte_mlx5_pci_driver mlx5_driver = {
+	.dev_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /**
@@ -2014,7 +2018,7 @@ RTE_INIT(rte_mlx5_pmd_init)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		rte_mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 5bd5acd9d..300b15a7d 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -129,7 +129,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 91c89d604..a2231f021 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
 CFLAGS += -I$(RTE_SDK)/lib/librte_sched
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..f4175c34e 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 1113d6cef..b223da7f2 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -6,7 +6,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <rte_bus_mlx5_pci.h>
 #include <rte_pci.h>
 
 #include <mlx5_glue.h>
@@ -428,7 +428,7 @@ mlx5_vdpa_roce_disable(struct rte_pci_addr *addr, struct ibv_device **ibv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI bus device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -598,14 +598,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct rte_mlx5_pci_driver mlx5_vdpa_driver = {
+	.dev_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 /**
@@ -618,7 +621,7 @@ RTE_INIT(rte_mlx5_vdpa_init)
 	if (mlx5_vdpa_logtype >= 0)
 		rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE);
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0ce8cf541..d7c1d90e4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF)      += -lrte_pmd_memif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD)       += -lrte_pmd_mlx4
 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
 _LDLIBS-y                                   += -lrte_common_mlx5
+_LDLIBS-y                                   += -lrte_bus_mlx5_pci
 endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)  += -lrte_pmd_mlx5_vdpa
-- 
2.25.4

[dpdk-dev] [PATCH v2 6/6] common/mlx5: Remove class checks from individual driver

[Parav Pandit]

Now that mlx5_pci bus does the check for enabled classes and performs
probe(), remove() of associated classes, individual class driver
doesn't need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v1->v2:
 - New patch
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  2 -
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 5 files changed, 51 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index db94d4aa8..ae7a0d6d6 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -19,43 +19,6 @@ const struct mlx5_glue *mlx5_glue;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 6cc961e99..fee988379 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -208,8 +208,6 @@ enum mlx5_class {
 	MLX5_CLASS_VDPA = RTE_BIT(1),
 };
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 68f120712..236a7ef7b 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,8 +1,6 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 60f42c5d1..944c3bf66 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1381,11 +1381,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_dev_config dev_config;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index b223da7f2..30758f7b6 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -451,11 +451,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.25.4

[dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
a new mlx5 PCI bus is created. This bus allows binding multiple
class drivers (such as net, vdpa, regex(future)) to bind to the
mlx5 PCI bus driver.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Introduces new RTE constructor priority for common
initialization
Patch-3 Fixes compilation error
Patch-4 Define mlx5 class as bit fields
Patch-5 Uses new RTE common priority
Patch-6 Adds mlx5 PCI bus
Patch-7 Implements a mlx5 PCI bus driver
Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
API instead of rte PCI bus API
Patch-9 Removed class check code as its already part of the bus now
Patch-10 Add maintainers for the new bus

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       -------------       /
         \______|   mlx5    |_____ /
                |   pci bus |
                -------------   
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD defines its
  specific PCI ID table and mlx5 bus driver probes matching
  class drivers.
- mlx5 pci bus driver is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI bus driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci bus, a common driver is
implemented which exposes class registration API.
However, bus model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common bus driver.
It also eliminates peer blind zone problem as bottom pci
bus layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

In future,
./testpmd -w <PCI BDF>,class=net:regex

Changelog:
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver


Parav Pandit (10):
  eal: introduce macros for getting value for bit
  eal: introduce RTE common initialization level
  common/mlx5: fix empty input style in glue wrappers
  common/mlx5: change mlx5 class enum values as bits
  common/mlx5: use common rte priority
  bus/mlx5_pci: add mlx5 PCI bus
  bus/mlx5_pci: register a PCI driver
  bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
  common/mlx5: remove class checks from individual driver
  maintainers: add maintainers for mlx5 pci bus

 MAINTAINERS                                   |   5 +
 drivers/bus/Makefile                          |   3 +
 drivers/bus/meson.build                       |   2 +-
 drivers/bus/mlx5_pci/Makefile                 |  40 ++
 drivers/bus/mlx5_pci/meson.build              |  19 +
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 522 ++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  84 +++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/mlx5_common.c             |  39 +-
 drivers/common/mlx5/mlx5_common.h             |   7 +-
 .../common/mlx5/rte_common_mlx5_version.map   |   2 -
 drivers/net/mlx5/Makefile                     |   3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/meson.build                  |   2 +-
 drivers/net/mlx5/mlx5.c                       |  26 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  30 +-
 lib/librte_eal/include/rte_bitops.h           |   2 +
 lib/librte_eal/include/rte_common.h           |   1 +
 mk/rte.app.mk                                 |   1 +
 23 files changed, 724 insertions(+), 85 deletions(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

-- 
2.26.2

[dpdk-dev] [PATCH v3 01/10] eal: introduce macros for getting value for bit

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..d72c7cd93 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,8 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+#define RTE_BIT(bit_num)	(1UL << (bit_num))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.26.2

[dpdk-dev] [PATCH v3 02/10] eal: introduce RTE common initialization level

[Parav Pandit]

Currently mlx5_common uses CLASS priority to initialize
common code before initializing the PMD.
However mlx5_common is not really a class, it is the pre-initialization
code needed for the PMDs.

In subsequent patch a needed initialization sequence is:
(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
for class driver initialization.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 lib/librte_eal/include/rte_common.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
index 0843ce69e..d4653ea77 100644
--- a/lib/librte_eal/include/rte_common.h
+++ b/lib/librte_eal/include/rte_common.h
@@ -126,6 +126,7 @@ typedef uint16_t unaligned_uint16_t;
 
 #define RTE_PRIORITY_LOG 101
 #define RTE_PRIORITY_BUS 110
+#define RTE_PRIORITY_COMMON 119
 #define RTE_PRIORITY_CLASS 120
 #define RTE_PRIORITY_LAST 65535
 
-- 
2.26.2

[dpdk-dev] [PATCH v3 03/10] common/mlx5: fix empty input style in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")
Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index 069d8540c..e5e052a6a 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v3 04/10] common/mlx5: change mlx5 class enum values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v1->v2:
 - Rebasd due to removal previous patch
---
 drivers/common/mlx5/mlx5_common.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index da01ffa1c..62861af05 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -207,9 +208,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT(0),
+	MLX5_CLASS_VDPA = RTE_BIT(1),
 };
 
 #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */
-- 
2.26.2

[dpdk-dev] [PATCH v3 05/10] common/mlx5: use common rte priority

[Parav Pandit]

Use RTE_PRIO_COMMON for mlx5 common initialization and use
RTE_PRIO_CLASS for mlx5 net, vdpa PMDs.

This enables to do following initialization sequence.

(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c | 2 +-
 drivers/net/mlx5/mlx5.c           | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 693e2c68c..0ce5e4db1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 /**
  * Initialization routine for run-time dependency on glue library.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+RTE_INIT_PRIO(mlx5_glue_init, COMMON)
 {
 	mlx5_glue_constructor();
 }
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 07c6addd5..bb10c63bb 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1907,7 +1907,7 @@ struct rte_pci_driver mlx5_driver = {
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_pmd_init)
+RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 {
 	/* Initialize driver log type. */
 	mlx5_logtype = rte_log_register("pmd.net.mlx5");
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index dbd36ab0c..f043166ef 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -842,7 +842,7 @@ static struct rte_pci_driver mlx5_vdpa_driver = {
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_vdpa_init)
+RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 {
 	/* Initialize common log type. */
 	mlx5_vdpa_logtype = rte_log_register("pmd.vdpa.mlx5");
-- 
2.26.2

[dpdk-dev] [PATCH v3 06/10] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
pci device.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Addressed comments from Thomas and Asaf
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address comments from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Changed alignedment to mlx5 specific aligment instead of standard
   DPDK
 - Using uint32_t instead of mlx5_class enum
v1->v2:
 - Address comments from Thomas and Gaetan
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
---
 drivers/bus/meson.build                       |  2 +-
 drivers/bus/mlx5_pci/Makefile                 | 38 +++++++++
 drivers/bus/mlx5_pci/meson.build              | 19 +++++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 84 +++++++++++++++++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
 6 files changed, 161 insertions(+), 1 deletion(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 80de2d91d..b1381838d 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
+drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
 std_deps = ['eal']
 config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
 driver_name_fmt = 'rte_bus_@0@'
diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
new file mode 100644
index 000000000..de4ccd83f
--- /dev/null
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Mellanox Technologies, Ltd
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_bus_mlx5_pci.a
+
+CFLAGS += -O3 -Wall -Wextra
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -Wno-strict-prototypes
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
+CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
+CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+LDLIBS += -lrte_eal
+LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# versioning export map
+EXPORT_MAP := rte_bus_mlx5_pci_version.map
+
+SRCS-y += mlx5_pci_bus.c
+
+# DEBUG which is usually provided on the command-line may enable
+# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
+ifeq ($(DEBUG),1)
+CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
+endif
+
+#
+# Export include files
+#
+SYMLINK-y-include += rte_bus_mlx5_pci.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
new file mode 100644
index 000000000..64a17cbad
--- /dev/null
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Mellanox Technologies Ltd
+
+deps += ['pci', 'bus_pci', 'common_mlx5']
+install_headers('rte_bus_mlx5_pci.h')
+sources = files('mlx5_pci_bus.c')
+
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600'
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
new file mode 100644
index 000000000..66db3c7b0
--- /dev/null
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include "rte_bus_mlx5_pci.h"
+
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
+{
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
new file mode 100644
index 000000000..9f8d22e2b
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _RTE_BUS_MLX5_PCI_H_
+#define _RTE_BUS_MLX5_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Bus Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This bus enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci bus driver.
+ *
+ * -----------    ------------    -----------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5        |
+ * | net pmd |    | vdpa pmd |    | new class pmd |
+ * -----------    ------------    -----------------
+ *      \              |                /
+ *       \             |               /
+ *        \       -------------       /
+ *         \______|   mlx5    |_____ /
+ *                |   pci bus |
+ *                -------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct rte_mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t dev_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a rte_mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
+
+#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \
+	static const char *mlx5_pci_drvinit_fn_ ## nm; \
+	RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \
+	{ \
+		(drv).driver.name = RTE_STR(nm); \
+		rte_mlx5_pci_driver_register(&drv); \
+	} \
+	RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _RTE_BUS_MLX5_PCI_H_ */
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
new file mode 100644
index 000000000..4cfd3db10
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
@@ -0,0 +1,5 @@
+INTERNAL {
+	global:
+
+	rte_mlx5_pci_driver_register;
+};
-- 
2.26.2

[dpdk-dev] [PATCH v3 07/10] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

Create a mlx5 bus driver framework for invoking drivers of
multiple classes who have registered with the mlx5_pci bus
driver.

Validate user class arguments for supported class combinations.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Addressed comments from Asaf
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Addressed comments from Matan.
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block.
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Maintaining class driver list to keep load/unload order symmetric and
   mirror of each other.
 - Deriving drv_flags based on the class drivers
 - Using PCI address comparision helper instead of pointer comparision
 - Fixed alignment for id_table
 - Continue to probe_err if device is already probed
 - Perform dma map on best effort basis for all supported drivers
 - Removed drv_flags check
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Address comments from Thomas and Gaetan
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - USe uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
---
 drivers/bus/mlx5_pci/Makefile       |   2 +
 drivers/bus/mlx5_pci/meson.build    |   2 +-
 drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++
 3 files changed, 511 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
index de4ccd83f..1a005ee32 100644
--- a/drivers/bus/mlx5_pci/Makefile
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -15,7 +15,9 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+CFLAGS += -D_DEFAULT_SOURCE
 LDLIBS += -lrte_eal
+LDLIBS += -lrte_kvargs
 LDLIBS += -lrte_common_mlx5
 LDLIBS += -lrte_pci -lrte_bus_pci
 
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
index 64a17cbad..0532a9dfd 100644
--- a/drivers/bus/mlx5_pci/meson.build
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Mellanox Technologies Ltd
 
-deps += ['pci', 'bus_pci', 'common_mlx5']
+deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
 install_headers('rte_bus_mlx5_pci.h')
 sources = files('mlx5_pci_bus.c')
 
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index 66db3c7b0..7344c2083 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -2,13 +2,521 @@
  * Copyright 2020 Mellanox Technologies, Ltd
  */
 
+#include <stdlib.h>
+#include <rte_malloc.h>
 #include "rte_bus_mlx5_pci.h"
+#include <mlx5_common_utils.h>
 
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of class drivers. */
 static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
 				TAILQ_HEAD_INITIALIZER(drv_list);
 
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int dev_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .dev_class = MLX5_CLASS_NET },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	/* New class combination should be added here.
+	 * For example a new multi class device combination
+	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
+	 */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].dev_class;
+	}
+	return -EINVAL;
+}
+
+static struct rte_mlx5_pci_driver *
+class_driver_get(uint32_t class)
+{
+	struct rte_mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->dev_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
 void
 rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
 {
 	TAILQ_INSERT_TAIL(&drv_list, driver, next);
 }
+
+static bool
+mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = class_driver_get(RTE_BIT(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+class_drivers_probe(struct mlx5_pci_device *dev,
+		    struct rte_pci_driver *pci_drv,
+		    struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->dev_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->dev_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load class driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->dev_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	class_drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple PCI class devices.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		   struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more class devices for a PCI device.
+ *
+ * This function removes all class devices belong to a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_bus_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload class drivers. */
+	ret = class_drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+		     uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	struct rte_mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_map) {
+			ret = class->pci_driver.dma_map(pci_dev, addr,
+							iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == class)
+			break;
+		if (device_class_enabled(dev, temp->dev_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+		       uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_unmap) {
+			local_ret = class->pci_driver.dma_unmap(pci_dev, addr,
+								iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5
+ * class drivers.
+ */
+static struct rte_pci_id *mlx5_bus_pci_id_map;
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == mlx5_bus_pci_id_map[i].device_id &&
+		    id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id &&
+		    id->subsystem_vendor_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_vendor_id &&
+		    id->subsystem_device_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(const struct rte_pci_id *id_table, int *next_idx)
+{
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, *next_idx)) {
+			/* New entry; add to the table. */
+			mlx5_bus_pci_id_map[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_build(void)
+{
+	struct rte_mlx5_pci_driver *first_driver;
+	struct rte_mlx5_pci_driver *driver;
+	const struct rte_pci_id *id_table;
+	int num_ids = 0;
+	int i = 0;
+
+	TAILQ_FOREACH(driver, &drv_list, next)
+		num_ids += pci_id_table_size_get(driver->pci_driver.id_table);
+	if (!num_ids)
+		return -ENODEV;
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map));
+	if (!mlx5_bus_pci_id_map)
+		return -ENOMEM;
+	first_driver = TAILQ_FIRST(&drv_list);
+	/* Copy the first class driver's ID table. */
+	for (id_table = first_driver->pci_driver.id_table;
+	     id_table->vendor_id != 0; id_table++, i++)
+		mlx5_bus_pci_id_map[i] = *id_table;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		/* We already added first driver; skip it. */
+		if (driver == first_driver)
+			continue;
+		pci_id_insert(driver->pci_driver.id_table, &i);
+	}
+	mlx5_bus_pci_id_map[i].vendor_id = 0;
+	return 0;
+}
+
+static bool mlx5_bus_registered;
+static struct rte_pci_driver mlx5_bus_driver = {
+	.driver = {
+		.name = "mlx5_bus_pci",
+	},
+	.probe = mlx5_bus_pci_probe,
+	.remove = mlx5_bus_pci_remove,
+	.dma_map = mlx5_bus_pci_dma_map,
+	.dma_unmap = mlx5_bus_pci_dma_unmap,
+};
+
+RTE_INIT(mlx5_bus_pci)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret;
+
+	ret = pci_ids_table_build();
+	if (ret)
+		return;
+	TAILQ_FOREACH(class, &drv_list, next)
+		mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags;
+	mlx5_bus_driver.id_table = mlx5_bus_pci_id_map;
+	rte_pci_register(&mlx5_bus_driver);
+	mlx5_bus_registered = true;
+}
+
+RTE_FINI(mlx5_bus_pci_finish)
+{
+	if (mlx5_bus_registered)
+		rte_pci_unregister(&mlx5_bus_driver);
+	if (mlx5_bus_pci_id_map)
+		free(mlx5_bus_pci_id_map);
+}
+RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
-- 
2.26.2

Re: [dpdk-dev] [PATCH v3 07/10] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

> From: Parav Pandit <parav@mellanox.com>
> Sent: Friday, July 3, 2020 2:43 PM
> 
> Create a mlx5 bus driver framework for invoking drivers of multiple classes
> who have registered with the mlx5_pci bus driver.

[..]
> diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> index 66db3c7b0..7344c2083 100644
> --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> @@ -2,13 +2,521 @@
>   * Copyright 2020 Mellanox Technologies, Ltd
>   */
> +static int
> +mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
> +		     uint64_t iova, size_t len)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +	struct rte_mlx5_pci_driver *temp;
> +	struct mlx5_pci_device *dev;
> +	int ret = -EINVAL;
> +
> +	dev = pci_to_mlx5_device(pci_dev);
> +	if (!dev)
> +		return -ENODEV;
> +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head,
> next) {
> +		if (device_class_enabled(dev, class->dev_class) &&
> +		    class->pci_driver.dma_map) {
> +			ret = class->pci_driver.dma_map(pci_dev, addr,
> +							iova, len);
> +			if (ret)
> +				goto map_err;
> +		}
> +	}
> +	return ret;
> +map_err:
> +	TAILQ_FOREACH(temp, &drv_list, next) {

I made a mistake in reusing the logic of the dma_unmap() callback here.
This loop needs to follow same sequence as that what dma_map() does.
I will send the fix. Will wait for some time to gather if there are comments.

> +		if (temp == class)
> +			break;
> +		if (device_class_enabled(dev, temp->dev_class) &&
> +		    temp->pci_driver.dma_map && temp-
> >pci_driver.dma_unmap)
> +			temp->pci_driver.dma_unmap(pci_dev, addr, iova,
> len);
> +	}
> +	return ret;
> +}

[dpdk-dev] [PATCH v3 08/10] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Parav Pandit]

Enable class driver to match with the mlx5 pci devices.
Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Avoid static table
v1->v2:
 - Migreate API from rte_driver to rte_pci_driver
---
 drivers/bus/Makefile             |  3 +++
 drivers/net/mlx5/Makefile        |  3 ++-
 drivers/net/mlx5/linux/mlx5_os.c |  1 -
 drivers/net/mlx5/linux/mlx5_os.h |  1 +
 drivers/net/mlx5/meson.build     |  2 +-
 drivers/net/mlx5/mlx5.c          | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h          |  1 -
 drivers/vdpa/mlx5/Makefile       |  3 ++-
 drivers/vdpa/mlx5/meson.build    |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c    | 23 +++++++++++++----------
 mk/rte.app.mk                    |  1 +
 11 files changed, 38 insertions(+), 26 deletions(-)

diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile
index cea3b55e6..f96caa5ec 100644
--- a/drivers/bus/Makefile
+++ b/drivers/bus/Makefile
@@ -9,6 +9,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
 DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
+DIRS-y += mlx5_pci
+endif
 DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
 DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus
 
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index a458402dc..1cea7cd07 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_bus_mlx5_pci
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 2dc57b20e..b6042b7ef 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1321,7 +1321,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 31add3988..695722520 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,4 +15,5 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
+
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index e95ce0267..26699a79b 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -8,7 +8,7 @@ if not (is_linux or is_windows)
 	subdir_done()
 endif
 
-deps += ['hash', 'common_mlx5']
+deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
 sources = files(
 	'mlx5.c',
 	'mlx5_ethdev.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index bb10c63bb..2270055fe 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -34,6 +34,7 @@
 #include <rte_spinlock.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
+#include <rte_bus_mlx5_pci.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
@@ -1892,16 +1893,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct rte_mlx5_pci_driver mlx5_driver = {
+	.dev_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /**
@@ -1919,7 +1923,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		rte_mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 46e66eb1c..df7b5eb87 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -129,7 +129,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..9bb5dfdbf 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
 CFLAGS += -I$(RTE_SDK)/lib/librte_sched
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..f4175c34e 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index f043166ef..09c9cb935 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,7 +11,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <rte_bus_mlx5_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI bus device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct rte_mlx5_pci_driver mlx5_vdpa_driver = {
+	.dev_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 /**
@@ -849,7 +852,7 @@ RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 	if (mlx5_vdpa_logtype >= 0)
 		rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE);
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0ce8cf541..d7c1d90e4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF)      += -lrte_pmd_memif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD)       += -lrte_pmd_mlx4
 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
 _LDLIBS-y                                   += -lrte_common_mlx5
+_LDLIBS-y                                   += -lrte_bus_mlx5_pci
 endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)  += -lrte_pmd_mlx5_vdpa
-- 
2.26.2

[dpdk-dev] [PATCH v3 09/10] common/mlx5: remove class checks from individual driver

[Parav Pandit]

Now that mlx5_pci bus does the check for enabled classes and performs
probe(), remove() of associated classes, individual class driver
doesn't need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Removed empty line
v1->v2:
 - New patch
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  2 -
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/net/mlx5/linux/mlx5_os.h              |  1 -
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 6 files changed, 52 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ce5e4db1..771e046c1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 62861af05..ca66c4033 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -243,8 +243,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index ae57ebdba..01b4358a0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,8 +1,6 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index b6042b7ef..2bd2b664d 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1308,11 +1308,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_dev_config dev_config;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 695722520..31add3988 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,5 +15,4 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
-
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 09c9cb935..41d69e8a8 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

[dpdk-dev] [PATCH v3 10/10] maintainers: add maintainers for mlx5 pci bus

[Parav Pandit]

Added maintainers for new mlx5 specific mlx5_pci bus.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 MAINTAINERS | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 53a5e9a9e..e3fec55ca 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -517,6 +517,11 @@ Intel FPGA bus
 M: Rosen Xu <rosen.xu@intel.com>
 F: drivers/bus/ifpga/
 
+Melllanox mlx5 PCI bus driver
+M: Parav Pandit <parav@mellaox.com>
+M: Matan Azrad <matan@mellanox.com>
+F: drivers/bus/mlx5_pci
+
 NXP buses
 M: Hemant Agrawal <hemant.agrawal@nxp.com>
 M: Sachin Saxena <sachin.saxena@nxp.com>
-- 
2.26.2

[dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
a new mlx5 PCI bus is created. This bus allows binding multiple
class drivers (such as net, vdpa, regex(future)) to bind to the
mlx5 PCI bus driver.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Introduces new RTE constructor priority for common
initialization
Patch-3 Fixes compilation error
Patch-4 Define mlx5 class as bit fields
Patch-5 Uses new RTE common priority
Patch-6 Adds mlx5 PCI bus
Patch-7 Implements a mlx5 PCI bus driver
Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
API instead of rte PCI bus API
Patch-9 Removed class check code as its already part of the bus now
Patch-10 Add maintainers for the new bus

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       -------------       /
         \______|   mlx5    |_____ /
                |   pci bus |
                -------------   
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD defines its
  specific PCI ID table and mlx5 bus driver probes matching
  class drivers.
- mlx5 pci bus driver is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI bus driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci bus, a common driver is
implemented which exposes class registration API.
However, bus model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common bus driver.
It also eliminates peer blind zone problem as bottom pci
bus layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

In future,
./testpmd -w <PCI BDF>,class=net:regex

Changelog:
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (10):
  eal: introduce macros for getting value for bit
  eal: introduce RTE common initialization level
  common/mlx5: fix empty input style in glue wrappers
  common/mlx5: change mlx5 class enum values as bits
  common/mlx5: use common rte priority
  bus/mlx5_pci: add mlx5 PCI bus
  bus/mlx5_pci: register a PCI driver
  bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
  common/mlx5: remove class checks from individual driver
  maintainers: add maintainers for mlx5 pci bus

 MAINTAINERS                                   |   5 +
 drivers/bus/Makefile                          |   3 +
 drivers/bus/meson.build                       |   2 +-
 drivers/bus/mlx5_pci/Makefile                 |  40 ++
 drivers/bus/mlx5_pci/meson.build              |  19 +
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 522 ++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  84 +++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/mlx5_common.c             |  39 +-
 drivers/common/mlx5/mlx5_common.h             |   7 +-
 .../common/mlx5/rte_common_mlx5_version.map   |   2 -
 drivers/net/mlx5/Makefile                     |   3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/meson.build                  |   2 +-
 drivers/net/mlx5/mlx5.c                       |  26 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  30 +-
 lib/librte_eal/include/rte_bitops.h           |   2 +
 lib/librte_eal/include/rte_common.h           |   1 +
 mk/rte.app.mk                                 |   1 +
 23 files changed, 724 insertions(+), 85 deletions(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

-- 
2.26.2

[dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting value for bit

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..d72c7cd93 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,8 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+#define RTE_BIT(bit_num)	(1UL << (bit_num))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.26.2

Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit

[Morten Brørup]

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Parav Pandit
> Sent: Friday, July 3, 2020 3:47 PM
> 
> There are several drivers which duplicate bit generation macro.
> Introduce a generic bit macros so that such drivers avoid redefining
> same in multiple drivers.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> Acked-by: Matan Azrad <matan@mellanox.com>
> ---
> Changelog:
> v1->v2:
>  - Addressed comments from Thomas and Gaten.
>  - Avoided new file, added macro to rte_bitops.h
> ---
>  lib/librte_eal/include/rte_bitops.h | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/lib/librte_eal/include/rte_bitops.h
> b/lib/librte_eal/include/rte_bitops.h
> index 740927f3b..d72c7cd93 100644
> --- a/lib/librte_eal/include/rte_bitops.h
> +++ b/lib/librte_eal/include/rte_bitops.h
> @@ -17,6 +17,8 @@
>  #include <rte_debug.h>
>  #include <rte_compat.h>
> 
> +#define RTE_BIT(bit_num)	(1UL << (bit_num))

Is the return value 32 or 64 bit, or is intended to depend on the target architecture?

Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of 1UL, if you want a specific size.

It could be a static inline __attribute__((__pure__)) function instead of a macro, but it's not important for me.

The macro/function needs a description for the documentation.

I'm also concerned about the name of the macro being too generic. But the effort of changing all the drivers where it is being used already could be too big if the name changes too.

And the macro/function is new, so shouldn't it - in theory - be marked as experimental?

> +
>  /*------------------------ 32-bit relaxed operations -----------------
> -------*/
> 
>  /**
> --
> 2.26.2
> 

Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit

[Parav Pandit]

> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Monday, July 6, 2020 4:24 PM
> 
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Parav Pandit
> > Sent: Friday, July 3, 2020 3:47 PM
> >
> > There are several drivers which duplicate bit generation macro.
> > Introduce a generic bit macros so that such drivers avoid redefining
> > same in multiple drivers.
> >
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > Acked-by: Matan Azrad <matan@mellanox.com>
> > ---
> > Changelog:
> > v1->v2:
> >  - Addressed comments from Thomas and Gaten.
> >  - Avoided new file, added macro to rte_bitops.h
> > ---
> >  lib/librte_eal/include/rte_bitops.h | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/lib/librte_eal/include/rte_bitops.h
> > b/lib/librte_eal/include/rte_bitops.h
> > index 740927f3b..d72c7cd93 100644
> > --- a/lib/librte_eal/include/rte_bitops.h
> > +++ b/lib/librte_eal/include/rte_bitops.h
> > @@ -17,6 +17,8 @@
> >  #include <rte_debug.h>
> >  #include <rte_compat.h>
> >
> > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> 
> Is the return value 32 or 64 bit, or is intended to depend on the target
> architecture?
>
It should be 64-bit.
 
> Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of 1UL, if you
> want a specific size.
> 
Will do UINT64_C(1).

> It could be a static inline __attribute__((__pure__)) function instead of a macro,
> but it's not important for me.
> 
> The macro/function needs a description for the documentation.
> 
In this header file or outside?

> I'm also concerned about the name of the macro being too generic. But the
> effort of changing all the drivers where it is being used already could be too big
> if the name changes too.
>
Right. Currently drivers have generic name as BIT(). Close to 3000 entries.
So doing at RTE_BIT to match other rte_ APIs.
Drivers can slowly migrate at their pace to this one.
 
> And the macro/function is new, so shouldn't it - in theory - be marked as
> experimental?

How to mark a macro as experimental?

Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit

[Thomas Monjalon]

07/07/2020 13:38, Parav Pandit:
> From: Morten Brørup <mb@smartsharesystems.com>
> > From: Parav Pandit
> > > --- a/lib/librte_eal/include/rte_bitops.h
> > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > @@ -17,6 +17,8 @@
> > >  #include <rte_debug.h>
> > >  #include <rte_compat.h>
> > >
> > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> > 
> > Is the return value 32 or 64 bit, or is intended to depend on the target
> > architecture?
> >
> It should be 64-bit.
>  
> > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of 1UL, if you
> > want a specific size.
> > 
> Will do UINT64_C(1).
> 
> > It could be a static inline __attribute__((__pure__)) function instead of a macro,
> > but it's not important for me.
> > 
> > The macro/function needs a description for the documentation.
> > 
> In this header file or outside?

It is asked to add a doxygen comment.


> > I'm also concerned about the name of the macro being too generic. But the
> > effort of changing all the drivers where it is being used already could be too big
> > if the name changes too.
> >
> Right. Currently drivers have generic name as BIT(). Close to 3000 entries.
> So doing at RTE_BIT to match other rte_ APIs.
> Drivers can slowly migrate at their pace to this one.
>  
> > And the macro/function is new, so shouldn't it - in theory - be marked as
> > experimental?
> 
> How to mark a macro as experimental?

A macro cannot be experimental.

Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit

[Morten Brørup]

Adding Joyce Kong to this discussion as the rte_bitops maintainer.

> From: Thomas Monjalon [mailto:thomas@monjalon.net]
> Sent: Tuesday, July 7, 2020 2:13 PM
> 
> 07/07/2020 13:38, Parav Pandit:
> > From: Morten Brørup <mb@smartsharesystems.com>
> > > From: Parav Pandit
> > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > @@ -17,6 +17,8 @@
> > > >  #include <rte_debug.h>
> > > >  #include <rte_compat.h>
> > > >
> > > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> > >
> > > Is the return value 32 or 64 bit, or is intended to depend on the
> target
> > > architecture?
> > >
> > It should be 64-bit.
> >
> > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of
> 1UL, if you
> > > want a specific size.
> > >
> > Will do UINT64_C(1).
> >
> > > It could be a static inline __attribute__((__pure__)) function
> instead of a macro,
> > > but it's not important for me.
> > >
> > > The macro/function needs a description for the documentation.
> > >
> > In this header file or outside?
> 
> It is asked to add a doxygen comment.
> 
> 
> > > I'm also concerned about the name of the macro being too generic.
> But the
> > > effort of changing all the drivers where it is being used already
> could be too big
> > > if the name changes too.
> > >
> > Right. Currently drivers have generic name as BIT(). Close to 3000
> entries.
> > So doing at RTE_BIT to match other rte_ APIs.
> > Drivers can slowly migrate at their pace to this one.
> >
> > > And the macro/function is new, so shouldn't it - in theory - be
> marked as
> > > experimental?
> >
> > How to mark a macro as experimental?
> 
> A macro cannot be experimental.
> 

OK. If the macro is given a future proof name, I guess it should be accepted.

If we want boundary checks, I suggest a macro like:

#define RTE_BIT64(nr)						\
	({								\
		typeof(nr) n = nr; 				\
		RTE_BUILD_BUG_ON((n > 64) || (n < 0)); 	\
		UINT64_C(1) << (n);				\
	})

Or a function:

__rte_experimental
static __rte_always_inline __attribute__((const)) uint64_t
rte_bit64(const unsigned int nr)
{
	RTE_ASSERT(nr < 64);

	return UINT64_C(1) << nr;
}

Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit

[Parav Pandit]

Hi Morten,

> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Tuesday, July 7, 2020 6:11 PM

> Adding Joyce Kong to this discussion as the rte_bitops maintainer.
> 
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > Sent: Tuesday, July 7, 2020 2:13 PM
> >
> > 07/07/2020 13:38, Parav Pandit:
> > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > From: Parav Pandit
> > > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > > @@ -17,6 +17,8 @@
> > > > >  #include <rte_debug.h>
> > > > >  #include <rte_compat.h>
> > > > >
> > > > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> > > >
> > > > Is the return value 32 or 64 bit, or is intended to depend on the
> > target
> > > > architecture?
> > > >
> > > It should be 64-bit.
> > >
> > > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of
> > 1UL, if you
> > > > want a specific size.
> > > >
> > > Will do UINT64_C(1).
> > >
> > > > It could be a static inline __attribute__((__pure__)) function
> > instead of a macro,
> > > > but it's not important for me.
> > > >
> > > > The macro/function needs a description for the documentation.
> > > >
> > > In this header file or outside?
> >
> > It is asked to add a doxygen comment.
Ok. will add.

> >
> >
> > > > I'm also concerned about the name of the macro being too generic.
> > But the
> > > > effort of changing all the drivers where it is being used already
> > could be too big
> > > > if the name changes too.
> > > >
> > > Right. Currently drivers have generic name as BIT(). Close to 3000
> > entries.
> > > So doing at RTE_BIT to match other rte_ APIs.
> > > Drivers can slowly migrate at their pace to this one.
> > >
> > > > And the macro/function is new, so shouldn't it - in theory - be
> > marked as
> > > > experimental?
> > >
> > > How to mark a macro as experimental?
> >
> > A macro cannot be experimental.
> >
> 
> OK. If the macro is given a future proof name, I guess it should be accepted.
> 
> If we want boundary checks, I suggest a macro like:
> 
> #define RTE_BIT64(nr)						\
> 	({								\
> 		typeof(nr) n = nr; 				\
> 		RTE_BUILD_BUG_ON((n > 64) || (n < 0)); 	\
> 		UINT64_C(1) << (n);				\
> 	})
>
Compiler doesn't like it.

../lib/librte_eal/include/rte_bitops.h:21:2: error: braced-group within expression allowed only inside a function
  ({      \
  ^
 
> Or a function:
> 
> __rte_experimental
> static __rte_always_inline __attribute__((const)) uint64_t rte_bit64(const
> unsigned int nr) {
> 	RTE_ASSERT(nr < 64);
> 
> 	return UINT64_C(1) << nr;
> }
> 
Value retrieved using this macro is used an enum. Don't see how a function call like above can solve it.

For a below macro definition, compiler is already catching for negative value when RTE_BIT64(-1) is done,

../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift count is negative [-Wshift-count-negative]
 #define RTE_BIT64(nr) (UINT64_C(1) << (nr))

And when RTE_BIT64(259) is done below error is done,

../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift count >= width of type [-Wshift-count-overflow]
 #define RTE_BIT64(nr) (UINT64_C(1) << (nr))

So below definition is good covering all needed cases.

#define RTE_BIT64(nr) (UINT64_C(1) << (nr))

Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit

[Morten Brørup]

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Parav Pandit
> Sent: Thursday, July 9, 2020 8:24 AM
> 
> Hi Morten,
> 
> > From: Morten Brørup <mb@smartsharesystems.com>
> > Sent: Tuesday, July 7, 2020 6:11 PM
> 
> > Adding Joyce Kong to this discussion as the rte_bitops maintainer.
> >
> > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > Sent: Tuesday, July 7, 2020 2:13 PM
> > >
> > > 07/07/2020 13:38, Parav Pandit:
> > > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > > From: Parav Pandit
> > > > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > > > @@ -17,6 +17,8 @@
> > > > > >  #include <rte_debug.h>
> > > > > >  #include <rte_compat.h>
> > > > > >
> > > > > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> > > > >
> > > > > Is the return value 32 or 64 bit, or is intended to depend on
> the
> > > target
> > > > > architecture?
> > > > >
> > > > It should be 64-bit.
> > > >
> > > > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead
> of
> > > 1UL, if you
> > > > > want a specific size.
> > > > >
> > > > Will do UINT64_C(1).
> > > >
> > > > > It could be a static inline __attribute__((__pure__)) function
> > > instead of a macro,
> > > > > but it's not important for me.
> > > > >
> > > > > The macro/function needs a description for the documentation.
> > > > >
> > > > In this header file or outside?
> > >
> > > It is asked to add a doxygen comment.
> Ok. will add.
> 
> > >
> > >
> > > > > I'm also concerned about the name of the macro being too
> generic.
> > > But the
> > > > > effort of changing all the drivers where it is being used
> already
> > > could be too big
> > > > > if the name changes too.
> > > > >
> > > > Right. Currently drivers have generic name as BIT(). Close to
> 3000
> > > entries.
> > > > So doing at RTE_BIT to match other rte_ APIs.
> > > > Drivers can slowly migrate at their pace to this one.
> > > >
> > > > > And the macro/function is new, so shouldn't it - in theory - be
> > > marked as
> > > > > experimental?
> > > >
> > > > How to mark a macro as experimental?
> > >
> > > A macro cannot be experimental.
> > >
> >
> > OK. If the macro is given a future proof name, I guess it should be
> accepted.
> >
> > If we want boundary checks, I suggest a macro like:
> >
> > #define RTE_BIT64(nr)						\
> > 	({								\
> > 		typeof(nr) n = nr; 				\
> > 		RTE_BUILD_BUG_ON((n > 64) || (n < 0)); 	\
> > 		UINT64_C(1) << (n);				\
> > 	})
> >
> Compiler doesn't like it.
> 
> ../lib/librte_eal/include/rte_bitops.h:21:2: error: braced-group within
> expression allowed only inside a function
>   ({      \
>   ^
> 
> > Or a function:
> >
> > __rte_experimental
> > static __rte_always_inline __attribute__((const)) uint64_t
> rte_bit64(const
> > unsigned int nr) {
> > 	RTE_ASSERT(nr < 64);
> >
> > 	return UINT64_C(1) << nr;
> > }
> >
> Value retrieved using this macro is used an enum. Don't see how a
> function call like above can solve it.
> 
> For a below macro definition, compiler is already catching for negative
> value when RTE_BIT64(-1) is done,
> 
> ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift count
> is negative [-Wshift-count-negative]
>  #define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> 
> And when RTE_BIT64(259) is done below error is done,
> 
> ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift count
> >= width of type [-Wshift-count-overflow]
>  #define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> 
> So below definition is good covering all needed cases.
> 
> #define RTE_BIT64(nr) (UINT64_C(1) << (nr))

Great. Then, when you have added a doxygen comment:

Acked-by: Morten Brørup <mb@smartsharesystems.com>

Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit

[Parav Pandit]

> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Thursday, July 9, 2020 12:46 PM
> 
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Parav Pandit
> > Sent: Thursday, July 9, 2020 8:24 AM
> >
> > Hi Morten,
> >
> > > From: Morten Brørup <mb@smartsharesystems.com>
> > > Sent: Tuesday, July 7, 2020 6:11 PM
> >
> > > Adding Joyce Kong to this discussion as the rte_bitops maintainer.
> > >
> > > > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > > > Sent: Tuesday, July 7, 2020 2:13 PM
> > > >
> > > > 07/07/2020 13:38, Parav Pandit:
> > > > > From: Morten Brørup <mb@smartsharesystems.com>
> > > > > > From: Parav Pandit
> > > > > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > > > > @@ -17,6 +17,8 @@
> > > > > > >  #include <rte_debug.h>
> > > > > > >  #include <rte_compat.h>
> > > > > > >
> > > > > > > +#define RTE_BIT(bit_num)	(1UL << (bit_num))
> > > > > >
> > > > > > Is the return value 32 or 64 bit, or is intended to depend on
> > the
> > > > target
> > > > > > architecture?
> > > > > >
> > > > > It should be 64-bit.
> > > > >
> > > > > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead
> > of
> > > > 1UL, if you
> > > > > > want a specific size.
> > > > > >
> > > > > Will do UINT64_C(1).
> > > > >
> > > > > > It could be a static inline __attribute__((__pure__)) function
> > > > instead of a macro,
> > > > > > but it's not important for me.
> > > > > >
> > > > > > The macro/function needs a description for the documentation.
> > > > > >
> > > > > In this header file or outside?
> > > >
> > > > It is asked to add a doxygen comment.
> > Ok. will add.
> >
> > > >
> > > >
> > > > > > I'm also concerned about the name of the macro being too
> > generic.
> > > > But the
> > > > > > effort of changing all the drivers where it is being used
> > already
> > > > could be too big
> > > > > > if the name changes too.
> > > > > >
> > > > > Right. Currently drivers have generic name as BIT(). Close to
> > 3000
> > > > entries.
> > > > > So doing at RTE_BIT to match other rte_ APIs.
> > > > > Drivers can slowly migrate at their pace to this one.
> > > > >
> > > > > > And the macro/function is new, so shouldn't it - in theory -
> > > > > > be
> > > > marked as
> > > > > > experimental?
> > > > >
> > > > > How to mark a macro as experimental?
> > > >
> > > > A macro cannot be experimental.
> > > >
> > >
> > > OK. If the macro is given a future proof name, I guess it should be
> > accepted.
> > >
> > > If we want boundary checks, I suggest a macro like:
> > >
> > > #define RTE_BIT64(nr)						\
> > > 	({								\
> > > 		typeof(nr) n = nr; 				\
> > > 		RTE_BUILD_BUG_ON((n > 64) || (n < 0)); 	\
> > > 		UINT64_C(1) << (n);				\
> > > 	})
> > >
> > Compiler doesn't like it.
> >
> > ../lib/librte_eal/include/rte_bitops.h:21:2: error: braced-group
> > within expression allowed only inside a function
> >   ({      \
> >   ^
> >
> > > Or a function:
> > >
> > > __rte_experimental
> > > static __rte_always_inline __attribute__((const)) uint64_t
> > rte_bit64(const
> > > unsigned int nr) {
> > > 	RTE_ASSERT(nr < 64);
> > >
> > > 	return UINT64_C(1) << nr;
> > > }
> > >
> > Value retrieved using this macro is used an enum. Don't see how a
> > function call like above can solve it.
> >
> > For a below macro definition, compiler is already catching for
> > negative value when RTE_BIT64(-1) is done,
> >
> > ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift
> > count is negative [-Wshift-count-negative]  #define RTE_BIT64(nr)
> > (UINT64_C(1) << (nr))
> >
> > And when RTE_BIT64(259) is done below error is done,
> >
> > ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift
> > count
> > >= width of type [-Wshift-count-overflow]
> >  #define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> >
> > So below definition is good covering all needed cases.
> >
> > #define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> 
> Great. Then, when you have added a doxygen comment:
> 
> Acked-by: Morten Brørup <mb@smartsharesystems.com>

Thanks Morten; adding it.

[dpdk-dev] [PATCH v4 02/10] eal: introduce RTE common initialization level

[Parav Pandit]

Currently mlx5_common uses CLASS priority to initialize
common code before initializing the PMD.
However mlx5_common is not really a class, it is the pre-initialization
code needed for the PMDs.

In subsequent patch a needed initialization sequence is:
(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
for class driver initialization.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 lib/librte_eal/include/rte_common.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
index 0843ce69e..d4653ea77 100644
--- a/lib/librte_eal/include/rte_common.h
+++ b/lib/librte_eal/include/rte_common.h
@@ -126,6 +126,7 @@ typedef uint16_t unaligned_uint16_t;
 
 #define RTE_PRIORITY_LOG 101
 #define RTE_PRIORITY_BUS 110
+#define RTE_PRIORITY_COMMON 119
 #define RTE_PRIORITY_CLASS 120
 #define RTE_PRIORITY_LAST 65535
 
-- 
2.26.2

[dpdk-dev] [PATCH v4 03/10] common/mlx5: fix empty input style in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")
Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index 069d8540c..e5e052a6a 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v4 04/10] common/mlx5: change mlx5 class enum values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v1->v2:
 - Rebasd due to removal previous patch
---
 drivers/common/mlx5/mlx5_common.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index da01ffa1c..62861af05 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -207,9 +208,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT(0),
+	MLX5_CLASS_VDPA = RTE_BIT(1),
 };
 
 #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */
-- 
2.26.2

[dpdk-dev] [PATCH v4 05/10] common/mlx5: use common rte priority

[Parav Pandit]

Use RTE_PRIO_COMMON for mlx5 common initialization and use
RTE_PRIO_CLASS for mlx5 net, vdpa PMDs.

This enables to do following initialization sequence.

(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c | 2 +-
 drivers/net/mlx5/mlx5.c           | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 693e2c68c..0ce5e4db1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 /**
  * Initialization routine for run-time dependency on glue library.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+RTE_INIT_PRIO(mlx5_glue_init, COMMON)
 {
 	mlx5_glue_constructor();
 }
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 07c6addd5..bb10c63bb 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1907,7 +1907,7 @@ struct rte_pci_driver mlx5_driver = {
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_pmd_init)
+RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 {
 	/* Initialize driver log type. */
 	mlx5_logtype = rte_log_register("pmd.net.mlx5");
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index dbd36ab0c..f043166ef 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -842,7 +842,7 @@ static struct rte_pci_driver mlx5_vdpa_driver = {
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_vdpa_init)
+RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 {
 	/* Initialize common log type. */
 	mlx5_vdpa_logtype = rte_log_register("pmd.vdpa.mlx5");
-- 
2.26.2

[dpdk-dev] [PATCH v4 06/10] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
pci device.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Addressed comments from Thomas and Asaf
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address comments from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Changed alignedment to mlx5 specific aligment instead of standard
   DPDK
 - Using uint32_t instead of mlx5_class enum
v1->v2:
 - Address comments from Thomas and Gaetan
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
---
 drivers/bus/meson.build                       |  2 +-
 drivers/bus/mlx5_pci/Makefile                 | 38 +++++++++
 drivers/bus/mlx5_pci/meson.build              | 19 +++++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 84 +++++++++++++++++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
 6 files changed, 161 insertions(+), 1 deletion(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 80de2d91d..b1381838d 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
+drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
 std_deps = ['eal']
 config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
 driver_name_fmt = 'rte_bus_@0@'
diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
new file mode 100644
index 000000000..de4ccd83f
--- /dev/null
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Mellanox Technologies, Ltd
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_bus_mlx5_pci.a
+
+CFLAGS += -O3 -Wall -Wextra
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -Wno-strict-prototypes
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
+CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
+CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+LDLIBS += -lrte_eal
+LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# versioning export map
+EXPORT_MAP := rte_bus_mlx5_pci_version.map
+
+SRCS-y += mlx5_pci_bus.c
+
+# DEBUG which is usually provided on the command-line may enable
+# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
+ifeq ($(DEBUG),1)
+CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
+endif
+
+#
+# Export include files
+#
+SYMLINK-y-include += rte_bus_mlx5_pci.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
new file mode 100644
index 000000000..64a17cbad
--- /dev/null
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Mellanox Technologies Ltd
+
+deps += ['pci', 'bus_pci', 'common_mlx5']
+install_headers('rte_bus_mlx5_pci.h')
+sources = files('mlx5_pci_bus.c')
+
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600'
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
new file mode 100644
index 000000000..66db3c7b0
--- /dev/null
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include "rte_bus_mlx5_pci.h"
+
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
+{
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
new file mode 100644
index 000000000..9f8d22e2b
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _RTE_BUS_MLX5_PCI_H_
+#define _RTE_BUS_MLX5_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Bus Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This bus enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci bus driver.
+ *
+ * -----------    ------------    -----------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5        |
+ * | net pmd |    | vdpa pmd |    | new class pmd |
+ * -----------    ------------    -----------------
+ *      \              |                /
+ *       \             |               /
+ *        \       -------------       /
+ *         \______|   mlx5    |_____ /
+ *                |   pci bus |
+ *                -------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct rte_mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t dev_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a rte_mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
+
+#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \
+	static const char *mlx5_pci_drvinit_fn_ ## nm; \
+	RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \
+	{ \
+		(drv).driver.name = RTE_STR(nm); \
+		rte_mlx5_pci_driver_register(&drv); \
+	} \
+	RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _RTE_BUS_MLX5_PCI_H_ */
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
new file mode 100644
index 000000000..4cfd3db10
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
@@ -0,0 +1,5 @@
+INTERNAL {
+	global:
+
+	rte_mlx5_pci_driver_register;
+};
-- 
2.26.2

[dpdk-dev] [PATCH v4 07/10] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

Create a mlx5 bus driver framework for invoking drivers of
multiple classes who have registered with the mlx5_pci bus
driver.

Validate user class arguments for supported class combinations.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Addressed comments from Asaf
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Addressed comments from Matan.
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block.
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Maintaining class driver list to keep load/unload order symmetric and
   mirror of each other.
 - Deriving drv_flags based on the class drivers
 - Using PCI address comparision helper instead of pointer comparision
 - Fixed alignment for id_table
 - Continue to probe_err if device is already probed
 - Perform dma map on best effort basis for all supported drivers
 - Removed drv_flags check
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Address comments from Thomas and Gaetan
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - USe uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
---
 drivers/bus/mlx5_pci/Makefile       |   2 +
 drivers/bus/mlx5_pci/meson.build    |   2 +-
 drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++
 3 files changed, 511 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
index de4ccd83f..1a005ee32 100644
--- a/drivers/bus/mlx5_pci/Makefile
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -15,7 +15,9 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+CFLAGS += -D_DEFAULT_SOURCE
 LDLIBS += -lrte_eal
+LDLIBS += -lrte_kvargs
 LDLIBS += -lrte_common_mlx5
 LDLIBS += -lrte_pci -lrte_bus_pci
 
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
index 64a17cbad..0532a9dfd 100644
--- a/drivers/bus/mlx5_pci/meson.build
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Mellanox Technologies Ltd
 
-deps += ['pci', 'bus_pci', 'common_mlx5']
+deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
 install_headers('rte_bus_mlx5_pci.h')
 sources = files('mlx5_pci_bus.c')
 
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index 66db3c7b0..d6e6da3dc 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -2,13 +2,521 @@
  * Copyright 2020 Mellanox Technologies, Ltd
  */
 
+#include <stdlib.h>
+#include <rte_malloc.h>
 #include "rte_bus_mlx5_pci.h"
+#include <mlx5_common_utils.h>
 
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of class drivers. */
 static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
 				TAILQ_HEAD_INITIALIZER(drv_list);
 
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int dev_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .dev_class = MLX5_CLASS_NET },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	/* New class combination should be added here.
+	 * For example a new multi class device combination
+	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
+	 */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].dev_class;
+	}
+	return -EINVAL;
+}
+
+static struct rte_mlx5_pci_driver *
+class_driver_get(uint32_t class)
+{
+	struct rte_mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->dev_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
 void
 rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
 {
 	TAILQ_INSERT_TAIL(&drv_list, driver, next);
 }
+
+static bool
+mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = class_driver_get(RTE_BIT(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+class_drivers_probe(struct mlx5_pci_device *dev,
+		    struct rte_pci_driver *pci_drv,
+		    struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->dev_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->dev_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load class driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->dev_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	class_drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple PCI class devices.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		   struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more class devices for a PCI device.
+ *
+ * This function removes all class devices belong to a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_bus_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload class drivers. */
+	ret = class_drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+		     uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class = NULL;
+	struct rte_mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_map) {
+			ret = class->pci_driver.dma_map(pci_dev, addr,
+							iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == class)
+			break;
+		if (device_class_enabled(dev, temp->dev_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+		       uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_unmap) {
+			local_ret = class->pci_driver.dma_unmap(pci_dev, addr,
+								iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5
+ * class drivers.
+ */
+static struct rte_pci_id *mlx5_bus_pci_id_map;
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == mlx5_bus_pci_id_map[i].device_id &&
+		    id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id &&
+		    id->subsystem_vendor_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_vendor_id &&
+		    id->subsystem_device_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(const struct rte_pci_id *id_table, int *next_idx)
+{
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, *next_idx)) {
+			/* New entry; add to the table. */
+			mlx5_bus_pci_id_map[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_build(void)
+{
+	struct rte_mlx5_pci_driver *first_driver;
+	struct rte_mlx5_pci_driver *driver;
+	const struct rte_pci_id *id_table;
+	int num_ids = 0;
+	int i = 0;
+
+	TAILQ_FOREACH(driver, &drv_list, next)
+		num_ids += pci_id_table_size_get(driver->pci_driver.id_table);
+	if (!num_ids)
+		return -ENODEV;
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map));
+	if (!mlx5_bus_pci_id_map)
+		return -ENOMEM;
+	first_driver = TAILQ_FIRST(&drv_list);
+	/* Copy the first class driver's ID table. */
+	for (id_table = first_driver->pci_driver.id_table;
+	     id_table->vendor_id != 0; id_table++, i++)
+		mlx5_bus_pci_id_map[i] = *id_table;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		/* We already added first driver; skip it. */
+		if (driver == first_driver)
+			continue;
+		pci_id_insert(driver->pci_driver.id_table, &i);
+	}
+	mlx5_bus_pci_id_map[i].vendor_id = 0;
+	return 0;
+}
+
+static bool mlx5_bus_registered;
+static struct rte_pci_driver mlx5_bus_driver = {
+	.driver = {
+		.name = "mlx5_bus_pci",
+	},
+	.probe = mlx5_bus_pci_probe,
+	.remove = mlx5_bus_pci_remove,
+	.dma_map = mlx5_bus_pci_dma_map,
+	.dma_unmap = mlx5_bus_pci_dma_unmap,
+};
+
+RTE_INIT(mlx5_bus_pci)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret;
+
+	ret = pci_ids_table_build();
+	if (ret)
+		return;
+	TAILQ_FOREACH(class, &drv_list, next)
+		mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags;
+	mlx5_bus_driver.id_table = mlx5_bus_pci_id_map;
+	rte_pci_register(&mlx5_bus_driver);
+	mlx5_bus_registered = true;
+}
+
+RTE_FINI(mlx5_bus_pci_finish)
+{
+	if (mlx5_bus_registered)
+		rte_pci_unregister(&mlx5_bus_driver);
+	if (mlx5_bus_pci_id_map)
+		free(mlx5_bus_pci_id_map);
+}
+RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
-- 
2.26.2

[dpdk-dev] [PATCH v4 08/10] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Parav Pandit]

Enable class driver to match with the mlx5 pci devices.
Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Avoid static table
v1->v2:
 - Migreate API from rte_driver to rte_pci_driver
---
 drivers/bus/Makefile             |  3 +++
 drivers/net/mlx5/Makefile        |  3 ++-
 drivers/net/mlx5/linux/mlx5_os.c |  1 -
 drivers/net/mlx5/linux/mlx5_os.h |  1 +
 drivers/net/mlx5/meson.build     |  2 +-
 drivers/net/mlx5/mlx5.c          | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h          |  1 -
 drivers/vdpa/mlx5/Makefile       |  3 ++-
 drivers/vdpa/mlx5/meson.build    |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c    | 23 +++++++++++++----------
 mk/rte.app.mk                    |  1 +
 11 files changed, 38 insertions(+), 26 deletions(-)

diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile
index cea3b55e6..f96caa5ec 100644
--- a/drivers/bus/Makefile
+++ b/drivers/bus/Makefile
@@ -9,6 +9,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
 DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
+DIRS-y += mlx5_pci
+endif
 DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
 DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus
 
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index a458402dc..1cea7cd07 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_bus_mlx5_pci
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 2dc57b20e..b6042b7ef 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1321,7 +1321,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 31add3988..695722520 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,4 +15,5 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
+
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index e95ce0267..26699a79b 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -8,7 +8,7 @@ if not (is_linux or is_windows)
 	subdir_done()
 endif
 
-deps += ['hash', 'common_mlx5']
+deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
 sources = files(
 	'mlx5.c',
 	'mlx5_ethdev.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index bb10c63bb..2270055fe 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -34,6 +34,7 @@
 #include <rte_spinlock.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
+#include <rte_bus_mlx5_pci.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
@@ -1892,16 +1893,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct rte_mlx5_pci_driver mlx5_driver = {
+	.dev_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /**
@@ -1919,7 +1923,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		rte_mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 46e66eb1c..df7b5eb87 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -129,7 +129,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..9bb5dfdbf 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
 CFLAGS += -I$(RTE_SDK)/lib/librte_sched
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..f4175c34e 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index f043166ef..09c9cb935 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,7 +11,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <rte_bus_mlx5_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI bus device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct rte_mlx5_pci_driver mlx5_vdpa_driver = {
+	.dev_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 /**
@@ -849,7 +852,7 @@ RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 	if (mlx5_vdpa_logtype >= 0)
 		rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE);
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0ce8cf541..d7c1d90e4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF)      += -lrte_pmd_memif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD)       += -lrte_pmd_mlx4
 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
 _LDLIBS-y                                   += -lrte_common_mlx5
+_LDLIBS-y                                   += -lrte_bus_mlx5_pci
 endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)  += -lrte_pmd_mlx5_vdpa
-- 
2.26.2

[dpdk-dev] [PATCH v4 09/10] common/mlx5: remove class checks from individual driver

[Parav Pandit]

Now that mlx5_pci bus does the check for enabled classes and performs
probe(), remove() of associated classes, individual class driver
doesn't need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Removed empty line
v1->v2:
 - New patch
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  2 -
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/net/mlx5/linux/mlx5_os.h              |  1 -
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 6 files changed, 52 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ce5e4db1..771e046c1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 62861af05..ca66c4033 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -243,8 +243,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index ae57ebdba..01b4358a0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,8 +1,6 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index b6042b7ef..2bd2b664d 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1308,11 +1308,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_dev_config dev_config;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 695722520..31add3988 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,5 +15,4 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
-
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 09c9cb935..41d69e8a8 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

[dpdk-dev] [PATCH v4 10/10] maintainers: add maintainers for mlx5 pci bus

[Parav Pandit]

Added maintainers for new mlx5 specific mlx5_pci bus.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 MAINTAINERS | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 53a5e9a9e..e3fec55ca 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -517,6 +517,11 @@ Intel FPGA bus
 M: Rosen Xu <rosen.xu@intel.com>
 F: drivers/bus/ifpga/
 
+Melllanox mlx5 PCI bus driver
+M: Parav Pandit <parav@mellaox.com>
+M: Matan Azrad <matan@mellanox.com>
+F: drivers/bus/mlx5_pci
+
 NXP buses
 M: Hemant Agrawal <hemant.agrawal@nxp.com>
 M: Sachin Saxena <sachin.saxena@nxp.com>
-- 
2.26.2

[dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
a new mlx5 PCI bus is created. This bus allows binding multiple
class drivers (such as net, vdpa, regex(future)) to bind to the
mlx5 PCI bus driver.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Introduces new RTE constructor priority for common
initialization
Patch-3 Fixes compilation error
Patch-4 Define mlx5 class as bit fields
Patch-5 Uses new RTE common priority
Patch-6 Adds mlx5 PCI bus
Patch-7 Implements a mlx5 PCI bus driver
Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
API instead of rte PCI bus API
Patch-9 Removed class check code as its already part of the bus now
Patch-10 Add maintainers for the new bus

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       -------------       /
         \______|   mlx5    |_____ /
                |   pci bus |
                -------------   
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD defines its
  specific PCI ID table and mlx5 bus driver probes matching
  class drivers.
- mlx5 pci bus driver is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI bus driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci bus, a common driver is
implemented which exposes class registration API.
However, bus model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common bus driver.
It also eliminates peer blind zone problem as bottom pci
bus layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

In future,
./testpmd -w <PCI BDF>,class=net:regex

Changelog:
v4->v5:
 - Squash the maintainers update path with other patch which adds the
   bus
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (9):
  eal: introduce macros for getting value for bit
  eal: introduce RTE common initialization level
  common/mlx5: fix empty input style in glue wrappers
  common/mlx5: change mlx5 class enum values as bits
  common/mlx5: use common rte priority
  bus/mlx5_pci: add mlx5 PCI bus
  bus/mlx5_pci: register a PCI driver
  bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
  common/mlx5: remove class checks from individual driver

 MAINTAINERS                                   |   5 +
 drivers/bus/Makefile                          |   3 +
 drivers/bus/meson.build                       |   2 +-
 drivers/bus/mlx5_pci/Makefile                 |  40 ++
 drivers/bus/mlx5_pci/meson.build              |  19 +
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 522 ++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  84 +++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/mlx5_common.c             |  39 +-
 drivers/common/mlx5/mlx5_common.h             |   7 +-
 .../common/mlx5/rte_common_mlx5_version.map   |   2 -
 drivers/net/mlx5/Makefile                     |   3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/meson.build                  |   2 +-
 drivers/net/mlx5/mlx5.c                       |  26 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  30 +-
 lib/librte_eal/include/rte_bitops.h           |   8 +
 lib/librte_eal/include/rte_common.h           |   1 +
 mk/rte.app.mk                                 |   1 +
 23 files changed, 730 insertions(+), 85 deletions(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

-- 
2.26.2

[dpdk-dev] [PATCH v5 1/9] eal: introduce macros for getting value for bit

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
Changelog:
v4->v5:
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..ca46a110f 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,14 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+/**
+ * Get the uint64_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 63.
+ */
+#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.26.2

[dpdk-dev] [PATCH v5 2/9] eal: introduce RTE common initialization level

[Parav Pandit]

Currently mlx5_common uses CLASS priority to initialize
common code before initializing the PMD.
However mlx5_common is not really a class, it is the pre-initialization
code needed for the PMDs.

In subsequent patch a needed initialization sequence is:
(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
for class driver initialization.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 lib/librte_eal/include/rte_common.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
index 0843ce69e..d4653ea77 100644
--- a/lib/librte_eal/include/rte_common.h
+++ b/lib/librte_eal/include/rte_common.h
@@ -126,6 +126,7 @@ typedef uint16_t unaligned_uint16_t;
 
 #define RTE_PRIORITY_LOG 101
 #define RTE_PRIORITY_BUS 110
+#define RTE_PRIORITY_COMMON 119
 #define RTE_PRIORITY_CLASS 120
 #define RTE_PRIORITY_LAST 65535
 
-- 
2.26.2

[dpdk-dev] [PATCH v5 3/9] common/mlx5: fix empty input style in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")
Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index 069d8540c..e5e052a6a 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v5 4/9] common/mlx5: change mlx5 class enum values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v1->v2:
 - Rebasd due to removal previous patch
---
 drivers/common/mlx5/mlx5_common.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index da01ffa1c..00ccba622 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -207,9 +208,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT64(0),
+	MLX5_CLASS_VDPA = RTE_BIT64(1),
 };
 
 #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */
-- 
2.26.2

[dpdk-dev] [PATCH v5 5/9] common/mlx5: use common rte priority

[Parav Pandit]

Use RTE_PRIO_COMMON for mlx5 common initialization and use
RTE_PRIO_CLASS for mlx5 net, vdpa PMDs.

This enables to do following initialization sequence.

(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c | 2 +-
 drivers/net/mlx5/mlx5.c           | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 693e2c68c..0ce5e4db1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 /**
  * Initialization routine for run-time dependency on glue library.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+RTE_INIT_PRIO(mlx5_glue_init, COMMON)
 {
 	mlx5_glue_constructor();
 }
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 07c6addd5..bb10c63bb 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1907,7 +1907,7 @@ struct rte_pci_driver mlx5_driver = {
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_pmd_init)
+RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 {
 	/* Initialize driver log type. */
 	mlx5_logtype = rte_log_register("pmd.net.mlx5");
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index dbd36ab0c..f043166ef 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -842,7 +842,7 @@ static struct rte_pci_driver mlx5_vdpa_driver = {
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_vdpa_init)
+RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 {
 	/* Initialize common log type. */
 	mlx5_vdpa_logtype = rte_log_register("pmd.vdpa.mlx5");
-- 
2.26.2

[dpdk-dev] [PATCH v5 6/9] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
pci device.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v4->v5:
 - Merged maintainers update patch with this patch
v2->v3:
 - Addressed comments from Thomas and Asaf
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address comments from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Changed alignedment to mlx5 specific aligment instead of standard
   DPDK
 - Using uint32_t instead of mlx5_class enum
v1->v2:
 - Address comments from Thomas and Gaetan
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
---
 MAINTAINERS                                   |  5 ++
 drivers/bus/meson.build                       |  2 +-
 drivers/bus/mlx5_pci/Makefile                 | 38 +++++++++
 drivers/bus/mlx5_pci/meson.build              | 19 +++++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 84 +++++++++++++++++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
 7 files changed, 166 insertions(+), 1 deletion(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 53a5e9a9e..e3fec55ca 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -517,6 +517,11 @@ Intel FPGA bus
 M: Rosen Xu <rosen.xu@intel.com>
 F: drivers/bus/ifpga/
 
+Melllanox mlx5 PCI bus driver
+M: Parav Pandit <parav@mellaox.com>
+M: Matan Azrad <matan@mellanox.com>
+F: drivers/bus/mlx5_pci
+
 NXP buses
 M: Hemant Agrawal <hemant.agrawal@nxp.com>
 M: Sachin Saxena <sachin.saxena@nxp.com>
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 80de2d91d..b1381838d 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
+drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
 std_deps = ['eal']
 config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
 driver_name_fmt = 'rte_bus_@0@'
diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
new file mode 100644
index 000000000..de4ccd83f
--- /dev/null
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Mellanox Technologies, Ltd
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_bus_mlx5_pci.a
+
+CFLAGS += -O3 -Wall -Wextra
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -Wno-strict-prototypes
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
+CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
+CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+LDLIBS += -lrte_eal
+LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# versioning export map
+EXPORT_MAP := rte_bus_mlx5_pci_version.map
+
+SRCS-y += mlx5_pci_bus.c
+
+# DEBUG which is usually provided on the command-line may enable
+# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
+ifeq ($(DEBUG),1)
+CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
+endif
+
+#
+# Export include files
+#
+SYMLINK-y-include += rte_bus_mlx5_pci.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
new file mode 100644
index 000000000..64a17cbad
--- /dev/null
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Mellanox Technologies Ltd
+
+deps += ['pci', 'bus_pci', 'common_mlx5']
+install_headers('rte_bus_mlx5_pci.h')
+sources = files('mlx5_pci_bus.c')
+
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600'
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
new file mode 100644
index 000000000..66db3c7b0
--- /dev/null
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include "rte_bus_mlx5_pci.h"
+
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
+{
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
new file mode 100644
index 000000000..9f8d22e2b
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _RTE_BUS_MLX5_PCI_H_
+#define _RTE_BUS_MLX5_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Bus Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This bus enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci bus driver.
+ *
+ * -----------    ------------    -----------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5        |
+ * | net pmd |    | vdpa pmd |    | new class pmd |
+ * -----------    ------------    -----------------
+ *      \              |                /
+ *       \             |               /
+ *        \       -------------       /
+ *         \______|   mlx5    |_____ /
+ *                |   pci bus |
+ *                -------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct rte_mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t dev_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a rte_mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
+
+#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \
+	static const char *mlx5_pci_drvinit_fn_ ## nm; \
+	RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \
+	{ \
+		(drv).driver.name = RTE_STR(nm); \
+		rte_mlx5_pci_driver_register(&drv); \
+	} \
+	RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _RTE_BUS_MLX5_PCI_H_ */
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
new file mode 100644
index 000000000..4cfd3db10
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
@@ -0,0 +1,5 @@
+INTERNAL {
+	global:
+
+	rte_mlx5_pci_driver_register;
+};
-- 
2.26.2

[dpdk-dev] [PATCH v5 7/9] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

Create a mlx5 bus driver framework for invoking drivers of
multiple classes who have registered with the mlx5_pci bus
driver.

Validate user class arguments for supported class combinations.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Addressed comments from Asaf
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Addressed comments from Matan.
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block.
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Maintaining class driver list to keep load/unload order symmetric and
   mirror of each other.
 - Deriving drv_flags based on the class drivers
 - Using PCI address comparision helper instead of pointer comparision
 - Fixed alignment for id_table
 - Continue to probe_err if device is already probed
 - Perform dma map on best effort basis for all supported drivers
 - Removed drv_flags check
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Address comments from Thomas and Gaetan
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - USe uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
---
 drivers/bus/mlx5_pci/Makefile       |   2 +
 drivers/bus/mlx5_pci/meson.build    |   2 +-
 drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++
 3 files changed, 511 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
index de4ccd83f..1a005ee32 100644
--- a/drivers/bus/mlx5_pci/Makefile
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -15,7 +15,9 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+CFLAGS += -D_DEFAULT_SOURCE
 LDLIBS += -lrte_eal
+LDLIBS += -lrte_kvargs
 LDLIBS += -lrte_common_mlx5
 LDLIBS += -lrte_pci -lrte_bus_pci
 
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
index 64a17cbad..0532a9dfd 100644
--- a/drivers/bus/mlx5_pci/meson.build
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Mellanox Technologies Ltd
 
-deps += ['pci', 'bus_pci', 'common_mlx5']
+deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
 install_headers('rte_bus_mlx5_pci.h')
 sources = files('mlx5_pci_bus.c')
 
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index 66db3c7b0..6f219730c 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -2,13 +2,521 @@
  * Copyright 2020 Mellanox Technologies, Ltd
  */
 
+#include <stdlib.h>
+#include <rte_malloc.h>
 #include "rte_bus_mlx5_pci.h"
+#include <mlx5_common_utils.h>
 
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of class drivers. */
 static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
 				TAILQ_HEAD_INITIALIZER(drv_list);
 
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int dev_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .dev_class = MLX5_CLASS_NET },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	/* New class combination should be added here.
+	 * For example a new multi class device combination
+	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
+	 */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].dev_class;
+	}
+	return -EINVAL;
+}
+
+static struct rte_mlx5_pci_driver *
+class_driver_get(uint32_t class)
+{
+	struct rte_mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->dev_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
 void
 rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
 {
 	TAILQ_INSERT_TAIL(&drv_list, driver, next);
 }
+
+static bool
+mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = class_driver_get(RTE_BIT64(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT64(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT64(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+class_drivers_probe(struct mlx5_pci_device *dev,
+		    struct rte_pci_driver *pci_drv,
+		    struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->dev_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->dev_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load class driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->dev_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	class_drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple PCI class devices.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		   struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more class devices for a PCI device.
+ *
+ * This function removes all class devices belong to a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_bus_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload class drivers. */
+	ret = class_drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+		     uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class = NULL;
+	struct rte_mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_map) {
+			ret = class->pci_driver.dma_map(pci_dev, addr,
+							iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == class)
+			break;
+		if (device_class_enabled(dev, temp->dev_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+		       uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_unmap) {
+			local_ret = class->pci_driver.dma_unmap(pci_dev, addr,
+								iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5
+ * class drivers.
+ */
+static struct rte_pci_id *mlx5_bus_pci_id_map;
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == mlx5_bus_pci_id_map[i].device_id &&
+		    id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id &&
+		    id->subsystem_vendor_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_vendor_id &&
+		    id->subsystem_device_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(const struct rte_pci_id *id_table, int *next_idx)
+{
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, *next_idx)) {
+			/* New entry; add to the table. */
+			mlx5_bus_pci_id_map[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_build(void)
+{
+	struct rte_mlx5_pci_driver *first_driver;
+	struct rte_mlx5_pci_driver *driver;
+	const struct rte_pci_id *id_table;
+	int num_ids = 0;
+	int i = 0;
+
+	TAILQ_FOREACH(driver, &drv_list, next)
+		num_ids += pci_id_table_size_get(driver->pci_driver.id_table);
+	if (!num_ids)
+		return -ENODEV;
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map));
+	if (!mlx5_bus_pci_id_map)
+		return -ENOMEM;
+	first_driver = TAILQ_FIRST(&drv_list);
+	/* Copy the first class driver's ID table. */
+	for (id_table = first_driver->pci_driver.id_table;
+	     id_table->vendor_id != 0; id_table++, i++)
+		mlx5_bus_pci_id_map[i] = *id_table;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		/* We already added first driver; skip it. */
+		if (driver == first_driver)
+			continue;
+		pci_id_insert(driver->pci_driver.id_table, &i);
+	}
+	mlx5_bus_pci_id_map[i].vendor_id = 0;
+	return 0;
+}
+
+static bool mlx5_bus_registered;
+static struct rte_pci_driver mlx5_bus_driver = {
+	.driver = {
+		.name = "mlx5_bus_pci",
+	},
+	.probe = mlx5_bus_pci_probe,
+	.remove = mlx5_bus_pci_remove,
+	.dma_map = mlx5_bus_pci_dma_map,
+	.dma_unmap = mlx5_bus_pci_dma_unmap,
+};
+
+RTE_INIT(mlx5_bus_pci)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret;
+
+	ret = pci_ids_table_build();
+	if (ret)
+		return;
+	TAILQ_FOREACH(class, &drv_list, next)
+		mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags;
+	mlx5_bus_driver.id_table = mlx5_bus_pci_id_map;
+	rte_pci_register(&mlx5_bus_driver);
+	mlx5_bus_registered = true;
+}
+
+RTE_FINI(mlx5_bus_pci_finish)
+{
+	if (mlx5_bus_registered)
+		rte_pci_unregister(&mlx5_bus_driver);
+	if (mlx5_bus_pci_id_map)
+		free(mlx5_bus_pci_id_map);
+}
+RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
-- 
2.26.2

[dpdk-dev] [PATCH v5 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Parav Pandit]

Enable class driver to match with the mlx5 pci devices.
Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Avoid static table
v1->v2:
 - Migreate API from rte_driver to rte_pci_driver
---
 drivers/bus/Makefile             |  3 +++
 drivers/net/mlx5/Makefile        |  3 ++-
 drivers/net/mlx5/linux/mlx5_os.c |  1 -
 drivers/net/mlx5/linux/mlx5_os.h |  1 +
 drivers/net/mlx5/meson.build     |  2 +-
 drivers/net/mlx5/mlx5.c          | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h          |  1 -
 drivers/vdpa/mlx5/Makefile       |  3 ++-
 drivers/vdpa/mlx5/meson.build    |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c    | 23 +++++++++++++----------
 mk/rte.app.mk                    |  1 +
 11 files changed, 38 insertions(+), 26 deletions(-)

diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile
index cea3b55e6..f96caa5ec 100644
--- a/drivers/bus/Makefile
+++ b/drivers/bus/Makefile
@@ -9,6 +9,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
 DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
+DIRS-y += mlx5_pci
+endif
 DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
 DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus
 
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index a458402dc..1cea7cd07 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_bus_mlx5_pci
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 2dc57b20e..b6042b7ef 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1321,7 +1321,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 31add3988..695722520 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,4 +15,5 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
+
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index e95ce0267..26699a79b 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -8,7 +8,7 @@ if not (is_linux or is_windows)
 	subdir_done()
 endif
 
-deps += ['hash', 'common_mlx5']
+deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
 sources = files(
 	'mlx5.c',
 	'mlx5_ethdev.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index bb10c63bb..2270055fe 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -34,6 +34,7 @@
 #include <rte_spinlock.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
+#include <rte_bus_mlx5_pci.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
@@ -1892,16 +1893,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct rte_mlx5_pci_driver mlx5_driver = {
+	.dev_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /**
@@ -1919,7 +1923,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		rte_mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 46e66eb1c..df7b5eb87 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -129,7 +129,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..9bb5dfdbf 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
 CFLAGS += -I$(RTE_SDK)/lib/librte_sched
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..f4175c34e 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index f043166ef..09c9cb935 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,7 +11,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <rte_bus_mlx5_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI bus device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct rte_mlx5_pci_driver mlx5_vdpa_driver = {
+	.dev_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 /**
@@ -849,7 +852,7 @@ RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 	if (mlx5_vdpa_logtype >= 0)
 		rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE);
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0ce8cf541..d7c1d90e4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF)      += -lrte_pmd_memif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD)       += -lrte_pmd_mlx4
 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
 _LDLIBS-y                                   += -lrte_common_mlx5
+_LDLIBS-y                                   += -lrte_bus_mlx5_pci
 endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)  += -lrte_pmd_mlx5_vdpa
-- 
2.26.2

[dpdk-dev] [PATCH v5 9/9] common/mlx5: remove class checks from individual driver

[Parav Pandit]

Now that mlx5_pci bus does the check for enabled classes and performs
probe(), remove() of associated classes, individual class driver
doesn't need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Removed empty line
v1->v2:
 - New patch
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  2 -
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/net/mlx5/linux/mlx5_os.h              |  1 -
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 6 files changed, 52 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ce5e4db1..771e046c1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 00ccba622..0effaab1d 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -243,8 +243,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index ae57ebdba..01b4358a0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,8 +1,6 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index b6042b7ef..2bd2b664d 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1308,11 +1308,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_dev_config dev_config;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 695722520..31add3988 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,5 +15,4 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
-
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 09c9cb935..41d69e8a8 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

[dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
a new mlx5 PCI bus is created. This bus allows binding multiple
class drivers (such as net, vdpa, regex(future)) to bind to the
mlx5 PCI bus driver.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Introduces new RTE constructor priority for common
initialization
Patch-3 Fixes compilation error
Patch-4 Define mlx5 class as bit fields
Patch-5 Uses new RTE common priority
Patch-6 Adds mlx5 PCI bus
Patch-7 Implements a mlx5 PCI bus driver
Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
API instead of rte PCI bus API
Patch-9 Removed class check code as its already part of the bus now
Patch-10 Add maintainers for the new bus

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       -------------       /
         \______|   mlx5    |_____ /
                |   pci bus |
                -------------   
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD defines its
  specific PCI ID table and mlx5 bus driver probes matching
  class drivers.
- mlx5 pci bus driver is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI bus driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci bus, a common driver is
implemented which exposes class registration API.
However, bus model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common bus driver.
It also eliminates peer blind zone problem as bottom pci
bus layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

In future,
./testpmd -w <PCI BDF>,class=net:regex

Changelog:
v5->v6:
 - Fixed compilation failure in parallel build for shared lib
v4->v5:
 - Squash the maintainers update path with other patch which adds the
   bus
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (9):
  eal: introduce macros for getting value for bit
  eal: introduce RTE common initialization level
  common/mlx5: fix empty input style in glue wrappers
  common/mlx5: change mlx5 class enum values as bits
  common/mlx5: use common rte priority
  bus/mlx5_pci: add mlx5 PCI bus
  bus/mlx5_pci: register a PCI driver
  bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
  common/mlx5: remove class checks from individual driver

 MAINTAINERS                                   |   5 +
 drivers/bus/Makefile                          |   4 +
 drivers/bus/meson.build                       |   2 +-
 drivers/bus/mlx5_pci/Makefile                 |  41 ++
 drivers/bus/mlx5_pci/meson.build              |  19 +
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 522 ++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  84 +++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/mlx5_common.c             |  39 +-
 drivers/common/mlx5/mlx5_common.h             |   7 +-
 .../common/mlx5/rte_common_mlx5_version.map   |   2 -
 drivers/net/mlx5/Makefile                     |   3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/meson.build                  |   2 +-
 drivers/net/mlx5/mlx5.c                       |  26 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  30 +-
 lib/librte_eal/include/rte_bitops.h           |   8 +
 lib/librte_eal/include/rte_common.h           |   1 +
 mk/rte.app.mk                                 |   1 +
 23 files changed, 732 insertions(+), 85 deletions(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

-- 
2.26.2

[dpdk-dev] [PATCH v6 1/9] eal: introduce macros for getting value for bit

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
Changelog:
v4->v5:
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..ca46a110f 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,14 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+/**
+ * Get the uint64_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 63.
+ */
+#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.26.2

[dpdk-dev] [PATCH v6 2/9] eal: introduce RTE common initialization level

[Parav Pandit]

Currently mlx5_common uses CLASS priority to initialize
common code before initializing the PMD.
However mlx5_common is not really a class, it is the pre-initialization
code needed for the PMDs.

In subsequent patch a needed initialization sequence is:
(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
for class driver initialization.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 lib/librte_eal/include/rte_common.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
index 8f487a563..522afe58e 100644
--- a/lib/librte_eal/include/rte_common.h
+++ b/lib/librte_eal/include/rte_common.h
@@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
 
 #define RTE_PRIORITY_LOG 101
 #define RTE_PRIORITY_BUS 110
+#define RTE_PRIORITY_COMMON 119
 #define RTE_PRIORITY_CLASS 120
 #define RTE_PRIORITY_LAST 65535
 
-- 
2.26.2

[dpdk-dev] [PATCH v6 3/9] common/mlx5: fix empty input style in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")
Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index 069d8540c..e5e052a6a 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v6 4/9] common/mlx5: change mlx5 class enum values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v1->v2:
 - Rebasd due to removal previous patch
---
 drivers/common/mlx5/mlx5_common.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 285150705..00e68edbb 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -208,9 +209,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT64(0),
+	MLX5_CLASS_VDPA = RTE_BIT64(1),
 };
 
 #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */
-- 
2.26.2

[dpdk-dev] [PATCH v6 5/9] common/mlx5: use common rte priority

[Parav Pandit]

Use RTE_PRIO_COMMON for mlx5 common initialization and use
RTE_PRIO_CLASS for mlx5 net, vdpa PMDs.

This enables to do following initialization sequence.

(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c | 2 +-
 drivers/net/mlx5/mlx5.c           | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 693e2c68c..0ce5e4db1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 /**
  * Initialization routine for run-time dependency on glue library.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+RTE_INIT_PRIO(mlx5_glue_init, COMMON)
 {
 	mlx5_glue_constructor();
 }
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 0c654ed8b..a6a65957e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1912,7 +1912,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_pmd_init)
+RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 {
 	/* Build the static tables for Verbs conversion. */
 	mlx5_set_ptype_table();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 67e77b11a..205a9e4ad 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -844,7 +844,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_vdpa_init)
+RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 {
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_vdpa_driver);
-- 
2.26.2

[dpdk-dev] [PATCH v6 6/9] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
pci device.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v5->v6:
 - Updated Makefile for parallel shared build support
v4->v5:
 - Merged maintainers update patch with this patch
v2->v3:
 - Addressed comments from Thomas and Asaf
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address comments from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Changed alignedment to mlx5 specific aligment instead of standard
   DPDK
 - Using uint32_t instead of mlx5_class enum
v1->v2:
 - Address comments from Thomas and Gaetan
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
---
 MAINTAINERS                                   |  5 ++
 drivers/bus/Makefile                          |  4 +
 drivers/bus/meson.build                       |  2 +-
 drivers/bus/mlx5_pci/Makefile                 | 39 +++++++++
 drivers/bus/mlx5_pci/meson.build              | 19 +++++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 84 +++++++++++++++++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
 8 files changed, 171 insertions(+), 1 deletion(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 3cd402b34..9e3bd1672 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -525,6 +525,11 @@ Intel FPGA bus
 M: Rosen Xu <rosen.xu@intel.com>
 F: drivers/bus/ifpga/
 
+Melllanox mlx5 PCI bus driver
+M: Parav Pandit <parav@mellaox.com>
+M: Matan Azrad <matan@mellanox.com>
+F: drivers/bus/mlx5_pci
+
 NXP buses
 M: Hemant Agrawal <hemant.agrawal@nxp.com>
 M: Sachin Saxena <sachin.saxena@nxp.com>
diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile
index cea3b55e6..a70f213c1 100644
--- a/drivers/bus/Makefile
+++ b/drivers/bus/Makefile
@@ -9,6 +9,10 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
 DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
+DEPDIRS-mlx5_pci := pci
+DIRS-y += mlx5_pci
+endif
 DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
 DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus
 
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 80de2d91d..b1381838d 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
+drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
 std_deps = ['eal']
 config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
 driver_name_fmt = 'rte_bus_@0@'
diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
new file mode 100644
index 000000000..dd24811a3
--- /dev/null
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -0,0 +1,39 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Mellanox Technologies, Ltd
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_bus_mlx5_pci.a
+
+CFLAGS += -O3 -Wall -Wextra
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -Wno-strict-prototypes
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
+CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
+CFLAGS += -I$(BUILDDIR)/drivers/bus/pci
+CFLAGS += -I$(BUILDDIR)/drivers/bus
+LDLIBS += -lrte_eal
+LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# versioning export map
+EXPORT_MAP := rte_bus_mlx5_pci_version.map
+
+SRCS-y += mlx5_pci_bus.c
+
+# DEBUG which is usually provided on the command-line may enable
+# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
+ifeq ($(DEBUG),1)
+CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
+endif
+
+#
+# Export include files
+#
+SYMLINK-y-include += rte_bus_mlx5_pci.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
new file mode 100644
index 000000000..64a17cbad
--- /dev/null
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Mellanox Technologies Ltd
+
+deps += ['pci', 'bus_pci', 'common_mlx5']
+install_headers('rte_bus_mlx5_pci.h')
+sources = files('mlx5_pci_bus.c')
+
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600'
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
new file mode 100644
index 000000000..66db3c7b0
--- /dev/null
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include "rte_bus_mlx5_pci.h"
+
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
+{
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
new file mode 100644
index 000000000..9f8d22e2b
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _RTE_BUS_MLX5_PCI_H_
+#define _RTE_BUS_MLX5_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Bus Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This bus enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci bus driver.
+ *
+ * -----------    ------------    -----------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5        |
+ * | net pmd |    | vdpa pmd |    | new class pmd |
+ * -----------    ------------    -----------------
+ *      \              |                /
+ *       \             |               /
+ *        \       -------------       /
+ *         \______|   mlx5    |_____ /
+ *                |   pci bus |
+ *                -------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct rte_mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t dev_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a rte_mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
+
+#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \
+	static const char *mlx5_pci_drvinit_fn_ ## nm; \
+	RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \
+	{ \
+		(drv).driver.name = RTE_STR(nm); \
+		rte_mlx5_pci_driver_register(&drv); \
+	} \
+	RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _RTE_BUS_MLX5_PCI_H_ */
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
new file mode 100644
index 000000000..4cfd3db10
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
@@ -0,0 +1,5 @@
+INTERNAL {
+	global:
+
+	rte_mlx5_pci_driver_register;
+};
-- 
2.26.2

[dpdk-dev] [PATCH v6 7/9] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

Create a mlx5 bus driver framework for invoking drivers of
multiple classes who have registered with the mlx5_pci bus
driver.

Validate user class arguments for supported class combinations.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Addressed comments from Asaf
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Addressed comments from Matan.
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block.
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Maintaining class driver list to keep load/unload order symmetric and
   mirror of each other.
 - Deriving drv_flags based on the class drivers
 - Using PCI address comparision helper instead of pointer comparision
 - Fixed alignment for id_table
 - Continue to probe_err if device is already probed
 - Perform dma map on best effort basis for all supported drivers
 - Removed drv_flags check
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Address comments from Thomas and Gaetan
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - USe uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
---
 drivers/bus/mlx5_pci/Makefile       |   2 +
 drivers/bus/mlx5_pci/meson.build    |   2 +-
 drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++
 3 files changed, 511 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
index dd24811a3..ee1af4dc6 100644
--- a/drivers/bus/mlx5_pci/Makefile
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -16,7 +16,9 @@ CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/bus/pci
 CFLAGS += -I$(BUILDDIR)/drivers/bus
+CFLAGS += -D_DEFAULT_SOURCE
 LDLIBS += -lrte_eal
+LDLIBS += -lrte_kvargs
 LDLIBS += -lrte_common_mlx5
 LDLIBS += -lrte_pci -lrte_bus_pci
 
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
index 64a17cbad..0532a9dfd 100644
--- a/drivers/bus/mlx5_pci/meson.build
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Mellanox Technologies Ltd
 
-deps += ['pci', 'bus_pci', 'common_mlx5']
+deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
 install_headers('rte_bus_mlx5_pci.h')
 sources = files('mlx5_pci_bus.c')
 
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index 66db3c7b0..6f219730c 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -2,13 +2,521 @@
  * Copyright 2020 Mellanox Technologies, Ltd
  */
 
+#include <stdlib.h>
+#include <rte_malloc.h>
 #include "rte_bus_mlx5_pci.h"
+#include <mlx5_common_utils.h>
 
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of class drivers. */
 static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
 				TAILQ_HEAD_INITIALIZER(drv_list);
 
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int dev_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .dev_class = MLX5_CLASS_NET },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	/* New class combination should be added here.
+	 * For example a new multi class device combination
+	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
+	 */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].dev_class;
+	}
+	return -EINVAL;
+}
+
+static struct rte_mlx5_pci_driver *
+class_driver_get(uint32_t class)
+{
+	struct rte_mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->dev_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
 void
 rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
 {
 	TAILQ_INSERT_TAIL(&drv_list, driver, next);
 }
+
+static bool
+mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = class_driver_get(RTE_BIT64(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT64(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT64(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+class_drivers_probe(struct mlx5_pci_device *dev,
+		    struct rte_pci_driver *pci_drv,
+		    struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->dev_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->dev_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load class driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->dev_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	class_drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple PCI class devices.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		   struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more class devices for a PCI device.
+ *
+ * This function removes all class devices belong to a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_bus_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload class drivers. */
+	ret = class_drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+		     uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class = NULL;
+	struct rte_mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_map) {
+			ret = class->pci_driver.dma_map(pci_dev, addr,
+							iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == class)
+			break;
+		if (device_class_enabled(dev, temp->dev_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+		       uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_unmap) {
+			local_ret = class->pci_driver.dma_unmap(pci_dev, addr,
+								iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5
+ * class drivers.
+ */
+static struct rte_pci_id *mlx5_bus_pci_id_map;
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == mlx5_bus_pci_id_map[i].device_id &&
+		    id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id &&
+		    id->subsystem_vendor_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_vendor_id &&
+		    id->subsystem_device_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(const struct rte_pci_id *id_table, int *next_idx)
+{
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, *next_idx)) {
+			/* New entry; add to the table. */
+			mlx5_bus_pci_id_map[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_build(void)
+{
+	struct rte_mlx5_pci_driver *first_driver;
+	struct rte_mlx5_pci_driver *driver;
+	const struct rte_pci_id *id_table;
+	int num_ids = 0;
+	int i = 0;
+
+	TAILQ_FOREACH(driver, &drv_list, next)
+		num_ids += pci_id_table_size_get(driver->pci_driver.id_table);
+	if (!num_ids)
+		return -ENODEV;
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map));
+	if (!mlx5_bus_pci_id_map)
+		return -ENOMEM;
+	first_driver = TAILQ_FIRST(&drv_list);
+	/* Copy the first class driver's ID table. */
+	for (id_table = first_driver->pci_driver.id_table;
+	     id_table->vendor_id != 0; id_table++, i++)
+		mlx5_bus_pci_id_map[i] = *id_table;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		/* We already added first driver; skip it. */
+		if (driver == first_driver)
+			continue;
+		pci_id_insert(driver->pci_driver.id_table, &i);
+	}
+	mlx5_bus_pci_id_map[i].vendor_id = 0;
+	return 0;
+}
+
+static bool mlx5_bus_registered;
+static struct rte_pci_driver mlx5_bus_driver = {
+	.driver = {
+		.name = "mlx5_bus_pci",
+	},
+	.probe = mlx5_bus_pci_probe,
+	.remove = mlx5_bus_pci_remove,
+	.dma_map = mlx5_bus_pci_dma_map,
+	.dma_unmap = mlx5_bus_pci_dma_unmap,
+};
+
+RTE_INIT(mlx5_bus_pci)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret;
+
+	ret = pci_ids_table_build();
+	if (ret)
+		return;
+	TAILQ_FOREACH(class, &drv_list, next)
+		mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags;
+	mlx5_bus_driver.id_table = mlx5_bus_pci_id_map;
+	rte_pci_register(&mlx5_bus_driver);
+	mlx5_bus_registered = true;
+}
+
+RTE_FINI(mlx5_bus_pci_finish)
+{
+	if (mlx5_bus_registered)
+		rte_pci_unregister(&mlx5_bus_driver);
+	if (mlx5_bus_pci_id_map)
+		free(mlx5_bus_pci_id_map);
+}
+RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
-- 
2.26.2

[dpdk-dev] [PATCH v6 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Parav Pandit]

Enable class driver to match with the mlx5 pci devices.
Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Avoid static table
v1->v2:
 - Migreate API from rte_driver to rte_pci_driver
---
 drivers/net/mlx5/Makefile        |  3 ++-
 drivers/net/mlx5/linux/mlx5_os.c |  1 -
 drivers/net/mlx5/linux/mlx5_os.h |  1 +
 drivers/net/mlx5/meson.build     |  2 +-
 drivers/net/mlx5/mlx5.c          | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h          |  1 -
 drivers/vdpa/mlx5/Makefile       |  3 ++-
 drivers/vdpa/mlx5/meson.build    |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c    | 23 +++++++++++++----------
 mk/rte.app.mk                    |  1 +
 10 files changed, 35 insertions(+), 26 deletions(-)

diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index a458402dc..1cea7cd07 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_bus_mlx5_pci
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 2dc57b20e..b6042b7ef 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1321,7 +1321,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 31add3988..695722520 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,4 +15,5 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
+
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index e95ce0267..26699a79b 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -8,7 +8,7 @@ if not (is_linux or is_windows)
 	subdir_done()
 endif
 
-deps += ['hash', 'common_mlx5']
+deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
 sources = files(
 	'mlx5.c',
 	'mlx5_ethdev.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index a6a65957e..8f853844b 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -34,6 +34,7 @@
 #include <rte_spinlock.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
+#include <rte_bus_mlx5_pci.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
@@ -1894,16 +1895,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct rte_mlx5_pci_driver mlx5_driver = {
+	.dev_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /* Initialize driver log type. */
@@ -1919,7 +1923,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		rte_mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 46e66eb1c..df7b5eb87 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -129,7 +129,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..9bb5dfdbf 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
 CFLAGS += -I$(RTE_SDK)/lib/librte_sched
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..f4175c34e 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 205a9e4ad..5d247e0a1 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,7 +11,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <rte_bus_mlx5_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI bus device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct rte_mlx5_pci_driver mlx5_vdpa_driver = {
+	.dev_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
@@ -847,7 +850,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
 RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 {
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0ce8cf541..d7c1d90e4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF)      += -lrte_pmd_memif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD)       += -lrte_pmd_mlx4
 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
 _LDLIBS-y                                   += -lrte_common_mlx5
+_LDLIBS-y                                   += -lrte_bus_mlx5_pci
 endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)  += -lrte_pmd_mlx5_vdpa
-- 
2.26.2

[dpdk-dev] [PATCH v6 9/9] common/mlx5: remove class checks from individual driver

[Parav Pandit]

Now that mlx5_pci bus does the check for enabled classes and performs
probe(), remove() of associated classes, individual class driver
does not need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Removed empty line
v1->v2:
 - New patch
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  2 -
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/net/mlx5/linux/mlx5_os.h              |  1 -
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 6 files changed, 52 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ce5e4db1..771e046c1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 00e68edbb..5b9b7bd5a 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -244,8 +244,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index ae57ebdba..01b4358a0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,8 +1,6 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index b6042b7ef..2bd2b664d 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1308,11 +1308,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_dev_config dev_config;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 695722520..31add3988 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,5 +15,4 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
-
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 5d247e0a1..70692ea1d 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

[dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
a new mlx5 PCI bus is created. This bus allows binding multiple
class drivers (such as net, vdpa, regex(future)) to bind to the
mlx5 PCI bus driver.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Introduces new RTE constructor priority for common
initialization
Patch-3 Fixes compilation error
Patch-4 Define mlx5 class as bit fields
Patch-5 Uses new RTE common priority
Patch-6 Adds mlx5 PCI bus
Patch-7 Implements a mlx5 PCI bus driver
Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
API instead of rte PCI bus API
Patch-9 Removed class check code as its already part of the bus now

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       -------------       /
         \______|   mlx5    |_____ /
                |   pci bus |
                -------------   
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD defines its
  specific PCI ID table and mlx5 bus driver probes matching
  class drivers.
- mlx5 pci bus driver is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI bus driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci bus, a common driver is
implemented which exposes class registration API.
However, bus model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common bus driver.
It also eliminates peer blind zone problem as bottom pci
bus layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

In future,
./testpmd -w <PCI BDF>,class=net:regex

Changelog:
v6->v7:
 - Updated release notes
v5->v6:
 - Fixed compilation failure in parallel build for shared lib
v4->v5:
 - Squash the maintainers update path with other patch which adds the
   bus
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (9):
  eal: introduce macros for getting value for bit
  eal: introduce RTE common initialization level
  common/mlx5: fix empty input style in glue wrappers
  common/mlx5: change mlx5 class enum values as bits
  common/mlx5: use common rte priority
  bus/mlx5_pci: add mlx5 PCI bus
  bus/mlx5_pci: register a PCI driver
  bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
  common/mlx5: remove class checks from individual driver

 MAINTAINERS                                   |   5 +
 doc/guides/rel_notes/release_20_08.rst        |   5 +
 drivers/bus/Makefile                          |   4 +
 drivers/bus/meson.build                       |   2 +-
 drivers/bus/mlx5_pci/Makefile                 |  41 ++
 drivers/bus/mlx5_pci/meson.build              |  19 +
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 522 ++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  84 +++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/mlx5_common.c             |  39 +-
 drivers/common/mlx5/mlx5_common.h             |   7 +-
 .../common/mlx5/rte_common_mlx5_version.map   |   2 -
 drivers/net/mlx5/Makefile                     |   3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/meson.build                  |   2 +-
 drivers/net/mlx5/mlx5.c                       |  26 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  30 +-
 lib/librte_eal/include/rte_bitops.h           |   8 +
 lib/librte_eal/include/rte_common.h           |   1 +
 mk/rte.app.mk                                 |   1 +
 24 files changed, 737 insertions(+), 85 deletions(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

-- 
2.26.2

[dpdk-dev] [PATCH v7 1/9] eal: introduce macros for getting value for bit

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
Changelog:
v4->v5:
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..ca46a110f 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,14 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+/**
+ * Get the uint64_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 63.
+ */
+#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.26.2

[dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Parav Pandit]

Currently mlx5_common uses CLASS priority to initialize
common code before initializing the PMD.
However mlx5_common is not really a class, it is the pre-initialization
code needed for the PMDs.

In subsequent patch a needed initialization sequence is:
(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
for class driver initialization.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 lib/librte_eal/include/rte_common.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
index 8f487a563..522afe58e 100644
--- a/lib/librte_eal/include/rte_common.h
+++ b/lib/librte_eal/include/rte_common.h
@@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
 
 #define RTE_PRIORITY_LOG 101
 #define RTE_PRIORITY_BUS 110
+#define RTE_PRIORITY_COMMON 119
 #define RTE_PRIORITY_CLASS 120
 #define RTE_PRIORITY_LAST 65535
 
-- 
2.26.2

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Ferruh Yigit]

On 7/17/2020 2:49 PM, Parav Pandit wrote:
> Currently mlx5_common uses CLASS priority to initialize
> common code before initializing the PMD.
> However mlx5_common is not really a class, it is the pre-initialization
> code needed for the PMDs.
> 
> In subsequent patch a needed initialization sequence is:
> (a) Initialize bus (say pci)
> (b) Initialize common code of a driver (mlx5_common)
> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
> Information registered by these PMDs is used by mlx5_bus_pci PMD.
> This mlx5 class PMDs should not confused with rte_class.
> (d) Register mlx5 PCI bus PMD
> 
> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
> can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
> for class driver initialization.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> Acked-by: Matan Azrad <matan@mellanox.com>
> ---
> Changelog:
> v2->v3:
>  - new patch
> ---
>  lib/librte_eal/include/rte_common.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
> index 8f487a563..522afe58e 100644
> --- a/lib/librte_eal/include/rte_common.h
> +++ b/lib/librte_eal/include/rte_common.h
> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
>  
>  #define RTE_PRIORITY_LOG 101
>  #define RTE_PRIORITY_BUS 110
> +#define RTE_PRIORITY_COMMON 119
>  #define RTE_PRIORITY_CLASS 120
>  #define RTE_PRIORITY_LAST 65535
>  
> 

I guess the name "common" selected because of the intention to use it by the
common piece of the driver, but only from eal perspective the name
"PRIORITY_COMMON" looks so vague, it doesn't describe any purpose.

Also the value doesn't leave any gap between the class priority, what else can
be needed in the future in between, right?


@Thomas, @David, I am reluctant to get this eal change through the next-net, can
you please review/ack it first?

Thanks,
ferruh

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Thomas Monjalon]

20/07/2020 18:21, Ferruh Yigit:
> On 7/17/2020 2:49 PM, Parav Pandit wrote:
> > Currently mlx5_common uses CLASS priority to initialize
> > common code before initializing the PMD.
> > However mlx5_common is not really a class, it is the pre-initialization
> > code needed for the PMDs.
> > 
> > In subsequent patch a needed initialization sequence is:
> > (a) Initialize bus (say pci)
> > (b) Initialize common code of a driver (mlx5_common)
> > (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
> > Information registered by these PMDs is used by mlx5_bus_pci PMD.
> > This mlx5 class PMDs should not confused with rte_class.
> > (d) Register mlx5 PCI bus PMD
> > 
> > Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
> > can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
> > for class driver initialization.
> > 
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > Acked-by: Matan Azrad <matan@mellanox.com>
> > ---
> > Changelog:
> > v2->v3:
> >  - new patch
> > ---
> >  lib/librte_eal/include/rte_common.h | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
> > index 8f487a563..522afe58e 100644
> > --- a/lib/librte_eal/include/rte_common.h
> > +++ b/lib/librte_eal/include/rte_common.h
> > @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
> >  
> >  #define RTE_PRIORITY_LOG 101
> >  #define RTE_PRIORITY_BUS 110
> > +#define RTE_PRIORITY_COMMON 119
> >  #define RTE_PRIORITY_CLASS 120
> >  #define RTE_PRIORITY_LAST 65535
> >  
> > 
> 
> I guess the name "common" selected because of the intention to use it by the
> common piece of the driver, but only from eal perspective the name
> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose.

You're right.

> Also the value doesn't leave any gap between the class priority, what else can
> be needed in the future in between, right?

And we can imagine a bus requiring a common lib
to be initialized before.

> @Thomas, @David, I am reluctant to get this eal change through the next-net, can
> you please review/ack it first?

What about skipping this patch and using "RTE_PRIORITY_CLASS - 1"
in the code?

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Ferruh Yigit]

On 7/20/2020 5:48 PM, Thomas Monjalon wrote:
> 20/07/2020 18:21, Ferruh Yigit:
>> On 7/17/2020 2:49 PM, Parav Pandit wrote:
>>> Currently mlx5_common uses CLASS priority to initialize
>>> common code before initializing the PMD.
>>> However mlx5_common is not really a class, it is the pre-initialization
>>> code needed for the PMDs.
>>>
>>> In subsequent patch a needed initialization sequence is:
>>> (a) Initialize bus (say pci)
>>> (b) Initialize common code of a driver (mlx5_common)
>>> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
>>> Information registered by these PMDs is used by mlx5_bus_pci PMD.
>>> This mlx5 class PMDs should not confused with rte_class.
>>> (d) Register mlx5 PCI bus PMD
>>>
>>> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
>>> can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
>>> for class driver initialization.
>>>
>>> Signed-off-by: Parav Pandit <parav@mellanox.com>
>>> Acked-by: Matan Azrad <matan@mellanox.com>
>>> ---
>>> Changelog:
>>> v2->v3:
>>>  - new patch
>>> ---
>>>  lib/librte_eal/include/rte_common.h | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
>>> index 8f487a563..522afe58e 100644
>>> --- a/lib/librte_eal/include/rte_common.h
>>> +++ b/lib/librte_eal/include/rte_common.h
>>> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
>>>  
>>>  #define RTE_PRIORITY_LOG 101
>>>  #define RTE_PRIORITY_BUS 110
>>> +#define RTE_PRIORITY_COMMON 119
>>>  #define RTE_PRIORITY_CLASS 120
>>>  #define RTE_PRIORITY_LAST 65535
>>>  
>>>
>>
>> I guess the name "common" selected because of the intention to use it by the
>> common piece of the driver, but only from eal perspective the name
>> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose.
> 
> You're right.
> 
>> Also the value doesn't leave any gap between the class priority, what else can
>> be needed in the future in between, right?
> 
> And we can imagine a bus requiring a common lib
> to be initialized before.
> 
>> @Thomas, @David, I am reluctant to get this eal change through the next-net, can
>> you please review/ack it first?
> 
> What about skipping this patch and using "RTE_PRIORITY_CLASS - 1"
> in the code?
> 

For now I think it is OK, in the future if more priority dependency involved we
can define the macro.

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Ori Kam]

Hi

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> On 7/20/2020 5:48 PM, Thomas Monjalon wrote:
> > 20/07/2020 18:21, Ferruh Yigit:
> >> On 7/17/2020 2:49 PM, Parav Pandit wrote:
> >>> Currently mlx5_common uses CLASS priority to initialize
> >>> common code before initializing the PMD.
> >>> However mlx5_common is not really a class, it is the pre-initialization
> >>> code needed for the PMDs.
> >>>
> >>> In subsequent patch a needed initialization sequence is:
> >>> (a) Initialize bus (say pci)
> >>> (b) Initialize common code of a driver (mlx5_common)
> >>> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
> >>> Information registered by these PMDs is used by mlx5_bus_pci PMD.
> >>> This mlx5 class PMDs should not confused with rte_class.
> >>> (d) Register mlx5 PCI bus PMD
> >>>
> >>> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
> >>> can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
> >>> for class driver initialization.
> >>>
> >>> Signed-off-by: Parav Pandit <parav@mellanox.com>
> >>> Acked-by: Matan Azrad <matan@mellanox.com>
> >>> ---
> >>> Changelog:
> >>> v2->v3:
> >>>  - new patch
> >>> ---
> >>>  lib/librte_eal/include/rte_common.h | 1 +
> >>>  1 file changed, 1 insertion(+)
> >>>
> >>> diff --git a/lib/librte_eal/include/rte_common.h
> b/lib/librte_eal/include/rte_common.h
> >>> index 8f487a563..522afe58e 100644
> >>> --- a/lib/librte_eal/include/rte_common.h
> >>> +++ b/lib/librte_eal/include/rte_common.h
> >>> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
> >>>
> >>>  #define RTE_PRIORITY_LOG 101
> >>>  #define RTE_PRIORITY_BUS 110
> >>> +#define RTE_PRIORITY_COMMON 119
> >>>  #define RTE_PRIORITY_CLASS 120
> >>>  #define RTE_PRIORITY_LAST 65535
> >>>
> >>>
> >>
> >> I guess the name "common" selected because of the intention to use it by
> the
> >> common piece of the driver, but only from eal perspective the name
> >> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose.
> >
> > You're right.
> >
> >> Also the value doesn't leave any gap between the class priority, what else
> can
> >> be needed in the future in between, right?
> >
> > And we can imagine a bus requiring a common lib
> > to be initialized before.
> >
> >> @Thomas, @David, I am reluctant to get this eal change through the next-
> net, can
> >> you please review/ack it first?
> >
> > What about skipping this patch and using "RTE_PRIORITY_CLASS - 1"
> > in the code?
> >
> 
> For now I think it is OK, in the future if more priority dependency involved we
> can define the macro.
> 
I'm concerned what if someone else will add priority there may be conflict and.
Also using -1 means that no one knows that there is use in such priority.
What about setting the value to 115?

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Ferruh Yigit]

On 7/20/2020 6:26 PM, Ori Kam wrote:
> Hi
> 
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>> On 7/20/2020 5:48 PM, Thomas Monjalon wrote:
>>> 20/07/2020 18:21, Ferruh Yigit:
>>>> On 7/17/2020 2:49 PM, Parav Pandit wrote:
>>>>> Currently mlx5_common uses CLASS priority to initialize
>>>>> common code before initializing the PMD.
>>>>> However mlx5_common is not really a class, it is the pre-initialization
>>>>> code needed for the PMDs.
>>>>>
>>>>> In subsequent patch a needed initialization sequence is:
>>>>> (a) Initialize bus (say pci)
>>>>> (b) Initialize common code of a driver (mlx5_common)
>>>>> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
>>>>> Information registered by these PMDs is used by mlx5_bus_pci PMD.
>>>>> This mlx5 class PMDs should not confused with rte_class.
>>>>> (d) Register mlx5 PCI bus PMD
>>>>>
>>>>> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
>>>>> can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
>>>>> for class driver initialization.
>>>>>
>>>>> Signed-off-by: Parav Pandit <parav@mellanox.com>
>>>>> Acked-by: Matan Azrad <matan@mellanox.com>
>>>>> ---
>>>>> Changelog:
>>>>> v2->v3:
>>>>>  - new patch
>>>>> ---
>>>>>  lib/librte_eal/include/rte_common.h | 1 +
>>>>>  1 file changed, 1 insertion(+)
>>>>>
>>>>> diff --git a/lib/librte_eal/include/rte_common.h
>> b/lib/librte_eal/include/rte_common.h
>>>>> index 8f487a563..522afe58e 100644
>>>>> --- a/lib/librte_eal/include/rte_common.h
>>>>> +++ b/lib/librte_eal/include/rte_common.h
>>>>> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
>>>>>
>>>>>  #define RTE_PRIORITY_LOG 101
>>>>>  #define RTE_PRIORITY_BUS 110
>>>>> +#define RTE_PRIORITY_COMMON 119
>>>>>  #define RTE_PRIORITY_CLASS 120
>>>>>  #define RTE_PRIORITY_LAST 65535
>>>>>
>>>>>
>>>>
>>>> I guess the name "common" selected because of the intention to use it by
>> the
>>>> common piece of the driver, but only from eal perspective the name
>>>> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose.
>>>
>>> You're right.
>>>
>>>> Also the value doesn't leave any gap between the class priority, what else
>> can
>>>> be needed in the future in between, right?
>>>
>>> And we can imagine a bus requiring a common lib
>>> to be initialized before.
>>>
>>>> @Thomas, @David, I am reluctant to get this eal change through the next-
>> net, can
>>>> you please review/ack it first?
>>>
>>> What about skipping this patch and using "RTE_PRIORITY_CLASS - 1"
>>> in the code?
>>>
>>
>> For now I think it is OK, in the future if more priority dependency involved we
>> can define the macro.
>>
> I'm concerned what if someone else will add priority there may be conflict and.
> Also using -1 means that no one knows that there is use in such priority.

Is the new constructor priority level a common need, or just specific to this
"mlx5 pci" bus usage?
I understand the need of new constructor priority level, but it seems it is not
clear enough to define it as a generic level, that is why I believe this can be
local to PMD for now, otherwise your concerns looks valid.

Also I am thinking if the multi class support can be done as generic bus
feature, instead of defining a new PMD specific bus for it, but I am aware it is
too late for this question.

> What about setting the value to 115?
> 

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Ori Kam]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> 
> On 7/20/2020 6:26 PM, Ori Kam wrote:
> > Hi
> >
> >> -----Original Message-----
> >> From: Ferruh Yigit <ferruh.yigit@intel.com>
> >> On 7/20/2020 5:48 PM, Thomas Monjalon wrote:
> >>> 20/07/2020 18:21, Ferruh Yigit:
> >>>> On 7/17/2020 2:49 PM, Parav Pandit wrote:
> >>>>> Currently mlx5_common uses CLASS priority to initialize
> >>>>> common code before initializing the PMD.
> >>>>> However mlx5_common is not really a class, it is the pre-initialization
> >>>>> code needed for the PMDs.
> >>>>>
> >>>>> In subsequent patch a needed initialization sequence is:
> >>>>> (a) Initialize bus (say pci)
> >>>>> (b) Initialize common code of a driver (mlx5_common)
> >>>>> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
> >>>>> Information registered by these PMDs is used by mlx5_bus_pci PMD.
> >>>>> This mlx5 class PMDs should not confused with rte_class.
> >>>>> (d) Register mlx5 PCI bus PMD
> >>>>>
> >>>>> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
> >>>>> can be used for common initialization and RTE_PRIO_CLASS by mlx5
> PMDs
> >>>>> for class driver initialization.
> >>>>>
> >>>>> Signed-off-by: Parav Pandit <parav@mellanox.com>
> >>>>> Acked-by: Matan Azrad <matan@mellanox.com>
> >>>>> ---
> >>>>> Changelog:
> >>>>> v2->v3:
> >>>>>  - new patch
> >>>>> ---
> >>>>>  lib/librte_eal/include/rte_common.h | 1 +
> >>>>>  1 file changed, 1 insertion(+)
> >>>>>
> >>>>> diff --git a/lib/librte_eal/include/rte_common.h
> >> b/lib/librte_eal/include/rte_common.h
> >>>>> index 8f487a563..522afe58e 100644
> >>>>> --- a/lib/librte_eal/include/rte_common.h
> >>>>> +++ b/lib/librte_eal/include/rte_common.h
> >>>>> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
> >>>>>
> >>>>>  #define RTE_PRIORITY_LOG 101
> >>>>>  #define RTE_PRIORITY_BUS 110
> >>>>> +#define RTE_PRIORITY_COMMON 119
> >>>>>  #define RTE_PRIORITY_CLASS 120
> >>>>>  #define RTE_PRIORITY_LAST 65535
> >>>>>
> >>>>>
> >>>>
> >>>> I guess the name "common" selected because of the intention to use it by
> >> the
> >>>> common piece of the driver, but only from eal perspective the name
> >>>> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose.
> >>>
> >>> You're right.
> >>>
> >>>> Also the value doesn't leave any gap between the class priority, what else
> >> can
> >>>> be needed in the future in between, right?
> >>>
> >>> And we can imagine a bus requiring a common lib
> >>> to be initialized before.
> >>>
> >>>> @Thomas, @David, I am reluctant to get this eal change through the
> next-
> >> net, can
> >>>> you please review/ack it first?
> >>>
> >>> What about skipping this patch and using "RTE_PRIORITY_CLASS - 1"
> >>> in the code?
> >>>
> >>
> >> For now I think it is OK, in the future if more priority dependency involved
> we
> >> can define the macro.
> >>
> > I'm concerned what if someone else will add priority there may be conflict
> and.
> > Also using -1 means that no one knows that there is use in such priority.
> 
> Is the new constructor priority level a common need, or just specific to this
> "mlx5 pci" bus usage?

I think it can be used by any other vendor that needs a bus for his own PMDs.

> I understand the need of new constructor priority level, but it seems it is not
> clear enough to define it as a generic level, that is why I believe this can be
> local to PMD for now, otherwise your concerns looks valid.
> 
If I understand it can be local to the bus, it is not exposed to the PMD.
But like I said above this may be used by other vendors that wish to combine
some of their devices.

> Also I am thinking if the multi class support can be done as generic bus
> feature, instead of defining a new PMD specific bus for it, but I am aware it is
> too late for this question.
> 
> > What about setting the value to 115?
> >

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[David Marchand]

On Mon, Jul 20, 2020 at 6:48 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 20/07/2020 18:21, Ferruh Yigit:
> > On 7/17/2020 2:49 PM, Parav Pandit wrote:
> > > Currently mlx5_common uses CLASS priority to initialize
> > > common code before initializing the PMD.
> > > However mlx5_common is not really a class, it is the pre-initialization
> > > code needed for the PMDs.
> > >
> > > In subsequent patch a needed initialization sequence is:
> > > (a) Initialize bus (say pci)
> > > (b) Initialize common code of a driver (mlx5_common)
> > > (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
> > > Information registered by these PMDs is used by mlx5_bus_pci PMD.
> > > This mlx5 class PMDs should not confused with rte_class.
> > > (d) Register mlx5 PCI bus PMD
> > >
> > > Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
> > > can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs
> > > for class driver initialization.
> > >
> > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > Acked-by: Matan Azrad <matan@mellanox.com>
> > > ---
> > > Changelog:
> > > v2->v3:
> > >  - new patch
> > > ---
> > >  lib/librte_eal/include/rte_common.h | 1 +
> > >  1 file changed, 1 insertion(+)
> > >
> > > diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h
> > > index 8f487a563..522afe58e 100644
> > > --- a/lib/librte_eal/include/rte_common.h
> > > +++ b/lib/librte_eal/include/rte_common.h
> > > @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
> > >
> > >  #define RTE_PRIORITY_LOG 101
> > >  #define RTE_PRIORITY_BUS 110
> > > +#define RTE_PRIORITY_COMMON 119
> > >  #define RTE_PRIORITY_CLASS 120
> > >  #define RTE_PRIORITY_LAST 65535
> > >
> > >
> >
> > I guess the name "common" selected because of the intention to use it by the
> > common piece of the driver, but only from eal perspective the name
> > "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose.
>
> You're right.
>
> > Also the value doesn't leave any gap between the class priority, what else can
> > be needed in the future in between, right?
>
> And we can imagine a bus requiring a common lib
> to be initialized before.
>
> > @Thomas, @David, I am reluctant to get this eal change through the next-net, can
> > you please review/ack it first?
>
> What about skipping this patch and using "RTE_PRIORITY_CLASS - 1"
> in the code?

net and vdpa code expect the common code being initialised.
It is a dependency internal to mlx5 drivers, I see nothing generic.

The common driver can provide a init function called by net and vdpa
drivers in constructor context (not that I like calling complex init,
but it should be equivalent to current state).
It expresses a clear dependency and there is no constructor semantic
to invent, nor runtime breakage possible because someone tweaks the
priority order in the future.

There is also some hack about a haswell/broadwell detection in LOG
priority that makes no sense.

I compile-tested following:

$ git diff next-net-mlx/master -- lib/librte_eal/ drivers/*/mlx5/
diff --git a/drivers/common/mlx5/mlx5_common.c
b/drivers/common/mlx5/mlx5_common.c
index 79cd5ba344..3fb9a8ab89 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -49,14 +49,6 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
                rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE);
 }

-/**
- * Initialization routine for run-time dependency on glue library.
- */
-RTE_INIT_PRIO(mlx5_glue_init, COMMON)
-{
-       mlx5_glue_constructor();
-}
-
 /**
  * This function is responsible of initializing the variable
  *  haswell_broadwell_cpu by checking if the cpu is intel
@@ -67,7 +59,7 @@ RTE_INIT_PRIO(mlx5_glue_init, COMMON)
  *  if the cpu is haswell or broadwell the variable will be set to 1
  *  otherwise it will be 0.
  */
-RTE_INIT_PRIO(mlx5_is_haswell_broadwell_cpu, LOG)
+static void mlx5_is_haswell_broadwell_cpu(void)
 {
 #ifdef RTE_ARCH_X86_64
        unsigned int broadwell_models[4] = {0x3d, 0x47, 0x4F, 0x56};
@@ -114,6 +106,21 @@ RTE_INIT_PRIO(mlx5_is_haswell_broadwell_cpu, LOG)
        haswell_broadwell_cpu = 0;
 }

+/**
+ * Initialization routine for run-time dependency on glue library.
+ */
+void mlx5_common_init(void)
+{
+       static bool init_once = false;
+
+       if (init_once)
+               return;
+
+       mlx5_glue_constructor();
+       mlx5_is_haswell_broadwell_cpu();
+       init_once = true;
+}
+
 /**
  * Allocate page of door-bells and register it using DevX API.
  *
diff --git a/drivers/common/mlx5/mlx5_common.h
b/drivers/common/mlx5/mlx5_common.h
index 5b9b7bd5a9..30961bc8cc 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -254,6 +254,10 @@ int64_t mlx5_get_dbr(void *ctx,  struct
mlx5_dbr_page_list *head,
 __rte_internal
 int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id,
                         uint64_t offset);
+
+__rte_internal
+void mlx5_common_init(void);
+
 extern uint8_t haswell_broadwell_cpu;

 #endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map
b/drivers/common/mlx5/rte_common_mlx5_version.map
index fe62fa2b2f..039d221333 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,6 +1,7 @@
 INTERNAL {
        global:

+       mlx5_common_init;
        mlx5_common_verbs_reg_mr;
        mlx5_common_verbs_dereg_mr;

diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 846398dd3d..d0e9345a55 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -1989,6 +1989,8 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
  */
 RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 {
+       mlx5_common_init();
+
        /* Build the static tables for Verbs conversion. */
        mlx5_set_ptype_table();
        mlx5_set_cksum_table();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 70692ea1d2..9dc3e8fa56 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -844,6 +844,8 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
  */
 RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 {
+       mlx5_common_init();
+
        if (mlx5_glue)
                rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
diff --git a/lib/librte_eal/include/rte_common.h
b/lib/librte_eal/include/rte_common.h
index 522afe58ed..8f487a563d 100644
--- a/lib/librte_eal/include/rte_common.h
+++ b/lib/librte_eal/include/rte_common.h
@@ -135,7 +135,6 @@ typedef uint16_t unaligned_uint16_t;

 #define RTE_PRIORITY_LOG 101
 #define RTE_PRIORITY_BUS 110
-#define RTE_PRIORITY_COMMON 119
 #define RTE_PRIORITY_CLASS 120
 #define RTE_PRIORITY_LAST 65535


-- 
David Marchand

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Ori Kam]

Hi David,

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> 
> On Mon, Jul 20, 2020 at 6:48 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> >
> > 20/07/2020 18:21, Ferruh Yigit:
> > > On 7/17/2020 2:49 PM, Parav Pandit wrote:
> > > > Currently mlx5_common uses CLASS priority to initialize
> > > > common code before initializing the PMD.
> > > > However mlx5_common is not really a class, it is the pre-initialization
> > > > code needed for the PMDs.
> > > >
> > > > In subsequent patch a needed initialization sequence is:
> > > > (a) Initialize bus (say pci)
> > > > (b) Initialize common code of a driver (mlx5_common)
> > > > (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
> > > > Information registered by these PMDs is used by mlx5_bus_pci PMD.
> > > > This mlx5 class PMDs should not confused with rte_class.
> > > > (d) Register mlx5 PCI bus PMD
> > > >
> > > > Hence, introduce a new RTE priority level RTE_PRIO_COMMON which
> > > > can be used for common initialization and RTE_PRIO_CLASS by mlx5
> PMDs
> > > > for class driver initialization.
> > > >
> > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > Acked-by: Matan Azrad <matan@mellanox.com>
> > > > ---
> > > > Changelog:
> > > > v2->v3:
> > > >  - new patch
> > > > ---
> > > >  lib/librte_eal/include/rte_common.h | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > >
> > > > diff --git a/lib/librte_eal/include/rte_common.h
> b/lib/librte_eal/include/rte_common.h
> > > > index 8f487a563..522afe58e 100644
> > > > --- a/lib/librte_eal/include/rte_common.h
> > > > +++ b/lib/librte_eal/include/rte_common.h
> > > > @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t;
> > > >
> > > >  #define RTE_PRIORITY_LOG 101
> > > >  #define RTE_PRIORITY_BUS 110
> > > > +#define RTE_PRIORITY_COMMON 119
> > > >  #define RTE_PRIORITY_CLASS 120
> > > >  #define RTE_PRIORITY_LAST 65535
> > > >
> > > >
> > >
> > > I guess the name "common" selected because of the intention to use it by
> the
> > > common piece of the driver, but only from eal perspective the name
> > > "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose.
> >
> > You're right.
> >
> > > Also the value doesn't leave any gap between the class priority, what else
> can
> > > be needed in the future in between, right?
> >
> > And we can imagine a bus requiring a common lib
> > to be initialized before.
> >
> > > @Thomas, @David, I am reluctant to get this eal change through the next-
> net, can
> > > you please review/ack it first?
> >
> > What about skipping this patch and using "RTE_PRIORITY_CLASS - 1"
> > in the code?
> 
> net and vdpa code expect the common code being initialised.
> It is a dependency internal to mlx5 drivers, I see nothing generic.
> 
First the idea was to declare a new bus not a PMD.
The issue is not from common code but from loading more than one
device on the same PCI.
So the logic is Mellanox PMD are registering to the new bus, the new bus
register to the PCI one.
So the new bus must have middle priority between the PMDs (class)  and the PCI (bus) bus.

May be a better name should have been:
RTE_PRIORITY_ MANUFACTORE _BUS  

> The common driver can provide a init function called by net and vdpa
> drivers in constructor context (not that I like calling complex init,
> but it should be equivalent to current state).
> It expresses a clear dependency and there is no constructor semantic
> to invent, nor runtime breakage possible because someone tweaks the
> priority order in the future.
> 
> There is also some hack about a haswell/broadwell detection in LOG
> priority that makes no sense.
> 
> I compile-tested following:
> 
> $ git diff next-net-mlx/master -- lib/librte_eal/ drivers/*/mlx5/
> diff --git a/drivers/common/mlx5/mlx5_common.c
> b/drivers/common/mlx5/mlx5_common.c
> index 79cd5ba344..3fb9a8ab89 100644
> --- a/drivers/common/mlx5/mlx5_common.c
> +++ b/drivers/common/mlx5/mlx5_common.c
> @@ -49,14 +49,6 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
>                 rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE);
>  }
> 
> -/**
> - * Initialization routine for run-time dependency on glue library.
> - */
> -RTE_INIT_PRIO(mlx5_glue_init, COMMON)
> -{
> -       mlx5_glue_constructor();
> -}
> -
>  /**
>   * This function is responsible of initializing the variable
>   *  haswell_broadwell_cpu by checking if the cpu is intel
> @@ -67,7 +59,7 @@ RTE_INIT_PRIO(mlx5_glue_init, COMMON)
>   *  if the cpu is haswell or broadwell the variable will be set to 1
>   *  otherwise it will be 0.
>   */
> -RTE_INIT_PRIO(mlx5_is_haswell_broadwell_cpu, LOG)
> +static void mlx5_is_haswell_broadwell_cpu(void)
>  {
>  #ifdef RTE_ARCH_X86_64
>         unsigned int broadwell_models[4] = {0x3d, 0x47, 0x4F, 0x56};
> @@ -114,6 +106,21 @@ RTE_INIT_PRIO(mlx5_is_haswell_broadwell_cpu,
> LOG)
>         haswell_broadwell_cpu = 0;
>  }
> 
> +/**
> + * Initialization routine for run-time dependency on glue library.
> + */
> +void mlx5_common_init(void)
> +{
> +       static bool init_once = false;
> +
> +       if (init_once)
> +               return;
> +
> +       mlx5_glue_constructor();
> +       mlx5_is_haswell_broadwell_cpu();
> +       init_once = true;
> +}
> +
>  /**
>   * Allocate page of door-bells and register it using DevX API.
>   *
> diff --git a/drivers/common/mlx5/mlx5_common.h
> b/drivers/common/mlx5/mlx5_common.h
> index 5b9b7bd5a9..30961bc8cc 100644
> --- a/drivers/common/mlx5/mlx5_common.h
> +++ b/drivers/common/mlx5/mlx5_common.h
> @@ -254,6 +254,10 @@ int64_t mlx5_get_dbr(void *ctx,  struct
> mlx5_dbr_page_list *head,
>  __rte_internal
>  int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t
> umem_id,
>                          uint64_t offset);
> +
> +__rte_internal
> +void mlx5_common_init(void);
> +
>  extern uint8_t haswell_broadwell_cpu;
> 
>  #endif /* RTE_PMD_MLX5_COMMON_H_ */
> diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map
> b/drivers/common/mlx5/rte_common_mlx5_version.map
> index fe62fa2b2f..039d221333 100644
> --- a/drivers/common/mlx5/rte_common_mlx5_version.map
> +++ b/drivers/common/mlx5/rte_common_mlx5_version.map
> @@ -1,6 +1,7 @@
>  INTERNAL {
>         global:
> 
> +       mlx5_common_init;
>         mlx5_common_verbs_reg_mr;
>         mlx5_common_verbs_dereg_mr;
> 
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
> index 846398dd3d..d0e9345a55 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -1989,6 +1989,8 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5,
> NOTICE)
>   */
>  RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
>  {
> +       mlx5_common_init();
> +
>         /* Build the static tables for Verbs conversion. */
>         mlx5_set_ptype_table();
>         mlx5_set_cksum_table();
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
> index 70692ea1d2..9dc3e8fa56 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
> @@ -844,6 +844,8 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype,
> pmd.vdpa.mlx5, NOTICE)
>   */
>  RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
>  {
> +       mlx5_common_init();
> +
>         if (mlx5_glue)
>                 rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
>  }
> diff --git a/lib/librte_eal/include/rte_common.h
> b/lib/librte_eal/include/rte_common.h
> index 522afe58ed..8f487a563d 100644
> --- a/lib/librte_eal/include/rte_common.h
> +++ b/lib/librte_eal/include/rte_common.h
> @@ -135,7 +135,6 @@ typedef uint16_t unaligned_uint16_t;
> 
>  #define RTE_PRIORITY_LOG 101
>  #define RTE_PRIORITY_BUS 110
> -#define RTE_PRIORITY_COMMON 119
>  #define RTE_PRIORITY_CLASS 120
>  #define RTE_PRIORITY_LAST 65535
> 
> 
> --
> David Marchand

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[David Marchand]

On Mon, Jul 20, 2020 at 9:30 PM Ori Kam <orika@mellanox.com> wrote:
> > net and vdpa code expect the common code being initialised.
> > It is a dependency internal to mlx5 drivers, I see nothing generic.
> >
> First the idea was to declare a new bus not a PMD.
> The issue is not from common code but from loading more than one
> device on the same PCI.
> So the logic is Mellanox PMD are registering to the new bus, the new bus
> register to the PCI one.

- Is there a problem with the snippet I sent yesterday that removes
the COMMON priority?


- Currently in the main branch, the net/mlx5 driver is a pci driver
registered to the pci bus driver.
This registration happens in a constructor which puts the net/mlx5 pci
driver struct (with a populated id map) into the pci driver list.
Later (out of constructor context), a probe on the pci bus uses this
driver list.


A new pci driver is introduced by Mellanox to solve resources
dispatching between mlx5 drivers.
I'd rather not call this new pci driver a bus, it is not registered as
such (no rte_bus object) => no reason its constructor should be called
from the RTE_PRIO_BUS priority.

This driver not being a bus, I would not put it in drivers/bus/,
rather in the common code.
If there is a build dependency issue with the pci bus driver, you can
probably use the trick from qat
(https://git.dpdk.org/dpdk/tree/drivers/Makefile#n17) and write an
explicit dependency in drivers/Makefile.
meson should be just fine.


net/mlx5 and vdpa/mlx5 are not classes => no reason their constructor
should be called from the RTE_PRIO_CLASS priority.
The only classes we have are ethdev and vdpa.


The new pci driver simply serves as a proxy for mlx5 net and vdpa drivers.
From what I see, the new pci driver constructs its pci device id
map/drv_flags when registering to the pci bus driver based on what
mlx5 net and vdpa drivers registered before.
This is where a dependency was created.

You can move the id_table/drv_flags construction from
RTE_INIT(mlx5_bus_pci) to rte_mlx5_pci_register() itself.
This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5
pci driver whether it is registered to the pci bus or not yet.


-- 
David Marchand

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Parav Pandit]

Hi David,

On 7/21/2020 3:04 PM, David Marchand wrote:
> On Mon, Jul 20, 2020 at 9:30 PM Ori Kam <orika@mellanox.com> wrote:
>>> net and vdpa code expect the common code being initialised.
>>> It is a dependency internal to mlx5 drivers, I see nothing generic.
>>>
>> First the idea was to declare a new bus not a PMD.
>> The issue is not from common code but from loading more than one
>> device on the same PCI.
>> So the logic is Mellanox PMD are registering to the new bus, the new bus
>> register to the PCI one.
> 
> - Is there a problem with the snippet I sent yesterday that removes
> the COMMON priority?
> 
> 
No. The problem as you describe below is in handling the inter
dependencies between mlx5_common, rte_pci and mlx5_{net/vdpa}.

mlx5_net/vdpa demands common to be loaded first.
mlx5_pci_bus expects mlx5_net_vdpa to register first.
More below.

> - Currently in the main branch, the net/mlx5 driver is a pci driver
> registered to the pci bus driver.
> This registration happens in a constructor which puts the net/mlx5 pci
> driver struct (with a populated id map) into the pci driver list.
> Later (out of constructor context), a probe on the pci bus uses this
> driver list.
> 
> 
> A new pci driver is introduced by Mellanox to solve resources
> dispatching between mlx5 drivers.
> I'd rather not call this new pci driver a bus, it is not registered as
> such (no rte_bus object) => no reason its constructor should be called
> from the RTE_PRIO_BUS priority.
> 
> This driver not being a bus, I would not put it in drivers/bus/,
> rather in the common code.
> If there is a build dependency issue with the pci bus driver, you can
> probably use the trick from qat
> (https://git.dpdk.org/dpdk/tree/drivers/Makefile#n17) and write an
> explicit dependency in drivers/Makefile.
> meson should be just fine.
> 
mlx5_common to depend on pci bus. (new Makefile change)
mlx5_net and mlx5_vdpa to depend on mlx5_common. (already there)

> 
> net/mlx5 and vdpa/mlx5 are not classes => no reason their constructor
> should be called from the RTE_PRIO_CLASS priority.
> The only classes we have are ethdev and vdpa.
> 
> 
> The new pci driver simply serves as a proxy for mlx5 net and vdpa drivers.
> From what I see, the new pci driver constructs its pci device id
> map/drv_flags when registering to the pci bus driver based on what
> mlx5 net and vdpa drivers registered before.
> This is where a dependency was created.
> 
> You can move the id_table/drv_flags construction from
> RTE_INIT(mlx5_bus_pci) to rte_mlx5_pci_register() itself.
ok. This is what were also discussing internally as well.

> This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5
> pci driver whether it is registered to the pci bus or not yet.

And here pci_id_table will not be const *.
Is that ok?

Basically mlx5_pci_bus, mlx5_net and mlx5_vdpa will be called with same
priority RTE_PRIO_LAST.
driver id table and drv_flags dynamically updated as PMDs register.

Should we have another API after rte_pci_register() to indicate that
some field of the driver are updated, instead of just silently updating
it in a PMD?

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[David Marchand]

On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> wrote:
> > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5
> > pci driver whether it is registered to the pci bus or not yet.
>
> And here pci_id_table will not be const *.
> Is that ok?

This is already the case in the current patch.
I see nothing wrong with it.
The pci code expects this pointer to be const and will it treat it as such.


On the other hand, you may remove the pci table exported by the common
driver as it is useless.
pmdinfogen only parses the .o and extracts the table symbol, so it
won't load other mlx drivers to generate the .pmd.c file.

I.e. with current patch,
$ ./usertools/dpdk-pmdinfo.py ./build/drivers/librte_bus_mlx5_pci.so
PMD NAME: mlx5_bus_pci


>
> Basically mlx5_pci_bus, mlx5_net and mlx5_vdpa will be called with same
> priority RTE_PRIO_LAST.
> driver id table and drv_flags dynamically updated as PMDs register.
>
> Should we have another API after rte_pci_register() to indicate that
> some field of the driver are updated, instead of just silently updating
> it in a PMD?

All of this happens in constructor context, this is a really early
stage used to resolve code dependencies.
No big init should happen there.

We can document this use of the existing API but I don't see which API to add.


-- 
David Marchand

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Thomas Monjalon]

21/07/2020 13:29, David Marchand:
> On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> wrote:
> > Basically mlx5_pci_bus, mlx5_net and mlx5_vdpa will be called with same
> > priority RTE_PRIO_LAST.
> > driver id table and drv_flags dynamically updated as PMDs register.
> >
> > Should we have another API after rte_pci_register() to indicate that
> > some field of the driver are updated, instead of just silently updating
> > it in a PMD?
> 
> All of this happens in constructor context, this is a really early
> stage used to resolve code dependencies.
> No big init should happen there.

In general, there is a misunderstanding of what should be done
in a constructor. I think we should document constructor usage properly.

For the record, init done in constructors should be small.
If the application is linked with DPDK but choose to not initialize it,
it should not have driver initialized (until it calls rte_eal_init).
Constructor init should be limited to adding a driver struct to a list.

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Parav Pandit]

> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, July 21, 2020 4:59 PM
> 
> On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> wrote:
> > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5
> > > pci driver whether it is registered to the pci bus or not yet.
> >
> > And here pci_id_table will not be const *.
> > Is that ok?
> 
> This is already the case in the current patch.
> I see nothing wrong with it.
> The pci code expects this pointer to be const and will it treat it as such.
> 
o.k.

Gaetan, Ferruh, Thomas,
Can you please ack as well?

> 
> On the other hand, you may remove the pci table exported by the common
> driver as it is useless.
> pmdinfogen only parses the .o and extracts the table symbol, so it won't load
> other mlx drivers to generate the .pmd.c file.
> 
Ok.

> I.e. with current patch,
> $ ./usertools/dpdk-pmdinfo.py ./build/drivers/librte_bus_mlx5_pci.so
> PMD NAME: mlx5_bus_pci
> 
> 
> >
> > Basically mlx5_pci_bus, mlx5_net and mlx5_vdpa will be called with
> > same priority RTE_PRIO_LAST.
> > driver id table and drv_flags dynamically updated as PMDs register.
> >
> > Should we have another API after rte_pci_register() to indicate that
> > some field of the driver are updated, instead of just silently
> > updating it in a PMD?
> 
> All of this happens in constructor context, this is a really early stage used to
> resolve code dependencies.
> No big init should happen there.
> 
Right.
Only init() is to dynamically build the pci id table to remove duplicates, how its done today.
And it requires reallocating new table as part of mlx5_pci_bus_register() and updating new id table pointer in the driver structure.

> We can document this use of the existing API but I don't see which API to
> add.
> 
Alright.
Will wait for ack on pci id table from Ferruh, Gaetan and Thomas to post v8.
Thanks a lot David for the review.

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Thomas Monjalon]

21/07/2020 14:13, Parav Pandit:
> From: David Marchand <david.marchand@redhat.com>
> > On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> wrote:
> > > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5
> > > > pci driver whether it is registered to the pci bus or not yet.
> > >
> > > And here pci_id_table will not be const *.
> > > Is that ok?
> > 
> > This is already the case in the current patch.
> > I see nothing wrong with it.
> > The pci code expects this pointer to be const and will it treat it as such.
> > 
> o.k.
> 
> Gaetan, Ferruh, Thomas,
> Can you please ack as well?

Yes of course it is OK updating the PCI table of the common layer
in runtime. The most important is to keep the fixed PCI table of the PMDs
the same as registered for pmdinfo usage.

> > On the other hand, you may remove the pci table exported by the common
> > driver as it is useless.
> > pmdinfogen only parses the .o and extracts the table symbol, so it won't load
> > other mlx drivers to generate the .pmd.c file.
> 
> Ok.

One more nit: please don't use "class" as variable name for drivers.

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Parav Pandit]

> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Tuesday, July 21, 2020 5:57 PM
> 
> 21/07/2020 14:13, Parav Pandit:
> > From: David Marchand <david.marchand@redhat.com>
> > > On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com>
> wrote:
> > > > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the
> > > > > mlx5 pci driver whether it is registered to the pci bus or not yet.
> > > >
> > > > And here pci_id_table will not be const *.
> > > > Is that ok?
> > >
> > > This is already the case in the current patch.
> > > I see nothing wrong with it.
> > > The pci code expects this pointer to be const and will it treat it as such.
> > >
> > o.k.
> >
> > Gaetan, Ferruh, Thomas,
> > Can you please ack as well?
> 
> Yes of course it is OK updating the PCI table of the common layer in runtime.
> The most important is to keep the fixed PCI table of the PMDs the same as
> registered for pmdinfo usage.
o.k. pci id table registered with PCI bus will be build dynamically, similar to how its done in v7.
Instead of doing it in constructor of mlx5_pci_bus, it will be done inside rte_mlx5_pci_bus_register().
Right?

And since we don't register it as rte_bus, shall I name it as rte_mlx5_pci_driver_register()/unregister().
Right?

> 
> > > On the other hand, you may remove the pci table exported by the
> > > common driver as it is useless.
> > > pmdinfogen only parses the .o and extracts the table symbol, so it
> > > won't load other mlx drivers to generate the .pmd.c file.
> >
> > Ok.
> 
> One more nit: please don't use "class" as variable name for drivers.
Ok. I will address that.

Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level

[Thomas Monjalon]

21/07/2020 14:51, Parav Pandit:
> 
> > From: Thomas Monjalon <thomas@monjalon.net>
> > Sent: Tuesday, July 21, 2020 5:57 PM
> > 
> > 21/07/2020 14:13, Parav Pandit:
> > > From: David Marchand <david.marchand@redhat.com>
> > > > On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com>
> > wrote:
> > > > > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the
> > > > > > mlx5 pci driver whether it is registered to the pci bus or not yet.
> > > > >
> > > > > And here pci_id_table will not be const *.
> > > > > Is that ok?
> > > >
> > > > This is already the case in the current patch.
> > > > I see nothing wrong with it.
> > > > The pci code expects this pointer to be const and will it treat it as such.
> > > >
> > > o.k.
> > >
> > > Gaetan, Ferruh, Thomas,
> > > Can you please ack as well?
> > 
> > Yes of course it is OK updating the PCI table of the common layer in runtime.
> > The most important is to keep the fixed PCI table of the PMDs the same as
> > registered for pmdinfo usage.
> 
> o.k. pci id table registered with PCI bus will be build dynamically, similar to how its done in v7.
> Instead of doing it in constructor of mlx5_pci_bus, it will be done inside rte_mlx5_pci_bus_register().
> Right?

Yes

> And since we don't register it as rte_bus, shall I name it as rte_mlx5_pci_driver_register()/unregister().
> Right?

Yes, if merging code in the common driver, no need to keep "bus" wording.

[dpdk-dev] [PATCH v7 3/9] common/mlx5: fix empty input style in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")
Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index c4f9b006a..734ace2a3 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v7 4/9] common/mlx5: change mlx5 class enum values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v1->v2:
 - Rebasd due to removal previous patch
---
 drivers/common/mlx5/mlx5_common.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 285150705..00e68edbb 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -208,9 +209,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT64(0),
+	MLX5_CLASS_VDPA = RTE_BIT64(1),
 };
 
 #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */
-- 
2.26.2

[dpdk-dev] [PATCH v7 5/9] common/mlx5: use common rte priority

[Parav Pandit]

Use RTE_PRIO_COMMON for mlx5 common initialization and use
RTE_PRIO_CLASS for mlx5 net, vdpa PMDs.

This enables to do following initialization sequence.

(a) Initialize bus (say pci)
(b) Initialize common code of a driver (mlx5_common)
(c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa)
Information registered by these PMDs is used by mlx5_bus_pci PMD.
This mlx5 class PMDs should not confused with rte_class.
(d) Register mlx5 PCI bus PMD

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c | 2 +-
 drivers/net/mlx5/mlx5.c           | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c     | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 693e2c68c..0ce5e4db1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 /**
  * Initialization routine for run-time dependency on glue library.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+RTE_INIT_PRIO(mlx5_glue_init, COMMON)
 {
 	mlx5_glue_constructor();
 }
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 723c1dd72..1647256be 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2074,7 +2074,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_pmd_init)
+RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 {
 	/* Build the static tables for Verbs conversion. */
 	mlx5_set_ptype_table();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 67e77b11a..205a9e4ad 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -844,7 +844,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
 /**
  * Driver initialization routine.
  */
-RTE_INIT(rte_mlx5_vdpa_init)
+RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 {
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_vdpa_driver);
-- 
2.26.2

[dpdk-dev] [PATCH v7 6/9] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
pci device.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v6->v7:
 - Updated release notes
v5->v6:
 - Updated Makefile for parallel shared build support
v4->v5:
 - Merged maintainers update patch with this patch
v2->v3:
 - Addressed comments from Thomas and Asaf
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address comments from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Changed alignedment to mlx5 specific aligment instead of standard
   DPDK
 - Using uint32_t instead of mlx5_class enum
v1->v2:
 - Address comments from Thomas and Gaetan
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
---
 MAINTAINERS                                   |  5 ++
 doc/guides/rel_notes/release_20_08.rst        |  5 ++
 drivers/bus/Makefile                          |  4 +
 drivers/bus/meson.build                       |  2 +-
 drivers/bus/mlx5_pci/Makefile                 | 39 +++++++++
 drivers/bus/mlx5_pci/meson.build              | 19 +++++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 84 +++++++++++++++++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
 9 files changed, 176 insertions(+), 1 deletion(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 3cd402b34..9e3bd1672 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -525,6 +525,11 @@ Intel FPGA bus
 M: Rosen Xu <rosen.xu@intel.com>
 F: drivers/bus/ifpga/
 
+Melllanox mlx5 PCI bus driver
+M: Parav Pandit <parav@mellaox.com>
+M: Matan Azrad <matan@mellanox.com>
+F: drivers/bus/mlx5_pci
+
 NXP buses
 M: Hemant Agrawal <hemant.agrawal@nxp.com>
 M: Sachin Saxena <sachin.saxena@nxp.com>
diff --git a/doc/guides/rel_notes/release_20_08.rst b/doc/guides/rel_notes/release_20_08.rst
index 6f44ffdfc..917a5b8b3 100644
--- a/doc/guides/rel_notes/release_20_08.rst
+++ b/doc/guides/rel_notes/release_20_08.rst
@@ -226,6 +226,11 @@ New Features
   See the :doc:`../sample_app_ug/l2_forward_real_virtual` for more
   details of this parameter usage.
 
+* **Added Mellanox mlx5_pci bus driver.**
+
+  Added mlx5_pci bus driver that supports loading multiple PMDs for
+  Mellanox mlx5 devices. This bus PMD is auto selected when its
+  upper layer PMD need it.
 
 Removed Items
 -------------
diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile
index cea3b55e6..a70f213c1 100644
--- a/drivers/bus/Makefile
+++ b/drivers/bus/Makefile
@@ -9,6 +9,10 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
 DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
+DEPDIRS-mlx5_pci := pci
+DIRS-y += mlx5_pci
+endif
 DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
 DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus
 
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 80de2d91d..b1381838d 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
+drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
 std_deps = ['eal']
 config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
 driver_name_fmt = 'rte_bus_@0@'
diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
new file mode 100644
index 000000000..dd24811a3
--- /dev/null
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -0,0 +1,39 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Mellanox Technologies, Ltd
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_bus_mlx5_pci.a
+
+CFLAGS += -O3 -Wall -Wextra
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -Wno-strict-prototypes
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
+CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
+CFLAGS += -I$(BUILDDIR)/drivers/bus/pci
+CFLAGS += -I$(BUILDDIR)/drivers/bus
+LDLIBS += -lrte_eal
+LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# versioning export map
+EXPORT_MAP := rte_bus_mlx5_pci_version.map
+
+SRCS-y += mlx5_pci_bus.c
+
+# DEBUG which is usually provided on the command-line may enable
+# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
+ifeq ($(DEBUG),1)
+CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
+endif
+
+#
+# Export include files
+#
+SYMLINK-y-include += rte_bus_mlx5_pci.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
new file mode 100644
index 000000000..64a17cbad
--- /dev/null
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Mellanox Technologies Ltd
+
+deps += ['pci', 'bus_pci', 'common_mlx5']
+install_headers('rte_bus_mlx5_pci.h')
+sources = files('mlx5_pci_bus.c')
+
+cflags_options = [
+	'-std=c11',
+	'-Wno-strict-prototypes',
+	'-D_BSD_SOURCE',
+	'-D_DEFAULT_SOURCE',
+	'-D_XOPEN_SOURCE=600'
+]
+foreach option:cflags_options
+	if cc.has_argument(option)
+		cflags += option
+	endif
+endforeach
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
new file mode 100644
index 000000000..66db3c7b0
--- /dev/null
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include "rte_bus_mlx5_pci.h"
+
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
+{
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
new file mode 100644
index 000000000..9f8d22e2b
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _RTE_BUS_MLX5_PCI_H_
+#define _RTE_BUS_MLX5_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Bus Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This bus enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci bus driver.
+ *
+ * -----------    ------------    -----------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5        |
+ * | net pmd |    | vdpa pmd |    | new class pmd |
+ * -----------    ------------    -----------------
+ *      \              |                /
+ *       \             |               /
+ *        \       -------------       /
+ *         \______|   mlx5    |_____ /
+ *                |   pci bus |
+ *                -------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct rte_mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t dev_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a rte_mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
+
+#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \
+	static const char *mlx5_pci_drvinit_fn_ ## nm; \
+	RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \
+	{ \
+		(drv).driver.name = RTE_STR(nm); \
+		rte_mlx5_pci_driver_register(&drv); \
+	} \
+	RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _RTE_BUS_MLX5_PCI_H_ */
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
new file mode 100644
index 000000000..4cfd3db10
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
@@ -0,0 +1,5 @@
+INTERNAL {
+	global:
+
+	rte_mlx5_pci_driver_register;
+};
-- 
2.26.2

[dpdk-dev] [PATCH v7 7/9] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

Create a mlx5 bus driver framework for invoking drivers of
multiple classes who have registered with the mlx5_pci bus
driver.

Validate user class arguments for supported class combinations.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Addressed comments from Asaf
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Addressed comments from Matan.
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block.
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Maintaining class driver list to keep load/unload order symmetric and
   mirror of each other.
 - Deriving drv_flags based on the class drivers
 - Using PCI address comparision helper instead of pointer comparision
 - Fixed alignment for id_table
 - Continue to probe_err if device is already probed
 - Perform dma map on best effort basis for all supported drivers
 - Removed drv_flags check
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Address comments from Thomas and Gaetan
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - USe uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
---
 drivers/bus/mlx5_pci/Makefile       |   2 +
 drivers/bus/mlx5_pci/meson.build    |   2 +-
 drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++
 3 files changed, 511 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
index dd24811a3..ee1af4dc6 100644
--- a/drivers/bus/mlx5_pci/Makefile
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -16,7 +16,9 @@ CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/bus/pci
 CFLAGS += -I$(BUILDDIR)/drivers/bus
+CFLAGS += -D_DEFAULT_SOURCE
 LDLIBS += -lrte_eal
+LDLIBS += -lrte_kvargs
 LDLIBS += -lrte_common_mlx5
 LDLIBS += -lrte_pci -lrte_bus_pci
 
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
index 64a17cbad..0532a9dfd 100644
--- a/drivers/bus/mlx5_pci/meson.build
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Mellanox Technologies Ltd
 
-deps += ['pci', 'bus_pci', 'common_mlx5']
+deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
 install_headers('rte_bus_mlx5_pci.h')
 sources = files('mlx5_pci_bus.c')
 
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index 66db3c7b0..6f219730c 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -2,13 +2,521 @@
  * Copyright 2020 Mellanox Technologies, Ltd
  */
 
+#include <stdlib.h>
+#include <rte_malloc.h>
 #include "rte_bus_mlx5_pci.h"
+#include <mlx5_common_utils.h>
 
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of class drivers. */
 static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
 				TAILQ_HEAD_INITIALIZER(drv_list);
 
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int dev_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .dev_class = MLX5_CLASS_NET },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	/* New class combination should be added here.
+	 * For example a new multi class device combination
+	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
+	 */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].dev_class;
+	}
+	return -EINVAL;
+}
+
+static struct rte_mlx5_pci_driver *
+class_driver_get(uint32_t class)
+{
+	struct rte_mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->dev_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
 void
 rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
 {
 	TAILQ_INSERT_TAIL(&drv_list, driver, next);
 }
+
+static bool
+mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = class_driver_get(RTE_BIT64(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT64(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT64(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+class_drivers_probe(struct mlx5_pci_device *dev,
+		    struct rte_pci_driver *pci_drv,
+		    struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct rte_mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->dev_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->dev_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load class driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->dev_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	class_drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple PCI class devices.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		   struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more class devices for a PCI device.
+ *
+ * This function removes all class devices belong to a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_bus_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload class drivers. */
+	ret = class_drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+		     uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class = NULL;
+	struct rte_mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_map) {
+			ret = class->pci_driver.dma_map(pci_dev, addr,
+							iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == class)
+			break;
+		if (device_class_enabled(dev, temp->dev_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+		       uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, class->dev_class) &&
+		    class->pci_driver.dma_unmap) {
+			local_ret = class->pci_driver.dma_unmap(pci_dev, addr,
+								iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5
+ * class drivers.
+ */
+static struct rte_pci_id *mlx5_bus_pci_id_map;
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == mlx5_bus_pci_id_map[i].device_id &&
+		    id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id &&
+		    id->subsystem_vendor_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_vendor_id &&
+		    id->subsystem_device_id ==
+		    mlx5_bus_pci_id_map[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(const struct rte_pci_id *id_table, int *next_idx)
+{
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, *next_idx)) {
+			/* New entry; add to the table. */
+			mlx5_bus_pci_id_map[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_build(void)
+{
+	struct rte_mlx5_pci_driver *first_driver;
+	struct rte_mlx5_pci_driver *driver;
+	const struct rte_pci_id *id_table;
+	int num_ids = 0;
+	int i = 0;
+
+	TAILQ_FOREACH(driver, &drv_list, next)
+		num_ids += pci_id_table_size_get(driver->pci_driver.id_table);
+	if (!num_ids)
+		return -ENODEV;
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map));
+	if (!mlx5_bus_pci_id_map)
+		return -ENOMEM;
+	first_driver = TAILQ_FIRST(&drv_list);
+	/* Copy the first class driver's ID table. */
+	for (id_table = first_driver->pci_driver.id_table;
+	     id_table->vendor_id != 0; id_table++, i++)
+		mlx5_bus_pci_id_map[i] = *id_table;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		/* We already added first driver; skip it. */
+		if (driver == first_driver)
+			continue;
+		pci_id_insert(driver->pci_driver.id_table, &i);
+	}
+	mlx5_bus_pci_id_map[i].vendor_id = 0;
+	return 0;
+}
+
+static bool mlx5_bus_registered;
+static struct rte_pci_driver mlx5_bus_driver = {
+	.driver = {
+		.name = "mlx5_bus_pci",
+	},
+	.probe = mlx5_bus_pci_probe,
+	.remove = mlx5_bus_pci_remove,
+	.dma_map = mlx5_bus_pci_dma_map,
+	.dma_unmap = mlx5_bus_pci_dma_unmap,
+};
+
+RTE_INIT(mlx5_bus_pci)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret;
+
+	ret = pci_ids_table_build();
+	if (ret)
+		return;
+	TAILQ_FOREACH(class, &drv_list, next)
+		mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags;
+	mlx5_bus_driver.id_table = mlx5_bus_pci_id_map;
+	rte_pci_register(&mlx5_bus_driver);
+	mlx5_bus_registered = true;
+}
+
+RTE_FINI(mlx5_bus_pci_finish)
+{
+	if (mlx5_bus_registered)
+		rte_pci_unregister(&mlx5_bus_driver);
+	if (mlx5_bus_pci_id_map)
+		free(mlx5_bus_pci_id_map);
+}
+RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__);
+RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
-- 
2.26.2

[dpdk-dev] [PATCH v7 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Parav Pandit]

Enable class driver to match with the mlx5 pci devices.
Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Avoid static table
v1->v2:
 - Migreate API from rte_driver to rte_pci_driver
---
 drivers/net/mlx5/Makefile        |  3 ++-
 drivers/net/mlx5/linux/mlx5_os.c |  1 -
 drivers/net/mlx5/linux/mlx5_os.h |  1 +
 drivers/net/mlx5/meson.build     |  2 +-
 drivers/net/mlx5/mlx5.c          | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h          |  1 -
 drivers/vdpa/mlx5/Makefile       |  3 ++-
 drivers/vdpa/mlx5/meson.build    |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c    | 23 +++++++++++++----------
 mk/rte.app.mk                    |  1 +
 10 files changed, 35 insertions(+), 26 deletions(-)

diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index 9eaac6b7e..3c5f486c3 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -46,16 +46,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_bus_mlx5_pci
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index f228babae..3450a961d 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1399,7 +1399,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 31add3988..695722520 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,4 +15,5 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
+
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index c06b1538a..e35d90bbb 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -8,7 +8,7 @@ if not (is_linux or is_windows)
 	subdir_done()
 endif
 
-deps += ['hash', 'common_mlx5']
+deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
 sources = files(
 	'mlx5.c',
 	'mlx5_ethdev.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 1647256be..4d1c3716f 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -34,6 +34,7 @@
 #include <rte_spinlock.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
+#include <rte_bus_mlx5_pci.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
@@ -2056,16 +2057,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct rte_mlx5_pci_driver mlx5_driver = {
+	.dev_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /* Initialize driver log type. */
@@ -2081,7 +2085,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		rte_mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 2e61d0cba..c5cb732c5 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -129,7 +129,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..9bb5dfdbf 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
 CFLAGS += -I$(RTE_SDK)/lib/librte_sched
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..f4175c34e 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 205a9e4ad..5d247e0a1 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,7 +11,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <rte_bus_mlx5_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI bus device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct rte_mlx5_pci_driver mlx5_vdpa_driver = {
+	.dev_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
@@ -847,7 +850,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
 RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS)
 {
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0ce8cf541..d7c1d90e4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF)      += -lrte_pmd_memif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD)       += -lrte_pmd_mlx4
 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
 _LDLIBS-y                                   += -lrte_common_mlx5
+_LDLIBS-y                                   += -lrte_bus_mlx5_pci
 endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)  += -lrte_pmd_mlx5_vdpa
-- 
2.26.2

[dpdk-dev] [PATCH v7 9/9] common/mlx5: remove class checks from individual driver

[Parav Pandit]

Now that mlx5_pci bus does the check for enabled classes and performs
probe(), remove() of associated classes, individual class driver
does not need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - Removed empty line
v1->v2:
 - New patch
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  2 -
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/net/mlx5/linux/mlx5_os.h              |  1 -
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 6 files changed, 52 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 0ce5e4db1..771e046c1 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 00e68edbb..5b9b7bd5a 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -244,8 +244,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 5aad2199a..ce91576b3 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,8 +1,6 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 3450a961d..0b3e1b674 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1386,11 +1386,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_dev_config dev_config;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index 695722520..31add3988 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -15,5 +15,4 @@ enum {
 #define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
 			RTE_PCI_DRV_INTR_RMV | \
 			RTE_PCI_DRV_PROBE_AGAIN)
-
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 5d247e0a1..70692ea1d 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

Re: [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes

[Raslan Darawsheh]

Hi,

> -----Original Message-----
> From: Parav Pandit <parav@mellanox.com>
> Sent: Friday, July 17, 2020 4:49 PM
> To: dev@dpdk.org; grive@u256.net; ferruh.yigit@intel.com; Thomas
> Monjalon <thomas@monjalon.net>
> Cc: Raslan Darawsheh <rasland@mellanox.com>; Ori Kam
> <orika@mellanox.com>; Matan Azrad <matan@mellanox.com>;
> joyce.kong@arm.com; Parav Pandit <parav@mellanox.com>
> Subject: [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple
> classes
> 
> This series introduces mlx5 bus to support multiple class of devices
> for a single PCI device.
> 
> Motivation and example
> ----------------------
> mlx5 PCI device supports multiple class of devices such as net, vdpa
> and regex devices.
> 
> Currently only one pmd (either net or vdpa) can bind to this device.
> This design limits use of PCI device only for single device class.
> 
> To support multiple classes simultaneously for a mlx5 PCI device,
> a new mlx5 PCI bus is created. This bus allows binding multiple
> class drivers (such as net, vdpa, regex(future)) to bind to the
> mlx5 PCI bus driver.
> 
> Change description
> ------------------
> Patch-1 Introduces RTE_BIT() macro
> Patch-2 Introduces new RTE constructor priority for common
> initialization
> Patch-3 Fixes compilation error
> Patch-4 Define mlx5 class as bit fields
> Patch-5 Uses new RTE common priority
> Patch-6 Adds mlx5 PCI bus
> Patch-7 Implements a mlx5 PCI bus driver
> Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
> API instead of rte PCI bus API
> Patch-9 Removed class check code as its already part of the bus now
> 
> Design overview
> ---------------
> 
>  -----------    ------------    -------------
>  |   mlx5  |    |   mlx5   |    |   mlx5    |
>  | net pmd |    | vdpa pmd |    | regex pmd |
>  -----------    ------------    -------------
>       \              |                /
>        \             |               /
>         \       -------------       /
>          \______|   mlx5    |_____ /
>                 |   pci bus |
>                 -------------
>                      |
>                  -----------
>                  |   mlx5  |
>                  | pci dev |
>                  -----------
> 
> - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
>   ID table of all related mlx5 PCI devices.
> - mlx5 class driver such as net, vdpa, regex PMD defines its
>   specific PCI ID table and mlx5 bus driver probes matching
>   class drivers.
> - mlx5 pci bus driver is cental place that validates supported
>   class combinations.
> - In future as code evolves, more device setup/cleanup and
>   resource creation code moves to mlx5 PCI bus driver.
> 
> Alternatives considered
> -----------------------
> 1. Instead of creating mlx5 pci bus, a common driver is
> implemented which exposes class registration API.
> However, bus model fits better with existing DPDK design
> similar to ifpga driver.
> Class registration API need to create a new callbacks
> and ID signature; instead it is better to utilize current
> well defined methods.
> 
> 2. Enhance pci core to allow multiple driver binding to
> single rte PCI device.
> This approach is not taken, because peer drivers using
> one PCI device won't be aware of other's presence. This
> requires cross-driver syncronization of who initializes
> common resources (such as irq, eq and more).
> This also requires refcounting common objects etc among
> peer drivers.
> Instead of layered approach delivers and allows putting
> common resource sharing, setup code in common bus driver.
> It also eliminates peer blind zone problem as bottom pci
> bus layer provides necessary setup without any reference
> counting.
> 
> 3. In future mlx5 prefers to use RDMA MR cache of the mbuf
> used between net and regex pmd so that same mbuf use across
> multiple device can be possible.
> 
> Examples:
> --------
> A user who wish to use a specific class(es) provides list
> of classes at command line such as,
> ./testpmd -w <PCI BDF>,class=net:vdpa
> ./testpmd -w <PCI BDF>,class=vdpa
> 
> In future,
> ./testpmd -w <PCI BDF>,class=net:regex
> 
> Changelog:
> v6->v7:
>  - Updated release notes
> v5->v6:
>  - Fixed compilation failure in parallel build for shared lib
> v4->v5:
>  - Squash the maintainers update path with other patch which adds the
>    bus
>  - Addressed comments from Morten Brørup
>  - Renamed newly added macro to RTE_BIT64
>  - Added doxygen comment section for the macro
> v3->v4:
>  - Fixed dma_map error unwinding flow to follow same order for unmap
> v2->v3:
>  - Added RTE priority for common driver initialization
>  - Addressed comments from Thomas and Asaf
>  - Fixed compilation error in glue wrapper
>  - Moved pci_driver structure instance as first in driver
>  - Removed white spaces at the end of line in diagram
>  - Address commnts from Matan
>  - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
>  - Renamed mlx5_valid_class_combo to mlx5_class_combinations
>  - Added cross check for class drivers to support only 3 flags for now
>  - Added full stop at the end of comment block
>  - Using full names in function names
>  - Added new line before function name in multiple functions
>  - Added example string to parse for multiple classes
>  - Dropped mlx5 prefix from static function
>  - Removed empty lines
>  - Fixed issue to remove multiple classes for a driver
>  - Using define for drv_flags at multiple places
>  - Deriving drv_flags based on the class drivers
>  - Fixed alignment for id_table
>  - Perform dma map on best effort basis for all supported drivers
>  - Dynamically build pci id table
>  - Using PCI to mlx5 device helper routines
> v1->v2:
>  - Addressed most comments from Thomas and Gaetan.
>  - Symbols starting with prefix rte_bus_pci_mlx5 may be
>    confusing as it may appear as it belong to rte_bus_pci module.
>    Hence it is kept as rte_bus_mlx5_pci which matches with other
>    modules as mlx5_vdpa, mlx5_net.
>  - Dropped 2nd patch and replace with new 6th patch.
>  - Avoided new file, added macro to rte_bitops.h
>  - Inheriting ret_pci_driver instead of rte_driver
>  - Added design and description of the mlx5_pci bus
>  - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
>  - Use anonymous structure for class search and code changes around it
>  - Define static for class comination array
>  - Use RTE_DIM to find array size
>  - Added OOM check for strdup()
>  - Renamed copy variable to nstr_orig
>  - Returning negagive error code
>  - Returning directly if match entry found
>  - Use compat condition check
>  - Avoided cutting error message string
>  - Use uint32_t datatype instead of enum mlx5_class
>  - Changed logic to parse device arguments only once during probe()
>  - Added check to fail driver probe if multiple classes register with
>    DMA ops
>  - Renamed function to parse_class_options
>  - Migreate API from rte_driver to rte_pci_driver
> 
> Parav Pandit (9):
>   eal: introduce macros for getting value for bit
>   eal: introduce RTE common initialization level
>   common/mlx5: fix empty input style in glue wrappers
>   common/mlx5: change mlx5 class enum values as bits
>   common/mlx5: use common rte priority
>   bus/mlx5_pci: add mlx5 PCI bus
>   bus/mlx5_pci: register a PCI driver
>   bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
>   common/mlx5: remove class checks from individual driver
> 
>  MAINTAINERS                                   |   5 +
>  doc/guides/rel_notes/release_20_08.rst        |   5 +
>  drivers/bus/Makefile                          |   4 +
>  drivers/bus/meson.build                       |   2 +-
>  drivers/bus/mlx5_pci/Makefile                 |  41 ++
>  drivers/bus/mlx5_pci/meson.build              |  19 +
>  drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 522 ++++++++++++++++++
>  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  84 +++
>  .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
>  drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
>  drivers/common/mlx5/mlx5_common.c             |  39 +-
>  drivers/common/mlx5/mlx5_common.h             |   7 +-
>  .../common/mlx5/rte_common_mlx5_version.map   |   2 -
>  drivers/net/mlx5/Makefile                     |   3 +-
>  drivers/net/mlx5/linux/mlx5_os.c              |   6 -
>  drivers/net/mlx5/meson.build                  |   2 +-
>  drivers/net/mlx5/mlx5.c                       |  26 +-
>  drivers/net/mlx5/mlx5.h                       |   1 -
>  drivers/vdpa/mlx5/Makefile                    |   3 +-
>  drivers/vdpa/mlx5/meson.build                 |   2 +-
>  drivers/vdpa/mlx5/mlx5_vdpa.c                 |  30 +-
>  lib/librte_eal/include/rte_bitops.h           |   8 +
>  lib/librte_eal/include/rte_common.h           |   1 +
>  mk/rte.app.mk                                 |   1 +
>  24 files changed, 737 insertions(+), 85 deletions(-)
>  create mode 100644 drivers/bus/mlx5_pci/Makefile
>  create mode 100644 drivers/bus/mlx5_pci/meson.build
>  create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
>  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
>  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> 
> --
> 2.26.2

Series applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh

[dpdk-dev] [PATCH v8 00/10] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 bus to support multiple class of devices
for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
mlx5 common module is extended as generic PCI PMD.
This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded
on a single PCI Device.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Fixed indentation for directory listing
Patch-3 Relax dependency order
Patch-4 Fixes compilation error
Patch-5 Fixes segmentation fault in regex probe error unwind path
Patch-6 Avoid using class constructor priority
Patch-7 Change class values as bits
Patch-8 Introduce mlx5 layer to support multiple class drivers
Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API
instead of rte PCI bus API
Patch-10 Removed class check code from class drivers 

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       ---------------     /
         \______| mlx5 common |____/
                |    pmd      |
                --------------- 
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 common driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD define their
  specific PCI ID tables. mlx5 common driver probes each
  individual class driver(s) for maching and enabled classes.
- mlx5 pci layer is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI common driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci common PMD, a common driver is
implemented which exposes class registration API.
However, common PMD model fits better with existing DPDK design
similar to ifpga driver.
Class registration API need to create a new callbacks
and ID signature; instead it is better to utilize current
well defined methods.

2. Enhance pci core to allow multiple driver binding to
single rte PCI device.
This approach is not taken, because peer drivers using
one PCI device won't be aware of other's presence. This
requires cross-driver syncronization of who initializes
common resources (such as irq, eq and more).
This also requires refcounting common objects etc among
peer drivers.
Instead of layered approach delivers and allows putting
common resource sharing, setup code in common driver.
It also eliminates peer blind zone problem as bottom pci
layer provides necessary setup without any reference
counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf
used between net and regex pmd so that same mbuf use across
multiple device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list
of classes at command line such as,
./testpmd -w <PCI BDF>,class=net:vdpa
./testpmd -w <PCI BDF>,class=vdpa

Changelog:
v7->v8:
 - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5
 - Avoided new RTE PRIORITY
 - Removed mlx5 common to use RTE_PRIORITY_CLASS
v6->v7:
 - Updated release notes
v5->v6:
 - Fixed compilation failure in parallel build for shared lib
v4->v5:
 - Squash the maintainers update path with other patch which adds the
   bus
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (7):
  eal: introduce macro for bit definition
  common/mlx5: fix void parameters in glue wrappers
  regex/mlx5: fix segmentation fault during error unwinding
  common/mlx5: avoid using class constructor priority
  common/mlx5: change class values as bits
  common/mlx5: register class drivers through common layer
  common/mlx5: remove class check from class drivers

Thomas Monjalon (3):
  drivers: fix indent of directory list
  drivers: relax dependency order
  common/mlx5: introduce layer to support multiple class drivers

 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   1 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |  52 +-
 drivers/common/mlx5/mlx5_common.h             |  12 +-
 drivers/common/mlx5/mlx5_common_pci.c         | 539 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   4 +-
 drivers/meson.build                           |  51 +-
 drivers/net/mlx5/Makefile                     |   1 -
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/mlx5.c                       |  25 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/regex/mlx5/meson.build                |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |  29 +-
 drivers/vdpa/mlx5/Makefile                    |   3 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  29 +-
 lib/librte_eal/include/rte_bitops.h           |   8 +
 22 files changed, 748 insertions(+), 123 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

-- 
2.25.4

[dpdk-dev] [PATCH v8 01/10] eal: introduce macro for bit definition

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
 lib/librte_eal/include/rte_bitops.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..ca46a110f 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,14 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+/**
+ * Get the uint64_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 63.
+ */
+#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.25.4

[dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Define each sub-directory on its own line ended with a comma,
and use a simple indent.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
 drivers/meson.build | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e76ebddfa..e2aeba931 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -2,17 +2,19 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 # Defines the order in which the drivers are buit.
-dpdk_driver_classes = ['common',
-	       'bus',
-	       'mempool', # depends on common and bus.
-	       'net',     # depends on common, bus, mempool
-	       'raw',     # depends on common, bus and net.
-	       'crypto',  # depends on common, bus and mempool (net in future).
-	       'compress', # depends on common, bus, mempool.
-	       'regex', # depends on common, bus, regexdev.
-	       'vdpa',    # depends on common, bus and mempool.
-	       'event',   # depends on common, bus, mempool and net.
-	       'baseband'] # depends on common and bus.
+dpdk_driver_classes = [
+	'common',
+	'bus',
+	'mempool', # depends on common and bus.
+	'net',     # depends on common, bus, mempool
+	'raw',     # depends on common, bus and net.
+	'crypto',  # depends on common, bus and mempool (net in future).
+	'compress', # depends on common, bus, mempool.
+	'regex', # depends on common, bus, regexdev.
+	'vdpa',    # depends on common, bus and mempool.
+	'event',   # depends on common, bus, mempool and net.
+	'baseband', # depends on common and bus.
+]
 
 disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
-- 
2.25.4

Re: [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list

[Bruce Richardson]

On Thu, Jul 23, 2020 at 11:09:02PM +0300, Parav Pandit wrote:
> From: Thomas Monjalon <thomas@monjalon.net>
> 
> Define each sub-directory on its own line ended with a comma,
> and use a simple indent.
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
>  drivers/meson.build | 24 +++++++++++++-----------
>  1 file changed, 13 insertions(+), 11 deletions(-)
> 
Thanks. Although we normally don't like doing cleanups for whitespace, this
is a small-enough change and worth doing, since the existing indentation
was just weird.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list

[David Marchand]

On Thu, Jul 23, 2020 at 10:10 PM Parav Pandit <parav@mellanox.com> wrote:
>
> From: Thomas Monjalon <thomas@monjalon.net>
>
> Define each sub-directory on its own line ended with a comma,
> and use a simple indent.
>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
>  drivers/meson.build | 24 +++++++++++++-----------
>  1 file changed, 13 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/meson.build b/drivers/meson.build
> index e76ebddfa..e2aeba931 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -2,17 +2,19 @@
>  # Copyright(c) 2017-2019 Intel Corporation
>
>  # Defines the order in which the drivers are buit.
> -dpdk_driver_classes = ['common',
> -              'bus',
> -              'mempool', # depends on common and bus.
> -              'net',     # depends on common, bus, mempool
> -              'raw',     # depends on common, bus and net.
> -              'crypto',  # depends on common, bus and mempool (net in future).
> -              'compress', # depends on common, bus, mempool.
> -              'regex', # depends on common, bus, regexdev.
> -              'vdpa',    # depends on common, bus and mempool.
> -              'event',   # depends on common, bus, mempool and net.
> -              'baseband'] # depends on common and bus.
> +dpdk_driver_classes = [
> +       'common',
> +       'bus',
> +       'mempool', # depends on common and bus.
> +       'net',     # depends on common, bus, mempool
> +       'raw',     # depends on common, bus and net.
> +       'crypto',  # depends on common, bus and mempool (net in future).
> +       'compress', # depends on common, bus, mempool.
> +       'regex', # depends on common, bus, regexdev.
> +       'vdpa',    # depends on common, bus and mempool.
> +       'event',   # depends on common, bus, mempool and net.
> +       'baseband', # depends on common and bus.
> +]
>
>  disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
>                 ).stdout().split()
> --
> 2.25.4
>

Reviewed-by: David Marchand <david.marchand@redhat.com>


-- 
David Marchand

[dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Drivers dependencies are evaluated in the order defined per
their parent directory (also called class).
This strict ordering prevent from having 2 different drivers
of the same class with different dependencies ordering.
This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci,
while drivers/bus/dpaa depends on drivers/common/dpaax.
Having a strict ordering between directories bus and common
is too much restrictive.

That's why it is made possible to have a more fine-grain directory list,
adding a driver sub-directory in the list.
In this case, the isolated driver must be removed from its class list,
and added directly in drivers/meson.build.
Also, the per-class variables must be duplicated in the isolated driver,
because the call "subdir(class)" is skipped in the isolated driver case.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
 drivers/meson.build | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e2aeba931..e6d0409aa 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-# Defines the order in which the drivers are buit.
-dpdk_driver_classes = [
+# Defines the order of dependencies evaluation
+subdirs = [
 	'common',
 	'bus',
 	'mempool', # depends on common and bus.
@@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation')
 	default_cflags += '-Wno-format-truncation'
 endif
 
-foreach class:dpdk_driver_classes
+foreach subpath:subdirs
 	drivers = []
 	std_deps = []
 	config_flag_fmt = '' # format string used to set the value in dpdk_conf
@@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes
 	                     # the library, the dependency and to find the
 	                     # version file for linking
 
-	subdir(class)
-	class_drivers = []
+	# subpath can be either "class" or "class/driver"
+	if subpath.contains('/')
+		driver_path = subpath.split('/')
+		class = driver_path[0]
+		drivers += driver_path[1]
+	else
+		class = subpath
+		subdir(class)
+	endif
+
+	# save class name on first occurence
+	if not dpdk_driver_classes.contains(class)
+		dpdk_driver_classes += class
+	endif
+	# get already enabled drivers of the same class
+	enabled_drivers = get_variable(class + '_drivers', [])
 
 	foreach drv:drivers
 		drv_path = join_paths(class, drv)
@@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes
 						'_disable_reason', reason)
 			endif
 		else
-			class_drivers += name
+			enabled_drivers += name
 
 			if fmt_name == ''
 				fmt_name = name
@@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes
 		endif # build
 	endforeach
 
-	set_variable(class + '_drivers', class_drivers)
+	set_variable(class + '_drivers', enabled_drivers)
 endforeach
-- 
2.25.4

Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order

[Bruce Richardson]

On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote:
> From: Thomas Monjalon <thomas@monjalon.net>
> 
> Drivers dependencies are evaluated in the order defined per
> their parent directory (also called class).
> This strict ordering prevent from having 2 different drivers
> of the same class with different dependencies ordering.
> This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci,
> while drivers/bus/dpaa depends on drivers/common/dpaax.
> Having a strict ordering between directories bus and common
> is too much restrictive.
> 
> That's why it is made possible to have a more fine-grain directory list,
> adding a driver sub-directory in the list.
> In this case, the isolated driver must be removed from its class list,
> and added directly in drivers/meson.build.
> Also, the per-class variables must be duplicated in the isolated driver,
> because the call "subdir(class)" is skipped in the isolated driver case.
> 
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

The commit log above has some strange word-wrapping, and occasionally
strange phrasing. I think it could be slightly reworded, perhaps as:

  Drivers dependencies are evaluated in the order defined per their parent
  directory (also called class). This strict ordering prevents from us
  from having pairs of drivers from two classes with different dependency
  ordering. For example, if the mlx5 common code depends on the pci bus
  driver, while the dpaax common code is itself a dependency of the dpaa
  bus driver.  Having a strict ordering between directories bus and common
  is too restrictive, as processing either common drivers or bus drivers
  first leads us to missing dependencies in this scenario.
   
  This patch makes it possible to have a more fine-grain directory list,
  adding a specific driver sub-directory in the top-level drivers
  subdirectory list. In this case, the isolated driver must also be removed
  from its class list, and the per-class variables must be duplicated in
  the isolated driver, because the call "subdir(class)" is skipped in the
  isolated driver case.
  

Apart from that, I think this is a good idea to give us some flexibility
in managing driver ordering which should help other drivers too -
perhaps QAT? Ideally, though, I'd like if we can limit the flexible
ordering to *only* common code, in which case we could move the
per-class variables for common to the top-level to prevent duplication,
and maybe even get rid of common/meson.build completely. That, however,
will depend on how much this feature gets used and by whom.

Therefore:
Review-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order

[Parav Pandit]

Hi Bruce,

> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Friday, July 24, 2020 4:37 PM
> 
> On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote:
> > From: Thomas Monjalon <thomas@monjalon.net>
> >
> > Drivers dependencies are evaluated in the order defined per their
> > parent directory (also called class).
> > This strict ordering prevent from having 2 different drivers of the
> > same class with different dependencies ordering.
> > This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci,
> > while drivers/bus/dpaa depends on drivers/common/dpaax.
> > Having a strict ordering between directories bus and common is too
> > much restrictive.
> >
> > That's why it is made possible to have a more fine-grain directory
> > list, adding a driver sub-directory in the list.
> > In this case, the isolated driver must be removed from its class list,
> > and added directly in drivers/meson.build.
> > Also, the per-class variables must be duplicated in the isolated
> > driver, because the call "subdir(class)" is skipped in the isolated driver case.
> >
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> 
> The commit log above has some strange word-wrapping, and occasionally
> strange phrasing. I think it could be slightly reworded, perhaps as:
> 
I updated the commit log as you suggested below along with RB, ack tag.
Thank you.

>   Drivers dependencies are evaluated in the order defined per their parent
>   directory (also called class). This strict ordering prevents from us
>   from having pairs of drivers from two classes with different dependency
>   ordering. For example, if the mlx5 common code depends on the pci bus
>   driver, while the dpaax common code is itself a dependency of the dpaa
>   bus driver.  Having a strict ordering between directories bus and common
>   is too restrictive, as processing either common drivers or bus drivers
>   first leads us to missing dependencies in this scenario.
> 
>   This patch makes it possible to have a more fine-grain directory list,
>   adding a specific driver sub-directory in the top-level drivers
>   subdirectory list. In this case, the isolated driver must also be removed
>   from its class list, and the per-class variables must be duplicated in
>   the isolated driver, because the call "subdir(class)" is skipped in the
>   isolated driver case.
> 
> 
> Apart from that, I think this is a good idea to give us some flexibility in
> managing driver ordering which should help other drivers too - perhaps QAT?
> Ideally, though, I'd like if we can limit the flexible ordering to *only* common
> code, in which case we could move the per-class variables for common to the
> top-level to prevent duplication, and maybe even get rid of
> common/meson.build completely. That, however, will depend on how much
> this feature gets used and by whom.
> 
> Therefore:
> Review-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order

[Thomas Monjalon]

24/07/2020 15:48, Parav Pandit:
> Hi Bruce,
> 
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Friday, July 24, 2020 4:37 PM
> > 
> > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote:
> > > From: Thomas Monjalon <thomas@monjalon.net>
> > >
> > > Drivers dependencies are evaluated in the order defined per their
> > > parent directory (also called class).
> > > This strict ordering prevent from having 2 different drivers of the
> > > same class with different dependencies ordering.
> > > This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci,
> > > while drivers/bus/dpaa depends on drivers/common/dpaax.
> > > Having a strict ordering between directories bus and common is too
> > > much restrictive.
> > >
> > > That's why it is made possible to have a more fine-grain directory
> > > list, adding a driver sub-directory in the list.
> > > In this case, the isolated driver must be removed from its class list,
> > > and added directly in drivers/meson.build.
> > > Also, the per-class variables must be duplicated in the isolated
> > > driver, because the call "subdir(class)" is skipped in the isolated driver case.
> > >
> > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > 
> > The commit log above has some strange word-wrapping, and occasionally
> > strange phrasing. I think it could be slightly reworded, perhaps as:
> > 
> I updated the commit log as you suggested below along with RB, ack tag.
> Thank you.
> 
> >   Drivers dependencies are evaluated in the order defined per their parent
> >   directory (also called class). This strict ordering prevents from us

Is "from us" too much?

> >   from having pairs of drivers from two classes with different dependency
> >   ordering. For example, if the mlx5 common code depends on the pci bus
> >   driver, while the dpaax common code is itself a dependency of the dpaa
> >   bus driver.  Having a strict ordering between directories bus and common
> >   is too restrictive, as processing either common drivers or bus drivers
> >   first leads us to missing dependencies in this scenario.
> > 
> >   This patch makes it possible to have a more fine-grain directory list,
> >   adding a specific driver sub-directory in the top-level drivers
> >   subdirectory list. In this case, the isolated driver must also be removed
> >   from its class list, and the per-class variables must be duplicated in
> >   the isolated driver, because the call "subdir(class)" is skipped in the
> >   isolated driver case.
> > 
> > 
> > Apart from that, I think this is a good idea to give us some flexibility in
> > managing driver ordering which should help other drivers too - perhaps QAT?
> > Ideally, though, I'd like if we can limit the flexible ordering to *only* common
> > code, in which case we could move the per-class variables for common to the
> > top-level to prevent duplication, and maybe even get rid of
> > common/meson.build completely. That, however, will depend on how much
> > this feature gets used and by whom.

For now it is used only to have common/mlx5 isolated of the rest
of common drivers, as an exception.
I think it is good to keep common/meson.build
until there are more exceptions than the rest.

Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order

[Bruce Richardson]

On Fri, Jul 24, 2020 at 03:54:33PM +0200, Thomas Monjalon wrote:
> 24/07/2020 15:48, Parav Pandit:
> > Hi Bruce,
> > 
> > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > Sent: Friday, July 24, 2020 4:37 PM
> > > 
> > > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote:
> > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > >
> > > > Drivers dependencies are evaluated in the order defined per their
> > > > parent directory (also called class).
> > > > This strict ordering prevent from having 2 different drivers of the
> > > > same class with different dependencies ordering.
> > > > This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci,
> > > > while drivers/bus/dpaa depends on drivers/common/dpaax.
> > > > Having a strict ordering between directories bus and common is too
> > > > much restrictive.
> > > >
> > > > That's why it is made possible to have a more fine-grain directory
> > > > list, adding a driver sub-directory in the list.
> > > > In this case, the isolated driver must be removed from its class list,
> > > > and added directly in drivers/meson.build.
> > > > Also, the per-class variables must be duplicated in the isolated
> > > > driver, because the call "subdir(class)" is skipped in the isolated driver case.
> > > >
> > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > > 
> > > The commit log above has some strange word-wrapping, and occasionally
> > > strange phrasing. I think it could be slightly reworded, perhaps as:
> > > 
> > I updated the commit log as you suggested below along with RB, ack tag.
> > Thank you.
> > 
> > >   Drivers dependencies are evaluated in the order defined per their parent
> > >   directory (also called class). This strict ordering prevents from us
> 
> Is "from us" too much?

The word "from" is incorrect, anyway, it should just be "prevents us".
Evidently I didn't proofread my own text well enough. :-(

> 
> > >   from having pairs of drivers from two classes with different dependency
> > >   ordering. For example, if the mlx5 common code depends on the pci bus
> > >   driver, while the dpaax common code is itself a dependency of the dpaa
> > >   bus driver.  Having a strict ordering between directories bus and common
> > >   is too restrictive, as processing either common drivers or bus drivers
> > >   first leads us to missing dependencies in this scenario.
> > > 
> > >   This patch makes it possible to have a more fine-grain directory list,
> > >   adding a specific driver sub-directory in the top-level drivers
> > >   subdirectory list. In this case, the isolated driver must also be removed
> > >   from its class list, and the per-class variables must be duplicated in
> > >   the isolated driver, because the call "subdir(class)" is skipped in the
> > >   isolated driver case.
> > > 
> > > 
> > > Apart from that, I think this is a good idea to give us some flexibility in
> > > managing driver ordering which should help other drivers too - perhaps QAT?
> > > Ideally, though, I'd like if we can limit the flexible ordering to *only* common
> > > code, in which case we could move the per-class variables for common to the
> > > top-level to prevent duplication, and maybe even get rid of
> > > common/meson.build completely. That, however, will depend on how much
> > > this feature gets used and by whom.
> 
> For now it is used only to have common/mlx5 isolated of the rest
> of common drivers, as an exception.
> I think it is good to keep common/meson.build
> until there are more exceptions than the rest.
> 

+1 agreed. 
Just suggesting possible future changes depending on how things
go, and also seeding the idea that any out-of-order build processing should
really be limited to the common folder.

/Bruce

Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order

[Parav Pandit]

> From: Bruce Richardson <bruce.richardson@intel.com>
> Sent: Friday, July 24, 2020 8:20 PM
> 
> On Fri, Jul 24, 2020 at 03:54:33PM +0200, Thomas Monjalon wrote:
> > 24/07/2020 15:48, Parav Pandit:
> > > Hi Bruce,
> > >
> > > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > > Sent: Friday, July 24, 2020 4:37 PM
> > > >
> > > > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote:
> > > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > >
> > > > > Drivers dependencies are evaluated in the order defined per
> > > > > their parent directory (also called class).
> > > > > This strict ordering prevent from having 2 different drivers of
> > > > > the same class with different dependencies ordering.
> > > > > This problem occurs if drivers/common/mlx5 depends on
> > > > > drivers/bus/pci, while drivers/bus/dpaa depends on
> drivers/common/dpaax.
> > > > > Having a strict ordering between directories bus and common is
> > > > > too much restrictive.
> > > > >
> > > > > That's why it is made possible to have a more fine-grain
> > > > > directory list, adding a driver sub-directory in the list.
> > > > > In this case, the isolated driver must be removed from its class
> > > > > list, and added directly in drivers/meson.build.
> > > > > Also, the per-class variables must be duplicated in the isolated
> > > > > driver, because the call "subdir(class)" is skipped in the isolated driver
> case.
> > > > >
> > > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > > >
> > > > The commit log above has some strange word-wrapping, and
> > > > occasionally strange phrasing. I think it could be slightly reworded,
> perhaps as:
> > > >
> > > I updated the commit log as you suggested below along with RB, ack tag.
> > > Thank you.
> > >
> > > >   Drivers dependencies are evaluated in the order defined per their
> parent
> > > >   directory (also called class). This strict ordering prevents
> > > > from us
> >
> > Is "from us" too much?
> 
> The word "from" is incorrect, anyway, it should just be "prevents us".
> Evidently I didn't proofread my own text well enough. :-(

In v10 I rephase it as "strict ordering prevents from having pairs of drivers...".

Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order

[Bruce Richardson]

On Fri, Jul 24, 2020 at 03:17:38PM +0000, Parav Pandit wrote:
> 
> 
> > From: Bruce Richardson <bruce.richardson@intel.com>
> > Sent: Friday, July 24, 2020 8:20 PM
> > 
> > On Fri, Jul 24, 2020 at 03:54:33PM +0200, Thomas Monjalon wrote:
> > > 24/07/2020 15:48, Parav Pandit:
> > > > Hi Bruce,
> > > >
> > > > > From: Bruce Richardson <bruce.richardson@intel.com>
> > > > > Sent: Friday, July 24, 2020 4:37 PM
> > > > >
> > > > > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote:
> > > > > > From: Thomas Monjalon <thomas@monjalon.net>
> > > > > >
> > > > > > Drivers dependencies are evaluated in the order defined per
> > > > > > their parent directory (also called class).
> > > > > > This strict ordering prevent from having 2 different drivers of
> > > > > > the same class with different dependencies ordering.
> > > > > > This problem occurs if drivers/common/mlx5 depends on
> > > > > > drivers/bus/pci, while drivers/bus/dpaa depends on
> > drivers/common/dpaax.
> > > > > > Having a strict ordering between directories bus and common is
> > > > > > too much restrictive.
> > > > > >
> > > > > > That's why it is made possible to have a more fine-grain
> > > > > > directory list, adding a driver sub-directory in the list.
> > > > > > In this case, the isolated driver must be removed from its class
> > > > > > list, and added directly in drivers/meson.build.
> > > > > > Also, the per-class variables must be duplicated in the isolated
> > > > > > driver, because the call "subdir(class)" is skipped in the isolated driver
> > case.
> > > > > >
> > > > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > > > >
> > > > > The commit log above has some strange word-wrapping, and
> > > > > occasionally strange phrasing. I think it could be slightly reworded,
> > perhaps as:
> > > > >
> > > > I updated the commit log as you suggested below along with RB, ack tag.
> > > > Thank you.
> > > >
> > > > >   Drivers dependencies are evaluated in the order defined per their
> > parent
> > > > >   directory (also called class). This strict ordering prevents
> > > > > from us
> > >
> > > Is "from us" too much?
> > 
> > The word "from" is incorrect, anyway, it should just be "prevents us".
> > Evidently I didn't proofread my own text well enough. :-(
> 
> In v10 I rephase it as "strict ordering prevents from having pairs of drivers...".

Great, thanks.

Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order

[David Marchand]

On Thu, Jul 23, 2020 at 10:10 PM Parav Pandit <parav@mellanox.com> wrote:
>
> From: Thomas Monjalon <thomas@monjalon.net>
>
> Drivers dependencies are evaluated in the order defined per
> their parent directory (also called class).
> This strict ordering prevent from having 2 different drivers
> of the same class with different dependencies ordering.
> This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci,
> while drivers/bus/dpaa depends on drivers/common/dpaax.
> Having a strict ordering between directories bus and common
> is too much restrictive.
>
> That's why it is made possible to have a more fine-grain directory list,
> adding a driver sub-directory in the list.
> In this case, the isolated driver must be removed from its class list,
> and added directly in drivers/meson.build.
> Also, the per-class variables must be duplicated in the isolated driver,
> because the call "subdir(class)" is skipped in the isolated driver case.
>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
>  drivers/meson.build | 28 +++++++++++++++++++++-------
>  1 file changed, 21 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/meson.build b/drivers/meson.build
> index e2aeba931..e6d0409aa 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -1,8 +1,8 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright(c) 2017-2019 Intel Corporation
>
> -# Defines the order in which the drivers are buit.
> -dpdk_driver_classes = [
> +# Defines the order of dependencies evaluation
> +subdirs = [
>         'common',
>         'bus',
>         'mempool', # depends on common and bus.
> @@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation')
>         default_cflags += '-Wno-format-truncation'
>  endif
>
> -foreach class:dpdk_driver_classes
> +foreach subpath:subdirs
>         drivers = []
>         std_deps = []
>         config_flag_fmt = '' # format string used to set the value in dpdk_conf
> @@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes
>                              # the library, the dependency and to find the
>                              # version file for linking
>
> -       subdir(class)
> -       class_drivers = []
> +       # subpath can be either "class" or "class/driver"
> +       if subpath.contains('/')
> +               driver_path = subpath.split('/')
> +               class = driver_path[0]
> +               drivers += driver_path[1]
> +       else
> +               class = subpath
> +               subdir(class)
> +       endif
> +
> +       # save class name on first occurence
> +       if not dpdk_driver_classes.contains(class)
> +               dpdk_driver_classes += class
> +       endif
> +       # get already enabled drivers of the same class
> +       enabled_drivers = get_variable(class + '_drivers', [])
>
>         foreach drv:drivers
>                 drv_path = join_paths(class, drv)
> @@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes
>                                                 '_disable_reason', reason)
>                         endif
>                 else
> -                       class_drivers += name
> +                       enabled_drivers += name
>
>                         if fmt_name == ''
>                                 fmt_name = name
> @@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes
>                 endif # build
>         endforeach
>
> -       set_variable(class + '_drivers', class_drivers)
> +       set_variable(class + '_drivers', enabled_drivers)
>  endforeach
> --
> 2.25.4
>

Reviewed-by: David Marchand <david.marchand@redhat.com>


-- 
David Marchand

[dpdk-dev] [PATCH v8 04/10] common/mlx5: fix void parameters in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index c4f9b006a..734ace2a3 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.25.4

[dpdk-dev] [PATCH v8 05/10] regex/mlx5: fix segmentation fault during error unwinding

[Parav Pandit]

When fail to initialize the device, avoid segmentation fault while
accessing unintialized priv.

Fixes: cfc672a90b74 ("regex/mlx5: support probing")
Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/regex/mlx5/mlx5_regex.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 1ca5bfe9b..36ae9f809 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -137,17 +137,17 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	} else if (!attr.regex || attr.regexp_num_of_engines == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
 			"old FW/OFED version?");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	}
 	if (mlx5_regex_engines_status(ctx, 2)) {
 		DRV_LOG(ERR, "RegEx engine error.");
 		rte_errno = ENOMEM;
-		goto error;
+		goto dev_error;
 	}
 	priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
 			   RTE_CACHE_LINE_SIZE);
@@ -200,6 +200,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		mlx5_glue->devx_free_uar(priv->uar);
 	if (priv->regexdev)
 		rte_regexdev_unregister(priv->regexdev);
+dev_error:
 	if (ctx)
 		mlx5_glue->close_device(ctx);
 	if (priv)
-- 
2.25.4

[dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority

[Parav Pandit]

mlx5_common is shared library between mlx5 net, VDPA and regex PMD.
It is better to use common initialization helper instead of using
RTE_INIT_CLASS priority.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c               | 13 +++++++++++--
 drivers/common/mlx5/mlx5_common.h               |  3 +++
 drivers/common/mlx5/rte_common_mlx5_version.map |  1 +
 drivers/net/mlx5/mlx5.c                         |  1 +
 drivers/regex/mlx5/mlx5_regex.c                 |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.c                   |  1 +
 6 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 1a67a1b16..2b336bb2d 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 		rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE);
 }
 
+static bool mlx5_common_initialized;
+
 /**
- * Initialization routine for run-time dependency on glue library.
+ * One time innitialization routine for run-time dependency on glue library
+ * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module,
+ * must invoke in its constructor.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+void
+mlx5_common_init(void)
 {
+	if (mlx5_common_initialized)
+		return;
+
 	mlx5_glue_constructor();
+	mlx5_common_initialized = true;
 }
 
 /**
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a811eb6c9..ebe4e9ced 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id,
 			 uint64_t offset);
 extern uint8_t haswell_broadwell_cpu;
 
+__rte_internal
+void mlx5_common_init(void);
+
 #endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 132a0695f..65f25252a 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -3,6 +3,7 @@ INTERNAL {
 
 	mlx5_class_get;
 
+	mlx5_common_init;
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 647ada339..037703d2e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_pmd_init)
 {
+	mlx5_common_init();
 	/* Build the static tables for Verbs conversion. */
 	mlx5_set_ptype_table();
 	mlx5_set_cksum_table();
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 36ae9f809..4e0367052 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = {
 
 RTE_INIT(rte_mlx5_regex_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_regex_driver);
 }
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 67e77b11a..85dbcf956 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_vdpa_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_vdpa_driver);
 }
-- 
2.25.4

Re: [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority

[David Marchand]

On Thu, Jul 23, 2020 at 10:10 PM Parav Pandit <parav@mellanox.com> wrote:
>
> mlx5_common is shared library between mlx5 net, VDPA and regex PMD.
> It is better to use common initialization helper instead of using
> RTE_INIT_CLASS priority.

RTE_INIT_CLASS does not exist, I suppose you meant RTE_PRIORITY_CLASS.

Suggested-by: David Marchand <david.marchand@redhat.com>

> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/common/mlx5/mlx5_common.c               | 13 +++++++++++--
>  drivers/common/mlx5/mlx5_common.h               |  3 +++
>  drivers/common/mlx5/rte_common_mlx5_version.map |  1 +
>  drivers/net/mlx5/mlx5.c                         |  1 +
>  drivers/regex/mlx5/mlx5_regex.c                 |  1 +
>  drivers/vdpa/mlx5/mlx5_vdpa.c                   |  1 +
>  6 files changed, 18 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
> index 1a67a1b16..2b336bb2d 100644
> --- a/drivers/common/mlx5/mlx5_common.c
> +++ b/drivers/common/mlx5/mlx5_common.c
> @@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
>                 rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE);
>  }
>
> +static bool mlx5_common_initialized;
> +
>  /**
> - * Initialization routine for run-time dependency on glue library.
> + * One time innitialization routine for run-time dependency on glue library
> + * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module,
> + * must invoke in its constructor.
>   */
> -RTE_INIT_PRIO(mlx5_glue_init, CLASS)
> +void
> +mlx5_common_init(void)
>  {
> +       if (mlx5_common_initialized)
> +               return;
> +
>         mlx5_glue_constructor();
> +       mlx5_common_initialized = true;
>  }
>
>  /**
> diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
> index a811eb6c9..ebe4e9ced 100644
> --- a/drivers/common/mlx5/mlx5_common.h
> +++ b/drivers/common/mlx5/mlx5_common.h
> @@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id,
>                          uint64_t offset);
>  extern uint8_t haswell_broadwell_cpu;
>
> +__rte_internal
> +void mlx5_common_init(void);
> +
>  #endif /* RTE_PMD_MLX5_COMMON_H_ */
> diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
> index 132a0695f..65f25252a 100644
> --- a/drivers/common/mlx5/rte_common_mlx5_version.map
> +++ b/drivers/common/mlx5/rte_common_mlx5_version.map
> @@ -3,6 +3,7 @@ INTERNAL {
>
>         mlx5_class_get;
>
> +       mlx5_common_init;
>         mlx5_common_verbs_reg_mr;
>         mlx5_common_verbs_dereg_mr;
>
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
> index 647ada339..037703d2e 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
>   */
>  RTE_INIT(rte_mlx5_pmd_init)
>  {
> +       mlx5_common_init();
>         /* Build the static tables for Verbs conversion. */
>         mlx5_set_ptype_table();
>         mlx5_set_cksum_table();
> diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
> index 36ae9f809..4e0367052 100644
> --- a/drivers/regex/mlx5/mlx5_regex.c
> +++ b/drivers/regex/mlx5/mlx5_regex.c
> @@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = {
>
>  RTE_INIT(rte_mlx5_regex_init)
>  {
> +       mlx5_common_init();
>         if (mlx5_glue)
>                 rte_pci_register(&mlx5_regex_driver);
>  }
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
> index 67e77b11a..85dbcf956 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
> @@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
>   */
>  RTE_INIT(rte_mlx5_vdpa_init)
>  {
> +       mlx5_common_init();
>         if (mlx5_glue)
>                 rte_pci_register(&mlx5_vdpa_driver);
>  }
> --
> 2.25.4
>

Reviewed-by: David Marchand <david.marchand@redhat.com>


-- 
David Marchand

Re: [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority

[Parav Pandit]

> From: David Marchand <david.marchand@redhat.com>
> Sent: Friday, July 24, 2020 7:15 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: dev <dev@dpdk.org>; Gaetan Rivet <grive@u256.net>; Yigit, Ferruh
> <ferruh.yigit@intel.com>; Thomas Monjalon <thomas@monjalon.net>;
> Raslan Darawsheh <rasland@mellanox.com>; Ori Kam
> <orika@mellanox.com>; Matan Azrad <matan@mellanox.com>; Joyce Kong
> <joyce.kong@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class
> constructor priority
> 
> On Thu, Jul 23, 2020 at 10:10 PM Parav Pandit <parav@mellanox.com> wrote:
> >
> > mlx5_common is shared library between mlx5 net, VDPA and regex PMD.
> > It is better to use common initialization helper instead of using
> > RTE_INIT_CLASS priority.
> 
> RTE_INIT_CLASS does not exist, I suppose you meant RTE_PRIORITY_CLASS.
> 
You are right.
Correcting it and adding below tags.

> Suggested-by: David Marchand <david.marchand@redhat.com>
> 
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > ---
> >  drivers/common/mlx5/mlx5_common.c               | 13 +++++++++++--
> >  drivers/common/mlx5/mlx5_common.h               |  3 +++
> >  drivers/common/mlx5/rte_common_mlx5_version.map |  1 +
> >  drivers/net/mlx5/mlx5.c                         |  1 +
> >  drivers/regex/mlx5/mlx5_regex.c                 |  1 +
> >  drivers/vdpa/mlx5/mlx5_vdpa.c                   |  1 +
> >  6 files changed, 18 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/common/mlx5/mlx5_common.c
> > b/drivers/common/mlx5/mlx5_common.c
> > index 1a67a1b16..2b336bb2d 100644
> > --- a/drivers/common/mlx5/mlx5_common.c
> > +++ b/drivers/common/mlx5/mlx5_common.c
> > @@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
> >                 rte_log_set_level(mlx5_common_logtype,
> > RTE_LOG_NOTICE);  }
> >
> > +static bool mlx5_common_initialized;
> > +
> >  /**
> > - * Initialization routine for run-time dependency on glue library.
> > + * One time innitialization routine for run-time dependency on glue
> > + library
> > + * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common
> > + module,
> > + * must invoke in its constructor.
> >   */
> > -RTE_INIT_PRIO(mlx5_glue_init, CLASS)
> > +void
> > +mlx5_common_init(void)
> >  {
> > +       if (mlx5_common_initialized)
> > +               return;
> > +
> >         mlx5_glue_constructor();
> > +       mlx5_common_initialized = true;
> >  }
> >
> >  /**
> > diff --git a/drivers/common/mlx5/mlx5_common.h
> > b/drivers/common/mlx5/mlx5_common.h
> > index a811eb6c9..ebe4e9ced 100644
> > --- a/drivers/common/mlx5/mlx5_common.h
> > +++ b/drivers/common/mlx5/mlx5_common.h
> > @@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct
> mlx5_dbr_page_list *head, uint32_t umem_id,
> >                          uint64_t offset);  extern uint8_t
> > haswell_broadwell_cpu;
> >
> > +__rte_internal
> > +void mlx5_common_init(void);
> > +
> >  #endif /* RTE_PMD_MLX5_COMMON_H_ */
> > diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map
> > b/drivers/common/mlx5/rte_common_mlx5_version.map
> > index 132a0695f..65f25252a 100644
> > --- a/drivers/common/mlx5/rte_common_mlx5_version.map
> > +++ b/drivers/common/mlx5/rte_common_mlx5_version.map
> > @@ -3,6 +3,7 @@ INTERNAL {
> >
> >         mlx5_class_get;
> >
> > +       mlx5_common_init;
> >         mlx5_common_verbs_reg_mr;
> >         mlx5_common_verbs_dereg_mr;
> >
> > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index
> > 647ada339..037703d2e 100644
> > --- a/drivers/net/mlx5/mlx5.c
> > +++ b/drivers/net/mlx5/mlx5.c
> > @@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype,
> pmd.net.mlx5, NOTICE)
> >   */
> >  RTE_INIT(rte_mlx5_pmd_init)
> >  {
> > +       mlx5_common_init();
> >         /* Build the static tables for Verbs conversion. */
> >         mlx5_set_ptype_table();
> >         mlx5_set_cksum_table();
> > diff --git a/drivers/regex/mlx5/mlx5_regex.c
> > b/drivers/regex/mlx5/mlx5_regex.c index 36ae9f809..4e0367052 100644
> > --- a/drivers/regex/mlx5/mlx5_regex.c
> > +++ b/drivers/regex/mlx5/mlx5_regex.c
> > @@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = {
> >
> >  RTE_INIT(rte_mlx5_regex_init)
> >  {
> > +       mlx5_common_init();
> >         if (mlx5_glue)
> >                 rte_pci_register(&mlx5_regex_driver);
> >  }
> > diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c
> > b/drivers/vdpa/mlx5/mlx5_vdpa.c index 67e77b11a..85dbcf956 100644
> > --- a/drivers/vdpa/mlx5/mlx5_vdpa.c
> > +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
> > @@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype,
> pmd.vdpa.mlx5, NOTICE)
> >   */
> >  RTE_INIT(rte_mlx5_vdpa_init)
> >  {
> > +       mlx5_common_init();
> >         if (mlx5_glue)
> >                 rte_pci_register(&mlx5_vdpa_driver);
> >  }
> > --
> > 2.25.4
> >
> 
> Reviewed-by: David Marchand <david.marchand@redhat.com>
> 
> 
> --
> David Marchand

[dpdk-dev] [PATCH v8 07/10] common/mlx5: change class values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.h | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index ebe4e9ced..864e82ff0 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -208,10 +209,10 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
-	MLX5_CLASS_REGEX,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT64(0),
+	MLX5_CLASS_VDPA = RTE_BIT64(1),
+	MLX5_CLASS_REGEX = RTE_BIT64(2),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
-- 
2.25.4

[dpdk-dev] [PATCH v8 08/10] common/mlx5: introduce layer to support multiple class drivers

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Add generic mlx5 PCI PMD layer as part of existing common_mlx5
module. This enables multiple classes (net, regex, vdpa) PMDs
to be supported at same time.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   1 +
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |   2 +
 drivers/common/mlx5/mlx5_common_pci.c         | 541 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   2 +
 drivers/meson.build                           |   1 +
 10 files changed, 638 insertions(+), 11 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

diff --git a/drivers/Makefile b/drivers/Makefile
index 1551272ef..7f06162dc 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -8,8 +8,12 @@ DIRS-y += bus
 DEPDIRS-bus := common
 DIRS-y += mempool
 DEPDIRS-mempool := common bus
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
+DIRS-y += common/mlx5
+DEPDIRS-common/mlx5 := bus
+endif
 DIRS-y += net
-DEPDIRS-net := common bus mempool
+DEPDIRS-net := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband
 DEPDIRS-baseband := common bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto
@@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress
 DEPDIRS-compress := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex
-DEPDIRS-regex := common bus
+DEPDIRS-regex := common bus common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa
-DEPDIRS-vdpa := common bus mempool
+DEPDIRS-vdpa := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event
 DEPDIRS-event := common bus mempool net crypto
 DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index cbc71077c..cfb6b4dc8 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y)))
 DIRS-y += iavf
 endif
 
-ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
-DIRS-y += mlx5
-endif
-
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/common/meson.build b/drivers/common/meson.build
index 5db7e29b1..9ed4c04ba 100644
--- a/drivers/common/meson.build
+++ b/drivers/common/meson.build
@@ -6,6 +6,6 @@ if is_windows
 endif
 
 std_deps = ['eal']
-drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat']
+drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat']
 config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
 driver_name_fmt = 'rte_common_@0@'
diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile
index 6b89a6c85..eaabb8c10 100644
--- a/drivers/common/mlx5/Makefile
+++ b/drivers/common/mlx5/Makefile
@@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c
 SRCS-y += mlx5_common_mp.c
 SRCS-y += mlx5_common_mr.c
 SRCS-y += mlx5_malloc.c
+SRCS-y += mlx5_common_pci.c
 ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y)
 INSTALL-y-lib += $(LIB_GLUE)
 endif
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index 70e2c1c32..8e5608703 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -1,19 +1,22 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2019 Mellanox Technologies, Ltd
 
-if not (is_linux or is_windows)
+if not is_linux
 	build = false
-	reason = 'only supported on Linux and Windows'
+	reason = 'only supported on Linux'
 	subdir_done()
 endif
 
-deps += ['hash', 'pci', 'net', 'eal', 'kvargs']
+config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
+driver_name_fmt = 'rte_common_@0@'
+deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs']
 sources += files(
 	'mlx5_devx_cmds.c',
 	'mlx5_common.c',
 	'mlx5_common_mp.c',
 	'mlx5_common_mr.c',
 	'mlx5_malloc.c',
+	'mlx5_common_pci.c',
 )
 
 cflags_options = [
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 2b336bb2d..fd818ef24 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -14,6 +14,7 @@
 #include "mlx5_common_os.h"
 #include "mlx5_common_utils.h"
 #include "mlx5_malloc.h"
+#include "mlx5_common_pci.h"
 
 int mlx5_common_logtype;
 
@@ -100,6 +101,7 @@ mlx5_common_init(void)
 		return;
 
 	mlx5_glue_constructor();
+	mlx5_common_pci_init();
 	mlx5_common_initialized = true;
 }
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
new file mode 100644
index 000000000..b42f7469f
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -0,0 +1,541 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies Ltd
+ */
+
+#include <stdlib.h>
+#include <rte_malloc.h>
+#include "mlx5_common_utils.h"
+#include "mlx5_common_pci.h"
+
+struct mlx5_pci_device {
+      struct rte_pci_device *pci_dev;
+      TAILQ_ENTRY(mlx5_pci_device) next;
+      uint32_t classes_loaded;
+};
+
+/* Head of list of drivers. */
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int driver_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .driver_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .driver_class = MLX5_CLASS_NET },
+	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	MLX5_CLASS_REGEX,
+	/* New class combination should be added here.
+	 * For example a new multi class device combination
+	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
+	 */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].driver_class;
+	}
+	return -EINVAL;
+}
+
+static struct mlx5_pci_driver *
+driver_get(uint32_t class)
+{
+	struct mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->driver_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+static bool
+mlx5_bus_match(const struct mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = driver_get(RTE_BIT64(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT64(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT64(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv,
+	      struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->driver_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->driver_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->driver_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple drivers for a PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		      struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more drivers for a PCI device.
+ *
+ * This function removes all drivers probed for a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_common_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload drivers. */
+	ret = drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+			uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver = NULL;
+	struct mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_map) {
+			ret = driver->pci_driver.dma_map(pci_dev, addr,
+							 iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == driver)
+			break;
+		if (device_class_enabled(dev, temp->driver_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+			  uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_unmap) {
+			local_ret = driver->pci_driver.dma_unmap(pci_dev, addr,
+								 iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5 drivers. */
+static struct rte_pci_id *mlx5_pci_id_table;
+
+static struct rte_pci_driver mlx5_pci_driver = {
+	.driver = {
+		.name = "mlx5_pci",
+	},
+	.probe = mlx5_common_pci_probe,
+	.remove = mlx5_common_pci_remove,
+	.dma_map = mlx5_common_pci_dma_map,
+	.dma_unmap = mlx5_common_pci_dma_unmap,
+};
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table,
+	      int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == table[i].device_id &&
+		    id->vendor_id == table[i].vendor_id &&
+		    id->subsystem_vendor_id == table[i].subsystem_vendor_id &&
+		    id->subsystem_device_id == table[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(struct rte_pci_id *new_table, int *next_idx,
+	      const struct rte_pci_id *id_table)
+{
+	/* Traverse the id_table, check if entry exists in new_table;
+	 * Add non duplicate entries to new table.
+	 */
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, new_table, *next_idx)) {
+			/* New entry; add to the table. */
+			new_table[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_update(const struct rte_pci_id *driver_id_table)
+{
+	const struct rte_pci_id *id_iter;
+	struct rte_pci_id *updated_table;
+	struct rte_pci_id *old_table;
+	int num_ids = 0;
+	int i = 0;
+
+	old_table = mlx5_pci_id_table;
+	if (old_table)
+		num_ids = pci_id_table_size_get(old_table);
+	num_ids += pci_id_table_size_get(driver_id_table);
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	updated_table = calloc(num_ids, sizeof(*updated_table));
+	if (!updated_table)
+		return -ENOMEM;
+	if (TAILQ_EMPTY(&drv_list)) {
+		/* Copy the first driver's ID table. */
+		for (id_iter = driver_id_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+	} else {
+		/* First copy existing table entries. */
+		for (id_iter = old_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+		/* New id to be added at the end of current ID table. */
+		pci_id_insert(updated_table, &i, driver_id_table);
+	}
+	/* Terminate table with empty entry. */
+	updated_table[i].vendor_id = 0;
+	mlx5_pci_driver.id_table = updated_table;
+	mlx5_pci_id_table = updated_table;
+	if (old_table)
+		free(old_table);
+	return 0;
+}
+
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver)
+{
+	int ret;
+
+	ret = pci_ids_table_update(driver->pci_driver.id_table);
+	if (ret)
+		return;
+	mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags;
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
+
+void mlx5_common_pci_init(void)
+{
+	const struct rte_pci_id empty_table[] = {
+		{
+			.vendor_id = 0
+		},
+	};
+
+	/* All mlx5 PMDs constructor runs at same priority. So any of the PMD
+	 * including this one can register the PCI table first. If any other
+	 * PMD(s) have registered the PCI ID table, No need to register an empty
+	 * default one.
+	 */
+	if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table))
+			return;
+	rte_pci_register(&mlx5_pci_driver);
+}
+
+RTE_FINI(mlx5_common_pci_finish)
+{
+	if (mlx5_pci_id_table != NULL) {
+		/* Constructor doesn't register with PCI bus if it failed
+		 * to build the table.
+		 */
+		rte_pci_unregister(&mlx5_pci_driver);
+		free(mlx5_pci_id_table);
+	}
+}
+RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__);
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
new file mode 100644
index 000000000..41b73e17a
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _MLX5_COMMON_PCI_H_
+#define _MLX5_COMMON_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Driver Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This layer enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci driver.
+ *
+ * -----------    ------------    -------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5    |
+ * | net pmd |    | vdpa pmd |    | regex pmd |
+ * -----------    ------------    -------------
+ *      \              |                 /
+ *       \             |                /
+ *        \       --------------       /
+ *         \______|   mlx5     |_____ /
+ *                | pci common |
+ *                --------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+void mlx5_common_pci_init(void);
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t driver_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _MLX5_COMMON_PCI_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 65f25252a..73cf72548 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -91,5 +91,7 @@ INTERNAL {
 	mlx5_malloc;
 	mlx5_realloc;
 	mlx5_free;
+
+	mlx5_pci_driver_register;
 };
 
diff --git a/drivers/meson.build b/drivers/meson.build
index e6d0409aa..95df7ef1d 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -5,6 +5,7 @@
 subdirs = [
 	'common',
 	'bus',
+	'common/mlx5', # depends on bus.
 	'mempool', # depends on common and bus.
 	'net',     # depends on common, bus, mempool
 	'raw',     # depends on common, bus and net.
-- 
2.25.4

Re: [dpdk-dev] [PATCH v8 08/10] common/mlx5: introduce layer to support multiple class drivers

[David Marchand]

This is not a thorough review, just caught one issue for makefile and some nits.


On Thu, Jul 23, 2020 at 10:11 PM Parav Pandit <parav@mellanox.com> wrote:
>
> From: Thomas Monjalon <thomas@monjalon.net>
>
> Add generic mlx5 PCI PMD layer as part of existing common_mlx5
> module. This enables multiple classes (net, regex, vdpa) PMDs
> to be supported at same time.
>
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> Acked-by: Matan Azrad <matan@mellanox.com>
> ---
>  drivers/Makefile                              |  10 +-
>  drivers/common/Makefile                       |   4 -
>  drivers/common/meson.build                    |   2 +-
>  drivers/common/mlx5/Makefile                  |   1 +
>  drivers/common/mlx5/meson.build               |   9 +-
>  drivers/common/mlx5/mlx5_common.c             |   2 +
>  drivers/common/mlx5/mlx5_common_pci.c         | 541 ++++++++++++++++++
>  drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
>  .../common/mlx5/rte_common_mlx5_version.map   |   2 +
>  drivers/meson.build                           |   1 +
>  10 files changed, 638 insertions(+), 11 deletions(-)
>  create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
>  create mode 100644 drivers/common/mlx5/mlx5_common_pci.h
>
> diff --git a/drivers/Makefile b/drivers/Makefile
> index 1551272ef..7f06162dc 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -8,8 +8,12 @@ DIRS-y += bus
>  DEPDIRS-bus := common
>  DIRS-y += mempool
>  DEPDIRS-mempool := common bus
> +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
> +DIRS-y += common/mlx5
> +DEPDIRS-common/mlx5 := bus
> +endif
>  DIRS-y += net
> -DEPDIRS-net := common bus mempool
> +DEPDIRS-net := common bus mempool common/mlx5

Will it still work for people not building with mlx5?

I would have seen something like:
DEPDIRS-net/mlx5 := xx xx xx common/mlx5
Or maybe
DEPDIRS-net/mlx5 += common/mlx5


>  DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband
>  DEPDIRS-baseband := common bus mempool
>  DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto
> @@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool
>  DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress
>  DEPDIRS-compress := bus mempool
>  DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex
> -DEPDIRS-regex := common bus
> +DEPDIRS-regex := common bus common/mlx5

DEPDIRS-regex/mlx5 := xx xx xx common/mlx5

>  DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa
> -DEPDIRS-vdpa := common bus mempool
> +DEPDIRS-vdpa := common bus mempool common/mlx5

Idem.


>  DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event
>  DEPDIRS-event := common bus mempool net crypto
>  DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw
> diff --git a/drivers/common/Makefile b/drivers/common/Makefile
> index cbc71077c..cfb6b4dc8 100644
> --- a/drivers/common/Makefile
> +++ b/drivers/common/Makefile
> @@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y)))
>  DIRS-y += iavf
>  endif
>
> -ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
> -DIRS-y += mlx5
> -endif
> -
>  include $(RTE_SDK)/mk/rte.subdir.mk
> diff --git a/drivers/common/meson.build b/drivers/common/meson.build
> index 5db7e29b1..9ed4c04ba 100644
> --- a/drivers/common/meson.build
> +++ b/drivers/common/meson.build
> @@ -6,6 +6,6 @@ if is_windows
>  endif
>
>  std_deps = ['eal']
> -drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat']
> +drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat']
>  config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
>  driver_name_fmt = 'rte_common_@0@'
> diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile
> index 6b89a6c85..eaabb8c10 100644
> --- a/drivers/common/mlx5/Makefile
> +++ b/drivers/common/mlx5/Makefile
> @@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c
>  SRCS-y += mlx5_common_mp.c
>  SRCS-y += mlx5_common_mr.c
>  SRCS-y += mlx5_malloc.c
> +SRCS-y += mlx5_common_pci.c

Alphabetical order?


>  ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y)
>  INSTALL-y-lib += $(LIB_GLUE)
>  endif
> diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
> index 70e2c1c32..8e5608703 100644
> --- a/drivers/common/mlx5/meson.build
> +++ b/drivers/common/mlx5/meson.build
> @@ -1,19 +1,22 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright 2019 Mellanox Technologies, Ltd
>
> -if not (is_linux or is_windows)
> +if not is_linux
>         build = false
> -       reason = 'only supported on Linux and Windows'
> +       reason = 'only supported on Linux'
>         subdir_done()
>  endif
>
> -deps += ['hash', 'pci', 'net', 'eal', 'kvargs']
> +config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
> +driver_name_fmt = 'rte_common_@0@'
> +deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs']
>  sources += files(
>         'mlx5_devx_cmds.c',
>         'mlx5_common.c',
>         'mlx5_common_mp.c',
>         'mlx5_common_mr.c',
>         'mlx5_malloc.c',
> +       'mlx5_common_pci.c',

Idem order.


>  )
>
>  cflags_options = [
> diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
> index 2b336bb2d..fd818ef24 100644
> --- a/drivers/common/mlx5/mlx5_common.c
> +++ b/drivers/common/mlx5/mlx5_common.c
> @@ -14,6 +14,7 @@
>  #include "mlx5_common_os.h"
>  #include "mlx5_common_utils.h"
>  #include "mlx5_malloc.h"
> +#include "mlx5_common_pci.h"

Idem order.


>
>  int mlx5_common_logtype;
>
> @@ -100,6 +101,7 @@ mlx5_common_init(void)
>                 return;
>
>         mlx5_glue_constructor();
> +       mlx5_common_pci_init();
>         mlx5_common_initialized = true;
>  }
>
> diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
> new file mode 100644
> index 000000000..b42f7469f
> --- /dev/null
> +++ b/drivers/common/mlx5/mlx5_common_pci.c
> @@ -0,0 +1,541 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2020 Mellanox Technologies Ltd
> + */
> +
> +#include <stdlib.h>

Nit: headers usually come in separate blocks, "system headers", "dpdk
headers", "drivers headers".
So missing an empty line here.

> +#include <rte_malloc.h>


> +#include "mlx5_common_utils.h"
> +#include "mlx5_common_pci.h"
> +
> +struct mlx5_pci_device {
> +      struct rte_pci_device *pci_dev;
> +      TAILQ_ENTRY(mlx5_pci_device) next;
> +      uint32_t classes_loaded;
> +};
> +
> +/* Head of list of drivers. */
> +static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list =
> +                               TAILQ_HEAD_INITIALIZER(drv_list);
> +
> +/* Head of mlx5 pci devices. */
> +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
> +                               TAILQ_HEAD_INITIALIZER(devices_list);
> +
> +static const struct {
> +       const char *name;
> +       unsigned int driver_class;
> +} mlx5_classes[] = {
> +       { .name = "vdpa", .driver_class = MLX5_CLASS_VDPA },
> +       { .name = "net", .driver_class = MLX5_CLASS_NET },
> +       { .name = "regex", .driver_class = MLX5_CLASS_REGEX },
> +};
> +
> +static const unsigned int mlx5_class_combinations[] = {
> +       MLX5_CLASS_NET,
> +       MLX5_CLASS_VDPA,
> +       MLX5_CLASS_REGEX,
> +       /* New class combination should be added here.
> +        * For example a new multi class device combination
> +        * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
> +        */
> +};
> +
> +static int
> +class_name_to_value(const char *class_name)
> +{
> +       unsigned int i;
> +
> +       for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
> +               if (strcmp(class_name, mlx5_classes[i].name) == 0)
> +                       return mlx5_classes[i].driver_class;
> +       }
> +       return -EINVAL;
> +}
> +
> +static struct mlx5_pci_driver *
> +driver_get(uint32_t class)
> +{
> +       struct mlx5_pci_driver *driver;
> +
> +       TAILQ_FOREACH(driver, &drv_list, next) {
> +               if (driver->driver_class == class)
> +                       return driver;
> +       }
> +       return NULL;
> +}
> +
> +static int
> +bus_cmdline_options_handler(__rte_unused const char *key,
> +                           const char *class_names, void *opaque)
> +{
> +       int *ret = opaque;
> +       char *nstr_org;
> +       int class_val;
> +       char *found;
> +       char *nstr;
> +
> +       *ret = 0;
> +       nstr = strdup(class_names);
> +       if (!nstr) {
> +               *ret = -ENOMEM;
> +               return *ret;
> +       }
> +       nstr_org = nstr;
> +       while (nstr) {
> +               /* Extract each individual class name. Multiple
> +                * class key,value is supplied as class=net:vdpa:foo:bar.
> +                */
> +               found = strsep(&nstr, ":");
> +               if (!found)
> +                       continue;
> +               /* Check if its a valid class. */
> +               class_val = class_name_to_value(found);
> +               if (class_val < 0) {
> +                       *ret = -EINVAL;
> +                       goto err;
> +               }
> +               *ret |= class_val;
> +       }
> +err:
> +       free(nstr_org);
> +       if (*ret < 0)
> +               DRV_LOG(ERR, "Invalid mlx5 class options %s."
> +                       " Maybe typo in device class argument setting?",
> +                       class_names);
> +       return *ret;
> +}
> +
> +static int
> +parse_class_options(const struct rte_devargs *devargs)
> +{
> +       const char *key = MLX5_CLASS_ARG_NAME;
> +       struct rte_kvargs *kvlist;
> +       int ret = 0;
> +
> +       if (devargs == NULL)
> +               return 0;
> +       kvlist = rte_kvargs_parse(devargs->args, NULL);
> +       if (kvlist == NULL)
> +               return 0;
> +       if (rte_kvargs_count(kvlist, key))
> +               rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
> +                                  &ret);
> +       rte_kvargs_free(kvlist);
> +       return ret;
> +}
> +
> +static bool
> +mlx5_bus_match(const struct mlx5_pci_driver *drv,
> +              const struct rte_pci_device *pci_dev)
> +{
> +       const struct rte_pci_id *id_table;
> +
> +       for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
> +            id_table++) {
> +               /* Check if device's ids match the class driver's ids. */
> +               if (id_table->vendor_id != pci_dev->id.vendor_id &&
> +                   id_table->vendor_id != PCI_ANY_ID)
> +                       continue;
> +               if (id_table->device_id != pci_dev->id.device_id &&
> +                   id_table->device_id != PCI_ANY_ID)
> +                       continue;
> +               if (id_table->subsystem_vendor_id !=
> +                   pci_dev->id.subsystem_vendor_id &&
> +                   id_table->subsystem_vendor_id != PCI_ANY_ID)
> +                       continue;
> +               if (id_table->subsystem_device_id !=
> +                   pci_dev->id.subsystem_device_id &&
> +                   id_table->subsystem_device_id != PCI_ANY_ID)
> +                       continue;
> +               if (id_table->class_id != pci_dev->id.class_id &&
> +                   id_table->class_id != RTE_CLASS_ANY_ID)
> +                       continue;
> +               return true;
> +       }
> +       return false;
> +}
> +
> +static int
> +is_valid_class_combination(uint32_t user_classes)
> +{
> +       unsigned int i;
> +
> +       /* Verify if user specified valid supported combination. */

Are there invalid supported combinations? :-)

/* Verify if the user specified a supported combination. */


> +       for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
> +               if (mlx5_class_combinations[i] == user_classes)
> +                       return 0;
> +       }
> +       /* Not found any valid class combination. */
> +       return -EINVAL;
> +}
> +
> +static struct mlx5_pci_device *
> +pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
> +{
> +       struct mlx5_pci_device *dev;
> +
> +       TAILQ_FOREACH(dev, &devices_list, next) {
> +               if (dev->pci_dev == pci_dev)
> +                       return dev;
> +       }
> +       return NULL;
> +}
> +
> +static bool
> +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
> +{
> +       return (device->classes_loaded & class) ? true : false;
> +}
> +
> +static void
> +dev_release(struct mlx5_pci_device *dev)
> +{
> +       TAILQ_REMOVE(&devices_list, dev, next);
> +       rte_free(dev);
> +}
> +
> +static int
> +drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
> +{
> +       struct mlx5_pci_driver *driver;
> +       int local_ret = -ENODEV;
> +       unsigned int i = 0;
> +       int ret = 0;
> +
> +       enabled_classes &= dev->classes_loaded;
> +       while (enabled_classes) {
> +               driver = driver_get(RTE_BIT64(i));
> +               if (driver) {
> +                       local_ret = driver->pci_driver.remove(dev->pci_dev);
> +                       if (!local_ret)
> +                               dev->classes_loaded &= ~RTE_BIT64(i);
> +                       else if (ret == 0)
> +                               ret = local_ret;
> +               }
> +               enabled_classes &= ~RTE_BIT64(i);
> +               i++;
> +       }
> +       if (local_ret)
> +               ret = local_ret;
> +       return ret;
> +}
> +
> +static int
> +drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv,
> +             struct rte_pci_device *pci_dev, uint32_t user_classes)
> +{
> +       struct mlx5_pci_driver *driver;
> +       uint32_t enabled_classes = 0;
> +       bool already_loaded;
> +       int ret;
> +
> +       TAILQ_FOREACH(driver, &drv_list, next) {
> +               if ((driver->driver_class & user_classes) == 0)
> +                       continue;
> +               if (!mlx5_bus_match(driver, pci_dev))
> +                       continue;
> +               already_loaded = dev->classes_loaded & driver->driver_class;
> +               if (already_loaded &&
> +                   !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
> +                       DRV_LOG(ERR, "Device %s is already probed\n",
> +                               pci_dev->device.name);
> +                       ret = -EEXIST;
> +                       goto probe_err;
> +               }
> +               ret = driver->pci_driver.probe(pci_drv, pci_dev);
> +               if (ret < 0) {
> +                       DRV_LOG(ERR, "Failed to load driver = %s.\n",
> +                               driver->pci_driver.driver.name);
> +                       goto probe_err;
> +               }
> +               enabled_classes |= driver->driver_class;
> +       }
> +       dev->classes_loaded |= enabled_classes;
> +       return 0;
> +probe_err:
> +       /* Only unload drivers which are enabled which were enabled
> +        * in this probe instance.
> +        */
> +       drivers_remove(dev, enabled_classes);
> +       return ret;
> +}
> +
> +/**
> + * DPDK callback to register to probe multiple drivers for a PCI device.
> + *
> + * @param[in] pci_drv
> + *   PCI driver structure.
> + * @param[in] dev
> + *   PCI device information.
> + *
> + * @return
> + *   0 on success, a negative errno value otherwise and rte_errno is set.
> + */
> +static int
> +mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
> +                     struct rte_pci_device *pci_dev)
> +{
> +       struct mlx5_pci_device *dev;
> +       uint32_t user_classes = 0;
> +       bool new_device = false;
> +       int ret;
> +
> +       ret = parse_class_options(pci_dev->device.devargs);
> +       if (ret < 0)
> +               return ret;
> +       user_classes = ret;
> +       if (user_classes) {
> +               /* Validate combination here. */
> +               ret = is_valid_class_combination(user_classes);
> +               if (ret) {
> +                       DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
> +                       return ret;
> +               }
> +       } else {
> +               /* Default to net class. */
> +               user_classes = MLX5_CLASS_NET;
> +       }
> +       dev = pci_to_mlx5_device(pci_dev);
> +       if (!dev) {
> +               dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
> +               if (!dev)
> +                       return -ENOMEM;
> +               dev->pci_dev = pci_dev;
> +               TAILQ_INSERT_HEAD(&devices_list, dev, next);
> +               new_device = true;
> +       }
> +       ret = drivers_probe(dev, pci_drv, pci_dev, user_classes);
> +       if (ret)
> +               goto class_err;
> +       return 0;
> +class_err:
> +       if (new_device)
> +               dev_release(dev);
> +       return ret;
> +}
> +
> +/**
> + * DPDK callback to remove one or more drivers for a PCI device.
> + *
> + * This function removes all drivers probed for a given PCI device.
> + *
> + * @param[in] pci_dev
> + *   Pointer to the PCI device.
> + *
> + * @return
> + *   0 on success, the function cannot fail.
> + */
> +static int
> +mlx5_common_pci_remove(struct rte_pci_device *pci_dev)
> +{
> +       struct mlx5_pci_device *dev;
> +       int ret;
> +
> +       dev = pci_to_mlx5_device(pci_dev);
> +       if (!dev)
> +               return -ENODEV;
> +       /* Matching device found, cleanup and unload drivers. */
> +       ret = drivers_remove(dev, dev->classes_loaded);
> +       if (!ret)
> +               dev_release(dev);
> +       return ret;
> +}
> +
> +static int
> +mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
> +                       uint64_t iova, size_t len)
> +{
> +       struct mlx5_pci_driver *driver = NULL;
> +       struct mlx5_pci_driver *temp;
> +       struct mlx5_pci_device *dev;
> +       int ret = -EINVAL;
> +
> +       dev = pci_to_mlx5_device(pci_dev);
> +       if (!dev)
> +               return -ENODEV;
> +       TAILQ_FOREACH(driver, &drv_list, next) {
> +               if (device_class_enabled(dev, driver->driver_class) &&
> +                   driver->pci_driver.dma_map) {
> +                       ret = driver->pci_driver.dma_map(pci_dev, addr,
> +                                                        iova, len);
> +                       if (ret)
> +                               goto map_err;
> +               }
> +       }
> +       return ret;
> +map_err:
> +       TAILQ_FOREACH(temp, &drv_list, next) {
> +               if (temp == driver)
> +                       break;
> +               if (device_class_enabled(dev, temp->driver_class) &&
> +                   temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
> +                       temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
> +       }
> +       return ret;
> +}
> +
> +static int
> +mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
> +                         uint64_t iova, size_t len)
> +{
> +       struct mlx5_pci_driver *driver;
> +       struct mlx5_pci_device *dev;
> +       int local_ret = -EINVAL;
> +       int ret;
> +
> +       dev = pci_to_mlx5_device(pci_dev);
> +       if (!dev)
> +               return -ENODEV;
> +       ret = 0;
> +       /* There is no unmap error recovery in current implementation. */
> +       TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) {
> +               if (device_class_enabled(dev, driver->driver_class) &&
> +                   driver->pci_driver.dma_unmap) {
> +                       local_ret = driver->pci_driver.dma_unmap(pci_dev, addr,
> +                                                                iova, len);
> +                       if (local_ret && (ret == 0))
> +                               ret = local_ret;
> +               }
> +       }
> +       if (local_ret)
> +               ret = local_ret;
> +       return ret;
> +}
> +
> +/* PCI ID table is build dynamically based on registered mlx5 drivers. */

built

> +static struct rte_pci_id *mlx5_pci_id_table;
> +
> +static struct rte_pci_driver mlx5_pci_driver = {
> +       .driver = {
> +               .name = "mlx5_pci",
> +       },
> +       .probe = mlx5_common_pci_probe,
> +       .remove = mlx5_common_pci_remove,
> +       .dma_map = mlx5_common_pci_dma_map,
> +       .dma_unmap = mlx5_common_pci_dma_unmap,
> +};
> +
> +static int
> +pci_id_table_size_get(const struct rte_pci_id *id_table)
> +{
> +       int table_size = 0;
> +
> +       for (; id_table->vendor_id != 0; id_table++)
> +               table_size++;
> +       return table_size;
> +}
> +
> +static bool
> +pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table,
> +             int next_idx)
> +{
> +       int current_size = next_idx - 1;
> +       int i;
> +
> +       for (i = 0; i < current_size; i++) {
> +               if (id->device_id == table[i].device_id &&
> +                   id->vendor_id == table[i].vendor_id &&
> +                   id->subsystem_vendor_id == table[i].subsystem_vendor_id &&
> +                   id->subsystem_device_id == table[i].subsystem_device_id)
> +                       return true;
> +       }
> +       return false;
> +}
> +
> +static void
> +pci_id_insert(struct rte_pci_id *new_table, int *next_idx,
> +             const struct rte_pci_id *id_table)
> +{
> +       /* Traverse the id_table, check if entry exists in new_table;
> +        * Add non duplicate entries to new table.
> +        */
> +       for (; id_table->vendor_id != 0; id_table++) {
> +               if (!pci_id_exists(id_table, new_table, *next_idx)) {
> +                       /* New entry; add to the table. */
> +                       new_table[*next_idx] = *id_table;
> +                       (*next_idx)++;
> +               }
> +       }
> +}
> +
> +static int
> +pci_ids_table_update(const struct rte_pci_id *driver_id_table)
> +{
> +       const struct rte_pci_id *id_iter;
> +       struct rte_pci_id *updated_table;
> +       struct rte_pci_id *old_table;
> +       int num_ids = 0;
> +       int i = 0;
> +
> +       old_table = mlx5_pci_id_table;
> +       if (old_table)
> +               num_ids = pci_id_table_size_get(old_table);
> +       num_ids += pci_id_table_size_get(driver_id_table);
> +       /* Increase size by one for the termination entry of vendor_id = 0. */
> +       num_ids += 1;
> +       updated_table = calloc(num_ids, sizeof(*updated_table));
> +       if (!updated_table)
> +               return -ENOMEM;
> +       if (TAILQ_EMPTY(&drv_list)) {
> +               /* Copy the first driver's ID table. */
> +               for (id_iter = driver_id_table; id_iter->vendor_id != 0;
> +                    id_iter++, i++)
> +                       updated_table[i] = *id_iter;
> +       } else {
> +               /* First copy existing table entries. */
> +               for (id_iter = old_table; id_iter->vendor_id != 0;
> +                    id_iter++, i++)
> +                       updated_table[i] = *id_iter;
> +               /* New id to be added at the end of current ID table. */
> +               pci_id_insert(updated_table, &i, driver_id_table);
> +       }
> +       /* Terminate table with empty entry. */
> +       updated_table[i].vendor_id = 0;
> +       mlx5_pci_driver.id_table = updated_table;
> +       mlx5_pci_id_table = updated_table;
> +       if (old_table)
> +               free(old_table);
> +       return 0;
> +}
> +
> +void
> +mlx5_pci_driver_register(struct mlx5_pci_driver *driver)
> +{
> +       int ret;
> +
> +       ret = pci_ids_table_update(driver->pci_driver.id_table);
> +       if (ret)
> +               return;
> +       mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags;
> +       TAILQ_INSERT_TAIL(&drv_list, driver, next);
> +}
> +
> +void mlx5_common_pci_init(void)
> +{
> +       const struct rte_pci_id empty_table[] = {
> +               {
> +                       .vendor_id = 0
> +               },
> +       };
> +
> +       /* All mlx5 PMDs constructor runs at same priority. So any of the PMD

/* All mlx5 PMDs constructors run at the same priority. So any of the PMD


> +        * including this one can register the PCI table first. If any other
> +        * PMD(s) have registered the PCI ID table, No need to register an empty
> +        * default one.
> +        */
> +       if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table))
> +                       return;
> +       rte_pci_register(&mlx5_pci_driver);
> +}
> +
> +RTE_FINI(mlx5_common_pci_finish)
> +{
> +       if (mlx5_pci_id_table != NULL) {
> +               /* Constructor doesn't register with PCI bus if it failed
> +                * to build the table.
> +                */
> +               rte_pci_unregister(&mlx5_pci_driver);
> +               free(mlx5_pci_id_table);
> +       }
> +}

I don't like the asymmetry between multiple constructors calling a
common init / and a single destructor in the common code.
But this is not really an issue.


> +RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__);

The pci driver name is mlx5_pci iirc.

But if there is no info about this driver to export, you can remove
RTE_PMD_EXPORT_NAME.


> diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
> new file mode 100644
> index 000000000..41b73e17a
> --- /dev/null
> +++ b/drivers/common/mlx5/mlx5_common_pci.h
> @@ -0,0 +1,77 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#ifndef _MLX5_COMMON_PCI_H_
> +#define _MLX5_COMMON_PCI_H_
> +
> +/**
> + * @file
> + *
> + * RTE Mellanox PCI Driver Interface
> + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
> + * devices. This layer enables creating such multiple class of devices on a
> + * single PCI device by allowing to bind multiple class specific device
> + * driver to attach to mlx5_pci driver.
> + *
> + * -----------    ------------    -------------
> + * |   mlx5  |    |   mlx5   |    |   mlx5    |
> + * | net pmd |    | vdpa pmd |    | regex pmd |
> + * -----------    ------------    -------------
> + *      \              |                 /
> + *       \             |                /
> + *        \       --------------       /
> + *         \______|   mlx5     |_____ /
> + *                | pci common |
> + *                --------------
> + *                     |
> + *                 -----------
> + *                 |   mlx5  |
> + *                 | pci dev |
> + *                 -----------
> + *
> + * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI
> + *   ID table of all related mlx5 PCI devices.
> + * - mlx5 class driver such as net, vdpa, regex PMD defines its
> + *   specific PCI ID table and mlx5 bus driver probes matching
> + *   class drivers.
> + * - mlx5 pci bus driver is cental place that validates supported

central


> + *   class combinations.
> + */
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif /* __cplusplus */
> +
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +
> +#include <mlx5_common.h>
> +
> +void mlx5_common_pci_init(void);
> +
> +/**
> + * A structure describing a mlx5 pci driver.
> + */
> +struct mlx5_pci_driver {
> +       struct rte_pci_driver pci_driver;       /**< Inherit core pci driver. */
> +       uint32_t driver_class;  /**< Class of this driver, enum mlx5_class */
> +       TAILQ_ENTRY(mlx5_pci_driver) next;
> +};
> +
> +/**
> + * Register a mlx5_pci device driver.
> + *
> + * @param driver
> + *   A pointer to a mlx5_pci_driver structure describing the driver
> + *   to be registered.
> + */
> +__rte_internal
> +void
> +mlx5_pci_driver_register(struct mlx5_pci_driver *driver);
> +
> +#ifdef __cplusplus
> +}
> +#endif /* __cplusplus */
> +
> +#endif /* _MLX5_COMMON_PCI_H_ */
> diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
> index 65f25252a..73cf72548 100644
> --- a/drivers/common/mlx5/rte_common_mlx5_version.map
> +++ b/drivers/common/mlx5/rte_common_mlx5_version.map
> @@ -91,5 +91,7 @@ INTERNAL {
>         mlx5_malloc;
>         mlx5_realloc;
>         mlx5_free;
> +
> +       mlx5_pci_driver_register;
>  };
>
> diff --git a/drivers/meson.build b/drivers/meson.build
> index e6d0409aa..95df7ef1d 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -5,6 +5,7 @@
>  subdirs = [
>         'common',
>         'bus',
> +       'common/mlx5', # depends on bus.
>         'mempool', # depends on common and bus.
>         'net',     # depends on common, bus, mempool
>         'raw',     # depends on common, bus and net.
> --
> 2.25.4
>

Reviewed-by: David Marchand <david.marchand@redhat.com>


-- 
David Marchand

[dpdk-dev] [PATCH v8 09/10] common/mlx5: register class drivers through common layer

[Parav Pandit]

Migrate mlx5 net, vdpa and regex PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v7->v8:
 - Extended support for newly added mlx5 regex driver
---
 drivers/common/mlx5/mlx5_common_pci.c |  6 ++----
 drivers/net/mlx5/Makefile             |  1 -
 drivers/net/mlx5/linux/mlx5_os.c      |  1 -
 drivers/net/mlx5/mlx5.c               | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h               |  1 -
 drivers/regex/mlx5/meson.build        |  2 +-
 drivers/regex/mlx5/mlx5_regex.c       | 21 ++++++++++++---------
 drivers/vdpa/mlx5/Makefile            |  3 ++-
 drivers/vdpa/mlx5/meson.build         |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c         | 23 +++++++++++++----------
 10 files changed, 45 insertions(+), 39 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index b42f7469f..08c8a5538 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -34,10 +34,8 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_NET,
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
-	/* New class combination should be added here.
-	 * For example a new multi class device combination
-	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
-	 */
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
+	/* New class combination should be added here. */
 };
 
 static int
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index 253faf909..568c77241 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -55,7 +55,6 @@ LDLIBS += -lrte_common_mlx5
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index fa3b02787..97d7b0e7b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1537,7 +1537,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 037703d2e..1e4c695f8 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -30,6 +30,7 @@
 #include <mlx5_common.h>
 #include <mlx5_common_os.h>
 #include <mlx5_common_mp.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_malloc.h>
 
 #include "mlx5_defs.h"
@@ -2091,16 +2092,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct mlx5_pci_driver mlx5_driver = {
+	.driver_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /* Initialize driver log type. */
@@ -2117,7 +2121,7 @@ RTE_INIT(rte_mlx5_pmd_init)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a92194d2d..78d6eb728 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -119,7 +119,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/regex/mlx5/meson.build b/drivers/regex/mlx5/meson.build
index 7f800f2e3..d7cb2c572 100644
--- a/drivers/regex/mlx5/meson.build
+++ b/drivers/regex/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_regex'
-deps += ['common_mlx5', 'bus_pci', 'eal', 'regexdev']
+deps += ['common_mlx5', 'eal', 'regexdev']
 sources = files(
 	'mlx5_regex.c',
 	'mlx5_rxp.c',
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 4e0367052..ae9f00189 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -5,12 +5,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_regexdev.h>
 #include <rte_regexdev_core.h>
 #include <rte_regexdev_driver.h>
 
+#include <mlx5_common_pci.h>
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
@@ -246,21 +246,24 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_regex_driver = {
-	.driver = {
-		.name = "mlx5_regex",
+static struct mlx5_pci_driver mlx5_regex_driver = {
+	.driver_class = MLX5_CLASS_REGEX,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_regex",
+		},
+		.id_table = mlx5_regex_pci_id_map,
+		.probe = mlx5_regex_pci_probe,
+		.remove = mlx5_regex_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_regex_pci_id_map,
-	.probe = mlx5_regex_pci_probe,
-	.remove = mlx5_regex_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_INIT(rte_mlx5_regex_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_regex_driver);
+		mlx5_pci_driver_register(&mlx5_regex_driver);
 }
 
 RTE_LOG_REGISTER(mlx5_regex_logtype, pmd.regex.mlx5, NOTICE)
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..1cbbcecf8 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
 CFLAGS += -I$(RTE_SDK)/lib/librte_sched
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..254a8ec14 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 85dbcf956..ffe2f00ac 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,12 +11,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
 #include <mlx5_nl.h>
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct mlx5_pci_driver mlx5_vdpa_driver = {
+	.driver_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
@@ -848,7 +851,7 @@ RTE_INIT(rte_mlx5_vdpa_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
-- 
2.25.4

[dpdk-dev] [PATCH v8 10/10] common/mlx5: remove class check from class drivers

[Parav Pandit]

Now that mlx5_pci PMD checks for enabled classes and performs
probe(), remove() of associated classes, individual class driver
does not need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 5 files changed, 1 insertion(+), 51 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index fd818ef24..06f0a6400 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -20,43 +20,6 @@ int mlx5_common_logtype;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 864e82ff0..2cdb226f3 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -247,8 +247,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 73cf72548..7729d0dd0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,9 +1,8 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_init;
+
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 97d7b0e7b..fc339affa 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1524,11 +1524,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	unsigned int dev_config_vf;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index ffe2f00ac..c0b87bcc0 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.25.4

[dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 common driver layer to support multiple
class of devices for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
mlx5 common module is extended as generic PCI PMD.
This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded
on a single PCI Device.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Fixed indentation for directory listing
Patch-3 Relax dependency order
Patch-4 Fixes compilation error
Patch-5 Fixes segmentation fault in regex probe error unwind path
Patch-6 Avoid using class constructor priority
Patch-7 Change class values as bits
Patch-8 Introduce mlx5 layer to support multiple class drivers
Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API
        instead of rte PCI bus API
Patch-10 Removed class check code from class drivers 

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       ---------------     /
         \______| mlx5 common |____/
                |    pmd      |
                --------------- 
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 common driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD define their
  specific PCI ID tables. mlx5 common driver probes each
  individual class driver(s) for maching and enabled classes.
- mlx5 pci layer is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI common driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci common PMD, a common driver is
implemented which exposes class registration API.
However, common PMD model fits better with existing DPDK design
similar to ifpga driver. Class registration API need to create a
new callbacks and ID signature; instead it is better to utilize
current well defined methods.

2. Enhance pci core to allow multiple driver binding to single
rte PCI device.
This approach is not taken, because peer drivers using one PCI
device won't be aware of other's presence. This requires
cross-driver syncronization of who initializes common resources
(such as irq, eq and more).
This also requires refcounting common objects etc among peer drivers.
Instead of layered approach delivers and allows putting common resource
sharing, setup code in common driver.
It also eliminates peer blind zone problem as bottom pci layer provides
necessary setup without any reference counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf used
between net and regex pmd so that same mbuf use across multiple
device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list of classes
at command line such as,
./testpmd -w <PCI BDF>,class=net:regex
./testpmd -w <PCI BDF>,class=vdpa

Changelog:
v8->v9:
 - Updated commit message
 - Fixed LD_FLAGS
 - Fixed white space to tab
 - Fixed Makefile to have dependency on common/mlx5 for parallel builds
v7->v8:
 - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5
 - Avoided new RTE PRIORITY
 - Removed mlx5 common to use RTE_PRIORITY_CLASS
v6->v7:
 - Updated release notes
v5->v6:
 - Fixed compilation failure in parallel build for shared lib
v4->v5:
 - Squash the maintainers update path with other patch which adds the
   bus
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (8):
  eal: introduce macro for bit definition
  common/mlx5: fix void parameters in glue wrappers
  regex/mlx5: fix segmentation fault during error unwinding
  common/mlx5: avoid using class constructor priority
  common/mlx5: change class values as bits
  common/mlx5: introduce layer to support multiple class drivers
  common/mlx5: register class drivers through common layer
  common/mlx5: remove class check from class drivers

Thomas Monjalon (2):
  drivers: fix indent of directory list
  drivers: relax dependency order

 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   2 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |  52 +-
 drivers/common/mlx5/mlx5_common.h             |  12 +-
 drivers/common/mlx5/mlx5_common_pci.c         | 539 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   4 +-
 drivers/meson.build                           |  51 +-
 drivers/net/mlx5/Makefile                     |   1 -
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/mlx5.c                       |  25 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/regex/mlx5/meson.build                |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |  29 +-
 drivers/vdpa/mlx5/Makefile                    |   2 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  29 +-
 lib/librte_eal/include/rte_bitops.h           |   8 +
 22 files changed, 748 insertions(+), 123 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

-- 
2.26.2

[dpdk-dev] [PATCH v9 01/10] eal: introduce macro for bit definition

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
Changelog:
v4->v5:
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..ca46a110f 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,14 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+/**
+ * Get the uint64_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 63.
+ */
+#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.26.2

[dpdk-dev] [PATCH v9 02/10] drivers: fix indent of directory list

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Define each sub-directory on its own line ended with a comma,
and use a simple indent.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog:
v7->8:
 - new patch
---
 drivers/meson.build | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e76ebddfa..e2aeba931 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -2,17 +2,19 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 # Defines the order in which the drivers are buit.
-dpdk_driver_classes = ['common',
-	       'bus',
-	       'mempool', # depends on common and bus.
-	       'net',     # depends on common, bus, mempool
-	       'raw',     # depends on common, bus and net.
-	       'crypto',  # depends on common, bus and mempool (net in future).
-	       'compress', # depends on common, bus, mempool.
-	       'regex', # depends on common, bus, regexdev.
-	       'vdpa',    # depends on common, bus and mempool.
-	       'event',   # depends on common, bus, mempool and net.
-	       'baseband'] # depends on common and bus.
+dpdk_driver_classes = [
+	'common',
+	'bus',
+	'mempool', # depends on common and bus.
+	'net',     # depends on common, bus, mempool
+	'raw',     # depends on common, bus and net.
+	'crypto',  # depends on common, bus and mempool (net in future).
+	'compress', # depends on common, bus, mempool.
+	'regex', # depends on common, bus, regexdev.
+	'vdpa',    # depends on common, bus and mempool.
+	'event',   # depends on common, bus, mempool and net.
+	'baseband', # depends on common and bus.
+]
 
 disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
-- 
2.26.2

[dpdk-dev] [PATCH v9 03/10] drivers: relax dependency order

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Drivers dependencies are evaluated in the order defined per their parent
directory (also called class). This strict ordering prevents from
having pairs of drivers from two classes with different dependency
ordering. For example, if the mlx5 common code depends on the pci bus
driver, while the dpaax common code is itself a dependency of the dpaa
bus driver.  Having a strict ordering between directories bus and common
is too restrictive, as processing either common drivers or bus drivers
first leads us to missing dependencies in this scenario.

This patch makes it possible to have a more fine-grain directory list,
adding a specific driver sub-directory in the top-level drivers
subdirectory list. In this case, the isolated driver must also be removed
from its class list, and the per-class variables must be duplicated in
the isolated driver, because the call "subdir(class)" is skipped in the
isolated driver case.

Review-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog:
v8->v9:
 - Updated commit message
 - Fixed typo
---
 drivers/meson.build | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e2aeba931..e6d0409aa 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-# Defines the order in which the drivers are buit.
-dpdk_driver_classes = [
+# Defines the order of dependencies evaluation
+subdirs = [
 	'common',
 	'bus',
 	'mempool', # depends on common and bus.
@@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation')
 	default_cflags += '-Wno-format-truncation'
 endif
 
-foreach class:dpdk_driver_classes
+foreach subpath:subdirs
 	drivers = []
 	std_deps = []
 	config_flag_fmt = '' # format string used to set the value in dpdk_conf
@@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes
 	                     # the library, the dependency and to find the
 	                     # version file for linking
 
-	subdir(class)
-	class_drivers = []
+	# subpath can be either "class" or "class/driver"
+	if subpath.contains('/')
+		driver_path = subpath.split('/')
+		class = driver_path[0]
+		drivers += driver_path[1]
+	else
+		class = subpath
+		subdir(class)
+	endif
+
+	# save class name on first occurence
+	if not dpdk_driver_classes.contains(class)
+		dpdk_driver_classes += class
+	endif
+	# get already enabled drivers of the same class
+	enabled_drivers = get_variable(class + '_drivers', [])
 
 	foreach drv:drivers
 		drv_path = join_paths(class, drv)
@@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes
 						'_disable_reason', reason)
 			endif
 		else
-			class_drivers += name
+			enabled_drivers += name
 
 			if fmt_name == ''
 				fmt_name = name
@@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes
 		endif # build
 	endforeach
 
-	set_variable(class + '_drivers', class_drivers)
+	set_variable(class + '_drivers', enabled_drivers)
 endforeach
-- 
2.26.2

[dpdk-dev] [PATCH v9 04/10] common/mlx5: fix void parameters in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index c4f9b006a..734ace2a3 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v9 05/10] regex/mlx5: fix segmentation fault during error unwinding

[Parav Pandit]

When fail to initialize the device, avoid segmentation fault while
accessing unintialized priv.

Fixes: cfc672a90b74 ("regex/mlx5: support probing")

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v7->v8:
 - Rebased
 - new patch
---
 drivers/regex/mlx5/mlx5_regex.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 1ca5bfe9b..36ae9f809 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -137,17 +137,17 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	} else if (!attr.regex || attr.regexp_num_of_engines == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
 			"old FW/OFED version?");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	}
 	if (mlx5_regex_engines_status(ctx, 2)) {
 		DRV_LOG(ERR, "RegEx engine error.");
 		rte_errno = ENOMEM;
-		goto error;
+		goto dev_error;
 	}
 	priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
 			   RTE_CACHE_LINE_SIZE);
@@ -200,6 +200,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		mlx5_glue->devx_free_uar(priv->uar);
 	if (priv->regexdev)
 		rte_regexdev_unregister(priv->regexdev);
+dev_error:
 	if (ctx)
 		mlx5_glue->close_device(ctx);
 	if (priv)
-- 
2.26.2

[dpdk-dev] [PATCH v9 06/10] common/mlx5: avoid using class constructor priority

[Parav Pandit]

mlx5_common is shared library between mlx5 net, VDPA and regex PMD.
It is better to use common initialization helper instead of using
RTE_PRIORITY_CLASS priority.

Reviewed-by: David Marchand <david.marchand@redhat.com>
Suggested-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v7->v8:
 - new patch
---
 drivers/common/mlx5/mlx5_common.c               | 13 +++++++++++--
 drivers/common/mlx5/mlx5_common.h               |  3 +++
 drivers/common/mlx5/rte_common_mlx5_version.map |  1 +
 drivers/net/mlx5/mlx5.c                         |  1 +
 drivers/regex/mlx5/mlx5_regex.c                 |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.c                   |  1 +
 6 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 1a67a1b16..2b336bb2d 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 		rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE);
 }
 
+static bool mlx5_common_initialized;
+
 /**
- * Initialization routine for run-time dependency on glue library.
+ * One time innitialization routine for run-time dependency on glue library
+ * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module,
+ * must invoke in its constructor.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+void
+mlx5_common_init(void)
 {
+	if (mlx5_common_initialized)
+		return;
+
 	mlx5_glue_constructor();
+	mlx5_common_initialized = true;
 }
 
 /**
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a811eb6c9..ebe4e9ced 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id,
 			 uint64_t offset);
 extern uint8_t haswell_broadwell_cpu;
 
+__rte_internal
+void mlx5_common_init(void);
+
 #endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 132a0695f..65f25252a 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -3,6 +3,7 @@ INTERNAL {
 
 	mlx5_class_get;
 
+	mlx5_common_init;
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 647ada339..037703d2e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_pmd_init)
 {
+	mlx5_common_init();
 	/* Build the static tables for Verbs conversion. */
 	mlx5_set_ptype_table();
 	mlx5_set_cksum_table();
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 36ae9f809..4e0367052 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = {
 
 RTE_INIT(rte_mlx5_regex_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_regex_driver);
 }
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 67e77b11a..85dbcf956 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_vdpa_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_vdpa_driver);
 }
-- 
2.26.2

[dpdk-dev] [PATCH v9 07/10] common/mlx5: change class values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v7->v8:
 - updated commit message
---
 drivers/common/mlx5/mlx5_common.h | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index ebe4e9ced..864e82ff0 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -208,10 +209,10 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
-	MLX5_CLASS_REGEX,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT64(0),
+	MLX5_CLASS_VDPA = RTE_BIT64(1),
+	MLX5_CLASS_REGEX = RTE_BIT64(2),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
-- 
2.26.2

[dpdk-dev] [PATCH v9 08/10] common/mlx5: introduce layer to support multiple class drivers

[Parav Pandit]

Add generic mlx5 PCI PMD layer as part of existing common_mlx5
module. This enables multiple classes (net, regex, vdpa) PMDs
to be supported at same time.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v8->v9:
 - Added missing LDFLAG for pci bus
 - Fixed white spaces at start of the line
---
 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   2 +
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |   2 +
 drivers/common/mlx5/mlx5_common_pci.c         | 541 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   2 +
 drivers/meson.build                           |   1 +
 10 files changed, 639 insertions(+), 11 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

diff --git a/drivers/Makefile b/drivers/Makefile
index 1551272ef..7f06162dc 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -8,8 +8,12 @@ DIRS-y += bus
 DEPDIRS-bus := common
 DIRS-y += mempool
 DEPDIRS-mempool := common bus
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
+DIRS-y += common/mlx5
+DEPDIRS-common/mlx5 := bus
+endif
 DIRS-y += net
-DEPDIRS-net := common bus mempool
+DEPDIRS-net := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband
 DEPDIRS-baseband := common bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto
@@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress
 DEPDIRS-compress := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex
-DEPDIRS-regex := common bus
+DEPDIRS-regex := common bus common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa
-DEPDIRS-vdpa := common bus mempool
+DEPDIRS-vdpa := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event
 DEPDIRS-event := common bus mempool net crypto
 DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index cbc71077c..cfb6b4dc8 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y)))
 DIRS-y += iavf
 endif
 
-ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
-DIRS-y += mlx5
-endif
-
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/common/meson.build b/drivers/common/meson.build
index 5db7e29b1..9ed4c04ba 100644
--- a/drivers/common/meson.build
+++ b/drivers/common/meson.build
@@ -6,6 +6,6 @@ if is_windows
 endif
 
 std_deps = ['eal']
-drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat']
+drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat']
 config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
 driver_name_fmt = 'rte_common_@0@'
diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile
index 6b89a6c85..4edd54104 100644
--- a/drivers/common/mlx5/Makefile
+++ b/drivers/common/mlx5/Makefile
@@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c
 SRCS-y += mlx5_common_mp.c
 SRCS-y += mlx5_common_mr.c
 SRCS-y += mlx5_malloc.c
+SRCS-y += mlx5_common_pci.c
 ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y)
 INSTALL-y-lib += $(LIB_GLUE)
 endif
@@ -51,6 +52,7 @@ LDLIBS += -libverbs -lmlx5
 endif
 
 LDLIBS += -lrte_eal -lrte_pci -lrte_kvargs -lrte_net
+LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual -UPEDANTIC
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index 70e2c1c32..8e5608703 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -1,19 +1,22 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2019 Mellanox Technologies, Ltd
 
-if not (is_linux or is_windows)
+if not is_linux
 	build = false
-	reason = 'only supported on Linux and Windows'
+	reason = 'only supported on Linux'
 	subdir_done()
 endif
 
-deps += ['hash', 'pci', 'net', 'eal', 'kvargs']
+config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
+driver_name_fmt = 'rte_common_@0@'
+deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs']
 sources += files(
 	'mlx5_devx_cmds.c',
 	'mlx5_common.c',
 	'mlx5_common_mp.c',
 	'mlx5_common_mr.c',
 	'mlx5_malloc.c',
+	'mlx5_common_pci.c',
 )
 
 cflags_options = [
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 2b336bb2d..fd818ef24 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -14,6 +14,7 @@
 #include "mlx5_common_os.h"
 #include "mlx5_common_utils.h"
 #include "mlx5_malloc.h"
+#include "mlx5_common_pci.h"
 
 int mlx5_common_logtype;
 
@@ -100,6 +101,7 @@ mlx5_common_init(void)
 		return;
 
 	mlx5_glue_constructor();
+	mlx5_common_pci_init();
 	mlx5_common_initialized = true;
 }
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
new file mode 100644
index 000000000..adb3f707d
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -0,0 +1,541 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies Ltd
+ */
+
+#include <stdlib.h>
+#include <rte_malloc.h>
+#include "mlx5_common_utils.h"
+#include "mlx5_common_pci.h"
+
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of drivers. */
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int driver_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .driver_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .driver_class = MLX5_CLASS_NET },
+	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	MLX5_CLASS_REGEX,
+	/* New class combination should be added here.
+	 * For example a new multi class device combination
+	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
+	 */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].driver_class;
+	}
+	return -EINVAL;
+}
+
+static struct mlx5_pci_driver *
+driver_get(uint32_t class)
+{
+	struct mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->driver_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+static bool
+mlx5_bus_match(const struct mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = driver_get(RTE_BIT64(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT64(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT64(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv,
+	      struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->driver_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->driver_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->driver_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple drivers for a PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		      struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more drivers for a PCI device.
+ *
+ * This function removes all drivers probed for a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_common_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload drivers. */
+	ret = drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+			uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver = NULL;
+	struct mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_map) {
+			ret = driver->pci_driver.dma_map(pci_dev, addr,
+							 iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == driver)
+			break;
+		if (device_class_enabled(dev, temp->driver_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+			  uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_unmap) {
+			local_ret = driver->pci_driver.dma_unmap(pci_dev, addr,
+								 iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5 drivers. */
+static struct rte_pci_id *mlx5_pci_id_table;
+
+static struct rte_pci_driver mlx5_pci_driver = {
+	.driver = {
+		.name = "mlx5_pci",
+	},
+	.probe = mlx5_common_pci_probe,
+	.remove = mlx5_common_pci_remove,
+	.dma_map = mlx5_common_pci_dma_map,
+	.dma_unmap = mlx5_common_pci_dma_unmap,
+};
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table,
+	      int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == table[i].device_id &&
+		    id->vendor_id == table[i].vendor_id &&
+		    id->subsystem_vendor_id == table[i].subsystem_vendor_id &&
+		    id->subsystem_device_id == table[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(struct rte_pci_id *new_table, int *next_idx,
+	      const struct rte_pci_id *id_table)
+{
+	/* Traverse the id_table, check if entry exists in new_table;
+	 * Add non duplicate entries to new table.
+	 */
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, new_table, *next_idx)) {
+			/* New entry; add to the table. */
+			new_table[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_update(const struct rte_pci_id *driver_id_table)
+{
+	const struct rte_pci_id *id_iter;
+	struct rte_pci_id *updated_table;
+	struct rte_pci_id *old_table;
+	int num_ids = 0;
+	int i = 0;
+
+	old_table = mlx5_pci_id_table;
+	if (old_table)
+		num_ids = pci_id_table_size_get(old_table);
+	num_ids += pci_id_table_size_get(driver_id_table);
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	updated_table = calloc(num_ids, sizeof(*updated_table));
+	if (!updated_table)
+		return -ENOMEM;
+	if (TAILQ_EMPTY(&drv_list)) {
+		/* Copy the first driver's ID table. */
+		for (id_iter = driver_id_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+	} else {
+		/* First copy existing table entries. */
+		for (id_iter = old_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+		/* New id to be added at the end of current ID table. */
+		pci_id_insert(updated_table, &i, driver_id_table);
+	}
+	/* Terminate table with empty entry. */
+	updated_table[i].vendor_id = 0;
+	mlx5_pci_driver.id_table = updated_table;
+	mlx5_pci_id_table = updated_table;
+	if (old_table)
+		free(old_table);
+	return 0;
+}
+
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver)
+{
+	int ret;
+
+	ret = pci_ids_table_update(driver->pci_driver.id_table);
+	if (ret)
+		return;
+	mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags;
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
+
+void mlx5_common_pci_init(void)
+{
+	const struct rte_pci_id empty_table[] = {
+		{
+			.vendor_id = 0
+		},
+	};
+
+	/* All mlx5 PMDs constructor runs at same priority. So any of the PMD
+	 * including this one can register the PCI table first. If any other
+	 * PMD(s) have registered the PCI ID table, No need to register an empty
+	 * default one.
+	 */
+	if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table))
+			return;
+	rte_pci_register(&mlx5_pci_driver);
+}
+
+RTE_FINI(mlx5_common_pci_finish)
+{
+	if (mlx5_pci_id_table != NULL) {
+		/* Constructor doesn't register with PCI bus if it failed
+		 * to build the table.
+		 */
+		rte_pci_unregister(&mlx5_pci_driver);
+		free(mlx5_pci_id_table);
+	}
+}
+RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__);
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
new file mode 100644
index 000000000..41b73e17a
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _MLX5_COMMON_PCI_H_
+#define _MLX5_COMMON_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Driver Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This layer enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci driver.
+ *
+ * -----------    ------------    -------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5    |
+ * | net pmd |    | vdpa pmd |    | regex pmd |
+ * -----------    ------------    -------------
+ *      \              |                 /
+ *       \             |                /
+ *        \       --------------       /
+ *         \______|   mlx5     |_____ /
+ *                | pci common |
+ *                --------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+void mlx5_common_pci_init(void);
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t driver_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _MLX5_COMMON_PCI_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 65f25252a..73cf72548 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -91,5 +91,7 @@ INTERNAL {
 	mlx5_malloc;
 	mlx5_realloc;
 	mlx5_free;
+
+	mlx5_pci_driver_register;
 };
 
diff --git a/drivers/meson.build b/drivers/meson.build
index e6d0409aa..95df7ef1d 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -5,6 +5,7 @@
 subdirs = [
 	'common',
 	'bus',
+	'common/mlx5', # depends on bus.
 	'mempool', # depends on common and bus.
 	'net',     # depends on common, bus, mempool
 	'raw',     # depends on common, bus and net.
-- 
2.26.2

[dpdk-dev] [PATCH v9 09/10] common/mlx5: register class drivers through common layer

[Parav Pandit]

Migrate mlx5 net, vdpa and regex PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v8->v9:
 - Removed leftover inclusion of mlx5_pci bus include directory
---
 drivers/common/mlx5/mlx5_common_pci.c |  6 ++----
 drivers/net/mlx5/Makefile             |  1 -
 drivers/net/mlx5/linux/mlx5_os.c      |  1 -
 drivers/net/mlx5/mlx5.c               | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h               |  1 -
 drivers/regex/mlx5/meson.build        |  2 +-
 drivers/regex/mlx5/mlx5_regex.c       | 21 ++++++++++++---------
 drivers/vdpa/mlx5/Makefile            |  2 +-
 drivers/vdpa/mlx5/meson.build         |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c         | 23 +++++++++++++----------
 10 files changed, 44 insertions(+), 39 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index adb3f707d..b5fcc16b1 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -34,10 +34,8 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_NET,
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
-	/* New class combination should be added here.
-	 * For example a new multi class device combination
-	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
-	 */
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
+	/* New class combination should be added here. */
 };
 
 static int
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index 253faf909..568c77241 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -55,7 +55,6 @@ LDLIBS += -lrte_common_mlx5
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index fa3b02787..97d7b0e7b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1537,7 +1537,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 037703d2e..1e4c695f8 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -30,6 +30,7 @@
 #include <mlx5_common.h>
 #include <mlx5_common_os.h>
 #include <mlx5_common_mp.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_malloc.h>
 
 #include "mlx5_defs.h"
@@ -2091,16 +2092,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct mlx5_pci_driver mlx5_driver = {
+	.driver_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /* Initialize driver log type. */
@@ -2117,7 +2121,7 @@ RTE_INIT(rte_mlx5_pmd_init)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a92194d2d..78d6eb728 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -119,7 +119,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/regex/mlx5/meson.build b/drivers/regex/mlx5/meson.build
index 7f800f2e3..d7cb2c572 100644
--- a/drivers/regex/mlx5/meson.build
+++ b/drivers/regex/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_regex'
-deps += ['common_mlx5', 'bus_pci', 'eal', 'regexdev']
+deps += ['common_mlx5', 'eal', 'regexdev']
 sources = files(
 	'mlx5_regex.c',
 	'mlx5_rxp.c',
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 4e0367052..ae9f00189 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -5,12 +5,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_regexdev.h>
 #include <rte_regexdev_core.h>
 #include <rte_regexdev_driver.h>
 
+#include <mlx5_common_pci.h>
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
@@ -246,21 +246,24 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_regex_driver = {
-	.driver = {
-		.name = "mlx5_regex",
+static struct mlx5_pci_driver mlx5_regex_driver = {
+	.driver_class = MLX5_CLASS_REGEX,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_regex",
+		},
+		.id_table = mlx5_regex_pci_id_map,
+		.probe = mlx5_regex_pci_probe,
+		.remove = mlx5_regex_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_regex_pci_id_map,
-	.probe = mlx5_regex_pci_probe,
-	.remove = mlx5_regex_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_INIT(rte_mlx5_regex_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_regex_driver);
+		mlx5_pci_driver_register(&mlx5_regex_driver);
 }
 
 RTE_LOG_REGISTER(mlx5_regex_logtype, pmd.regex.mlx5, NOTICE)
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..d46699fbc 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -30,7 +30,7 @@ CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..254a8ec14 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 85dbcf956..ffe2f00ac 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,12 +11,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
 #include <mlx5_nl.h>
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct mlx5_pci_driver mlx5_vdpa_driver = {
+	.driver_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
@@ -848,7 +851,7 @@ RTE_INIT(rte_mlx5_vdpa_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
-- 
2.26.2

[dpdk-dev] [PATCH v9 10/10] common/mlx5: remove class check from class drivers

[Parav Pandit]

Now that mlx5_pci PMD checks for enabled classes and performs
probe(), remove() of associated classes, individual class driver
does not need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 5 files changed, 1 insertion(+), 51 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index fd818ef24..06f0a6400 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -20,43 +20,6 @@ int mlx5_common_logtype;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 864e82ff0..2cdb226f3 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -247,8 +247,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 73cf72548..7729d0dd0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,9 +1,8 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_init;
+
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 97d7b0e7b..fc339affa 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1524,11 +1524,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	unsigned int dev_config_vf;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index ffe2f00ac..c0b87bcc0 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

Re: [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

> From: Parav Pandit <parav@mellanox.com>
> Sent: Friday, July 24, 2020 7:54 PM
> 
> This series introduces mlx5 common driver layer to support multiple class of
> devices for a single PCI device.
> 
[..]

> 
> Changelog:
> v8->v9:
>  - Updated commit message
>  - Fixed LD_FLAGS
>  - Fixed white space to tab
>  - Fixed Makefile to have dependency on common/mlx5 for parallel builds

Sorry for missing out the checkpatch warnings. I missed to address them.
Sending v10.

[dpdk-dev] [PATCH v10 00/10] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 common driver layer to support multiple
class of devices for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
mlx5 common module is extended as generic PCI PMD.
This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded
on a single PCI Device.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Fixed indentation for directory listing
Patch-3 Relax dependency order
Patch-4 Fixes compilation error
Patch-5 Fixes segmentation fault in regex probe error unwind path
Patch-6 Avoid using class constructor priority
Patch-7 Change class values as bits
Patch-8 Introduce mlx5 layer to support multiple class drivers
Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API
        instead of rte PCI bus API
Patch-10 Removed class check code from class drivers 

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       ---------------     /
         \______| mlx5 common |____/
                |    pmd      |
                --------------- 
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 common driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD define their
  specific PCI ID tables. mlx5 common driver probes each
  individual class driver(s) for maching and enabled classes.
- mlx5 pci layer is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI common driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci common PMD, a common driver is
implemented which exposes class registration API.
However, common PMD model fits better with existing DPDK design
similar to ifpga driver. Class registration API need to create a
new callbacks and ID signature; instead it is better to utilize
current well defined methods.

2. Enhance pci core to allow multiple driver binding to single
rte PCI device.
This approach is not taken, because peer drivers using one PCI
device won't be aware of other's presence. This requires
cross-driver syncronization of who initializes common resources
(such as irq, eq and more).
This also requires refcounting common objects etc among peer drivers.
Instead of layered approach delivers and allows putting common resource
sharing, setup code in common driver.
It also eliminates peer blind zone problem as bottom pci layer provides
necessary setup without any reference counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf used
between net and regex pmd so that same mbuf use across multiple
device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list of classes
at command line such as,
./testpmd -w <PCI BDF>,class=net:regex
./testpmd -w <PCI BDF>,class=vdpa

Changelog:
v9->v10:
 - Fixed checkpatch warnings related to spelling mistakes and alignment
v8->v9:
 - Updated commit message
 - Fixed LD_FLAGS
 - Fixed white space to tab
 - Fixed Makefile to have dependency on common/mlx5 for parallel builds
v7->v8:
 - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5
 - Avoided new RTE PRIORITY
 - Removed mlx5 common to use RTE_PRIORITY_CLASS
v6->v7:
 - Updated release notes
v5->v6:
 - Fixed compilation failure in parallel build for shared lib
v4->v5:
 - Squash the maintainers update path with other patch which adds the
   bus
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (8):
  eal: introduce macro for bit definition
  common/mlx5: fix void parameters in glue wrappers
  regex/mlx5: fix segmentation fault during error unwinding
  common/mlx5: avoid using class constructor priority
  common/mlx5: change class values as bits
  common/mlx5: introduce layer to support multiple class drivers
  common/mlx5: register class drivers through common layer
  common/mlx5: remove class check from class drivers

Thomas Monjalon (2):
  drivers: fix indent of directory list
  drivers: relax dependency order

 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   2 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |  52 +-
 drivers/common/mlx5/mlx5_common.h             |  12 +-
 drivers/common/mlx5/mlx5_common_pci.c         | 539 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   4 +-
 drivers/meson.build                           |  51 +-
 drivers/net/mlx5/Makefile                     |   1 -
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/mlx5.c                       |  25 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/regex/mlx5/meson.build                |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |  29 +-
 drivers/vdpa/mlx5/Makefile                    |   2 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  29 +-
 lib/librte_eal/include/rte_bitops.h           |   8 +
 22 files changed, 748 insertions(+), 123 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

-- 
2.26.2

[dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
Changelog:
v4->v5:
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..ca46a110f 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,14 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+/**
+ * Get the uint64_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 63.
+ */
+#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.26.2

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Honnappa Nagarahalli]

<snip>

> Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition
> 
> There are several drivers which duplicate bit generation macro.
> Introduce a generic bit macros so that such drivers avoid redefining same in
> multiple drivers.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> Acked-by: Matan Azrad <matan@mellanox.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> ---
> Changelog:
> v4->v5:
>  - Addressed comments from Morten Brørup
>  - Renamed newly added macro to RTE_BIT64
>  - Added doxygen comment section for the macro
> v1->v2:
>  - Addressed comments from Thomas and Gaten.
>  - Avoided new file, added macro to rte_bitops.h
> ---
>  lib/librte_eal/include/rte_bitops.h | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/lib/librte_eal/include/rte_bitops.h
> b/lib/librte_eal/include/rte_bitops.h
> index 740927f3b..ca46a110f 100644
> --- a/lib/librte_eal/include/rte_bitops.h
> +++ b/lib/librte_eal/include/rte_bitops.h
> @@ -17,6 +17,14 @@
>  #include <rte_debug.h>
>  #include <rte_compat.h>
> 
> +/**
> + * Get the uint64_t value for a specified bit set.
> + *
> + * @param nr
> + *   The bit number in range of 0 to 63.
> + */
> +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
In general, the macros have been avoided in this file. Suggest changing this to an inline function.

Also, this file has uses of this macro, it would be good to replace them with the new inline function.

> +
>  /*------------------------ 32-bit relaxed operations ------------------------*/
> 
>  /**
> --
> 2.26.2

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Morten Brørup]

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Honnappa
> Nagarahalli
> Sent: Friday, July 24, 2020 8:31 PM
> 
> <snip>
> 
> > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit
> definition
> >
> > There are several drivers which duplicate bit generation macro.
> > Introduce a generic bit macros so that such drivers avoid redefining
> same in
> > multiple drivers.
> >
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > Acked-by: Matan Azrad <matan@mellanox.com>
> > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > ---
> > Changelog:
> > v4->v5:
> >  - Addressed comments from Morten Brørup
> >  - Renamed newly added macro to RTE_BIT64
> >  - Added doxygen comment section for the macro
> > v1->v2:
> >  - Addressed comments from Thomas and Gaten.
> >  - Avoided new file, added macro to rte_bitops.h
> > ---
> >  lib/librte_eal/include/rte_bitops.h | 8 ++++++++
> >  1 file changed, 8 insertions(+)
> >
> > diff --git a/lib/librte_eal/include/rte_bitops.h
> > b/lib/librte_eal/include/rte_bitops.h
> > index 740927f3b..ca46a110f 100644
> > --- a/lib/librte_eal/include/rte_bitops.h
> > +++ b/lib/librte_eal/include/rte_bitops.h
> > @@ -17,6 +17,14 @@
> >  #include <rte_debug.h>
> >  #include <rte_compat.h>
> >
> > +/**
> > + * Get the uint64_t value for a specified bit set.
> > + *
> > + * @param nr
> > + *   The bit number in range of 0 to 63.
> > + */
> > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> In general, the macros have been avoided in this file. Suggest changing
> this to an inline function.

That has been discussed already, and rejected for good reasons:
http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@AM0PR05MB4866.eurprd05.prod.outlook.com/

> Also, this file has uses of this macro, it would be good to replace
> them with the new inline function.

Makes sense.
And for consistency, it would require adding an RTE_BIT32() macro too.

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Parav Pandit]

> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Monday, July 27, 2020 1:52 PM
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Honnappa
> > Nagarahalli
> > Sent: Friday, July 24, 2020 8:31 PM
> >
> > <snip>
> >
> > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit
> > definition
> > >
> > > There are several drivers which duplicate bit generation macro.
> > > Introduce a generic bit macros so that such drivers avoid redefining
> > same in
> > > multiple drivers.
> > >
> > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > Acked-by: Matan Azrad <matan@mellanox.com>
> > > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > > ---
> > > Changelog:
> > > v4->v5:
> > >  - Addressed comments from Morten Brørup
> > >  - Renamed newly added macro to RTE_BIT64
> > >  - Added doxygen comment section for the macro
> > > v1->v2:
> > >  - Addressed comments from Thomas and Gaten.
> > >  - Avoided new file, added macro to rte_bitops.h
> > > ---
> > >  lib/librte_eal/include/rte_bitops.h | 8 ++++++++
> > >  1 file changed, 8 insertions(+)
> > >
> > > diff --git a/lib/librte_eal/include/rte_bitops.h
> > > b/lib/librte_eal/include/rte_bitops.h
> > > index 740927f3b..ca46a110f 100644
> > > --- a/lib/librte_eal/include/rte_bitops.h
> > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > @@ -17,6 +17,14 @@
> > >  #include <rte_debug.h>
> > >  #include <rte_compat.h>
> > >
> > > +/**
> > > + * Get the uint64_t value for a specified bit set.
> > > + *
> > > + * @param nr
> > > + *   The bit number in range of 0 to 63.
> > > + */
> > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> > In general, the macros have been avoided in this file. Suggest
> > changing this to an inline function.
> 
> That has been discussed already, and rejected for good reasons:
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox.
> dpdk.org%2Fdev%2FAM0PR05MB4866823B0170B90F679A2765D1640%40AM
> 0PR05MB4866.eurprd05.prod.outlook.com%2F&amp;data=02%7C01%7Cpara
> v%40mellanox.com%7Cdb43e7a083ae44a063a208d8320619b6%7Ca652971c7d
> 2e4d9ba6a4d149256f461b%7C0%7C0%7C637314349075254981&amp;sdata=k
> R3fSOickIY2HbpmodNlB3ERF%2F2Qc7th55SGF40xmB0%3D&amp;reserved=0
> 
> > Also, this file has uses of this macro, it would be good to replace
> > them with the new inline function.
> 
> Makes sense.
> And for consistency, it would require adding an RTE_BIT32() macro too.
Ok. Sending v12 addressing it.

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Honnappa Nagarahalli]

<snip>
> >
> > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit
> > definition
> > >
> > > There are several drivers which duplicate bit generation macro.
> > > Introduce a generic bit macros so that such drivers avoid redefining
> > same in
> > > multiple drivers.
> > >
> > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > Acked-by: Matan Azrad <matan@mellanox.com>
> > > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > > ---
> > > Changelog:
> > > v4->v5:
> > >  - Addressed comments from Morten Brørup
> > >  - Renamed newly added macro to RTE_BIT64
> > >  - Added doxygen comment section for the macro
> > > v1->v2:
> > >  - Addressed comments from Thomas and Gaten.
> > >  - Avoided new file, added macro to rte_bitops.h
> > > ---
> > >  lib/librte_eal/include/rte_bitops.h | 8 ++++++++
> > >  1 file changed, 8 insertions(+)
> > >
> > > diff --git a/lib/librte_eal/include/rte_bitops.h
> > > b/lib/librte_eal/include/rte_bitops.h
> > > index 740927f3b..ca46a110f 100644
> > > --- a/lib/librte_eal/include/rte_bitops.h
> > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > @@ -17,6 +17,14 @@
> > >  #include <rte_debug.h>
> > >  #include <rte_compat.h>
> > >
> > > +/**
> > > + * Get the uint64_t value for a specified bit set.
> > > + *
> > > + * @param nr
> > > + *   The bit number in range of 0 to 63.
> > > + */
> > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> > In general, the macros have been avoided in this file. Suggest
> > changing this to an inline function.
> 
> That has been discussed already, and rejected for good reasons:
> http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@
> AM0PR05MB4866.eurprd05.prod.outlook.com/
Thank you for the link.
In this patch series, I see the macro being used in enum initialization (7/10 in v11) as well as in functions (8/10 in v11). Does it make sense to introduce use inline functions and use the inline functions for 8/10?
If we do this, we should document in rte_bitops.h that inline functions should be used wherever possible.

> 
> > Also, this file has uses of this macro, it would be good to replace
> > them with the new inline function.
> 
> Makes sense.
> And for consistency, it would require adding an RTE_BIT32() macro too.

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Morten Brørup]

+ Ray and Neil as ABI Policy maintainers.

> From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com]
> Sent: Tuesday, July 28, 2020 4:19 AM
> 
> <snip>
> > >
> > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for
> bit
> > > definition
> > > >
> > > > There are several drivers which duplicate bit generation macro.
> > > > Introduce a generic bit macros so that such drivers avoid
> redefining
> > > same in
> > > > multiple drivers.
> > > >
> > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > Acked-by: Matan Azrad <matan@mellanox.com>
> > > > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > > > ---
> > > > Changelog:
> > > > v4->v5:
> > > >  - Addressed comments from Morten Brørup
> > > >  - Renamed newly added macro to RTE_BIT64
> > > >  - Added doxygen comment section for the macro
> > > > v1->v2:
> > > >  - Addressed comments from Thomas and Gaten.
> > > >  - Avoided new file, added macro to rte_bitops.h
> > > > ---
> > > >  lib/librte_eal/include/rte_bitops.h | 8 ++++++++
> > > >  1 file changed, 8 insertions(+)
> > > >
> > > > diff --git a/lib/librte_eal/include/rte_bitops.h
> > > > b/lib/librte_eal/include/rte_bitops.h
> > > > index 740927f3b..ca46a110f 100644
> > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > @@ -17,6 +17,14 @@
> > > >  #include <rte_debug.h>
> > > >  #include <rte_compat.h>
> > > >
> > > > +/**
> > > > + * Get the uint64_t value for a specified bit set.
> > > > + *
> > > > + * @param nr
> > > > + *   The bit number in range of 0 to 63.
> > > > + */
> > > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> > > In general, the macros have been avoided in this file. Suggest
> > > changing this to an inline function.
> >
> > That has been discussed already, and rejected for good reasons:
> > http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@
> > AM0PR05MB4866.eurprd05.prod.outlook.com/
> Thank you for the link.
> In this patch series, I see the macro being used in enum initialization
> (7/10 in v11) as well as in functions (8/10 in v11). Does it make sense
> to introduce use inline functions and use the inline functions for
> 8/10?
> If we do this, we should document in rte_bitops.h that inline functions
> should be used wherever possible.

I would agree, but only in theory. I disagree in reality, and argue that there should only be macros for this. Here is why:

rte_byteorder.h has both RTE_BEnn() macros and rte_cpu_to_be_nn() functions, for doing the same thing at compile time or at run time. There are no compile time warnings if the wrong one is being used, so I am certain that we can find code that uses the macro where the function should be used, or vice versa.

Which opens another, higher level, question: Would it be possible to add a compile time check macro in rte_common.h for these and similar?

Furthermore: For the RTE_BITnn() operations in this patch set, I expect the compiler to generate perfectly efficient code using the macro for run time use. I.e. there would be no performance advantage by also implementing the macros as functions for run time use.

> >
> > > Also, this file has uses of this macro, it would be good to replace
> > > them with the new inline function.
> >
> > Makes sense.
> > And for consistency, it would require adding an RTE_BIT32() macro
> too.

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Gaëtan Rivet]

On 28/07/20 10:24 +0200, Morten Brørup wrote:
> + Ray and Neil as ABI Policy maintainers.
> 
> > From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com]
> > Sent: Tuesday, July 28, 2020 4:19 AM
> > 
> > <snip>
> > > >
> > > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for
> > bit
> > > > definition
> > > > >
> > > > > There are several drivers which duplicate bit generation macro.
> > > > > Introduce a generic bit macros so that such drivers avoid
> > redefining
> > > > same in
> > > > > multiple drivers.
> > > > >
> > > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > > Acked-by: Matan Azrad <matan@mellanox.com>
> > > > > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > > > > ---
> > > > > Changelog:
> > > > > v4->v5:
> > > > >  - Addressed comments from Morten Brørup
> > > > >  - Renamed newly added macro to RTE_BIT64
> > > > >  - Added doxygen comment section for the macro
> > > > > v1->v2:
> > > > >  - Addressed comments from Thomas and Gaten.
> > > > >  - Avoided new file, added macro to rte_bitops.h
> > > > > ---
> > > > >  lib/librte_eal/include/rte_bitops.h | 8 ++++++++
> > > > >  1 file changed, 8 insertions(+)
> > > > >
> > > > > diff --git a/lib/librte_eal/include/rte_bitops.h
> > > > > b/lib/librte_eal/include/rte_bitops.h
> > > > > index 740927f3b..ca46a110f 100644
> > > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > > @@ -17,6 +17,14 @@
> > > > >  #include <rte_debug.h>
> > > > >  #include <rte_compat.h>
> > > > >
> > > > > +/**
> > > > > + * Get the uint64_t value for a specified bit set.
> > > > > + *
> > > > > + * @param nr
> > > > > + *   The bit number in range of 0 to 63.
> > > > > + */
> > > > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> > > > In general, the macros have been avoided in this file. Suggest
> > > > changing this to an inline function.
> > >
> > > That has been discussed already, and rejected for good reasons:
> > > http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@
> > > AM0PR05MB4866.eurprd05.prod.outlook.com/
> > Thank you for the link.
> > In this patch series, I see the macro being used in enum initialization
> > (7/10 in v11) as well as in functions (8/10 in v11). Does it make sense
> > to introduce use inline functions and use the inline functions for
> > 8/10?
> > If we do this, we should document in rte_bitops.h that inline functions
> > should be used wherever possible.
> 
> I would agree, but only in theory. I disagree in reality, and argue that there should only be macros for this. Here is why:
> 
> rte_byteorder.h has both RTE_BEnn() macros and rte_cpu_to_be_nn() functions, for doing the same thing at compile time or at run time. There are no compile time warnings if the wrong one is being used, so I am certain that we can find code that uses the macro where the function should be used, or vice versa.
> 

Hi,

It is not clear to me, reading this thread, what is the motivation to
enforce use of inline functions? Is it perf, compiler type checking, or
usage checks?

Macros are checked at compile time when possible, though it can be
improved upon. But I agree with Morten, proposing two forms ensures devs
will sometimes use the wrong one, and we would need a practical way to
check usages.

> Which opens another, higher level, question: Would it be possible to add a compile time check macro in rte_common.h for these and similar?
> 

Can you clarify your idea? Is is something similar to:

    #define _BIT64(n) (UINT64_C(1) << (n))
    static inline uint64_t
    bit64(uint64_t n)
    {
            assert(n < 64);
            return (UINT64_C(1) << n);
    }
    /* Integer Constant Expression? */
    #define ICE_P(x) (sizeof(int) == sizeof(*(1 ? ((void*)((x) * 0l)) : (int*)1)))
    #define BIT64(n) (ICE_P(n) ? _BIT64(n) : bit64(n))

I don't think so, but this is as close as automatic compile-time check
and automatic use of proper macro vs. function I know of, did you have
something else in mind?

In this kind of code:

   #include <stdio.h>
   #include <stdint.h>
   #include <inttypes.h>
   #include <assert.h>

   enum vals {
           ZERO = 0,
           ONE = BIT64(1),
           TWO = BIT64(2),
           THREE = BIT64(3),
   };
   
   int main(void)
   {
           uint64_t x = ONE;
   
           x = BIT64(0);
           x = BIT64(1);
           x = BIT64(60);
           x = BIT64(64);
           x = BIT64(x);
   
           printf("x: 0x%" PRIx64 "\n", x);
   
           return 0;
   }

The enum is defined using the macro, x = BIT64(64); triggers the
following warning with GCC:

constant_bitop.c:6:32: warning: left shift count >= width of type [-Wshift-count-overflow]
    6 | #define _BIT64(n) (UINT64_C(1) << (n))

and x = BIT64(x); triggers the assert() at runtime.

> Furthermore: For the RTE_BITnn() operations in this patch set, I expect the compiler to generate perfectly efficient code using the macro for run time use. I.e. there would be no performance advantage by also implementing the macros as functions for run time use.
> 

Regards,
-- 
Gaëtan

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Morten Brørup]

> From: Gaëtan Rivet [mailto:grive@u256.net]
> Sent: Tuesday, July 28, 2020 11:29 AM
> 
> On 28/07/20 10:24 +0200, Morten Brørup wrote:
> > + Ray and Neil as ABI Policy maintainers.
> >
> > > From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com]
> > > Sent: Tuesday, July 28, 2020 4:19 AM
> > >
> > > <snip>
> > > > >
> > > > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro
> for
> > > bit
> > > > > definition
> > > > > >
> > > > > > There are several drivers which duplicate bit generation
> macro.
> > > > > > Introduce a generic bit macros so that such drivers avoid
> > > redefining
> > > > > same in
> > > > > > multiple drivers.
> > > > > >
> > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > > > Acked-by: Matan Azrad <matan@mellanox.com>
> > > > > > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > > > > > ---
> > > > > > Changelog:
> > > > > > v4->v5:
> > > > > >  - Addressed comments from Morten Brørup
> > > > > >  - Renamed newly added macro to RTE_BIT64
> > > > > >  - Added doxygen comment section for the macro
> > > > > > v1->v2:
> > > > > >  - Addressed comments from Thomas and Gaten.
> > > > > >  - Avoided new file, added macro to rte_bitops.h
> > > > > > ---
> > > > > >  lib/librte_eal/include/rte_bitops.h | 8 ++++++++
> > > > > >  1 file changed, 8 insertions(+)
> > > > > >
> > > > > > diff --git a/lib/librte_eal/include/rte_bitops.h
> > > > > > b/lib/librte_eal/include/rte_bitops.h
> > > > > > index 740927f3b..ca46a110f 100644
> > > > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > > > @@ -17,6 +17,14 @@
> > > > > >  #include <rte_debug.h>
> > > > > >  #include <rte_compat.h>
> > > > > >
> > > > > > +/**
> > > > > > + * Get the uint64_t value for a specified bit set.
> > > > > > + *
> > > > > > + * @param nr
> > > > > > + *   The bit number in range of 0 to 63.
> > > > > > + */
> > > > > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> > > > > In general, the macros have been avoided in this file. Suggest
> > > > > changing this to an inline function.
> > > >
> > > > That has been discussed already, and rejected for good reasons:
> > > > http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@
> > > > AM0PR05MB4866.eurprd05.prod.outlook.com/
> > > Thank you for the link.
> > > In this patch series, I see the macro being used in enum
> initialization
> > > (7/10 in v11) as well as in functions (8/10 in v11). Does it make
> sense
> > > to introduce use inline functions and use the inline functions for
> > > 8/10?
> > > If we do this, we should document in rte_bitops.h that inline
> functions
> > > should be used wherever possible.
> >
> > I would agree, but only in theory. I disagree in reality, and argue
> that there should only be macros for this. Here is why:
> >
> > rte_byteorder.h has both RTE_BEnn() macros and rte_cpu_to_be_nn()
> functions, for doing the same thing at compile time or at run time.
> There are no compile time warnings if the wrong one is being used, so I
> am certain that we can find code that uses the macro where the function
> should be used, or vice versa.
> >
> 
> Hi,
> 
> It is not clear to me, reading this thread, what is the motivation to
> enforce use of inline functions? Is it perf, compiler type checking, or
> usage checks?
> 
> Macros are checked at compile time when possible, though it can be
> improved upon. But I agree with Morten, proposing two forms ensures
> devs
> will sometimes use the wrong one, and we would need a practical way to
> check usages.
> 
> > Which opens another, higher level, question: Would it be possible to
> add a compile time check macro in rte_common.h for these and similar?
> >
> 
> Can you clarify your idea? Is is something similar to:
> 
>     #define _BIT64(n) (UINT64_C(1) << (n))
>     static inline uint64_t
>     bit64(uint64_t n)
>     {
>             assert(n < 64);
>             return (UINT64_C(1) << n);
>     }
>     /* Integer Constant Expression? */
>     #define ICE_P(x) (sizeof(int) == sizeof(*(1 ? ((void*)((x) * 0l)) :
> (int*)1)))
>     #define BIT64(n) (ICE_P(n) ? _BIT64(n) : bit64(n))
> 
> I don't think so, but this is as close as automatic compile-time check
> and automatic use of proper macro vs. function I know of, did you have
> something else in mind?

I was only thinking of adding a compile time warning if the function was being used where the macro should be used, and vice versa.

Your proposed solution for automatic use of the function or macro is even better. Thank you! And it could be used in rte_byteorder.h too.

But as I mentioned, it is a higher level discussion, so for this patch, let's settle with the macro as already provided by Parav. And the higher level discussion about how to do this generally in DPDK libraries, where both macros and functions for the same calculation are provided, can be resumed later.

> 
> In this kind of code:
> 
>    #include <stdio.h>
>    #include <stdint.h>
>    #include <inttypes.h>
>    #include <assert.h>
> 
>    enum vals {
>            ZERO = 0,
>            ONE = BIT64(1),
>            TWO = BIT64(2),
>            THREE = BIT64(3),
>    };
> 
>    int main(void)
>    {
>            uint64_t x = ONE;
> 
>            x = BIT64(0);
>            x = BIT64(1);
>            x = BIT64(60);
>            x = BIT64(64);
>            x = BIT64(x);
> 
>            printf("x: 0x%" PRIx64 "\n", x);
> 
>            return 0;
>    }
> 
> The enum is defined using the macro, x = BIT64(64); triggers the
> following warning with GCC:
> 
> constant_bitop.c:6:32: warning: left shift count >= width of type [-
> Wshift-count-overflow]
>     6 | #define _BIT64(n) (UINT64_C(1) << (n))
> 
> and x = BIT64(x); triggers the assert() at runtime.
> 
> > Furthermore: For the RTE_BITnn() operations in this patch set, I
> expect the compiler to generate perfectly efficient code using the
> macro for run time use. I.e. there would be no performance advantage by
> also implementing the macros as functions for run time use.
> >
> 
> Regards,
> --
> Gaëtan

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Honnappa Nagarahalli]

<snip>

> 
> On 28/07/20 10:24 +0200, Morten Brørup wrote:
> > + Ray and Neil as ABI Policy maintainers.
> >
> > > From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com]
> > > Sent: Tuesday, July 28, 2020 4:19 AM
> > >
> > > <snip>
> > > > >
> > > > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for
> > > bit
> > > > > definition
> > > > > >
> > > > > > There are several drivers which duplicate bit generation macro.
> > > > > > Introduce a generic bit macros so that such drivers avoid
> > > redefining
> > > > > same in
> > > > > > multiple drivers.
> > > > > >
> > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > > > Acked-by: Matan Azrad <matan@mellanox.com>
> > > > > > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> > > > > > ---
> > > > > > Changelog:
> > > > > > v4->v5:
> > > > > >  - Addressed comments from Morten Brørup
> > > > > >  - Renamed newly added macro to RTE_BIT64
> > > > > >  - Added doxygen comment section for the macro
> > > > > > v1->v2:
> > > > > >  - Addressed comments from Thomas and Gaten.
> > > > > >  - Avoided new file, added macro to rte_bitops.h
> > > > > > ---
> > > > > >  lib/librte_eal/include/rte_bitops.h | 8 ++++++++
> > > > > >  1 file changed, 8 insertions(+)
> > > > > >
> > > > > > diff --git a/lib/librte_eal/include/rte_bitops.h
> > > > > > b/lib/librte_eal/include/rte_bitops.h
> > > > > > index 740927f3b..ca46a110f 100644
> > > > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > > > @@ -17,6 +17,14 @@
> > > > > >  #include <rte_debug.h>
> > > > > >  #include <rte_compat.h>
> > > > > >
> > > > > > +/**
> > > > > > + * Get the uint64_t value for a specified bit set.
> > > > > > + *
> > > > > > + * @param nr
> > > > > > + *   The bit number in range of 0 to 63.
> > > > > > + */
> > > > > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> > > > > In general, the macros have been avoided in this file. Suggest
> > > > > changing this to an inline function.
> > > >
> > > > That has been discussed already, and rejected for good reasons:
> > > >
> http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@
> > > > AM0PR05MB4866.eurprd05.prod.outlook.com/
> > > Thank you for the link.
> > > In this patch series, I see the macro being used in enum
> > > initialization
> > > (7/10 in v11) as well as in functions (8/10 in v11). Does it make
> > > sense to introduce use inline functions and use the inline functions
> > > for 8/10?
> > > If we do this, we should document in rte_bitops.h that inline
> > > functions should be used wherever possible.
> >
> > I would agree, but only in theory. I disagree in reality, and argue that there
> should only be macros for this. Here is why:
> >
> > rte_byteorder.h has both RTE_BEnn() macros and rte_cpu_to_be_nn()
> functions, for doing the same thing at compile time or at run time. There are
> no compile time warnings if the wrong one is being used, so I am certain that
> we can find code that uses the macro where the function should be used, or
> vice versa.
Agree, there is not a suitable way to enforce the use of one over the other (other than code review).

When the APIs in rte_bitops.h were introduced, there was a discussion around using the macros. I was for using macros as it would have kept the code as well as number of APIs smaller. However, there was a decision made not to use macros and instead provide inline functions. It was nothing to do with performance. So, I am just saying that we need to follow the same principles at least for this file.

> >
> 
> Hi,
> 
> It is not clear to me, reading this thread, what is the motivation to enforce
> use of inline functions? Is it perf, compiler type checking, or usage checks?
> 
> Macros are checked at compile time when possible, though it can be
> improved upon. But I agree with Morten, proposing two forms ensures devs
> will sometimes use the wrong one, and we would need a practical way to
> check usages.
> 
> > Which opens another, higher level, question: Would it be possible to add a
> compile time check macro in rte_common.h for these and similar?
> >
> 
> Can you clarify your idea? Is is something similar to:
> 
>     #define _BIT64(n) (UINT64_C(1) << (n))
>     static inline uint64_t
>     bit64(uint64_t n)
>     {
>             assert(n < 64);
>             return (UINT64_C(1) << n);
>     }
>     /* Integer Constant Expression? */
>     #define ICE_P(x) (sizeof(int) == sizeof(*(1 ? ((void*)((x) * 0l)) : (int*)1)))
>     #define BIT64(n) (ICE_P(n) ? _BIT64(n) : bit64(n))
> 
> I don't think so, but this is as close as automatic compile-time check and
> automatic use of proper macro vs. function I know of, did you have something
> else in mind?
> 
> In this kind of code:
> 
>    #include <stdio.h>
>    #include <stdint.h>
>    #include <inttypes.h>
>    #include <assert.h>
> 
>    enum vals {
>            ZERO = 0,
>            ONE = BIT64(1),
>            TWO = BIT64(2),
>            THREE = BIT64(3),
>    };
> 
>    int main(void)
>    {
>            uint64_t x = ONE;
> 
>            x = BIT64(0);
>            x = BIT64(1);
>            x = BIT64(60);
>            x = BIT64(64);
>            x = BIT64(x);
> 
>            printf("x: 0x%" PRIx64 "\n", x);
> 
>            return 0;
>    }
> 
> The enum is defined using the macro, x = BIT64(64); triggers the following
> warning with GCC:
> 
> constant_bitop.c:6:32: warning: left shift count >= width of type [-Wshift-
> count-overflow]
>     6 | #define _BIT64(n) (UINT64_C(1) << (n))
> 
> and x = BIT64(x); triggers the assert() at runtime.
> 
> > Furthermore: For the RTE_BITnn() operations in this patch set, I expect the
> compiler to generate perfectly efficient code using the macro for run time use.
> I.e. there would be no performance advantage by also implementing the
> macros as functions for run time use.
> >
> 
> Regards,
> --
> Gaëtan

Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition

[Thomas Monjalon]

28/07/2020 17:39, Honnappa Nagarahalli:
> > On 28/07/20 10:24 +0200, Morten Brørup wrote:
> > > > From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com]
> > > > > > > --- a/lib/librte_eal/include/rte_bitops.h
> > > > > > > +++ b/lib/librte_eal/include/rte_bitops.h
> > > > > > > @@ -17,6 +17,14 @@
> > > > > > >  #include <rte_debug.h>
> > > > > > >  #include <rte_compat.h>
> > > > > > >
> > > > > > > +/**
> > > > > > > + * Get the uint64_t value for a specified bit set.
> > > > > > > + *
> > > > > > > + * @param nr
> > > > > > > + *   The bit number in range of 0 to 63.
> > > > > > > + */
> > > > > > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
> > > > > > In general, the macros have been avoided in this file. Suggest
> > > > > > changing this to an inline function.
> > > > >
> > > > > That has been discussed already, and rejected for good reasons:
> > > > >
> > http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@
> > > > > AM0PR05MB4866.eurprd05.prod.outlook.com/
> > > > Thank you for the link.
> > > > In this patch series, I see the macro being used in enum
> > > > initialization
> > > > (7/10 in v11) as well as in functions (8/10 in v11). Does it make
> > > > sense to introduce use inline functions and use the inline functions
> > > > for 8/10?
> > > > If we do this, we should document in rte_bitops.h that inline
> > > > functions should be used wherever possible.
> > >
> > > I would agree, but only in theory. I disagree in reality, and argue that there
> > should only be macros for this. Here is why:
> > >
> > > rte_byteorder.h has both RTE_BEnn() macros and rte_cpu_to_be_nn()
> > functions, for doing the same thing at compile time or at run time. There are
> > no compile time warnings if the wrong one is being used, so I am certain that
> > we can find code that uses the macro where the function should be used, or
> > vice versa.
> Agree, there is not a suitable way to enforce the use of one over the other (other than code review).
> 
> When the APIs in rte_bitops.h were introduced, there was a discussion around using the macros. I was for using macros as it would have kept the code as well as number of APIs smaller. However, there was a decision made not to use macros and instead provide inline functions. It was nothing to do with performance. So, I am just saying that we need to follow the same principles at least for this file.

I think bit definitions should be simple macros.
Even macro is a bit overkill for this simple thing.
I will merge this series.

If we want to change the philosophy of macro definitions,
it may be a larger discussion.

[dpdk-dev] [PATCH v10 02/10] drivers: fix indent of directory list

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Define each sub-directory on its own line ended with a comma,
and use a simple indent.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog:
v7->8:
 - new patch
---
 drivers/meson.build | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e76ebddfa..e2aeba931 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -2,17 +2,19 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 # Defines the order in which the drivers are buit.
-dpdk_driver_classes = ['common',
-	       'bus',
-	       'mempool', # depends on common and bus.
-	       'net',     # depends on common, bus, mempool
-	       'raw',     # depends on common, bus and net.
-	       'crypto',  # depends on common, bus and mempool (net in future).
-	       'compress', # depends on common, bus, mempool.
-	       'regex', # depends on common, bus, regexdev.
-	       'vdpa',    # depends on common, bus and mempool.
-	       'event',   # depends on common, bus, mempool and net.
-	       'baseband'] # depends on common and bus.
+dpdk_driver_classes = [
+	'common',
+	'bus',
+	'mempool', # depends on common and bus.
+	'net',     # depends on common, bus, mempool
+	'raw',     # depends on common, bus and net.
+	'crypto',  # depends on common, bus and mempool (net in future).
+	'compress', # depends on common, bus, mempool.
+	'regex', # depends on common, bus, regexdev.
+	'vdpa',    # depends on common, bus and mempool.
+	'event',   # depends on common, bus, mempool and net.
+	'baseband', # depends on common and bus.
+]
 
 disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
-- 
2.26.2

[dpdk-dev] [PATCH v10 03/10] drivers: relax dependency order

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Drivers dependencies are evaluated in the order defined per their parent
directory (also called class). This strict ordering prevents from
having pairs of drivers from two classes with different dependency
ordering. For example, if the mlx5 common code depends on the pci bus
driver, while the dpaax common code is itself a dependency of the dpaa
bus driver.  Having a strict ordering between directories bus and common
is too restrictive, as processing either common drivers or bus drivers
first leads us to missing dependencies in this scenario.

This patch makes it possible to have a more fine-grain directory list,
adding a specific driver sub-directory in the top-level drivers
subdirectory list. In this case, the isolated driver must also be removed
from its class list, and the per-class variables must be duplicated in
the isolated driver, because the call "subdir(class)" is skipped in the
isolated driver case.

Review-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog:
v8->v9:
 - Updated commit message
 - Fixed typo
---
 drivers/meson.build | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e2aeba931..038957460 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-# Defines the order in which the drivers are buit.
-dpdk_driver_classes = [
+# Defines the order of dependencies evaluation
+subdirs = [
 	'common',
 	'bus',
 	'mempool', # depends on common and bus.
@@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation')
 	default_cflags += '-Wno-format-truncation'
 endif
 
-foreach class:dpdk_driver_classes
+foreach subpath:subdirs
 	drivers = []
 	std_deps = []
 	config_flag_fmt = '' # format string used to set the value in dpdk_conf
@@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes
 	                     # the library, the dependency and to find the
 	                     # version file for linking
 
-	subdir(class)
-	class_drivers = []
+	# subpath can be either "class" or "class/driver"
+	if subpath.contains('/')
+		driver_path = subpath.split('/')
+		class = driver_path[0]
+		drivers += driver_path[1]
+	else
+		class = subpath
+		subdir(class)
+	endif
+
+	# save class name on first occurrence
+	if not dpdk_driver_classes.contains(class)
+		dpdk_driver_classes += class
+	endif
+	# get already enabled drivers of the same class
+	enabled_drivers = get_variable(class + '_drivers', [])
 
 	foreach drv:drivers
 		drv_path = join_paths(class, drv)
@@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes
 						'_disable_reason', reason)
 			endif
 		else
-			class_drivers += name
+			enabled_drivers += name
 
 			if fmt_name == ''
 				fmt_name = name
@@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes
 		endif # build
 	endforeach
 
-	set_variable(class + '_drivers', class_drivers)
+	set_variable(class + '_drivers', enabled_drivers)
 endforeach
-- 
2.26.2

[dpdk-dev] [PATCH v10 04/10] common/mlx5: fix void parameters in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index c4f9b006a..734ace2a3 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v10 05/10] regex/mlx5: fix segmentation fault during error unwinding

[Parav Pandit]

When fail to initialize the device, avoid segmentation fault while
accessing unintialized priv.

Fixes: cfc672a90b74 ("regex/mlx5: support probing")

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v9->v10:
 - Corrected type in commit log
v7->v8:
 - Rebased
 - new patch
---
 drivers/regex/mlx5/mlx5_regex.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 1ca5bfe9b..36ae9f809 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -137,17 +137,17 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	} else if (!attr.regex || attr.regexp_num_of_engines == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
 			"old FW/OFED version?");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	}
 	if (mlx5_regex_engines_status(ctx, 2)) {
 		DRV_LOG(ERR, "RegEx engine error.");
 		rte_errno = ENOMEM;
-		goto error;
+		goto dev_error;
 	}
 	priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
 			   RTE_CACHE_LINE_SIZE);
@@ -200,6 +200,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		mlx5_glue->devx_free_uar(priv->uar);
 	if (priv->regexdev)
 		rte_regexdev_unregister(priv->regexdev);
+dev_error:
 	if (ctx)
 		mlx5_glue->close_device(ctx);
 	if (priv)
-- 
2.26.2

[dpdk-dev] [PATCH v10 06/10] common/mlx5: avoid using class constructor priority

[Parav Pandit]

mlx5_common is shared library between mlx5 net, VDPA and regex PMD.
It is better to use common initialization helper instead of using
RTE_PRIORITY_CLASS priority.

Reviewed-by: David Marchand <david.marchand@redhat.com>
Suggested-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v7->v8:
 - new patch
---
 drivers/common/mlx5/mlx5_common.c               | 13 +++++++++++--
 drivers/common/mlx5/mlx5_common.h               |  3 +++
 drivers/common/mlx5/rte_common_mlx5_version.map |  1 +
 drivers/net/mlx5/mlx5.c                         |  1 +
 drivers/regex/mlx5/mlx5_regex.c                 |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.c                   |  1 +
 6 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 1a67a1b16..2b336bb2d 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 		rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE);
 }
 
+static bool mlx5_common_initialized;
+
 /**
- * Initialization routine for run-time dependency on glue library.
+ * One time innitialization routine for run-time dependency on glue library
+ * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module,
+ * must invoke in its constructor.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+void
+mlx5_common_init(void)
 {
+	if (mlx5_common_initialized)
+		return;
+
 	mlx5_glue_constructor();
+	mlx5_common_initialized = true;
 }
 
 /**
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a811eb6c9..ebe4e9ced 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id,
 			 uint64_t offset);
 extern uint8_t haswell_broadwell_cpu;
 
+__rte_internal
+void mlx5_common_init(void);
+
 #endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 132a0695f..65f25252a 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -3,6 +3,7 @@ INTERNAL {
 
 	mlx5_class_get;
 
+	mlx5_common_init;
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 647ada339..037703d2e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_pmd_init)
 {
+	mlx5_common_init();
 	/* Build the static tables for Verbs conversion. */
 	mlx5_set_ptype_table();
 	mlx5_set_cksum_table();
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 36ae9f809..4e0367052 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = {
 
 RTE_INIT(rte_mlx5_regex_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_regex_driver);
 }
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 67e77b11a..85dbcf956 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_vdpa_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_vdpa_driver);
 }
-- 
2.26.2

[dpdk-dev] [PATCH v10 07/10] common/mlx5: change class values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v7->v8:
 - updated commit message
---
 drivers/common/mlx5/mlx5_common.h | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index ebe4e9ced..864e82ff0 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -208,10 +209,10 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
-	MLX5_CLASS_REGEX,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT64(0),
+	MLX5_CLASS_VDPA = RTE_BIT64(1),
+	MLX5_CLASS_REGEX = RTE_BIT64(2),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
-- 
2.26.2

[dpdk-dev] [PATCH v10 08/10] common/mlx5: introduce layer to support multiple class drivers

[Parav Pandit]

Add generic mlx5 PCI PMD layer as part of existing common_mlx5
module. This enables multiple classes (net, regex, vdpa) PMDs
to be supported at same time.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v9->v10:
 - Fixed alignment
v8->v9:
 - Added missing LDFLAG for pci bus
 - Fixed white spaces at start of the line
---
 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   2 +
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |   2 +
 drivers/common/mlx5/mlx5_common_pci.c         | 541 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   2 +
 drivers/meson.build                           |   1 +
 10 files changed, 639 insertions(+), 11 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

diff --git a/drivers/Makefile b/drivers/Makefile
index 1551272ef..7f06162dc 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -8,8 +8,12 @@ DIRS-y += bus
 DEPDIRS-bus := common
 DIRS-y += mempool
 DEPDIRS-mempool := common bus
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
+DIRS-y += common/mlx5
+DEPDIRS-common/mlx5 := bus
+endif
 DIRS-y += net
-DEPDIRS-net := common bus mempool
+DEPDIRS-net := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband
 DEPDIRS-baseband := common bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto
@@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress
 DEPDIRS-compress := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex
-DEPDIRS-regex := common bus
+DEPDIRS-regex := common bus common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa
-DEPDIRS-vdpa := common bus mempool
+DEPDIRS-vdpa := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event
 DEPDIRS-event := common bus mempool net crypto
 DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index cbc71077c..cfb6b4dc8 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y)))
 DIRS-y += iavf
 endif
 
-ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
-DIRS-y += mlx5
-endif
-
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/common/meson.build b/drivers/common/meson.build
index 5db7e29b1..9ed4c04ba 100644
--- a/drivers/common/meson.build
+++ b/drivers/common/meson.build
@@ -6,6 +6,6 @@ if is_windows
 endif
 
 std_deps = ['eal']
-drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat']
+drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat']
 config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
 driver_name_fmt = 'rte_common_@0@'
diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile
index 6b89a6c85..4edd54104 100644
--- a/drivers/common/mlx5/Makefile
+++ b/drivers/common/mlx5/Makefile
@@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c
 SRCS-y += mlx5_common_mp.c
 SRCS-y += mlx5_common_mr.c
 SRCS-y += mlx5_malloc.c
+SRCS-y += mlx5_common_pci.c
 ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y)
 INSTALL-y-lib += $(LIB_GLUE)
 endif
@@ -51,6 +52,7 @@ LDLIBS += -libverbs -lmlx5
 endif
 
 LDLIBS += -lrte_eal -lrte_pci -lrte_kvargs -lrte_net
+LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual -UPEDANTIC
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index 70e2c1c32..8e5608703 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -1,19 +1,22 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2019 Mellanox Technologies, Ltd
 
-if not (is_linux or is_windows)
+if not is_linux
 	build = false
-	reason = 'only supported on Linux and Windows'
+	reason = 'only supported on Linux'
 	subdir_done()
 endif
 
-deps += ['hash', 'pci', 'net', 'eal', 'kvargs']
+config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
+driver_name_fmt = 'rte_common_@0@'
+deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs']
 sources += files(
 	'mlx5_devx_cmds.c',
 	'mlx5_common.c',
 	'mlx5_common_mp.c',
 	'mlx5_common_mr.c',
 	'mlx5_malloc.c',
+	'mlx5_common_pci.c',
 )
 
 cflags_options = [
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 2b336bb2d..fd818ef24 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -14,6 +14,7 @@
 #include "mlx5_common_os.h"
 #include "mlx5_common_utils.h"
 #include "mlx5_malloc.h"
+#include "mlx5_common_pci.h"
 
 int mlx5_common_logtype;
 
@@ -100,6 +101,7 @@ mlx5_common_init(void)
 		return;
 
 	mlx5_glue_constructor();
+	mlx5_common_pci_init();
 	mlx5_common_initialized = true;
 }
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
new file mode 100644
index 000000000..1c778b72b
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -0,0 +1,541 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies Ltd
+ */
+
+#include <stdlib.h>
+#include <rte_malloc.h>
+#include "mlx5_common_utils.h"
+#include "mlx5_common_pci.h"
+
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of drivers. */
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int driver_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .driver_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .driver_class = MLX5_CLASS_NET },
+	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	MLX5_CLASS_REGEX,
+	/* New class combination should be added here.
+	 * For example a new multi class device combination
+	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
+	 */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].driver_class;
+	}
+	return -EINVAL;
+}
+
+static struct mlx5_pci_driver *
+driver_get(uint32_t class)
+{
+	struct mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->driver_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+static bool
+mlx5_bus_match(const struct mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = driver_get(RTE_BIT64(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT64(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT64(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv,
+	      struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->driver_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->driver_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->driver_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple drivers for a PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		      struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more drivers for a PCI device.
+ *
+ * This function removes all drivers probed for a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_common_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload drivers. */
+	ret = drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+			uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver = NULL;
+	struct mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_map) {
+			ret = driver->pci_driver.dma_map(pci_dev, addr,
+							 iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == driver)
+			break;
+		if (device_class_enabled(dev, temp->driver_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+			  uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_unmap) {
+			local_ret = driver->pci_driver.dma_unmap(pci_dev, addr,
+								 iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5 drivers. */
+static struct rte_pci_id *mlx5_pci_id_table;
+
+static struct rte_pci_driver mlx5_pci_driver = {
+	.driver = {
+		.name = "mlx5_pci",
+	},
+	.probe = mlx5_common_pci_probe,
+	.remove = mlx5_common_pci_remove,
+	.dma_map = mlx5_common_pci_dma_map,
+	.dma_unmap = mlx5_common_pci_dma_unmap,
+};
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table,
+	      int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == table[i].device_id &&
+		    id->vendor_id == table[i].vendor_id &&
+		    id->subsystem_vendor_id == table[i].subsystem_vendor_id &&
+		    id->subsystem_device_id == table[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(struct rte_pci_id *new_table, int *next_idx,
+	      const struct rte_pci_id *id_table)
+{
+	/* Traverse the id_table, check if entry exists in new_table;
+	 * Add non duplicate entries to new table.
+	 */
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, new_table, *next_idx)) {
+			/* New entry; add to the table. */
+			new_table[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_update(const struct rte_pci_id *driver_id_table)
+{
+	const struct rte_pci_id *id_iter;
+	struct rte_pci_id *updated_table;
+	struct rte_pci_id *old_table;
+	int num_ids = 0;
+	int i = 0;
+
+	old_table = mlx5_pci_id_table;
+	if (old_table)
+		num_ids = pci_id_table_size_get(old_table);
+	num_ids += pci_id_table_size_get(driver_id_table);
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	updated_table = calloc(num_ids, sizeof(*updated_table));
+	if (!updated_table)
+		return -ENOMEM;
+	if (TAILQ_EMPTY(&drv_list)) {
+		/* Copy the first driver's ID table. */
+		for (id_iter = driver_id_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+	} else {
+		/* First copy existing table entries. */
+		for (id_iter = old_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+		/* New id to be added at the end of current ID table. */
+		pci_id_insert(updated_table, &i, driver_id_table);
+	}
+	/* Terminate table with empty entry. */
+	updated_table[i].vendor_id = 0;
+	mlx5_pci_driver.id_table = updated_table;
+	mlx5_pci_id_table = updated_table;
+	if (old_table)
+		free(old_table);
+	return 0;
+}
+
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver)
+{
+	int ret;
+
+	ret = pci_ids_table_update(driver->pci_driver.id_table);
+	if (ret)
+		return;
+	mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags;
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
+
+void mlx5_common_pci_init(void)
+{
+	const struct rte_pci_id empty_table[] = {
+		{
+			.vendor_id = 0
+		},
+	};
+
+	/* All mlx5 PMDs constructor runs at same priority. So any of the PMD
+	 * including this one can register the PCI table first. If any other
+	 * PMD(s) have registered the PCI ID table, No need to register an empty
+	 * default one.
+	 */
+	if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table))
+		return;
+	rte_pci_register(&mlx5_pci_driver);
+}
+
+RTE_FINI(mlx5_common_pci_finish)
+{
+	if (mlx5_pci_id_table != NULL) {
+		/* Constructor doesn't register with PCI bus if it failed
+		 * to build the table.
+		 */
+		rte_pci_unregister(&mlx5_pci_driver);
+		free(mlx5_pci_id_table);
+	}
+}
+RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__);
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
new file mode 100644
index 000000000..41b73e17a
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _MLX5_COMMON_PCI_H_
+#define _MLX5_COMMON_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Driver Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This layer enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci driver.
+ *
+ * -----------    ------------    -------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5    |
+ * | net pmd |    | vdpa pmd |    | regex pmd |
+ * -----------    ------------    -------------
+ *      \              |                 /
+ *       \             |                /
+ *        \       --------------       /
+ *         \______|   mlx5     |_____ /
+ *                | pci common |
+ *                --------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+void mlx5_common_pci_init(void);
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t driver_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _MLX5_COMMON_PCI_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 65f25252a..73cf72548 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -91,5 +91,7 @@ INTERNAL {
 	mlx5_malloc;
 	mlx5_realloc;
 	mlx5_free;
+
+	mlx5_pci_driver_register;
 };
 
diff --git a/drivers/meson.build b/drivers/meson.build
index 038957460..5f9526557 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -5,6 +5,7 @@
 subdirs = [
 	'common',
 	'bus',
+	'common/mlx5', # depends on bus.
 	'mempool', # depends on common and bus.
 	'net',     # depends on common, bus, mempool
 	'raw',     # depends on common, bus and net.
-- 
2.26.2

[dpdk-dev] [PATCH v10 09/10] common/mlx5: register class drivers through common layer

[Parav Pandit]

Migrate mlx5 net, vdpa and regex PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v8->v9:
 - Removed leftover inclusion of mlx5_pci bus include directory
---
 drivers/common/mlx5/mlx5_common_pci.c |  6 ++----
 drivers/net/mlx5/Makefile             |  1 -
 drivers/net/mlx5/linux/mlx5_os.c      |  1 -
 drivers/net/mlx5/mlx5.c               | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h               |  1 -
 drivers/regex/mlx5/meson.build        |  2 +-
 drivers/regex/mlx5/mlx5_regex.c       | 21 ++++++++++++---------
 drivers/vdpa/mlx5/Makefile            |  2 +-
 drivers/vdpa/mlx5/meson.build         |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c         | 23 +++++++++++++----------
 10 files changed, 44 insertions(+), 39 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
index 1c778b72b..7bb5440c3 100644
--- a/drivers/common/mlx5/mlx5_common_pci.c
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -34,10 +34,8 @@ static const unsigned int mlx5_class_combinations[] = {
 	MLX5_CLASS_NET,
 	MLX5_CLASS_VDPA,
 	MLX5_CLASS_REGEX,
-	/* New class combination should be added here.
-	 * For example a new multi class device combination
-	 * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR.
-	 */
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
+	/* New class combination should be added here. */
 };
 
 static int
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index 253faf909..568c77241 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -55,7 +55,6 @@ LDLIBS += -lrte_common_mlx5
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index fa3b02787..97d7b0e7b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1537,7 +1537,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 037703d2e..1e4c695f8 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -30,6 +30,7 @@
 #include <mlx5_common.h>
 #include <mlx5_common_os.h>
 #include <mlx5_common_mp.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_malloc.h>
 
 #include "mlx5_defs.h"
@@ -2091,16 +2092,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct mlx5_pci_driver mlx5_driver = {
+	.driver_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /* Initialize driver log type. */
@@ -2117,7 +2121,7 @@ RTE_INIT(rte_mlx5_pmd_init)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a92194d2d..78d6eb728 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -119,7 +119,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/regex/mlx5/meson.build b/drivers/regex/mlx5/meson.build
index 7f800f2e3..d7cb2c572 100644
--- a/drivers/regex/mlx5/meson.build
+++ b/drivers/regex/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_regex'
-deps += ['common_mlx5', 'bus_pci', 'eal', 'regexdev']
+deps += ['common_mlx5', 'eal', 'regexdev']
 sources = files(
 	'mlx5_regex.c',
 	'mlx5_rxp.c',
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 4e0367052..ae9f00189 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -5,12 +5,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_regexdev.h>
 #include <rte_regexdev_core.h>
 #include <rte_regexdev_driver.h>
 
+#include <mlx5_common_pci.h>
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
@@ -246,21 +246,24 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_regex_driver = {
-	.driver = {
-		.name = "mlx5_regex",
+static struct mlx5_pci_driver mlx5_regex_driver = {
+	.driver_class = MLX5_CLASS_REGEX,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_regex",
+		},
+		.id_table = mlx5_regex_pci_id_map,
+		.probe = mlx5_regex_pci_probe,
+		.remove = mlx5_regex_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_regex_pci_id_map,
-	.probe = mlx5_regex_pci_probe,
-	.remove = mlx5_regex_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_INIT(rte_mlx5_regex_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_regex_driver);
+		mlx5_pci_driver_register(&mlx5_regex_driver);
 }
 
 RTE_LOG_REGISTER(mlx5_regex_logtype, pmd.regex.mlx5, NOTICE)
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..d46699fbc 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -30,7 +30,7 @@ CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..254a8ec14 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 85dbcf956..ffe2f00ac 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,12 +11,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
 #include <mlx5_nl.h>
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct mlx5_pci_driver mlx5_vdpa_driver = {
+	.driver_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
@@ -848,7 +851,7 @@ RTE_INIT(rte_mlx5_vdpa_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
-- 
2.26.2

[dpdk-dev] [PATCH v10 10/10] common/mlx5: remove class check from class drivers

[Parav Pandit]

Now that mlx5_pci PMD checks for enabled classes and performs
probe(), remove() of associated classes, individual class driver
does not need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 5 files changed, 1 insertion(+), 51 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index fd818ef24..06f0a6400 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -20,43 +20,6 @@ int mlx5_common_logtype;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 864e82ff0..2cdb226f3 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -247,8 +247,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 73cf72548..7729d0dd0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,9 +1,8 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_init;
+
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 97d7b0e7b..fc339affa 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1524,11 +1524,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	unsigned int dev_config_vf;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index ffe2f00ac..c0b87bcc0 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

[dpdk-dev] [PATCH v11 00/10] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 common driver layer to support multiple
class of devices for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
mlx5 common module is extended as generic PCI PMD.
This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded
on a single PCI Device.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Fixed indentation for directory listing
Patch-3 Relax dependency order
Patch-4 Fixes compilation error
Patch-5 Fixes segmentation fault in regex probe error unwind path
Patch-6 Avoid using class constructor priority
Patch-7 Change class values as bits
Patch-8 Introduce mlx5 layer to support multiple class drivers
Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API
        instead of rte PCI bus API
Patch-10 Removed class check code from class drivers 

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       ---------------     /
         \______| mlx5 common |____/
                |    pmd      |
                --------------- 
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 common driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD define their
  specific PCI ID tables. mlx5 common driver probes each
  individual class driver(s) for maching and enabled classes.
- mlx5 pci layer is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI common driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci common PMD, a common driver is
implemented which exposes class registration API.
However, common PMD model fits better with existing DPDK design
similar to ifpga driver. Class registration API need to create a
new callbacks and ID signature; instead it is better to utilize
current well defined methods.

2. Enhance pci core to allow multiple driver binding to single
rte PCI device.
This approach is not taken, because peer drivers using one PCI
device won't be aware of other's presence. This requires
cross-driver syncronization of who initializes common resources
(such as irq, eq and more).
This also requires refcounting common objects etc among peer drivers.
Instead of layered approach delivers and allows putting common resource
sharing, setup code in common driver.
It also eliminates peer blind zone problem as bottom pci layer provides
necessary setup without any reference counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf used
between net and regex pmd so that same mbuf use across multiple
device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list of classes
at command line such as,
./testpmd -w <PCI BDF>,class=net:regex
./testpmd -w <PCI BDF>,class=vdpa

Changelog:
v10->v11:
 - Added vdpa + regex combination to class combination table
 - All class definitions are moved to single patch
v9->v10:
 - Fixed checkpatch warnings related to spelling mistakes and alignment
v8->v9:
 - Updated commit message
 - Fixed LD_FLAGS
 - Fixed white space to tab
 - Fixed Makefile to have dependency on common/mlx5 for parallel builds
v7->v8:
 - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5
 - Avoided new RTE PRIORITY
 - Removed mlx5 common to use RTE_PRIORITY_CLASS
v6->v7:
 - Updated release notes
v5->v6:
 - Fixed compilation failure in parallel build for shared lib
v4->v5:
 - Squash the maintainers update path with other patch which adds the
   bus
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver


Parav Pandit (8):
  eal: introduce macro for bit definition
  common/mlx5: fix void parameters in glue wrappers
  regex/mlx5: fix segmentation fault during error unwinding
  common/mlx5: avoid using class constructor priority
  common/mlx5: change class values as bits
  common/mlx5: introduce layer to support multiple class drivers
  common/mlx5: register class drivers through common layer
  common/mlx5: remove class check from class drivers

Thomas Monjalon (2):
  drivers: fix indent of directory list
  drivers: relax dependency order

 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   2 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |  52 +-
 drivers/common/mlx5/mlx5_common.h             |  12 +-
 drivers/common/mlx5/mlx5_common_pci.c         | 540 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   4 +-
 drivers/meson.build                           |  51 +-
 drivers/net/mlx5/Makefile                     |   1 -
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/mlx5.c                       |  25 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/regex/mlx5/meson.build                |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |  29 +-
 drivers/vdpa/mlx5/Makefile                    |   2 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  29 +-
 lib/librte_eal/include/rte_bitops.h           |   8 +
 22 files changed, 749 insertions(+), 123 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

-- 
2.26.2

[dpdk-dev] [PATCH v11 01/10] eal: introduce macro for bit definition

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
Changelog:
v4->v5:
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..ca46a110f 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,14 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+/**
+ * Get the uint64_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 63.
+ */
+#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
-- 
2.26.2

[dpdk-dev] [PATCH v11 02/10] drivers: fix indent of directory list

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Define each sub-directory on its own line ended with a comma,
and use a simple indent.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog:
v7->8:
 - new patch
---
 drivers/meson.build | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e76ebddfa..e2aeba931 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -2,17 +2,19 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 # Defines the order in which the drivers are buit.
-dpdk_driver_classes = ['common',
-	       'bus',
-	       'mempool', # depends on common and bus.
-	       'net',     # depends on common, bus, mempool
-	       'raw',     # depends on common, bus and net.
-	       'crypto',  # depends on common, bus and mempool (net in future).
-	       'compress', # depends on common, bus, mempool.
-	       'regex', # depends on common, bus, regexdev.
-	       'vdpa',    # depends on common, bus and mempool.
-	       'event',   # depends on common, bus, mempool and net.
-	       'baseband'] # depends on common and bus.
+dpdk_driver_classes = [
+	'common',
+	'bus',
+	'mempool', # depends on common and bus.
+	'net',     # depends on common, bus, mempool
+	'raw',     # depends on common, bus and net.
+	'crypto',  # depends on common, bus and mempool (net in future).
+	'compress', # depends on common, bus, mempool.
+	'regex', # depends on common, bus, regexdev.
+	'vdpa',    # depends on common, bus and mempool.
+	'event',   # depends on common, bus, mempool and net.
+	'baseband', # depends on common and bus.
+]
 
 disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
-- 
2.26.2

[dpdk-dev] [PATCH v11 03/10] drivers: relax dependency order

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Drivers dependencies are evaluated in the order defined per their parent
directory (also called class). This strict ordering prevents from
having pairs of drivers from two classes with different dependency
ordering. For example, if the mlx5 common code depends on the pci bus
driver, while the dpaax common code is itself a dependency of the dpaa
bus driver.  Having a strict ordering between directories bus and common
is too restrictive, as processing either common drivers or bus drivers
first leads us to missing dependencies in this scenario.

This patch makes it possible to have a more fine-grain directory list,
adding a specific driver sub-directory in the top-level drivers
subdirectory list. In this case, the isolated driver must also be removed
from its class list, and the per-class variables must be duplicated in
the isolated driver, because the call "subdir(class)" is skipped in the
isolated driver case.

Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog:
v8->v9:
 - Updated commit message
 - Fixed typo
---
 drivers/meson.build | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e2aeba931..038957460 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-# Defines the order in which the drivers are buit.
-dpdk_driver_classes = [
+# Defines the order of dependencies evaluation
+subdirs = [
 	'common',
 	'bus',
 	'mempool', # depends on common and bus.
@@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation')
 	default_cflags += '-Wno-format-truncation'
 endif
 
-foreach class:dpdk_driver_classes
+foreach subpath:subdirs
 	drivers = []
 	std_deps = []
 	config_flag_fmt = '' # format string used to set the value in dpdk_conf
@@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes
 	                     # the library, the dependency and to find the
 	                     # version file for linking
 
-	subdir(class)
-	class_drivers = []
+	# subpath can be either "class" or "class/driver"
+	if subpath.contains('/')
+		driver_path = subpath.split('/')
+		class = driver_path[0]
+		drivers += driver_path[1]
+	else
+		class = subpath
+		subdir(class)
+	endif
+
+	# save class name on first occurrence
+	if not dpdk_driver_classes.contains(class)
+		dpdk_driver_classes += class
+	endif
+	# get already enabled drivers of the same class
+	enabled_drivers = get_variable(class + '_drivers', [])
 
 	foreach drv:drivers
 		drv_path = join_paths(class, drv)
@@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes
 						'_disable_reason', reason)
 			endif
 		else
-			class_drivers += name
+			enabled_drivers += name
 
 			if fmt_name == ''
 				fmt_name = name
@@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes
 		endif # build
 	endforeach
 
-	set_variable(class + '_drivers', class_drivers)
+	set_variable(class + '_drivers', enabled_drivers)
 endforeach
-- 
2.26.2

[dpdk-dev] [PATCH v11 04/10] common/mlx5: fix void parameters in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index c4f9b006a..734ace2a3 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v11 05/10] regex/mlx5: fix segmentation fault during error unwinding

[Parav Pandit]

When fail to initialize the device, avoid segmentation fault while
accessing uninitialized priv.

Fixes: cfc672a90b74 ("regex/mlx5: support probing")

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v9->v10:
 - Corrected type in commit log
v7->v8:
 - Rebased
 - new patch
---
 drivers/regex/mlx5/mlx5_regex.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 1ca5bfe9b..36ae9f809 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -137,17 +137,17 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	} else if (!attr.regex || attr.regexp_num_of_engines == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
 			"old FW/OFED version?");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	}
 	if (mlx5_regex_engines_status(ctx, 2)) {
 		DRV_LOG(ERR, "RegEx engine error.");
 		rte_errno = ENOMEM;
-		goto error;
+		goto dev_error;
 	}
 	priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
 			   RTE_CACHE_LINE_SIZE);
@@ -200,6 +200,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		mlx5_glue->devx_free_uar(priv->uar);
 	if (priv->regexdev)
 		rte_regexdev_unregister(priv->regexdev);
+dev_error:
 	if (ctx)
 		mlx5_glue->close_device(ctx);
 	if (priv)
-- 
2.26.2

[dpdk-dev] [PATCH v11 06/10] common/mlx5: avoid using class constructor priority

[Parav Pandit]

mlx5_common is shared library between mlx5 net, VDPA and regex PMD.
It is better to use common initialization helper instead of using
RTE_PRIORITY_CLASS priority.

Reviewed-by: David Marchand <david.marchand@redhat.com>
Suggested-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v7->v8:
 - new patch
---
 drivers/common/mlx5/mlx5_common.c               | 13 +++++++++++--
 drivers/common/mlx5/mlx5_common.h               |  3 +++
 drivers/common/mlx5/rte_common_mlx5_version.map |  1 +
 drivers/net/mlx5/mlx5.c                         |  1 +
 drivers/regex/mlx5/mlx5_regex.c                 |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.c                   |  1 +
 6 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 1a67a1b16..2b336bb2d 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 		rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE);
 }
 
+static bool mlx5_common_initialized;
+
 /**
- * Initialization routine for run-time dependency on glue library.
+ * One time innitialization routine for run-time dependency on glue library
+ * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module,
+ * must invoke in its constructor.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+void
+mlx5_common_init(void)
 {
+	if (mlx5_common_initialized)
+		return;
+
 	mlx5_glue_constructor();
+	mlx5_common_initialized = true;
 }
 
 /**
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a811eb6c9..ebe4e9ced 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id,
 			 uint64_t offset);
 extern uint8_t haswell_broadwell_cpu;
 
+__rte_internal
+void mlx5_common_init(void);
+
 #endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 132a0695f..65f25252a 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -3,6 +3,7 @@ INTERNAL {
 
 	mlx5_class_get;
 
+	mlx5_common_init;
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 647ada339..037703d2e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_pmd_init)
 {
+	mlx5_common_init();
 	/* Build the static tables for Verbs conversion. */
 	mlx5_set_ptype_table();
 	mlx5_set_cksum_table();
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 36ae9f809..4e0367052 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = {
 
 RTE_INIT(rte_mlx5_regex_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_regex_driver);
 }
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 67e77b11a..85dbcf956 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_vdpa_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_vdpa_driver);
 }
-- 
2.26.2

[dpdk-dev] [PATCH v11 07/10] common/mlx5: change class values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v7->v8:
 - updated commit message
---
 drivers/common/mlx5/mlx5_common.h | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index ebe4e9ced..864e82ff0 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -208,10 +209,10 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
-	MLX5_CLASS_REGEX,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT64(0),
+	MLX5_CLASS_VDPA = RTE_BIT64(1),
+	MLX5_CLASS_REGEX = RTE_BIT64(2),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
-- 
2.26.2

[dpdk-dev] [PATCH v11 08/10] common/mlx5: introduce layer to support multiple class drivers

[Parav Pandit]

Add generic mlx5 PCI PMD layer as part of existing common_mlx5
module. This enables multiple classes (net, regex, vdpa) PMDs
to be supported at same time.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v10->v11:
 - Moved class combination hunk from next patch to this one
v9->v10:
 - Fixed alignment
v8->v9:
 - Added missing LDFLAG for pci bus
 - Fixed white spaces at start of the line
---
 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   2 +
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |   2 +
 drivers/common/mlx5/mlx5_common_pci.c         | 540 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   2 +
 drivers/meson.build                           |   1 +
 10 files changed, 638 insertions(+), 11 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

diff --git a/drivers/Makefile b/drivers/Makefile
index 1551272ef..7f06162dc 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -8,8 +8,12 @@ DIRS-y += bus
 DEPDIRS-bus := common
 DIRS-y += mempool
 DEPDIRS-mempool := common bus
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
+DIRS-y += common/mlx5
+DEPDIRS-common/mlx5 := bus
+endif
 DIRS-y += net
-DEPDIRS-net := common bus mempool
+DEPDIRS-net := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband
 DEPDIRS-baseband := common bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto
@@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress
 DEPDIRS-compress := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex
-DEPDIRS-regex := common bus
+DEPDIRS-regex := common bus common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa
-DEPDIRS-vdpa := common bus mempool
+DEPDIRS-vdpa := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event
 DEPDIRS-event := common bus mempool net crypto
 DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index cbc71077c..cfb6b4dc8 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y)))
 DIRS-y += iavf
 endif
 
-ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
-DIRS-y += mlx5
-endif
-
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/common/meson.build b/drivers/common/meson.build
index 5db7e29b1..9ed4c04ba 100644
--- a/drivers/common/meson.build
+++ b/drivers/common/meson.build
@@ -6,6 +6,6 @@ if is_windows
 endif
 
 std_deps = ['eal']
-drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat']
+drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat']
 config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
 driver_name_fmt = 'rte_common_@0@'
diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile
index 6b89a6c85..4edd54104 100644
--- a/drivers/common/mlx5/Makefile
+++ b/drivers/common/mlx5/Makefile
@@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c
 SRCS-y += mlx5_common_mp.c
 SRCS-y += mlx5_common_mr.c
 SRCS-y += mlx5_malloc.c
+SRCS-y += mlx5_common_pci.c
 ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y)
 INSTALL-y-lib += $(LIB_GLUE)
 endif
@@ -51,6 +52,7 @@ LDLIBS += -libverbs -lmlx5
 endif
 
 LDLIBS += -lrte_eal -lrte_pci -lrte_kvargs -lrte_net
+LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual -UPEDANTIC
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index 70e2c1c32..8e5608703 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -1,19 +1,22 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2019 Mellanox Technologies, Ltd
 
-if not (is_linux or is_windows)
+if not is_linux
 	build = false
-	reason = 'only supported on Linux and Windows'
+	reason = 'only supported on Linux'
 	subdir_done()
 endif
 
-deps += ['hash', 'pci', 'net', 'eal', 'kvargs']
+config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
+driver_name_fmt = 'rte_common_@0@'
+deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs']
 sources += files(
 	'mlx5_devx_cmds.c',
 	'mlx5_common.c',
 	'mlx5_common_mp.c',
 	'mlx5_common_mr.c',
 	'mlx5_malloc.c',
+	'mlx5_common_pci.c',
 )
 
 cflags_options = [
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 2b336bb2d..fd818ef24 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -14,6 +14,7 @@
 #include "mlx5_common_os.h"
 #include "mlx5_common_utils.h"
 #include "mlx5_malloc.h"
+#include "mlx5_common_pci.h"
 
 int mlx5_common_logtype;
 
@@ -100,6 +101,7 @@ mlx5_common_init(void)
 		return;
 
 	mlx5_glue_constructor();
+	mlx5_common_pci_init();
 	mlx5_common_initialized = true;
 }
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
new file mode 100644
index 000000000..d4ff0392f
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -0,0 +1,540 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies Ltd
+ */
+
+#include <stdlib.h>
+#include <rte_malloc.h>
+#include "mlx5_common_utils.h"
+#include "mlx5_common_pci.h"
+
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of drivers. */
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int driver_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .driver_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .driver_class = MLX5_CLASS_NET },
+	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	MLX5_CLASS_REGEX,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
+	/* New class combination should be added here. */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].driver_class;
+	}
+	return -EINVAL;
+}
+
+static struct mlx5_pci_driver *
+driver_get(uint32_t class)
+{
+	struct mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->driver_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+static bool
+mlx5_bus_match(const struct mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = driver_get(RTE_BIT64(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT64(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT64(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv,
+	      struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->driver_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->driver_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->driver_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple drivers for a PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		      struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more drivers for a PCI device.
+ *
+ * This function removes all drivers probed for a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_common_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload drivers. */
+	ret = drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+			uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver = NULL;
+	struct mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_map) {
+			ret = driver->pci_driver.dma_map(pci_dev, addr,
+							 iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == driver)
+			break;
+		if (device_class_enabled(dev, temp->driver_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+			  uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_unmap) {
+			local_ret = driver->pci_driver.dma_unmap(pci_dev, addr,
+								 iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5 drivers. */
+static struct rte_pci_id *mlx5_pci_id_table;
+
+static struct rte_pci_driver mlx5_pci_driver = {
+	.driver = {
+		.name = "mlx5_pci",
+	},
+	.probe = mlx5_common_pci_probe,
+	.remove = mlx5_common_pci_remove,
+	.dma_map = mlx5_common_pci_dma_map,
+	.dma_unmap = mlx5_common_pci_dma_unmap,
+};
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table,
+	      int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == table[i].device_id &&
+		    id->vendor_id == table[i].vendor_id &&
+		    id->subsystem_vendor_id == table[i].subsystem_vendor_id &&
+		    id->subsystem_device_id == table[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(struct rte_pci_id *new_table, int *next_idx,
+	      const struct rte_pci_id *id_table)
+{
+	/* Traverse the id_table, check if entry exists in new_table;
+	 * Add non duplicate entries to new table.
+	 */
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, new_table, *next_idx)) {
+			/* New entry; add to the table. */
+			new_table[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_update(const struct rte_pci_id *driver_id_table)
+{
+	const struct rte_pci_id *id_iter;
+	struct rte_pci_id *updated_table;
+	struct rte_pci_id *old_table;
+	int num_ids = 0;
+	int i = 0;
+
+	old_table = mlx5_pci_id_table;
+	if (old_table)
+		num_ids = pci_id_table_size_get(old_table);
+	num_ids += pci_id_table_size_get(driver_id_table);
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	updated_table = calloc(num_ids, sizeof(*updated_table));
+	if (!updated_table)
+		return -ENOMEM;
+	if (TAILQ_EMPTY(&drv_list)) {
+		/* Copy the first driver's ID table. */
+		for (id_iter = driver_id_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+	} else {
+		/* First copy existing table entries. */
+		for (id_iter = old_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+		/* New id to be added at the end of current ID table. */
+		pci_id_insert(updated_table, &i, driver_id_table);
+	}
+	/* Terminate table with empty entry. */
+	updated_table[i].vendor_id = 0;
+	mlx5_pci_driver.id_table = updated_table;
+	mlx5_pci_id_table = updated_table;
+	if (old_table)
+		free(old_table);
+	return 0;
+}
+
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver)
+{
+	int ret;
+
+	ret = pci_ids_table_update(driver->pci_driver.id_table);
+	if (ret)
+		return;
+	mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags;
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
+
+void mlx5_common_pci_init(void)
+{
+	const struct rte_pci_id empty_table[] = {
+		{
+			.vendor_id = 0
+		},
+	};
+
+	/* All mlx5 PMDs constructor runs at same priority. So any of the PMD
+	 * including this one can register the PCI table first. If any other
+	 * PMD(s) have registered the PCI ID table, No need to register an empty
+	 * default one.
+	 */
+	if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table))
+		return;
+	rte_pci_register(&mlx5_pci_driver);
+}
+
+RTE_FINI(mlx5_common_pci_finish)
+{
+	if (mlx5_pci_id_table != NULL) {
+		/* Constructor doesn't register with PCI bus if it failed
+		 * to build the table.
+		 */
+		rte_pci_unregister(&mlx5_pci_driver);
+		free(mlx5_pci_id_table);
+	}
+}
+RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__);
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
new file mode 100644
index 000000000..41b73e17a
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _MLX5_COMMON_PCI_H_
+#define _MLX5_COMMON_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Driver Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This layer enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci driver.
+ *
+ * -----------    ------------    -------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5    |
+ * | net pmd |    | vdpa pmd |    | regex pmd |
+ * -----------    ------------    -------------
+ *      \              |                 /
+ *       \             |                /
+ *        \       --------------       /
+ *         \______|   mlx5     |_____ /
+ *                | pci common |
+ *                --------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+void mlx5_common_pci_init(void);
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t driver_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _MLX5_COMMON_PCI_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 65f25252a..73cf72548 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -91,5 +91,7 @@ INTERNAL {
 	mlx5_malloc;
 	mlx5_realloc;
 	mlx5_free;
+
+	mlx5_pci_driver_register;
 };
 
diff --git a/drivers/meson.build b/drivers/meson.build
index 038957460..5f9526557 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -5,6 +5,7 @@
 subdirs = [
 	'common',
 	'bus',
+	'common/mlx5', # depends on bus.
 	'mempool', # depends on common and bus.
 	'net',     # depends on common, bus, mempool
 	'raw',     # depends on common, bus and net.
-- 
2.26.2

[dpdk-dev] [PATCH v11 09/10] common/mlx5: register class drivers through common layer

[Parav Pandit]

Migrate mlx5 net, vdpa and regex PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v10->v11:
 - Moved new class combinations to previous patch
v8->v9:
 - Removed leftover inclusion of mlx5_pci bus include directory
---
 drivers/net/mlx5/Makefile        |  1 -
 drivers/net/mlx5/linux/mlx5_os.c |  1 -
 drivers/net/mlx5/mlx5.c          | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h          |  1 -
 drivers/regex/mlx5/meson.build   |  2 +-
 drivers/regex/mlx5/mlx5_regex.c  | 21 ++++++++++++---------
 drivers/vdpa/mlx5/Makefile       |  2 +-
 drivers/vdpa/mlx5/meson.build    |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c    | 23 +++++++++++++----------
 9 files changed, 42 insertions(+), 35 deletions(-)

diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index 253faf909..568c77241 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -55,7 +55,6 @@ LDLIBS += -lrte_common_mlx5
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index fa3b02787..97d7b0e7b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1537,7 +1537,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 037703d2e..1e4c695f8 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -30,6 +30,7 @@
 #include <mlx5_common.h>
 #include <mlx5_common_os.h>
 #include <mlx5_common_mp.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_malloc.h>
 
 #include "mlx5_defs.h"
@@ -2091,16 +2092,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct mlx5_pci_driver mlx5_driver = {
+	.driver_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /* Initialize driver log type. */
@@ -2117,7 +2121,7 @@ RTE_INIT(rte_mlx5_pmd_init)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a92194d2d..78d6eb728 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -119,7 +119,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/regex/mlx5/meson.build b/drivers/regex/mlx5/meson.build
index 7f800f2e3..d7cb2c572 100644
--- a/drivers/regex/mlx5/meson.build
+++ b/drivers/regex/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_regex'
-deps += ['common_mlx5', 'bus_pci', 'eal', 'regexdev']
+deps += ['common_mlx5', 'eal', 'regexdev']
 sources = files(
 	'mlx5_regex.c',
 	'mlx5_rxp.c',
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 4e0367052..ae9f00189 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -5,12 +5,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_regexdev.h>
 #include <rte_regexdev_core.h>
 #include <rte_regexdev_driver.h>
 
+#include <mlx5_common_pci.h>
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
@@ -246,21 +246,24 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_regex_driver = {
-	.driver = {
-		.name = "mlx5_regex",
+static struct mlx5_pci_driver mlx5_regex_driver = {
+	.driver_class = MLX5_CLASS_REGEX,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_regex",
+		},
+		.id_table = mlx5_regex_pci_id_map,
+		.probe = mlx5_regex_pci_probe,
+		.remove = mlx5_regex_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_regex_pci_id_map,
-	.probe = mlx5_regex_pci_probe,
-	.remove = mlx5_regex_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_INIT(rte_mlx5_regex_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_regex_driver);
+		mlx5_pci_driver_register(&mlx5_regex_driver);
 }
 
 RTE_LOG_REGISTER(mlx5_regex_logtype, pmd.regex.mlx5, NOTICE)
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..d46699fbc 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -30,7 +30,7 @@ CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..254a8ec14 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 85dbcf956..ffe2f00ac 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,12 +11,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
 #include <mlx5_nl.h>
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct mlx5_pci_driver mlx5_vdpa_driver = {
+	.driver_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
@@ -848,7 +851,7 @@ RTE_INIT(rte_mlx5_vdpa_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
-- 
2.26.2

[dpdk-dev] [PATCH v11 10/10] common/mlx5: remove class check from class drivers

[Parav Pandit]

Now that mlx5_pci PMD checks for enabled classes and performs
probe(), remove() of associated classes, individual class driver
does not need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 5 files changed, 1 insertion(+), 51 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index fd818ef24..06f0a6400 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -20,43 +20,6 @@ int mlx5_common_logtype;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 864e82ff0..2cdb226f3 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -247,8 +247,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 73cf72548..7729d0dd0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,9 +1,8 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_init;
+
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 97d7b0e7b..fc339affa 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1524,11 +1524,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	unsigned int dev_config_vf;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index ffe2f00ac..c0b87bcc0 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

[dpdk-dev] [PATCH v12 00/10] Improve mlx5 PMD driver framework for multiple classes

[Parav Pandit]

This series introduces mlx5 common driver layer to support multiple
class of devices for a single PCI device.

Motivation and example
----------------------
mlx5 PCI device supports multiple class of devices such as net, vdpa
and regex devices.

Currently only one pmd (either net or vdpa) can bind to this device.
This design limits use of PCI device only for single device class.

To support multiple classes simultaneously for a mlx5 PCI device,
mlx5 common module is extended as generic PCI PMD.
This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded
on a single PCI Device.

Change description
------------------
Patch-1 Introduces RTE_BIT() macro
Patch-2 Fixed indentation for directory listing
Patch-3 Relax dependency order
Patch-4 Fixes compilation error
Patch-5 Fixes segmentation fault in regex probe error unwind path
Patch-6 Avoid using class constructor priority
Patch-7 Change class values as bits
Patch-8 Introduce mlx5 layer to support multiple class drivers
Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API
        instead of rte PCI bus API
Patch-10 Removed class check code from class drivers 

Design overview
---------------

 -----------    ------------    -------------
 |   mlx5  |    |   mlx5   |    |   mlx5    |
 | net pmd |    | vdpa pmd |    | regex pmd |
 -----------    ------------    -------------
      \              |                /
       \             |               /
        \       ---------------     /
         \______| mlx5 common |____/
                |    pmd      |
                --------------- 
                     |
                 ----------- 
                 |   mlx5  | 
                 | pci dev | 
                 ----------- 

- mlx5 common driver binds to mlx5 PCI devices defined by PCI
  ID table of all related mlx5 PCI devices.
- mlx5 class driver such as net, vdpa, regex PMD define their
  specific PCI ID tables. mlx5 common driver probes each
  individual class driver(s) for maching and enabled classes.
- mlx5 pci layer is cental place that validates supported
  class combinations.
- In future as code evolves, more device setup/cleanup and
  resource creation code moves to mlx5 PCI common driver.

Alternatives considered
-----------------------
1. Instead of creating mlx5 pci common PMD, a common driver is
implemented which exposes class registration API.
However, common PMD model fits better with existing DPDK design
similar to ifpga driver. Class registration API need to create a
new callbacks and ID signature; instead it is better to utilize
current well defined methods.

2. Enhance pci core to allow multiple driver binding to single
rte PCI device.
This approach is not taken, because peer drivers using one PCI
device won't be aware of other's presence. This requires
cross-driver syncronization of who initializes common resources
(such as irq, eq and more).
This also requires refcounting common objects etc among peer drivers.
Instead of layered approach delivers and allows putting common resource
sharing, setup code in common driver.
It also eliminates peer blind zone problem as bottom pci layer provides
necessary setup without any reference counting.

3. In future mlx5 prefers to use RDMA MR cache of the mbuf used
between net and regex pmd so that same mbuf use across multiple
device can be possible.

Examples:
--------
A user who wish to use a specific class(es) provides list of classes
at command line such as,
./testpmd -w <PCI BDF>,class=net:regex
./testpmd -w <PCI BDF>,class=vdpa

Changelog:
v11->v12:
 - Define RTE_BIT32 macro as well
 - Use RTE_BIT32 and RTE_BIT64 macros in bitops as well
 - Updated release notes to cover mlx5 changes
v10->v11:
 - Added vdpa + regex combination to class combination table
 - All class defintions are moved to single patch
v9->v10:
 - Fixed checkpatch warnings related to spelling mistakes and alignment
v8->v9:
 - Updated commit message
 - Fixed LD_FLAGS
 - Fixed white space to tab
 - Fixed Makefile to have dependency on common/mlx5 for parallel builds
v7->v8:
 - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5
 - Avoided new RTE PRIORITY
 - Removed mlx5 common to use RTE_PRIORITY_CLASS
v6->v7:
 - Updated release notes
v5->v6:
 - Fixed compilation failure in parallel build for shared lib
v4->v5:
 - Squash the maintainers update path with other patch which adds the
   bus
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v3->v4:
 - Fixed dma_map error unwinding flow to follow same order for unmap
v2->v3:
 - Added RTE priority for common driver initialization
 - Addressed comments from Thomas and Asaf
 - Fixed compilation error in glue wrapper
 - Moved pci_driver structure instance as first in driver
 - Removed white spaces at the end of line in diagram
 - Address commnts from Matan
 - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
 - Renamed mlx5_valid_class_combo to mlx5_class_combinations
 - Added cross check for class drivers to support only 3 flags for now
 - Added full stop at the end of comment block
 - Using full names in function names
 - Added new line before function name in multiple functions
 - Added example string to parse for multiple classes
 - Dropped mlx5 prefix from static function
 - Removed empty lines
 - Fixed issue to remove multiple classes for a driver
 - Using define for drv_flags at multiple places
 - Deriving drv_flags based on the class drivers
 - Fixed alignment for id_table
 - Perform dma map on best effort basis for all supported drivers
 - Dynamically build pci id table
 - Using PCI to mlx5 device helper routines
v1->v2:
 - Addressed most comments from Thomas and Gaetan.
 - Symbols starting with prefix rte_bus_pci_mlx5 may be
   confusing as it may appear as it belong to rte_bus_pci module.
   Hence it is kept as rte_bus_mlx5_pci which matches with other
   modules as mlx5_vdpa, mlx5_net.
 - Dropped 2nd patch and replace with new 6th patch.
 - Avoided new file, added macro to rte_bitops.h
 - Inheriting ret_pci_driver instead of rte_driver
 - Added design and description of the mlx5_pci bus
 - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
 - Use anonymous structure for class search and code changes around it
 - Define static for class comination array
 - Use RTE_DIM to find array size
 - Added OOM check for strdup()
 - Renamed copy variable to nstr_orig
 - Returning negagive error code
 - Returning directly if match entry found
 - Use compat condition check
 - Avoided cutting error message string
 - Use uint32_t datatype instead of enum mlx5_class
 - Changed logic to parse device arguments only once during probe()
 - Added check to fail driver probe if multiple classes register with
   DMA ops
 - Renamed function to parse_class_options
 - Migreate API from rte_driver to rte_pci_driver

Parav Pandit (8):
  eal: introduce macro for bit definition
  common/mlx5: fix void parameters in glue wrappers
  regex/mlx5: fix segmentation fault during error unwinding
  common/mlx5: avoid using class constructor priority
  common/mlx5: change class values as bits
  common/mlx5: introduce layer to support multiple class drivers
  common/mlx5: register class drivers through common layer
  common/mlx5: remove class check from class drivers

Thomas Monjalon (2):
  drivers: fix indent of directory list
  drivers: relax dependency order

 doc/guides/rel_notes/release_20_08.rst        |   6 +
 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   2 +
 drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |  52 +-
 drivers/common/mlx5/mlx5_common.h             |  12 +-
 drivers/common/mlx5/mlx5_common_pci.c         | 540 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   4 +-
 drivers/meson.build                           |  51 +-
 drivers/net/mlx5/Makefile                     |   1 -
 drivers/net/mlx5/linux/mlx5_os.c              |   6 -
 drivers/net/mlx5/mlx5.c                       |  25 +-
 drivers/net/mlx5/mlx5.h                       |   1 -
 drivers/regex/mlx5/meson.build                |   2 +-
 drivers/regex/mlx5/mlx5_regex.c               |  29 +-
 drivers/vdpa/mlx5/Makefile                    |   2 +-
 drivers/vdpa/mlx5/meson.build                 |   2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  29 +-
 lib/librte_eal/include/rte_bitops.h           |  34 +-
 23 files changed, 772 insertions(+), 132 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

-- 
2.26.2

[dpdk-dev] [PATCH v12 01/10] eal: introduce macro for bit definition

[Parav Pandit]

There are several drivers which duplicate bit generation macro.
Introduce a generic bit macros so that such drivers avoid redefining
same in multiple drivers.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
Changelog:
v11->v12:
 - Define RTE_BIT32 macro as well
 - Use RTE_BIT32 and RTE_BIT64 macros in bitops as well
v4->v5:
 - Addressed comments from Morten Brørup
 - Renamed newly added macro to RTE_BIT64
 - Added doxygen comment section for the macro
v1->v2:
 - Addressed comments from Thomas and Gaten.
 - Avoided new file, added macro to rte_bitops.h
---
 lib/librte_eal/include/rte_bitops.h | 34 +++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 9 deletions(-)

diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h
index 740927f3b..141e8ea73 100644
--- a/lib/librte_eal/include/rte_bitops.h
+++ b/lib/librte_eal/include/rte_bitops.h
@@ -17,6 +17,22 @@
 #include <rte_debug.h>
 #include <rte_compat.h>
 
+/**
+ * Get the uint64_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 63.
+ */
+#define RTE_BIT64(nr) (UINT64_C(1) << (nr))
+
+/**
+ * Get the uint32_t value for a specified bit set.
+ *
+ * @param nr
+ *   The bit number in range of 0 to 31.
+ */
+#define RTE_BIT32(nr) (UINT32_C(1) << (nr))
+
 /*------------------------ 32-bit relaxed operations ------------------------*/
 
 /**
@@ -59,7 +75,7 @@ rte_bit_relaxed_set32(unsigned int nr, volatile uint32_t *addr)
 {
 	RTE_ASSERT(nr < 32);
 
-	uint32_t mask = UINT32_C(1) << nr;
+	uint32_t mask = RTE_BIT32(nr);
 	*addr = (*addr) | mask;
 }
 
@@ -80,7 +96,7 @@ rte_bit_relaxed_clear32(unsigned int nr, volatile uint32_t *addr)
 {
 	RTE_ASSERT(nr < 32);
 
-	uint32_t mask = UINT32_C(1) << nr;
+	uint32_t mask = RTE_BIT32(nr);
 	*addr = (*addr) & (~mask);
 }
 
@@ -104,7 +120,7 @@ rte_bit_relaxed_test_and_set32(unsigned int nr, volatile uint32_t *addr)
 {
 	RTE_ASSERT(nr < 32);
 
-	uint32_t mask = UINT32_C(1) << nr;
+	uint32_t mask = RTE_BIT32(nr);
 	uint32_t val = *addr;
 	*addr = val | mask;
 	return val & mask;
@@ -130,7 +146,7 @@ rte_bit_relaxed_test_and_clear32(unsigned int nr, volatile uint32_t *addr)
 {
 	RTE_ASSERT(nr < 32);
 
-	uint32_t mask = UINT32_C(1) << nr;
+	uint32_t mask = RTE_BIT32(nr);
 	uint32_t val = *addr;
 	*addr = val & (~mask);
 	return val & mask;
@@ -157,7 +173,7 @@ rte_bit_relaxed_get64(unsigned int nr, volatile uint64_t *addr)
 {
 	RTE_ASSERT(nr < 64);
 
-	uint64_t mask = UINT64_C(1) << nr;
+	uint64_t mask = RTE_BIT64(nr);
 	return (*addr) & mask;
 }
 
@@ -178,7 +194,7 @@ rte_bit_relaxed_set64(unsigned int nr, volatile uint64_t *addr)
 {
 	RTE_ASSERT(nr < 64);
 
-	uint64_t mask = UINT64_C(1) << nr;
+	uint64_t mask = RTE_BIT64(nr);
 	(*addr) = (*addr) | mask;
 }
 
@@ -199,7 +215,7 @@ rte_bit_relaxed_clear64(unsigned int nr, volatile uint64_t *addr)
 {
 	RTE_ASSERT(nr < 64);
 
-	uint64_t mask = UINT64_C(1) << nr;
+	uint64_t mask = RTE_BIT64(nr);
 	*addr = (*addr) & (~mask);
 }
 
@@ -223,7 +239,7 @@ rte_bit_relaxed_test_and_set64(unsigned int nr, volatile uint64_t *addr)
 {
 	RTE_ASSERT(nr < 64);
 
-	uint64_t mask = UINT64_C(1) << nr;
+	uint64_t mask = RTE_BIT64(nr);
 	uint64_t val = *addr;
 	*addr = val | mask;
 	return val;
@@ -249,7 +265,7 @@ rte_bit_relaxed_test_and_clear64(unsigned int nr, volatile uint64_t *addr)
 {
 	RTE_ASSERT(nr < 64);
 
-	uint64_t mask = UINT64_C(1) << nr;
+	uint64_t mask = RTE_BIT64(nr);
 	uint64_t val = *addr;
 	*addr = val & (~mask);
 	return val & mask;
-- 
2.26.2

[dpdk-dev] [PATCH v12 02/10] drivers: fix indent of directory list

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Define each sub-directory on its own line ended with a comma,
and use a simple indent.

Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog:
v7->8:
 - new patch
---
 drivers/meson.build | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e76ebddfa..e2aeba931 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -2,17 +2,19 @@
 # Copyright(c) 2017-2019 Intel Corporation
 
 # Defines the order in which the drivers are buit.
-dpdk_driver_classes = ['common',
-	       'bus',
-	       'mempool', # depends on common and bus.
-	       'net',     # depends on common, bus, mempool
-	       'raw',     # depends on common, bus and net.
-	       'crypto',  # depends on common, bus and mempool (net in future).
-	       'compress', # depends on common, bus, mempool.
-	       'regex', # depends on common, bus, regexdev.
-	       'vdpa',    # depends on common, bus and mempool.
-	       'event',   # depends on common, bus, mempool and net.
-	       'baseband'] # depends on common and bus.
+dpdk_driver_classes = [
+	'common',
+	'bus',
+	'mempool', # depends on common and bus.
+	'net',     # depends on common, bus, mempool
+	'raw',     # depends on common, bus and net.
+	'crypto',  # depends on common, bus and mempool (net in future).
+	'compress', # depends on common, bus, mempool.
+	'regex', # depends on common, bus, regexdev.
+	'vdpa',    # depends on common, bus and mempool.
+	'event',   # depends on common, bus, mempool and net.
+	'baseband', # depends on common and bus.
+]
 
 disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'),
 		).stdout().split()
-- 
2.26.2

[dpdk-dev] [PATCH v12 03/10] drivers: relax dependency order

[Parav Pandit]

From: Thomas Monjalon <thomas@monjalon.net>

Drivers dependencies are evaluated in the order defined per their parent
directory (also called class). This strict ordering prevents from
having pairs of drivers from two classes with different dependency
ordering. For example, if the mlx5 common code depends on the pci bus
driver, while the dpaax common code is itself a dependency of the dpaa
bus driver.  Having a strict ordering between directories bus and common
is too restrictive, as processing either common drivers or bus drivers
first leads us to missing dependencies in this scenario.

This patch makes it possible to have a more fine-grain directory list,
adding a specific driver sub-directory in the top-level drivers
subdirectory list. In this case, the isolated driver must also be removed
from its class list, and the per-class variables must be duplicated in
the isolated driver, because the call "subdir(class)" is skipped in the
isolated driver case.

Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Changelog:
v8->v9:
 - Updated commit message
 - Fixed typo
---
 drivers/meson.build | 28 +++++++++++++++++++++-------
 1 file changed, 21 insertions(+), 7 deletions(-)

diff --git a/drivers/meson.build b/drivers/meson.build
index e2aeba931..038957460 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017-2019 Intel Corporation
 
-# Defines the order in which the drivers are buit.
-dpdk_driver_classes = [
+# Defines the order of dependencies evaluation
+subdirs = [
 	'common',
 	'bus',
 	'mempool', # depends on common and bus.
@@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation')
 	default_cflags += '-Wno-format-truncation'
 endif
 
-foreach class:dpdk_driver_classes
+foreach subpath:subdirs
 	drivers = []
 	std_deps = []
 	config_flag_fmt = '' # format string used to set the value in dpdk_conf
@@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes
 	                     # the library, the dependency and to find the
 	                     # version file for linking
 
-	subdir(class)
-	class_drivers = []
+	# subpath can be either "class" or "class/driver"
+	if subpath.contains('/')
+		driver_path = subpath.split('/')
+		class = driver_path[0]
+		drivers += driver_path[1]
+	else
+		class = subpath
+		subdir(class)
+	endif
+
+	# save class name on first occurrence
+	if not dpdk_driver_classes.contains(class)
+		dpdk_driver_classes += class
+	endif
+	# get already enabled drivers of the same class
+	enabled_drivers = get_variable(class + '_drivers', [])
 
 	foreach drv:drivers
 		drv_path = join_paths(class, drv)
@@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes
 						'_disable_reason', reason)
 			endif
 		else
-			class_drivers += name
+			enabled_drivers += name
 
 			if fmt_name == ''
 				fmt_name = name
@@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes
 		endif # build
 	endforeach
 
-	set_variable(class + '_drivers', class_drivers)
+	set_variable(class + '_drivers', enabled_drivers)
 endforeach
-- 
2.26.2

[dpdk-dev] [PATCH v12 04/10] common/mlx5: fix void parameters in glue wrappers

[Parav Pandit]

Following two errors are reported when compiled with
gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5).

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

drivers/common/mlx5/linux/mlx5_glue.h:188:2:
error: function declaration isn't a prototype [-Werror=strict-prototypes]

Fix them by adding void data type in empty argument list.

Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch")
Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue")

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v2->v3:
 - new patch
---
 drivers/common/mlx5/linux/mlx5_glue.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h
index c4f9b006a..734ace2a3 100644
--- a/drivers/common/mlx5/linux/mlx5_glue.h
+++ b/drivers/common/mlx5/linux/mlx5_glue.h
@@ -185,11 +185,11 @@ struct mlx5_glue {
 	void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl);
 	void *(*dr_create_flow_action_dest_port)(void *domain,
 						 uint32_t port);
-	void *(*dr_create_flow_action_drop)();
+	void *(*dr_create_flow_action_drop)(void);
 	void *(*dr_create_flow_action_push_vlan)
 					(struct mlx5dv_dr_domain *domain,
 					 rte_be32_t vlan_tag);
-	void *(*dr_create_flow_action_pop_vlan)();
+	void *(*dr_create_flow_action_pop_vlan)(void);
 	void *(*dr_create_flow_tbl)(void *domain, uint32_t level);
 	int (*dr_destroy_flow_tbl)(void *tbl);
 	void *(*dr_create_domain)(struct ibv_context *ctx,
-- 
2.26.2

[dpdk-dev] [PATCH v12 05/10] regex/mlx5: fix segmentation fault during error unwinding

[Parav Pandit]

When fail to initialize the device, avoid segmentation fault while
accessing uninitialized priv.

Fixes: cfc672a90b74 ("regex/mlx5: support probing")

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v9->v10:
 - Corrected type in commit log
v7->v8:
 - Rebased
 - new patch
---
 drivers/regex/mlx5/mlx5_regex.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 1ca5bfe9b..36ae9f809 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -137,17 +137,17 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	if (ret) {
 		DRV_LOG(ERR, "Unable to read HCA capabilities.");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	} else if (!attr.regex || attr.regexp_num_of_engines == 0) {
 		DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe "
 			"old FW/OFED version?");
 		rte_errno = ENOTSUP;
-		goto error;
+		goto dev_error;
 	}
 	if (mlx5_regex_engines_status(ctx, 2)) {
 		DRV_LOG(ERR, "RegEx engine error.");
 		rte_errno = ENOMEM;
-		goto error;
+		goto dev_error;
 	}
 	priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv),
 			   RTE_CACHE_LINE_SIZE);
@@ -200,6 +200,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		mlx5_glue->devx_free_uar(priv->uar);
 	if (priv->regexdev)
 		rte_regexdev_unregister(priv->regexdev);
+dev_error:
 	if (ctx)
 		mlx5_glue->close_device(ctx);
 	if (priv)
-- 
2.26.2

[dpdk-dev] [PATCH v12 06/10] common/mlx5: avoid using class constructor priority

[Parav Pandit]

mlx5_common is shared library between mlx5 net, VDPA and regex PMD.
It is better to use common initialization helper instead of using
RTE_PRIORITY_CLASS priority.

Reviewed-by: David Marchand <david.marchand@redhat.com>
Suggested-by: David Marchand <david.marchand@redhat.com>
Signed-off-by: Parav Pandit <parav@mellanox.com>
---
Changelog:
v7->v8:
 - new patch
---
 drivers/common/mlx5/mlx5_common.c               | 13 +++++++++++--
 drivers/common/mlx5/mlx5_common.h               |  3 +++
 drivers/common/mlx5/rte_common_mlx5_version.map |  1 +
 drivers/net/mlx5/mlx5.c                         |  1 +
 drivers/regex/mlx5/mlx5_regex.c                 |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa.c                   |  1 +
 6 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 1a67a1b16..2b336bb2d 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG)
 		rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE);
 }
 
+static bool mlx5_common_initialized;
+
 /**
- * Initialization routine for run-time dependency on glue library.
+ * One time innitialization routine for run-time dependency on glue library
+ * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module,
+ * must invoke in its constructor.
  */
-RTE_INIT_PRIO(mlx5_glue_init, CLASS)
+void
+mlx5_common_init(void)
 {
+	if (mlx5_common_initialized)
+		return;
+
 	mlx5_glue_constructor();
+	mlx5_common_initialized = true;
 }
 
 /**
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index a811eb6c9..ebe4e9ced 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id,
 			 uint64_t offset);
 extern uint8_t haswell_broadwell_cpu;
 
+__rte_internal
+void mlx5_common_init(void);
+
 #endif /* RTE_PMD_MLX5_COMMON_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 132a0695f..65f25252a 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -3,6 +3,7 @@ INTERNAL {
 
 	mlx5_class_get;
 
+	mlx5_common_init;
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 647ada339..037703d2e 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_pmd_init)
 {
+	mlx5_common_init();
 	/* Build the static tables for Verbs conversion. */
 	mlx5_set_ptype_table();
 	mlx5_set_cksum_table();
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 36ae9f809..4e0367052 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = {
 
 RTE_INIT(rte_mlx5_regex_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_regex_driver);
 }
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 67e77b11a..85dbcf956 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
  */
 RTE_INIT(rte_mlx5_vdpa_init)
 {
+	mlx5_common_init();
 	if (mlx5_glue)
 		rte_pci_register(&mlx5_vdpa_driver);
 }
-- 
2.26.2

[dpdk-dev] [PATCH v12 07/10] common/mlx5: change class values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be parsed.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v7->v8:
 - updated commit message
---
 drivers/common/mlx5/mlx5_common.h | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index ebe4e9ced..864e82ff0 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bitops.h>
 
 #include "mlx5_prm.h"
 #include "mlx5_devx_cmds.h"
@@ -208,10 +209,10 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
-	MLX5_CLASS_REGEX,
 	MLX5_CLASS_INVALID,
+	MLX5_CLASS_NET = RTE_BIT64(0),
+	MLX5_CLASS_VDPA = RTE_BIT64(1),
+	MLX5_CLASS_REGEX = RTE_BIT64(2),
 };
 
 #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE
-- 
2.26.2

[dpdk-dev] [PATCH v12 08/10] common/mlx5: introduce layer to support multiple class drivers

[Parav Pandit]

Add generic mlx5 PCI PMD layer as part of existing common_mlx5
module. This enables multiple classes (net, regex, vdpa) PMDs
to be supported at same time.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v11->v12:
 - Added releases note back
v10->v11:
 - Moved class combination hunk from next patch to this one
v9->v10:
 - Fixed alignment
v8->v9:
 - Added missing LDFLAG for pci bus
 - Fixed white spaces at start of the line
---
 doc/guides/rel_notes/release_20_08.rst        |   6 +
 drivers/Makefile                              |  10 +-
 drivers/common/Makefile                       |   4 -
 drivers/common/meson.build                    |   2 +-
 drivers/common/mlx5/Makefile                  |   2 +
 drivers/common/mlx5/meson.build               |   9 +-
 drivers/common/mlx5/mlx5_common.c             |   2 +
 drivers/common/mlx5/mlx5_common_pci.c         | 540 ++++++++++++++++++
 drivers/common/mlx5/mlx5_common_pci.h         |  77 +++
 .../common/mlx5/rte_common_mlx5_version.map   |   2 +
 drivers/meson.build                           |   1 +
 11 files changed, 644 insertions(+), 11 deletions(-)
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.c
 create mode 100644 drivers/common/mlx5/mlx5_common_pci.h

diff --git a/doc/guides/rel_notes/release_20_08.rst b/doc/guides/rel_notes/release_20_08.rst
index d65f77250..619a16adf 100644
--- a/doc/guides/rel_notes/release_20_08.rst
+++ b/doc/guides/rel_notes/release_20_08.rst
@@ -144,6 +144,12 @@ New Features
   * Added support for virtio queue statistics.
   * Added support for MTU update.
 
+* **Updated Mellanox mlx5 common, net, vdpa and regex drivers.**
+
+  Updated Mellanox mlx5 drivers to share a PCI device among multiple PMDs.
+
+  * Added mlx5 PCI layer as common to all mlx5 PMDs.
+
 * **Updated Marvell octeontx2 ethdev PMD.**
 
   Updated Marvell octeontx2 driver with cn98xx support.
diff --git a/drivers/Makefile b/drivers/Makefile
index 1551272ef..7f06162dc 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -8,8 +8,12 @@ DIRS-y += bus
 DEPDIRS-bus := common
 DIRS-y += mempool
 DEPDIRS-mempool := common bus
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
+DIRS-y += common/mlx5
+DEPDIRS-common/mlx5 := bus
+endif
 DIRS-y += net
-DEPDIRS-net := common bus mempool
+DEPDIRS-net := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband
 DEPDIRS-baseband := common bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto
@@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress
 DEPDIRS-compress := bus mempool
 DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex
-DEPDIRS-regex := common bus
+DEPDIRS-regex := common bus common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa
-DEPDIRS-vdpa := common bus mempool
+DEPDIRS-vdpa := common bus mempool common/mlx5
 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event
 DEPDIRS-event := common bus mempool net crypto
 DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw
diff --git a/drivers/common/Makefile b/drivers/common/Makefile
index cbc71077c..cfb6b4dc8 100644
--- a/drivers/common/Makefile
+++ b/drivers/common/Makefile
@@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y)))
 DIRS-y += iavf
 endif
 
-ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y)
-DIRS-y += mlx5
-endif
-
 include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/common/meson.build b/drivers/common/meson.build
index 5db7e29b1..9ed4c04ba 100644
--- a/drivers/common/meson.build
+++ b/drivers/common/meson.build
@@ -6,6 +6,6 @@ if is_windows
 endif
 
 std_deps = ['eal']
-drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat']
+drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat']
 config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
 driver_name_fmt = 'rte_common_@0@'
diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile
index 6b89a6c85..4edd54104 100644
--- a/drivers/common/mlx5/Makefile
+++ b/drivers/common/mlx5/Makefile
@@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c
 SRCS-y += mlx5_common_mp.c
 SRCS-y += mlx5_common_mr.c
 SRCS-y += mlx5_malloc.c
+SRCS-y += mlx5_common_pci.c
 ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y)
 INSTALL-y-lib += $(LIB_GLUE)
 endif
@@ -51,6 +52,7 @@ LDLIBS += -libverbs -lmlx5
 endif
 
 LDLIBS += -lrte_eal -lrte_pci -lrte_kvargs -lrte_net
+LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual -UPEDANTIC
diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build
index 70e2c1c32..8e5608703 100644
--- a/drivers/common/mlx5/meson.build
+++ b/drivers/common/mlx5/meson.build
@@ -1,19 +1,22 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright 2019 Mellanox Technologies, Ltd
 
-if not (is_linux or is_windows)
+if not is_linux
 	build = false
-	reason = 'only supported on Linux and Windows'
+	reason = 'only supported on Linux'
 	subdir_done()
 endif
 
-deps += ['hash', 'pci', 'net', 'eal', 'kvargs']
+config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON'
+driver_name_fmt = 'rte_common_@0@'
+deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs']
 sources += files(
 	'mlx5_devx_cmds.c',
 	'mlx5_common.c',
 	'mlx5_common_mp.c',
 	'mlx5_common_mr.c',
 	'mlx5_malloc.c',
+	'mlx5_common_pci.c',
 )
 
 cflags_options = [
diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index 2b336bb2d..fd818ef24 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -14,6 +14,7 @@
 #include "mlx5_common_os.h"
 #include "mlx5_common_utils.h"
 #include "mlx5_malloc.h"
+#include "mlx5_common_pci.h"
 
 int mlx5_common_logtype;
 
@@ -100,6 +101,7 @@ mlx5_common_init(void)
 		return;
 
 	mlx5_glue_constructor();
+	mlx5_common_pci_init();
 	mlx5_common_initialized = true;
 }
 
diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c
new file mode 100644
index 000000000..d4ff0392f
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.c
@@ -0,0 +1,540 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies Ltd
+ */
+
+#include <stdlib.h>
+#include <rte_malloc.h>
+#include "mlx5_common_utils.h"
+#include "mlx5_common_pci.h"
+
+struct mlx5_pci_device {
+	struct rte_pci_device *pci_dev;
+	TAILQ_ENTRY(mlx5_pci_device) next;
+	uint32_t classes_loaded;
+};
+
+/* Head of list of drivers. */
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+/* Head of mlx5 pci devices. */
+static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list =
+				TAILQ_HEAD_INITIALIZER(devices_list);
+
+static const struct {
+	const char *name;
+	unsigned int driver_class;
+} mlx5_classes[] = {
+	{ .name = "vdpa", .driver_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .driver_class = MLX5_CLASS_NET },
+	{ .name = "regex", .driver_class = MLX5_CLASS_REGEX },
+};
+
+static const unsigned int mlx5_class_combinations[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	MLX5_CLASS_REGEX,
+	MLX5_CLASS_NET | MLX5_CLASS_REGEX,
+	MLX5_CLASS_VDPA | MLX5_CLASS_REGEX,
+	/* New class combination should be added here. */
+};
+
+static int
+class_name_to_value(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < RTE_DIM(mlx5_classes); i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return mlx5_classes[i].driver_class;
+	}
+	return -EINVAL;
+}
+
+static struct mlx5_pci_driver *
+driver_get(uint32_t class)
+{
+	struct mlx5_pci_driver *driver;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (driver->driver_class == class)
+			return driver;
+	}
+	return NULL;
+}
+
+static int
+bus_cmdline_options_handler(__rte_unused const char *key,
+			    const char *class_names, void *opaque)
+{
+	int *ret = opaque;
+	char *nstr_org;
+	int class_val;
+	char *found;
+	char *nstr;
+
+	*ret = 0;
+	nstr = strdup(class_names);
+	if (!nstr) {
+		*ret = -ENOMEM;
+		return *ret;
+	}
+	nstr_org = nstr;
+	while (nstr) {
+		/* Extract each individual class name. Multiple
+		 * class key,value is supplied as class=net:vdpa:foo:bar.
+		 */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+		/* Check if its a valid class. */
+		class_val = class_name_to_value(found);
+		if (class_val < 0) {
+			*ret = -EINVAL;
+			goto err;
+		}
+		*ret |= class_val;
+	}
+err:
+	free(nstr_org);
+	if (*ret < 0)
+		DRV_LOG(ERR, "Invalid mlx5 class options %s."
+			" Maybe typo in device class argument setting?",
+			class_names);
+	return *ret;
+}
+
+static int
+parse_class_options(const struct rte_devargs *devargs)
+{
+	const char *key = MLX5_CLASS_ARG_NAME;
+	struct rte_kvargs *kvlist;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, bus_cmdline_options_handler,
+				   &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
+static bool
+mlx5_bus_match(const struct mlx5_pci_driver *drv,
+	       const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0;
+	     id_table++) {
+		/* Check if device's ids match the class driver's ids. */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+		    id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+		    id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+		    id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+		return true;
+	}
+	return false;
+}
+
+static int
+is_valid_class_combination(uint32_t user_classes)
+{
+	unsigned int i;
+
+	/* Verify if user specified valid supported combination. */
+	for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) {
+		if (mlx5_class_combinations[i] == user_classes)
+			return 0;
+	}
+	/* Not found any valid class combination. */
+	return -EINVAL;
+}
+
+static struct mlx5_pci_device *
+pci_to_mlx5_device(const struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+
+	TAILQ_FOREACH(dev, &devices_list, next) {
+		if (dev->pci_dev == pci_dev)
+			return dev;
+	}
+	return NULL;
+}
+
+static bool
+device_class_enabled(const struct mlx5_pci_device *device, uint32_t class)
+{
+	return (device->classes_loaded & class) ? true : false;
+}
+
+static void
+dev_release(struct mlx5_pci_device *dev)
+{
+	TAILQ_REMOVE(&devices_list, dev, next);
+	rte_free(dev);
+}
+
+static int
+drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes)
+{
+	struct mlx5_pci_driver *driver;
+	int local_ret = -ENODEV;
+	unsigned int i = 0;
+	int ret = 0;
+
+	enabled_classes &= dev->classes_loaded;
+	while (enabled_classes) {
+		driver = driver_get(RTE_BIT64(i));
+		if (driver) {
+			local_ret = driver->pci_driver.remove(dev->pci_dev);
+			if (!local_ret)
+				dev->classes_loaded &= ~RTE_BIT64(i);
+			else if (ret == 0)
+				ret = local_ret;
+		}
+		enabled_classes &= ~RTE_BIT64(i);
+		i++;
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+static int
+drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv,
+	      struct rte_pci_device *pci_dev, uint32_t user_classes)
+{
+	struct mlx5_pci_driver *driver;
+	uint32_t enabled_classes = 0;
+	bool already_loaded;
+	int ret;
+
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if ((driver->driver_class & user_classes) == 0)
+			continue;
+		if (!mlx5_bus_match(driver, pci_dev))
+			continue;
+		already_loaded = dev->classes_loaded & driver->driver_class;
+		if (already_loaded &&
+		    !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) {
+			DRV_LOG(ERR, "Device %s is already probed\n",
+				pci_dev->device.name);
+			ret = -EEXIST;
+			goto probe_err;
+		}
+		ret = driver->pci_driver.probe(pci_drv, pci_dev);
+		if (ret < 0) {
+			DRV_LOG(ERR, "Failed to load driver = %s.\n",
+				driver->pci_driver.driver.name);
+			goto probe_err;
+		}
+		enabled_classes |= driver->driver_class;
+	}
+	dev->classes_loaded |= enabled_classes;
+	return 0;
+probe_err:
+	/* Only unload drivers which are enabled which were enabled
+	 * in this probe instance.
+	 */
+	drivers_remove(dev, enabled_classes);
+	return ret;
+}
+
+/**
+ * DPDK callback to register to probe multiple drivers for a PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		      struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	uint32_t user_classes = 0;
+	bool new_device = false;
+	int ret;
+
+	ret = parse_class_options(pci_dev->device.devargs);
+	if (ret < 0)
+		return ret;
+	user_classes = ret;
+	if (user_classes) {
+		/* Validate combination here. */
+		ret = is_valid_class_combination(user_classes);
+		if (ret) {
+			DRV_LOG(ERR, "Unsupported mlx5 classes supplied.");
+			return ret;
+		}
+	} else {
+		/* Default to net class. */
+		user_classes = MLX5_CLASS_NET;
+	}
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev) {
+		dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0);
+		if (!dev)
+			return -ENOMEM;
+		dev->pci_dev = pci_dev;
+		TAILQ_INSERT_HEAD(&devices_list, dev, next);
+		new_device = true;
+	}
+	ret = drivers_probe(dev, pci_drv, pci_dev, user_classes);
+	if (ret)
+		goto class_err;
+	return 0;
+class_err:
+	if (new_device)
+		dev_release(dev);
+	return ret;
+}
+
+/**
+ * DPDK callback to remove one or more drivers for a PCI device.
+ *
+ * This function removes all drivers probed for a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_common_pci_remove(struct rte_pci_device *pci_dev)
+{
+	struct mlx5_pci_device *dev;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	/* Matching device found, cleanup and unload drivers. */
+	ret = drivers_remove(dev, dev->classes_loaded);
+	if (!ret)
+		dev_release(dev);
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr,
+			uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver = NULL;
+	struct mlx5_pci_driver *temp;
+	struct mlx5_pci_device *dev;
+	int ret = -EINVAL;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	TAILQ_FOREACH(driver, &drv_list, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_map) {
+			ret = driver->pci_driver.dma_map(pci_dev, addr,
+							 iova, len);
+			if (ret)
+				goto map_err;
+		}
+	}
+	return ret;
+map_err:
+	TAILQ_FOREACH(temp, &drv_list, next) {
+		if (temp == driver)
+			break;
+		if (device_class_enabled(dev, temp->driver_class) &&
+		    temp->pci_driver.dma_map && temp->pci_driver.dma_unmap)
+			temp->pci_driver.dma_unmap(pci_dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr,
+			  uint64_t iova, size_t len)
+{
+	struct mlx5_pci_driver *driver;
+	struct mlx5_pci_device *dev;
+	int local_ret = -EINVAL;
+	int ret;
+
+	dev = pci_to_mlx5_device(pci_dev);
+	if (!dev)
+		return -ENODEV;
+	ret = 0;
+	/* There is no unmap error recovery in current implementation. */
+	TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (device_class_enabled(dev, driver->driver_class) &&
+		    driver->pci_driver.dma_unmap) {
+			local_ret = driver->pci_driver.dma_unmap(pci_dev, addr,
+								 iova, len);
+			if (local_ret && (ret == 0))
+				ret = local_ret;
+		}
+	}
+	if (local_ret)
+		ret = local_ret;
+	return ret;
+}
+
+/* PCI ID table is build dynamically based on registered mlx5 drivers. */
+static struct rte_pci_id *mlx5_pci_id_table;
+
+static struct rte_pci_driver mlx5_pci_driver = {
+	.driver = {
+		.name = "mlx5_pci",
+	},
+	.probe = mlx5_common_pci_probe,
+	.remove = mlx5_common_pci_remove,
+	.dma_map = mlx5_common_pci_dma_map,
+	.dma_unmap = mlx5_common_pci_dma_unmap,
+};
+
+static int
+pci_id_table_size_get(const struct rte_pci_id *id_table)
+{
+	int table_size = 0;
+
+	for (; id_table->vendor_id != 0; id_table++)
+		table_size++;
+	return table_size;
+}
+
+static bool
+pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table,
+	      int next_idx)
+{
+	int current_size = next_idx - 1;
+	int i;
+
+	for (i = 0; i < current_size; i++) {
+		if (id->device_id == table[i].device_id &&
+		    id->vendor_id == table[i].vendor_id &&
+		    id->subsystem_vendor_id == table[i].subsystem_vendor_id &&
+		    id->subsystem_device_id == table[i].subsystem_device_id)
+			return true;
+	}
+	return false;
+}
+
+static void
+pci_id_insert(struct rte_pci_id *new_table, int *next_idx,
+	      const struct rte_pci_id *id_table)
+{
+	/* Traverse the id_table, check if entry exists in new_table;
+	 * Add non duplicate entries to new table.
+	 */
+	for (; id_table->vendor_id != 0; id_table++) {
+		if (!pci_id_exists(id_table, new_table, *next_idx)) {
+			/* New entry; add to the table. */
+			new_table[*next_idx] = *id_table;
+			(*next_idx)++;
+		}
+	}
+}
+
+static int
+pci_ids_table_update(const struct rte_pci_id *driver_id_table)
+{
+	const struct rte_pci_id *id_iter;
+	struct rte_pci_id *updated_table;
+	struct rte_pci_id *old_table;
+	int num_ids = 0;
+	int i = 0;
+
+	old_table = mlx5_pci_id_table;
+	if (old_table)
+		num_ids = pci_id_table_size_get(old_table);
+	num_ids += pci_id_table_size_get(driver_id_table);
+	/* Increase size by one for the termination entry of vendor_id = 0. */
+	num_ids += 1;
+	updated_table = calloc(num_ids, sizeof(*updated_table));
+	if (!updated_table)
+		return -ENOMEM;
+	if (TAILQ_EMPTY(&drv_list)) {
+		/* Copy the first driver's ID table. */
+		for (id_iter = driver_id_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+	} else {
+		/* First copy existing table entries. */
+		for (id_iter = old_table; id_iter->vendor_id != 0;
+		     id_iter++, i++)
+			updated_table[i] = *id_iter;
+		/* New id to be added at the end of current ID table. */
+		pci_id_insert(updated_table, &i, driver_id_table);
+	}
+	/* Terminate table with empty entry. */
+	updated_table[i].vendor_id = 0;
+	mlx5_pci_driver.id_table = updated_table;
+	mlx5_pci_id_table = updated_table;
+	if (old_table)
+		free(old_table);
+	return 0;
+}
+
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver)
+{
+	int ret;
+
+	ret = pci_ids_table_update(driver->pci_driver.id_table);
+	if (ret)
+		return;
+	mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags;
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
+
+void mlx5_common_pci_init(void)
+{
+	const struct rte_pci_id empty_table[] = {
+		{
+			.vendor_id = 0
+		},
+	};
+
+	/* All mlx5 PMDs constructor runs at same priority. So any of the PMD
+	 * including this one can register the PCI table first. If any other
+	 * PMD(s) have registered the PCI ID table, No need to register an empty
+	 * default one.
+	 */
+	if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table))
+		return;
+	rte_pci_register(&mlx5_pci_driver);
+}
+
+RTE_FINI(mlx5_common_pci_finish)
+{
+	if (mlx5_pci_id_table != NULL) {
+		/* Constructor doesn't register with PCI bus if it failed
+		 * to build the table.
+		 */
+		rte_pci_unregister(&mlx5_pci_driver);
+		free(mlx5_pci_id_table);
+	}
+}
+RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__);
diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h
new file mode 100644
index 000000000..41b73e17a
--- /dev/null
+++ b/drivers/common/mlx5/mlx5_common_pci.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _MLX5_COMMON_PCI_H_
+#define _MLX5_COMMON_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Driver Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This layer enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci driver.
+ *
+ * -----------    ------------    -------------
+ * |   mlx5  |    |   mlx5   |    |   mlx5    |
+ * | net pmd |    | vdpa pmd |    | regex pmd |
+ * -----------    ------------    -------------
+ *      \              |                 /
+ *       \             |                /
+ *        \       --------------       /
+ *         \______|   mlx5     |_____ /
+ *                | pci common |
+ *                --------------
+ *                     |
+ *                 -----------
+ *                 |   mlx5  |
+ *                 | pci dev |
+ *                 -----------
+ *
+ * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI
+ *   ID table of all related mlx5 PCI devices.
+ * - mlx5 class driver such as net, vdpa, regex PMD defines its
+ *   specific PCI ID table and mlx5 bus driver probes matching
+ *   class drivers.
+ * - mlx5 pci bus driver is cental place that validates supported
+ *   class combinations.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+void mlx5_common_pci_init(void);
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct mlx5_pci_driver {
+	struct rte_pci_driver pci_driver;	/**< Inherit core pci driver. */
+	uint32_t driver_class;	/**< Class of this driver, enum mlx5_class */
+	TAILQ_ENTRY(mlx5_pci_driver) next;
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+mlx5_pci_driver_register(struct mlx5_pci_driver *driver);
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _MLX5_COMMON_PCI_H_ */
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 65f25252a..73cf72548 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -91,5 +91,7 @@ INTERNAL {
 	mlx5_malloc;
 	mlx5_realloc;
 	mlx5_free;
+
+	mlx5_pci_driver_register;
 };
 
diff --git a/drivers/meson.build b/drivers/meson.build
index 038957460..5f9526557 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -5,6 +5,7 @@
 subdirs = [
 	'common',
 	'bus',
+	'common/mlx5', # depends on bus.
 	'mempool', # depends on common and bus.
 	'net',     # depends on common, bus, mempool
 	'raw',     # depends on common, bus and net.
-- 
2.26.2

[dpdk-dev] [PATCH v12 09/10] common/mlx5: register class drivers through common layer

[Parav Pandit]

Migrate mlx5 net, vdpa and regex PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
Changelog:
v10->v11:
 - Moved new class combinations to previous patch
v8->v9:
 - Removed leftover inclusion of mlx5_pci bus include directory
---
 drivers/net/mlx5/Makefile        |  1 -
 drivers/net/mlx5/linux/mlx5_os.c |  1 -
 drivers/net/mlx5/mlx5.c          | 24 ++++++++++++++----------
 drivers/net/mlx5/mlx5.h          |  1 -
 drivers/regex/mlx5/meson.build   |  2 +-
 drivers/regex/mlx5/mlx5_regex.c  | 21 ++++++++++++---------
 drivers/vdpa/mlx5/Makefile       |  2 +-
 drivers/vdpa/mlx5/meson.build    |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c    | 23 +++++++++++++----------
 9 files changed, 42 insertions(+), 35 deletions(-)

diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index 253faf909..568c77241 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -55,7 +55,6 @@ LDLIBS += -lrte_common_mlx5
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index fa3b02787..97d7b0e7b 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1537,7 +1537,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 037703d2e..1e4c695f8 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -30,6 +30,7 @@
 #include <mlx5_common.h>
 #include <mlx5_common_os.h>
 #include <mlx5_common_mp.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_malloc.h>
 
 #include "mlx5_defs.h"
@@ -2091,16 +2092,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
-	.driver = {
-		.name = MLX5_DRIVER_NAME
+static struct mlx5_pci_driver mlx5_driver = {
+	.driver_class = MLX5_CLASS_NET,
+	.pci_driver = {
+		.driver = {
+			.name = MLX5_DRIVER_NAME,
+		},
+		.id_table = mlx5_pci_id_map,
+		.probe = mlx5_os_pci_probe,
+		.remove = mlx5_pci_remove,
+		.dma_map = mlx5_dma_map,
+		.dma_unmap = mlx5_dma_unmap,
+		.drv_flags = PCI_DRV_FLAGS,
 	},
-	.id_table = mlx5_pci_id_map,
-	.probe = mlx5_os_pci_probe,
-	.remove = mlx5_pci_remove,
-	.dma_map = mlx5_dma_map,
-	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /* Initialize driver log type. */
@@ -2117,7 +2121,7 @@ RTE_INIT(rte_mlx5_pmd_init)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index a92194d2d..78d6eb728 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -119,7 +119,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_os_dev_ops;
diff --git a/drivers/regex/mlx5/meson.build b/drivers/regex/mlx5/meson.build
index 7f800f2e3..d7cb2c572 100644
--- a/drivers/regex/mlx5/meson.build
+++ b/drivers/regex/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_regex'
-deps += ['common_mlx5', 'bus_pci', 'eal', 'regexdev']
+deps += ['common_mlx5', 'eal', 'regexdev']
 sources = files(
 	'mlx5_regex.c',
 	'mlx5_rxp.c',
diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c
index 4e0367052..ae9f00189 100644
--- a/drivers/regex/mlx5/mlx5_regex.c
+++ b/drivers/regex/mlx5/mlx5_regex.c
@@ -5,12 +5,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_regexdev.h>
 #include <rte_regexdev_core.h>
 #include <rte_regexdev_driver.h>
 
+#include <mlx5_common_pci.h>
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
@@ -246,21 +246,24 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_regex_driver = {
-	.driver = {
-		.name = "mlx5_regex",
+static struct mlx5_pci_driver mlx5_regex_driver = {
+	.driver_class = MLX5_CLASS_REGEX,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_regex",
+		},
+		.id_table = mlx5_regex_pci_id_map,
+		.probe = mlx5_regex_pci_probe,
+		.remove = mlx5_regex_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_regex_pci_id_map,
-	.probe = mlx5_regex_pci_probe,
-	.remove = mlx5_regex_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_INIT(rte_mlx5_regex_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_regex_driver);
+		mlx5_pci_driver_register(&mlx5_regex_driver);
 }
 
 RTE_LOG_REGISTER(mlx5_regex_logtype, pmd.regex.mlx5, NOTICE)
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 8a1c2eab5..d46699fbc 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -30,7 +30,7 @@ CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_sched
 LDLIBS += -pthread
 
 # A few warnings cannot be avoided in external headers.
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..254a8ec14 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 85dbcf956..ffe2f00ac 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -11,12 +11,12 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_common.h>
+#include <mlx5_common_pci.h>
 #include <mlx5_devx_cmds.h>
 #include <mlx5_prm.h>
 #include <mlx5_nl.h>
@@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
-	.driver = {
-		.name = "mlx5_vdpa",
+static struct mlx5_pci_driver mlx5_vdpa_driver = {
+	.driver_class = MLX5_CLASS_VDPA,
+	.pci_driver = {
+		.driver = {
+			.name = "mlx5_vdpa",
+		},
+		.id_table = mlx5_vdpa_pci_id_map,
+		.probe = mlx5_vdpa_pci_probe,
+		.remove = mlx5_vdpa_pci_remove,
+		.drv_flags = 0,
 	},
-	.id_table = mlx5_vdpa_pci_id_map,
-	.probe = mlx5_vdpa_pci_probe,
-	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE)
@@ -848,7 +851,7 @@ RTE_INIT(rte_mlx5_vdpa_init)
 {
 	mlx5_common_init();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
-- 
2.26.2

[dpdk-dev] [PATCH v12 10/10] common/mlx5: remove class check from class drivers

[Parav Pandit]

Now that mlx5_pci PMD checks for enabled classes and performs
probe(), remove() of associated classes, individual class driver
does not need to check if other driver is enabled.

Signed-off-by: Parav Pandit <parav@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c             | 37 -------------------
 drivers/common/mlx5/mlx5_common.h             |  2 -
 .../common/mlx5/rte_common_mlx5_version.map   |  3 +-
 drivers/net/mlx5/linux/mlx5_os.c              |  5 ---
 drivers/vdpa/mlx5/mlx5_vdpa.c                 |  5 ---
 5 files changed, 1 insertion(+), 51 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index fd818ef24..06f0a6400 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -20,43 +20,6 @@ int mlx5_common_logtype;
 
 uint8_t haswell_broadwell_cpu;
 
-static int
-mlx5_class_check_handler(__rte_unused const char *key, const char *value,
-			 void *opaque)
-{
-	enum mlx5_class *ret = opaque;
-
-	if (strcmp(value, "vdpa") == 0) {
-		*ret = MLX5_CLASS_VDPA;
-	} else if (strcmp(value, "net") == 0) {
-		*ret = MLX5_CLASS_NET;
-	} else {
-		DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device"
-			" class argument setting?", value);
-		*ret = MLX5_CLASS_INVALID;
-	}
-	return 0;
-}
-
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
-{
-	struct rte_kvargs *kvlist;
-	const char *key = MLX5_CLASS_ARG_NAME;
-	enum mlx5_class ret = MLX5_CLASS_NET;
-
-	if (devargs == NULL)
-		return ret;
-	kvlist = rte_kvargs_parse(devargs->args, NULL);
-	if (kvlist == NULL)
-		return ret;
-	if (rte_kvargs_count(kvlist, key))
-		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
-	rte_kvargs_free(kvlist);
-	return ret;
-}
-
-
 /* In case this is an x86_64 intel processor to check if
  * we should use relaxed ordering.
  */
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 864e82ff0..2cdb226f3 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -247,8 +247,6 @@ struct mlx5_klm {
 
 LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page);
 
-__rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 73cf72548..7729d0dd0 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,9 +1,8 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
-
 	mlx5_common_init;
+
 	mlx5_common_verbs_reg_mr;
 	mlx5_common_verbs_dereg_mr;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 97d7b0e7b..fc339affa 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1524,11 +1524,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	unsigned int dev_config_vf;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index ffe2f00ac..c0b87bcc0 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
-		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
-			" driver.");
-		return 1;
-	}
 	ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr);
 	if (!ibv) {
 		DRV_LOG(ERR, "No matching IB device for PCI slot "
-- 
2.26.2

Re: [dpdk-dev] [PATCH v12 00/10] Improve mlx5 PMD driver framework for multiple classes

[Thomas Monjalon]

27/07/2020 19:47, Parav Pandit:
> This series introduces mlx5 common driver layer to support multiple
> class of devices for a single PCI device.

Applied, thanks

[dpdk-dev] [RFC PATCH 2/6] common/mlx5: use class enable check helper function

[Parav Pandit]

Currently mlx5_class_get() returns enabled single valid class.
To support multiple class and to improve readability of code, change it
to mlx5_class_enabled().
With this function, each class enablement can be checked, to load class
specific driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.c               | 12 +++++++-----
 drivers/common/mlx5/mlx5_common.h               |  5 +++--
 drivers/common/mlx5/rte_common_mlx5_version.map |  2 +-
 drivers/net/mlx5/linux/mlx5_os.c                |  3 ++-
 drivers/vdpa/mlx5/mlx5_vdpa.c                   |  2 +-
 5 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
index db94d4aa8..96c415842 100644
--- a/drivers/common/mlx5/mlx5_common.c
+++ b/drivers/common/mlx5/mlx5_common.c
@@ -37,22 +37,24 @@ mlx5_class_check_handler(__rte_unused const char *key, const char *value,
 	return 0;
 }
 
-enum mlx5_class
-mlx5_class_get(struct rte_devargs *devargs)
+bool
+mlx5_class_enabled(const struct rte_devargs *devargs, enum mlx5_class dev_class)
 {
 	struct rte_kvargs *kvlist;
 	const char *key = MLX5_CLASS_ARG_NAME;
+	/* Default NET CLASS is enabled if user didn't specify the class */
 	enum mlx5_class ret = MLX5_CLASS_NET;
 
 	if (devargs == NULL)
-		return ret;
+		return dev_class == MLX5_CLASS_NET ? true : false;
 	kvlist = rte_kvargs_parse(devargs->args, NULL);
 	if (kvlist == NULL)
-		return ret;
+		return dev_class == MLX5_CLASS_NET ? true : false;
 	if (rte_kvargs_count(kvlist, key))
 		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
 	rte_kvargs_free(kvlist);
-	return ret;
+
+	return (ret & dev_class) ? true : false;
 }
 
 
diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 8e679c699..1d59873c8 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -202,13 +202,14 @@ int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);
 #define MLX5_CLASS_ARG_NAME "class"
 
 enum mlx5_class {
+	MLX5_CLASS_INVALID,
 	MLX5_CLASS_NET,
 	MLX5_CLASS_VDPA,
-	MLX5_CLASS_INVALID,
 };
 
 __rte_internal
-enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
+bool mlx5_class_enabled(const struct rte_devargs *devargs,
+			enum mlx5_class dev_class);
 __rte_internal
 void mlx5_translate_port_name(const char *port_name_in,
 			      struct mlx5_switch_info *port_info_out);
diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
index 350e77140..01fa0cc25 100644
--- a/drivers/common/mlx5/rte_common_mlx5_version.map
+++ b/drivers/common/mlx5/rte_common_mlx5_version.map
@@ -1,7 +1,7 @@
 INTERNAL {
 	global:
 
-	mlx5_class_get;
+	mlx5_class_enabled;
 
 	mlx5_create_mr_ext;
 
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 92422dbe6..06772b7ae 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1377,11 +1377,12 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_dev_config dev_config;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
+	if (!mlx5_class_enabled(pci_dev->device.devargs, MLX5_CLASS_NET)) {
 		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
 			" driver.");
 		return 1;
 	}
+
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
 		mlx5_pmd_socket_init();
 	ret = mlx5_init_once();
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 1113d6cef..96776b64e 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -451,7 +451,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 	struct mlx5_hca_attr attr;
 	int ret;
 
-	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
+	if (!mlx5_class_enabled(pci_dev->device.devargs, MLX5_CLASS_VDPA)) {
 		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
 			" driver.");
 		return 1;
-- 
2.25.4

Re: [dpdk-dev] [RFC PATCH 2/6] common/mlx5: use class enable check helper function

[Gaëtan Rivet]

On 10/06/20 17:17 +0000, Parav Pandit wrote:
> Currently mlx5_class_get() returns enabled single valid class.
> To support multiple class and to improve readability of code, change it
> to mlx5_class_enabled().
> With this function, each class enablement can be checked, to load class
> specific driver.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/common/mlx5/mlx5_common.c               | 12 +++++++-----
>  drivers/common/mlx5/mlx5_common.h               |  5 +++--
>  drivers/common/mlx5/rte_common_mlx5_version.map |  2 +-
>  drivers/net/mlx5/linux/mlx5_os.c                |  3 ++-
>  drivers/vdpa/mlx5/mlx5_vdpa.c                   |  2 +-
>  5 files changed, 14 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c
> index db94d4aa8..96c415842 100644
> --- a/drivers/common/mlx5/mlx5_common.c
> +++ b/drivers/common/mlx5/mlx5_common.c
> @@ -37,22 +37,24 @@ mlx5_class_check_handler(__rte_unused const char *key, const char *value,
>  	return 0;
>  }
>  
> -enum mlx5_class
> -mlx5_class_get(struct rte_devargs *devargs)
> +bool
> +mlx5_class_enabled(const struct rte_devargs *devargs, enum mlx5_class dev_class)
>  {
>  	struct rte_kvargs *kvlist;
>  	const char *key = MLX5_CLASS_ARG_NAME;
> +	/* Default NET CLASS is enabled if user didn't specify the class */
>  	enum mlx5_class ret = MLX5_CLASS_NET;
>  
>  	if (devargs == NULL)
> -		return ret;
> +		return dev_class == MLX5_CLASS_NET ? true : false;
>  	kvlist = rte_kvargs_parse(devargs->args, NULL);
>  	if (kvlist == NULL)
> -		return ret;
> +		return dev_class == MLX5_CLASS_NET ? true : false;
>  	if (rte_kvargs_count(kvlist, key))
>  		rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret);
>  	rte_kvargs_free(kvlist);
> -	return ret;
> +
> +	return (ret & dev_class) ? true : false;
>  }

I think it would be simpler to transform the enum mlx5_class into a
value that could support multiple states simultaneously, instead of
going through this function.

As it is this function cannot properly work. You return CLASS_NET on
devargs NULL (fine), but also on kvlist error, meaning on devargs
invalid keys passed to the device. An error signal should be worked out
at this point, but it does not play well with the binary true/false.

I think you should instead change enum mlx5_class into a bitfield.
Return some u8 (I guess u32 if you want to keep the enum width) with the proper bits set,
and MLX5_CLASS_INVALID bit set on error (EOM if kvlist malloc failed, or other
errors due to invalid inputs).

>  
>  
> diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
> index 8e679c699..1d59873c8 100644
> --- a/drivers/common/mlx5/mlx5_common.h
> +++ b/drivers/common/mlx5/mlx5_common.h
> @@ -202,13 +202,14 @@ int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);
>  #define MLX5_CLASS_ARG_NAME "class"
>  
>  enum mlx5_class {
> +	MLX5_CLASS_INVALID,
>  	MLX5_CLASS_NET,
>  	MLX5_CLASS_VDPA,
> -	MLX5_CLASS_INVALID,
>  };
>  
>  __rte_internal
> -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs);
> +bool mlx5_class_enabled(const struct rte_devargs *devargs,
> +			enum mlx5_class dev_class);
>  __rte_internal
>  void mlx5_translate_port_name(const char *port_name_in,
>  			      struct mlx5_switch_info *port_info_out);
> diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map
> index 350e77140..01fa0cc25 100644
> --- a/drivers/common/mlx5/rte_common_mlx5_version.map
> +++ b/drivers/common/mlx5/rte_common_mlx5_version.map
> @@ -1,7 +1,7 @@
>  INTERNAL {
>  	global:
>  
> -	mlx5_class_get;
> +	mlx5_class_enabled;
>  
>  	mlx5_create_mr_ext;
>  
> diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
> index 92422dbe6..06772b7ae 100644
> --- a/drivers/net/mlx5/linux/mlx5_os.c
> +++ b/drivers/net/mlx5/linux/mlx5_os.c
> @@ -1377,11 +1377,12 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
>  	struct mlx5_dev_config dev_config;
>  	int ret;
>  
> -	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) {
> +	if (!mlx5_class_enabled(pci_dev->device.devargs, MLX5_CLASS_NET)) {
>  		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
>  			" driver.");
>  		return 1;
>  	}
> +
>  	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
>  		mlx5_pmd_socket_init();
>  	ret = mlx5_init_once();
> diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
> index 1113d6cef..96776b64e 100644
> --- a/drivers/vdpa/mlx5/mlx5_vdpa.c
> +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
> @@ -451,7 +451,7 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
>  	struct mlx5_hca_attr attr;
>  	int ret;
>  
> -	if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) {
> +	if (!mlx5_class_enabled(pci_dev->device.devargs, MLX5_CLASS_VDPA)) {
>  		DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5"
>  			" driver.");
>  		return 1;
> -- 
> 2.25.4
> 

-- 
Gaëtan

[dpdk-dev] [RFC PATCH 3/6] common/mlx5: change mlx5 class enum values as bits

[Parav Pandit]

mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
and/or regex.
To support these multiple classes, change mlx5_class to a
bitmap values so that if users asks to enable multiple of them, all
supported classes can be returned by mlx5_class_supported().

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/common/mlx5/mlx5_common.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
index 1d59873c8..9fdbd341d 100644
--- a/drivers/common/mlx5/mlx5_common.h
+++ b/drivers/common/mlx5/mlx5_common.h
@@ -13,6 +13,7 @@
 #include <rte_log.h>
 #include <rte_kvargs.h>
 #include <rte_devargs.h>
+#include <rte_bits.h>
 
 #include "mlx5_prm.h"
 
@@ -203,8 +204,8 @@ int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);
 
 enum mlx5_class {
 	MLX5_CLASS_INVALID,
-	MLX5_CLASS_NET,
-	MLX5_CLASS_VDPA,
+	MLX5_CLASS_NET = RTE_BIT(0),
+	MLX5_CLASS_VDPA = RTE_BIT(1),
 };
 
 __rte_internal
-- 
2.25.4

Re: [dpdk-dev] [RFC PATCH 3/6] common/mlx5: change mlx5 class enum values as bits

[Gaëtan Rivet]

On 10/06/20 17:17 +0000, Parav Pandit wrote:
> mlx5 PCI Device supports multiple classes of devices such as net, vdpa,
> and/or regex.
> To support these multiple classes, change mlx5_class to a
> bitmap values so that if users asks to enable multiple of them, all
> supported classes can be returned by mlx5_class_supported().
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/common/mlx5/mlx5_common.h | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h
> index 1d59873c8..9fdbd341d 100644
> --- a/drivers/common/mlx5/mlx5_common.h
> +++ b/drivers/common/mlx5/mlx5_common.h
> @@ -13,6 +13,7 @@
>  #include <rte_log.h>
>  #include <rte_kvargs.h>
>  #include <rte_devargs.h>
> +#include <rte_bits.h>
>  
>  #include "mlx5_prm.h"
>  
> @@ -203,8 +204,8 @@ int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr);
>  
>  enum mlx5_class {
>  	MLX5_CLASS_INVALID,
> -	MLX5_CLASS_NET,
> -	MLX5_CLASS_VDPA,
> +	MLX5_CLASS_NET = RTE_BIT(0),
> +	MLX5_CLASS_VDPA = RTE_BIT(1),
>  };
>  
>  __rte_internal
> -- 
> 2.25.4
> 

Well, I should have read further after all :)

You should merge this commit with the previous one, accompanying the
class getter API change.

I am not sure if an enum width is fixed however. I think it is
implementation defined. If so, using a define would be better.

-- 
Gaëtan

Re: [dpdk-dev] [RFC PATCH 3/6] common/mlx5: change mlx5 class enum values as bits

[Parav Pandit]

> From: Gaëtan Rivet <grive@u256.net>
> Sent: Tuesday, June 16, 2020 1:25 AM
> 
> On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > mlx5 PCI Device supports multiple classes of devices such as net,
> > vdpa, and/or regex.
> > To support these multiple classes, change mlx5_class to a bitmap
> > values so that if users asks to enable multiple of them, all supported
> > classes can be returned by mlx5_class_supported().
> >
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > ---
> >  drivers/common/mlx5/mlx5_common.h | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/common/mlx5/mlx5_common.h
> > b/drivers/common/mlx5/mlx5_common.h
> > index 1d59873c8..9fdbd341d 100644
> > --- a/drivers/common/mlx5/mlx5_common.h
> > +++ b/drivers/common/mlx5/mlx5_common.h
> > @@ -13,6 +13,7 @@
> >  #include <rte_log.h>
> >  #include <rte_kvargs.h>
> >  #include <rte_devargs.h>
> > +#include <rte_bits.h>
> >
> >  #include "mlx5_prm.h"
> >
> > @@ -203,8 +204,8 @@ int mlx5_dev_to_pci_addr(const char *dev_path,
> > struct rte_pci_addr *pci_addr);
> >
> >  enum mlx5_class {
> >  	MLX5_CLASS_INVALID,
> > -	MLX5_CLASS_NET,
> > -	MLX5_CLASS_VDPA,
> > +	MLX5_CLASS_NET = RTE_BIT(0),
> > +	MLX5_CLASS_VDPA = RTE_BIT(1),
> >  };
> >
> >  __rte_internal
> > --
> > 2.25.4
> >
> 
> Well, I should have read further after all :)
> 
No problem. :-)

> You should merge this commit with the previous one, accompanying the
> class getter API change.
> 
Ok. Usually two changes in two different patches. But its ok to merge.

> I am not sure if an enum width is fixed however. I think it is
> implementation defined. If so, using a define would be better.
Will change to #define.

[dpdk-dev] [RFC PATCH 4/6] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
pci device.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 config/common_base                            |  6 ++
 config/defconfig_arm64-bluefield-linuxapp-gcc |  6 ++
 drivers/bus/meson.build                       |  2 +-
 drivers/bus/mlx5_pci/Makefile                 | 48 ++++++++++++++
 drivers/bus/mlx5_pci/meson.build              |  6 ++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 65 +++++++++++++++++++
 .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
 8 files changed, 151 insertions(+), 1 deletion(-)
 create mode 100644 drivers/bus/mlx5_pci/Makefile
 create mode 100644 drivers/bus/mlx5_pci/meson.build
 create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
 create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map

diff --git a/config/common_base b/config/common_base
index c7d5c7321..f75b333f9 100644
--- a/config/common_base
+++ b/config/common_base
@@ -366,6 +366,12 @@ CONFIG_RTE_LIBRTE_IGC_DEBUG_TX=n
 CONFIG_RTE_LIBRTE_MLX4_PMD=n
 CONFIG_RTE_LIBRTE_MLX4_DEBUG=n
 
+#
+# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5,
+# ConnectX-6 & BlueField (MLX5) PMD
+#
+CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
+
 #
 # Compile burst-oriented Mellanox ConnectX-4, ConnectX-5,
 # ConnectX-6 & BlueField (MLX5) PMD
diff --git a/config/defconfig_arm64-bluefield-linuxapp-gcc b/config/defconfig_arm64-bluefield-linuxapp-gcc
index b49653881..15ade7ebc 100644
--- a/config/defconfig_arm64-bluefield-linuxapp-gcc
+++ b/config/defconfig_arm64-bluefield-linuxapp-gcc
@@ -14,5 +14,11 @@ CONFIG_RTE_CACHE_LINE_SIZE=64
 CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
 CONFIG_RTE_LIBRTE_VHOST_NUMA=n
 
+#
+# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5,
+# ConnectX-6 & BlueField (MLX5) PMD
+#
+CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
+
 # PMD for ConnectX-5
 CONFIG_RTE_LIBRTE_MLX5_PMD=y
diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
index 80de2d91d..b1381838d 100644
--- a/drivers/bus/meson.build
+++ b/drivers/bus/meson.build
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
-drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
+drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
 std_deps = ['eal']
 config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
 driver_name_fmt = 'rte_bus_@0@'
diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
new file mode 100644
index 000000000..b36916e52
--- /dev/null
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -0,0 +1,48 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright 2020 Mellanox Technologies, Ltd
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+#
+# library name
+#
+LIB = librte_bus_mlx5_pci.a
+
+CFLAGS += -O3
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
+CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
+CFLAGS += -Wno-strict-prototypes
+LDLIBS += -lrte_eal
+LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_pci -lrte_bus_pci
+
+# versioning export map
+EXPORT_MAP := rte_bus_mlx5_pci_version.map
+
+SRCS-y += mlx5_pci_bus.c
+
+# DEBUG which is usually provided on the command-line may enable
+# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
+ifeq ($(DEBUG),1)
+CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
+endif
+
+# User-defined CFLAGS.
+ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y)
+CFLAGS += -pedantic
+ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y)
+CFLAGS += -DPEDANTIC
+endif
+AUTO_CONFIG_CFLAGS += -Wno-pedantic
+else
+CFLAGS += -UPEDANTIC
+endif
+
+#
+# Export include files
+#
+SYMLINK-y-include += rte_bus_mlx5_pci.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
new file mode 100644
index 000000000..cc4a84e23
--- /dev/null
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2020 Mellanox Technologies Ltd
+
+deps += ['pci', 'bus_pci', 'common_mlx5']
+install_headers('rte_bus_mlx5_pci.h')
+sources = files('mlx5_pci_bus.c')
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
new file mode 100644
index 000000000..66db3c7b0
--- /dev/null
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#include "rte_bus_mlx5_pci.h"
+
+static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
+				TAILQ_HEAD_INITIALIZER(drv_list);
+
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
+{
+	TAILQ_INSERT_TAIL(&drv_list, driver, next);
+}
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
new file mode 100644
index 000000000..b0423f99e
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright 2020 Mellanox Technologies, Ltd
+ */
+
+#ifndef _RTE_BUS_MLX5_PCI_H_
+#define _RTE_BUS_MLX5_PCI_H_
+
+/**
+ * @file
+ *
+ * RTE Mellanox PCI Bus Interface
+ * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
+ * devices. This bus enables creating such multiple class of devices on a
+ * single PCI device by allowing to bind multiple class specific device
+ * driver to attach to mlx5_pci bus driver.
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif /* __cplusplus */
+
+#include <rte_pci.h>
+#include <rte_bus_pci.h>
+
+#include <mlx5_common.h>
+
+/**
+ * A structure describing a mlx5 pci driver.
+ */
+struct rte_mlx5_pci_driver {
+	enum mlx5_class dev_class;    /**< Class of this driver */
+	struct rte_driver driver;     /**< Inherit core driver. */
+	pci_probe_t *probe;           /**< Class device probe function. */
+	pci_remove_t *remove;         /**< Class device remove function. */
+	pci_dma_map_t *dma_map;       /**< Class device dma map function. */
+	pci_dma_unmap_t *dma_unmap;   /**< Class device dma unmap function. */
+	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
+	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
+};
+
+/**
+ * Register a mlx5_pci device driver.
+ *
+ * @param driver
+ *   A pointer to a rte_mlx5_pci_driver structure describing the driver
+ *   to be registered.
+ */
+__rte_internal
+void
+rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
+
+#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv)	\
+static const char *mlx5_pci_drvinit_fn_ ## nm;	\
+RTE_INIT(mlx5_pci_drvinit_fn_ ##drv)	\
+{	\
+	(drv).driver.name = RTE_STR(nm);	\
+	rte_mlx5_pci_driver_register(&drv);	\
+}	\
+RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
+
+#ifdef __cplusplus
+}
+#endif /* __cplusplus */
+
+#endif /* _RTE_BUS_MLX5_PCI_H_ */
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
new file mode 100644
index 000000000..4cfd3db10
--- /dev/null
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
@@ -0,0 +1,5 @@
+INTERNAL {
+	global:
+
+	rte_mlx5_pci_driver_register;
+};
-- 
2.25.4

Re: [dpdk-dev] [RFC PATCH 4/6] bus/mlx5_pci: add mlx5 PCI bus

[Gaëtan Rivet]

On 10/06/20 17:17 +0000, Parav Pandit wrote:
> Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
> pci device.
> 

This is a little quick to explain the architecture here.
First it should be clear that this is not, in fact, a bus.

You only define a PCI driver, that will demux PCI ops towards
several sub-drivers. We can call it a bus in the sense that it
will support multiple devices and carry on some control, but this should
be made clear here that no rte_bus is being added.

> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  config/common_base                            |  6 ++
>  config/defconfig_arm64-bluefield-linuxapp-gcc |  6 ++
>  drivers/bus/meson.build                       |  2 +-
>  drivers/bus/mlx5_pci/Makefile                 | 48 ++++++++++++++
>  drivers/bus/mlx5_pci/meson.build              |  6 ++
>  drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
>  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 65 +++++++++++++++++++
>  .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
>  8 files changed, 151 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/bus/mlx5_pci/Makefile
>  create mode 100644 drivers/bus/mlx5_pci/meson.build
>  create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
>  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
>  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> 
> diff --git a/config/common_base b/config/common_base
> index c7d5c7321..f75b333f9 100644
> --- a/config/common_base
> +++ b/config/common_base
> @@ -366,6 +366,12 @@ CONFIG_RTE_LIBRTE_IGC_DEBUG_TX=n
>  CONFIG_RTE_LIBRTE_MLX4_PMD=n
>  CONFIG_RTE_LIBRTE_MLX4_DEBUG=n
>  
> +#
> +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5,
> +# ConnectX-6 & BlueField (MLX5) PMD
> +#
> +CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
> +

I'm not a fan of having yet another CONFIG_ toggle to enable in build
systems. Ideally, this should be enabled implicitly by enabling any of
its dependents (MLX5 PMD, MLX5_VDPA_PMD, REGEX I guess, etc).

You can find such similar constructs already in some makefiles:

mk/rte.app.mk:204:ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)

Actually, reading further commits, you already use this construct when
you enable the build for VDPA and MLX5 PMDs, I think this option is not
needed then?

>  #
>  # Compile burst-oriented Mellanox ConnectX-4, ConnectX-5,
>  # ConnectX-6 & BlueField (MLX5) PMD
> diff --git a/config/defconfig_arm64-bluefield-linuxapp-gcc b/config/defconfig_arm64-bluefield-linuxapp-gcc
> index b49653881..15ade7ebc 100644
> --- a/config/defconfig_arm64-bluefield-linuxapp-gcc
> +++ b/config/defconfig_arm64-bluefield-linuxapp-gcc
> @@ -14,5 +14,11 @@ CONFIG_RTE_CACHE_LINE_SIZE=64
>  CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
>  CONFIG_RTE_LIBRTE_VHOST_NUMA=n
>  
> +#
> +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5,
> +# ConnectX-6 & BlueField (MLX5) PMD
> +#
> +CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
> +
>  # PMD for ConnectX-5
>  CONFIG_RTE_LIBRTE_MLX5_PMD=y
> diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build
> index 80de2d91d..b1381838d 100644
> --- a/drivers/bus/meson.build
> +++ b/drivers/bus/meson.build
> @@ -1,7 +1,7 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright(c) 2017 Intel Corporation
>  
> -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
> +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus']
>  std_deps = ['eal']
>  config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
>  driver_name_fmt = 'rte_bus_@0@'
> diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
> new file mode 100644
> index 000000000..b36916e52
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/Makefile
> @@ -0,0 +1,48 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright 2020 Mellanox Technologies, Ltd
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +#
> +# library name
> +#
> +LIB = librte_bus_mlx5_pci.a
> +
> +CFLAGS += -O3
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5
> +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
> +CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
> +CFLAGS += -Wno-strict-prototypes

Why no-strict-prototypes by the way?

> +LDLIBS += -lrte_eal
> +LDLIBS += -lrte_common_mlx5
> +LDLIBS += -lrte_pci -lrte_bus_pci
> +
> +# versioning export map
> +EXPORT_MAP := rte_bus_mlx5_pci_version.map
> +
> +SRCS-y += mlx5_pci_bus.c
> +
> +# DEBUG which is usually provided on the command-line may enable
> +# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
> +ifeq ($(DEBUG),1)
> +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
> +endif
> +
> +# User-defined CFLAGS.
> +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y)
> +CFLAGS += -pedantic
> +ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y)
> +CFLAGS += -DPEDANTIC
> +endif
> +AUTO_CONFIG_CFLAGS += -Wno-pedantic
> +else
> +CFLAGS += -UPEDANTIC
> +endif
> +

At this point why not define some
$(RTE_SDK)/drivers/common/mlx5/mlx5_common.mk

That should be included by vdpa, mlx5, this one?
This would force-align flag behavior, this is becoming untidy.

(Make is disappearing soon I heard, but still.)

> +#
> +# Export include files
> +#
> +SYMLINK-y-include += rte_bus_mlx5_pci.h
> +
> +include $(RTE_SDK)/mk/rte.lib.mk
> diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
> new file mode 100644
> index 000000000..cc4a84e23
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/meson.build
> @@ -0,0 +1,6 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2020 Mellanox Technologies Ltd
> +
> +deps += ['pci', 'bus_pci', 'common_mlx5']
> +install_headers('rte_bus_mlx5_pci.h')
> +sources = files('mlx5_pci_bus.c')
> diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> new file mode 100644
> index 000000000..66db3c7b0
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> @@ -0,0 +1,14 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#include "rte_bus_mlx5_pci.h"
> +
> +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
> +				TAILQ_HEAD_INITIALIZER(drv_list);
> +
> +void
> +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
> +{
> +	TAILQ_INSERT_TAIL(&drv_list, driver, next);
> +}
> diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> new file mode 100644
> index 000000000..b0423f99e
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> @@ -0,0 +1,65 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright 2020 Mellanox Technologies, Ltd
> + */
> +
> +#ifndef _RTE_BUS_MLX5_PCI_H_
> +#define _RTE_BUS_MLX5_PCI_H_
> +
> +/**
> + * @file
> + *
> + * RTE Mellanox PCI Bus Interface
> + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex)
> + * devices. This bus enables creating such multiple class of devices on a
> + * single PCI device by allowing to bind multiple class specific device
> + * driver to attach to mlx5_pci bus driver.
> + */

I think it would be better to explain that this bus is mostly a PCI
driver demuxing to several device classes (you could copy here the
explanation you'd write in the commit log).

> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif /* __cplusplus */
> +
> +#include <rte_pci.h>
> +#include <rte_bus_pci.h>
> +
> +#include <mlx5_common.h>
> +
> +/**
> + * A structure describing a mlx5 pci driver.
> + */
> +struct rte_mlx5_pci_driver {

A note on the namespace: rte_mlx5_pci seems heavy.
Do you expect other types of "super-driver", other than PCI?
Wouldn't rte_mlx5_driver be ok for example?

> +	enum mlx5_class dev_class;    /**< Class of this driver */
> +	struct rte_driver driver;     /**< Inherit core driver. */
> +	pci_probe_t *probe;           /**< Class device probe function. */
> +	pci_remove_t *remove;         /**< Class device remove function. */
> +	pci_dma_map_t *dma_map;       /**< Class device dma map function. */
> +	pci_dma_unmap_t *dma_unmap;   /**< Class device dma unmap function. */
> +	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
> +	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */

At this point, why not inherit an rte_pci_driver instead of the core
rte_driver?

> +};
> +
> +/**
> + * Register a mlx5_pci device driver.
> + *
> + * @param driver
> + *   A pointer to a rte_mlx5_pci_driver structure describing the driver
> + *   to be registered.
> + */
> +__rte_internal
> +void
> +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
> +
> +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv)	\
> +static const char *mlx5_pci_drvinit_fn_ ## nm;	\
> +RTE_INIT(mlx5_pci_drvinit_fn_ ##drv)	\
> +{	\
> +	(drv).driver.name = RTE_STR(nm);	\
> +	rte_mlx5_pci_driver_register(&drv);	\
> +}	\
> +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> +
> +#ifdef __cplusplus
> +}
> +#endif /* __cplusplus */
> +
> +#endif /* _RTE_BUS_MLX5_PCI_H_ */

I'm not sure something is gained by cutting this definition here.
You added the build system but this is an empty shell. Why not merge
this commit and the next?

> diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> new file mode 100644
> index 000000000..4cfd3db10
> --- /dev/null
> +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> @@ -0,0 +1,5 @@
> +INTERNAL {
> +	global:
> +
> +	rte_mlx5_pci_driver_register;
> +};
> -- 
> 2.25.4
> 

-- 
Gaëtan

Re: [dpdk-dev] [RFC PATCH 4/6] bus/mlx5_pci: add mlx5 PCI bus

[Thomas Monjalon]

15/06/2020 23:00, Gaëtan Rivet:
> On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > +# DEBUG which is usually provided on the command-line may enable
> > +# CONFIG_RTE_LIBRTE_MLX5_DEBUG.
> > +ifeq ($(DEBUG),1)
> > +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
> > +endif
> > +
> > +# User-defined CFLAGS.
> > +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y)
> > +CFLAGS += -pedantic
> > +ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y)
> > +CFLAGS += -DPEDANTIC
> > +endif
> > +AUTO_CONFIG_CFLAGS += -Wno-pedantic
> > +else
> > +CFLAGS += -UPEDANTIC
> > +endif
> > +
> 
> At this point why not define some
> $(RTE_SDK)/drivers/common/mlx5/mlx5_common.mk
> 
> That should be included by vdpa, mlx5, this one?
> This would force-align flag behavior, this is becoming untidy.
> 
> (Make is disappearing soon I heard, but still.)

Yes makefiles will be removed in 2 months.
Please do not move makefiles at this point.

[...]
> > +/**
> > + * A structure describing a mlx5 pci driver.
> > + */
> > +struct rte_mlx5_pci_driver {
> 
> A note on the namespace: rte_mlx5_pci seems heavy.
> Do you expect other types of "super-driver", other than PCI?
> Wouldn't rte_mlx5_driver be ok for example?
> 
> > +	enum mlx5_class dev_class;    /**< Class of this driver */
> > +	struct rte_driver driver;     /**< Inherit core driver. */
> > +	pci_probe_t *probe;           /**< Class device probe function. */
> > +	pci_remove_t *remove;         /**< Class device remove function. */
> > +	pci_dma_map_t *dma_map;       /**< Class device dma map function. */
> > +	pci_dma_unmap_t *dma_unmap;   /**< Class device dma unmap function. */
> > +	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
> > +	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
> 
> At this point, why not inherit an rte_pci_driver instead of the core
> rte_driver?

I agree we expect inheriting rte_pci_driver.

Re: [dpdk-dev] [RFC PATCH 4/6] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, June 17, 2020 1:44 PM
> To: Parav Pandit <parav@mellanox.com>; Gaëtan Rivet <grive@u256.net>
> Cc: dev@dpdk.org; ferruh.yigit@intel.com; Ori Kam <orika@mellanox.com>;
> Matan Azrad <matan@mellanox.com>
> Subject: Re: [dpdk-dev] [RFC PATCH 4/6] bus/mlx5_pci: add mlx5 PCI bus
> 
> 15/06/2020 23:00, Gaëtan Rivet:
> > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > +# DEBUG which is usually provided on the command-line may enable #
> > > +CONFIG_RTE_LIBRTE_MLX5_DEBUG.
> > > +ifeq ($(DEBUG),1)
> > > +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
> > > +endif
> > > +
> > > +# User-defined CFLAGS.
> > > +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y)
> > > +CFLAGS += -pedantic
> > > +ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y) CFLAGS += -DPEDANTIC
> endif
> > > +AUTO_CONFIG_CFLAGS += -Wno-pedantic else CFLAGS += -UPEDANTIC
> endif
> > > +
> >
> > At this point why not define some
> > $(RTE_SDK)/drivers/common/mlx5/mlx5_common.mk
> >
> > That should be included by vdpa, mlx5, this one?
> > This would force-align flag behavior, this is becoming untidy.
> >
> > (Make is disappearing soon I heard, but still.)
> 
> Yes makefiles will be removed in 2 months.
> Please do not move makefiles at this point.
> 
> [...]
> > > +/**
> > > + * A structure describing a mlx5 pci driver.
> > > + */
> > > +struct rte_mlx5_pci_driver {
> >
> > A note on the namespace: rte_mlx5_pci seems heavy.
> > Do you expect other types of "super-driver", other than PCI?
> > Wouldn't rte_mlx5_driver be ok for example?
> >
> > > +	enum mlx5_class dev_class;    /**< Class of this driver */
> > > +	struct rte_driver driver;     /**< Inherit core driver. */
> > > +	pci_probe_t *probe;           /**< Class device probe function. */
> > > +	pci_remove_t *remove;         /**< Class device remove function. */
> > > +	pci_dma_map_t *dma_map;       /**< Class device dma map function.
> */
> > > +	pci_dma_unmap_t *dma_unmap;   /**< Class device dma unmap
> function. */
> > > +	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
> > > +	const struct rte_pci_id *id_table; /**< ID table, NULL terminated.
> > > +*/
> >
> > At this point, why not inherit an rte_pci_driver instead of the core
> > rte_driver?
> 
> I agree we expect inheriting rte_pci_driver.
> 
Ok. I will attempt this.

Re: [dpdk-dev] [RFC PATCH 4/6] bus/mlx5_pci: add mlx5 PCI bus

[Parav Pandit]

> From: Gaëtan Rivet <grive@u256.net>
> Sent: Tuesday, June 16, 2020 2:31 AM
> 
> On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single
> > pci device.
> >
> 
> This is a little quick to explain the architecture here.
> First it should be clear that this is not, in fact, a bus.
> 
> You only define a PCI driver, that will demux PCI ops towards several sub-
> drivers. We can call it a bus in the sense that it will support multiple devices
> and carry on some control, but this should be made clear here that no
> rte_bus is being added.
Yes. will describe here.

> 
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > ---
> >  config/common_base                            |  6 ++
> >  config/defconfig_arm64-bluefield-linuxapp-gcc |  6 ++
> >  drivers/bus/meson.build                       |  2 +-
> >  drivers/bus/mlx5_pci/Makefile                 | 48 ++++++++++++++
> >  drivers/bus/mlx5_pci/meson.build              |  6 ++
> >  drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 14 ++++
> >  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       | 65
> +++++++++++++++++++
> >  .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |  5 ++
> >  8 files changed, 151 insertions(+), 1 deletion(-)  create mode 100644
> > drivers/bus/mlx5_pci/Makefile  create mode 100644
> > drivers/bus/mlx5_pci/meson.build  create mode 100644
> > drivers/bus/mlx5_pci/mlx5_pci_bus.c
> >  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> >  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> >
> > diff --git a/config/common_base b/config/common_base index
> > c7d5c7321..f75b333f9 100644
> > --- a/config/common_base
> > +++ b/config/common_base
> > @@ -366,6 +366,12 @@ CONFIG_RTE_LIBRTE_IGC_DEBUG_TX=n
> > CONFIG_RTE_LIBRTE_MLX4_PMD=n
> CONFIG_RTE_LIBRTE_MLX4_DEBUG=n
> >
> > +#
> > +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5, # ConnectX-6 &
> > +BlueField (MLX5) PMD # CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
> > +
> 
> I'm not a fan of having yet another CONFIG_ toggle to enable in build
> systems. Ideally, this should be enabled implicitly by enabling any of its
> dependents (MLX5 PMD, MLX5_VDPA_PMD, REGEX I guess, etc).
> 
> You can find such similar constructs already in some makefiles:
> 
> mk/rte.app.mk:204:ifeq ($(findstring
> y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_
> PMD)),y)
> 
> Actually, reading further commits, you already use this construct when you
> enable the build for VDPA and MLX5 PMDs, I think this option is not needed
> then?
As Thomas described of makefile removal, I will keep it this way for now.

> 
> >  #
> >  # Compile burst-oriented Mellanox ConnectX-4, ConnectX-5,  #
> > ConnectX-6 & BlueField (MLX5) PMD diff --git
> > a/config/defconfig_arm64-bluefield-linuxapp-gcc
> > b/config/defconfig_arm64-bluefield-linuxapp-gcc
> > index b49653881..15ade7ebc 100644
> > --- a/config/defconfig_arm64-bluefield-linuxapp-gcc
> > +++ b/config/defconfig_arm64-bluefield-linuxapp-gcc
> > @@ -14,5 +14,11 @@ CONFIG_RTE_CACHE_LINE_SIZE=64
> > CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n
> >  CONFIG_RTE_LIBRTE_VHOST_NUMA=n
> >
> > +#
> > +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5, # ConnectX-6 &
> > +BlueField (MLX5) PMD # CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n
> > +
> >  # PMD for ConnectX-5
> >  CONFIG_RTE_LIBRTE_MLX5_PMD=y
> > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index
> > 80de2d91d..b1381838d 100644
> > --- a/drivers/bus/meson.build
> > +++ b/drivers/bus/meson.build
> > @@ -1,7 +1,7 @@
> >  # SPDX-License-Identifier: BSD-3-Clause  # Copyright(c) 2017 Intel
> > Corporation
> >
> > -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus']
> > +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev',
> > +'vmbus']
> >  std_deps = ['eal']
> >  config_flag_fmt = 'RTE_LIBRTE_@0@_BUS'
> >  driver_name_fmt = 'rte_bus_@0@'
> > diff --git a/drivers/bus/mlx5_pci/Makefile
> > b/drivers/bus/mlx5_pci/Makefile new file mode 100644 index
> > 000000000..b36916e52
> > --- /dev/null
> > +++ b/drivers/bus/mlx5_pci/Makefile
> > @@ -0,0 +1,48 @@
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright 2020 Mellanox
> > +Technologies, Ltd
> > +
> > +include $(RTE_SDK)/mk/rte.vars.mk
> > +
> > +#
> > +# library name
> > +#
> > +LIB = librte_bus_mlx5_pci.a
> > +
> > +CFLAGS += -O3
> > +CFLAGS += $(WERROR_FLAGS)
> > +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 CFLAGS +=
> > +-I$(BUILDDIR)/drivers/common/mlx5
> > +CFLAGS += -I$(RTE_SDK)/drivers/bus/pci CFLAGS +=
> > +-Wno-strict-prototypes
> 
> Why no-strict-prototypes by the way?
I should use strict prototypes. Will do.

> 
> > +LDLIBS += -lrte_eal
> > +LDLIBS += -lrte_common_mlx5
> > +LDLIBS += -lrte_pci -lrte_bus_pci
> > +
> > +# versioning export map
> > +EXPORT_MAP := rte_bus_mlx5_pci_version.map
> > +
> > +SRCS-y += mlx5_pci_bus.c
> > +
> > +# DEBUG which is usually provided on the command-line may enable #
> > +CONFIG_RTE_LIBRTE_MLX5_DEBUG.
> > +ifeq ($(DEBUG),1)
> > +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y
> > +endif
> > +
> > +# User-defined CFLAGS.
> > +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y)
> > +CFLAGS += -pedantic
> > +ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y) CFLAGS += -DPEDANTIC endif
> > +AUTO_CONFIG_CFLAGS += -Wno-pedantic else CFLAGS += -UPEDANTIC
> endif
> > +
> 
> At this point why not define some
> $(RTE_SDK)/drivers/common/mlx5/mlx5_common.mk
> 
Yes. will keep this refactor in different series, mostly after Makefiles are removed.

> That should be included by vdpa, mlx5, this one?
> This would force-align flag behavior, this is becoming untidy.
> 
> (Make is disappearing soon I heard, but still.)
> 
> > +#
> > +# Export include files
> > +#
> > +SYMLINK-y-include += rte_bus_mlx5_pci.h
> > +
> > +include $(RTE_SDK)/mk/rte.lib.mk
> > diff --git a/drivers/bus/mlx5_pci/meson.build
> > b/drivers/bus/mlx5_pci/meson.build
> > new file mode 100644
> > index 000000000..cc4a84e23
> > --- /dev/null
> > +++ b/drivers/bus/mlx5_pci/meson.build
> > @@ -0,0 +1,6 @@
> > +# SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Mellanox
> > +Technologies Ltd
> > +
> > +deps += ['pci', 'bus_pci', 'common_mlx5']
> > +install_headers('rte_bus_mlx5_pci.h')
> > +sources = files('mlx5_pci_bus.c')
> > diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> > b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> > new file mode 100644
> > index 000000000..66db3c7b0
> > --- /dev/null
> > +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> > @@ -0,0 +1,14 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright 2020 Mellanox Technologies, Ltd  */
> > +
> > +#include "rte_bus_mlx5_pci.h"
> > +
> > +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver)
> drv_list =
> > +				TAILQ_HEAD_INITIALIZER(drv_list);
> > +
> > +void
> > +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) {
> > +	TAILQ_INSERT_TAIL(&drv_list, driver, next); }
> > diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> > b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> > new file mode 100644
> > index 000000000..b0423f99e
> > --- /dev/null
> > +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> > @@ -0,0 +1,65 @@
> > +/* SPDX-License-Identifier: BSD-3-Clause
> > + * Copyright 2020 Mellanox Technologies, Ltd  */
> > +
> > +#ifndef _RTE_BUS_MLX5_PCI_H_
> > +#define _RTE_BUS_MLX5_PCI_H_
> > +
> > +/**
> > + * @file
> > + *
> > + * RTE Mellanox PCI Bus Interface
> > + * Mellanox ConnectX PCI device supports multiple class
> > +(net/vdpa/regex)
> > + * devices. This bus enables creating such multiple class of devices
> > +on a
> > + * single PCI device by allowing to bind multiple class specific
> > +device
> > + * driver to attach to mlx5_pci bus driver.
> > + */
> 
> I think it would be better to explain that this bus is mostly a PCI driver
> demuxing to several device classes (you could copy here the explanation
> you'd write in the commit log).
> 
Sure. Will do.

> > +
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif /* __cplusplus */
> > +
> > +#include <rte_pci.h>
> > +#include <rte_bus_pci.h>
> > +
> > +#include <mlx5_common.h>
> > +
> > +/**
> > + * A structure describing a mlx5 pci driver.
> > + */
> > +struct rte_mlx5_pci_driver {
> 
> A note on the namespace: rte_mlx5_pci seems heavy.
> Do you expect other types of "super-driver", other than PCI?
> Wouldn't rte_mlx5_driver be ok for example?
Yes. I am working on virtbus devices. A new virbus is on horizon in kernel.
Mellanox sub-function devices will be anchored on such virtbus device.
At that point we will have rte_mlx5_virtbus_driver.
So I prefer to keep _pci prefix.

> 
> > +	enum mlx5_class dev_class;    /**< Class of this driver */
> > +	struct rte_driver driver;     /**< Inherit core driver. */
> > +	pci_probe_t *probe;           /**< Class device probe function. */
> > +	pci_remove_t *remove;         /**< Class device remove function. */
> > +	pci_dma_map_t *dma_map;       /**< Class device dma map function.
> */
> > +	pci_dma_unmap_t *dma_unmap;   /**< Class device dma unmap
> function. */
> > +	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
> > +	const struct rte_pci_id *id_table; /**< ID table, NULL terminated.
> > +*/
> 
> At this point, why not inherit an rte_pci_driver instead of the core
> rte_driver?
It should be possible. I will try it out.

> 
> > +};
> > +
> > +/**
> > + * Register a mlx5_pci device driver.
> > + *
> > + * @param driver
> > + *   A pointer to a rte_mlx5_pci_driver structure describing the driver
> > + *   to be registered.
> > + */
> > +__rte_internal
> > +void
> > +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver);
> > +
> > +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv)	\
> > +static const char *mlx5_pci_drvinit_fn_ ## nm;	\
> > +RTE_INIT(mlx5_pci_drvinit_fn_ ##drv)	\
> > +{	\
> > +	(drv).driver.name = RTE_STR(nm);	\
> > +	rte_mlx5_pci_driver_register(&drv);	\
> > +}	\
> > +RTE_PMD_EXPORT_NAME(nm, __COUNTER__)
> > +
> > +#ifdef __cplusplus
> > +}
> > +#endif /* __cplusplus */
> > +
> > +#endif /* _RTE_BUS_MLX5_PCI_H_ */
> 
> I'm not sure something is gained by cutting this definition here.
> You added the build system but this is an empty shell. Why not merge this
> commit and the next?
> 
Combining both patches in one looks a big patch.
2nd patch contains the core bus logic, this one contains the framework.
So functionality level split is possible, hence two patches.

[dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

Create a mlx5 bus driver framework for invoking drivers of
multiple classes who have registered with the mlx5_pci bus
driver.

Validate user class arguments for supported class combinations.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/bus/mlx5_pci/Makefile           |   1 +
 drivers/bus/mlx5_pci/meson.build        |   2 +-
 drivers/bus/mlx5_pci/mlx5_pci_bus.c     | 253 ++++++++++++++++++++++++
 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h |   1 +
 4 files changed, 256 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
index b36916e52..327076fe4 100644
--- a/drivers/bus/mlx5_pci/Makefile
+++ b/drivers/bus/mlx5_pci/Makefile
@@ -15,6 +15,7 @@ CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_eal
+LDLIBS += -lrte_kvargs
 LDLIBS += -lrte_common_mlx5
 LDLIBS += -lrte_pci -lrte_bus_pci
 
diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
index cc4a84e23..5111baa4e 100644
--- a/drivers/bus/mlx5_pci/meson.build
+++ b/drivers/bus/mlx5_pci/meson.build
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2020 Mellanox Technologies Ltd
 
-deps += ['pci', 'bus_pci', 'common_mlx5']
+deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
 install_headers('rte_bus_mlx5_pci.h')
 sources = files('mlx5_pci_bus.c')
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index 66db3c7b0..8108e35ea 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -3,12 +3,265 @@
  */
 
 #include "rte_bus_mlx5_pci.h"
+#include <mlx5_common_utils.h>
 
 static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
 				TAILQ_HEAD_INITIALIZER(drv_list);
 
+struct class_map {
+	const char *name;
+	enum mlx5_class dev_class;
+};
+
+const struct class_map mlx5_classes[] = {
+	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
+	{ .name = "net", .dev_class = MLX5_CLASS_NET },
+};
+
+static const enum mlx5_class mlx5_valid_class_combo[] = {
+	MLX5_CLASS_NET,
+	MLX5_CLASS_VDPA,
+	/* New class combination should be added here */
+};
+
+static const struct class_map *is_valid_class(const char *class_name)
+{
+	unsigned int i;
+
+	for (i = 0; i < sizeof(mlx5_classes) / sizeof(struct class_map);
+	     i++) {
+		if (strcmp(class_name, mlx5_classes[i].name) == 0)
+			return &mlx5_classes[i];
+
+	}
+	return NULL;
+}
+
+static int is_valid_class_combo(const char *class_names)
+{
+	enum mlx5_class user_classes = 0;
+	char *nstr = strdup(class_names);
+	const struct class_map *entry;
+	char *copy = nstr;
+	int invalid = 0;
+	unsigned int i;
+	char *found;
+
+	while (nstr) {
+		/* Extract each individual class name */
+		found = strsep(&nstr, ":");
+		if (!found)
+			continue;
+
+		/* Check if its a valid class */
+		entry = is_valid_class(found);
+		if (!entry) {
+			invalid = EINVAL;
+			break;
+		}
+		user_classes |= entry->dev_class;
+	}
+	if (copy)
+		free(copy);
+	if (invalid)
+		return invalid;
+
+	/* Verify if user specified valid supported combination */
+	for (i = 0;
+	     i < sizeof(mlx5_valid_class_combo) / sizeof(enum mlx5_class);
+	     i++) {
+		if (mlx5_valid_class_combo[i] == user_classes)
+			break;
+	}
+	/* Not found any valid class combination */
+	if (i == sizeof(mlx5_valid_class_combo) / sizeof(enum mlx5_class))
+		return -EINVAL;
+	else
+		return 0;
+}
+
+static int
+mlx5_bus_opt_handler(__rte_unused const char *key, const char *value,
+		       void *opaque)
+{
+	int *ret = opaque;
+
+	*ret = is_valid_class_combo(value);
+	if (*ret)
+		DRV_LOG(ERR, "Invalid mlx5 classes %s. Maybe typo in device"
+			" class argument setting?", value);
+	return 0;
+}
+
+static int
+mlx5_bus_options_valid(const struct rte_devargs *devargs)
+{
+	struct rte_kvargs *kvlist;
+	const char *key = MLX5_CLASS_ARG_NAME;
+	int ret = 0;
+
+	if (devargs == NULL)
+		return 0;
+	kvlist = rte_kvargs_parse(devargs->args, NULL);
+	if (kvlist == NULL)
+		return 0;
+	if (rte_kvargs_count(kvlist, key))
+		rte_kvargs_process(kvlist, key, mlx5_bus_opt_handler, &ret);
+	rte_kvargs_free(kvlist);
+	return ret;
+}
+
 void
 rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
 {
 	TAILQ_INSERT_TAIL(&drv_list, driver, next);
 }
+
+static bool
+mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
+		 const struct rte_pci_device *pci_dev)
+{
+	const struct rte_pci_id *id_table;
+
+	for (id_table = drv->id_table; id_table->vendor_id != 0; id_table++) {
+		/* check if device's ids match the class driver's ones */
+		if (id_table->vendor_id != pci_dev->id.vendor_id &&
+				id_table->vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->device_id != pci_dev->id.device_id &&
+				id_table->device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_vendor_id !=
+		    pci_dev->id.subsystem_vendor_id &&
+		    id_table->subsystem_vendor_id != PCI_ANY_ID)
+			continue;
+		if (id_table->subsystem_device_id !=
+		    pci_dev->id.subsystem_device_id &&
+		    id_table->subsystem_device_id != PCI_ANY_ID)
+			continue;
+		if (id_table->class_id != pci_dev->id.class_id &&
+				id_table->class_id != RTE_CLASS_ANY_ID)
+			continue;
+
+		return true;
+	}
+	return false;
+}
+
+/**
+ * DPDK callback to register to probe multiple PCI class devices.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure.
+ * @param[in] dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, 1 to skip this driver, a negative errno value otherwise
+ *   and rte_errno is set.
+ */
+static int
+mlx5_bus_pci_probe(struct rte_pci_driver *drv __rte_unused,
+		     struct rte_pci_device *dev)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret = 0;
+
+	ret = mlx5_bus_options_valid(dev->device.devargs);
+	if (ret)
+		return ret;
+
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (!mlx5_bus_match(class, dev))
+			continue;
+
+		if (!mlx5_class_enabled(dev->device.devargs, class->dev_class))
+			continue;
+
+		ret = class->probe(drv, dev);
+		if (!ret)
+			class->loaded = true;
+	}
+	return 0;
+}
+
+/**
+ * DPDK callback to remove one or more class devices for a PCI device.
+ *
+ * This function removes all class devices belong to a given PCI device.
+ *
+ * @param[in] pci_dev
+ *   Pointer to the PCI device.
+ *
+ * @return
+ *   0 on success, the function cannot fail.
+ */
+static int
+mlx5_bus_pci_remove(struct rte_pci_device *dev)
+{
+	struct rte_mlx5_pci_driver *class;
+
+	/* Remove each class driver in reverse order */
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (!mlx5_class_enabled(dev->device.devargs, class->dev_class))
+			continue;
+
+		if (class->loaded)
+			class->remove(dev);
+	}
+	return 0;
+}
+
+static int
+mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr,
+		     uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret = -EINVAL;
+
+	TAILQ_FOREACH(class, &drv_list, next) {
+		if (!class->dma_map)
+			continue;
+
+		return class->dma_map(dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static int
+mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
+		       uint64_t iova, size_t len)
+{
+	struct rte_mlx5_pci_driver *class;
+	int ret = -EINVAL;
+
+	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
+		if (!class->dma_unmap)
+			continue;
+
+		return class->dma_unmap(dev, addr, iova, len);
+	}
+	return ret;
+}
+
+static const struct rte_pci_id mlx5_bus_pci_id_map[] = {
+	{
+		.vendor_id = 0
+	}
+};
+
+static struct rte_pci_driver mlx5_bus_driver = {
+	.driver = {
+		.name = "mlx5_bus_pci",
+	},
+	.id_table = mlx5_bus_pci_id_map,
+	.probe = mlx5_bus_pci_probe,
+	.remove = mlx5_bus_pci_remove,
+	.dma_map = mlx5_bus_pci_dma_map,
+	.dma_unmap = mlx5_bus_pci_dma_unmap,
+	.drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV |
+		     RTE_PCI_DRV_PROBE_AGAIN,
+};
+
+RTE_PMD_REGISTER_PCI(mlx5_bus, mlx5_bus_driver);
+RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
index b0423f99e..7be9a15cd 100644
--- a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
+++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
@@ -36,6 +36,7 @@ struct rte_mlx5_pci_driver {
 	pci_dma_unmap_t *dma_unmap;   /**< Class device dma unmap function. */
 	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
 	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
+	bool loaded;
 };
 
 /**
-- 
2.25.4

Re: [dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver

[Gaëtan Rivet]

On 10/06/20 17:17 +0000, Parav Pandit wrote:
> Create a mlx5 bus driver framework for invoking drivers of
> multiple classes who have registered with the mlx5_pci bus
> driver.
> 
> Validate user class arguments for supported class combinations.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/bus/mlx5_pci/Makefile           |   1 +
>  drivers/bus/mlx5_pci/meson.build        |   2 +-
>  drivers/bus/mlx5_pci/mlx5_pci_bus.c     | 253 ++++++++++++++++++++++++
>  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h |   1 +
>  4 files changed, 256 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile
> index b36916e52..327076fe4 100644
> --- a/drivers/bus/mlx5_pci/Makefile
> +++ b/drivers/bus/mlx5_pci/Makefile
> @@ -15,6 +15,7 @@ CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
>  CFLAGS += -I$(RTE_SDK)/drivers/bus/pci
>  CFLAGS += -Wno-strict-prototypes
>  LDLIBS += -lrte_eal
> +LDLIBS += -lrte_kvargs
>  LDLIBS += -lrte_common_mlx5
>  LDLIBS += -lrte_pci -lrte_bus_pci
>  
> diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build
> index cc4a84e23..5111baa4e 100644
> --- a/drivers/bus/mlx5_pci/meson.build
> +++ b/drivers/bus/mlx5_pci/meson.build
> @@ -1,6 +1,6 @@
>  # SPDX-License-Identifier: BSD-3-Clause
>  # Copyright(c) 2020 Mellanox Technologies Ltd
>  
> -deps += ['pci', 'bus_pci', 'common_mlx5']
> +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
>  install_headers('rte_bus_mlx5_pci.h')
>  sources = files('mlx5_pci_bus.c')
> diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> index 66db3c7b0..8108e35ea 100644
> --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> @@ -3,12 +3,265 @@
>   */
>  
>  #include "rte_bus_mlx5_pci.h"
> +#include <mlx5_common_utils.h>
>  
>  static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list =
>  				TAILQ_HEAD_INITIALIZER(drv_list);
>  
> +struct class_map {
> +	const char *name;
> +	enum mlx5_class dev_class;
> +};

Defining a type here does not seem useful.
You could return an "enum mlx5_class" from the function is_valid_class()
further below, instead of a class_map entry. You have the
MLX5_CLASS_INVALID sentinel value to mark the inexistant mapping.

Making this type anonymous, you should merge it with the array below:

static const struct {
	const char *name;
	enum mlx5_class class; // Remember to change this enum to a
			       // fixed width type by the way.
} mlx5_classes[] = {
	...
};

> +
> +const struct class_map mlx5_classes[] = {

This is not really a class list, but as the type describes,
a mapping between names and binary id.

I think mlx5_class_names would fit better.

> +	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
> +	{ .name = "net", .dev_class = MLX5_CLASS_NET },
> +};

Globals should be static even if not exposed through header.

> +
> +static const enum mlx5_class mlx5_valid_class_combo[] = {
> +	MLX5_CLASS_NET,
> +	MLX5_CLASS_VDPA,

How do you describe future combos?
Should we expect MLX5_CLASS_NET | MLX5_CLASS_REGEX for example?

> +	/* New class combination should be added here */
> +};
> +
> +static const struct class_map *is_valid_class(const char *class_name)

I don't think the function name conveys its actual use.
mlx5_class_from_name() for example would align with other DPDK APIs.

> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < sizeof(mlx5_classes) / sizeof(struct class_map);

RTE_DIM(mlx5_classes) must be used instead.

> +	     i++) {
> +		if (strcmp(class_name, mlx5_classes[i].name) == 0)
> +			return &mlx5_classes[i];
> +
> +	}
> +	return NULL;
> +}
> +
> +static int is_valid_class_combo(const char *class_names)
> +{
> +	enum mlx5_class user_classes = 0;
> +	char *nstr = strdup(class_names);

Not a fan of assignment with malloc directly at var declaration,
you are missing some checks here.

> +	const struct class_map *entry;
> +	char *copy = nstr;

copy is nondescript and dangerous. I'd use
nstr_orig instead.

> +	int invalid = 0;
> +	unsigned int i;
> +	char *found;

Reading the code below, it seems that the kvlist should only
be defined if the devargs length is > 0, so class_names here should be
defined.

However you do not handle the OOM case explicitly here,
if nstr cannot be allocated on strdup().

> +
> +	while (nstr) {
> +		/* Extract each individual class name */
> +		found = strsep(&nstr, ":");

I have not seen the feature test macros (_DEFAULT_SOURCE) in
the Makefile, it seems required for strsep()?

> +		if (!found)
> +			continue;
> +
> +		/* Check if its a valid class */
> +		entry = is_valid_class(found);

As said earlier, you have no use for the full map entry,
you could return the mlx5_class type instead, as you have
the MLX5_CLASS_INVALID sentinel available to mark the !found case.

> +		if (!entry) {
> +			invalid = EINVAL;

Just toggling invalid = 1; here would be better.
Return EINVAL explicitly if (invalid).

> +			break;
> +		}
> +		user_classes |= entry->dev_class;
> +	}
> +	if (copy)
> +		free(copy);
> +	if (invalid)
> +		return invalid;

You are returning EINVAL here, but -EINVAL below.
It should be aligned, in DPDK usually error return values are negative.
positive errno should only be used for rte_errno.

> +
> +	/* Verify if user specified valid supported combination */
> +	for (i = 0;
> +	     i < sizeof(mlx5_valid_class_combo) / sizeof(enum mlx5_class);

RTE_DIM() here;

> +	     i++) {
> +		if (mlx5_valid_class_combo[i] == user_classes)

return 0; directly?

> +			break;
> +	}
> +	/* Not found any valid class combination */
> +	if (i == sizeof(mlx5_valid_class_combo) / sizeof(enum mlx5_class))

This would simplify this check, where you can directly return -ENODEV
for example to differentiate from invalid class name and invalid combo.

> +		return -EINVAL;
> +	else
> +		return 0;
> +}
> +
> +static int
> +mlx5_bus_opt_handler(__rte_unused const char *key, const char *value,
> +		       void *opaque)
> +{
> +	int *ret = opaque;
> +
> +	*ret = is_valid_class_combo(value);
> +	if (*ret)
> +		DRV_LOG(ERR, "Invalid mlx5 classes %s. Maybe typo in device"
> +			" class argument setting?", value);

Error message should not be cut in half, it makes it difficult to grep.
If you differentiate between typo in name and invalid combo you could
directly warn the user about the proper error.

(You can ignore the warning from checkpatch.sh about the long lines on a
string if there is one.)

> +	return 0;
> +}
> +
> +static int
> +mlx5_bus_options_valid(const struct rte_devargs *devargs)
> +{
> +	struct rte_kvargs *kvlist;
> +	const char *key = MLX5_CLASS_ARG_NAME;
> +	int ret = 0;
> +
> +	if (devargs == NULL)
> +		return 0;
> +	kvlist = rte_kvargs_parse(devargs->args, NULL);
> +	if (kvlist == NULL)
> +		return 0;
> +	if (rte_kvargs_count(kvlist, key))
> +		rte_kvargs_process(kvlist, key, mlx5_bus_opt_handler, &ret);
> +	rte_kvargs_free(kvlist);
> +	return ret;
> +}
> +
>  void
>  rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)
>  {
>  	TAILQ_INSERT_TAIL(&drv_list, driver, next);
>  }
> +
> +static bool
> +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
> +		 const struct rte_pci_device *pci_dev)
> +{
> +	const struct rte_pci_id *id_table;
> +
> +	for (id_table = drv->id_table; id_table->vendor_id != 0; id_table++) {
> +		/* check if device's ids match the class driver's ones */
> +		if (id_table->vendor_id != pci_dev->id.vendor_id &&
> +				id_table->vendor_id != PCI_ANY_ID)
> +			continue;
> +		if (id_table->device_id != pci_dev->id.device_id &&
> +				id_table->device_id != PCI_ANY_ID)
> +			continue;
> +		if (id_table->subsystem_vendor_id !=
> +		    pci_dev->id.subsystem_vendor_id &&
> +		    id_table->subsystem_vendor_id != PCI_ANY_ID)
> +			continue;
> +		if (id_table->subsystem_device_id !=
> +		    pci_dev->id.subsystem_device_id &&
> +		    id_table->subsystem_device_id != PCI_ANY_ID)
> +			continue;
> +		if (id_table->class_id != pci_dev->id.class_id &&
> +				id_table->class_id != RTE_CLASS_ANY_ID)
> +			continue;
> +
> +		return true;
> +	}
> +	return false;
> +}
> +
> +/**
> + * DPDK callback to register to probe multiple PCI class devices.
> + *
> + * @param[in] pci_drv
> + *   PCI driver structure.
> + * @param[in] dev
> + *   PCI device information.
> + *
> + * @return
> + *   0 on success, 1 to skip this driver, a negative errno value otherwise
> + *   and rte_errno is set.
> + */
> +static int
> +mlx5_bus_pci_probe(struct rte_pci_driver *drv __rte_unused,
> +		     struct rte_pci_device *dev)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +	int ret = 0;
> +
> +	ret = mlx5_bus_options_valid(dev->device.devargs);
> +	if (ret)
> +		return ret;
> +
> +	TAILQ_FOREACH(class, &drv_list, next) {
> +		if (!mlx5_bus_match(class, dev))
> +			continue;
> +
> +		if (!mlx5_class_enabled(dev->device.devargs, class->dev_class))
> +			continue;
> +
> +		ret = class->probe(drv, dev);
> +		if (!ret)
> +			class->loaded = true;
> +	}
> +	return 0;
> +}
> +
> +/**
> + * DPDK callback to remove one or more class devices for a PCI device.
> + *
> + * This function removes all class devices belong to a given PCI device.
> + *
> + * @param[in] pci_dev
> + *   Pointer to the PCI device.
> + *
> + * @return
> + *   0 on success, the function cannot fail.
> + */
> +static int
> +mlx5_bus_pci_remove(struct rte_pci_device *dev)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +
> +	/* Remove each class driver in reverse order */

Why use a revese order specifically?

> +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
> +		if (!mlx5_class_enabled(dev->device.devargs, class->dev_class))

You are parsing the devargs each time to check a single class.
If you return the proper bitmap instead, do not parse the devargs within
this loop, do it beforehand and check only that
(devargs_class_bits & class->dev_class).

Same thing in the probe().

> +			continue;
> +
> +		if (class->loaded)
> +			class->remove(dev);
> +	}
> +	return 0;
> +}
> +
> +static int
> +mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr,
> +		     uint64_t iova, size_t len)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +	int ret = -EINVAL;
> +
> +	TAILQ_FOREACH(class, &drv_list, next) {
> +		if (!class->dma_map)
> +			continue;
> +
> +		return class->dma_map(dev, addr, iova, len);

Is there a specific class that could have priority for the DMA?

> +	}
> +	return ret;
> +}
> +
> +static int
> +mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
> +		       uint64_t iova, size_t len)
> +{
> +	struct rte_mlx5_pci_driver *class;
> +	int ret = -EINVAL;
> +
> +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) {
> +		if (!class->dma_unmap)
> +			continue;
> +
> +		return class->dma_unmap(dev, addr, iova, len);

If you have two classes A -> B having dma_map() + dma_unmap(),
you will dma_map() with A then dma_unmap() with B, due to
the _REVERSE() iteration?

Why use reversed iteration at all by the way for dinit? If your ops is
sound any order should be ok.

> +	}
> +	return ret;
> +}
> +
> +static const struct rte_pci_id mlx5_bus_pci_id_map[] = {
> +	{
> +		.vendor_id = 0
> +	}
> +};
> +
> +static struct rte_pci_driver mlx5_bus_driver = {
> +	.driver = {
> +		.name = "mlx5_bus_pci",
> +	},
> +	.id_table = mlx5_bus_pci_id_map,
> +	.probe = mlx5_bus_pci_probe,
> +	.remove = mlx5_bus_pci_remove,
> +	.dma_map = mlx5_bus_pci_dma_map,
> +	.dma_unmap = mlx5_bus_pci_dma_unmap,
> +	.drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV |
> +		     RTE_PCI_DRV_PROBE_AGAIN,
> +};
> +
> +RTE_PMD_REGISTER_PCI(mlx5_bus, mlx5_bus_driver);
> +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map);
> diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> index b0423f99e..7be9a15cd 100644
> --- a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
> @@ -36,6 +36,7 @@ struct rte_mlx5_pci_driver {
>  	pci_dma_unmap_t *dma_unmap;   /**< Class device dma unmap function. */
>  	TAILQ_ENTRY(rte_mlx5_pci_driver) next;
>  	const struct rte_pci_id *id_table; /**< ID table, NULL terminated. */
> +	bool loaded;
>  };
>  
>  /**
> -- 
> 2.25.4
> 

-- 
Gaëtan

Re: [dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver

[Thomas Monjalon]

15/06/2020 23:46, Gaëtan Rivet:
> On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > +		DRV_LOG(ERR, "Invalid mlx5 classes %s. Maybe typo in device"
> > +			" class argument setting?", value);
> 
> Error message should not be cut in half, it makes it difficult to grep.
> If you differentiate between typo in name and invalid combo you could
> directly warn the user about the proper error.
> 
> (You can ignore the warning from checkpatch.sh about the long lines on a
> string if there is one.)

The best is to cut the message after variable placeholder.
Here you can cut at the end of the first sentence "%s. "

Re: [dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Wednesday, June 17, 2020 1:49 PM
> To: Parav Pandit <parav@mellanox.com>; Gaëtan Rivet <grive@u256.net>
> Cc: dev@dpdk.org; ferruh.yigit@intel.com; Ori Kam <orika@mellanox.com>;
> Matan Azrad <matan@mellanox.com>
> Subject: Re: [dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver
> 
> 15/06/2020 23:46, Gaëtan Rivet:
> > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > +		DRV_LOG(ERR, "Invalid mlx5 classes %s. Maybe typo in
> device"
> > > +			" class argument setting?", value);
> >
> > Error message should not be cut in half, it makes it difficult to grep.
> > If you differentiate between typo in name and invalid combo you could
> > directly warn the user about the proper error.
> >
> > (You can ignore the warning from checkpatch.sh about the long lines on
> > a string if there is one.)
> 
> The best is to cut the message after variable placeholder.
> Here you can cut at the end of the first sentence "%s. "
> 
o.k.

Re: [dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

> From: Gaëtan Rivet <grive@u256.net>
> Sent: Tuesday, June 16, 2020 3:17 AM
> 
> On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > Create a mlx5 bus driver framework for invoking drivers of multiple
> > classes who have registered with the mlx5_pci bus driver.
> >
> > Validate user class arguments for supported class combinations.
> >
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > ---
> >  drivers/bus/mlx5_pci/Makefile           |   1 +
> >  drivers/bus/mlx5_pci/meson.build        |   2 +-
> >  drivers/bus/mlx5_pci/mlx5_pci_bus.c     | 253
> ++++++++++++++++++++++++
> >  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h |   1 +
> >  4 files changed, 256 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/bus/mlx5_pci/Makefile
> > b/drivers/bus/mlx5_pci/Makefile index b36916e52..327076fe4 100644
> > --- a/drivers/bus/mlx5_pci/Makefile
> > +++ b/drivers/bus/mlx5_pci/Makefile
> > @@ -15,6 +15,7 @@ CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
> CFLAGS
> > += -I$(RTE_SDK)/drivers/bus/pci  CFLAGS += -Wno-strict-prototypes
> > LDLIBS += -lrte_eal
> > +LDLIBS += -lrte_kvargs
> >  LDLIBS += -lrte_common_mlx5
> >  LDLIBS += -lrte_pci -lrte_bus_pci
> >
> > diff --git a/drivers/bus/mlx5_pci/meson.build
> > b/drivers/bus/mlx5_pci/meson.build
> > index cc4a84e23..5111baa4e 100644
> > --- a/drivers/bus/mlx5_pci/meson.build
> > +++ b/drivers/bus/mlx5_pci/meson.build
> > @@ -1,6 +1,6 @@
> >  # SPDX-License-Identifier: BSD-3-Clause  # Copyright(c) 2020 Mellanox
> > Technologies Ltd
> >
> > -deps += ['pci', 'bus_pci', 'common_mlx5']
> > +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs']
> >  install_headers('rte_bus_mlx5_pci.h')
> >  sources = files('mlx5_pci_bus.c')
> > diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> > b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> > index 66db3c7b0..8108e35ea 100644
> > --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> > +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
> > @@ -3,12 +3,265 @@
> >   */
> >
> >  #include "rte_bus_mlx5_pci.h"
> > +#include <mlx5_common_utils.h>
> >
> >  static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list
> =
> >  				TAILQ_HEAD_INITIALIZER(drv_list);
> >
> > +struct class_map {
> > +	const char *name;
> > +	enum mlx5_class dev_class;
> > +};
> 
> Defining a type here does not seem useful.
> You could return an "enum mlx5_class" from the function is_valid_class()
> further below, instead of a class_map entry. You have the
> MLX5_CLASS_INVALID sentinel value to mark the inexistant mapping.
> 
> Making this type anonymous, you should merge it with the array below:
> 
> static const struct {
> 	const char *name;
> 	enum mlx5_class class; // Remember to change this enum to a
> 			       // fixed width type by the way.
Yes, I acked to changed to define, but I forgot that I use the enum here.
So I am going to keep the enum as code reads more clear with enum.

> } mlx5_classes[] = {
> 	...
> };
> 
> > +
> > +const struct class_map mlx5_classes[] = {
> 
> This is not really a class list, but as the type describes, a mapping between
> names and binary id.
> 
> I think mlx5_class_names would fit better.
o.k.
> 
> > +	{ .name = "vdpa", .dev_class = MLX5_CLASS_VDPA },
> > +	{ .name = "net", .dev_class = MLX5_CLASS_NET }, };
> 
> Globals should be static even if not exposed through header.
Yes.
> 
> > +
> > +static const enum mlx5_class mlx5_valid_class_combo[] = {
> > +	MLX5_CLASS_NET,
> > +	MLX5_CLASS_VDPA,
> 
> How do you describe future combos?
> Should we expect MLX5_CLASS_NET | MLX5_CLASS_REGEX for example?
> 
Correct.

> > +	/* New class combination should be added here */ };
> > +
> > +static const struct class_map *is_valid_class(const char *class_name)
> 
> I don't think the function name conveys its actual use.
> mlx5_class_from_name() for example would align with other DPDK APIs.
Make sense. Will change.
> 
> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < sizeof(mlx5_classes) / sizeof(struct class_map);
> 
> RTE_DIM(mlx5_classes) must be used instead.
> 
> > +	     i++) {
> > +		if (strcmp(class_name, mlx5_classes[i].name) == 0)
> > +			return &mlx5_classes[i];
> > +
> > +	}
> > +	return NULL;
> > +}
> > +
> > +static int is_valid_class_combo(const char *class_names) {
> > +	enum mlx5_class user_classes = 0;
> > +	char *nstr = strdup(class_names);
> 
> Not a fan of assignment with malloc directly at var declaration, you are
> missing some checks here.
> 
Ok. will do assignment explicitly and do the check.

> > +	const struct class_map *entry;
> > +	char *copy = nstr;
> 
> copy is nondescript and dangerous. I'd use nstr_orig instead.
Ok. will change varialble name from copy to nstr_orig.

> 
> > +	int invalid = 0;
> > +	unsigned int i;
> > +	char *found;
> 
> Reading the code below, it seems that the kvlist should only be defined if the
> devargs length is > 0, so class_names here should be defined.
> 
> However you do not handle the OOM case explicitly here, if nstr cannot be
> allocated on strdup().
> 
Will return error on OOM.

> > +
> > +	while (nstr) {
> > +		/* Extract each individual class name */
> > +		found = strsep(&nstr, ":");
> 
> I have not seen the feature test macros (_DEFAULT_SOURCE) in the
> Makefile, it seems required for strsep()?
> 
If its mandatory meson build should have complained?

> > +		if (!found)
> > +			continue;
> > +
> > +		/* Check if its a valid class */
> > +		entry = is_valid_class(found);
> 
> As said earlier, you have no use for the full map entry, you could return the
> mlx5_class type instead, as you have the MLX5_CLASS_INVALID sentinel
> available to mark the !found case.
> 
> > +		if (!entry) {
> > +			invalid = EINVAL;
> 
> Just toggling invalid = 1; here would be better.
> Return EINVAL explicitly if (invalid).
> 
> > +			break;
> > +		}
> > +		user_classes |= entry->dev_class;
> > +	}
> > +	if (copy)
> > +		free(copy);
> > +	if (invalid)
> > +		return invalid;
> 
> You are returning EINVAL here, but -EINVAL below.
> It should be aligned, in DPDK usually error return values are negative.
> positive errno should only be used for rte_errno.
> 
> > +
> > +	/* Verify if user specified valid supported combination */
> > +	for (i = 0;
> > +	     i < sizeof(mlx5_valid_class_combo) / sizeof(enum mlx5_class);
> 
> RTE_DIM() here;
> 
Ack.

> > +	     i++) {
> > +		if (mlx5_valid_class_combo[i] == user_classes)
> 
> return 0; directly?
> 
Yes.

> > +			break;
> > +	}
> > +	/* Not found any valid class combination */
> > +	if (i == sizeof(mlx5_valid_class_combo) / sizeof(enum mlx5_class))
> 
> This would simplify this check, where you can directly return -ENODEV for
> example to differentiate from invalid class name and invalid combo.
> 
> > +		return -EINVAL;
> > +	else
> > +		return 0;
> > +}
> > +
> > +static int
> > +mlx5_bus_opt_handler(__rte_unused const char *key, const char *value,
> > +		       void *opaque)
> > +{
> > +	int *ret = opaque;
> > +
> > +	*ret = is_valid_class_combo(value);
> > +	if (*ret)
> > +		DRV_LOG(ERR, "Invalid mlx5 classes %s. Maybe typo in
> device"
> > +			" class argument setting?", value);
> 
> Error message should not be cut in half, it makes it difficult to grep.
> If you differentiate between typo in name and invalid combo you could
> directly warn the user about the proper error.
> 
Will have it in one line.

> (You can ignore the warning from checkpatch.sh about the long lines on a
> string if there is one.)
> 
> > +	return 0;
> > +}
> > +
> > +static int
> > +mlx5_bus_options_valid(const struct rte_devargs *devargs) {
> > +	struct rte_kvargs *kvlist;
> > +	const char *key = MLX5_CLASS_ARG_NAME;
> > +	int ret = 0;
> > +
> > +	if (devargs == NULL)
> > +		return 0;
> > +	kvlist = rte_kvargs_parse(devargs->args, NULL);
> > +	if (kvlist == NULL)
> > +		return 0;
> > +	if (rte_kvargs_count(kvlist, key))
> > +		rte_kvargs_process(kvlist, key, mlx5_bus_opt_handler,
> &ret);
> > +	rte_kvargs_free(kvlist);
> > +	return ret;
> > +}
> > +
> >  void
> >  rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver)  {
> >  	TAILQ_INSERT_TAIL(&drv_list, driver, next);  }
> > +
> > +static bool
> > +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv,
> > +		 const struct rte_pci_device *pci_dev) {
> > +	const struct rte_pci_id *id_table;
> > +
> > +	for (id_table = drv->id_table; id_table->vendor_id != 0; id_table++) {
> > +		/* check if device's ids match the class driver's ones */
> > +		if (id_table->vendor_id != pci_dev->id.vendor_id &&
> > +				id_table->vendor_id != PCI_ANY_ID)
> > +			continue;
> > +		if (id_table->device_id != pci_dev->id.device_id &&
> > +				id_table->device_id != PCI_ANY_ID)
> > +			continue;
> > +		if (id_table->subsystem_vendor_id !=
> > +		    pci_dev->id.subsystem_vendor_id &&
> > +		    id_table->subsystem_vendor_id != PCI_ANY_ID)
> > +			continue;
> > +		if (id_table->subsystem_device_id !=
> > +		    pci_dev->id.subsystem_device_id &&
> > +		    id_table->subsystem_device_id != PCI_ANY_ID)
> > +			continue;
> > +		if (id_table->class_id != pci_dev->id.class_id &&
> > +				id_table->class_id != RTE_CLASS_ANY_ID)
> > +			continue;
> > +
> > +		return true;
> > +	}
> > +	return false;
> > +}
> > +
> > +/**
> > + * DPDK callback to register to probe multiple PCI class devices.
> > + *
> > + * @param[in] pci_drv
> > + *   PCI driver structure.
> > + * @param[in] dev
> > + *   PCI device information.
> > + *
> > + * @return
> > + *   0 on success, 1 to skip this driver, a negative errno value otherwise
> > + *   and rte_errno is set.
> > + */
> > +static int
> > +mlx5_bus_pci_probe(struct rte_pci_driver *drv __rte_unused,
> > +		     struct rte_pci_device *dev)
> > +{
> > +	struct rte_mlx5_pci_driver *class;
> > +	int ret = 0;
> > +
> > +	ret = mlx5_bus_options_valid(dev->device.devargs);
> > +	if (ret)
> > +		return ret;
> > +
> > +	TAILQ_FOREACH(class, &drv_list, next) {
> > +		if (!mlx5_bus_match(class, dev))
> > +			continue;
> > +
> > +		if (!mlx5_class_enabled(dev->device.devargs, class-
> >dev_class))
> > +			continue;
> > +
> > +		ret = class->probe(drv, dev);
> > +		if (!ret)
> > +			class->loaded = true;
> > +	}
> > +	return 0;
> > +}
> > +
> > +/**
> > + * DPDK callback to remove one or more class devices for a PCI device.
> > + *
> > + * This function removes all class devices belong to a given PCI device.
> > + *
> > + * @param[in] pci_dev
> > + *   Pointer to the PCI device.
> > + *
> > + * @return
> > + *   0 on success, the function cannot fail.
> > + */
> > +static int
> > +mlx5_bus_pci_remove(struct rte_pci_device *dev) {
> > +	struct rte_mlx5_pci_driver *class;
> > +
> > +	/* Remove each class driver in reverse order */
> 
> Why use a revese order specifically?
> 
> > +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head,
> next) {
> > +		if (!mlx5_class_enabled(dev->device.devargs, class-
> >dev_class))
> 
> You are parsing the devargs each time to check a single class.
> If you return the proper bitmap instead, do not parse the devargs within this
> loop, do it beforehand and check only that (devargs_class_bits & class-
> >dev_class).
> 
> Same thing in the probe().
> 
> > +			continue;
> > +
> > +		if (class->loaded)
> > +			class->remove(dev);
> > +	}
> > +	return 0;
> > +}
> > +
> > +static int
> > +mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr,
> > +		     uint64_t iova, size_t len)
> > +{
> > +	struct rte_mlx5_pci_driver *class;
> > +	int ret = -EINVAL;
> > +
> > +	TAILQ_FOREACH(class, &drv_list, next) {
> > +		if (!class->dma_map)
> > +			continue;
> > +
> > +		return class->dma_map(dev, addr, iova, len);
> 
> Is there a specific class that could have priority for the DMA?
> 
No.

> > +	}
> > +	return ret;
> > +}
> > +
> > +static int
> > +mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
> > +		       uint64_t iova, size_t len)
> > +{
> > +	struct rte_mlx5_pci_driver *class;
> > +	int ret = -EINVAL;
> > +
> > +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head,
> next) {
> > +		if (!class->dma_unmap)
> > +			continue;
> > +
> > +		return class->dma_unmap(dev, addr, iova, len);
> 
> If you have two classes A -> B having dma_map() + dma_unmap(), you will
> dma_map() with A then dma_unmap() with B, due to the _REVERSE()
> iteration?
> 
There isn't plan for two drivers to do so.
If two classes do that its source of an error.
Will enhance the bus when that need arise.

> Why use reversed iteration at all by the way for dinit? If your ops is sound
> any order should be ok.
> 
Because deinit must be always reverse of init() code regardless.

> > +	}
> > +	return ret;
> > +}

Re: [dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver

[Gaëtan Rivet]

On 18/06/20 10:03 +0000, Parav Pandit wrote:
> 
> > From: Gaëtan Rivet <grive@u256.net>
> > Sent: Tuesday, June 16, 2020 3:17 AM
> > 
> > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > Create a mlx5 bus driver framework for invoking drivers of multiple
> > > classes who have registered with the mlx5_pci bus driver.
> > >
> > > Validate user class arguments for supported class combinations.
> > >
> > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > ---
> > >  drivers/bus/mlx5_pci/Makefile           |   1 +
> > >  drivers/bus/mlx5_pci/meson.build        |   2 +-
> > >  drivers/bus/mlx5_pci/mlx5_pci_bus.c     | 253
> > ++++++++++++++++++++++++
> > >  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h |   1 +
> > >  4 files changed, 256 insertions(+), 1 deletion(-)
> > >

[...]

> > > +
> > > +	while (nstr) {
> > > +		/* Extract each individual class name */
> > > +		found = strsep(&nstr, ":");
> > 
> > I have not seen the feature test macros (_DEFAULT_SOURCE) in the
> > Makefile, it seems required for strsep()?
> > 
> If its mandatory meson build should have complained?
> 

Invoking the compiler without specific standard conformance will work.
If someone adds for example -std=c11 however then _DEFAULT_SOURCE
becomes necessary.

It all depends on the range of compiler versions targeted by this code.
I don't know the full coverage, but I see -std=c11 + -D_DEFAULT_SOURCE
in most mlx5 code, which is why I'm asking for a double check here.


[...]

> > > +			continue;
> > > +
> > > +		if (class->loaded)
> > > +			class->remove(dev);
> > > +	}
> > > +	return 0;
> > > +}
> > > +
> > > +static int
> > > +mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr,
> > > +		     uint64_t iova, size_t len)
> > > +{
> > > +	struct rte_mlx5_pci_driver *class;
> > > +	int ret = -EINVAL;
> > > +
> > > +	TAILQ_FOREACH(class, &drv_list, next) {
> > > +		if (!class->dma_map)
> > > +			continue;
> > > +
> > > +		return class->dma_map(dev, addr, iova, len);
> > 
> > Is there a specific class that could have priority for the DMA?
> > 
> No.
> 

The code being written this way seems to point to multiple classes being
able to have DMA ops. If that's not the case, you can add a sanity check
to enforce than only the right classes have DMA ops defined.

> > > +	}
> > > +	return ret;
> > > +}
> > > +
> > > +static int
> > > +mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
> > > +		       uint64_t iova, size_t len)
> > > +{
> > > +	struct rte_mlx5_pci_driver *class;
> > > +	int ret = -EINVAL;
> > > +
> > > +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head,
> > next) {
> > > +		if (!class->dma_unmap)
> > > +			continue;
> > > +
> > > +		return class->dma_unmap(dev, addr, iova, len);
> > 
> > If you have two classes A -> B having dma_map() + dma_unmap(), you will
> > dma_map() with A then dma_unmap() with B, due to the _REVERSE()
> > iteration?
> > 
> There isn't plan for two drivers to do so.
> If two classes do that its source of an error.
> Will enhance the bus when that need arise.
> 

You have well-defined edge-cases, but they are not apparent reading
the code. Such error could be warned about and / or documented.

> > Why use reversed iteration at all by the way for dinit? If your ops is sound
> > any order should be ok.
> > 
> Because deinit must be always reverse of init() code regardless.
> 

This is a strong statement :)

If this is a requirement for driver inter-dependencies to be properly
implemented, this should be documented as such. Probably also explained
in the high-level design documentation in the header exposing this
driver API.

Best,
-- 
Gaëtan

Re: [dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver

[Parav Pandit]

> From: Gaëtan Rivet <grive@u256.net>
> Sent: Thursday, June 18, 2020 8:05 PM
> To: Parav Pandit <parav@mellanox.com>
> Cc: dev@dpdk.org; ferruh.yigit@intel.com; Ori Kam <orika@mellanox.com>;
> Matan Azrad <matan@mellanox.com>
> Subject: Re: [dpdk-dev] [RFC PATCH 5/6] bus/mlx5_pci: register a PCI driver
> 
> On 18/06/20 10:03 +0000, Parav Pandit wrote:
> >
> > > From: Gaëtan Rivet <grive@u256.net>
> > > Sent: Tuesday, June 16, 2020 3:17 AM
> > >
> > > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > > Create a mlx5 bus driver framework for invoking drivers of
> > > > multiple classes who have registered with the mlx5_pci bus driver.
> > > >
> > > > Validate user class arguments for supported class combinations.
> > > >
> > > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > > ---
> > > >  drivers/bus/mlx5_pci/Makefile           |   1 +
> > > >  drivers/bus/mlx5_pci/meson.build        |   2 +-
> > > >  drivers/bus/mlx5_pci/mlx5_pci_bus.c     | 253
> > > ++++++++++++++++++++++++
> > > >  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h |   1 +
> > > >  4 files changed, 256 insertions(+), 1 deletion(-)
> > > >
> 
> [...]
> 
> > > > +
> > > > +	while (nstr) {
> > > > +		/* Extract each individual class name */
> > > > +		found = strsep(&nstr, ":");
> > >
> > > I have not seen the feature test macros (_DEFAULT_SOURCE) in the
> > > Makefile, it seems required for strsep()?
> > >
> > If its mandatory meson build should have complained?
> >
> 
> Invoking the compiler without specific standard conformance will work.
> If someone adds for example -std=c11 however then _DEFAULT_SOURCE
> becomes necessary.
> 
> It all depends on the range of compiler versions targeted by this code.
> I don't know the full coverage, but I see -std=c11 + -D_DEFAULT_SOURCE in
> most mlx5 code, which is why I'm asking for a double check here.
> 
> 
> [...]
> 
> > > > +			continue;
> > > > +
> > > > +		if (class->loaded)
> > > > +			class->remove(dev);
> > > > +	}
> > > > +	return 0;
> > > > +}
> > > > +
> > > > +static int
> > > > +mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr,
> > > > +		     uint64_t iova, size_t len) {
> > > > +	struct rte_mlx5_pci_driver *class;
> > > > +	int ret = -EINVAL;
> > > > +
> > > > +	TAILQ_FOREACH(class, &drv_list, next) {
> > > > +		if (!class->dma_map)
> > > > +			continue;
> > > > +
> > > > +		return class->dma_map(dev, addr, iova, len);
> > >
> > > Is there a specific class that could have priority for the DMA?
> > >
> > No.
> >
> 
> The code being written this way seems to point to multiple classes being able
> to have DMA ops. If that's not the case, you can add a sanity check to enforce
> than only the right classes have DMA ops defined.
> 
Yes. I will add it. good point.

> > > > +	}
> > > > +	return ret;
> > > > +}
> > > > +
> > > > +static int
> > > > +mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
> > > > +		       uint64_t iova, size_t len) {
> > > > +	struct rte_mlx5_pci_driver *class;
> > > > +	int ret = -EINVAL;
> > > > +
> > > > +	TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head,
> > > next) {
> > > > +		if (!class->dma_unmap)
> > > > +			continue;
> > > > +
> > > > +		return class->dma_unmap(dev, addr, iova, len);
> > >
> > > If you have two classes A -> B having dma_map() + dma_unmap(), you
> > > will
> > > dma_map() with A then dma_unmap() with B, due to the _REVERSE()
> > > iteration?
> > >
> > There isn't plan for two drivers to do so.
> > If two classes do that its source of an error.
> > Will enhance the bus when that need arise.
> >
> 
> You have well-defined edge-cases, but they are not apparent reading the
> code. Such error could be warned about and / or documented.
> 
> > > Why use reversed iteration at all by the way for dinit? If your ops
> > > is sound any order should be ok.
> > >
> > Because deinit must be always reverse of init() code regardless.
> >
> 
> This is a strong statement :)
> 
:-)
Deinit not following reverse is almost an invitation to bugs.
Given there are two different drivers, there shouldn't be much dependency that way.
But it is always a good practice to follow de-init as reverse sequence.
It eliminates plenty of null and other validation checks all over.

> If this is a requirement for driver inter-dependencies to be properly
> implemented, this should be documented as such. Probably also explained in
> the high-level design documentation in the header exposing this driver API.
> 
At the moment we don't have such dependency and will continue to steer the design that way.
Only dependency to have is between bus driver (rte_bus_pci_mlx5) and its upper layer class drivers.

> Best,
> --
> Gaëtan

[dpdk-dev] [RFC PATCH 6/6] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Parav Pandit]

Enable class driver to match with the mlx5 pci devices.
Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
driver.

Signed-off-by: Parav Pandit <parav@mellanox.com>
---
 drivers/bus/Makefile                |  3 ++
 drivers/bus/mlx5_pci/mlx5_pci_bus.c | 60 +++++++++++++++++++++++++++++
 drivers/net/mlx5/Makefile           |  3 +-
 drivers/net/mlx5/linux/mlx5_os.c    |  1 -
 drivers/net/mlx5/linux/mlx5_os.h    |  3 --
 drivers/net/mlx5/meson.build        |  2 +-
 drivers/net/mlx5/mlx5.c             |  7 ++--
 drivers/net/mlx5/mlx5.h             |  1 -
 drivers/vdpa/mlx5/Makefile          |  3 +-
 drivers/vdpa/mlx5/meson.build       |  2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c       | 10 ++---
 mk/rte.app.mk                       |  1 +
 12 files changed, 79 insertions(+), 17 deletions(-)

diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile
index cea3b55e6..3840ac51c 100644
--- a/drivers/bus/Makefile
+++ b/drivers/bus/Makefile
@@ -8,6 +8,9 @@ ifeq ($(CONFIG_RTE_EAL_VFIO),y)
 DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga
+ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
+DIRS-y += mlx5_pci
+endif
 DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci
 DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev
 DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus
diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
index 8108e35ea..ca001f3c6 100644
--- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c
+++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c
@@ -245,6 +245,66 @@ mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr,
 }
 
 static const struct rte_pci_id mlx5_bus_pci_id_map[] = {
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX4)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX4VF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX4LX)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5VF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5EX)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+			       PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6VF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6DX)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6DXVF)
+	},
+	{
+		RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX,
+				PCI_DEVICE_ID_MELLANOX_CONNECTX6DXBF)
+	},
 	{
 		.vendor_id = 0
 	}
diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index 41ab73e5c..ccb38245d 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -43,16 +43,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
+LDLIBS += -lrte_bus_mlx5_pci
 LDLIBS += -lm
 LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring
 LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs
-LDLIBS += -lrte_bus_pci
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 06772b7ae..61e3b8a80 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1391,7 +1391,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
 			strerror(rte_errno));
 		return -rte_errno;
 	}
-	MLX5_ASSERT(pci_drv == &mlx5_driver);
 	errno = 0;
 	ibv_list = mlx5_glue->get_device_list(&ret);
 	if (!ibv_list) {
diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
index f310f1773..70972244f 100644
--- a/drivers/net/mlx5/linux/mlx5_os.h
+++ b/drivers/net/mlx5/linux/mlx5_os.h
@@ -12,7 +12,4 @@ enum {
 	DEV_SYSFS_PATH_MAX = IBV_SYSFS_PATH_MAX
 };
 
-#define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
-			RTE_PCI_DRV_INTR_RMV | \
-			RTE_PCI_DRV_PROBE_AGAIN)
 #endif /* RTE_PMD_MLX5_OS_H_ */
diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index e71b2c515..fd9f41d6e 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -8,7 +8,7 @@ if not (is_linux or is_windows)
 	subdir_done()
 endif
 
-deps += ['hash', 'common_mlx5']
+deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
 sources = files(
 	'mlx5.c',
 	'mlx5_ethdev.c',
diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
index 7c5e23d9f..6b250771f 100644
--- a/drivers/net/mlx5/mlx5.c
+++ b/drivers/net/mlx5/mlx5.c
@@ -34,6 +34,7 @@
 #include <rte_spinlock.h>
 #include <rte_string_fns.h>
 #include <rte_alarm.h>
+#include <rte_bus_mlx5_pci.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
@@ -2106,16 +2107,16 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
 	}
 };
 
-struct rte_pci_driver mlx5_driver = {
+static struct rte_mlx5_pci_driver mlx5_driver = {
 	.driver = {
 		.name = MLX5_DRIVER_NAME
 	},
+	.dev_class = MLX5_CLASS_NET,
 	.id_table = mlx5_pci_id_map,
 	.probe = mlx5_os_pci_probe,
 	.remove = mlx5_pci_remove,
 	.dma_map = mlx5_dma_map,
 	.dma_unmap = mlx5_dma_unmap,
-	.drv_flags = PCI_DRV_FLAGS,
 };
 
 /**
@@ -2133,7 +2134,7 @@ RTE_INIT(rte_mlx5_pmd_init)
 	mlx5_set_cksum_table();
 	mlx5_set_swp_types_table();
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_driver);
+		rte_mlx5_pci_driver_register(&mlx5_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
index 8c4b234e5..b357543dd 100644
--- a/drivers/net/mlx5/mlx5.h
+++ b/drivers/net/mlx5/mlx5.h
@@ -129,7 +129,6 @@ struct mlx5_local_data {
 };
 
 extern struct mlx5_shared_data *mlx5_shared_data;
-extern struct rte_pci_driver mlx5_driver;
 
 /* Dev ops structs */
 extern const struct eth_dev_ops mlx5_dev_sec_ops;
diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
index 91c89d604..a2231f021 100644
--- a/drivers/vdpa/mlx5/Makefile
+++ b/drivers/vdpa/mlx5/Makefile
@@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
 CFLAGS += -I$(RTE_SDK)/lib/librte_sched
 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
+CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
 CFLAGS += -D_BSD_SOURCE
 CFLAGS += -D_DEFAULT_SOURCE
 CFLAGS += -D_XOPEN_SOURCE=600
 CFLAGS += $(WERROR_FLAGS)
 CFLAGS += -Wno-strict-prototypes
 LDLIBS += -lrte_common_mlx5
-LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
+LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched
 
 # A few warnings cannot be avoided in external headers.
 CFLAGS += -Wno-error=cast-qual
diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
index 2963aad71..f4175c34e 100644
--- a/drivers/vdpa/mlx5/meson.build
+++ b/drivers/vdpa/mlx5/meson.build
@@ -8,7 +8,7 @@ if not is_linux
 endif
 
 fmt_name = 'mlx5_vdpa'
-deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
+deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched']
 sources = files(
 	'mlx5_vdpa.c',
 	'mlx5_vdpa_mem.c',
diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c
index 96776b64e..c85283f65 100644
--- a/drivers/vdpa/mlx5/mlx5_vdpa.c
+++ b/drivers/vdpa/mlx5/mlx5_vdpa.c
@@ -6,7 +6,7 @@
 #include <rte_malloc.h>
 #include <rte_log.h>
 #include <rte_errno.h>
-#include <rte_bus_pci.h>
+#include <rte_bus_mlx5_pci.h>
 #include <rte_pci.h>
 
 #include <mlx5_glue.h>
@@ -428,7 +428,7 @@ mlx5_vdpa_roce_disable(struct rte_pci_addr *addr, struct ibv_device **ibv)
 }
 
 /**
- * DPDK callback to register a PCI device.
+ * DPDK callback to register a mlx5 PCI bus device.
  *
  * This function spawns vdpa device out of a given PCI device.
  *
@@ -598,14 +598,14 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = {
 	}
 };
 
-static struct rte_pci_driver mlx5_vdpa_driver = {
+static struct rte_mlx5_pci_driver mlx5_vdpa_driver = {
+	.dev_class = MLX5_CLASS_VDPA,
 	.driver = {
 		.name = "mlx5_vdpa",
 	},
 	.id_table = mlx5_vdpa_pci_id_map,
 	.probe = mlx5_vdpa_pci_probe,
 	.remove = mlx5_vdpa_pci_remove,
-	.drv_flags = 0,
 };
 
 /**
@@ -618,7 +618,7 @@ RTE_INIT(rte_mlx5_vdpa_init)
 	if (mlx5_vdpa_logtype >= 0)
 		rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE);
 	if (mlx5_glue)
-		rte_pci_register(&mlx5_vdpa_driver);
+		rte_mlx5_pci_driver_register(&mlx5_vdpa_driver);
 }
 
 RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__);
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 0ce8cf541..d7c1d90e4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF)      += -lrte_pmd_memif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD)       += -lrte_pmd_mlx4
 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y)
 _LDLIBS-y                                   += -lrte_common_mlx5
+_LDLIBS-y                                   += -lrte_bus_mlx5_pci
 endif
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD)       += -lrte_pmd_mlx5
 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)  += -lrte_pmd_mlx5_vdpa
-- 
2.25.4

Re: [dpdk-dev] [RFC PATCH 6/6] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Gaëtan Rivet]

On 10/06/20 17:17 +0000, Parav Pandit wrote:
> Enable class driver to match with the mlx5 pci devices.
> Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
> driver.
> 
> Signed-off-by: Parav Pandit <parav@mellanox.com>
> ---
>  drivers/bus/Makefile                |  3 ++
>  drivers/bus/mlx5_pci/mlx5_pci_bus.c | 60 +++++++++++++++++++++++++++++
>  drivers/net/mlx5/Makefile           |  3 +-
>  drivers/net/mlx5/linux/mlx5_os.c    |  1 -
>  drivers/net/mlx5/linux/mlx5_os.h    |  3 --
>  drivers/net/mlx5/meson.build        |  2 +-
>  drivers/net/mlx5/mlx5.c             |  7 ++--
>  drivers/net/mlx5/mlx5.h             |  1 -
>  drivers/vdpa/mlx5/Makefile          |  3 +-
>  drivers/vdpa/mlx5/meson.build       |  2 +-
>  drivers/vdpa/mlx5/mlx5_vdpa.c       | 10 ++---
>  mk/rte.app.mk                       |  1 +
>  12 files changed, 79 insertions(+), 17 deletions(-)
> 

[...]

> diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h
> index f310f1773..70972244f 100644
> --- a/drivers/net/mlx5/linux/mlx5_os.h
> +++ b/drivers/net/mlx5/linux/mlx5_os.h
> @@ -12,7 +12,4 @@ enum {
>  	DEV_SYSFS_PATH_MAX = IBV_SYSFS_PATH_MAX
>  };
>  
> -#define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
> -			RTE_PCI_DRV_INTR_RMV | \
> -			RTE_PCI_DRV_PROBE_AGAIN)
>  #endif /* RTE_PMD_MLX5_OS_H_ */
> diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
> index e71b2c515..fd9f41d6e 100644
> --- a/drivers/net/mlx5/meson.build
> +++ b/drivers/net/mlx5/meson.build
> @@ -8,7 +8,7 @@ if not (is_linux or is_windows)
>  	subdir_done()
>  endif
>  
> -deps += ['hash', 'common_mlx5']
> +deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
>  sources = files(
>  	'mlx5.c',
>  	'mlx5_ethdev.c',
> diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c
> index 7c5e23d9f..6b250771f 100644
> --- a/drivers/net/mlx5/mlx5.c
> +++ b/drivers/net/mlx5/mlx5.c
> @@ -34,6 +34,7 @@
>  #include <rte_spinlock.h>
>  #include <rte_string_fns.h>
>  #include <rte_alarm.h>
> +#include <rte_bus_mlx5_pci.h>
>  
>  #include <mlx5_glue.h>
>  #include <mlx5_devx_cmds.h>
> @@ -2106,16 +2107,16 @@ static const struct rte_pci_id mlx5_pci_id_map[] = {
>  	}
>  };
>  
> -struct rte_pci_driver mlx5_driver = {
> +static struct rte_mlx5_pci_driver mlx5_driver = {
>  	.driver = {
>  		.name = MLX5_DRIVER_NAME
>  	},
> +	.dev_class = MLX5_CLASS_NET,
>  	.id_table = mlx5_pci_id_map,
>  	.probe = mlx5_os_pci_probe,
>  	.remove = mlx5_pci_remove,
>  	.dma_map = mlx5_dma_map,
>  	.dma_unmap = mlx5_dma_unmap,
> -	.drv_flags = PCI_DRV_FLAGS,

You mask the drv_flags, do you plan on merging those flags
in the demuxing drivers above? It seems LSC | RMV | PROBE_AGAIN
is not used afterward, seems missing.

Using an rte_pci_driver instead of an rte_driver as core object
for you driver would allow managing this by the way.

>  };
>  
>  /**
> @@ -2133,7 +2134,7 @@ RTE_INIT(rte_mlx5_pmd_init)
>  	mlx5_set_cksum_table();
>  	mlx5_set_swp_types_table();
>  	if (mlx5_glue)
> -		rte_pci_register(&mlx5_driver);
> +		rte_mlx5_pci_driver_register(&mlx5_driver);
>  }
>  
>  RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__);
> diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h
> index 8c4b234e5..b357543dd 100644
> --- a/drivers/net/mlx5/mlx5.h
> +++ b/drivers/net/mlx5/mlx5.h
> @@ -129,7 +129,6 @@ struct mlx5_local_data {
>  };
>  
>  extern struct mlx5_shared_data *mlx5_shared_data;
> -extern struct rte_pci_driver mlx5_driver;
>  
>  /* Dev ops structs */
>  extern const struct eth_dev_ops mlx5_dev_sec_ops;
> diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile
> index 91c89d604..a2231f021 100644
> --- a/drivers/vdpa/mlx5/Makefile
> +++ b/drivers/vdpa/mlx5/Makefile
> @@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux
>  CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
>  CFLAGS += -I$(RTE_SDK)/lib/librte_sched
>  CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5
> +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
>  CFLAGS += -D_BSD_SOURCE
>  CFLAGS += -D_DEFAULT_SOURCE
>  CFLAGS += -D_XOPEN_SOURCE=600
>  CFLAGS += $(WERROR_FLAGS)
>  CFLAGS += -Wno-strict-prototypes
>  LDLIBS += -lrte_common_mlx5
> -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched
> +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched
>  
>  # A few warnings cannot be avoided in external headers.
>  CFLAGS += -Wno-error=cast-qual
> diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build
> index 2963aad71..f4175c34e 100644
> --- a/drivers/vdpa/mlx5/meson.build
> +++ b/drivers/vdpa/mlx5/meson.build
> @@ -8,7 +8,7 @@ if not is_linux
>  endif
>  
>  fmt_name = 'mlx5_vdpa'
> -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched']
> +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched']

After reading it more, I think bus_pci_mlx5 would work better.
From the more general to the specific (bus >> pci >> mlx5 demux).


Regards,
-- 
Gaëtan

Re: [dpdk-dev] [RFC PATCH 6/6] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Parav Pandit]

> From: Gaëtan Rivet <grive@u256.net>
> Sent: Tuesday, June 16, 2020 3:26 AM
> 
> On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > Enable class driver to match with the mlx5 pci devices.
> > Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
> > driver.
> >
> > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > ---
> >  drivers/bus/Makefile                |  3 ++
> >  drivers/bus/mlx5_pci/mlx5_pci_bus.c | 60
> +++++++++++++++++++++++++++++
> >  drivers/net/mlx5/Makefile           |  3 +-
> >  drivers/net/mlx5/linux/mlx5_os.c    |  1 -
> >  drivers/net/mlx5/linux/mlx5_os.h    |  3 --
> >  drivers/net/mlx5/meson.build        |  2 +-
> >  drivers/net/mlx5/mlx5.c             |  7 ++--
> >  drivers/net/mlx5/mlx5.h             |  1 -
> >  drivers/vdpa/mlx5/Makefile          |  3 +-
> >  drivers/vdpa/mlx5/meson.build       |  2 +-
> >  drivers/vdpa/mlx5/mlx5_vdpa.c       | 10 ++---
> >  mk/rte.app.mk                       |  1 +
> >  12 files changed, 79 insertions(+), 17 deletions(-)
> >
> 
> [...]
> 
> > diff --git a/drivers/net/mlx5/linux/mlx5_os.h
> > b/drivers/net/mlx5/linux/mlx5_os.h
> > index f310f1773..70972244f 100644
> > --- a/drivers/net/mlx5/linux/mlx5_os.h
> > +++ b/drivers/net/mlx5/linux/mlx5_os.h
> > @@ -12,7 +12,4 @@ enum {
> >  	DEV_SYSFS_PATH_MAX = IBV_SYSFS_PATH_MAX  };
> >
> > -#define PCI_DRV_FLAGS  (RTE_PCI_DRV_INTR_LSC | \
> > -			RTE_PCI_DRV_INTR_RMV | \
> > -			RTE_PCI_DRV_PROBE_AGAIN)
> >  #endif /* RTE_PMD_MLX5_OS_H_ */
> > diff --git a/drivers/net/mlx5/meson.build
> > b/drivers/net/mlx5/meson.build index e71b2c515..fd9f41d6e 100644
> > --- a/drivers/net/mlx5/meson.build
> > +++ b/drivers/net/mlx5/meson.build
> > @@ -8,7 +8,7 @@ if not (is_linux or is_windows)
> >  	subdir_done()
> >  endif
> >
> > -deps += ['hash', 'common_mlx5']
> > +deps += ['hash', 'common_mlx5', 'bus_mlx5_pci']
> >  sources = files(
> >  	'mlx5.c',
> >  	'mlx5_ethdev.c',
> > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index
> > 7c5e23d9f..6b250771f 100644
> > --- a/drivers/net/mlx5/mlx5.c
> > +++ b/drivers/net/mlx5/mlx5.c
> > @@ -34,6 +34,7 @@
> >  #include <rte_spinlock.h>
> >  #include <rte_string_fns.h>
> >  #include <rte_alarm.h>
> > +#include <rte_bus_mlx5_pci.h>
> >
> >  #include <mlx5_glue.h>
> >  #include <mlx5_devx_cmds.h>
> > @@ -2106,16 +2107,16 @@ static const struct rte_pci_id
> mlx5_pci_id_map[] = {
> >  	}
> >  };
> >
> > -struct rte_pci_driver mlx5_driver = {
> > +static struct rte_mlx5_pci_driver mlx5_driver = {
> >  	.driver = {
> >  		.name = MLX5_DRIVER_NAME
> >  	},
> > +	.dev_class = MLX5_CLASS_NET,
> >  	.id_table = mlx5_pci_id_map,
> >  	.probe = mlx5_os_pci_probe,
> >  	.remove = mlx5_pci_remove,
> >  	.dma_map = mlx5_dma_map,
> >  	.dma_unmap = mlx5_dma_unmap,
> > -	.drv_flags = PCI_DRV_FLAGS,
> 
> You mask the drv_flags, do you plan on merging those flags in the demuxing
> drivers above? It seems LSC | RMV | PROBE_AGAIN is not used afterward,
> seems missing.
> 
> Using an rte_pci_driver instead of an rte_driver as core object for you driver
> would allow managing this by the way.
Yes. but how would mlx5_pci bus will process the drv_flags passed by the class drivers such a vdpa/net?

> 
> >  };
> >
> >  /**
> > @@ -2133,7 +2134,7 @@ RTE_INIT(rte_mlx5_pmd_init)
> >  	mlx5_set_cksum_table();
> >  	mlx5_set_swp_types_table();
> >  	if (mlx5_glue)
> > -		rte_pci_register(&mlx5_driver);
> > +		rte_mlx5_pci_driver_register(&mlx5_driver);
> >  }
> >
> >  RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git
> > a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index
> > 8c4b234e5..b357543dd 100644
> > --- a/drivers/net/mlx5/mlx5.h
> > +++ b/drivers/net/mlx5/mlx5.h
> > @@ -129,7 +129,6 @@ struct mlx5_local_data {  };
> >
> >  extern struct mlx5_shared_data *mlx5_shared_data; -extern struct
> > rte_pci_driver mlx5_driver;
> >
> >  /* Dev ops structs */
> >  extern const struct eth_dev_ops mlx5_dev_sec_ops; diff --git
> > a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index
> > 91c89d604..a2231f021 100644
> > --- a/drivers/vdpa/mlx5/Makefile
> > +++ b/drivers/vdpa/mlx5/Makefile
> > @@ -24,13 +24,14 @@ CFLAGS += -
> I$(RTE_SDK)/drivers/common/mlx5/linux
> >  CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa
> >  CFLAGS += -I$(RTE_SDK)/lib/librte_sched  CFLAGS +=
> > -I$(BUILDDIR)/drivers/common/mlx5
> > +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci
> >  CFLAGS += -D_BSD_SOURCE
> >  CFLAGS += -D_DEFAULT_SOURCE
> >  CFLAGS += -D_XOPEN_SOURCE=600
> >  CFLAGS += $(WERROR_FLAGS)
> >  CFLAGS += -Wno-strict-prototypes
> >  LDLIBS += -lrte_common_mlx5
> > -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci
> > -lrte_sched
> > +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci
> > +-lrte_bus_mlx5_pci -lrte_sched
> >
> >  # A few warnings cannot be avoided in external headers.
> >  CFLAGS += -Wno-error=cast-qual
> > diff --git a/drivers/vdpa/mlx5/meson.build
> > b/drivers/vdpa/mlx5/meson.build index 2963aad71..f4175c34e 100644
> > --- a/drivers/vdpa/mlx5/meson.build
> > +++ b/drivers/vdpa/mlx5/meson.build
> > @@ -8,7 +8,7 @@ if not is_linux
> >  endif
> >
> >  fmt_name = 'mlx5_vdpa'
> > -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal',
> > 'sched']
> > +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci',
> > +'eal', 'sched']
> 
> After reading it more, I think bus_pci_mlx5 would work better.
> From the more general to the specific (bus >> pci >> mlx5 demux).
> 
Ok. that is fine too.
Will change.

Re: [dpdk-dev] [RFC PATCH 6/6] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver

[Gaëtan Rivet]

On 18/06/20 10:06 +0000, Parav Pandit wrote:
> 
> 
> > From: Gaëtan Rivet <grive@u256.net>
> > Sent: Tuesday, June 16, 2020 3:26 AM
> > 
> > On 10/06/20 17:17 +0000, Parav Pandit wrote:
> > > Enable class driver to match with the mlx5 pci devices.
> > > Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class
> > > driver.
> > >
> > > Signed-off-by: Parav Pandit <parav@mellanox.com>
> > > ---
> > >  drivers/bus/Makefile                |  3 ++
> > >  drivers/bus/mlx5_pci/mlx5_pci_bus.c | 60
> > +++++++++++++++++++++++++++++
> > >  drivers/net/mlx5/Makefile           |  3 +-
> > >  drivers/net/mlx5/linux/mlx5_os.c    |  1 -
> > >  drivers/net/mlx5/linux/mlx5_os.h    |  3 --
> > >  drivers/net/mlx5/meson.build        |  2 +-
> > >  drivers/net/mlx5/mlx5.c             |  7 ++--
> > >  drivers/net/mlx5/mlx5.h             |  1 -
> > >  drivers/vdpa/mlx5/Makefile          |  3 +-
> > >  drivers/vdpa/mlx5/meson.build       |  2 +-
> > >  drivers/vdpa/mlx5/mlx5_vdpa.c       | 10 ++---
> > >  mk/rte.app.mk                       |  1 +
> > >  12 files changed, 79 insertions(+), 17 deletions(-)
> > >
> > 

[...]

> > > -struct rte_pci_driver mlx5_driver = {
> > > +static struct rte_mlx5_pci_driver mlx5_driver = {
> > >  	.driver = {
> > >  		.name = MLX5_DRIVER_NAME
> > >  	},
> > > +	.dev_class = MLX5_CLASS_NET,
> > >  	.id_table = mlx5_pci_id_map,
> > >  	.probe = mlx5_os_pci_probe,
> > >  	.remove = mlx5_pci_remove,
> > >  	.dma_map = mlx5_dma_map,
> > >  	.dma_unmap = mlx5_dma_unmap,
> > > -	.drv_flags = PCI_DRV_FLAGS,
> > 
> > You mask the drv_flags, do you plan on merging those flags in the demuxing
> > drivers above? It seems LSC | RMV | PROBE_AGAIN is not used afterward,
> > seems missing.
> > 
> > Using an rte_pci_driver instead of an rte_driver as core object for you driver
> > would allow managing this by the way.
> Yes. but how would mlx5_pci bus will process the drv_flags passed by the class drivers such a vdpa/net?
> 

LSC and RMV are used within rte_eth_copy_pci_info(), which is called by
each individual netdevice PCI probe callback, after creating an ethdev.

PROBE_AGAIN is used by the PCI bus to determine whether to throw -EEXIST
when attempting to probe again a device.

They will all access it through the demux PCI device -> PCI driver,
which will be in that case your demux driver. It needs to have a correct
value that will not mask underlying device capability or behavior.

Binary-AND between classes is bare-minimum, and would probably not work
for LSC. Some finer-grained control could be used to work well with
other layers.

-- 
Gaëtan