* [dpdk-dev] [RFC PATCH 0/6] Improve mlx5 PMD common driver framework for multiple classes @ 2020-06-10 17:17 Parav Pandit 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (5 more replies) 0 siblings, 6 replies; 193+ messages in thread From: Parav Pandit @ 2020-06-10 17:17 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, thomasm, orika, matan, Parav Pandit This commit introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, a new mlx5 PCI bus is created. This bus allows binding multiple class drivers (such as net, vdpa, regex(future)) to bind to the mlx5 PCI bus driver. Change description ------------------ Patch-1 prepares the code to have RTE_BIT() macro defined in a common header. Patch-2 Prepares the code to check for enabled class Patch-3 Changes class value to a bit field Patch-4 Exposes mlx5_pci class driver registration APIs PAtch-5 Implements mlx5 PCI bus Patch-6 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus API instead of rte PCI bus API Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ ------------- / \______| mlx5 |_____ / | pci bus | ------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD defines its specific PCI ID table and mlx5 bus driver probes matching class drivers. - mlx5 pci bus driver is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI bus driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci bus, a common driver is implemented which exposes class registration API. However, bus model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common bus driver. It also eliminates peer blind zone problem as bottom pci bus layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:vdpa ./testpmd -w <PCI BDF>,class=vdpa In future, ./testpmd -w <PCI BDF>,class=net:regex Parav Pandit (6): eal: introduce macros for getting value for bit common/mlx5: use class enable check helper function common/mlx5: change mlx5 class enum values as bits bus/mlx5_pci: add mlx5 PCI bus bus/mlx5_pci: register a PCI driver bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver config/common_base | 6 + config/defconfig_arm64-bluefield-linuxapp-gcc | 6 + drivers/bus/Makefile | 3 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 49 +++ drivers/bus/mlx5_pci/meson.build | 6 + drivers/bus/mlx5_pci/mlx5_pci_bus.c | 327 ++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 66 ++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + drivers/common/mlx5/mlx5_common.c | 12 +- drivers/common/mlx5/mlx5_common.h | 8 +- .../common/mlx5/rte_common_mlx5_version.map | 2 +- drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 4 +- drivers/net/mlx5/linux/mlx5_os.h | 3 - drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 7 +- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 12 +- lib/librte_eal/include/rte_bits.h | 10 + mk/rte.app.mk | 1 + 23 files changed, 511 insertions(+), 29 deletions(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map create mode 100644 lib/librte_eal/include/rte_bits.h -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit 2020-06-10 17:17 [dpdk-dev] [RFC PATCH 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit @ 2020-06-10 17:17 ` Parav Pandit 2020-06-15 19:33 ` Gaëtan Rivet ` (11 more replies) 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 2/6] common/mlx5: use class enable check helper function Parav Pandit ` (4 subsequent siblings) 5 siblings, 12 replies; 193+ messages in thread From: Parav Pandit @ 2020-06-10 17:17 UTC (permalink / raw) To: dev; +Cc: ferruh.yigit, thomasm, orika, matan, Parav Pandit There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> --- lib/librte_eal/include/rte_bits.h | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 lib/librte_eal/include/rte_bits.h diff --git a/lib/librte_eal/include/rte_bits.h b/lib/librte_eal/include/rte_bits.h new file mode 100644 index 000000000..37f284971 --- /dev/null +++ b/lib/librte_eal/include/rte_bits.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd. + */ + +#ifndef _RTE_BITS_H_ +#define _RTE_BITS_H_ + +#define RTE_BIT(bit_num) (1UL << (bit_num)) + +#endif -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit @ 2020-06-15 19:33 ` Gaëtan Rivet 2020-06-17 8:05 ` Thomas Monjalon 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit ` (10 subsequent siblings) 11 siblings, 1 reply; 193+ messages in thread From: Gaëtan Rivet @ 2020-06-15 19:33 UTC (permalink / raw) To: Parav Pandit; +Cc: dev, ferruh.yigit, thomasm, orika, matan Hello Parav, On 10/06/20 17:17 +0000, Parav Pandit wrote: > There are several drivers which duplicate bit generation macro. > Introduce a generic bit macros so that such drivers avoid redefining > same in multiple drivers. > > Signed-off-by: Parav Pandit <parav@mellanox.com> > --- > lib/librte_eal/include/rte_bits.h | 10 ++++++++++ > 1 file changed, 10 insertions(+) > create mode 100644 lib/librte_eal/include/rte_bits.h > > diff --git a/lib/librte_eal/include/rte_bits.h b/lib/librte_eal/include/rte_bits.h > new file mode 100644 > index 000000000..37f284971 > --- /dev/null > +++ b/lib/librte_eal/include/rte_bits.h > @@ -0,0 +1,10 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2020 Mellanox Technologies, Ltd. > + */ > + > +#ifndef _RTE_BITS_H_ > +#define _RTE_BITS_H_ > + > +#define RTE_BIT(bit_num) (1UL << (bit_num)) ^ The tab here should be replaced by a space. > + > +#endif > -- > 2.25.4 > I'm not sure this kind of macro is needed, but if multiple drivers are using the patterns let's say ok. However I don't think it needs its own header. Would it be ok in lib/librte_eal/include/rte_common.h for example? -- Gaëtan ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit 2020-06-15 19:33 ` Gaëtan Rivet @ 2020-06-17 8:05 ` Thomas Monjalon 2020-06-18 9:25 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: Thomas Monjalon @ 2020-06-17 8:05 UTC (permalink / raw) To: Parav Pandit, Gaëtan Rivet; +Cc: dev, ferruh.yigit, orika, matan 15/06/2020 21:33, Gaëtan Rivet: > On 10/06/20 17:17 +0000, Parav Pandit wrote: > > There are several drivers which duplicate bit generation macro. > > Introduce a generic bit macros so that such drivers avoid redefining > > same in multiple drivers. > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > --- [...] > > --- /dev/null > > +++ b/lib/librte_eal/include/rte_bits.h > > @@ -0,0 +1,10 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright 2020 Mellanox Technologies, Ltd. > > + */ > > + > > +#ifndef _RTE_BITS_H_ > > +#define _RTE_BITS_H_ > > + > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > ^ The tab here should be replaced by a space. > > + > > +#endif > > I'm not sure this kind of macro is needed, but if multiple > drivers are using the patterns let's say ok. > > However I don't think it needs its own header. Would it be ok in > lib/librte_eal/include/rte_common.h for example? If we want to reuse an existing file, it could be lib/librte_eal/include/rte_bitops.h ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit 2020-06-17 8:05 ` Thomas Monjalon @ 2020-06-18 9:25 ` Parav Pandit 2020-06-18 12:16 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-06-18 9:25 UTC (permalink / raw) To: Thomas Monjalon, Gaëtan Rivet Cc: dev, ferruh.yigit, Ori Kam, Matan Azrad > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Wednesday, June 17, 2020 1:35 PM > > 15/06/2020 21:33, Gaëtan Rivet: > > On 10/06/20 17:17 +0000, Parav Pandit wrote: > > > There are several drivers which duplicate bit generation macro. > > > Introduce a generic bit macros so that such drivers avoid redefining > > > same in multiple drivers. > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > --- > [...] > > > --- /dev/null > > > +++ b/lib/librte_eal/include/rte_bits.h > > > @@ -0,0 +1,10 @@ > > > +/* SPDX-License-Identifier: BSD-3-Clause > > > + * Copyright 2020 Mellanox Technologies, Ltd. > > > + */ > > > + > > > +#ifndef _RTE_BITS_H_ > > > +#define _RTE_BITS_H_ > > > + > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > ^ The tab here should be replaced by a space. > > > + > > > +#endif > > > > I'm not sure this kind of macro is needed, but if multiple drivers are > > using the patterns let's say ok. > > Yes. we certainly need it. Currently BIT() macro is used at 3000+ locations and defined in 10+ drivers. Once we have this macro, it can gradually be replaced. > > However I don't think it needs its own header. Would it be ok in > > lib/librte_eal/include/rte_common.h for example? > > If we want to reuse an existing file, it could be > lib/librte_eal/include/rte_bitops.h > o.k. I will rename file from rte_bits.h to rte_bitops.h More such macros will be available here to avoid redefinitions in drivers. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit 2020-06-18 9:25 ` Parav Pandit @ 2020-06-18 12:16 ` Parav Pandit 2020-06-18 12:22 ` Thomas Monjalon 0 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-06-18 12:16 UTC (permalink / raw) To: Parav Pandit, Thomas Monjalon, Gaëtan Rivet Cc: dev, ferruh.yigit, Ori Kam, Matan Azrad Hi Thomas, > From: dev <dev-bounces@dpdk.org> On Behalf Of Parav Pandit > Sent: Thursday, June 18, 2020 2:55 PM > > > From: Thomas Monjalon <thomas@monjalon.net> > > Sent: Wednesday, June 17, 2020 1:35 PM > > > > 15/06/2020 21:33, Gaëtan Rivet: > > > On 10/06/20 17:17 +0000, Parav Pandit wrote: > > > > There are several drivers which duplicate bit generation macro. > > > > Introduce a generic bit macros so that such drivers avoid > > > > redefining same in multiple drivers. > > > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > > --- > > [...] > > > > --- /dev/null > > > > +++ b/lib/librte_eal/include/rte_bits.h > > > > @@ -0,0 +1,10 @@ > > > > +/* SPDX-License-Identifier: BSD-3-Clause > > > > + * Copyright 2020 Mellanox Technologies, Ltd. > > > > + */ > > > > + > > > > +#ifndef _RTE_BITS_H_ > > > > +#define _RTE_BITS_H_ > > > > + > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > > ^ The tab here should be replaced by a space. > > > > + > > > > +#endif > > > > > > I'm not sure this kind of macro is needed, but if multiple drivers > > > are using the patterns let's say ok. > > > > Yes. we certainly need it. Currently BIT() macro is used at 3000+ locations and > defined in 10+ drivers. > Once we have this macro, it can gradually be replaced. > > > > However I don't think it needs its own header. Would it be ok in > > > lib/librte_eal/include/rte_common.h for example? > > > > If we want to reuse an existing file, it could be > > lib/librte_eal/include/rte_bitops.h > > > o.k. > I will rename file from rte_bits.h to rte_bitops.h More such macros will be > available here to avoid redefinitions in drivers. I see that rte_bitops.h already exist and this new definition doesn't fit well in the rte_bitops.h. The intent of the rte_bitops.h is to have generic mostly macros to replace current duplication of: Such as BIT() BIT_ULL() BITS_PER_BYTE() BITS_TO_LONGS() They do not fit well in rte_bitops.h which works on the 'bitmap'. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit 2020-06-18 12:16 ` Parav Pandit @ 2020-06-18 12:22 ` Thomas Monjalon 2020-06-18 13:20 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: Thomas Monjalon @ 2020-06-18 12:22 UTC (permalink / raw) To: Parav Pandit, Gaëtan Rivet Cc: dev, ferruh.yigit, Ori Kam, Matan Azrad, joyce.kong, phil.yang, honnappa.nagarahalli, mb 18/06/2020 14:16, Parav Pandit: > From: Parav Pandit > > From: Thomas Monjalon <thomas@monjalon.net> > > > 15/06/2020 21:33, Gaëtan Rivet: > > > > On 10/06/20 17:17 +0000, Parav Pandit wrote: > > > > > There are several drivers which duplicate bit generation macro. > > > > > Introduce a generic bit macros so that such drivers avoid > > > > > redefining same in multiple drivers. > > > > > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > > > --- > > > [...] > > > > > --- /dev/null > > > > > +++ b/lib/librte_eal/include/rte_bits.h > > > > > @@ -0,0 +1,10 @@ > > > > > +/* SPDX-License-Identifier: BSD-3-Clause > > > > > + * Copyright 2020 Mellanox Technologies, Ltd. > > > > > + */ > > > > > + > > > > > +#ifndef _RTE_BITS_H_ > > > > > +#define _RTE_BITS_H_ > > > > > + > > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > > > ^ The tab here should be replaced by a space. > > > > > + > > > > > +#endif > > > > > > > > I'm not sure this kind of macro is needed, but if multiple drivers > > > > are using the patterns let's say ok. > > > > > > Yes. we certainly need it. Currently BIT() macro is used at 3000+ locations and > > defined in 10+ drivers. > > Once we have this macro, it can gradually be replaced. > > > > > > However I don't think it needs its own header. Would it be ok in > > > > lib/librte_eal/include/rte_common.h for example? > > > > > > If we want to reuse an existing file, it could be > > > lib/librte_eal/include/rte_bitops.h > > > > > o.k. > > I will rename file from rte_bits.h to rte_bitops.h More such macros will be > > available here to avoid redefinitions in drivers. > > I see that rte_bitops.h already exist and this new definition doesn't fit well in the rte_bitops.h. > The intent of the rte_bitops.h is to have generic mostly macros to replace current duplication of: > > Such as > BIT() > BIT_ULL() > BITS_PER_BYTE() > BITS_TO_LONGS() > > They do not fit well in rte_bitops.h which works on the 'bitmap'. Current bitops functions are getting and setting a bit. I don't undertand why you say "works on the bitmap". In my opinion, bit definition fits in this include file. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit 2020-06-18 12:22 ` Thomas Monjalon @ 2020-06-18 13:20 ` Parav Pandit 0 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-06-18 13:20 UTC (permalink / raw) To: Thomas Monjalon, Gaëtan Rivet Cc: dev, ferruh.yigit, Ori Kam, Matan Azrad, joyce.kong, phil.yang, honnappa.nagarahalli, mb > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Thursday, June 18, 2020 5:52 PM > > 18/06/2020 14:16, Parav Pandit: > > From: Parav Pandit > > > From: Thomas Monjalon <thomas@monjalon.net> > > > > 15/06/2020 21:33, Gaëtan Rivet: > > > > > On 10/06/20 17:17 +0000, Parav Pandit wrote: > > > > > > There are several drivers which duplicate bit generation macro. > > > > > > Introduce a generic bit macros so that such drivers avoid > > > > > > redefining same in multiple drivers. > > > > > > > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > > > > --- > > > > [...] > > > > > > --- /dev/null > > > > > > +++ b/lib/librte_eal/include/rte_bits.h > > > > > > @@ -0,0 +1,10 @@ > > > > > > +/* SPDX-License-Identifier: BSD-3-Clause > > > > > > + * Copyright 2020 Mellanox Technologies, Ltd. > > > > > > + */ > > > > > > + > > > > > > +#ifndef _RTE_BITS_H_ > > > > > > +#define _RTE_BITS_H_ > > > > > > + > > > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > > > > ^ The tab here should be replaced by a space. > > > > > > + > > > > > > +#endif > > > > > > > > > > I'm not sure this kind of macro is needed, but if multiple > > > > > drivers are using the patterns let's say ok. > > > > > > > > Yes. we certainly need it. Currently BIT() macro is used at 3000+ > > > locations and defined in 10+ drivers. > > > Once we have this macro, it can gradually be replaced. > > > > > > > > However I don't think it needs its own header. Would it be ok in > > > > > lib/librte_eal/include/rte_common.h for example? > > > > > > > > If we want to reuse an existing file, it could be > > > > lib/librte_eal/include/rte_bitops.h > > > > > > > o.k. > > > I will rename file from rte_bits.h to rte_bitops.h More such macros > > > will be available here to avoid redefinitions in drivers. > > > > I see that rte_bitops.h already exist and this new definition doesn't fit well > in the rte_bitops.h. > > The intent of the rte_bitops.h is to have generic mostly macros to replace > current duplication of: > > > > Such as > > BIT() > > BIT_ULL() > > BITS_PER_BYTE() > > BITS_TO_LONGS() > > > > They do not fit well in rte_bitops.h which works on the 'bitmap'. > > Current bitops functions are getting and setting a bit. > I don't undertand why you say "works on the bitmap". It sets in 32-bit and 64-bit addr. Eventually it will be possibly extended to more than 64-bit. It will be a bitmap as generic API. > In my opinion, bit definition fits in this include file. > Ok. I am fine. I don't have strong opinion to create a new file. I will place it here. ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit 2020-06-15 19:33 ` Gaëtan Rivet @ 2020-06-21 19:11 ` Parav Pandit 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (5 more replies) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (9 subsequent siblings) 11 siblings, 6 replies; 193+ messages in thread From: Parav Pandit @ 2020-06-21 19:11 UTC (permalink / raw) To: grive, ferruh.yigit, thomas, dev; +Cc: orika, matan, Parav Pandit This commit introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, a new mlx5 PCI bus is created. This bus allows binding multiple class drivers (such as net, vdpa, regex(future)) to bind to the mlx5 PCI bus driver. Change description ------------------ Patch-1 prepares the code to have RTE_BIT() macro defined in a common header. Patch-2 Changes class value to a bit field Patch-3 Exposes mlx5_pci class driver registration APIs PAtch-4 Implements mlx5 PCI bus Patch-5 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus API instead of rte PCI bus API Patch-6 Removed class check code as its already part of the bus now Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ ------------- / \______| mlx5 |_____ / | pci bus | ------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD defines its specific PCI ID table and mlx5 bus driver probes matching class drivers. - mlx5 pci bus driver is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI bus driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci bus, a common driver is implemented which exposes class registration API. However, bus model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common bus driver. It also eliminates peer blind zone problem as bottom pci bus layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:vdpa ./testpmd -w <PCI BDF>,class=vdpa In future, ./testpmd -w <PCI BDF>,class=net:regex Changelog: v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - USe uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (6): eal: introduce macros for getting value for bit common/mlx5: change mlx5 class enum values as bits bus/mlx5_pci: add mlx5 PCI bus bus/mlx5_pci: register a PCI driver bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver common/mlx5: Remove class checks from individual driver config/common_base | 6 + config/defconfig_arm64-bluefield-linuxapp-gcc | 6 + drivers/bus/Makefile | 3 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 49 +++ drivers/bus/mlx5_pci/meson.build | 6 + drivers/bus/mlx5_pci/mlx5_pci_bus.c | 364 ++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 85 ++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + drivers/common/mlx5/mlx5_common.c | 37 -- drivers/common/mlx5/mlx5_common.h | 7 +- .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/linux/mlx5_os.h | 1 + drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 24 +- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 28 +- lib/librte_eal/include/rte_bitops.h | 2 + mk/rte.app.mk | 1 + 23 files changed, 565 insertions(+), 80 deletions(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v2 1/6] eal: introduce macros for getting value for bit 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit @ 2020-06-21 19:11 ` Parav Pandit 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 2/6] common/mlx5: change mlx5 class enum values as bits Parav Pandit ` (4 subsequent siblings) 5 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-06-21 19:11 UTC (permalink / raw) To: grive, ferruh.yigit, thomas, dev; +Cc: orika, matan, Parav Pandit There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> --- Changelog: v1->v2: - Addressed comments from Thomas and Gaten. - Avoided new file, added macro to rte_bitops.h --- lib/librte_eal/include/rte_bitops.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..d72c7cd93 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,8 @@ #include <rte_debug.h> #include <rte_compat.h> +#define RTE_BIT(bit_num) (1UL << (bit_num)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v2 2/6] common/mlx5: change mlx5 class enum values as bits 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 1/6] eal: introduce macros for getting value for bit Parav Pandit @ 2020-06-21 19:11 ` Parav Pandit 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 3/6] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit ` (3 subsequent siblings) 5 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-06-21 19:11 UTC (permalink / raw) To: grive, ferruh.yigit, thomas, dev; +Cc: orika, matan, Parav Pandit mlx5 PCI Device supports multiple classes of devices such as net, vdpa, and/or regex. To support these multiple classes, change mlx5_class to a bitmap values so that if users asks to enable multiple of them, all supported classes can be parsed. Signed-off-by: Parav Pandit <parav@mellanox.com> --- Changelog: v1->v2: - Rebasd due to removal previous patch --- drivers/common/mlx5/mlx5_common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 77f10e676..6cc961e99 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -13,6 +13,7 @@ #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h> +#include <rte_bitops.h> #include "mlx5_prm.h" @@ -202,9 +203,9 @@ int mlx5_dev_to_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr); #define MLX5_CLASS_ARG_NAME "class" enum mlx5_class { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, MLX5_CLASS_INVALID, + MLX5_CLASS_NET = RTE_BIT(0), + MLX5_CLASS_VDPA = RTE_BIT(1), }; __rte_internal -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v2 3/6] bus/mlx5_pci: add mlx5 PCI bus 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 1/6] eal: introduce macros for getting value for bit Parav Pandit 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 2/6] common/mlx5: change mlx5 class enum values as bits Parav Pandit @ 2020-06-21 19:11 ` Parav Pandit 2020-06-29 14:01 ` Gaëtan Rivet 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 4/6] bus/mlx5_pci: register a PCI driver Parav Pandit ` (2 subsequent siblings) 5 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-06-21 19:11 UTC (permalink / raw) To: grive, ferruh.yigit, thomas, dev; +Cc: orika, matan, Parav Pandit Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single pci device. Signed-off-by: Parav Pandit <parav@mellanox.com> --- Changelog: v1->v2: - Address comments from Thomas and Gaetan - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus --- config/common_base | 6 ++ config/defconfig_arm64-bluefield-linuxapp-gcc | 6 ++ drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 47 +++++++++++ drivers/bus/mlx5_pci/meson.build | 6 ++ drivers/bus/mlx5_pci/mlx5_pci_bus.c | 14 ++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++++++++++++++++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 ++ 8 files changed, 169 insertions(+), 1 deletion(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map diff --git a/config/common_base b/config/common_base index c7d5c7321..f75b333f9 100644 --- a/config/common_base +++ b/config/common_base @@ -366,6 +366,12 @@ CONFIG_RTE_LIBRTE_IGC_DEBUG_TX=n CONFIG_RTE_LIBRTE_MLX4_PMD=n CONFIG_RTE_LIBRTE_MLX4_DEBUG=n +# +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5, +# ConnectX-6 & BlueField (MLX5) PMD +# +CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n + # # Compile burst-oriented Mellanox ConnectX-4, ConnectX-5, # ConnectX-6 & BlueField (MLX5) PMD diff --git a/config/defconfig_arm64-bluefield-linuxapp-gcc b/config/defconfig_arm64-bluefield-linuxapp-gcc index b49653881..15ade7ebc 100644 --- a/config/defconfig_arm64-bluefield-linuxapp-gcc +++ b/config/defconfig_arm64-bluefield-linuxapp-gcc @@ -14,5 +14,11 @@ CONFIG_RTE_CACHE_LINE_SIZE=64 CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n CONFIG_RTE_LIBRTE_VHOST_NUMA=n +# +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5, +# ConnectX-6 & BlueField (MLX5) PMD +# +CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n + # PMD for ConnectX-5 CONFIG_RTE_LIBRTE_MLX5_PMD=y diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index 80de2d91d..b1381838d 100644 --- a/drivers/bus/meson.build +++ b/drivers/bus/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel Corporation -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus'] +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus'] std_deps = ['eal'] config_flag_fmt = 'RTE_LIBRTE_@0@_BUS' driver_name_fmt = 'rte_bus_@0@' diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile new file mode 100644 index 000000000..7db977ba8 --- /dev/null +++ b/drivers/bus/mlx5_pci/Makefile @@ -0,0 +1,47 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2020 Mellanox Technologies, Ltd + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_bus_mlx5_pci.a + +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +LDLIBS += -lrte_eal +LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_bus_mlx5_pci_version.map + +SRCS-y += mlx5_pci_bus.c + +# DEBUG which is usually provided on the command-line may enable +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. +ifeq ($(DEBUG),1) +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y +endif + +# User-defined CFLAGS. +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y) +CFLAGS += -pedantic +ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y) +CFLAGS += -DPEDANTIC +endif +AUTO_CONFIG_CFLAGS += -Wno-pedantic +else +CFLAGS += -UPEDANTIC +endif + +# +# Export include files +# +SYMLINK-y-include += rte_bus_mlx5_pci.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build new file mode 100644 index 000000000..cc4a84e23 --- /dev/null +++ b/drivers/bus/mlx5_pci/meson.build @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020 Mellanox Technologies Ltd + +deps += ['pci', 'bus_pci', 'common_mlx5'] +install_headers('rte_bus_mlx5_pci.h') +sources = files('mlx5_pci_bus.c') diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c new file mode 100644 index 000000000..66db3c7b0 --- /dev/null +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#include "rte_bus_mlx5_pci.h" + +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = + TAILQ_HEAD_INITIALIZER(drv_list); + +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) +{ + TAILQ_INSERT_TAIL(&drv_list, driver, next); +} diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h new file mode 100644 index 000000000..571f7dfd6 --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef _RTE_BUS_MLX5_PCI_H_ +#define _RTE_BUS_MLX5_PCI_H_ + +/** + * @file + * + * RTE Mellanox PCI Bus Interface + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) + * devices. This bus enables creating such multiple class of devices on a + * single PCI device by allowing to bind multiple class specific device + * driver to attach to mlx5_pci bus driver. + * + * ----------- ------------ ----------------- + * | mlx5 | | mlx5 | | mlx5 | + * | net pmd | | vdpa pmd | | new class pmd | + * ----------- ------------ ----------------- + * \ | / + * \ | / + * \ ------------- / + * \______| mlx5 |_____ / + * | pci bus | + * ------------- + * | + * ----------- + * | mlx5 | + * | pci dev | + * ----------- + * + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 pci bus driver is cental place that validates supported + * class combinations. + */ + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +#include <rte_pci.h> +#include <rte_bus_pci.h> + +#include <mlx5_common.h> + +/** + * A structure describing a mlx5 pci driver. + */ +struct rte_mlx5_pci_driver { + enum mlx5_class dev_class; /**< Class of this driver */ + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ + TAILQ_ENTRY(rte_mlx5_pci_driver) next; +}; + +/** + * Register a mlx5_pci device driver. + * + * @param driver + * A pointer to a rte_mlx5_pci_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver); + +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \ +static const char *mlx5_pci_drvinit_fn_ ## nm; \ +RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \ +{ \ + (drv).driver.name = RTE_STR(nm); \ + rte_mlx5_pci_driver_register(&drv); \ +} \ +RTE_PMD_EXPORT_NAME(nm, __COUNTER__) + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _RTE_BUS_MLX5_PCI_H_ */ diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map new file mode 100644 index 000000000..4cfd3db10 --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map @@ -0,0 +1,5 @@ +INTERNAL { + global: + + rte_mlx5_pci_driver_register; +}; -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v2 3/6] bus/mlx5_pci: add mlx5 PCI bus 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 3/6] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit @ 2020-06-29 14:01 ` Gaëtan Rivet 0 siblings, 0 replies; 193+ messages in thread From: Gaëtan Rivet @ 2020-06-29 14:01 UTC (permalink / raw) To: Parav Pandit; +Cc: ferruh.yigit, thomas, dev, orika, matan Hi Parav, On 21/06/20 19:11 +0000, Parav Pandit wrote: > Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single > pci device. > > Signed-off-by: Parav Pandit <parav@mellanox.com> > --- > Changelog: > v1->v2: > - Address comments from Thomas and Gaetan > - Inheriting ret_pci_driver instead of rte_driver > - Added design and description of the mlx5_pci bus > --- > config/common_base | 6 ++ > config/defconfig_arm64-bluefield-linuxapp-gcc | 6 ++ > drivers/bus/meson.build | 2 +- > drivers/bus/mlx5_pci/Makefile | 47 +++++++++++ > drivers/bus/mlx5_pci/meson.build | 6 ++ > drivers/bus/mlx5_pci/mlx5_pci_bus.c | 14 ++++ > drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++++++++++++++++++ > .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 ++ > 8 files changed, 169 insertions(+), 1 deletion(-) > create mode 100644 drivers/bus/mlx5_pci/Makefile > create mode 100644 drivers/bus/mlx5_pci/meson.build > create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c > create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h > create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map > > diff --git a/config/common_base b/config/common_base > index c7d5c7321..f75b333f9 100644 > --- a/config/common_base > +++ b/config/common_base > @@ -366,6 +366,12 @@ CONFIG_RTE_LIBRTE_IGC_DEBUG_TX=n > CONFIG_RTE_LIBRTE_MLX4_PMD=n > CONFIG_RTE_LIBRTE_MLX4_DEBUG=n > > +# > +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5, > +# ConnectX-6 & BlueField (MLX5) PMD > +# > +CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n > + I told you that you were already using a construct in Makefiles that was making this config useless, but you insisted to add it. Meson does not need it and you can write your makefiles in such a way, that this bus will be enabled automatically if any of its dependents is enabled. This is the way to go, please remove these config lines. > # > # Compile burst-oriented Mellanox ConnectX-4, ConnectX-5, > # ConnectX-6 & BlueField (MLX5) PMD > diff --git a/config/defconfig_arm64-bluefield-linuxapp-gcc b/config/defconfig_arm64-bluefield-linuxapp-gcc > index b49653881..15ade7ebc 100644 > --- a/config/defconfig_arm64-bluefield-linuxapp-gcc > +++ b/config/defconfig_arm64-bluefield-linuxapp-gcc > @@ -14,5 +14,11 @@ CONFIG_RTE_CACHE_LINE_SIZE=64 > CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n > CONFIG_RTE_LIBRTE_VHOST_NUMA=n > > +# > +# Compile Mellanox PCI BUS for ConnectX-4, ConnectX-5, > +# ConnectX-6 & BlueField (MLX5) PMD > +# > +CONFIG_RTE_LIBRTE_MLX5_PCI_BUS=n > + > # PMD for ConnectX-5 > CONFIG_RTE_LIBRTE_MLX5_PMD=y > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build > index 80de2d91d..b1381838d 100644 > --- a/drivers/bus/meson.build > +++ b/drivers/bus/meson.build > @@ -1,7 +1,7 @@ > # SPDX-License-Identifier: BSD-3-Clause > # Copyright(c) 2017 Intel Corporation > > -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus'] > +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus'] > std_deps = ['eal'] > config_flag_fmt = 'RTE_LIBRTE_@0@_BUS' > driver_name_fmt = 'rte_bus_@0@' It seems you only added your bus to Meson. I see no modification to drivers/bus/Makefile > diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile > new file mode 100644 > index 000000000..7db977ba8 > --- /dev/null > +++ b/drivers/bus/mlx5_pci/Makefile > @@ -0,0 +1,47 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright 2020 Mellanox Technologies, Ltd > + > +include $(RTE_SDK)/mk/rte.vars.mk > + > +# > +# library name > +# > +LIB = librte_bus_mlx5_pci.a > + > +CFLAGS += -O3 > +CFLAGS += $(WERROR_FLAGS) > +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 > +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 > +CFLAGS += -I$(RTE_SDK)/drivers/bus/pci > +LDLIBS += -lrte_eal > +LDLIBS += -lrte_common_mlx5 > +LDLIBS += -lrte_pci -lrte_bus_pci > + > +# versioning export map > +EXPORT_MAP := rte_bus_mlx5_pci_version.map > + > +SRCS-y += mlx5_pci_bus.c > + > +# DEBUG which is usually provided on the command-line may enable > +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. > +ifeq ($(DEBUG),1) > +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y > +endif > + > +# User-defined CFLAGS. > +ifeq ($(CONFIG_RTE_LIBRTE_MLX5_DEBUG),y) > +CFLAGS += -pedantic > +ifneq ($(CONFIG_RTE_TOOLCHAIN_ICC),y) > +CFLAGS += -DPEDANTIC > +endif > +AUTO_CONFIG_CFLAGS += -Wno-pedantic > +else > +CFLAGS += -UPEDANTIC > +endif > + > +# > +# Export include files > +# > +SYMLINK-y-include += rte_bus_mlx5_pci.h > + > +include $(RTE_SDK)/mk/rte.lib.mk > diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build > new file mode 100644 > index 000000000..cc4a84e23 > --- /dev/null > +++ b/drivers/bus/mlx5_pci/meson.build > @@ -0,0 +1,6 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright(c) 2020 Mellanox Technologies Ltd > + > +deps += ['pci', 'bus_pci', 'common_mlx5'] > +install_headers('rte_bus_mlx5_pci.h') > +sources = files('mlx5_pci_bus.c') > diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c > new file mode 100644 > index 000000000..66db3c7b0 > --- /dev/null > +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c > @@ -0,0 +1,14 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2020 Mellanox Technologies, Ltd > + */ > + > +#include "rte_bus_mlx5_pci.h" > + > +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = > + TAILQ_HEAD_INITIALIZER(drv_list); > + > +void > +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) > +{ > + TAILQ_INSERT_TAIL(&drv_list, driver, next); > +} > diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h > new file mode 100644 > index 000000000..571f7dfd6 > --- /dev/null > +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h > @@ -0,0 +1,84 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2020 Mellanox Technologies, Ltd > + */ > + > +#ifndef _RTE_BUS_MLX5_PCI_H_ > +#define _RTE_BUS_MLX5_PCI_H_ > + > +/** > + * @file > + * > + * RTE Mellanox PCI Bus Interface > + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) ^ this should only appear in doc once supported > + * devices. This bus enables creating such multiple class of devices on a > + * single PCI device by allowing to bind multiple class specific device > + * driver to attach to mlx5_pci bus driver. Formulation is not great here. Here's a fix: Mellanox ConnectX PCI devices can be used in several ways, each represented by a device class (net, VDPA). This bus allows using multiple device classes with a single PCI device. Currently only net and VDPA classes exists and no combination of the two is allowed. New classes are expected to be introduced, this bus is the groundwork for their support. This last paragraph is to be removed once regex dev are supported, when combinations becomes valid and it's not future anymore. It should then be replaced by a description of the rules to combine several classes. > + * > + * ----------- ------------ ----------------- > + * | mlx5 | | mlx5 | | mlx5 | > + * | net pmd | | vdpa pmd | | new class pmd | > + * ----------- ------------ ----------------- > + * \ | / > + * \ | / > + * \ ------------- / > + * \______| mlx5 |_____ / > + * | pci bus | > + * ------------- ^^^-- trailing spaces here > + * | > + * ----------- < > + * | mlx5 | < > + * | pci dev | < > + * ----------- <--- trailing spaces here > + * > + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI > + * ID table of all related mlx5 PCI devices. > + * - mlx5 class driver such as net, vdpa, regex PMD defines its > + * specific PCI ID table and mlx5 bus driver probes matching > + * class drivers. > + * - mlx5 pci bus driver is cental place that validates supported > + * class combinations. Multiple formulation issues, simpler for me to rewrite: (please leave the empty newlines between each points.) - The mlx5 PCI bus driver binds to each PCI devices matching a PCI ID within the mlx5 table of supported PCI devices. - mlx5 class drivers such as net or VDPA PMD define their supported PCI ID table. The mlx5 bus driver will probe each matching class driver. - The mlx5 PCI bus driver is the central place validating the rules to combine and match multiple device classes. > + */ > + > +#ifdef __cplusplus > +extern "C" { > +#endif /* __cplusplus */ > + > +#include <rte_pci.h> > +#include <rte_bus_pci.h> > + > +#include <mlx5_common.h> > + > +/** > + * A structure describing a mlx5 pci driver. > + */ > +struct rte_mlx5_pci_driver { > + enum mlx5_class dev_class; /**< Class of this driver */ ^^^^^^^^^^^^^^^ ^ ^ missing period here. \ \ \ Please do not align doc elements in your structs, \ they will be awkward to edit later when adding long-name fields. \ enums width is implementation defined, I thinks it's better to use uint32_t instead. Especially considering you use offsetof-based stuff after (TAILQ). > + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ > + TAILQ_ENTRY(rte_mlx5_pci_driver) next; > +}; > + > +/** > + * Register a mlx5_pci device driver. > + * > + * @param driver > + * A pointer to a rte_mlx5_pci_driver structure describing the driver > + * to be registered. > + */ > +__rte_internal > +void > +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver); > + > +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \ > +static const char *mlx5_pci_drvinit_fn_ ## nm; \ > +RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \ > +{ \ > + (drv).driver.name = RTE_STR(nm); \ > + rte_mlx5_pci_driver_register(&drv); \ > +} \ > +RTE_PMD_EXPORT_NAME(nm, __COUNTER__) > + > +#ifdef __cplusplus > +} > +#endif /* __cplusplus */ > + > +#endif /* _RTE_BUS_MLX5_PCI_H_ */ > diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map > new file mode 100644 > index 000000000..4cfd3db10 > --- /dev/null > +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map > @@ -0,0 +1,5 @@ > +INTERNAL { > + global: > + > + rte_mlx5_pci_driver_register; > +}; > -- > 2.25.4 > -- Gaëtan ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v2 4/6] bus/mlx5_pci: register a PCI driver 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit ` (2 preceding siblings ...) 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 3/6] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit @ 2020-06-21 19:11 ` Parav Pandit 2020-06-29 15:49 ` Gaëtan Rivet 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 5/6] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit 2020-06-21 19:12 ` [dpdk-dev] [PATCH v2 6/6] common/mlx5: Remove class checks from individual driver Parav Pandit 5 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-06-21 19:11 UTC (permalink / raw) To: grive, ferruh.yigit, thomas, dev; +Cc: orika, matan, Parav Pandit Create a mlx5 bus driver framework for invoking drivers of multiple classes who have registered with the mlx5_pci bus driver. Validate user class arguments for supported class combinations. Signed-off-by: Parav Pandit <parav@mellanox.com> --- Changelog: v1->v2: - Address comments from Thomas and Gaetan - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - USe uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options --- drivers/bus/mlx5_pci/Makefile | 2 + drivers/bus/mlx5_pci/meson.build | 2 +- drivers/bus/mlx5_pci/mlx5_pci_bus.c | 290 ++++++++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 1 + 4 files changed, 294 insertions(+), 1 deletion(-) diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile index 7db977ba8..e53ed8856 100644 --- a/drivers/bus/mlx5_pci/Makefile +++ b/drivers/bus/mlx5_pci/Makefile @@ -13,7 +13,9 @@ CFLAGS += $(WERROR_FLAGS) CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +CFLAGS += -D_DEFAULT_SOURCE LDLIBS += -lrte_eal +LDLIBS += -lrte_kvargs LDLIBS += -lrte_common_mlx5 LDLIBS += -lrte_pci -lrte_bus_pci diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build index cc4a84e23..5111baa4e 100644 --- a/drivers/bus/mlx5_pci/meson.build +++ b/drivers/bus/mlx5_pci/meson.build @@ -1,6 +1,6 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Mellanox Technologies Ltd -deps += ['pci', 'bus_pci', 'common_mlx5'] +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs'] install_headers('rte_bus_mlx5_pci.h') sources = files('mlx5_pci_bus.c') diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c index 66db3c7b0..e8f1649a3 100644 --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -3,12 +3,302 @@ */ #include "rte_bus_mlx5_pci.h" +#include <mlx5_common_utils.h> static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = TAILQ_HEAD_INITIALIZER(drv_list); +static const struct { + const char *name; + unsigned int dev_class; +} mlx5_classes[] = { + { .name = "vdpa", .dev_class = MLX5_CLASS_VDPA }, + { .name = "net", .dev_class = MLX5_CLASS_NET }, +}; + +static const unsigned int mlx5_valid_class_combo[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + /* New class combination should be added here */ +}; + +static int class_name_to_val(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].dev_class; + + } + return -EINVAL; +} + +static int +mlx5_bus_opt_handler(__rte_unused const char *key, const char *class_names, + void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name */ + found = strsep(&nstr, ":"); + if (!found) + continue; + + /* Check if its a valid class */ + class_val = class_name_to_val(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s. Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, mlx5_bus_opt_handler, &ret); + rte_kvargs_free(kvlist); + return ret; +} + void rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) { TAILQ_INSERT_TAIL(&drv_list, driver, next); } + +static bool +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* check if device's ids match the class driver's ones */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + + return true; + } + return false; +} + +static int is_valid_class_combo(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination */ + for (i = 0; i < RTE_DIM(mlx5_valid_class_combo); i++) { + if (mlx5_valid_class_combo[i] == user_classes) + return 0; + } + /* Not found any valid class combination */ + return -EINVAL; +} + +static int validate_single_class_dma_ops(void) +{ + struct rte_mlx5_pci_driver *class; + int dma_map_classes = 0; + + TAILQ_FOREACH(class, &drv_list, next) { + if (class->pci_driver.dma_map) + dma_map_classes++; + } + if (dma_map_classes > 1) { + DRV_LOG(ERR, "Multiple classes with DMA ops is unsupported"); + return -EINVAL; + } + return 0; +} + +/** + * DPDK callback to register to probe multiple PCI class devices. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, 1 to skip this driver, a negative errno value otherwise + * and rte_errno is set. + */ +static int +mlx5_bus_pci_probe(struct rte_pci_driver *drv __rte_unused, + struct rte_pci_device *dev) +{ + struct rte_mlx5_pci_driver *class; + uint32_t user_classes = 0; + int ret; + + ret = validate_single_class_dma_ops(); + if (ret) + return ret; + + ret = parse_class_options(dev->device.devargs); + if (ret < 0) + return ret; + + user_classes = ret; + if (user_classes) { + /* Validate combination here */ + ret = is_valid_class_combo(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied"); + return ret; + } + } + + /* Default to net class */ + if (user_classes == 0) + user_classes = MLX5_CLASS_NET; + + TAILQ_FOREACH(class, &drv_list, next) { + if (!mlx5_bus_match(class, dev)) + continue; + + if ((class->dev_class & user_classes) == 0) + continue; + + ret = -EINVAL; + if (class->loaded) { + /* If already loaded and class driver can handle + * reprobe, probe such class driver again. + */ + if (class->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN) + ret = class->pci_driver.probe(drv, dev); + } else { + ret = class->pci_driver.probe(drv, dev); + } + if (!ret) + class->loaded = true; + } + return 0; +} + +/** + * DPDK callback to remove one or more class devices for a PCI device. + * + * This function removes all class devices belong to a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_bus_pci_remove(struct rte_pci_device *dev) +{ + struct rte_mlx5_pci_driver *class; + + /* Remove each class driver in reverse order */ + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (class->loaded) + class->pci_driver.remove(dev); + } + return 0; +} + +static int +mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + int ret = -EINVAL; + + TAILQ_FOREACH(class, &drv_list, next) { + if (!class->pci_driver.dma_map) + continue; + + return class->pci_driver.dma_map(dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + int ret = -EINVAL; + + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (!class->pci_driver.dma_unmap) + continue; + + return class->pci_driver.dma_unmap(dev, addr, iova, len); + } + return ret; +} + +static const struct rte_pci_id mlx5_bus_pci_id_map[] = { + { + .vendor_id = 0 + } +}; + +static struct rte_pci_driver mlx5_bus_driver = { + .driver = { + .name = "mlx5_bus_pci", + }, + .id_table = mlx5_bus_pci_id_map, + .probe = mlx5_bus_pci_probe, + .remove = mlx5_bus_pci_remove, + .dma_map = mlx5_bus_pci_dma_map, + .dma_unmap = mlx5_bus_pci_dma_unmap, + .drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV | + RTE_PCI_DRV_PROBE_AGAIN, +}; + +RTE_PMD_REGISTER_PCI(mlx5_bus, mlx5_bus_driver); +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map); diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h index 571f7dfd6..c8cd7187b 100644 --- a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h @@ -55,6 +55,7 @@ struct rte_mlx5_pci_driver { enum mlx5_class dev_class; /**< Class of this driver */ struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ TAILQ_ENTRY(rte_mlx5_pci_driver) next; + bool loaded; }; /** -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v2 4/6] bus/mlx5_pci: register a PCI driver 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 4/6] bus/mlx5_pci: register a PCI driver Parav Pandit @ 2020-06-29 15:49 ` Gaëtan Rivet 0 siblings, 0 replies; 193+ messages in thread From: Gaëtan Rivet @ 2020-06-29 15:49 UTC (permalink / raw) To: Parav Pandit; +Cc: ferruh.yigit, thomas, dev, orika, matan On 21/06/20 19:11 +0000, Parav Pandit wrote: > Create a mlx5 bus driver framework for invoking drivers of > multiple classes who have registered with the mlx5_pci bus > driver. > > Validate user class arguments for supported class combinations. > > Signed-off-by: Parav Pandit <parav@mellanox.com> > --- > Changelog: > v1->v2: > - Address comments from Thomas and Gaetan > - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag > - Use anonymous structure for class search and code changes around it > - Define static for class comination array > - Use RTE_DIM to find array size > - Added OOM check for strdup() > - Renamed copy variable to nstr_orig > - Returning negagive error code > - Returning directly if match entry found > - Use compat condition check > - Avoided cutting error message string > - USe uint32_t datatype instead of enum mlx5_class > - Changed logic to parse device arguments only once during probe() > - Added check to fail driver probe if multiple classes register with > DMA ops > - Renamed function to parse_class_options > --- > drivers/bus/mlx5_pci/Makefile | 2 + > drivers/bus/mlx5_pci/meson.build | 2 +- > drivers/bus/mlx5_pci/mlx5_pci_bus.c | 290 ++++++++++++++++++++++++ > drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 1 + > 4 files changed, 294 insertions(+), 1 deletion(-) > > diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile > index 7db977ba8..e53ed8856 100644 > --- a/drivers/bus/mlx5_pci/Makefile > +++ b/drivers/bus/mlx5_pci/Makefile > @@ -13,7 +13,9 @@ CFLAGS += $(WERROR_FLAGS) > CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 > CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 > CFLAGS += -I$(RTE_SDK)/drivers/bus/pci > +CFLAGS += -D_DEFAULT_SOURCE > LDLIBS += -lrte_eal > +LDLIBS += -lrte_kvargs > LDLIBS += -lrte_common_mlx5 > LDLIBS += -lrte_pci -lrte_bus_pci > > diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build > index cc4a84e23..5111baa4e 100644 > --- a/drivers/bus/mlx5_pci/meson.build > +++ b/drivers/bus/mlx5_pci/meson.build > @@ -1,6 +1,6 @@ > # SPDX-License-Identifier: BSD-3-Clause > # Copyright(c) 2020 Mellanox Technologies Ltd > > -deps += ['pci', 'bus_pci', 'common_mlx5'] > +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs'] > install_headers('rte_bus_mlx5_pci.h') > sources = files('mlx5_pci_bus.c') > diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c > index 66db3c7b0..e8f1649a3 100644 > --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c > +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c > @@ -3,12 +3,302 @@ > */ > > #include "rte_bus_mlx5_pci.h" > +#include <mlx5_common_utils.h> > > static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = > TAILQ_HEAD_INITIALIZER(drv_list); > > +static const struct { > + const char *name; > + unsigned int dev_class; Let me quote you when you refused to follow my comment: >> Yes, I acked to changed to define, but I forgot that I use the enum here. >> So I am going to keep the enum as code reads more clear with enum. You refused to use a fixed-width integer type as per my past comments, for readability reasons, but changed the type to "unsigned int" instead. I insisted in the previous commit on uint32_t for exposed ABI (even if between internal libs). Here I accept some leeway given the compilation-unit scope of the definition. But at this point, your choice is certainly *NOT* to use a vague type instead. > +} mlx5_classes[] = { mlx5_class_names. > + { .name = "vdpa", .dev_class = MLX5_CLASS_VDPA }, > + { .name = "net", .dev_class = MLX5_CLASS_NET }, > +}; > + > +static const unsigned int mlx5_valid_class_combo[] = { > + MLX5_CLASS_NET, > + MLX5_CLASS_VDPA, > + /* New class combination should be added here */ This comment seems redundant, new class combo will be added wherever appropriate, leave it to future dev. > +}; > + > +static int class_name_to_val(const char *class_name) I think mlx5_class_from_name() is better. (with mlx5_ namespace.) > +{ > + unsigned int i; In general, size_t is the type of array iterators in C. > + > + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { > + if (strcmp(class_name, mlx5_classes[i].name) == 0) > + return mlx5_classes[i].dev_class; > + > + } > + return -EINVAL; You're mixing signed int and enum mlx5_class as return type. Please find another way of signaling error that will make you keep the enum. You have a sentinel value describing explicitly an invalid class, it seems perfectly suited instead of -EINVAL. Use it instead. > +} > + > +static int > +mlx5_bus_opt_handler(__rte_unused const char *key, const char *class_names, > + void *opaque) > +{ > + int *ret = opaque; > + char *nstr_org; > + int class_val; > + char *found; > + char *nstr; > + > + *ret = 0; > + nstr = strdup(class_names); > + if (!nstr) { Please be explicit and use (nstr == NULL). > + *ret = -ENOMEM; > + return *ret; > + } > + > + nstr_org = nstr; nstr_orig is more readable. > + while (nstr) { while (nstr != NULL) { > + /* Extract each individual class name */ > + found = strsep(&nstr, ":"); > + if (!found) ditto > + continue; > + > + /* Check if its a valid class */ > + class_val = class_name_to_val(found); > + if (class_val < 0) { if (class_val == MLX5_CLASS_INVALID), with the proper API change. > + *ret = -EINVAL; > + goto err; > + } > + > + *ret |= class_val; Once again, mixing ints and enum mlx5_class. You don't *have* to set *ret on error. * Change your opaque out_arg to uint32_t, stop using variable width types for bitmaps. * Do not set it on error, use a tmp u32 for parsing and only set it once everything is ok. * rte_kvargs_process() will mask your error values anyway, so instead set rte_errno and return -1. On negative return, it will itself return -1. Check for < 0 in bus_options_valid() > + } > +err: > + free(nstr_org); > + if (*ret < 0) > + DRV_LOG(ERR, "Invalid mlx5 class options %s. Maybe typo in device class argument setting?", Find a way to give the exact source of error. If it is an invalid name, show which token failed to be parsed (meaning move your error code before nstr_orig is freed). Remove the "Maybe" formulation. By the way, Thomas' comment was correct instead of mine, you should just cut your format string after the "%s.". > + class_names); > + return *ret; > +} > + > +static int > +parse_class_options(const struct rte_devargs *devargs) > +{ > + const char *key = MLX5_CLASS_ARG_NAME; > + struct rte_kvargs *kvlist; > + int ret = 0; > + > + if (devargs == NULL) > + return 0; > + kvlist = rte_kvargs_parse(devargs->args, NULL); > + if (kvlist == NULL) > + return 0; > + if (rte_kvargs_count(kvlist, key)) > + rte_kvargs_process(kvlist, key, mlx5_bus_opt_handler, &ret); Set ret to rte_kvargs_process() return value instead, define a specific u32 for bitmap. Find a way to output the bitmap *separately* from the error code, or set MLX5_CLASS_INVALID in the bitmap before returning it as sole return value for this function. (meaning having a proper bit value for MLX5_CLASS_INVALID, if you go this way.) I already said it in previous review, I will reformulate: stop overloading your types, relying on implicit casts between correct and incorrect values, and merging your returned values and the error channel. Please be proactive into cleaning up your APIs. > + rte_kvargs_free(kvlist); > + return ret; > +} > + > void > rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) > { > TAILQ_INSERT_TAIL(&drv_list, driver, next); > } > + > +static bool > +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv, > + const struct rte_pci_device *pci_dev) > +{ > + const struct rte_pci_id *id_table; > + > + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; > + id_table++) { > + /* check if device's ids match the class driver's ones */ > + if (id_table->vendor_id != pci_dev->id.vendor_id && > + id_table->vendor_id != PCI_ANY_ID) > + continue; > + if (id_table->device_id != pci_dev->id.device_id && > + id_table->device_id != PCI_ANY_ID) > + continue; > + if (id_table->subsystem_vendor_id != > + pci_dev->id.subsystem_vendor_id && > + id_table->subsystem_vendor_id != PCI_ANY_ID) > + continue; > + if (id_table->subsystem_device_id != > + pci_dev->id.subsystem_device_id && > + id_table->subsystem_device_id != PCI_ANY_ID) > + continue; > + if (id_table->class_id != pci_dev->id.class_id && > + id_table->class_id != RTE_CLASS_ANY_ID) > + continue; > + > + return true; > + } > + return false; > +} > + > +static int is_valid_class_combo(uint32_t user_classes) > +{ > + unsigned int i; size_t > + > + /* Verify if user specified valid supported combination */ a valid combination. > + for (i = 0; i < RTE_DIM(mlx5_valid_class_combo); i++) { > + if (mlx5_valid_class_combo[i] == user_classes) You simplified the scope of this function, which is good. However, given the more limited scope, now it becomes a boolean yes/no. You are returning (0 | false) for yes, which is not ok. reading if (is_valid_class_combo(combo)) { handle_error(combo); } is pretty awkward. While you're at it, you might want to use a proper bool instead. > + return 0; > + } > + /* Not found any valid class combination */ > + return -EINVAL; > +} > + > +static int validate_single_class_dma_ops(void) > +{ > + struct rte_mlx5_pci_driver *class; > + int dma_map_classes = 0; > + > + TAILQ_FOREACH(class, &drv_list, next) { > + if (class->pci_driver.dma_map) > + dma_map_classes++; > + } > + if (dma_map_classes > 1) { > + DRV_LOG(ERR, "Multiple classes with DMA ops is unsupported"); > + return -EINVAL; > + } > + return 0; > +} > + > +/** > + * DPDK callback to register to probe multiple PCI class devices. > + * > + * @param[in] pci_drv > + * PCI driver structure. > + * @param[in] dev > + * PCI device information. > + * > + * @return > + * 0 on success, 1 to skip this driver, a negative errno value otherwise > + * and rte_errno is set. > + */ > +static int > +mlx5_bus_pci_probe(struct rte_pci_driver *drv __rte_unused, drv is not unused, you are passing it to sub-drivers below. > + struct rte_pci_device *dev) > +{ > + struct rte_mlx5_pci_driver *class; This compilation unit targets a C compiler. I think only headers should ensure compat with C++, but this name is not great still. driver seems more appropriate anyway to designate a driver. > + uint32_t user_classes = 0; > + int ret; > + Mixing ret and user_classes as you do afterward is the result of the above API issues already outlined. I won't go over them again, please fix everything to have proper type discipline. > + ret = validate_single_class_dma_ops(); > + if (ret) > + return ret; > + > + ret = parse_class_options(dev->device.devargs); > + if (ret < 0) > + return ret; > + > + user_classes = ret; > + if (user_classes) { > + /* Validate combination here */ > + ret = is_valid_class_combo(user_classes); > + if (ret) { > + DRV_LOG(ERR, "Unsupported mlx5 classes supplied"); > + return ret; > + } > + } > + > + /* Default to net class */ > + if (user_classes == 0) > + user_classes = MLX5_CLASS_NET; > + > + TAILQ_FOREACH(class, &drv_list, next) { > + if (!mlx5_bus_match(class, dev)) > + continue; > + > + if ((class->dev_class & user_classes) == 0) > + continue; > + > + ret = -EINVAL; > + if (class->loaded) { > + /* If already loaded and class driver can handle > + * reprobe, probe such class driver again. > + */ > + if (class->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN) > + ret = class->pci_driver.probe(drv, dev); Using "drv" here instead of "class" means you are overriding the DRV_FLAG set by the sub-driver. Why not use "class" instead? dev->driver is setup by the upper layer, so will be correctly set to drv instead of class. > + } else { > + ret = class->pci_driver.probe(drv, dev); > + } You are ignoring probe() < 0 here, seems wrong. > + if (!ret) > + class->loaded = true; loaded flag is not properly set. You will set it on first successful probe, even on further errors. Instead, use a u32 to mark each properly probed classes, then set loaded outside of this loop, only if this "probed" bitmap matches exactly the "user_classes" bitmap. This means also not silently ignoring dev and class mismatch. If this is the behavior you explicitly want, then you will need to unset the mismatched class in the user_classes, so that the exact match on probed is correct. Otherwise, logging an error is more appropriate. > + } > + return 0; > +} > + > +/** > + * DPDK callback to remove one or more class devices for a PCI device. > + * > + * This function removes all class devices belong to a given PCI device. > + * > + * @param[in] pci_dev > + * Pointer to the PCI device. > + * > + * @return > + * 0 on success, the function cannot fail. > + */ > +static int > +mlx5_bus_pci_remove(struct rte_pci_device *dev) > +{ > + struct rte_mlx5_pci_driver *class; > + > + /* Remove each class driver in reverse order */ > + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { > + if (class->loaded) > + class->pci_driver.remove(dev); > + } > + return 0; > +} > + > +static int > +mlx5_bus_pci_dma_map(struct rte_pci_device *dev, void *addr, > + uint64_t iova, size_t len) > +{ > + struct rte_mlx5_pci_driver *class; > + int ret = -EINVAL; > + > + TAILQ_FOREACH(class, &drv_list, next) { > + if (!class->pci_driver.dma_map) > + continue; > + > + return class->pci_driver.dma_map(dev, addr, iova, len); > + } > + return ret; > +} > + > +static int > +mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr, > + uint64_t iova, size_t len) > +{ > + struct rte_mlx5_pci_driver *class; > + int ret = -EINVAL; > + > + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { > + if (!class->pci_driver.dma_unmap) > + continue; > + I see no additional logging about edge-cases that were discussed previously. You can add them to the register function. > + return class->pci_driver.dma_unmap(dev, addr, iova, len); > + } > + return ret; > +} > + > +static const struct rte_pci_id mlx5_bus_pci_id_map[] = { > + { > + .vendor_id = 0 > + } > +}; > + > +static struct rte_pci_driver mlx5_bus_driver = { > + .driver = { > + .name = "mlx5_bus_pci", > + }, > + .id_table = mlx5_bus_pci_id_map, > + .probe = mlx5_bus_pci_probe, > + .remove = mlx5_bus_pci_remove, > + .dma_map = mlx5_bus_pci_dma_map, > + .dma_unmap = mlx5_bus_pci_dma_unmap, > + .drv_flags = RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV | > + RTE_PCI_DRV_PROBE_AGAIN, > +}; > + > +RTE_PMD_REGISTER_PCI(mlx5_bus, mlx5_bus_driver); > +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map); > diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h > index 571f7dfd6..c8cd7187b 100644 > --- a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h > +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h > @@ -55,6 +55,7 @@ struct rte_mlx5_pci_driver { > enum mlx5_class dev_class; /**< Class of this driver */ > struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ > TAILQ_ENTRY(rte_mlx5_pci_driver) next; > + bool loaded; > }; > > /** > -- > 2.25.4 > -- Gaëtan ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v2 5/6] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit ` (3 preceding siblings ...) 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 4/6] bus/mlx5_pci: register a PCI driver Parav Pandit @ 2020-06-21 19:11 ` Parav Pandit 2020-06-21 19:12 ` [dpdk-dev] [PATCH v2 6/6] common/mlx5: Remove class checks from individual driver Parav Pandit 5 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-06-21 19:11 UTC (permalink / raw) To: grive, ferruh.yigit, thomas, dev; +Cc: orika, matan, Parav Pandit Enable class driver to match with the mlx5 pci devices. Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class driver. Signed-off-by: Parav Pandit <parav@mellanox.com> --- Changelog: v1->v2: - Migreate API from rte_driver to rte_pci_driver --- drivers/bus/Makefile | 3 ++ drivers/bus/mlx5_pci/mlx5_pci_bus.c | 60 +++++++++++++++++++++++++++++ drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 1 - drivers/net/mlx5/linux/mlx5_os.h | 1 + drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 24 +++++++----- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 ++++++----- mk/rte.app.mk | 1 + 12 files changed, 98 insertions(+), 26 deletions(-) diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index cea3b55e6..3840ac51c 100644 --- a/drivers/bus/Makefile +++ b/drivers/bus/Makefile @@ -8,6 +8,9 @@ ifeq ($(CONFIG_RTE_EAL_VFIO),y) DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) +DIRS-y += mlx5_pci +endif DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c index e8f1649a3..6844c1961 100644 --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -282,6 +282,66 @@ mlx5_bus_pci_dma_unmap(struct rte_pci_device *dev, void *addr, } static const struct rte_pci_id mlx5_bus_pci_id_map[] = { + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX4) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX4VF) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX4LX) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX5) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX5VF) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX5EX) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX5BF) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX6) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX6VF) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX6DX) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX6DXVF) + }, + { + RTE_PCI_DEVICE(PCI_VENDOR_ID_MELLANOX, + PCI_DEVICE_ID_MELLANOX_CONNECTX6DXBF) + }, { .vendor_id = 0 } diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index a458402dc..1cea7cd07 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_bus_mlx5_pci LDLIBS += -lm LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 3792371c3..60f42c5d1 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1394,7 +1394,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); return -rte_errno; } - MLX5_ASSERT(pci_drv == &mlx5_driver); errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index f310f1773..3479c6428 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,4 +15,5 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) + #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index e95ce0267..26699a79b 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -8,7 +8,7 @@ if not (is_linux or is_windows) subdir_done() endif -deps += ['hash', 'common_mlx5'] +deps += ['hash', 'common_mlx5', 'bus_mlx5_pci'] sources = files( 'mlx5.c', 'mlx5_ethdev.c', diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 5c86f6fb9..22eaa64d3 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -34,6 +34,7 @@ #include <rte_spinlock.h> #include <rte_string_fns.h> #include <rte_alarm.h> +#include <rte_bus_mlx5_pci.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h> @@ -1987,16 +1988,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -struct rte_pci_driver mlx5_driver = { - .driver = { - .name = MLX5_DRIVER_NAME +static struct rte_mlx5_pci_driver mlx5_driver = { + .dev_class = MLX5_CLASS_NET, + .pci_driver = { + .driver = { + .name = MLX5_DRIVER_NAME, + }, + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_dma_map, + .dma_unmap = mlx5_dma_unmap, + .drv_flags = PCI_DRV_FLAGS, }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, }; /** @@ -2014,7 +2018,7 @@ RTE_INIT(rte_mlx5_pmd_init) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - rte_pci_register(&mlx5_driver); + rte_mlx5_pci_driver_register(&mlx5_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 5bd5acd9d..300b15a7d 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -129,7 +129,6 @@ struct mlx5_local_data { }; extern struct mlx5_shared_data *mlx5_shared_data; -extern struct rte_pci_driver mlx5_driver; /* Dev ops structs */ extern const struct eth_dev_ops mlx5_os_dev_ops; diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index 91c89d604..a2231f021 100644 --- a/drivers/vdpa/mlx5/Makefile +++ b/drivers/vdpa/mlx5/Makefile @@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa CFLAGS += -I$(RTE_SDK)/lib/librte_sched CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build index 2963aad71..f4175c34e 100644 --- a/drivers/vdpa/mlx5/meson.build +++ b/drivers/vdpa/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_vdpa' -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched'] +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched'] sources = files( 'mlx5_vdpa.c', 'mlx5_vdpa_mem.c', diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 1113d6cef..b223da7f2 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -6,7 +6,7 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> +#include <rte_bus_mlx5_pci.h> #include <rte_pci.h> #include <mlx5_glue.h> @@ -428,7 +428,7 @@ mlx5_vdpa_roce_disable(struct rte_pci_addr *addr, struct ibv_device **ibv) } /** - * DPDK callback to register a PCI device. + * DPDK callback to register a mlx5 PCI bus device. * * This function spawns vdpa device out of a given PCI device. * @@ -598,14 +598,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_vdpa_driver = { - .driver = { - .name = "mlx5_vdpa", +static struct rte_mlx5_pci_driver mlx5_vdpa_driver = { + .dev_class = MLX5_CLASS_VDPA, + .pci_driver = { + .driver = { + .name = "mlx5_vdpa", + }, + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_pci_probe, + .remove = mlx5_vdpa_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, }; /** @@ -618,7 +621,7 @@ RTE_INIT(rte_mlx5_vdpa_init) if (mlx5_vdpa_logtype >= 0) rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE); if (mlx5_glue) - rte_pci_register(&mlx5_vdpa_driver); + rte_mlx5_pci_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 0ce8cf541..d7c1d90e4 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF) += -lrte_pmd_memif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) _LDLIBS-y += -lrte_common_mlx5 +_LDLIBS-y += -lrte_bus_mlx5_pci endif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD) += -lrte_pmd_mlx5_vdpa -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v2 6/6] common/mlx5: Remove class checks from individual driver 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit ` (4 preceding siblings ...) 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 5/6] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit @ 2020-06-21 19:12 ` Parav Pandit 5 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-06-21 19:12 UTC (permalink / raw) To: grive, ferruh.yigit, thomas, dev; +Cc: orika, matan, Parav Pandit Now that mlx5_pci bus does the check for enabled classes and performs probe(), remove() of associated classes, individual class driver doesn't need to check if other driver is enabled. Signed-off-by: Parav Pandit <parav@mellanox.com> --- Changelog: v1->v2: - New patch --- drivers/common/mlx5/mlx5_common.c | 37 ------------------- drivers/common/mlx5/mlx5_common.h | 2 - .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/linux/mlx5_os.c | 5 --- drivers/vdpa/mlx5/mlx5_vdpa.c | 5 --- 5 files changed, 51 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index db94d4aa8..ae7a0d6d6 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -19,43 +19,6 @@ const struct mlx5_glue *mlx5_glue; uint8_t haswell_broadwell_cpu; -static int -mlx5_class_check_handler(__rte_unused const char *key, const char *value, - void *opaque) -{ - enum mlx5_class *ret = opaque; - - if (strcmp(value, "vdpa") == 0) { - *ret = MLX5_CLASS_VDPA; - } else if (strcmp(value, "net") == 0) { - *ret = MLX5_CLASS_NET; - } else { - DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device" - " class argument setting?", value); - *ret = MLX5_CLASS_INVALID; - } - return 0; -} - -enum mlx5_class -mlx5_class_get(struct rte_devargs *devargs) -{ - struct rte_kvargs *kvlist; - const char *key = MLX5_CLASS_ARG_NAME; - enum mlx5_class ret = MLX5_CLASS_NET; - - if (devargs == NULL) - return ret; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return ret; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret); - rte_kvargs_free(kvlist); - return ret; -} - - /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 6cc961e99..fee988379 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -208,8 +208,6 @@ enum mlx5_class { MLX5_CLASS_VDPA = RTE_BIT(1), }; -__rte_internal -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs); __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 68f120712..236a7ef7b 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,8 +1,6 @@ INTERNAL { global: - mlx5_class_get; - mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 60f42c5d1..944c3bf66 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1381,11 +1381,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_dev_config dev_config; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } if (rte_eal_process_type() == RTE_PROC_PRIMARY) mlx5_pmd_socket_init(); ret = mlx5_init_once(); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index b223da7f2..30758f7b6 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -451,11 +451,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_hca_attr attr; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr); if (!ibv) { DRV_LOG(ERR, "No matching IB device for PCI slot " -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit 2020-06-15 19:33 ` Gaëtan Rivet 2020-06-21 19:11 ` [dpdk-dev] [PATCH v2 0/6] Improve mlx5 PMD common driver framework for multiple classes Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 01/10] eal: introduce macros for getting value for bit Parav Pandit ` (9 more replies) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (8 subsequent siblings) 11 siblings, 10 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit This series introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, a new mlx5 PCI bus is created. This bus allows binding multiple class drivers (such as net, vdpa, regex(future)) to bind to the mlx5 PCI bus driver. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Introduces new RTE constructor priority for common initialization Patch-3 Fixes compilation error Patch-4 Define mlx5 class as bit fields Patch-5 Uses new RTE common priority Patch-6 Adds mlx5 PCI bus Patch-7 Implements a mlx5 PCI bus driver Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus API instead of rte PCI bus API Patch-9 Removed class check code as its already part of the bus now Patch-10 Add maintainers for the new bus Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ ------------- / \______| mlx5 |_____ / | pci bus | ------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD defines its specific PCI ID table and mlx5 bus driver probes matching class drivers. - mlx5 pci bus driver is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI bus driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci bus, a common driver is implemented which exposes class registration API. However, bus model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common bus driver. It also eliminates peer blind zone problem as bottom pci bus layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:vdpa ./testpmd -w <PCI BDF>,class=vdpa In future, ./testpmd -w <PCI BDF>,class=net:regex Changelog: v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (10): eal: introduce macros for getting value for bit eal: introduce RTE common initialization level common/mlx5: fix empty input style in glue wrappers common/mlx5: change mlx5 class enum values as bits common/mlx5: use common rte priority bus/mlx5_pci: add mlx5 PCI bus bus/mlx5_pci: register a PCI driver bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver common/mlx5: remove class checks from individual driver maintainers: add maintainers for mlx5 pci bus MAINTAINERS | 5 + drivers/bus/Makefile | 3 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 40 ++ drivers/bus/mlx5_pci/meson.build | 19 + drivers/bus/mlx5_pci/mlx5_pci_bus.c | 522 ++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/mlx5_common.c | 39 +- drivers/common/mlx5/mlx5_common.h | 7 +- .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 26 +- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 30 +- lib/librte_eal/include/rte_bitops.h | 2 + lib/librte_eal/include/rte_common.h | 1 + mk/rte.app.mk | 1 + 23 files changed, 724 insertions(+), 85 deletions(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 01/10] eal: introduce macros for getting value for bit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 02/10] eal: introduce RTE common initialization level Parav Pandit ` (8 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v1->v2: - Addressed comments from Thomas and Gaten. - Avoided new file, added macro to rte_bitops.h --- lib/librte_eal/include/rte_bitops.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..d72c7cd93 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,8 @@ #include <rte_debug.h> #include <rte_compat.h> +#define RTE_BIT(bit_num) (1UL << (bit_num)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 02/10] eal: introduce RTE common initialization level 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 01/10] eal: introduce macros for getting value for bit Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 03/10] common/mlx5: fix empty input style in glue wrappers Parav Pandit ` (7 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Currently mlx5_common uses CLASS priority to initialize common code before initializing the PMD. However mlx5_common is not really a class, it is the pre-initialization code needed for the PMDs. In subsequent patch a needed initialization sequence is: (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Hence, introduce a new RTE priority level RTE_PRIO_COMMON which can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs for class driver initialization. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- lib/librte_eal/include/rte_common.h | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h index 0843ce69e..d4653ea77 100644 --- a/lib/librte_eal/include/rte_common.h +++ b/lib/librte_eal/include/rte_common.h @@ -126,6 +126,7 @@ typedef uint16_t unaligned_uint16_t; #define RTE_PRIORITY_LOG 101 #define RTE_PRIORITY_BUS 110 +#define RTE_PRIORITY_COMMON 119 #define RTE_PRIORITY_CLASS 120 #define RTE_PRIORITY_LAST 65535 -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 03/10] common/mlx5: fix empty input style in glue wrappers 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 01/10] eal: introduce macros for getting value for bit Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 02/10] eal: introduce RTE common initialization level Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 04/10] common/mlx5: change mlx5 class enum values as bits Parav Pandit ` (6 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Following two errors are reported when compiled with gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5). drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] Fix them by adding void data type in empty argument list. Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch") Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue") Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- drivers/common/mlx5/linux/mlx5_glue.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h index 069d8540c..e5e052a6a 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.h +++ b/drivers/common/mlx5/linux/mlx5_glue.h @@ -185,11 +185,11 @@ struct mlx5_glue { void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl); void *(*dr_create_flow_action_dest_port)(void *domain, uint32_t port); - void *(*dr_create_flow_action_drop)(); + void *(*dr_create_flow_action_drop)(void); void *(*dr_create_flow_action_push_vlan) (struct mlx5dv_dr_domain *domain, rte_be32_t vlan_tag); - void *(*dr_create_flow_action_pop_vlan)(); + void *(*dr_create_flow_action_pop_vlan)(void); void *(*dr_create_flow_tbl)(void *domain, uint32_t level); int (*dr_destroy_flow_tbl)(void *tbl); void *(*dr_create_domain)(struct ibv_context *ctx, -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 04/10] common/mlx5: change mlx5 class enum values as bits 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (2 preceding siblings ...) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 03/10] common/mlx5: fix empty input style in glue wrappers Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 05/10] common/mlx5: use common rte priority Parav Pandit ` (5 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit mlx5 PCI Device supports multiple classes of devices such as net, vdpa, and/or regex. To support these multiple classes, change mlx5_class to a bitmap values so that if users asks to enable multiple of them, all supported classes can be parsed. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v1->v2: - Rebasd due to removal previous patch --- drivers/common/mlx5/mlx5_common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index da01ffa1c..62861af05 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -13,6 +13,7 @@ #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h> +#include <rte_bitops.h> #include "mlx5_prm.h" #include "mlx5_devx_cmds.h" @@ -207,9 +208,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); #define MLX5_CLASS_ARG_NAME "class" enum mlx5_class { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, MLX5_CLASS_INVALID, + MLX5_CLASS_NET = RTE_BIT(0), + MLX5_CLASS_VDPA = RTE_BIT(1), }; #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */ -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 05/10] common/mlx5: use common rte priority 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (3 preceding siblings ...) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 04/10] common/mlx5: change mlx5 class enum values as bits Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 06/10] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit ` (4 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Use RTE_PRIO_COMMON for mlx5 common initialization and use RTE_PRIO_CLASS for mlx5 net, vdpa PMDs. This enables to do following initialization sequence. (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/mlx5_common.c | 2 +- drivers/net/mlx5/mlx5.c | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 693e2c68c..0ce5e4db1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) /** * Initialization routine for run-time dependency on glue library. */ -RTE_INIT_PRIO(mlx5_glue_init, CLASS) +RTE_INIT_PRIO(mlx5_glue_init, COMMON) { mlx5_glue_constructor(); } diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 07c6addd5..bb10c63bb 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1907,7 +1907,7 @@ struct rte_pci_driver mlx5_driver = { /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_pmd_init) +RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) { /* Initialize driver log type. */ mlx5_logtype = rte_log_register("pmd.net.mlx5"); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index dbd36ab0c..f043166ef 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -842,7 +842,7 @@ static struct rte_pci_driver mlx5_vdpa_driver = { /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_vdpa_init) +RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) { /* Initialize common log type. */ mlx5_vdpa_logtype = rte_log_register("pmd.vdpa.mlx5"); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 06/10] bus/mlx5_pci: add mlx5 PCI bus 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (4 preceding siblings ...) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 05/10] common/mlx5: use common rte priority Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 07/10] bus/mlx5_pci: register a PCI driver Parav Pandit ` (3 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single pci device. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Addressed comments from Thomas and Asaf - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address comments from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Changed alignedment to mlx5 specific aligment instead of standard DPDK - Using uint32_t instead of mlx5_class enum v1->v2: - Address comments from Thomas and Gaetan - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus --- drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 38 +++++++++ drivers/bus/mlx5_pci/meson.build | 19 +++++ drivers/bus/mlx5_pci/mlx5_pci_bus.c | 14 ++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++++++++++++++++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 ++ 6 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index 80de2d91d..b1381838d 100644 --- a/drivers/bus/meson.build +++ b/drivers/bus/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel Corporation -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus'] +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus'] std_deps = ['eal'] config_flag_fmt = 'RTE_LIBRTE_@0@_BUS' driver_name_fmt = 'rte_bus_@0@' diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile new file mode 100644 index 000000000..de4ccd83f --- /dev/null +++ b/drivers/bus/mlx5_pci/Makefile @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2020 Mellanox Technologies, Ltd + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_bus_mlx5_pci.a + +CFLAGS += -O3 -Wall -Wextra +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -Wno-strict-prototypes +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux +CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +LDLIBS += -lrte_eal +LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_bus_mlx5_pci_version.map + +SRCS-y += mlx5_pci_bus.c + +# DEBUG which is usually provided on the command-line may enable +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. +ifeq ($(DEBUG),1) +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y +endif + +# +# Export include files +# +SYMLINK-y-include += rte_bus_mlx5_pci.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build new file mode 100644 index 000000000..64a17cbad --- /dev/null +++ b/drivers/bus/mlx5_pci/meson.build @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020 Mellanox Technologies Ltd + +deps += ['pci', 'bus_pci', 'common_mlx5'] +install_headers('rte_bus_mlx5_pci.h') +sources = files('mlx5_pci_bus.c') + +cflags_options = [ + '-std=c11', + '-Wno-strict-prototypes', + '-D_BSD_SOURCE', + '-D_DEFAULT_SOURCE', + '-D_XOPEN_SOURCE=600' +] +foreach option:cflags_options + if cc.has_argument(option) + cflags += option + endif +endforeach diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c new file mode 100644 index 000000000..66db3c7b0 --- /dev/null +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#include "rte_bus_mlx5_pci.h" + +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = + TAILQ_HEAD_INITIALIZER(drv_list); + +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) +{ + TAILQ_INSERT_TAIL(&drv_list, driver, next); +} diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h new file mode 100644 index 000000000..9f8d22e2b --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef _RTE_BUS_MLX5_PCI_H_ +#define _RTE_BUS_MLX5_PCI_H_ + +/** + * @file + * + * RTE Mellanox PCI Bus Interface + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) + * devices. This bus enables creating such multiple class of devices on a + * single PCI device by allowing to bind multiple class specific device + * driver to attach to mlx5_pci bus driver. + * + * ----------- ------------ ----------------- + * | mlx5 | | mlx5 | | mlx5 | + * | net pmd | | vdpa pmd | | new class pmd | + * ----------- ------------ ----------------- + * \ | / + * \ | / + * \ ------------- / + * \______| mlx5 |_____ / + * | pci bus | + * ------------- + * | + * ----------- + * | mlx5 | + * | pci dev | + * ----------- + * + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 pci bus driver is cental place that validates supported + * class combinations. + */ + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +#include <rte_pci.h> +#include <rte_bus_pci.h> + +#include <mlx5_common.h> + +/** + * A structure describing a mlx5 pci driver. + */ +struct rte_mlx5_pci_driver { + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ + uint32_t dev_class; /**< Class of this driver, enum mlx5_class */ + TAILQ_ENTRY(rte_mlx5_pci_driver) next; +}; + +/** + * Register a mlx5_pci device driver. + * + * @param driver + * A pointer to a rte_mlx5_pci_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver); + +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \ + static const char *mlx5_pci_drvinit_fn_ ## nm; \ + RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \ + { \ + (drv).driver.name = RTE_STR(nm); \ + rte_mlx5_pci_driver_register(&drv); \ + } \ + RTE_PMD_EXPORT_NAME(nm, __COUNTER__) + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _RTE_BUS_MLX5_PCI_H_ */ diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map new file mode 100644 index 000000000..4cfd3db10 --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map @@ -0,0 +1,5 @@ +INTERNAL { + global: + + rte_mlx5_pci_driver_register; +}; -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 07/10] bus/mlx5_pci: register a PCI driver 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (5 preceding siblings ...) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 06/10] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 12:53 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 08/10] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit ` (2 subsequent siblings) 9 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Create a mlx5 bus driver framework for invoking drivers of multiple classes who have registered with the mlx5_pci bus driver. Validate user class arguments for supported class combinations. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Addressed comments from Asaf - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Addressed comments from Matan. - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block. - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Maintaining class driver list to keep load/unload order symmetric and mirror of each other. - Deriving drv_flags based on the class drivers - Using PCI address comparision helper instead of pointer comparision - Fixed alignment for id_table - Continue to probe_err if device is already probed - Perform dma map on best effort basis for all supported drivers - Removed drv_flags check - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Address comments from Thomas and Gaetan - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - USe uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options --- drivers/bus/mlx5_pci/Makefile | 2 + drivers/bus/mlx5_pci/meson.build | 2 +- drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++ 3 files changed, 511 insertions(+), 1 deletion(-) diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile index de4ccd83f..1a005ee32 100644 --- a/drivers/bus/mlx5_pci/Makefile +++ b/drivers/bus/mlx5_pci/Makefile @@ -15,7 +15,9 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +CFLAGS += -D_DEFAULT_SOURCE LDLIBS += -lrte_eal +LDLIBS += -lrte_kvargs LDLIBS += -lrte_common_mlx5 LDLIBS += -lrte_pci -lrte_bus_pci diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build index 64a17cbad..0532a9dfd 100644 --- a/drivers/bus/mlx5_pci/meson.build +++ b/drivers/bus/mlx5_pci/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Mellanox Technologies Ltd -deps += ['pci', 'bus_pci', 'common_mlx5'] +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs'] install_headers('rte_bus_mlx5_pci.h') sources = files('mlx5_pci_bus.c') diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c index 66db3c7b0..7344c2083 100644 --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -2,13 +2,521 @@ * Copyright 2020 Mellanox Technologies, Ltd */ +#include <stdlib.h> +#include <rte_malloc.h> #include "rte_bus_mlx5_pci.h" +#include <mlx5_common_utils.h> +struct mlx5_pci_device { + struct rte_pci_device *pci_dev; + TAILQ_ENTRY(mlx5_pci_device) next; + uint32_t classes_loaded; +}; + +/* Head of list of class drivers. */ static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = TAILQ_HEAD_INITIALIZER(drv_list); +/* Head of mlx5 pci devices. */ +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = + TAILQ_HEAD_INITIALIZER(devices_list); + +static const struct { + const char *name; + unsigned int dev_class; +} mlx5_classes[] = { + { .name = "vdpa", .dev_class = MLX5_CLASS_VDPA }, + { .name = "net", .dev_class = MLX5_CLASS_NET }, +}; + +static const unsigned int mlx5_class_combinations[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + /* New class combination should be added here. + * For example a new multi class device combination + * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. + */ +}; + +static int +class_name_to_value(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].dev_class; + } + return -EINVAL; +} + +static struct rte_mlx5_pci_driver * +class_driver_get(uint32_t class) +{ + struct rte_mlx5_pci_driver *driver; + + TAILQ_FOREACH(driver, &drv_list, next) { + if (driver->dev_class == class) + return driver; + } + return NULL; +} + +static int +bus_cmdline_options_handler(__rte_unused const char *key, + const char *class_names, void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name. Multiple + * class key,value is supplied as class=net:vdpa:foo:bar. + */ + found = strsep(&nstr, ":"); + if (!found) + continue; + /* Check if its a valid class. */ + class_val = class_name_to_value(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s." + " Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, + &ret); + rte_kvargs_free(kvlist); + return ret; +} + void rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) { TAILQ_INSERT_TAIL(&drv_list, driver, next); } + +static bool +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* Check if device's ids match the class driver's ids. */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + return true; + } + return false; +} + +static int +is_valid_class_combination(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination. */ + for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { + if (mlx5_class_combinations[i] == user_classes) + return 0; + } + /* Not found any valid class combination. */ + return -EINVAL; +} + +static struct mlx5_pci_device * +pci_to_mlx5_device(const struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + + TAILQ_FOREACH(dev, &devices_list, next) { + if (dev->pci_dev == pci_dev) + return dev; + } + return NULL; +} + +static bool +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) +{ + return (device->classes_loaded & class) ? true : false; +} + +static void +dev_release(struct mlx5_pci_device *dev) +{ + TAILQ_REMOVE(&devices_list, dev, next); + rte_free(dev); +} + +static int +class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) +{ + struct rte_mlx5_pci_driver *driver; + int local_ret = -ENODEV; + unsigned int i = 0; + int ret = 0; + + enabled_classes &= dev->classes_loaded; + while (enabled_classes) { + driver = class_driver_get(RTE_BIT(i)); + if (driver) { + local_ret = driver->pci_driver.remove(dev->pci_dev); + if (!local_ret) + dev->classes_loaded &= ~RTE_BIT(i); + else if (ret == 0) + ret = local_ret; + } + enabled_classes &= ~RTE_BIT(i); + i++; + } + if (local_ret) + ret = local_ret; + return ret; +} + +static int +class_drivers_probe(struct mlx5_pci_device *dev, + struct rte_pci_driver *pci_drv, + struct rte_pci_device *pci_dev, uint32_t user_classes) +{ + struct rte_mlx5_pci_driver *driver; + uint32_t enabled_classes = 0; + bool already_loaded; + int ret; + + TAILQ_FOREACH(driver, &drv_list, next) { + if ((driver->dev_class & user_classes) == 0) + continue; + if (!mlx5_bus_match(driver, pci_dev)) + continue; + already_loaded = dev->classes_loaded & driver->dev_class; + if (already_loaded && + !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { + DRV_LOG(ERR, "Device %s is already probed\n", + pci_dev->device.name); + ret = -EEXIST; + goto probe_err; + } + ret = driver->pci_driver.probe(pci_drv, pci_dev); + if (ret < 0) { + DRV_LOG(ERR, "Failed to load class driver = %s.\n", + driver->pci_driver.driver.name); + goto probe_err; + } + enabled_classes |= driver->dev_class; + } + dev->classes_loaded |= enabled_classes; + return 0; +probe_err: + /* Only unload drivers which are enabled which were enabled + * in this probe instance. + */ + class_drivers_remove(dev, enabled_classes); + return ret; +} + +/** + * DPDK callback to register to probe multiple PCI class devices. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + uint32_t user_classes = 0; + bool new_device = false; + int ret; + + ret = parse_class_options(pci_dev->device.devargs); + if (ret < 0) + return ret; + user_classes = ret; + if (user_classes) { + /* Validate combination here. */ + ret = is_valid_class_combination(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); + return ret; + } + } else { + /* Default to net class. */ + user_classes = MLX5_CLASS_NET; + } + dev = pci_to_mlx5_device(pci_dev); + if (!dev) { + dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); + if (!dev) + return -ENOMEM; + dev->pci_dev = pci_dev; + TAILQ_INSERT_HEAD(&devices_list, dev, next); + new_device = true; + } + ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes); + if (ret) + goto class_err; + return 0; +class_err: + if (new_device) + dev_release(dev); + return ret; +} + +/** + * DPDK callback to remove one or more class devices for a PCI device. + * + * This function removes all class devices belong to a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_bus_pci_remove(struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + /* Matching device found, cleanup and unload class drivers. */ + ret = class_drivers_remove(dev, dev->classes_loaded); + if (!ret) + dev_release(dev); + return ret; +} + +static int +mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + struct rte_mlx5_pci_driver *temp; + struct mlx5_pci_device *dev; + int ret = -EINVAL; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_map) { + ret = class->pci_driver.dma_map(pci_dev, addr, + iova, len); + if (ret) + goto map_err; + } + } + return ret; +map_err: + TAILQ_FOREACH(temp, &drv_list, next) { + if (temp == class) + break; + if (device_class_enabled(dev, temp->dev_class) && + temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) + temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + struct mlx5_pci_device *dev; + int local_ret = -EINVAL; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + ret = 0; + /* There is no unmap error recovery in current implementation. */ + TAILQ_FOREACH(class, &drv_list, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_unmap) { + local_ret = class->pci_driver.dma_unmap(pci_dev, addr, + iova, len); + if (local_ret && (ret == 0)) + ret = local_ret; + } + } + if (local_ret) + ret = local_ret; + return ret; +} + +/* PCI ID table is build dynamically based on registered mlx5 + * class drivers. + */ +static struct rte_pci_id *mlx5_bus_pci_id_map; + +static int +pci_id_table_size_get(const struct rte_pci_id *id_table) +{ + int table_size = 0; + + for (; id_table->vendor_id != 0; id_table++) + table_size++; + return table_size; +} + +static bool +pci_id_exists(const struct rte_pci_id *id, int next_idx) +{ + int current_size = next_idx - 1; + int i; + + for (i = 0; i < current_size; i++) { + if (id->device_id == mlx5_bus_pci_id_map[i].device_id && + id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id && + id->subsystem_vendor_id == + mlx5_bus_pci_id_map[i].subsystem_vendor_id && + id->subsystem_device_id == + mlx5_bus_pci_id_map[i].subsystem_device_id) + return true; + } + return false; +} + +static void +pci_id_insert(const struct rte_pci_id *id_table, int *next_idx) +{ + for (; id_table->vendor_id != 0; id_table++) { + if (!pci_id_exists(id_table, *next_idx)) { + /* New entry; add to the table. */ + mlx5_bus_pci_id_map[*next_idx] = *id_table; + (*next_idx)++; + } + } +} + +static int +pci_ids_table_build(void) +{ + struct rte_mlx5_pci_driver *first_driver; + struct rte_mlx5_pci_driver *driver; + const struct rte_pci_id *id_table; + int num_ids = 0; + int i = 0; + + TAILQ_FOREACH(driver, &drv_list, next) + num_ids += pci_id_table_size_get(driver->pci_driver.id_table); + if (!num_ids) + return -ENODEV; + /* Increase size by one for the termination entry of vendor_id = 0. */ + num_ids += 1; + mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map)); + if (!mlx5_bus_pci_id_map) + return -ENOMEM; + first_driver = TAILQ_FIRST(&drv_list); + /* Copy the first class driver's ID table. */ + for (id_table = first_driver->pci_driver.id_table; + id_table->vendor_id != 0; id_table++, i++) + mlx5_bus_pci_id_map[i] = *id_table; + TAILQ_FOREACH(driver, &drv_list, next) { + /* We already added first driver; skip it. */ + if (driver == first_driver) + continue; + pci_id_insert(driver->pci_driver.id_table, &i); + } + mlx5_bus_pci_id_map[i].vendor_id = 0; + return 0; +} + +static bool mlx5_bus_registered; +static struct rte_pci_driver mlx5_bus_driver = { + .driver = { + .name = "mlx5_bus_pci", + }, + .probe = mlx5_bus_pci_probe, + .remove = mlx5_bus_pci_remove, + .dma_map = mlx5_bus_pci_dma_map, + .dma_unmap = mlx5_bus_pci_dma_unmap, +}; + +RTE_INIT(mlx5_bus_pci) +{ + struct rte_mlx5_pci_driver *class; + int ret; + + ret = pci_ids_table_build(); + if (ret) + return; + TAILQ_FOREACH(class, &drv_list, next) + mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags; + mlx5_bus_driver.id_table = mlx5_bus_pci_id_map; + rte_pci_register(&mlx5_bus_driver); + mlx5_bus_registered = true; +} + +RTE_FINI(mlx5_bus_pci_finish) +{ + if (mlx5_bus_registered) + rte_pci_unregister(&mlx5_bus_driver); + if (mlx5_bus_pci_id_map) + free(mlx5_bus_pci_id_map); +} +RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v3 07/10] bus/mlx5_pci: register a PCI driver 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 07/10] bus/mlx5_pci: register a PCI driver Parav Pandit @ 2020-07-03 12:53 ` Parav Pandit 0 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 12:53 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: Thomas Monjalon, Ori Kam, Matan Azrad > From: Parav Pandit <parav@mellanox.com> > Sent: Friday, July 3, 2020 2:43 PM > > Create a mlx5 bus driver framework for invoking drivers of multiple classes > who have registered with the mlx5_pci bus driver. [..] > diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c > b/drivers/bus/mlx5_pci/mlx5_pci_bus.c > index 66db3c7b0..7344c2083 100644 > --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c > +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c > @@ -2,13 +2,521 @@ > * Copyright 2020 Mellanox Technologies, Ltd > */ > +static int > +mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, > + uint64_t iova, size_t len) > +{ > + struct rte_mlx5_pci_driver *class; > + struct rte_mlx5_pci_driver *temp; > + struct mlx5_pci_device *dev; > + int ret = -EINVAL; > + > + dev = pci_to_mlx5_device(pci_dev); > + if (!dev) > + return -ENODEV; > + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, > next) { > + if (device_class_enabled(dev, class->dev_class) && > + class->pci_driver.dma_map) { > + ret = class->pci_driver.dma_map(pci_dev, addr, > + iova, len); > + if (ret) > + goto map_err; > + } > + } > + return ret; > +map_err: > + TAILQ_FOREACH(temp, &drv_list, next) { I made a mistake in reusing the logic of the dma_unmap() callback here. This loop needs to follow same sequence as that what dma_map() does. I will send the fix. Will wait for some time to gather if there are comments. > + if (temp == class) > + break; > + if (device_class_enabled(dev, temp->dev_class) && > + temp->pci_driver.dma_map && temp- > >pci_driver.dma_unmap) > + temp->pci_driver.dma_unmap(pci_dev, addr, iova, > len); > + } > + return ret; > +} ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 08/10] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (6 preceding siblings ...) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 07/10] bus/mlx5_pci: register a PCI driver Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 09/10] common/mlx5: remove class checks from individual driver Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 10/10] maintainers: add maintainers for mlx5 pci bus Parav Pandit 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Enable class driver to match with the mlx5 pci devices. Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class driver. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Avoid static table v1->v2: - Migreate API from rte_driver to rte_pci_driver --- drivers/bus/Makefile | 3 +++ drivers/net/mlx5/Makefile | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 1 - drivers/net/mlx5/linux/mlx5_os.h | 1 + drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 24 ++++++++++++++---------- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 ++- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++---------- mk/rte.app.mk | 1 + 11 files changed, 38 insertions(+), 26 deletions(-) diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index cea3b55e6..f96caa5ec 100644 --- a/drivers/bus/Makefile +++ b/drivers/bus/Makefile @@ -9,6 +9,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) +DIRS-y += mlx5_pci +endif DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index a458402dc..1cea7cd07 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_bus_mlx5_pci LDLIBS += -lm LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 2dc57b20e..b6042b7ef 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1321,7 +1321,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); return -rte_errno; } - MLX5_ASSERT(pci_drv == &mlx5_driver); errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 31add3988..695722520 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,4 +15,5 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) + #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index e95ce0267..26699a79b 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -8,7 +8,7 @@ if not (is_linux or is_windows) subdir_done() endif -deps += ['hash', 'common_mlx5'] +deps += ['hash', 'common_mlx5', 'bus_mlx5_pci'] sources = files( 'mlx5.c', 'mlx5_ethdev.c', diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index bb10c63bb..2270055fe 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -34,6 +34,7 @@ #include <rte_spinlock.h> #include <rte_string_fns.h> #include <rte_alarm.h> +#include <rte_bus_mlx5_pci.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h> @@ -1892,16 +1893,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -struct rte_pci_driver mlx5_driver = { - .driver = { - .name = MLX5_DRIVER_NAME +static struct rte_mlx5_pci_driver mlx5_driver = { + .dev_class = MLX5_CLASS_NET, + .pci_driver = { + .driver = { + .name = MLX5_DRIVER_NAME, + }, + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_dma_map, + .dma_unmap = mlx5_dma_unmap, + .drv_flags = PCI_DRV_FLAGS, }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, }; /** @@ -1919,7 +1923,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - rte_pci_register(&mlx5_driver); + rte_mlx5_pci_driver_register(&mlx5_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 46e66eb1c..df7b5eb87 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -129,7 +129,6 @@ struct mlx5_local_data { }; extern struct mlx5_shared_data *mlx5_shared_data; -extern struct rte_pci_driver mlx5_driver; /* Dev ops structs */ extern const struct eth_dev_ops mlx5_os_dev_ops; diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index 8a1c2eab5..9bb5dfdbf 100644 --- a/drivers/vdpa/mlx5/Makefile +++ b/drivers/vdpa/mlx5/Makefile @@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa CFLAGS += -I$(RTE_SDK)/lib/librte_sched CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched LDLIBS += -pthread # A few warnings cannot be avoided in external headers. diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build index 2963aad71..f4175c34e 100644 --- a/drivers/vdpa/mlx5/meson.build +++ b/drivers/vdpa/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_vdpa' -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched'] +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched'] sources = files( 'mlx5_vdpa.c', 'mlx5_vdpa_mem.c', diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index f043166ef..09c9cb935 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -11,7 +11,7 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> +#include <rte_bus_mlx5_pci.h> #include <rte_pci.h> #include <rte_string_fns.h> @@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) } /** - * DPDK callback to register a PCI device. + * DPDK callback to register a mlx5 PCI bus device. * * This function spawns vdpa device out of a given PCI device. * @@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_vdpa_driver = { - .driver = { - .name = "mlx5_vdpa", +static struct rte_mlx5_pci_driver mlx5_vdpa_driver = { + .dev_class = MLX5_CLASS_VDPA, + .pci_driver = { + .driver = { + .name = "mlx5_vdpa", + }, + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_pci_probe, + .remove = mlx5_vdpa_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, }; /** @@ -849,7 +852,7 @@ RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) if (mlx5_vdpa_logtype >= 0) rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE); if (mlx5_glue) - rte_pci_register(&mlx5_vdpa_driver); + rte_mlx5_pci_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 0ce8cf541..d7c1d90e4 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF) += -lrte_pmd_memif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) _LDLIBS-y += -lrte_common_mlx5 +_LDLIBS-y += -lrte_bus_mlx5_pci endif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD) += -lrte_pmd_mlx5_vdpa -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 09/10] common/mlx5: remove class checks from individual driver 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (7 preceding siblings ...) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 08/10] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 10/10] maintainers: add maintainers for mlx5 pci bus Parav Pandit 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Now that mlx5_pci bus does the check for enabled classes and performs probe(), remove() of associated classes, individual class driver doesn't need to check if other driver is enabled. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Removed empty line v1->v2: - New patch --- drivers/common/mlx5/mlx5_common.c | 37 ------------------- drivers/common/mlx5/mlx5_common.h | 2 - .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/linux/mlx5_os.c | 5 --- drivers/net/mlx5/linux/mlx5_os.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 5 --- 6 files changed, 52 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0ce5e4db1..771e046c1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue; uint8_t haswell_broadwell_cpu; -static int -mlx5_class_check_handler(__rte_unused const char *key, const char *value, - void *opaque) -{ - enum mlx5_class *ret = opaque; - - if (strcmp(value, "vdpa") == 0) { - *ret = MLX5_CLASS_VDPA; - } else if (strcmp(value, "net") == 0) { - *ret = MLX5_CLASS_NET; - } else { - DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device" - " class argument setting?", value); - *ret = MLX5_CLASS_INVALID; - } - return 0; -} - -enum mlx5_class -mlx5_class_get(struct rte_devargs *devargs) -{ - struct rte_kvargs *kvlist; - const char *key = MLX5_CLASS_ARG_NAME; - enum mlx5_class ret = MLX5_CLASS_NET; - - if (devargs == NULL) - return ret; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return ret; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret); - rte_kvargs_free(kvlist); - return ret; -} - - /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 62861af05..ca66c4033 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -243,8 +243,6 @@ struct mlx5_klm { LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page); -__rte_internal -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs); __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index ae57ebdba..01b4358a0 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,8 +1,6 @@ INTERNAL { global: - mlx5_class_get; - mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index b6042b7ef..2bd2b664d 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1308,11 +1308,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_dev_config dev_config; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } if (rte_eal_process_type() == RTE_PROC_PRIMARY) mlx5_pmd_socket_init(); ret = mlx5_init_once(); diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 695722520..31add3988 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,5 +15,4 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) - #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 09c9cb935..41d69e8a8 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_hca_attr attr; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr); if (!ibv) { DRV_LOG(ERR, "No matching IB device for PCI slot " -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v3 10/10] maintainers: add maintainers for mlx5 pci bus 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (8 preceding siblings ...) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 09/10] common/mlx5: remove class checks from individual driver Parav Pandit @ 2020-07-03 9:12 ` Parav Pandit 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 9:12 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Added maintainers for new mlx5 specific mlx5_pci bus. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- MAINTAINERS | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 53a5e9a9e..e3fec55ca 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -517,6 +517,11 @@ Intel FPGA bus M: Rosen Xu <rosen.xu@intel.com> F: drivers/bus/ifpga/ +Melllanox mlx5 PCI bus driver +M: Parav Pandit <parav@mellaox.com> +M: Matan Azrad <matan@mellanox.com> +F: drivers/bus/mlx5_pci + NXP buses M: Hemant Agrawal <hemant.agrawal@nxp.com> M: Sachin Saxena <sachin.saxena@nxp.com> -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (2 preceding siblings ...) 2020-07-03 9:12 ` [dpdk-dev] [PATCH v3 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting value for bit Parav Pandit ` (9 more replies) 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (7 subsequent siblings) 11 siblings, 10 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit This series introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, a new mlx5 PCI bus is created. This bus allows binding multiple class drivers (such as net, vdpa, regex(future)) to bind to the mlx5 PCI bus driver. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Introduces new RTE constructor priority for common initialization Patch-3 Fixes compilation error Patch-4 Define mlx5 class as bit fields Patch-5 Uses new RTE common priority Patch-6 Adds mlx5 PCI bus Patch-7 Implements a mlx5 PCI bus driver Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus API instead of rte PCI bus API Patch-9 Removed class check code as its already part of the bus now Patch-10 Add maintainers for the new bus Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ ------------- / \______| mlx5 |_____ / | pci bus | ------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD defines its specific PCI ID table and mlx5 bus driver probes matching class drivers. - mlx5 pci bus driver is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI bus driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci bus, a common driver is implemented which exposes class registration API. However, bus model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common bus driver. It also eliminates peer blind zone problem as bottom pci bus layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:vdpa ./testpmd -w <PCI BDF>,class=vdpa In future, ./testpmd -w <PCI BDF>,class=net:regex Changelog: v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (10): eal: introduce macros for getting value for bit eal: introduce RTE common initialization level common/mlx5: fix empty input style in glue wrappers common/mlx5: change mlx5 class enum values as bits common/mlx5: use common rte priority bus/mlx5_pci: add mlx5 PCI bus bus/mlx5_pci: register a PCI driver bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver common/mlx5: remove class checks from individual driver maintainers: add maintainers for mlx5 pci bus MAINTAINERS | 5 + drivers/bus/Makefile | 3 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 40 ++ drivers/bus/mlx5_pci/meson.build | 19 + drivers/bus/mlx5_pci/mlx5_pci_bus.c | 522 ++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/mlx5_common.c | 39 +- drivers/common/mlx5/mlx5_common.h | 7 +- .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 26 +- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 30 +- lib/librte_eal/include/rte_bitops.h | 2 + lib/librte_eal/include/rte_common.h | 1 + mk/rte.app.mk | 1 + 23 files changed, 724 insertions(+), 85 deletions(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting value for bit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-06 10:53 ` [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit Morten Brørup 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 02/10] eal: introduce RTE common initialization level Parav Pandit ` (8 subsequent siblings) 9 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v1->v2: - Addressed comments from Thomas and Gaten. - Avoided new file, added macro to rte_bitops.h --- lib/librte_eal/include/rte_bitops.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..d72c7cd93 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,8 @@ #include <rte_debug.h> #include <rte_compat.h> +#define RTE_BIT(bit_num) (1UL << (bit_num)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting value for bit Parav Pandit @ 2020-07-06 10:53 ` Morten Brørup 2020-07-07 11:38 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: Morten Brørup @ 2020-07-06 10:53 UTC (permalink / raw) To: Parav Pandit, dev, grive, ferruh.yigit; +Cc: thomas, orika, matan > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Parav Pandit > Sent: Friday, July 3, 2020 3:47 PM > > There are several drivers which duplicate bit generation macro. > Introduce a generic bit macros so that such drivers avoid redefining > same in multiple drivers. > > Signed-off-by: Parav Pandit <parav@mellanox.com> > Acked-by: Matan Azrad <matan@mellanox.com> > --- > Changelog: > v1->v2: > - Addressed comments from Thomas and Gaten. > - Avoided new file, added macro to rte_bitops.h > --- > lib/librte_eal/include/rte_bitops.h | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/lib/librte_eal/include/rte_bitops.h > b/lib/librte_eal/include/rte_bitops.h > index 740927f3b..d72c7cd93 100644 > --- a/lib/librte_eal/include/rte_bitops.h > +++ b/lib/librte_eal/include/rte_bitops.h > @@ -17,6 +17,8 @@ > #include <rte_debug.h> > #include <rte_compat.h> > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) Is the return value 32 or 64 bit, or is intended to depend on the target architecture? Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of 1UL, if you want a specific size. It could be a static inline __attribute__((__pure__)) function instead of a macro, but it's not important for me. The macro/function needs a description for the documentation. I'm also concerned about the name of the macro being too generic. But the effort of changing all the drivers where it is being used already could be too big if the name changes too. And the macro/function is new, so shouldn't it - in theory - be marked as experimental? > + > /*------------------------ 32-bit relaxed operations ----------------- > -------*/ > > /** > -- > 2.26.2 > ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit 2020-07-06 10:53 ` [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit Morten Brørup @ 2020-07-07 11:38 ` Parav Pandit 2020-07-07 12:13 ` Thomas Monjalon 0 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-07 11:38 UTC (permalink / raw) To: Morten Brørup, dev, grive, ferruh.yigit Cc: Thomas Monjalon, Ori Kam, Matan Azrad > From: Morten Brørup <mb@smartsharesystems.com> > Sent: Monday, July 6, 2020 4:24 PM > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Parav Pandit > > Sent: Friday, July 3, 2020 3:47 PM > > > > There are several drivers which duplicate bit generation macro. > > Introduce a generic bit macros so that such drivers avoid redefining > > same in multiple drivers. > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > Acked-by: Matan Azrad <matan@mellanox.com> > > --- > > Changelog: > > v1->v2: > > - Addressed comments from Thomas and Gaten. > > - Avoided new file, added macro to rte_bitops.h > > --- > > lib/librte_eal/include/rte_bitops.h | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/lib/librte_eal/include/rte_bitops.h > > b/lib/librte_eal/include/rte_bitops.h > > index 740927f3b..d72c7cd93 100644 > > --- a/lib/librte_eal/include/rte_bitops.h > > +++ b/lib/librte_eal/include/rte_bitops.h > > @@ -17,6 +17,8 @@ > > #include <rte_debug.h> > > #include <rte_compat.h> > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > Is the return value 32 or 64 bit, or is intended to depend on the target > architecture? > It should be 64-bit. > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of 1UL, if you > want a specific size. > Will do UINT64_C(1). > It could be a static inline __attribute__((__pure__)) function instead of a macro, > but it's not important for me. > > The macro/function needs a description for the documentation. > In this header file or outside? > I'm also concerned about the name of the macro being too generic. But the > effort of changing all the drivers where it is being used already could be too big > if the name changes too. > Right. Currently drivers have generic name as BIT(). Close to 3000 entries. So doing at RTE_BIT to match other rte_ APIs. Drivers can slowly migrate at their pace to this one. > And the macro/function is new, so shouldn't it - in theory - be marked as > experimental? How to mark a macro as experimental? ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit 2020-07-07 11:38 ` Parav Pandit @ 2020-07-07 12:13 ` Thomas Monjalon 2020-07-07 12:40 ` Morten Brørup 0 siblings, 1 reply; 193+ messages in thread From: Thomas Monjalon @ 2020-07-07 12:13 UTC (permalink / raw) To: Parav Pandit Cc: Morten Brørup, dev, grive, ferruh.yigit, Ori Kam, Matan Azrad 07/07/2020 13:38, Parav Pandit: > From: Morten Brørup <mb@smartsharesystems.com> > > From: Parav Pandit > > > --- a/lib/librte_eal/include/rte_bitops.h > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > @@ -17,6 +17,8 @@ > > > #include <rte_debug.h> > > > #include <rte_compat.h> > > > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > > > Is the return value 32 or 64 bit, or is intended to depend on the target > > architecture? > > > It should be 64-bit. > > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of 1UL, if you > > want a specific size. > > > Will do UINT64_C(1). > > > It could be a static inline __attribute__((__pure__)) function instead of a macro, > > but it's not important for me. > > > > The macro/function needs a description for the documentation. > > > In this header file or outside? It is asked to add a doxygen comment. > > I'm also concerned about the name of the macro being too generic. But the > > effort of changing all the drivers where it is being used already could be too big > > if the name changes too. > > > Right. Currently drivers have generic name as BIT(). Close to 3000 entries. > So doing at RTE_BIT to match other rte_ APIs. > Drivers can slowly migrate at their pace to this one. > > > And the macro/function is new, so shouldn't it - in theory - be marked as > > experimental? > > How to mark a macro as experimental? A macro cannot be experimental. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit 2020-07-07 12:13 ` Thomas Monjalon @ 2020-07-07 12:40 ` Morten Brørup 2020-07-09 6:23 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: Morten Brørup @ 2020-07-07 12:40 UTC (permalink / raw) To: Thomas Monjalon, Parav Pandit, Joyce Kong Cc: dev, grive, ferruh.yigit, Ori Kam, Matan Azrad Adding Joyce Kong to this discussion as the rte_bitops maintainer. > From: Thomas Monjalon [mailto:thomas@monjalon.net] > Sent: Tuesday, July 7, 2020 2:13 PM > > 07/07/2020 13:38, Parav Pandit: > > From: Morten Brørup <mb@smartsharesystems.com> > > > From: Parav Pandit > > > > --- a/lib/librte_eal/include/rte_bitops.h > > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > > @@ -17,6 +17,8 @@ > > > > #include <rte_debug.h> > > > > #include <rte_compat.h> > > > > > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > > > > > Is the return value 32 or 64 bit, or is intended to depend on the > target > > > architecture? > > > > > It should be 64-bit. > > > > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of > 1UL, if you > > > want a specific size. > > > > > Will do UINT64_C(1). > > > > > It could be a static inline __attribute__((__pure__)) function > instead of a macro, > > > but it's not important for me. > > > > > > The macro/function needs a description for the documentation. > > > > > In this header file or outside? > > It is asked to add a doxygen comment. > > > > > I'm also concerned about the name of the macro being too generic. > But the > > > effort of changing all the drivers where it is being used already > could be too big > > > if the name changes too. > > > > > Right. Currently drivers have generic name as BIT(). Close to 3000 > entries. > > So doing at RTE_BIT to match other rte_ APIs. > > Drivers can slowly migrate at their pace to this one. > > > > > And the macro/function is new, so shouldn't it - in theory - be > marked as > > > experimental? > > > > How to mark a macro as experimental? > > A macro cannot be experimental. > OK. If the macro is given a future proof name, I guess it should be accepted. If we want boundary checks, I suggest a macro like: #define RTE_BIT64(nr) \ ({ \ typeof(nr) n = nr; \ RTE_BUILD_BUG_ON((n > 64) || (n < 0)); \ UINT64_C(1) << (n); \ }) Or a function: __rte_experimental static __rte_always_inline __attribute__((const)) uint64_t rte_bit64(const unsigned int nr) { RTE_ASSERT(nr < 64); return UINT64_C(1) << nr; } ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit 2020-07-07 12:40 ` Morten Brørup @ 2020-07-09 6:23 ` Parav Pandit 2020-07-09 7:15 ` Morten Brørup 0 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-09 6:23 UTC (permalink / raw) To: Morten Brørup, Thomas Monjalon, Joyce Kong Cc: dev, grive, ferruh.yigit, Ori Kam, Matan Azrad Hi Morten, > From: Morten Brørup <mb@smartsharesystems.com> > Sent: Tuesday, July 7, 2020 6:11 PM > Adding Joyce Kong to this discussion as the rte_bitops maintainer. > > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > Sent: Tuesday, July 7, 2020 2:13 PM > > > > 07/07/2020 13:38, Parav Pandit: > > > From: Morten Brørup <mb@smartsharesystems.com> > > > > From: Parav Pandit > > > > > --- a/lib/librte_eal/include/rte_bitops.h > > > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > > > @@ -17,6 +17,8 @@ > > > > > #include <rte_debug.h> > > > > > #include <rte_compat.h> > > > > > > > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > > > > > > > Is the return value 32 or 64 bit, or is intended to depend on the > > target > > > > architecture? > > > > > > > It should be 64-bit. > > > > > > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead of > > 1UL, if you > > > > want a specific size. > > > > > > > Will do UINT64_C(1). > > > > > > > It could be a static inline __attribute__((__pure__)) function > > instead of a macro, > > > > but it's not important for me. > > > > > > > > The macro/function needs a description for the documentation. > > > > > > > In this header file or outside? > > > > It is asked to add a doxygen comment. Ok. will add. > > > > > > > > I'm also concerned about the name of the macro being too generic. > > But the > > > > effort of changing all the drivers where it is being used already > > could be too big > > > > if the name changes too. > > > > > > > Right. Currently drivers have generic name as BIT(). Close to 3000 > > entries. > > > So doing at RTE_BIT to match other rte_ APIs. > > > Drivers can slowly migrate at their pace to this one. > > > > > > > And the macro/function is new, so shouldn't it - in theory - be > > marked as > > > > experimental? > > > > > > How to mark a macro as experimental? > > > > A macro cannot be experimental. > > > > OK. If the macro is given a future proof name, I guess it should be accepted. > > If we want boundary checks, I suggest a macro like: > > #define RTE_BIT64(nr) \ > ({ \ > typeof(nr) n = nr; \ > RTE_BUILD_BUG_ON((n > 64) || (n < 0)); \ > UINT64_C(1) << (n); \ > }) > Compiler doesn't like it. ../lib/librte_eal/include/rte_bitops.h:21:2: error: braced-group within expression allowed only inside a function ({ \ ^ > Or a function: > > __rte_experimental > static __rte_always_inline __attribute__((const)) uint64_t rte_bit64(const > unsigned int nr) { > RTE_ASSERT(nr < 64); > > return UINT64_C(1) << nr; > } > Value retrieved using this macro is used an enum. Don't see how a function call like above can solve it. For a below macro definition, compiler is already catching for negative value when RTE_BIT64(-1) is done, ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift count is negative [-Wshift-count-negative] #define RTE_BIT64(nr) (UINT64_C(1) << (nr)) And when RTE_BIT64(259) is done below error is done, ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift count >= width of type [-Wshift-count-overflow] #define RTE_BIT64(nr) (UINT64_C(1) << (nr)) So below definition is good covering all needed cases. #define RTE_BIT64(nr) (UINT64_C(1) << (nr)) ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit 2020-07-09 6:23 ` Parav Pandit @ 2020-07-09 7:15 ` Morten Brørup 2020-07-09 7:30 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: Morten Brørup @ 2020-07-09 7:15 UTC (permalink / raw) To: Parav Pandit, Thomas Monjalon, Joyce Kong Cc: dev, grive, ferruh.yigit, Ori Kam, Matan Azrad > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Parav Pandit > Sent: Thursday, July 9, 2020 8:24 AM > > Hi Morten, > > > From: Morten Brørup <mb@smartsharesystems.com> > > Sent: Tuesday, July 7, 2020 6:11 PM > > > Adding Joyce Kong to this discussion as the rte_bitops maintainer. > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > > Sent: Tuesday, July 7, 2020 2:13 PM > > > > > > 07/07/2020 13:38, Parav Pandit: > > > > From: Morten Brørup <mb@smartsharesystems.com> > > > > > From: Parav Pandit > > > > > > --- a/lib/librte_eal/include/rte_bitops.h > > > > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > > > > @@ -17,6 +17,8 @@ > > > > > > #include <rte_debug.h> > > > > > > #include <rte_compat.h> > > > > > > > > > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > > > > > > > > > Is the return value 32 or 64 bit, or is intended to depend on > the > > > target > > > > > architecture? > > > > > > > > > It should be 64-bit. > > > > > > > > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead > of > > > 1UL, if you > > > > > want a specific size. > > > > > > > > > Will do UINT64_C(1). > > > > > > > > > It could be a static inline __attribute__((__pure__)) function > > > instead of a macro, > > > > > but it's not important for me. > > > > > > > > > > The macro/function needs a description for the documentation. > > > > > > > > > In this header file or outside? > > > > > > It is asked to add a doxygen comment. > Ok. will add. > > > > > > > > > > > > I'm also concerned about the name of the macro being too > generic. > > > But the > > > > > effort of changing all the drivers where it is being used > already > > > could be too big > > > > > if the name changes too. > > > > > > > > > Right. Currently drivers have generic name as BIT(). Close to > 3000 > > > entries. > > > > So doing at RTE_BIT to match other rte_ APIs. > > > > Drivers can slowly migrate at their pace to this one. > > > > > > > > > And the macro/function is new, so shouldn't it - in theory - be > > > marked as > > > > > experimental? > > > > > > > > How to mark a macro as experimental? > > > > > > A macro cannot be experimental. > > > > > > > OK. If the macro is given a future proof name, I guess it should be > accepted. > > > > If we want boundary checks, I suggest a macro like: > > > > #define RTE_BIT64(nr) \ > > ({ \ > > typeof(nr) n = nr; \ > > RTE_BUILD_BUG_ON((n > 64) || (n < 0)); \ > > UINT64_C(1) << (n); \ > > }) > > > Compiler doesn't like it. > > ../lib/librte_eal/include/rte_bitops.h:21:2: error: braced-group within > expression allowed only inside a function > ({ \ > ^ > > > Or a function: > > > > __rte_experimental > > static __rte_always_inline __attribute__((const)) uint64_t > rte_bit64(const > > unsigned int nr) { > > RTE_ASSERT(nr < 64); > > > > return UINT64_C(1) << nr; > > } > > > Value retrieved using this macro is used an enum. Don't see how a > function call like above can solve it. > > For a below macro definition, compiler is already catching for negative > value when RTE_BIT64(-1) is done, > > ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift count > is negative [-Wshift-count-negative] > #define RTE_BIT64(nr) (UINT64_C(1) << (nr)) > > And when RTE_BIT64(259) is done below error is done, > > ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift count > >= width of type [-Wshift-count-overflow] > #define RTE_BIT64(nr) (UINT64_C(1) << (nr)) > > So below definition is good covering all needed cases. > > #define RTE_BIT64(nr) (UINT64_C(1) << (nr)) Great. Then, when you have added a doxygen comment: Acked-by: Morten Brørup <mb@smartsharesystems.com> ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting valuefor bit 2020-07-09 7:15 ` Morten Brørup @ 2020-07-09 7:30 ` Parav Pandit 0 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:30 UTC (permalink / raw) To: Morten Brørup, Thomas Monjalon, Joyce Kong Cc: dev, grive, ferruh.yigit, Ori Kam, Matan Azrad > From: Morten Brørup <mb@smartsharesystems.com> > Sent: Thursday, July 9, 2020 12:46 PM > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Parav Pandit > > Sent: Thursday, July 9, 2020 8:24 AM > > > > Hi Morten, > > > > > From: Morten Brørup <mb@smartsharesystems.com> > > > Sent: Tuesday, July 7, 2020 6:11 PM > > > > > Adding Joyce Kong to this discussion as the rte_bitops maintainer. > > > > > > > From: Thomas Monjalon [mailto:thomas@monjalon.net] > > > > Sent: Tuesday, July 7, 2020 2:13 PM > > > > > > > > 07/07/2020 13:38, Parav Pandit: > > > > > From: Morten Brørup <mb@smartsharesystems.com> > > > > > > From: Parav Pandit > > > > > > > --- a/lib/librte_eal/include/rte_bitops.h > > > > > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > > > > > @@ -17,6 +17,8 @@ > > > > > > > #include <rte_debug.h> > > > > > > > #include <rte_compat.h> > > > > > > > > > > > > > > +#define RTE_BIT(bit_num) (1UL << (bit_num)) > > > > > > > > > > > > Is the return value 32 or 64 bit, or is intended to depend on > > the > > > > target > > > > > > architecture? > > > > > > > > > > > It should be 64-bit. > > > > > > > > > > > Please be explicit by using UINT32_C(1) or UINT64_C(1) instead > > of > > > > 1UL, if you > > > > > > want a specific size. > > > > > > > > > > > Will do UINT64_C(1). > > > > > > > > > > > It could be a static inline __attribute__((__pure__)) function > > > > instead of a macro, > > > > > > but it's not important for me. > > > > > > > > > > > > The macro/function needs a description for the documentation. > > > > > > > > > > > In this header file or outside? > > > > > > > > It is asked to add a doxygen comment. > > Ok. will add. > > > > > > > > > > > > > > > > I'm also concerned about the name of the macro being too > > generic. > > > > But the > > > > > > effort of changing all the drivers where it is being used > > already > > > > could be too big > > > > > > if the name changes too. > > > > > > > > > > > Right. Currently drivers have generic name as BIT(). Close to > > 3000 > > > > entries. > > > > > So doing at RTE_BIT to match other rte_ APIs. > > > > > Drivers can slowly migrate at their pace to this one. > > > > > > > > > > > And the macro/function is new, so shouldn't it - in theory - > > > > > > be > > > > marked as > > > > > > experimental? > > > > > > > > > > How to mark a macro as experimental? > > > > > > > > A macro cannot be experimental. > > > > > > > > > > OK. If the macro is given a future proof name, I guess it should be > > accepted. > > > > > > If we want boundary checks, I suggest a macro like: > > > > > > #define RTE_BIT64(nr) \ > > > ({ \ > > > typeof(nr) n = nr; \ > > > RTE_BUILD_BUG_ON((n > 64) || (n < 0)); \ > > > UINT64_C(1) << (n); \ > > > }) > > > > > Compiler doesn't like it. > > > > ../lib/librte_eal/include/rte_bitops.h:21:2: error: braced-group > > within expression allowed only inside a function > > ({ \ > > ^ > > > > > Or a function: > > > > > > __rte_experimental > > > static __rte_always_inline __attribute__((const)) uint64_t > > rte_bit64(const > > > unsigned int nr) { > > > RTE_ASSERT(nr < 64); > > > > > > return UINT64_C(1) << nr; > > > } > > > > > Value retrieved using this macro is used an enum. Don't see how a > > function call like above can solve it. > > > > For a below macro definition, compiler is already catching for > > negative value when RTE_BIT64(-1) is done, > > > > ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift > > count is negative [-Wshift-count-negative] #define RTE_BIT64(nr) > > (UINT64_C(1) << (nr)) > > > > And when RTE_BIT64(259) is done below error is done, > > > > ../lib/librte_eal/include/rte_bitops.h:36:36: warning: left shift > > count > > >= width of type [-Wshift-count-overflow] > > #define RTE_BIT64(nr) (UINT64_C(1) << (nr)) > > > > So below definition is good covering all needed cases. > > > > #define RTE_BIT64(nr) (UINT64_C(1) << (nr)) > > Great. Then, when you have added a doxygen comment: > > Acked-by: Morten Brørup <mb@smartsharesystems.com> Thanks Morten; adding it. ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 02/10] eal: introduce RTE common initialization level 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting value for bit Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 03/10] common/mlx5: fix empty input style in glue wrappers Parav Pandit ` (7 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Currently mlx5_common uses CLASS priority to initialize common code before initializing the PMD. However mlx5_common is not really a class, it is the pre-initialization code needed for the PMDs. In subsequent patch a needed initialization sequence is: (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Hence, introduce a new RTE priority level RTE_PRIO_COMMON which can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs for class driver initialization. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- lib/librte_eal/include/rte_common.h | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h index 0843ce69e..d4653ea77 100644 --- a/lib/librte_eal/include/rte_common.h +++ b/lib/librte_eal/include/rte_common.h @@ -126,6 +126,7 @@ typedef uint16_t unaligned_uint16_t; #define RTE_PRIORITY_LOG 101 #define RTE_PRIORITY_BUS 110 +#define RTE_PRIORITY_COMMON 119 #define RTE_PRIORITY_CLASS 120 #define RTE_PRIORITY_LAST 65535 -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 03/10] common/mlx5: fix empty input style in glue wrappers 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 01/10] eal: introduce macros for getting value for bit Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 02/10] eal: introduce RTE common initialization level Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 04/10] common/mlx5: change mlx5 class enum values as bits Parav Pandit ` (6 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Following two errors are reported when compiled with gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5). drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] Fix them by adding void data type in empty argument list. Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch") Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue") Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- drivers/common/mlx5/linux/mlx5_glue.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h index 069d8540c..e5e052a6a 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.h +++ b/drivers/common/mlx5/linux/mlx5_glue.h @@ -185,11 +185,11 @@ struct mlx5_glue { void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl); void *(*dr_create_flow_action_dest_port)(void *domain, uint32_t port); - void *(*dr_create_flow_action_drop)(); + void *(*dr_create_flow_action_drop)(void); void *(*dr_create_flow_action_push_vlan) (struct mlx5dv_dr_domain *domain, rte_be32_t vlan_tag); - void *(*dr_create_flow_action_pop_vlan)(); + void *(*dr_create_flow_action_pop_vlan)(void); void *(*dr_create_flow_tbl)(void *domain, uint32_t level); int (*dr_destroy_flow_tbl)(void *tbl); void *(*dr_create_domain)(struct ibv_context *ctx, -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 04/10] common/mlx5: change mlx5 class enum values as bits 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (2 preceding siblings ...) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 03/10] common/mlx5: fix empty input style in glue wrappers Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 05/10] common/mlx5: use common rte priority Parav Pandit ` (5 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit mlx5 PCI Device supports multiple classes of devices such as net, vdpa, and/or regex. To support these multiple classes, change mlx5_class to a bitmap values so that if users asks to enable multiple of them, all supported classes can be parsed. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v1->v2: - Rebasd due to removal previous patch --- drivers/common/mlx5/mlx5_common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index da01ffa1c..62861af05 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -13,6 +13,7 @@ #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h> +#include <rte_bitops.h> #include "mlx5_prm.h" #include "mlx5_devx_cmds.h" @@ -207,9 +208,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); #define MLX5_CLASS_ARG_NAME "class" enum mlx5_class { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, MLX5_CLASS_INVALID, + MLX5_CLASS_NET = RTE_BIT(0), + MLX5_CLASS_VDPA = RTE_BIT(1), }; #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */ -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 05/10] common/mlx5: use common rte priority 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (3 preceding siblings ...) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 04/10] common/mlx5: change mlx5 class enum values as bits Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 06/10] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit ` (4 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Use RTE_PRIO_COMMON for mlx5 common initialization and use RTE_PRIO_CLASS for mlx5 net, vdpa PMDs. This enables to do following initialization sequence. (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/mlx5_common.c | 2 +- drivers/net/mlx5/mlx5.c | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 693e2c68c..0ce5e4db1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) /** * Initialization routine for run-time dependency on glue library. */ -RTE_INIT_PRIO(mlx5_glue_init, CLASS) +RTE_INIT_PRIO(mlx5_glue_init, COMMON) { mlx5_glue_constructor(); } diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 07c6addd5..bb10c63bb 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1907,7 +1907,7 @@ struct rte_pci_driver mlx5_driver = { /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_pmd_init) +RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) { /* Initialize driver log type. */ mlx5_logtype = rte_log_register("pmd.net.mlx5"); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index dbd36ab0c..f043166ef 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -842,7 +842,7 @@ static struct rte_pci_driver mlx5_vdpa_driver = { /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_vdpa_init) +RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) { /* Initialize common log type. */ mlx5_vdpa_logtype = rte_log_register("pmd.vdpa.mlx5"); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 06/10] bus/mlx5_pci: add mlx5 PCI bus 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (4 preceding siblings ...) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 05/10] common/mlx5: use common rte priority Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 07/10] bus/mlx5_pci: register a PCI driver Parav Pandit ` (3 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single pci device. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Addressed comments from Thomas and Asaf - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address comments from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Changed alignedment to mlx5 specific aligment instead of standard DPDK - Using uint32_t instead of mlx5_class enum v1->v2: - Address comments from Thomas and Gaetan - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus --- drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 38 +++++++++ drivers/bus/mlx5_pci/meson.build | 19 +++++ drivers/bus/mlx5_pci/mlx5_pci_bus.c | 14 ++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++++++++++++++++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 ++ 6 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index 80de2d91d..b1381838d 100644 --- a/drivers/bus/meson.build +++ b/drivers/bus/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel Corporation -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus'] +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus'] std_deps = ['eal'] config_flag_fmt = 'RTE_LIBRTE_@0@_BUS' driver_name_fmt = 'rte_bus_@0@' diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile new file mode 100644 index 000000000..de4ccd83f --- /dev/null +++ b/drivers/bus/mlx5_pci/Makefile @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2020 Mellanox Technologies, Ltd + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_bus_mlx5_pci.a + +CFLAGS += -O3 -Wall -Wextra +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -Wno-strict-prototypes +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux +CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +LDLIBS += -lrte_eal +LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_bus_mlx5_pci_version.map + +SRCS-y += mlx5_pci_bus.c + +# DEBUG which is usually provided on the command-line may enable +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. +ifeq ($(DEBUG),1) +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y +endif + +# +# Export include files +# +SYMLINK-y-include += rte_bus_mlx5_pci.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build new file mode 100644 index 000000000..64a17cbad --- /dev/null +++ b/drivers/bus/mlx5_pci/meson.build @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020 Mellanox Technologies Ltd + +deps += ['pci', 'bus_pci', 'common_mlx5'] +install_headers('rte_bus_mlx5_pci.h') +sources = files('mlx5_pci_bus.c') + +cflags_options = [ + '-std=c11', + '-Wno-strict-prototypes', + '-D_BSD_SOURCE', + '-D_DEFAULT_SOURCE', + '-D_XOPEN_SOURCE=600' +] +foreach option:cflags_options + if cc.has_argument(option) + cflags += option + endif +endforeach diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c new file mode 100644 index 000000000..66db3c7b0 --- /dev/null +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#include "rte_bus_mlx5_pci.h" + +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = + TAILQ_HEAD_INITIALIZER(drv_list); + +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) +{ + TAILQ_INSERT_TAIL(&drv_list, driver, next); +} diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h new file mode 100644 index 000000000..9f8d22e2b --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef _RTE_BUS_MLX5_PCI_H_ +#define _RTE_BUS_MLX5_PCI_H_ + +/** + * @file + * + * RTE Mellanox PCI Bus Interface + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) + * devices. This bus enables creating such multiple class of devices on a + * single PCI device by allowing to bind multiple class specific device + * driver to attach to mlx5_pci bus driver. + * + * ----------- ------------ ----------------- + * | mlx5 | | mlx5 | | mlx5 | + * | net pmd | | vdpa pmd | | new class pmd | + * ----------- ------------ ----------------- + * \ | / + * \ | / + * \ ------------- / + * \______| mlx5 |_____ / + * | pci bus | + * ------------- + * | + * ----------- + * | mlx5 | + * | pci dev | + * ----------- + * + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 pci bus driver is cental place that validates supported + * class combinations. + */ + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +#include <rte_pci.h> +#include <rte_bus_pci.h> + +#include <mlx5_common.h> + +/** + * A structure describing a mlx5 pci driver. + */ +struct rte_mlx5_pci_driver { + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ + uint32_t dev_class; /**< Class of this driver, enum mlx5_class */ + TAILQ_ENTRY(rte_mlx5_pci_driver) next; +}; + +/** + * Register a mlx5_pci device driver. + * + * @param driver + * A pointer to a rte_mlx5_pci_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver); + +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \ + static const char *mlx5_pci_drvinit_fn_ ## nm; \ + RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \ + { \ + (drv).driver.name = RTE_STR(nm); \ + rte_mlx5_pci_driver_register(&drv); \ + } \ + RTE_PMD_EXPORT_NAME(nm, __COUNTER__) + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _RTE_BUS_MLX5_PCI_H_ */ diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map new file mode 100644 index 000000000..4cfd3db10 --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map @@ -0,0 +1,5 @@ +INTERNAL { + global: + + rte_mlx5_pci_driver_register; +}; -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 07/10] bus/mlx5_pci: register a PCI driver 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (5 preceding siblings ...) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 06/10] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 08/10] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit ` (2 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Create a mlx5 bus driver framework for invoking drivers of multiple classes who have registered with the mlx5_pci bus driver. Validate user class arguments for supported class combinations. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Addressed comments from Asaf - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Addressed comments from Matan. - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block. - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Maintaining class driver list to keep load/unload order symmetric and mirror of each other. - Deriving drv_flags based on the class drivers - Using PCI address comparision helper instead of pointer comparision - Fixed alignment for id_table - Continue to probe_err if device is already probed - Perform dma map on best effort basis for all supported drivers - Removed drv_flags check - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Address comments from Thomas and Gaetan - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - USe uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options --- drivers/bus/mlx5_pci/Makefile | 2 + drivers/bus/mlx5_pci/meson.build | 2 +- drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++ 3 files changed, 511 insertions(+), 1 deletion(-) diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile index de4ccd83f..1a005ee32 100644 --- a/drivers/bus/mlx5_pci/Makefile +++ b/drivers/bus/mlx5_pci/Makefile @@ -15,7 +15,9 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +CFLAGS += -D_DEFAULT_SOURCE LDLIBS += -lrte_eal +LDLIBS += -lrte_kvargs LDLIBS += -lrte_common_mlx5 LDLIBS += -lrte_pci -lrte_bus_pci diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build index 64a17cbad..0532a9dfd 100644 --- a/drivers/bus/mlx5_pci/meson.build +++ b/drivers/bus/mlx5_pci/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Mellanox Technologies Ltd -deps += ['pci', 'bus_pci', 'common_mlx5'] +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs'] install_headers('rte_bus_mlx5_pci.h') sources = files('mlx5_pci_bus.c') diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c index 66db3c7b0..d6e6da3dc 100644 --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -2,13 +2,521 @@ * Copyright 2020 Mellanox Technologies, Ltd */ +#include <stdlib.h> +#include <rte_malloc.h> #include "rte_bus_mlx5_pci.h" +#include <mlx5_common_utils.h> +struct mlx5_pci_device { + struct rte_pci_device *pci_dev; + TAILQ_ENTRY(mlx5_pci_device) next; + uint32_t classes_loaded; +}; + +/* Head of list of class drivers. */ static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = TAILQ_HEAD_INITIALIZER(drv_list); +/* Head of mlx5 pci devices. */ +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = + TAILQ_HEAD_INITIALIZER(devices_list); + +static const struct { + const char *name; + unsigned int dev_class; +} mlx5_classes[] = { + { .name = "vdpa", .dev_class = MLX5_CLASS_VDPA }, + { .name = "net", .dev_class = MLX5_CLASS_NET }, +}; + +static const unsigned int mlx5_class_combinations[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + /* New class combination should be added here. + * For example a new multi class device combination + * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. + */ +}; + +static int +class_name_to_value(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].dev_class; + } + return -EINVAL; +} + +static struct rte_mlx5_pci_driver * +class_driver_get(uint32_t class) +{ + struct rte_mlx5_pci_driver *driver; + + TAILQ_FOREACH(driver, &drv_list, next) { + if (driver->dev_class == class) + return driver; + } + return NULL; +} + +static int +bus_cmdline_options_handler(__rte_unused const char *key, + const char *class_names, void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name. Multiple + * class key,value is supplied as class=net:vdpa:foo:bar. + */ + found = strsep(&nstr, ":"); + if (!found) + continue; + /* Check if its a valid class. */ + class_val = class_name_to_value(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s." + " Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, + &ret); + rte_kvargs_free(kvlist); + return ret; +} + void rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) { TAILQ_INSERT_TAIL(&drv_list, driver, next); } + +static bool +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* Check if device's ids match the class driver's ids. */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + return true; + } + return false; +} + +static int +is_valid_class_combination(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination. */ + for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { + if (mlx5_class_combinations[i] == user_classes) + return 0; + } + /* Not found any valid class combination. */ + return -EINVAL; +} + +static struct mlx5_pci_device * +pci_to_mlx5_device(const struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + + TAILQ_FOREACH(dev, &devices_list, next) { + if (dev->pci_dev == pci_dev) + return dev; + } + return NULL; +} + +static bool +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) +{ + return (device->classes_loaded & class) ? true : false; +} + +static void +dev_release(struct mlx5_pci_device *dev) +{ + TAILQ_REMOVE(&devices_list, dev, next); + rte_free(dev); +} + +static int +class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) +{ + struct rte_mlx5_pci_driver *driver; + int local_ret = -ENODEV; + unsigned int i = 0; + int ret = 0; + + enabled_classes &= dev->classes_loaded; + while (enabled_classes) { + driver = class_driver_get(RTE_BIT(i)); + if (driver) { + local_ret = driver->pci_driver.remove(dev->pci_dev); + if (!local_ret) + dev->classes_loaded &= ~RTE_BIT(i); + else if (ret == 0) + ret = local_ret; + } + enabled_classes &= ~RTE_BIT(i); + i++; + } + if (local_ret) + ret = local_ret; + return ret; +} + +static int +class_drivers_probe(struct mlx5_pci_device *dev, + struct rte_pci_driver *pci_drv, + struct rte_pci_device *pci_dev, uint32_t user_classes) +{ + struct rte_mlx5_pci_driver *driver; + uint32_t enabled_classes = 0; + bool already_loaded; + int ret; + + TAILQ_FOREACH(driver, &drv_list, next) { + if ((driver->dev_class & user_classes) == 0) + continue; + if (!mlx5_bus_match(driver, pci_dev)) + continue; + already_loaded = dev->classes_loaded & driver->dev_class; + if (already_loaded && + !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { + DRV_LOG(ERR, "Device %s is already probed\n", + pci_dev->device.name); + ret = -EEXIST; + goto probe_err; + } + ret = driver->pci_driver.probe(pci_drv, pci_dev); + if (ret < 0) { + DRV_LOG(ERR, "Failed to load class driver = %s.\n", + driver->pci_driver.driver.name); + goto probe_err; + } + enabled_classes |= driver->dev_class; + } + dev->classes_loaded |= enabled_classes; + return 0; +probe_err: + /* Only unload drivers which are enabled which were enabled + * in this probe instance. + */ + class_drivers_remove(dev, enabled_classes); + return ret; +} + +/** + * DPDK callback to register to probe multiple PCI class devices. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + uint32_t user_classes = 0; + bool new_device = false; + int ret; + + ret = parse_class_options(pci_dev->device.devargs); + if (ret < 0) + return ret; + user_classes = ret; + if (user_classes) { + /* Validate combination here. */ + ret = is_valid_class_combination(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); + return ret; + } + } else { + /* Default to net class. */ + user_classes = MLX5_CLASS_NET; + } + dev = pci_to_mlx5_device(pci_dev); + if (!dev) { + dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); + if (!dev) + return -ENOMEM; + dev->pci_dev = pci_dev; + TAILQ_INSERT_HEAD(&devices_list, dev, next); + new_device = true; + } + ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes); + if (ret) + goto class_err; + return 0; +class_err: + if (new_device) + dev_release(dev); + return ret; +} + +/** + * DPDK callback to remove one or more class devices for a PCI device. + * + * This function removes all class devices belong to a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_bus_pci_remove(struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + /* Matching device found, cleanup and unload class drivers. */ + ret = class_drivers_remove(dev, dev->classes_loaded); + if (!ret) + dev_release(dev); + return ret; +} + +static int +mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class = NULL; + struct rte_mlx5_pci_driver *temp; + struct mlx5_pci_device *dev; + int ret = -EINVAL; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + TAILQ_FOREACH(class, &drv_list, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_map) { + ret = class->pci_driver.dma_map(pci_dev, addr, + iova, len); + if (ret) + goto map_err; + } + } + return ret; +map_err: + TAILQ_FOREACH(temp, &drv_list, next) { + if (temp == class) + break; + if (device_class_enabled(dev, temp->dev_class) && + temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) + temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + struct mlx5_pci_device *dev; + int local_ret = -EINVAL; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + ret = 0; + /* There is no unmap error recovery in current implementation. */ + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_unmap) { + local_ret = class->pci_driver.dma_unmap(pci_dev, addr, + iova, len); + if (local_ret && (ret == 0)) + ret = local_ret; + } + } + if (local_ret) + ret = local_ret; + return ret; +} + +/* PCI ID table is build dynamically based on registered mlx5 + * class drivers. + */ +static struct rte_pci_id *mlx5_bus_pci_id_map; + +static int +pci_id_table_size_get(const struct rte_pci_id *id_table) +{ + int table_size = 0; + + for (; id_table->vendor_id != 0; id_table++) + table_size++; + return table_size; +} + +static bool +pci_id_exists(const struct rte_pci_id *id, int next_idx) +{ + int current_size = next_idx - 1; + int i; + + for (i = 0; i < current_size; i++) { + if (id->device_id == mlx5_bus_pci_id_map[i].device_id && + id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id && + id->subsystem_vendor_id == + mlx5_bus_pci_id_map[i].subsystem_vendor_id && + id->subsystem_device_id == + mlx5_bus_pci_id_map[i].subsystem_device_id) + return true; + } + return false; +} + +static void +pci_id_insert(const struct rte_pci_id *id_table, int *next_idx) +{ + for (; id_table->vendor_id != 0; id_table++) { + if (!pci_id_exists(id_table, *next_idx)) { + /* New entry; add to the table. */ + mlx5_bus_pci_id_map[*next_idx] = *id_table; + (*next_idx)++; + } + } +} + +static int +pci_ids_table_build(void) +{ + struct rte_mlx5_pci_driver *first_driver; + struct rte_mlx5_pci_driver *driver; + const struct rte_pci_id *id_table; + int num_ids = 0; + int i = 0; + + TAILQ_FOREACH(driver, &drv_list, next) + num_ids += pci_id_table_size_get(driver->pci_driver.id_table); + if (!num_ids) + return -ENODEV; + /* Increase size by one for the termination entry of vendor_id = 0. */ + num_ids += 1; + mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map)); + if (!mlx5_bus_pci_id_map) + return -ENOMEM; + first_driver = TAILQ_FIRST(&drv_list); + /* Copy the first class driver's ID table. */ + for (id_table = first_driver->pci_driver.id_table; + id_table->vendor_id != 0; id_table++, i++) + mlx5_bus_pci_id_map[i] = *id_table; + TAILQ_FOREACH(driver, &drv_list, next) { + /* We already added first driver; skip it. */ + if (driver == first_driver) + continue; + pci_id_insert(driver->pci_driver.id_table, &i); + } + mlx5_bus_pci_id_map[i].vendor_id = 0; + return 0; +} + +static bool mlx5_bus_registered; +static struct rte_pci_driver mlx5_bus_driver = { + .driver = { + .name = "mlx5_bus_pci", + }, + .probe = mlx5_bus_pci_probe, + .remove = mlx5_bus_pci_remove, + .dma_map = mlx5_bus_pci_dma_map, + .dma_unmap = mlx5_bus_pci_dma_unmap, +}; + +RTE_INIT(mlx5_bus_pci) +{ + struct rte_mlx5_pci_driver *class; + int ret; + + ret = pci_ids_table_build(); + if (ret) + return; + TAILQ_FOREACH(class, &drv_list, next) + mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags; + mlx5_bus_driver.id_table = mlx5_bus_pci_id_map; + rte_pci_register(&mlx5_bus_driver); + mlx5_bus_registered = true; +} + +RTE_FINI(mlx5_bus_pci_finish) +{ + if (mlx5_bus_registered) + rte_pci_unregister(&mlx5_bus_driver); + if (mlx5_bus_pci_id_map) + free(mlx5_bus_pci_id_map); +} +RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 08/10] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (6 preceding siblings ...) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 07/10] bus/mlx5_pci: register a PCI driver Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 09/10] common/mlx5: remove class checks from individual driver Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 10/10] maintainers: add maintainers for mlx5 pci bus Parav Pandit 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Enable class driver to match with the mlx5 pci devices. Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class driver. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Avoid static table v1->v2: - Migreate API from rte_driver to rte_pci_driver --- drivers/bus/Makefile | 3 +++ drivers/net/mlx5/Makefile | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 1 - drivers/net/mlx5/linux/mlx5_os.h | 1 + drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 24 ++++++++++++++---------- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 ++- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++---------- mk/rte.app.mk | 1 + 11 files changed, 38 insertions(+), 26 deletions(-) diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index cea3b55e6..f96caa5ec 100644 --- a/drivers/bus/Makefile +++ b/drivers/bus/Makefile @@ -9,6 +9,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) +DIRS-y += mlx5_pci +endif DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index a458402dc..1cea7cd07 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_bus_mlx5_pci LDLIBS += -lm LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 2dc57b20e..b6042b7ef 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1321,7 +1321,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); return -rte_errno; } - MLX5_ASSERT(pci_drv == &mlx5_driver); errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 31add3988..695722520 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,4 +15,5 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) + #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index e95ce0267..26699a79b 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -8,7 +8,7 @@ if not (is_linux or is_windows) subdir_done() endif -deps += ['hash', 'common_mlx5'] +deps += ['hash', 'common_mlx5', 'bus_mlx5_pci'] sources = files( 'mlx5.c', 'mlx5_ethdev.c', diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index bb10c63bb..2270055fe 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -34,6 +34,7 @@ #include <rte_spinlock.h> #include <rte_string_fns.h> #include <rte_alarm.h> +#include <rte_bus_mlx5_pci.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h> @@ -1892,16 +1893,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -struct rte_pci_driver mlx5_driver = { - .driver = { - .name = MLX5_DRIVER_NAME +static struct rte_mlx5_pci_driver mlx5_driver = { + .dev_class = MLX5_CLASS_NET, + .pci_driver = { + .driver = { + .name = MLX5_DRIVER_NAME, + }, + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_dma_map, + .dma_unmap = mlx5_dma_unmap, + .drv_flags = PCI_DRV_FLAGS, }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, }; /** @@ -1919,7 +1923,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - rte_pci_register(&mlx5_driver); + rte_mlx5_pci_driver_register(&mlx5_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 46e66eb1c..df7b5eb87 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -129,7 +129,6 @@ struct mlx5_local_data { }; extern struct mlx5_shared_data *mlx5_shared_data; -extern struct rte_pci_driver mlx5_driver; /* Dev ops structs */ extern const struct eth_dev_ops mlx5_os_dev_ops; diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index 8a1c2eab5..9bb5dfdbf 100644 --- a/drivers/vdpa/mlx5/Makefile +++ b/drivers/vdpa/mlx5/Makefile @@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa CFLAGS += -I$(RTE_SDK)/lib/librte_sched CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched LDLIBS += -pthread # A few warnings cannot be avoided in external headers. diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build index 2963aad71..f4175c34e 100644 --- a/drivers/vdpa/mlx5/meson.build +++ b/drivers/vdpa/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_vdpa' -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched'] +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched'] sources = files( 'mlx5_vdpa.c', 'mlx5_vdpa_mem.c', diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index f043166ef..09c9cb935 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -11,7 +11,7 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> +#include <rte_bus_mlx5_pci.h> #include <rte_pci.h> #include <rte_string_fns.h> @@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) } /** - * DPDK callback to register a PCI device. + * DPDK callback to register a mlx5 PCI bus device. * * This function spawns vdpa device out of a given PCI device. * @@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_vdpa_driver = { - .driver = { - .name = "mlx5_vdpa", +static struct rte_mlx5_pci_driver mlx5_vdpa_driver = { + .dev_class = MLX5_CLASS_VDPA, + .pci_driver = { + .driver = { + .name = "mlx5_vdpa", + }, + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_pci_probe, + .remove = mlx5_vdpa_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, }; /** @@ -849,7 +852,7 @@ RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) if (mlx5_vdpa_logtype >= 0) rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE); if (mlx5_glue) - rte_pci_register(&mlx5_vdpa_driver); + rte_mlx5_pci_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 0ce8cf541..d7c1d90e4 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF) += -lrte_pmd_memif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) _LDLIBS-y += -lrte_common_mlx5 +_LDLIBS-y += -lrte_bus_mlx5_pci endif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD) += -lrte_pmd_mlx5_vdpa -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 09/10] common/mlx5: remove class checks from individual driver 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (7 preceding siblings ...) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 08/10] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 10/10] maintainers: add maintainers for mlx5 pci bus Parav Pandit 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Now that mlx5_pci bus does the check for enabled classes and performs probe(), remove() of associated classes, individual class driver doesn't need to check if other driver is enabled. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Removed empty line v1->v2: - New patch --- drivers/common/mlx5/mlx5_common.c | 37 ------------------- drivers/common/mlx5/mlx5_common.h | 2 - .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/linux/mlx5_os.c | 5 --- drivers/net/mlx5/linux/mlx5_os.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 5 --- 6 files changed, 52 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0ce5e4db1..771e046c1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue; uint8_t haswell_broadwell_cpu; -static int -mlx5_class_check_handler(__rte_unused const char *key, const char *value, - void *opaque) -{ - enum mlx5_class *ret = opaque; - - if (strcmp(value, "vdpa") == 0) { - *ret = MLX5_CLASS_VDPA; - } else if (strcmp(value, "net") == 0) { - *ret = MLX5_CLASS_NET; - } else { - DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device" - " class argument setting?", value); - *ret = MLX5_CLASS_INVALID; - } - return 0; -} - -enum mlx5_class -mlx5_class_get(struct rte_devargs *devargs) -{ - struct rte_kvargs *kvlist; - const char *key = MLX5_CLASS_ARG_NAME; - enum mlx5_class ret = MLX5_CLASS_NET; - - if (devargs == NULL) - return ret; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return ret; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret); - rte_kvargs_free(kvlist); - return ret; -} - - /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 62861af05..ca66c4033 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -243,8 +243,6 @@ struct mlx5_klm { LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page); -__rte_internal -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs); __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index ae57ebdba..01b4358a0 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,8 +1,6 @@ INTERNAL { global: - mlx5_class_get; - mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index b6042b7ef..2bd2b664d 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1308,11 +1308,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_dev_config dev_config; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } if (rte_eal_process_type() == RTE_PROC_PRIMARY) mlx5_pmd_socket_init(); ret = mlx5_init_once(); diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 695722520..31add3988 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,5 +15,4 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) - #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 09c9cb935..41d69e8a8 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_hca_attr attr; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr); if (!ibv) { DRV_LOG(ERR, "No matching IB device for PCI slot " -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v4 10/10] maintainers: add maintainers for mlx5 pci bus 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (8 preceding siblings ...) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 09/10] common/mlx5: remove class checks from individual driver Parav Pandit @ 2020-07-03 13:46 ` Parav Pandit 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-03 13:46 UTC (permalink / raw) To: dev, grive, ferruh.yigit; +Cc: thomas, orika, matan, Parav Pandit Added maintainers for new mlx5 specific mlx5_pci bus. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- MAINTAINERS | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 53a5e9a9e..e3fec55ca 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -517,6 +517,11 @@ Intel FPGA bus M: Rosen Xu <rosen.xu@intel.com> F: drivers/bus/ifpga/ +Melllanox mlx5 PCI bus driver +M: Parav Pandit <parav@mellaox.com> +M: Matan Azrad <matan@mellanox.com> +F: drivers/bus/mlx5_pci + NXP buses M: Hemant Agrawal <hemant.agrawal@nxp.com> M: Sachin Saxena <sachin.saxena@nxp.com> -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (3 preceding siblings ...) 2020-07-03 13:46 ` [dpdk-dev] [PATCH v4 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 1/9] eal: introduce macros for getting value for bit Parav Pandit ` (8 more replies) 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (6 subsequent siblings) 11 siblings, 9 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit This series introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, a new mlx5 PCI bus is created. This bus allows binding multiple class drivers (such as net, vdpa, regex(future)) to bind to the mlx5 PCI bus driver. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Introduces new RTE constructor priority for common initialization Patch-3 Fixes compilation error Patch-4 Define mlx5 class as bit fields Patch-5 Uses new RTE common priority Patch-6 Adds mlx5 PCI bus Patch-7 Implements a mlx5 PCI bus driver Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus API instead of rte PCI bus API Patch-9 Removed class check code as its already part of the bus now Patch-10 Add maintainers for the new bus Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ ------------- / \______| mlx5 |_____ / | pci bus | ------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD defines its specific PCI ID table and mlx5 bus driver probes matching class drivers. - mlx5 pci bus driver is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI bus driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci bus, a common driver is implemented which exposes class registration API. However, bus model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common bus driver. It also eliminates peer blind zone problem as bottom pci bus layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:vdpa ./testpmd -w <PCI BDF>,class=vdpa In future, ./testpmd -w <PCI BDF>,class=net:regex Changelog: v4->v5: - Squash the maintainers update path with other patch which adds the bus - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (9): eal: introduce macros for getting value for bit eal: introduce RTE common initialization level common/mlx5: fix empty input style in glue wrappers common/mlx5: change mlx5 class enum values as bits common/mlx5: use common rte priority bus/mlx5_pci: add mlx5 PCI bus bus/mlx5_pci: register a PCI driver bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver common/mlx5: remove class checks from individual driver MAINTAINERS | 5 + drivers/bus/Makefile | 3 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 40 ++ drivers/bus/mlx5_pci/meson.build | 19 + drivers/bus/mlx5_pci/mlx5_pci_bus.c | 522 ++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/mlx5_common.c | 39 +- drivers/common/mlx5/mlx5_common.h | 7 +- .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 26 +- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 30 +- lib/librte_eal/include/rte_bitops.h | 8 + lib/librte_eal/include/rte_common.h | 1 + mk/rte.app.mk | 1 + 23 files changed, 730 insertions(+), 85 deletions(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 1/9] eal: introduce macros for getting value for bit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 2/9] eal: introduce RTE common initialization level Parav Pandit ` (7 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit, Morten Brørup There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> Acked-by: Morten Brørup <mb@smartsharesystems.com> --- Changelog: v4->v5: - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v1->v2: - Addressed comments from Thomas and Gaten. - Avoided new file, added macro to rte_bitops.h --- lib/librte_eal/include/rte_bitops.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..ca46a110f 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,14 @@ #include <rte_debug.h> #include <rte_compat.h> +/** + * Get the uint64_t value for a specified bit set. + * + * @param nr + * The bit number in range of 0 to 63. + */ +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 2/9] eal: introduce RTE common initialization level 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 1/9] eal: introduce macros for getting value for bit Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 3/9] common/mlx5: fix empty input style in glue wrappers Parav Pandit ` (6 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Currently mlx5_common uses CLASS priority to initialize common code before initializing the PMD. However mlx5_common is not really a class, it is the pre-initialization code needed for the PMDs. In subsequent patch a needed initialization sequence is: (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Hence, introduce a new RTE priority level RTE_PRIO_COMMON which can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs for class driver initialization. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- lib/librte_eal/include/rte_common.h | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h index 0843ce69e..d4653ea77 100644 --- a/lib/librte_eal/include/rte_common.h +++ b/lib/librte_eal/include/rte_common.h @@ -126,6 +126,7 @@ typedef uint16_t unaligned_uint16_t; #define RTE_PRIORITY_LOG 101 #define RTE_PRIORITY_BUS 110 +#define RTE_PRIORITY_COMMON 119 #define RTE_PRIORITY_CLASS 120 #define RTE_PRIORITY_LAST 65535 -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 3/9] common/mlx5: fix empty input style in glue wrappers 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 1/9] eal: introduce macros for getting value for bit Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 2/9] eal: introduce RTE common initialization level Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 4/9] common/mlx5: change mlx5 class enum values as bits Parav Pandit ` (5 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Following two errors are reported when compiled with gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5). drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] Fix them by adding void data type in empty argument list. Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch") Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue") Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- drivers/common/mlx5/linux/mlx5_glue.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h index 069d8540c..e5e052a6a 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.h +++ b/drivers/common/mlx5/linux/mlx5_glue.h @@ -185,11 +185,11 @@ struct mlx5_glue { void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl); void *(*dr_create_flow_action_dest_port)(void *domain, uint32_t port); - void *(*dr_create_flow_action_drop)(); + void *(*dr_create_flow_action_drop)(void); void *(*dr_create_flow_action_push_vlan) (struct mlx5dv_dr_domain *domain, rte_be32_t vlan_tag); - void *(*dr_create_flow_action_pop_vlan)(); + void *(*dr_create_flow_action_pop_vlan)(void); void *(*dr_create_flow_tbl)(void *domain, uint32_t level); int (*dr_destroy_flow_tbl)(void *tbl); void *(*dr_create_domain)(struct ibv_context *ctx, -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 4/9] common/mlx5: change mlx5 class enum values as bits 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (2 preceding siblings ...) 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 3/9] common/mlx5: fix empty input style in glue wrappers Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 5/9] common/mlx5: use common rte priority Parav Pandit ` (4 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit mlx5 PCI Device supports multiple classes of devices such as net, vdpa, and/or regex. To support these multiple classes, change mlx5_class to a bitmap values so that if users asks to enable multiple of them, all supported classes can be parsed. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v1->v2: - Rebasd due to removal previous patch --- drivers/common/mlx5/mlx5_common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index da01ffa1c..00ccba622 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -13,6 +13,7 @@ #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h> +#include <rte_bitops.h> #include "mlx5_prm.h" #include "mlx5_devx_cmds.h" @@ -207,9 +208,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); #define MLX5_CLASS_ARG_NAME "class" enum mlx5_class { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, MLX5_CLASS_INVALID, + MLX5_CLASS_NET = RTE_BIT64(0), + MLX5_CLASS_VDPA = RTE_BIT64(1), }; #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */ -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 5/9] common/mlx5: use common rte priority 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (3 preceding siblings ...) 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 4/9] common/mlx5: change mlx5 class enum values as bits Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 6/9] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit ` (3 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Use RTE_PRIO_COMMON for mlx5 common initialization and use RTE_PRIO_CLASS for mlx5 net, vdpa PMDs. This enables to do following initialization sequence. (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/mlx5_common.c | 2 +- drivers/net/mlx5/mlx5.c | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 693e2c68c..0ce5e4db1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) /** * Initialization routine for run-time dependency on glue library. */ -RTE_INIT_PRIO(mlx5_glue_init, CLASS) +RTE_INIT_PRIO(mlx5_glue_init, COMMON) { mlx5_glue_constructor(); } diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 07c6addd5..bb10c63bb 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1907,7 +1907,7 @@ struct rte_pci_driver mlx5_driver = { /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_pmd_init) +RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) { /* Initialize driver log type. */ mlx5_logtype = rte_log_register("pmd.net.mlx5"); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index dbd36ab0c..f043166ef 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -842,7 +842,7 @@ static struct rte_pci_driver mlx5_vdpa_driver = { /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_vdpa_init) +RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) { /* Initialize common log type. */ mlx5_vdpa_logtype = rte_log_register("pmd.vdpa.mlx5"); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 6/9] bus/mlx5_pci: add mlx5 PCI bus 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (4 preceding siblings ...) 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 5/9] common/mlx5: use common rte priority Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 7/9] bus/mlx5_pci: register a PCI driver Parav Pandit ` (2 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single pci device. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v4->v5: - Merged maintainers update patch with this patch v2->v3: - Addressed comments from Thomas and Asaf - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address comments from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Changed alignedment to mlx5 specific aligment instead of standard DPDK - Using uint32_t instead of mlx5_class enum v1->v2: - Address comments from Thomas and Gaetan - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus --- MAINTAINERS | 5 ++ drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 38 +++++++++ drivers/bus/mlx5_pci/meson.build | 19 +++++ drivers/bus/mlx5_pci/mlx5_pci_bus.c | 14 ++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++++++++++++++++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 ++ 7 files changed, 166 insertions(+), 1 deletion(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map diff --git a/MAINTAINERS b/MAINTAINERS index 53a5e9a9e..e3fec55ca 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -517,6 +517,11 @@ Intel FPGA bus M: Rosen Xu <rosen.xu@intel.com> F: drivers/bus/ifpga/ +Melllanox mlx5 PCI bus driver +M: Parav Pandit <parav@mellaox.com> +M: Matan Azrad <matan@mellanox.com> +F: drivers/bus/mlx5_pci + NXP buses M: Hemant Agrawal <hemant.agrawal@nxp.com> M: Sachin Saxena <sachin.saxena@nxp.com> diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index 80de2d91d..b1381838d 100644 --- a/drivers/bus/meson.build +++ b/drivers/bus/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel Corporation -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus'] +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus'] std_deps = ['eal'] config_flag_fmt = 'RTE_LIBRTE_@0@_BUS' driver_name_fmt = 'rte_bus_@0@' diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile new file mode 100644 index 000000000..de4ccd83f --- /dev/null +++ b/drivers/bus/mlx5_pci/Makefile @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2020 Mellanox Technologies, Ltd + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_bus_mlx5_pci.a + +CFLAGS += -O3 -Wall -Wextra +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -Wno-strict-prototypes +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux +CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +LDLIBS += -lrte_eal +LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_bus_mlx5_pci_version.map + +SRCS-y += mlx5_pci_bus.c + +# DEBUG which is usually provided on the command-line may enable +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. +ifeq ($(DEBUG),1) +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y +endif + +# +# Export include files +# +SYMLINK-y-include += rte_bus_mlx5_pci.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build new file mode 100644 index 000000000..64a17cbad --- /dev/null +++ b/drivers/bus/mlx5_pci/meson.build @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020 Mellanox Technologies Ltd + +deps += ['pci', 'bus_pci', 'common_mlx5'] +install_headers('rte_bus_mlx5_pci.h') +sources = files('mlx5_pci_bus.c') + +cflags_options = [ + '-std=c11', + '-Wno-strict-prototypes', + '-D_BSD_SOURCE', + '-D_DEFAULT_SOURCE', + '-D_XOPEN_SOURCE=600' +] +foreach option:cflags_options + if cc.has_argument(option) + cflags += option + endif +endforeach diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c new file mode 100644 index 000000000..66db3c7b0 --- /dev/null +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#include "rte_bus_mlx5_pci.h" + +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = + TAILQ_HEAD_INITIALIZER(drv_list); + +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) +{ + TAILQ_INSERT_TAIL(&drv_list, driver, next); +} diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h new file mode 100644 index 000000000..9f8d22e2b --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef _RTE_BUS_MLX5_PCI_H_ +#define _RTE_BUS_MLX5_PCI_H_ + +/** + * @file + * + * RTE Mellanox PCI Bus Interface + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) + * devices. This bus enables creating such multiple class of devices on a + * single PCI device by allowing to bind multiple class specific device + * driver to attach to mlx5_pci bus driver. + * + * ----------- ------------ ----------------- + * | mlx5 | | mlx5 | | mlx5 | + * | net pmd | | vdpa pmd | | new class pmd | + * ----------- ------------ ----------------- + * \ | / + * \ | / + * \ ------------- / + * \______| mlx5 |_____ / + * | pci bus | + * ------------- + * | + * ----------- + * | mlx5 | + * | pci dev | + * ----------- + * + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 pci bus driver is cental place that validates supported + * class combinations. + */ + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +#include <rte_pci.h> +#include <rte_bus_pci.h> + +#include <mlx5_common.h> + +/** + * A structure describing a mlx5 pci driver. + */ +struct rte_mlx5_pci_driver { + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ + uint32_t dev_class; /**< Class of this driver, enum mlx5_class */ + TAILQ_ENTRY(rte_mlx5_pci_driver) next; +}; + +/** + * Register a mlx5_pci device driver. + * + * @param driver + * A pointer to a rte_mlx5_pci_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver); + +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \ + static const char *mlx5_pci_drvinit_fn_ ## nm; \ + RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \ + { \ + (drv).driver.name = RTE_STR(nm); \ + rte_mlx5_pci_driver_register(&drv); \ + } \ + RTE_PMD_EXPORT_NAME(nm, __COUNTER__) + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _RTE_BUS_MLX5_PCI_H_ */ diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map new file mode 100644 index 000000000..4cfd3db10 --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map @@ -0,0 +1,5 @@ +INTERNAL { + global: + + rte_mlx5_pci_driver_register; +}; -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 7/9] bus/mlx5_pci: register a PCI driver 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (5 preceding siblings ...) 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 6/9] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 9/9] common/mlx5: remove class checks from individual driver Parav Pandit 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Create a mlx5 bus driver framework for invoking drivers of multiple classes who have registered with the mlx5_pci bus driver. Validate user class arguments for supported class combinations. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Addressed comments from Asaf - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Addressed comments from Matan. - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block. - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Maintaining class driver list to keep load/unload order symmetric and mirror of each other. - Deriving drv_flags based on the class drivers - Using PCI address comparision helper instead of pointer comparision - Fixed alignment for id_table - Continue to probe_err if device is already probed - Perform dma map on best effort basis for all supported drivers - Removed drv_flags check - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Address comments from Thomas and Gaetan - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - USe uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options --- drivers/bus/mlx5_pci/Makefile | 2 + drivers/bus/mlx5_pci/meson.build | 2 +- drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++ 3 files changed, 511 insertions(+), 1 deletion(-) diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile index de4ccd83f..1a005ee32 100644 --- a/drivers/bus/mlx5_pci/Makefile +++ b/drivers/bus/mlx5_pci/Makefile @@ -15,7 +15,9 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/bus/pci +CFLAGS += -D_DEFAULT_SOURCE LDLIBS += -lrte_eal +LDLIBS += -lrte_kvargs LDLIBS += -lrte_common_mlx5 LDLIBS += -lrte_pci -lrte_bus_pci diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build index 64a17cbad..0532a9dfd 100644 --- a/drivers/bus/mlx5_pci/meson.build +++ b/drivers/bus/mlx5_pci/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Mellanox Technologies Ltd -deps += ['pci', 'bus_pci', 'common_mlx5'] +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs'] install_headers('rte_bus_mlx5_pci.h') sources = files('mlx5_pci_bus.c') diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c index 66db3c7b0..6f219730c 100644 --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -2,13 +2,521 @@ * Copyright 2020 Mellanox Technologies, Ltd */ +#include <stdlib.h> +#include <rte_malloc.h> #include "rte_bus_mlx5_pci.h" +#include <mlx5_common_utils.h> +struct mlx5_pci_device { + struct rte_pci_device *pci_dev; + TAILQ_ENTRY(mlx5_pci_device) next; + uint32_t classes_loaded; +}; + +/* Head of list of class drivers. */ static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = TAILQ_HEAD_INITIALIZER(drv_list); +/* Head of mlx5 pci devices. */ +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = + TAILQ_HEAD_INITIALIZER(devices_list); + +static const struct { + const char *name; + unsigned int dev_class; +} mlx5_classes[] = { + { .name = "vdpa", .dev_class = MLX5_CLASS_VDPA }, + { .name = "net", .dev_class = MLX5_CLASS_NET }, +}; + +static const unsigned int mlx5_class_combinations[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + /* New class combination should be added here. + * For example a new multi class device combination + * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. + */ +}; + +static int +class_name_to_value(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].dev_class; + } + return -EINVAL; +} + +static struct rte_mlx5_pci_driver * +class_driver_get(uint32_t class) +{ + struct rte_mlx5_pci_driver *driver; + + TAILQ_FOREACH(driver, &drv_list, next) { + if (driver->dev_class == class) + return driver; + } + return NULL; +} + +static int +bus_cmdline_options_handler(__rte_unused const char *key, + const char *class_names, void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name. Multiple + * class key,value is supplied as class=net:vdpa:foo:bar. + */ + found = strsep(&nstr, ":"); + if (!found) + continue; + /* Check if its a valid class. */ + class_val = class_name_to_value(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s." + " Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, + &ret); + rte_kvargs_free(kvlist); + return ret; +} + void rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) { TAILQ_INSERT_TAIL(&drv_list, driver, next); } + +static bool +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* Check if device's ids match the class driver's ids. */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + return true; + } + return false; +} + +static int +is_valid_class_combination(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination. */ + for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { + if (mlx5_class_combinations[i] == user_classes) + return 0; + } + /* Not found any valid class combination. */ + return -EINVAL; +} + +static struct mlx5_pci_device * +pci_to_mlx5_device(const struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + + TAILQ_FOREACH(dev, &devices_list, next) { + if (dev->pci_dev == pci_dev) + return dev; + } + return NULL; +} + +static bool +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) +{ + return (device->classes_loaded & class) ? true : false; +} + +static void +dev_release(struct mlx5_pci_device *dev) +{ + TAILQ_REMOVE(&devices_list, dev, next); + rte_free(dev); +} + +static int +class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) +{ + struct rte_mlx5_pci_driver *driver; + int local_ret = -ENODEV; + unsigned int i = 0; + int ret = 0; + + enabled_classes &= dev->classes_loaded; + while (enabled_classes) { + driver = class_driver_get(RTE_BIT64(i)); + if (driver) { + local_ret = driver->pci_driver.remove(dev->pci_dev); + if (!local_ret) + dev->classes_loaded &= ~RTE_BIT64(i); + else if (ret == 0) + ret = local_ret; + } + enabled_classes &= ~RTE_BIT64(i); + i++; + } + if (local_ret) + ret = local_ret; + return ret; +} + +static int +class_drivers_probe(struct mlx5_pci_device *dev, + struct rte_pci_driver *pci_drv, + struct rte_pci_device *pci_dev, uint32_t user_classes) +{ + struct rte_mlx5_pci_driver *driver; + uint32_t enabled_classes = 0; + bool already_loaded; + int ret; + + TAILQ_FOREACH(driver, &drv_list, next) { + if ((driver->dev_class & user_classes) == 0) + continue; + if (!mlx5_bus_match(driver, pci_dev)) + continue; + already_loaded = dev->classes_loaded & driver->dev_class; + if (already_loaded && + !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { + DRV_LOG(ERR, "Device %s is already probed\n", + pci_dev->device.name); + ret = -EEXIST; + goto probe_err; + } + ret = driver->pci_driver.probe(pci_drv, pci_dev); + if (ret < 0) { + DRV_LOG(ERR, "Failed to load class driver = %s.\n", + driver->pci_driver.driver.name); + goto probe_err; + } + enabled_classes |= driver->dev_class; + } + dev->classes_loaded |= enabled_classes; + return 0; +probe_err: + /* Only unload drivers which are enabled which were enabled + * in this probe instance. + */ + class_drivers_remove(dev, enabled_classes); + return ret; +} + +/** + * DPDK callback to register to probe multiple PCI class devices. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + uint32_t user_classes = 0; + bool new_device = false; + int ret; + + ret = parse_class_options(pci_dev->device.devargs); + if (ret < 0) + return ret; + user_classes = ret; + if (user_classes) { + /* Validate combination here. */ + ret = is_valid_class_combination(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); + return ret; + } + } else { + /* Default to net class. */ + user_classes = MLX5_CLASS_NET; + } + dev = pci_to_mlx5_device(pci_dev); + if (!dev) { + dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); + if (!dev) + return -ENOMEM; + dev->pci_dev = pci_dev; + TAILQ_INSERT_HEAD(&devices_list, dev, next); + new_device = true; + } + ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes); + if (ret) + goto class_err; + return 0; +class_err: + if (new_device) + dev_release(dev); + return ret; +} + +/** + * DPDK callback to remove one or more class devices for a PCI device. + * + * This function removes all class devices belong to a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_bus_pci_remove(struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + /* Matching device found, cleanup and unload class drivers. */ + ret = class_drivers_remove(dev, dev->classes_loaded); + if (!ret) + dev_release(dev); + return ret; +} + +static int +mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class = NULL; + struct rte_mlx5_pci_driver *temp; + struct mlx5_pci_device *dev; + int ret = -EINVAL; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + TAILQ_FOREACH(class, &drv_list, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_map) { + ret = class->pci_driver.dma_map(pci_dev, addr, + iova, len); + if (ret) + goto map_err; + } + } + return ret; +map_err: + TAILQ_FOREACH(temp, &drv_list, next) { + if (temp == class) + break; + if (device_class_enabled(dev, temp->dev_class) && + temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) + temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + struct mlx5_pci_device *dev; + int local_ret = -EINVAL; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + ret = 0; + /* There is no unmap error recovery in current implementation. */ + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_unmap) { + local_ret = class->pci_driver.dma_unmap(pci_dev, addr, + iova, len); + if (local_ret && (ret == 0)) + ret = local_ret; + } + } + if (local_ret) + ret = local_ret; + return ret; +} + +/* PCI ID table is build dynamically based on registered mlx5 + * class drivers. + */ +static struct rte_pci_id *mlx5_bus_pci_id_map; + +static int +pci_id_table_size_get(const struct rte_pci_id *id_table) +{ + int table_size = 0; + + for (; id_table->vendor_id != 0; id_table++) + table_size++; + return table_size; +} + +static bool +pci_id_exists(const struct rte_pci_id *id, int next_idx) +{ + int current_size = next_idx - 1; + int i; + + for (i = 0; i < current_size; i++) { + if (id->device_id == mlx5_bus_pci_id_map[i].device_id && + id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id && + id->subsystem_vendor_id == + mlx5_bus_pci_id_map[i].subsystem_vendor_id && + id->subsystem_device_id == + mlx5_bus_pci_id_map[i].subsystem_device_id) + return true; + } + return false; +} + +static void +pci_id_insert(const struct rte_pci_id *id_table, int *next_idx) +{ + for (; id_table->vendor_id != 0; id_table++) { + if (!pci_id_exists(id_table, *next_idx)) { + /* New entry; add to the table. */ + mlx5_bus_pci_id_map[*next_idx] = *id_table; + (*next_idx)++; + } + } +} + +static int +pci_ids_table_build(void) +{ + struct rte_mlx5_pci_driver *first_driver; + struct rte_mlx5_pci_driver *driver; + const struct rte_pci_id *id_table; + int num_ids = 0; + int i = 0; + + TAILQ_FOREACH(driver, &drv_list, next) + num_ids += pci_id_table_size_get(driver->pci_driver.id_table); + if (!num_ids) + return -ENODEV; + /* Increase size by one for the termination entry of vendor_id = 0. */ + num_ids += 1; + mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map)); + if (!mlx5_bus_pci_id_map) + return -ENOMEM; + first_driver = TAILQ_FIRST(&drv_list); + /* Copy the first class driver's ID table. */ + for (id_table = first_driver->pci_driver.id_table; + id_table->vendor_id != 0; id_table++, i++) + mlx5_bus_pci_id_map[i] = *id_table; + TAILQ_FOREACH(driver, &drv_list, next) { + /* We already added first driver; skip it. */ + if (driver == first_driver) + continue; + pci_id_insert(driver->pci_driver.id_table, &i); + } + mlx5_bus_pci_id_map[i].vendor_id = 0; + return 0; +} + +static bool mlx5_bus_registered; +static struct rte_pci_driver mlx5_bus_driver = { + .driver = { + .name = "mlx5_bus_pci", + }, + .probe = mlx5_bus_pci_probe, + .remove = mlx5_bus_pci_remove, + .dma_map = mlx5_bus_pci_dma_map, + .dma_unmap = mlx5_bus_pci_dma_unmap, +}; + +RTE_INIT(mlx5_bus_pci) +{ + struct rte_mlx5_pci_driver *class; + int ret; + + ret = pci_ids_table_build(); + if (ret) + return; + TAILQ_FOREACH(class, &drv_list, next) + mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags; + mlx5_bus_driver.id_table = mlx5_bus_pci_id_map; + rte_pci_register(&mlx5_bus_driver); + mlx5_bus_registered = true; +} + +RTE_FINI(mlx5_bus_pci_finish) +{ + if (mlx5_bus_registered) + rte_pci_unregister(&mlx5_bus_driver); + if (mlx5_bus_pci_id_map) + free(mlx5_bus_pci_id_map); +} +RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (6 preceding siblings ...) 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 7/9] bus/mlx5_pci: register a PCI driver Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 9/9] common/mlx5: remove class checks from individual driver Parav Pandit 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Enable class driver to match with the mlx5 pci devices. Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class driver. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Avoid static table v1->v2: - Migreate API from rte_driver to rte_pci_driver --- drivers/bus/Makefile | 3 +++ drivers/net/mlx5/Makefile | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 1 - drivers/net/mlx5/linux/mlx5_os.h | 1 + drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 24 ++++++++++++++---------- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 ++- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++---------- mk/rte.app.mk | 1 + 11 files changed, 38 insertions(+), 26 deletions(-) diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index cea3b55e6..f96caa5ec 100644 --- a/drivers/bus/Makefile +++ b/drivers/bus/Makefile @@ -9,6 +9,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) +DIRS-y += mlx5_pci +endif DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index a458402dc..1cea7cd07 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_bus_mlx5_pci LDLIBS += -lm LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 2dc57b20e..b6042b7ef 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1321,7 +1321,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); return -rte_errno; } - MLX5_ASSERT(pci_drv == &mlx5_driver); errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 31add3988..695722520 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,4 +15,5 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) + #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index e95ce0267..26699a79b 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -8,7 +8,7 @@ if not (is_linux or is_windows) subdir_done() endif -deps += ['hash', 'common_mlx5'] +deps += ['hash', 'common_mlx5', 'bus_mlx5_pci'] sources = files( 'mlx5.c', 'mlx5_ethdev.c', diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index bb10c63bb..2270055fe 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -34,6 +34,7 @@ #include <rte_spinlock.h> #include <rte_string_fns.h> #include <rte_alarm.h> +#include <rte_bus_mlx5_pci.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h> @@ -1892,16 +1893,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -struct rte_pci_driver mlx5_driver = { - .driver = { - .name = MLX5_DRIVER_NAME +static struct rte_mlx5_pci_driver mlx5_driver = { + .dev_class = MLX5_CLASS_NET, + .pci_driver = { + .driver = { + .name = MLX5_DRIVER_NAME, + }, + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_dma_map, + .dma_unmap = mlx5_dma_unmap, + .drv_flags = PCI_DRV_FLAGS, }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, }; /** @@ -1919,7 +1923,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - rte_pci_register(&mlx5_driver); + rte_mlx5_pci_driver_register(&mlx5_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 46e66eb1c..df7b5eb87 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -129,7 +129,6 @@ struct mlx5_local_data { }; extern struct mlx5_shared_data *mlx5_shared_data; -extern struct rte_pci_driver mlx5_driver; /* Dev ops structs */ extern const struct eth_dev_ops mlx5_os_dev_ops; diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index 8a1c2eab5..9bb5dfdbf 100644 --- a/drivers/vdpa/mlx5/Makefile +++ b/drivers/vdpa/mlx5/Makefile @@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa CFLAGS += -I$(RTE_SDK)/lib/librte_sched CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched LDLIBS += -pthread # A few warnings cannot be avoided in external headers. diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build index 2963aad71..f4175c34e 100644 --- a/drivers/vdpa/mlx5/meson.build +++ b/drivers/vdpa/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_vdpa' -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched'] +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched'] sources = files( 'mlx5_vdpa.c', 'mlx5_vdpa_mem.c', diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index f043166ef..09c9cb935 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -11,7 +11,7 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> +#include <rte_bus_mlx5_pci.h> #include <rte_pci.h> #include <rte_string_fns.h> @@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) } /** - * DPDK callback to register a PCI device. + * DPDK callback to register a mlx5 PCI bus device. * * This function spawns vdpa device out of a given PCI device. * @@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_vdpa_driver = { - .driver = { - .name = "mlx5_vdpa", +static struct rte_mlx5_pci_driver mlx5_vdpa_driver = { + .dev_class = MLX5_CLASS_VDPA, + .pci_driver = { + .driver = { + .name = "mlx5_vdpa", + }, + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_pci_probe, + .remove = mlx5_vdpa_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, }; /** @@ -849,7 +852,7 @@ RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) if (mlx5_vdpa_logtype >= 0) rte_log_set_level(mlx5_vdpa_logtype, RTE_LOG_NOTICE); if (mlx5_glue) - rte_pci_register(&mlx5_vdpa_driver); + rte_mlx5_pci_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 0ce8cf541..d7c1d90e4 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF) += -lrte_pmd_memif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) _LDLIBS-y += -lrte_common_mlx5 +_LDLIBS-y += -lrte_bus_mlx5_pci endif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD) += -lrte_pmd_mlx5_vdpa -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v5 9/9] common/mlx5: remove class checks from individual driver 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (7 preceding siblings ...) 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit @ 2020-07-09 7:34 ` Parav Pandit 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-09 7:34 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Now that mlx5_pci bus does the check for enabled classes and performs probe(), remove() of associated classes, individual class driver doesn't need to check if other driver is enabled. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Removed empty line v1->v2: - New patch --- drivers/common/mlx5/mlx5_common.c | 37 ------------------- drivers/common/mlx5/mlx5_common.h | 2 - .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/linux/mlx5_os.c | 5 --- drivers/net/mlx5/linux/mlx5_os.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 5 --- 6 files changed, 52 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0ce5e4db1..771e046c1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue; uint8_t haswell_broadwell_cpu; -static int -mlx5_class_check_handler(__rte_unused const char *key, const char *value, - void *opaque) -{ - enum mlx5_class *ret = opaque; - - if (strcmp(value, "vdpa") == 0) { - *ret = MLX5_CLASS_VDPA; - } else if (strcmp(value, "net") == 0) { - *ret = MLX5_CLASS_NET; - } else { - DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device" - " class argument setting?", value); - *ret = MLX5_CLASS_INVALID; - } - return 0; -} - -enum mlx5_class -mlx5_class_get(struct rte_devargs *devargs) -{ - struct rte_kvargs *kvlist; - const char *key = MLX5_CLASS_ARG_NAME; - enum mlx5_class ret = MLX5_CLASS_NET; - - if (devargs == NULL) - return ret; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return ret; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret); - rte_kvargs_free(kvlist); - return ret; -} - - /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 00ccba622..0effaab1d 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -243,8 +243,6 @@ struct mlx5_klm { LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page); -__rte_internal -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs); __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index ae57ebdba..01b4358a0 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,8 +1,6 @@ INTERNAL { global: - mlx5_class_get; - mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index b6042b7ef..2bd2b664d 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1308,11 +1308,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_dev_config dev_config; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } if (rte_eal_process_type() == RTE_PROC_PRIMARY) mlx5_pmd_socket_init(); ret = mlx5_init_once(); diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 695722520..31add3988 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,5 +15,4 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) - #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 09c9cb935..41d69e8a8 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_hca_attr attr; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr); if (!ibv) { DRV_LOG(ERR, "No matching IB device for PCI slot " -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (4 preceding siblings ...) 2020-07-09 7:34 ` [dpdk-dev] [PATCH v5 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 1/9] eal: introduce macros for getting value for bit Parav Pandit ` (8 more replies) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (5 subsequent siblings) 11 siblings, 9 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit This series introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, a new mlx5 PCI bus is created. This bus allows binding multiple class drivers (such as net, vdpa, regex(future)) to bind to the mlx5 PCI bus driver. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Introduces new RTE constructor priority for common initialization Patch-3 Fixes compilation error Patch-4 Define mlx5 class as bit fields Patch-5 Uses new RTE common priority Patch-6 Adds mlx5 PCI bus Patch-7 Implements a mlx5 PCI bus driver Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus API instead of rte PCI bus API Patch-9 Removed class check code as its already part of the bus now Patch-10 Add maintainers for the new bus Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ ------------- / \______| mlx5 |_____ / | pci bus | ------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD defines its specific PCI ID table and mlx5 bus driver probes matching class drivers. - mlx5 pci bus driver is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI bus driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci bus, a common driver is implemented which exposes class registration API. However, bus model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common bus driver. It also eliminates peer blind zone problem as bottom pci bus layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:vdpa ./testpmd -w <PCI BDF>,class=vdpa In future, ./testpmd -w <PCI BDF>,class=net:regex Changelog: v5->v6: - Fixed compilation failure in parallel build for shared lib v4->v5: - Squash the maintainers update path with other patch which adds the bus - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (9): eal: introduce macros for getting value for bit eal: introduce RTE common initialization level common/mlx5: fix empty input style in glue wrappers common/mlx5: change mlx5 class enum values as bits common/mlx5: use common rte priority bus/mlx5_pci: add mlx5 PCI bus bus/mlx5_pci: register a PCI driver bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver common/mlx5: remove class checks from individual driver MAINTAINERS | 5 + drivers/bus/Makefile | 4 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 41 ++ drivers/bus/mlx5_pci/meson.build | 19 + drivers/bus/mlx5_pci/mlx5_pci_bus.c | 522 ++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/mlx5_common.c | 39 +- drivers/common/mlx5/mlx5_common.h | 7 +- .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 26 +- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 30 +- lib/librte_eal/include/rte_bitops.h | 8 + lib/librte_eal/include/rte_common.h | 1 + mk/rte.app.mk | 1 + 23 files changed, 732 insertions(+), 85 deletions(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 1/9] eal: introduce macros for getting value for bit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 2/9] eal: introduce RTE common initialization level Parav Pandit ` (7 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit, Morten Brørup There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> Acked-by: Morten Brørup <mb@smartsharesystems.com> --- Changelog: v4->v5: - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v1->v2: - Addressed comments from Thomas and Gaten. - Avoided new file, added macro to rte_bitops.h --- lib/librte_eal/include/rte_bitops.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..ca46a110f 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,14 @@ #include <rte_debug.h> #include <rte_compat.h> +/** + * Get the uint64_t value for a specified bit set. + * + * @param nr + * The bit number in range of 0 to 63. + */ +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 2/9] eal: introduce RTE common initialization level 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 1/9] eal: introduce macros for getting value for bit Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 3/9] common/mlx5: fix empty input style in glue wrappers Parav Pandit ` (6 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Currently mlx5_common uses CLASS priority to initialize common code before initializing the PMD. However mlx5_common is not really a class, it is the pre-initialization code needed for the PMDs. In subsequent patch a needed initialization sequence is: (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Hence, introduce a new RTE priority level RTE_PRIO_COMMON which can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs for class driver initialization. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- lib/librte_eal/include/rte_common.h | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h index 8f487a563..522afe58e 100644 --- a/lib/librte_eal/include/rte_common.h +++ b/lib/librte_eal/include/rte_common.h @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; #define RTE_PRIORITY_LOG 101 #define RTE_PRIORITY_BUS 110 +#define RTE_PRIORITY_COMMON 119 #define RTE_PRIORITY_CLASS 120 #define RTE_PRIORITY_LAST 65535 -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 3/9] common/mlx5: fix empty input style in glue wrappers 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 1/9] eal: introduce macros for getting value for bit Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 2/9] eal: introduce RTE common initialization level Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 4/9] common/mlx5: change mlx5 class enum values as bits Parav Pandit ` (5 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Following two errors are reported when compiled with gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5). drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] Fix them by adding void data type in empty argument list. Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch") Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue") Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- drivers/common/mlx5/linux/mlx5_glue.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h index 069d8540c..e5e052a6a 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.h +++ b/drivers/common/mlx5/linux/mlx5_glue.h @@ -185,11 +185,11 @@ struct mlx5_glue { void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl); void *(*dr_create_flow_action_dest_port)(void *domain, uint32_t port); - void *(*dr_create_flow_action_drop)(); + void *(*dr_create_flow_action_drop)(void); void *(*dr_create_flow_action_push_vlan) (struct mlx5dv_dr_domain *domain, rte_be32_t vlan_tag); - void *(*dr_create_flow_action_pop_vlan)(); + void *(*dr_create_flow_action_pop_vlan)(void); void *(*dr_create_flow_tbl)(void *domain, uint32_t level); int (*dr_destroy_flow_tbl)(void *tbl); void *(*dr_create_domain)(struct ibv_context *ctx, -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 4/9] common/mlx5: change mlx5 class enum values as bits 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (2 preceding siblings ...) 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 3/9] common/mlx5: fix empty input style in glue wrappers Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 5/9] common/mlx5: use common rte priority Parav Pandit ` (4 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit mlx5 PCI Device supports multiple classes of devices such as net, vdpa, and/or regex. To support these multiple classes, change mlx5_class to a bitmap values so that if users asks to enable multiple of them, all supported classes can be parsed. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v1->v2: - Rebasd due to removal previous patch --- drivers/common/mlx5/mlx5_common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 285150705..00e68edbb 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -13,6 +13,7 @@ #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h> +#include <rte_bitops.h> #include "mlx5_prm.h" #include "mlx5_devx_cmds.h" @@ -208,9 +209,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); #define MLX5_CLASS_ARG_NAME "class" enum mlx5_class { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, MLX5_CLASS_INVALID, + MLX5_CLASS_NET = RTE_BIT64(0), + MLX5_CLASS_VDPA = RTE_BIT64(1), }; #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */ -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 5/9] common/mlx5: use common rte priority 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (3 preceding siblings ...) 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 4/9] common/mlx5: change mlx5 class enum values as bits Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 6/9] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit ` (3 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Use RTE_PRIO_COMMON for mlx5 common initialization and use RTE_PRIO_CLASS for mlx5 net, vdpa PMDs. This enables to do following initialization sequence. (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/mlx5_common.c | 2 +- drivers/net/mlx5/mlx5.c | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 693e2c68c..0ce5e4db1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) /** * Initialization routine for run-time dependency on glue library. */ -RTE_INIT_PRIO(mlx5_glue_init, CLASS) +RTE_INIT_PRIO(mlx5_glue_init, COMMON) { mlx5_glue_constructor(); } diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 0c654ed8b..a6a65957e 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1912,7 +1912,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE) /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_pmd_init) +RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) { /* Build the static tables for Verbs conversion. */ mlx5_set_ptype_table(); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 67e77b11a..205a9e4ad 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -844,7 +844,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_vdpa_init) +RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) { if (mlx5_glue) rte_pci_register(&mlx5_vdpa_driver); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 6/9] bus/mlx5_pci: add mlx5 PCI bus 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (4 preceding siblings ...) 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 5/9] common/mlx5: use common rte priority Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 7/9] bus/mlx5_pci: register a PCI driver Parav Pandit ` (2 subsequent siblings) 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single pci device. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v5->v6: - Updated Makefile for parallel shared build support v4->v5: - Merged maintainers update patch with this patch v2->v3: - Addressed comments from Thomas and Asaf - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address comments from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Changed alignedment to mlx5 specific aligment instead of standard DPDK - Using uint32_t instead of mlx5_class enum v1->v2: - Address comments from Thomas and Gaetan - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus --- MAINTAINERS | 5 ++ drivers/bus/Makefile | 4 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 39 +++++++++ drivers/bus/mlx5_pci/meson.build | 19 +++++ drivers/bus/mlx5_pci/mlx5_pci_bus.c | 14 ++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++++++++++++++++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 ++ 8 files changed, 171 insertions(+), 1 deletion(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map diff --git a/MAINTAINERS b/MAINTAINERS index 3cd402b34..9e3bd1672 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -525,6 +525,11 @@ Intel FPGA bus M: Rosen Xu <rosen.xu@intel.com> F: drivers/bus/ifpga/ +Melllanox mlx5 PCI bus driver +M: Parav Pandit <parav@mellaox.com> +M: Matan Azrad <matan@mellanox.com> +F: drivers/bus/mlx5_pci + NXP buses M: Hemant Agrawal <hemant.agrawal@nxp.com> M: Sachin Saxena <sachin.saxena@nxp.com> diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index cea3b55e6..a70f213c1 100644 --- a/drivers/bus/Makefile +++ b/drivers/bus/Makefile @@ -9,6 +9,10 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) +DEPDIRS-mlx5_pci := pci +DIRS-y += mlx5_pci +endif DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index 80de2d91d..b1381838d 100644 --- a/drivers/bus/meson.build +++ b/drivers/bus/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel Corporation -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus'] +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus'] std_deps = ['eal'] config_flag_fmt = 'RTE_LIBRTE_@0@_BUS' driver_name_fmt = 'rte_bus_@0@' diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile new file mode 100644 index 000000000..dd24811a3 --- /dev/null +++ b/drivers/bus/mlx5_pci/Makefile @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2020 Mellanox Technologies, Ltd + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_bus_mlx5_pci.a + +CFLAGS += -O3 -Wall -Wextra +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -Wno-strict-prototypes +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux +CFLAGS += -I$(BUILDDIR)/drivers/bus/pci +CFLAGS += -I$(BUILDDIR)/drivers/bus +LDLIBS += -lrte_eal +LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_bus_mlx5_pci_version.map + +SRCS-y += mlx5_pci_bus.c + +# DEBUG which is usually provided on the command-line may enable +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. +ifeq ($(DEBUG),1) +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y +endif + +# +# Export include files +# +SYMLINK-y-include += rte_bus_mlx5_pci.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build new file mode 100644 index 000000000..64a17cbad --- /dev/null +++ b/drivers/bus/mlx5_pci/meson.build @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020 Mellanox Technologies Ltd + +deps += ['pci', 'bus_pci', 'common_mlx5'] +install_headers('rte_bus_mlx5_pci.h') +sources = files('mlx5_pci_bus.c') + +cflags_options = [ + '-std=c11', + '-Wno-strict-prototypes', + '-D_BSD_SOURCE', + '-D_DEFAULT_SOURCE', + '-D_XOPEN_SOURCE=600' +] +foreach option:cflags_options + if cc.has_argument(option) + cflags += option + endif +endforeach diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c new file mode 100644 index 000000000..66db3c7b0 --- /dev/null +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#include "rte_bus_mlx5_pci.h" + +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = + TAILQ_HEAD_INITIALIZER(drv_list); + +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) +{ + TAILQ_INSERT_TAIL(&drv_list, driver, next); +} diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h new file mode 100644 index 000000000..9f8d22e2b --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef _RTE_BUS_MLX5_PCI_H_ +#define _RTE_BUS_MLX5_PCI_H_ + +/** + * @file + * + * RTE Mellanox PCI Bus Interface + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) + * devices. This bus enables creating such multiple class of devices on a + * single PCI device by allowing to bind multiple class specific device + * driver to attach to mlx5_pci bus driver. + * + * ----------- ------------ ----------------- + * | mlx5 | | mlx5 | | mlx5 | + * | net pmd | | vdpa pmd | | new class pmd | + * ----------- ------------ ----------------- + * \ | / + * \ | / + * \ ------------- / + * \______| mlx5 |_____ / + * | pci bus | + * ------------- + * | + * ----------- + * | mlx5 | + * | pci dev | + * ----------- + * + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 pci bus driver is cental place that validates supported + * class combinations. + */ + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +#include <rte_pci.h> +#include <rte_bus_pci.h> + +#include <mlx5_common.h> + +/** + * A structure describing a mlx5 pci driver. + */ +struct rte_mlx5_pci_driver { + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ + uint32_t dev_class; /**< Class of this driver, enum mlx5_class */ + TAILQ_ENTRY(rte_mlx5_pci_driver) next; +}; + +/** + * Register a mlx5_pci device driver. + * + * @param driver + * A pointer to a rte_mlx5_pci_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver); + +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \ + static const char *mlx5_pci_drvinit_fn_ ## nm; \ + RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \ + { \ + (drv).driver.name = RTE_STR(nm); \ + rte_mlx5_pci_driver_register(&drv); \ + } \ + RTE_PMD_EXPORT_NAME(nm, __COUNTER__) + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _RTE_BUS_MLX5_PCI_H_ */ diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map new file mode 100644 index 000000000..4cfd3db10 --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map @@ -0,0 +1,5 @@ +INTERNAL { + global: + + rte_mlx5_pci_driver_register; +}; -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 7/9] bus/mlx5_pci: register a PCI driver 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (5 preceding siblings ...) 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 6/9] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 9/9] common/mlx5: remove class checks from individual driver Parav Pandit 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Create a mlx5 bus driver framework for invoking drivers of multiple classes who have registered with the mlx5_pci bus driver. Validate user class arguments for supported class combinations. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Addressed comments from Asaf - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Addressed comments from Matan. - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block. - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Maintaining class driver list to keep load/unload order symmetric and mirror of each other. - Deriving drv_flags based on the class drivers - Using PCI address comparision helper instead of pointer comparision - Fixed alignment for id_table - Continue to probe_err if device is already probed - Perform dma map on best effort basis for all supported drivers - Removed drv_flags check - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Address comments from Thomas and Gaetan - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - USe uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options --- drivers/bus/mlx5_pci/Makefile | 2 + drivers/bus/mlx5_pci/meson.build | 2 +- drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++ 3 files changed, 511 insertions(+), 1 deletion(-) diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile index dd24811a3..ee1af4dc6 100644 --- a/drivers/bus/mlx5_pci/Makefile +++ b/drivers/bus/mlx5_pci/Makefile @@ -16,7 +16,9 @@ CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(BUILDDIR)/drivers/bus/pci CFLAGS += -I$(BUILDDIR)/drivers/bus +CFLAGS += -D_DEFAULT_SOURCE LDLIBS += -lrte_eal +LDLIBS += -lrte_kvargs LDLIBS += -lrte_common_mlx5 LDLIBS += -lrte_pci -lrte_bus_pci diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build index 64a17cbad..0532a9dfd 100644 --- a/drivers/bus/mlx5_pci/meson.build +++ b/drivers/bus/mlx5_pci/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Mellanox Technologies Ltd -deps += ['pci', 'bus_pci', 'common_mlx5'] +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs'] install_headers('rte_bus_mlx5_pci.h') sources = files('mlx5_pci_bus.c') diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c index 66db3c7b0..6f219730c 100644 --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -2,13 +2,521 @@ * Copyright 2020 Mellanox Technologies, Ltd */ +#include <stdlib.h> +#include <rte_malloc.h> #include "rte_bus_mlx5_pci.h" +#include <mlx5_common_utils.h> +struct mlx5_pci_device { + struct rte_pci_device *pci_dev; + TAILQ_ENTRY(mlx5_pci_device) next; + uint32_t classes_loaded; +}; + +/* Head of list of class drivers. */ static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = TAILQ_HEAD_INITIALIZER(drv_list); +/* Head of mlx5 pci devices. */ +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = + TAILQ_HEAD_INITIALIZER(devices_list); + +static const struct { + const char *name; + unsigned int dev_class; +} mlx5_classes[] = { + { .name = "vdpa", .dev_class = MLX5_CLASS_VDPA }, + { .name = "net", .dev_class = MLX5_CLASS_NET }, +}; + +static const unsigned int mlx5_class_combinations[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + /* New class combination should be added here. + * For example a new multi class device combination + * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. + */ +}; + +static int +class_name_to_value(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].dev_class; + } + return -EINVAL; +} + +static struct rte_mlx5_pci_driver * +class_driver_get(uint32_t class) +{ + struct rte_mlx5_pci_driver *driver; + + TAILQ_FOREACH(driver, &drv_list, next) { + if (driver->dev_class == class) + return driver; + } + return NULL; +} + +static int +bus_cmdline_options_handler(__rte_unused const char *key, + const char *class_names, void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name. Multiple + * class key,value is supplied as class=net:vdpa:foo:bar. + */ + found = strsep(&nstr, ":"); + if (!found) + continue; + /* Check if its a valid class. */ + class_val = class_name_to_value(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s." + " Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, + &ret); + rte_kvargs_free(kvlist); + return ret; +} + void rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) { TAILQ_INSERT_TAIL(&drv_list, driver, next); } + +static bool +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* Check if device's ids match the class driver's ids. */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + return true; + } + return false; +} + +static int +is_valid_class_combination(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination. */ + for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { + if (mlx5_class_combinations[i] == user_classes) + return 0; + } + /* Not found any valid class combination. */ + return -EINVAL; +} + +static struct mlx5_pci_device * +pci_to_mlx5_device(const struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + + TAILQ_FOREACH(dev, &devices_list, next) { + if (dev->pci_dev == pci_dev) + return dev; + } + return NULL; +} + +static bool +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) +{ + return (device->classes_loaded & class) ? true : false; +} + +static void +dev_release(struct mlx5_pci_device *dev) +{ + TAILQ_REMOVE(&devices_list, dev, next); + rte_free(dev); +} + +static int +class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) +{ + struct rte_mlx5_pci_driver *driver; + int local_ret = -ENODEV; + unsigned int i = 0; + int ret = 0; + + enabled_classes &= dev->classes_loaded; + while (enabled_classes) { + driver = class_driver_get(RTE_BIT64(i)); + if (driver) { + local_ret = driver->pci_driver.remove(dev->pci_dev); + if (!local_ret) + dev->classes_loaded &= ~RTE_BIT64(i); + else if (ret == 0) + ret = local_ret; + } + enabled_classes &= ~RTE_BIT64(i); + i++; + } + if (local_ret) + ret = local_ret; + return ret; +} + +static int +class_drivers_probe(struct mlx5_pci_device *dev, + struct rte_pci_driver *pci_drv, + struct rte_pci_device *pci_dev, uint32_t user_classes) +{ + struct rte_mlx5_pci_driver *driver; + uint32_t enabled_classes = 0; + bool already_loaded; + int ret; + + TAILQ_FOREACH(driver, &drv_list, next) { + if ((driver->dev_class & user_classes) == 0) + continue; + if (!mlx5_bus_match(driver, pci_dev)) + continue; + already_loaded = dev->classes_loaded & driver->dev_class; + if (already_loaded && + !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { + DRV_LOG(ERR, "Device %s is already probed\n", + pci_dev->device.name); + ret = -EEXIST; + goto probe_err; + } + ret = driver->pci_driver.probe(pci_drv, pci_dev); + if (ret < 0) { + DRV_LOG(ERR, "Failed to load class driver = %s.\n", + driver->pci_driver.driver.name); + goto probe_err; + } + enabled_classes |= driver->dev_class; + } + dev->classes_loaded |= enabled_classes; + return 0; +probe_err: + /* Only unload drivers which are enabled which were enabled + * in this probe instance. + */ + class_drivers_remove(dev, enabled_classes); + return ret; +} + +/** + * DPDK callback to register to probe multiple PCI class devices. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + uint32_t user_classes = 0; + bool new_device = false; + int ret; + + ret = parse_class_options(pci_dev->device.devargs); + if (ret < 0) + return ret; + user_classes = ret; + if (user_classes) { + /* Validate combination here. */ + ret = is_valid_class_combination(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); + return ret; + } + } else { + /* Default to net class. */ + user_classes = MLX5_CLASS_NET; + } + dev = pci_to_mlx5_device(pci_dev); + if (!dev) { + dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); + if (!dev) + return -ENOMEM; + dev->pci_dev = pci_dev; + TAILQ_INSERT_HEAD(&devices_list, dev, next); + new_device = true; + } + ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes); + if (ret) + goto class_err; + return 0; +class_err: + if (new_device) + dev_release(dev); + return ret; +} + +/** + * DPDK callback to remove one or more class devices for a PCI device. + * + * This function removes all class devices belong to a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_bus_pci_remove(struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + /* Matching device found, cleanup and unload class drivers. */ + ret = class_drivers_remove(dev, dev->classes_loaded); + if (!ret) + dev_release(dev); + return ret; +} + +static int +mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class = NULL; + struct rte_mlx5_pci_driver *temp; + struct mlx5_pci_device *dev; + int ret = -EINVAL; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + TAILQ_FOREACH(class, &drv_list, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_map) { + ret = class->pci_driver.dma_map(pci_dev, addr, + iova, len); + if (ret) + goto map_err; + } + } + return ret; +map_err: + TAILQ_FOREACH(temp, &drv_list, next) { + if (temp == class) + break; + if (device_class_enabled(dev, temp->dev_class) && + temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) + temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + struct mlx5_pci_device *dev; + int local_ret = -EINVAL; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + ret = 0; + /* There is no unmap error recovery in current implementation. */ + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_unmap) { + local_ret = class->pci_driver.dma_unmap(pci_dev, addr, + iova, len); + if (local_ret && (ret == 0)) + ret = local_ret; + } + } + if (local_ret) + ret = local_ret; + return ret; +} + +/* PCI ID table is build dynamically based on registered mlx5 + * class drivers. + */ +static struct rte_pci_id *mlx5_bus_pci_id_map; + +static int +pci_id_table_size_get(const struct rte_pci_id *id_table) +{ + int table_size = 0; + + for (; id_table->vendor_id != 0; id_table++) + table_size++; + return table_size; +} + +static bool +pci_id_exists(const struct rte_pci_id *id, int next_idx) +{ + int current_size = next_idx - 1; + int i; + + for (i = 0; i < current_size; i++) { + if (id->device_id == mlx5_bus_pci_id_map[i].device_id && + id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id && + id->subsystem_vendor_id == + mlx5_bus_pci_id_map[i].subsystem_vendor_id && + id->subsystem_device_id == + mlx5_bus_pci_id_map[i].subsystem_device_id) + return true; + } + return false; +} + +static void +pci_id_insert(const struct rte_pci_id *id_table, int *next_idx) +{ + for (; id_table->vendor_id != 0; id_table++) { + if (!pci_id_exists(id_table, *next_idx)) { + /* New entry; add to the table. */ + mlx5_bus_pci_id_map[*next_idx] = *id_table; + (*next_idx)++; + } + } +} + +static int +pci_ids_table_build(void) +{ + struct rte_mlx5_pci_driver *first_driver; + struct rte_mlx5_pci_driver *driver; + const struct rte_pci_id *id_table; + int num_ids = 0; + int i = 0; + + TAILQ_FOREACH(driver, &drv_list, next) + num_ids += pci_id_table_size_get(driver->pci_driver.id_table); + if (!num_ids) + return -ENODEV; + /* Increase size by one for the termination entry of vendor_id = 0. */ + num_ids += 1; + mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map)); + if (!mlx5_bus_pci_id_map) + return -ENOMEM; + first_driver = TAILQ_FIRST(&drv_list); + /* Copy the first class driver's ID table. */ + for (id_table = first_driver->pci_driver.id_table; + id_table->vendor_id != 0; id_table++, i++) + mlx5_bus_pci_id_map[i] = *id_table; + TAILQ_FOREACH(driver, &drv_list, next) { + /* We already added first driver; skip it. */ + if (driver == first_driver) + continue; + pci_id_insert(driver->pci_driver.id_table, &i); + } + mlx5_bus_pci_id_map[i].vendor_id = 0; + return 0; +} + +static bool mlx5_bus_registered; +static struct rte_pci_driver mlx5_bus_driver = { + .driver = { + .name = "mlx5_bus_pci", + }, + .probe = mlx5_bus_pci_probe, + .remove = mlx5_bus_pci_remove, + .dma_map = mlx5_bus_pci_dma_map, + .dma_unmap = mlx5_bus_pci_dma_unmap, +}; + +RTE_INIT(mlx5_bus_pci) +{ + struct rte_mlx5_pci_driver *class; + int ret; + + ret = pci_ids_table_build(); + if (ret) + return; + TAILQ_FOREACH(class, &drv_list, next) + mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags; + mlx5_bus_driver.id_table = mlx5_bus_pci_id_map; + rte_pci_register(&mlx5_bus_driver); + mlx5_bus_registered = true; +} + +RTE_FINI(mlx5_bus_pci_finish) +{ + if (mlx5_bus_registered) + rte_pci_unregister(&mlx5_bus_driver); + if (mlx5_bus_pci_id_map) + free(mlx5_bus_pci_id_map); +} +RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (6 preceding siblings ...) 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 7/9] bus/mlx5_pci: register a PCI driver Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 9/9] common/mlx5: remove class checks from individual driver Parav Pandit 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Enable class driver to match with the mlx5 pci devices. Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class driver. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Avoid static table v1->v2: - Migreate API from rte_driver to rte_pci_driver --- drivers/net/mlx5/Makefile | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 1 - drivers/net/mlx5/linux/mlx5_os.h | 1 + drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 24 ++++++++++++++---------- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 ++- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++---------- mk/rte.app.mk | 1 + 10 files changed, 35 insertions(+), 26 deletions(-) diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index a458402dc..1cea7cd07 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -45,16 +45,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_bus_mlx5_pci LDLIBS += -lm LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 2dc57b20e..b6042b7ef 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1321,7 +1321,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); return -rte_errno; } - MLX5_ASSERT(pci_drv == &mlx5_driver); errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 31add3988..695722520 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,4 +15,5 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) + #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index e95ce0267..26699a79b 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -8,7 +8,7 @@ if not (is_linux or is_windows) subdir_done() endif -deps += ['hash', 'common_mlx5'] +deps += ['hash', 'common_mlx5', 'bus_mlx5_pci'] sources = files( 'mlx5.c', 'mlx5_ethdev.c', diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index a6a65957e..8f853844b 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -34,6 +34,7 @@ #include <rte_spinlock.h> #include <rte_string_fns.h> #include <rte_alarm.h> +#include <rte_bus_mlx5_pci.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h> @@ -1894,16 +1895,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -struct rte_pci_driver mlx5_driver = { - .driver = { - .name = MLX5_DRIVER_NAME +static struct rte_mlx5_pci_driver mlx5_driver = { + .dev_class = MLX5_CLASS_NET, + .pci_driver = { + .driver = { + .name = MLX5_DRIVER_NAME, + }, + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_dma_map, + .dma_unmap = mlx5_dma_unmap, + .drv_flags = PCI_DRV_FLAGS, }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, }; /* Initialize driver log type. */ @@ -1919,7 +1923,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - rte_pci_register(&mlx5_driver); + rte_mlx5_pci_driver_register(&mlx5_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 46e66eb1c..df7b5eb87 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -129,7 +129,6 @@ struct mlx5_local_data { }; extern struct mlx5_shared_data *mlx5_shared_data; -extern struct rte_pci_driver mlx5_driver; /* Dev ops structs */ extern const struct eth_dev_ops mlx5_os_dev_ops; diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index 8a1c2eab5..9bb5dfdbf 100644 --- a/drivers/vdpa/mlx5/Makefile +++ b/drivers/vdpa/mlx5/Makefile @@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa CFLAGS += -I$(RTE_SDK)/lib/librte_sched CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched LDLIBS += -pthread # A few warnings cannot be avoided in external headers. diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build index 2963aad71..f4175c34e 100644 --- a/drivers/vdpa/mlx5/meson.build +++ b/drivers/vdpa/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_vdpa' -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched'] +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched'] sources = files( 'mlx5_vdpa.c', 'mlx5_vdpa_mem.c', diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 205a9e4ad..5d247e0a1 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -11,7 +11,7 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> +#include <rte_bus_mlx5_pci.h> #include <rte_pci.h> #include <rte_string_fns.h> @@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) } /** - * DPDK callback to register a PCI device. + * DPDK callback to register a mlx5 PCI bus device. * * This function spawns vdpa device out of a given PCI device. * @@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_vdpa_driver = { - .driver = { - .name = "mlx5_vdpa", +static struct rte_mlx5_pci_driver mlx5_vdpa_driver = { + .dev_class = MLX5_CLASS_VDPA, + .pci_driver = { + .driver = { + .name = "mlx5_vdpa", + }, + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_pci_probe, + .remove = mlx5_vdpa_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, }; RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) @@ -847,7 +850,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) { if (mlx5_glue) - rte_pci_register(&mlx5_vdpa_driver); + rte_mlx5_pci_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 0ce8cf541..d7c1d90e4 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF) += -lrte_pmd_memif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) _LDLIBS-y += -lrte_common_mlx5 +_LDLIBS-y += -lrte_bus_mlx5_pci endif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD) += -lrte_pmd_mlx5_vdpa -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v6 9/9] common/mlx5: remove class checks from individual driver 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (7 preceding siblings ...) 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit @ 2020-07-16 7:29 ` Parav Pandit 8 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-16 7:29 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Now that mlx5_pci bus does the check for enabled classes and performs probe(), remove() of associated classes, individual class driver does not need to check if other driver is enabled. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Removed empty line v1->v2: - New patch --- drivers/common/mlx5/mlx5_common.c | 37 ------------------- drivers/common/mlx5/mlx5_common.h | 2 - .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/linux/mlx5_os.c | 5 --- drivers/net/mlx5/linux/mlx5_os.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 5 --- 6 files changed, 52 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0ce5e4db1..771e046c1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue; uint8_t haswell_broadwell_cpu; -static int -mlx5_class_check_handler(__rte_unused const char *key, const char *value, - void *opaque) -{ - enum mlx5_class *ret = opaque; - - if (strcmp(value, "vdpa") == 0) { - *ret = MLX5_CLASS_VDPA; - } else if (strcmp(value, "net") == 0) { - *ret = MLX5_CLASS_NET; - } else { - DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device" - " class argument setting?", value); - *ret = MLX5_CLASS_INVALID; - } - return 0; -} - -enum mlx5_class -mlx5_class_get(struct rte_devargs *devargs) -{ - struct rte_kvargs *kvlist; - const char *key = MLX5_CLASS_ARG_NAME; - enum mlx5_class ret = MLX5_CLASS_NET; - - if (devargs == NULL) - return ret; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return ret; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret); - rte_kvargs_free(kvlist); - return ret; -} - - /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 00e68edbb..5b9b7bd5a 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -244,8 +244,6 @@ struct mlx5_klm { LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page); -__rte_internal -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs); __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index ae57ebdba..01b4358a0 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,8 +1,6 @@ INTERNAL { global: - mlx5_class_get; - mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index b6042b7ef..2bd2b664d 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1308,11 +1308,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_dev_config dev_config; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } if (rte_eal_process_type() == RTE_PROC_PRIMARY) mlx5_pmd_socket_init(); ret = mlx5_init_once(); diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 695722520..31add3988 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,5 +15,4 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) - #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 5d247e0a1..70692ea1d 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_hca_attr attr; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr); if (!ibv) { DRV_LOG(ERR, "No matching IB device for PCI slot " -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (5 preceding siblings ...) 2020-07-16 7:29 ` [dpdk-dev] [PATCH v6 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 1/9] eal: introduce macros for getting value for bit Parav Pandit ` (9 more replies) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit ` (4 subsequent siblings) 11 siblings, 10 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit This series introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, a new mlx5 PCI bus is created. This bus allows binding multiple class drivers (such as net, vdpa, regex(future)) to bind to the mlx5 PCI bus driver. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Introduces new RTE constructor priority for common initialization Patch-3 Fixes compilation error Patch-4 Define mlx5 class as bit fields Patch-5 Uses new RTE common priority Patch-6 Adds mlx5 PCI bus Patch-7 Implements a mlx5 PCI bus driver Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus API instead of rte PCI bus API Patch-9 Removed class check code as its already part of the bus now Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ ------------- / \______| mlx5 |_____ / | pci bus | ------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD defines its specific PCI ID table and mlx5 bus driver probes matching class drivers. - mlx5 pci bus driver is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI bus driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci bus, a common driver is implemented which exposes class registration API. However, bus model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common bus driver. It also eliminates peer blind zone problem as bottom pci bus layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:vdpa ./testpmd -w <PCI BDF>,class=vdpa In future, ./testpmd -w <PCI BDF>,class=net:regex Changelog: v6->v7: - Updated release notes v5->v6: - Fixed compilation failure in parallel build for shared lib v4->v5: - Squash the maintainers update path with other patch which adds the bus - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (9): eal: introduce macros for getting value for bit eal: introduce RTE common initialization level common/mlx5: fix empty input style in glue wrappers common/mlx5: change mlx5 class enum values as bits common/mlx5: use common rte priority bus/mlx5_pci: add mlx5 PCI bus bus/mlx5_pci: register a PCI driver bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver common/mlx5: remove class checks from individual driver MAINTAINERS | 5 + doc/guides/rel_notes/release_20_08.rst | 5 + drivers/bus/Makefile | 4 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 41 ++ drivers/bus/mlx5_pci/meson.build | 19 + drivers/bus/mlx5_pci/mlx5_pci_bus.c | 522 ++++++++++++++++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/mlx5_common.c | 39 +- drivers/common/mlx5/mlx5_common.h | 7 +- .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/Makefile | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 26 +- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 30 +- lib/librte_eal/include/rte_bitops.h | 8 + lib/librte_eal/include/rte_common.h | 1 + mk/rte.app.mk | 1 + 24 files changed, 737 insertions(+), 85 deletions(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 1/9] eal: introduce macros for getting value for bit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level Parav Pandit ` (8 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit, Morten Brørup There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> Acked-by: Morten Brørup <mb@smartsharesystems.com> --- Changelog: v4->v5: - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v1->v2: - Addressed comments from Thomas and Gaten. - Avoided new file, added macro to rte_bitops.h --- lib/librte_eal/include/rte_bitops.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..ca46a110f 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,14 @@ #include <rte_debug.h> #include <rte_compat.h> +/** + * Get the uint64_t value for a specified bit set. + * + * @param nr + * The bit number in range of 0 to 63. + */ +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 1/9] eal: introduce macros for getting value for bit Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-20 16:21 ` Ferruh Yigit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 3/9] common/mlx5: fix empty input style in glue wrappers Parav Pandit ` (7 subsequent siblings) 9 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Currently mlx5_common uses CLASS priority to initialize common code before initializing the PMD. However mlx5_common is not really a class, it is the pre-initialization code needed for the PMDs. In subsequent patch a needed initialization sequence is: (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Hence, introduce a new RTE priority level RTE_PRIO_COMMON which can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs for class driver initialization. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- lib/librte_eal/include/rte_common.h | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h index 8f487a563..522afe58e 100644 --- a/lib/librte_eal/include/rte_common.h +++ b/lib/librte_eal/include/rte_common.h @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; #define RTE_PRIORITY_LOG 101 #define RTE_PRIORITY_BUS 110 +#define RTE_PRIORITY_COMMON 119 #define RTE_PRIORITY_CLASS 120 #define RTE_PRIORITY_LAST 65535 -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level Parav Pandit @ 2020-07-20 16:21 ` Ferruh Yigit 2020-07-20 16:48 ` Thomas Monjalon 0 siblings, 1 reply; 193+ messages in thread From: Ferruh Yigit @ 2020-07-20 16:21 UTC (permalink / raw) To: Parav Pandit, dev, grive, thomas Cc: rasland, orika, matan, joyce.kong, David Marchand On 7/17/2020 2:49 PM, Parav Pandit wrote: > Currently mlx5_common uses CLASS priority to initialize > common code before initializing the PMD. > However mlx5_common is not really a class, it is the pre-initialization > code needed for the PMDs. > > In subsequent patch a needed initialization sequence is: > (a) Initialize bus (say pci) > (b) Initialize common code of a driver (mlx5_common) > (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) > Information registered by these PMDs is used by mlx5_bus_pci PMD. > This mlx5 class PMDs should not confused with rte_class. > (d) Register mlx5 PCI bus PMD > > Hence, introduce a new RTE priority level RTE_PRIO_COMMON which > can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs > for class driver initialization. > > Signed-off-by: Parav Pandit <parav@mellanox.com> > Acked-by: Matan Azrad <matan@mellanox.com> > --- > Changelog: > v2->v3: > - new patch > --- > lib/librte_eal/include/rte_common.h | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h > index 8f487a563..522afe58e 100644 > --- a/lib/librte_eal/include/rte_common.h > +++ b/lib/librte_eal/include/rte_common.h > @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; > > #define RTE_PRIORITY_LOG 101 > #define RTE_PRIORITY_BUS 110 > +#define RTE_PRIORITY_COMMON 119 > #define RTE_PRIORITY_CLASS 120 > #define RTE_PRIORITY_LAST 65535 > > I guess the name "common" selected because of the intention to use it by the common piece of the driver, but only from eal perspective the name "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose. Also the value doesn't leave any gap between the class priority, what else can be needed in the future in between, right? @Thomas, @David, I am reluctant to get this eal change through the next-net, can you please review/ack it first? Thanks, ferruh ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-20 16:21 ` Ferruh Yigit @ 2020-07-20 16:48 ` Thomas Monjalon 2020-07-20 16:58 ` Ferruh Yigit 2020-07-20 19:08 ` David Marchand 0 siblings, 2 replies; 193+ messages in thread From: Thomas Monjalon @ 2020-07-20 16:48 UTC (permalink / raw) To: Parav Pandit, Ferruh Yigit Cc: dev, grive, rasland, orika, matan, joyce.kong, David Marchand 20/07/2020 18:21, Ferruh Yigit: > On 7/17/2020 2:49 PM, Parav Pandit wrote: > > Currently mlx5_common uses CLASS priority to initialize > > common code before initializing the PMD. > > However mlx5_common is not really a class, it is the pre-initialization > > code needed for the PMDs. > > > > In subsequent patch a needed initialization sequence is: > > (a) Initialize bus (say pci) > > (b) Initialize common code of a driver (mlx5_common) > > (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) > > Information registered by these PMDs is used by mlx5_bus_pci PMD. > > This mlx5 class PMDs should not confused with rte_class. > > (d) Register mlx5 PCI bus PMD > > > > Hence, introduce a new RTE priority level RTE_PRIO_COMMON which > > can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs > > for class driver initialization. > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > Acked-by: Matan Azrad <matan@mellanox.com> > > --- > > Changelog: > > v2->v3: > > - new patch > > --- > > lib/librte_eal/include/rte_common.h | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h > > index 8f487a563..522afe58e 100644 > > --- a/lib/librte_eal/include/rte_common.h > > +++ b/lib/librte_eal/include/rte_common.h > > @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; > > > > #define RTE_PRIORITY_LOG 101 > > #define RTE_PRIORITY_BUS 110 > > +#define RTE_PRIORITY_COMMON 119 > > #define RTE_PRIORITY_CLASS 120 > > #define RTE_PRIORITY_LAST 65535 > > > > > > I guess the name "common" selected because of the intention to use it by the > common piece of the driver, but only from eal perspective the name > "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose. You're right. > Also the value doesn't leave any gap between the class priority, what else can > be needed in the future in between, right? And we can imagine a bus requiring a common lib to be initialized before. > @Thomas, @David, I am reluctant to get this eal change through the next-net, can > you please review/ack it first? What about skipping this patch and using "RTE_PRIORITY_CLASS - 1" in the code? ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-20 16:48 ` Thomas Monjalon @ 2020-07-20 16:58 ` Ferruh Yigit 2020-07-20 17:26 ` Ori Kam 2020-07-20 19:08 ` David Marchand 1 sibling, 1 reply; 193+ messages in thread From: Ferruh Yigit @ 2020-07-20 16:58 UTC (permalink / raw) To: Thomas Monjalon, Parav Pandit Cc: dev, grive, rasland, orika, matan, joyce.kong, David Marchand On 7/20/2020 5:48 PM, Thomas Monjalon wrote: > 20/07/2020 18:21, Ferruh Yigit: >> On 7/17/2020 2:49 PM, Parav Pandit wrote: >>> Currently mlx5_common uses CLASS priority to initialize >>> common code before initializing the PMD. >>> However mlx5_common is not really a class, it is the pre-initialization >>> code needed for the PMDs. >>> >>> In subsequent patch a needed initialization sequence is: >>> (a) Initialize bus (say pci) >>> (b) Initialize common code of a driver (mlx5_common) >>> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) >>> Information registered by these PMDs is used by mlx5_bus_pci PMD. >>> This mlx5 class PMDs should not confused with rte_class. >>> (d) Register mlx5 PCI bus PMD >>> >>> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which >>> can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs >>> for class driver initialization. >>> >>> Signed-off-by: Parav Pandit <parav@mellanox.com> >>> Acked-by: Matan Azrad <matan@mellanox.com> >>> --- >>> Changelog: >>> v2->v3: >>> - new patch >>> --- >>> lib/librte_eal/include/rte_common.h | 1 + >>> 1 file changed, 1 insertion(+) >>> >>> diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h >>> index 8f487a563..522afe58e 100644 >>> --- a/lib/librte_eal/include/rte_common.h >>> +++ b/lib/librte_eal/include/rte_common.h >>> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; >>> >>> #define RTE_PRIORITY_LOG 101 >>> #define RTE_PRIORITY_BUS 110 >>> +#define RTE_PRIORITY_COMMON 119 >>> #define RTE_PRIORITY_CLASS 120 >>> #define RTE_PRIORITY_LAST 65535 >>> >>> >> >> I guess the name "common" selected because of the intention to use it by the >> common piece of the driver, but only from eal perspective the name >> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose. > > You're right. > >> Also the value doesn't leave any gap between the class priority, what else can >> be needed in the future in between, right? > > And we can imagine a bus requiring a common lib > to be initialized before. > >> @Thomas, @David, I am reluctant to get this eal change through the next-net, can >> you please review/ack it first? > > What about skipping this patch and using "RTE_PRIORITY_CLASS - 1" > in the code? > For now I think it is OK, in the future if more priority dependency involved we can define the macro. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-20 16:58 ` Ferruh Yigit @ 2020-07-20 17:26 ` Ori Kam 2020-07-20 18:28 ` Ferruh Yigit 0 siblings, 1 reply; 193+ messages in thread From: Ori Kam @ 2020-07-20 17:26 UTC (permalink / raw) To: Ferruh Yigit, Thomas Monjalon, Parav Pandit Cc: dev, grive, Raslan Darawsheh, Matan Azrad, joyce.kong, David Marchand Hi > -----Original Message----- > From: Ferruh Yigit <ferruh.yigit@intel.com> > On 7/20/2020 5:48 PM, Thomas Monjalon wrote: > > 20/07/2020 18:21, Ferruh Yigit: > >> On 7/17/2020 2:49 PM, Parav Pandit wrote: > >>> Currently mlx5_common uses CLASS priority to initialize > >>> common code before initializing the PMD. > >>> However mlx5_common is not really a class, it is the pre-initialization > >>> code needed for the PMDs. > >>> > >>> In subsequent patch a needed initialization sequence is: > >>> (a) Initialize bus (say pci) > >>> (b) Initialize common code of a driver (mlx5_common) > >>> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) > >>> Information registered by these PMDs is used by mlx5_bus_pci PMD. > >>> This mlx5 class PMDs should not confused with rte_class. > >>> (d) Register mlx5 PCI bus PMD > >>> > >>> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which > >>> can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs > >>> for class driver initialization. > >>> > >>> Signed-off-by: Parav Pandit <parav@mellanox.com> > >>> Acked-by: Matan Azrad <matan@mellanox.com> > >>> --- > >>> Changelog: > >>> v2->v3: > >>> - new patch > >>> --- > >>> lib/librte_eal/include/rte_common.h | 1 + > >>> 1 file changed, 1 insertion(+) > >>> > >>> diff --git a/lib/librte_eal/include/rte_common.h > b/lib/librte_eal/include/rte_common.h > >>> index 8f487a563..522afe58e 100644 > >>> --- a/lib/librte_eal/include/rte_common.h > >>> +++ b/lib/librte_eal/include/rte_common.h > >>> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; > >>> > >>> #define RTE_PRIORITY_LOG 101 > >>> #define RTE_PRIORITY_BUS 110 > >>> +#define RTE_PRIORITY_COMMON 119 > >>> #define RTE_PRIORITY_CLASS 120 > >>> #define RTE_PRIORITY_LAST 65535 > >>> > >>> > >> > >> I guess the name "common" selected because of the intention to use it by > the > >> common piece of the driver, but only from eal perspective the name > >> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose. > > > > You're right. > > > >> Also the value doesn't leave any gap between the class priority, what else > can > >> be needed in the future in between, right? > > > > And we can imagine a bus requiring a common lib > > to be initialized before. > > > >> @Thomas, @David, I am reluctant to get this eal change through the next- > net, can > >> you please review/ack it first? > > > > What about skipping this patch and using "RTE_PRIORITY_CLASS - 1" > > in the code? > > > > For now I think it is OK, in the future if more priority dependency involved we > can define the macro. > I'm concerned what if someone else will add priority there may be conflict and. Also using -1 means that no one knows that there is use in such priority. What about setting the value to 115? ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-20 17:26 ` Ori Kam @ 2020-07-20 18:28 ` Ferruh Yigit 2020-07-20 19:19 ` Ori Kam 0 siblings, 1 reply; 193+ messages in thread From: Ferruh Yigit @ 2020-07-20 18:28 UTC (permalink / raw) To: Ori Kam, Thomas Monjalon, Parav Pandit Cc: dev, grive, Raslan Darawsheh, Matan Azrad, joyce.kong, David Marchand On 7/20/2020 6:26 PM, Ori Kam wrote: > Hi > >> -----Original Message----- >> From: Ferruh Yigit <ferruh.yigit@intel.com> >> On 7/20/2020 5:48 PM, Thomas Monjalon wrote: >>> 20/07/2020 18:21, Ferruh Yigit: >>>> On 7/17/2020 2:49 PM, Parav Pandit wrote: >>>>> Currently mlx5_common uses CLASS priority to initialize >>>>> common code before initializing the PMD. >>>>> However mlx5_common is not really a class, it is the pre-initialization >>>>> code needed for the PMDs. >>>>> >>>>> In subsequent patch a needed initialization sequence is: >>>>> (a) Initialize bus (say pci) >>>>> (b) Initialize common code of a driver (mlx5_common) >>>>> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) >>>>> Information registered by these PMDs is used by mlx5_bus_pci PMD. >>>>> This mlx5 class PMDs should not confused with rte_class. >>>>> (d) Register mlx5 PCI bus PMD >>>>> >>>>> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which >>>>> can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs >>>>> for class driver initialization. >>>>> >>>>> Signed-off-by: Parav Pandit <parav@mellanox.com> >>>>> Acked-by: Matan Azrad <matan@mellanox.com> >>>>> --- >>>>> Changelog: >>>>> v2->v3: >>>>> - new patch >>>>> --- >>>>> lib/librte_eal/include/rte_common.h | 1 + >>>>> 1 file changed, 1 insertion(+) >>>>> >>>>> diff --git a/lib/librte_eal/include/rte_common.h >> b/lib/librte_eal/include/rte_common.h >>>>> index 8f487a563..522afe58e 100644 >>>>> --- a/lib/librte_eal/include/rte_common.h >>>>> +++ b/lib/librte_eal/include/rte_common.h >>>>> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; >>>>> >>>>> #define RTE_PRIORITY_LOG 101 >>>>> #define RTE_PRIORITY_BUS 110 >>>>> +#define RTE_PRIORITY_COMMON 119 >>>>> #define RTE_PRIORITY_CLASS 120 >>>>> #define RTE_PRIORITY_LAST 65535 >>>>> >>>>> >>>> >>>> I guess the name "common" selected because of the intention to use it by >> the >>>> common piece of the driver, but only from eal perspective the name >>>> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose. >>> >>> You're right. >>> >>>> Also the value doesn't leave any gap between the class priority, what else >> can >>>> be needed in the future in between, right? >>> >>> And we can imagine a bus requiring a common lib >>> to be initialized before. >>> >>>> @Thomas, @David, I am reluctant to get this eal change through the next- >> net, can >>>> you please review/ack it first? >>> >>> What about skipping this patch and using "RTE_PRIORITY_CLASS - 1" >>> in the code? >>> >> >> For now I think it is OK, in the future if more priority dependency involved we >> can define the macro. >> > I'm concerned what if someone else will add priority there may be conflict and. > Also using -1 means that no one knows that there is use in such priority. Is the new constructor priority level a common need, or just specific to this "mlx5 pci" bus usage? I understand the need of new constructor priority level, but it seems it is not clear enough to define it as a generic level, that is why I believe this can be local to PMD for now, otherwise your concerns looks valid. Also I am thinking if the multi class support can be done as generic bus feature, instead of defining a new PMD specific bus for it, but I am aware it is too late for this question. > What about setting the value to 115? > ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-20 18:28 ` Ferruh Yigit @ 2020-07-20 19:19 ` Ori Kam 0 siblings, 0 replies; 193+ messages in thread From: Ori Kam @ 2020-07-20 19:19 UTC (permalink / raw) To: Ferruh Yigit, Thomas Monjalon, Parav Pandit Cc: dev, grive, Raslan Darawsheh, Matan Azrad, joyce.kong, David Marchand > -----Original Message----- > From: Ferruh Yigit <ferruh.yigit@intel.com> > > On 7/20/2020 6:26 PM, Ori Kam wrote: > > Hi > > > >> -----Original Message----- > >> From: Ferruh Yigit <ferruh.yigit@intel.com> > >> On 7/20/2020 5:48 PM, Thomas Monjalon wrote: > >>> 20/07/2020 18:21, Ferruh Yigit: > >>>> On 7/17/2020 2:49 PM, Parav Pandit wrote: > >>>>> Currently mlx5_common uses CLASS priority to initialize > >>>>> common code before initializing the PMD. > >>>>> However mlx5_common is not really a class, it is the pre-initialization > >>>>> code needed for the PMDs. > >>>>> > >>>>> In subsequent patch a needed initialization sequence is: > >>>>> (a) Initialize bus (say pci) > >>>>> (b) Initialize common code of a driver (mlx5_common) > >>>>> (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) > >>>>> Information registered by these PMDs is used by mlx5_bus_pci PMD. > >>>>> This mlx5 class PMDs should not confused with rte_class. > >>>>> (d) Register mlx5 PCI bus PMD > >>>>> > >>>>> Hence, introduce a new RTE priority level RTE_PRIO_COMMON which > >>>>> can be used for common initialization and RTE_PRIO_CLASS by mlx5 > PMDs > >>>>> for class driver initialization. > >>>>> > >>>>> Signed-off-by: Parav Pandit <parav@mellanox.com> > >>>>> Acked-by: Matan Azrad <matan@mellanox.com> > >>>>> --- > >>>>> Changelog: > >>>>> v2->v3: > >>>>> - new patch > >>>>> --- > >>>>> lib/librte_eal/include/rte_common.h | 1 + > >>>>> 1 file changed, 1 insertion(+) > >>>>> > >>>>> diff --git a/lib/librte_eal/include/rte_common.h > >> b/lib/librte_eal/include/rte_common.h > >>>>> index 8f487a563..522afe58e 100644 > >>>>> --- a/lib/librte_eal/include/rte_common.h > >>>>> +++ b/lib/librte_eal/include/rte_common.h > >>>>> @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; > >>>>> > >>>>> #define RTE_PRIORITY_LOG 101 > >>>>> #define RTE_PRIORITY_BUS 110 > >>>>> +#define RTE_PRIORITY_COMMON 119 > >>>>> #define RTE_PRIORITY_CLASS 120 > >>>>> #define RTE_PRIORITY_LAST 65535 > >>>>> > >>>>> > >>>> > >>>> I guess the name "common" selected because of the intention to use it by > >> the > >>>> common piece of the driver, but only from eal perspective the name > >>>> "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose. > >>> > >>> You're right. > >>> > >>>> Also the value doesn't leave any gap between the class priority, what else > >> can > >>>> be needed in the future in between, right? > >>> > >>> And we can imagine a bus requiring a common lib > >>> to be initialized before. > >>> > >>>> @Thomas, @David, I am reluctant to get this eal change through the > next- > >> net, can > >>>> you please review/ack it first? > >>> > >>> What about skipping this patch and using "RTE_PRIORITY_CLASS - 1" > >>> in the code? > >>> > >> > >> For now I think it is OK, in the future if more priority dependency involved > we > >> can define the macro. > >> > > I'm concerned what if someone else will add priority there may be conflict > and. > > Also using -1 means that no one knows that there is use in such priority. > > Is the new constructor priority level a common need, or just specific to this > "mlx5 pci" bus usage? I think it can be used by any other vendor that needs a bus for his own PMDs. > I understand the need of new constructor priority level, but it seems it is not > clear enough to define it as a generic level, that is why I believe this can be > local to PMD for now, otherwise your concerns looks valid. > If I understand it can be local to the bus, it is not exposed to the PMD. But like I said above this may be used by other vendors that wish to combine some of their devices. > Also I am thinking if the multi class support can be done as generic bus > feature, instead of defining a new PMD specific bus for it, but I am aware it is > too late for this question. > > > What about setting the value to 115? > > ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-20 16:48 ` Thomas Monjalon 2020-07-20 16:58 ` Ferruh Yigit @ 2020-07-20 19:08 ` David Marchand 2020-07-20 19:30 ` Ori Kam 1 sibling, 1 reply; 193+ messages in thread From: David Marchand @ 2020-07-20 19:08 UTC (permalink / raw) To: Thomas Monjalon, Ori Kam Cc: Parav Pandit, Ferruh Yigit, dev, Gaetan Rivet, Raslan, Matan Azrad, Joyce Kong On Mon, Jul 20, 2020 at 6:48 PM Thomas Monjalon <thomas@monjalon.net> wrote: > > 20/07/2020 18:21, Ferruh Yigit: > > On 7/17/2020 2:49 PM, Parav Pandit wrote: > > > Currently mlx5_common uses CLASS priority to initialize > > > common code before initializing the PMD. > > > However mlx5_common is not really a class, it is the pre-initialization > > > code needed for the PMDs. > > > > > > In subsequent patch a needed initialization sequence is: > > > (a) Initialize bus (say pci) > > > (b) Initialize common code of a driver (mlx5_common) > > > (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) > > > Information registered by these PMDs is used by mlx5_bus_pci PMD. > > > This mlx5 class PMDs should not confused with rte_class. > > > (d) Register mlx5 PCI bus PMD > > > > > > Hence, introduce a new RTE priority level RTE_PRIO_COMMON which > > > can be used for common initialization and RTE_PRIO_CLASS by mlx5 PMDs > > > for class driver initialization. > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > Acked-by: Matan Azrad <matan@mellanox.com> > > > --- > > > Changelog: > > > v2->v3: > > > - new patch > > > --- > > > lib/librte_eal/include/rte_common.h | 1 + > > > 1 file changed, 1 insertion(+) > > > > > > diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h > > > index 8f487a563..522afe58e 100644 > > > --- a/lib/librte_eal/include/rte_common.h > > > +++ b/lib/librte_eal/include/rte_common.h > > > @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; > > > > > > #define RTE_PRIORITY_LOG 101 > > > #define RTE_PRIORITY_BUS 110 > > > +#define RTE_PRIORITY_COMMON 119 > > > #define RTE_PRIORITY_CLASS 120 > > > #define RTE_PRIORITY_LAST 65535 > > > > > > > > > > I guess the name "common" selected because of the intention to use it by the > > common piece of the driver, but only from eal perspective the name > > "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose. > > You're right. > > > Also the value doesn't leave any gap between the class priority, what else can > > be needed in the future in between, right? > > And we can imagine a bus requiring a common lib > to be initialized before. > > > @Thomas, @David, I am reluctant to get this eal change through the next-net, can > > you please review/ack it first? > > What about skipping this patch and using "RTE_PRIORITY_CLASS - 1" > in the code? net and vdpa code expect the common code being initialised. It is a dependency internal to mlx5 drivers, I see nothing generic. The common driver can provide a init function called by net and vdpa drivers in constructor context (not that I like calling complex init, but it should be equivalent to current state). It expresses a clear dependency and there is no constructor semantic to invent, nor runtime breakage possible because someone tweaks the priority order in the future. There is also some hack about a haswell/broadwell detection in LOG priority that makes no sense. I compile-tested following: $ git diff next-net-mlx/master -- lib/librte_eal/ drivers/*/mlx5/ diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 79cd5ba344..3fb9a8ab89 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -49,14 +49,6 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE); } -/** - * Initialization routine for run-time dependency on glue library. - */ -RTE_INIT_PRIO(mlx5_glue_init, COMMON) -{ - mlx5_glue_constructor(); -} - /** * This function is responsible of initializing the variable * haswell_broadwell_cpu by checking if the cpu is intel @@ -67,7 +59,7 @@ RTE_INIT_PRIO(mlx5_glue_init, COMMON) * if the cpu is haswell or broadwell the variable will be set to 1 * otherwise it will be 0. */ -RTE_INIT_PRIO(mlx5_is_haswell_broadwell_cpu, LOG) +static void mlx5_is_haswell_broadwell_cpu(void) { #ifdef RTE_ARCH_X86_64 unsigned int broadwell_models[4] = {0x3d, 0x47, 0x4F, 0x56}; @@ -114,6 +106,21 @@ RTE_INIT_PRIO(mlx5_is_haswell_broadwell_cpu, LOG) haswell_broadwell_cpu = 0; } +/** + * Initialization routine for run-time dependency on glue library. + */ +void mlx5_common_init(void) +{ + static bool init_once = false; + + if (init_once) + return; + + mlx5_glue_constructor(); + mlx5_is_haswell_broadwell_cpu(); + init_once = true; +} + /** * Allocate page of door-bells and register it using DevX API. * diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 5b9b7bd5a9..30961bc8cc 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -254,6 +254,10 @@ int64_t mlx5_get_dbr(void *ctx, struct mlx5_dbr_page_list *head, __rte_internal int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id, uint64_t offset); + +__rte_internal +void mlx5_common_init(void); + extern uint8_t haswell_broadwell_cpu; #endif /* RTE_PMD_MLX5_COMMON_H_ */ diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index fe62fa2b2f..039d221333 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,6 +1,7 @@ INTERNAL { global: + mlx5_common_init; mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 846398dd3d..d0e9345a55 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -1989,6 +1989,8 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE) */ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) { + mlx5_common_init(); + /* Build the static tables for Verbs conversion. */ mlx5_set_ptype_table(); mlx5_set_cksum_table(); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 70692ea1d2..9dc3e8fa56 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -844,6 +844,8 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) */ RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) { + mlx5_common_init(); + if (mlx5_glue) rte_mlx5_pci_driver_register(&mlx5_vdpa_driver); } diff --git a/lib/librte_eal/include/rte_common.h b/lib/librte_eal/include/rte_common.h index 522afe58ed..8f487a563d 100644 --- a/lib/librte_eal/include/rte_common.h +++ b/lib/librte_eal/include/rte_common.h @@ -135,7 +135,6 @@ typedef uint16_t unaligned_uint16_t; #define RTE_PRIORITY_LOG 101 #define RTE_PRIORITY_BUS 110 -#define RTE_PRIORITY_COMMON 119 #define RTE_PRIORITY_CLASS 120 #define RTE_PRIORITY_LAST 65535 -- David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-20 19:08 ` David Marchand @ 2020-07-20 19:30 ` Ori Kam 2020-07-21 9:34 ` David Marchand 0 siblings, 1 reply; 193+ messages in thread From: Ori Kam @ 2020-07-20 19:30 UTC (permalink / raw) To: David Marchand, Thomas Monjalon Cc: Parav Pandit, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Matan Azrad, Joyce Kong Hi David, > -----Original Message----- > From: David Marchand <david.marchand@redhat.com> > > On Mon, Jul 20, 2020 at 6:48 PM Thomas Monjalon <thomas@monjalon.net> > wrote: > > > > 20/07/2020 18:21, Ferruh Yigit: > > > On 7/17/2020 2:49 PM, Parav Pandit wrote: > > > > Currently mlx5_common uses CLASS priority to initialize > > > > common code before initializing the PMD. > > > > However mlx5_common is not really a class, it is the pre-initialization > > > > code needed for the PMDs. > > > > > > > > In subsequent patch a needed initialization sequence is: > > > > (a) Initialize bus (say pci) > > > > (b) Initialize common code of a driver (mlx5_common) > > > > (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) > > > > Information registered by these PMDs is used by mlx5_bus_pci PMD. > > > > This mlx5 class PMDs should not confused with rte_class. > > > > (d) Register mlx5 PCI bus PMD > > > > > > > > Hence, introduce a new RTE priority level RTE_PRIO_COMMON which > > > > can be used for common initialization and RTE_PRIO_CLASS by mlx5 > PMDs > > > > for class driver initialization. > > > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > > Acked-by: Matan Azrad <matan@mellanox.com> > > > > --- > > > > Changelog: > > > > v2->v3: > > > > - new patch > > > > --- > > > > lib/librte_eal/include/rte_common.h | 1 + > > > > 1 file changed, 1 insertion(+) > > > > > > > > diff --git a/lib/librte_eal/include/rte_common.h > b/lib/librte_eal/include/rte_common.h > > > > index 8f487a563..522afe58e 100644 > > > > --- a/lib/librte_eal/include/rte_common.h > > > > +++ b/lib/librte_eal/include/rte_common.h > > > > @@ -135,6 +135,7 @@ typedef uint16_t unaligned_uint16_t; > > > > > > > > #define RTE_PRIORITY_LOG 101 > > > > #define RTE_PRIORITY_BUS 110 > > > > +#define RTE_PRIORITY_COMMON 119 > > > > #define RTE_PRIORITY_CLASS 120 > > > > #define RTE_PRIORITY_LAST 65535 > > > > > > > > > > > > > > I guess the name "common" selected because of the intention to use it by > the > > > common piece of the driver, but only from eal perspective the name > > > "PRIORITY_COMMON" looks so vague, it doesn't describe any purpose. > > > > You're right. > > > > > Also the value doesn't leave any gap between the class priority, what else > can > > > be needed in the future in between, right? > > > > And we can imagine a bus requiring a common lib > > to be initialized before. > > > > > @Thomas, @David, I am reluctant to get this eal change through the next- > net, can > > > you please review/ack it first? > > > > What about skipping this patch and using "RTE_PRIORITY_CLASS - 1" > > in the code? > > net and vdpa code expect the common code being initialised. > It is a dependency internal to mlx5 drivers, I see nothing generic. > First the idea was to declare a new bus not a PMD. The issue is not from common code but from loading more than one device on the same PCI. So the logic is Mellanox PMD are registering to the new bus, the new bus register to the PCI one. So the new bus must have middle priority between the PMDs (class) and the PCI (bus) bus. May be a better name should have been: RTE_PRIORITY_ MANUFACTORE _BUS > The common driver can provide a init function called by net and vdpa > drivers in constructor context (not that I like calling complex init, > but it should be equivalent to current state). > It expresses a clear dependency and there is no constructor semantic > to invent, nor runtime breakage possible because someone tweaks the > priority order in the future. > > There is also some hack about a haswell/broadwell detection in LOG > priority that makes no sense. > > I compile-tested following: > > $ git diff next-net-mlx/master -- lib/librte_eal/ drivers/*/mlx5/ > diff --git a/drivers/common/mlx5/mlx5_common.c > b/drivers/common/mlx5/mlx5_common.c > index 79cd5ba344..3fb9a8ab89 100644 > --- a/drivers/common/mlx5/mlx5_common.c > +++ b/drivers/common/mlx5/mlx5_common.c > @@ -49,14 +49,6 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) > rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE); > } > > -/** > - * Initialization routine for run-time dependency on glue library. > - */ > -RTE_INIT_PRIO(mlx5_glue_init, COMMON) > -{ > - mlx5_glue_constructor(); > -} > - > /** > * This function is responsible of initializing the variable > * haswell_broadwell_cpu by checking if the cpu is intel > @@ -67,7 +59,7 @@ RTE_INIT_PRIO(mlx5_glue_init, COMMON) > * if the cpu is haswell or broadwell the variable will be set to 1 > * otherwise it will be 0. > */ > -RTE_INIT_PRIO(mlx5_is_haswell_broadwell_cpu, LOG) > +static void mlx5_is_haswell_broadwell_cpu(void) > { > #ifdef RTE_ARCH_X86_64 > unsigned int broadwell_models[4] = {0x3d, 0x47, 0x4F, 0x56}; > @@ -114,6 +106,21 @@ RTE_INIT_PRIO(mlx5_is_haswell_broadwell_cpu, > LOG) > haswell_broadwell_cpu = 0; > } > > +/** > + * Initialization routine for run-time dependency on glue library. > + */ > +void mlx5_common_init(void) > +{ > + static bool init_once = false; > + > + if (init_once) > + return; > + > + mlx5_glue_constructor(); > + mlx5_is_haswell_broadwell_cpu(); > + init_once = true; > +} > + > /** > * Allocate page of door-bells and register it using DevX API. > * > diff --git a/drivers/common/mlx5/mlx5_common.h > b/drivers/common/mlx5/mlx5_common.h > index 5b9b7bd5a9..30961bc8cc 100644 > --- a/drivers/common/mlx5/mlx5_common.h > +++ b/drivers/common/mlx5/mlx5_common.h > @@ -254,6 +254,10 @@ int64_t mlx5_get_dbr(void *ctx, struct > mlx5_dbr_page_list *head, > __rte_internal > int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t > umem_id, > uint64_t offset); > + > +__rte_internal > +void mlx5_common_init(void); > + > extern uint8_t haswell_broadwell_cpu; > > #endif /* RTE_PMD_MLX5_COMMON_H_ */ > diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map > b/drivers/common/mlx5/rte_common_mlx5_version.map > index fe62fa2b2f..039d221333 100644 > --- a/drivers/common/mlx5/rte_common_mlx5_version.map > +++ b/drivers/common/mlx5/rte_common_mlx5_version.map > @@ -1,6 +1,7 @@ > INTERNAL { > global: > > + mlx5_common_init; > mlx5_common_verbs_reg_mr; > mlx5_common_verbs_dereg_mr; > > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c > index 846398dd3d..d0e9345a55 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -1989,6 +1989,8 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, > NOTICE) > */ > RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) > { > + mlx5_common_init(); > + > /* Build the static tables for Verbs conversion. */ > mlx5_set_ptype_table(); > mlx5_set_cksum_table(); > diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c > index 70692ea1d2..9dc3e8fa56 100644 > --- a/drivers/vdpa/mlx5/mlx5_vdpa.c > +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c > @@ -844,6 +844,8 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, > pmd.vdpa.mlx5, NOTICE) > */ > RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) > { > + mlx5_common_init(); > + > if (mlx5_glue) > rte_mlx5_pci_driver_register(&mlx5_vdpa_driver); > } > diff --git a/lib/librte_eal/include/rte_common.h > b/lib/librte_eal/include/rte_common.h > index 522afe58ed..8f487a563d 100644 > --- a/lib/librte_eal/include/rte_common.h > +++ b/lib/librte_eal/include/rte_common.h > @@ -135,7 +135,6 @@ typedef uint16_t unaligned_uint16_t; > > #define RTE_PRIORITY_LOG 101 > #define RTE_PRIORITY_BUS 110 > -#define RTE_PRIORITY_COMMON 119 > #define RTE_PRIORITY_CLASS 120 > #define RTE_PRIORITY_LAST 65535 > > > -- > David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-20 19:30 ` Ori Kam @ 2020-07-21 9:34 ` David Marchand 2020-07-21 11:18 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: David Marchand @ 2020-07-21 9:34 UTC (permalink / raw) To: Ori Kam Cc: Thomas Monjalon, Parav Pandit, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Matan Azrad, Joyce Kong On Mon, Jul 20, 2020 at 9:30 PM Ori Kam <orika@mellanox.com> wrote: > > net and vdpa code expect the common code being initialised. > > It is a dependency internal to mlx5 drivers, I see nothing generic. > > > First the idea was to declare a new bus not a PMD. > The issue is not from common code but from loading more than one > device on the same PCI. > So the logic is Mellanox PMD are registering to the new bus, the new bus > register to the PCI one. - Is there a problem with the snippet I sent yesterday that removes the COMMON priority? - Currently in the main branch, the net/mlx5 driver is a pci driver registered to the pci bus driver. This registration happens in a constructor which puts the net/mlx5 pci driver struct (with a populated id map) into the pci driver list. Later (out of constructor context), a probe on the pci bus uses this driver list. A new pci driver is introduced by Mellanox to solve resources dispatching between mlx5 drivers. I'd rather not call this new pci driver a bus, it is not registered as such (no rte_bus object) => no reason its constructor should be called from the RTE_PRIO_BUS priority. This driver not being a bus, I would not put it in drivers/bus/, rather in the common code. If there is a build dependency issue with the pci bus driver, you can probably use the trick from qat (https://git.dpdk.org/dpdk/tree/drivers/Makefile#n17) and write an explicit dependency in drivers/Makefile. meson should be just fine. net/mlx5 and vdpa/mlx5 are not classes => no reason their constructor should be called from the RTE_PRIO_CLASS priority. The only classes we have are ethdev and vdpa. The new pci driver simply serves as a proxy for mlx5 net and vdpa drivers. From what I see, the new pci driver constructs its pci device id map/drv_flags when registering to the pci bus driver based on what mlx5 net and vdpa drivers registered before. This is where a dependency was created. You can move the id_table/drv_flags construction from RTE_INIT(mlx5_bus_pci) to rte_mlx5_pci_register() itself. This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5 pci driver whether it is registered to the pci bus or not yet. -- David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-21 9:34 ` David Marchand @ 2020-07-21 11:18 ` Parav Pandit 2020-07-21 11:29 ` David Marchand 0 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-21 11:18 UTC (permalink / raw) To: David Marchand, Ori Kam Cc: Thomas Monjalon, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Matan Azrad, Joyce Kong Hi David, On 7/21/2020 3:04 PM, David Marchand wrote: > On Mon, Jul 20, 2020 at 9:30 PM Ori Kam <orika@mellanox.com> wrote: >>> net and vdpa code expect the common code being initialised. >>> It is a dependency internal to mlx5 drivers, I see nothing generic. >>> >> First the idea was to declare a new bus not a PMD. >> The issue is not from common code but from loading more than one >> device on the same PCI. >> So the logic is Mellanox PMD are registering to the new bus, the new bus >> register to the PCI one. > > - Is there a problem with the snippet I sent yesterday that removes > the COMMON priority? > > No. The problem as you describe below is in handling the inter dependencies between mlx5_common, rte_pci and mlx5_{net/vdpa}. mlx5_net/vdpa demands common to be loaded first. mlx5_pci_bus expects mlx5_net_vdpa to register first. More below. > - Currently in the main branch, the net/mlx5 driver is a pci driver > registered to the pci bus driver. > This registration happens in a constructor which puts the net/mlx5 pci > driver struct (with a populated id map) into the pci driver list. > Later (out of constructor context), a probe on the pci bus uses this > driver list. > > > A new pci driver is introduced by Mellanox to solve resources > dispatching between mlx5 drivers. > I'd rather not call this new pci driver a bus, it is not registered as > such (no rte_bus object) => no reason its constructor should be called > from the RTE_PRIO_BUS priority. > > This driver not being a bus, I would not put it in drivers/bus/, > rather in the common code. > If there is a build dependency issue with the pci bus driver, you can > probably use the trick from qat > (https://git.dpdk.org/dpdk/tree/drivers/Makefile#n17) and write an > explicit dependency in drivers/Makefile. > meson should be just fine. > mlx5_common to depend on pci bus. (new Makefile change) mlx5_net and mlx5_vdpa to depend on mlx5_common. (already there) > > net/mlx5 and vdpa/mlx5 are not classes => no reason their constructor > should be called from the RTE_PRIO_CLASS priority. > The only classes we have are ethdev and vdpa. > > > The new pci driver simply serves as a proxy for mlx5 net and vdpa drivers. > From what I see, the new pci driver constructs its pci device id > map/drv_flags when registering to the pci bus driver based on what > mlx5 net and vdpa drivers registered before. > This is where a dependency was created. > > You can move the id_table/drv_flags construction from > RTE_INIT(mlx5_bus_pci) to rte_mlx5_pci_register() itself. ok. This is what were also discussing internally as well. > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5 > pci driver whether it is registered to the pci bus or not yet. And here pci_id_table will not be const *. Is that ok? Basically mlx5_pci_bus, mlx5_net and mlx5_vdpa will be called with same priority RTE_PRIO_LAST. driver id table and drv_flags dynamically updated as PMDs register. Should we have another API after rte_pci_register() to indicate that some field of the driver are updated, instead of just silently updating it in a PMD? ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-21 11:18 ` Parav Pandit @ 2020-07-21 11:29 ` David Marchand 2020-07-21 12:10 ` Thomas Monjalon 2020-07-21 12:13 ` Parav Pandit 0 siblings, 2 replies; 193+ messages in thread From: David Marchand @ 2020-07-21 11:29 UTC (permalink / raw) To: Parav Pandit Cc: Ori Kam, Thomas Monjalon, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Matan Azrad, Joyce Kong On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> wrote: > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5 > > pci driver whether it is registered to the pci bus or not yet. > > And here pci_id_table will not be const *. > Is that ok? This is already the case in the current patch. I see nothing wrong with it. The pci code expects this pointer to be const and will it treat it as such. On the other hand, you may remove the pci table exported by the common driver as it is useless. pmdinfogen only parses the .o and extracts the table symbol, so it won't load other mlx drivers to generate the .pmd.c file. I.e. with current patch, $ ./usertools/dpdk-pmdinfo.py ./build/drivers/librte_bus_mlx5_pci.so PMD NAME: mlx5_bus_pci > > Basically mlx5_pci_bus, mlx5_net and mlx5_vdpa will be called with same > priority RTE_PRIO_LAST. > driver id table and drv_flags dynamically updated as PMDs register. > > Should we have another API after rte_pci_register() to indicate that > some field of the driver are updated, instead of just silently updating > it in a PMD? All of this happens in constructor context, this is a really early stage used to resolve code dependencies. No big init should happen there. We can document this use of the existing API but I don't see which API to add. -- David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-21 11:29 ` David Marchand @ 2020-07-21 12:10 ` Thomas Monjalon 2020-07-21 12:13 ` Parav Pandit 1 sibling, 0 replies; 193+ messages in thread From: Thomas Monjalon @ 2020-07-21 12:10 UTC (permalink / raw) To: Parav Pandit, dev, Matan Azrad, David Marchand Cc: Ori Kam, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Joyce Kong 21/07/2020 13:29, David Marchand: > On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> wrote: > > Basically mlx5_pci_bus, mlx5_net and mlx5_vdpa will be called with same > > priority RTE_PRIO_LAST. > > driver id table and drv_flags dynamically updated as PMDs register. > > > > Should we have another API after rte_pci_register() to indicate that > > some field of the driver are updated, instead of just silently updating > > it in a PMD? > > All of this happens in constructor context, this is a really early > stage used to resolve code dependencies. > No big init should happen there. In general, there is a misunderstanding of what should be done in a constructor. I think we should document constructor usage properly. For the record, init done in constructors should be small. If the application is linked with DPDK but choose to not initialize it, it should not have driver initialized (until it calls rte_eal_init). Constructor init should be limited to adding a driver struct to a list. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-21 11:29 ` David Marchand 2020-07-21 12:10 ` Thomas Monjalon @ 2020-07-21 12:13 ` Parav Pandit 2020-07-21 12:26 ` Thomas Monjalon 1 sibling, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-21 12:13 UTC (permalink / raw) To: David Marchand Cc: Ori Kam, Thomas Monjalon, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Matan Azrad, Joyce Kong > From: David Marchand <david.marchand@redhat.com> > Sent: Tuesday, July 21, 2020 4:59 PM > > On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> wrote: > > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5 > > > pci driver whether it is registered to the pci bus or not yet. > > > > And here pci_id_table will not be const *. > > Is that ok? > > This is already the case in the current patch. > I see nothing wrong with it. > The pci code expects this pointer to be const and will it treat it as such. > o.k. Gaetan, Ferruh, Thomas, Can you please ack as well? > > On the other hand, you may remove the pci table exported by the common > driver as it is useless. > pmdinfogen only parses the .o and extracts the table symbol, so it won't load > other mlx drivers to generate the .pmd.c file. > Ok. > I.e. with current patch, > $ ./usertools/dpdk-pmdinfo.py ./build/drivers/librte_bus_mlx5_pci.so > PMD NAME: mlx5_bus_pci > > > > > > Basically mlx5_pci_bus, mlx5_net and mlx5_vdpa will be called with > > same priority RTE_PRIO_LAST. > > driver id table and drv_flags dynamically updated as PMDs register. > > > > Should we have another API after rte_pci_register() to indicate that > > some field of the driver are updated, instead of just silently > > updating it in a PMD? > > All of this happens in constructor context, this is a really early stage used to > resolve code dependencies. > No big init should happen there. > Right. Only init() is to dynamically build the pci id table to remove duplicates, how its done today. And it requires reallocating new table as part of mlx5_pci_bus_register() and updating new id table pointer in the driver structure. > We can document this use of the existing API but I don't see which API to > add. > Alright. Will wait for ack on pci id table from Ferruh, Gaetan and Thomas to post v8. Thanks a lot David for the review. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-21 12:13 ` Parav Pandit @ 2020-07-21 12:26 ` Thomas Monjalon 2020-07-21 12:51 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: Thomas Monjalon @ 2020-07-21 12:26 UTC (permalink / raw) To: Parav Pandit Cc: David Marchand, dev, Ori Kam, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Matan Azrad, Joyce Kong 21/07/2020 14:13, Parav Pandit: > From: David Marchand <david.marchand@redhat.com> > > On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> wrote: > > > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the mlx5 > > > > pci driver whether it is registered to the pci bus or not yet. > > > > > > And here pci_id_table will not be const *. > > > Is that ok? > > > > This is already the case in the current patch. > > I see nothing wrong with it. > > The pci code expects this pointer to be const and will it treat it as such. > > > o.k. > > Gaetan, Ferruh, Thomas, > Can you please ack as well? Yes of course it is OK updating the PCI table of the common layer in runtime. The most important is to keep the fixed PCI table of the PMDs the same as registered for pmdinfo usage. > > On the other hand, you may remove the pci table exported by the common > > driver as it is useless. > > pmdinfogen only parses the .o and extracts the table symbol, so it won't load > > other mlx drivers to generate the .pmd.c file. > > Ok. One more nit: please don't use "class" as variable name for drivers. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-21 12:26 ` Thomas Monjalon @ 2020-07-21 12:51 ` Parav Pandit 2020-07-21 13:07 ` Thomas Monjalon 0 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-21 12:51 UTC (permalink / raw) To: Thomas Monjalon Cc: David Marchand, dev, Ori Kam, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Matan Azrad, Joyce Kong > From: Thomas Monjalon <thomas@monjalon.net> > Sent: Tuesday, July 21, 2020 5:57 PM > > 21/07/2020 14:13, Parav Pandit: > > From: David Marchand <david.marchand@redhat.com> > > > On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> > wrote: > > > > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the > > > > > mlx5 pci driver whether it is registered to the pci bus or not yet. > > > > > > > > And here pci_id_table will not be const *. > > > > Is that ok? > > > > > > This is already the case in the current patch. > > > I see nothing wrong with it. > > > The pci code expects this pointer to be const and will it treat it as such. > > > > > o.k. > > > > Gaetan, Ferruh, Thomas, > > Can you please ack as well? > > Yes of course it is OK updating the PCI table of the common layer in runtime. > The most important is to keep the fixed PCI table of the PMDs the same as > registered for pmdinfo usage. o.k. pci id table registered with PCI bus will be build dynamically, similar to how its done in v7. Instead of doing it in constructor of mlx5_pci_bus, it will be done inside rte_mlx5_pci_bus_register(). Right? And since we don't register it as rte_bus, shall I name it as rte_mlx5_pci_driver_register()/unregister(). Right? > > > > On the other hand, you may remove the pci table exported by the > > > common driver as it is useless. > > > pmdinfogen only parses the .o and extracts the table symbol, so it > > > won't load other mlx drivers to generate the .pmd.c file. > > > > Ok. > > One more nit: please don't use "class" as variable name for drivers. Ok. I will address that. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level 2020-07-21 12:51 ` Parav Pandit @ 2020-07-21 13:07 ` Thomas Monjalon 0 siblings, 0 replies; 193+ messages in thread From: Thomas Monjalon @ 2020-07-21 13:07 UTC (permalink / raw) To: Parav Pandit Cc: David Marchand, dev, Ori Kam, Ferruh Yigit, dev, Gaetan Rivet, Raslan Darawsheh, Matan Azrad, Joyce Kong 21/07/2020 14:51, Parav Pandit: > > > From: Thomas Monjalon <thomas@monjalon.net> > > Sent: Tuesday, July 21, 2020 5:57 PM > > > > 21/07/2020 14:13, Parav Pandit: > > > From: David Marchand <david.marchand@redhat.com> > > > > On Tue, Jul 21, 2020 at 1:19 PM Parav Pandit <parav@mellanox.com> > > wrote: > > > > > > This way, net/mlx5 and vdpa/mlx5 will pass their id_map to the > > > > > > mlx5 pci driver whether it is registered to the pci bus or not yet. > > > > > > > > > > And here pci_id_table will not be const *. > > > > > Is that ok? > > > > > > > > This is already the case in the current patch. > > > > I see nothing wrong with it. > > > > The pci code expects this pointer to be const and will it treat it as such. > > > > > > > o.k. > > > > > > Gaetan, Ferruh, Thomas, > > > Can you please ack as well? > > > > Yes of course it is OK updating the PCI table of the common layer in runtime. > > The most important is to keep the fixed PCI table of the PMDs the same as > > registered for pmdinfo usage. > > o.k. pci id table registered with PCI bus will be build dynamically, similar to how its done in v7. > Instead of doing it in constructor of mlx5_pci_bus, it will be done inside rte_mlx5_pci_bus_register(). > Right? Yes > And since we don't register it as rte_bus, shall I name it as rte_mlx5_pci_driver_register()/unregister(). > Right? Yes, if merging code in the common driver, no need to keep "bus" wording. ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 3/9] common/mlx5: fix empty input style in glue wrappers 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 1/9] eal: introduce macros for getting value for bit Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 2/9] eal: introduce RTE common initialization level Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 4/9] common/mlx5: change mlx5 class enum values as bits Parav Pandit ` (6 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Following two errors are reported when compiled with gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5). drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] Fix them by adding void data type in empty argument list. Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch") Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue") Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- drivers/common/mlx5/linux/mlx5_glue.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h index c4f9b006a..734ace2a3 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.h +++ b/drivers/common/mlx5/linux/mlx5_glue.h @@ -185,11 +185,11 @@ struct mlx5_glue { void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl); void *(*dr_create_flow_action_dest_port)(void *domain, uint32_t port); - void *(*dr_create_flow_action_drop)(); + void *(*dr_create_flow_action_drop)(void); void *(*dr_create_flow_action_push_vlan) (struct mlx5dv_dr_domain *domain, rte_be32_t vlan_tag); - void *(*dr_create_flow_action_pop_vlan)(); + void *(*dr_create_flow_action_pop_vlan)(void); void *(*dr_create_flow_tbl)(void *domain, uint32_t level); int (*dr_destroy_flow_tbl)(void *tbl); void *(*dr_create_domain)(struct ibv_context *ctx, -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 4/9] common/mlx5: change mlx5 class enum values as bits 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (2 preceding siblings ...) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 3/9] common/mlx5: fix empty input style in glue wrappers Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 5/9] common/mlx5: use common rte priority Parav Pandit ` (5 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit mlx5 PCI Device supports multiple classes of devices such as net, vdpa, and/or regex. To support these multiple classes, change mlx5_class to a bitmap values so that if users asks to enable multiple of them, all supported classes can be parsed. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v1->v2: - Rebasd due to removal previous patch --- drivers/common/mlx5/mlx5_common.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 285150705..00e68edbb 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -13,6 +13,7 @@ #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h> +#include <rte_bitops.h> #include "mlx5_prm.h" #include "mlx5_devx_cmds.h" @@ -208,9 +209,9 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); #define MLX5_CLASS_ARG_NAME "class" enum mlx5_class { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, MLX5_CLASS_INVALID, + MLX5_CLASS_NET = RTE_BIT64(0), + MLX5_CLASS_VDPA = RTE_BIT64(1), }; #define MLX5_DBR_PAGE_SIZE 4096 /* Must be >= 512. */ -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 5/9] common/mlx5: use common rte priority 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (3 preceding siblings ...) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 4/9] common/mlx5: change mlx5 class enum values as bits Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 6/9] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit ` (4 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Use RTE_PRIO_COMMON for mlx5 common initialization and use RTE_PRIO_CLASS for mlx5 net, vdpa PMDs. This enables to do following initialization sequence. (a) Initialize bus (say pci) (b) Initialize common code of a driver (mlx5_common) (c) Register mlx5 class PMDs (mlx5 net, mlx5 vdpa) Information registered by these PMDs is used by mlx5_bus_pci PMD. This mlx5 class PMDs should not confused with rte_class. (d) Register mlx5 PCI bus PMD Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/mlx5_common.c | 2 +- drivers/net/mlx5/mlx5.c | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 693e2c68c..0ce5e4db1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -92,7 +92,7 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) /** * Initialization routine for run-time dependency on glue library. */ -RTE_INIT_PRIO(mlx5_glue_init, CLASS) +RTE_INIT_PRIO(mlx5_glue_init, COMMON) { mlx5_glue_constructor(); } diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 723c1dd72..1647256be 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -2074,7 +2074,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE) /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_pmd_init) +RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) { /* Build the static tables for Verbs conversion. */ mlx5_set_ptype_table(); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 67e77b11a..205a9e4ad 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -844,7 +844,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) /** * Driver initialization routine. */ -RTE_INIT(rte_mlx5_vdpa_init) +RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) { if (mlx5_glue) rte_pci_register(&mlx5_vdpa_driver); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 6/9] bus/mlx5_pci: add mlx5 PCI bus 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (4 preceding siblings ...) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 5/9] common/mlx5: use common rte priority Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 7/9] bus/mlx5_pci: register a PCI driver Parav Pandit ` (3 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Add mlx5 PCI bus which enables multiple mlx5 drivers to bind to single pci device. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v6->v7: - Updated release notes v5->v6: - Updated Makefile for parallel shared build support v4->v5: - Merged maintainers update patch with this patch v2->v3: - Addressed comments from Thomas and Asaf - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address comments from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Changed alignedment to mlx5 specific aligment instead of standard DPDK - Using uint32_t instead of mlx5_class enum v1->v2: - Address comments from Thomas and Gaetan - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus --- MAINTAINERS | 5 ++ doc/guides/rel_notes/release_20_08.rst | 5 ++ drivers/bus/Makefile | 4 + drivers/bus/meson.build | 2 +- drivers/bus/mlx5_pci/Makefile | 39 +++++++++ drivers/bus/mlx5_pci/meson.build | 19 +++++ drivers/bus/mlx5_pci/mlx5_pci_bus.c | 14 ++++ drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++++++++++++++++++ .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 ++ 9 files changed, 176 insertions(+), 1 deletion(-) create mode 100644 drivers/bus/mlx5_pci/Makefile create mode 100644 drivers/bus/mlx5_pci/meson.build create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map diff --git a/MAINTAINERS b/MAINTAINERS index 3cd402b34..9e3bd1672 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -525,6 +525,11 @@ Intel FPGA bus M: Rosen Xu <rosen.xu@intel.com> F: drivers/bus/ifpga/ +Melllanox mlx5 PCI bus driver +M: Parav Pandit <parav@mellaox.com> +M: Matan Azrad <matan@mellanox.com> +F: drivers/bus/mlx5_pci + NXP buses M: Hemant Agrawal <hemant.agrawal@nxp.com> M: Sachin Saxena <sachin.saxena@nxp.com> diff --git a/doc/guides/rel_notes/release_20_08.rst b/doc/guides/rel_notes/release_20_08.rst index 6f44ffdfc..917a5b8b3 100644 --- a/doc/guides/rel_notes/release_20_08.rst +++ b/doc/guides/rel_notes/release_20_08.rst @@ -226,6 +226,11 @@ New Features See the :doc:`../sample_app_ug/l2_forward_real_virtual` for more details of this parameter usage. +* **Added Mellanox mlx5_pci bus driver.** + + Added mlx5_pci bus driver that supports loading multiple PMDs for + Mellanox mlx5 devices. This bus PMD is auto selected when its + upper layer PMD need it. Removed Items ------------- diff --git a/drivers/bus/Makefile b/drivers/bus/Makefile index cea3b55e6..a70f213c1 100644 --- a/drivers/bus/Makefile +++ b/drivers/bus/Makefile @@ -9,6 +9,10 @@ DIRS-$(CONFIG_RTE_LIBRTE_FSLMC_BUS) += fslmc endif DIRS-$(CONFIG_RTE_LIBRTE_IFPGA_BUS) += ifpga DIRS-$(CONFIG_RTE_LIBRTE_PCI_BUS) += pci +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) +DEPDIRS-mlx5_pci := pci +DIRS-y += mlx5_pci +endif DIRS-$(CONFIG_RTE_LIBRTE_VDEV_BUS) += vdev DIRS-$(CONFIG_RTE_LIBRTE_VMBUS) += vmbus diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build index 80de2d91d..b1381838d 100644 --- a/drivers/bus/meson.build +++ b/drivers/bus/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017 Intel Corporation -drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'vdev', 'vmbus'] +drivers = ['dpaa', 'fslmc', 'ifpga', 'pci', 'mlx5_pci', 'vdev', 'vmbus'] std_deps = ['eal'] config_flag_fmt = 'RTE_LIBRTE_@0@_BUS' driver_name_fmt = 'rte_bus_@0@' diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile new file mode 100644 index 000000000..dd24811a3 --- /dev/null +++ b/drivers/bus/mlx5_pci/Makefile @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2020 Mellanox Technologies, Ltd + +include $(RTE_SDK)/mk/rte.vars.mk + +# +# library name +# +LIB = librte_bus_mlx5_pci.a + +CFLAGS += -O3 -Wall -Wextra +CFLAGS += $(WERROR_FLAGS) +CFLAGS += -Wno-strict-prototypes +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5 +CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux +CFLAGS += -I$(BUILDDIR)/drivers/bus/pci +CFLAGS += -I$(BUILDDIR)/drivers/bus +LDLIBS += -lrte_eal +LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_pci -lrte_bus_pci + +# versioning export map +EXPORT_MAP := rte_bus_mlx5_pci_version.map + +SRCS-y += mlx5_pci_bus.c + +# DEBUG which is usually provided on the command-line may enable +# CONFIG_RTE_LIBRTE_MLX5_DEBUG. +ifeq ($(DEBUG),1) +CONFIG_RTE_LIBRTE_MLX5_DEBUG := y +endif + +# +# Export include files +# +SYMLINK-y-include += rte_bus_mlx5_pci.h + +include $(RTE_SDK)/mk/rte.lib.mk diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build new file mode 100644 index 000000000..64a17cbad --- /dev/null +++ b/drivers/bus/mlx5_pci/meson.build @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2020 Mellanox Technologies Ltd + +deps += ['pci', 'bus_pci', 'common_mlx5'] +install_headers('rte_bus_mlx5_pci.h') +sources = files('mlx5_pci_bus.c') + +cflags_options = [ + '-std=c11', + '-Wno-strict-prototypes', + '-D_BSD_SOURCE', + '-D_DEFAULT_SOURCE', + '-D_XOPEN_SOURCE=600' +] +foreach option:cflags_options + if cc.has_argument(option) + cflags += option + endif +endforeach diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c new file mode 100644 index 000000000..66db3c7b0 --- /dev/null +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#include "rte_bus_mlx5_pci.h" + +static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = + TAILQ_HEAD_INITIALIZER(drv_list); + +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) +{ + TAILQ_INSERT_TAIL(&drv_list, driver, next); +} diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h new file mode 100644 index 000000000..9f8d22e2b --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h @@ -0,0 +1,84 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef _RTE_BUS_MLX5_PCI_H_ +#define _RTE_BUS_MLX5_PCI_H_ + +/** + * @file + * + * RTE Mellanox PCI Bus Interface + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) + * devices. This bus enables creating such multiple class of devices on a + * single PCI device by allowing to bind multiple class specific device + * driver to attach to mlx5_pci bus driver. + * + * ----------- ------------ ----------------- + * | mlx5 | | mlx5 | | mlx5 | + * | net pmd | | vdpa pmd | | new class pmd | + * ----------- ------------ ----------------- + * \ | / + * \ | / + * \ ------------- / + * \______| mlx5 |_____ / + * | pci bus | + * ------------- + * | + * ----------- + * | mlx5 | + * | pci dev | + * ----------- + * + * - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 pci bus driver is cental place that validates supported + * class combinations. + */ + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +#include <rte_pci.h> +#include <rte_bus_pci.h> + +#include <mlx5_common.h> + +/** + * A structure describing a mlx5 pci driver. + */ +struct rte_mlx5_pci_driver { + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ + uint32_t dev_class; /**< Class of this driver, enum mlx5_class */ + TAILQ_ENTRY(rte_mlx5_pci_driver) next; +}; + +/** + * Register a mlx5_pci device driver. + * + * @param driver + * A pointer to a rte_mlx5_pci_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver); + +#define RTE_PMD_REGISTER_MLX5_PCI(nm, drv) \ + static const char *mlx5_pci_drvinit_fn_ ## nm; \ + RTE_INIT(mlx5_pci_drvinit_fn_ ##drv) \ + { \ + (drv).driver.name = RTE_STR(nm); \ + rte_mlx5_pci_driver_register(&drv); \ + } \ + RTE_PMD_EXPORT_NAME(nm, __COUNTER__) + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _RTE_BUS_MLX5_PCI_H_ */ diff --git a/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map new file mode 100644 index 000000000..4cfd3db10 --- /dev/null +++ b/drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map @@ -0,0 +1,5 @@ +INTERNAL { + global: + + rte_mlx5_pci_driver_register; +}; -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 7/9] bus/mlx5_pci: register a PCI driver 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (5 preceding siblings ...) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 6/9] bus/mlx5_pci: add mlx5 PCI bus Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit ` (2 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Create a mlx5 bus driver framework for invoking drivers of multiple classes who have registered with the mlx5_pci bus driver. Validate user class arguments for supported class combinations. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Addressed comments from Asaf - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Addressed comments from Matan. - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block. - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Maintaining class driver list to keep load/unload order symmetric and mirror of each other. - Deriving drv_flags based on the class drivers - Using PCI address comparision helper instead of pointer comparision - Fixed alignment for id_table - Continue to probe_err if device is already probed - Perform dma map on best effort basis for all supported drivers - Removed drv_flags check - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Address comments from Thomas and Gaetan - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - USe uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options --- drivers/bus/mlx5_pci/Makefile | 2 + drivers/bus/mlx5_pci/meson.build | 2 +- drivers/bus/mlx5_pci/mlx5_pci_bus.c | 508 ++++++++++++++++++++++++++++ 3 files changed, 511 insertions(+), 1 deletion(-) diff --git a/drivers/bus/mlx5_pci/Makefile b/drivers/bus/mlx5_pci/Makefile index dd24811a3..ee1af4dc6 100644 --- a/drivers/bus/mlx5_pci/Makefile +++ b/drivers/bus/mlx5_pci/Makefile @@ -16,7 +16,9 @@ CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(BUILDDIR)/drivers/bus/pci CFLAGS += -I$(BUILDDIR)/drivers/bus +CFLAGS += -D_DEFAULT_SOURCE LDLIBS += -lrte_eal +LDLIBS += -lrte_kvargs LDLIBS += -lrte_common_mlx5 LDLIBS += -lrte_pci -lrte_bus_pci diff --git a/drivers/bus/mlx5_pci/meson.build b/drivers/bus/mlx5_pci/meson.build index 64a17cbad..0532a9dfd 100644 --- a/drivers/bus/mlx5_pci/meson.build +++ b/drivers/bus/mlx5_pci/meson.build @@ -1,7 +1,7 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2020 Mellanox Technologies Ltd -deps += ['pci', 'bus_pci', 'common_mlx5'] +deps += ['pci', 'bus_pci', 'common_mlx5', 'kvargs'] install_headers('rte_bus_mlx5_pci.h') sources = files('mlx5_pci_bus.c') diff --git a/drivers/bus/mlx5_pci/mlx5_pci_bus.c b/drivers/bus/mlx5_pci/mlx5_pci_bus.c index 66db3c7b0..6f219730c 100644 --- a/drivers/bus/mlx5_pci/mlx5_pci_bus.c +++ b/drivers/bus/mlx5_pci/mlx5_pci_bus.c @@ -2,13 +2,521 @@ * Copyright 2020 Mellanox Technologies, Ltd */ +#include <stdlib.h> +#include <rte_malloc.h> #include "rte_bus_mlx5_pci.h" +#include <mlx5_common_utils.h> +struct mlx5_pci_device { + struct rte_pci_device *pci_dev; + TAILQ_ENTRY(mlx5_pci_device) next; + uint32_t classes_loaded; +}; + +/* Head of list of class drivers. */ static TAILQ_HEAD(mlx5_pci_bus_drv_head, rte_mlx5_pci_driver) drv_list = TAILQ_HEAD_INITIALIZER(drv_list); +/* Head of mlx5 pci devices. */ +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = + TAILQ_HEAD_INITIALIZER(devices_list); + +static const struct { + const char *name; + unsigned int dev_class; +} mlx5_classes[] = { + { .name = "vdpa", .dev_class = MLX5_CLASS_VDPA }, + { .name = "net", .dev_class = MLX5_CLASS_NET }, +}; + +static const unsigned int mlx5_class_combinations[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + /* New class combination should be added here. + * For example a new multi class device combination + * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. + */ +}; + +static int +class_name_to_value(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].dev_class; + } + return -EINVAL; +} + +static struct rte_mlx5_pci_driver * +class_driver_get(uint32_t class) +{ + struct rte_mlx5_pci_driver *driver; + + TAILQ_FOREACH(driver, &drv_list, next) { + if (driver->dev_class == class) + return driver; + } + return NULL; +} + +static int +bus_cmdline_options_handler(__rte_unused const char *key, + const char *class_names, void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name. Multiple + * class key,value is supplied as class=net:vdpa:foo:bar. + */ + found = strsep(&nstr, ":"); + if (!found) + continue; + /* Check if its a valid class. */ + class_val = class_name_to_value(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s." + " Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, + &ret); + rte_kvargs_free(kvlist); + return ret; +} + void rte_mlx5_pci_driver_register(struct rte_mlx5_pci_driver *driver) { TAILQ_INSERT_TAIL(&drv_list, driver, next); } + +static bool +mlx5_bus_match(const struct rte_mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* Check if device's ids match the class driver's ids. */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + return true; + } + return false; +} + +static int +is_valid_class_combination(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination. */ + for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { + if (mlx5_class_combinations[i] == user_classes) + return 0; + } + /* Not found any valid class combination. */ + return -EINVAL; +} + +static struct mlx5_pci_device * +pci_to_mlx5_device(const struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + + TAILQ_FOREACH(dev, &devices_list, next) { + if (dev->pci_dev == pci_dev) + return dev; + } + return NULL; +} + +static bool +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) +{ + return (device->classes_loaded & class) ? true : false; +} + +static void +dev_release(struct mlx5_pci_device *dev) +{ + TAILQ_REMOVE(&devices_list, dev, next); + rte_free(dev); +} + +static int +class_drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) +{ + struct rte_mlx5_pci_driver *driver; + int local_ret = -ENODEV; + unsigned int i = 0; + int ret = 0; + + enabled_classes &= dev->classes_loaded; + while (enabled_classes) { + driver = class_driver_get(RTE_BIT64(i)); + if (driver) { + local_ret = driver->pci_driver.remove(dev->pci_dev); + if (!local_ret) + dev->classes_loaded &= ~RTE_BIT64(i); + else if (ret == 0) + ret = local_ret; + } + enabled_classes &= ~RTE_BIT64(i); + i++; + } + if (local_ret) + ret = local_ret; + return ret; +} + +static int +class_drivers_probe(struct mlx5_pci_device *dev, + struct rte_pci_driver *pci_drv, + struct rte_pci_device *pci_dev, uint32_t user_classes) +{ + struct rte_mlx5_pci_driver *driver; + uint32_t enabled_classes = 0; + bool already_loaded; + int ret; + + TAILQ_FOREACH(driver, &drv_list, next) { + if ((driver->dev_class & user_classes) == 0) + continue; + if (!mlx5_bus_match(driver, pci_dev)) + continue; + already_loaded = dev->classes_loaded & driver->dev_class; + if (already_loaded && + !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { + DRV_LOG(ERR, "Device %s is already probed\n", + pci_dev->device.name); + ret = -EEXIST; + goto probe_err; + } + ret = driver->pci_driver.probe(pci_drv, pci_dev); + if (ret < 0) { + DRV_LOG(ERR, "Failed to load class driver = %s.\n", + driver->pci_driver.driver.name); + goto probe_err; + } + enabled_classes |= driver->dev_class; + } + dev->classes_loaded |= enabled_classes; + return 0; +probe_err: + /* Only unload drivers which are enabled which were enabled + * in this probe instance. + */ + class_drivers_remove(dev, enabled_classes); + return ret; +} + +/** + * DPDK callback to register to probe multiple PCI class devices. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_bus_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + uint32_t user_classes = 0; + bool new_device = false; + int ret; + + ret = parse_class_options(pci_dev->device.devargs); + if (ret < 0) + return ret; + user_classes = ret; + if (user_classes) { + /* Validate combination here. */ + ret = is_valid_class_combination(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); + return ret; + } + } else { + /* Default to net class. */ + user_classes = MLX5_CLASS_NET; + } + dev = pci_to_mlx5_device(pci_dev); + if (!dev) { + dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); + if (!dev) + return -ENOMEM; + dev->pci_dev = pci_dev; + TAILQ_INSERT_HEAD(&devices_list, dev, next); + new_device = true; + } + ret = class_drivers_probe(dev, pci_drv, pci_dev, user_classes); + if (ret) + goto class_err; + return 0; +class_err: + if (new_device) + dev_release(dev); + return ret; +} + +/** + * DPDK callback to remove one or more class devices for a PCI device. + * + * This function removes all class devices belong to a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_bus_pci_remove(struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + /* Matching device found, cleanup and unload class drivers. */ + ret = class_drivers_remove(dev, dev->classes_loaded); + if (!ret) + dev_release(dev); + return ret; +} + +static int +mlx5_bus_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class = NULL; + struct rte_mlx5_pci_driver *temp; + struct mlx5_pci_device *dev; + int ret = -EINVAL; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + TAILQ_FOREACH(class, &drv_list, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_map) { + ret = class->pci_driver.dma_map(pci_dev, addr, + iova, len); + if (ret) + goto map_err; + } + } + return ret; +map_err: + TAILQ_FOREACH(temp, &drv_list, next) { + if (temp == class) + break; + if (device_class_enabled(dev, temp->dev_class) && + temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) + temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_bus_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct rte_mlx5_pci_driver *class; + struct mlx5_pci_device *dev; + int local_ret = -EINVAL; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + ret = 0; + /* There is no unmap error recovery in current implementation. */ + TAILQ_FOREACH_REVERSE(class, &drv_list, mlx5_pci_bus_drv_head, next) { + if (device_class_enabled(dev, class->dev_class) && + class->pci_driver.dma_unmap) { + local_ret = class->pci_driver.dma_unmap(pci_dev, addr, + iova, len); + if (local_ret && (ret == 0)) + ret = local_ret; + } + } + if (local_ret) + ret = local_ret; + return ret; +} + +/* PCI ID table is build dynamically based on registered mlx5 + * class drivers. + */ +static struct rte_pci_id *mlx5_bus_pci_id_map; + +static int +pci_id_table_size_get(const struct rte_pci_id *id_table) +{ + int table_size = 0; + + for (; id_table->vendor_id != 0; id_table++) + table_size++; + return table_size; +} + +static bool +pci_id_exists(const struct rte_pci_id *id, int next_idx) +{ + int current_size = next_idx - 1; + int i; + + for (i = 0; i < current_size; i++) { + if (id->device_id == mlx5_bus_pci_id_map[i].device_id && + id->vendor_id == mlx5_bus_pci_id_map[i].vendor_id && + id->subsystem_vendor_id == + mlx5_bus_pci_id_map[i].subsystem_vendor_id && + id->subsystem_device_id == + mlx5_bus_pci_id_map[i].subsystem_device_id) + return true; + } + return false; +} + +static void +pci_id_insert(const struct rte_pci_id *id_table, int *next_idx) +{ + for (; id_table->vendor_id != 0; id_table++) { + if (!pci_id_exists(id_table, *next_idx)) { + /* New entry; add to the table. */ + mlx5_bus_pci_id_map[*next_idx] = *id_table; + (*next_idx)++; + } + } +} + +static int +pci_ids_table_build(void) +{ + struct rte_mlx5_pci_driver *first_driver; + struct rte_mlx5_pci_driver *driver; + const struct rte_pci_id *id_table; + int num_ids = 0; + int i = 0; + + TAILQ_FOREACH(driver, &drv_list, next) + num_ids += pci_id_table_size_get(driver->pci_driver.id_table); + if (!num_ids) + return -ENODEV; + /* Increase size by one for the termination entry of vendor_id = 0. */ + num_ids += 1; + mlx5_bus_pci_id_map = calloc(num_ids, sizeof(*mlx5_bus_pci_id_map)); + if (!mlx5_bus_pci_id_map) + return -ENOMEM; + first_driver = TAILQ_FIRST(&drv_list); + /* Copy the first class driver's ID table. */ + for (id_table = first_driver->pci_driver.id_table; + id_table->vendor_id != 0; id_table++, i++) + mlx5_bus_pci_id_map[i] = *id_table; + TAILQ_FOREACH(driver, &drv_list, next) { + /* We already added first driver; skip it. */ + if (driver == first_driver) + continue; + pci_id_insert(driver->pci_driver.id_table, &i); + } + mlx5_bus_pci_id_map[i].vendor_id = 0; + return 0; +} + +static bool mlx5_bus_registered; +static struct rte_pci_driver mlx5_bus_driver = { + .driver = { + .name = "mlx5_bus_pci", + }, + .probe = mlx5_bus_pci_probe, + .remove = mlx5_bus_pci_remove, + .dma_map = mlx5_bus_pci_dma_map, + .dma_unmap = mlx5_bus_pci_dma_unmap, +}; + +RTE_INIT(mlx5_bus_pci) +{ + struct rte_mlx5_pci_driver *class; + int ret; + + ret = pci_ids_table_build(); + if (ret) + return; + TAILQ_FOREACH(class, &drv_list, next) + mlx5_bus_driver.drv_flags |= class->pci_driver.drv_flags; + mlx5_bus_driver.id_table = mlx5_bus_pci_id_map; + rte_pci_register(&mlx5_bus_driver); + mlx5_bus_registered = true; +} + +RTE_FINI(mlx5_bus_pci_finish) +{ + if (mlx5_bus_registered) + rte_pci_unregister(&mlx5_bus_driver); + if (mlx5_bus_pci_id_map) + free(mlx5_bus_pci_id_map); +} +RTE_PMD_EXPORT_NAME(mlx5_bus_pci, __COUNTER__); +RTE_PMD_REGISTER_PCI_TABLE(mlx5_bus, mlx5_bus_pci_id_map); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (6 preceding siblings ...) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 7/9] bus/mlx5_pci: register a PCI driver Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 9/9] common/mlx5: remove class checks from individual driver Parav Pandit 2020-07-19 7:28 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Raslan Darawsheh 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Enable class driver to match with the mlx5 pci devices. Migrate mlx5 net PMD and vdpa PMD to start using mlx5 common class driver. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Avoid static table v1->v2: - Migreate API from rte_driver to rte_pci_driver --- drivers/net/mlx5/Makefile | 3 ++- drivers/net/mlx5/linux/mlx5_os.c | 1 - drivers/net/mlx5/linux/mlx5_os.h | 1 + drivers/net/mlx5/meson.build | 2 +- drivers/net/mlx5/mlx5.c | 24 ++++++++++++++---------- drivers/net/mlx5/mlx5.h | 1 - drivers/vdpa/mlx5/Makefile | 3 ++- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++---------- mk/rte.app.mk | 1 + 10 files changed, 35 insertions(+), 26 deletions(-) diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 9eaac6b7e..3c5f486c3 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -46,16 +46,17 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5 CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5/linux CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 +LDLIBS += -lrte_bus_mlx5_pci LDLIBS += -lm LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index f228babae..3450a961d 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1399,7 +1399,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); return -rte_errno; } - MLX5_ASSERT(pci_drv == &mlx5_driver); errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 31add3988..695722520 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,4 +15,5 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) + #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index c06b1538a..e35d90bbb 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -8,7 +8,7 @@ if not (is_linux or is_windows) subdir_done() endif -deps += ['hash', 'common_mlx5'] +deps += ['hash', 'common_mlx5', 'bus_mlx5_pci'] sources = files( 'mlx5.c', 'mlx5_ethdev.c', diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 1647256be..4d1c3716f 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -34,6 +34,7 @@ #include <rte_spinlock.h> #include <rte_string_fns.h> #include <rte_alarm.h> +#include <rte_bus_mlx5_pci.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h> @@ -2056,16 +2057,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -struct rte_pci_driver mlx5_driver = { - .driver = { - .name = MLX5_DRIVER_NAME +static struct rte_mlx5_pci_driver mlx5_driver = { + .dev_class = MLX5_CLASS_NET, + .pci_driver = { + .driver = { + .name = MLX5_DRIVER_NAME, + }, + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_dma_map, + .dma_unmap = mlx5_dma_unmap, + .drv_flags = PCI_DRV_FLAGS, }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, }; /* Initialize driver log type. */ @@ -2081,7 +2085,7 @@ RTE_INIT_PRIO(rte_mlx5_pmd_init, CLASS) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - rte_pci_register(&mlx5_driver); + rte_mlx5_pci_driver_register(&mlx5_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 2e61d0cba..c5cb732c5 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -129,7 +129,6 @@ struct mlx5_local_data { }; extern struct mlx5_shared_data *mlx5_shared_data; -extern struct rte_pci_driver mlx5_driver; /* Dev ops structs */ extern const struct eth_dev_ops mlx5_os_dev_ops; diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index 8a1c2eab5..9bb5dfdbf 100644 --- a/drivers/vdpa/mlx5/Makefile +++ b/drivers/vdpa/mlx5/Makefile @@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa CFLAGS += -I$(RTE_SDK)/lib/librte_sched CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_mlx5_pci -lrte_sched LDLIBS += -pthread # A few warnings cannot be avoided in external headers. diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build index 2963aad71..f4175c34e 100644 --- a/drivers/vdpa/mlx5/meson.build +++ b/drivers/vdpa/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_vdpa' -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched'] +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_mlx5_pci', 'eal', 'sched'] sources = files( 'mlx5_vdpa.c', 'mlx5_vdpa_mem.c', diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 205a9e4ad..5d247e0a1 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -11,7 +11,7 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> +#include <rte_bus_mlx5_pci.h> #include <rte_pci.h> #include <rte_string_fns.h> @@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) } /** - * DPDK callback to register a PCI device. + * DPDK callback to register a mlx5 PCI bus device. * * This function spawns vdpa device out of a given PCI device. * @@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_vdpa_driver = { - .driver = { - .name = "mlx5_vdpa", +static struct rte_mlx5_pci_driver mlx5_vdpa_driver = { + .dev_class = MLX5_CLASS_VDPA, + .pci_driver = { + .driver = { + .name = "mlx5_vdpa", + }, + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_pci_probe, + .remove = mlx5_vdpa_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, }; RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) @@ -847,7 +850,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) RTE_INIT_PRIO(rte_mlx5_vdpa_init, CLASS) { if (mlx5_glue) - rte_pci_register(&mlx5_vdpa_driver); + rte_mlx5_pci_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); diff --git a/mk/rte.app.mk b/mk/rte.app.mk index 0ce8cf541..d7c1d90e4 100644 --- a/mk/rte.app.mk +++ b/mk/rte.app.mk @@ -203,6 +203,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_MEMIF) += -lrte_pmd_memif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -lrte_pmd_mlx4 ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)),y) _LDLIBS-y += -lrte_common_mlx5 +_LDLIBS-y += -lrte_bus_mlx5_pci endif _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) += -lrte_pmd_mlx5 _LDLIBS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD) += -lrte_pmd_mlx5_vdpa -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v7 9/9] common/mlx5: remove class checks from individual driver 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (7 preceding siblings ...) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 8/9] bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver Parav Pandit @ 2020-07-17 13:49 ` Parav Pandit 2020-07-19 7:28 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Raslan Darawsheh 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-17 13:49 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Now that mlx5_pci bus does the check for enabled classes and performs probe(), remove() of associated classes, individual class driver does not need to check if other driver is enabled. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - Removed empty line v1->v2: - New patch --- drivers/common/mlx5/mlx5_common.c | 37 ------------------- drivers/common/mlx5/mlx5_common.h | 2 - .../common/mlx5/rte_common_mlx5_version.map | 2 - drivers/net/mlx5/linux/mlx5_os.c | 5 --- drivers/net/mlx5/linux/mlx5_os.h | 1 - drivers/vdpa/mlx5/mlx5_vdpa.c | 5 --- 6 files changed, 52 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 0ce5e4db1..771e046c1 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -22,43 +22,6 @@ const struct mlx5_glue *mlx5_glue; uint8_t haswell_broadwell_cpu; -static int -mlx5_class_check_handler(__rte_unused const char *key, const char *value, - void *opaque) -{ - enum mlx5_class *ret = opaque; - - if (strcmp(value, "vdpa") == 0) { - *ret = MLX5_CLASS_VDPA; - } else if (strcmp(value, "net") == 0) { - *ret = MLX5_CLASS_NET; - } else { - DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device" - " class argument setting?", value); - *ret = MLX5_CLASS_INVALID; - } - return 0; -} - -enum mlx5_class -mlx5_class_get(struct rte_devargs *devargs) -{ - struct rte_kvargs *kvlist; - const char *key = MLX5_CLASS_ARG_NAME; - enum mlx5_class ret = MLX5_CLASS_NET; - - if (devargs == NULL) - return ret; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return ret; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret); - rte_kvargs_free(kvlist); - return ret; -} - - /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 00e68edbb..5b9b7bd5a 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -244,8 +244,6 @@ struct mlx5_klm { LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page); -__rte_internal -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs); __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 5aad2199a..ce91576b3 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,8 +1,6 @@ INTERNAL { global: - mlx5_class_get; - mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 3450a961d..0b3e1b674 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1386,11 +1386,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_dev_config dev_config; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } if (rte_eal_process_type() == RTE_PROC_PRIMARY) mlx5_pmd_socket_init(); ret = mlx5_init_once(); diff --git a/drivers/net/mlx5/linux/mlx5_os.h b/drivers/net/mlx5/linux/mlx5_os.h index 695722520..31add3988 100644 --- a/drivers/net/mlx5/linux/mlx5_os.h +++ b/drivers/net/mlx5/linux/mlx5_os.h @@ -15,5 +15,4 @@ enum { #define PCI_DRV_FLAGS (RTE_PCI_DRV_INTR_LSC | \ RTE_PCI_DRV_INTR_RMV | \ RTE_PCI_DRV_PROBE_AGAIN) - #endif /* RTE_PMD_MLX5_OS_H_ */ diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 5d247e0a1..70692ea1d 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_hca_attr attr; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr); if (!ibv) { DRV_LOG(ERR, "No matching IB device for PCI slot " -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (8 preceding siblings ...) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 9/9] common/mlx5: remove class checks from individual driver Parav Pandit @ 2020-07-19 7:28 ` Raslan Darawsheh 9 siblings, 0 replies; 193+ messages in thread From: Raslan Darawsheh @ 2020-07-19 7:28 UTC (permalink / raw) To: Parav Pandit, dev, grive, ferruh.yigit, Thomas Monjalon Cc: Ori Kam, Matan Azrad, joyce.kong Hi, > -----Original Message----- > From: Parav Pandit <parav@mellanox.com> > Sent: Friday, July 17, 2020 4:49 PM > To: dev@dpdk.org; grive@u256.net; ferruh.yigit@intel.com; Thomas > Monjalon <thomas@monjalon.net> > Cc: Raslan Darawsheh <rasland@mellanox.com>; Ori Kam > <orika@mellanox.com>; Matan Azrad <matan@mellanox.com>; > joyce.kong@arm.com; Parav Pandit <parav@mellanox.com> > Subject: [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple > classes > > This series introduces mlx5 bus to support multiple class of devices > for a single PCI device. > > Motivation and example > ---------------------- > mlx5 PCI device supports multiple class of devices such as net, vdpa > and regex devices. > > Currently only one pmd (either net or vdpa) can bind to this device. > This design limits use of PCI device only for single device class. > > To support multiple classes simultaneously for a mlx5 PCI device, > a new mlx5 PCI bus is created. This bus allows binding multiple > class drivers (such as net, vdpa, regex(future)) to bind to the > mlx5 PCI bus driver. > > Change description > ------------------ > Patch-1 Introduces RTE_BIT() macro > Patch-2 Introduces new RTE constructor priority for common > initialization > Patch-3 Fixes compilation error > Patch-4 Define mlx5 class as bit fields > Patch-5 Uses new RTE common priority > Patch-6 Adds mlx5 PCI bus > Patch-7 Implements a mlx5 PCI bus driver > Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus > API instead of rte PCI bus API > Patch-9 Removed class check code as its already part of the bus now > > Design overview > --------------- > > ----------- ------------ ------------- > | mlx5 | | mlx5 | | mlx5 | > | net pmd | | vdpa pmd | | regex pmd | > ----------- ------------ ------------- > \ | / > \ | / > \ ------------- / > \______| mlx5 |_____ / > | pci bus | > ------------- > | > ----------- > | mlx5 | > | pci dev | > ----------- > > - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI > ID table of all related mlx5 PCI devices. > - mlx5 class driver such as net, vdpa, regex PMD defines its > specific PCI ID table and mlx5 bus driver probes matching > class drivers. > - mlx5 pci bus driver is cental place that validates supported > class combinations. > - In future as code evolves, more device setup/cleanup and > resource creation code moves to mlx5 PCI bus driver. > > Alternatives considered > ----------------------- > 1. Instead of creating mlx5 pci bus, a common driver is > implemented which exposes class registration API. > However, bus model fits better with existing DPDK design > similar to ifpga driver. > Class registration API need to create a new callbacks > and ID signature; instead it is better to utilize current > well defined methods. > > 2. Enhance pci core to allow multiple driver binding to > single rte PCI device. > This approach is not taken, because peer drivers using > one PCI device won't be aware of other's presence. This > requires cross-driver syncronization of who initializes > common resources (such as irq, eq and more). > This also requires refcounting common objects etc among > peer drivers. > Instead of layered approach delivers and allows putting > common resource sharing, setup code in common bus driver. > It also eliminates peer blind zone problem as bottom pci > bus layer provides necessary setup without any reference > counting. > > 3. In future mlx5 prefers to use RDMA MR cache of the mbuf > used between net and regex pmd so that same mbuf use across > multiple device can be possible. > > Examples: > -------- > A user who wish to use a specific class(es) provides list > of classes at command line such as, > ./testpmd -w <PCI BDF>,class=net:vdpa > ./testpmd -w <PCI BDF>,class=vdpa > > In future, > ./testpmd -w <PCI BDF>,class=net:regex > > Changelog: > v6->v7: > - Updated release notes > v5->v6: > - Fixed compilation failure in parallel build for shared lib > v4->v5: > - Squash the maintainers update path with other patch which adds the > bus > - Addressed comments from Morten Brørup > - Renamed newly added macro to RTE_BIT64 > - Added doxygen comment section for the macro > v3->v4: > - Fixed dma_map error unwinding flow to follow same order for unmap > v2->v3: > - Added RTE priority for common driver initialization > - Addressed comments from Thomas and Asaf > - Fixed compilation error in glue wrapper > - Moved pci_driver structure instance as first in driver > - Removed white spaces at the end of line in diagram > - Address commnts from Matan > - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files > - Renamed mlx5_valid_class_combo to mlx5_class_combinations > - Added cross check for class drivers to support only 3 flags for now > - Added full stop at the end of comment block > - Using full names in function names > - Added new line before function name in multiple functions > - Added example string to parse for multiple classes > - Dropped mlx5 prefix from static function > - Removed empty lines > - Fixed issue to remove multiple classes for a driver > - Using define for drv_flags at multiple places > - Deriving drv_flags based on the class drivers > - Fixed alignment for id_table > - Perform dma map on best effort basis for all supported drivers > - Dynamically build pci id table > - Using PCI to mlx5 device helper routines > v1->v2: > - Addressed most comments from Thomas and Gaetan. > - Symbols starting with prefix rte_bus_pci_mlx5 may be > confusing as it may appear as it belong to rte_bus_pci module. > Hence it is kept as rte_bus_mlx5_pci which matches with other > modules as mlx5_vdpa, mlx5_net. > - Dropped 2nd patch and replace with new 6th patch. > - Avoided new file, added macro to rte_bitops.h > - Inheriting ret_pci_driver instead of rte_driver > - Added design and description of the mlx5_pci bus > - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag > - Use anonymous structure for class search and code changes around it > - Define static for class comination array > - Use RTE_DIM to find array size > - Added OOM check for strdup() > - Renamed copy variable to nstr_orig > - Returning negagive error code > - Returning directly if match entry found > - Use compat condition check > - Avoided cutting error message string > - Use uint32_t datatype instead of enum mlx5_class > - Changed logic to parse device arguments only once during probe() > - Added check to fail driver probe if multiple classes register with > DMA ops > - Renamed function to parse_class_options > - Migreate API from rte_driver to rte_pci_driver > > Parav Pandit (9): > eal: introduce macros for getting value for bit > eal: introduce RTE common initialization level > common/mlx5: fix empty input style in glue wrappers > common/mlx5: change mlx5 class enum values as bits > common/mlx5: use common rte priority > bus/mlx5_pci: add mlx5 PCI bus > bus/mlx5_pci: register a PCI driver > bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver > common/mlx5: remove class checks from individual driver > > MAINTAINERS | 5 + > doc/guides/rel_notes/release_20_08.rst | 5 + > drivers/bus/Makefile | 4 + > drivers/bus/meson.build | 2 +- > drivers/bus/mlx5_pci/Makefile | 41 ++ > drivers/bus/mlx5_pci/meson.build | 19 + > drivers/bus/mlx5_pci/mlx5_pci_bus.c | 522 ++++++++++++++++++ > drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h | 84 +++ > .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map | 5 + > drivers/common/mlx5/linux/mlx5_glue.h | 4 +- > drivers/common/mlx5/mlx5_common.c | 39 +- > drivers/common/mlx5/mlx5_common.h | 7 +- > .../common/mlx5/rte_common_mlx5_version.map | 2 - > drivers/net/mlx5/Makefile | 3 +- > drivers/net/mlx5/linux/mlx5_os.c | 6 - > drivers/net/mlx5/meson.build | 2 +- > drivers/net/mlx5/mlx5.c | 26 +- > drivers/net/mlx5/mlx5.h | 1 - > drivers/vdpa/mlx5/Makefile | 3 +- > drivers/vdpa/mlx5/meson.build | 2 +- > drivers/vdpa/mlx5/mlx5_vdpa.c | 30 +- > lib/librte_eal/include/rte_bitops.h | 8 + > lib/librte_eal/include/rte_common.h | 1 + > mk/rte.app.mk | 1 + > 24 files changed, 737 insertions(+), 85 deletions(-) > create mode 100644 drivers/bus/mlx5_pci/Makefile > create mode 100644 drivers/bus/mlx5_pci/meson.build > create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c > create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h > create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map > > -- > 2.26.2 Series applied to next-net-mlx, Kindest regards, Raslan Darawsheh ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 00/10] Improve mlx5 PMD driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (6 preceding siblings ...) 2020-07-17 13:49 ` [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 01/10] eal: introduce macro for bit definition Parav Pandit ` (9 more replies) 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (3 subsequent siblings) 11 siblings, 10 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit This series introduces mlx5 bus to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, mlx5 common module is extended as generic PCI PMD. This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded on a single PCI Device. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Fixed indentation for directory listing Patch-3 Relax dependency order Patch-4 Fixes compilation error Patch-5 Fixes segmentation fault in regex probe error unwind path Patch-6 Avoid using class constructor priority Patch-7 Change class values as bits Patch-8 Introduce mlx5 layer to support multiple class drivers Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API instead of rte PCI bus API Patch-10 Removed class check code from class drivers Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ --------------- / \______| mlx5 common |____/ | pmd | --------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 common driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD define their specific PCI ID tables. mlx5 common driver probes each individual class driver(s) for maching and enabled classes. - mlx5 pci layer is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI common driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci common PMD, a common driver is implemented which exposes class registration API. However, common PMD model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common driver. It also eliminates peer blind zone problem as bottom pci layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:vdpa ./testpmd -w <PCI BDF>,class=vdpa Changelog: v7->v8: - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5 - Avoided new RTE PRIORITY - Removed mlx5 common to use RTE_PRIORITY_CLASS v6->v7: - Updated release notes v5->v6: - Fixed compilation failure in parallel build for shared lib v4->v5: - Squash the maintainers update path with other patch which adds the bus - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (7): eal: introduce macro for bit definition common/mlx5: fix void parameters in glue wrappers regex/mlx5: fix segmentation fault during error unwinding common/mlx5: avoid using class constructor priority common/mlx5: change class values as bits common/mlx5: register class drivers through common layer common/mlx5: remove class check from class drivers Thomas Monjalon (3): drivers: fix indent of directory list drivers: relax dependency order common/mlx5: introduce layer to support multiple class drivers drivers/Makefile | 10 +- drivers/common/Makefile | 4 - drivers/common/meson.build | 2 +- drivers/common/mlx5/Makefile | 1 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/meson.build | 9 +- drivers/common/mlx5/mlx5_common.c | 52 +- drivers/common/mlx5/mlx5_common.h | 12 +- drivers/common/mlx5/mlx5_common_pci.c | 539 ++++++++++++++++++ drivers/common/mlx5/mlx5_common_pci.h | 77 +++ .../common/mlx5/rte_common_mlx5_version.map | 4 +- drivers/meson.build | 51 +- drivers/net/mlx5/Makefile | 1 - drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/mlx5.c | 25 +- drivers/net/mlx5/mlx5.h | 1 - drivers/regex/mlx5/meson.build | 2 +- drivers/regex/mlx5/mlx5_regex.c | 29 +- drivers/vdpa/mlx5/Makefile | 3 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 29 +- lib/librte_eal/include/rte_bitops.h | 8 + 22 files changed, 748 insertions(+), 123 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_pci.c create mode 100644 drivers/common/mlx5/mlx5_common_pci.h -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 01/10] eal: introduce macro for bit definition 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list Parav Pandit ` (8 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit, Morten Brørup There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> Acked-by: Morten Brørup <mb@smartsharesystems.com> --- lib/librte_eal/include/rte_bitops.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..ca46a110f 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,14 @@ #include <rte_debug.h> #include <rte_compat.h> +/** + * Get the uint64_t value for a specified bit set. + * + * @param nr + * The bit number in range of 0 to 63. + */ +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 01/10] eal: introduce macro for bit definition Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-24 10:44 ` Bruce Richardson 2020-07-24 13:27 ` David Marchand 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order Parav Pandit ` (7 subsequent siblings) 9 siblings, 2 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas; +Cc: rasland, orika, matan, joyce.kong From: Thomas Monjalon <thomas@monjalon.net> Define each sub-directory on its own line ended with a comma, and use a simple indent. Signed-off-by: Thomas Monjalon <thomas@monjalon.net> --- drivers/meson.build | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/drivers/meson.build b/drivers/meson.build index e76ebddfa..e2aeba931 100644 --- a/drivers/meson.build +++ b/drivers/meson.build @@ -2,17 +2,19 @@ # Copyright(c) 2017-2019 Intel Corporation # Defines the order in which the drivers are buit. -dpdk_driver_classes = ['common', - 'bus', - 'mempool', # depends on common and bus. - 'net', # depends on common, bus, mempool - 'raw', # depends on common, bus and net. - 'crypto', # depends on common, bus and mempool (net in future). - 'compress', # depends on common, bus, mempool. - 'regex', # depends on common, bus, regexdev. - 'vdpa', # depends on common, bus and mempool. - 'event', # depends on common, bus, mempool and net. - 'baseband'] # depends on common and bus. +dpdk_driver_classes = [ + 'common', + 'bus', + 'mempool', # depends on common and bus. + 'net', # depends on common, bus, mempool + 'raw', # depends on common, bus and net. + 'crypto', # depends on common, bus and mempool (net in future). + 'compress', # depends on common, bus, mempool. + 'regex', # depends on common, bus, regexdev. + 'vdpa', # depends on common, bus and mempool. + 'event', # depends on common, bus, mempool and net. + 'baseband', # depends on common and bus. +] disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'), ).stdout().split() -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list Parav Pandit @ 2020-07-24 10:44 ` Bruce Richardson 2020-07-24 13:27 ` David Marchand 1 sibling, 0 replies; 193+ messages in thread From: Bruce Richardson @ 2020-07-24 10:44 UTC (permalink / raw) To: Parav Pandit Cc: dev, grive, ferruh.yigit, thomas, rasland, orika, matan, joyce.kong On Thu, Jul 23, 2020 at 11:09:02PM +0300, Parav Pandit wrote: > From: Thomas Monjalon <thomas@monjalon.net> > > Define each sub-directory on its own line ended with a comma, > and use a simple indent. > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > --- > drivers/meson.build | 24 +++++++++++++----------- > 1 file changed, 13 insertions(+), 11 deletions(-) > Thanks. Although we normally don't like doing cleanups for whitespace, this is a small-enough change and worth doing, since the existing indentation was just weird. Acked-by: Bruce Richardson <bruce.richardson@intel.com> ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list Parav Pandit 2020-07-24 10:44 ` Bruce Richardson @ 2020-07-24 13:27 ` David Marchand 1 sibling, 0 replies; 193+ messages in thread From: David Marchand @ 2020-07-24 13:27 UTC (permalink / raw) To: Parav Pandit Cc: dev, Gaetan Rivet, Yigit, Ferruh, Thomas Monjalon, Raslan, Ori Kam, Matan Azrad, Joyce Kong On Thu, Jul 23, 2020 at 10:10 PM Parav Pandit <parav@mellanox.com> wrote: > > From: Thomas Monjalon <thomas@monjalon.net> > > Define each sub-directory on its own line ended with a comma, > and use a simple indent. > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > --- > drivers/meson.build | 24 +++++++++++++----------- > 1 file changed, 13 insertions(+), 11 deletions(-) > > diff --git a/drivers/meson.build b/drivers/meson.build > index e76ebddfa..e2aeba931 100644 > --- a/drivers/meson.build > +++ b/drivers/meson.build > @@ -2,17 +2,19 @@ > # Copyright(c) 2017-2019 Intel Corporation > > # Defines the order in which the drivers are buit. > -dpdk_driver_classes = ['common', > - 'bus', > - 'mempool', # depends on common and bus. > - 'net', # depends on common, bus, mempool > - 'raw', # depends on common, bus and net. > - 'crypto', # depends on common, bus and mempool (net in future). > - 'compress', # depends on common, bus, mempool. > - 'regex', # depends on common, bus, regexdev. > - 'vdpa', # depends on common, bus and mempool. > - 'event', # depends on common, bus, mempool and net. > - 'baseband'] # depends on common and bus. > +dpdk_driver_classes = [ > + 'common', > + 'bus', > + 'mempool', # depends on common and bus. > + 'net', # depends on common, bus, mempool > + 'raw', # depends on common, bus and net. > + 'crypto', # depends on common, bus and mempool (net in future). > + 'compress', # depends on common, bus, mempool. > + 'regex', # depends on common, bus, regexdev. > + 'vdpa', # depends on common, bus and mempool. > + 'event', # depends on common, bus, mempool and net. > + 'baseband', # depends on common and bus. > +] > > disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'), > ).stdout().split() > -- > 2.25.4 > Reviewed-by: David Marchand <david.marchand@redhat.com> -- David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 01/10] eal: introduce macro for bit definition Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 02/10] drivers: fix indent of directory list Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-24 11:07 ` Bruce Richardson 2020-07-24 13:41 ` David Marchand 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 04/10] common/mlx5: fix void parameters in glue wrappers Parav Pandit ` (6 subsequent siblings) 9 siblings, 2 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas; +Cc: rasland, orika, matan, joyce.kong From: Thomas Monjalon <thomas@monjalon.net> Drivers dependencies are evaluated in the order defined per their parent directory (also called class). This strict ordering prevent from having 2 different drivers of the same class with different dependencies ordering. This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci, while drivers/bus/dpaa depends on drivers/common/dpaax. Having a strict ordering between directories bus and common is too much restrictive. That's why it is made possible to have a more fine-grain directory list, adding a driver sub-directory in the list. In this case, the isolated driver must be removed from its class list, and added directly in drivers/meson.build. Also, the per-class variables must be duplicated in the isolated driver, because the call "subdir(class)" is skipped in the isolated driver case. Signed-off-by: Thomas Monjalon <thomas@monjalon.net> --- drivers/meson.build | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/drivers/meson.build b/drivers/meson.build index e2aeba931..e6d0409aa 100644 --- a/drivers/meson.build +++ b/drivers/meson.build @@ -1,8 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017-2019 Intel Corporation -# Defines the order in which the drivers are buit. -dpdk_driver_classes = [ +# Defines the order of dependencies evaluation +subdirs = [ 'common', 'bus', 'mempool', # depends on common and bus. @@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation') default_cflags += '-Wno-format-truncation' endif -foreach class:dpdk_driver_classes +foreach subpath:subdirs drivers = [] std_deps = [] config_flag_fmt = '' # format string used to set the value in dpdk_conf @@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes # the library, the dependency and to find the # version file for linking - subdir(class) - class_drivers = [] + # subpath can be either "class" or "class/driver" + if subpath.contains('/') + driver_path = subpath.split('/') + class = driver_path[0] + drivers += driver_path[1] + else + class = subpath + subdir(class) + endif + + # save class name on first occurence + if not dpdk_driver_classes.contains(class) + dpdk_driver_classes += class + endif + # get already enabled drivers of the same class + enabled_drivers = get_variable(class + '_drivers', []) foreach drv:drivers drv_path = join_paths(class, drv) @@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes '_disable_reason', reason) endif else - class_drivers += name + enabled_drivers += name if fmt_name == '' fmt_name = name @@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes endif # build endforeach - set_variable(class + '_drivers', class_drivers) + set_variable(class + '_drivers', enabled_drivers) endforeach -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order Parav Pandit @ 2020-07-24 11:07 ` Bruce Richardson 2020-07-24 13:48 ` Parav Pandit 2020-07-24 13:41 ` David Marchand 1 sibling, 1 reply; 193+ messages in thread From: Bruce Richardson @ 2020-07-24 11:07 UTC (permalink / raw) To: Parav Pandit Cc: dev, grive, ferruh.yigit, thomas, rasland, orika, matan, joyce.kong On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote: > From: Thomas Monjalon <thomas@monjalon.net> > > Drivers dependencies are evaluated in the order defined per > their parent directory (also called class). > This strict ordering prevent from having 2 different drivers > of the same class with different dependencies ordering. > This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci, > while drivers/bus/dpaa depends on drivers/common/dpaax. > Having a strict ordering between directories bus and common > is too much restrictive. > > That's why it is made possible to have a more fine-grain directory list, > adding a driver sub-directory in the list. > In this case, the isolated driver must be removed from its class list, > and added directly in drivers/meson.build. > Also, the per-class variables must be duplicated in the isolated driver, > because the call "subdir(class)" is skipped in the isolated driver case. > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> The commit log above has some strange word-wrapping, and occasionally strange phrasing. I think it could be slightly reworded, perhaps as: Drivers dependencies are evaluated in the order defined per their parent directory (also called class). This strict ordering prevents from us from having pairs of drivers from two classes with different dependency ordering. For example, if the mlx5 common code depends on the pci bus driver, while the dpaax common code is itself a dependency of the dpaa bus driver. Having a strict ordering between directories bus and common is too restrictive, as processing either common drivers or bus drivers first leads us to missing dependencies in this scenario. This patch makes it possible to have a more fine-grain directory list, adding a specific driver sub-directory in the top-level drivers subdirectory list. In this case, the isolated driver must also be removed from its class list, and the per-class variables must be duplicated in the isolated driver, because the call "subdir(class)" is skipped in the isolated driver case. Apart from that, I think this is a good idea to give us some flexibility in managing driver ordering which should help other drivers too - perhaps QAT? Ideally, though, I'd like if we can limit the flexible ordering to *only* common code, in which case we could move the per-class variables for common to the top-level to prevent duplication, and maybe even get rid of common/meson.build completely. That, however, will depend on how much this feature gets used and by whom. Therefore: Review-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order 2020-07-24 11:07 ` Bruce Richardson @ 2020-07-24 13:48 ` Parav Pandit 2020-07-24 13:54 ` Thomas Monjalon 0 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-24 13:48 UTC (permalink / raw) To: Bruce Richardson Cc: dev, grive, ferruh.yigit, Thomas Monjalon, Raslan Darawsheh, Ori Kam, Matan Azrad, joyce.kong Hi Bruce, > From: Bruce Richardson <bruce.richardson@intel.com> > Sent: Friday, July 24, 2020 4:37 PM > > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote: > > From: Thomas Monjalon <thomas@monjalon.net> > > > > Drivers dependencies are evaluated in the order defined per their > > parent directory (also called class). > > This strict ordering prevent from having 2 different drivers of the > > same class with different dependencies ordering. > > This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci, > > while drivers/bus/dpaa depends on drivers/common/dpaax. > > Having a strict ordering between directories bus and common is too > > much restrictive. > > > > That's why it is made possible to have a more fine-grain directory > > list, adding a driver sub-directory in the list. > > In this case, the isolated driver must be removed from its class list, > > and added directly in drivers/meson.build. > > Also, the per-class variables must be duplicated in the isolated > > driver, because the call "subdir(class)" is skipped in the isolated driver case. > > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > > The commit log above has some strange word-wrapping, and occasionally > strange phrasing. I think it could be slightly reworded, perhaps as: > I updated the commit log as you suggested below along with RB, ack tag. Thank you. > Drivers dependencies are evaluated in the order defined per their parent > directory (also called class). This strict ordering prevents from us > from having pairs of drivers from two classes with different dependency > ordering. For example, if the mlx5 common code depends on the pci bus > driver, while the dpaax common code is itself a dependency of the dpaa > bus driver. Having a strict ordering between directories bus and common > is too restrictive, as processing either common drivers or bus drivers > first leads us to missing dependencies in this scenario. > > This patch makes it possible to have a more fine-grain directory list, > adding a specific driver sub-directory in the top-level drivers > subdirectory list. In this case, the isolated driver must also be removed > from its class list, and the per-class variables must be duplicated in > the isolated driver, because the call "subdir(class)" is skipped in the > isolated driver case. > > > Apart from that, I think this is a good idea to give us some flexibility in > managing driver ordering which should help other drivers too - perhaps QAT? > Ideally, though, I'd like if we can limit the flexible ordering to *only* common > code, in which case we could move the per-class variables for common to the > top-level to prevent duplication, and maybe even get rid of > common/meson.build completely. That, however, will depend on how much > this feature gets used and by whom. > > Therefore: > Review-by: Bruce Richardson <bruce.richardson@intel.com> > Acked-by: Bruce Richardson <bruce.richardson@intel.com> ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order 2020-07-24 13:48 ` Parav Pandit @ 2020-07-24 13:54 ` Thomas Monjalon 2020-07-24 14:50 ` Bruce Richardson 0 siblings, 1 reply; 193+ messages in thread From: Thomas Monjalon @ 2020-07-24 13:54 UTC (permalink / raw) To: Bruce Richardson, Parav Pandit Cc: dev, grive, ferruh.yigit, Raslan Darawsheh, Ori Kam, Matan Azrad, joyce.kong 24/07/2020 15:48, Parav Pandit: > Hi Bruce, > > > From: Bruce Richardson <bruce.richardson@intel.com> > > Sent: Friday, July 24, 2020 4:37 PM > > > > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote: > > > From: Thomas Monjalon <thomas@monjalon.net> > > > > > > Drivers dependencies are evaluated in the order defined per their > > > parent directory (also called class). > > > This strict ordering prevent from having 2 different drivers of the > > > same class with different dependencies ordering. > > > This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci, > > > while drivers/bus/dpaa depends on drivers/common/dpaax. > > > Having a strict ordering between directories bus and common is too > > > much restrictive. > > > > > > That's why it is made possible to have a more fine-grain directory > > > list, adding a driver sub-directory in the list. > > > In this case, the isolated driver must be removed from its class list, > > > and added directly in drivers/meson.build. > > > Also, the per-class variables must be duplicated in the isolated > > > driver, because the call "subdir(class)" is skipped in the isolated driver case. > > > > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > > > > The commit log above has some strange word-wrapping, and occasionally > > strange phrasing. I think it could be slightly reworded, perhaps as: > > > I updated the commit log as you suggested below along with RB, ack tag. > Thank you. > > > Drivers dependencies are evaluated in the order defined per their parent > > directory (also called class). This strict ordering prevents from us Is "from us" too much? > > from having pairs of drivers from two classes with different dependency > > ordering. For example, if the mlx5 common code depends on the pci bus > > driver, while the dpaax common code is itself a dependency of the dpaa > > bus driver. Having a strict ordering between directories bus and common > > is too restrictive, as processing either common drivers or bus drivers > > first leads us to missing dependencies in this scenario. > > > > This patch makes it possible to have a more fine-grain directory list, > > adding a specific driver sub-directory in the top-level drivers > > subdirectory list. In this case, the isolated driver must also be removed > > from its class list, and the per-class variables must be duplicated in > > the isolated driver, because the call "subdir(class)" is skipped in the > > isolated driver case. > > > > > > Apart from that, I think this is a good idea to give us some flexibility in > > managing driver ordering which should help other drivers too - perhaps QAT? > > Ideally, though, I'd like if we can limit the flexible ordering to *only* common > > code, in which case we could move the per-class variables for common to the > > top-level to prevent duplication, and maybe even get rid of > > common/meson.build completely. That, however, will depend on how much > > this feature gets used and by whom. For now it is used only to have common/mlx5 isolated of the rest of common drivers, as an exception. I think it is good to keep common/meson.build until there are more exceptions than the rest. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order 2020-07-24 13:54 ` Thomas Monjalon @ 2020-07-24 14:50 ` Bruce Richardson 2020-07-24 15:17 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: Bruce Richardson @ 2020-07-24 14:50 UTC (permalink / raw) To: Thomas Monjalon Cc: Parav Pandit, dev, grive, ferruh.yigit, Raslan Darawsheh, Ori Kam, Matan Azrad, joyce.kong On Fri, Jul 24, 2020 at 03:54:33PM +0200, Thomas Monjalon wrote: > 24/07/2020 15:48, Parav Pandit: > > Hi Bruce, > > > > > From: Bruce Richardson <bruce.richardson@intel.com> > > > Sent: Friday, July 24, 2020 4:37 PM > > > > > > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote: > > > > From: Thomas Monjalon <thomas@monjalon.net> > > > > > > > > Drivers dependencies are evaluated in the order defined per their > > > > parent directory (also called class). > > > > This strict ordering prevent from having 2 different drivers of the > > > > same class with different dependencies ordering. > > > > This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci, > > > > while drivers/bus/dpaa depends on drivers/common/dpaax. > > > > Having a strict ordering between directories bus and common is too > > > > much restrictive. > > > > > > > > That's why it is made possible to have a more fine-grain directory > > > > list, adding a driver sub-directory in the list. > > > > In this case, the isolated driver must be removed from its class list, > > > > and added directly in drivers/meson.build. > > > > Also, the per-class variables must be duplicated in the isolated > > > > driver, because the call "subdir(class)" is skipped in the isolated driver case. > > > > > > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > > > > > > The commit log above has some strange word-wrapping, and occasionally > > > strange phrasing. I think it could be slightly reworded, perhaps as: > > > > > I updated the commit log as you suggested below along with RB, ack tag. > > Thank you. > > > > > Drivers dependencies are evaluated in the order defined per their parent > > > directory (also called class). This strict ordering prevents from us > > Is "from us" too much? The word "from" is incorrect, anyway, it should just be "prevents us". Evidently I didn't proofread my own text well enough. :-( > > > > from having pairs of drivers from two classes with different dependency > > > ordering. For example, if the mlx5 common code depends on the pci bus > > > driver, while the dpaax common code is itself a dependency of the dpaa > > > bus driver. Having a strict ordering between directories bus and common > > > is too restrictive, as processing either common drivers or bus drivers > > > first leads us to missing dependencies in this scenario. > > > > > > This patch makes it possible to have a more fine-grain directory list, > > > adding a specific driver sub-directory in the top-level drivers > > > subdirectory list. In this case, the isolated driver must also be removed > > > from its class list, and the per-class variables must be duplicated in > > > the isolated driver, because the call "subdir(class)" is skipped in the > > > isolated driver case. > > > > > > > > > Apart from that, I think this is a good idea to give us some flexibility in > > > managing driver ordering which should help other drivers too - perhaps QAT? > > > Ideally, though, I'd like if we can limit the flexible ordering to *only* common > > > code, in which case we could move the per-class variables for common to the > > > top-level to prevent duplication, and maybe even get rid of > > > common/meson.build completely. That, however, will depend on how much > > > this feature gets used and by whom. > > For now it is used only to have common/mlx5 isolated of the rest > of common drivers, as an exception. > I think it is good to keep common/meson.build > until there are more exceptions than the rest. > +1 agreed. Just suggesting possible future changes depending on how things go, and also seeding the idea that any out-of-order build processing should really be limited to the common folder. /Bruce ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order 2020-07-24 14:50 ` Bruce Richardson @ 2020-07-24 15:17 ` Parav Pandit 2020-07-24 15:29 ` Bruce Richardson 0 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-24 15:17 UTC (permalink / raw) To: Bruce Richardson, Thomas Monjalon Cc: dev, grive, ferruh.yigit, Raslan Darawsheh, Ori Kam, Matan Azrad, joyce.kong > From: Bruce Richardson <bruce.richardson@intel.com> > Sent: Friday, July 24, 2020 8:20 PM > > On Fri, Jul 24, 2020 at 03:54:33PM +0200, Thomas Monjalon wrote: > > 24/07/2020 15:48, Parav Pandit: > > > Hi Bruce, > > > > > > > From: Bruce Richardson <bruce.richardson@intel.com> > > > > Sent: Friday, July 24, 2020 4:37 PM > > > > > > > > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote: > > > > > From: Thomas Monjalon <thomas@monjalon.net> > > > > > > > > > > Drivers dependencies are evaluated in the order defined per > > > > > their parent directory (also called class). > > > > > This strict ordering prevent from having 2 different drivers of > > > > > the same class with different dependencies ordering. > > > > > This problem occurs if drivers/common/mlx5 depends on > > > > > drivers/bus/pci, while drivers/bus/dpaa depends on > drivers/common/dpaax. > > > > > Having a strict ordering between directories bus and common is > > > > > too much restrictive. > > > > > > > > > > That's why it is made possible to have a more fine-grain > > > > > directory list, adding a driver sub-directory in the list. > > > > > In this case, the isolated driver must be removed from its class > > > > > list, and added directly in drivers/meson.build. > > > > > Also, the per-class variables must be duplicated in the isolated > > > > > driver, because the call "subdir(class)" is skipped in the isolated driver > case. > > > > > > > > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > > > > > > > > The commit log above has some strange word-wrapping, and > > > > occasionally strange phrasing. I think it could be slightly reworded, > perhaps as: > > > > > > > I updated the commit log as you suggested below along with RB, ack tag. > > > Thank you. > > > > > > > Drivers dependencies are evaluated in the order defined per their > parent > > > > directory (also called class). This strict ordering prevents > > > > from us > > > > Is "from us" too much? > > The word "from" is incorrect, anyway, it should just be "prevents us". > Evidently I didn't proofread my own text well enough. :-( In v10 I rephase it as "strict ordering prevents from having pairs of drivers...". ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order 2020-07-24 15:17 ` Parav Pandit @ 2020-07-24 15:29 ` Bruce Richardson 0 siblings, 0 replies; 193+ messages in thread From: Bruce Richardson @ 2020-07-24 15:29 UTC (permalink / raw) To: Parav Pandit Cc: Thomas Monjalon, dev, grive, ferruh.yigit, Raslan Darawsheh, Ori Kam, Matan Azrad, joyce.kong On Fri, Jul 24, 2020 at 03:17:38PM +0000, Parav Pandit wrote: > > > > From: Bruce Richardson <bruce.richardson@intel.com> > > Sent: Friday, July 24, 2020 8:20 PM > > > > On Fri, Jul 24, 2020 at 03:54:33PM +0200, Thomas Monjalon wrote: > > > 24/07/2020 15:48, Parav Pandit: > > > > Hi Bruce, > > > > > > > > > From: Bruce Richardson <bruce.richardson@intel.com> > > > > > Sent: Friday, July 24, 2020 4:37 PM > > > > > > > > > > On Thu, Jul 23, 2020 at 11:09:03PM +0300, Parav Pandit wrote: > > > > > > From: Thomas Monjalon <thomas@monjalon.net> > > > > > > > > > > > > Drivers dependencies are evaluated in the order defined per > > > > > > their parent directory (also called class). > > > > > > This strict ordering prevent from having 2 different drivers of > > > > > > the same class with different dependencies ordering. > > > > > > This problem occurs if drivers/common/mlx5 depends on > > > > > > drivers/bus/pci, while drivers/bus/dpaa depends on > > drivers/common/dpaax. > > > > > > Having a strict ordering between directories bus and common is > > > > > > too much restrictive. > > > > > > > > > > > > That's why it is made possible to have a more fine-grain > > > > > > directory list, adding a driver sub-directory in the list. > > > > > > In this case, the isolated driver must be removed from its class > > > > > > list, and added directly in drivers/meson.build. > > > > > > Also, the per-class variables must be duplicated in the isolated > > > > > > driver, because the call "subdir(class)" is skipped in the isolated driver > > case. > > > > > > > > > > > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > > > > > > > > > > The commit log above has some strange word-wrapping, and > > > > > occasionally strange phrasing. I think it could be slightly reworded, > > perhaps as: > > > > > > > > > I updated the commit log as you suggested below along with RB, ack tag. > > > > Thank you. > > > > > > > > > Drivers dependencies are evaluated in the order defined per their > > parent > > > > > directory (also called class). This strict ordering prevents > > > > > from us > > > > > > Is "from us" too much? > > > > The word "from" is incorrect, anyway, it should just be "prevents us". > > Evidently I didn't proofread my own text well enough. :-( > > In v10 I rephase it as "strict ordering prevents from having pairs of drivers...". Great, thanks. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order Parav Pandit 2020-07-24 11:07 ` Bruce Richardson @ 2020-07-24 13:41 ` David Marchand 1 sibling, 0 replies; 193+ messages in thread From: David Marchand @ 2020-07-24 13:41 UTC (permalink / raw) To: Parav Pandit Cc: dev, Gaetan Rivet, Yigit, Ferruh, Thomas Monjalon, Raslan, Ori Kam, Matan Azrad, Joyce Kong On Thu, Jul 23, 2020 at 10:10 PM Parav Pandit <parav@mellanox.com> wrote: > > From: Thomas Monjalon <thomas@monjalon.net> > > Drivers dependencies are evaluated in the order defined per > their parent directory (also called class). > This strict ordering prevent from having 2 different drivers > of the same class with different dependencies ordering. > This problem occurs if drivers/common/mlx5 depends on drivers/bus/pci, > while drivers/bus/dpaa depends on drivers/common/dpaax. > Having a strict ordering between directories bus and common > is too much restrictive. > > That's why it is made possible to have a more fine-grain directory list, > adding a driver sub-directory in the list. > In this case, the isolated driver must be removed from its class list, > and added directly in drivers/meson.build. > Also, the per-class variables must be duplicated in the isolated driver, > because the call "subdir(class)" is skipped in the isolated driver case. > > Signed-off-by: Thomas Monjalon <thomas@monjalon.net> > --- > drivers/meson.build | 28 +++++++++++++++++++++------- > 1 file changed, 21 insertions(+), 7 deletions(-) > > diff --git a/drivers/meson.build b/drivers/meson.build > index e2aeba931..e6d0409aa 100644 > --- a/drivers/meson.build > +++ b/drivers/meson.build > @@ -1,8 +1,8 @@ > # SPDX-License-Identifier: BSD-3-Clause > # Copyright(c) 2017-2019 Intel Corporation > > -# Defines the order in which the drivers are buit. > -dpdk_driver_classes = [ > +# Defines the order of dependencies evaluation > +subdirs = [ > 'common', > 'bus', > 'mempool', # depends on common and bus. > @@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation') > default_cflags += '-Wno-format-truncation' > endif > > -foreach class:dpdk_driver_classes > +foreach subpath:subdirs > drivers = [] > std_deps = [] > config_flag_fmt = '' # format string used to set the value in dpdk_conf > @@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes > # the library, the dependency and to find the > # version file for linking > > - subdir(class) > - class_drivers = [] > + # subpath can be either "class" or "class/driver" > + if subpath.contains('/') > + driver_path = subpath.split('/') > + class = driver_path[0] > + drivers += driver_path[1] > + else > + class = subpath > + subdir(class) > + endif > + > + # save class name on first occurence > + if not dpdk_driver_classes.contains(class) > + dpdk_driver_classes += class > + endif > + # get already enabled drivers of the same class > + enabled_drivers = get_variable(class + '_drivers', []) > > foreach drv:drivers > drv_path = join_paths(class, drv) > @@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes > '_disable_reason', reason) > endif > else > - class_drivers += name > + enabled_drivers += name > > if fmt_name == '' > fmt_name = name > @@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes > endif # build > endforeach > > - set_variable(class + '_drivers', class_drivers) > + set_variable(class + '_drivers', enabled_drivers) > endforeach > -- > 2.25.4 > Reviewed-by: David Marchand <david.marchand@redhat.com> -- David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 04/10] common/mlx5: fix void parameters in glue wrappers 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit ` (2 preceding siblings ...) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 03/10] drivers: relax dependency order Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 05/10] regex/mlx5: fix segmentation fault during error unwinding Parav Pandit ` (5 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Following two errors are reported when compiled with gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5). drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] Fix them by adding void data type in empty argument list. Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch") Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue") Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/linux/mlx5_glue.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h index c4f9b006a..734ace2a3 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.h +++ b/drivers/common/mlx5/linux/mlx5_glue.h @@ -185,11 +185,11 @@ struct mlx5_glue { void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl); void *(*dr_create_flow_action_dest_port)(void *domain, uint32_t port); - void *(*dr_create_flow_action_drop)(); + void *(*dr_create_flow_action_drop)(void); void *(*dr_create_flow_action_push_vlan) (struct mlx5dv_dr_domain *domain, rte_be32_t vlan_tag); - void *(*dr_create_flow_action_pop_vlan)(); + void *(*dr_create_flow_action_pop_vlan)(void); void *(*dr_create_flow_tbl)(void *domain, uint32_t level); int (*dr_destroy_flow_tbl)(void *tbl); void *(*dr_create_domain)(struct ibv_context *ctx, -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 05/10] regex/mlx5: fix segmentation fault during error unwinding 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit ` (3 preceding siblings ...) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 04/10] common/mlx5: fix void parameters in glue wrappers Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority Parav Pandit ` (4 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit When fail to initialize the device, avoid segmentation fault while accessing unintialized priv. Fixes: cfc672a90b74 ("regex/mlx5: support probing") Signed-off-by: Parav Pandit <parav@mellanox.com> --- drivers/regex/mlx5/mlx5_regex.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c index 1ca5bfe9b..36ae9f809 100644 --- a/drivers/regex/mlx5/mlx5_regex.c +++ b/drivers/regex/mlx5/mlx5_regex.c @@ -137,17 +137,17 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (ret) { DRV_LOG(ERR, "Unable to read HCA capabilities."); rte_errno = ENOTSUP; - goto error; + goto dev_error; } else if (!attr.regex || attr.regexp_num_of_engines == 0) { DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe " "old FW/OFED version?"); rte_errno = ENOTSUP; - goto error; + goto dev_error; } if (mlx5_regex_engines_status(ctx, 2)) { DRV_LOG(ERR, "RegEx engine error."); rte_errno = ENOMEM; - goto error; + goto dev_error; } priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv), RTE_CACHE_LINE_SIZE); @@ -200,6 +200,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mlx5_glue->devx_free_uar(priv->uar); if (priv->regexdev) rte_regexdev_unregister(priv->regexdev); +dev_error: if (ctx) mlx5_glue->close_device(ctx); if (priv) -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit ` (4 preceding siblings ...) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 05/10] regex/mlx5: fix segmentation fault during error unwinding Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-24 13:45 ` David Marchand 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 07/10] common/mlx5: change class values as bits Parav Pandit ` (3 subsequent siblings) 9 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit mlx5_common is shared library between mlx5 net, VDPA and regex PMD. It is better to use common initialization helper instead of using RTE_INIT_CLASS priority. Signed-off-by: Parav Pandit <parav@mellanox.com> --- drivers/common/mlx5/mlx5_common.c | 13 +++++++++++-- drivers/common/mlx5/mlx5_common.h | 3 +++ drivers/common/mlx5/rte_common_mlx5_version.map | 1 + drivers/net/mlx5/mlx5.c | 1 + drivers/regex/mlx5/mlx5_regex.c | 1 + drivers/vdpa/mlx5/mlx5_vdpa.c | 1 + 6 files changed, 18 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 1a67a1b16..2b336bb2d 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE); } +static bool mlx5_common_initialized; + /** - * Initialization routine for run-time dependency on glue library. + * One time innitialization routine for run-time dependency on glue library + * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module, + * must invoke in its constructor. */ -RTE_INIT_PRIO(mlx5_glue_init, CLASS) +void +mlx5_common_init(void) { + if (mlx5_common_initialized) + return; + mlx5_glue_constructor(); + mlx5_common_initialized = true; } /** diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index a811eb6c9..ebe4e9ced 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id, uint64_t offset); extern uint8_t haswell_broadwell_cpu; +__rte_internal +void mlx5_common_init(void); + #endif /* RTE_PMD_MLX5_COMMON_H_ */ diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 132a0695f..65f25252a 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -3,6 +3,7 @@ INTERNAL { mlx5_class_get; + mlx5_common_init; mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 647ada339..037703d2e 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE) */ RTE_INIT(rte_mlx5_pmd_init) { + mlx5_common_init(); /* Build the static tables for Verbs conversion. */ mlx5_set_ptype_table(); mlx5_set_cksum_table(); diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c index 36ae9f809..4e0367052 100644 --- a/drivers/regex/mlx5/mlx5_regex.c +++ b/drivers/regex/mlx5/mlx5_regex.c @@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = { RTE_INIT(rte_mlx5_regex_init) { + mlx5_common_init(); if (mlx5_glue) rte_pci_register(&mlx5_regex_driver); } diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 67e77b11a..85dbcf956 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) */ RTE_INIT(rte_mlx5_vdpa_init) { + mlx5_common_init(); if (mlx5_glue) rte_pci_register(&mlx5_vdpa_driver); } -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority Parav Pandit @ 2020-07-24 13:45 ` David Marchand 2020-07-24 13:47 ` Parav Pandit 0 siblings, 1 reply; 193+ messages in thread From: David Marchand @ 2020-07-24 13:45 UTC (permalink / raw) To: Parav Pandit Cc: dev, Gaetan Rivet, Yigit, Ferruh, Thomas Monjalon, Raslan, Ori Kam, Matan Azrad, Joyce Kong On Thu, Jul 23, 2020 at 10:10 PM Parav Pandit <parav@mellanox.com> wrote: > > mlx5_common is shared library between mlx5 net, VDPA and regex PMD. > It is better to use common initialization helper instead of using > RTE_INIT_CLASS priority. RTE_INIT_CLASS does not exist, I suppose you meant RTE_PRIORITY_CLASS. Suggested-by: David Marchand <david.marchand@redhat.com> > Signed-off-by: Parav Pandit <parav@mellanox.com> > --- > drivers/common/mlx5/mlx5_common.c | 13 +++++++++++-- > drivers/common/mlx5/mlx5_common.h | 3 +++ > drivers/common/mlx5/rte_common_mlx5_version.map | 1 + > drivers/net/mlx5/mlx5.c | 1 + > drivers/regex/mlx5/mlx5_regex.c | 1 + > drivers/vdpa/mlx5/mlx5_vdpa.c | 1 + > 6 files changed, 18 insertions(+), 2 deletions(-) > > diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c > index 1a67a1b16..2b336bb2d 100644 > --- a/drivers/common/mlx5/mlx5_common.c > +++ b/drivers/common/mlx5/mlx5_common.c > @@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) > rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE); > } > > +static bool mlx5_common_initialized; > + > /** > - * Initialization routine for run-time dependency on glue library. > + * One time innitialization routine for run-time dependency on glue library > + * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module, > + * must invoke in its constructor. > */ > -RTE_INIT_PRIO(mlx5_glue_init, CLASS) > +void > +mlx5_common_init(void) > { > + if (mlx5_common_initialized) > + return; > + > mlx5_glue_constructor(); > + mlx5_common_initialized = true; > } > > /** > diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h > index a811eb6c9..ebe4e9ced 100644 > --- a/drivers/common/mlx5/mlx5_common.h > +++ b/drivers/common/mlx5/mlx5_common.h > @@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id, > uint64_t offset); > extern uint8_t haswell_broadwell_cpu; > > +__rte_internal > +void mlx5_common_init(void); > + > #endif /* RTE_PMD_MLX5_COMMON_H_ */ > diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map > index 132a0695f..65f25252a 100644 > --- a/drivers/common/mlx5/rte_common_mlx5_version.map > +++ b/drivers/common/mlx5/rte_common_mlx5_version.map > @@ -3,6 +3,7 @@ INTERNAL { > > mlx5_class_get; > > + mlx5_common_init; > mlx5_common_verbs_reg_mr; > mlx5_common_verbs_dereg_mr; > > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c > index 647ada339..037703d2e 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE) > */ > RTE_INIT(rte_mlx5_pmd_init) > { > + mlx5_common_init(); > /* Build the static tables for Verbs conversion. */ > mlx5_set_ptype_table(); > mlx5_set_cksum_table(); > diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c > index 36ae9f809..4e0367052 100644 > --- a/drivers/regex/mlx5/mlx5_regex.c > +++ b/drivers/regex/mlx5/mlx5_regex.c > @@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = { > > RTE_INIT(rte_mlx5_regex_init) > { > + mlx5_common_init(); > if (mlx5_glue) > rte_pci_register(&mlx5_regex_driver); > } > diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c > index 67e77b11a..85dbcf956 100644 > --- a/drivers/vdpa/mlx5/mlx5_vdpa.c > +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c > @@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) > */ > RTE_INIT(rte_mlx5_vdpa_init) > { > + mlx5_common_init(); > if (mlx5_glue) > rte_pci_register(&mlx5_vdpa_driver); > } > -- > 2.25.4 > Reviewed-by: David Marchand <david.marchand@redhat.com> -- David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority 2020-07-24 13:45 ` David Marchand @ 2020-07-24 13:47 ` Parav Pandit 0 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 13:47 UTC (permalink / raw) To: David Marchand Cc: dev, Gaetan Rivet, Yigit, Ferruh, Thomas Monjalon, Raslan Darawsheh, Ori Kam, Matan Azrad, Joyce Kong > From: David Marchand <david.marchand@redhat.com> > Sent: Friday, July 24, 2020 7:15 PM > To: Parav Pandit <parav@mellanox.com> > Cc: dev <dev@dpdk.org>; Gaetan Rivet <grive@u256.net>; Yigit, Ferruh > <ferruh.yigit@intel.com>; Thomas Monjalon <thomas@monjalon.net>; > Raslan Darawsheh <rasland@mellanox.com>; Ori Kam > <orika@mellanox.com>; Matan Azrad <matan@mellanox.com>; Joyce Kong > <joyce.kong@arm.com> > Subject: Re: [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class > constructor priority > > On Thu, Jul 23, 2020 at 10:10 PM Parav Pandit <parav@mellanox.com> wrote: > > > > mlx5_common is shared library between mlx5 net, VDPA and regex PMD. > > It is better to use common initialization helper instead of using > > RTE_INIT_CLASS priority. > > RTE_INIT_CLASS does not exist, I suppose you meant RTE_PRIORITY_CLASS. > You are right. Correcting it and adding below tags. > Suggested-by: David Marchand <david.marchand@redhat.com> > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > --- > > drivers/common/mlx5/mlx5_common.c | 13 +++++++++++-- > > drivers/common/mlx5/mlx5_common.h | 3 +++ > > drivers/common/mlx5/rte_common_mlx5_version.map | 1 + > > drivers/net/mlx5/mlx5.c | 1 + > > drivers/regex/mlx5/mlx5_regex.c | 1 + > > drivers/vdpa/mlx5/mlx5_vdpa.c | 1 + > > 6 files changed, 18 insertions(+), 2 deletions(-) > > > > diff --git a/drivers/common/mlx5/mlx5_common.c > > b/drivers/common/mlx5/mlx5_common.c > > index 1a67a1b16..2b336bb2d 100644 > > --- a/drivers/common/mlx5/mlx5_common.c > > +++ b/drivers/common/mlx5/mlx5_common.c > > @@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) > > rte_log_set_level(mlx5_common_logtype, > > RTE_LOG_NOTICE); } > > > > +static bool mlx5_common_initialized; > > + > > /** > > - * Initialization routine for run-time dependency on glue library. > > + * One time innitialization routine for run-time dependency on glue > > + library > > + * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common > > + module, > > + * must invoke in its constructor. > > */ > > -RTE_INIT_PRIO(mlx5_glue_init, CLASS) > > +void > > +mlx5_common_init(void) > > { > > + if (mlx5_common_initialized) > > + return; > > + > > mlx5_glue_constructor(); > > + mlx5_common_initialized = true; > > } > > > > /** > > diff --git a/drivers/common/mlx5/mlx5_common.h > > b/drivers/common/mlx5/mlx5_common.h > > index a811eb6c9..ebe4e9ced 100644 > > --- a/drivers/common/mlx5/mlx5_common.h > > +++ b/drivers/common/mlx5/mlx5_common.h > > @@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct > mlx5_dbr_page_list *head, uint32_t umem_id, > > uint64_t offset); extern uint8_t > > haswell_broadwell_cpu; > > > > +__rte_internal > > +void mlx5_common_init(void); > > + > > #endif /* RTE_PMD_MLX5_COMMON_H_ */ > > diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map > > b/drivers/common/mlx5/rte_common_mlx5_version.map > > index 132a0695f..65f25252a 100644 > > --- a/drivers/common/mlx5/rte_common_mlx5_version.map > > +++ b/drivers/common/mlx5/rte_common_mlx5_version.map > > @@ -3,6 +3,7 @@ INTERNAL { > > > > mlx5_class_get; > > > > + mlx5_common_init; > > mlx5_common_verbs_reg_mr; > > mlx5_common_verbs_dereg_mr; > > > > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > > 647ada339..037703d2e 100644 > > --- a/drivers/net/mlx5/mlx5.c > > +++ b/drivers/net/mlx5/mlx5.c > > @@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, > pmd.net.mlx5, NOTICE) > > */ > > RTE_INIT(rte_mlx5_pmd_init) > > { > > + mlx5_common_init(); > > /* Build the static tables for Verbs conversion. */ > > mlx5_set_ptype_table(); > > mlx5_set_cksum_table(); > > diff --git a/drivers/regex/mlx5/mlx5_regex.c > > b/drivers/regex/mlx5/mlx5_regex.c index 36ae9f809..4e0367052 100644 > > --- a/drivers/regex/mlx5/mlx5_regex.c > > +++ b/drivers/regex/mlx5/mlx5_regex.c > > @@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = { > > > > RTE_INIT(rte_mlx5_regex_init) > > { > > + mlx5_common_init(); > > if (mlx5_glue) > > rte_pci_register(&mlx5_regex_driver); > > } > > diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c > > b/drivers/vdpa/mlx5/mlx5_vdpa.c index 67e77b11a..85dbcf956 100644 > > --- a/drivers/vdpa/mlx5/mlx5_vdpa.c > > +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c > > @@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, > pmd.vdpa.mlx5, NOTICE) > > */ > > RTE_INIT(rte_mlx5_vdpa_init) > > { > > + mlx5_common_init(); > > if (mlx5_glue) > > rte_pci_register(&mlx5_vdpa_driver); > > } > > -- > > 2.25.4 > > > > Reviewed-by: David Marchand <david.marchand@redhat.com> > > > -- > David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 07/10] common/mlx5: change class values as bits 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit ` (5 preceding siblings ...) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 06/10] common/mlx5: avoid using class constructor priority Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 08/10] common/mlx5: introduce layer to support multiple class drivers Parav Pandit ` (2 subsequent siblings) 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit mlx5 PCI Device supports multiple classes of devices such as net, vdpa, and/or regex. To support these multiple classes, change mlx5_class to a bitmap values so that if users asks to enable multiple of them, all supported classes can be parsed. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/mlx5_common.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index ebe4e9ced..864e82ff0 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -13,6 +13,7 @@ #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h> +#include <rte_bitops.h> #include "mlx5_prm.h" #include "mlx5_devx_cmds.h" @@ -208,10 +209,10 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); #define MLX5_CLASS_ARG_NAME "class" enum mlx5_class { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, - MLX5_CLASS_REGEX, MLX5_CLASS_INVALID, + MLX5_CLASS_NET = RTE_BIT64(0), + MLX5_CLASS_VDPA = RTE_BIT64(1), + MLX5_CLASS_REGEX = RTE_BIT64(2), }; #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 08/10] common/mlx5: introduce layer to support multiple class drivers 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit ` (6 preceding siblings ...) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 07/10] common/mlx5: change class values as bits Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-24 14:40 ` David Marchand 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 09/10] common/mlx5: register class drivers through common layer Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 10/10] common/mlx5: remove class check from class drivers Parav Pandit 9 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit From: Thomas Monjalon <thomas@monjalon.net> Add generic mlx5 PCI PMD layer as part of existing common_mlx5 module. This enables multiple classes (net, regex, vdpa) PMDs to be supported at same time. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/Makefile | 10 +- drivers/common/Makefile | 4 - drivers/common/meson.build | 2 +- drivers/common/mlx5/Makefile | 1 + drivers/common/mlx5/meson.build | 9 +- drivers/common/mlx5/mlx5_common.c | 2 + drivers/common/mlx5/mlx5_common_pci.c | 541 ++++++++++++++++++ drivers/common/mlx5/mlx5_common_pci.h | 77 +++ .../common/mlx5/rte_common_mlx5_version.map | 2 + drivers/meson.build | 1 + 10 files changed, 638 insertions(+), 11 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_pci.c create mode 100644 drivers/common/mlx5/mlx5_common_pci.h diff --git a/drivers/Makefile b/drivers/Makefile index 1551272ef..7f06162dc 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -8,8 +8,12 @@ DIRS-y += bus DEPDIRS-bus := common DIRS-y += mempool DEPDIRS-mempool := common bus +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y) +DIRS-y += common/mlx5 +DEPDIRS-common/mlx5 := bus +endif DIRS-y += net -DEPDIRS-net := common bus mempool +DEPDIRS-net := common bus mempool common/mlx5 DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband DEPDIRS-baseband := common bus mempool DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto @@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress DEPDIRS-compress := bus mempool DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex -DEPDIRS-regex := common bus +DEPDIRS-regex := common bus common/mlx5 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa -DEPDIRS-vdpa := common bus mempool +DEPDIRS-vdpa := common bus mempool common/mlx5 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event DEPDIRS-event := common bus mempool net crypto DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw diff --git a/drivers/common/Makefile b/drivers/common/Makefile index cbc71077c..cfb6b4dc8 100644 --- a/drivers/common/Makefile +++ b/drivers/common/Makefile @@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y))) DIRS-y += iavf endif -ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y) -DIRS-y += mlx5 -endif - include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/common/meson.build b/drivers/common/meson.build index 5db7e29b1..9ed4c04ba 100644 --- a/drivers/common/meson.build +++ b/drivers/common/meson.build @@ -6,6 +6,6 @@ if is_windows endif std_deps = ['eal'] -drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat'] +drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat'] config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON' driver_name_fmt = 'rte_common_@0@' diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile index 6b89a6c85..eaabb8c10 100644 --- a/drivers/common/mlx5/Makefile +++ b/drivers/common/mlx5/Makefile @@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c SRCS-y += mlx5_common_mp.c SRCS-y += mlx5_common_mr.c SRCS-y += mlx5_malloc.c +SRCS-y += mlx5_common_pci.c ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) INSTALL-y-lib += $(LIB_GLUE) endif diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build index 70e2c1c32..8e5608703 100644 --- a/drivers/common/mlx5/meson.build +++ b/drivers/common/mlx5/meson.build @@ -1,19 +1,22 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2019 Mellanox Technologies, Ltd -if not (is_linux or is_windows) +if not is_linux build = false - reason = 'only supported on Linux and Windows' + reason = 'only supported on Linux' subdir_done() endif -deps += ['hash', 'pci', 'net', 'eal', 'kvargs'] +config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON' +driver_name_fmt = 'rte_common_@0@' +deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c', 'mlx5_common_mp.c', 'mlx5_common_mr.c', 'mlx5_malloc.c', + 'mlx5_common_pci.c', ) cflags_options = [ diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 2b336bb2d..fd818ef24 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -14,6 +14,7 @@ #include "mlx5_common_os.h" #include "mlx5_common_utils.h" #include "mlx5_malloc.h" +#include "mlx5_common_pci.h" int mlx5_common_logtype; @@ -100,6 +101,7 @@ mlx5_common_init(void) return; mlx5_glue_constructor(); + mlx5_common_pci_init(); mlx5_common_initialized = true; } diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c new file mode 100644 index 000000000..b42f7469f --- /dev/null +++ b/drivers/common/mlx5/mlx5_common_pci.c @@ -0,0 +1,541 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies Ltd + */ + +#include <stdlib.h> +#include <rte_malloc.h> +#include "mlx5_common_utils.h" +#include "mlx5_common_pci.h" + +struct mlx5_pci_device { + struct rte_pci_device *pci_dev; + TAILQ_ENTRY(mlx5_pci_device) next; + uint32_t classes_loaded; +}; + +/* Head of list of drivers. */ +static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list = + TAILQ_HEAD_INITIALIZER(drv_list); + +/* Head of mlx5 pci devices. */ +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = + TAILQ_HEAD_INITIALIZER(devices_list); + +static const struct { + const char *name; + unsigned int driver_class; +} mlx5_classes[] = { + { .name = "vdpa", .driver_class = MLX5_CLASS_VDPA }, + { .name = "net", .driver_class = MLX5_CLASS_NET }, + { .name = "regex", .driver_class = MLX5_CLASS_REGEX }, +}; + +static const unsigned int mlx5_class_combinations[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + MLX5_CLASS_REGEX, + /* New class combination should be added here. + * For example a new multi class device combination + * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. + */ +}; + +static int +class_name_to_value(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].driver_class; + } + return -EINVAL; +} + +static struct mlx5_pci_driver * +driver_get(uint32_t class) +{ + struct mlx5_pci_driver *driver; + + TAILQ_FOREACH(driver, &drv_list, next) { + if (driver->driver_class == class) + return driver; + } + return NULL; +} + +static int +bus_cmdline_options_handler(__rte_unused const char *key, + const char *class_names, void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name. Multiple + * class key,value is supplied as class=net:vdpa:foo:bar. + */ + found = strsep(&nstr, ":"); + if (!found) + continue; + /* Check if its a valid class. */ + class_val = class_name_to_value(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s." + " Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, + &ret); + rte_kvargs_free(kvlist); + return ret; +} + +static bool +mlx5_bus_match(const struct mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* Check if device's ids match the class driver's ids. */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + return true; + } + return false; +} + +static int +is_valid_class_combination(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination. */ + for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { + if (mlx5_class_combinations[i] == user_classes) + return 0; + } + /* Not found any valid class combination. */ + return -EINVAL; +} + +static struct mlx5_pci_device * +pci_to_mlx5_device(const struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + + TAILQ_FOREACH(dev, &devices_list, next) { + if (dev->pci_dev == pci_dev) + return dev; + } + return NULL; +} + +static bool +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) +{ + return (device->classes_loaded & class) ? true : false; +} + +static void +dev_release(struct mlx5_pci_device *dev) +{ + TAILQ_REMOVE(&devices_list, dev, next); + rte_free(dev); +} + +static int +drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) +{ + struct mlx5_pci_driver *driver; + int local_ret = -ENODEV; + unsigned int i = 0; + int ret = 0; + + enabled_classes &= dev->classes_loaded; + while (enabled_classes) { + driver = driver_get(RTE_BIT64(i)); + if (driver) { + local_ret = driver->pci_driver.remove(dev->pci_dev); + if (!local_ret) + dev->classes_loaded &= ~RTE_BIT64(i); + else if (ret == 0) + ret = local_ret; + } + enabled_classes &= ~RTE_BIT64(i); + i++; + } + if (local_ret) + ret = local_ret; + return ret; +} + +static int +drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv, + struct rte_pci_device *pci_dev, uint32_t user_classes) +{ + struct mlx5_pci_driver *driver; + uint32_t enabled_classes = 0; + bool already_loaded; + int ret; + + TAILQ_FOREACH(driver, &drv_list, next) { + if ((driver->driver_class & user_classes) == 0) + continue; + if (!mlx5_bus_match(driver, pci_dev)) + continue; + already_loaded = dev->classes_loaded & driver->driver_class; + if (already_loaded && + !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { + DRV_LOG(ERR, "Device %s is already probed\n", + pci_dev->device.name); + ret = -EEXIST; + goto probe_err; + } + ret = driver->pci_driver.probe(pci_drv, pci_dev); + if (ret < 0) { + DRV_LOG(ERR, "Failed to load driver = %s.\n", + driver->pci_driver.driver.name); + goto probe_err; + } + enabled_classes |= driver->driver_class; + } + dev->classes_loaded |= enabled_classes; + return 0; +probe_err: + /* Only unload drivers which are enabled which were enabled + * in this probe instance. + */ + drivers_remove(dev, enabled_classes); + return ret; +} + +/** + * DPDK callback to register to probe multiple drivers for a PCI device. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + uint32_t user_classes = 0; + bool new_device = false; + int ret; + + ret = parse_class_options(pci_dev->device.devargs); + if (ret < 0) + return ret; + user_classes = ret; + if (user_classes) { + /* Validate combination here. */ + ret = is_valid_class_combination(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); + return ret; + } + } else { + /* Default to net class. */ + user_classes = MLX5_CLASS_NET; + } + dev = pci_to_mlx5_device(pci_dev); + if (!dev) { + dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); + if (!dev) + return -ENOMEM; + dev->pci_dev = pci_dev; + TAILQ_INSERT_HEAD(&devices_list, dev, next); + new_device = true; + } + ret = drivers_probe(dev, pci_drv, pci_dev, user_classes); + if (ret) + goto class_err; + return 0; +class_err: + if (new_device) + dev_release(dev); + return ret; +} + +/** + * DPDK callback to remove one or more drivers for a PCI device. + * + * This function removes all drivers probed for a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_common_pci_remove(struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + /* Matching device found, cleanup and unload drivers. */ + ret = drivers_remove(dev, dev->classes_loaded); + if (!ret) + dev_release(dev); + return ret; +} + +static int +mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct mlx5_pci_driver *driver = NULL; + struct mlx5_pci_driver *temp; + struct mlx5_pci_device *dev; + int ret = -EINVAL; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + TAILQ_FOREACH(driver, &drv_list, next) { + if (device_class_enabled(dev, driver->driver_class) && + driver->pci_driver.dma_map) { + ret = driver->pci_driver.dma_map(pci_dev, addr, + iova, len); + if (ret) + goto map_err; + } + } + return ret; +map_err: + TAILQ_FOREACH(temp, &drv_list, next) { + if (temp == driver) + break; + if (device_class_enabled(dev, temp->driver_class) && + temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) + temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct mlx5_pci_driver *driver; + struct mlx5_pci_device *dev; + int local_ret = -EINVAL; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + ret = 0; + /* There is no unmap error recovery in current implementation. */ + TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) { + if (device_class_enabled(dev, driver->driver_class) && + driver->pci_driver.dma_unmap) { + local_ret = driver->pci_driver.dma_unmap(pci_dev, addr, + iova, len); + if (local_ret && (ret == 0)) + ret = local_ret; + } + } + if (local_ret) + ret = local_ret; + return ret; +} + +/* PCI ID table is build dynamically based on registered mlx5 drivers. */ +static struct rte_pci_id *mlx5_pci_id_table; + +static struct rte_pci_driver mlx5_pci_driver = { + .driver = { + .name = "mlx5_pci", + }, + .probe = mlx5_common_pci_probe, + .remove = mlx5_common_pci_remove, + .dma_map = mlx5_common_pci_dma_map, + .dma_unmap = mlx5_common_pci_dma_unmap, +}; + +static int +pci_id_table_size_get(const struct rte_pci_id *id_table) +{ + int table_size = 0; + + for (; id_table->vendor_id != 0; id_table++) + table_size++; + return table_size; +} + +static bool +pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table, + int next_idx) +{ + int current_size = next_idx - 1; + int i; + + for (i = 0; i < current_size; i++) { + if (id->device_id == table[i].device_id && + id->vendor_id == table[i].vendor_id && + id->subsystem_vendor_id == table[i].subsystem_vendor_id && + id->subsystem_device_id == table[i].subsystem_device_id) + return true; + } + return false; +} + +static void +pci_id_insert(struct rte_pci_id *new_table, int *next_idx, + const struct rte_pci_id *id_table) +{ + /* Traverse the id_table, check if entry exists in new_table; + * Add non duplicate entries to new table. + */ + for (; id_table->vendor_id != 0; id_table++) { + if (!pci_id_exists(id_table, new_table, *next_idx)) { + /* New entry; add to the table. */ + new_table[*next_idx] = *id_table; + (*next_idx)++; + } + } +} + +static int +pci_ids_table_update(const struct rte_pci_id *driver_id_table) +{ + const struct rte_pci_id *id_iter; + struct rte_pci_id *updated_table; + struct rte_pci_id *old_table; + int num_ids = 0; + int i = 0; + + old_table = mlx5_pci_id_table; + if (old_table) + num_ids = pci_id_table_size_get(old_table); + num_ids += pci_id_table_size_get(driver_id_table); + /* Increase size by one for the termination entry of vendor_id = 0. */ + num_ids += 1; + updated_table = calloc(num_ids, sizeof(*updated_table)); + if (!updated_table) + return -ENOMEM; + if (TAILQ_EMPTY(&drv_list)) { + /* Copy the first driver's ID table. */ + for (id_iter = driver_id_table; id_iter->vendor_id != 0; + id_iter++, i++) + updated_table[i] = *id_iter; + } else { + /* First copy existing table entries. */ + for (id_iter = old_table; id_iter->vendor_id != 0; + id_iter++, i++) + updated_table[i] = *id_iter; + /* New id to be added at the end of current ID table. */ + pci_id_insert(updated_table, &i, driver_id_table); + } + /* Terminate table with empty entry. */ + updated_table[i].vendor_id = 0; + mlx5_pci_driver.id_table = updated_table; + mlx5_pci_id_table = updated_table; + if (old_table) + free(old_table); + return 0; +} + +void +mlx5_pci_driver_register(struct mlx5_pci_driver *driver) +{ + int ret; + + ret = pci_ids_table_update(driver->pci_driver.id_table); + if (ret) + return; + mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags; + TAILQ_INSERT_TAIL(&drv_list, driver, next); +} + +void mlx5_common_pci_init(void) +{ + const struct rte_pci_id empty_table[] = { + { + .vendor_id = 0 + }, + }; + + /* All mlx5 PMDs constructor runs at same priority. So any of the PMD + * including this one can register the PCI table first. If any other + * PMD(s) have registered the PCI ID table, No need to register an empty + * default one. + */ + if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table)) + return; + rte_pci_register(&mlx5_pci_driver); +} + +RTE_FINI(mlx5_common_pci_finish) +{ + if (mlx5_pci_id_table != NULL) { + /* Constructor doesn't register with PCI bus if it failed + * to build the table. + */ + rte_pci_unregister(&mlx5_pci_driver); + free(mlx5_pci_id_table); + } +} +RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__); diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h new file mode 100644 index 000000000..41b73e17a --- /dev/null +++ b/drivers/common/mlx5/mlx5_common_pci.h @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef _MLX5_COMMON_PCI_H_ +#define _MLX5_COMMON_PCI_H_ + +/** + * @file + * + * RTE Mellanox PCI Driver Interface + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) + * devices. This layer enables creating such multiple class of devices on a + * single PCI device by allowing to bind multiple class specific device + * driver to attach to mlx5_pci driver. + * + * ----------- ------------ ------------- + * | mlx5 | | mlx5 | | mlx5 | + * | net pmd | | vdpa pmd | | regex pmd | + * ----------- ------------ ------------- + * \ | / + * \ | / + * \ -------------- / + * \______| mlx5 |_____ / + * | pci common | + * -------------- + * | + * ----------- + * | mlx5 | + * | pci dev | + * ----------- + * + * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 pci bus driver is cental place that validates supported + * class combinations. + */ + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +#include <rte_pci.h> +#include <rte_bus_pci.h> + +#include <mlx5_common.h> + +void mlx5_common_pci_init(void); + +/** + * A structure describing a mlx5 pci driver. + */ +struct mlx5_pci_driver { + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ + uint32_t driver_class; /**< Class of this driver, enum mlx5_class */ + TAILQ_ENTRY(mlx5_pci_driver) next; +}; + +/** + * Register a mlx5_pci device driver. + * + * @param driver + * A pointer to a mlx5_pci_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +mlx5_pci_driver_register(struct mlx5_pci_driver *driver); + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _MLX5_COMMON_PCI_H_ */ diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 65f25252a..73cf72548 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -91,5 +91,7 @@ INTERNAL { mlx5_malloc; mlx5_realloc; mlx5_free; + + mlx5_pci_driver_register; }; diff --git a/drivers/meson.build b/drivers/meson.build index e6d0409aa..95df7ef1d 100644 --- a/drivers/meson.build +++ b/drivers/meson.build @@ -5,6 +5,7 @@ subdirs = [ 'common', 'bus', + 'common/mlx5', # depends on bus. 'mempool', # depends on common and bus. 'net', # depends on common, bus, mempool 'raw', # depends on common, bus and net. -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v8 08/10] common/mlx5: introduce layer to support multiple class drivers 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 08/10] common/mlx5: introduce layer to support multiple class drivers Parav Pandit @ 2020-07-24 14:40 ` David Marchand 0 siblings, 0 replies; 193+ messages in thread From: David Marchand @ 2020-07-24 14:40 UTC (permalink / raw) To: Parav Pandit Cc: dev, Gaetan Rivet, Yigit, Ferruh, Thomas Monjalon, Raslan, Ori Kam, Matan Azrad, Joyce Kong This is not a thorough review, just caught one issue for makefile and some nits. On Thu, Jul 23, 2020 at 10:11 PM Parav Pandit <parav@mellanox.com> wrote: > > From: Thomas Monjalon <thomas@monjalon.net> > > Add generic mlx5 PCI PMD layer as part of existing common_mlx5 > module. This enables multiple classes (net, regex, vdpa) PMDs > to be supported at same time. > > Signed-off-by: Parav Pandit <parav@mellanox.com> > Acked-by: Matan Azrad <matan@mellanox.com> > --- > drivers/Makefile | 10 +- > drivers/common/Makefile | 4 - > drivers/common/meson.build | 2 +- > drivers/common/mlx5/Makefile | 1 + > drivers/common/mlx5/meson.build | 9 +- > drivers/common/mlx5/mlx5_common.c | 2 + > drivers/common/mlx5/mlx5_common_pci.c | 541 ++++++++++++++++++ > drivers/common/mlx5/mlx5_common_pci.h | 77 +++ > .../common/mlx5/rte_common_mlx5_version.map | 2 + > drivers/meson.build | 1 + > 10 files changed, 638 insertions(+), 11 deletions(-) > create mode 100644 drivers/common/mlx5/mlx5_common_pci.c > create mode 100644 drivers/common/mlx5/mlx5_common_pci.h > > diff --git a/drivers/Makefile b/drivers/Makefile > index 1551272ef..7f06162dc 100644 > --- a/drivers/Makefile > +++ b/drivers/Makefile > @@ -8,8 +8,12 @@ DIRS-y += bus > DEPDIRS-bus := common > DIRS-y += mempool > DEPDIRS-mempool := common bus > +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y) > +DIRS-y += common/mlx5 > +DEPDIRS-common/mlx5 := bus > +endif > DIRS-y += net > -DEPDIRS-net := common bus mempool > +DEPDIRS-net := common bus mempool common/mlx5 Will it still work for people not building with mlx5? I would have seen something like: DEPDIRS-net/mlx5 := xx xx xx common/mlx5 Or maybe DEPDIRS-net/mlx5 += common/mlx5 > DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband > DEPDIRS-baseband := common bus mempool > DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto > @@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool > DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress > DEPDIRS-compress := bus mempool > DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex > -DEPDIRS-regex := common bus > +DEPDIRS-regex := common bus common/mlx5 DEPDIRS-regex/mlx5 := xx xx xx common/mlx5 > DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa > -DEPDIRS-vdpa := common bus mempool > +DEPDIRS-vdpa := common bus mempool common/mlx5 Idem. > DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event > DEPDIRS-event := common bus mempool net crypto > DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw > diff --git a/drivers/common/Makefile b/drivers/common/Makefile > index cbc71077c..cfb6b4dc8 100644 > --- a/drivers/common/Makefile > +++ b/drivers/common/Makefile > @@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y))) > DIRS-y += iavf > endif > > -ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y) > -DIRS-y += mlx5 > -endif > - > include $(RTE_SDK)/mk/rte.subdir.mk > diff --git a/drivers/common/meson.build b/drivers/common/meson.build > index 5db7e29b1..9ed4c04ba 100644 > --- a/drivers/common/meson.build > +++ b/drivers/common/meson.build > @@ -6,6 +6,6 @@ if is_windows > endif > > std_deps = ['eal'] > -drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat'] > +drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat'] > config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON' > driver_name_fmt = 'rte_common_@0@' > diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile > index 6b89a6c85..eaabb8c10 100644 > --- a/drivers/common/mlx5/Makefile > +++ b/drivers/common/mlx5/Makefile > @@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c > SRCS-y += mlx5_common_mp.c > SRCS-y += mlx5_common_mr.c > SRCS-y += mlx5_malloc.c > +SRCS-y += mlx5_common_pci.c Alphabetical order? > ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) > INSTALL-y-lib += $(LIB_GLUE) > endif > diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build > index 70e2c1c32..8e5608703 100644 > --- a/drivers/common/mlx5/meson.build > +++ b/drivers/common/mlx5/meson.build > @@ -1,19 +1,22 @@ > # SPDX-License-Identifier: BSD-3-Clause > # Copyright 2019 Mellanox Technologies, Ltd > > -if not (is_linux or is_windows) > +if not is_linux > build = false > - reason = 'only supported on Linux and Windows' > + reason = 'only supported on Linux' > subdir_done() > endif > > -deps += ['hash', 'pci', 'net', 'eal', 'kvargs'] > +config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON' > +driver_name_fmt = 'rte_common_@0@' > +deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] > sources += files( > 'mlx5_devx_cmds.c', > 'mlx5_common.c', > 'mlx5_common_mp.c', > 'mlx5_common_mr.c', > 'mlx5_malloc.c', > + 'mlx5_common_pci.c', Idem order. > ) > > cflags_options = [ > diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c > index 2b336bb2d..fd818ef24 100644 > --- a/drivers/common/mlx5/mlx5_common.c > +++ b/drivers/common/mlx5/mlx5_common.c > @@ -14,6 +14,7 @@ > #include "mlx5_common_os.h" > #include "mlx5_common_utils.h" > #include "mlx5_malloc.h" > +#include "mlx5_common_pci.h" Idem order. > > int mlx5_common_logtype; > > @@ -100,6 +101,7 @@ mlx5_common_init(void) > return; > > mlx5_glue_constructor(); > + mlx5_common_pci_init(); > mlx5_common_initialized = true; > } > > diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c > new file mode 100644 > index 000000000..b42f7469f > --- /dev/null > +++ b/drivers/common/mlx5/mlx5_common_pci.c > @@ -0,0 +1,541 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2020 Mellanox Technologies Ltd > + */ > + > +#include <stdlib.h> Nit: headers usually come in separate blocks, "system headers", "dpdk headers", "drivers headers". So missing an empty line here. > +#include <rte_malloc.h> > +#include "mlx5_common_utils.h" > +#include "mlx5_common_pci.h" > + > +struct mlx5_pci_device { > + struct rte_pci_device *pci_dev; > + TAILQ_ENTRY(mlx5_pci_device) next; > + uint32_t classes_loaded; > +}; > + > +/* Head of list of drivers. */ > +static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list = > + TAILQ_HEAD_INITIALIZER(drv_list); > + > +/* Head of mlx5 pci devices. */ > +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = > + TAILQ_HEAD_INITIALIZER(devices_list); > + > +static const struct { > + const char *name; > + unsigned int driver_class; > +} mlx5_classes[] = { > + { .name = "vdpa", .driver_class = MLX5_CLASS_VDPA }, > + { .name = "net", .driver_class = MLX5_CLASS_NET }, > + { .name = "regex", .driver_class = MLX5_CLASS_REGEX }, > +}; > + > +static const unsigned int mlx5_class_combinations[] = { > + MLX5_CLASS_NET, > + MLX5_CLASS_VDPA, > + MLX5_CLASS_REGEX, > + /* New class combination should be added here. > + * For example a new multi class device combination > + * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. > + */ > +}; > + > +static int > +class_name_to_value(const char *class_name) > +{ > + unsigned int i; > + > + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { > + if (strcmp(class_name, mlx5_classes[i].name) == 0) > + return mlx5_classes[i].driver_class; > + } > + return -EINVAL; > +} > + > +static struct mlx5_pci_driver * > +driver_get(uint32_t class) > +{ > + struct mlx5_pci_driver *driver; > + > + TAILQ_FOREACH(driver, &drv_list, next) { > + if (driver->driver_class == class) > + return driver; > + } > + return NULL; > +} > + > +static int > +bus_cmdline_options_handler(__rte_unused const char *key, > + const char *class_names, void *opaque) > +{ > + int *ret = opaque; > + char *nstr_org; > + int class_val; > + char *found; > + char *nstr; > + > + *ret = 0; > + nstr = strdup(class_names); > + if (!nstr) { > + *ret = -ENOMEM; > + return *ret; > + } > + nstr_org = nstr; > + while (nstr) { > + /* Extract each individual class name. Multiple > + * class key,value is supplied as class=net:vdpa:foo:bar. > + */ > + found = strsep(&nstr, ":"); > + if (!found) > + continue; > + /* Check if its a valid class. */ > + class_val = class_name_to_value(found); > + if (class_val < 0) { > + *ret = -EINVAL; > + goto err; > + } > + *ret |= class_val; > + } > +err: > + free(nstr_org); > + if (*ret < 0) > + DRV_LOG(ERR, "Invalid mlx5 class options %s." > + " Maybe typo in device class argument setting?", > + class_names); > + return *ret; > +} > + > +static int > +parse_class_options(const struct rte_devargs *devargs) > +{ > + const char *key = MLX5_CLASS_ARG_NAME; > + struct rte_kvargs *kvlist; > + int ret = 0; > + > + if (devargs == NULL) > + return 0; > + kvlist = rte_kvargs_parse(devargs->args, NULL); > + if (kvlist == NULL) > + return 0; > + if (rte_kvargs_count(kvlist, key)) > + rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, > + &ret); > + rte_kvargs_free(kvlist); > + return ret; > +} > + > +static bool > +mlx5_bus_match(const struct mlx5_pci_driver *drv, > + const struct rte_pci_device *pci_dev) > +{ > + const struct rte_pci_id *id_table; > + > + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; > + id_table++) { > + /* Check if device's ids match the class driver's ids. */ > + if (id_table->vendor_id != pci_dev->id.vendor_id && > + id_table->vendor_id != PCI_ANY_ID) > + continue; > + if (id_table->device_id != pci_dev->id.device_id && > + id_table->device_id != PCI_ANY_ID) > + continue; > + if (id_table->subsystem_vendor_id != > + pci_dev->id.subsystem_vendor_id && > + id_table->subsystem_vendor_id != PCI_ANY_ID) > + continue; > + if (id_table->subsystem_device_id != > + pci_dev->id.subsystem_device_id && > + id_table->subsystem_device_id != PCI_ANY_ID) > + continue; > + if (id_table->class_id != pci_dev->id.class_id && > + id_table->class_id != RTE_CLASS_ANY_ID) > + continue; > + return true; > + } > + return false; > +} > + > +static int > +is_valid_class_combination(uint32_t user_classes) > +{ > + unsigned int i; > + > + /* Verify if user specified valid supported combination. */ Are there invalid supported combinations? :-) /* Verify if the user specified a supported combination. */ > + for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { > + if (mlx5_class_combinations[i] == user_classes) > + return 0; > + } > + /* Not found any valid class combination. */ > + return -EINVAL; > +} > + > +static struct mlx5_pci_device * > +pci_to_mlx5_device(const struct rte_pci_device *pci_dev) > +{ > + struct mlx5_pci_device *dev; > + > + TAILQ_FOREACH(dev, &devices_list, next) { > + if (dev->pci_dev == pci_dev) > + return dev; > + } > + return NULL; > +} > + > +static bool > +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) > +{ > + return (device->classes_loaded & class) ? true : false; > +} > + > +static void > +dev_release(struct mlx5_pci_device *dev) > +{ > + TAILQ_REMOVE(&devices_list, dev, next); > + rte_free(dev); > +} > + > +static int > +drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) > +{ > + struct mlx5_pci_driver *driver; > + int local_ret = -ENODEV; > + unsigned int i = 0; > + int ret = 0; > + > + enabled_classes &= dev->classes_loaded; > + while (enabled_classes) { > + driver = driver_get(RTE_BIT64(i)); > + if (driver) { > + local_ret = driver->pci_driver.remove(dev->pci_dev); > + if (!local_ret) > + dev->classes_loaded &= ~RTE_BIT64(i); > + else if (ret == 0) > + ret = local_ret; > + } > + enabled_classes &= ~RTE_BIT64(i); > + i++; > + } > + if (local_ret) > + ret = local_ret; > + return ret; > +} > + > +static int > +drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv, > + struct rte_pci_device *pci_dev, uint32_t user_classes) > +{ > + struct mlx5_pci_driver *driver; > + uint32_t enabled_classes = 0; > + bool already_loaded; > + int ret; > + > + TAILQ_FOREACH(driver, &drv_list, next) { > + if ((driver->driver_class & user_classes) == 0) > + continue; > + if (!mlx5_bus_match(driver, pci_dev)) > + continue; > + already_loaded = dev->classes_loaded & driver->driver_class; > + if (already_loaded && > + !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { > + DRV_LOG(ERR, "Device %s is already probed\n", > + pci_dev->device.name); > + ret = -EEXIST; > + goto probe_err; > + } > + ret = driver->pci_driver.probe(pci_drv, pci_dev); > + if (ret < 0) { > + DRV_LOG(ERR, "Failed to load driver = %s.\n", > + driver->pci_driver.driver.name); > + goto probe_err; > + } > + enabled_classes |= driver->driver_class; > + } > + dev->classes_loaded |= enabled_classes; > + return 0; > +probe_err: > + /* Only unload drivers which are enabled which were enabled > + * in this probe instance. > + */ > + drivers_remove(dev, enabled_classes); > + return ret; > +} > + > +/** > + * DPDK callback to register to probe multiple drivers for a PCI device. > + * > + * @param[in] pci_drv > + * PCI driver structure. > + * @param[in] dev > + * PCI device information. > + * > + * @return > + * 0 on success, a negative errno value otherwise and rte_errno is set. > + */ > +static int > +mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, > + struct rte_pci_device *pci_dev) > +{ > + struct mlx5_pci_device *dev; > + uint32_t user_classes = 0; > + bool new_device = false; > + int ret; > + > + ret = parse_class_options(pci_dev->device.devargs); > + if (ret < 0) > + return ret; > + user_classes = ret; > + if (user_classes) { > + /* Validate combination here. */ > + ret = is_valid_class_combination(user_classes); > + if (ret) { > + DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); > + return ret; > + } > + } else { > + /* Default to net class. */ > + user_classes = MLX5_CLASS_NET; > + } > + dev = pci_to_mlx5_device(pci_dev); > + if (!dev) { > + dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); > + if (!dev) > + return -ENOMEM; > + dev->pci_dev = pci_dev; > + TAILQ_INSERT_HEAD(&devices_list, dev, next); > + new_device = true; > + } > + ret = drivers_probe(dev, pci_drv, pci_dev, user_classes); > + if (ret) > + goto class_err; > + return 0; > +class_err: > + if (new_device) > + dev_release(dev); > + return ret; > +} > + > +/** > + * DPDK callback to remove one or more drivers for a PCI device. > + * > + * This function removes all drivers probed for a given PCI device. > + * > + * @param[in] pci_dev > + * Pointer to the PCI device. > + * > + * @return > + * 0 on success, the function cannot fail. > + */ > +static int > +mlx5_common_pci_remove(struct rte_pci_device *pci_dev) > +{ > + struct mlx5_pci_device *dev; > + int ret; > + > + dev = pci_to_mlx5_device(pci_dev); > + if (!dev) > + return -ENODEV; > + /* Matching device found, cleanup and unload drivers. */ > + ret = drivers_remove(dev, dev->classes_loaded); > + if (!ret) > + dev_release(dev); > + return ret; > +} > + > +static int > +mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, > + uint64_t iova, size_t len) > +{ > + struct mlx5_pci_driver *driver = NULL; > + struct mlx5_pci_driver *temp; > + struct mlx5_pci_device *dev; > + int ret = -EINVAL; > + > + dev = pci_to_mlx5_device(pci_dev); > + if (!dev) > + return -ENODEV; > + TAILQ_FOREACH(driver, &drv_list, next) { > + if (device_class_enabled(dev, driver->driver_class) && > + driver->pci_driver.dma_map) { > + ret = driver->pci_driver.dma_map(pci_dev, addr, > + iova, len); > + if (ret) > + goto map_err; > + } > + } > + return ret; > +map_err: > + TAILQ_FOREACH(temp, &drv_list, next) { > + if (temp == driver) > + break; > + if (device_class_enabled(dev, temp->driver_class) && > + temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) > + temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); > + } > + return ret; > +} > + > +static int > +mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, > + uint64_t iova, size_t len) > +{ > + struct mlx5_pci_driver *driver; > + struct mlx5_pci_device *dev; > + int local_ret = -EINVAL; > + int ret; > + > + dev = pci_to_mlx5_device(pci_dev); > + if (!dev) > + return -ENODEV; > + ret = 0; > + /* There is no unmap error recovery in current implementation. */ > + TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) { > + if (device_class_enabled(dev, driver->driver_class) && > + driver->pci_driver.dma_unmap) { > + local_ret = driver->pci_driver.dma_unmap(pci_dev, addr, > + iova, len); > + if (local_ret && (ret == 0)) > + ret = local_ret; > + } > + } > + if (local_ret) > + ret = local_ret; > + return ret; > +} > + > +/* PCI ID table is build dynamically based on registered mlx5 drivers. */ built > +static struct rte_pci_id *mlx5_pci_id_table; > + > +static struct rte_pci_driver mlx5_pci_driver = { > + .driver = { > + .name = "mlx5_pci", > + }, > + .probe = mlx5_common_pci_probe, > + .remove = mlx5_common_pci_remove, > + .dma_map = mlx5_common_pci_dma_map, > + .dma_unmap = mlx5_common_pci_dma_unmap, > +}; > + > +static int > +pci_id_table_size_get(const struct rte_pci_id *id_table) > +{ > + int table_size = 0; > + > + for (; id_table->vendor_id != 0; id_table++) > + table_size++; > + return table_size; > +} > + > +static bool > +pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table, > + int next_idx) > +{ > + int current_size = next_idx - 1; > + int i; > + > + for (i = 0; i < current_size; i++) { > + if (id->device_id == table[i].device_id && > + id->vendor_id == table[i].vendor_id && > + id->subsystem_vendor_id == table[i].subsystem_vendor_id && > + id->subsystem_device_id == table[i].subsystem_device_id) > + return true; > + } > + return false; > +} > + > +static void > +pci_id_insert(struct rte_pci_id *new_table, int *next_idx, > + const struct rte_pci_id *id_table) > +{ > + /* Traverse the id_table, check if entry exists in new_table; > + * Add non duplicate entries to new table. > + */ > + for (; id_table->vendor_id != 0; id_table++) { > + if (!pci_id_exists(id_table, new_table, *next_idx)) { > + /* New entry; add to the table. */ > + new_table[*next_idx] = *id_table; > + (*next_idx)++; > + } > + } > +} > + > +static int > +pci_ids_table_update(const struct rte_pci_id *driver_id_table) > +{ > + const struct rte_pci_id *id_iter; > + struct rte_pci_id *updated_table; > + struct rte_pci_id *old_table; > + int num_ids = 0; > + int i = 0; > + > + old_table = mlx5_pci_id_table; > + if (old_table) > + num_ids = pci_id_table_size_get(old_table); > + num_ids += pci_id_table_size_get(driver_id_table); > + /* Increase size by one for the termination entry of vendor_id = 0. */ > + num_ids += 1; > + updated_table = calloc(num_ids, sizeof(*updated_table)); > + if (!updated_table) > + return -ENOMEM; > + if (TAILQ_EMPTY(&drv_list)) { > + /* Copy the first driver's ID table. */ > + for (id_iter = driver_id_table; id_iter->vendor_id != 0; > + id_iter++, i++) > + updated_table[i] = *id_iter; > + } else { > + /* First copy existing table entries. */ > + for (id_iter = old_table; id_iter->vendor_id != 0; > + id_iter++, i++) > + updated_table[i] = *id_iter; > + /* New id to be added at the end of current ID table. */ > + pci_id_insert(updated_table, &i, driver_id_table); > + } > + /* Terminate table with empty entry. */ > + updated_table[i].vendor_id = 0; > + mlx5_pci_driver.id_table = updated_table; > + mlx5_pci_id_table = updated_table; > + if (old_table) > + free(old_table); > + return 0; > +} > + > +void > +mlx5_pci_driver_register(struct mlx5_pci_driver *driver) > +{ > + int ret; > + > + ret = pci_ids_table_update(driver->pci_driver.id_table); > + if (ret) > + return; > + mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags; > + TAILQ_INSERT_TAIL(&drv_list, driver, next); > +} > + > +void mlx5_common_pci_init(void) > +{ > + const struct rte_pci_id empty_table[] = { > + { > + .vendor_id = 0 > + }, > + }; > + > + /* All mlx5 PMDs constructor runs at same priority. So any of the PMD /* All mlx5 PMDs constructors run at the same priority. So any of the PMD > + * including this one can register the PCI table first. If any other > + * PMD(s) have registered the PCI ID table, No need to register an empty > + * default one. > + */ > + if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table)) > + return; > + rte_pci_register(&mlx5_pci_driver); > +} > + > +RTE_FINI(mlx5_common_pci_finish) > +{ > + if (mlx5_pci_id_table != NULL) { > + /* Constructor doesn't register with PCI bus if it failed > + * to build the table. > + */ > + rte_pci_unregister(&mlx5_pci_driver); > + free(mlx5_pci_id_table); > + } > +} I don't like the asymmetry between multiple constructors calling a common init / and a single destructor in the common code. But this is not really an issue. > +RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__); The pci driver name is mlx5_pci iirc. But if there is no info about this driver to export, you can remove RTE_PMD_EXPORT_NAME. > diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h > new file mode 100644 > index 000000000..41b73e17a > --- /dev/null > +++ b/drivers/common/mlx5/mlx5_common_pci.h > @@ -0,0 +1,77 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2020 Mellanox Technologies, Ltd > + */ > + > +#ifndef _MLX5_COMMON_PCI_H_ > +#define _MLX5_COMMON_PCI_H_ > + > +/** > + * @file > + * > + * RTE Mellanox PCI Driver Interface > + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) > + * devices. This layer enables creating such multiple class of devices on a > + * single PCI device by allowing to bind multiple class specific device > + * driver to attach to mlx5_pci driver. > + * > + * ----------- ------------ ------------- > + * | mlx5 | | mlx5 | | mlx5 | > + * | net pmd | | vdpa pmd | | regex pmd | > + * ----------- ------------ ------------- > + * \ | / > + * \ | / > + * \ -------------- / > + * \______| mlx5 |_____ / > + * | pci common | > + * -------------- > + * | > + * ----------- > + * | mlx5 | > + * | pci dev | > + * ----------- > + * > + * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI > + * ID table of all related mlx5 PCI devices. > + * - mlx5 class driver such as net, vdpa, regex PMD defines its > + * specific PCI ID table and mlx5 bus driver probes matching > + * class drivers. > + * - mlx5 pci bus driver is cental place that validates supported central > + * class combinations. > + */ > + > +#ifdef __cplusplus > +extern "C" { > +#endif /* __cplusplus */ > + > +#include <rte_pci.h> > +#include <rte_bus_pci.h> > + > +#include <mlx5_common.h> > + > +void mlx5_common_pci_init(void); > + > +/** > + * A structure describing a mlx5 pci driver. > + */ > +struct mlx5_pci_driver { > + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ > + uint32_t driver_class; /**< Class of this driver, enum mlx5_class */ > + TAILQ_ENTRY(mlx5_pci_driver) next; > +}; > + > +/** > + * Register a mlx5_pci device driver. > + * > + * @param driver > + * A pointer to a mlx5_pci_driver structure describing the driver > + * to be registered. > + */ > +__rte_internal > +void > +mlx5_pci_driver_register(struct mlx5_pci_driver *driver); > + > +#ifdef __cplusplus > +} > +#endif /* __cplusplus */ > + > +#endif /* _MLX5_COMMON_PCI_H_ */ > diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map > index 65f25252a..73cf72548 100644 > --- a/drivers/common/mlx5/rte_common_mlx5_version.map > +++ b/drivers/common/mlx5/rte_common_mlx5_version.map > @@ -91,5 +91,7 @@ INTERNAL { > mlx5_malloc; > mlx5_realloc; > mlx5_free; > + > + mlx5_pci_driver_register; > }; > > diff --git a/drivers/meson.build b/drivers/meson.build > index e6d0409aa..95df7ef1d 100644 > --- a/drivers/meson.build > +++ b/drivers/meson.build > @@ -5,6 +5,7 @@ > subdirs = [ > 'common', > 'bus', > + 'common/mlx5', # depends on bus. > 'mempool', # depends on common and bus. > 'net', # depends on common, bus, mempool > 'raw', # depends on common, bus and net. > -- > 2.25.4 > Reviewed-by: David Marchand <david.marchand@redhat.com> -- David Marchand ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 09/10] common/mlx5: register class drivers through common layer 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit ` (7 preceding siblings ...) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 08/10] common/mlx5: introduce layer to support multiple class drivers Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 10/10] common/mlx5: remove class check from class drivers Parav Pandit 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Migrate mlx5 net, vdpa and regex PMD to start using mlx5 common class driver. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v7->v8: - Extended support for newly added mlx5 regex driver --- drivers/common/mlx5/mlx5_common_pci.c | 6 ++---- drivers/net/mlx5/Makefile | 1 - drivers/net/mlx5/linux/mlx5_os.c | 1 - drivers/net/mlx5/mlx5.c | 24 ++++++++++++++---------- drivers/net/mlx5/mlx5.h | 1 - drivers/regex/mlx5/meson.build | 2 +- drivers/regex/mlx5/mlx5_regex.c | 21 ++++++++++++--------- drivers/vdpa/mlx5/Makefile | 3 ++- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++---------- 10 files changed, 45 insertions(+), 39 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c index b42f7469f..08c8a5538 100644 --- a/drivers/common/mlx5/mlx5_common_pci.c +++ b/drivers/common/mlx5/mlx5_common_pci.c @@ -34,10 +34,8 @@ static const unsigned int mlx5_class_combinations[] = { MLX5_CLASS_NET, MLX5_CLASS_VDPA, MLX5_CLASS_REGEX, - /* New class combination should be added here. - * For example a new multi class device combination - * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. - */ + MLX5_CLASS_NET | MLX5_CLASS_REGEX, + /* New class combination should be added here. */ }; static int diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 253faf909..568c77241 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -55,7 +55,6 @@ LDLIBS += -lrte_common_mlx5 LDLIBS += -lm LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index fa3b02787..97d7b0e7b 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1537,7 +1537,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); return -rte_errno; } - MLX5_ASSERT(pci_drv == &mlx5_driver); errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 037703d2e..1e4c695f8 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -30,6 +30,7 @@ #include <mlx5_common.h> #include <mlx5_common_os.h> #include <mlx5_common_mp.h> +#include <mlx5_common_pci.h> #include <mlx5_malloc.h> #include "mlx5_defs.h" @@ -2091,16 +2092,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -struct rte_pci_driver mlx5_driver = { - .driver = { - .name = MLX5_DRIVER_NAME +static struct mlx5_pci_driver mlx5_driver = { + .driver_class = MLX5_CLASS_NET, + .pci_driver = { + .driver = { + .name = MLX5_DRIVER_NAME, + }, + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_dma_map, + .dma_unmap = mlx5_dma_unmap, + .drv_flags = PCI_DRV_FLAGS, }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, }; /* Initialize driver log type. */ @@ -2117,7 +2121,7 @@ RTE_INIT(rte_mlx5_pmd_init) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - rte_pci_register(&mlx5_driver); + mlx5_pci_driver_register(&mlx5_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index a92194d2d..78d6eb728 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -119,7 +119,6 @@ struct mlx5_local_data { }; extern struct mlx5_shared_data *mlx5_shared_data; -extern struct rte_pci_driver mlx5_driver; /* Dev ops structs */ extern const struct eth_dev_ops mlx5_os_dev_ops; diff --git a/drivers/regex/mlx5/meson.build b/drivers/regex/mlx5/meson.build index 7f800f2e3..d7cb2c572 100644 --- a/drivers/regex/mlx5/meson.build +++ b/drivers/regex/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_regex' -deps += ['common_mlx5', 'bus_pci', 'eal', 'regexdev'] +deps += ['common_mlx5', 'eal', 'regexdev'] sources = files( 'mlx5_regex.c', 'mlx5_rxp.c', diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c index 4e0367052..ae9f00189 100644 --- a/drivers/regex/mlx5/mlx5_regex.c +++ b/drivers/regex/mlx5/mlx5_regex.c @@ -5,12 +5,12 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> #include <rte_pci.h> #include <rte_regexdev.h> #include <rte_regexdev_core.h> #include <rte_regexdev_driver.h> +#include <mlx5_common_pci.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> @@ -246,21 +246,24 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_regex_driver = { - .driver = { - .name = "mlx5_regex", +static struct mlx5_pci_driver mlx5_regex_driver = { + .driver_class = MLX5_CLASS_REGEX, + .pci_driver = { + .driver = { + .name = "mlx5_regex", + }, + .id_table = mlx5_regex_pci_id_map, + .probe = mlx5_regex_pci_probe, + .remove = mlx5_regex_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_regex_pci_id_map, - .probe = mlx5_regex_pci_probe, - .remove = mlx5_regex_pci_remove, - .drv_flags = 0, }; RTE_INIT(rte_mlx5_regex_init) { mlx5_common_init(); if (mlx5_glue) - rte_pci_register(&mlx5_regex_driver); + mlx5_pci_driver_register(&mlx5_regex_driver); } RTE_LOG_REGISTER(mlx5_regex_logtype, pmd.regex.mlx5, NOTICE) diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index 8a1c2eab5..1cbbcecf8 100644 --- a/drivers/vdpa/mlx5/Makefile +++ b/drivers/vdpa/mlx5/Makefile @@ -24,13 +24,14 @@ CFLAGS += -I$(RTE_SDK)/drivers/common/mlx5/linux CFLAGS += -I$(RTE_SDK)/drivers/net/mlx5_vdpa CFLAGS += -I$(RTE_SDK)/lib/librte_sched CFLAGS += -I$(BUILDDIR)/drivers/common/mlx5 +CFLAGS += -I$(RTE_SDK)/drivers/bus/mlx5_pci CFLAGS += -D_BSD_SOURCE CFLAGS += -D_DEFAULT_SOURCE CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_sched LDLIBS += -pthread # A few warnings cannot be avoided in external headers. diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build index 2963aad71..254a8ec14 100644 --- a/drivers/vdpa/mlx5/meson.build +++ b/drivers/vdpa/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_vdpa' -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched'] +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'eal', 'sched'] sources = files( 'mlx5_vdpa.c', 'mlx5_vdpa_mem.c', diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 85dbcf956..ffe2f00ac 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -11,12 +11,12 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> #include <rte_pci.h> #include <rte_string_fns.h> #include <mlx5_glue.h> #include <mlx5_common.h> +#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> #include <mlx5_nl.h> @@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) } /** - * DPDK callback to register a PCI device. + * DPDK callback to register a mlx5 PCI device. * * This function spawns vdpa device out of a given PCI device. * @@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_vdpa_driver = { - .driver = { - .name = "mlx5_vdpa", +static struct mlx5_pci_driver mlx5_vdpa_driver = { + .driver_class = MLX5_CLASS_VDPA, + .pci_driver = { + .driver = { + .name = "mlx5_vdpa", + }, + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_pci_probe, + .remove = mlx5_vdpa_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, }; RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) @@ -848,7 +851,7 @@ RTE_INIT(rte_mlx5_vdpa_init) { mlx5_common_init(); if (mlx5_glue) - rte_pci_register(&mlx5_vdpa_driver); + mlx5_pci_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v8 10/10] common/mlx5: remove class check from class drivers 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit ` (8 preceding siblings ...) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 09/10] common/mlx5: register class drivers through common layer Parav Pandit @ 2020-07-23 20:09 ` Parav Pandit 9 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-23 20:09 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Now that mlx5_pci PMD checks for enabled classes and performs probe(), remove() of associated classes, individual class driver does not need to check if other driver is enabled. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/mlx5_common.c | 37 ------------------- drivers/common/mlx5/mlx5_common.h | 2 - .../common/mlx5/rte_common_mlx5_version.map | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 5 --- drivers/vdpa/mlx5/mlx5_vdpa.c | 5 --- 5 files changed, 1 insertion(+), 51 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index fd818ef24..06f0a6400 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -20,43 +20,6 @@ int mlx5_common_logtype; uint8_t haswell_broadwell_cpu; -static int -mlx5_class_check_handler(__rte_unused const char *key, const char *value, - void *opaque) -{ - enum mlx5_class *ret = opaque; - - if (strcmp(value, "vdpa") == 0) { - *ret = MLX5_CLASS_VDPA; - } else if (strcmp(value, "net") == 0) { - *ret = MLX5_CLASS_NET; - } else { - DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device" - " class argument setting?", value); - *ret = MLX5_CLASS_INVALID; - } - return 0; -} - -enum mlx5_class -mlx5_class_get(struct rte_devargs *devargs) -{ - struct rte_kvargs *kvlist; - const char *key = MLX5_CLASS_ARG_NAME; - enum mlx5_class ret = MLX5_CLASS_NET; - - if (devargs == NULL) - return ret; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return ret; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret); - rte_kvargs_free(kvlist); - return ret; -} - - /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 864e82ff0..2cdb226f3 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -247,8 +247,6 @@ struct mlx5_klm { LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page); -__rte_internal -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs); __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 73cf72548..7729d0dd0 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,9 +1,8 @@ INTERNAL { global: - mlx5_class_get; - mlx5_common_init; + mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 97d7b0e7b..fc339affa 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1524,11 +1524,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int dev_config_vf; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } if (rte_eal_process_type() == RTE_PROC_PRIMARY) mlx5_pmd_socket_init(); ret = mlx5_init_once(); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index ffe2f00ac..c0b87bcc0 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_hca_attr attr; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr); if (!ibv) { DRV_LOG(ERR, "No matching IB device for PCI slot " -- 2.25.4 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (7 preceding siblings ...) 2020-07-23 20:09 ` [dpdk-dev] [PATCH v8 00/10] " Parav Pandit @ 2020-07-24 14:23 ` Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 01/10] eal: introduce macro for bit definition Parav Pandit ` (10 more replies) 2020-07-24 14:38 ` [dpdk-dev] [PATCH v10 " Parav Pandit ` (2 subsequent siblings) 11 siblings, 11 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:23 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit This series introduces mlx5 common driver layer to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, mlx5 common module is extended as generic PCI PMD. This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded on a single PCI Device. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Fixed indentation for directory listing Patch-3 Relax dependency order Patch-4 Fixes compilation error Patch-5 Fixes segmentation fault in regex probe error unwind path Patch-6 Avoid using class constructor priority Patch-7 Change class values as bits Patch-8 Introduce mlx5 layer to support multiple class drivers Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API instead of rte PCI bus API Patch-10 Removed class check code from class drivers Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ --------------- / \______| mlx5 common |____/ | pmd | --------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 common driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD define their specific PCI ID tables. mlx5 common driver probes each individual class driver(s) for maching and enabled classes. - mlx5 pci layer is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI common driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci common PMD, a common driver is implemented which exposes class registration API. However, common PMD model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common driver. It also eliminates peer blind zone problem as bottom pci layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:regex ./testpmd -w <PCI BDF>,class=vdpa Changelog: v8->v9: - Updated commit message - Fixed LD_FLAGS - Fixed white space to tab - Fixed Makefile to have dependency on common/mlx5 for parallel builds v7->v8: - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5 - Avoided new RTE PRIORITY - Removed mlx5 common to use RTE_PRIORITY_CLASS v6->v7: - Updated release notes v5->v6: - Fixed compilation failure in parallel build for shared lib v4->v5: - Squash the maintainers update path with other patch which adds the bus - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (8): eal: introduce macro for bit definition common/mlx5: fix void parameters in glue wrappers regex/mlx5: fix segmentation fault during error unwinding common/mlx5: avoid using class constructor priority common/mlx5: change class values as bits common/mlx5: introduce layer to support multiple class drivers common/mlx5: register class drivers through common layer common/mlx5: remove class check from class drivers Thomas Monjalon (2): drivers: fix indent of directory list drivers: relax dependency order drivers/Makefile | 10 +- drivers/common/Makefile | 4 - drivers/common/meson.build | 2 +- drivers/common/mlx5/Makefile | 2 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/meson.build | 9 +- drivers/common/mlx5/mlx5_common.c | 52 +- drivers/common/mlx5/mlx5_common.h | 12 +- drivers/common/mlx5/mlx5_common_pci.c | 539 ++++++++++++++++++ drivers/common/mlx5/mlx5_common_pci.h | 77 +++ .../common/mlx5/rte_common_mlx5_version.map | 4 +- drivers/meson.build | 51 +- drivers/net/mlx5/Makefile | 1 - drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/mlx5.c | 25 +- drivers/net/mlx5/mlx5.h | 1 - drivers/regex/mlx5/meson.build | 2 +- drivers/regex/mlx5/mlx5_regex.c | 29 +- drivers/vdpa/mlx5/Makefile | 2 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 29 +- lib/librte_eal/include/rte_bitops.h | 8 + 22 files changed, 748 insertions(+), 123 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_pci.c create mode 100644 drivers/common/mlx5/mlx5_common_pci.h -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 01/10] eal: introduce macro for bit definition 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-24 14:23 ` Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 02/10] drivers: fix indent of directory list Parav Pandit ` (9 subsequent siblings) 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:23 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit, Morten Brørup There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> Acked-by: Morten Brørup <mb@smartsharesystems.com> --- Changelog: v4->v5: - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v1->v2: - Addressed comments from Thomas and Gaten. - Avoided new file, added macro to rte_bitops.h --- lib/librte_eal/include/rte_bitops.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..ca46a110f 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,14 @@ #include <rte_debug.h> #include <rte_compat.h> +/** + * Get the uint64_t value for a specified bit set. + * + * @param nr + * The bit number in range of 0 to 63. + */ +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 02/10] drivers: fix indent of directory list 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 01/10] eal: introduce macro for bit definition Parav Pandit @ 2020-07-24 14:23 ` Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 03/10] drivers: relax dependency order Parav Pandit ` (8 subsequent siblings) 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:23 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Bruce Richardson, David Marchand From: Thomas Monjalon <thomas@monjalon.net> Define each sub-directory on its own line ended with a comma, and use a simple indent. Acked-by: Bruce Richardson <bruce.richardson@intel.com> Reviewed-by: David Marchand <david.marchand@redhat.com> Signed-off-by: Thomas Monjalon <thomas@monjalon.net> --- Changelog: v7->8: - new patch --- drivers/meson.build | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/drivers/meson.build b/drivers/meson.build index e76ebddfa..e2aeba931 100644 --- a/drivers/meson.build +++ b/drivers/meson.build @@ -2,17 +2,19 @@ # Copyright(c) 2017-2019 Intel Corporation # Defines the order in which the drivers are buit. -dpdk_driver_classes = ['common', - 'bus', - 'mempool', # depends on common and bus. - 'net', # depends on common, bus, mempool - 'raw', # depends on common, bus and net. - 'crypto', # depends on common, bus and mempool (net in future). - 'compress', # depends on common, bus, mempool. - 'regex', # depends on common, bus, regexdev. - 'vdpa', # depends on common, bus and mempool. - 'event', # depends on common, bus, mempool and net. - 'baseband'] # depends on common and bus. +dpdk_driver_classes = [ + 'common', + 'bus', + 'mempool', # depends on common and bus. + 'net', # depends on common, bus, mempool + 'raw', # depends on common, bus and net. + 'crypto', # depends on common, bus and mempool (net in future). + 'compress', # depends on common, bus, mempool. + 'regex', # depends on common, bus, regexdev. + 'vdpa', # depends on common, bus and mempool. + 'event', # depends on common, bus, mempool and net. + 'baseband', # depends on common and bus. +] disabled_drivers = run_command(list_dir_globs, get_option('disable_drivers'), ).stdout().split() -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 03/10] drivers: relax dependency order 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 01/10] eal: introduce macro for bit definition Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 02/10] drivers: fix indent of directory list Parav Pandit @ 2020-07-24 14:23 ` Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 04/10] common/mlx5: fix void parameters in glue wrappers Parav Pandit ` (7 subsequent siblings) 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:23 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Bruce Richardson From: Thomas Monjalon <thomas@monjalon.net> Drivers dependencies are evaluated in the order defined per their parent directory (also called class). This strict ordering prevents from having pairs of drivers from two classes with different dependency ordering. For example, if the mlx5 common code depends on the pci bus driver, while the dpaax common code is itself a dependency of the dpaa bus driver. Having a strict ordering between directories bus and common is too restrictive, as processing either common drivers or bus drivers first leads us to missing dependencies in this scenario. This patch makes it possible to have a more fine-grain directory list, adding a specific driver sub-directory in the top-level drivers subdirectory list. In this case, the isolated driver must also be removed from its class list, and the per-class variables must be duplicated in the isolated driver, because the call "subdir(class)" is skipped in the isolated driver case. Review-by: Bruce Richardson <bruce.richardson@intel.com> Acked-by: Bruce Richardson <bruce.richardson@intel.com> Signed-off-by: Thomas Monjalon <thomas@monjalon.net> --- Changelog: v8->v9: - Updated commit message - Fixed typo --- drivers/meson.build | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/drivers/meson.build b/drivers/meson.build index e2aeba931..e6d0409aa 100644 --- a/drivers/meson.build +++ b/drivers/meson.build @@ -1,8 +1,8 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright(c) 2017-2019 Intel Corporation -# Defines the order in which the drivers are buit. -dpdk_driver_classes = [ +# Defines the order of dependencies evaluation +subdirs = [ 'common', 'bus', 'mempool', # depends on common and bus. @@ -27,7 +27,7 @@ if cc.has_argument('-Wno-format-truncation') default_cflags += '-Wno-format-truncation' endif -foreach class:dpdk_driver_classes +foreach subpath:subdirs drivers = [] std_deps = [] config_flag_fmt = '' # format string used to set the value in dpdk_conf @@ -35,8 +35,22 @@ foreach class:dpdk_driver_classes # the library, the dependency and to find the # version file for linking - subdir(class) - class_drivers = [] + # subpath can be either "class" or "class/driver" + if subpath.contains('/') + driver_path = subpath.split('/') + class = driver_path[0] + drivers += driver_path[1] + else + class = subpath + subdir(class) + endif + + # save class name on first occurence + if not dpdk_driver_classes.contains(class) + dpdk_driver_classes += class + endif + # get already enabled drivers of the same class + enabled_drivers = get_variable(class + '_drivers', []) foreach drv:drivers drv_path = join_paths(class, drv) @@ -96,7 +110,7 @@ foreach class:dpdk_driver_classes '_disable_reason', reason) endif else - class_drivers += name + enabled_drivers += name if fmt_name == '' fmt_name = name @@ -203,5 +217,5 @@ foreach class:dpdk_driver_classes endif # build endforeach - set_variable(class + '_drivers', class_drivers) + set_variable(class + '_drivers', enabled_drivers) endforeach -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 04/10] common/mlx5: fix void parameters in glue wrappers 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (2 preceding siblings ...) 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 03/10] drivers: relax dependency order Parav Pandit @ 2020-07-24 14:23 ` Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 05/10] regex/mlx5: fix segmentation fault during error unwinding Parav Pandit ` (6 subsequent siblings) 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:23 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Following two errors are reported when compiled with gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5). drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] drivers/common/mlx5/linux/mlx5_glue.h:188:2: error: function declaration isn't a prototype [-Werror=strict-prototypes] Fix them by adding void data type in empty argument list. Fixes: 34fa7c0268e7 ("net/mlx5: add drop action to Direct Verbs E-Switch") Fixes: 400d985eb586 ("net/mlx5: add VLAN push/pop DR commands to glue") Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v2->v3: - new patch --- drivers/common/mlx5/linux/mlx5_glue.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/linux/mlx5_glue.h b/drivers/common/mlx5/linux/mlx5_glue.h index c4f9b006a..734ace2a3 100644 --- a/drivers/common/mlx5/linux/mlx5_glue.h +++ b/drivers/common/mlx5/linux/mlx5_glue.h @@ -185,11 +185,11 @@ struct mlx5_glue { void *(*dr_create_flow_action_dest_flow_tbl)(void *tbl); void *(*dr_create_flow_action_dest_port)(void *domain, uint32_t port); - void *(*dr_create_flow_action_drop)(); + void *(*dr_create_flow_action_drop)(void); void *(*dr_create_flow_action_push_vlan) (struct mlx5dv_dr_domain *domain, rte_be32_t vlan_tag); - void *(*dr_create_flow_action_pop_vlan)(); + void *(*dr_create_flow_action_pop_vlan)(void); void *(*dr_create_flow_tbl)(void *domain, uint32_t level); int (*dr_destroy_flow_tbl)(void *tbl); void *(*dr_create_domain)(struct ibv_context *ctx, -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 05/10] regex/mlx5: fix segmentation fault during error unwinding 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (3 preceding siblings ...) 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 04/10] common/mlx5: fix void parameters in glue wrappers Parav Pandit @ 2020-07-24 14:23 ` Parav Pandit 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 06/10] common/mlx5: avoid using class constructor priority Parav Pandit ` (5 subsequent siblings) 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:23 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit When fail to initialize the device, avoid segmentation fault while accessing unintialized priv. Fixes: cfc672a90b74 ("regex/mlx5: support probing") Signed-off-by: Parav Pandit <parav@mellanox.com> --- Changelog: v7->v8: - Rebased - new patch --- drivers/regex/mlx5/mlx5_regex.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c index 1ca5bfe9b..36ae9f809 100644 --- a/drivers/regex/mlx5/mlx5_regex.c +++ b/drivers/regex/mlx5/mlx5_regex.c @@ -137,17 +137,17 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, if (ret) { DRV_LOG(ERR, "Unable to read HCA capabilities."); rte_errno = ENOTSUP; - goto error; + goto dev_error; } else if (!attr.regex || attr.regexp_num_of_engines == 0) { DRV_LOG(ERR, "Not enough capabilities to support RegEx, maybe " "old FW/OFED version?"); rte_errno = ENOTSUP; - goto error; + goto dev_error; } if (mlx5_regex_engines_status(ctx, 2)) { DRV_LOG(ERR, "RegEx engine error."); rte_errno = ENOMEM; - goto error; + goto dev_error; } priv = rte_zmalloc("mlx5 regex device private", sizeof(*priv), RTE_CACHE_LINE_SIZE); @@ -200,6 +200,7 @@ mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, mlx5_glue->devx_free_uar(priv->uar); if (priv->regexdev) rte_regexdev_unregister(priv->regexdev); +dev_error: if (ctx) mlx5_glue->close_device(ctx); if (priv) -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 06/10] common/mlx5: avoid using class constructor priority 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (4 preceding siblings ...) 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 05/10] regex/mlx5: fix segmentation fault during error unwinding Parav Pandit @ 2020-07-24 14:23 ` Parav Pandit 2020-07-24 14:24 ` [dpdk-dev] [PATCH v9 07/10] common/mlx5: change class values as bits Parav Pandit ` (4 subsequent siblings) 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:23 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit, David Marchand mlx5_common is shared library between mlx5 net, VDPA and regex PMD. It is better to use common initialization helper instead of using RTE_PRIORITY_CLASS priority. Reviewed-by: David Marchand <david.marchand@redhat.com> Suggested-by: David Marchand <david.marchand@redhat.com> Signed-off-by: Parav Pandit <parav@mellanox.com> --- Changelog: v7->v8: - new patch --- drivers/common/mlx5/mlx5_common.c | 13 +++++++++++-- drivers/common/mlx5/mlx5_common.h | 3 +++ drivers/common/mlx5/rte_common_mlx5_version.map | 1 + drivers/net/mlx5/mlx5.c | 1 + drivers/regex/mlx5/mlx5_regex.c | 1 + drivers/vdpa/mlx5/mlx5_vdpa.c | 1 + 6 files changed, 18 insertions(+), 2 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 1a67a1b16..2b336bb2d 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -86,12 +86,21 @@ RTE_INIT_PRIO(mlx5_log_init, LOG) rte_log_set_level(mlx5_common_logtype, RTE_LOG_NOTICE); } +static bool mlx5_common_initialized; + /** - * Initialization routine for run-time dependency on glue library. + * One time innitialization routine for run-time dependency on glue library + * for multiple PMDs. Each mlx5 PMD that depends on mlx5_common module, + * must invoke in its constructor. */ -RTE_INIT_PRIO(mlx5_glue_init, CLASS) +void +mlx5_common_init(void) { + if (mlx5_common_initialized) + return; + mlx5_glue_constructor(); + mlx5_common_initialized = true; } /** diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index a811eb6c9..ebe4e9ced 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -260,4 +260,7 @@ int32_t mlx5_release_dbr(struct mlx5_dbr_page_list *head, uint32_t umem_id, uint64_t offset); extern uint8_t haswell_broadwell_cpu; +__rte_internal +void mlx5_common_init(void); + #endif /* RTE_PMD_MLX5_COMMON_H_ */ diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 132a0695f..65f25252a 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -3,6 +3,7 @@ INTERNAL { mlx5_class_get; + mlx5_common_init; mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 647ada339..037703d2e 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -2111,6 +2111,7 @@ RTE_LOG_REGISTER(mlx5_logtype, pmd.net.mlx5, NOTICE) */ RTE_INIT(rte_mlx5_pmd_init) { + mlx5_common_init(); /* Build the static tables for Verbs conversion. */ mlx5_set_ptype_table(); mlx5_set_cksum_table(); diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c index 36ae9f809..4e0367052 100644 --- a/drivers/regex/mlx5/mlx5_regex.c +++ b/drivers/regex/mlx5/mlx5_regex.c @@ -258,6 +258,7 @@ static struct rte_pci_driver mlx5_regex_driver = { RTE_INIT(rte_mlx5_regex_init) { + mlx5_common_init(); if (mlx5_glue) rte_pci_register(&mlx5_regex_driver); } diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 67e77b11a..85dbcf956 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -846,6 +846,7 @@ RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) */ RTE_INIT(rte_mlx5_vdpa_init) { + mlx5_common_init(); if (mlx5_glue) rte_pci_register(&mlx5_vdpa_driver); } -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 07/10] common/mlx5: change class values as bits 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (5 preceding siblings ...) 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 06/10] common/mlx5: avoid using class constructor priority Parav Pandit @ 2020-07-24 14:24 ` Parav Pandit 2020-07-24 14:24 ` [dpdk-dev] [PATCH v9 08/10] common/mlx5: introduce layer to support multiple class drivers Parav Pandit ` (3 subsequent siblings) 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:24 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit mlx5 PCI Device supports multiple classes of devices such as net, vdpa, and/or regex. To support these multiple classes, change mlx5_class to a bitmap values so that if users asks to enable multiple of them, all supported classes can be parsed. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v7->v8: - updated commit message --- drivers/common/mlx5/mlx5_common.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index ebe4e9ced..864e82ff0 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -13,6 +13,7 @@ #include <rte_log.h> #include <rte_kvargs.h> #include <rte_devargs.h> +#include <rte_bitops.h> #include "mlx5_prm.h" #include "mlx5_devx_cmds.h" @@ -208,10 +209,10 @@ int mlx5_get_ifname_sysfs(const char *ibdev_path, char *ifname); #define MLX5_CLASS_ARG_NAME "class" enum mlx5_class { - MLX5_CLASS_NET, - MLX5_CLASS_VDPA, - MLX5_CLASS_REGEX, MLX5_CLASS_INVALID, + MLX5_CLASS_NET = RTE_BIT64(0), + MLX5_CLASS_VDPA = RTE_BIT64(1), + MLX5_CLASS_REGEX = RTE_BIT64(2), }; #define MLX5_DBR_SIZE RTE_CACHE_LINE_SIZE -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 08/10] common/mlx5: introduce layer to support multiple class drivers 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (6 preceding siblings ...) 2020-07-24 14:24 ` [dpdk-dev] [PATCH v9 07/10] common/mlx5: change class values as bits Parav Pandit @ 2020-07-24 14:24 ` Parav Pandit 2020-07-24 14:24 ` [dpdk-dev] [PATCH v9 09/10] common/mlx5: register class drivers through common layer Parav Pandit ` (2 subsequent siblings) 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:24 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Add generic mlx5 PCI PMD layer as part of existing common_mlx5 module. This enables multiple classes (net, regex, vdpa) PMDs to be supported at same time. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v8->v9: - Added missing LDFLAG for pci bus - Fixed white spaces at start of the line --- drivers/Makefile | 10 +- drivers/common/Makefile | 4 - drivers/common/meson.build | 2 +- drivers/common/mlx5/Makefile | 2 + drivers/common/mlx5/meson.build | 9 +- drivers/common/mlx5/mlx5_common.c | 2 + drivers/common/mlx5/mlx5_common_pci.c | 541 ++++++++++++++++++ drivers/common/mlx5/mlx5_common_pci.h | 77 +++ .../common/mlx5/rte_common_mlx5_version.map | 2 + drivers/meson.build | 1 + 10 files changed, 639 insertions(+), 11 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_pci.c create mode 100644 drivers/common/mlx5/mlx5_common_pci.h diff --git a/drivers/Makefile b/drivers/Makefile index 1551272ef..7f06162dc 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -8,8 +8,12 @@ DIRS-y += bus DEPDIRS-bus := common DIRS-y += mempool DEPDIRS-mempool := common bus +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y) +DIRS-y += common/mlx5 +DEPDIRS-common/mlx5 := bus +endif DIRS-y += net -DEPDIRS-net := common bus mempool +DEPDIRS-net := common bus mempool common/mlx5 DIRS-$(CONFIG_RTE_LIBRTE_BBDEV) += baseband DEPDIRS-baseband := common bus mempool DIRS-$(CONFIG_RTE_LIBRTE_CRYPTODEV) += crypto @@ -19,9 +23,9 @@ DEPDIRS-common/qat := bus mempool DIRS-$(CONFIG_RTE_LIBRTE_COMPRESSDEV) += compress DEPDIRS-compress := bus mempool DIRS-$(CONFIG_RTE_LIBRTE_REGEXDEV) += regex -DEPDIRS-regex := common bus +DEPDIRS-regex := common bus common/mlx5 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vdpa -DEPDIRS-vdpa := common bus mempool +DEPDIRS-vdpa := common bus mempool common/mlx5 DIRS-$(CONFIG_RTE_LIBRTE_EVENTDEV) += event DEPDIRS-event := common bus mempool net crypto DIRS-$(CONFIG_RTE_LIBRTE_RAWDEV) += raw diff --git a/drivers/common/Makefile b/drivers/common/Makefile index cbc71077c..cfb6b4dc8 100644 --- a/drivers/common/Makefile +++ b/drivers/common/Makefile @@ -36,8 +36,4 @@ ifneq (,$(findstring y,$(IAVF-y))) DIRS-y += iavf endif -ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD)$(CONFIG_RTE_LIBRTE_MLX5_REGEX_PMD)),y) -DIRS-y += mlx5 -endif - include $(RTE_SDK)/mk/rte.subdir.mk diff --git a/drivers/common/meson.build b/drivers/common/meson.build index 5db7e29b1..9ed4c04ba 100644 --- a/drivers/common/meson.build +++ b/drivers/common/meson.build @@ -6,6 +6,6 @@ if is_windows endif std_deps = ['eal'] -drivers = ['cpt', 'dpaax', 'iavf', 'mlx5', 'mvep', 'octeontx', 'octeontx2', 'qat'] +drivers = ['cpt', 'dpaax', 'iavf', 'mvep', 'octeontx', 'octeontx2', 'qat'] config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON' driver_name_fmt = 'rte_common_@0@' diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile index 6b89a6c85..4edd54104 100644 --- a/drivers/common/mlx5/Makefile +++ b/drivers/common/mlx5/Makefile @@ -22,6 +22,7 @@ SRCS-y += linux/mlx5_common_verbs.c SRCS-y += mlx5_common_mp.c SRCS-y += mlx5_common_mr.c SRCS-y += mlx5_malloc.c +SRCS-y += mlx5_common_pci.c ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) INSTALL-y-lib += $(LIB_GLUE) endif @@ -51,6 +52,7 @@ LDLIBS += -libverbs -lmlx5 endif LDLIBS += -lrte_eal -lrte_pci -lrte_kvargs -lrte_net +LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual -UPEDANTIC diff --git a/drivers/common/mlx5/meson.build b/drivers/common/mlx5/meson.build index 70e2c1c32..8e5608703 100644 --- a/drivers/common/mlx5/meson.build +++ b/drivers/common/mlx5/meson.build @@ -1,19 +1,22 @@ # SPDX-License-Identifier: BSD-3-Clause # Copyright 2019 Mellanox Technologies, Ltd -if not (is_linux or is_windows) +if not is_linux build = false - reason = 'only supported on Linux and Windows' + reason = 'only supported on Linux' subdir_done() endif -deps += ['hash', 'pci', 'net', 'eal', 'kvargs'] +config_flag_fmt = 'RTE_LIBRTE_@0@_COMMON' +driver_name_fmt = 'rte_common_@0@' +deps += ['hash', 'pci', 'bus_pci', 'net', 'eal', 'kvargs'] sources += files( 'mlx5_devx_cmds.c', 'mlx5_common.c', 'mlx5_common_mp.c', 'mlx5_common_mr.c', 'mlx5_malloc.c', + 'mlx5_common_pci.c', ) cflags_options = [ diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index 2b336bb2d..fd818ef24 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -14,6 +14,7 @@ #include "mlx5_common_os.h" #include "mlx5_common_utils.h" #include "mlx5_malloc.h" +#include "mlx5_common_pci.h" int mlx5_common_logtype; @@ -100,6 +101,7 @@ mlx5_common_init(void) return; mlx5_glue_constructor(); + mlx5_common_pci_init(); mlx5_common_initialized = true; } diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c new file mode 100644 index 000000000..adb3f707d --- /dev/null +++ b/drivers/common/mlx5/mlx5_common_pci.c @@ -0,0 +1,541 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies Ltd + */ + +#include <stdlib.h> +#include <rte_malloc.h> +#include "mlx5_common_utils.h" +#include "mlx5_common_pci.h" + +struct mlx5_pci_device { + struct rte_pci_device *pci_dev; + TAILQ_ENTRY(mlx5_pci_device) next; + uint32_t classes_loaded; +}; + +/* Head of list of drivers. */ +static TAILQ_HEAD(mlx5_pci_bus_drv_head, mlx5_pci_driver) drv_list = + TAILQ_HEAD_INITIALIZER(drv_list); + +/* Head of mlx5 pci devices. */ +static TAILQ_HEAD(mlx5_pci_devices_head, mlx5_pci_device) devices_list = + TAILQ_HEAD_INITIALIZER(devices_list); + +static const struct { + const char *name; + unsigned int driver_class; +} mlx5_classes[] = { + { .name = "vdpa", .driver_class = MLX5_CLASS_VDPA }, + { .name = "net", .driver_class = MLX5_CLASS_NET }, + { .name = "regex", .driver_class = MLX5_CLASS_REGEX }, +}; + +static const unsigned int mlx5_class_combinations[] = { + MLX5_CLASS_NET, + MLX5_CLASS_VDPA, + MLX5_CLASS_REGEX, + /* New class combination should be added here. + * For example a new multi class device combination + * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. + */ +}; + +static int +class_name_to_value(const char *class_name) +{ + unsigned int i; + + for (i = 0; i < RTE_DIM(mlx5_classes); i++) { + if (strcmp(class_name, mlx5_classes[i].name) == 0) + return mlx5_classes[i].driver_class; + } + return -EINVAL; +} + +static struct mlx5_pci_driver * +driver_get(uint32_t class) +{ + struct mlx5_pci_driver *driver; + + TAILQ_FOREACH(driver, &drv_list, next) { + if (driver->driver_class == class) + return driver; + } + return NULL; +} + +static int +bus_cmdline_options_handler(__rte_unused const char *key, + const char *class_names, void *opaque) +{ + int *ret = opaque; + char *nstr_org; + int class_val; + char *found; + char *nstr; + + *ret = 0; + nstr = strdup(class_names); + if (!nstr) { + *ret = -ENOMEM; + return *ret; + } + nstr_org = nstr; + while (nstr) { + /* Extract each individual class name. Multiple + * class key,value is supplied as class=net:vdpa:foo:bar. + */ + found = strsep(&nstr, ":"); + if (!found) + continue; + /* Check if its a valid class. */ + class_val = class_name_to_value(found); + if (class_val < 0) { + *ret = -EINVAL; + goto err; + } + *ret |= class_val; + } +err: + free(nstr_org); + if (*ret < 0) + DRV_LOG(ERR, "Invalid mlx5 class options %s." + " Maybe typo in device class argument setting?", + class_names); + return *ret; +} + +static int +parse_class_options(const struct rte_devargs *devargs) +{ + const char *key = MLX5_CLASS_ARG_NAME; + struct rte_kvargs *kvlist; + int ret = 0; + + if (devargs == NULL) + return 0; + kvlist = rte_kvargs_parse(devargs->args, NULL); + if (kvlist == NULL) + return 0; + if (rte_kvargs_count(kvlist, key)) + rte_kvargs_process(kvlist, key, bus_cmdline_options_handler, + &ret); + rte_kvargs_free(kvlist); + return ret; +} + +static bool +mlx5_bus_match(const struct mlx5_pci_driver *drv, + const struct rte_pci_device *pci_dev) +{ + const struct rte_pci_id *id_table; + + for (id_table = drv->pci_driver.id_table; id_table->vendor_id != 0; + id_table++) { + /* Check if device's ids match the class driver's ids. */ + if (id_table->vendor_id != pci_dev->id.vendor_id && + id_table->vendor_id != PCI_ANY_ID) + continue; + if (id_table->device_id != pci_dev->id.device_id && + id_table->device_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_vendor_id != + pci_dev->id.subsystem_vendor_id && + id_table->subsystem_vendor_id != PCI_ANY_ID) + continue; + if (id_table->subsystem_device_id != + pci_dev->id.subsystem_device_id && + id_table->subsystem_device_id != PCI_ANY_ID) + continue; + if (id_table->class_id != pci_dev->id.class_id && + id_table->class_id != RTE_CLASS_ANY_ID) + continue; + return true; + } + return false; +} + +static int +is_valid_class_combination(uint32_t user_classes) +{ + unsigned int i; + + /* Verify if user specified valid supported combination. */ + for (i = 0; i < RTE_DIM(mlx5_class_combinations); i++) { + if (mlx5_class_combinations[i] == user_classes) + return 0; + } + /* Not found any valid class combination. */ + return -EINVAL; +} + +static struct mlx5_pci_device * +pci_to_mlx5_device(const struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + + TAILQ_FOREACH(dev, &devices_list, next) { + if (dev->pci_dev == pci_dev) + return dev; + } + return NULL; +} + +static bool +device_class_enabled(const struct mlx5_pci_device *device, uint32_t class) +{ + return (device->classes_loaded & class) ? true : false; +} + +static void +dev_release(struct mlx5_pci_device *dev) +{ + TAILQ_REMOVE(&devices_list, dev, next); + rte_free(dev); +} + +static int +drivers_remove(struct mlx5_pci_device *dev, uint32_t enabled_classes) +{ + struct mlx5_pci_driver *driver; + int local_ret = -ENODEV; + unsigned int i = 0; + int ret = 0; + + enabled_classes &= dev->classes_loaded; + while (enabled_classes) { + driver = driver_get(RTE_BIT64(i)); + if (driver) { + local_ret = driver->pci_driver.remove(dev->pci_dev); + if (!local_ret) + dev->classes_loaded &= ~RTE_BIT64(i); + else if (ret == 0) + ret = local_ret; + } + enabled_classes &= ~RTE_BIT64(i); + i++; + } + if (local_ret) + ret = local_ret; + return ret; +} + +static int +drivers_probe(struct mlx5_pci_device *dev, struct rte_pci_driver *pci_drv, + struct rte_pci_device *pci_dev, uint32_t user_classes) +{ + struct mlx5_pci_driver *driver; + uint32_t enabled_classes = 0; + bool already_loaded; + int ret; + + TAILQ_FOREACH(driver, &drv_list, next) { + if ((driver->driver_class & user_classes) == 0) + continue; + if (!mlx5_bus_match(driver, pci_dev)) + continue; + already_loaded = dev->classes_loaded & driver->driver_class; + if (already_loaded && + !(driver->pci_driver.drv_flags & RTE_PCI_DRV_PROBE_AGAIN)) { + DRV_LOG(ERR, "Device %s is already probed\n", + pci_dev->device.name); + ret = -EEXIST; + goto probe_err; + } + ret = driver->pci_driver.probe(pci_drv, pci_dev); + if (ret < 0) { + DRV_LOG(ERR, "Failed to load driver = %s.\n", + driver->pci_driver.driver.name); + goto probe_err; + } + enabled_classes |= driver->driver_class; + } + dev->classes_loaded |= enabled_classes; + return 0; +probe_err: + /* Only unload drivers which are enabled which were enabled + * in this probe instance. + */ + drivers_remove(dev, enabled_classes); + return ret; +} + +/** + * DPDK callback to register to probe multiple drivers for a PCI device. + * + * @param[in] pci_drv + * PCI driver structure. + * @param[in] dev + * PCI device information. + * + * @return + * 0 on success, a negative errno value otherwise and rte_errno is set. + */ +static int +mlx5_common_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, + struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + uint32_t user_classes = 0; + bool new_device = false; + int ret; + + ret = parse_class_options(pci_dev->device.devargs); + if (ret < 0) + return ret; + user_classes = ret; + if (user_classes) { + /* Validate combination here. */ + ret = is_valid_class_combination(user_classes); + if (ret) { + DRV_LOG(ERR, "Unsupported mlx5 classes supplied."); + return ret; + } + } else { + /* Default to net class. */ + user_classes = MLX5_CLASS_NET; + } + dev = pci_to_mlx5_device(pci_dev); + if (!dev) { + dev = rte_zmalloc("mlx5_pci_device", sizeof(*dev), 0); + if (!dev) + return -ENOMEM; + dev->pci_dev = pci_dev; + TAILQ_INSERT_HEAD(&devices_list, dev, next); + new_device = true; + } + ret = drivers_probe(dev, pci_drv, pci_dev, user_classes); + if (ret) + goto class_err; + return 0; +class_err: + if (new_device) + dev_release(dev); + return ret; +} + +/** + * DPDK callback to remove one or more drivers for a PCI device. + * + * This function removes all drivers probed for a given PCI device. + * + * @param[in] pci_dev + * Pointer to the PCI device. + * + * @return + * 0 on success, the function cannot fail. + */ +static int +mlx5_common_pci_remove(struct rte_pci_device *pci_dev) +{ + struct mlx5_pci_device *dev; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + /* Matching device found, cleanup and unload drivers. */ + ret = drivers_remove(dev, dev->classes_loaded); + if (!ret) + dev_release(dev); + return ret; +} + +static int +mlx5_common_pci_dma_map(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct mlx5_pci_driver *driver = NULL; + struct mlx5_pci_driver *temp; + struct mlx5_pci_device *dev; + int ret = -EINVAL; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + TAILQ_FOREACH(driver, &drv_list, next) { + if (device_class_enabled(dev, driver->driver_class) && + driver->pci_driver.dma_map) { + ret = driver->pci_driver.dma_map(pci_dev, addr, + iova, len); + if (ret) + goto map_err; + } + } + return ret; +map_err: + TAILQ_FOREACH(temp, &drv_list, next) { + if (temp == driver) + break; + if (device_class_enabled(dev, temp->driver_class) && + temp->pci_driver.dma_map && temp->pci_driver.dma_unmap) + temp->pci_driver.dma_unmap(pci_dev, addr, iova, len); + } + return ret; +} + +static int +mlx5_common_pci_dma_unmap(struct rte_pci_device *pci_dev, void *addr, + uint64_t iova, size_t len) +{ + struct mlx5_pci_driver *driver; + struct mlx5_pci_device *dev; + int local_ret = -EINVAL; + int ret; + + dev = pci_to_mlx5_device(pci_dev); + if (!dev) + return -ENODEV; + ret = 0; + /* There is no unmap error recovery in current implementation. */ + TAILQ_FOREACH_REVERSE(driver, &drv_list, mlx5_pci_bus_drv_head, next) { + if (device_class_enabled(dev, driver->driver_class) && + driver->pci_driver.dma_unmap) { + local_ret = driver->pci_driver.dma_unmap(pci_dev, addr, + iova, len); + if (local_ret && (ret == 0)) + ret = local_ret; + } + } + if (local_ret) + ret = local_ret; + return ret; +} + +/* PCI ID table is build dynamically based on registered mlx5 drivers. */ +static struct rte_pci_id *mlx5_pci_id_table; + +static struct rte_pci_driver mlx5_pci_driver = { + .driver = { + .name = "mlx5_pci", + }, + .probe = mlx5_common_pci_probe, + .remove = mlx5_common_pci_remove, + .dma_map = mlx5_common_pci_dma_map, + .dma_unmap = mlx5_common_pci_dma_unmap, +}; + +static int +pci_id_table_size_get(const struct rte_pci_id *id_table) +{ + int table_size = 0; + + for (; id_table->vendor_id != 0; id_table++) + table_size++; + return table_size; +} + +static bool +pci_id_exists(const struct rte_pci_id *id, const struct rte_pci_id *table, + int next_idx) +{ + int current_size = next_idx - 1; + int i; + + for (i = 0; i < current_size; i++) { + if (id->device_id == table[i].device_id && + id->vendor_id == table[i].vendor_id && + id->subsystem_vendor_id == table[i].subsystem_vendor_id && + id->subsystem_device_id == table[i].subsystem_device_id) + return true; + } + return false; +} + +static void +pci_id_insert(struct rte_pci_id *new_table, int *next_idx, + const struct rte_pci_id *id_table) +{ + /* Traverse the id_table, check if entry exists in new_table; + * Add non duplicate entries to new table. + */ + for (; id_table->vendor_id != 0; id_table++) { + if (!pci_id_exists(id_table, new_table, *next_idx)) { + /* New entry; add to the table. */ + new_table[*next_idx] = *id_table; + (*next_idx)++; + } + } +} + +static int +pci_ids_table_update(const struct rte_pci_id *driver_id_table) +{ + const struct rte_pci_id *id_iter; + struct rte_pci_id *updated_table; + struct rte_pci_id *old_table; + int num_ids = 0; + int i = 0; + + old_table = mlx5_pci_id_table; + if (old_table) + num_ids = pci_id_table_size_get(old_table); + num_ids += pci_id_table_size_get(driver_id_table); + /* Increase size by one for the termination entry of vendor_id = 0. */ + num_ids += 1; + updated_table = calloc(num_ids, sizeof(*updated_table)); + if (!updated_table) + return -ENOMEM; + if (TAILQ_EMPTY(&drv_list)) { + /* Copy the first driver's ID table. */ + for (id_iter = driver_id_table; id_iter->vendor_id != 0; + id_iter++, i++) + updated_table[i] = *id_iter; + } else { + /* First copy existing table entries. */ + for (id_iter = old_table; id_iter->vendor_id != 0; + id_iter++, i++) + updated_table[i] = *id_iter; + /* New id to be added at the end of current ID table. */ + pci_id_insert(updated_table, &i, driver_id_table); + } + /* Terminate table with empty entry. */ + updated_table[i].vendor_id = 0; + mlx5_pci_driver.id_table = updated_table; + mlx5_pci_id_table = updated_table; + if (old_table) + free(old_table); + return 0; +} + +void +mlx5_pci_driver_register(struct mlx5_pci_driver *driver) +{ + int ret; + + ret = pci_ids_table_update(driver->pci_driver.id_table); + if (ret) + return; + mlx5_pci_driver.drv_flags |= driver->pci_driver.drv_flags; + TAILQ_INSERT_TAIL(&drv_list, driver, next); +} + +void mlx5_common_pci_init(void) +{ + const struct rte_pci_id empty_table[] = { + { + .vendor_id = 0 + }, + }; + + /* All mlx5 PMDs constructor runs at same priority. So any of the PMD + * including this one can register the PCI table first. If any other + * PMD(s) have registered the PCI ID table, No need to register an empty + * default one. + */ + if (mlx5_pci_id_table == NULL && pci_ids_table_update(empty_table)) + return; + rte_pci_register(&mlx5_pci_driver); +} + +RTE_FINI(mlx5_common_pci_finish) +{ + if (mlx5_pci_id_table != NULL) { + /* Constructor doesn't register with PCI bus if it failed + * to build the table. + */ + rte_pci_unregister(&mlx5_pci_driver); + free(mlx5_pci_id_table); + } +} +RTE_PMD_EXPORT_NAME(mlx5_common_pci, __COUNTER__); diff --git a/drivers/common/mlx5/mlx5_common_pci.h b/drivers/common/mlx5/mlx5_common_pci.h new file mode 100644 index 000000000..41b73e17a --- /dev/null +++ b/drivers/common/mlx5/mlx5_common_pci.h @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2020 Mellanox Technologies, Ltd + */ + +#ifndef _MLX5_COMMON_PCI_H_ +#define _MLX5_COMMON_PCI_H_ + +/** + * @file + * + * RTE Mellanox PCI Driver Interface + * Mellanox ConnectX PCI device supports multiple class (net/vdpa/regex) + * devices. This layer enables creating such multiple class of devices on a + * single PCI device by allowing to bind multiple class specific device + * driver to attach to mlx5_pci driver. + * + * ----------- ------------ ------------- + * | mlx5 | | mlx5 | | mlx5 | + * | net pmd | | vdpa pmd | | regex pmd | + * ----------- ------------ ------------- + * \ | / + * \ | / + * \ -------------- / + * \______| mlx5 |_____ / + * | pci common | + * -------------- + * | + * ----------- + * | mlx5 | + * | pci dev | + * ----------- + * + * - mlx5 pci driver binds to mlx5 PCI devices defined by PCI + * ID table of all related mlx5 PCI devices. + * - mlx5 class driver such as net, vdpa, regex PMD defines its + * specific PCI ID table and mlx5 bus driver probes matching + * class drivers. + * - mlx5 pci bus driver is cental place that validates supported + * class combinations. + */ + +#ifdef __cplusplus +extern "C" { +#endif /* __cplusplus */ + +#include <rte_pci.h> +#include <rte_bus_pci.h> + +#include <mlx5_common.h> + +void mlx5_common_pci_init(void); + +/** + * A structure describing a mlx5 pci driver. + */ +struct mlx5_pci_driver { + struct rte_pci_driver pci_driver; /**< Inherit core pci driver. */ + uint32_t driver_class; /**< Class of this driver, enum mlx5_class */ + TAILQ_ENTRY(mlx5_pci_driver) next; +}; + +/** + * Register a mlx5_pci device driver. + * + * @param driver + * A pointer to a mlx5_pci_driver structure describing the driver + * to be registered. + */ +__rte_internal +void +mlx5_pci_driver_register(struct mlx5_pci_driver *driver); + +#ifdef __cplusplus +} +#endif /* __cplusplus */ + +#endif /* _MLX5_COMMON_PCI_H_ */ diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 65f25252a..73cf72548 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -91,5 +91,7 @@ INTERNAL { mlx5_malloc; mlx5_realloc; mlx5_free; + + mlx5_pci_driver_register; }; diff --git a/drivers/meson.build b/drivers/meson.build index e6d0409aa..95df7ef1d 100644 --- a/drivers/meson.build +++ b/drivers/meson.build @@ -5,6 +5,7 @@ subdirs = [ 'common', 'bus', + 'common/mlx5', # depends on bus. 'mempool', # depends on common and bus. 'net', # depends on common, bus, mempool 'raw', # depends on common, bus and net. -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 09/10] common/mlx5: register class drivers through common layer 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (7 preceding siblings ...) 2020-07-24 14:24 ` [dpdk-dev] [PATCH v9 08/10] common/mlx5: introduce layer to support multiple class drivers Parav Pandit @ 2020-07-24 14:24 ` Parav Pandit 2020-07-24 14:24 ` [dpdk-dev] [PATCH v9 10/10] common/mlx5: remove class check from class drivers Parav Pandit 2020-07-24 14:37 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:24 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Migrate mlx5 net, vdpa and regex PMD to start using mlx5 common class driver. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- Changelog: v8->v9: - Removed leftover inclusion of mlx5_pci bus include directory --- drivers/common/mlx5/mlx5_common_pci.c | 6 ++---- drivers/net/mlx5/Makefile | 1 - drivers/net/mlx5/linux/mlx5_os.c | 1 - drivers/net/mlx5/mlx5.c | 24 ++++++++++++++---------- drivers/net/mlx5/mlx5.h | 1 - drivers/regex/mlx5/meson.build | 2 +- drivers/regex/mlx5/mlx5_regex.c | 21 ++++++++++++--------- drivers/vdpa/mlx5/Makefile | 2 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++---------- 10 files changed, 44 insertions(+), 39 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common_pci.c b/drivers/common/mlx5/mlx5_common_pci.c index adb3f707d..b5fcc16b1 100644 --- a/drivers/common/mlx5/mlx5_common_pci.c +++ b/drivers/common/mlx5/mlx5_common_pci.c @@ -34,10 +34,8 @@ static const unsigned int mlx5_class_combinations[] = { MLX5_CLASS_NET, MLX5_CLASS_VDPA, MLX5_CLASS_REGEX, - /* New class combination should be added here. - * For example a new multi class device combination - * can be MLX5_CLASS_FOO | MLX5_CLASS_BAR. - */ + MLX5_CLASS_NET | MLX5_CLASS_REGEX, + /* New class combination should be added here. */ }; static int diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile index 253faf909..568c77241 100644 --- a/drivers/net/mlx5/Makefile +++ b/drivers/net/mlx5/Makefile @@ -55,7 +55,6 @@ LDLIBS += -lrte_common_mlx5 LDLIBS += -lm LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -LDLIBS += -lrte_bus_pci # A few warnings cannot be avoided in external headers. CFLAGS += -Wno-error=cast-qual diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index fa3b02787..97d7b0e7b 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1537,7 +1537,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, strerror(rte_errno)); return -rte_errno; } - MLX5_ASSERT(pci_drv == &mlx5_driver); errno = 0; ibv_list = mlx5_glue->get_device_list(&ret); if (!ibv_list) { diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 037703d2e..1e4c695f8 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -30,6 +30,7 @@ #include <mlx5_common.h> #include <mlx5_common_os.h> #include <mlx5_common_mp.h> +#include <mlx5_common_pci.h> #include <mlx5_malloc.h> #include "mlx5_defs.h" @@ -2091,16 +2092,19 @@ static const struct rte_pci_id mlx5_pci_id_map[] = { } }; -struct rte_pci_driver mlx5_driver = { - .driver = { - .name = MLX5_DRIVER_NAME +static struct mlx5_pci_driver mlx5_driver = { + .driver_class = MLX5_CLASS_NET, + .pci_driver = { + .driver = { + .name = MLX5_DRIVER_NAME, + }, + .id_table = mlx5_pci_id_map, + .probe = mlx5_os_pci_probe, + .remove = mlx5_pci_remove, + .dma_map = mlx5_dma_map, + .dma_unmap = mlx5_dma_unmap, + .drv_flags = PCI_DRV_FLAGS, }, - .id_table = mlx5_pci_id_map, - .probe = mlx5_os_pci_probe, - .remove = mlx5_pci_remove, - .dma_map = mlx5_dma_map, - .dma_unmap = mlx5_dma_unmap, - .drv_flags = PCI_DRV_FLAGS, }; /* Initialize driver log type. */ @@ -2117,7 +2121,7 @@ RTE_INIT(rte_mlx5_pmd_init) mlx5_set_cksum_table(); mlx5_set_swp_types_table(); if (mlx5_glue) - rte_pci_register(&mlx5_driver); + mlx5_pci_driver_register(&mlx5_driver); } RTE_PMD_EXPORT_NAME(net_mlx5, __COUNTER__); diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index a92194d2d..78d6eb728 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -119,7 +119,6 @@ struct mlx5_local_data { }; extern struct mlx5_shared_data *mlx5_shared_data; -extern struct rte_pci_driver mlx5_driver; /* Dev ops structs */ extern const struct eth_dev_ops mlx5_os_dev_ops; diff --git a/drivers/regex/mlx5/meson.build b/drivers/regex/mlx5/meson.build index 7f800f2e3..d7cb2c572 100644 --- a/drivers/regex/mlx5/meson.build +++ b/drivers/regex/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_regex' -deps += ['common_mlx5', 'bus_pci', 'eal', 'regexdev'] +deps += ['common_mlx5', 'eal', 'regexdev'] sources = files( 'mlx5_regex.c', 'mlx5_rxp.c', diff --git a/drivers/regex/mlx5/mlx5_regex.c b/drivers/regex/mlx5/mlx5_regex.c index 4e0367052..ae9f00189 100644 --- a/drivers/regex/mlx5/mlx5_regex.c +++ b/drivers/regex/mlx5/mlx5_regex.c @@ -5,12 +5,12 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> #include <rte_pci.h> #include <rte_regexdev.h> #include <rte_regexdev_core.h> #include <rte_regexdev_driver.h> +#include <mlx5_common_pci.h> #include <mlx5_glue.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> @@ -246,21 +246,24 @@ static const struct rte_pci_id mlx5_regex_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_regex_driver = { - .driver = { - .name = "mlx5_regex", +static struct mlx5_pci_driver mlx5_regex_driver = { + .driver_class = MLX5_CLASS_REGEX, + .pci_driver = { + .driver = { + .name = "mlx5_regex", + }, + .id_table = mlx5_regex_pci_id_map, + .probe = mlx5_regex_pci_probe, + .remove = mlx5_regex_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_regex_pci_id_map, - .probe = mlx5_regex_pci_probe, - .remove = mlx5_regex_pci_remove, - .drv_flags = 0, }; RTE_INIT(rte_mlx5_regex_init) { mlx5_common_init(); if (mlx5_glue) - rte_pci_register(&mlx5_regex_driver); + mlx5_pci_driver_register(&mlx5_regex_driver); } RTE_LOG_REGISTER(mlx5_regex_logtype, pmd.regex.mlx5, NOTICE) diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile index 8a1c2eab5..d46699fbc 100644 --- a/drivers/vdpa/mlx5/Makefile +++ b/drivers/vdpa/mlx5/Makefile @@ -30,7 +30,7 @@ CFLAGS += -D_XOPEN_SOURCE=600 CFLAGS += $(WERROR_FLAGS) CFLAGS += -Wno-strict-prototypes LDLIBS += -lrte_common_mlx5 -LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_bus_pci -lrte_sched +LDLIBS += -lrte_eal -lrte_vhost -lrte_kvargs -lrte_pci -lrte_sched LDLIBS += -pthread # A few warnings cannot be avoided in external headers. diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.build index 2963aad71..254a8ec14 100644 --- a/drivers/vdpa/mlx5/meson.build +++ b/drivers/vdpa/mlx5/meson.build @@ -8,7 +8,7 @@ if not is_linux endif fmt_name = 'mlx5_vdpa' -deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'bus_pci', 'eal', 'sched'] +deps += ['hash', 'common_mlx5', 'vhost', 'pci', 'eal', 'sched'] sources = files( 'mlx5_vdpa.c', 'mlx5_vdpa_mem.c', diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index 85dbcf956..ffe2f00ac 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -11,12 +11,12 @@ #include <rte_malloc.h> #include <rte_log.h> #include <rte_errno.h> -#include <rte_bus_pci.h> #include <rte_pci.h> #include <rte_string_fns.h> #include <mlx5_glue.h> #include <mlx5_common.h> +#include <mlx5_common_pci.h> #include <mlx5_devx_cmds.h> #include <mlx5_prm.h> #include <mlx5_nl.h> @@ -657,7 +657,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) } /** - * DPDK callback to register a PCI device. + * DPDK callback to register a mlx5 PCI device. * * This function spawns vdpa device out of a given PCI device. * @@ -829,14 +829,17 @@ static const struct rte_pci_id mlx5_vdpa_pci_id_map[] = { } }; -static struct rte_pci_driver mlx5_vdpa_driver = { - .driver = { - .name = "mlx5_vdpa", +static struct mlx5_pci_driver mlx5_vdpa_driver = { + .driver_class = MLX5_CLASS_VDPA, + .pci_driver = { + .driver = { + .name = "mlx5_vdpa", + }, + .id_table = mlx5_vdpa_pci_id_map, + .probe = mlx5_vdpa_pci_probe, + .remove = mlx5_vdpa_pci_remove, + .drv_flags = 0, }, - .id_table = mlx5_vdpa_pci_id_map, - .probe = mlx5_vdpa_pci_probe, - .remove = mlx5_vdpa_pci_remove, - .drv_flags = 0, }; RTE_LOG_REGISTER(mlx5_vdpa_logtype, pmd.vdpa.mlx5, NOTICE) @@ -848,7 +851,7 @@ RTE_INIT(rte_mlx5_vdpa_init) { mlx5_common_init(); if (mlx5_glue) - rte_pci_register(&mlx5_vdpa_driver); + mlx5_pci_driver_register(&mlx5_vdpa_driver); } RTE_PMD_EXPORT_NAME(net_mlx5_vdpa, __COUNTER__); -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v9 10/10] common/mlx5: remove class check from class drivers 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (8 preceding siblings ...) 2020-07-24 14:24 ` [dpdk-dev] [PATCH v9 09/10] common/mlx5: register class drivers through common layer Parav Pandit @ 2020-07-24 14:24 ` Parav Pandit 2020-07-24 14:37 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:24 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit Now that mlx5_pci PMD checks for enabled classes and performs probe(), remove() of associated classes, individual class driver does not need to check if other driver is enabled. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> --- drivers/common/mlx5/mlx5_common.c | 37 ------------------- drivers/common/mlx5/mlx5_common.h | 2 - .../common/mlx5/rte_common_mlx5_version.map | 3 +- drivers/net/mlx5/linux/mlx5_os.c | 5 --- drivers/vdpa/mlx5/mlx5_vdpa.c | 5 --- 5 files changed, 1 insertion(+), 51 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common.c b/drivers/common/mlx5/mlx5_common.c index fd818ef24..06f0a6400 100644 --- a/drivers/common/mlx5/mlx5_common.c +++ b/drivers/common/mlx5/mlx5_common.c @@ -20,43 +20,6 @@ int mlx5_common_logtype; uint8_t haswell_broadwell_cpu; -static int -mlx5_class_check_handler(__rte_unused const char *key, const char *value, - void *opaque) -{ - enum mlx5_class *ret = opaque; - - if (strcmp(value, "vdpa") == 0) { - *ret = MLX5_CLASS_VDPA; - } else if (strcmp(value, "net") == 0) { - *ret = MLX5_CLASS_NET; - } else { - DRV_LOG(ERR, "Invalid mlx5 class %s. Maybe typo in device" - " class argument setting?", value); - *ret = MLX5_CLASS_INVALID; - } - return 0; -} - -enum mlx5_class -mlx5_class_get(struct rte_devargs *devargs) -{ - struct rte_kvargs *kvlist; - const char *key = MLX5_CLASS_ARG_NAME; - enum mlx5_class ret = MLX5_CLASS_NET; - - if (devargs == NULL) - return ret; - kvlist = rte_kvargs_parse(devargs->args, NULL); - if (kvlist == NULL) - return ret; - if (rte_kvargs_count(kvlist, key)) - rte_kvargs_process(kvlist, key, mlx5_class_check_handler, &ret); - rte_kvargs_free(kvlist); - return ret; -} - - /* In case this is an x86_64 intel processor to check if * we should use relaxed ordering. */ diff --git a/drivers/common/mlx5/mlx5_common.h b/drivers/common/mlx5/mlx5_common.h index 864e82ff0..2cdb226f3 100644 --- a/drivers/common/mlx5/mlx5_common.h +++ b/drivers/common/mlx5/mlx5_common.h @@ -247,8 +247,6 @@ struct mlx5_klm { LIST_HEAD(mlx5_dbr_page_list, mlx5_devx_dbr_page); -__rte_internal -enum mlx5_class mlx5_class_get(struct rte_devargs *devargs); __rte_internal void mlx5_translate_port_name(const char *port_name_in, struct mlx5_switch_info *port_info_out); diff --git a/drivers/common/mlx5/rte_common_mlx5_version.map b/drivers/common/mlx5/rte_common_mlx5_version.map index 73cf72548..7729d0dd0 100644 --- a/drivers/common/mlx5/rte_common_mlx5_version.map +++ b/drivers/common/mlx5/rte_common_mlx5_version.map @@ -1,9 +1,8 @@ INTERNAL { global: - mlx5_class_get; - mlx5_common_init; + mlx5_common_verbs_reg_mr; mlx5_common_verbs_dereg_mr; diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 97d7b0e7b..fc339affa 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1524,11 +1524,6 @@ mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, unsigned int dev_config_vf; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_NET) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } if (rte_eal_process_type() == RTE_PROC_PRIMARY) mlx5_pmd_socket_init(); ret = mlx5_init_once(); diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index ffe2f00ac..c0b87bcc0 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -680,11 +680,6 @@ mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, struct mlx5_hca_attr attr; int ret; - if (mlx5_class_get(pci_dev->device.devargs) != MLX5_CLASS_VDPA) { - DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" - " driver."); - return 1; - } ibv = mlx5_vdpa_get_ib_device_match(&pci_dev->addr); if (!ibv) { DRV_LOG(ERR, "No matching IB device for PCI slot " -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit ` (9 preceding siblings ...) 2020-07-24 14:24 ` [dpdk-dev] [PATCH v9 10/10] common/mlx5: remove class check from class drivers Parav Pandit @ 2020-07-24 14:37 ` Parav Pandit 10 siblings, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:37 UTC (permalink / raw) To: dev, grive, ferruh.yigit, Thomas Monjalon Cc: Raslan Darawsheh, Ori Kam, Matan Azrad, joyce.kong > From: Parav Pandit <parav@mellanox.com> > Sent: Friday, July 24, 2020 7:54 PM > > This series introduces mlx5 common driver layer to support multiple class of > devices for a single PCI device. > [..] > > Changelog: > v8->v9: > - Updated commit message > - Fixed LD_FLAGS > - Fixed white space to tab > - Fixed Makefile to have dependency on common/mlx5 for parallel builds Sorry for missing out the checkpatch warnings. I missed to address them. Sending v10. ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v10 00/10] Improve mlx5 PMD driver framework for multiple classes 2020-06-10 17:17 ` [dpdk-dev] [RFC PATCH 1/6] eal: introduce macros for getting value for bit Parav Pandit ` (8 preceding siblings ...) 2020-07-24 14:23 ` [dpdk-dev] [PATCH v9 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit @ 2020-07-24 14:38 ` Parav Pandit 2020-07-24 14:38 ` [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition Parav Pandit ` (9 more replies) 2020-07-26 14:55 ` [dpdk-dev] [PATCH v11 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 2020-07-27 17:47 ` [dpdk-dev] [PATCH v12 00/10] Improve mlx5 PMD driver framework for multiple classes Parav Pandit 11 siblings, 10 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:38 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit This series introduces mlx5 common driver layer to support multiple class of devices for a single PCI device. Motivation and example ---------------------- mlx5 PCI device supports multiple class of devices such as net, vdpa and regex devices. Currently only one pmd (either net or vdpa) can bind to this device. This design limits use of PCI device only for single device class. To support multiple classes simultaneously for a mlx5 PCI device, mlx5 common module is extended as generic PCI PMD. This PMD enables multiple class PMDS (net, regex, vdpa) to be loaded on a single PCI Device. Change description ------------------ Patch-1 Introduces RTE_BIT() macro Patch-2 Fixed indentation for directory listing Patch-3 Relax dependency order Patch-4 Fixes compilation error Patch-5 Fixes segmentation fault in regex probe error unwind path Patch-6 Avoid using class constructor priority Patch-7 Change class values as bits Patch-8 Introduce mlx5 layer to support multiple class drivers Patch-9 Migrates mlx5 net and vdpa driver to use mlx5 PCI driver API instead of rte PCI bus API Patch-10 Removed class check code from class drivers Design overview --------------- ----------- ------------ ------------- | mlx5 | | mlx5 | | mlx5 | | net pmd | | vdpa pmd | | regex pmd | ----------- ------------ ------------- \ | / \ | / \ --------------- / \______| mlx5 common |____/ | pmd | --------------- | ----------- | mlx5 | | pci dev | ----------- - mlx5 common driver binds to mlx5 PCI devices defined by PCI ID table of all related mlx5 PCI devices. - mlx5 class driver such as net, vdpa, regex PMD define their specific PCI ID tables. mlx5 common driver probes each individual class driver(s) for maching and enabled classes. - mlx5 pci layer is cental place that validates supported class combinations. - In future as code evolves, more device setup/cleanup and resource creation code moves to mlx5 PCI common driver. Alternatives considered ----------------------- 1. Instead of creating mlx5 pci common PMD, a common driver is implemented which exposes class registration API. However, common PMD model fits better with existing DPDK design similar to ifpga driver. Class registration API need to create a new callbacks and ID signature; instead it is better to utilize current well defined methods. 2. Enhance pci core to allow multiple driver binding to single rte PCI device. This approach is not taken, because peer drivers using one PCI device won't be aware of other's presence. This requires cross-driver syncronization of who initializes common resources (such as irq, eq and more). This also requires refcounting common objects etc among peer drivers. Instead of layered approach delivers and allows putting common resource sharing, setup code in common driver. It also eliminates peer blind zone problem as bottom pci layer provides necessary setup without any reference counting. 3. In future mlx5 prefers to use RDMA MR cache of the mbuf used between net and regex pmd so that same mbuf use across multiple device can be possible. Examples: -------- A user who wish to use a specific class(es) provides list of classes at command line such as, ./testpmd -w <PCI BDF>,class=net:regex ./testpmd -w <PCI BDF>,class=vdpa Changelog: v9->v10: - Fixed checkpatch warnings related to spelling mistakes and alignment v8->v9: - Updated commit message - Fixed LD_FLAGS - Fixed white space to tab - Fixed Makefile to have dependency on common/mlx5 for parallel builds v7->v8: - Instead of dedicated mlx5 bus driver, merged the PMD to common_mlx5 - Avoided new RTE PRIORITY - Removed mlx5 common to use RTE_PRIORITY_CLASS v6->v7: - Updated release notes v5->v6: - Fixed compilation failure in parallel build for shared lib v4->v5: - Squash the maintainers update path with other patch which adds the bus - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v3->v4: - Fixed dma_map error unwinding flow to follow same order for unmap v2->v3: - Added RTE priority for common driver initialization - Addressed comments from Thomas and Asaf - Fixed compilation error in glue wrapper - Moved pci_driver structure instance as first in driver - Removed white spaces at the end of line in diagram - Address commnts from Matan - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files - Renamed mlx5_valid_class_combo to mlx5_class_combinations - Added cross check for class drivers to support only 3 flags for now - Added full stop at the end of comment block - Using full names in function names - Added new line before function name in multiple functions - Added example string to parse for multiple classes - Dropped mlx5 prefix from static function - Removed empty lines - Fixed issue to remove multiple classes for a driver - Using define for drv_flags at multiple places - Deriving drv_flags based on the class drivers - Fixed alignment for id_table - Perform dma map on best effort basis for all supported drivers - Dynamically build pci id table - Using PCI to mlx5 device helper routines v1->v2: - Addressed most comments from Thomas and Gaetan. - Symbols starting with prefix rte_bus_pci_mlx5 may be confusing as it may appear as it belong to rte_bus_pci module. Hence it is kept as rte_bus_mlx5_pci which matches with other modules as mlx5_vdpa, mlx5_net. - Dropped 2nd patch and replace with new 6th patch. - Avoided new file, added macro to rte_bitops.h - Inheriting ret_pci_driver instead of rte_driver - Added design and description of the mlx5_pci bus - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag - Use anonymous structure for class search and code changes around it - Define static for class comination array - Use RTE_DIM to find array size - Added OOM check for strdup() - Renamed copy variable to nstr_orig - Returning negagive error code - Returning directly if match entry found - Use compat condition check - Avoided cutting error message string - Use uint32_t datatype instead of enum mlx5_class - Changed logic to parse device arguments only once during probe() - Added check to fail driver probe if multiple classes register with DMA ops - Renamed function to parse_class_options - Migreate API from rte_driver to rte_pci_driver Parav Pandit (8): eal: introduce macro for bit definition common/mlx5: fix void parameters in glue wrappers regex/mlx5: fix segmentation fault during error unwinding common/mlx5: avoid using class constructor priority common/mlx5: change class values as bits common/mlx5: introduce layer to support multiple class drivers common/mlx5: register class drivers through common layer common/mlx5: remove class check from class drivers Thomas Monjalon (2): drivers: fix indent of directory list drivers: relax dependency order drivers/Makefile | 10 +- drivers/common/Makefile | 4 - drivers/common/meson.build | 2 +- drivers/common/mlx5/Makefile | 2 + drivers/common/mlx5/linux/mlx5_glue.h | 4 +- drivers/common/mlx5/meson.build | 9 +- drivers/common/mlx5/mlx5_common.c | 52 +- drivers/common/mlx5/mlx5_common.h | 12 +- drivers/common/mlx5/mlx5_common_pci.c | 539 ++++++++++++++++++ drivers/common/mlx5/mlx5_common_pci.h | 77 +++ .../common/mlx5/rte_common_mlx5_version.map | 4 +- drivers/meson.build | 51 +- drivers/net/mlx5/Makefile | 1 - drivers/net/mlx5/linux/mlx5_os.c | 6 - drivers/net/mlx5/mlx5.c | 25 +- drivers/net/mlx5/mlx5.h | 1 - drivers/regex/mlx5/meson.build | 2 +- drivers/regex/mlx5/mlx5_regex.c | 29 +- drivers/vdpa/mlx5/Makefile | 2 +- drivers/vdpa/mlx5/meson.build | 2 +- drivers/vdpa/mlx5/mlx5_vdpa.c | 29 +- lib/librte_eal/include/rte_bitops.h | 8 + 22 files changed, 748 insertions(+), 123 deletions(-) create mode 100644 drivers/common/mlx5/mlx5_common_pci.c create mode 100644 drivers/common/mlx5/mlx5_common_pci.h -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition 2020-07-24 14:38 ` [dpdk-dev] [PATCH v10 " Parav Pandit @ 2020-07-24 14:38 ` Parav Pandit 2020-07-24 18:31 ` Honnappa Nagarahalli 2020-07-24 14:38 ` [dpdk-dev] [PATCH v10 02/10] drivers: fix indent of directory list Parav Pandit ` (8 subsequent siblings) 9 siblings, 1 reply; 193+ messages in thread From: Parav Pandit @ 2020-07-24 14:38 UTC (permalink / raw) To: dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, joyce.kong, Parav Pandit, Morten Brørup There are several drivers which duplicate bit generation macro. Introduce a generic bit macros so that such drivers avoid redefining same in multiple drivers. Signed-off-by: Parav Pandit <parav@mellanox.com> Acked-by: Matan Azrad <matan@mellanox.com> Acked-by: Morten Brørup <mb@smartsharesystems.com> --- Changelog: v4->v5: - Addressed comments from Morten Brørup - Renamed newly added macro to RTE_BIT64 - Added doxygen comment section for the macro v1->v2: - Addressed comments from Thomas and Gaten. - Avoided new file, added macro to rte_bitops.h --- lib/librte_eal/include/rte_bitops.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/lib/librte_eal/include/rte_bitops.h b/lib/librte_eal/include/rte_bitops.h index 740927f3b..ca46a110f 100644 --- a/lib/librte_eal/include/rte_bitops.h +++ b/lib/librte_eal/include/rte_bitops.h @@ -17,6 +17,14 @@ #include <rte_debug.h> #include <rte_compat.h> +/** + * Get the uint64_t value for a specified bit set. + * + * @param nr + * The bit number in range of 0 to 63. + */ +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) + /*------------------------ 32-bit relaxed operations ------------------------*/ /** -- 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition 2020-07-24 14:38 ` [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition Parav Pandit @ 2020-07-24 18:31 ` Honnappa Nagarahalli 2020-07-27 8:21 ` Morten Brørup 0 siblings, 1 reply; 193+ messages in thread From: Honnappa Nagarahalli @ 2020-07-24 18:31 UTC (permalink / raw) To: Parav Pandit, dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, Joyce Kong, Morten Brørup, nd, Honnappa Nagarahalli, nd <snip> > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition > > There are several drivers which duplicate bit generation macro. > Introduce a generic bit macros so that such drivers avoid redefining same in > multiple drivers. > > Signed-off-by: Parav Pandit <parav@mellanox.com> > Acked-by: Matan Azrad <matan@mellanox.com> > Acked-by: Morten Brørup <mb@smartsharesystems.com> > --- > Changelog: > v4->v5: > - Addressed comments from Morten Brørup > - Renamed newly added macro to RTE_BIT64 > - Added doxygen comment section for the macro > v1->v2: > - Addressed comments from Thomas and Gaten. > - Avoided new file, added macro to rte_bitops.h > --- > lib/librte_eal/include/rte_bitops.h | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/lib/librte_eal/include/rte_bitops.h > b/lib/librte_eal/include/rte_bitops.h > index 740927f3b..ca46a110f 100644 > --- a/lib/librte_eal/include/rte_bitops.h > +++ b/lib/librte_eal/include/rte_bitops.h > @@ -17,6 +17,14 @@ > #include <rte_debug.h> > #include <rte_compat.h> > > +/** > + * Get the uint64_t value for a specified bit set. > + * > + * @param nr > + * The bit number in range of 0 to 63. > + */ > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) In general, the macros have been avoided in this file. Suggest changing this to an inline function. Also, this file has uses of this macro, it would be good to replace them with the new inline function. > + > /*------------------------ 32-bit relaxed operations ------------------------*/ > > /** > -- > 2.26.2 ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition 2020-07-24 18:31 ` Honnappa Nagarahalli @ 2020-07-27 8:21 ` Morten Brørup 2020-07-27 17:01 ` Parav Pandit 2020-07-28 2:18 ` Honnappa Nagarahalli 0 siblings, 2 replies; 193+ messages in thread From: Morten Brørup @ 2020-07-27 8:21 UTC (permalink / raw) To: Honnappa Nagarahalli, Parav Pandit, dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, Joyce Kong, nd, nd > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Honnappa > Nagarahalli > Sent: Friday, July 24, 2020 8:31 PM > > <snip> > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit > definition > > > > There are several drivers which duplicate bit generation macro. > > Introduce a generic bit macros so that such drivers avoid redefining > same in > > multiple drivers. > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > Acked-by: Matan Azrad <matan@mellanox.com> > > Acked-by: Morten Brørup <mb@smartsharesystems.com> > > --- > > Changelog: > > v4->v5: > > - Addressed comments from Morten Brørup > > - Renamed newly added macro to RTE_BIT64 > > - Added doxygen comment section for the macro > > v1->v2: > > - Addressed comments from Thomas and Gaten. > > - Avoided new file, added macro to rte_bitops.h > > --- > > lib/librte_eal/include/rte_bitops.h | 8 ++++++++ > > 1 file changed, 8 insertions(+) > > > > diff --git a/lib/librte_eal/include/rte_bitops.h > > b/lib/librte_eal/include/rte_bitops.h > > index 740927f3b..ca46a110f 100644 > > --- a/lib/librte_eal/include/rte_bitops.h > > +++ b/lib/librte_eal/include/rte_bitops.h > > @@ -17,6 +17,14 @@ > > #include <rte_debug.h> > > #include <rte_compat.h> > > > > +/** > > + * Get the uint64_t value for a specified bit set. > > + * > > + * @param nr > > + * The bit number in range of 0 to 63. > > + */ > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) > In general, the macros have been avoided in this file. Suggest changing > this to an inline function. That has been discussed already, and rejected for good reasons: http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@AM0PR05MB4866.eurprd05.prod.outlook.com/ > Also, this file has uses of this macro, it would be good to replace > them with the new inline function. Makes sense. And for consistency, it would require adding an RTE_BIT32() macro too. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition 2020-07-27 8:21 ` Morten Brørup @ 2020-07-27 17:01 ` Parav Pandit 2020-07-28 2:18 ` Honnappa Nagarahalli 1 sibling, 0 replies; 193+ messages in thread From: Parav Pandit @ 2020-07-27 17:01 UTC (permalink / raw) To: Morten Brørup, Honnappa Nagarahalli, dev, grive, ferruh.yigit, Thomas Monjalon Cc: Raslan Darawsheh, Ori Kam, Matan Azrad, Joyce Kong, nd, nd > From: Morten Brørup <mb@smartsharesystems.com> > Sent: Monday, July 27, 2020 1:52 PM > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Honnappa > > Nagarahalli > > Sent: Friday, July 24, 2020 8:31 PM > > > > <snip> > > > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit > > definition > > > > > > There are several drivers which duplicate bit generation macro. > > > Introduce a generic bit macros so that such drivers avoid redefining > > same in > > > multiple drivers. > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > Acked-by: Matan Azrad <matan@mellanox.com> > > > Acked-by: Morten Brørup <mb@smartsharesystems.com> > > > --- > > > Changelog: > > > v4->v5: > > > - Addressed comments from Morten Brørup > > > - Renamed newly added macro to RTE_BIT64 > > > - Added doxygen comment section for the macro > > > v1->v2: > > > - Addressed comments from Thomas and Gaten. > > > - Avoided new file, added macro to rte_bitops.h > > > --- > > > lib/librte_eal/include/rte_bitops.h | 8 ++++++++ > > > 1 file changed, 8 insertions(+) > > > > > > diff --git a/lib/librte_eal/include/rte_bitops.h > > > b/lib/librte_eal/include/rte_bitops.h > > > index 740927f3b..ca46a110f 100644 > > > --- a/lib/librte_eal/include/rte_bitops.h > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > @@ -17,6 +17,14 @@ > > > #include <rte_debug.h> > > > #include <rte_compat.h> > > > > > > +/** > > > + * Get the uint64_t value for a specified bit set. > > > + * > > > + * @param nr > > > + * The bit number in range of 0 to 63. > > > + */ > > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) > > In general, the macros have been avoided in this file. Suggest > > changing this to an inline function. > > That has been discussed already, and rejected for good reasons: > https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Finbox. > dpdk.org%2Fdev%2FAM0PR05MB4866823B0170B90F679A2765D1640%40AM > 0PR05MB4866.eurprd05.prod.outlook.com%2F&data=02%7C01%7Cpara > v%40mellanox.com%7Cdb43e7a083ae44a063a208d8320619b6%7Ca652971c7d > 2e4d9ba6a4d149256f461b%7C0%7C0%7C637314349075254981&sdata=k > R3fSOickIY2HbpmodNlB3ERF%2F2Qc7th55SGF40xmB0%3D&reserved=0 > > > Also, this file has uses of this macro, it would be good to replace > > them with the new inline function. > > Makes sense. > And for consistency, it would require adding an RTE_BIT32() macro too. Ok. Sending v12 addressing it. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition 2020-07-27 8:21 ` Morten Brørup 2020-07-27 17:01 ` Parav Pandit @ 2020-07-28 2:18 ` Honnappa Nagarahalli 2020-07-28 8:24 ` Morten Brørup 1 sibling, 1 reply; 193+ messages in thread From: Honnappa Nagarahalli @ 2020-07-28 2:18 UTC (permalink / raw) To: Morten Brørup, Parav Pandit, dev, grive, ferruh.yigit, thomas Cc: rasland, orika, matan, Joyce Kong, nd, Honnappa Nagarahalli, nd <snip> > > > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit > > definition > > > > > > There are several drivers which duplicate bit generation macro. > > > Introduce a generic bit macros so that such drivers avoid redefining > > same in > > > multiple drivers. > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > Acked-by: Matan Azrad <matan@mellanox.com> > > > Acked-by: Morten Brørup <mb@smartsharesystems.com> > > > --- > > > Changelog: > > > v4->v5: > > > - Addressed comments from Morten Brørup > > > - Renamed newly added macro to RTE_BIT64 > > > - Added doxygen comment section for the macro > > > v1->v2: > > > - Addressed comments from Thomas and Gaten. > > > - Avoided new file, added macro to rte_bitops.h > > > --- > > > lib/librte_eal/include/rte_bitops.h | 8 ++++++++ > > > 1 file changed, 8 insertions(+) > > > > > > diff --git a/lib/librte_eal/include/rte_bitops.h > > > b/lib/librte_eal/include/rte_bitops.h > > > index 740927f3b..ca46a110f 100644 > > > --- a/lib/librte_eal/include/rte_bitops.h > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > @@ -17,6 +17,14 @@ > > > #include <rte_debug.h> > > > #include <rte_compat.h> > > > > > > +/** > > > + * Get the uint64_t value for a specified bit set. > > > + * > > > + * @param nr > > > + * The bit number in range of 0 to 63. > > > + */ > > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) > > In general, the macros have been avoided in this file. Suggest > > changing this to an inline function. > > That has been discussed already, and rejected for good reasons: > http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@ > AM0PR05MB4866.eurprd05.prod.outlook.com/ Thank you for the link. In this patch series, I see the macro being used in enum initialization (7/10 in v11) as well as in functions (8/10 in v11). Does it make sense to introduce use inline functions and use the inline functions for 8/10? If we do this, we should document in rte_bitops.h that inline functions should be used wherever possible. > > > Also, this file has uses of this macro, it would be good to replace > > them with the new inline function. > > Makes sense. > And for consistency, it would require adding an RTE_BIT32() macro too. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition 2020-07-28 2:18 ` Honnappa Nagarahalli @ 2020-07-28 8:24 ` Morten Brørup 2020-07-28 9:29 ` Gaëtan Rivet 0 siblings, 1 reply; 193+ messages in thread From: Morten Brørup @ 2020-07-28 8:24 UTC (permalink / raw) To: Honnappa Nagarahalli, Parav Pandit, dev, grive, ferruh.yigit, thomas, Ray Kinsella, Neil Horman Cc: rasland, orika, matan, Joyce Kong, nd, nd + Ray and Neil as ABI Policy maintainers. > From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com] > Sent: Tuesday, July 28, 2020 4:19 AM > > <snip> > > > > > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for > bit > > > definition > > > > > > > > There are several drivers which duplicate bit generation macro. > > > > Introduce a generic bit macros so that such drivers avoid > redefining > > > same in > > > > multiple drivers. > > > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > > Acked-by: Matan Azrad <matan@mellanox.com> > > > > Acked-by: Morten Brørup <mb@smartsharesystems.com> > > > > --- > > > > Changelog: > > > > v4->v5: > > > > - Addressed comments from Morten Brørup > > > > - Renamed newly added macro to RTE_BIT64 > > > > - Added doxygen comment section for the macro > > > > v1->v2: > > > > - Addressed comments from Thomas and Gaten. > > > > - Avoided new file, added macro to rte_bitops.h > > > > --- > > > > lib/librte_eal/include/rte_bitops.h | 8 ++++++++ > > > > 1 file changed, 8 insertions(+) > > > > > > > > diff --git a/lib/librte_eal/include/rte_bitops.h > > > > b/lib/librte_eal/include/rte_bitops.h > > > > index 740927f3b..ca46a110f 100644 > > > > --- a/lib/librte_eal/include/rte_bitops.h > > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > > @@ -17,6 +17,14 @@ > > > > #include <rte_debug.h> > > > > #include <rte_compat.h> > > > > > > > > +/** > > > > + * Get the uint64_t value for a specified bit set. > > > > + * > > > > + * @param nr > > > > + * The bit number in range of 0 to 63. > > > > + */ > > > > +#define RTE_BIT64(nr) (UINT64_C(1) << (nr)) > > > In general, the macros have been avoided in this file. Suggest > > > changing this to an inline function. > > > > That has been discussed already, and rejected for good reasons: > > http://inbox.dpdk.org/dev/AM0PR05MB4866823B0170B90F679A2765D1640@ > > AM0PR05MB4866.eurprd05.prod.outlook.com/ > Thank you for the link. > In this patch series, I see the macro being used in enum initialization > (7/10 in v11) as well as in functions (8/10 in v11). Does it make sense > to introduce use inline functions and use the inline functions for > 8/10? > If we do this, we should document in rte_bitops.h that inline functions > should be used wherever possible. I would agree, but only in theory. I disagree in reality, and argue that there should only be macros for this. Here is why: rte_byteorder.h has both RTE_BEnn() macros and rte_cpu_to_be_nn() functions, for doing the same thing at compile time or at run time. There are no compile time warnings if the wrong one is being used, so I am certain that we can find code that uses the macro where the function should be used, or vice versa. Which opens another, higher level, question: Would it be possible to add a compile time check macro in rte_common.h for these and similar? Furthermore: For the RTE_BITnn() operations in this patch set, I expect the compiler to generate perfectly efficient code using the macro for run time use. I.e. there would be no performance advantage by also implementing the macros as functions for run time use. > > > > > Also, this file has uses of this macro, it would be good to replace > > > them with the new inline function. > > > > Makes sense. > > And for consistency, it would require adding an RTE_BIT32() macro > too. ^ permalink raw reply [flat|nested] 193+ messages in thread
* Re: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for bit definition 2020-07-28 8:24 ` Morten Brørup @ 2020-07-28 9:29 ` Gaëtan Rivet 2020-07-28 11:11 ` Morten Brørup 2020-07-28 15:39 ` Honnappa Nagarahalli 0 siblings, 2 replies; 193+ messages in thread From: Gaëtan Rivet @ 2020-07-28 9:29 UTC (permalink / raw) To: Morten Brørup Cc: Honnappa Nagarahalli, Parav Pandit, dev, ferruh.yigit, thomas, Ray Kinsella, Neil Horman, rasland, orika, matan, Joyce Kong, nd On 28/07/20 10:24 +0200, Morten Brørup wrote: > + Ray and Neil as ABI Policy maintainers. > > > From: Honnappa Nagarahalli [mailto:Honnappa.Nagarahalli@arm.com] > > Sent: Tuesday, July 28, 2020 4:19 AM > > > > <snip> > > > > > > > > > Subject: [dpdk-dev] [PATCH v10 01/10] eal: introduce macro for > > bit > > > > definition > > > > > > > > > > There are several drivers which duplicate bit generation macro. > > > > > Introduce a generic bit macros so that such drivers avoid > > redefining > > > > same in > > > > > multiple drivers. > > > > > > > > > > Signed-off-by: Parav Pandit <parav@mellanox.com> > > > > > Acked-by: Matan Azrad <matan@mellanox.com> > > > > > Acked-by: Morten Brørup <mb@smartsharesystems.com> > > > > > --- > > > > > Changelog: > > > > > v4->v5: > > > > > - Addressed comments from Morten Brørup > > > > > - Renamed newly added macro to RTE_BIT64 > > > > > - Added doxygen comment section for the macro > > > > > v1->v2: > > > > > - Addressed comments from Thomas and Gaten. > > > > > - Avoided new file, added macro to rte_bitops.h > > > > > --- > > > > > lib/librte_eal/include/rte_bitops.h | 8 ++++++++ > > > > > 1 file changed, 8 insertions(+) > > > > > > > > > > diff --git a/lib/librte_eal/include/rte_bitops.h > > > > > b/lib/librte_eal/include/rte_bitops.h > > > > > index 740927f3b..ca46a110f 100644 > > > > > --- a/lib/librte_eal/include/rte_bitops.h > > > > > +++ b/lib/librte_eal/include/rte_bitops.h > > > > > @@ -17,6 +17,14 @@ > > > > > #include <