Re: [dpdk-dev] [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple classes

[Raslan Darawsheh <rasland@mellanox.com>]

Hi,

> -----Original Message-----
> From: Parav Pandit <parav@mellanox.com>
> Sent: Friday, July 17, 2020 4:49 PM
> To: dev@dpdk.org; grive@u256.net; ferruh.yigit@intel.com; Thomas
> Monjalon <thomas@monjalon.net>
> Cc: Raslan Darawsheh <rasland@mellanox.com>; Ori Kam
> <orika@mellanox.com>; Matan Azrad <matan@mellanox.com>;
> joyce.kong@arm.com; Parav Pandit <parav@mellanox.com>
> Subject: [PATCH v7 0/9] Improve mlx5 PMD driver framework for multiple
> classes
> 
> This series introduces mlx5 bus to support multiple class of devices
> for a single PCI device.
> 
> Motivation and example
> ----------------------
> mlx5 PCI device supports multiple class of devices such as net, vdpa
> and regex devices.
> 
> Currently only one pmd (either net or vdpa) can bind to this device.
> This design limits use of PCI device only for single device class.
> 
> To support multiple classes simultaneously for a mlx5 PCI device,
> a new mlx5 PCI bus is created. This bus allows binding multiple
> class drivers (such as net, vdpa, regex(future)) to bind to the
> mlx5 PCI bus driver.
> 
> Change description
> ------------------
> Patch-1 Introduces RTE_BIT() macro
> Patch-2 Introduces new RTE constructor priority for common
> initialization
> Patch-3 Fixes compilation error
> Patch-4 Define mlx5 class as bit fields
> Patch-5 Uses new RTE common priority
> Patch-6 Adds mlx5 PCI bus
> Patch-7 Implements a mlx5 PCI bus driver
> Patch-8 Migrates mlx5 net and vdpa driver to use mlx5 PCI bus
> API instead of rte PCI bus API
> Patch-9 Removed class check code as its already part of the bus now
> 
> Design overview
> ---------------
> 
>  -----------    ------------    -------------
>  |   mlx5  |    |   mlx5   |    |   mlx5    |
>  | net pmd |    | vdpa pmd |    | regex pmd |
>  -----------    ------------    -------------
>       \              |                /
>        \             |               /
>         \       -------------       /
>          \______|   mlx5    |_____ /
>                 |   pci bus |
>                 -------------
>                      |
>                  -----------
>                  |   mlx5  |
>                  | pci dev |
>                  -----------
> 
> - mlx5 pci bus driver binds to mlx5 PCI devices defined by PCI
>   ID table of all related mlx5 PCI devices.
> - mlx5 class driver such as net, vdpa, regex PMD defines its
>   specific PCI ID table and mlx5 bus driver probes matching
>   class drivers.
> - mlx5 pci bus driver is cental place that validates supported
>   class combinations.
> - In future as code evolves, more device setup/cleanup and
>   resource creation code moves to mlx5 PCI bus driver.
> 
> Alternatives considered
> -----------------------
> 1. Instead of creating mlx5 pci bus, a common driver is
> implemented which exposes class registration API.
> However, bus model fits better with existing DPDK design
> similar to ifpga driver.
> Class registration API need to create a new callbacks
> and ID signature; instead it is better to utilize current
> well defined methods.
> 
> 2. Enhance pci core to allow multiple driver binding to
> single rte PCI device.
> This approach is not taken, because peer drivers using
> one PCI device won't be aware of other's presence. This
> requires cross-driver syncronization of who initializes
> common resources (such as irq, eq and more).
> This also requires refcounting common objects etc among
> peer drivers.
> Instead of layered approach delivers and allows putting
> common resource sharing, setup code in common bus driver.
> It also eliminates peer blind zone problem as bottom pci
> bus layer provides necessary setup without any reference
> counting.
> 
> 3. In future mlx5 prefers to use RDMA MR cache of the mbuf
> used between net and regex pmd so that same mbuf use across
> multiple device can be possible.
> 
> Examples:
> --------
> A user who wish to use a specific class(es) provides list
> of classes at command line such as,
> ./testpmd -w <PCI BDF>,class=net:vdpa
> ./testpmd -w <PCI BDF>,class=vdpa
> 
> In future,
> ./testpmd -w <PCI BDF>,class=net:regex
> 
> Changelog:
> v6->v7:
>  - Updated release notes
> v5->v6:
>  - Fixed compilation failure in parallel build for shared lib
> v4->v5:
>  - Squash the maintainers update path with other patch which adds the
>    bus
>  - Addressed comments from Morten Brørup
>  - Renamed newly added macro to RTE_BIT64
>  - Added doxygen comment section for the macro
> v3->v4:
>  - Fixed dma_map error unwinding flow to follow same order for unmap
> v2->v3:
>  - Added RTE priority for common driver initialization
>  - Addressed comments from Thomas and Asaf
>  - Fixed compilation error in glue wrapper
>  - Moved pci_driver structure instance as first in driver
>  - Removed white spaces at the end of line in diagram
>  - Address commnts from Matan
>  - Removed CONFIG_RTE_LIBRTE_MLX5_PCI_BUS from config files
>  - Renamed mlx5_valid_class_combo to mlx5_class_combinations
>  - Added cross check for class drivers to support only 3 flags for now
>  - Added full stop at the end of comment block
>  - Using full names in function names
>  - Added new line before function name in multiple functions
>  - Added example string to parse for multiple classes
>  - Dropped mlx5 prefix from static function
>  - Removed empty lines
>  - Fixed issue to remove multiple classes for a driver
>  - Using define for drv_flags at multiple places
>  - Deriving drv_flags based on the class drivers
>  - Fixed alignment for id_table
>  - Perform dma map on best effort basis for all supported drivers
>  - Dynamically build pci id table
>  - Using PCI to mlx5 device helper routines
> v1->v2:
>  - Addressed most comments from Thomas and Gaetan.
>  - Symbols starting with prefix rte_bus_pci_mlx5 may be
>    confusing as it may appear as it belong to rte_bus_pci module.
>    Hence it is kept as rte_bus_mlx5_pci which matches with other
>    modules as mlx5_vdpa, mlx5_net.
>  - Dropped 2nd patch and replace with new 6th patch.
>  - Avoided new file, added macro to rte_bitops.h
>  - Inheriting ret_pci_driver instead of rte_driver
>  - Added design and description of the mlx5_pci bus
>  - Enhanced driver to honor RTE_PCI_DRV_PROBE_AGAIN drv_flag
>  - Use anonymous structure for class search and code changes around it
>  - Define static for class comination array
>  - Use RTE_DIM to find array size
>  - Added OOM check for strdup()
>  - Renamed copy variable to nstr_orig
>  - Returning negagive error code
>  - Returning directly if match entry found
>  - Use compat condition check
>  - Avoided cutting error message string
>  - Use uint32_t datatype instead of enum mlx5_class
>  - Changed logic to parse device arguments only once during probe()
>  - Added check to fail driver probe if multiple classes register with
>    DMA ops
>  - Renamed function to parse_class_options
>  - Migreate API from rte_driver to rte_pci_driver
> 
> Parav Pandit (9):
>   eal: introduce macros for getting value for bit
>   eal: introduce RTE common initialization level
>   common/mlx5: fix empty input style in glue wrappers
>   common/mlx5: change mlx5 class enum values as bits
>   common/mlx5: use common rte priority
>   bus/mlx5_pci: add mlx5 PCI bus
>   bus/mlx5_pci: register a PCI driver
>   bus/mlx5_pci: enable net and vDPA to use mlx5 PCI bus driver
>   common/mlx5: remove class checks from individual driver
> 
>  MAINTAINERS                                   |   5 +
>  doc/guides/rel_notes/release_20_08.rst        |   5 +
>  drivers/bus/Makefile                          |   4 +
>  drivers/bus/meson.build                       |   2 +-
>  drivers/bus/mlx5_pci/Makefile                 |  41 ++
>  drivers/bus/mlx5_pci/meson.build              |  19 +
>  drivers/bus/mlx5_pci/mlx5_pci_bus.c           | 522 ++++++++++++++++++
>  drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h       |  84 +++
>  .../bus/mlx5_pci/rte_bus_mlx5_pci_version.map |   5 +
>  drivers/common/mlx5/linux/mlx5_glue.h         |   4 +-
>  drivers/common/mlx5/mlx5_common.c             |  39 +-
>  drivers/common/mlx5/mlx5_common.h             |   7 +-
>  .../common/mlx5/rte_common_mlx5_version.map   |   2 -
>  drivers/net/mlx5/Makefile                     |   3 +-
>  drivers/net/mlx5/linux/mlx5_os.c              |   6 -
>  drivers/net/mlx5/meson.build                  |   2 +-
>  drivers/net/mlx5/mlx5.c                       |  26 +-
>  drivers/net/mlx5/mlx5.h                       |   1 -
>  drivers/vdpa/mlx5/Makefile                    |   3 +-
>  drivers/vdpa/mlx5/meson.build                 |   2 +-
>  drivers/vdpa/mlx5/mlx5_vdpa.c                 |  30 +-
>  lib/librte_eal/include/rte_bitops.h           |   8 +
>  lib/librte_eal/include/rte_common.h           |   1 +
>  mk/rte.app.mk                                 |   1 +
>  24 files changed, 737 insertions(+), 85 deletions(-)
>  create mode 100644 drivers/bus/mlx5_pci/Makefile
>  create mode 100644 drivers/bus/mlx5_pci/meson.build
>  create mode 100644 drivers/bus/mlx5_pci/mlx5_pci_bus.c
>  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci.h
>  create mode 100644 drivers/bus/mlx5_pci/rte_bus_mlx5_pci_version.map
> 
> --
> 2.26.2

Series applied to next-net-mlx,

Kindest regards,
Raslan Darawsheh