From: Michael Baum <michaelba@nvidia.com>
To: <dev@dpdk.org>
Cc: Matan Azrad <matan@nvidia.com>,
Raslan Darawsheh <rasland@nvidia.com>,
Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Subject: [dpdk-dev] [RFC 00/21] mlx5: sharing global MR cache between drivers
Date: Tue, 17 Aug 2021 16:44:20 +0300 [thread overview]
Message-ID: <20210817134441.1966618-1-michaelba@nvidia.com> (raw)
There are 5 classes of mlx5 drivers (net\eth, RegEx, vDPA, compress and
crypto). The various drivers are registered under the common mlx5 driver
and are managed by it.
The common driver probing calls in a loop to the probe function of each
driver registered to it.
Each driver creates for itself all the objects required for
communication with the hardware and a global MR cache that manages
memory mappings.
The management of the caches separately by the different drivers is not
very efficient. In fact, the same memory is mapped multiple times to the
HW when more than 1 class use the device.
This feature will move management to the common driver in two phases.
Phase 1: sharing HW objects between drivers on the same device
The communication with the hardware - for any MR handle - is conducted
by the Protection Domain, so we are motivated to share it between the
drivers. However, to create it, we need to give the context of the
device, so the context must also be shared between the drivers.
At this point, we will share the next trio between the drivers (CTX, PD,
pdn) to create an infrastructure that will allow sharing of dependent
objects, particularly the global MR cache.
The common driver itself will create this trio individually for all
drivers before calling their probe function. Then, as a parameter to the
probe function, it will give them a pointer to the structure containing
the trio.
Phase 2: sharing global MR cache between drivers on the same device
The common driver will add to the structure containing the trio and the
structure that manages the global MR cache and keep a list of such
structures for memory management. In each driver, each queue will manage
its own local MR cache. If the queue does not find its cache, it will
search the global MR cache shared by all. Caching access will be through
the pointer that the driver received as a parameter in probing.
Michael Baum (21):
net/mlx5: fix shared device context creation error flow
net/mlx5: fix PCI probing error flow
common/mlx5: add context device structure
compress/mlx5: use context device structure
crypto/mlx5: use context device structure
regex/mlx5: use context device structure
net/mlx5: improve probe function on Windows
net/mlx5: improve probe function on Linux
net/mlx5: improve spawn function
net/mlx5: use context device structure
net/mlx5: move NUMA node field to context device
common/mlx5: add ROCE disable in context device creation
vdpa/mlx5: use context device structure
mlx5: update device sent to probing
mlx5: share context device structure between drivers
common/mlx5: add HCA attributes to context device structure
regex/mlx5: use HCA attributes from context device
vdpa/mlx5: use HCA attributes from context device
compress/mlx5: use HCA attributes from context device
crypto/mlx5: use HCA attributes from context device
net/mlx5: use HCA attributes from context device
drivers/common/mlx5/linux/mlx5_common_os.c | 268 ++++++++-
drivers/common/mlx5/mlx5_common.c | 273 +++++++++-
drivers/common/mlx5/mlx5_common.h | 35 +-
drivers/common/mlx5/mlx5_common_private.h | 6 -
drivers/common/mlx5/version.map | 2 +
drivers/common/mlx5/windows/mlx5_common_os.c | 207 ++++++-
drivers/compress/mlx5/mlx5_compress.c | 112 +---
drivers/crypto/mlx5/mlx5_crypto.c | 111 +---
drivers/crypto/mlx5/mlx5_crypto.h | 4 +-
drivers/crypto/mlx5/mlx5_crypto_dek.c | 5 +-
drivers/net/mlx5/linux/mlx5_ethdev_os.c | 8 +-
drivers/net/mlx5/linux/mlx5_mp_os.c | 9 +-
drivers/net/mlx5/linux/mlx5_os.c | 543 +++++++++----------
drivers/net/mlx5/linux/mlx5_verbs.c | 55 +-
drivers/net/mlx5/mlx5.c | 85 ++-
drivers/net/mlx5/mlx5.h | 17 +-
drivers/net/mlx5/mlx5_devx.c | 35 +-
drivers/net/mlx5/mlx5_flow.c | 6 +-
drivers/net/mlx5/mlx5_flow_aso.c | 24 +-
drivers/net/mlx5/mlx5_flow_dv.c | 51 +-
drivers/net/mlx5/mlx5_flow_verbs.c | 4 +-
drivers/net/mlx5/mlx5_mr.c | 14 +-
drivers/net/mlx5/mlx5_txpp.c | 27 +-
drivers/net/mlx5/windows/mlx5_ethdev_os.c | 14 +-
drivers/net/mlx5/windows/mlx5_os.c | 285 ++--------
drivers/regex/mlx5/mlx5_regex.c | 74 +--
drivers/regex/mlx5/mlx5_regex.h | 23 +-
drivers/regex/mlx5/mlx5_regex_control.c | 12 +-
drivers/regex/mlx5/mlx5_regex_fastpath.c | 18 +-
drivers/regex/mlx5/mlx5_rxp.c | 64 ++-
drivers/vdpa/mlx5/mlx5_vdpa.c | 210 +------
drivers/vdpa/mlx5/mlx5_vdpa.h | 4 +-
drivers/vdpa/mlx5/mlx5_vdpa_event.c | 19 +-
drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 6 +-
drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 13 +-
drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 10 +-
drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 16 +-
37 files changed, 1414 insertions(+), 1255 deletions(-)
--
2.25.1
next reply other threads:[~2021-08-17 13:45 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-17 13:44 Michael Baum [this message]
2021-08-17 13:44 ` [dpdk-dev] [RFC 01/21] net/mlx5: fix shared device context creation error flow Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 02/21] net/mlx5: fix PCI probing " Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 03/21] common/mlx5: add context device structure Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 04/21] compress/mlx5: use " Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 05/21] crypto/mlx5: " Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 06/21] regex/mlx5: " Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 07/21] net/mlx5: improve probe function on Windows Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 08/21] net/mlx5: improve probe function on Linux Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 09/21] net/mlx5: improve spawn function Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 10/21] net/mlx5: use context device structure Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 11/21] net/mlx5: move NUMA node field to context device Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 12/21] common/mlx5: add ROCE disable in context device creation Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 13/21] vdpa/mlx5: use context device structure Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 14/21] mlx5: update device sent to probing Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 15/21] mlx5: share context device structure between drivers Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 16/21] common/mlx5: add HCA attributes to context device structure Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 17/21] regex/mlx5: use HCA attributes from context device Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 18/21] vdpa/mlx5: " Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 19/21] compress/mlx5: " Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 20/21] crypto/mlx5: " Michael Baum
2021-08-17 13:44 ` [dpdk-dev] [RFC 21/21] net/mlx5: " Michael Baum
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210817134441.1966618-1-michaelba@nvidia.com \
--to=michaelba@nvidia.com \
--cc=dev@dpdk.org \
--cc=matan@nvidia.com \
--cc=rasland@nvidia.com \
--cc=viacheslavo@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).