From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B4A53A0524; Thu, 30 Jan 2020 15:39:00 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 139D51BFFC; Thu, 30 Jan 2020 15:39:00 +0100 (CET) Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) by dpdk.org (Postfix) with ESMTP id 9C1871BFE6 for ; Thu, 30 Jan 2020 15:38:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1580395137; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=Hwhs1+w3M9Lk1Lfst5XMLLdkxa0a0oG9ueOwIhZ9oE8=; b=BkgK5z3pAB6VyarnX8qPa/Z5VjKHkh8rssPsRpS5YNwABehAYD7wyFx2YfvDA7nxNzRlcV 94W5j4lxENs+j4SMXjVATRdg9+xHHjqTCp1m13YxgmJBYgvFU+L7tmiJOAbyIsdcMdHOjL mPwXz7NQxMYmcGfxCFhqYP6vzh+HQm0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-101-1yn3wkkMMTyPRzoI1tuBIA-1; Thu, 30 Jan 2020 09:38:54 -0500 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1A8E7A0CBF; Thu, 30 Jan 2020 14:38:53 +0000 (UTC) Received: from [10.36.112.51] (ovpn-112-51.ams2.redhat.com [10.36.112.51]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9C9D07FB5B; Thu, 30 Jan 2020 14:38:51 +0000 (UTC) To: Matan Azrad , dev@dpdk.org, Viacheslav Ovsiienko References: <1579539790-3882-1-git-send-email-matan@mellanox.com> <1580292549-27439-1-git-send-email-matan@mellanox.com> <1580292549-27439-2-git-send-email-matan@mellanox.com> From: Maxime Coquelin Autocrypt: addr=maxime.coquelin@redhat.com; keydata= mQINBFOEQQIBEADjNLYZZqghYuWv1nlLisptPJp+TSxE/KuP7x47e1Gr5/oMDJ1OKNG8rlNg kLgBQUki3voWhUbMb69ybqdMUHOl21DGCj0BTU3lXwapYXOAnsh8q6RRM+deUpasyT+Jvf3a gU35dgZcomRh5HPmKMU4KfeA38cVUebsFec1HuJAWzOb/UdtQkYyZR4rbzw8SbsOemtMtwOx YdXodneQD7KuRU9IhJKiEfipwqk2pufm2VSGl570l5ANyWMA/XADNhcEXhpkZ1Iwj3TWO7XR uH4xfvPl8nBsLo/EbEI7fbuUULcAnHfowQslPUm6/yaGv6cT5160SPXT1t8U9QDO6aTSo59N jH519JS8oeKZB1n1eLDslCfBpIpWkW8ZElGkOGWAN0vmpLfdyiqBNNyS3eGAfMkJ6b1A24un /TKc6j2QxM0QK4yZGfAxDxtvDv9LFXec8ENJYsbiR6WHRHq7wXl/n8guyh5AuBNQ3LIK44x0 KjGXP1FJkUhUuruGyZsMrDLBRHYi+hhDAgRjqHgoXi5XGETA1PAiNBNnQwMf5aubt+mE2Q5r qLNTgwSo2dpTU3+mJ3y3KlsIfoaxYI7XNsPRXGnZi4hbxmeb2NSXgdCXhX3nELUNYm4ArKBP LugOIT/zRwk0H0+RVwL2zHdMO1Tht1UOFGfOZpvuBF60jhMzbQARAQABtCxNYXhpbWUgQ29x dWVsaW4gPG1heGltZS5jb3F1ZWxpbkByZWRoYXQuY29tPokCOAQTAQIAIgUCV3u/5QIbAwYL CQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQyjiNKEaHD4ma2g/+P+Hg9WkONPaY1J4AR7Uf kBneosS4NO3CRy0x4WYmUSLYMLx1I3VH6SVjqZ6uBoYy6Fs6TbF6SHNc7QbB6Qjo3neqnQR1 71Ua1MFvIob8vUEl3jAR/+oaE1UJKrxjWztpppQTukIk4oJOmXbL0nj3d8dA2QgHdTyttZ1H xzZJWWz6vqxCrUqHU7RSH9iWg9R2iuTzii4/vk1oi4Qz7y/q8ONOq6ffOy/t5xSZOMtZCspu Mll2Szzpc/trFO0pLH4LZZfz/nXh2uuUbk8qRIJBIjZH3ZQfACffgfNefLe2PxMqJZ8mFJXc RQO0ONZvwoOoHL6CcnFZp2i0P5ddduzwPdGsPq1bnIXnZqJSl3dUfh3xG5ArkliZ/++zGF1O wvpGvpIuOgLqjyCNNRoR7cP7y8F24gWE/HqJBXs1qzdj/5Hr68NVPV1Tu/l2D1KMOcL5sOrz 2jLXauqDWn1Okk9hkXAP7+0Cmi6QwAPuBT3i6t2e8UdtMtCE4sLesWS/XohnSFFscZR6Vaf3 gKdWiJ/fW64L6b9gjkWtHd4jAJBAIAx1JM6xcA1xMbAFsD8gA2oDBWogHGYcScY/4riDNKXi lw92d6IEHnSf6y7KJCKq8F+Jrj2BwRJiFKTJ6ChbOpyyR6nGTckzsLgday2KxBIyuh4w+hMq TGDSp2rmWGJjASq5Ag0EVPSbkwEQAMkaNc084Qvql+XW+wcUIY+Dn9A2D1gMr2BVwdSfVDN7 0ZYxo9PvSkzh6eQmnZNQtl8WSHl3VG3IEDQzsMQ2ftZn2sxjcCadexrQQv3Lu60Tgj7YVYRM H+fLYt9W5YuWduJ+FPLbjIKynBf6JCRMWr75QAOhhhaI0tsie3eDsKQBA0w7WCuPiZiheJaL 4MDe9hcH4rM3ybnRW7K2dLszWNhHVoYSFlZGYh+MGpuODeQKDS035+4H2rEWgg+iaOwqD7bg CQXwTZ1kSrm8NxIRVD3MBtzp9SZdUHLfmBl/tLVwDSZvHZhhvJHC6Lj6VL4jPXF5K2+Nn/Su CQmEBisOmwnXZhhu8ulAZ7S2tcl94DCo60ReheDoPBU8PR2TLg8rS5f9w6mLYarvQWL7cDtT d2eX3Z6TggfNINr/RTFrrAd7NHl5h3OnlXj7PQ1f0kfufduOeCQddJN4gsQfxo/qvWVB7PaE 1WTIggPmWS+Xxijk7xG6x9McTdmGhYaPZBpAxewK8ypl5+yubVsE9yOOhKMVo9DoVCjh5To5 aph7CQWfQsV7cd9PfSJjI2lXI0dhEXhQ7lRCFpf3V3mD6CyrhpcJpV6XVGjxJvGUale7+IOp sQIbPKUHpB2F+ZUPWds9yyVxGwDxD8WLqKKy0WLIjkkSsOb9UBNzgRyzrEC9lgQ/ABEBAAGJ Ah8EGAECAAkFAlT0m5MCGwwACgkQyjiNKEaHD4nU8hAAtt0xFJAy0sOWqSmyxTc7FUcX+pbD KVyPlpl6urKKMk1XtVMUPuae/+UwvIt0urk1mXi6DnrAN50TmQqvdjcPTQ6uoZ8zjgGeASZg jj0/bJGhgUr9U7oG7Hh2F8vzpOqZrdd65MRkxmc7bWj1k81tOU2woR/Gy8xLzi0k0KUa8ueB iYOcZcIGTcs9CssVwQjYaXRoeT65LJnTxYZif2pfNxfINFzCGw42s3EtZFteczClKcVSJ1+L +QUY/J24x0/ocQX/M1PwtZbB4c/2Pg/t5FS+s6UB1Ce08xsJDcwyOPIH6O3tccZuriHgvqKP yKz/Ble76+NFlTK1mpUlfM7PVhD5XzrDUEHWRTeTJSvJ8TIPL4uyfzhjHhlkCU0mw7Pscyxn DE8G0UYMEaNgaZap8dcGMYH/96EfE5s/nTX0M6MXV0yots7U2BDb4soLCxLOJz4tAFDtNFtA wLBhXRSvWhdBJZiig/9CG3dXmKfi2H+wdUCSvEFHRpgo7GK8/Kh3vGhgKmnnxhl8ACBaGy9n fxjSxjSO6rj4/MeenmlJw1yebzkX8ZmaSi8BHe+n6jTGEFNrbiOdWpJgc5yHIZZnwXaW54QT UhhSjDL1rV2B4F28w30jYmlRmm2RdN7iCZfbyP3dvFQTzQ4ySquuPkIGcOOHrvZzxbRjzMx1 Mwqu3GQ= Message-ID: Date: Thu, 30 Jan 2020 15:38:49 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: <1580292549-27439-2-git-send-email-matan@mellanox.com> Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-MC-Unique: 1yn3wkkMMTyPRzoI1tuBIA-1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [PATCH v2 01/13] drivers: introduce mlx5 vDPA driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 1/29/20 11:08 AM, Matan Azrad wrote: > Add a new driver to support vDPA operations by Mellanox devices. >=20 > The first Mellanox devices which support vDPA operations are > ConnectX6DX and Bluefield1 HCA for their PF ports and VF ports. >=20 > This driver is depending on rdma-core like the mlx5 PMD, also it is > going to use mlx5 DevX to create HW objects directly by the FW. > Hence, the common/mlx5 library is linked to the mlx5_vdpa driver. If possible, I would really appreciate to have the information on the versions required for the above dependencies. Better if it is also mentionned in the guide. > This driver will not be compiled by default due to the above > dependencies. >=20 > Register a new log type for this driver. >=20 > Signed-off-by: Matan Azrad > Acked-by: Viacheslav Ovsiienko > --- > MAINTAINERS | 7 + > config/common_base | 5 + > doc/guides/rel_notes/release_20_02.rst | 5 + > doc/guides/vdpadevs/features/mlx5.ini | 14 ++ > doc/guides/vdpadevs/index.rst | 1 + > doc/guides/vdpadevs/mlx5.rst | 111 ++++++++++++ > drivers/common/Makefile | 2 +- > drivers/common/mlx5/Makefile | 17 +- > drivers/meson.build | 8 +- > drivers/vdpa/Makefile | 2 + > drivers/vdpa/meson.build | 3 +- > drivers/vdpa/mlx5/Makefile | 36 ++++ > drivers/vdpa/mlx5/meson.build | 29 +++ > drivers/vdpa/mlx5/mlx5_vdpa.c | 227 ++++++++++++++++++= ++++++ > drivers/vdpa/mlx5/mlx5_vdpa_utils.h | 20 +++ > drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map | 3 + > mk/rte.app.mk | 15 +- > 17 files changed, 488 insertions(+), 17 deletions(-) > create mode 100644 doc/guides/vdpadevs/features/mlx5.ini > create mode 100644 doc/guides/vdpadevs/mlx5.rst > create mode 100644 drivers/vdpa/mlx5/Makefile > create mode 100644 drivers/vdpa/mlx5/meson.build > create mode 100644 drivers/vdpa/mlx5/mlx5_vdpa.c > create mode 100644 drivers/vdpa/mlx5/mlx5_vdpa_utils.h > create mode 100644 drivers/vdpa/mlx5/rte_pmd_mlx5_vdpa_version.map >=20 > diff --git a/MAINTAINERS b/MAINTAINERS > index 150d507..f697e9a 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -1103,6 +1103,13 @@ F: drivers/vdpa/ifc/ > F: doc/guides/vdpadevs/ifc.rst > F: doc/guides/vdpadevs/features/ifcvf.ini > =20 > +Mellanox mlx5 vDPA > +M: Matan Azrad > +M: Viacheslav Ovsiienko > +F: drivers/vdpa/mlx5/ > +F: doc/guides/vdpadevs/mlx5.rst > +F: doc/guides/vdpadevs/features/mlx5.ini > + > =20 > Eventdev Drivers > ---------------- > diff --git a/config/common_base b/config/common_base > index c897dd0..6ea9c63 100644 > --- a/config/common_base > +++ b/config/common_base > @@ -366,6 +366,11 @@ CONFIG_RTE_LIBRTE_MLX4_DEBUG=3Dn > CONFIG_RTE_LIBRTE_MLX5_PMD=3Dn > CONFIG_RTE_LIBRTE_MLX5_DEBUG=3Dn > =20 > +# > +# Compile vdpa-oriented Mellanox ConnectX-6 & Bluefield (MLX5) PMD > +# > +CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD=3Dn > + > # Linking method for mlx4/5 dependency on ibverbs and related libraries > # Default linking is dynamic by linker. > # Other options are: dynamic by dlopen at run-time, or statically embedd= ed. > diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_note= s/release_20_02.rst > index 50e2c14..690e7db 100644 > --- a/doc/guides/rel_notes/release_20_02.rst > +++ b/doc/guides/rel_notes/release_20_02.rst > @@ -113,6 +113,11 @@ New Features > * Added support for RSS using L3/L4 source/destination only. > * Added support for matching on GTP tunnel header item. > =20 > +* **Add new vDPA PMD based on Mellanox devices** > + > + Added a new Mellanox vDPA (``mlx5_vdpa``) PMD. > + See the :doc:`../vdpadevs/mlx5` guide for more details on this driver. > + > * **Updated testpmd application.** > =20 > Added support for ESP and L2TPv3 over IP rte_flow patterns to the test= pmd > diff --git a/doc/guides/vdpadevs/features/mlx5.ini b/doc/guides/vdpadevs/= features/mlx5.ini > new file mode 100644 > index 0000000..d635bdf > --- /dev/null > +++ b/doc/guides/vdpadevs/features/mlx5.ini > @@ -0,0 +1,14 @@ > +; > +; Supported features of the 'mlx5' VDPA driver. > +; > +; Refer to default.ini for the full list of available driver features. > +; > +[Features] > +Other kdrv =3D Y > +ARMv8 =3D Y > +Power8 =3D Y > +x86-32 =3D Y > +x86-64 =3D Y > +Usage doc =3D Y > +Design doc =3D Y > + > diff --git a/doc/guides/vdpadevs/index.rst b/doc/guides/vdpadevs/index.rs= t > index 9657108..1a13efe 100644 > --- a/doc/guides/vdpadevs/index.rst > +++ b/doc/guides/vdpadevs/index.rst > @@ -13,3 +13,4 @@ which can be used from an application through vhost API= . > =20 > features_overview > ifc > + mlx5 > diff --git a/doc/guides/vdpadevs/mlx5.rst b/doc/guides/vdpadevs/mlx5.rst > new file mode 100644 > index 0000000..1861e71 > --- /dev/null > +++ b/doc/guides/vdpadevs/mlx5.rst > @@ -0,0 +1,111 @@ > +.. SPDX-License-Identifier: BSD-3-Clause > + Copyright 2019 Mellanox Technologies, Ltd > + > +MLX5 vDPA driver > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + > +The MLX5 vDPA (vhost data path acceleration) driver library > +(**librte_pmd_mlx5_vdpa**) provides support for **Mellanox ConnectX-6**, > +**Mellanox ConnectX-6DX** and **Mellanox BlueField** families of > +10/25/40/50/100/200 Gb/s adapters as well as their virtual functions (VF= ) in > +SR-IOV context. > + > +.. note:: > + > + Due to external dependencies, this driver is disabled in default > + configuration of the "make" build. It can be enabled with > + ``CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD=3Dy`` or by using "meson" build sys= tem which > + will detect dependencies. > + > + > +Design > +------ > + > +For security reasons and robustness, this driver only deals with virtual > +memory addresses. The way resources allocations are handled by the kerne= l, > +combined with hardware specifications that allow to handle virtual memor= y > +addresses directly, ensure that DPDK applications cannot access random > +physical memory (or memory that does not belong to the current process). > + > +The PMD can use libibverbs and libmlx5 to access the device firmware > +or directly the hardware components. > +There are different levels of objects and bypassing abilities > +to get the best performances: > + > +- Verbs is a complete high-level generic API > +- Direct Verbs is a device-specific API > +- DevX allows to access firmware objects > +- Direct Rules manages flow steering at low-level hardware layer > + > +Enabling librte_pmd_mlx5_vdpa causes DPDK applications to be linked agai= nst > +libibverbs. > + > +A Mellanox mlx5 PCI device can be probed by either net/mlx5 driver or vd= pa/mlx5 > +driver but not in parallel. Hence, the user should decide the driver by = the > +``class`` parameter in the device argument list. > +By default, the mlx5 device will be probed by the net/mlx5 driver.=20 > + > +Supported NICs > +-------------- > + > +* Mellanox(R) ConnectX(R)-6 200G MCX654106A-HCAT (4x200G) > +* Mellanox(R) ConnectX(R)-6DX EN 100G MCX623106AN-CDAT (2*100G) > +* Mellanox(R) ConnectX(R)-6DX EN 200G MCX623105AN-VDAT (1*200G) > +* Mellanox(R) BlueField SmartNIC 25G MBF1M332A-ASCAT (2*25G) > + > +Prerequisites > +------------- > + > +- Mellanox OFED version: **4.7** > + see :doc:`../../nics/mlx5` guide for more Mellanox OFED details. > + > +Compilation options > +~~~~~~~~~~~~~~~~~~~ > + > +These options can be modified in the ``.config`` file. > + > +- ``CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD`` (default **n**) > + > + Toggle compilation of librte_pmd_mlx5 itself. > + > +- ``CONFIG_RTE_IBVERBS_LINK_DLOPEN`` (default **n**) > + > + Build PMD with additional code to make it loadable without hard > + dependencies on **libibverbs** nor **libmlx5**, which may not be insta= lled > + on the target system. > + > + In this mode, their presence is still required for it to run properly, > + however their absence won't prevent a DPDK application from starting (= with > + ``CONFIG_RTE_BUILD_SHARED_LIB`` disabled) and they won't show up as > + missing with ``ldd(1)``. > + > + It works by moving these dependencies to a purpose-built rdma-core "gl= ue" > + plug-in which must either be installed in a directory whose name is ba= sed > + on ``CONFIG_RTE_EAL_PMD_PATH`` suffixed with ``-glue`` if set, or in a > + standard location for the dynamic linker (e.g. ``/lib``) if left to th= e > + default empty string (``""``). > + > + This option has no performance impact. > + > +- ``CONFIG_RTE_IBVERBS_LINK_STATIC`` (default **n**) > + > + Embed static flavor of the dependencies **libibverbs** and **libmlx5** > + in the PMD shared library or the executable static binary. > + > +.. note:: > + > + For BlueField, target should be set to ``arm64-bluefield-linux-gcc``.= This > + will enable ``CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD`` and set > + ``RTE_CACHE_LINE_SIZE`` to 64. Default armv8a configuration of make b= uild and > + meson build set it to 128 then brings performance degradation. > + > +Run-time configuration > +~~~~~~~~~~~~~~~~~~~~~~ > + > +- **ethtool** operations on related kernel interfaces also affect the PM= D. > + > +- ``class`` parameter [string] > + > + Select the class of the driver that should probe the device. > + `vdpa` for the mlx5 vDPA driver. > + > diff --git a/drivers/common/Makefile b/drivers/common/Makefile > index 4775d4b..96bd7ac 100644 > --- a/drivers/common/Makefile > +++ b/drivers/common/Makefile > @@ -35,7 +35,7 @@ ifneq (,$(findstring y,$(IAVF-y))) > DIRS-y +=3D iavf > endif > =20 > -ifeq ($(CONFIG_RTE_LIBRTE_MLX5_PMD),y) > +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_ML= X5_VDPA_PMD)),y) > DIRS-y +=3D mlx5 > endif > =20 > diff --git a/drivers/common/mlx5/Makefile b/drivers/common/mlx5/Makefile > index 9d4d81f..c4b7999 100644 > --- a/drivers/common/mlx5/Makefile > +++ b/drivers/common/mlx5/Makefile > @@ -10,15 +10,16 @@ LIB_GLUE_BASE =3D librte_pmd_mlx5_glue.so > LIB_GLUE_VERSION =3D 20.02.0 > =20 > # Sources. > +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_ML= X5_VDPA_PMD)),y) > ifneq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) > -SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) +=3D mlx5_glue.c > +SRCS-y +=3D mlx5_glue.c > endif > -SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) +=3D mlx5_devx_cmds.c > -SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) +=3D mlx5_common.c > -SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD) +=3D mlx5_nl.c > - > +SRCS-y +=3D mlx5_devx_cmds.c > +SRCS-y +=3D mlx5_common.c > +SRCS-y +=3D mlx5_nl.c > ifeq ($(CONFIG_RTE_IBVERBS_LINK_DLOPEN),y) > -INSTALL-$(CONFIG_RTE_LIBRTE_MLX5_PMD)-lib +=3D $(LIB_GLUE) > +INSTALL-y-lib +=3D $(LIB_GLUE) > +endif > endif > =20 > # Basic CFLAGS. > @@ -317,7 +318,9 @@ mlx5_autoconf.h: mlx5_autoconf.h.new > =09=09cmp '$<' '$@' $(AUTOCONF_OUTPUT) || \ > =09=09mv '$<' '$@' > =20 > -$(SRCS-$(CONFIG_RTE_LIBRTE_MLX5_PMD):.c=3D.o): mlx5_autoconf.h > +ifeq ($(findstring y,$(CONFIG_RTE_LIBRTE_MLX5_PMD)$(CONFIG_RTE_LIBRTE_ML= X5_VDPA_PMD)),y) > +$(SRCS-y:.c=3D.o): mlx5_autoconf.h > +endif > =20 > # Generate dependency plug-in for rdma-core when the PMD must not be lin= ked > # directly, so that applications do not inherit this dependency. > diff --git a/drivers/meson.build b/drivers/meson.build > index 29708cc..bd154fa 100644 > --- a/drivers/meson.build > +++ b/drivers/meson.build > @@ -42,6 +42,7 @@ foreach class:dpdk_driver_classes > =09=09build =3D true # set to false to disable, e.g. missing deps > =09=09reason =3D '' # set if build =3D=3D false to expla= in > =09=09name =3D drv > +=09=09fmt_name =3D '' > =09=09allow_experimental_apis =3D false > =09=09sources =3D [] > =09=09objs =3D [] > @@ -98,8 +99,11 @@ foreach class:dpdk_driver_classes > =09=09else > =09=09=09class_drivers +=3D name > =20 > -=09=09=09dpdk_conf.set(config_flag_fmt.format(name.to_upper()),1) > -=09=09=09lib_name =3D driver_name_fmt.format(name) > +=09=09=09if fmt_name =3D=3D '' > +=09=09=09=09fmt_name =3D name > +=09=09=09endif > +=09=09=09dpdk_conf.set(config_flag_fmt.format(fmt_name.to_upper()),1) > +=09=09=09lib_name =3D driver_name_fmt.format(fmt_name) > =20 > =09=09=09if allow_experimental_apis > =09=09=09=09cflags +=3D '-DALLOW_EXPERIMENTAL_API' > diff --git a/drivers/vdpa/Makefile b/drivers/vdpa/Makefile > index b5a7a11..6e88359 100644 > --- a/drivers/vdpa/Makefile > +++ b/drivers/vdpa/Makefile > @@ -7,4 +7,6 @@ ifeq ($(CONFIG_RTE_EAL_VFIO),y) > DIRS-$(CONFIG_RTE_LIBRTE_IFC_PMD) +=3D ifc > endif > =20 > +DIRS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD) +=3D mlx5 > + > include $(RTE_SDK)/mk/rte.subdir.mk > diff --git a/drivers/vdpa/meson.build b/drivers/vdpa/meson.build > index 2f047b5..e3ed54a 100644 > --- a/drivers/vdpa/meson.build > +++ b/drivers/vdpa/meson.build > @@ -1,7 +1,8 @@ > # SPDX-License-Identifier: BSD-3-Clause > # Copyright 2019 Mellanox Technologies, Ltd > =20 > -drivers =3D ['ifc'] > +drivers =3D ['ifc', > +=09 'mlx5',] > std_deps =3D ['bus_pci', 'kvargs'] > std_deps +=3D ['vhost'] > config_flag_fmt =3D 'RTE_LIBRTE_@0@_PMD' > diff --git a/drivers/vdpa/mlx5/Makefile b/drivers/vdpa/mlx5/Makefile > new file mode 100644 > index 0000000..c1c8cc0 > --- /dev/null > +++ b/drivers/vdpa/mlx5/Makefile > @@ -0,0 +1,36 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright 2019 Mellanox Technologies, Ltd > + > +include $(RTE_SDK)/mk/rte.vars.mk > + > +# Library name. > +LIB =3D librte_pmd_mlx5_vdpa.a > + > +# Sources. > +SRCS-$(CONFIG_RTE_LIBRTE_MLX5_VDPA_PMD) +=3D mlx5_vdpa.c > + > +# Basic CFLAGS. > +CFLAGS +=3D -O3 > +CFLAGS +=3D -std=3Dc11 -Wall -Wextra > +CFLAGS +=3D -g > +CFLAGS +=3D -I$(RTE_SDK)/drivers/common/mlx5 > +CFLAGS +=3D -I$(RTE_SDK)/drivers/net/mlx5_vdpa > +CFLAGS +=3D -I$(BUILDDIR)/drivers/common/mlx5 > +CFLAGS +=3D -D_BSD_SOURCE > +CFLAGS +=3D -D_DEFAULT_SOURCE > +CFLAGS +=3D -D_XOPEN_SOURCE=3D600 > +CFLAGS +=3D $(WERROR_FLAGS) > +CFLAGS +=3D -Wno-strict-prototypes > +LDLIBS +=3D -lrte_common_mlx5 > +LDLIBS +=3D -lrte_eal -lrte_vhost -lrte_kvargs -lrte_bus_pci > + > +# A few warnings cannot be avoided in external headers. > +CFLAGS +=3D -Wno-error=3Dcast-qual > + > +EXPORT_MAP :=3D rte_pmd_mlx5_vdpa_version.map > +# memseg walk is not part of stable API > +CFLAGS +=3D -DALLOW_EXPERIMENTAL_API > + > +CFLAGS +=3D -DNDEBUG -UPEDANTIC > + > +include $(RTE_SDK)/mk/rte.lib.mk > diff --git a/drivers/vdpa/mlx5/meson.build b/drivers/vdpa/mlx5/meson.buil= d > new file mode 100644 > index 0000000..4bca6ea > --- /dev/null > +++ b/drivers/vdpa/mlx5/meson.build > @@ -0,0 +1,29 @@ > +# SPDX-License-Identifier: BSD-3-Clause > +# Copyright 2019 Mellanox Technologies, Ltd > + > +if not is_linux > +=09build =3D false > +=09reason =3D 'only supported on Linux' > +=09subdir_done() > +endif > + > +fmt_name =3D 'mlx5_vdpa' > +allow_experimental_apis =3D true > +deps +=3D ['hash', 'common_mlx5', 'vhost', 'bus_pci', 'eal'] > +sources =3D files( > +=09'mlx5_vdpa.c', > +) > +cflags_options =3D [ > +=09'-std=3Dc11', > +=09'-Wno-strict-prototypes', > +=09'-D_BSD_SOURCE', > +=09'-D_DEFAULT_SOURCE', > +=09'-D_XOPEN_SOURCE=3D600' > +] > +foreach option:cflags_options > +=09if cc.has_argument(option) > +=09=09cflags +=3D option > +=09endif > +endforeach > + > +cflags +=3D [ '-DNDEBUG', '-UPEDANTIC' ] > diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.= c > new file mode 100644 > index 0000000..6286d7a > --- /dev/null > +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c > @@ -0,0 +1,227 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2019 Mellanox Technologies, Ltd > + */ > +#include > +#include > +#include > +#include > +#include > + > +#include > +#include > + > +#include "mlx5_vdpa_utils.h" > + > + > +struct mlx5_vdpa_priv { > +=09TAILQ_ENTRY(mlx5_vdpa_priv) next; > +=09int id; /* vDPA device id. */ > +=09struct ibv_context *ctx; /* Device context. */ > +=09struct rte_vdpa_dev_addr dev_addr; > +}; > + > +TAILQ_HEAD(mlx5_vdpa_privs, mlx5_vdpa_priv) priv_list =3D > +=09=09=09=09=09 TAILQ_HEAD_INITIALIZER(priv_list); > +static pthread_mutex_t priv_list_lock =3D PTHREAD_MUTEX_INITIALIZER; > +int mlx5_vdpa_logtype; > + > +static struct rte_vdpa_dev_ops mlx5_vdpa_ops =3D { > +=09.get_queue_num =3D NULL, > +=09.get_features =3D NULL, > +=09.get_protocol_features =3D NULL, > +=09.dev_conf =3D NULL, > +=09.dev_close =3D NULL, > +=09.set_vring_state =3D NULL, > +=09.set_features =3D NULL, > +=09.migration_done =3D NULL, > +=09.get_vfio_group_fd =3D NULL, > +=09.get_vfio_device_fd =3D NULL, > +=09.get_notify_area =3D NULL, > +}; > + > +/** > + * DPDK callback to register a PCI device. > + * > + * This function spawns vdpa device out of a given PCI device. > + * > + * @param[in] pci_drv > + * PCI driver structure (mlx5_vpda_driver). > + * @param[in] pci_dev > + * PCI device information. > + * > + * @return > + * 0 on success, a negative errno value otherwise and rte_errno is set= . > + */ > +static int > +mlx5_vdpa_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, > +=09=09 struct rte_pci_device *pci_dev __rte_unused) > +{ > +=09struct ibv_device **ibv_list; > +=09struct ibv_device *ibv_match =3D NULL; > +=09struct mlx5_vdpa_priv *priv =3D NULL; > +=09struct ibv_context *ctx =3D NULL; > +=09int ret; > + > +=09if (mlx5_class_get(pci_dev->device.devargs) !=3D MLX5_CLASS_VDPA) { > +=09=09DRV_LOG(DEBUG, "Skip probing - should be probed by other mlx5" > +=09=09=09" driver."); > +=09=09return 1; Maybe the function doc should be implemented for return 1 case: * @return * 0 on success, a negative errno value otherwise and rte_errno is set. > +=09} > +=09errno =3D 0; > +=09ibv_list =3D mlx5_glue->get_device_list(&ret); > +=09if (!ibv_list) { > +=09=09rte_errno =3D errno; > +=09=09DRV_LOG(ERR, "Failed to get device list, is ib_uverbs loaded?"); > +=09=09return -ENOSYS; Shouldn't return -rte_errno to be consistent with the rest of the function? For the sake of consistency, you could also goto error instead. > +=09} > +=09while (ret-- > 0) { > +=09=09struct rte_pci_addr pci_addr; > + > +=09=09DRV_LOG(DEBUG, "Checking device \"%s\"..", ibv_list[ret]->name); > +=09=09if (mlx5_dev_to_pci_addr(ibv_list[ret]->ibdev_path, &pci_addr)) > +=09=09=09continue; > +=09=09if (pci_dev->addr.domain !=3D pci_addr.domain || > +=09=09 pci_dev->addr.bus !=3D pci_addr.bus || > +=09=09 pci_dev->addr.devid !=3D pci_addr.devid || > +=09=09 pci_dev->addr.function !=3D pci_addr.function) > +=09=09=09continue; > +=09=09DRV_LOG(INFO, "PCI information matches for device \"%s\".", > +=09=09=09ibv_list[ret]->name); > +=09=09ibv_match =3D ibv_list[ret]; > +=09=09break; > +=09} > +=09mlx5_glue->free_device_list(ibv_list); > +=09if (!ibv_match) { > +=09=09DRV_LOG(ERR, "No matching IB device for PCI slot " > +=09=09=09"%" SCNx32 ":%" SCNx8 ":%" SCNx8 ".%" SCNx8 ".", > +=09=09=09pci_dev->addr.domain, pci_dev->addr.bus, > +=09=09=09pci_dev->addr.devid, pci_dev->addr.function); > +=09=09rte_errno =3D ENOENT; > +=09=09return -rte_errno; > +=09} > +=09ctx =3D mlx5_glue->dv_open_device(ibv_match); > +=09if (!ctx) { > +=09=09DRV_LOG(ERR, "Failed to open IB device \"%s\".", > +=09=09=09ibv_match->name); > +=09=09rte_errno =3D ENODEV; > +=09=09return -rte_errno; > +=09} > +=09priv =3D rte_zmalloc("mlx5 vDPA device private", sizeof(*priv), > +=09=09=09 RTE_CACHE_LINE_SIZE); > +=09if (!priv) { > +=09=09DRV_LOG(ERR, "Failed to allocate private memory."); > +=09=09rte_errno =3D ENOMEM; > +=09=09goto error; > +=09} > +=09priv->ctx =3D ctx; > +=09priv->dev_addr.pci_addr =3D pci_dev->addr; > +=09priv->dev_addr.type =3D PCI_ADDR; > +=09priv->id =3D rte_vdpa_register_device(&priv->dev_addr, &mlx5_vdpa_ops= ); > +=09if (priv->id < 0) { > +=09=09DRV_LOG(ERR, "Failed to register vDPA device."); > +=09=09rte_errno =3D rte_errno ? rte_errno : EINVAL; > +=09=09goto error; > +=09} > +=09pthread_mutex_lock(&priv_list_lock); > +=09TAILQ_INSERT_TAIL(&priv_list, priv, next); > +=09pthread_mutex_unlock(&priv_list_lock); > +=09return 0; > + > +error: > +=09if (priv) > +=09=09rte_free(priv); > +=09if (ctx) > +=09=09mlx5_glue->close_device(ctx); > +=09return -rte_errno; > +} > + These are minor comments. If directly fixed in v3, feel free to add my: Reviewed-by: Maxime Coquelin Thanks, Maxime