From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AB17CA034F; Tue, 30 Mar 2021 10:05:03 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3ECCD406FF; Tue, 30 Mar 2021 10:05:03 +0200 (CEST) Received: from NAM04-BN3-obe.outbound.protection.outlook.com (mail-eopbgr680086.outbound.protection.outlook.com [40.107.68.86]) by mails.dpdk.org (Postfix) with ESMTP id F3938406B4 for ; Tue, 30 Mar 2021 10:05:01 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CINlWWUjAosvjFnYwbqjCbTrhVqoLJiEttzTuahxRp1YeRb6Ez3L3wdqoFS3aqi8wFcfN2r9X8b4liGq9GXOW+E4lnKsuxMupJAKSUJUiScSxp+/u40F8W2UbNADbLTBlag/YxJ1H3LUkLH00Dsa4GDa+he1SWGJWITrTT/2jUEHeEyZvCtDDh7dB3vVmbUoayN3MZjmY0vaN1bjmIxxPotbTMDmfNaRdzIV+/daRfYCAeDctfHVb9//BjOAjmoH1lHtBYTjaY/AU9Z28xx0F2aN62PhSJC6faDnTPQvSZfjKluDkPJajc81CKK2/FpZGmOjAZP6JRMoF9SJx9OOHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=onmFT1qVMAiV6CC/Qz/ZPLdoiyyNK8wNWb2uARjAOcY=; b=YnlyuoY+t9xIqMvt85ue7Em049eHb2Swrcfpyt2PwcxKHK2M/cqWtkzTc7qiRYqmDpER6nY4AVdzY8J2Gfs4CqXLtgEYU+/NN6cYsLPe+eLj2MzqYz+i8W8W6tzBcso5a6CfsIHipMG50ai9tZ2a2fEhJ0ILpHhyRy/kC00PcjhZxGIDbCPTTCeJqzFFyAJ1s3/8bZGkC7jMc8VuREnTuBwR6wlpY/X5zEujdVVIQKB13oauyCbAGRuUONCjlniAuaPbHu/PoNlsmXZ7bzVw9SHEUfLeiVIC98uyOniG+JxpE5dsbHXJGNESZ501KyC42dBGiZmJgNgv6n3PcwgnQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=onmFT1qVMAiV6CC/Qz/ZPLdoiyyNK8wNWb2uARjAOcY=; b=phVVi9g/a1MLovkGrZT1nQVPhl0aQ2/2X7Oei7LEpJjjRBJc81iWDDKzo+wBSYhBtO6x4SLvoMTgkCJvoej5HLaKE4AzDOniuzUNb6H9xwZly+iUNuJJ8lx5LgtrNpC+SRDrDdmKnovcH/iQYXrg9jkecdIsZLe99T6NiEEXEGrcDgnd+EhFbqGbmKCYJAL8ggT9p12Xq7fz/vEcsZ8eTe1spXo8QDz4thwwN4V/bmWXUuMtHUyzgzAZVCAxN2nUMzXlTK2DjVQ8XgV8vi5W7daC2+zPUqDz7ELBf9YFwMyZrocQg492TvgfjSACyvpDnjXskXpc7lCX+eDnlFYO2w== Received: from DM6PR12MB3753.namprd12.prod.outlook.com (2603:10b6:5:1c7::18) by DM6PR12MB5024.namprd12.prod.outlook.com (2603:10b6:5:20a::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.26; Tue, 30 Mar 2021 08:05:00 +0000 Received: from DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::20f7:11fc:9d46:9258]) by DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::20f7:11fc:9d46:9258%6]) with mapi id 15.20.3977.033; Tue, 30 Mar 2021 08:05:00 +0000 From: Slava Ovsiienko To: Suanming Mou , Ori Kam CC: "dev@dpdk.org" , Matan Azrad , Raslan Darawsheh Thread-Topic: [PATCH v3 2/4] regex/mlx5: add data path scattered mbuf process Thread-Index: AQHXJQWI7lt3Nlvmu06mE0jgtoCKeaqcKh6g Date: Tue, 30 Mar 2021 08:05:00 +0000 Message-ID: References: <20210309235732.3952418-1-suanmingm@nvidia.com> <20210330013916.1319266-1-suanmingm@nvidia.com> <20210330013916.1319266-3-suanmingm@nvidia.com> In-Reply-To: <20210330013916.1319266-3-suanmingm@nvidia.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: nvidia.com; dkim=none (message not signed) header.d=none;nvidia.com; dmarc=none action=none header.from=nvidia.com; x-originating-ip: [95.164.10.10] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3f3310d7-3e4d-46e7-0daf-08d8f3528447 x-ms-traffictypediagnostic: DM6PR12MB5024: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:138; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: m1Eoc3DXQ4J+q1BV+JTvH5iX2X0/8C3i1XIv5HFOZN7blKFKOiVZT1ZA0w8VxFSghiJx3HkQO0zcu2MAIhCzuGe4os1VRrSGaBp456t4izSQFbzWyFsMuYMuT8CCpx+VBEbx7nkGos2uaRCXS8PlnJG99XwZvbGcQBWXlm6rv6eLMghqMJ2pUyLBy1LMRT2ghFgeZU/bJtVYS50US9s54rcbpaNrl8Y5mxQC/9G0t8ETiFXmY7IiCk0B6CkUJZYtKO/ECTAqx4T56lT3kk0nQ7nN0iWLkt7/yIcEHobNryW5TsdjNVGPK14uhWNsB0xdJHAJy5IbDZ+HpQ9AmTw6KNiPc5hJaA9oCIPdYQDWjXMzKJdEjXMvXeq3p798sAa/Es14XivELAmFvOufW4VyEUYhPSzesFYypdsDo6nDvnX+M9aL21hLRAqKBigawipGb/FdJdBw1Orqto9dlX8q9pT8/IwjUCLXCh7sxcaIXj280WZPfFRfChyhkHLXNoT4z7rOQ1PDYCnZMSj9H85UbGO49PduhqiK/ACiWhyfVHOFbF5N76IoXv/RMYpKO22196FsFzcC65v+nPjlKvPgzkg8/7xrAkn3q7gPlrzC+2v2+yKfRlwDb4wnDxJJETi5E/aCwGO+NNlZ1D7QZcmN2x8u7RO47CF6HEf2s6hW4ZE= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB3753.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(366004)(136003)(376002)(396003)(346002)(5660300002)(33656002)(107886003)(2906002)(30864003)(316002)(86362001)(8936002)(66946007)(110136005)(7696005)(52536014)(4326008)(83380400001)(64756008)(55016002)(8676002)(38100700001)(478600001)(54906003)(9686003)(186003)(6636002)(66476007)(66556008)(53546011)(6506007)(66446008)(71200400001)(76116006)(26005)(579004)(559001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?WtP3+zUblZsxojKmTC3eHW6tU1KEfg0kef2BLt+1XGbzegW9phQm8883A1LZ?= =?us-ascii?Q?QqgwP+E+SfD597yNnrwoeOCR99Xo9g1zQT2bBBw+orVSlshV8v7epzkCZ4JB?= =?us-ascii?Q?qBEGCRiijD9fW/6YyfjOTjDpLzheOO5/Wm7bcVheM4RTRtM6rau56YkyWdSM?= =?us-ascii?Q?bG8zk/mKJolR5zxo3+Eg0OFmOzmvgh98Z10puUZhyp0igANZO8SxXGCUfoxG?= =?us-ascii?Q?UCQBjqbHJQgGkarpNfippqvEVKAUrqs51g6dQJfVvmj4K6LiQAEIBwpG1nMw?= =?us-ascii?Q?4AWmJz/wPI41JoBpdnZlW+YsBKu981GcKODu/NfUUBgLkasG+yk+VSDz9d6R?= =?us-ascii?Q?bhGD6nU36vole37pi4qCVwCcdRJJO5APc3TVQDD0nWMlLwnz6ZypBVx8WuuE?= =?us-ascii?Q?jlDWCCOshEBLXVK65iRUPmGIKHjfVpK0d0LUeXEmeGh0RNylmG49XsRPQJzA?= =?us-ascii?Q?dBqxgP0sHYEIdeeks5nzZx0DzFK/pO+hPE2AquUmWkL0tcYqp9OPdGUfe8OD?= =?us-ascii?Q?di+VtW7YFX724Jjl6BqhBF0TpPhmfs5szmg1BzhrQuBoErj6qeCTi0YMnTbO?= =?us-ascii?Q?wAe46YuXM+ExEibnEQE8KlUl79YShI2n9uc/knrzq0gcLwJYCtFUTkS6jYVZ?= =?us-ascii?Q?cCFyEk7ZrMnI6Oekd68kNnil6+HwtCzW1AjtbXrEcVrDM+PEd4qI3rvHTeUn?= =?us-ascii?Q?LBGDHQ3l8sIXah+4YEKg+hfc89pBObWG1eYjL/ndfFGdHDI5hRLB2Oi3feWQ?= =?us-ascii?Q?LWDiI5fEcYDoLHidFknRRx2D/CcBSMip3Yds+tXkBVck/9ZoBk7HJo/s3WEG?= =?us-ascii?Q?YnTDbzWnR1pNRGd2ClMJfjkaEXv/jKp8WsoQgpbkvL5i0jNvzmibzITGYkd6?= =?us-ascii?Q?cliQJE28FO+MKBeKeOsXWIvHz4xqUj5mJx1EH1Z2T1fe2zpP2Ub9IGtzCeN5?= =?us-ascii?Q?ymhBw5ciWfUWPVlIG06i8U+G/pN3RIc13SWL3grgVYKaK0imZPpeSvXDab7M?= =?us-ascii?Q?thhj3rR2rgfuTBlen8NGqRzVGAZ6/G4pMH7b5vB+2sTX5CzZjSOaGZfnFbTS?= =?us-ascii?Q?xXz5TyFJ0U8pZF+pEPZnfRaOqN3MDnU04yqsp0sdp/KRQOjtxaE26jSSpFjK?= =?us-ascii?Q?3323EaQZgUru53ppbu7WadP/OSZz/qD1spj6uYmKkwNN5ebLyU3AC8oRd2Wt?= =?us-ascii?Q?15q9cue+5fYTxPSVwnEwvOf0vB0QafCuFBnPlDoU0Fvf3OAvj/dxCDsqusPm?= =?us-ascii?Q?xiJ3XezN/cXe0QegyoEVCsX9tEDpx2E3P7LcGl9vkX0bc5ynCM2RbTum7kuB?= =?us-ascii?Q?24YVF/zrEtt3RTqeaTuzxaU+?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3753.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3f3310d7-3e4d-46e7-0daf-08d8f3528447 X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Mar 2021 08:05:00.3323 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: jZThSlriWeCUhckIxiG5kdppSKJ6ONbrTpM43C6Qn32yWhV9pxnTVMwla3RBDL1AQIDrZHs9FZYfxnCyEcXcIg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB5024 Subject: Re: [dpdk-dev] [PATCH v3 2/4] regex/mlx5: add data path scattered mbuf process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Suanming Mou > Sent: Tuesday, March 30, 2021 4:39 > To: Ori Kam > Cc: dev@dpdk.org; Slava Ovsiienko ; Matan Azrad > ; Raslan Darawsheh > Subject: [PATCH v3 2/4] regex/mlx5: add data path scattered mbuf process >=20 Nice feature, but I would fix the typos and reword a bit: > UMR WQE can convert multiple mkey's memory sapce to contiguous space. Typo: "sapce?" And rather not "convert mkey" but "present data buffers scattered within multiple mbufs with single indirect mkey". > Take advantage of the UMR WQE, scattered mbuf in one operation can be > converted to an indirect mkey. The RegEx which only accepts one mkey can > now process the whole scattered mbuf. I would add "in one operation." >=20 > The maximum scattered mbuf can be supported in one UMR WQE is now > defined as 64. Multiple operations scattered mbufs can be add to one UMR Typos: "THE multiple", "added" I would reword - "The mbufs from multiple operations can be combined into=20 one UMR. Also, I would add few words what UMR is. > WQE if there is enough space in the KLM array, since the operations can > address their own mbuf's content by the mkey's address and length. > However, one operation's scattered mbuf's can't be placed in two differen= t > UMR WQE's KLM array, if the UMR WQE's KLM does not has enough free > space for one operation, a new UMR WQE will be required. I would say "the extra UMR WQE will be engaged" >=20 > In case the UMR WQE's indirect mkey will be over wrapped by the SQ's WQE > move, the meky's index used by the UMR WQE should be the index of last typo: "meky" > the RegEX WQE in the operations. As one operation consumes one WQE set, > build the RegEx WQE by reverse helps address the mkey more efficiently. typo: TO address With best regards, Slava > Once the operations in one burst consumes multiple mkeys, when the mkey > KLM array is full, the reverse WQE set index will always be the last of t= he new > mkey's for the new UMR WQE. >=20 > In GGA mode, the SQ WQE's memory layout becomes UMR/NOP and RegEx > WQE by interleave. The UMR and RegEx WQE can be called as WQE set. The > SQ's pi and ci will also be increased as WQE set not as WQE. >=20 > For operations don't have scattered mbuf, uses the mbuf's mkey directly, > the WQE set combination is NOP + RegEx. > For operations have scattered mubf but share the UMR WQE with others, > the WQE set combination is NOP + RegEx. > For operations complete the UMR WQE, the WQE set combination is UMR + > RegEx. >=20 > Signed-off-by: Suanming Mou > Acked-by: Ori Kam > --- > doc/guides/regexdevs/mlx5.rst | 5 + > doc/guides/rel_notes/release_21_05.rst | 4 + > drivers/regex/mlx5/mlx5_regex.c | 9 + > drivers/regex/mlx5/mlx5_regex.h | 26 +- > drivers/regex/mlx5/mlx5_regex_control.c | 43 ++- > drivers/regex/mlx5/mlx5_regex_fastpath.c | 378 > +++++++++++++++++++++-- > 6 files changed, 407 insertions(+), 58 deletions(-) >=20 > diff --git a/doc/guides/regexdevs/mlx5.rst b/doc/guides/regexdevs/mlx5.rs= t > index faaa6ac11d..45a0b96980 100644 > --- a/doc/guides/regexdevs/mlx5.rst > +++ b/doc/guides/regexdevs/mlx5.rst > @@ -35,6 +35,11 @@ be specified as device parameter. The RegEx device > can be probed and used with other Mellanox devices, by adding more > options in the class. > For example: ``class=3Dnet:regex`` will probe both the net PMD and the R= egEx > PMD. >=20 > +Features > +-------- > + > +- Multi segments mbuf support. > + > Supported NICs > -------------- >=20 > diff --git a/doc/guides/rel_notes/release_21_05.rst > b/doc/guides/rel_notes/release_21_05.rst > index 3c76148b11..c3d6b8e8ae 100644 > --- a/doc/guides/rel_notes/release_21_05.rst > +++ b/doc/guides/rel_notes/release_21_05.rst > @@ -119,6 +119,10 @@ New Features > * Added command to display Rx queue used descriptor count. > ``show port (port_id) rxq (queue_id) desc used count`` >=20 > +* **Updated Mellanox RegEx PMD.** > + > + * Added support for multi segments mbuf. > + >=20 > Removed Items > ------------- > diff --git a/drivers/regex/mlx5/mlx5_regex.c > b/drivers/regex/mlx5/mlx5_regex.c index ac5b205fa9..82c485e50c 100644 > --- a/drivers/regex/mlx5/mlx5_regex.c > +++ b/drivers/regex/mlx5/mlx5_regex.c > @@ -199,6 +199,13 @@ mlx5_regex_pci_probe(struct rte_pci_driver > *pci_drv __rte_unused, > } > priv->regexdev->dev_ops =3D &mlx5_regexdev_ops; > priv->regexdev->enqueue =3D mlx5_regexdev_enqueue; > +#ifdef HAVE_MLX5_UMR_IMKEY > + if (!attr.umr_indirect_mkey_disabled && > + !attr.umr_modify_entity_size_disabled) > + priv->has_umr =3D 1; > + if (priv->has_umr) > + priv->regexdev->enqueue =3D mlx5_regexdev_enqueue_gga; > #endif > priv->regexdev->dequeue =3D mlx5_regexdev_dequeue; > priv->regexdev->device =3D (struct rte_device *)pci_dev; > priv->regexdev->data->dev_private =3D priv; @@ -213,6 +220,8 @@ > mlx5_regex_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, > rte_errno =3D ENOMEM; > goto error; > } > + DRV_LOG(INFO, "RegEx GGA is %s.", > + priv->has_umr ? "supported" : "unsupported"); > return 0; >=20 > error: > diff --git a/drivers/regex/mlx5/mlx5_regex.h > b/drivers/regex/mlx5/mlx5_regex.h index a2b3f0d9f3..51a2101e53 100644 > --- a/drivers/regex/mlx5/mlx5_regex.h > +++ b/drivers/regex/mlx5/mlx5_regex.h > @@ -15,6 +15,7 @@ > #include >=20 > #include "mlx5_rxp.h" > +#include "mlx5_regex_utils.h" >=20 > struct mlx5_regex_sq { > uint16_t log_nb_desc; /* Log 2 number of desc for this object. */ > @@ -40,6 +41,7 @@ struct mlx5_regex_qp { > struct mlx5_regex_job *jobs; > struct ibv_mr *metadata; > struct ibv_mr *outputs; > + struct ibv_mr *imkey_addr; /* Indirect mkey array region. */ > size_t ci, pi; > struct mlx5_mr_ctrl mr_ctrl; > }; > @@ -71,8 +73,29 @@ struct mlx5_regex_priv { > struct mlx5_mr_share_cache mr_scache; /* Global shared MR cache. > */ > uint8_t is_bf2; /* The device is BF2 device. */ > uint8_t sq_ts_format; /* Whether SQ supports timestamp formats. > */ > + uint8_t has_umr; /* The device supports UMR. */ > }; >=20 > +#ifdef HAVE_IBV_FLOW_DV_SUPPORT > +static inline int > +regex_get_pdn(void *pd, uint32_t *pdn) > +{ > + struct mlx5dv_obj obj; > + struct mlx5dv_pd pd_info; > + int ret =3D 0; > + > + obj.pd.in =3D pd; > + obj.pd.out =3D &pd_info; > + ret =3D mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD); > + if (ret) { > + DRV_LOG(DEBUG, "Fail to get PD object info"); > + return ret; > + } > + *pdn =3D pd_info.pdn; > + return 0; > +} > +#endif > + > /* mlx5_regex.c */ > int mlx5_regex_start(struct rte_regexdev *dev); int > mlx5_regex_stop(struct rte_regexdev *dev); @@ -108,5 +131,6 @@ > uint16_t mlx5_regexdev_enqueue(struct rte_regexdev *dev, uint16_t > qp_id, > struct rte_regex_ops **ops, uint16_t nb_ops); uint16_t > mlx5_regexdev_dequeue(struct rte_regexdev *dev, uint16_t qp_id, > struct rte_regex_ops **ops, uint16_t nb_ops); > - > +uint16_t mlx5_regexdev_enqueue_gga(struct rte_regexdev *dev, uint16_t > qp_id, > + struct rte_regex_ops **ops, uint16_t nb_ops); > #endif /* MLX5_REGEX_H */ > diff --git a/drivers/regex/mlx5/mlx5_regex_control.c > b/drivers/regex/mlx5/mlx5_regex_control.c > index 55fbb419ed..eef0fe579d 100644 > --- a/drivers/regex/mlx5/mlx5_regex_control.c > +++ b/drivers/regex/mlx5/mlx5_regex_control.c > @@ -27,6 +27,9 @@ >=20 > #define MLX5_REGEX_NUM_WQE_PER_PAGE (4096/64) >=20 > +#define MLX5_REGEX_WQE_LOG_NUM(has_umr, log_desc) \ > + ((has_umr) ? ((log_desc) + 2) : (log_desc)) > + > /** > * Returns the number of qp obj to be created. > * > @@ -91,26 +94,6 @@ regex_ctrl_create_cq(struct mlx5_regex_priv *priv, > struct mlx5_regex_cq *cq) > return 0; > } >=20 > -#ifdef HAVE_IBV_FLOW_DV_SUPPORT > -static int > -regex_get_pdn(void *pd, uint32_t *pdn) > -{ > - struct mlx5dv_obj obj; > - struct mlx5dv_pd pd_info; > - int ret =3D 0; > - > - obj.pd.in =3D pd; > - obj.pd.out =3D &pd_info; > - ret =3D mlx5_glue->dv_init_obj(&obj, MLX5DV_OBJ_PD); > - if (ret) { > - DRV_LOG(DEBUG, "Fail to get PD object info"); > - return ret; > - } > - *pdn =3D pd_info.pdn; > - return 0; > -} > -#endif > - > /** > * Destroy the SQ object. > * > @@ -168,14 +151,16 @@ regex_ctrl_create_sq(struct mlx5_regex_priv *priv, > struct mlx5_regex_qp *qp, > int ret; >=20 > sq->log_nb_desc =3D log_nb_desc; > + sq->sqn =3D q_ind; > sq->ci =3D 0; > sq->pi =3D 0; > ret =3D regex_get_pdn(priv->pd, &pd_num); > if (ret) > return ret; > attr.wq_attr.pd =3D pd_num; > - ret =3D mlx5_devx_sq_create(priv->ctx, &sq->sq_obj, log_nb_desc, > &attr, > - SOCKET_ID_ANY); > + ret =3D mlx5_devx_sq_create(priv->ctx, &sq->sq_obj, > + MLX5_REGEX_WQE_LOG_NUM(priv->has_umr, > log_nb_desc), > + &attr, SOCKET_ID_ANY); > if (ret) { > DRV_LOG(ERR, "Can't create SQ object."); > rte_errno =3D ENOMEM; > @@ -225,10 +210,18 @@ mlx5_regex_qp_setup(struct rte_regexdev *dev, > uint16_t qp_ind, >=20 > qp =3D &priv->qps[qp_ind]; > qp->flags =3D cfg->qp_conf_flags; > - qp->cq.log_nb_desc =3D rte_log2_u32(cfg->nb_desc); > - qp->nb_desc =3D 1 << qp->cq.log_nb_desc; > + log_desc =3D rte_log2_u32(cfg->nb_desc); > + /* > + * UMR mode requires two WQEs(UMR and RegEx WQE) for one > descriptor. > + * For CQ, expand the CQE number multiple with 2. > + * For SQ, the UMR and RegEx WQE for one descriptor consumes 4 > WQEBBS, > + * expand the WQE number multiple with 4. > + */ > + qp->cq.log_nb_desc =3D log_desc + (!!priv->has_umr); > + qp->nb_desc =3D 1 << log_desc; > if (qp->flags & RTE_REGEX_QUEUE_PAIR_CFG_OOS_F) > - qp->nb_obj =3D regex_ctrl_get_nb_obj(qp->nb_desc); > + qp->nb_obj =3D regex_ctrl_get_nb_obj > + (1 << MLX5_REGEX_WQE_LOG_NUM(priv- > >has_umr, log_desc)); > else > qp->nb_obj =3D 1; > qp->sqs =3D rte_malloc(NULL, > diff --git a/drivers/regex/mlx5/mlx5_regex_fastpath.c > b/drivers/regex/mlx5/mlx5_regex_fastpath.c > index beaea7b63f..4f9402c583 100644 > --- a/drivers/regex/mlx5/mlx5_regex_fastpath.c > +++ b/drivers/regex/mlx5/mlx5_regex_fastpath.c > @@ -32,6 +32,15 @@ > #define MLX5_REGEX_WQE_GATHER_OFFSET 32 #define > MLX5_REGEX_WQE_SCATTER_OFFSET 48 #define > MLX5_REGEX_METADATA_OFF 32 > +#define MLX5_REGEX_UMR_WQE_SIZE 192 > +/* The maximum KLMs can be added to one UMR indirect mkey. */ #define > +MLX5_REGEX_MAX_KLM_NUM 128 > +/* The KLM array size for one job. */ > +#define MLX5_REGEX_KLMS_SIZE \ > + ((MLX5_REGEX_MAX_KLM_NUM) * sizeof(struct mlx5_klm)) > +/* In WQE set mode, the pi should be quarter of the > +MLX5_REGEX_MAX_WQE_INDEX. */ #define > MLX5_REGEX_UMR_SQ_PI_IDX(pi, ops) \ > + (((pi) + (ops)) & (MLX5_REGEX_MAX_WQE_INDEX >> 2)) >=20 > static inline uint32_t > sq_size_get(struct mlx5_regex_sq *sq) > @@ -49,6 +58,8 @@ struct mlx5_regex_job { > uint64_t user_id; > volatile uint8_t *output; > volatile uint8_t *metadata; > + struct mlx5_klm *imkey_array; /* Indirect mkey's KLM array. */ > + struct mlx5_devx_obj *imkey; /* UMR WQE's indirect meky. */ > } __rte_cached_aligned; >=20 > static inline void > @@ -99,12 +110,13 @@ set_wqe_ctrl_seg(struct mlx5_wqe_ctrl_seg *seg, > uint16_t pi, uint8_t opcode, } >=20 > static inline void > -prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp, > - struct mlx5_regex_sq *sq, struct rte_regex_ops *op, > - struct mlx5_regex_job *job) > +__prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_sq *sq, > + struct rte_regex_ops *op, struct mlx5_regex_job *job, > + size_t pi, struct mlx5_klm *klm) > { > - size_t wqe_offset =3D (sq->pi & (sq_size_get(sq) - 1)) * > MLX5_SEND_WQE_BB; > - uint32_t lkey; > + size_t wqe_offset =3D (pi & (sq_size_get(sq) - 1)) * > + (MLX5_SEND_WQE_BB << (priv->has_umr ? 2 : 0)) > + > + (priv->has_umr ? MLX5_REGEX_UMR_WQE_SIZE : > 0); > uint16_t group0 =3D op->req_flags & > RTE_REGEX_OPS_REQ_GROUP_ID0_VALID_F ? > op->group_id0 : 0; > uint16_t group1 =3D op->req_flags & > RTE_REGEX_OPS_REQ_GROUP_ID1_VALID_F ? > @@ -122,14 +134,11 @@ prep_one(struct mlx5_regex_priv *priv, struct > mlx5_regex_qp *qp, > RTE_REGEX_OPS_REQ_GROUP_ID2_VALID_F | > RTE_REGEX_OPS_REQ_GROUP_ID3_VALID_F))) > group0 =3D op->group_id0; > - lkey =3D mlx5_mr_addr2mr_bh(priv->pd, 0, > - &priv->mr_scache, &qp->mr_ctrl, > - rte_pktmbuf_mtod(op->mbuf, uintptr_t), > - !!(op->mbuf->ol_flags & > EXT_ATTACHED_MBUF)); > uint8_t *wqe =3D (uint8_t *)(uintptr_t)sq->sq_obj.wqes + wqe_offset; > int ds =3D 4; /* ctrl + meta + input + output */ >=20 > - set_wqe_ctrl_seg((struct mlx5_wqe_ctrl_seg *)wqe, sq->pi, > + set_wqe_ctrl_seg((struct mlx5_wqe_ctrl_seg *)wqe, > + (priv->has_umr ? (pi * 4 + 3) : pi), > MLX5_OPCODE_MMO, > MLX5_OPC_MOD_MMO_REGEX, > sq->sq_obj.sq->id, 0, ds, 0, 0); > set_regex_ctrl_seg(wqe + 12, 0, group0, group1, group2, group3, > @@ -137,36 +146,54 @@ prep_one(struct mlx5_regex_priv *priv, struct > mlx5_regex_qp *qp, > struct mlx5_wqe_data_seg *input_seg =3D > (struct mlx5_wqe_data_seg *)(wqe + >=20 > MLX5_REGEX_WQE_GATHER_OFFSET); > - input_seg->byte_count =3D > - rte_cpu_to_be_32(rte_pktmbuf_data_len(op->mbuf)); > - input_seg->addr =3D rte_cpu_to_be_64(rte_pktmbuf_mtod(op- > >mbuf, > - uintptr_t)); > - input_seg->lkey =3D lkey; > + input_seg->byte_count =3D rte_cpu_to_be_32(klm->byte_count); > + input_seg->addr =3D rte_cpu_to_be_64(klm->address); > + input_seg->lkey =3D klm->mkey; > job->user_id =3D op->user_id; > +} > + > +static inline void > +prep_one(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp, > + struct mlx5_regex_sq *sq, struct rte_regex_ops *op, > + struct mlx5_regex_job *job) > +{ > + struct mlx5_klm klm; > + > + klm.byte_count =3D rte_pktmbuf_data_len(op->mbuf); > + klm.mkey =3D mlx5_mr_addr2mr_bh(priv->pd, 0, > + &priv->mr_scache, &qp->mr_ctrl, > + rte_pktmbuf_mtod(op->mbuf, uintptr_t), > + !!(op->mbuf->ol_flags & > EXT_ATTACHED_MBUF)); > + klm.address =3D rte_pktmbuf_mtod(op->mbuf, uintptr_t); > + __prep_one(priv, sq, op, job, sq->pi, &klm); > sq->db_pi =3D sq->pi; > sq->pi =3D (sq->pi + 1) & MLX5_REGEX_MAX_WQE_INDEX; } >=20 > static inline void > -send_doorbell(struct mlx5dv_devx_uar *uar, struct mlx5_regex_sq *sq) > +send_doorbell(struct mlx5_regex_priv *priv, struct mlx5_regex_sq *sq) > { > + struct mlx5dv_devx_uar *uar =3D priv->uar; > size_t wqe_offset =3D (sq->db_pi & (sq_size_get(sq) - 1)) * > - MLX5_SEND_WQE_BB; > + (MLX5_SEND_WQE_BB << (priv->has_umr ? 2 : 0)) + > + (priv->has_umr ? MLX5_REGEX_UMR_WQE_SIZE : 0); > uint8_t *wqe =3D (uint8_t *)(uintptr_t)sq->sq_obj.wqes + wqe_offset; > - ((struct mlx5_wqe_ctrl_seg *)wqe)->fm_ce_se =3D > MLX5_WQE_CTRL_CQ_UPDATE; > + /* Or the fm_ce_se instead of set, avoid the fence be cleared. */ > + ((struct mlx5_wqe_ctrl_seg *)wqe)->fm_ce_se |=3D > +MLX5_WQE_CTRL_CQ_UPDATE; > uint64_t *doorbell_addr =3D > (uint64_t *)((uint8_t *)uar->base_addr + 0x800); > rte_io_wmb(); > - sq->sq_obj.db_rec[MLX5_SND_DBR] =3D rte_cpu_to_be_32((sq- > >db_pi + 1) & > - > MLX5_REGEX_MAX_WQE_INDEX); > + sq->sq_obj.db_rec[MLX5_SND_DBR] =3D rte_cpu_to_be_32((priv- > >has_umr ? > + (sq->db_pi * 4 + 3) : sq->db_pi) & > + MLX5_REGEX_MAX_WQE_INDEX); > rte_wmb(); > *doorbell_addr =3D *(volatile uint64_t *)wqe; > rte_wmb(); > } >=20 > static inline int > -can_send(struct mlx5_regex_sq *sq) { > - return ((uint16_t)(sq->pi - sq->ci) < sq_size_get(sq)); > +get_free(struct mlx5_regex_sq *sq) { > + return (sq_size_get(sq) - (uint16_t)(sq->pi - sq->ci)); > } >=20 > static inline uint32_t > @@ -174,6 +201,211 @@ job_id_get(uint32_t qid, size_t sq_size, size_t > index) { > return qid * sq_size + (index & (sq_size - 1)); } >=20 > +#ifdef HAVE_MLX5_UMR_IMKEY > +static inline int > +mkey_klm_available(struct mlx5_klm *klm, uint32_t pos, uint32_t new) { > + return (klm && ((pos + new) <=3D MLX5_REGEX_MAX_KLM_NUM)); } > + > +static inline void > +complete_umr_wqe(struct mlx5_regex_qp *qp, struct mlx5_regex_sq *sq, > + struct mlx5_regex_job *mkey_job, > + size_t umr_index, uint32_t klm_size, uint32_t total_len) { > + size_t wqe_offset =3D (umr_index & (sq_size_get(sq) - 1)) * > + (MLX5_SEND_WQE_BB * 4); > + struct mlx5_wqe_ctrl_seg *wqe =3D (struct mlx5_wqe_ctrl_seg > *)((uint8_t *) > + (uintptr_t)sq->sq_obj.wqes + wqe_offset); > + struct mlx5_wqe_umr_ctrl_seg *ucseg =3D > + (struct mlx5_wqe_umr_ctrl_seg *)(wqe + 1); > + struct mlx5_wqe_mkey_context_seg *mkc =3D > + (struct mlx5_wqe_mkey_context_seg > *)(ucseg + 1); > + struct mlx5_klm *iklm =3D (struct mlx5_klm *)(mkc + 1); > + uint16_t klm_align =3D RTE_ALIGN(klm_size, 4); > + > + memset(wqe, 0, MLX5_REGEX_UMR_WQE_SIZE); > + /* Set WQE control seg. Non-inline KLM UMR WQE size must be 9 > WQE_DS. */ > + set_wqe_ctrl_seg(wqe, (umr_index * 4), MLX5_OPCODE_UMR, > + 0, sq->sq_obj.sq->id, 0, 9, 0, > + rte_cpu_to_be_32(mkey_job->imkey->id)); > + /* Set UMR WQE control seg. */ > + ucseg->mkey_mask |=3D > rte_cpu_to_be_64(MLX5_WQE_UMR_CTRL_MKEY_MASK_LEN | > + > MLX5_WQE_UMR_CTRL_FLAG_TRNSLATION_OFFSET | > + > MLX5_WQE_UMR_CTRL_MKEY_MASK_ACCESS_LOCAL_WRITE); > + ucseg->klm_octowords =3D rte_cpu_to_be_16(klm_align); > + /* Set mkey context seg. */ > + mkc->len =3D rte_cpu_to_be_64(total_len); > + mkc->qpn_mkey =3D rte_cpu_to_be_32(0xffffff00 | > + (mkey_job->imkey->id & 0xff)); > + /* Set UMR pointer to data seg. */ > + iklm->address =3D rte_cpu_to_be_64 > + ((uintptr_t)((char *)mkey_job- > >imkey_array)); > + iklm->mkey =3D rte_cpu_to_be_32(qp->imkey_addr->lkey); > + iklm->byte_count =3D rte_cpu_to_be_32(klm_align); > + /* Clear the padding memory. */ > + memset((uint8_t *)&mkey_job->imkey_array[klm_size], 0, > + sizeof(struct mlx5_klm) * (klm_align - klm_size)); > + > + /* Add the following RegEx WQE with fence. */ > + wqe =3D (struct mlx5_wqe_ctrl_seg *) > + (((uint8_t *)wqe) + > MLX5_REGEX_UMR_WQE_SIZE); > + wqe->fm_ce_se |=3D MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE; > +} > + > +static inline void > +prep_nop_regex_wqe_set(struct mlx5_regex_priv *priv, struct > mlx5_regex_sq *sq, > + struct rte_regex_ops *op, struct mlx5_regex_job *job, > + size_t pi, struct mlx5_klm *klm) { > + size_t wqe_offset =3D (pi & (sq_size_get(sq) - 1)) * > + (MLX5_SEND_WQE_BB << 2); > + struct mlx5_wqe_ctrl_seg *wqe =3D (struct mlx5_wqe_ctrl_seg > *)((uint8_t *) > + (uintptr_t)sq->sq_obj.wqes + wqe_offset); > + > + /* Clear the WQE memory used as UMR WQE previously. */ > + if ((rte_be_to_cpu_32(wqe->opmod_idx_opcode) & 0xff) !=3D > MLX5_OPCODE_NOP) > + memset(wqe, 0, MLX5_REGEX_UMR_WQE_SIZE); > + /* UMR WQE size is 9 DS, align nop WQE to 3 WQEBBS(12 DS). */ > + set_wqe_ctrl_seg(wqe, pi * 4, MLX5_OPCODE_NOP, 0, sq- > >sq_obj.sq->id, > + 0, 12, 0, 0); > + __prep_one(priv, sq, op, job, pi, klm); } > + > +static inline void > +prep_regex_umr_wqe_set(struct mlx5_regex_priv *priv, struct > mlx5_regex_qp *qp, > + struct mlx5_regex_sq *sq, struct rte_regex_ops **op, size_t > nb_ops) { > + struct mlx5_regex_job *job =3D NULL; > + size_t sqid =3D sq->sqn, mkey_job_id =3D 0; > + size_t left_ops =3D nb_ops; > + uint32_t klm_num =3D 0, len; > + struct mlx5_klm *mkey_klm =3D NULL; > + struct mlx5_klm klm; > + > + sqid =3D sq->sqn; > + while (left_ops--) > + rte_prefetch0(op[left_ops]); > + left_ops =3D nb_ops; > + /* > + * Build the WQE set by reverse. In case the burst may consume > + * multiple mkeys, build the WQE set as normal will hard to > + * address the last mkey index, since we will only know the last > + * RegEx WQE's index when finishes building. > + */ > + while (left_ops--) { > + struct rte_mbuf *mbuf =3D op[left_ops]->mbuf; > + size_t pi =3D MLX5_REGEX_UMR_SQ_PI_IDX(sq->pi, left_ops); > + > + if (mbuf->nb_segs > 1) { > + size_t scatter_size =3D 0; > + > + if (!mkey_klm_available(mkey_klm, klm_num, > + mbuf->nb_segs)) { > + /* > + * The mkey's KLM is full, create the UMR > + * WQE in the next WQE set. > + */ > + if (mkey_klm) > + complete_umr_wqe(qp, sq, > + &qp->jobs[mkey_job_id], > + > MLX5_REGEX_UMR_SQ_PI_IDX(pi, 1), > + klm_num, len); > + /* > + * Get the indircet mkey and KLM array index > + * from the last WQE set. > + */ > + mkey_job_id =3D job_id_get(sqid, > + sq_size_get(sq), pi); > + mkey_klm =3D qp- > >jobs[mkey_job_id].imkey_array; > + klm_num =3D 0; > + len =3D 0; > + } > + /* Build RegEx WQE's data segment KLM. */ > + klm.address =3D len; > + klm.mkey =3D rte_cpu_to_be_32 > + (qp->jobs[mkey_job_id].imkey->id); > + while (mbuf) { > + /* Build indirect mkey seg's KLM. */ > + mkey_klm->mkey =3D > mlx5_mr_addr2mr_bh(priv->pd, > + NULL, &priv->mr_scache, &qp- > >mr_ctrl, > + rte_pktmbuf_mtod(mbuf, uintptr_t), > + !!(mbuf->ol_flags & > EXT_ATTACHED_MBUF)); > + mkey_klm->address =3D rte_cpu_to_be_64 > + (rte_pktmbuf_mtod(mbuf, > uintptr_t)); > + mkey_klm->byte_count =3D rte_cpu_to_be_32 > + > (rte_pktmbuf_data_len(mbuf)); > + /* > + * Save the mbuf's total size for RegEx data > + * segment. > + */ > + scatter_size +=3D > rte_pktmbuf_data_len(mbuf); > + mkey_klm++; > + klm_num++; > + mbuf =3D mbuf->next; > + } > + len +=3D scatter_size; > + klm.byte_count =3D scatter_size; > + } else { > + /* The single mubf case. Build the KLM directly. */ > + klm.mkey =3D mlx5_mr_addr2mr_bh(priv->pd, NULL, > + &priv->mr_scache, &qp->mr_ctrl, > + rte_pktmbuf_mtod(mbuf, uintptr_t), > + !!(mbuf->ol_flags & > EXT_ATTACHED_MBUF)); > + klm.address =3D rte_pktmbuf_mtod(mbuf, uintptr_t); > + klm.byte_count =3D rte_pktmbuf_data_len(mbuf); > + } > + job =3D &qp->jobs[job_id_get(sqid, sq_size_get(sq), pi)]; > + /* > + * Build the nop + RegEx WQE set by default. The fist nop > WQE > + * will be updated later as UMR WQE if scattered mubf exist. > + */ > + prep_nop_regex_wqe_set(priv, sq, op[left_ops], job, pi, > &klm); > + } > + /* > + * Scattered mbuf have been added to the KLM array. Complete the > build > + * of UMR WQE, update the first nop WQE as UMR WQE. > + */ > + if (mkey_klm) > + complete_umr_wqe(qp, sq, &qp->jobs[mkey_job_id], sq- > >pi, > + klm_num, len); > + sq->db_pi =3D MLX5_REGEX_UMR_SQ_PI_IDX(sq->pi, nb_ops - 1); > + sq->pi =3D MLX5_REGEX_UMR_SQ_PI_IDX(sq->pi, nb_ops); } > + > +uint16_t > +mlx5_regexdev_enqueue_gga(struct rte_regexdev *dev, uint16_t qp_id, > + struct rte_regex_ops **ops, uint16_t nb_ops) { > + struct mlx5_regex_priv *priv =3D dev->data->dev_private; > + struct mlx5_regex_qp *queue =3D &priv->qps[qp_id]; > + struct mlx5_regex_sq *sq; > + size_t sqid, nb_left =3D nb_ops, nb_desc; > + > + while ((sqid =3D ffs(queue->free_sqs))) { > + sqid--; /* ffs returns 1 for bit 0 */ > + sq =3D &queue->sqs[sqid]; > + nb_desc =3D get_free(sq); > + if (nb_desc) { > + /* The ops be handled can't exceed nb_ops. */ > + if (nb_desc > nb_left) > + nb_desc =3D nb_left; > + else > + queue->free_sqs &=3D ~(1 << sqid); > + prep_regex_umr_wqe_set(priv, queue, sq, ops, > nb_desc); > + send_doorbell(priv, sq); > + nb_left -=3D nb_desc; > + } > + if (!nb_left) > + break; > + ops +=3D nb_desc; > + } > + nb_ops -=3D nb_left; > + queue->pi +=3D nb_ops; > + return nb_ops; > +} > +#endif > + > uint16_t > mlx5_regexdev_enqueue(struct rte_regexdev *dev, uint16_t qp_id, > struct rte_regex_ops **ops, uint16_t nb_ops) @@ - > 186,17 +418,17 @@ mlx5_regexdev_enqueue(struct rte_regexdev *dev, > uint16_t qp_id, > while ((sqid =3D ffs(queue->free_sqs))) { > sqid--; /* ffs returns 1 for bit 0 */ > sq =3D &queue->sqs[sqid]; > - while (can_send(sq)) { > + while (get_free(sq)) { > job_id =3D job_id_get(sqid, sq_size_get(sq), sq->pi); > prep_one(priv, queue, sq, ops[i], &queue- > >jobs[job_id]); > i++; > if (unlikely(i =3D=3D nb_ops)) { > - send_doorbell(priv->uar, sq); > + send_doorbell(priv, sq); > goto out; > } > } > queue->free_sqs &=3D ~(1 << sqid); > - send_doorbell(priv->uar, sq); > + send_doorbell(priv, sq); > } >=20 > out: > @@ -308,6 +540,10 @@ mlx5_regexdev_dequeue(struct rte_regexdev > *dev, uint16_t qp_id, > MLX5_REGEX_MAX_WQE_INDEX; > size_t sqid =3D cqe->rsvd3[2]; > struct mlx5_regex_sq *sq =3D &queue->sqs[sqid]; > + > + /* UMR mode WQE counter move as WQE set(4 WQEBBS).*/ > + if (priv->has_umr) > + wq_counter >>=3D 2; > while (sq->ci !=3D wq_counter) { > if (unlikely(i =3D=3D nb_ops)) { > /* Return without updating cq->ci */ @@ - > 316,7 +552,9 @@ mlx5_regexdev_dequeue(struct rte_regexdev *dev, > uint16_t qp_id, > uint32_t job_id =3D job_id_get(sqid, sq_size_get(sq), > sq->ci); > extract_result(ops[i], &queue->jobs[job_id]); > - sq->ci =3D (sq->ci + 1) & > MLX5_REGEX_MAX_WQE_INDEX; > + sq->ci =3D (sq->ci + 1) & (priv->has_umr ? > + (MLX5_REGEX_MAX_WQE_INDEX >> 2) : > + MLX5_REGEX_MAX_WQE_INDEX); > i++; > } > cq->ci =3D (cq->ci + 1) & 0xffffff; > @@ -331,7 +569,7 @@ mlx5_regexdev_dequeue(struct rte_regexdev *dev, > uint16_t qp_id, } >=20 > static void > -setup_sqs(struct mlx5_regex_qp *queue) > +setup_sqs(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *queue) > { > size_t sqid, entry; > uint32_t job_id; > @@ -342,6 +580,14 @@ setup_sqs(struct mlx5_regex_qp *queue) > job_id =3D sqid * sq_size_get(sq) + entry; > struct mlx5_regex_job *job =3D &queue->jobs[job_id]; >=20 > + /* Fill UMR WQE with NOP in advanced. */ > + if (priv->has_umr) { > + set_wqe_ctrl_seg > + ((struct mlx5_wqe_ctrl_seg *)wqe, > + entry * 2, MLX5_OPCODE_NOP, 0, > + sq->sq_obj.sq->id, 0, 12, 0, 0); > + wqe +=3D MLX5_REGEX_UMR_WQE_SIZE; > + } > set_metadata_seg((struct mlx5_wqe_metadata_seg > *) > (wqe + > MLX5_REGEX_WQE_METADATA_OFFSET), > 0, queue->metadata->lkey, > @@ -358,8 +604,9 @@ setup_sqs(struct mlx5_regex_qp *queue) } >=20 > static int > -setup_buffers(struct mlx5_regex_qp *qp, struct ibv_pd *pd) > +setup_buffers(struct mlx5_regex_priv *priv, struct mlx5_regex_qp *qp) > { > + struct ibv_pd *pd =3D priv->pd; > uint32_t i; > int err; >=20 > @@ -395,6 +642,24 @@ setup_buffers(struct mlx5_regex_qp *qp, struct > ibv_pd *pd) > goto err_output; > } >=20 > + if (priv->has_umr) { > + ptr =3D rte_calloc(__func__, qp->nb_desc, > MLX5_REGEX_KLMS_SIZE, > + MLX5_REGEX_KLMS_SIZE); > + if (!ptr) { > + err =3D -ENOMEM; > + goto err_imkey; > + } > + qp->imkey_addr =3D mlx5_glue->reg_mr(pd, ptr, > + MLX5_REGEX_KLMS_SIZE * qp- > >nb_desc, > + IBV_ACCESS_LOCAL_WRITE); > + if (!qp->imkey_addr) { > + rte_free(ptr); > + DRV_LOG(ERR, "Failed to register output"); > + err =3D -EINVAL; > + goto err_imkey; > + } > + } > + > /* distribute buffers to jobs */ > for (i =3D 0; i < qp->nb_desc; i++) { > qp->jobs[i].output =3D > @@ -403,9 +668,18 @@ setup_buffers(struct mlx5_regex_qp *qp, struct > ibv_pd *pd) > qp->jobs[i].metadata =3D > (uint8_t *)qp->metadata->addr + > (i % qp->nb_desc) * MLX5_REGEX_METADATA_SIZE; > + if (qp->imkey_addr) > + qp->jobs[i].imkey_array =3D (struct mlx5_klm *) > + qp->imkey_addr->addr + > + (i % qp->nb_desc) * > MLX5_REGEX_MAX_KLM_NUM; > } > + > return 0; >=20 > +err_imkey: > + ptr =3D qp->outputs->addr; > + rte_free(ptr); > + mlx5_glue->dereg_mr(qp->outputs); > err_output: > ptr =3D qp->metadata->addr; > rte_free(ptr); > @@ -417,23 +691,57 @@ int > mlx5_regexdev_setup_fastpath(struct mlx5_regex_priv *priv, uint32_t > qp_id) { > struct mlx5_regex_qp *qp =3D &priv->qps[qp_id]; > - int err; > + struct mlx5_klm klm =3D { 0 }; > + struct mlx5_devx_mkey_attr attr =3D { > + .klm_array =3D &klm, > + .klm_num =3D 1, > + .umr_en =3D 1, > + }; > + uint32_t i; > + int err =3D 0; >=20 > qp->jobs =3D rte_calloc(__func__, qp->nb_desc, sizeof(*qp->jobs), > 64); > if (!qp->jobs) > return -ENOMEM; > - err =3D setup_buffers(qp, priv->pd); > + err =3D setup_buffers(priv, qp); > if (err) { > rte_free(qp->jobs); > return err; > } > - setup_sqs(qp); > - return 0; > + > + setup_sqs(priv, qp); > + > + if (priv->has_umr) { > +#ifdef HAVE_IBV_FLOW_DV_SUPPORT > + if (regex_get_pdn(priv->pd, &attr.pd)) { > + err =3D -rte_errno; > + DRV_LOG(ERR, "Failed to get pdn."); > + mlx5_regexdev_teardown_fastpath(priv, qp_id); > + return err; > + } > +#endif > + for (i =3D 0; i < qp->nb_desc; i++) { > + attr.klm_num =3D MLX5_REGEX_MAX_KLM_NUM; > + attr.klm_array =3D qp->jobs[i].imkey_array; > + qp->jobs[i].imkey =3D > mlx5_devx_cmd_mkey_create(priv->ctx, > + &attr); > + if (!qp->jobs[i].imkey) { > + err =3D -rte_errno; > + DRV_LOG(ERR, "Failed to allocate imkey."); > + mlx5_regexdev_teardown_fastpath(priv, > qp_id); > + } > + } > + } > + return err; > } >=20 > static void > free_buffers(struct mlx5_regex_qp *qp) > { > + if (qp->imkey_addr) { > + mlx5_glue->dereg_mr(qp->imkey_addr); > + rte_free(qp->imkey_addr->addr); > + } > if (qp->metadata) { > mlx5_glue->dereg_mr(qp->metadata); > rte_free(qp->metadata->addr); > @@ -448,8 +756,14 @@ void > mlx5_regexdev_teardown_fastpath(struct mlx5_regex_priv *priv, uint32_t > qp_id) { > struct mlx5_regex_qp *qp =3D &priv->qps[qp_id]; > + uint32_t i; >=20 > if (qp) { > + for (i =3D 0; i < qp->nb_desc; i++) { > + if (qp->jobs[i].imkey) > + claim_zero(mlx5_devx_cmd_destroy > + (qp->jobs[i].imkey)); > + } > free_buffers(qp); > if (qp->jobs) > rte_free(qp->jobs); > -- > 2.25.1