From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 30D46A0A02; Tue, 6 Apr 2021 11:28:18 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 177C3140FCA; Tue, 6 Apr 2021 11:28:18 +0200 (CEST) Received: from NAM04-CO1-obe.outbound.protection.outlook.com (mail-eopbgr690057.outbound.protection.outlook.com [40.107.69.57]) by mails.dpdk.org (Postfix) with ESMTP id 7DFE5406A2 for ; Tue, 6 Apr 2021 11:28:16 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CDlwRCXVODAUyjwwrMwP++DylUpBPQ+hAhpDMqAUmz+LvHhNbRcj/x7ewlR+xsbhb6O7+y5RHN5H4BfL/0mjFGkpFo4yoSO/LCR/ThTvmOnlTEz1PLaI5Ko6OaxvsEyy/gsjwSkynxKq+wcxmPWwtuwS27De3nrbEUcFxDx94eHFr0Kk/UOOEg2GZKFhXRwsJU1vCkBOTkH31jlKp8I1aSJH9H5XB71BGXILTUizjEyGrm5Y1jMqdS1RKj3qo8BqMBsWJ7qv79wvgn4weeFkg1rEk48bPxcXwOXLO0gLkHy71dRwHWschFZonBe8lgmATI1hKpISSNpXyMb18yMpnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T2uYviqrQrmJ0sn8KTYaNBXKfYAYKA1CDy1PDOWS1+o=; b=RteOYex7ykODAqAUNJKyBdxYA3EioSY6ErS6w2imnVa8Q9aE2Zyuk/daTylaf+nFFgenTBZy43hbON++B0Y0amTV0cKgScfAgi5ODuVMF/Dj6lWjyzY9+/Tig9t+JpqLtCOLD5J3o1cQY9zOkt3RyY6rcZ03bQ71NFiU86gwgKTBvOV+aAguNutqPOWzVVsClCiC7Fb5AsElhlVc859FoOhDUagcicaQObQqdK4h3gq6/1B0RRDtAje7cPF86OnH3FjbaHyBEZv1xYYhhpnFNrR5CJNP+zeXElRk5qs41QezdJza/5seXxmNo4NIuQPZmPNWOa+usUNYwCyGGb7sxw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T2uYviqrQrmJ0sn8KTYaNBXKfYAYKA1CDy1PDOWS1+o=; b=urOjUh23bOcxAfL+5vwfeHYt/O42LrgHpJJ2K2FHJfwXn1YJGUc6bZcK1NvzJGyD7uKfJQ/rvHCOmOUBbWxzO7TyWNFT5bVdmnkql6yTPQiZcbGoJpaQm9uSI6l1aKwUxTRfTZvAChDpfRtRbCAB4ekwJMfnEUOPVCfowRhsztyjpQeJQvqxZVdy0+mBsyV4GJbtHAOAQl3NrXB3ppwr1OhOBn/qO4mCqPT5q+6MF6V/anpNuMatfZDN2lSPoT9RD40Z9lNQ1FliDvA5igee+Id8GQc1T1aOEZLbt8lE9aZsdLYYbUs5bpJa0m+6WNA2HbIR41Uo/RhuIfkxM8EZwQ== Received: from DM6PR12MB3753.namprd12.prod.outlook.com (2603:10b6:5:1c7::18) by DM6PR12MB4481.namprd12.prod.outlook.com (2603:10b6:5:2af::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Tue, 6 Apr 2021 09:28:14 +0000 Received: from DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::20f7:11fc:9d46:9258]) by DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::20f7:11fc:9d46:9258%6]) with mapi id 15.20.3999.032; Tue, 6 Apr 2021 09:28:14 +0000 From: Slava Ovsiienko To: Michael Baum , "dev@dpdk.org" CC: Matan Azrad , Raslan Darawsheh Thread-Topic: [PATCH 3/6] net/mlx5: separate Tx function declarations to another file Thread-Index: AQHXKiQyoMmOtfuBoUWtgMln4L53UqqnOf4Q Date: Tue, 6 Apr 2021 09:28:14 +0000 Message-ID: References: <1617631256-3018-1-git-send-email-michaelba@nvidia.com> <1617631256-3018-4-git-send-email-michaelba@nvidia.com> In-Reply-To: <1617631256-3018-4-git-send-email-michaelba@nvidia.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: nvidia.com; dkim=none (message not signed) header.d=none;nvidia.com; dmarc=none action=none header.from=nvidia.com; x-originating-ip: [95.164.10.10] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1542ccc2-d915-4fcc-ce94-08d8f8de4e01 x-ms-traffictypediagnostic: DM6PR12MB4481: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:549; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 1SytIy3fcSQXlqdvv4MbPiZzJLaXnCf+coy1/+hG0tsnAvsYUXxuY1cw3WvHqekH/MbqXtQAgTJq7B8nmkjkoPn9+brTVGCXj6jsg68IAgFAaMLQHc2wIg2Hu/8KEsxoeonjYhjdeFdB6GBKrdLF/UKSsAgS65/71uia939IiqbtBQQ1O2/YKPaFZGuZHKkVp6wrx7SjyIf56Fg3mmqEE8+baJF5kkzKKsRFrvwHbYSdxI+AZlRBtEpj/y8l14fxcsWxaYQEBJKB2Q2Rmg8sHPRVWdLuKfTYMpf0zfEP9iJaItYqkSQLnBv95drsoKynYmpt0SFP06hkb03prGcG9KRk0PTNnwkfOG8nbXE52cimU8cNPKC+/xxNPYa72KpVVoWu5JIBVn6jB/V5JP33W0UoNF9/AZ6OCmThaFjrNCjJ3dxL8zxxK2E4e5JLC3+YitSJOa1spdx73T+t8KrDPy9qq6apxebDXP3sWmT9sR2cn9+aZemv1rhhe0dsBDpaPc2abOzH95rZZLIjPF1o2vB0EQkRrol3BKjuH8Ukq3/0GOMveNevDukNQKA8lVP/r9IMlVxEfY/ZsxEr3RzjjVs1T6cVDl0HurYODd2pN89EZN3GUDZFsP1q+FWJ1xJ0syecoF4zXOCCQWPXJHwZHVTo90fMtPPOCjqQnEfElS0= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB3753.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(346002)(136003)(376002)(396003)(39860400002)(366004)(76116006)(33656002)(64756008)(54906003)(66446008)(186003)(66556008)(66476007)(4326008)(71200400001)(110136005)(66946007)(5660300002)(7696005)(86362001)(316002)(6506007)(55016002)(30864003)(52536014)(9686003)(38100700001)(83380400001)(8676002)(8936002)(2906002)(26005)(107886003)(53546011)(478600001)(579004)(559001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?B72oS/USnabfvRdg91ZCjcdFRYLeZBFgbkM3anJoCxhDW8hUAzH2/ZThqreV?= =?us-ascii?Q?sPdzuedcTFZh2XBwkn8AoaZmq0yK/SjnbIRtxY5PK2RzMgfbjuha1uB3EHtQ?= =?us-ascii?Q?O5piJCb2akdIwhSfES1zze1AhRmDlxst43p9engMNIGQAXCRG+Eb1XcZMnMN?= =?us-ascii?Q?wLa8324sd5P3ySFVrfI8xA3wcutqZjPlC9t+x+Lfq5cJv/KKl+M0AeIwLP9R?= =?us-ascii?Q?vGFrE8kOCrE/DHEPuxBtU1Dy404OM1NJVvoG48bPhFjKM4gF6snDP7pPLzqj?= =?us-ascii?Q?ycayKQxES0O3k6/5VIhztMPEnbmeqwrm1tZdN488pzpLkom+REaq4Lyi6QmJ?= =?us-ascii?Q?8f5DIbSqWVVPVJaaYz4OTlvXfQ+vMsoLpu5nueiumxpR1L2HVNrixafraXb/?= =?us-ascii?Q?g2BxzXYFmoGgybY3NkwM1LZKa4rnNVoYFvDGYYnlSYbbqWrKkpg9rKpNtUkv?= =?us-ascii?Q?8kjk4QDTICFJAiemNeABnHkCaJ9bRFo18vjb5hUgHFHv+kAj6+jfMvRa0WdX?= =?us-ascii?Q?DxUbiCuRC4Y/gsaoLeOL3gEBFhwOScNCc8p9pQRACjJ9/xw3THWqXVRpjnJS?= =?us-ascii?Q?WkOZ4uiFu6KKYBRaMWDHDhQr7QSHUdpQJC/qtc8flueQfYR722Cm5BS+uvAU?= =?us-ascii?Q?b5mHC1/frsljIoslN0h2qz5o3rRd6wdxHx0qbTq3/nhLm4QCohCwhutGt7BH?= =?us-ascii?Q?4qhjLluGhwqGpgaMD2By67bXoVVs9nH7eptV8Ux2Hn/V7eOPxeA/V1el+sO3?= =?us-ascii?Q?6QQoKOgYuDaliSq77owf49K+oH9/j2Th1FtDdzw2hIGB/A4KLkTP1R4gkQ1m?= =?us-ascii?Q?v0IrAMCHPOJYgVK7WudpPbTBJItp6stY8/onNmMdURtYqz48j/NnDJ7S/ZhM?= =?us-ascii?Q?UXGZUpVSsHNko9Ish602UsgVhDdJGts98H+/4gPtsXIU8a+jvxE3nqU5Fc5F?= =?us-ascii?Q?GuaL401losuq3EneNv6nMOBs2rdloRszjhynlU51wBlKhir1v3Exz7ulRjph?= =?us-ascii?Q?w+CNd405pM4QedZGFtzcef+KIVWFNg0Vzu/j5DGRKoy5VXmpdoxegwRgqO9C?= =?us-ascii?Q?IdM6kai08Boa0jdVEDgUabMTSd2hNRhZJvnjLx2ttSpDoxlXgd8ickjiHkU+?= =?us-ascii?Q?E/OFxBheQ82466YkRIFAQz/B6KUCufiuo9PsgDzzlXlT50ijZ1kjrm6x39mS?= =?us-ascii?Q?6C4xZsTWlq3yr25W3VUVHfMSXTEvo8o2qqypUDWBQGrUbqvPd5SjALaNq4H8?= =?us-ascii?Q?gqVkeGXSw8BWCRwEhkWJ8MWKFaeVFH3h04EbzZekp0N2CJnpI8IBFrHcXkjq?= =?us-ascii?Q?Ep3m7xJO3hIimND5QHzO8+lU?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3753.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1542ccc2-d915-4fcc-ce94-08d8f8de4e01 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Apr 2021 09:28:14.7015 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 5elpBbbnbtryNd6sW/aENdww41qcQod5LTUJy5BbUCWnG68Y/NxvFdCyqaQDc5J5W4lEDjkzD5M6/mQByQIoJQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4481 Subject: Re: [dpdk-dev] [PATCH 3/6] net/mlx5: separate Tx function declarations to another file X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Michael Baum > Sent: Monday, April 5, 2021 17:01 > To: dev@dpdk.org > Cc: Matan Azrad ; Raslan Darawsheh > ; Slava Ovsiienko > Subject: [PATCH 3/6] net/mlx5: separate Tx function declarations to anoth= er > file >=20 > This patch separates Tx function declarations to different header file in > preparation for removing their implementation from the source file and as= an > optional preparation for Tx cleanup. >=20 > Signed-off-by: Michael Baum Acked-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/linux/mlx5_mp_os.c | 1 + > drivers/net/mlx5/linux/mlx5_os.c | 1 + > drivers/net/mlx5/linux/mlx5_verbs.c | 2 +- > drivers/net/mlx5/mlx5.c | 1 + > drivers/net/mlx5/mlx5_devx.c | 2 +- > drivers/net/mlx5/mlx5_ethdev.c | 1 + > drivers/net/mlx5/mlx5_flow.c | 2 +- > drivers/net/mlx5/mlx5_flow_dv.c | 2 +- > drivers/net/mlx5/mlx5_flow_verbs.c | 1 - > drivers/net/mlx5/mlx5_mr.c | 1 + > drivers/net/mlx5/mlx5_rxmode.c | 1 - > drivers/net/mlx5/mlx5_rxq.c | 2 +- > drivers/net/mlx5/mlx5_rxtx.c | 1 + > drivers/net/mlx5/mlx5_rxtx.h | 344 ------------------------------= --- > drivers/net/mlx5/mlx5_stats.c | 2 +- > drivers/net/mlx5/mlx5_trigger.c | 2 +- > drivers/net/mlx5/mlx5_tx.h | 371 > ++++++++++++++++++++++++++++++++++++ > drivers/net/mlx5/mlx5_txpp.c | 2 +- > drivers/net/mlx5/mlx5_txq.c | 2 +- > drivers/net/mlx5/windows/mlx5_os.c | 1 + > 20 files changed, 387 insertions(+), 355 deletions(-) create mode 10064= 4 > drivers/net/mlx5/mlx5_tx.h >=20 > diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c > b/drivers/net/mlx5/linux/mlx5_mp_os.c > index 63fa278..ca529b6 100644 > --- a/drivers/net/mlx5/linux/mlx5_mp_os.c > +++ b/drivers/net/mlx5/linux/mlx5_mp_os.c > @@ -17,6 +17,7 @@ > #include "mlx5.h" > #include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_utils.h" >=20 > int > diff --git a/drivers/net/mlx5/linux/mlx5_os.c > b/drivers/net/mlx5/linux/mlx5_os.c > index 97a28ec..026423b 100644 > --- a/drivers/net/mlx5/linux/mlx5_os.c > +++ b/drivers/net/mlx5/linux/mlx5_os.c > @@ -41,6 +41,7 @@ > #include "mlx5_utils.h" > #include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_autoconf.h" > #include "mlx5_mr.h" > #include "mlx5_flow.h" > diff --git a/drivers/net/mlx5/linux/mlx5_verbs.c > b/drivers/net/mlx5/linux/mlx5_verbs.c > index 73096af..0b0759f 100644 > --- a/drivers/net/mlx5/linux/mlx5_verbs.c > +++ b/drivers/net/mlx5/linux/mlx5_verbs.c > @@ -20,9 +20,9 @@ > #include > #include > #include > -#include > #include > #include > +#include > #include > #include >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > 6f77bc2..02cc2c7 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -36,6 +36,7 @@ > #include "mlx5_utils.h" > #include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_autoconf.h" > #include "mlx5_mr.h" > #include "mlx5_flow.h" > diff --git a/drivers/net/mlx5/mlx5_devx.c b/drivers/net/mlx5/mlx5_devx.c > index 76935f6..76d31f5 100644 > --- a/drivers/net/mlx5/mlx5_devx.c > +++ b/drivers/net/mlx5/mlx5_devx.c > @@ -20,7 +20,7 @@ >=20 > #include "mlx5.h" > #include "mlx5_common_os.h" > -#include "mlx5_rxtx.h" > +#include "mlx5_tx.h" > #include "mlx5_rx.h" > #include "mlx5_utils.h" > #include "mlx5_devx.h" > diff --git a/drivers/net/mlx5/mlx5_ethdev.c > b/drivers/net/mlx5/mlx5_ethdev.c index 708e3a3..90baee5 100644 > --- a/drivers/net/mlx5/mlx5_ethdev.c > +++ b/drivers/net/mlx5/mlx5_ethdev.c > @@ -24,6 +24,7 @@ >=20 > #include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_autoconf.h" >=20 > /** > diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c > index b3877a1..4dea006 100644 > --- a/drivers/net/mlx5/mlx5_flow.c > +++ b/drivers/net/mlx5/mlx5_flow.c > @@ -29,8 +29,8 @@ > #include "mlx5.h" > #include "mlx5_flow.h" > #include "mlx5_flow_os.h" > -#include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_common_os.h" > #include "rte_pmd_mlx5.h" >=20 > diff --git a/drivers/net/mlx5/mlx5_flow_dv.c > b/drivers/net/mlx5/mlx5_flow_dv.c index cac05fb..cb5b3c9 100644 > --- a/drivers/net/mlx5/mlx5_flow_dv.c > +++ b/drivers/net/mlx5/mlx5_flow_dv.c > @@ -32,8 +32,8 @@ > #include "mlx5_common_os.h" > #include "mlx5_flow.h" > #include "mlx5_flow_os.h" > -#include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "rte_pmd_mlx5.h" >=20 > #if defined(HAVE_IBV_FLOW_DV_SUPPORT) || > !defined(HAVE_INFINIBAND_VERBS_H) diff --git > a/drivers/net/mlx5/mlx5_flow_verbs.c > b/drivers/net/mlx5/mlx5_flow_verbs.c > index c331350..0fdafbb 100644 > --- a/drivers/net/mlx5/mlx5_flow_verbs.c > +++ b/drivers/net/mlx5/mlx5_flow_verbs.c > @@ -23,7 +23,6 @@ > #include "mlx5_defs.h" > #include "mlx5.h" > #include "mlx5_flow.h" > -#include "mlx5_rxtx.h" > #include "mlx5_rx.h" >=20 > #define VERBS_SPEC_INNER(item_flags) \ > diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c inde= x > 2014936..e791b63 100644 > --- a/drivers/net/mlx5/mlx5_mr.c > +++ b/drivers/net/mlx5/mlx5_mr.c > @@ -16,6 +16,7 @@ > #include "mlx5_mr.h" > #include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" >=20 > struct mr_find_contig_memsegs_data { > uintptr_t addr; > diff --git a/drivers/net/mlx5/mlx5_rxmode.c > b/drivers/net/mlx5/mlx5_rxmode.c index cf93cca..25fb47c 100644 > --- a/drivers/net/mlx5/mlx5_rxmode.c > +++ b/drivers/net/mlx5/mlx5_rxmode.c > @@ -11,7 +11,6 @@ >=20 > #include > #include "mlx5.h" > -#include "mlx5_rxtx.h" > #include "mlx5_utils.h" >=20 > /** > diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c > index 19df0fa..bb9a908 100644 > --- a/drivers/net/mlx5/mlx5_rxq.c > +++ b/drivers/net/mlx5/mlx5_rxq.c > @@ -24,7 +24,7 @@ >=20 > #include "mlx5_defs.h" > #include "mlx5.h" > -#include "mlx5_rxtx.h" > +#include "mlx5_tx.h" > #include "mlx5_rx.h" > #include "mlx5_utils.h" > #include "mlx5_autoconf.h" > diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c > index c7f2605..57ff407 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.c > +++ b/drivers/net/mlx5/mlx5_rxtx.c > @@ -26,6 +26,7 @@ > #include "mlx5_utils.h" > #include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" >=20 > /* TX burst subroutines return codes. */ enum mlx5_txcmp_code { diff --= git > a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index > f1ebc99..e168dd4 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.h > +++ b/drivers/net/mlx5/mlx5_rxtx.h > @@ -17,169 +17,18 @@ > #include > #include > #include > -#include > #include >=20 > -#include > -#include > #include > #include >=20 > -#include "mlx5_defs.h" > #include "mlx5_utils.h" > #include "mlx5.h" > #include "mlx5_autoconf.h" > #include "mlx5_mr.h" >=20 > - > -/* Mbuf dynamic flag offset for inline. */ -extern uint64_t > rte_net_mlx5_dynf_inline_mask; > - > -struct mlx5_txq_stats { > -#ifdef MLX5_PMD_SOFT_COUNTERS > - uint64_t opackets; /**< Total of successfully sent packets. */ > - uint64_t obytes; /**< Total of successfully sent bytes. */ > -#endif > - uint64_t oerrors; /**< Total number of failed transmitted packets. */ > -}; > - > struct mlx5_priv; >=20 > -/* TX queue send local data. */ > -__extension__ > -struct mlx5_txq_local { > - struct mlx5_wqe *wqe_last; /* last sent WQE pointer. */ > - struct rte_mbuf *mbuf; /* first mbuf to process. */ > - uint16_t pkts_copy; /* packets copied to elts. */ > - uint16_t pkts_sent; /* packets sent. */ > - uint16_t pkts_loop; /* packets sent on loop entry. */ > - uint16_t elts_free; /* available elts remain. */ > - uint16_t wqe_free; /* available wqe remain. */ > - uint16_t mbuf_off; /* data offset in current mbuf. */ > - uint16_t mbuf_nseg; /* number of remaining mbuf. */ > - uint16_t mbuf_free; /* number of inline mbufs to free. */ > -}; > - > -/* TX queue descriptor. */ > -__extension__ > -struct mlx5_txq_data { > - uint16_t elts_head; /* Current counter in (*elts)[]. */ > - uint16_t elts_tail; /* Counter of first element awaiting completion. */ > - uint16_t elts_comp; /* elts index since last completion request. */ > - uint16_t elts_s; /* Number of mbuf elements. */ > - uint16_t elts_m; /* Mask for mbuf elements indices. */ > - /* Fields related to elts mbuf storage. */ > - uint16_t wqe_ci; /* Consumer index for work queue. */ > - uint16_t wqe_pi; /* Producer index for work queue. */ > - uint16_t wqe_s; /* Number of WQ elements. */ > - uint16_t wqe_m; /* Mask Number for WQ elements. */ > - uint16_t wqe_comp; /* WQE index since last completion request. */ > - uint16_t wqe_thres; /* WQE threshold to request completion in CQ. > */ > - /* WQ related fields. */ > - uint16_t cq_ci; /* Consumer index for completion queue. */ > - uint16_t cq_pi; /* Production index for completion queue. */ > - uint16_t cqe_s; /* Number of CQ elements. */ > - uint16_t cqe_m; /* Mask for CQ indices. */ > - /* CQ related fields. */ > - uint16_t elts_n:4; /* elts[] length (in log2). */ > - uint16_t cqe_n:4; /* Number of CQ elements (in log2). */ > - uint16_t wqe_n:4; /* Number of WQ elements (in log2). */ > - uint16_t tso_en:1; /* When set hardware TSO is enabled. */ > - uint16_t tunnel_en:1; > - /* When set TX offload for tunneled packets are supported. */ > - uint16_t swp_en:1; /* Whether SW parser is enabled. */ > - uint16_t vlan_en:1; /* VLAN insertion in WQE is supported. */ > - uint16_t db_nc:1; /* Doorbell mapped to non-cached region. */ > - uint16_t db_heu:1; /* Doorbell heuristic write barrier. */ > - uint16_t fast_free:1; /* mbuf fast free on Tx is enabled. */ > - uint16_t inlen_send; /* Ordinary send data inline size. */ > - uint16_t inlen_empw; /* eMPW max packet size to inline. */ > - uint16_t inlen_mode; /* Minimal data length to inline. */ > - uint32_t qp_num_8s; /* QP number shifted by 8. */ > - uint64_t offloads; /* Offloads for Tx Queue. */ > - struct mlx5_mr_ctrl mr_ctrl; /* MR control descriptor. */ > - struct mlx5_wqe *wqes; /* Work queue. */ > - struct mlx5_wqe *wqes_end; /* Work queue array limit. */ > -#ifdef RTE_LIBRTE_MLX5_DEBUG > - uint32_t *fcqs; /* Free completion queue (debug extended). */ > -#else > - uint16_t *fcqs; /* Free completion queue. */ > -#endif > - volatile struct mlx5_cqe *cqes; /* Completion queue. */ > - volatile uint32_t *qp_db; /* Work queue doorbell. */ > - volatile uint32_t *cq_db; /* Completion queue doorbell. */ > - uint16_t port_id; /* Port ID of device. */ > - uint16_t idx; /* Queue index. */ > - uint64_t ts_mask; /* Timestamp flag dynamic mask. */ > - int32_t ts_offset; /* Timestamp field dynamic offset. */ > - struct mlx5_dev_ctx_shared *sh; /* Shared context. */ > - struct mlx5_txq_stats stats; /* TX queue counters. */ > -#ifndef RTE_ARCH_64 > - rte_spinlock_t *uar_lock; > - /* UAR access lock required for 32bit implementations */ > -#endif > - struct rte_mbuf *elts[0]; > - /* Storage for queued packets, must be the last field. */ > -} __rte_cache_aligned; > - > -enum mlx5_txq_type { > - MLX5_TXQ_TYPE_STANDARD, /* Standard Tx queue. */ > - MLX5_TXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ > -}; > - > -/* TX queue control descriptor. */ > -struct mlx5_txq_ctrl { > - LIST_ENTRY(mlx5_txq_ctrl) next; /* Pointer to the next element. */ > - uint32_t refcnt; /* Reference counter. */ > - unsigned int socket; /* CPU socket ID for allocations. */ > - enum mlx5_txq_type type; /* The txq ctrl type. */ > - unsigned int max_inline_data; /* Max inline data. */ > - unsigned int max_tso_header; /* Max TSO header size. */ > - struct mlx5_txq_obj *obj; /* Verbs/DevX queue object. */ > - struct mlx5_priv *priv; /* Back pointer to private data. */ > - off_t uar_mmap_offset; /* UAR mmap offset for non-primary > process. */ > - void *bf_reg; /* BlueFlame register from Verbs. */ > - uint16_t dump_file_n; /* Number of dump files. */ > - struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ > - uint32_t hairpin_status; /* Hairpin binding status. */ > - struct mlx5_txq_data txq; /* Data path structure. */ > - /* Must be the last field in the structure, contains elts[]. */ > -}; > - > -#define MLX5_TX_BFREG(txq) \ > - (MLX5_PROC_PRIV((txq)->port_id)->uar_table[(txq)->idx]) > - > -/* mlx5_txq.c */ > - > -int mlx5_tx_queue_start(struct rte_eth_dev *dev, uint16_t queue_id); -in= t > mlx5_tx_queue_stop(struct rte_eth_dev *dev, uint16_t queue_id); -int > mlx5_tx_queue_start_primary(struct rte_eth_dev *dev, uint16_t > queue_id); -int mlx5_tx_queue_stop_primary(struct rte_eth_dev *dev, > uint16_t queue_id); -int mlx5_tx_queue_setup(struct rte_eth_dev *dev, > uint16_t idx, uint16_t desc, > - unsigned int socket, const struct rte_eth_txconf > *conf); > -int mlx5_tx_hairpin_queue_setup > - (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, > - const struct rte_eth_hairpin_conf *hairpin_conf); > -void mlx5_tx_queue_release(void *dpdk_txq); -void txq_uar_init(struct > mlx5_txq_ctrl *txq_ctrl); -int mlx5_tx_uar_init_secondary(struct > rte_eth_dev *dev, int fd); -void mlx5_tx_uar_uninit_secondary(struct > rte_eth_dev *dev); -int mlx5_txq_obj_verify(struct rte_eth_dev *dev); - > struct mlx5_txq_ctrl *mlx5_txq_new(struct rte_eth_dev *dev, uint16_t idx, > - uint16_t desc, unsigned int socket, > - const struct rte_eth_txconf *conf); > -struct mlx5_txq_ctrl *mlx5_txq_hairpin_new > - (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, > - const struct rte_eth_hairpin_conf *hairpin_conf); > -struct mlx5_txq_ctrl *mlx5_txq_get(struct rte_eth_dev *dev, uint16_t idx= ); > -int mlx5_txq_release(struct rte_eth_dev *dev, uint16_t idx); -int > mlx5_txq_releasable(struct rte_eth_dev *dev, uint16_t idx); -int > mlx5_txq_verify(struct rte_eth_dev *dev); -void txq_alloc_elts(struct > mlx5_txq_ctrl *txq_ctrl); -void txq_free_elts(struct mlx5_txq_ctrl *txq_c= trl); > -uint64_t mlx5_get_tx_port_offloads(struct rte_eth_dev *dev); -void > mlx5_txq_dynf_timestamp_set(struct rte_eth_dev *dev); > - > /* mlx5_rxtx.c */ >=20 > extern uint32_t mlx5_ptype_table[]; > @@ -189,88 +38,22 @@ struct mlx5_txq_ctrl *mlx5_txq_hairpin_new void > mlx5_set_ptype_table(void); void mlx5_set_cksum_table(void); void > mlx5_set_swp_types_table(void); -uint16_t removed_tx_burst(void > *dpdk_txq, struct rte_mbuf **pkts, > - uint16_t pkts_n); > -int mlx5_tx_descriptor_status(void *tx_queue, uint16_t offset); void > mlx5_dump_debug_information(const char *path, const char *title, > const void *buf, unsigned int len); int > mlx5_queue_state_modify_primary(struct rte_eth_dev *dev, > const struct mlx5_mp_arg_queue_state_modify > *sm); int mlx5_queue_state_modify(struct rte_eth_dev *dev, > struct mlx5_mp_arg_queue_state_modify *sm); - > void mlx5_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, > - struct rte_eth_txq_info *qinfo); > -int mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t > tx_queue_id, > - struct rte_eth_burst_mode *mode); >=20 > /* mlx5_mr.c */ >=20 > void mlx5_mr_flush_local_cache(struct mlx5_mr_ctrl *mr_ctrl); -uint32_t > mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb); - > uint32_t mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t > addr, > - struct rte_mempool *mp); > int mlx5_dma_map(struct rte_pci_device *pdev, void *addr, uint64_t iova, > size_t len); > int mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, uint64_t > iova, > size_t len); >=20 > /** > - * Provide safe 64bit store operation to mlx5 UAR region for both 32bit = and > - * 64bit architectures. > - * > - * @param val > - * value to write in CPU endian format. > - * @param addr > - * Address to write to. > - * @param lock > - * Address of the lock to use for that UAR access. > - */ > -static __rte_always_inline void > -__mlx5_uar_write64_relaxed(uint64_t val, void *addr, > - rte_spinlock_t *lock __rte_unused) > -{ > -#ifdef RTE_ARCH_64 > - *(uint64_t *)addr =3D val; > -#else /* !RTE_ARCH_64 */ > - rte_spinlock_lock(lock); > - *(uint32_t *)addr =3D val; > - rte_io_wmb(); > - *((uint32_t *)addr + 1) =3D val >> 32; > - rte_spinlock_unlock(lock); > -#endif > -} > - > -/** > - * Provide safe 64bit store operation to mlx5 UAR region for both 32bit = and > - * 64bit architectures while guaranteeing the order of execution with th= e > - * code being executed. > - * > - * @param val > - * value to write in CPU endian format. > - * @param addr > - * Address to write to. > - * @param lock > - * Address of the lock to use for that UAR access. > - */ > -static __rte_always_inline void > -__mlx5_uar_write64(uint64_t val, void *addr, rte_spinlock_t *lock) -{ > - rte_io_wmb(); > - __mlx5_uar_write64_relaxed(val, addr, lock); > -} > - > -/* Assist macros, used instead of directly calling the functions they wr= ap. */ > -#ifdef RTE_ARCH_64 -#define mlx5_uar_write64_relaxed(val, dst, lock) \ > - __mlx5_uar_write64_relaxed(val, dst, NULL) > -#define mlx5_uar_write64(val, dst, lock) __mlx5_uar_write64(val, dst, > NULL) -#else -#define mlx5_uar_write64_relaxed(val, dst, lock) \ > - __mlx5_uar_write64_relaxed(val, dst, lock) > -#define mlx5_uar_write64(val, dst, lock) __mlx5_uar_write64(val, dst, lo= ck) > -#endif > - > -/** > * Get Memory Pool (MP) from mbuf. If mbuf is indirect, the pool from > which the > * cloned mbuf is allocated is returned instead. > * > @@ -288,131 +71,4 @@ int mlx5_dma_unmap(struct rte_pci_device *pdev, > void *addr, uint64_t iova, > return buf->pool; > } >=20 > -/** > - * Query LKey from a packet buffer for Tx. If not found, add the mempool= . > - * > - * @param txq > - * Pointer to Tx queue structure. > - * @param addr > - * Address to search. > - * > - * @return > - * Searched LKey on success, UINT32_MAX on no match. > - */ > -static __rte_always_inline uint32_t > -mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb) -{ > - struct mlx5_mr_ctrl *mr_ctrl =3D &txq->mr_ctrl; > - uintptr_t addr =3D (uintptr_t)mb->buf_addr; > - uint32_t lkey; > - > - /* Check generation bit to see if there's any change on existing MRs. > */ > - if (unlikely(*mr_ctrl->dev_gen_ptr !=3D mr_ctrl->cur_gen)) > - mlx5_mr_flush_local_cache(mr_ctrl); > - /* Linear search on MR cache array. */ > - lkey =3D mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru, > - MLX5_MR_CACHE_N, addr); > - if (likely(lkey !=3D UINT32_MAX)) > - return lkey; > - /* Take slower bottom-half on miss. */ > - return mlx5_tx_mb2mr_bh(txq, mb); > -} > - > -/** > - * Ring TX queue doorbell and flush the update if requested. > - * > - * @param txq > - * Pointer to TX queue structure. > - * @param wqe > - * Pointer to the last WQE posted in the NIC. > - * @param cond > - * Request for write memory barrier after BlueFlame update. > - */ > -static __rte_always_inline void > -mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volatile struct > mlx5_wqe *wqe, > - int cond) > -{ > - uint64_t *dst =3D MLX5_TX_BFREG(txq); > - volatile uint64_t *src =3D ((volatile uint64_t *)wqe); > - > - rte_io_wmb(); > - *txq->qp_db =3D rte_cpu_to_be_32(txq->wqe_ci); > - /* Ensure ordering between DB record and BF copy. */ > - rte_wmb(); > - mlx5_uar_write64_relaxed(*src, dst, txq->uar_lock); > - if (cond) > - rte_wmb(); > -} > - > -/** > - * Ring TX queue doorbell and flush the update by write memory barrier. > - * > - * @param txq > - * Pointer to TX queue structure. > - * @param wqe > - * Pointer to the last WQE posted in the NIC. > - */ > -static __rte_always_inline void > -mlx5_tx_dbrec(struct mlx5_txq_data *txq, volatile struct mlx5_wqe *wqe) = - > { > - mlx5_tx_dbrec_cond_wmb(txq, wqe, 1); > -} > - > -/** > - * Convert timestamp from mbuf format to linear counter > - * of Clock Queue completions (24 bits) > - * > - * @param sh > - * Pointer to the device shared context to fetch Tx > - * packet pacing timestamp and parameters. > - * @param ts > - * Timestamp from mbuf to convert. > - * @return > - * positive or zero value - completion ID to wait > - * negative value - conversion error > - */ > -static __rte_always_inline int32_t > -mlx5_txpp_convert_tx_ts(struct mlx5_dev_ctx_shared *sh, uint64_t mts) -{ > - uint64_t ts, ci; > - uint32_t tick; > - > - do { > - /* > - * Read atomically two uint64_t fields and compare lsb bits. > - * It there is no match - the timestamp was updated in > - * the service thread, data should be re-read. > - */ > - rte_compiler_barrier(); > - ci =3D __atomic_load_n(&sh->txpp.ts.ci_ts, > __ATOMIC_RELAXED); > - ts =3D __atomic_load_n(&sh->txpp.ts.ts, > __ATOMIC_RELAXED); > - rte_compiler_barrier(); > - if (!((ts ^ ci) << (64 - MLX5_CQ_INDEX_WIDTH))) > - break; > - } while (true); > - /* Perform the skew correction, positive value to send earlier. */ > - mts -=3D sh->txpp.skew; > - mts -=3D ts; > - if (unlikely(mts >=3D UINT64_MAX / 2)) { > - /* We have negative integer, mts is in the past. */ > - __atomic_fetch_add(&sh->txpp.err_ts_past, > - 1, __ATOMIC_RELAXED); > - return -1; > - } > - tick =3D sh->txpp.tick; > - MLX5_ASSERT(tick); > - /* Convert delta to completions, round up. */ > - mts =3D (mts + tick - 1) / tick; > - if (unlikely(mts >=3D (1 << MLX5_CQ_INDEX_WIDTH) / 2 - 1)) { > - /* We have mts is too distant future. */ > - __atomic_fetch_add(&sh->txpp.err_ts_future, > - 1, __ATOMIC_RELAXED); > - return -1; > - } > - mts <<=3D 64 - MLX5_CQ_INDEX_WIDTH; > - ci +=3D mts; > - ci >>=3D 64 - MLX5_CQ_INDEX_WIDTH; > - return ci; > -} > - > #endif /* RTE_PMD_MLX5_RXTX_H_ */ > diff --git a/drivers/net/mlx5/mlx5_stats.c b/drivers/net/mlx5/mlx5_stats.= c > index 4dbd831..ae2f566 100644 > --- a/drivers/net/mlx5/mlx5_stats.c > +++ b/drivers/net/mlx5/mlx5_stats.c > @@ -16,8 +16,8 @@ >=20 > #include "mlx5_defs.h" > #include "mlx5.h" > -#include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_malloc.h" >=20 > /** > diff --git a/drivers/net/mlx5/mlx5_trigger.c > b/drivers/net/mlx5/mlx5_trigger.c index c88cb22..001c0b5 100644 > --- a/drivers/net/mlx5/mlx5_trigger.c > +++ b/drivers/net/mlx5/mlx5_trigger.c > @@ -15,8 +15,8 @@ >=20 > #include "mlx5.h" > #include "mlx5_mr.h" > -#include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_utils.h" > #include "rte_pmd_mlx5.h" >=20 > diff --git a/drivers/net/mlx5/mlx5_tx.h b/drivers/net/mlx5/mlx5_tx.h new > file mode 100644 index 0000000..7f91d04 > --- /dev/null > +++ b/drivers/net/mlx5/mlx5_tx.h > @@ -0,0 +1,371 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright 2021 6WIND S.A. > + * Copyright 2021 Mellanox Technologies, Ltd */ > + > +#ifndef RTE_PMD_MLX5_TX_H_ > +#define RTE_PMD_MLX5_TX_H_ > + > +#include > +#include > + > +#include > +#include > +#include > +#include > + > +#include > + > +#include "mlx5.h" > +#include "mlx5_autoconf.h" > +#include "mlx5_mr.h" > + > +/* Mbuf dynamic flag offset for inline. */ extern uint64_t > +rte_net_mlx5_dynf_inline_mask; > + > +struct mlx5_txq_stats { > +#ifdef MLX5_PMD_SOFT_COUNTERS > + uint64_t opackets; /**< Total of successfully sent packets. */ > + uint64_t obytes; /**< Total of successfully sent bytes. */ #endif > + uint64_t oerrors; /**< Total number of failed transmitted packets. */ > +}; > + > +/* TX queue send local data. */ > +__extension__ > +struct mlx5_txq_local { > + struct mlx5_wqe *wqe_last; /* last sent WQE pointer. */ > + struct rte_mbuf *mbuf; /* first mbuf to process. */ > + uint16_t pkts_copy; /* packets copied to elts. */ > + uint16_t pkts_sent; /* packets sent. */ > + uint16_t pkts_loop; /* packets sent on loop entry. */ > + uint16_t elts_free; /* available elts remain. */ > + uint16_t wqe_free; /* available wqe remain. */ > + uint16_t mbuf_off; /* data offset in current mbuf. */ > + uint16_t mbuf_nseg; /* number of remaining mbuf. */ > + uint16_t mbuf_free; /* number of inline mbufs to free. */ }; > + > +/* TX queue descriptor. */ > +__extension__ > +struct mlx5_txq_data { > + uint16_t elts_head; /* Current counter in (*elts)[]. */ > + uint16_t elts_tail; /* Counter of first element awaiting completion. */ > + uint16_t elts_comp; /* elts index since last completion request. */ > + uint16_t elts_s; /* Number of mbuf elements. */ > + uint16_t elts_m; /* Mask for mbuf elements indices. */ > + /* Fields related to elts mbuf storage. */ > + uint16_t wqe_ci; /* Consumer index for work queue. */ > + uint16_t wqe_pi; /* Producer index for work queue. */ > + uint16_t wqe_s; /* Number of WQ elements. */ > + uint16_t wqe_m; /* Mask Number for WQ elements. */ > + uint16_t wqe_comp; /* WQE index since last completion request. */ > + uint16_t wqe_thres; /* WQE threshold to request completion in CQ. > */ > + /* WQ related fields. */ > + uint16_t cq_ci; /* Consumer index for completion queue. */ > + uint16_t cq_pi; /* Production index for completion queue. */ > + uint16_t cqe_s; /* Number of CQ elements. */ > + uint16_t cqe_m; /* Mask for CQ indices. */ > + /* CQ related fields. */ > + uint16_t elts_n:4; /* elts[] length (in log2). */ > + uint16_t cqe_n:4; /* Number of CQ elements (in log2). */ > + uint16_t wqe_n:4; /* Number of WQ elements (in log2). */ > + uint16_t tso_en:1; /* When set hardware TSO is enabled. */ > + uint16_t tunnel_en:1; > + /* When set TX offload for tunneled packets are supported. */ > + uint16_t swp_en:1; /* Whether SW parser is enabled. */ > + uint16_t vlan_en:1; /* VLAN insertion in WQE is supported. */ > + uint16_t db_nc:1; /* Doorbell mapped to non-cached region. */ > + uint16_t db_heu:1; /* Doorbell heuristic write barrier. */ > + uint16_t fast_free:1; /* mbuf fast free on Tx is enabled. */ > + uint16_t inlen_send; /* Ordinary send data inline size. */ > + uint16_t inlen_empw; /* eMPW max packet size to inline. */ > + uint16_t inlen_mode; /* Minimal data length to inline. */ > + uint32_t qp_num_8s; /* QP number shifted by 8. */ > + uint64_t offloads; /* Offloads for Tx Queue. */ > + struct mlx5_mr_ctrl mr_ctrl; /* MR control descriptor. */ > + struct mlx5_wqe *wqes; /* Work queue. */ > + struct mlx5_wqe *wqes_end; /* Work queue array limit. */ #ifdef > +RTE_LIBRTE_MLX5_DEBUG > + uint32_t *fcqs; /* Free completion queue (debug extended). */ > #else > + uint16_t *fcqs; /* Free completion queue. */ #endif > + volatile struct mlx5_cqe *cqes; /* Completion queue. */ > + volatile uint32_t *qp_db; /* Work queue doorbell. */ > + volatile uint32_t *cq_db; /* Completion queue doorbell. */ > + uint16_t port_id; /* Port ID of device. */ > + uint16_t idx; /* Queue index. */ > + uint64_t ts_mask; /* Timestamp flag dynamic mask. */ > + int32_t ts_offset; /* Timestamp field dynamic offset. */ > + struct mlx5_dev_ctx_shared *sh; /* Shared context. */ > + struct mlx5_txq_stats stats; /* TX queue counters. */ #ifndef > +RTE_ARCH_64 > + rte_spinlock_t *uar_lock; > + /* UAR access lock required for 32bit implementations */ #endif > + struct rte_mbuf *elts[0]; > + /* Storage for queued packets, must be the last field. */ } > +__rte_cache_aligned; > + > +enum mlx5_txq_type { > + MLX5_TXQ_TYPE_STANDARD, /* Standard Tx queue. */ > + MLX5_TXQ_TYPE_HAIRPIN, /* Hairpin Tx queue. */ }; > + > +/* TX queue control descriptor. */ > +struct mlx5_txq_ctrl { > + LIST_ENTRY(mlx5_txq_ctrl) next; /* Pointer to the next element. */ > + uint32_t refcnt; /* Reference counter. */ > + unsigned int socket; /* CPU socket ID for allocations. */ > + enum mlx5_txq_type type; /* The txq ctrl type. */ > + unsigned int max_inline_data; /* Max inline data. */ > + unsigned int max_tso_header; /* Max TSO header size. */ > + struct mlx5_txq_obj *obj; /* Verbs/DevX queue object. */ > + struct mlx5_priv *priv; /* Back pointer to private data. */ > + off_t uar_mmap_offset; /* UAR mmap offset for non-primary > process. */ > + void *bf_reg; /* BlueFlame register from Verbs. */ > + uint16_t dump_file_n; /* Number of dump files. */ > + struct rte_eth_hairpin_conf hairpin_conf; /* Hairpin configuration. */ > + uint32_t hairpin_status; /* Hairpin binding status. */ > + struct mlx5_txq_data txq; /* Data path structure. */ > + /* Must be the last field in the structure, contains elts[]. */ }; > + > +/* mlx5_txq.c */ > + > +int mlx5_tx_queue_start(struct rte_eth_dev *dev, uint16_t queue_id); > +int mlx5_tx_queue_stop(struct rte_eth_dev *dev, uint16_t queue_id); int > +mlx5_tx_queue_start_primary(struct rte_eth_dev *dev, uint16_t > +queue_id); int mlx5_tx_queue_stop_primary(struct rte_eth_dev *dev, > +uint16_t queue_id); int mlx5_tx_queue_setup(struct rte_eth_dev *dev, > uint16_t idx, uint16_t desc, > + unsigned int socket, const struct rte_eth_txconf > *conf); int > +mlx5_tx_hairpin_queue_setup > + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, > + const struct rte_eth_hairpin_conf *hairpin_conf); void > +mlx5_tx_queue_release(void *dpdk_txq); void txq_uar_init(struct > +mlx5_txq_ctrl *txq_ctrl); int mlx5_tx_uar_init_secondary(struct > +rte_eth_dev *dev, int fd); void mlx5_tx_uar_uninit_secondary(struct > +rte_eth_dev *dev); int mlx5_txq_obj_verify(struct rte_eth_dev *dev); > +struct mlx5_txq_ctrl *mlx5_txq_new(struct rte_eth_dev *dev, uint16_t > idx, > + uint16_t desc, unsigned int socket, > + const struct rte_eth_txconf *conf); struct > mlx5_txq_ctrl > +*mlx5_txq_hairpin_new > + (struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, > + const struct rte_eth_hairpin_conf *hairpin_conf); struct > +mlx5_txq_ctrl *mlx5_txq_get(struct rte_eth_dev *dev, uint16_t idx); int > +mlx5_txq_release(struct rte_eth_dev *dev, uint16_t idx); int > +mlx5_txq_releasable(struct rte_eth_dev *dev, uint16_t idx); int > +mlx5_txq_verify(struct rte_eth_dev *dev); void txq_alloc_elts(struct > +mlx5_txq_ctrl *txq_ctrl); void txq_free_elts(struct mlx5_txq_ctrl > +*txq_ctrl); uint64_t mlx5_get_tx_port_offloads(struct rte_eth_dev > +*dev); void mlx5_txq_dynf_timestamp_set(struct rte_eth_dev *dev); > + > +/* mlx5_tx.c */ > + > +uint16_t removed_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, > + uint16_t pkts_n); > +int mlx5_tx_descriptor_status(void *tx_queue, uint16_t offset); void > +mlx5_txq_info_get(struct rte_eth_dev *dev, uint16_t queue_id, > + struct rte_eth_txq_info *qinfo); int > +mlx5_tx_burst_mode_get(struct rte_eth_dev *dev, uint16_t tx_queue_id, > + struct rte_eth_burst_mode *mode); > + > +/* mlx5_mr.c */ > + > +uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf > +*mb); uint32_t mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, > uintptr_t addr, > + struct rte_mempool *mp); > + > +static __rte_always_inline uint64_t * > +mlx5_tx_bfreg(struct mlx5_txq_data *txq) { > + return MLX5_PROC_PRIV(txq->port_id)->uar_table[txq->idx]; > +} > + > +/** > + * Provide safe 64bit store operation to mlx5 UAR region for both 32bit > +and > + * 64bit architectures. > + * > + * @param val > + * value to write in CPU endian format. > + * @param addr > + * Address to write to. > + * @param lock > + * Address of the lock to use for that UAR access. > + */ > +static __rte_always_inline void > +__mlx5_uar_write64_relaxed(uint64_t val, void *addr, > + rte_spinlock_t *lock __rte_unused) { #ifdef > RTE_ARCH_64 > + *(uint64_t *)addr =3D val; > +#else /* !RTE_ARCH_64 */ > + rte_spinlock_lock(lock); > + *(uint32_t *)addr =3D val; > + rte_io_wmb(); > + *((uint32_t *)addr + 1) =3D val >> 32; > + rte_spinlock_unlock(lock); > +#endif > +} > + > +/** > + * Provide safe 64bit store operation to mlx5 UAR region for both 32bit > +and > + * 64bit architectures while guaranteeing the order of execution with > +the > + * code being executed. > + * > + * @param val > + * value to write in CPU endian format. > + * @param addr > + * Address to write to. > + * @param lock > + * Address of the lock to use for that UAR access. > + */ > +static __rte_always_inline void > +__mlx5_uar_write64(uint64_t val, void *addr, rte_spinlock_t *lock) { > + rte_io_wmb(); > + __mlx5_uar_write64_relaxed(val, addr, lock); } > + > +/* Assist macros, used instead of directly calling the functions they > +wrap. */ #ifdef RTE_ARCH_64 #define mlx5_uar_write64_relaxed(val, dst, > +lock) \ > + __mlx5_uar_write64_relaxed(val, dst, NULL) #define > +mlx5_uar_write64(val, dst, lock) __mlx5_uar_write64(val, dst, NULL) > +#else #define mlx5_uar_write64_relaxed(val, dst, lock) \ > + __mlx5_uar_write64_relaxed(val, dst, lock) #define > +mlx5_uar_write64(val, dst, lock) __mlx5_uar_write64(val, dst, lock) > +#endif > + > +/** > + * Query LKey from a packet buffer for Tx. If not found, add the mempool= . > + * > + * @param txq > + * Pointer to Tx queue structure. > + * @param addr > + * Address to search. > + * > + * @return > + * Searched LKey on success, UINT32_MAX on no match. > + */ > +static __rte_always_inline uint32_t > +mlx5_tx_mb2mr(struct mlx5_txq_data *txq, struct rte_mbuf *mb) { > + struct mlx5_mr_ctrl *mr_ctrl =3D &txq->mr_ctrl; > + uintptr_t addr =3D (uintptr_t)mb->buf_addr; > + uint32_t lkey; > + > + /* Check generation bit to see if there's any change on existing MRs. > */ > + if (unlikely(*mr_ctrl->dev_gen_ptr !=3D mr_ctrl->cur_gen)) > + mlx5_mr_flush_local_cache(mr_ctrl); > + /* Linear search on MR cache array. */ > + lkey =3D mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru, > + MLX5_MR_CACHE_N, addr); > + if (likely(lkey !=3D UINT32_MAX)) > + return lkey; > + /* Take slower bottom-half on miss. */ > + return mlx5_tx_mb2mr_bh(txq, mb); > +} > + > +/** > + * Ring TX queue doorbell and flush the update if requested. > + * > + * @param txq > + * Pointer to TX queue structure. > + * @param wqe > + * Pointer to the last WQE posted in the NIC. > + * @param cond > + * Request for write memory barrier after BlueFlame update. > + */ > +static __rte_always_inline void > +mlx5_tx_dbrec_cond_wmb(struct mlx5_txq_data *txq, volatile struct > mlx5_wqe *wqe, > + int cond) > +{ > + uint64_t *dst =3D mlx5_tx_bfreg(txq); > + volatile uint64_t *src =3D ((volatile uint64_t *)wqe); > + > + rte_io_wmb(); > + *txq->qp_db =3D rte_cpu_to_be_32(txq->wqe_ci); > + /* Ensure ordering between DB record and BF copy. */ > + rte_wmb(); > + mlx5_uar_write64_relaxed(*src, dst, txq->uar_lock); > + if (cond) > + rte_wmb(); > +} > + > +/** > + * Ring TX queue doorbell and flush the update by write memory barrier. > + * > + * @param txq > + * Pointer to TX queue structure. > + * @param wqe > + * Pointer to the last WQE posted in the NIC. > + */ > +static __rte_always_inline void > +mlx5_tx_dbrec(struct mlx5_txq_data *txq, volatile struct mlx5_wqe *wqe) > +{ > + mlx5_tx_dbrec_cond_wmb(txq, wqe, 1); > +} > + > +/** > + * Convert timestamp from mbuf format to linear counter > + * of Clock Queue completions (24 bits). > + * > + * @param sh > + * Pointer to the device shared context to fetch Tx > + * packet pacing timestamp and parameters. > + * @param ts > + * Timestamp from mbuf to convert. > + * @return > + * positive or zero value - completion ID to wait. > + * negative value - conversion error. > + */ > +static __rte_always_inline int32_t > +mlx5_txpp_convert_tx_ts(struct mlx5_dev_ctx_shared *sh, uint64_t mts) { > + uint64_t ts, ci; > + uint32_t tick; > + > + do { > + /* > + * Read atomically two uint64_t fields and compare lsb bits. > + * It there is no match - the timestamp was updated in > + * the service thread, data should be re-read. > + */ > + rte_compiler_barrier(); > + ci =3D __atomic_load_n(&sh->txpp.ts.ci_ts, > __ATOMIC_RELAXED); > + ts =3D __atomic_load_n(&sh->txpp.ts.ts, > __ATOMIC_RELAXED); > + rte_compiler_barrier(); > + if (!((ts ^ ci) << (64 - MLX5_CQ_INDEX_WIDTH))) > + break; > + } while (true); > + /* Perform the skew correction, positive value to send earlier. */ > + mts -=3D sh->txpp.skew; > + mts -=3D ts; > + if (unlikely(mts >=3D UINT64_MAX / 2)) { > + /* We have negative integer, mts is in the past. */ > + __atomic_fetch_add(&sh->txpp.err_ts_past, > + 1, __ATOMIC_RELAXED); > + return -1; > + } > + tick =3D sh->txpp.tick; > + MLX5_ASSERT(tick); > + /* Convert delta to completions, round up. */ > + mts =3D (mts + tick - 1) / tick; > + if (unlikely(mts >=3D (1 << MLX5_CQ_INDEX_WIDTH) / 2 - 1)) { > + /* We have mts is too distant future. */ > + __atomic_fetch_add(&sh->txpp.err_ts_future, > + 1, __ATOMIC_RELAXED); > + return -1; > + } > + mts <<=3D 64 - MLX5_CQ_INDEX_WIDTH; > + ci +=3D mts; > + ci >>=3D 64 - MLX5_CQ_INDEX_WIDTH; > + return ci; > +} > + > +#endif /* RTE_PMD_MLX5_TX_H_ */ > diff --git a/drivers/net/mlx5/mlx5_txpp.c b/drivers/net/mlx5/mlx5_txpp.c > index 89e1c5d..d90399a 100644 > --- a/drivers/net/mlx5/mlx5_txpp.c > +++ b/drivers/net/mlx5/mlx5_txpp.c > @@ -16,8 +16,8 @@ > #include >=20 > #include "mlx5.h" > -#include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_common_os.h" >=20 > static_assert(sizeof(struct mlx5_cqe_ts) =3D=3D sizeof(rte_int128_t), di= ff --git > a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index > cd13eb9..b8a1657 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -23,7 +23,7 @@ > #include "mlx5_defs.h" > #include "mlx5_utils.h" > #include "mlx5.h" > -#include "mlx5_rxtx.h" > +#include "mlx5_tx.h" > #include "mlx5_autoconf.h" >=20 > /** > diff --git a/drivers/net/mlx5/windows/mlx5_os.c > b/drivers/net/mlx5/windows/mlx5_os.c > index 79eac80..814063b 100644 > --- a/drivers/net/mlx5/windows/mlx5_os.c > +++ b/drivers/net/mlx5/windows/mlx5_os.c > @@ -24,6 +24,7 @@ > #include "mlx5_utils.h" > #include "mlx5_rxtx.h" > #include "mlx5_rx.h" > +#include "mlx5_tx.h" > #include "mlx5_autoconf.h" > #include "mlx5_mr.h" > #include "mlx5_flow.h" > -- > 1.8.3.1