From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id C13E7A00C2;
	Thu,  6 Oct 2022 13:02:04 +0200 (CEST)
Received: from [217.70.189.124] (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 7599B42BF0;
	Thu,  6 Oct 2022 13:02:04 +0200 (CEST)
Received: from NAM10-BN7-obe.outbound.protection.outlook.com
 (mail-bn7nam10on2049.outbound.protection.outlook.com [40.107.92.49])
 by mails.dpdk.org (Postfix) with ESMTP id F292B42B70
 for <dev@dpdk.org>; Thu,  6 Oct 2022 13:02:02 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=e16Gfs+YDEZlSBBiryqp2Va6USRym6ofmyYC8ra7ZnJdNNut0KDxJuRgYVPrt8v4XRNIkJ63LnGeoqcGC48xZTdLjIsf6L3aIDayDHV/DbwrXS+B3290zOnGkqGY6nxTDnnjWOQjLlG5jR2r7wxfKrtvFSqbDCUyG+bd+BRnLwoxiyLFFi9A3yaNXBA1zOidWhIRgPa/XEPlMK8S+cQYcxbucbFJymk7I35y9cLIxTTbyLOhZH21I0gTHqLdsM7msIyhxYSzPeiM6PR30Ysii4rTHrVwlRLHCS4SYoJ67Kk21Osu2TeyBxXMgOXOwTVis+jodSXARmeAQS4ICH8YQg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=yxrLcY4OSGa/76HmkoqKydHWRfmgUJznAupUDmYsGys=;
 b=An7yc/jdEUbrsAJ8AZI1dEE3ijB/BzUlNK+QtlIZwEjIvMYXvGKPtWIrR8a1aRZ/0OHjRFTnDTZuqDDmRxDkoNJL3y9dOSlfsP6qm6KRbbhNCXbZAQuUxJRmSU5qagJ6l77m5dJjTsC+s6WOKc4iYSsmtEmseSkZhZeGGh8xtqvMEUvM/rWNWrmXSfLLTYTKKuetcKx3xD860ZG2aRlDq/TW87SEbNyXRshSRF+ftzeD9JmUhVYlKA362iSANNVIsFfQd0JOIZ5KvFDPonuNzjqm6wUfshGDEnxrf5t7YH+23SnbAJx24KNf0GNkFAyd1kJxcMkF+FFCfIuGOO+PXA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
 216.228.117.161) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com;
 dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com;
 dkim=none (message not signed); arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;
 s=selector2;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=yxrLcY4OSGa/76HmkoqKydHWRfmgUJznAupUDmYsGys=;
 b=XHt9bYBQ2eQJxVV5A2lnU3LZBKc7St7oBHwgB/ih679QKk7UeiHWCdF8xphqkIqANEkpy/zv447peNYqC7dxnO9A9xhx7Q7jcthbi15IpH0CrfZFeqnXLW4hnESCl02ppMEjmVJtGO7e/APwi73NLH67Xi9wOapW7lVOzZ45lhtZX8gf/XcTUbgvL2vIDtaPc7pMKFgS7ktgN/K38Xk0DqEXdWtsgw5BHmrE7AVElXCdSwy6dgN9hVbjxbJI24/uPCswq39BfaMfGJ7yuWbGoIN8T93EE96yef0S8wKLXYaCYAvQrn2u98lW4VmRlBBjNN+DazE/4vKfF98yrpEudg==
Received: from DM6PR02CA0125.namprd02.prod.outlook.com (2603:10b6:5:1b4::27)
 by MW4PR12MB7142.namprd12.prod.outlook.com (2603:10b6:303:220::6) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.19; Thu, 6 Oct
 2022 11:02:00 +0000
Received: from DM6NAM11FT037.eop-nam11.prod.protection.outlook.com
 (2603:10b6:5:1b4:cafe::a5) by DM6PR02CA0125.outlook.office365.com
 (2603:10b6:5:1b4::27) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.28 via Frontend
 Transport; Thu, 6 Oct 2022 11:02:00 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161)
 smtp.mailfrom=nvidia.com;
 dkim=none (message not signed)
 header.d=none;dmarc=pass action=none header.from=nvidia.com;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.161 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C
Received: from mail.nvidia.com (216.228.117.161) by
 DM6NAM11FT037.mail.protection.outlook.com (10.13.172.122) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.5709.10 via Frontend Transport; Thu, 6 Oct 2022 11:02:00 +0000
Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com
 (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Thu, 6 Oct 2022
 04:01:45 -0700
Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com
 (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Thu, 6 Oct 2022
 04:01:42 -0700
From: Dariusz Sosnowski <dsosnowski@nvidia.com>
To: Thomas Monjalon <thomas@monjalon.net>, Ferruh Yigit
 <ferruh.yigit@xilinx.com>, Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
CC: <dev@dpdk.org>, Viacheslav Ovsiienko <viacheslavo@nvidia.com>, Matan Azrad
 <matan@nvidia.com>, Ori Kam <orika@nvidia.com>, Wisam Jaddo
 <wisamm@nvidia.com>, Aman Singh <aman.deep.singh@intel.com>, Yuying Zhang
 <yuying.zhang@intel.com>
Subject: [PATCH v2 0/8] ethdev: introduce hairpin memory capabilities
Date: Thu, 6 Oct 2022 11:00:57 +0000
Message-ID: <20221006110105.2986966-1-dsosnowski@nvidia.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20220919163731.1540454-1-dsosnowski@nvidia.com>
References: 
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-Originating-IP: [10.126.230.35]
X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To
 rnnvmail201.nvidia.com (10.129.68.8)
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: DM6NAM11FT037:EE_|MW4PR12MB7142:EE_
X-MS-Office365-Filtering-Correlation-Id: 244079b8-7a7f-4ab5-7dc6-08daa78a3172
X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: Of8Fpu8UiwJNCVzOUo5ldYAFEd6koLmW/WbrDUw16UHFnNcZqebl8UuQAZWe6qxNgsVon0jD26L/PKQCEpMLVysRrDXbtBx3QLCyFvIZLRrczeyE2AibWrDQIdDUJz37Aj974vJ7HI70vdXpME3OUFgxPFGCQVablhYaoIiAvO60R2lttSDjCTtc66hzNKCjt8cXJT5MOwZWL+dFXlcnPxU9jxyVLlGuHfn6Jlte0w4r4V2z7N4OMiQP540bHB1mCI0iOfjdo5IjWqcdusZ0a/6U9r/ReB2i1S9Gj3KtcrN6UR/WVJaODCuuANlWwV6hcJAqYAYT43BqzMki8bh0aewFD+n4F9AgQcvSokBQ6Ogo4xodaBxfdLCA7f0JkCWuQjJ6G1X9zsvPUfqvYHEk5nlk86gVq72deJWLHOFapoowShzz7TNDualO1sHWRThCIljNXEB4zUvhVTAb6j+xdD0MeSSpmBbFGmiFmGBmr9kPlTRpmxj/22OtdGEhUWLjEe/WMfT3vhVds5MF/FQXpn90CYIXg+ybQVeRQVTE+zwNT1WAuYiV/oXjrT4Ns6YTOhLQkEQFKbEBKPKTNe8zNfQn3CDBv4f3HYr584ChNnoaIjGIKyZbqWiaNPwX/zV/zOttKpXlaPrOgHtEmOVBKAYpagYPORvd81MDJkb1ui/+iX8BSHSoFEE+Sek+IyU+BhvYORRbekkQqf1WknKZmL6cMud2vxdWaPvVtD2l6/2kgz9FZArB5L5eEZw7kNbAb1yKr8v6+MG5rYe/5xAI3Sl73ULzOkdtkrub1FW28XtYx1WPBFPMOB3kx+TB2Vc3eiN9TlOP2k+98W3XY/9t0a/BDBKddTJiVM+YpIJFV/c=
X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE;
 SFS:(13230022)(4636009)(136003)(396003)(376002)(346002)(39860400002)(451199015)(36840700001)(40470700004)(46966006)(36756003)(86362001)(110136005)(36860700001)(336012)(16526019)(1076003)(186003)(2616005)(356005)(7636003)(82740400003)(47076005)(426003)(6286002)(26005)(7696005)(966005)(478600001)(316002)(6666004)(54906003)(70586007)(8676002)(4326008)(70206006)(8936002)(82310400005)(5660300002)(2906002)(83380400001)(40460700003)(40480700001)(55016003)(41300700001);
 DIR:OUT; SFP:1101; 
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2022 11:02:00.1383 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 244079b8-7a7f-4ab5-7dc6-08daa78a3172
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161];
 Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT037.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7142
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

The hairpin queues are used to transmit packets received on the wire, back to the wire.
How hairpin queues are implemented and configured is decided internally by the PMD and
applications have no control over the configuration of Rx and Tx hairpin queues.
This patchset addresses that by:

- Extending hairpin queue capabilities reported by PMDs.
- Exposing new configuration options for Rx and Tx hairpin queues.

Main goal of this patchset is to allow applications to provide configuration hints
regarding memory placement of hairpin queues.
These hints specify whether buffers of hairpin queues should be placed in host memory
or in dedicated device memory.

For example, in context of NVIDIA Connect-X and BlueField devices,
this distinction is important for several reasons:

- By default, data buffers and packet descriptors are placed in device memory region
  which is shared with other resources (e.g. flow rules).
  This results in memory contention on the device,
  which may lead to degraded performance under heavy load.
- Placing hairpin queues in dedicated device memory can decrease latency of hairpinned traffic,
  since hairpin queue processing will not be memory starved by other operations.
  Side effect of this memory configuration is that it leaves less memory for other resources,
  possibly causing memory contention in non-hairpin traffic.
- Placing hairpin queues in host memory can increase throughput of hairpinned
  traffic at the cost of increasing latency.
  Each packet processed by hairpin queues will incur additional PCI transactions (increase in latency),
  but memory contention on the device is avoided.

Depending on the workload and whether throughput or latency has a higher priority for developers,
it would be beneficial if developers could choose the best hairpin configuration for their use case.

To address that, this patchset adds the following configuration options (in rte_eth_hairpin_conf struct):

- use_locked_device_memory - If set, PMD will allocate specialized on-device memory for the queue.
- use_rte_memory - If set, PMD will use DPDK-managed memory for the queue.
- force_memory - If set, PMD will be forced to use provided memory configuration.
  If no appropriate resources are available, the queue allocation will fail.
  If unset and no appropriate resources are available, PMD will fallback to its default behavior.

Implementing support for these flags is optional and applications should be allowed to not set any of these new flags.
This will result in default memory configuration provided by the PMD.
Application developers should consult the PMD documentation in that case.

These changes were originally proposed in http://patches.dpdk.org/project/dpdk/patch/20220811120530.191683-1-dsosnowski@nvidia.com/.

Dariusz Sosnowski (8):
  ethdev: introduce hairpin memory capabilities
  common/mlx5: add hairpin SQ buffer type capabilities
  common/mlx5: add hairpin RQ buffer type capabilities
  net/mlx5: allow hairpin Tx queue in RTE memory
  net/mlx5: allow hairpin Rx queue in locked memory
  doc: add notes for hairpin to mlx5 documentation
  app/testpmd: add hairpin queues memory modes
  app/flow-perf: add hairpin queue memory config

 app/test-flow-perf/main.c              |  32 +++++
 app/test-pmd/parameters.c              |   2 +-
 app/test-pmd/testpmd.c                 |  24 +++-
 app/test-pmd/testpmd.h                 |   2 +-
 doc/guides/nics/mlx5.rst               |  37 ++++++
 doc/guides/platform/mlx5.rst           |   5 +
 doc/guides/rel_notes/release_22_11.rst |  10 ++
 doc/guides/testpmd_app_ug/run_app.rst  |  10 +-
 drivers/common/mlx5/mlx5_devx_cmds.c   |   8 ++
 drivers/common/mlx5/mlx5_devx_cmds.h   |   5 +
 drivers/common/mlx5/mlx5_prm.h         |  25 +++-
 drivers/net/mlx5/mlx5.h                |   2 +
 drivers/net/mlx5/mlx5_devx.c           | 170 ++++++++++++++++++++++---
 drivers/net/mlx5/mlx5_ethdev.c         |   6 +
 lib/ethdev/rte_ethdev.c                |  44 +++++++
 lib/ethdev/rte_ethdev.h                |  68 +++++++++-
 16 files changed, 422 insertions(+), 28 deletions(-)

-- 
v2:
* Fix Windows build by using mlx5_os_umem_dereg defined on both platforms to allocate memory for Tx hairpin queue.
* Added hairpin section to mlx5 PMD.
* Added info about new hairpin configuration options to DPDK release notes.

2.25.1