From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C13E7A00C2; Thu, 6 Oct 2022 13:02:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7599B42BF0; Thu, 6 Oct 2022 13:02:04 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2049.outbound.protection.outlook.com [40.107.92.49]) by mails.dpdk.org (Postfix) with ESMTP id F292B42B70 for <dev@dpdk.org>; Thu, 6 Oct 2022 13:02:02 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=e16Gfs+YDEZlSBBiryqp2Va6USRym6ofmyYC8ra7ZnJdNNut0KDxJuRgYVPrt8v4XRNIkJ63LnGeoqcGC48xZTdLjIsf6L3aIDayDHV/DbwrXS+B3290zOnGkqGY6nxTDnnjWOQjLlG5jR2r7wxfKrtvFSqbDCUyG+bd+BRnLwoxiyLFFi9A3yaNXBA1zOidWhIRgPa/XEPlMK8S+cQYcxbucbFJymk7I35y9cLIxTTbyLOhZH21I0gTHqLdsM7msIyhxYSzPeiM6PR30Ysii4rTHrVwlRLHCS4SYoJ67Kk21Osu2TeyBxXMgOXOwTVis+jodSXARmeAQS4ICH8YQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yxrLcY4OSGa/76HmkoqKydHWRfmgUJznAupUDmYsGys=; b=An7yc/jdEUbrsAJ8AZI1dEE3ijB/BzUlNK+QtlIZwEjIvMYXvGKPtWIrR8a1aRZ/0OHjRFTnDTZuqDDmRxDkoNJL3y9dOSlfsP6qm6KRbbhNCXbZAQuUxJRmSU5qagJ6l77m5dJjTsC+s6WOKc4iYSsmtEmseSkZhZeGGh8xtqvMEUvM/rWNWrmXSfLLTYTKKuetcKx3xD860ZG2aRlDq/TW87SEbNyXRshSRF+ftzeD9JmUhVYlKA362iSANNVIsFfQd0JOIZ5KvFDPonuNzjqm6wUfshGDEnxrf5t7YH+23SnbAJx24KNf0GNkFAyd1kJxcMkF+FFCfIuGOO+PXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yxrLcY4OSGa/76HmkoqKydHWRfmgUJznAupUDmYsGys=; b=XHt9bYBQ2eQJxVV5A2lnU3LZBKc7St7oBHwgB/ih679QKk7UeiHWCdF8xphqkIqANEkpy/zv447peNYqC7dxnO9A9xhx7Q7jcthbi15IpH0CrfZFeqnXLW4hnESCl02ppMEjmVJtGO7e/APwi73NLH67Xi9wOapW7lVOzZ45lhtZX8gf/XcTUbgvL2vIDtaPc7pMKFgS7ktgN/K38Xk0DqEXdWtsgw5BHmrE7AVElXCdSwy6dgN9hVbjxbJI24/uPCswq39BfaMfGJ7yuWbGoIN8T93EE96yef0S8wKLXYaCYAvQrn2u98lW4VmRlBBjNN+DazE/4vKfF98yrpEudg== Received: from DM6PR02CA0125.namprd02.prod.outlook.com (2603:10b6:5:1b4::27) by MW4PR12MB7142.namprd12.prod.outlook.com (2603:10b6:303:220::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.19; Thu, 6 Oct 2022 11:02:00 +0000 Received: from DM6NAM11FT037.eop-nam11.prod.protection.outlook.com (2603:10b6:5:1b4:cafe::a5) by DM6PR02CA0125.outlook.office365.com (2603:10b6:5:1b4::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.28 via Frontend Transport; Thu, 6 Oct 2022 11:02:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DM6NAM11FT037.mail.protection.outlook.com (10.13.172.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5709.10 via Frontend Transport; Thu, 6 Oct 2022 11:02:00 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.26; Thu, 6 Oct 2022 04:01:45 -0700 Received: from nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Thu, 6 Oct 2022 04:01:42 -0700 From: Dariusz Sosnowski <dsosnowski@nvidia.com> To: Thomas Monjalon <thomas@monjalon.net>, Ferruh Yigit <ferruh.yigit@xilinx.com>, Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> CC: <dev@dpdk.org>, Viacheslav Ovsiienko <viacheslavo@nvidia.com>, Matan Azrad <matan@nvidia.com>, Ori Kam <orika@nvidia.com>, Wisam Jaddo <wisamm@nvidia.com>, Aman Singh <aman.deep.singh@intel.com>, Yuying Zhang <yuying.zhang@intel.com> Subject: [PATCH v2 0/8] ethdev: introduce hairpin memory capabilities Date: Thu, 6 Oct 2022 11:00:57 +0000 Message-ID: <20221006110105.2986966-1-dsosnowski@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220919163731.1540454-1-dsosnowski@nvidia.com> References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT037:EE_|MW4PR12MB7142:EE_ X-MS-Office365-Filtering-Correlation-Id: 244079b8-7a7f-4ab5-7dc6-08daa78a3172 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Of8Fpu8UiwJNCVzOUo5ldYAFEd6koLmW/WbrDUw16UHFnNcZqebl8UuQAZWe6qxNgsVon0jD26L/PKQCEpMLVysRrDXbtBx3QLCyFvIZLRrczeyE2AibWrDQIdDUJz37Aj974vJ7HI70vdXpME3OUFgxPFGCQVablhYaoIiAvO60R2lttSDjCTtc66hzNKCjt8cXJT5MOwZWL+dFXlcnPxU9jxyVLlGuHfn6Jlte0w4r4V2z7N4OMiQP540bHB1mCI0iOfjdo5IjWqcdusZ0a/6U9r/ReB2i1S9Gj3KtcrN6UR/WVJaODCuuANlWwV6hcJAqYAYT43BqzMki8bh0aewFD+n4F9AgQcvSokBQ6Ogo4xodaBxfdLCA7f0JkCWuQjJ6G1X9zsvPUfqvYHEk5nlk86gVq72deJWLHOFapoowShzz7TNDualO1sHWRThCIljNXEB4zUvhVTAb6j+xdD0MeSSpmBbFGmiFmGBmr9kPlTRpmxj/22OtdGEhUWLjEe/WMfT3vhVds5MF/FQXpn90CYIXg+ybQVeRQVTE+zwNT1WAuYiV/oXjrT4Ns6YTOhLQkEQFKbEBKPKTNe8zNfQn3CDBv4f3HYr584ChNnoaIjGIKyZbqWiaNPwX/zV/zOttKpXlaPrOgHtEmOVBKAYpagYPORvd81MDJkb1ui/+iX8BSHSoFEE+Sek+IyU+BhvYORRbekkQqf1WknKZmL6cMud2vxdWaPvVtD2l6/2kgz9FZArB5L5eEZw7kNbAb1yKr8v6+MG5rYe/5xAI3Sl73ULzOkdtkrub1FW28XtYx1WPBFPMOB3kx+TB2Vc3eiN9TlOP2k+98W3XY/9t0a/BDBKddTJiVM+YpIJFV/c= X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230022)(4636009)(136003)(396003)(376002)(346002)(39860400002)(451199015)(36840700001)(40470700004)(46966006)(36756003)(86362001)(110136005)(36860700001)(336012)(16526019)(1076003)(186003)(2616005)(356005)(7636003)(82740400003)(47076005)(426003)(6286002)(26005)(7696005)(966005)(478600001)(316002)(6666004)(54906003)(70586007)(8676002)(4326008)(70206006)(8936002)(82310400005)(5660300002)(2906002)(83380400001)(40460700003)(40480700001)(55016003)(41300700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2022 11:02:00.1383 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 244079b8-7a7f-4ab5-7dc6-08daa78a3172 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT037.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB7142 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org The hairpin queues are used to transmit packets received on the wire, back to the wire. How hairpin queues are implemented and configured is decided internally by the PMD and applications have no control over the configuration of Rx and Tx hairpin queues. This patchset addresses that by: - Extending hairpin queue capabilities reported by PMDs. - Exposing new configuration options for Rx and Tx hairpin queues. Main goal of this patchset is to allow applications to provide configuration hints regarding memory placement of hairpin queues. These hints specify whether buffers of hairpin queues should be placed in host memory or in dedicated device memory. For example, in context of NVIDIA Connect-X and BlueField devices, this distinction is important for several reasons: - By default, data buffers and packet descriptors are placed in device memory region which is shared with other resources (e.g. flow rules). This results in memory contention on the device, which may lead to degraded performance under heavy load. - Placing hairpin queues in dedicated device memory can decrease latency of hairpinned traffic, since hairpin queue processing will not be memory starved by other operations. Side effect of this memory configuration is that it leaves less memory for other resources, possibly causing memory contention in non-hairpin traffic. - Placing hairpin queues in host memory can increase throughput of hairpinned traffic at the cost of increasing latency. Each packet processed by hairpin queues will incur additional PCI transactions (increase in latency), but memory contention on the device is avoided. Depending on the workload and whether throughput or latency has a higher priority for developers, it would be beneficial if developers could choose the best hairpin configuration for their use case. To address that, this patchset adds the following configuration options (in rte_eth_hairpin_conf struct): - use_locked_device_memory - If set, PMD will allocate specialized on-device memory for the queue. - use_rte_memory - If set, PMD will use DPDK-managed memory for the queue. - force_memory - If set, PMD will be forced to use provided memory configuration. If no appropriate resources are available, the queue allocation will fail. If unset and no appropriate resources are available, PMD will fallback to its default behavior. Implementing support for these flags is optional and applications should be allowed to not set any of these new flags. This will result in default memory configuration provided by the PMD. Application developers should consult the PMD documentation in that case. These changes were originally proposed in http://patches.dpdk.org/project/dpdk/patch/20220811120530.191683-1-dsosnowski@nvidia.com/. Dariusz Sosnowski (8): ethdev: introduce hairpin memory capabilities common/mlx5: add hairpin SQ buffer type capabilities common/mlx5: add hairpin RQ buffer type capabilities net/mlx5: allow hairpin Tx queue in RTE memory net/mlx5: allow hairpin Rx queue in locked memory doc: add notes for hairpin to mlx5 documentation app/testpmd: add hairpin queues memory modes app/flow-perf: add hairpin queue memory config app/test-flow-perf/main.c | 32 +++++ app/test-pmd/parameters.c | 2 +- app/test-pmd/testpmd.c | 24 +++- app/test-pmd/testpmd.h | 2 +- doc/guides/nics/mlx5.rst | 37 ++++++ doc/guides/platform/mlx5.rst | 5 + doc/guides/rel_notes/release_22_11.rst | 10 ++ doc/guides/testpmd_app_ug/run_app.rst | 10 +- drivers/common/mlx5/mlx5_devx_cmds.c | 8 ++ drivers/common/mlx5/mlx5_devx_cmds.h | 5 + drivers/common/mlx5/mlx5_prm.h | 25 +++- drivers/net/mlx5/mlx5.h | 2 + drivers/net/mlx5/mlx5_devx.c | 170 ++++++++++++++++++++++--- drivers/net/mlx5/mlx5_ethdev.c | 6 + lib/ethdev/rte_ethdev.c | 44 +++++++ lib/ethdev/rte_ethdev.h | 68 +++++++++- 16 files changed, 422 insertions(+), 28 deletions(-) -- v2: * Fix Windows build by using mlx5_os_umem_dereg defined on both platforms to allocate memory for Tx hairpin queue. * Added hairpin section to mlx5 PMD. * Added info about new hairpin configuration options to DPDK release notes. 2.25.1