From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 04DD042CFF; Tue, 20 Jun 2023 03:24:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 93AB942D3E; Tue, 20 Jun 2023 03:24:39 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2079.outbound.protection.outlook.com [40.107.237.79]) by mails.dpdk.org (Postfix) with ESMTP id 0FA3E42D12 for ; Tue, 20 Jun 2023 03:24:38 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H7F5fRnZ15n6KXrPf6dPXsvZUfPaJaDSB2ANBk3BAAY7S8XU5R5bu7AAn1OUHLCOL56wP53eK9wy91AWns7guactOBH7DsEhVnHQFW8MLPIYB+nx9q8W9HkyGbcH6CrHtKOb4uNe85OrMTm8MCJ90Op0uuoSbdOa21+p9WXomq9/mEtCybdYJQEGf5qKJ1skkYTWDJnCUDtsU4uw44euzSruVNLe/sXekPZGFhYCK0GcSjmslh2TpEkrcbrA23H1HkfNS5dqbc28WnmmbmMoBj5Zm2EEPvL684esow6aiHLEZ273uuG3NIaQDU0/8UvVb4xUHqvKa+XIl0Np2yPCnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EDHHzyJQoIeey/4TxKs4B02+RV+6GjJFXo2CvyGx0Js=; b=mDOYT2IPBbqDKcROd2LPTis5ZrnZQkFvFAH1Nxsk4NWzppsEZ95uY6GMFt5VKz90Kz6/G9Bp20XpxdFhceCcnz0ksgZ2fPr2UwvorU1u7ZKCtX9RRoRupFnO7IgBt8bEj6ukiRaMMGLZqm+RNtKTg9cF0M7qDQwIYG9OSJwACpexmVir9ffzAPJ6aYE8hsVJPEEOuiP0QJIrH5KAePdXN4/vHL1KGerd/BAXHHOu1F4OBqpS/bfFnlQ0Dsb5XzbczplxeCHTz9vq06Rn8nzxjb5FrnQnMoXodDW4eVYfhf5lLC0R5pUBeXowZq7/7x/Pe+mLJ4GOz0wuXoXj9QyjNg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EDHHzyJQoIeey/4TxKs4B02+RV+6GjJFXo2CvyGx0Js=; b=pvtB29PfCe0LIHL7y5F5+YNx3nw27kAh3UgnBiQ5Mkwf9c7rpjeWjZBevKiJpIC9JMeIjOKG2KMWCPEAUVRgIBtOHBCF9IHmd3pKaK9aGtkZaJJSUATcu+V4YbDCTKkvz9s4IPn4LVhnpQ6QVS5d4nUTAflVxPAVLTpoS6HXaEBT6T6t1p4B58TKNsB2mzCdnbAFu+GmDeRr1SYsRGr6uNOlX8IwHrjpAHNFefU5UixB22iLJfM39UgRCwcY/Ftg0DPQ+dNPKFLygrGgeRzJ7HjjHpUH1fyqnX4f0KVQWHwxjlDhpo6kF1h70Ds3XgQhV18umC6o2MHp+zdWhR14uQ== Received: from SA0PR11CA0009.namprd11.prod.outlook.com (2603:10b6:806:d3::14) by MW4PR12MB6732.namprd12.prod.outlook.com (2603:10b6:303:1ea::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.37; Tue, 20 Jun 2023 01:24:35 +0000 Received: from SN1PEPF000252A3.namprd05.prod.outlook.com (2603:10b6:806:d3:cafe::a) by SA0PR11CA0009.outlook.office365.com (2603:10b6:806:d3::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.37 via Frontend Transport; Tue, 20 Jun 2023 01:24:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF000252A3.mail.protection.outlook.com (10.167.242.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.27 via Frontend Transport; Tue, 20 Jun 2023 01:24:35 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Mon, 19 Jun 2023 18:24:24 -0700 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Mon, 19 Jun 2023 18:24:21 -0700 From: Suanming Mou To: Matan Azrad , Viacheslav Ovsiienko , Ori Kam CC: , , Subject: [PATCH v3 6/9] common/mlx5: add WQE-based QP synchronous basics Date: Tue, 20 Jun 2023 04:23:19 +0300 Message-ID: <20230620012322.788099-7-suanmingm@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230620012322.788099-1-suanmingm@nvidia.com> References: <20230418092325.2578712-1-suanmingm@nvidia.com> <20230620012322.788099-1-suanmingm@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF000252A3:EE_|MW4PR12MB6732:EE_ X-MS-Office365-Filtering-Correlation-Id: 92ca5bf4-83f9-4fd1-4d50-08db712d1b9e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zgWh/9GiMFqgrai+Xdh0b19zkPl6LGyoSxV+L21U73P+uDYF/WTdwpM5mUHZuXDe6OYuWZyEaBiXY92DvKXeGUKva1lz+CNBKP0CSVkTxdFqBBZE8PWEU4VAF5qA6rJN4hidpwpkncbPXDXtRi1S6b8XmY9h+TejVXzR2CkoSj5kCqPwDLiQfw7Q2+DPpzx+U+iN+Ogk3jYiWoumAOc2xAvUW0X4kT9/VEsI+b46LnDreRGGhEl6D2psk/Zm44oE6obs+chS2JCt3Qa8LbKH57XJetXt5LF+0djE85jsBwBIREwKTTDJJjP8PVAdiwKjN3AQ4OAHH0HwVVDyk+YvRmhYMr1Twj99p0Zevs44d77g3mcs/a8s2WLfXnBwOBR/7XamEGGhXIsI7YNINH84uGkeNqvxLoWOekOPAr6W1ZU9/ghyAywwxT4Fr9xqXvzO9fgS+TyRmD1RlPLI2T67rW7H/AT4Wz+Jla76X3uY0/X03NrJWyvw/e7yETpZBTz/ODuXzUSndieqKumlVGx54FKzHl+FGTQxxeoCa+Dz8dkh4bYDzGJshJdTirRUaUpxPzYnce3sMQuJJA4lCRhCdSyw2hFCZRjAKnAx/Lea/HZplf0d41QDzEGVkaO7BJg9TSCTjRcaB9ku3qvOO//NyYL7VH7s4wu9E/Phsfx2D563pgtis35ZhndWWXLTw79GRIZEJ9QWjiu7LtKRb8sOXH5M1QI/1Pv8GraTPTif3fo= X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230028)(4636009)(376002)(396003)(39860400002)(346002)(136003)(451199021)(36840700001)(40470700004)(46966006)(40460700003)(2906002)(41300700001)(5660300002)(8676002)(8936002)(82310400005)(36756003)(55016003)(40480700001)(86362001)(7696005)(478600001)(336012)(47076005)(426003)(36860700001)(186003)(16526019)(6286002)(26005)(1076003)(2616005)(54906003)(110136005)(356005)(82740400003)(316002)(7636003)(4326008)(6636002)(70586007)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jun 2023 01:24:35.2321 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 92ca5bf4-83f9-4fd1-4d50-08db712d1b9e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF000252A3.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6732 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Nvidia HW provides a synchronous mechanism between QPs. When creating the QPs, user can set one as primary and another as follower. The follower QP's WQE execution can be controlled by primary QP via SEND_EN WQE. This commit introduces the SEND_EN WQE to improve the WQE execution sync-up between primary and follower QPs. Signed-off-by: Suanming Mou --- drivers/common/mlx5/mlx5_devx_cmds.c | 6 ++++++ drivers/common/mlx5/mlx5_devx_cmds.h | 3 +++ drivers/common/mlx5/mlx5_prm.h | 11 +++++++++++ 3 files changed, 20 insertions(+) diff --git a/drivers/common/mlx5/mlx5_devx_cmds.c b/drivers/common/mlx5/mlx5_devx_cmds.c index 4332081165..ef87862a6d 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.c +++ b/drivers/common/mlx5/mlx5_devx_cmds.c @@ -2475,6 +2475,12 @@ mlx5_devx_cmd_create_qp(void *ctx, attr->dbr_umem_valid); MLX5_SET(qpc, qpc, dbr_umem_id, attr->dbr_umem_id); } + if (attr->cd_master) + MLX5_SET(qpc, qpc, cd_master, attr->cd_master); + if (attr->cd_slave_send) + MLX5_SET(qpc, qpc, cd_slave_send, attr->cd_slave_send); + if (attr->cd_slave_recv) + MLX5_SET(qpc, qpc, cd_slave_receive, attr->cd_slave_recv); MLX5_SET64(qpc, qpc, dbr_addr, attr->dbr_address); MLX5_SET64(create_qp_in, in, wq_umem_offset, attr->wq_umem_offset); diff --git a/drivers/common/mlx5/mlx5_devx_cmds.h b/drivers/common/mlx5/mlx5_devx_cmds.h index cb3f3a211b..e071cd841f 100644 --- a/drivers/common/mlx5/mlx5_devx_cmds.h +++ b/drivers/common/mlx5/mlx5_devx_cmds.h @@ -559,6 +559,9 @@ struct mlx5_devx_qp_attr { uint64_t wq_umem_offset; uint32_t user_index:24; uint32_t mmo:1; + uint32_t cd_master:1; + uint32_t cd_slave_send:1; + uint32_t cd_slave_recv:1; }; struct mlx5_devx_virtio_q_couners_attr { diff --git a/drivers/common/mlx5/mlx5_prm.h b/drivers/common/mlx5/mlx5_prm.h index 6b48c6ca32..2d0a34ffbc 100644 --- a/drivers/common/mlx5/mlx5_prm.h +++ b/drivers/common/mlx5/mlx5_prm.h @@ -589,6 +589,17 @@ struct mlx5_rdma_write_wqe { struct mlx5_wqe_dseg dseg[]; } __rte_packed; +struct mlx5_wqe_send_en_seg { + uint32_t reserve[2]; + uint32_t sqnpc; + uint32_t qpn; +} __rte_packed; + +struct mlx5_wqe_send_en_wqe { + struct mlx5_wqe_cseg ctr; + struct mlx5_wqe_send_en_seg sseg; +} __rte_packed; + #ifdef PEDANTIC #pragma GCC diagnostic error "-Wpedantic" #endif -- 2.25.1