From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 433BEA0542; Mon, 6 Jun 2022 13:47:18 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 34A834067E; Mon, 6 Jun 2022 13:47:18 +0200 (CEST) Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2084.outbound.protection.outlook.com [40.107.95.84]) by mails.dpdk.org (Postfix) with ESMTP id 9FEA84021E for ; Mon, 6 Jun 2022 13:47:16 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IIsx5yDNd2d4xyj7roAIwqj00kyQoqFp+rzbYvAZ1zYpJeb/oyWKdfGK6g0Gh/PK3eR9idJnUv9rnOOGi63ep3BgN1Mx0Pcj3+gEUTQw3mLhPNs2rZ/nlDYZz7QXMcnUGxnbd/GZN2H5D7ud9QSQvlsQDJxjmf4w4wMrzJHBlRc0ZcPjDoC4R08PjpLBQbX6Yy5J+UgnYQAfgtvAzSMst+BVmCpsZakYdpXEaYgNPnFa2jw2MrzJfgGKW4L312LZY4hmoffC1jz98UjeIAcJBRaQySntpZ5ZYactkJ8bzj4z9CkDvCF0iG7d3r3jd528IRJh/uXx0owJ+7RAETOnqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=S8fahRVD/la/8/eqrCfbWa6cSFPwRa/ORUQpmkC9iHg=; b=dONSEnBmrbzb4iiAC4TbJ1tA9a97Jr50jvCMinQ4AYS4RYDFikQxD87NsK+gu8im9lD3U6TG7O7jEf/Ly+Haz+6GZKxcHMM2SANI5W0q2NeX98zPq6xT/PmPEiE/pmjr0Sdt8Ogs2iOv7y6x+sf2Op6VGM+yZJBxYp3LCCLMsRgVqXozUmvcvU5c0elew2dprQceyYc+/HcKEYZurzKgfDExEXPFl3DCppVaaU9U2pYE1f8E3W/KGjRDfEoFoK33U10k4yoqtAbFieT/XafLQUf/4W9WWWUfrT6zRxHBaJzRV/eO5bP1aWkpELoST8dpE1bCQ+9syEg+Mt7p8zosTw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=S8fahRVD/la/8/eqrCfbWa6cSFPwRa/ORUQpmkC9iHg=; b=nSC1ZAuxlH1medCP9l2hx+LYbBykK0Kr0UJxoloH6Q0EWt1eq/ySZA1mcV4kFDhE5L6o0X1O9aNr4czRxlAfkrJs3b2n+Mnf+9TaoQ+vDxCZuPXiNM/ucCHChW9kNs3MyRYsHAl3hmTQuQzvb8f78x7qR+JzOBXUmN4tb7KJyXteopbb22wdt0D/BSk6R1UVGyZLAXOYS4SZ1QhHCJ2jDyUnqdbCmhsWoMXME/HR/ul3MnfbXn/aAH7wpigHbYrLAqYAl0CHbAWWXnGMHLaoFwvHbrvX9gXwjhgjaWRo+CUSrnRHVxfyq+Tk3+htupSm80uwOT3t5SoB7mbOkqHJ3w== Received: from CO2PR07CA0065.namprd07.prod.outlook.com (2603:10b6:100::33) by BN6PR12MB1923.namprd12.prod.outlook.com (2603:10b6:404:107::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13; Mon, 6 Jun 2022 11:47:15 +0000 Received: from CO1NAM11FT011.eop-nam11.prod.protection.outlook.com (2603:10b6:100:0:cafe::7d) by CO2PR07CA0065.outlook.office365.com (2603:10b6:100::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5314.13 via Frontend Transport; Mon, 6 Jun 2022 11:47:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.234) by CO1NAM11FT011.mail.protection.outlook.com (10.13.175.186) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5314.12 via Frontend Transport; Mon, 6 Jun 2022 11:47:13 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Mon, 6 Jun 2022 11:47:13 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Mon, 6 Jun 2022 04:47:10 -0700 From: Li Zhang To: , , , CC: , , , Subject: [PATCH v1 00/17] Add vDPA multi-threads optiomization Date: Mon, 6 Jun 2022 14:46:33 +0300 Message-ID: <20220606114650.209612-1-lizh@nvidia.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220408075606.33056-1-lizh@nvidia.com> References: <20220408075606.33056-1-lizh@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 32d8d6cb-0aa0-4291-3244-08da47b24c92 X-MS-TrafficTypeDiagnostic: BN6PR12MB1923:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4NilIvOiXtYvDEXubiOk4h509MFK+bs4q3xk+pkBrMl17eQir9Y9lFh76WvJ/s8x2wNtbnlMzaUkZrXYUBoIFBIr99WEEdLMsJfXA6seM9JZunD9Ue56sv5/+pgQmKPm4gOs2IHbouttT5lNmS/7QRCaPqYipwnBEQtssrFdcqareWnYL2pDWigfPP9h74zxLZeNMln6gGjI+aRpTnsEs+1lOKLJ4j6oLfK59vl/sfMQ/6L+zgoeE5a69LLPlw9vWP7rdpep7SWwpd9Q1h2YGR3LQ1+bOxubPKb4sUCR2ZxFd0W6CUn4/r8/RG0lfFaLmz/+RFhkcg4sQRx9gWvBOqFWM/8ROWSPEPfRRp3p3mVa1GKfHm0WAIUgW1441pTECWXT4REOducGc/wJLIouclNQfKXcX6CQzJ4qCnKO6z5yToDCxo4YrH7HC+RdBwb/xHHs5mSIHnvVl4MgkP/G2r3J9o335QBaKr+qHMPMzJusPoP+1kIDapV8MXFgd+hWb9PD7+iKB6aH/fhJWP/xifdmFYGoNKIooAXUt9+p8mj15ILzowzWxA/2Gqpsw5Xv0hsYbNkET4RrpnbqiUBAk8VGcAXgomENpOMuoOSMzt21bufPjSEGuvQanFlKbTtClF28PbmndXJbCU7o/OBrwdVDGA0PUFE1m3XngSEzzO0iWSmC2Mjt8SxkiN+9mk1r3z5sk4R9Zojwc1/GOuf39ZaTH2xFLSxqwZSeDP3J+GbZGn1FJEJHghjqeU7AxGsitAU/sOFkOOJUtYEjU+9a4ELoOTHVmuLdYFm9N+l0bcAVtbXf03+dZ7FpKzKZL4P3YK1Ne1JGiDJ7D/ZfU1vIDA== X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(46966006)(36840700001)(336012)(47076005)(1076003)(107886003)(6666004)(426003)(36756003)(966005)(186003)(55016003)(2616005)(16526019)(6286002)(83380400001)(36860700001)(26005)(7696005)(81166007)(40460700003)(110136005)(8936002)(356005)(316002)(82310400005)(508600001)(4326008)(8676002)(86362001)(5660300002)(6636002)(70586007)(54906003)(70206006)(2906002)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Jun 2022 11:47:13.9653 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32d8d6cb-0aa0-4291-3244-08da47b24c92 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT011.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR12MB1923 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Allow the driver to use internal threads to obtain fast configuration. All the threads will be open on the same core of the event completion queue scheduling thread. Add max_conf_threads parameter to configure the maximum number of internal threads in addition to the caller thread (8 is suggested). These internal threads to pipeline handle VDPA tasks in system and shared with all VDPA devices. Default is 0, don't use internal threads for configuration. Depends-on: series=21868 ("vdpa/mlx5: improve device shutdown time") http://patchwork.dpdk.org/project/dpdk/list/?series=21868 RFC ("Add vDPA multi-threads optiomization") https://patchwork.dpdk.org/project/dpdk/cover/20220408075606.33056-1-lizh@nvidia.com/ Li Zhang (12): vdpa/mlx5: fix usage of capability for max number of virtqs common/mlx5: extend virtq modifiable fields vdpa/mlx5: pre-create virtq in the prob vdpa/mlx5: optimize datapath-control synchronization vdpa/mlx5: add multi-thread management for configuration vdpa/mlx5: add task ring for MT management vdpa/mlx5: add MT task for VM memory registration vdpa/mlx5: add virtq creation task for MT management vdpa/mlx5: add virtq LM log task vdpa/mlx5: add device close task vdpa/mlx5: add virtq sub-resources creation vdpa/mlx5: prepare virtqueue resource creation Yajun Wu (5): eal: add device removal in rte cleanup examples/vdpa: fix devices cleanup vdpa/mlx5: support pre create virtq resource common/mlx5: add DevX API to move QP to reset state vdpa/mlx5: support event qp reuse doc/guides/vdpadevs/mlx5.rst | 25 + drivers/common/mlx5/mlx5_devx_cmds.c | 77 ++- drivers/common/mlx5/mlx5_devx_cmds.h | 6 +- drivers/common/mlx5/mlx5_prm.h | 30 +- drivers/vdpa/mlx5/meson.build | 1 + drivers/vdpa/mlx5/mlx5_vdpa.c | 270 +++++++++-- drivers/vdpa/mlx5/mlx5_vdpa.h | 152 +++++- drivers/vdpa/mlx5/mlx5_vdpa_cthread.c | 360 ++++++++++++++ drivers/vdpa/mlx5/mlx5_vdpa_event.c | 160 +++++-- drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 128 ++++- drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 270 +++++++---- drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 22 +- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 654 +++++++++++++++++++------- examples/vdpa/main.c | 5 +- lib/eal/freebsd/eal.c | 33 ++ lib/eal/include/rte_dev.h | 6 + lib/eal/linux/eal.c | 33 ++ lib/eal/windows/eal.c | 33 ++ 18 files changed, 1878 insertions(+), 387 deletions(-) create mode 100644 drivers/vdpa/mlx5/mlx5_vdpa_cthread.c -- 2.31.1