From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 27AC6A00BE; Fri, 8 Apr 2022 09:56:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C6FEF4067E; Fri, 8 Apr 2022 09:56:46 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2063.outbound.protection.outlook.com [40.107.92.63]) by mails.dpdk.org (Postfix) with ESMTP id A874C4003F for ; Fri, 8 Apr 2022 09:56:45 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iwKjXNLJEwULz1RUCLbEPaAEDwGUqal9YIr4x10y2QCo//NNFVO1JOrh15iPLwmDb9HRQvM9z9Vf7y4nOaRe7y26zkViltg18UzE0E/E0xEXfa15psZ4zoRG3ujxhzL1D3PSBaW3q9b8K7CTdhvW3G4mYlPbVB7H3czpchrFoISgRwgLWL0/wiZPLAnnESl2YGJKoe4jEgxnQXSfIxDz9UXtLRZJHrQCQyKsTcUngtYSk7y8qgVZxNWcDeBS6eyct21fbSFSIkVoV9B/Rx5Cm7m9ruQRi8a65r//m2/MggJnrhvgDiPO+0bbLuiN+kqlmXCHGtzBOW4U2WnwgnEYaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MEZEvNxbzk+rUr3LGEumbMytr13IpYfmq/njri8w6Rs=; b=GoDFwxXrUf9dfru1yFjOKXupV+36rWTi9umV6R3fjqyMWP460qcQ6cRE8eBP4IPQdYjb7KCK2PrL9ggNwq9h2vbjINe1+6TidvoXtMgAqP0S1lE6eUl3aBKaTG+2KFk+pTyb/DTEUDs8lNdwqVmnOcL9jIQF/s6uTBhJUn47266R79xcV3gxJ09OsWhwgKU1VrLX1rQD3sWULSLOWnathuP5ZhzR6zLnVusiHssVIqER0yCLG2B8wDTbr3ZwJ/B+TVdeSCnSm9Ys+yo3B2rUfnfvhBiiNB8Ft06CXxgrMAfhiWCtjJzHyqtX103wwA0wy5ae8hC1faVx3hixXYbTzA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MEZEvNxbzk+rUr3LGEumbMytr13IpYfmq/njri8w6Rs=; b=pK45DeOVM3vhzlvqx2LUF4FeOW2IIqcCTaCWgWldFSBHxQi0yHQGJgVlz8s6iaWfe5Lc3J2FJm1Dn+0oWdZqF2vu/zDFjd3IWOaBASSI/nU0io++27/nQAIJvVu0880r2xTgKVatN+eSGPCgnwYVgwKG/tn55CXrtvbq/bHnXvk8Xa5X95VZVvgzksu2CfQVRXm3ItDMM7scrU6n6riFgxMeqk8chQsYgf09Iv2g7cs9LfpvjW44nWa6RdOz+ds0DnVH4t8yqaI+Ve/UCwkVUkw3w/SAVPy3eVGy3489jj/doYphUsZxkEht6jpY48SzN0TTaD8T7vD7TkHTUAFh5Q== Received: from BN9PR03CA0948.namprd03.prod.outlook.com (2603:10b6:408:108::23) by DM5PR12MB2391.namprd12.prod.outlook.com (2603:10b6:4:b3::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5123.31; Fri, 8 Apr 2022 07:56:44 +0000 Received: from BN8NAM11FT052.eop-nam11.prod.protection.outlook.com (2603:10b6:408:108:cafe::95) by BN9PR03CA0948.outlook.office365.com (2603:10b6:408:108::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5144.25 via Frontend Transport; Fri, 8 Apr 2022 07:56:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.234) by BN8NAM11FT052.mail.protection.outlook.com (10.13.177.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5144.20 via Frontend Transport; Fri, 8 Apr 2022 07:56:43 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Fri, 8 Apr 2022 07:56:42 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Fri, 8 Apr 2022 00:56:40 -0700 From: Li Zhang To: , , , CC: , Subject: [RFC 00/15] Add vDPA multi-threads optiomization Date: Fri, 8 Apr 2022 10:55:50 +0300 Message-ID: <20220408075606.33056-1-lizh@nvidia.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 936c2271-d978-4bc6-9545-08da1935529c X-MS-TrafficTypeDiagnostic: DM5PR12MB2391:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7JW9PTIWy6iyRKrznQ3hxCvJn9cPTzWDdaq7pMwbMnIRPd6v4Xfh1/DjQsoS3RSHsOX2fp7Ohq9x9TAWDGEVzDJkcgMah0Wd3/k3Vl+u7d4wCukAF9lAjBk/XjyPBo0APHWD33XuHejt0GkJnsXwkfoNKB0scrQsoFo5cyQZln05Ve+S8f+vZUP3+04kaZoC9c2h5faG1ieyz+6ldJMPTRjQ1McIpp47XACPcO4x26PSxPSZGLtCHuPvMy6B586HigNLb6FKL+gC6ORBUaYMeUgb6rMmWCVSILsc9FpLDmB/LVQSjKm0+YLRFHl9+zV5/xU6tVk5AHX9TwxVC10rpQ8IPOiIgrLuuyIuFQtc9thm+jw+JeKn+ESHI/nYZ4w1vneqZCQU6VhizfT8w8meoL+qTRVTXp1zqTOuJ8vpQwvypCV4WlrzJ060NTkKBcqvs+aML9CHsBHxHQoJGaH9akkcSUC/CNiJRi+iooMIbj5/nzKkLZCynjFBqfAnC3brutqhETAIa1PPGbjyWCIL+XQ6qqa4dp0o/kyu1tvz+gF+hsNWLLVSRLTomHv1NeGgn0ao05pqS2R7LoBccWH/37KQ0CYiKrK+nvhglHtnaROcpi+rk3G4u20AFuVY/TlYxA6fyC/jZYehH/SNSJeM5+TiI5PKVhTGon/Kjzg2cH7pDEUCOpOxnyfnf0XVytbbFyDw05FNtEm/LyZHBtN7M+EOFtq6qiFcPd+MHdm8Re2C/LmkVQ2K78yYAyG4VA/rtUKkJdgJOlniL0yDDabELIAX4QXQQixWzCEBdCTb4P4= X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(70206006)(55016003)(70586007)(26005)(107886003)(54906003)(83380400001)(2616005)(8676002)(110136005)(356005)(2906002)(47076005)(16526019)(316002)(186003)(1076003)(4326008)(8936002)(82310400005)(7696005)(36860700001)(508600001)(81166007)(36756003)(426003)(336012)(6286002)(6666004)(40460700003)(86362001)(5660300002)(966005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Apr 2022 07:56:43.4582 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 936c2271-d978-4bc6-9545-08da1935529c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT052.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB2391 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Allow the driver to use internal threads to obtain fast configuration. All the threads will be open on the same core of the event completion queue scheduling thread. Add max_conf_threads parameter to configure the maximum number of internal threads in addition to the caller thread (8 is suggested). These internal threads to pipeline handle VDPA tasks in system and shared with all VDPA devices. Default is 0, don't use internal threads for configuration. Depends-on: series=21868 ("vdpa/mlx5: improve device shutdown time") http://patchwork.dpdk.org/project/dpdk/list/?series=21868 Li Zhang (11): common/mlx5: extend virtq modifiable fields vdpa/mlx5: pre-create virtq in the prob vdpa/mlx5: optimize datapath-control synchronization vdpa/mlx5: add multi-thread management for configuration vdpa/mlx5: add task ring for MT management vdpa/mlx5: add MT task for VM memory registration vdpa/mlx5: add virtq creation task for MT management vdpa/mlx5: add virtq LM log task vdpa/mlx5: add device close task vdpa/mlx5: add virtq sub-resources creation vdpa/mlx5: prepare virtqueue resource creation Yajun Wu (4): examples/vdpa: fix vDPA device remove vdpa/mlx5: support pre create virtq resource common/mlx5: add DevX API to move QP to reset state vdpa/mlx5: support event qp reuse doc/guides/vdpadevs/mlx5.rst | 25 ++ drivers/common/mlx5/mlx5_devx_cmds.c | 77 +++- drivers/common/mlx5/mlx5_devx_cmds.h | 6 +- drivers/common/mlx5/mlx5_prm.h | 30 +- drivers/vdpa/mlx5/meson.build | 1 + drivers/vdpa/mlx5/mlx5_vdpa.c | 227 +++++++++- drivers/vdpa/mlx5/mlx5_vdpa.h | 147 ++++++- drivers/vdpa/mlx5/mlx5_vdpa_cthread.c | 362 ++++++++++++++++ drivers/vdpa/mlx5/mlx5_vdpa_event.c | 160 +++++-- drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 133 ++++-- drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 268 ++++++++---- drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 20 +- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 582 ++++++++++++++++++-------- examples/vdpa/main.c | 4 + 14 files changed, 1674 insertions(+), 368 deletions(-) create mode 100644 drivers/vdpa/mlx5/mlx5_vdpa_cthread.c -- 2.27.0