From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 351F8A0032; Sat, 18 Jun 2022 10:48:27 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C346A40F19; Sat, 18 Jun 2022 10:48:26 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2052.outbound.protection.outlook.com [40.107.223.52]) by mails.dpdk.org (Postfix) with ESMTP id 2317E40E2D for ; Sat, 18 Jun 2022 10:48:26 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Sn16mFkg9FmQZ6aUFh8sStWoPihWDCrkExsOFsNhtTEHPhAGqXDLwloucfSH4ogwVwmbMUJd6tJ1j/3S5WfrriqIl3j8i6UcjMHmW++6sW4lRngRyu0KJUj2wfdA978wgmvpsaKjHdFlC+byyByJXs6HiSlEIhbTL0HDkehUpkuNyGS9gf4Kw2Wm5OBA23pWeqFJNAELicm1ndm+lxe4faVDuvyJeZrQuunpBsYiKpLYgTE20ThB83e1ZBddvblgIZFzr9ruJPBPG8rxdOJM8pmh/MRN9XWUiP76MKj9V84XyE/LAG9B8rEbKTrFEmuBdU1oAIWNFxt/cyOR7qaRZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wd1rInazf5xzU+J1j+LTzKA+ZOvqk4VqHWzQTq798iE=; b=lgKvEpOvH+DDVTsYSryLRCF+JySfIhyryt3aBLhjdzNAWwtOyWvoDM3xE4PCKFwBP+qil3Rx3ZsF4s5zyWS+TWTsCuoVd+H5CSFb/nd/ZNZGekiTwQ6HbEPztmyNmj8Qx0Yh1y/led+QmDyJne5SuAsyMyOgNqRjHtpwf9YgVncQDYlDmQEbLlrxLaod2v2opjt4ukqvZxSdf1fMR1F5ljxLHkZMg46+1+sNmaSAvZaK1AM6peRGpngZTrzhhlYTGd6QT94BM8DkRNXQrpJE75Goj+0eyaVSWr5391SruPFwByHf0TVu+/Hv2jplz8qZhoy6VlRpEp9wJbJHif6w3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wd1rInazf5xzU+J1j+LTzKA+ZOvqk4VqHWzQTq798iE=; b=JrGApKDkFg6wtn4gJdx9/ZXM5UisHqWZB9FdC7ukBhWDkn+BBzmy2L3ZQMMKY1kI3+CrK7Pscc63YUXwaZOCc5UYRZd3rSh6U0dGvkBnEX70LzAMaUbZVuPSG9RaKtiRYs8AmUyov2TAPslNr1N7onnOztz2zoGQ/AAsCNOvziOgkoMUT+KKmoahp+pu9mkcKZslcAgozdAcl53XRHtYtQNxY53GRLySnRoY3P9th2/sjUa1E8bQ1K4AjjOoBCZzl8UVzRhv0ZmFjkbuj1eOL7ZzxffOx45ZTqRifvw5/6n6RuIUka5F3IaVswhvzoeifVq7P7ubu1ysQlxF2Jp14w== Received: from MWHPR1601CA0003.namprd16.prod.outlook.com (2603:10b6:300:da::13) by DM6PR12MB3708.namprd12.prod.outlook.com (2603:10b6:5:1c5::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.15; Sat, 18 Jun 2022 08:48:24 +0000 Received: from CO1NAM11FT014.eop-nam11.prod.protection.outlook.com (2603:10b6:300:da:cafe::2d) by MWHPR1601CA0003.outlook.office365.com (2603:10b6:300:da::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.19 via Frontend Transport; Sat, 18 Jun 2022 08:48:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.234) by CO1NAM11FT014.mail.protection.outlook.com (10.13.175.99) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5353.14 via Frontend Transport; Sat, 18 Jun 2022 08:48:23 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Sat, 18 Jun 2022 08:48:23 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Sat, 18 Jun 2022 01:48:20 -0700 From: Li Zhang To: , , , CC: , , , , Subject: [PATCH v3 00/15] mlx5/vdpa: optimize live migration time Date: Sat, 18 Jun 2022 11:47:50 +0300 Message-ID: <20220618084805.87315-1-lizh@nvidia.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220408075606.33056-1-lizh@nvidia.com> References: <20220408075606.33056-1-lizh@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: af2f9355-08aa-46fa-32f0-08da51074dc2 X-MS-TrafficTypeDiagnostic: DM6PR12MB3708:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0SXZpuwlF92pEFcZjNIOSGv1hcRXFhfv9/FMgA86FqMxsgJqsSwDemQ2BryiRJaph8l3uJ3R51/qWkm3fpIEC3PbT8CyUKx57Ggk31s6AOBxJCf0kVSwhFS4HAyKzvh0mAEiq6RRv5MwxrrpmEqpJ/1g3zBeZJZ8bxghcjZKy4VjPVKjErSOrWiQ9u9kLSDtSDjq/VnTb5Kvh1/NS9RCgwC9NB842F0XwyPC+yi/OhMb0hgx9i9wjLwq4fQZmyS/xjDrdo0S/1L9FkgQks8uWNgq3LZWEkSXbduJLlog43SkQUKMQ45Hi1+Wl6luOetzGu/cq4997to2E8zILSvNFLGToGEo2/cHMsK+VRZid1FvqHmxv+C4fJ/F+D3NN0L4FxLFxjwaff8Q6WOBl3srgO9bge4arITIfsdjp0BtlBu2XCtpLNDA6Ttj/GwtXKpjfWvVhB2ShCXDLXk2aVIguNvoCxLCg+vzOFxdI1HfEOZq64A8i+jdNMuadE5TR14kaIZohJhJk22rU4m1CiZONhCeO+PI33X2uvG+/+KdHjmQQc5N4PVAFngIjq6onSIpMdLFX8jOTjl8AcUptS70bl04el5IxNMWbINXtoVCyyc5LD2uxnYNgsWASZdyfcQhFJb5J1lBWpxZfl4qzakPBJeHwcw84HxK9MYlDSyaMdYWWR+6ogsVnDM5fLdP0wUimF/7oWbvH0bO5d7tzUdAmbgtWcp/qCT00Skx+97LdVE5jXaffHFf9lUQZO4MsyRPX5ucv0Un4T0eCMHpKCGsmVVfnjech0tcjyYkpcuUu6y9J8tt7IrVzmjnVMjaFVz5+k86WxTVBHFi9AZjFKzpSw== X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230016)(4636009)(46966006)(36840700001)(40470700004)(83380400001)(8676002)(8936002)(70206006)(54906003)(2906002)(186003)(55016003)(1076003)(5660300002)(16526019)(81166007)(966005)(498600001)(40460700003)(86362001)(2616005)(36860700001)(110136005)(7696005)(6666004)(356005)(6636002)(82310400005)(26005)(4326008)(426003)(316002)(70586007)(36756003)(336012)(6286002)(47076005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Jun 2022 08:48:23.6446 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: af2f9355-08aa-46fa-32f0-08da51074dc2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT014.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3708 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Allow the driver to use internal threads to obtain fast configuration. All the threads will be open on the same core of the event completion queue scheduling thread. Add max_conf_threads parameter to configure the maximum number of internal threads in addition to the caller thread (8 is suggested). These internal threads to pipeline handle VDPA tasks in system and shared with all VDPA devices. Default is 0, don't use internal threads for configuration. Depends-on: series=21868 ("vdpa/mlx5: improve device shutdown time") http://patchwork.dpdk.org/project/dpdk/list/?series=21868 RFC ("Add vDPA multi-threads optiomization") https://patchwork.dpdk.org/project/dpdk/cover/20220408075606.33056-1-lizh@nvidia.com/ V2: * Drop eal device removal patch in series. * Add release note in release_22_07.rst. V3: * Fix comments about commit log issue. * Avoid cutting logs. Li Zhang (12): vdpa/mlx5: fix usage of capability for max number of virtqs common/mlx5: extend virtq modifiable fields vdpa/mlx5: pre-create virtq at probe time vdpa/mlx5: optimize datapath-control synchronization vdpa/mlx5: add multi-thread management for configuration vdpa/mlx5: add task ring for MT management vdpa/mlx5: add MT task for VM memory registration vdpa/mlx5: add virtq creation task for MT management vdpa/mlx5: add virtq LM log task vdpa/mlx5: add device close task vdpa/mlx5: add virtq sub-resources creation vdpa/mlx5: prepare virtqueue resource creation Yajun Wu (3): vdpa/mlx5: support pre create virtq resource common/mlx5: add DevX API to move QP to reset state vdpa/mlx5: support event qp reuse doc/guides/rel_notes/release_22_07.rst | 5 + doc/guides/vdpadevs/mlx5.rst | 25 + drivers/common/mlx5/mlx5_devx_cmds.c | 77 ++- drivers/common/mlx5/mlx5_devx_cmds.h | 6 +- drivers/common/mlx5/mlx5_prm.h | 30 +- drivers/vdpa/mlx5/meson.build | 1 + drivers/vdpa/mlx5/mlx5_vdpa.c | 270 ++++++++-- drivers/vdpa/mlx5/mlx5_vdpa.h | 152 +++++- drivers/vdpa/mlx5/mlx5_vdpa_cthread.c | 360 ++++++++++++++ drivers/vdpa/mlx5/mlx5_vdpa_event.c | 160 ++++-- drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 132 +++-- drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 270 ++++++---- drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 22 +- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 654 ++++++++++++++++++------- 14 files changed, 1777 insertions(+), 387 deletions(-) create mode 100644 drivers/vdpa/mlx5/mlx5_vdpa_cthread.c -- 2.31.1