From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AB1A0A0547; Thu, 16 Jun 2022 04:31:23 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A0FDF42BB4; Thu, 16 Jun 2022 04:31:23 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2059.outbound.protection.outlook.com [40.107.94.59]) by mails.dpdk.org (Postfix) with ESMTP id EF9D5410D0 for ; Thu, 16 Jun 2022 04:31:21 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=enxMa6Pp8PE9WmvMpF+YpSj/0FretuG2ExXU4gZWrlNTVRJvIR0V7Jd3l3a3OVOKVq3LGuIcC/qgTgjQ0c4zjhN2pVToDf0A9MfTkuWbOzQXzZpaXHswsDm6o5eTmAjWNL1B0TdChP4o2Np/WG2gZnWcmcubyU2F8UmUVn/EQLtoPXUFS+KtoqbaVKDe2S/J3cEloxyWM9VE/jMGXNRvPAOh9zue859MVvzNVsmXFtE5DL7xq42w+j1tVXssQUqqnRPXb+9CTjEPMAm0V5Xr+aRrrXExhXKpD4S6NfyZG/vg5Czf58eZyUmKnB0XrIboJIomD59Q2I7XaiLPJhLWyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Yo+6LUjTN9jVvdTG7q36KEfFKStqsPgd9cLBqttrJsQ=; b=iXJe9XxD4jM1XJyPL1uqYtCWYWreiU5SMwVE2vmIpIQt2lOPG4W7Xx5wWS3wTTM3oCKsHSk7+k/FWZmqvZtalzbf+SivTPSvJlX60SyCd5mNnq99RZi+rEmfyH4F66L0K5+qOnX4gW4qFGeFim+qqMW+KVddy4XTIuISDjhh1l0JqE+smBzASD3w45E4M1OtfrpRZrVR0epYK+gbwlVXHjdd3mCW5mm9/cnOich6pyBqu+MBMuHCYllJ3KDNAaHlSIx1knQuNNdee38tuyd9/NgdY38Q+WLnTuoPvTrUbS4a08qcnB6eFjPE7gZj0CZMtYZ6j9Q8etfGZ+at6wiV/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=monjalon.net smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yo+6LUjTN9jVvdTG7q36KEfFKStqsPgd9cLBqttrJsQ=; b=S7m/FW241s9LM+GqVm83DJYev0q0Wom809n75GitxpVyknBeqdA4koCbUZsDDyxH/qPk6m6C8kRUxN25IMJJ0ZDtZpUKixPNB46HpFCOylZDpxsmDbP9Di2FiXYG/jmc7O1idkx+djLaMlzedMVHLG7nZtSYd0O2qyzBcEcmKA3YAFUxI4BqsUOwK/afz61qT/OZXToG69Wnn9eU5TAC3FKZr/Mwla0lzvAx13UJCSYvHfshedtvqWW7BmRGUBF5pT0Kg+50AWzp2l8GVumcWHaVQ1YoaIG9VBhj4CfbXqK1fV+p/QfRnuVdozTSsTiY5CR/61vtzNAaTkKlWkEnZw== Received: from MW4PR03CA0104.namprd03.prod.outlook.com (2603:10b6:303:b7::19) by SN6PR12MB2621.namprd12.prod.outlook.com (2603:10b6:805:73::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.16; Thu, 16 Jun 2022 02:31:18 +0000 Received: from CO1NAM11FT068.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b7:cafe::c3) by MW4PR03CA0104.outlook.office365.com (2603:10b6:303:b7::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.14 via Frontend Transport; Thu, 16 Jun 2022 02:31:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT068.mail.protection.outlook.com (10.13.175.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5332.12 via Frontend Transport; Thu, 16 Jun 2022 02:31:18 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Thu, 16 Jun 2022 02:30:43 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Wed, 15 Jun 2022 19:30:40 -0700 From: Li Zhang To: , , , CC: , , , Subject: [PATCH v2 00/15] mlx5/vdpa: optimize live migration time Date: Thu, 16 Jun 2022 05:29:57 +0300 Message-ID: <20220616023012.16013-1-lizh@nvidia.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220408075606.33056-1-lizh@nvidia.com> References: <20220408075606.33056-1-lizh@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f1c02a0a-7aa4-40e0-9216-08da4f404b2c X-MS-TrafficTypeDiagnostic: SN6PR12MB2621:EE_ X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8BRS9Qrlb+DEhBigWSiL2GBURENs56x9+CfimDg8kY/ZGE/AvVOSL9KiTQwGH/KQ4R7c/ajsqb3NwWbBnjxP75XOD4hMPF04tMZcs3iMWyv7gtH6PZzyPAlhnB6VLsEMkHhj73n+e6sz3voMnqtW8a/WlOUMmpveM4t128YENy0jpuh3EGs60Ma9HzUbOvsbYnILIyGxN8ZsfzCCORXIDRYArLvOwKlvdihC+pcLt4hepeC2zfCuU+4gz50//XDbTY8Q/gb7RTJQ+wDpsuAzGJXV1z0Ur10dFEcq29kTupkBk9PPSwC6bHxR3u54Xu5pybct10jf8RbmlzsQ+w1CAJPxJUaXr8X3Vv4aKRSDmN7B9zSzUl0lQ1/iBV0KFfLIgPEWYIox5mQmI0awtzcIQCfiSuSil14CSn+jjFTgo//wPByARjEMpASO9C08esbW3SXZpp9piNwH5jSh3gHoq2I4+bqp66C7Y/42tqOkOTbm88W+3CTVslhXdX2O1SHI68j0Zh2YLyuA372rH8LlFItfub+ln6XgVd5zLq48rH28ghwltAq91p5eRsozN6V0ZY5ePhZsz4EwvE/HOWx3NWJzOrZOC4LL9EEqx+4NfS2ugIJhvN0q7UL5Hc/z+MamOqWEWIZCgO9eSNy9CW4pga/SU74Mns3doBSmf7XqPRdldM5Tm143IIODNdKZdYGWKmKeAoY4NxKoRE7NMQCdYJlS2UwjZwyhgUkI0yEMagq9wXT2Dx1G2LHzbnB8Mzl/wHO5R3CiVB+7WqUf4z9p+7dj9TXJA8jvXXaLHhjWBxfh/f5M2UMBozv/3D0BOSsncRJ2rtTxquzKCA7YAiEwCQ== X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230016)(4636009)(40470700004)(36840700001)(46966006)(82310400005)(36860700001)(36756003)(6636002)(55016003)(54906003)(2616005)(186003)(316002)(83380400001)(426003)(336012)(16526019)(47076005)(1076003)(107886003)(6666004)(966005)(8936002)(5660300002)(70206006)(8676002)(4326008)(70586007)(2906002)(508600001)(7696005)(110136005)(356005)(40460700003)(26005)(6286002)(86362001)(81166007)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2022 02:31:18.2966 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f1c02a0a-7aa4-40e0-9216-08da4f404b2c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT068.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR12MB2621 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Allow the driver to use internal threads to obtain fast configuration. All the threads will be open on the same core of the event completion queue scheduling thread. Add max_conf_threads parameter to configure the maximum number of internal threads in addition to the caller thread (8 is suggested). These internal threads to pipeline handle VDPA tasks in system and shared with all VDPA devices. Default is 0, don't use internal threads for configuration. Depends-on: series=21868 ("vdpa/mlx5: improve device shutdown time") http://patchwork.dpdk.org/project/dpdk/list/?series=21868 RFC ("Add vDPA multi-threads optiomization") https://patchwork.dpdk.org/project/dpdk/cover/20220408075606.33056-1-lizh@nvidia.com/ V2: * Drop eal device removal patch in series. * Add release note in release_22_07.rst. Li Zhang (12): vdpa/mlx5: fix usage of capability for max number of virtqs common/mlx5: extend virtq modifiable fields vdpa/mlx5: pre-create virtq in the prob vdpa/mlx5: optimize datapath-control synchronization vdpa/mlx5: add multi-thread management for configuration vdpa/mlx5: add task ring for MT management vdpa/mlx5: add MT task for VM memory registration vdpa/mlx5: add virtq creation task for MT management vdpa/mlx5: add virtq LM log task vdpa/mlx5: add device close task vdpa/mlx5: add virtq sub-resources creation vdpa/mlx5: prepare virtqueue resource creation Yajun Wu (3): vdpa/mlx5: support pre create virtq resource common/mlx5: add DevX API to move QP to reset state vdpa/mlx5: support event qp reuse doc/guides/rel_notes/release_22_07.rst | 5 + doc/guides/vdpadevs/mlx5.rst | 25 + drivers/common/mlx5/mlx5_devx_cmds.c | 77 ++- drivers/common/mlx5/mlx5_devx_cmds.h | 6 +- drivers/common/mlx5/mlx5_prm.h | 30 +- drivers/vdpa/mlx5/meson.build | 1 + drivers/vdpa/mlx5/mlx5_vdpa.c | 270 ++++++++-- drivers/vdpa/mlx5/mlx5_vdpa.h | 152 +++++- drivers/vdpa/mlx5/mlx5_vdpa_cthread.c | 360 ++++++++++++++ drivers/vdpa/mlx5/mlx5_vdpa_event.c | 160 ++++-- drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 128 ++++- drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 270 ++++++---- drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 22 +- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 654 ++++++++++++++++++------- 14 files changed, 1776 insertions(+), 384 deletions(-) create mode 100644 drivers/vdpa/mlx5/mlx5_vdpa_cthread.c -- 2.30.2