From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F15F7A034E; Thu, 24 Feb 2022 15:39:02 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CD46942725; Thu, 24 Feb 2022 15:38:54 +0100 (CET) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2071.outbound.protection.outlook.com [40.107.92.71]) by mails.dpdk.org (Postfix) with ESMTP id DDBE24271E for ; Thu, 24 Feb 2022 15:38:53 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fjhuH6jzJVnTR2airCamv1m/+L/y5BHQRGBfmrdEdzBTUkS1p2nUDOKcbO1dlcR8sa3uuJcNly+tan65VIJHA5kdPi1kHaZSyppI1dhRoYxuPt/WK+BiU2/Ir/RRb8nLi3X4q4HCli0pOZEE9oK7BCYEN553CtccX6TwW2toMS1Xv2OC/ywLjlDOtDhtnz/7sTu2D4LtxXAbBgxilYxfIfDwGECNYv1rEIWCqgzxtuzlDuHSPsPNylNmOPtyG2EzZiJ03GBVPJAO+ZTA16d+WZwCMUAnGc9Ki6147JUN8Uw96r6DbWUQTHTS/cKjiw/hgv0Ls07Ko+QAdXc6FZG87w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Iyql/w7x/AehFOc05lYomCru352xQrt3ksbCdqDB7Io=; b=HGamZ4EQhQKRvOJxU+vyP3wjlON6OqkGNOo6Prh+v3jtGcySCDmnZRP1Cmxs9bSNTt8Va58gYTWOe/gbUlPE/2/ebTJaDcss+YPkxbYEjTEAoTE8DBbpcrPXMOoRqbbLHtgvc0I5pd0waavl1vbQGPZ+yc1S/LAgepK6v/81fr4UnMWle24dldnnBt3jq9/7JUiuAPY+v9XwE92C2VHalP/PAY4iTYku4dwv+sN1pd/JukMzMxet89SvnKztyPYDn8zJCRCN3nMEen8NERPfkL9s/Ze/g1AJA6S6w6mSPDqAaz1lAfk8sZoE6UYfLJs78+UnwkUP/4jy4ATyVeBHdg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Iyql/w7x/AehFOc05lYomCru352xQrt3ksbCdqDB7Io=; b=XS5o9ez1l3q6hV3GCPkAjGjxoU4J74aVPDhHSSdgazlhhUOPvjCJPrGvHqagLxu1wdiJz0eAWx+b8bDBU98v/AOe1qwudbZXm1KOIPUAFole90AQ2rT3YAGu/pArkQAGE36DzmT7x8VnC32fft1Nx8Bc/+cmlFcEyxEeinDn3LDZv6vnfoVwMckwnRN07VW+PjFUdUnKNfH1U+etM7zoPmRVdo5pIPZkKByb7jg7Up1Fh2t3x4zHTHfprBrOEAHPTKRKQR7t8577hm14T0QeBKKPVzxyQ567tckf7bKzEdp3Gt+YJzRcHQhFl9RfIT8fEbKCeJ8WKNi45EEeiNp7Ow== Received: from MW2PR16CA0057.namprd16.prod.outlook.com (2603:10b6:907:1::34) by BY5PR12MB3729.namprd12.prod.outlook.com (2603:10b6:a03:1ad::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4995.17; Thu, 24 Feb 2022 14:38:52 +0000 Received: from CO1NAM11FT049.eop-nam11.prod.protection.outlook.com (2603:10b6:907:1:cafe::47) by MW2PR16CA0057.outlook.office365.com (2603:10b6:907:1::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.25 via Frontend Transport; Thu, 24 Feb 2022 14:38:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT049.mail.protection.outlook.com (10.13.175.50) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5017.22 via Frontend Transport; Thu, 24 Feb 2022 14:38:51 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 24 Feb 2022 14:38:51 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Thu, 24 Feb 2022 06:38:48 -0800 From: Xueming Li To: CC: , Matan Azrad , "Viacheslav Ovsiienko" Subject: [PATCH v1 6/7] vdpa/mlx5: support device cleanup callback Date: Thu, 24 Feb 2022 22:38:08 +0800 Message-ID: <20220224143809.1977642-7-xuemingl@nvidia.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220224143809.1977642-1-xuemingl@nvidia.com> References: <20220224132820.1939650-1-xuemingl@nvidia.com> <20220224143809.1977642-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail201.nvidia.com (10.129.68.8) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a068aaa2-5563-4894-4eac-08d9f7a36083 X-MS-TrafficTypeDiagnostic: BY5PR12MB3729:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MSHXPMCs1zDwVt0EeaPNBYzxc4/mcZYQ5uhlUn3aIg87UiqiGClxUalMtZLExpH83Wg4YwXcBsvfFIiEgYMI1bdONFEdwYIE0vPjviQqrhDAf80aUqly3QtjENe6dz3F1Jc0gDQvD9n3wP3BFs4FOqi8PdfRLj+b2d1KRKfUQbqFyGv3uZTz3xfI/OyHZBufzSAt1d+ivzdEuk7+zs82vPa0s/I8LT/8knSBxg3u1LoGmQQ7rWQwZiKx/tQyl6ldkeIvsM2UmkfBW3KC4z9iWG6Hm6Z66ZzD7Fw3HAXSo7CF1Zq4Zby8YVuRAV4tPsJL3/AkAvJ1pmpiGi1NpscSGy5vfw/6R7hidASGUxIc//vIQl9bhTRlZDOF9hQQ+OgjRS5tReUmBk7qKX89pGAtliE17JfvsEhMdd9xSqDR2H36DRl/5bqs1RfkX4OsGzZ9X9jw5z9NWwDnJ3J4PlSuVSsNOCOJPLbvJybHc38CHGLQoExUHM/STMMSg42EkVQRDJKYfCTxLpTGgOcH/Z4LRiG5ib4p3wAr+Ey6GnvqpLJvGvivUhmasxJRE6RvED8A6mb21E+fqewkQQA9aR2PUkOI07CMmCANcwhMZP02Y7XxtjMg6Rhx/YeW4CGvI/SsOoK9pER4GuoQ/S2WeE13WwJ4cRE+3F2n72mgUqpFGsoHmdTI1v1b+6sGzfOhYmpDwmWSd7Jkb9VDl7Yy/jnxNA== X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(40470700004)(36840700001)(46966006)(36756003)(508600001)(107886003)(336012)(16526019)(7696005)(70206006)(5660300002)(4326008)(1076003)(426003)(83380400001)(6666004)(70586007)(47076005)(6286002)(2616005)(8676002)(8936002)(316002)(26005)(186003)(54906003)(36860700001)(86362001)(2906002)(356005)(81166007)(55016003)(6916009)(82310400004)(40460700003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 14:38:51.9333 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a068aaa2-5563-4894-4eac-08d9f7a36083 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT049.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB3729 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch supports device cleanup callback API which called when device disconnected with VM. Cached resources like VM MR and VQ memory are released. Signed-off-by: Xueming Li --- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 + 2 files changed, 24 insertions(+) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index f794cb9bd61..a64445cd8b5 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -270,6 +270,8 @@ mlx5_vdpa_dev_close(int vid) if (priv->lm_mr.addr) mlx5_os_wrapped_mkey_destroy(&priv->lm_mr); priv->state = MLX5_VDPA_STATE_PROBED; + if (!priv->connected) + mlx5_vdpa_dev_cache_clean(priv); priv->vid = 0; /* The mutex may stay locked after event thread cancel - initiate it. */ pthread_mutex_init(&priv->vq_config_lock, NULL); @@ -294,6 +296,7 @@ mlx5_vdpa_dev_config(int vid) return -1; } priv->vid = vid; + priv->connected = true; if (mlx5_vdpa_mtu_set(priv)) DRV_LOG(WARNING, "MTU cannot be set on device %s.", vdev->device->name); @@ -431,12 +434,32 @@ mlx5_vdpa_reset_stats(struct rte_vdpa_device *vdev, int qid) return mlx5_vdpa_virtq_stats_reset(priv, qid); } +static int +mlx5_vdpa_dev_clean(int vid) +{ + struct rte_vdpa_device *vdev = rte_vhost_get_vdpa_device(vid); + struct mlx5_vdpa_priv *priv; + + if (vdev == NULL) + return -1; + priv = mlx5_vdpa_find_priv_resource_by_vdev(vdev); + if (priv == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name); + return -1; + } + if (priv->state == MLX5_VDPA_STATE_PROBED) + mlx5_vdpa_dev_cache_clean(priv); + priv->connected = false; + return 0; +} + static struct rte_vdpa_dev_ops mlx5_vdpa_ops = { .get_queue_num = mlx5_vdpa_get_queue_num, .get_features = mlx5_vdpa_get_vdpa_features, .get_protocol_features = mlx5_vdpa_get_protocol_features, .dev_conf = mlx5_vdpa_dev_config, .dev_close = mlx5_vdpa_dev_close, + .dev_cleanup = mlx5_vdpa_dev_clean, .set_vring_state = mlx5_vdpa_set_vring_state, .set_features = mlx5_vdpa_features_set, .migration_done = NULL, diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 540bf87a352..24bafe85b44 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -121,6 +121,7 @@ enum mlx5_dev_state { struct mlx5_vdpa_priv { TAILQ_ENTRY(mlx5_vdpa_priv) next; + bool connected; enum mlx5_dev_state state; pthread_mutex_t vq_config_lock; uint64_t no_traffic_counter; -- 2.35.1