From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9ECE4A034E; Thu, 24 Feb 2022 14:29:36 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6EC3D411FE; Thu, 24 Feb 2022 14:29:36 +0100 (CET) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2081.outbound.protection.outlook.com [40.107.243.81]) by mails.dpdk.org (Postfix) with ESMTP id D8BF141156 for ; Thu, 24 Feb 2022 14:29:34 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=L/9kjZMF6i6snYBBWsaXXv9Ut2dpDotJF4gEF/T09JQbvJ8nE84kE5Y7I9Xwt+7wCYGUjXHZyAtPD2ZSQmNIl3LuISzaiIXr67ZE9pWffIyz98d/XYvZ6oP9Gt93uzWWmPFqlvyS2VNb1RZA+Wisc/zBrzx9Vy98qGotKxmVSOjCKaB/TjR5Tu6Ep80uzXqhPnVY4mlfjLDbFUONnnKScRpWM9UQp61mHp8Cc4D39Zg7KlBisGPbMxgCdV4YAlbGy5f5/khyeAa1T9+hg4g/JzwiqLYPe1pdbE34dDPUEQNGOXG7P6KSrTS5NLNSDxUORR1s+sUfLLNqnBTDre8EOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8aQ7eNQuHaDd+4s8/8IcNCX9EiUXryhr6Quzw0kcNlA=; b=GwH88Y/ToPwtyo63gkupO6awuQKQ0dw04bE71LU1KgupPlteVrx7XcfDDJRT0VDuyq5iXGRVQv8FRe2leUhmHo9hcz4zUI0hhnaIHU//3XbM+wfrlcqfYfkIMtPp7ZtEVd61ykEO3BCimxr4SxCheZxodqs1B4bjotxit1hFmOWSv3YquW+67S43miQjp8GabxpsSe3viGWrHDqvvQMo1sIDVxu6b5XXLhlBi591j5xSYqtE2Ocj+ox2jWccFZMoPSELKIOmuevAQ4Iho+qCL2TAdEuFcVsEWaxvUxR4Hwsq/EiuS8CfKH9WqasOfragxWiw2OikIe1yjkgmVoscew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8aQ7eNQuHaDd+4s8/8IcNCX9EiUXryhr6Quzw0kcNlA=; b=eZ9MazIEbzYsNOlI6jTs3VsdwmAnN8yGFQqcxIOMbtYoGudkKZ/1LukWsGaPOf8lhwoC+BacMfenYWLix1WrOgu/ZoUw4qKbWpsCT1qI4kbJl5WXAGCRQ3KQ3Q0Jj7xJZKJH+I4hAoYFt0K0C9/Qr+muA0h4dUFLrVh98v5KdPAMY0jAxzsFhSWPuQIr2puN4fsHypRidalezFt+5mjurnEUThBtzKz7s3wXp2KM9PfqP/jMm8Be5xKog1ZXgsXNFqr3uwnims4zeyWRi4A5DIAIq1FwqaDhnp4EPeKrKkIb1SeXQfeBlEPaHIfeLtS+xVQwu5Io9yPHbruwa/Ni/Q== Received: from BN8PR03CA0009.namprd03.prod.outlook.com (2603:10b6:408:94::22) by MW2PR12MB2444.namprd12.prod.outlook.com (2603:10b6:907:11::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.22; Thu, 24 Feb 2022 13:29:32 +0000 Received: from BN8NAM11FT057.eop-nam11.prod.protection.outlook.com (2603:10b6:408:94:cafe::51) by BN8PR03CA0009.outlook.office365.com (2603:10b6:408:94::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.23 via Frontend Transport; Thu, 24 Feb 2022 13:29:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.234) by BN8NAM11FT057.mail.protection.outlook.com (10.13.177.49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5017.22 via Frontend Transport; Thu, 24 Feb 2022 13:29:32 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Thu, 24 Feb 2022 13:29:31 +0000 Received: from nvidia.com (10.126.231.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Thu, 24 Feb 2022 05:29:29 -0800 From: Xueming Li To: CC: , Matan Azrad , "Viacheslav Ovsiienko" Subject: [PATCH 6/7] vdpa/mlx5: support device cleanup callback Date: Thu, 24 Feb 2022 21:28:19 +0800 Message-ID: <20220224132820.1939650-7-xuemingl@nvidia.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220224132820.1939650-1-xuemingl@nvidia.com> References: <20220224132820.1939650-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.126.231.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2865dcb1-0bee-4891-54fb-08d9f799b130 X-MS-TrafficTypeDiagnostic: MW2PR12MB2444:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: v3dc1psHAUBlozJCL/ME01FIsnpcbsgyORtowoFBL9TIGc47OZcizDUAaY+P861lwDCmUHI1Zw/3FdyyxxhOSJYIDXUROj4XhrQOD27WndRO1fLORJOQEC3/9mTeEi478h67wDIL3dmhDmwpBV1O6GCriKLuXLHv0mr+QwxK1wZbbmfsXRhNZ1c/wfVGWQGnAhjsxNObTOKwilShuq491vBxXHDfV1lWX1A0B0BnYldI73kYoRPAunn8zev54WYLq0FIeBq2c3rp+WQ5JmerGPruw4lI9QQxmKnz9r1l5iFT8kE5DbvBmwVoimdfjydR+KZuOtwbvGfVP1fU8xtTWkxuQJcGQxZOAj/hHQvGRODglJ2Q7RI0NbjgWvUojXtU2oAIlPHvwmwVncIR7jG6Rw2UkjbzwvBuJ6wFQSSTmpMU4GwO3cmwkpqvQgqVbG3B6esTbkwbDjG5Imj737+pdx8k+zHp8+yBgHhiB9MgN/O9pUoX26dh21zBAuImR0FlFtcGiwNW0nDmCuYtAlb5DZCpnyl+94uDGIYWFFXpoOLS882idGz6CYWkY9e3txKqb1piIxZh3Q0Qz+vrVvKaqsSvxpCPz85i2ssD9WgrWJF3qZfenujsoqBJ/B117ULDLQV1h16YJVKwN2uFYceZwhhO4thmnzscCU6tLzEkHpybvQgh9NCO0tfJFgjxLzwFwZbMQFRoN7rjAdlCe+rH/Q== X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(40470700004)(46966006)(36860700001)(356005)(47076005)(2906002)(81166007)(82310400004)(70206006)(8936002)(4326008)(8676002)(40460700003)(55016003)(70586007)(6666004)(336012)(508600001)(107886003)(6916009)(7696005)(186003)(1076003)(86362001)(54906003)(2616005)(6286002)(83380400001)(5660300002)(426003)(26005)(36756003)(316002)(16526019)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Feb 2022 13:29:32.2552 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2865dcb1-0bee-4891-54fb-08d9f799b130 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT057.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR12MB2444 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch supports device cleanup callback API which called when device disconected with VM. Cached resources like VM MR and VQ memory are released. Signed-off-by: Xueming Li --- drivers/vdpa/mlx5/mlx5_vdpa.c | 23 +++++++++++++++++++++++ drivers/vdpa/mlx5/mlx5_vdpa.h | 1 + 2 files changed, 24 insertions(+) diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.c b/drivers/vdpa/mlx5/mlx5_vdpa.c index dbaa590d5d1..c83b1141482 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.c +++ b/drivers/vdpa/mlx5/mlx5_vdpa.c @@ -270,6 +270,8 @@ mlx5_vdpa_dev_close(int vid) if (priv->lm_mr.addr) mlx5_os_wrapped_mkey_destroy(&priv->lm_mr); priv->state = MLX5_VDPA_STATE_PROBED; + if (!priv->connected) + mlx5_vdpa_dev_cache_clean(priv); priv->vid = 0; /* The mutex may stay locked after event thread cancel - initiate it. */ pthread_mutex_init(&priv->vq_config_lock, NULL); @@ -294,6 +296,7 @@ mlx5_vdpa_dev_config(int vid) return -1; } priv->vid = vid; + priv->connected = true; if (mlx5_vdpa_mtu_set(priv)) DRV_LOG(WARNING, "MTU cannot be set on device %s.", vdev->device->name); @@ -431,12 +434,32 @@ mlx5_vdpa_reset_stats(struct rte_vdpa_device *vdev, int qid) return mlx5_vdpa_virtq_stats_reset(priv, qid); } +static int +mlx5_vdpa_dev_clean(int vid) +{ + struct rte_vdpa_device *vdev = rte_vhost_get_vdpa_device(vid); + struct mlx5_vdpa_priv *priv; + + if (vdev == NULL) + return -1; + priv = mlx5_vdpa_find_priv_resource_by_vdev(vdev); + if (priv == NULL) { + DRV_LOG(ERR, "Invalid vDPA device: %s.", vdev->device->name); + return -1; + } + if (priv->state == MLX5_VDPA_STATE_PROBED) + mlx5_vdpa_dev_cache_clean(priv); + priv->connected = false; + return 0; +} + static struct rte_vdpa_dev_ops mlx5_vdpa_ops = { .get_queue_num = mlx5_vdpa_get_queue_num, .get_features = mlx5_vdpa_get_vdpa_features, .get_protocol_features = mlx5_vdpa_get_protocol_features, .dev_conf = mlx5_vdpa_dev_config, .dev_close = mlx5_vdpa_dev_close, + .dev_cleanup = mlx5_vdpa_dev_clean, .set_vring_state = mlx5_vdpa_set_vring_state, .set_features = mlx5_vdpa_features_set, .migration_done = NULL, diff --git a/drivers/vdpa/mlx5/mlx5_vdpa.h b/drivers/vdpa/mlx5/mlx5_vdpa.h index 540bf87a352..24bafe85b44 100644 --- a/drivers/vdpa/mlx5/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/mlx5_vdpa.h @@ -121,6 +121,7 @@ enum mlx5_dev_state { struct mlx5_vdpa_priv { TAILQ_ENTRY(mlx5_vdpa_priv) next; + bool connected; enum mlx5_dev_state state; pthread_mutex_t vq_config_lock; uint64_t no_traffic_counter; -- 2.35.1