From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 59C724320F; Fri, 27 Oct 2023 05:04:43 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8FEAC42E95; Fri, 27 Oct 2023 05:01:34 +0200 (CEST) Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04on2093.outbound.protection.outlook.com [40.107.102.93]) by mails.dpdk.org (Postfix) with ESMTP id 0C8E342E5A for ; Fri, 27 Oct 2023 05:01:14 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lD98fyfz9BOzJps8x3zODg/58Z01/vwIqkaZzBfy1eNkqpECrBJZu5naLuIXTKCRMC/RxLo9wzgBpu96OH+Upkwq2Ytq4wPSe+NNH9wMrXn7Q4FNg5hQvgq1JsqkPlgBCWNciIO/QObhanhWwPB3+L24Fip7DC8aNfaTx/+SOStTTGEu0HnBGc3WZoIeq7l/keynbl/TrcKrA8kpVT0rjeeid7D3r6uMwuGxZBZw4shfvMfnENEiog3MHedVjPFluZnkAaDUW2UzHccFOoBTYgAZ49mJN3isoGAN8tAQypcqLJDIu7l0nrWphA23oOZpUfdkY2uU5TSzJ9fsrqZQGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zs3+5JTfr5a7DpPcAx9Hen1IISiYu2wPu83zcbKT8Pg=; b=Ln2Q7LZprlKKQBN3vWFK2vY8PkaUwkbgVi3K/UT/f8Mz0cxn7t86BTUsaEvwo58pm3O7FOsAVf/hJX7zAssyCp/yDNkAvZBnNTCMTPP47EzzY8I9zf9EoGE7w6l0hPrw0AUhYH3EP8ccgFeYOK3j9AMUcakpHxHI3M4Q5tOU90CxpEOV/ZR06oZvY+Hem/KjuC8YGKbc/f/GSPSpcW5rIoYwtV5gUyBaWdqKssPg+YMt25EbQRHvKIEm0jleMgbvHObRuCrtE6h9wQaPkDHkP3tlo/mzj7dLdmlqvxyYMVqAejPU5IRAJ3rLVSzGJs1+Nfsmq9qjWnnJxBPoQSuLkg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=corigine.com; dmarc=pass action=none header.from=corigine.com; dkim=pass header.d=corigine.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=corigine.onmicrosoft.com; s=selector2-corigine-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zs3+5JTfr5a7DpPcAx9Hen1IISiYu2wPu83zcbKT8Pg=; b=uush6V9R4yPX0SIRLhmuY8hTPDvv2jLMgDCH2258ftKkP43MZbiWP0oMUMBpINZF3nzG1fRjgDV9ndupHaSYeN7A7KG9wjjJnnisk0vZC5t7XzOzRI38WLTnHeXJpaCCIvE3MvrZ1vbwXpMTHUs86a5h53RQ+IuuYhbKE8WZ9jU= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=corigine.com; Received: from SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) by SA0PR13MB4048.namprd13.prod.outlook.com (2603:10b6:806:97::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.24; Fri, 27 Oct 2023 03:01:12 +0000 Received: from SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b]) by SJ0PR13MB5545.namprd13.prod.outlook.com ([fe80::45b3:d83c:cff2:4a1b%6]) with mapi id 15.20.6933.022; Fri, 27 Oct 2023 03:01:12 +0000 From: Chaoyong He To: dev@dpdk.org Cc: oss-drivers@corigine.com, Chaoyong He , Shujing Dong , Long Wu , Peng Zhang Subject: [PATCH v4 24/24] vdpa/nfp: add device operations Date: Fri, 27 Oct 2023 11:00:01 +0800 Message-Id: <20231027030001.602639-25-chaoyong.he@corigine.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20231027030001.602639-1-chaoyong.he@corigine.com> References: <20231026064324.177531-1-chaoyong.he@corigine.com> <20231027030001.602639-1-chaoyong.he@corigine.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BYAPR08CA0065.namprd08.prod.outlook.com (2603:10b6:a03:117::42) To SJ0PR13MB5545.namprd13.prod.outlook.com (2603:10b6:a03:424::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR13MB5545:EE_|SA0PR13MB4048:EE_ X-MS-Office365-Filtering-Correlation-Id: aef199c3-e08a-4e58-2600-08dbd698fa0c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: McrXvi3E/zOQukbjw4xPWVx2uVbDloFB2HjSN2PjvINWWb5tez6zRWPl90o+c7bbwB64Y3kug3F3dMq20170Qmphqxa/oRaaCuFaFttIK/l6G0bp/79Hs2uQ8VoS+d+TFxTk6+QuvA9XkkE1pp1zqEwuv4g5WsGlY1zszOV2FzFvL4T6Qa/KuQ5+pzBkXsooNcportI5RS3hqGRYyMyDBkvvk9rIQvt+LOjPlIAuY6aijy9803WmdI5Nv27G4yrdbBinasC6t09qa+ylrt9E6ptHf96pp3WNai+LMc7dJ0TbGeGwZcS83LAW/SDv9famDwmBs31MDWK99CZsnPNvoLnDabhqhHyYnttYb5old0SBt6iUzIwzpZGrMs8HikdswxoBfYzTF24dV9/0Xsk6M/I5i1F9rNWTIrCV7MjNhtMcoFA0nbFK0jex+B39U65LWwBI7pxznN/fggo/i+74LSU3dmqG6fzY7VWvCnU2zHUGMQ3+SP6aFZegw6z5R3PKvzj1PsHtOZxhNes5u2lgFdBZRzqwVA9tk042Hgvuiouh9PMAMg/uziuZrQM8b0XTDqJX6O4yjQ98C+W/UC0CDUmhPe8EpST1xQ9DPNRnOJezFr5ZC0gb1bPKLlbz9zKnFzf1Dpu49u6LrceSFGHMfjnhtbQ+QVpnMdpkEfUz1ak= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR13MB5545.namprd13.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(376002)(396003)(39840400004)(366004)(346002)(136003)(230922051799003)(64100799003)(186009)(1800799009)(451199024)(6512007)(6666004)(52116002)(6506007)(6486002)(86362001)(36756003)(478600001)(38100700002)(2906002)(6916009)(41300700001)(83380400001)(1076003)(107886003)(26005)(2616005)(38350700005)(66946007)(316002)(66556008)(66476007)(4326008)(8936002)(8676002)(44832011)(5660300002)(54906003); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?lyi6Sw0xcMe6SCiVC0dOZ/myM3B2KLd4ZhqmUVh4finAc9c1+Pn7zZMToxS5?= =?us-ascii?Q?zQg++YZVkk/e19Am/NQC2jRxqiDYWLsQOGiqZs+mKM0V9AalaLU2+pjzNF6h?= =?us-ascii?Q?LQHrI5wMHxz+LBCBlT/yVAJYRMtwefNpZL6/zMdLrY4mJ2CPBsW5Jf7Mefs8?= =?us-ascii?Q?108SjIi7i/NulQ2KPkE59Uv0t63eBjKGG3IcOreJzc6HVVzl/zb0IzEB8RJ+?= =?us-ascii?Q?rbUBlC+JzxxbqkQso4gPHGxdlp1OkhjoswyB0Kx9xo3DWhpXJMTYIx0GAJ7t?= =?us-ascii?Q?y73YURLziZkITudPj7Gxkx/q+lznCYh8b0htuvbWbdEKZgJOjR1yWAyIwvKU?= =?us-ascii?Q?B8jGqB1Rx4b2FevftCj68DyzlSYml9kRbUs/DRVf797rl8cf+KqHy4llVaHd?= =?us-ascii?Q?MLdPGnC9YlQiTbLZAxpbu/NizHRRhMCD5vVI/Ou7QAUNL5VDJSbTmWsu5jCV?= =?us-ascii?Q?hofzseE/iv14G3FeNWr0VXKU3YRvQJbUy5k+7EMaM1jSkLFpc3mNts0mOzp+?= =?us-ascii?Q?0rBzLCkDVTAI36/jDg+VtTpb9cM4EMFB9Jug7QfngcjfHfiCkQaSV3XNd+RA?= =?us-ascii?Q?x+e4snoo0D8ujStnDpjMESH0D008Il75nyZfVSXMDFtJEMBmZ9DEKkaZP2hP?= =?us-ascii?Q?2wM0UoG561OcpeQkgkXWuBZVL6xKyNCJm4NLnuxMEBsGeyL9DI9xeQCAxiwt?= =?us-ascii?Q?yIBV3PuhJ2orVqfcLCeHmjHhYDRGoEwCz3mZItMNHOe23cVJoSGr+9Zxn9rH?= =?us-ascii?Q?/Q6S9zwh97RK6DewPTlsZTBd5VViZFP08xDqGA6gXzPVNAUi3nKCGFFV1K4U?= =?us-ascii?Q?7G0v3NDj3hHMLpgWGy1DK85jgD5P2s4R/6135joI2qaJJYzStfP2t67YOAWe?= =?us-ascii?Q?QJktjfgumFrHs4m3pTAXy8FdXWdVx+xSmrBePmRFEdRL68vOuPF7aMZoyYPa?= =?us-ascii?Q?XKPzDq26KzvsxZxfQRNe/9hQOCKrOEYuC6eRDRVUaRW+13O0N8zr+nHHCfeX?= =?us-ascii?Q?aALXXLSuuVtDJQc8TG0q54Yt/z57XqN2wEh1SXdeA0kSqRQmJgUuulUnG5Np?= =?us-ascii?Q?MWfGhwiKYsYllbpf5OeIzs1hxhNR+Au2T1YTK1JU+PWpPqXpPHZ3pFRKEeIg?= =?us-ascii?Q?QHghRpKIiU+0L7/360x1qXBnTVAj+4jTWS/EwiAUhp6kTVhGu3zBNVNuq/Bz?= =?us-ascii?Q?2lc74OUr0qqEVxFUjEZ0nbpckTc5z261PT+GHIAHj1KZ/sfPEIZVqT8eXmrX?= =?us-ascii?Q?7V8a64dUDX8o2j1HkLFNxQQBoRuGLwWScKmL8+Ou0uvci7JpF+vnpsSq3m0Q?= =?us-ascii?Q?ckngwVCfJZ9Y+3rjUYl3/XTyb/yKO8h+Nnsa3Ght5nDSmgpgd2seDwQ8ds1R?= =?us-ascii?Q?xdKH6pMI1dzrDcob/lXlSdgSSigRYPpVv2j6D7Kh1EWF/6RBT9JGXg9FpQCB?= =?us-ascii?Q?iv9+C8IX/QuplYsyvwB0VaDBl+LotKjyPdnLbMkWyyNAK4IYVHo9Y3LeQv8i?= =?us-ascii?Q?jLg6ZITAkN0GrTzUtOi78iTG0ieuNZ4GuRguYUc4xCk5B6Y2x26/Dc5rd6jg?= =?us-ascii?Q?TVi+pKKyYhXcGc4KJfT+L8LKXa1yluoVCnzJKsVN4SRGp0USnx2+YS2OaCKd?= =?us-ascii?Q?mQ=3D=3D?= X-OriginatorOrg: corigine.com X-MS-Exchange-CrossTenant-Network-Message-Id: aef199c3-e08a-4e58-2600-08dbd698fa0c X-MS-Exchange-CrossTenant-AuthSource: SJ0PR13MB5545.namprd13.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2023 03:01:12.4232 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: fe128f2c-073b-4c20-818e-7246a585940c X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: R4TgzP6UsDQSlsWp4zEoLNk5aHOdxHIsQsAudE08PzPhjxwXOB4Hj/pUTvnLkW4YSGiPiwgWrbI8E56GPRPD366mEgsgEOoMZmrpDaKxGME= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR13MB4048 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Implement the corresponding nfp vDPA operation functions. Signed-off-by: Shujing Dong Signed-off-by: Chaoyong He Reviewed-by: Long Wu Reviewed-by: Peng Zhang --- drivers/vdpa/nfp/nfp_vdpa.c | 213 ++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) diff --git a/drivers/vdpa/nfp/nfp_vdpa.c b/drivers/vdpa/nfp/nfp_vdpa.c index c104588663..a1d1dc4f73 100644 --- a/drivers/vdpa/nfp/nfp_vdpa.c +++ b/drivers/vdpa/nfp/nfp_vdpa.c @@ -57,6 +57,29 @@ static struct vdpa_dev_list_head vdpa_dev_list = static pthread_mutex_t vdpa_list_lock = PTHREAD_MUTEX_INITIALIZER; +static struct nfp_vdpa_dev_node * +nfp_vdpa_find_node_by_vdev(struct rte_vdpa_device *vdev) +{ + bool found = false; + struct nfp_vdpa_dev_node *node; + + pthread_mutex_lock(&vdpa_list_lock); + + TAILQ_FOREACH(node, &vdpa_dev_list, next) { + if (vdev == node->device->vdev) { + found = true; + break; + } + } + + pthread_mutex_unlock(&vdpa_list_lock); + + if (found) + return node; + + return NULL; +} + static struct nfp_vdpa_dev_node * nfp_vdpa_find_node_by_pdev(struct rte_pci_device *pdev) { @@ -578,7 +601,197 @@ update_datapath(struct nfp_vdpa_dev *device) return ret; } +static int +nfp_vdpa_dev_config(int vid) +{ + int ret; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + device->vid = vid; + rte_atomic_store_explicit(&device->dev_attached, 1, rte_memory_order_relaxed); + update_datapath(device); + + ret = rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, true); + if (ret != 0) + DRV_VDPA_LOG(INFO, "vDPA (%s): software relay is used.", + vdev->device->name); + + return 0; +} + +static int +nfp_vdpa_dev_close(int vid) +{ + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + rte_atomic_store_explicit(&device->dev_attached, 0, rte_memory_order_relaxed); + update_datapath(device); + + return 0; +} + +static int +nfp_vdpa_get_vfio_group_fd(int vid) +{ + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + return node->device->vfio_group_fd; +} + +static int +nfp_vdpa_get_vfio_device_fd(int vid) +{ + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + return node->device->vfio_dev_fd; +} + +static int +nfp_vdpa_get_notify_area(int vid, + int qid, + uint64_t *offset, + uint64_t *size) +{ + int ret; + struct nfp_vdpa_dev *device; + struct rte_vdpa_device *vdev; + struct nfp_vdpa_dev_node *node; + struct vfio_region_info region = { + .argsz = sizeof(region) + }; + + vdev = rte_vhost_get_vdpa_device(vid); + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + device = node->device; + region.index = device->hw.notify_region; + + ret = ioctl(device->vfio_dev_fd, VFIO_DEVICE_GET_REGION_INFO, ®ion); + if (ret != 0) { + DRV_VDPA_LOG(ERR, "Get not get device region info."); + return -EIO; + } + + *offset = nfp_vdpa_get_queue_notify_offset(&device->hw, qid) + region.offset; + *size = NFP_VDPA_NOTIFY_ADDR_INTERVAL; + + return 0; +} + +static int +nfp_vdpa_get_queue_num(struct rte_vdpa_device *vdev, + uint32_t *queue_num) +{ + struct nfp_vdpa_dev_node *node; + + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + *queue_num = node->device->max_queues; + + return 0; +} + +static int +nfp_vdpa_get_vdpa_features(struct rte_vdpa_device *vdev, + uint64_t *features) +{ + struct nfp_vdpa_dev_node *node; + + node = nfp_vdpa_find_node_by_vdev(vdev); + if (node == NULL) { + DRV_VDPA_LOG(ERR, "Invalid vDPA device: %p", vdev); + return -ENODEV; + } + + *features = node->device->hw.features; + + return 0; +} + +static int +nfp_vdpa_get_protocol_features(struct rte_vdpa_device *vdev __rte_unused, + uint64_t *features) +{ + *features = 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | + 1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK | + 1ULL << VHOST_USER_PROTOCOL_F_BACKEND_REQ | + 1ULL << VHOST_USER_PROTOCOL_F_BACKEND_SEND_FD | + 1ULL << VHOST_USER_PROTOCOL_F_HOST_NOTIFIER; + + return 0; +} + +static int +nfp_vdpa_set_features(int32_t vid) +{ + DRV_VDPA_LOG(DEBUG, "Start vid=%d", vid); + return 0; +} + +static int +nfp_vdpa_set_vring_state(int vid, + int vring, + int state) +{ + DRV_VDPA_LOG(DEBUG, "Start vid=%d, vring=%d, state=%d", vid, vring, state); + return 0; +} + struct rte_vdpa_dev_ops nfp_vdpa_ops = { + .get_queue_num = nfp_vdpa_get_queue_num, + .get_features = nfp_vdpa_get_vdpa_features, + .get_protocol_features = nfp_vdpa_get_protocol_features, + .dev_conf = nfp_vdpa_dev_config, + .dev_close = nfp_vdpa_dev_close, + .set_vring_state = nfp_vdpa_set_vring_state, + .set_features = nfp_vdpa_set_features, + .get_vfio_group_fd = nfp_vdpa_get_vfio_group_fd, + .get_vfio_device_fd = nfp_vdpa_get_vfio_device_fd, + .get_notify_area = nfp_vdpa_get_notify_area, }; static int -- 2.39.1