From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8BB9443279; Fri, 3 Nov 2023 12:20:41 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 733C840691; Fri, 3 Nov 2023 12:20:40 +0100 (CET) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2075.outbound.protection.outlook.com [40.107.243.75]) by mails.dpdk.org (Postfix) with ESMTP id DF8574067D for ; Fri, 3 Nov 2023 12:20:38 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GVS96u8jUKbSbJkX5DhF5He6acjxA85qlDBELvC6PNORshSkFX1Jiac1yufCFHnncDkg/faYN01+AHByW0Zvv0/0QmDXUv8WQ4sXF+1zoA4WV+nwbM1jEMH4VyhK6lfzBAllE99Wn7eKbMS6vNJB6XJnxfiEexClkuUrPw803QPFJIkDM+EAEPR9+OV/60QBePeQI2iUpyXrU15hxO/w4EezGUZSdyGhKCTtWFmTjj2P5s1U5Vb/scGxDm6xHlY4KJndwPo4EFguvhegTU2+YyhHLXT5fkJ524IQOuFxPjYkT08aPE/xU9VlQp747d7ntSYtY5tIcU1BAd9NA7mbEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=R6lmhbqBY/pzmgv1VGaKpQpMhXOUpjORDh1a5RMZqCE=; b=HeBlO5eUUDZy4aOKqfUAhmigGmgcDrQeJ2OoJa1zrTrXijWafyKS1Lsnr4eJQfxE///6HFEpbEZiTbFiaEsqPwnftwcWJl7H1V7x9vB+G4RvgY2szUhPHxbIbfbhepDYAnIYcUV4s/WU4fOzJ5LeB4lPeJ1EnDCLoSoLrylXjSzmCMDtVz1LYwEP1P1w2be4rdKHrNIOYzZC9yrVlgUKjEUUGKJbj9Q6EEolfUB/wlUwW350g/yH0ob3hWpNPQySDO4o9hHdzSBM3a71bBtmmDeRRi+BOGkDlV7X4rkPgW7BOMMrLA3zlSV9W7kaX8Dm0f+nez/wX1gvahDbmro6Jw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R6lmhbqBY/pzmgv1VGaKpQpMhXOUpjORDh1a5RMZqCE=; b=hHsiyD3JBQg9uAICoF1EkLuqsfbz8qhy18COpNf+uCa1U9pLj3efCO0aavQpNn3MLkQkrzoCD2Ht0pJbZtbfOzQ6geGJwpNqc5Wp7OXjfFkRPeqn1jbMXlUR5ZDv4EpwSVf7Afc3LKBfjoDljkJ763Y1EJnxYKVHlPrVsomsrbQ= Received: from MN2PR19CA0031.namprd19.prod.outlook.com (2603:10b6:208:178::44) by SJ2PR12MB7964.namprd12.prod.outlook.com (2603:10b6:a03:4cf::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.24; Fri, 3 Nov 2023 11:20:36 +0000 Received: from BL6PEPF0001AB4A.namprd04.prod.outlook.com (2603:10b6:208:178:cafe::a4) by MN2PR19CA0031.outlook.office365.com (2603:10b6:208:178::44) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.22 via Frontend Transport; Fri, 3 Nov 2023 11:20:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL6PEPF0001AB4A.mail.protection.outlook.com (10.167.242.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6954.19 via Frontend Transport; Fri, 3 Nov 2023 11:20:35 +0000 Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Fri, 3 Nov 2023 06:20:35 -0500 Received: from xhdipdslab61.xilinx.com (10.180.168.240) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server id 15.1.2507.32 via Frontend Transport; Fri, 3 Nov 2023 06:20:33 -0500 From: Shubham Rohila To: , , , CC: , , , Shubham Rohila Subject: [PATCH v2 2/2] bus/cdx: add bus master enable/disable APIs Date: Fri, 3 Nov 2023 16:50:16 +0530 Message-ID: <20231103112016.1945684-2-shubham.rohila@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231103112016.1945684-1-shubham.rohila@amd.com> References: <20231027162212.3353973-1-ferruh.yigit@amd.com> <20231103112016.1945684-1-shubham.rohila@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4A:EE_|SJ2PR12MB7964:EE_ X-MS-Office365-Filtering-Correlation-Id: 7b45d533-4ba6-491a-b894-08dbdc5ee6c7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: StfUmH/LqGaTvf8xHmK6V4bMu4Wkon03ODfdh7C27mw6fNYcooFQNyhzlFKCyc67RynJhzNKg3BeeKPuA2PcaOZxOqljT5j5kHWYDdzh3OFlI3oq2UEjEu5xYYxShArqqV4Iqg4K6Cj5eLl53AhMLRTin04H3GfAKNDgWK8k38CVYvja2NSL3BmNHWj2lF7Iqkx+UISoOeDpFC9moT983TmQQjal5BFUcTgkoGcUlleqh2nBLUK6Nr6EMisBLL1BRr/AZQk8z3Bc1DHbhedq44BTnDFXxPeFgvjFPCsigB67+mhxs78p48C/MOXAEKqrnHZh9+XWH5Yct4vFHwSyTr32xgsXSlEjvN91Hh4X0tq4i4CTKzm/pbTCPllzo2DUhIdV/hrBRJrLA0+j2GP70pY3fDqKzLtEsW+1aEBnIoZNe3nwlUEh0FjJM6ClocCL0zKMXHtJxt6uemkmMhjMme58/5BmuAk3mrsH5CyNQ9ArH339talaZ+2iWfL9QRdy/zLiJ9E41pm9QdOrwb3/t9TPxOUDoRrpf/MuWnfdCGoaDOenuz40StuN2vClxKb19qS6mXIuFzmuBUySraExHmrrcuSBN0EBpp5uixJBc02UKTE73USW8YJxLWzdaqMVQmjzcOWX5CMbYaQCvi8Do2AB3OpPsCiIqUFlAByJ48dhC6Lxp6RDtGrCh3Cueq7R0FiMvkPATjMqIcQ0qqs0ZloxbffI8VFGrDPZAdEDTQ0+FfYQpGDUX6cNQdSwY8WbTLhoCywK7zLFXuYZ5WjcmA== X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(4636009)(376002)(396003)(136003)(346002)(39860400002)(230922051799003)(82310400011)(186009)(451199024)(64100799003)(1800799009)(46966006)(36840700001)(40470700004)(47076005)(26005)(6666004)(478600001)(54906003)(2906002)(81166007)(40480700001)(70206006)(110136005)(70586007)(1076003)(82740400003)(2616005)(36756003)(336012)(426003)(40460700003)(41300700001)(44832011)(316002)(5660300002)(356005)(83380400001)(86362001)(36860700001)(4326008)(8676002)(8936002)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2023 11:20:35.9473 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7b45d533-4ba6-491a-b894-08dbdc5ee6c7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7964 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Define rte_cdx_vfio_bm_enable and rte_cdx_vfio_bm_disable to enable or disable bus master functionality for cdx devices. Signed-off-by: Shubham Rohila --- v2 - Fix indentations and alphabetical orderings. - Wrap checks in parenthesis for readability - Assign argsz before using in rte_cdx_vfio_bm_disable - Remove unecessary variable initialisation drivers/bus/cdx/bus_cdx_driver.h | 25 ++++++++ drivers/bus/cdx/cdx_vfio.c | 103 +++++++++++++++++++++++++++++++ drivers/bus/cdx/version.map | 2 + lib/eal/include/rte_vfio.h | 27 ++++++++ 4 files changed, 157 insertions(+) diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h index 1c9a64c87a..211f8e406b 100644 --- a/drivers/bus/cdx/bus_cdx_driver.h +++ b/drivers/bus/cdx/bus_cdx_driver.h @@ -16,6 +16,7 @@ extern "C" { #include #include +#include #include #include @@ -178,6 +179,30 @@ int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle); __rte_internal int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle); +/** + * Enable Bus Mastering for CDX bus devices. + * + * @param dev + * Pointer to the cdx device. + * + * @return + * 0 on success, -1 on error. + */ +__rte_internal +int rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev); + +/** + * Disable Bus Mastering for CDX bus devices. + * + * @param dev + * Pointer to the cdx device. + * + * @return + * 0 on success, -1 on error. + */ +__rte_internal +int rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev); + /** * Unregister a CDX driver. * diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c index 8cac79782e..1481022f13 100644 --- a/drivers/bus/cdx/cdx_vfio.c +++ b/drivers/bus/cdx/cdx_vfio.c @@ -102,6 +102,10 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev) int ret, vfio_dev_fd; if (rte_intr_fd_get(dev->intr_handle) >= 0) { + if (rte_cdx_vfio_bm_disable(dev) < 0) + CDX_BUS_ERR("Error when disabling bus master for %s", + dev->device.name); + if (close(rte_intr_fd_get(dev->intr_handle)) < 0) { CDX_BUS_ERR("Error when closing eventfd file descriptor for %s", dev->device.name); @@ -255,6 +259,16 @@ cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd, return -1; } + /* + * Enable Bus mastering for the device. errno is set as ENOTTY if + * device does not support configuring bus master. + */ + if (rte_cdx_vfio_bm_enable(dev) && (errno != -ENOTTY)) { + CDX_BUS_ERR("Bus master enable failure! Error: %d (%s)", errno, + strerror(errno)); + return -1; + } + return 0; } @@ -595,3 +609,92 @@ rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle) return ret; } + +/* enable Bus Mastering */ +int +rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev) +{ + struct vfio_device_info device_info = { .argsz = sizeof(device_info) }; + struct vfio_device_feature_bus_master *vfio_bm_feature; + struct vfio_device_feature *feature; + int vfio_dev_fd, ret; + size_t argsz; + + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle); + if (vfio_dev_fd < 0) + return -1; + + argsz = sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_bus_master); + + feature = (struct vfio_device_feature *)malloc(argsz); + if (!feature) + return -ENOMEM; + + vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data; + + feature->argsz = argsz; + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE; + feature->flags |= VFIO_DEVICE_FEATURE_SET; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret) { + CDX_BUS_ERR("Bus Master configuring not supported for device: %s, error: %d (%s)\n", + dev->name, errno, strerror(errno)); + free(feature); + return ret; + } + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET; + vfio_bm_feature->op = VFIO_DEVICE_FEATURE_SET_MASTER; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret < 0) + CDX_BUS_ERR("BM Enable Error for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + + free(feature); + return ret; +} + +/* Disable Bus Mastering */ +int +rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev) +{ + struct vfio_device_feature_bus_master *vfio_bm_feature; + struct vfio_device_feature *feature; + int vfio_dev_fd, ret; + size_t argsz; + + vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle); + if (vfio_dev_fd < 0) + return -1; + + argsz = sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_bus_master); + + feature = (struct vfio_device_feature *)malloc(argsz); + if (!feature) + return -ENOMEM; + + vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data; + + feature->argsz = argsz; + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE; + feature->flags |= VFIO_DEVICE_FEATURE_SET; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret) { + CDX_BUS_ERR("Bus Master configuring not supported for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + free(feature); + return ret; + } + + feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET; + vfio_bm_feature->op = VFIO_DEVICE_FEATURE_CLEAR_MASTER; + ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature); + if (ret < 0) + CDX_BUS_ERR("BM Disable Error for device: %s, Error: %d (%s)\n", + dev->name, errno, strerror(errno)); + + free(feature); + return ret; +} diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map index 0a15d39ae8..7a22cf5b4b 100644 --- a/drivers/bus/cdx/version.map +++ b/drivers/bus/cdx/version.map @@ -5,6 +5,8 @@ INTERNAL { rte_cdx_register; rte_cdx_unmap_device; rte_cdx_unregister; + rte_cdx_vfio_bm_disable; + rte_cdx_vfio_bm_enable; rte_cdx_vfio_intr_disable; rte_cdx_vfio_intr_enable; diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h index 3487c4f2a2..22832afd0f 100644 --- a/lib/eal/include/rte_vfio.h +++ b/lib/eal/include/rte_vfio.h @@ -73,6 +73,33 @@ struct vfio_info_cap_header { #define RTE_VFIO_CAP_MSIX_MAPPABLE 3 #endif +/* VFIO_DEVICE_FEATURE is defined for kernel version 5.7 and newer. */ +#ifdef VFIO_DEVICE_FEATURE +#define RTE_VFIO_DEVICE_FEATURE VFIO_DEVICE_FEATURE +#else +#define RTE_VFIO_DEVICE_FEATURE _IO(VFIO_TYPE, VFIO_BASE + 17) +struct vfio_device_feature { + __u32 argsz; + __u32 flags; +#define VFIO_DEVICE_FEATURE_MASK (0xffff) /* 16-bit feature index */ +#define VFIO_DEVICE_FEATURE_GET (1 << 16) /* Get feature into data[] */ +#define VFIO_DEVICE_FEATURE_SET (1 << 17) /* Set feature from data[] */ +#define VFIO_DEVICE_FEATURE_PROBE (1 << 18) /* Probe feature support */ + __u8 data[]; +}; +#endif + +#ifdef VFIO_DEVICE_FEATURE_BUS_MASTER +#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER VFIO_DEVICE_FEATURE_BUS_MASTER +#else +#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER 10 +struct vfio_device_feature_bus_master { + __u32 op; +#define VFIO_DEVICE_FEATURE_CLEAR_MASTER 0 /* Clear Bus Master */ +#define VFIO_DEVICE_FEATURE_SET_MASTER 1 /* Set Bus Master */ +}; +#endif + #else /* not VFIO_PRESENT */ /* we don't need an actual definition, only pointer is used */ -- 2.25.1