From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id 8BB9443279;
Fri, 3 Nov 2023 12:20:41 +0100 (CET)
Received: from mails.dpdk.org (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id 733C840691;
Fri, 3 Nov 2023 12:20:40 +0100 (CET)
Received: from NAM12-DM6-obe.outbound.protection.outlook.com
(mail-dm6nam12on2075.outbound.protection.outlook.com [40.107.243.75])
by mails.dpdk.org (Postfix) with ESMTP id DF8574067D
for <dev@dpdk.org>; Fri, 3 Nov 2023 12:20:38 +0100 (CET)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=GVS96u8jUKbSbJkX5DhF5He6acjxA85qlDBELvC6PNORshSkFX1Jiac1yufCFHnncDkg/faYN01+AHByW0Zvv0/0QmDXUv8WQ4sXF+1zoA4WV+nwbM1jEMH4VyhK6lfzBAllE99Wn7eKbMS6vNJB6XJnxfiEexClkuUrPw803QPFJIkDM+EAEPR9+OV/60QBePeQI2iUpyXrU15hxO/w4EezGUZSdyGhKCTtWFmTjj2P5s1U5Vb/scGxDm6xHlY4KJndwPo4EFguvhegTU2+YyhHLXT5fkJ524IQOuFxPjYkT08aPE/xU9VlQp747d7ntSYtY5tIcU1BAd9NA7mbEg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
bh=R6lmhbqBY/pzmgv1VGaKpQpMhXOUpjORDh1a5RMZqCE=;
b=HeBlO5eUUDZy4aOKqfUAhmigGmgcDrQeJ2OoJa1zrTrXijWafyKS1Lsnr4eJQfxE///6HFEpbEZiTbFiaEsqPwnftwcWJl7H1V7x9vB+G4RvgY2szUhPHxbIbfbhepDYAnIYcUV4s/WU4fOzJ5LeB4lPeJ1EnDCLoSoLrylXjSzmCMDtVz1LYwEP1P1w2be4rdKHrNIOYzZC9yrVlgUKjEUUGKJbj9Q6EEolfUB/wlUwW350g/yH0ob3hWpNPQySDO4o9hHdzSBM3a71bBtmmDeRRi+BOGkDlV7X4rkPgW7BOMMrLA3zlSV9W7kaX8Dm0f+nez/wX1gvahDbmro6Jw==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is
165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass
(p=quarantine sp=quarantine pct=100) action=none header.from=amd.com;
dkim=none (message not signed); arc=none (0)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=R6lmhbqBY/pzmgv1VGaKpQpMhXOUpjORDh1a5RMZqCE=;
b=hHsiyD3JBQg9uAICoF1EkLuqsfbz8qhy18COpNf+uCa1U9pLj3efCO0aavQpNn3MLkQkrzoCD2Ht0pJbZtbfOzQ6geGJwpNqc5Wp7OXjfFkRPeqn1jbMXlUR5ZDv4EpwSVf7Afc3LKBfjoDljkJ763Y1EJnxYKVHlPrVsomsrbQ=
Received: from MN2PR19CA0031.namprd19.prod.outlook.com (2603:10b6:208:178::44)
by SJ2PR12MB7964.namprd12.prod.outlook.com (2603:10b6:a03:4cf::7)
with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.24; Fri, 3 Nov
2023 11:20:36 +0000
Received: from BL6PEPF0001AB4A.namprd04.prod.outlook.com
(2603:10b6:208:178:cafe::a4) by MN2PR19CA0031.outlook.office365.com
(2603:10b6:208:178::44) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.22 via Frontend
Transport; Fri, 3 Nov 2023 11:20:36 +0000
X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17)
smtp.mailfrom=amd.com; dkim=none (message not signed)
header.d=none;dmarc=pass action=none header.from=amd.com;
Received-SPF: Pass (protection.outlook.com: domain of amd.com designates
165.204.84.17 as permitted sender) receiver=protection.outlook.com;
client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C
Received: from SATLEXMB04.amd.com (165.204.84.17) by
BL6PEPF0001AB4A.mail.protection.outlook.com (10.167.242.68) with Microsoft
SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
15.20.6954.19 via Frontend Transport; Fri, 3 Nov 2023 11:20:35 +0000
Received: from SATLEXMB03.amd.com (10.181.40.144) by SATLEXMB04.amd.com
(10.181.40.145) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Fri, 3 Nov
2023 06:20:35 -0500
Received: from xhdipdslab61.xilinx.com (10.180.168.240) by SATLEXMB03.amd.com
(10.181.40.144) with Microsoft SMTP Server id 15.1.2507.32 via
Frontend Transport; Fri, 3 Nov 2023 06:20:33 -0500
From: Shubham Rohila <shubham.rohila@amd.com>
To: <nikhil.agarwal@amd.com>, <david.marchand@redhat.com>,
<nipun.gupta@amd.com>, <dev@dpdk.org>
CC: <thomas@monjalon.net>, <ferruh.yigit@amd.com>,
<anatoly.burakov@intel.com>, Shubham Rohila <shubham.rohila@amd.com>
Subject: [PATCH v2 2/2] bus/cdx: add bus master enable/disable APIs
Date: Fri, 3 Nov 2023 16:50:16 +0530
Message-ID: <20231103112016.1945684-2-shubham.rohila@amd.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20231103112016.1945684-1-shubham.rohila@amd.com>
References: <20231027162212.3353973-1-ferruh.yigit@amd.com>
<20231103112016.1945684-1-shubham.rohila@amd.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-EOPAttributedMessage: 0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4A:EE_|SJ2PR12MB7964:EE_
X-MS-Office365-Filtering-Correlation-Id: 7b45d533-4ba6-491a-b894-08dbdc5ee6c7
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: StfUmH/LqGaTvf8xHmK6V4bMu4Wkon03ODfdh7C27mw6fNYcooFQNyhzlFKCyc67RynJhzNKg3BeeKPuA2PcaOZxOqljT5j5kHWYDdzh3OFlI3oq2UEjEu5xYYxShArqqV4Iqg4K6Cj5eLl53AhMLRTin04H3GfAKNDgWK8k38CVYvja2NSL3BmNHWj2lF7Iqkx+UISoOeDpFC9moT983TmQQjal5BFUcTgkoGcUlleqh2nBLUK6Nr6EMisBLL1BRr/AZQk8z3Bc1DHbhedq44BTnDFXxPeFgvjFPCsigB67+mhxs78p48C/MOXAEKqrnHZh9+XWH5Yct4vFHwSyTr32xgsXSlEjvN91Hh4X0tq4i4CTKzm/pbTCPllzo2DUhIdV/hrBRJrLA0+j2GP70pY3fDqKzLtEsW+1aEBnIoZNe3nwlUEh0FjJM6ClocCL0zKMXHtJxt6uemkmMhjMme58/5BmuAk3mrsH5CyNQ9ArH339talaZ+2iWfL9QRdy/zLiJ9E41pm9QdOrwb3/t9TPxOUDoRrpf/MuWnfdCGoaDOenuz40StuN2vClxKb19qS6mXIuFzmuBUySraExHmrrcuSBN0EBpp5uixJBc02UKTE73USW8YJxLWzdaqMVQmjzcOWX5CMbYaQCvi8Do2AB3OpPsCiIqUFlAByJ48dhC6Lxp6RDtGrCh3Cueq7R0FiMvkPATjMqIcQ0qqs0ZloxbffI8VFGrDPZAdEDTQ0+FfYQpGDUX6cNQdSwY8WbTLhoCywK7zLFXuYZ5WjcmA==
X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:;
IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE;
SFS:(13230031)(4636009)(376002)(396003)(136003)(346002)(39860400002)(230922051799003)(82310400011)(186009)(451199024)(64100799003)(1800799009)(46966006)(36840700001)(40470700004)(47076005)(26005)(6666004)(478600001)(54906003)(2906002)(81166007)(40480700001)(70206006)(110136005)(70586007)(1076003)(82740400003)(2616005)(36756003)(336012)(426003)(40460700003)(41300700001)(44832011)(316002)(5660300002)(356005)(83380400001)(86362001)(36860700001)(4326008)(8676002)(8936002)(36900700001);
DIR:OUT; SFP:1101;
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2023 11:20:35.9473 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 7b45d533-4ba6-491a-b894-08dbdc5ee6c7
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17];
Helo=[SATLEXMB04.amd.com]
X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4A.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7964
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Define rte_cdx_vfio_bm_enable and rte_cdx_vfio_bm_disable to
enable or disable bus master functionality for cdx devices.
Signed-off-by: Shubham Rohila <shubham.rohila@amd.com>
---
v2
- Fix indentations and alphabetical orderings.
- Wrap checks in parenthesis for readability
- Assign argsz before using in rte_cdx_vfio_bm_disable
- Remove unecessary variable initialisation
drivers/bus/cdx/bus_cdx_driver.h | 25 ++++++++
drivers/bus/cdx/cdx_vfio.c | 103 +++++++++++++++++++++++++++++++
drivers/bus/cdx/version.map | 2 +
lib/eal/include/rte_vfio.h | 27 ++++++++
4 files changed, 157 insertions(+)
diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
index 1c9a64c87a..211f8e406b 100644
--- a/drivers/bus/cdx/bus_cdx_driver.h
+++ b/drivers/bus/cdx/bus_cdx_driver.h
@@ -16,6 +16,7 @@ extern "C" {
#include <stdlib.h>
#include <inttypes.h>
+#include <linux/types.h>
#include <bus_driver.h>
#include <dev_driver.h>
@@ -178,6 +179,30 @@ int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
__rte_internal
int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
+/**
+ * Enable Bus Mastering for CDX bus devices.
+ *
+ * @param dev
+ * Pointer to the cdx device.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev);
+
+/**
+ * Disable Bus Mastering for CDX bus devices.
+ *
+ * @param dev
+ * Pointer to the cdx device.
+ *
+ * @return
+ * 0 on success, -1 on error.
+ */
+__rte_internal
+int rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev);
+
/**
* Unregister a CDX driver.
*
diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
index 8cac79782e..1481022f13 100644
--- a/drivers/bus/cdx/cdx_vfio.c
+++ b/drivers/bus/cdx/cdx_vfio.c
@@ -102,6 +102,10 @@ cdx_vfio_unmap_resource_primary(struct rte_cdx_device *dev)
int ret, vfio_dev_fd;
if (rte_intr_fd_get(dev->intr_handle) >= 0) {
+ if (rte_cdx_vfio_bm_disable(dev) < 0)
+ CDX_BUS_ERR("Error when disabling bus master for %s",
+ dev->device.name);
+
if (close(rte_intr_fd_get(dev->intr_handle)) < 0) {
CDX_BUS_ERR("Error when closing eventfd file descriptor for %s",
dev->device.name);
@@ -255,6 +259,16 @@ cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
return -1;
}
+ /*
+ * Enable Bus mastering for the device. errno is set as ENOTTY if
+ * device does not support configuring bus master.
+ */
+ if (rte_cdx_vfio_bm_enable(dev) && (errno != -ENOTTY)) {
+ CDX_BUS_ERR("Bus master enable failure! Error: %d (%s)", errno,
+ strerror(errno));
+ return -1;
+ }
+
return 0;
}
@@ -595,3 +609,92 @@ rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
return ret;
}
+
+/* enable Bus Mastering */
+int
+rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev)
+{
+ struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
+ struct vfio_device_feature_bus_master *vfio_bm_feature;
+ struct vfio_device_feature *feature;
+ int vfio_dev_fd, ret;
+ size_t argsz;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ argsz = sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_bus_master);
+
+ feature = (struct vfio_device_feature *)malloc(argsz);
+ if (!feature)
+ return -ENOMEM;
+
+ vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data;
+
+ feature->argsz = argsz;
+
+ feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE;
+ feature->flags |= VFIO_DEVICE_FEATURE_SET;
+ ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature);
+ if (ret) {
+ CDX_BUS_ERR("Bus Master configuring not supported for device: %s, error: %d (%s)\n",
+ dev->name, errno, strerror(errno));
+ free(feature);
+ return ret;
+ }
+
+ feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET;
+ vfio_bm_feature->op = VFIO_DEVICE_FEATURE_SET_MASTER;
+ ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature);
+ if (ret < 0)
+ CDX_BUS_ERR("BM Enable Error for device: %s, Error: %d (%s)\n",
+ dev->name, errno, strerror(errno));
+
+ free(feature);
+ return ret;
+}
+
+/* Disable Bus Mastering */
+int
+rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev)
+{
+ struct vfio_device_feature_bus_master *vfio_bm_feature;
+ struct vfio_device_feature *feature;
+ int vfio_dev_fd, ret;
+ size_t argsz;
+
+ vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
+ if (vfio_dev_fd < 0)
+ return -1;
+
+ argsz = sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_bus_master);
+
+ feature = (struct vfio_device_feature *)malloc(argsz);
+ if (!feature)
+ return -ENOMEM;
+
+ vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data;
+
+ feature->argsz = argsz;
+
+ feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE;
+ feature->flags |= VFIO_DEVICE_FEATURE_SET;
+ ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature);
+ if (ret) {
+ CDX_BUS_ERR("Bus Master configuring not supported for device: %s, Error: %d (%s)\n",
+ dev->name, errno, strerror(errno));
+ free(feature);
+ return ret;
+ }
+
+ feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET;
+ vfio_bm_feature->op = VFIO_DEVICE_FEATURE_CLEAR_MASTER;
+ ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature);
+ if (ret < 0)
+ CDX_BUS_ERR("BM Disable Error for device: %s, Error: %d (%s)\n",
+ dev->name, errno, strerror(errno));
+
+ free(feature);
+ return ret;
+}
diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
index 0a15d39ae8..7a22cf5b4b 100644
--- a/drivers/bus/cdx/version.map
+++ b/drivers/bus/cdx/version.map
@@ -5,6 +5,8 @@ INTERNAL {
rte_cdx_register;
rte_cdx_unmap_device;
rte_cdx_unregister;
+ rte_cdx_vfio_bm_disable;
+ rte_cdx_vfio_bm_enable;
rte_cdx_vfio_intr_disable;
rte_cdx_vfio_intr_enable;
diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
index 3487c4f2a2..22832afd0f 100644
--- a/lib/eal/include/rte_vfio.h
+++ b/lib/eal/include/rte_vfio.h
@@ -73,6 +73,33 @@ struct vfio_info_cap_header {
#define RTE_VFIO_CAP_MSIX_MAPPABLE 3
#endif
+/* VFIO_DEVICE_FEATURE is defined for kernel version 5.7 and newer. */
+#ifdef VFIO_DEVICE_FEATURE
+#define RTE_VFIO_DEVICE_FEATURE VFIO_DEVICE_FEATURE
+#else
+#define RTE_VFIO_DEVICE_FEATURE _IO(VFIO_TYPE, VFIO_BASE + 17)
+struct vfio_device_feature {
+ __u32 argsz;
+ __u32 flags;
+#define VFIO_DEVICE_FEATURE_MASK (0xffff) /* 16-bit feature index */
+#define VFIO_DEVICE_FEATURE_GET (1 << 16) /* Get feature into data[] */
+#define VFIO_DEVICE_FEATURE_SET (1 << 17) /* Set feature from data[] */
+#define VFIO_DEVICE_FEATURE_PROBE (1 << 18) /* Probe feature support */
+ __u8 data[];
+};
+#endif
+
+#ifdef VFIO_DEVICE_FEATURE_BUS_MASTER
+#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER VFIO_DEVICE_FEATURE_BUS_MASTER
+#else
+#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER 10
+struct vfio_device_feature_bus_master {
+ __u32 op;
+#define VFIO_DEVICE_FEATURE_CLEAR_MASTER 0 /* Clear Bus Master */
+#define VFIO_DEVICE_FEATURE_SET_MASTER 1 /* Set Bus Master */
+};
+#endif
+
#else /* not VFIO_PRESENT */
/* we don't need an actual definition, only pointer is used */
--
2.25.1