From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 8A0E943215;
	Fri, 27 Oct 2023 15:34:14 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 7ADAA40A8A;
	Fri, 27 Oct 2023 15:34:14 +0200 (CEST)
Received: from NAM10-DM6-obe.outbound.protection.outlook.com
 (mail-dm6nam10on2048.outbound.protection.outlook.com [40.107.93.48])
 by mails.dpdk.org (Postfix) with ESMTP id 2055B402B9
 for <dev@dpdk.org>; Fri, 27 Oct 2023 15:34:12 +0200 (CEST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=iEu6+kimLpxnPq+5JqyOnAkbf0RxCS9SnFrrhEsZhuKVSiyVI8jOSH12L2untucjSap2f20zorwgybAfjaXrh0/Gx/t24AQR4lnX6V2Kf80KhezviHpdCjRwm8ixZOcTrkuHnqR6FCANl4jGKmE8VUC3PvXybRx9BcO5c9sROwqncdbV89Z2m9d2HW/uu3PabkdSZkPLclU3Tz5p0/rBImin0DO9sh9kV+meoI+K6t+ewbE/e/NsWlPqIQPDWMPIuJ7CXeugee1w5N/mMRgdHO8B8QrOysoQjSgEPVG4QrVgWjxomy+MFaSyz50fvY6VcZFmFlZt6LfFd40qPMHAWA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=VOLW2qZxVeQ0BCqUe6biR3aAdPxe8Q62XxtHMe/3++M=;
 b=aR7lawJ39vyUdEZtP6pluJvsLo6Oo55YVSSTAubj4mPIznUOYqf28NTxyMdlG83HIC6FYXLjGK6+y5CVhD5KPIUxgn7L4MdiHMkgj9lYH/h/eVNbP1301u4uySoetO8GMjBai37ZVhm9PBpb0bx2TA3KeXtAGj5gBeu2uRww0Jee6trsaxvFZ7LR+uBTitEU2CtsXMd6ST9U4sClfjPy9x713rSgqJTdnd1MlwctKHAvU5OmbzsCEv+vxI8ostp/z1vW956KbvhzlmmnOAXe8JcaD/WXMgxU/1+MwgXK4aOQaNUdiUbCyJj19sD3CUHDkt/BUO3Sk6qyf/GomUKxIQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass
 header.d=amd.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; 
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=VOLW2qZxVeQ0BCqUe6biR3aAdPxe8Q62XxtHMe/3++M=;
 b=aQTr8xq18k09soQ4fgeCVHNVx7hkFBktB5vgUXfVDj5sga0cLgYszk1SA558/LhU1jUPrncyDIm62DAoxcQQW76CboVlmsmiqlyEemVGoDX/GnF+X4QbtUAEptC98//9+AeiSCgnxIWyjqfR6ctYkwzpj4d76NS3mGswb3dLfOM=
Authentication-Results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=amd.com;
Received: from CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8)
 by BY5PR12MB4324.namprd12.prod.outlook.com (2603:10b6:a03:209::10)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.24; Fri, 27 Oct
 2023 13:34:09 +0000
Received: from CH3PR12MB8308.namprd12.prod.outlook.com
 ([fe80::1ff1:d311:1f3a:5b42]) by CH3PR12MB8308.namprd12.prod.outlook.com
 ([fe80::1ff1:d311:1f3a:5b42%7]) with mapi id 15.20.6933.019; Fri, 27 Oct 2023
 13:34:09 +0000
Message-ID: <4bcbd199-13f0-4155-831e-6d1d1e3b18d0@amd.com>
Date: Fri, 27 Oct 2023 19:04:00 +0530
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] Add bus master enable/disable APIs for CDX devices
Content-Language: en-US
To: Shubham Rohila <shubham.rohila@amd.com>, nikhil.agarwal@amd.com,
 david.marchand@redhat.com, dev@dpdk.org
Cc: thomas@monjalon.net, ferruh.yigit@amd.com, anatoly.burakov@intel.com
References: <20231026032009.3275263-1-shubham.rohila@amd.com>
From: "Gupta, Nipun" <nipun.gupta@amd.com>
In-Reply-To: <20231026032009.3275263-1-shubham.rohila@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ClientProxiedBy: BM1P287CA0001.INDP287.PROD.OUTLOOK.COM
 (2603:1096:b00:40::21) To CH3PR12MB8308.namprd12.prod.outlook.com
 (2603:10b6:610:131::8)
MIME-Version: 1.0
X-MS-PublicTrafficType: Email
X-MS-TrafficTypeDiagnostic: CH3PR12MB8308:EE_|BY5PR12MB4324:EE_
X-MS-Office365-Filtering-Correlation-Id: 8afb5c9b-4d9c-47cf-9648-08dbd6f16608
X-MS-Exchange-SenderADCheck: 1
X-MS-Exchange-AntiSpam-Relay: 0
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: o0hPH6zTkjj62Q0S48WZwHfi3ye6Fnp1+BMtho33gcra+GMVwTRR1zfHz2I3FjjV36bkq3GjuPCAhkF4x6zeIT0QxWBhy3tyB9XhtZYDtkZlAUx/I6d16wkNr7w57FCS0gxuJwe7RnGulTR/TqC44wLmYrtKnz6+m/eWXLqZhbfrCnTNY+Dyg9NqzVU0JpXpH+kKc9q27I5Mni7J3Id0khiLSjUGu3ynjFqSnqphnEchBaQE+svfJvUFlK74kGr3ucE5ovYvP2mP+q1u3qRgNj1kc950v2NrBWA5VkATzgvoTS9H0vzzOM/ZL2/eVrvJ+4YheduOi9tnUc0C802PDF0JbXiHXD5UzcWow4K7HpcbmhtM5i3AM/sbUPy8YARwqol1nkfglG0CBI9JgpO++of18Qsac7B0EfP8sE0Xtngvv0EF4j/oAEl0y3wsIk58uwsHUrvC+NbRPneFng3pOdF6YbOrsMjUDXwv3TYs3lsgJhXeubrA4/j4JXGJlLLNK0EJdVA2htYdToV0xn/Z9R0l2DfaEWLgaH0/9jebz0QxR924NZtg6p00N2LUszxGzV/u2OW4yMX5BzPLxNbbckaeCM3bJoslCg4eT7xR02sV55u+IlCgwnmXe5QW+9LeoHuHfT+2cGhJ/U4Qbh2RCw==
X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:CH3PR12MB8308.namprd12.prod.outlook.com; PTR:; CAT:NONE;
 SFS:(13230031)(39860400002)(396003)(136003)(366004)(346002)(376002)(230922051799003)(64100799003)(186009)(1800799009)(451199024)(2906002)(31696002)(2616005)(86362001)(36756003)(26005)(5660300002)(41300700001)(31686004)(8676002)(8936002)(4326008)(38100700002)(316002)(66556008)(83380400001)(66476007)(66946007)(478600001)(6486002)(53546011)(6512007)(6666004)(6506007)(45980500001)(43740500002);
 DIR:OUT; SFP:1101; 
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YmZ6WlhpNlZkdFZlVjIvS0FYS1JoYUpKNHZnaGgzbXRxNHArdStWRUtYS21n?=
 =?utf-8?B?WFRGdE5JOXh4RWJWaVkyUlhlR2oyMEZaWGpsUkRtNi9hdzBjT2xaY3UzYzRl?=
 =?utf-8?B?aXdRdno0VmlLc1Vta0I5WnB0SkFKekE0Qkc0K0lvK3FBTFNqZW8vcnM5N3p0?=
 =?utf-8?B?Y2FPdklHTklmUE1FaG1ablhvdFFDb1lTNXNjaWd6ZDRNWEJzYUU0U3Y1dXps?=
 =?utf-8?B?dlhWSFNxVUhDck0zU2lnYVNhampXWUhwaHFWYXlISmVoNlVrWGpIcEpwMk5j?=
 =?utf-8?B?aHZWUzkzcHVTd3l5SXVBZ1U4eTkyUklWdEZlVFprbUovN0NLN1U4UE1pYW1n?=
 =?utf-8?B?WWJzQUFHRzZ5UTJuRktsanRyZDI3Q09EcVFFeVMvZWdBT2tveDdRM3ZjTk1j?=
 =?utf-8?B?bzhMVzdKWXlqNWNzcGdCbldJU2hxVWFRSHVQWkN6d0RrTUdtY1dGRmw1YWJI?=
 =?utf-8?B?R05lenI4em5Ram9URG9zcmtDYWtmNkgvKzVQeUQyMDdMSUdSUlVMTUFVZTYw?=
 =?utf-8?B?UFAwM3lwS09sRzkzRnpPam53c0VHMWRoTWVoWXFQMkFvZk1zc2NHMlBZUHBq?=
 =?utf-8?B?T0hZNE1USGJXZDI4KzFkYVNnQ1hXTlNCOXZuYTVoMmZ2UXpNNmF4WGMwM1l6?=
 =?utf-8?B?SnJkZUNVbHAzcHdEUjVWdllUVHMvK0lyYnFzM3FDWVE1OUNEK0EySVl0UTdm?=
 =?utf-8?B?WXc1N05tdTZwcWFld2VJRDhJdmV5amJSK0JLK3BGL3RpWEVQSW96TFQ2VTlM?=
 =?utf-8?B?OU1sY203M0pIWUtnYXBtb3poMzJyZXFpNStHZjJLTG9pbWpxUWpUayt0Rk9a?=
 =?utf-8?B?N0hpYWZSclI3cTZ3NmplTjRKQkduU0h6bTl5RnJTRDJBeCtWL0F1d3lJakF6?=
 =?utf-8?B?NmxLamU2OG0xdDZ3REJQdkZtb3Fta2xrOUtQZUlncHcwbTFSdjdscFd0ZWRL?=
 =?utf-8?B?NkVqY2ZwanJEc3dXUG56cnkxSUhmWDIreEJ3WldBYnJvR2NKMGNJYVJSZkM5?=
 =?utf-8?B?eGx6MnlpQnlOdTMwaWd6eURxQk92c0FvQU1nd002M21ZNXR3ZTVBZXZFbndX?=
 =?utf-8?B?cjFnc0UwUXZqZUtkTERCTGhBb3o4eE14UzduNXRVK0kvRlNqT05PNm54Smk1?=
 =?utf-8?B?dzR3d05QejE2UThYS2o2aXhFMmF6VENacmhhK040dVlad3JnbHRmVFJlMVhs?=
 =?utf-8?B?R3VSN240Skl6YVBTaW1HbEUwckpPenl4RHptZUtVN3ZXQVkrQi84TjZyaStH?=
 =?utf-8?B?YmJsVzFGdEJoa1NjUlB4cVZJSldjNk1kUFZNTU05SWhMdlIyVU1BZVZSMnlr?=
 =?utf-8?B?VGlHTDdNQWd4bHcyYWkrQVpLeW0yY0I5Mk5NdktRZEZEVGh5eFgvbWxpS2Y5?=
 =?utf-8?B?VW5TYVZpUlNmNVZySGxQODdaRlV3MlovR3p4SlNnTWZWOHpvNWR3M0FXTW1l?=
 =?utf-8?B?SGx0aTNSNjJsZkhZV0M4TG9Fd3FCNVpuTEQzdHBtVllLTzJyUDNjeEpVVzRS?=
 =?utf-8?B?TEt6TGpLTEdYL1FJQWxWUzhoMmkybm5ab1ZuTDNJTG9yTHJsNTQ3cXZGa291?=
 =?utf-8?B?UlBtWUJ2TXBOKzc0T2tycTBZK0lpanBXMTJ4RHhaYnM3d0w5bStISk5ZUFVR?=
 =?utf-8?B?ZnhsYUpQWXhiR1lRdE1JM0MzNGVDY2loc1Z2eHdPKzNIMEZHK0trV2FtQW9T?=
 =?utf-8?B?QnBTT0lKcWdnalJtQUx2dzB4elR6NFVlSWF5SDJPYzZyTnFld1Z4NmFTaHcr?=
 =?utf-8?B?aXY4NVlwcFJyQjZsM1cvSVNZYmNmWkd5a0J4ZEZWeXNRanJab096R3I5VUZh?=
 =?utf-8?B?MWczNEVybVJsNUFxbkdZNVUyeGlxL1p1ZG03NEdVZ1AvdVF3WlFvUllNNGNE?=
 =?utf-8?B?NzVUdHhQaEQ2S1Y5eU12NTlhR3FBWUlTOS8wTlE2ZFcxRE8rUGVidHZCV2FZ?=
 =?utf-8?B?cmFRWUJ1YlhNNENIbW90WEU1TlloL0VGaUd6UzdGOGNnc2xzZTArSHZCZk8w?=
 =?utf-8?B?UFpYRUxScnZjeWorSE5GYWRBazZXVEsvZGZpUmU0NlE3SVRKOWdSTGU3aE5F?=
 =?utf-8?B?eVNVYUErZGhkUGRJdGhLaHRwOEpXek9URUQ2SVJpUlVPclZxMnlud3BhOTdV?=
 =?utf-8?Q?4NNoMpYqfxU6dlSvyjupNOuXY?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8afb5c9b-4d9c-47cf-9648-08dbd6f16608
X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8308.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2023 13:34:09.3331 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 2uMZC1pC6U/RmPAqse8uyZFktU9pzXuISjYWM34ccc7EmNXbjcT7Y4rbHfkN7wZx
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4324
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org



On 10/26/2023 8:50 AM, Shubham Rohila wrote:
> Define rte_cdx_vfio_bm_enable and rte_cdx_vfio_bm_disable to
> enable or disable bus master functionality for cdx devices.
> 
> Signed-off-by: Shubham Rohila <shubham.rohila@amd.com>
> ---
>   drivers/bus/cdx/bus_cdx_driver.h | 25 ++++++++
>   drivers/bus/cdx/cdx_vfio.c       | 97 ++++++++++++++++++++++++++++++++
>   drivers/bus/cdx/version.map      |  2 +
>   lib/eal/include/rte_vfio.h       | 27 +++++++++
>   4 files changed, 151 insertions(+)
> 
> diff --git a/drivers/bus/cdx/bus_cdx_driver.h b/drivers/bus/cdx/bus_cdx_driver.h
> index 1c9a64c87a..2f94a6c3af 100644
> --- a/drivers/bus/cdx/bus_cdx_driver.h
> +++ b/drivers/bus/cdx/bus_cdx_driver.h
> @@ -16,6 +16,7 @@ extern "C" {
>   
>   #include <stdlib.h>
>   #include <inttypes.h>
> +#include <linux/types.h>
>   
>   #include <bus_driver.h>
>   #include <dev_driver.h>
> @@ -178,6 +179,30 @@ int rte_cdx_vfio_intr_enable(const struct rte_intr_handle *intr_handle);
>   __rte_internal
>   int rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle);
>   
> +/**
> + * Enable Bus Mastering for CDX bus devices.
> + *
> + * @param dev
> + *   Pointer to the cdx device.
> + *
> + *  @return
> + *  0 on success, -1 on error.

alignment should be similar to @param dev

> + */
> +__rte_internal
> +int rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev);
> +
> +/**
> + * Disable Bus Mastering for CDX bus devices.
> + *
> + * @param dev
> + *   Pointer to the cdx device.
> + *
> + *  @return
> + *  0 on success, -1 on error.

here too

> + */
> +__rte_internal
> +int rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev);
> +
>   /**
>    * Unregister a CDX driver.
>    *
> diff --git a/drivers/bus/cdx/cdx_vfio.c b/drivers/bus/cdx/cdx_vfio.c
> index 8a3ac0b995..a0abf71483 100644
> --- a/drivers/bus/cdx/cdx_vfio.c
> +++ b/drivers/bus/cdx/cdx_vfio.c
> @@ -256,6 +256,16 @@ cdx_vfio_setup_device(struct rte_cdx_device *dev, int vfio_dev_fd,
>   		return -1;
>   	}
>   
> +	/*
> +	 * Enable Bus mastering for the device. errno is set as ENOTTY if
> +	 * device does not support configuring bus master.
> +	 */
> +	if (rte_cdx_vfio_bm_enable(dev) && errno != -ENOTTY) {

errno != -ENOTTY can be in brackets for better readability.

As we are probing the functionality in the 
rte_cdx_vfio_bm_enable/disable API, so why do we need a check for errno?

> +		CDX_BUS_ERR("Bus master enable failure! Error: %d (%s)", errno,
> +			strerror(errno));
> +		return -1;
> +	}

Disable in cdx_vfio_unmap_resource_primary?

> +
>   	return 0;
>   }
>   
> @@ -596,3 +606,90 @@ rte_cdx_vfio_intr_disable(const struct rte_intr_handle *intr_handle)
>   
>   	return ret;
>   }
> +
> +/* enable Bus Mastering */
> +int
> +rte_cdx_vfio_bm_enable(struct rte_cdx_device *dev)
> +{
> +	struct vfio_device_info device_info = { .argsz = sizeof(device_info) };
> +	struct vfio_device_feature_bus_master *vfio_bm_feature;
> +	struct vfio_device_feature *feature;
> +	int vfio_dev_fd, ret = 0;
> +	size_t argsz = 0;

no need to initialize argsz and ret here.

> +
> +	vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
> +	if (vfio_dev_fd < 0)
> +		return -1;
> +
> +	argsz = sizeof(struct vfio_device_feature) + sizeof(struct vfio_device_feature_bus_master);
> +
> +	feature = (struct vfio_device_feature *)malloc(argsz);
> +	if (!feature)
> +		return -ENOMEM;
> +
> +	vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data;
> +
> +	feature->argsz = argsz;
> +
> +	feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE;
> +	feature->flags |= VFIO_DEVICE_FEATURE_SET;
> +	ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature);
> +	if (ret) {
> +		CDX_BUS_ERR("Bus Master configuring not supported for device: %s, error: %d (%s)\n",
> +			dev->name, errno, strerror(errno));
> +		free(feature);
> +		return ret;
> +	}
> +
> +	feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET;
> +	vfio_bm_feature->op = VFIO_DEVICE_FEATURE_SET_MASTER;
> +	ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature);
> +	if (ret < 0)
> +		CDX_BUS_ERR("BM Enable Error for device: %s, Error: %d (%s)\n",
> +			dev->name, errno, strerror(errno));
> +
> +	free(feature);
> +	return ret;
> +}
> +
> +/* Disable Bus Mastering */
> +int
> +rte_cdx_vfio_bm_disable(struct rte_cdx_device *dev)
> +{
> +	struct vfio_device_feature_bus_master *vfio_bm_feature;
> +	struct vfio_device_feature *feature;
> +	int vfio_dev_fd, ret = 0;
> +	size_t argsz = 0;

no need to initialize argsz and ret here.

> +
> +	vfio_dev_fd = rte_intr_dev_fd_get(dev->intr_handle);
> +	if (vfio_dev_fd < 0)
> +		return -1;
> +
> +	feature = (struct vfio_device_feature *)malloc(argsz);

need to assign argsz before?

> +	if (!feature)
> +		return -ENOMEM;
> +
> +	vfio_bm_feature = (struct vfio_device_feature_bus_master *) feature->data;
> +
> +	feature->argsz = argsz;
> +
> +	feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_PROBE;
> +	feature->flags |= VFIO_DEVICE_FEATURE_SET;
> +	ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature);
> +	if (ret) {
> +		CDX_BUS_ERR("Bus Master configuring not supported for device: %s, Error: %d (%s)\n",
> +			dev->name, errno, strerror(errno));
> +		free(feature);
> +		return ret;
> +	}
> +
> +	feature->flags = RTE_VFIO_DEVICE_FEATURE_BUS_MASTER | VFIO_DEVICE_FEATURE_SET;
> +	vfio_bm_feature->op = VFIO_DEVICE_FEATURE_CLEAR_MASTER;
> +	ret = ioctl(vfio_dev_fd, RTE_VFIO_DEVICE_FEATURE, feature);
> +	if (ret < 0)
> +		CDX_BUS_ERR("BM Disable Error for device: %s, Error: %d (%s)\n",
> +			dev->name, errno, strerror(errno));
> +
> +	free(feature);
> +	return ret;
> +}
> diff --git a/drivers/bus/cdx/version.map b/drivers/bus/cdx/version.map
> index 0a15d39ae8..5438f76029 100644
> --- a/drivers/bus/cdx/version.map
> +++ b/drivers/bus/cdx/version.map
> @@ -7,6 +7,8 @@ INTERNAL {
>   	rte_cdx_unregister;
>   	rte_cdx_vfio_intr_disable;
>   	rte_cdx_vfio_intr_enable;
> +	rte_cdx_vfio_bm_enable;
> +	rte_cdx_vfio_bm_disable;

Alphabetical order.

>   
>   	local: *;
>   };
> diff --git a/lib/eal/include/rte_vfio.h b/lib/eal/include/rte_vfio.h
> index 3487c4f2a2..1c791b3636 100644
> --- a/lib/eal/include/rte_vfio.h
> +++ b/lib/eal/include/rte_vfio.h
> @@ -73,6 +73,33 @@ struct vfio_info_cap_header {
>   #define RTE_VFIO_CAP_MSIX_MAPPABLE 3
>   #endif
>   
> +/* VFIO_DEVICE_FEATURE is defined for kernel version 5.7 and newer. */
> +#ifdef VFIO_DEVICE_FEATURE
> +#define RTE_VFIO_DEVICE_FEATURE VFIO_DEVICE_FEATURE
> +#else
> +#define RTE_VFIO_DEVICE_FEATURE             _IO(VFIO_TYPE, VFIO_BASE + 17)

use tabs.

> +struct vfio_device_feature {
> +	__u32   argsz;
> +	__u32   flags;
> +#define VFIO_DEVICE_FEATURE_MASK        (0xffff) /* 16-bit feature index */
> +#define VFIO_DEVICE_FEATURE_GET         (1 << 16) /* Get feature into data[] */
> +#define VFIO_DEVICE_FEATURE_SET         (1 << 17) /* Set feature from data[] */
> +#define VFIO_DEVICE_FEATURE_PROBE       (1 << 18) /* Probe feature support */
> +	__u8    data[];
> +};
> +#endif
> +
> +#ifdef VFIO_DEVICE_FEATURE_BUS_MASTER
> +#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER VFIO_DEVICE_FEATURE_BUS_MASTER
> +#else
> +#define RTE_VFIO_DEVICE_FEATURE_BUS_MASTER  10
> +struct vfio_device_feature_bus_master {
> +	__u32 op;
> +#define  VFIO_DEVICE_FEATURE_CLEAR_MASTER       0       /* Clear Bus Master */
> +#define  VFIO_DEVICE_FEATURE_SET_MASTER         1       /* Set Bus Master */

and tabs in all above here.

Regards,
Nipun