From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 1B3E242A90;
	Mon,  8 May 2023 04:13:29 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id A896D40A84;
	Mon,  8 May 2023 04:13:28 +0200 (CEST)
Received: from mga14.intel.com (mga14.intel.com [192.55.52.115])
 by mails.dpdk.org (Postfix) with ESMTP id 3512D4014F;
 Mon,  8 May 2023 04:13:26 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;
 d=intel.com; i=@intel.com; q=dns/txt; s=Intel;
 t=1683512007; x=1715048007;
 h=from:to:cc:subject:date:message-id:references:
 in-reply-to:content-transfer-encoding:mime-version;
 bh=BDt/mO2rad5pq0m4L2RBobAuXTJobBGLq1fc8wQDF10=;
 b=RXSTS/l29cwsRjuNH9uwllMyv9AcFSKDfwbu5SSLde11AIRyGjOowX1R
 kFcUIpD6MEGPFl7uCnL1bxhAv9dxdxXSC2S10OHsatpshYzVRU/qZz8H2
 sGc7vJDGVOsqwmZz93D6cgKrMdMIoNMk8X99isuc+BA9mxgwHgihYPpCr
 iOojzz3oSj+T+Oc9Zyx2LNjfpf1ImW7r3fqc9nAE9lST199iZzlQ+3mp6
 E1DlXXccqs3QOrkDSOtq5qi0/uLYFOqCcESODGmaL3wUgUGdsDDoHNkpD
 EVv6EF18m4iAXn/hQQHJ6ORNa9m/zP+njcNdxHYwEbnR++4+YGLr388Tc w==;
X-IronPort-AV: E=McAfee;i="6600,9927,10703"; a="349568983"
X-IronPort-AV: E=Sophos;i="5.99,258,1677571200"; d="scan'208";a="349568983"
Received: from orsmga001.jf.intel.com ([10.7.209.18])
 by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384;
 07 May 2023 19:13:25 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=McAfee;i="6600,9927,10703"; a="731119878"
X-IronPort-AV: E=Sophos;i="5.99,258,1677571200"; d="scan'208";a="731119878"
Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16])
 by orsmga001.jf.intel.com with ESMTP; 07 May 2023 19:13:25 -0700
Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by
 ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server
 (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
 15.1.2507.23; Sun, 7 May 2023 19:13:25 -0700
Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by
 ORSMSX610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server
 (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
 15.1.2507.23; Sun, 7 May 2023 19:13:25 -0700
Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by
 orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server
 (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
 15.1.2507.23 via Frontend Transport; Sun, 7 May 2023 19:13:25 -0700
Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.102)
 by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server
 (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.1.2507.23; Sun, 7 May 2023 19:13:24 -0700
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=TxvZmhft91WGp8y079KUVSHf/i01BL9Ef8ID34yEnTYGW0oVVCWOWXMkDHTPtiuBp6hMozv/C/EPtdjfCVtv2II9CqLxIOZjb6vePKw6hb8vFJNCqnqP+82a/rN3Te4nlrkCIR87gAhDLztQRUZrqk8mq4OCXr/TVurQiV0sXLqCUGWM05A7ScJnvQVmEtPwsuCtOKt2aJob1F5YKg1gVdFuJQiEEhwWFnUW786/eDcIxkAIiCClf31fjJSLIlKuw4w/x18eL3wFTwuQwmZFoSj6uPnysM+nRi+xdpSPS5IfASjgz7FRfDMnKsJWKtKe2ud1oWdvu+pubWR3oXgneQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;
 bh=kIE6FzjZwLGFlFD8x6nFC+V4k8Xcg+MTK0uU1y0ZmXg=;
 b=JslGqy4WWJQnWmdP2pX3c7PyE+6yAgwN3VNIfy0HHAalBsM08SqDe9lBSIYajFz+2cq2cJaYVU3+ImQQRz12Pv2cI57rz+XqU/x2X9Hubqphll7eKRuNdwaGFH48E7eTulz/3rA/OwhXc03FETc+Av+G197YXDO1O+REO3i/MIKgThXrvJzBsJawPh86GgdvW8ECk2yVCIAIaL/IAM0SPVjrZGLyE13HjR/DSmf2SnU1/J6Glqimn2EcO1CBq1eWPH+jr3yaWrEQBnqrINhS5gVu+kO3kHdewI+SL2a1eCmgmzdAA0B6H4ABsEIzUzLXi/I2RFq67FecSVCYDGfBtA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
 smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com;
 dkim=pass header.d=intel.com; arc=none
Received: from SN6PR11MB3504.namprd11.prod.outlook.com (2603:10b6:805:d0::17)
 by MW4PR11MB6692.namprd11.prod.outlook.com (2603:10b6:303:20e::7)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.31; Mon, 8 May
 2023 02:13:22 +0000
Received: from SN6PR11MB3504.namprd11.prod.outlook.com
 ([fe80::5797:849:7a10:560d]) by SN6PR11MB3504.namprd11.prod.outlook.com
 ([fe80::5797:849:7a10:560d%6]) with mapi id 15.20.6363.031; Mon, 8 May 2023
 02:13:21 +0000
From: "Xia, Chenbo" <chenbo.xia@intel.com>
To: "dev@dpdk.org" <dev@dpdk.org>
CC: "skori@marvell.com" <skori@marvell.com>, "techboard@dpdk.org"
 <techboard@dpdk.org>, "thomas@monjalon.net" <thomas@monjalon.net>,
 "Richardson, Bruce" <bruce.richardson@intel.com>, "ferruh.yigit@amd.com"
 <ferruh.yigit@amd.com>, "david.marchand@redhat.com"
 <david.marchand@redhat.com>, "Cao, Yahui" <yahui.cao@intel.com>, "Li, Miao"
 <miao.li@intel.com>
Subject: RE: [RFC 0/4] Support VFIO sparse mmap in PCI bus
Thread-Topic: [RFC 0/4] Support VFIO sparse mmap in PCI bus
Thread-Index: AQHZcbmsi0+H/0NwHUa5BU1IpGPXgK9PwEgg
Date: Mon, 8 May 2023 02:13:21 +0000
Message-ID: <SN6PR11MB3504CBB8C1B0B24D3216C8149C719@SN6PR11MB3504.namprd11.prod.outlook.com>
References: <20230418053012.10667-1-chenbo.xia@intel.com>
In-Reply-To: <20230418053012.10667-1-chenbo.xia@intel.com>
Accept-Language: en-US, zh-CN
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
authentication-results: dkim=none (message not signed)
 header.d=none;dmarc=none action=none header.from=intel.com;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: SN6PR11MB3504:EE_|MW4PR11MB6692:EE_
x-ms-office365-filtering-correlation-id: 21783605-1a0a-4009-bc54-08db4f69cbd7
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: 1dRZG77vsUw1xk0d8XghKW6bt4l9vKrORB/+Wco68ZfxhoBSR0VGXKtVdXWx8a7LL+XiDSIXAOgMi/qZXLjEQTTx5uDiNswTMRhTDXgzG6C8/n+OxYDFYRGYkrdu4ODqX5iTHCImIzYaVBlDGp3VtV0gIHXzJLphhbTAPswzFWLR0KM9n8o1VPXWF7SF+mE5DS/cPD0I+vK+QfnBOagl72yIl3c1xzhwEWDvIdNuy7knynMmt0xbWnXBhMjup3taw7hVpPyEc4Vo6deiHqBUTBZb9wZckVcwr20p+pDV1Z+yYvpoEHMwbIe8eFDxYloRYdvVlJBR2cRhF2Lat+Rpdr2sMiPj0Clln7B0jwXuIXH0g5yEPExh/Y4xlUba1SU8C7ndTJPjpcinJBbjufaXvkDqzZc3tpZ6iNqM3vVSCxsp0//uM6vj20jS/2L51i5seWu14IzMUvT5e78oQbxRFcEmPN33UNSqKkvprluXk0Dp5XQCL4HXn9NgmWCd1EodAEkPdJb4YH3zXZXuHr7VGbhXYRlUymnTTxwGRNiTHFOpRFhYsuddfp3/hBSkS6H3+8uBY9cvgK9hMx9nRe13TwFJHDRAxzLAz7ilVrVVPxIio6umICMzC/XUvMTQi8fT
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
 IPV:NLI; SFV:NSPM; H:SN6PR11MB3504.namprd11.prod.outlook.com; PTR:; CAT:NONE;
 SFS:(13230028)(396003)(366004)(376002)(39860400002)(346002)(136003)(451199021)(76116006)(66946007)(66556008)(66476007)(66446008)(64756008)(6916009)(4326008)(478600001)(7696005)(316002)(54906003)(86362001)(33656002)(83380400001)(107886003)(53546011)(9686003)(6506007)(26005)(71200400001)(8676002)(8936002)(5660300002)(52536014)(41300700001)(2906002)(55016003)(82960400001)(38070700005)(38100700002)(122000001)(186003);
 DIR:OUT; SFP:1102; 
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?BU2IKvDcxlxDTMqDupXGeufuMx2S1iLKd51Amx8avBrXwkZoGFM34pF+u6Zb?=
 =?us-ascii?Q?w6xvzOL6RTGC0B381347TL2SeQiu7sEE7jL2jlYrH12UXm+3gaKuMVEkuCvI?=
 =?us-ascii?Q?OWWUAfq/Q+XKGX7/ZY/wOwTQMctlUn1iLeM3a0eSmxKbafAVp150ipv0MVna?=
 =?us-ascii?Q?TVXX4z+9Z3XyzpdwuzoauHZKpeqTZtb05OkDyOblnqBPxjnIsj1YnyWXqKEQ?=
 =?us-ascii?Q?pwHcvAe9DXucgiNfW9WSeOq7HIwANC6jJIcR9rO8LK7KCqH2P/w9xrZjlsmF?=
 =?us-ascii?Q?dSiXgBl6uIkJ2/bRE0F36zkVCOW2nRbrnCynl70kF45HW1nMnPJLLfJCUETd?=
 =?us-ascii?Q?3D7wqAaNhTOAXoJIe2DLNvBsVl3297xHgND2Ra4mJ9Mzq/pMjCzP92oslacx?=
 =?us-ascii?Q?8lDqyBTLlSeUbK39kQkHRMAhx7ewQmtEVHVH5ENwrhCwHcWRPUZMNCf5PGhh?=
 =?us-ascii?Q?MGrk2d0FaqSlKJR0zQNRO+ASmMI4NRW5UBuLa42LHCJWbQPOxOn9naTyYctG?=
 =?us-ascii?Q?55zpjKGMChHDOQgZ0+TR6G+GG7czUwwE8u/A57EZSywXPvq/PNUoUjX0iS/X?=
 =?us-ascii?Q?T2wa9HFUzWTeYcjucIXx3bbtIOzTICBzrGZ/kgchcCiYE1wtPzvd3UFiw3WA?=
 =?us-ascii?Q?0Oe1AnWdXKWNulvlcqghLZcMMi60IdZalTa1Jh+v1ZRW2+BuH4Aw8E7QvuMZ?=
 =?us-ascii?Q?S2n1w0gJT+dhkthv1+X1HWWJFIV11+/Bfmyg3BimF9bEo+GUgNIISorIO6vj?=
 =?us-ascii?Q?gFMRFVc8RECJfGa1eU+089vWWyxiqSMAAlhfsojm13C8An35AXMzE4o4a+TO?=
 =?us-ascii?Q?8o9fyQNrS+dLs4fJo1s9aWwc51kJ0AaJPdTK4h4fU4sXYhYI6FCTm6/yepqB?=
 =?us-ascii?Q?I9ZvBZGETxIRhRIBz5pSy3CprkudrJ7b1ZAJIbb3XGPdDl4j72+i1xTSPKV2?=
 =?us-ascii?Q?APGkbQkXQCVOy3bTQ66GvMv3toCiGq13EUg3IvCt6ybIbvntU4J6x8CvV+FJ?=
 =?us-ascii?Q?ABzCC64loQamCjLaINF6l6jWAuTlMe6oFnKDi/hWDwBIKVE5YYl9VibECTsa?=
 =?us-ascii?Q?/np2LBLUMLFZZh9KgSWO/rkazx0GCG77SjDZ4IMf04Sj3D3XFhgwv0ukSaxt?=
 =?us-ascii?Q?6zJiFMiSppo7j3sbjwNSwbBm6w4oSb40HP7hwBym8b4ZOYIU8ToDoryveBGD?=
 =?us-ascii?Q?GNgG4biK5W7qYxO06Nwu/KEKcUlSQOv71eJlF5tGWQNfeDIICw6J1ROY+W0k?=
 =?us-ascii?Q?89WVlfsKT7XvovuBiBZJDAe/BGKbP/BaRgSG5CLeNSmEA8M8ehjlz01fr9h7?=
 =?us-ascii?Q?LMLHR6G/4KNwHWQyhF44Ewn6+bbRkPmgVr2du3Ukhraecz5QIIe75kKr0nwj?=
 =?us-ascii?Q?t/4gDH3Rvmw8Moo8zCNPMGah0zJ4rzWfDKmUOeTHQ1zbUvhPankV62oeRoOt?=
 =?us-ascii?Q?GEisSb1vIiEm6enRBp3yRwyIzPRDz39CTxb5b1mrBuNFlGzFh0MZ+fL1uweX?=
 =?us-ascii?Q?1JEQPSS7FBTPHZN/rT/y5PpcekX3f4lUehtHjeY4gbjw4KHI//YzH4WKyB5j?=
 =?us-ascii?Q?kSkCBxOLKsa+VFTaQO8lUnecrj5rRq/5wMSWoVOJ?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3504.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 21783605-1a0a-4009-bc54-08db4f69cbd7
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 May 2023 02:13:21.2777 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: h6lJfTxqPodBJFWPTBfFtBYxafrNzZCo4CLsfFrqjm5gf/iFqqYGetyDirL9yVqrTVUWqZtF2ajcL4APGLtpog==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6692
X-OriginatorOrg: intel.com
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org

Gentle ping for some comments..

After rethink, personally I may choose option 2 as it will have no driver A=
PI change for
PMDs and as I look at current code, that's how we do when MSI-X table can't=
 be mmap-ed.

Thanks,
Chenbo

> -----Original Message-----
> From: Chenbo Xia <chenbo.xia@intel.com>
> Sent: Tuesday, April 18, 2023 1:30 PM
> To: dev@dpdk.org
> Cc: skori@marvell.com
> Subject: [RFC 0/4] Support VFIO sparse mmap in PCI bus
>=20
> This series introduces a VFIO standard capability, called sparse
> mmap to PCI bus. In linux kernel, it's defined as
> VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of
> mmap whole BAR region into DPDK process, only mmap part of the
> BAR region after getting sparse mmap information from kernel.
> For the rest of BAR region that is not mmap-ed, DPDK process
> can use pread/pwrite system calls to access. Sparse mmap is
> useful when kernel does not want userspace to mmap whole BAR
> region, or kernel wants to control over access to specific BAR
> region. Vendors can choose to enable this feature or not for
> their devices in their specific kernel modules.
>=20
> In this patchset:
>=20
> Patch 1-3 is mainly for introducing BAR access APIs so that
> driver could use them to access specific BAR using pread/pwrite
> system calls when part of the BAR is not mmap-able.
>=20
> Patch 4 adds the VFIO sparse mmap support finally. A question
> is for all sparse mmap regions, should they be mapped to a
> continuous virtual address region that follows device-specific
> BAR layout or not. In theory, there could be three options to
> support this feature.
>=20
> Option 1: Map sparse mmap regions independently
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
> In this approach, we mmap each sparse mmap region one by one
> and each region could be located anywhere in process address
> space. But accessing the mmaped BAR will not be as easy as
> 'bar_base_address + bar_offset', driver needs to check the
> sparse mmap information to access specific BAR register.
>=20
> Patch 4 in this patchset adopts this option. Driver API change
> is introduced in bus_pci_driver.h. Corresponding changes in
> all drivers are also done and currently I am assuming drivers
> do not support this feature so they will not check the
> 'is_sparse' flag but assumes it to be false. Note that it will
> not break any driver and each vendor can add related logic when
> they start to support this feature. This is only because I don't
> want to introduce complexity to drivers that do not want to
> support this feature.
>=20
> Option 2: Map sparse mmap regions based on device-specific BAR layout
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> In this approach, the sparse mmap regions are mapped to continuous
> virtual address region that follows device-specific BAR layout.
> For example, the BAR size is 0x4000 and only 0-0x1000 (sparse mmap
> region #1) and 0x3000-0x4000 (sparse mmap region #2) could be
> mmaped. Region #1 will be mapped at 'base_addr' and region #2
> will be mapped at 'base_addr + 0x3000'. The good thing is if
> we implement like this, driver can still access all BAR registers
> using 'bar_base_address + bar_offset' way and we don't need
> to introduce any driver API change. But the address space
> range 'base_addr + 0x1000' to 'base_addr + 0x3000' may need to
> be reserved so it could result in waste of address space or memory
> (when we use MAP_ANONYMOUS and MAP_PRIVATE flag to reserve this
> range). Meanwhile, driver needs to know which part of BAR is
> mmaped (this is possible since the range is defined by vendor's
> specific kernel module).
>=20
> Option 3: Support both option 1 & 2
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> We could define a driver flag to let driver choose which way it
> perfers since either option has its own Pros & Cons.
>=20
> Please share your comments, Thanks!
>=20
>=20
> Chenbo Xia (4):
>   bus/pci: introduce an internal representation of PCI device
>   bus/pci: avoid depending on private value in kernel source
>   bus/pci: introduce helper for MMIO read and write
>   bus/pci: add VFIO sparse mmap support
>=20
>  drivers/baseband/acc/rte_acc100_pmd.c         |   6 +-
>  drivers/baseband/acc/rte_vrb_pmd.c            |   6 +-
>  .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c         |   6 +-
>  drivers/baseband/fpga_lte_fec/fpga_lte_fec.c  |   6 +-
>  drivers/bus/pci/bsd/pci.c                     |  43 +-
>  drivers/bus/pci/bus_pci_driver.h              |  24 +-
>  drivers/bus/pci/linux/pci.c                   |  91 +++-
>  drivers/bus/pci/linux/pci_init.h              |  14 +-
>  drivers/bus/pci/linux/pci_uio.c               |  34 +-
>  drivers/bus/pci/linux/pci_vfio.c              | 445 ++++++++++++++----
>  drivers/bus/pci/pci_common.c                  |  57 ++-
>  drivers/bus/pci/pci_common_uio.c              |  12 +-
>  drivers/bus/pci/private.h                     |  25 +-
>  drivers/bus/pci/rte_bus_pci.h                 |  48 ++
>  drivers/bus/pci/version.map                   |   3 +
>  drivers/common/cnxk/roc_dev.c                 |   4 +-
>  drivers/common/cnxk/roc_dpi.c                 |   2 +-
>  drivers/common/cnxk/roc_ml.c                  |  22 +-
>  drivers/common/qat/dev/qat_dev_gen1.c         |   2 +-
>  drivers/common/qat/dev/qat_dev_gen4.c         |   4 +-
>  drivers/common/sfc_efx/sfc_efx.c              |   2 +-
>  drivers/compress/octeontx/otx_zip.c           |   4 +-
>  drivers/crypto/ccp/ccp_dev.c                  |   4 +-
>  drivers/crypto/cnxk/cnxk_cryptodev_ops.c      |   2 +-
>  drivers/crypto/nitrox/nitrox_device.c         |   4 +-
>  drivers/crypto/octeontx/otx_cryptodev_ops.c   |   6 +-
>  drivers/crypto/virtio/virtio_pci.c            |   6 +-
>  drivers/dma/cnxk/cnxk_dmadev.c                |   2 +-
>  drivers/dma/hisilicon/hisi_dmadev.c           |   6 +-
>  drivers/dma/idxd/idxd_pci.c                   |   4 +-
>  drivers/dma/ioat/ioat_dmadev.c                |   2 +-
>  drivers/event/dlb2/pf/dlb2_main.c             |  16 +-
>  drivers/event/octeontx/ssovf_probe.c          |  38 +-
>  drivers/event/octeontx/timvf_probe.c          |  18 +-
>  drivers/event/skeleton/skeleton_eventdev.c    |   2 +-
>  drivers/mempool/octeontx/octeontx_fpavf.c     |   6 +-
>  drivers/net/ark/ark_ethdev.c                  |   4 +-
>  drivers/net/atlantic/atl_ethdev.c             |   2 +-
>  drivers/net/avp/avp_ethdev.c                  |  20 +-
>  drivers/net/axgbe/axgbe_ethdev.c              |   4 +-
>  drivers/net/bnx2x/bnx2x_ethdev.c              |   6 +-
>  drivers/net/bnxt/bnxt_ethdev.c                |   8 +-
>  drivers/net/cpfl/cpfl_ethdev.c                |   4 +-
>  drivers/net/cxgbe/cxgbe_ethdev.c              |   2 +-
>  drivers/net/cxgbe/cxgbe_main.c                |   2 +-
>  drivers/net/cxgbe/cxgbevf_ethdev.c            |   2 +-
>  drivers/net/cxgbe/cxgbevf_main.c              |   2 +-
>  drivers/net/e1000/em_ethdev.c                 |   4 +-
>  drivers/net/e1000/igb_ethdev.c                |   4 +-
>  drivers/net/ena/ena_ethdev.c                  |   4 +-
>  drivers/net/enetc/enetc_ethdev.c              |   2 +-
>  drivers/net/enic/enic_main.c                  |   4 +-
>  drivers/net/fm10k/fm10k_ethdev.c              |   2 +-
>  drivers/net/gve/gve_ethdev.c                  |   4 +-
>  drivers/net/hinic/base/hinic_pmd_hwif.c       |  14 +-
>  drivers/net/hns3/hns3_ethdev.c                |   2 +-
>  drivers/net/hns3/hns3_ethdev_vf.c             |   2 +-
>  drivers/net/hns3/hns3_rxtx.c                  |   4 +-
>  drivers/net/i40e/i40e_ethdev.c                |   2 +-
>  drivers/net/iavf/iavf_ethdev.c                |   2 +-
>  drivers/net/ice/ice_dcf.c                     |   2 +-
>  drivers/net/ice/ice_ethdev.c                  |   2 +-
>  drivers/net/idpf/idpf_ethdev.c                |   4 +-
>  drivers/net/igc/igc_ethdev.c                  |   2 +-
>  drivers/net/ionic/ionic_dev_pci.c             |   2 +-
>  drivers/net/ixgbe/ixgbe_ethdev.c              |   4 +-
>  drivers/net/liquidio/lio_ethdev.c             |   4 +-
>  drivers/net/nfp/nfp_ethdev.c                  |   2 +-
>  drivers/net/nfp/nfp_ethdev_vf.c               |   6 +-
>  drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c    |   4 +-
>  drivers/net/ngbe/ngbe_ethdev.c                |   2 +-
>  drivers/net/octeon_ep/otx_ep_ethdev.c         |   2 +-
>  drivers/net/octeontx/base/octeontx_pkivf.c    |   6 +-
>  drivers/net/octeontx/base/octeontx_pkovf.c    |  12 +-
>  drivers/net/qede/qede_main.c                  |   6 +-
>  drivers/net/sfc/sfc.c                         |   2 +-
>  drivers/net/thunderx/nicvf_ethdev.c           |   2 +-
>  drivers/net/txgbe/txgbe_ethdev.c              |   2 +-
>  drivers/net/txgbe/txgbe_ethdev_vf.c           |   2 +-
>  drivers/net/virtio/virtio_pci.c               |   6 +-
>  drivers/net/vmxnet3/vmxnet3_ethdev.c          |   4 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy.c             |  10 +-
>  drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c         |   6 +-
>  drivers/raw/ifpga/afu_pmd_n3000.c             |   4 +-
>  drivers/raw/ifpga/ifpga_rawdev.c              |   6 +-
>  drivers/raw/ntb/ntb_hw_intel.c                |   8 +-
>  drivers/vdpa/ifc/ifcvf_vdpa.c                 |   6 +-
>  drivers/vdpa/sfc/sfc_vdpa_hw.c                |   2 +-
>  drivers/vdpa/sfc/sfc_vdpa_ops.c               |   2 +-
>  lib/eal/include/rte_vfio.h                    |   1 -
>  90 files changed, 853 insertions(+), 352 deletions(-)
>=20
> --
> 2.17.1