[dpdk-dev] [RFC] vfio: only map regions VFIO supports

[dpdk-dev] [RFC] vfio: only map regions VFIO supports

[Stephen Hemminger]

The FM10K driver has 3 BARS numbered 0, 4, and 8. But the kernel
VFIO driver only allows mapping 0-5 anything bigger than that will
return -EINVAL (see kernel source vfio_pci.c:vfio_pci_mmap).

The workaround is to limit the DPDK EAL VFIO support only map
the regions that will work. The FM10K driver is not using BAR8
in current code anyway.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
index 426953a..e269e75 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
@@ -549,7 +549,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 	int iommu_group_no;
 	char pci_addr[PATH_MAX] = {0};
 	struct rte_pci_addr *loc = &dev->addr;
-	int i, ret, msix_bar;
+	int i, ret, msix_bar, nmaps;
 	struct mapped_pci_resource *vfio_res = NULL;
 	struct mapped_pci_res_list *vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
 
@@ -724,7 +724,9 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
 	/* map BARs */
 	maps = vfio_res->maps;
 
-	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
+	/* VFIO supports limited (0-5) maps */
+	nmaps = RTE_MIN(vfio_res->nb_maps, VFIO_PCI_ROM_REGION_INDEX - 1);
+	for (i = 0; i < nmaps; i++) {
 		struct vfio_region_info reg = { .argsz = sizeof(reg) };
 		void *bar_addr;
 		struct memreg {
-- 
2.1.4

Re: [dpdk-dev] [RFC] vfio: only map regions VFIO supports

[Qiu, Michael]

Hi, Stephen

This patch does not work for fm10k with vfio, see error below:

EAL: PCI device 0000:84:00.0 on NUMA socket 1
EAL:   probe driver: 8086:15a4 rte_pmd_fm10k
EAL:   PCI memory mapped at 0x7f1980000000
EAL: Trying to map BAR 2 that contains the MSI-X table. Trying offsets:
0000:0000, 1000:1000
EAL:   PCI memory mapped at 0x7f1980401000
EAL: pci_map_resource(): cannot mmap(105, 0x7f1980402000, 0x4000000,
0x0): Invalid argument (0xffffffffffffffff)
EAL:   0000:84:00.0 mapping BAR4 failed: Invalid argument
EAL: Error - exiting with code: 1
  Cause: Requested device 0000:84:00.0 cannot be used


Thanks,
Michael

On 7/9/2015 6:39 AM, Stephen Hemminger wrote:
> The FM10K driver has 3 BARS numbered 0, 4, and 8. But the kernel
> VFIO driver only allows mapping 0-5 anything bigger than that will
> return -EINVAL (see kernel source vfio_pci.c:vfio_pci_mmap).
>
> The workaround is to limit the DPDK EAL VFIO support only map
> the regions that will work. The FM10K driver is not using BAR8
> in current code anyway.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
>  lib/librte_eal/linuxapp/eal/eal_pci_vfio.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> index 426953a..e269e75 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_vfio.c
> @@ -549,7 +549,7 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
>  	int iommu_group_no;
>  	char pci_addr[PATH_MAX] = {0};
>  	struct rte_pci_addr *loc = &dev->addr;
> -	int i, ret, msix_bar;
> +	int i, ret, msix_bar, nmaps;
>  	struct mapped_pci_resource *vfio_res = NULL;
>  	struct mapped_pci_res_list *vfio_res_list = RTE_TAILQ_CAST(rte_vfio_tailq.head, mapped_pci_res_list);
>  
> @@ -724,7 +724,9 @@ pci_vfio_map_resource(struct rte_pci_device *dev)
>  	/* map BARs */
>  	maps = vfio_res->maps;
>  
> -	for (i = 0; i < (int) vfio_res->nb_maps; i++) {
> +	/* VFIO supports limited (0-5) maps */
> +	nmaps = RTE_MIN(vfio_res->nb_maps, VFIO_PCI_ROM_REGION_INDEX - 1);
> +	for (i = 0; i < nmaps; i++) {
>  		struct vfio_region_info reg = { .argsz = sizeof(reg) };
>  		void *bar_addr;
>  		struct memreg {

Re: [dpdk-dev] [RFC] vfio: only map regions VFIO supports

[Stephen Hemminger]

On Fri, 10 Jul 2015 07:54:10 +0000
"Qiu, Michael" <michael.qiu@intel.com> wrote:

> Hi, Stephen
> 
> This patch does not work for fm10k with vfio, see error below:
> 
> EAL: PCI device 0000:84:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:15a4 rte_pmd_fm10k
> EAL:   PCI memory mapped at 0x7f1980000000
> EAL: Trying to map BAR 2 that contains the MSI-X table. Trying offsets:
> 0000:0000, 1000:1000
> EAL:   PCI memory mapped at 0x7f1980401000
> EAL: pci_map_resource(): cannot mmap(105, 0x7f1980402000, 0x4000000,
> 0x0): Invalid argument (0xffffffffffffffff)
> EAL:   0000:84:00.0 mapping BAR4 failed: Invalid argument
> EAL: Error - exiting with code: 1
>   Cause: Requested device 0000:84:00.0 cannot be used

Yes. The patch doesn't solve the problem (but might be needed in some future weird hw).

Re: [dpdk-dev] [RFC] vfio: only map regions VFIO supports

[Qiu, Michael]

Hi, Stephen

I have found out the root cause of this bug, and generate a patch.

Will send out later after :)

Thanks,
Michael

On 7/11/2015 1:24 AM, Stephen Hemminger wrote:
> On Fri, 10 Jul 2015 07:54:10 +0000
> "Qiu, Michael" <michael.qiu@intel.com> wrote:
>
>> Hi, Stephen
>>
>> This patch does not work for fm10k with vfio, see error below:
>>
>> EAL: PCI device 0000:84:00.0 on NUMA socket 1
>> EAL:   probe driver: 8086:15a4 rte_pmd_fm10k
>> EAL:   PCI memory mapped at 0x7f1980000000
>> EAL: Trying to map BAR 2 that contains the MSI-X table. Trying offsets:
>> 0000:0000, 1000:1000
>> EAL:   PCI memory mapped at 0x7f1980401000
>> EAL: pci_map_resource(): cannot mmap(105, 0x7f1980402000, 0x4000000,
>> 0x0): Invalid argument (0xffffffffffffffff)
>> EAL:   0000:84:00.0 mapping BAR4 failed: Invalid argument
>> EAL: Error - exiting with code: 1
>>   Cause: Requested device 0000:84:00.0 cannot be used
> Yes. The patch doesn't solve the problem (but might be needed in some future weird hw).
>
>

Re: [dpdk-dev] [RFC] vfio: only map regions VFIO supports

[Qiu, Michael]

Hi, all

While I check the maillist,  I found a patch:

[dpdk-dev] [PATCH v3] vfio: Fix overflow while assigning vfio BAR region
offset and size

This patch has fixed this issue :), and will be merged.

This patch is same with mine.

So I will not post the patch again.

Thanks,
Michael
On 7/11/2015 1:24 AM, Stephen Hemminger wrote:
> On Fri, 10 Jul 2015 07:54:10 +0000
> "Qiu, Michael" <michael.qiu@intel.com> wrote:
>
>> Hi, Stephen
>>
>> This patch does not work for fm10k with vfio, see error below:
>>
>> EAL: PCI device 0000:84:00.0 on NUMA socket 1
>> EAL:   probe driver: 8086:15a4 rte_pmd_fm10k
>> EAL:   PCI memory mapped at 0x7f1980000000
>> EAL: Trying to map BAR 2 that contains the MSI-X table. Trying offsets:
>> 0000:0000, 1000:1000
>> EAL:   PCI memory mapped at 0x7f1980401000
>> EAL: pci_map_resource(): cannot mmap(105, 0x7f1980402000, 0x4000000,
>> 0x0): Invalid argument (0xffffffffffffffff)
>> EAL:   0000:84:00.0 mapping BAR4 failed: Invalid argument
>> EAL: Error - exiting with code: 1
>>   Cause: Requested device 0000:84:00.0 cannot be used
> Yes. The patch doesn't solve the problem (but might be needed in some future weird hw).
>
>