RE: [PATCH] vfio: do not coalesce DMA mappings

["Ding, Xuan" <xuan.ding@intel.com>]

Hi Nipun,

> -----Original Message-----
> From: Gupta, Nipun <nipun.gupta@amd.com>
> Sent: Tuesday, July 4, 2023 5:23 PM
> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org; thomas@monjalon.net;
> Burakov, Anatoly <anatoly.burakov@intel.com>; Yigit, Ferruh
> <Ferruh.Yigit@amd.com>; David Marchand <david.marchand@redhat.com>
> Cc: Agarwal, Nikhil <nikhil.agarwal@amd.com>; He, Xingguang
> <xingguang.he@intel.com>; Ling, WeiX <weix.ling@intel.com>
> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings
> 
> Hi Xuan,
> 
> On 7/4/2023 1:36 PM, Ding, Xuan wrote:
> > Hi Nipun,
> >
> >> -----Original Message-----
> >> From: Gupta, Nipun <Nipun.Gupta@amd.com>
> >> Sent: Tuesday, July 4, 2023 2:54 PM
> >> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org;
> >> thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> >> Yigit, Ferruh <Ferruh.Yigit@amd.com>; David Marchand
> >> <david.marchand@redhat.com>
> >> Cc: Agarwal, Nikhil <nikhil.agarwal@amd.com>; He, Xingguang
> >> <xingguang.he@intel.com>; Ling, WeiX <weix.ling@intel.com>
> >> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
> >>
> >> Hi Xuan,
> >>
> >> Please see inline.
> >>
> >>> -----Original Message-----
> >>> From: Ding, Xuan <xuan.ding@intel.com>
> >>> Sent: Tuesday, July 4, 2023 10:43 AM
> >>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> >>> thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> >>> Yigit, Ferruh <Ferruh.Yigit@amd.com>
> >>> Cc: Agarwal, Nikhil <nikhil.agarwal@amd.com>; He, Xingguang
> >>> <xingguang.he@intel.com>; Ling, WeiX <weix.ling@intel.com>
> >>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
> >>>
> >>> Hi Nipun,
> >>>
> >>>> -----Original Message-----
> >>>> From: Ding, Xuan
> >>>> Sent: Friday, June 30, 2023 1:58 PM
> >>>> To: Nipun Gupta <nipun.gupta@amd.com>; dev@dpdk.org;
> >>>> thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> >>>> ferruh.yigit@amd.com
> >>>> Cc: nikhil.agarwal@amd.com; He, Xingguang <xingguang.he@intel.com>;
> >>>> Ling, WeiX <weix.ling@intel.com>
> >>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
> >>>>
> >>>> Hi Nipun,
> >>>>
> >>>> Replies are inline.
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Nipun Gupta <nipun.gupta@amd.com>
> >>>>> Sent: Friday, June 30, 2023 9:46 AM
> >>>>> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org;
> >>>>> thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> >>>>> ferruh.yigit@amd.com
> >>>>> Cc: nikhil.agarwal@amd.com; He, Xingguang
> >>>>> <xingguang.he@intel.com>; Ling, WeiX <weix.ling@intel.com>
> >>>>> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings
> >>>>>
> >>>>> Hi Xuan,
> >>>>>
> >>>>> Thanks for pointing out the issue and figuring out the patch which
> >>>>> introduced this. If you have answers to below queries, please let me
> know:
> >>>>>
> >>>>> Is there any other test cases which tests "--no-huge" which pass?
> >>>>
> >>>> Yes, there are test cases adding "--no-huge" option to validate 4k
> >>>> page size in async vhost.
> >>>> Actually, the page size is decided by front-end, so I think this
> >>>> case can be removed.
> >>>>
> >>>> Previously, testpmd can start with "--no-huge" options (not sure if
> >>>> there are test cases).
> >>>> Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i
> >>>>
> >>>>>
> >>>>> Also, if we change the "-m" option to provide lower memory, does
> >>>>> the test pass?
> >>>>
> >>>> "-m" option is also added and does not work.
> >>>>
> >>>>>
> >>>>> When you mention too many pages exceed the capability of IOMMU,
> >>>>> you are referring to HW capability to create multiple pages? Here
> >>>>> it seems in case of 4K page size we need 256K pages which is
> >>>>> limiting the
> >> capacity?
> >>>>
> >>>> Yes, this is the result of my initial debugging.
> >>>> The direct impact is that this kind of testpmd cases cannot start now.
> >>>> If this is expected, I think we can close this defect and ignore
> >>>> the "--no-
> >> huge"
> >>>> option when start.
> >>>
> >>> Any insights? Should we just ignore the "--no-huge" option and close
> >>> this
> >> defect?
> >>> Now we did this as a workaround. Seems no one uses the "--no-huge"
> >>> option in testpmd now.
> >>
> >> VFIO supports dma_entry_limit as a module parameter, which has a
> >> default value of U16_MAX i.e. 64K, most likely which is limiting
> >> creation of 256K entries for 4K pages here. This can be modified while
> inserting vfio module:
> >>          modprobe vfio_iommu_type1 dma_entry_limit=1000000
> >
> > Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work.
> > The reason I think is vfio-pci is build-in in kernel driver (since 20.04) and it
> does not support dynamic insmod/rmmod.
> >
> > Does this command need to rmmod vfio first and then modprobe again?
> >
> 
> If it is inserted as a module then you can remove using rmmod and then
> modprobe again with the dma_entry_limit parameter. Also note,
> vfio_iommu_type1 is the module which is limiting the entries to 64K, so this
> module needs to be inserted again providing the dma_entry_limit module
> param.
> 
> In case the module is built-in you can provide via kernel command line
> parameter (ref:
> https://www.kernel.org/doc/html/v4.12/admin-guide/kernel-
> parameters.html).
> As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000"
> should be used in the bootargs to set the module parameters.
> 
> FYI.. DPDK documentation also mentions the limitation at:
> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html

Thanks for pointing out these references.

Add supplement for configuring build-in vfio module:
Except adding " vfio_iommu_type1.dma_entry_limit=1000000" in bootargs,
we can use kernel command line: "echo 1000000 > /sys/module/vfio_iommu_type1/parameters/dma_entry_limit".

Both methods works, thanks again.

Regards,
Xuan

> 
> Thanks,
> Nipun