From: "Ding, Xuan" <xuan.ding@intel.com>
To: "Gupta, Nipun" <nipun.gupta@amd.com>,
"dev@dpdk.org" <dev@dpdk.org>,
"thomas@monjalon.net" <thomas@monjalon.net>,
"Burakov, Anatoly" <anatoly.burakov@intel.com>,
"Yigit, Ferruh" <Ferruh.Yigit@amd.com>,
"David Marchand" <david.marchand@redhat.com>
Cc: "Agarwal, Nikhil" <nikhil.agarwal@amd.com>,
"He, Xingguang" <xingguang.he@intel.com>,
"Ling, WeiX" <weix.ling@intel.com>
Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
Date: Tue, 4 Jul 2023 15:11:10 +0000 [thread overview]
Message-ID: <BN9PR11MB55132453CD05A1542DE829AEE72EA@BN9PR11MB5513.namprd11.prod.outlook.com> (raw)
In-Reply-To: <f602bb08-60a4-102a-b56f-e38a80f8460e@amd.com>
Hi Nipun,
> -----Original Message-----
> From: Gupta, Nipun <nipun.gupta@amd.com>
> Sent: Tuesday, July 4, 2023 5:23 PM
> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org; thomas@monjalon.net;
> Burakov, Anatoly <anatoly.burakov@intel.com>; Yigit, Ferruh
> <Ferruh.Yigit@amd.com>; David Marchand <david.marchand@redhat.com>
> Cc: Agarwal, Nikhil <nikhil.agarwal@amd.com>; He, Xingguang
> <xingguang.he@intel.com>; Ling, WeiX <weix.ling@intel.com>
> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings
>
> Hi Xuan,
>
> On 7/4/2023 1:36 PM, Ding, Xuan wrote:
> > Hi Nipun,
> >
> >> -----Original Message-----
> >> From: Gupta, Nipun <Nipun.Gupta@amd.com>
> >> Sent: Tuesday, July 4, 2023 2:54 PM
> >> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org;
> >> thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> >> Yigit, Ferruh <Ferruh.Yigit@amd.com>; David Marchand
> >> <david.marchand@redhat.com>
> >> Cc: Agarwal, Nikhil <nikhil.agarwal@amd.com>; He, Xingguang
> >> <xingguang.he@intel.com>; Ling, WeiX <weix.ling@intel.com>
> >> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
> >>
> >> Hi Xuan,
> >>
> >> Please see inline.
> >>
> >>> -----Original Message-----
> >>> From: Ding, Xuan <xuan.ding@intel.com>
> >>> Sent: Tuesday, July 4, 2023 10:43 AM
> >>> To: Gupta, Nipun <Nipun.Gupta@amd.com>; dev@dpdk.org;
> >>> thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> >>> Yigit, Ferruh <Ferruh.Yigit@amd.com>
> >>> Cc: Agarwal, Nikhil <nikhil.agarwal@amd.com>; He, Xingguang
> >>> <xingguang.he@intel.com>; Ling, WeiX <weix.ling@intel.com>
> >>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
> >>>
> >>> Hi Nipun,
> >>>
> >>>> -----Original Message-----
> >>>> From: Ding, Xuan
> >>>> Sent: Friday, June 30, 2023 1:58 PM
> >>>> To: Nipun Gupta <nipun.gupta@amd.com>; dev@dpdk.org;
> >>>> thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> >>>> ferruh.yigit@amd.com
> >>>> Cc: nikhil.agarwal@amd.com; He, Xingguang <xingguang.he@intel.com>;
> >>>> Ling, WeiX <weix.ling@intel.com>
> >>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
> >>>>
> >>>> Hi Nipun,
> >>>>
> >>>> Replies are inline.
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Nipun Gupta <nipun.gupta@amd.com>
> >>>>> Sent: Friday, June 30, 2023 9:46 AM
> >>>>> To: Ding, Xuan <xuan.ding@intel.com>; dev@dpdk.org;
> >>>>> thomas@monjalon.net; Burakov, Anatoly <anatoly.burakov@intel.com>;
> >>>>> ferruh.yigit@amd.com
> >>>>> Cc: nikhil.agarwal@amd.com; He, Xingguang
> >>>>> <xingguang.he@intel.com>; Ling, WeiX <weix.ling@intel.com>
> >>>>> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings
> >>>>>
> >>>>> Hi Xuan,
> >>>>>
> >>>>> Thanks for pointing out the issue and figuring out the patch which
> >>>>> introduced this. If you have answers to below queries, please let me
> know:
> >>>>>
> >>>>> Is there any other test cases which tests "--no-huge" which pass?
> >>>>
> >>>> Yes, there are test cases adding "--no-huge" option to validate 4k
> >>>> page size in async vhost.
> >>>> Actually, the page size is decided by front-end, so I think this
> >>>> case can be removed.
> >>>>
> >>>> Previously, testpmd can start with "--no-huge" options (not sure if
> >>>> there are test cases).
> >>>> Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i
> >>>>
> >>>>>
> >>>>> Also, if we change the "-m" option to provide lower memory, does
> >>>>> the test pass?
> >>>>
> >>>> "-m" option is also added and does not work.
> >>>>
> >>>>>
> >>>>> When you mention too many pages exceed the capability of IOMMU,
> >>>>> you are referring to HW capability to create multiple pages? Here
> >>>>> it seems in case of 4K page size we need 256K pages which is
> >>>>> limiting the
> >> capacity?
> >>>>
> >>>> Yes, this is the result of my initial debugging.
> >>>> The direct impact is that this kind of testpmd cases cannot start now.
> >>>> If this is expected, I think we can close this defect and ignore
> >>>> the "--no-
> >> huge"
> >>>> option when start.
> >>>
> >>> Any insights? Should we just ignore the "--no-huge" option and close
> >>> this
> >> defect?
> >>> Now we did this as a workaround. Seems no one uses the "--no-huge"
> >>> option in testpmd now.
> >>
> >> VFIO supports dma_entry_limit as a module parameter, which has a
> >> default value of U16_MAX i.e. 64K, most likely which is limiting
> >> creation of 256K entries for 4K pages here. This can be modified while
> inserting vfio module:
> >> modprobe vfio_iommu_type1 dma_entry_limit=1000000
> >
> > Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work.
> > The reason I think is vfio-pci is build-in in kernel driver (since 20.04) and it
> does not support dynamic insmod/rmmod.
> >
> > Does this command need to rmmod vfio first and then modprobe again?
> >
>
> If it is inserted as a module then you can remove using rmmod and then
> modprobe again with the dma_entry_limit parameter. Also note,
> vfio_iommu_type1 is the module which is limiting the entries to 64K, so this
> module needs to be inserted again providing the dma_entry_limit module
> param.
>
> In case the module is built-in you can provide via kernel command line
> parameter (ref:
> https://www.kernel.org/doc/html/v4.12/admin-guide/kernel-
> parameters.html).
> As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000"
> should be used in the bootargs to set the module parameters.
>
> FYI.. DPDK documentation also mentions the limitation at:
> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html
Thanks for pointing out these references.
Add supplement for configuring build-in vfio module:
Except adding " vfio_iommu_type1.dma_entry_limit=1000000" in bootargs,
we can use kernel command line: "echo 1000000 > /sys/module/vfio_iommu_type1/parameters/dma_entry_limit".
Both methods works, thanks again.
Regards,
Xuan
>
> Thanks,
> Nipun
next prev parent reply other threads:[~2023-07-04 15:11 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-30 9:58 Nipun Gupta
2023-01-04 5:19 ` [PATCH v2] " Nipun Gupta
2023-02-02 10:48 ` David Marchand
2023-02-07 8:56 ` Gupta, Nipun
2023-04-04 14:57 ` Gupta, Nipun
2023-04-04 15:13 ` Burakov, Anatoly
2023-04-04 16:32 ` Nipun Gupta
2023-04-05 14:16 ` Burakov, Anatoly
2023-04-05 15:06 ` Gupta, Nipun
2023-04-24 15:22 ` David Marchand
2023-04-24 16:10 ` Stephen Hemminger
2023-04-24 16:16 ` Gupta, Nipun
2023-05-10 12:58 ` Nipun Gupta
2023-05-11 14:08 ` Burakov, Anatoly
2023-04-11 15:13 ` Thomas Monjalon
2023-04-11 16:51 ` Gupta, Nipun
2023-04-07 6:13 ` Nipun Gupta
2023-04-07 7:26 ` David Marchand
2023-05-15 11:16 ` David Marchand
2023-06-29 8:21 ` [PATCH] " Ding, Xuan
2023-06-30 1:45 ` Nipun Gupta
2023-06-30 5:58 ` Ding, Xuan
2023-07-04 5:13 ` Ding, Xuan
2023-07-04 6:53 ` Gupta, Nipun
2023-07-04 8:06 ` Ding, Xuan
2023-07-04 9:23 ` Gupta, Nipun
2023-07-04 14:09 ` Thomas Monjalon
2023-07-04 16:39 ` Gupta, Nipun
2023-07-04 15:11 ` Ding, Xuan [this message]
2023-07-04 16:42 ` Gupta, Nipun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=BN9PR11MB55132453CD05A1542DE829AEE72EA@BN9PR11MB5513.namprd11.prod.outlook.com \
--to=xuan.ding@intel.com \
--cc=Ferruh.Yigit@amd.com \
--cc=anatoly.burakov@intel.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=nikhil.agarwal@amd.com \
--cc=nipun.gupta@amd.com \
--cc=thomas@monjalon.net \
--cc=weix.ling@intel.com \
--cc=xingguang.he@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).