From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3BEA842DAA; Tue, 4 Jul 2023 16:09:14 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AC73440F18; Tue, 4 Jul 2023 16:09:13 +0200 (CEST) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) by mails.dpdk.org (Postfix) with ESMTP id 2E0BC40E03 for ; Tue, 4 Jul 2023 16:09:12 +0200 (CEST) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 11A6F5C0370; Tue, 4 Jul 2023 10:09:09 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 04 Jul 2023 10:09:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1688479749; x=1688566149; bh=dfxV87vrGpgJUb1joUFIRG+wlWs4u1qAe3M 4KQ+gtUA=; b=Qde+F77MEo6cWqu2lm3Qy3wIZWkkF1g3FHSqPdtac3jWi2J/8so kVio7PtoEN47YI6XUb0LwwdvhFBloCKZI6uLsLbrM9bjNsAkACEbqCSUk1lfl1Hr 8/tf1UCpJGTr8ozBSSfNkXYTugERtNJOivYxMKUQIF+9GknbCopeLv9Uvne6Ergk 02D1V9uptCe0xPJs4JKdVrPXkbr1HnSFtfJ5b/o5hoo/P7U2hAwAtm7SYDuDiRko DadNJzNGL8nBHrF0yArCCnqLCMqbO7fLZ5KonqxkYhfwtC/9Fa1GJg5dWTJKid7Y qrQ+5vr7PtM7kqxmRXP4lVZzbZykIGcpFyQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1688479749; x=1688566149; bh=dfxV87vrGpgJUb1joUFIRG+wlWs4u1qAe3M 4KQ+gtUA=; b=Oe6bnxBDRTZNue4JHiooSiX2ZevGCByQpI0+7SxHYhn4U4iWI+R m+rKNPTXTHU5MzDgMwev2OtF2CWDTTUlJ0k1Hhoehfa9Bsz0KvPtd5Ml+p3YKHme UPL8qcnM0mGK64ft33LsJEpjDVGkIyh5cAkm9/Cu/aSeCXCD8/z8GqE0SrTZrhbV 2RHftw7VcQtVird6NxKxNrBRx7ZPY+WWRUVTvijoB5vODqIzu1V5X27dIzUQXF7m 5MVCSeBOYTtk6hw5+VXMAPFwvIf7JdxMlRdovnhueP0JqDpbTAbVc6u5sqVq4y8Y +M+I2TxGFDoLosfC2M6e738DPTbYPrnHPpQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeggdejfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkfgjfhgggfgtsehtufertddttddvnecuhfhrohhmpefvhhhomhgr shcuofhonhhjrghlohhnuceothhhohhmrghssehmohhnjhgrlhhonhdrnhgvtheqnecugg ftrfgrthhtvghrnhepgfdtiedvudejgfevteekheffueejudelffekhffhleduleegteet iefgjeejfeevnecuffhomhgrihhnpehkvghrnhgvlhdrohhrghdpughpughkrdhorhhgne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhm rghssehmohhnjhgrlhhonhdrnhgvth X-ME-Proxy: Feedback-ID: i47234305:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 4 Jul 2023 10:09:07 -0400 (EDT) From: Thomas Monjalon To: "Ding, Xuan" , "Burakov, Anatoly" , "Gupta, Nipun" Cc: "dev@dpdk.org" , "Yigit, Ferruh" , David Marchand , "Agarwal, Nikhil" , "He, Xingguang" , "Ling, WeiX" Subject: Re: [PATCH] vfio: do not coalesce DMA mappings Date: Tue, 04 Jul 2023 16:09:06 +0200 Message-ID: <1780903.TLkxdtWsSY@thomas> In-Reply-To: References: <20221230095853.1323616-1-nipun.gupta@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org 04/07/2023 11:23, Gupta, Nipun: > On 7/4/2023 1:36 PM, Ding, Xuan wrote: >> From: Gupta, Nipun >>> From: Ding, Xuan >>>> From: Ding, Xuan >>>>> From: Nipun Gupta > >>>>> Hi Xuan, > >>>>> > >>>>> Thanks for pointing out the issue and figuring out the patch which > >>>>> introduced this. If you have answers to below queries, please let me know: > >>>>> > >>>>> Is there any other test cases which tests "--no-huge" which pass? > >>>> > >>>> Yes, there are test cases adding "--no-huge" option to validate 4k > >>>> page size in async vhost. > >>>> Actually, the page size is decided by front-end, so I think this > >>>> case can be removed. > >>>> > >>>> Previously, testpmd can start with "--no-huge" options (not sure if > >>>> there are test cases). > >>>> Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i > >>>> > >>>>> > >>>>> Also, if we change the "-m" option to provide lower memory, does > >>>>> the test pass? > >>>> > >>>> "-m" option is also added and does not work. > >>>> > >>>>> > >>>>> When you mention too many pages exceed the capability of IOMMU, > >>>>> you are referring to HW capability to create multiple pages? Here > >>>>> it seems in case of 4K page size we need 256K pages which is limiting the > >> capacity? > >>>> > >>>> Yes, this is the result of my initial debugging. > >>>> The direct impact is that this kind of testpmd cases cannot start now. > >>>> If this is expected, I think we can close this defect and ignore the "--no- > >> huge" > >>>> option when start. > >>> > >>> Any insights? Should we just ignore the "--no-huge" option and close this > >> defect? > >>> Now we did this as a workaround. Seems no one uses the "--no-huge" > >>> option in testpmd now. > >> > >> VFIO supports dma_entry_limit as a module parameter, which has a default > >> value of U16_MAX i.e. 64K, most likely which is limiting creation of 256K > >> entries for 4K pages here. This can be modified while inserting vfio module: > >> modprobe vfio_iommu_type1 dma_entry_limit=1000000 > > > > Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work. > > The reason I think is vfio-pci is build-in in kernel driver (since 20.04) and it does not support dynamic insmod/rmmod. > > > > Does this command need to rmmod vfio first and then modprobe again? > > > > If it is inserted as a module then you can remove using rmmod and then > modprobe again with the dma_entry_limit parameter. Also note, > vfio_iommu_type1 is the module which is limiting the entries to 64K, so > this module needs to be inserted again providing the dma_entry_limit > module param. > > In case the module is built-in you can provide via kernel command line > parameter (ref: > https://www.kernel.org/doc/html/v4.12/admin-guide/kernel-parameters.html). > As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000" > should be used in the bootargs to set the module parameters. > > FYI.. DPDK documentation also mentions the limitation at: > https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html Yes the parameter is discussed in https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio-memory-mapping-limits but it does not mention we may need to decrease it with --no-huge. Please could you add this to the documentation?