From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3EFD142DD2; Tue, 4 Jul 2023 18:42:37 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0A45D40E03; Tue, 4 Jul 2023 18:42:37 +0200 (CEST) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2066.outbound.protection.outlook.com [40.107.243.66]) by mails.dpdk.org (Postfix) with ESMTP id 2BC7440A4B for ; Tue, 4 Jul 2023 18:42:35 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WXbaBqPM0F+9KlSlqo0HcmHC5ODabM38PVOPJy+Cb2VINSleJAmzGs4G5aOyS8YXYVabKts884tlPVdJA2I/wCqQLjfd2vQx4mX2jqkvm0FwnGmJXbKsi8GRykkRFHXrj6CD2oxXM2QRYLfNuTCVN6w1lHUbp4YPtLqgMWoCj5Je4LrIxmQhgLn0fMxvAVJBwstgErGZx4nHoqFn27MITw7c7jqKEACUHMfBsTBQReHaNxOfU7/DnVqcDQBqRleVGjEX5tM21IEjmZ739SgVjvDxRMZrf016SaTnzjPbMSo5xyvzAxpbRYnUJM73XJbnKAX1Ac6fzioVVFdkwDxNrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bSav5K12a8WgyAIBe0U2NfnFhPmNWWHTulfVJ68nlh4=; b=CMW0eQc5vfO+k/cvstp7mMxCkkbxX7W7ii8MUpR2iVz6h1kPBKiVHSTVxHxLStv89c/ATqIdtqacLdy6LwPvj9LP95CydtQNE+e2dEX18nRggUH+wKKpEKaDzGLMQtn4w1eOwON85w20UPQ3XIMRG348t5koPIL51z6H4drG2Xaqmfqy5A1QWCsFZzdTdDLMK2YtoLsrbJ9LTcgjIgKnoD790gfth4Qi7cks0ffpA6FLHD8wbxJNSxBlDCxIVYotLQ95j/z67vyl8O6n8CJtUVgtYXBV9T1eGIKA6vAVD4myovV1TTum2WpHWRsxuuE8Zb0HDBWP2lJrl3lNOdYdEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bSav5K12a8WgyAIBe0U2NfnFhPmNWWHTulfVJ68nlh4=; b=H6FTXEru0JM8jkWQBl8nbP0mMdi6nCkuFzRSG2KzLB8oPtH4cksIAxN0p8gjPJQhNZZ3Q0xYuRk5zf6kb2BPvJEw/xzjQGVcIzgoFPXZ85zD6wHhjOfhEOMPrx2fLnby2z1pXYyd20uEAQOLD7BEbvekeww0/fFfC2zovjtIgE4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) by CY8PR12MB7684.namprd12.prod.outlook.com (2603:10b6:930:87::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Tue, 4 Jul 2023 16:42:33 +0000 Received: from CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::e452:1914:1054:de79]) by CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::e452:1914:1054:de79%7]) with mapi id 15.20.6565.016; Tue, 4 Jul 2023 16:42:33 +0000 Message-ID: Date: Tue, 4 Jul 2023 22:12:20 +0530 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH] vfio: do not coalesce DMA mappings Content-Language: en-US To: "Ding, Xuan" , "dev@dpdk.org" , "thomas@monjalon.net" , "Burakov, Anatoly" , "Yigit, Ferruh" , David Marchand Cc: "Agarwal, Nikhil" , "He, Xingguang" , "Ling, WeiX" References: <20221230095853.1323616-1-nipun.gupta@amd.com> From: "Gupta, Nipun" In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: PN2PR01CA0074.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c01:23::19) To CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8308:EE_|CY8PR12MB7684:EE_ X-MS-Office365-Filtering-Correlation-Id: 58cea19a-cf3c-45a3-2f44-08db7cadaa02 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YJ0YGGbBF2bfgauIw/n0as0fOxYucrDH9UR9QslhLIJn64SjTH7yFXKquSWFsEDjmKvx+Rji7/Wchk66ea34kya0tyDqkrYmEZcetnwwh4kYJ/0kUAq275NngyKypVl8gdLHDM9Y6EelTvPSxnMyY2SYNGG6IvNcH1FV6fzgyT8tiJbMr8xomzELIbkX80TtNj/lZxraESa7ChAmHhXPij9JNVWS8aCBMFzilgsJdPS3bd0sXmF4ImY+dZ3WF8JY6xJfLp3wk2c6k81zVem6RrAI/wsAgvB0B7jSKWpMLyFShUfQIsXRhCg+Jpm3gUHr9lAk/P7bofz+THjStg+TjvtpUcwffs7C+X0hP0w0GVuJ/xWjQIy/jug4v9Cbr/T1V8xkuWmr1R2xX2Bb+csYs9e0gxQ8Wkwj7FPeyS0I2BgOI/6PPL77147v3YO/zW3N50ffUkXFWwB1kdya5vzwZoDHwnAxBRH5wd4y/Jv72cy4UxSa1C2naBG2USPdi9Z9tUoASuNbRH5sMAMZtUl7xfOnL3M704pRbfkWUJ7IlKH1f0glFcJHzyo/Mt8+PrOebBfF+SJjSFgY3Mf4tfcCGN8vPzGMhD13RFnvhLwLQSM/IvrSpt011dFtrEwyLopgN9Vj39sX4y16hLOijR7aKZGpareGsgmmwqeQZwYGA4IPSLXqrwcOKlpMx7cUj8Ne X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR12MB8308.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(366004)(396003)(346002)(376002)(136003)(39860400002)(451199021)(6506007)(53546011)(6512007)(316002)(4326008)(66946007)(66476007)(66556008)(38100700002)(2616005)(83380400001)(186003)(26005)(478600001)(8676002)(110136005)(54906003)(8936002)(31686004)(31696002)(2906002)(6486002)(5660300002)(966005)(86362001)(41300700001)(6666004)(36756003)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SVo3S2NBTXRnNnp1RDd2aHVnMjRpZzI5ZjNBQjJKVFIxQlc2RlBuUm94TTdB?= =?utf-8?B?YnFOYUNzVXBTb1g4Wk5mVmNnMTlsMUppU0V5THJGUjJBMmcxbUNXYnJWeEoy?= =?utf-8?B?dGxTMWhQL1JKUDkzdWFSdWVac0hwblAvUHVObWR1MGdWbGEycG12OVlYdkI0?= =?utf-8?B?SVB6cHBzejRnRVIxUkVXbDNrNk5UYXdhM29DZW1aOXlRL1ZMYTFjeGFCZzVV?= =?utf-8?B?L0hNM2RFb01CVUViRzZHcTN4L1NoRUFiOGxJVmJQVGpRakhGMnkwNmthZGdK?= =?utf-8?B?RTFhWnRMdExWTm91UnZQZDYwWFhUejRBSHdZcEdPZGRWTmhHTWRlOGNWbk14?= =?utf-8?B?eTdYZzhXL2RFYjJUR2Q3RkxrVjVVY0VjVWtGbmtiUzBBeDJjQWRMcGdTdGR2?= =?utf-8?B?bTN4NkIvVFRDbmtXQzg3ZStJV1NZYUlNYldYbzdxT0wrNWJmQnJTVElEcWVz?= =?utf-8?B?ZDRmcDJCbmVFMy9LSXBKMTNNL2tDU0RmL1hCWkY4UXlKM3JWWWRKRE1pWWdh?= =?utf-8?B?T1VGZjVBTVhUTjg1LzNFcFo0dEdsenBWS1B6b0lyZVF1L011WTZvN3RwQ1I1?= =?utf-8?B?OFZKbmwyQnRTbnVHdGpmOXZzN056dWlpcFAyV1RpWGE5cmhsYlBWanJzZFpr?= =?utf-8?B?MnBmSkswY2NaZlhsMU1LTDJVSFExVkR2Um9TcXJ6RlJwNFIydEdJeWJiUzhW?= =?utf-8?B?eXI0NUlPSjBEZTNmSVlSc1hES0pJRUN4WjIrTWdTR2twa01uM1pwMlNkODJI?= =?utf-8?B?SUk5R0RLTVZJS0xmTVVnUHIrZzZUMUo4NWNGTGlTd2t1aUlIQnI1ZCtNajVu?= =?utf-8?B?WmFWeHV0WGRzTU5Rc3R6TzVaWGxrMmM2Q2VWblpGZk5QL1FESXNPYVJkRVVT?= =?utf-8?B?ckNzK1dXUm8rYklhTGR6ZGhqcWEwdDJaYXQ3aThtRXpoTDRXSW56TExEVWdp?= =?utf-8?B?VllzeHFBNWtTMU9YUXBjaFZVNFowcWNEVmdDZWdrWmlsNUM3cXdkQ2VmNzJ5?= =?utf-8?B?ajEyakJnbW9iczJJS3Z1QVhLaXJ4ZStySU1DY3BGN1RjWUkwNlNQVUJ6Y1ZO?= =?utf-8?B?U1JNbmwrSVppR2tIeU9NREFtVk9TcTNJeSt3akhoKzAvc0ZPNmxYRHcxQ2cz?= =?utf-8?B?ajBvOFg1dmdoVU5iSllTM1dxVXpnVkN1b2FMS09qOGJtY1NGZXdwc3dFQzA2?= =?utf-8?B?VEx1L2ExSlpEaDlpOEJsOWJuV1k5WXpXV29KS0lncTk3OFYvOEpBbHU4VlZQ?= =?utf-8?B?Z0ZTcGhwS0NGeWFBenF1VXJHRW8rTTM3MzVsYWVHbVBwUjZlTlJ5cWl3L0pm?= =?utf-8?B?Y1Q0d2VvcU5ZZlhaaVI2bGJia05iUFVYUHZjRUxZemQ0QTY0U3FPQitoeEh5?= =?utf-8?B?RUFuQTdpTm10QTlXODF0Q2VoMnErTlBiUEhvbmlFQldENW5NOU12QytNOW9J?= =?utf-8?B?Z3JjaEtHZVlQd1R6MWovUkJiVEE5azNYNkR0QjhkU29JaWg4ekl6eFhPNXIz?= =?utf-8?B?aTlEZlpiaG9wUzB2ZjVUOEE5aXFPYlN1TlVuaVhOWitwR0RaeTB4WVF2Ynl1?= =?utf-8?B?ekdtemt5Q3EwT0gyREdYOWF1VVIvcWhPcUw5WVZabytSNEYyS24rTmd2bGo4?= =?utf-8?B?TWpuVGVYeWJBSXdNNzZnWGdYdnRDOENOOTJ3TmI4SnpvWWJPeHgvNSs5Z0Z0?= =?utf-8?B?c3VPckJwTlQzdU1TbDd3TW9odEtCRnluYnN0NjhDOHJrbE9iVmwvZGhKb1J0?= =?utf-8?B?MElaaU5rQTRoTGx5WDRmNzJpQ1ZuS2VzRXBCV0pHTytDU1RJdUNmamZKWUtm?= =?utf-8?B?bm4xbXpYZGYyTGhBdllLUlhiWTMrelorV1BZY095b0FVOVNscWJIV0tQNk5D?= =?utf-8?B?NVlSMUgzaGVVZHF2aGcvVWkvNnE3QmRzMGc2QWlqR25YMmE5eUF2eDgxSG9T?= =?utf-8?B?U0c4STI4SDl5TjAxdjYrVEQ5S1dsZGdTemFybWpaVVRDTzlnUHh2NUtVYkY3?= =?utf-8?B?UUVkNDc2RWoyRFNqQ1hWWERSaG56RnpKRndscWxsMys0K3ZHMk54QUFmbmNt?= =?utf-8?B?R0t0d1k2eml3TlQwcnRaSWZLOGViOGt1WTdxU0tRR1dwQ0VHZ0JKVkNod3NJ?= =?utf-8?Q?Wm7yjbac1ociv/ZEwLODrukhA?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58cea19a-cf3c-45a3-2f44-08db7cadaa02 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8308.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 16:42:32.9973 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CslMk+ECett9gnGcB4MDbb1Tui3vRpzuj4LUYMgf8ka9JS0exqYxYDMax8N0jKEM X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7684 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 7/4/2023 8:41 PM, Ding, Xuan wrote: > Hi Nipun, > >> -----Original Message----- >> From: Gupta, Nipun >> Sent: Tuesday, July 4, 2023 5:23 PM >> To: Ding, Xuan ; dev@dpdk.org; thomas@monjalon.net; >> Burakov, Anatoly ; Yigit, Ferruh >> ; David Marchand >> Cc: Agarwal, Nikhil ; He, Xingguang >> ; Ling, WeiX >> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings >> >> Hi Xuan, >> >> On 7/4/2023 1:36 PM, Ding, Xuan wrote: >>> Hi Nipun, >>> >>>> -----Original Message----- >>>> From: Gupta, Nipun >>>> Sent: Tuesday, July 4, 2023 2:54 PM >>>> To: Ding, Xuan ; dev@dpdk.org; >>>> thomas@monjalon.net; Burakov, Anatoly ; >>>> Yigit, Ferruh ; David Marchand >>>> >>>> Cc: Agarwal, Nikhil ; He, Xingguang >>>> ; Ling, WeiX >>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings >>>> >>>> Hi Xuan, >>>> >>>> Please see inline. >>>> >>>>> -----Original Message----- >>>>> From: Ding, Xuan >>>>> Sent: Tuesday, July 4, 2023 10:43 AM >>>>> To: Gupta, Nipun ; dev@dpdk.org; >>>>> thomas@monjalon.net; Burakov, Anatoly ; >>>>> Yigit, Ferruh >>>>> Cc: Agarwal, Nikhil ; He, Xingguang >>>>> ; Ling, WeiX >>>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings >>>>> >>>>> Hi Nipun, >>>>> >>>>>> -----Original Message----- >>>>>> From: Ding, Xuan >>>>>> Sent: Friday, June 30, 2023 1:58 PM >>>>>> To: Nipun Gupta ; dev@dpdk.org; >>>>>> thomas@monjalon.net; Burakov, Anatoly ; >>>>>> ferruh.yigit@amd.com >>>>>> Cc: nikhil.agarwal@amd.com; He, Xingguang ; >>>>>> Ling, WeiX >>>>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings >>>>>> >>>>>> Hi Nipun, >>>>>> >>>>>> Replies are inline. >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Nipun Gupta >>>>>>> Sent: Friday, June 30, 2023 9:46 AM >>>>>>> To: Ding, Xuan ; dev@dpdk.org; >>>>>>> thomas@monjalon.net; Burakov, Anatoly ; >>>>>>> ferruh.yigit@amd.com >>>>>>> Cc: nikhil.agarwal@amd.com; He, Xingguang >>>>>>> ; Ling, WeiX >>>>>>> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings >>>>>>> >>>>>>> Hi Xuan, >>>>>>> >>>>>>> Thanks for pointing out the issue and figuring out the patch which >>>>>>> introduced this. If you have answers to below queries, please let me >> know: >>>>>>> >>>>>>> Is there any other test cases which tests "--no-huge" which pass? >>>>>> >>>>>> Yes, there are test cases adding "--no-huge" option to validate 4k >>>>>> page size in async vhost. >>>>>> Actually, the page size is decided by front-end, so I think this >>>>>> case can be removed. >>>>>> >>>>>> Previously, testpmd can start with "--no-huge" options (not sure if >>>>>> there are test cases). >>>>>> Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i >>>>>> >>>>>>> >>>>>>> Also, if we change the "-m" option to provide lower memory, does >>>>>>> the test pass? >>>>>> >>>>>> "-m" option is also added and does not work. >>>>>> >>>>>>> >>>>>>> When you mention too many pages exceed the capability of IOMMU, >>>>>>> you are referring to HW capability to create multiple pages? Here >>>>>>> it seems in case of 4K page size we need 256K pages which is >>>>>>> limiting the >>>> capacity? >>>>>> >>>>>> Yes, this is the result of my initial debugging. >>>>>> The direct impact is that this kind of testpmd cases cannot start now. >>>>>> If this is expected, I think we can close this defect and ignore >>>>>> the "--no- >>>> huge" >>>>>> option when start. >>>>> >>>>> Any insights? Should we just ignore the "--no-huge" option and close >>>>> this >>>> defect? >>>>> Now we did this as a workaround. Seems no one uses the "--no-huge" >>>>> option in testpmd now. >>>> >>>> VFIO supports dma_entry_limit as a module parameter, which has a >>>> default value of U16_MAX i.e. 64K, most likely which is limiting >>>> creation of 256K entries for 4K pages here. This can be modified while >> inserting vfio module: >>>> modprobe vfio_iommu_type1 dma_entry_limit=1000000 >>> >>> Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work. >>> The reason I think is vfio-pci is build-in in kernel driver (since 20.04) and it >> does not support dynamic insmod/rmmod. >>> >>> Does this command need to rmmod vfio first and then modprobe again? >>> >> >> If it is inserted as a module then you can remove using rmmod and then >> modprobe again with the dma_entry_limit parameter. Also note, >> vfio_iommu_type1 is the module which is limiting the entries to 64K, so this >> module needs to be inserted again providing the dma_entry_limit module >> param. >> >> In case the module is built-in you can provide via kernel command line >> parameter (ref: >> https://www.kernel.org/doc/html/v4.12/admin-guide/kernel- >> parameters.html). >> As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000" >> should be used in the bootargs to set the module parameters. >> >> FYI.. DPDK documentation also mentions the limitation at: >> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html > > Thanks for pointing out these references. > > Add supplement for configuring build-in vfio module: > Except adding " vfio_iommu_type1.dma_entry_limit=1000000" in bootargs, > we can use kernel command line: "echo 1000000 > /sys/module/vfio_iommu_type1/parameters/dma_entry_limit". > > Both methods works, thanks again. Good to know this works! > > Regards, > Xuan > >> >> Thanks, >> Nipun