From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2909A42DD0; Tue, 4 Jul 2023 11:23:21 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 02EE640E03; Tue, 4 Jul 2023 11:23:21 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2073.outbound.protection.outlook.com [40.107.92.73]) by mails.dpdk.org (Postfix) with ESMTP id 7A00B40042 for ; Tue, 4 Jul 2023 11:23:19 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=adBha5dBOF21SGcP1viYlJ9lEhKGHll7+FhNqzte8qTKtny/dPlmSinrxv4nJAw4eY9ldwnZkIrG8PHB1hv8TUbcfr/qKUhjoNDdNyOlCrTXorooQFjsJnupqLtjMRKo8cY+DlN50rjONb4aRKsd1yDVS+rsNTD5RqhMjtejlu2ogH2UThvWiWLfLI5VsqR4UVgaA0uTcupGhswcIGciZpVh7PMKrLIv6Zg9r+Kw5+VRrXzSZn07ixtgQcZ5OfBwYAIwWNsrF+/I9+1sYG0K4lgUXxveOw8jGwW4y9BVEcAqIyza+aDr5B9hozEFslI+zjGqnLQ3moYx9dpf06i2mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7fCL34ekPxSRxrJU+m5CGszGWR8s6E9xIJxoeq5xUuQ=; b=bwjMmEie6+HIIGa89NaiOhlmudPTJwXBcgrpr8o8nLcYKYLOu/0RVeCSMxJyChQmPI69bC8PFM+nZ9/h75W/FYd5aas+g0v/0o9wQScy01iGL6hewuUHBD1FsRKi++o3n7YhbX8u+ulbAussGlUUu7bKAZLhpQ3fadyLM5lfAKVFDSScGrdBg7PvPdC76m6fnlpvEq9jyEhlMWPwUPzehdFrFzRrQHtgrt+Zf2etFQ/SIPRBR5MCB6DufeOTpz82LwJebwM5T5zWdb2/FgmB2cB6frqrybrXF0GPKqv02TbMGgWUiQm9/ecd8BxF6ef7QnlUAPKZL6CmvBjxB+ID6Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7fCL34ekPxSRxrJU+m5CGszGWR8s6E9xIJxoeq5xUuQ=; b=pseyqXfUNpUx9jysWvCgMY2I2vXmfXtqOebxxUuAfFdGnTPAc5YKQmbYImBZ1LWBaDwXqzIALcju2Ty7Ddz+wKXTR0AYXL7L7PBhqhRoGqJGY0qOn66HskOue6dVsuHVVy+b6Fm9KvZyJ1yL7qjrCkrgkdO2qygfSIa8AG/FcOA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from DS7PR12MB8322.namprd12.prod.outlook.com (2603:10b6:8:ed::13) by CY8PR12MB7413.namprd12.prod.outlook.com (2603:10b6:930:5f::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6565.17; Tue, 4 Jul 2023 09:23:15 +0000 Received: from DS7PR12MB8322.namprd12.prod.outlook.com ([fe80::6545:fad5:746c:fae2]) by DS7PR12MB8322.namprd12.prod.outlook.com ([fe80::6545:fad5:746c:fae2%7]) with mapi id 15.20.6544.024; Tue, 4 Jul 2023 09:23:15 +0000 Message-ID: Date: Tue, 4 Jul 2023 14:53:02 +0530 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH] vfio: do not coalesce DMA mappings Content-Language: en-US To: "Ding, Xuan" , "dev@dpdk.org" , "thomas@monjalon.net" , "Burakov, Anatoly" , "Yigit, Ferruh" , David Marchand Cc: "Agarwal, Nikhil" , "He, Xingguang" , "Ling, WeiX" References: <20221230095853.1323616-1-nipun.gupta@amd.com> From: "Gupta, Nipun" In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: PN2PR01CA0210.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c01:e9::19) To DS7PR12MB8322.namprd12.prod.outlook.com (2603:10b6:8:ed::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB8322:EE_|CY8PR12MB7413:EE_ X-MS-Office365-Filtering-Correlation-Id: 2754964e-ee17-4dd8-d9ce-08db7c704bb9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: f/5biRutbvMFUhbrOsb1v6Rx65zuatWwtcAI9M3jp06UYEXrUFNZopqjvubknrj3UGa6LgEQ7XQFynmNBgs0xoZyONuUUBMuv/+e6OJEm+3QTLNErdWQj5JFJ+G8JA4iNv7PiErnElPpqmoMDJMbdBsN4uI8pAgf1h+8akkSaBNf9OuRTDPfC1D4URwaloyw1Fp8MiizzY7pvLMTBkpgYCfQm3oDBiDmMJoVmdb/YaXo3V8nP4pQs6enkKkz1G9AsFfxSNlBbA4+C0BhZnLMXJ2xC7/pOZu1khmqQPMckTU9MgcpP86qQ+JNqFLCjCjm5jgqKE6ipspUjm3qPVj7b6bPH30PcRiMQwHNofn6JUSSB3Hir58PKCWuiVO8iPfSy8KS/yFS7n1xiA1mJML6kzI5yP5Rp3QtNxoy/hwboF/iyswMRh+MSFDrAd7iUu5Jr7k9o6J5h/6e5TEiXg3gINqd1kd0OKQT+fSPtsaYTdq+rf4GS2po3qz6w1GiTVNZzKb6mbOOJuEtwkQhkB5CT2aUcbAfFYRpPuNA0PlBQs/ZCdMonb2PZJTwIBAOhYVt6sz5FuzmlX1Kr51oeedsYoXEpkiqu7YgjK18Cr0QnNB8tRLNUQW19lO+bz3cptUL1/RXslzD3eL0L7o0K40UgUY9o3zonbtfMHRCH2rJBO7Z1gf+u0JudH2g/m1Taw4v X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS7PR12MB8322.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(39860400002)(396003)(136003)(376002)(366004)(346002)(451199021)(41300700001)(31686004)(8936002)(8676002)(110136005)(54906003)(36756003)(6486002)(316002)(5660300002)(2906002)(6666004)(4326008)(66476007)(66556008)(6512007)(966005)(478600001)(38100700002)(186003)(86362001)(31696002)(66946007)(2616005)(83380400001)(6506007)(53546011)(26005)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QnE5ZTFBRkEvOEJEcXY5eXdEdk5VVXpWeU5uL3p3eWZLMk04WGlDMEpZUE9Y?= =?utf-8?B?N29CRXR1ek92VGdERzVJOExwZkNFMkx5VkNWZThiMlFWa3pGRGVKMVBGN0xO?= =?utf-8?B?eXFIU1g1NlJnM200V2IvYWhFTWRlR2ZWRGxkbmdXUzFaYloxQ25STjUyRzZE?= =?utf-8?B?cFJSSGlxYnQvR2IyVUdPaWxMK045YTFsSU94a2FEbWtKT1hTNXRSTitha280?= =?utf-8?B?MmJSMy9xRkJSWFV6bzI3ekxDRUU1dVJiSjhRSjJOZ085bDVqQ1dxQjRxc1c0?= =?utf-8?B?RmEyV29EcXpncU0yRWgvTUhSTWF2SUgraEhEQ1lLVFgwZ253RllaWUlRNXJ6?= =?utf-8?B?ZWt3dThuL2l6U1hJYnV0OUJUdzFhMTJxSlJEbk1LWVFSQXRQSFVUTlFNaE1q?= =?utf-8?B?MVVNQmRrc1pBc2U2TDNybWxmZWU2NjVEM2pGSEdFTkRmdk9qaXlmcE8yMWk2?= =?utf-8?B?QmxrU1IrajhnaTRUb1hwMkxJZ2lnMC9KOTY5Tk9YelZjWnpCRWplaFZSbFlC?= =?utf-8?B?ZktVcytiUW1Ld2ppZ204cFVLVnZmN0thblA4dUZzSlFjbXV2dndES2kveVBF?= =?utf-8?B?S21jd0dnbzROdWozYkZGQ2VBeWswVHdFaHR1dDE3aGZjTTVZL2lrS2pZSEZ2?= =?utf-8?B?OHdrU3N2WGtVVHY4c0kyMHIwbUg1dkFRVFFDd1JlNDU1V0ZLQXhCOGRrcHVy?= =?utf-8?B?OGxrY0pDdXlCOVNLc09SRUFSWisybkJQY2lxeHp2QjVva1Z4bFdmcVJTYjBX?= =?utf-8?B?bG1lQUdtYWxxTCsrdkdTN003eERSb2RXcWYvRHdtUmFPT1ZjQm9odWVHdGNB?= =?utf-8?B?VXJ6MGR2RTM3YzRlajBCTkVUL1dmdHVnSGljMkZYYjBqRWlOVGduM3lQQmpO?= =?utf-8?B?SFZMNGV4WURCaWhJdXNwWnA3YWJpUFdnOThOdjIyeHF5UWp5ZHhhNUlxRTc3?= =?utf-8?B?b0N3TXhsWjBxTStNaUNGVEh1QjhGUnNQSGZtUFJwMEoyTFZ2b2tlb1RSbUpq?= =?utf-8?B?Mno3NThkYjZGSC9YWjBGOHZicHpIbVhnQXVEUmdJT0I5Tnp5U0dmdy92ZHd2?= =?utf-8?B?UHBXTStWdTFKQlJ2anQ1U0I4U1M5SEt2QmZIazFMMGREMG5QMS9QakFONEU5?= =?utf-8?B?eTlrMmJWc2pTMnlYVWRYMlFQcHo4aXNjYXBDeHNHMk9BMDNsUURKUUFWYldk?= =?utf-8?B?UTZkV2xSMldmN1BiNkVUdWswN3FyMTJvUEI3QXBSeU80V3RISldPd3Nqc0Rq?= =?utf-8?B?bnhMbUlnTEJ5Rks4NTJiYUFTT0lLcUYvZmlqL2F6bVAwK2hBMGRzL0U2SW81?= =?utf-8?B?WUxGTUtTNFo0VkFVSHRLci9Kem1vVTZyZW04MXhYajEwcHpKRGJhMlNxek8z?= =?utf-8?B?TmlyMEhwRDFZODNWaWdrQ2ZSN0xiMlBWQnhrZ2F4amdYUGVOWjVYM1NNcWJ5?= =?utf-8?B?M1ZDcjI4eEdtL1V0OVdMa2p2Wnd5aXBuRzJwMU1aeWZlUFkwRm9mWHpRdGdH?= =?utf-8?B?RTZIVUlNV0dZbUNvNFd1SDNMN1l3SmVHMEZSUENMa3R3dElzMHlaSUFCUTdG?= =?utf-8?B?R0NiejNRdnpMY0tIcmFXdW1NZ2t5SEpsZDZ5RXlIY2JJZUtNcmhQZTFQN3JJ?= =?utf-8?B?YWwyNStFRmtudWd4Q09LS01zNmR4bjlQMGJFdFA3cEZHd2NYbnpaSEt1RVJ2?= =?utf-8?B?NHVkZDRVZzZjUDVZeVkyZ3M4dG5ZSSs1SEFkY1hSaWZrSkl0TU5aNGVxVWs4?= =?utf-8?B?aTRSakFBMXV3TzFNdkM0bEVTS05vQVpQS2lpa2RKUHBYK0dCLzhYa3ZUTXFB?= =?utf-8?B?QU9FSEVLKzY0endlL0xxT2czdXloWVRDZnkrNWE0VWN3SkY1Y2xTMUluUmdT?= =?utf-8?B?SHhyazEzc0tRMU1RVkdmTSthckNxczFhYzBTYll0azZFUTNGcC9rekE4c3l6?= =?utf-8?B?dHBadUYvcVhqZThiRFdXNnJBaGRwYkd0QkFYNnYxUS94WXpvckwzUHZ6eXRC?= =?utf-8?B?VW5NdnpTc1owN3hjVU92R0FmaDZ5V0dJMVZBaWtwdTJUaGRxMGZSRTVvL0xC?= =?utf-8?B?akI5ME9RQ2VWUm5iVkQ0eFFSQzhyTXBLVzZtclA0K1ZvcWpUK1dvWEtCQ1RX?= =?utf-8?Q?uJg8lZ7efOtXwQw9GJICvkbor?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2754964e-ee17-4dd8-d9ce-08db7c704bb9 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB8322.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 09:23:15.4126 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MQQUqOgAMY6JXslAUjApyaBx5pvJsSWCyl9C+8ZErbszLwugjT8S8R0ewX/nLfEl X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7413 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Xuan, On 7/4/2023 1:36 PM, Ding, Xuan wrote: > Hi Nipun, > >> -----Original Message----- >> From: Gupta, Nipun >> Sent: Tuesday, July 4, 2023 2:54 PM >> To: Ding, Xuan ; dev@dpdk.org; thomas@monjalon.net; >> Burakov, Anatoly ; Yigit, Ferruh >> ; David Marchand >> Cc: Agarwal, Nikhil ; He, Xingguang >> ; Ling, WeiX >> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings >> >> Hi Xuan, >> >> Please see inline. >> >>> -----Original Message----- >>> From: Ding, Xuan >>> Sent: Tuesday, July 4, 2023 10:43 AM >>> To: Gupta, Nipun ; dev@dpdk.org; >>> thomas@monjalon.net; Burakov, Anatoly ; >>> Yigit, Ferruh >>> Cc: Agarwal, Nikhil ; He, Xingguang >>> ; Ling, WeiX >>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings >>> >>> Hi Nipun, >>> >>>> -----Original Message----- >>>> From: Ding, Xuan >>>> Sent: Friday, June 30, 2023 1:58 PM >>>> To: Nipun Gupta ; dev@dpdk.org; >>>> thomas@monjalon.net; Burakov, Anatoly ; >>>> ferruh.yigit@amd.com >>>> Cc: nikhil.agarwal@amd.com; He, Xingguang ; >>>> Ling, WeiX >>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings >>>> >>>> Hi Nipun, >>>> >>>> Replies are inline. >>>> >>>>> -----Original Message----- >>>>> From: Nipun Gupta >>>>> Sent: Friday, June 30, 2023 9:46 AM >>>>> To: Ding, Xuan ; dev@dpdk.org; >>>>> thomas@monjalon.net; Burakov, Anatoly ; >>>>> ferruh.yigit@amd.com >>>>> Cc: nikhil.agarwal@amd.com; He, Xingguang >>>>> ; Ling, WeiX >>>>> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings >>>>> >>>>> Hi Xuan, >>>>> >>>>> Thanks for pointing out the issue and figuring out the patch which >>>>> introduced this. If you have answers to below queries, please let me know: >>>>> >>>>> Is there any other test cases which tests "--no-huge" which pass? >>>> >>>> Yes, there are test cases adding "--no-huge" option to validate 4k >>>> page size in async vhost. >>>> Actually, the page size is decided by front-end, so I think this >>>> case can be removed. >>>> >>>> Previously, testpmd can start with "--no-huge" options (not sure if >>>> there are test cases). >>>> Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i >>>> >>>>> >>>>> Also, if we change the "-m" option to provide lower memory, does >>>>> the test pass? >>>> >>>> "-m" option is also added and does not work. >>>> >>>>> >>>>> When you mention too many pages exceed the capability of IOMMU, >>>>> you are referring to HW capability to create multiple pages? Here >>>>> it seems in case of 4K page size we need 256K pages which is limiting the >> capacity? >>>> >>>> Yes, this is the result of my initial debugging. >>>> The direct impact is that this kind of testpmd cases cannot start now. >>>> If this is expected, I think we can close this defect and ignore the "--no- >> huge" >>>> option when start. >>> >>> Any insights? Should we just ignore the "--no-huge" option and close this >> defect? >>> Now we did this as a workaround. Seems no one uses the "--no-huge" >>> option in testpmd now. >> >> VFIO supports dma_entry_limit as a module parameter, which has a default >> value of U16_MAX i.e. 64K, most likely which is limiting creation of 256K >> entries for 4K pages here. This can be modified while inserting vfio module: >> modprobe vfio_iommu_type1 dma_entry_limit=1000000 > > Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work. > The reason I think is vfio-pci is build-in in kernel driver (since 20.04) and it does not support dynamic insmod/rmmod. > > Does this command need to rmmod vfio first and then modprobe again? > If it is inserted as a module then you can remove using rmmod and then modprobe again with the dma_entry_limit parameter. Also note, vfio_iommu_type1 is the module which is limiting the entries to 64K, so this module needs to be inserted again providing the dma_entry_limit module param. In case the module is built-in you can provide via kernel command line parameter (ref: https://www.kernel.org/doc/html/v4.12/admin-guide/kernel-parameters.html). As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000" should be used in the bootargs to set the module parameters. FYI.. DPDK documentation also mentions the limitation at: https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html Thanks, Nipun