From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5725C42DAE; Tue, 4 Jul 2023 18:39:42 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2B96B40E03; Tue, 4 Jul 2023 18:39:42 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2052.outbound.protection.outlook.com [40.107.93.52]) by mails.dpdk.org (Postfix) with ESMTP id 1457D40042 for ; Tue, 4 Jul 2023 18:39:41 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bUf/o8NeyWZwkNdRlcIjLg4KFCBDBS4JbN7pc7bko+XINiLjc+nNR+z+zE5VGmy7urtU8XOfQU2tODijRA3oCDte37XJLXZHHERSg5EfxzRHwH2+WQp9DVv5PJMWPFveZNQjuz9D0MmwywoEkwILRUdZGltDExxmM/ORLvHA8ZnPyRUgRGeDaQyS57kL8+7qg8byKADXLr3a96ZmN5a1vsI2ZYBCZm4ng7bkTVR9OtMVsRzKZSdNfKkYBQphjvH6lw3+0ZBQUsr7lKg8ysfTp/N7HNvSnMInkEBEUAseuU4TXLVS6SFo0iyE3DkEpsKwLBJsAnikhgVhnkUduBzMeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kLEEmdrrCIVYzFHloNkPAQbhuVK2fqfV4tfYrkFPeek=; b=ew2Xr0oqSxxBIxFrqop7O5xrDWAqIbZXhACu1VOk3xaike903Z5MEEJypCF0V7LHHSLZEwkjTxyh4e5GAjMb+niHonGnn9Z5WrLwlKX5VGFCfkPTmMkpAMcR6DBTUnbChLA2QIF4GgE9uwjeYwJy7eXRyeed0wuoSQErJt5d75K0WIikjp69xFeVgOwneI2WBg5pzIUI6tHXmHHEGwvPWRgiqGeYDijSf6VbFikQc3BM1kPOBcsaEh4oTTb2rQc4i3LNxU4zwh4u1HiIQIZ6NhGNL8+HZC2Gan0SgTJYzUXL3WKc6N1H5JgfnILy32rt+mmFQGZnYwaLp4KPDuPCmA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kLEEmdrrCIVYzFHloNkPAQbhuVK2fqfV4tfYrkFPeek=; b=UeG+SKdx3jr4Yz9UrQNJwqe1ric2mFkzd1eZxf9NspMRJlF1E2X1mDbHx8DyTHNQnedKz73M1vfUuL1iDC6jHrXzHEPKjR01yyR8Q9kctcn+rhRWbOPK41kobKrsHk2PG1YyUzRACPqIhYJ7umjUfI9KEIRZ5p3PYD/yIGVbHD4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) by CY8PR12MB7684.namprd12.prod.outlook.com (2603:10b6:930:87::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Tue, 4 Jul 2023 16:39:38 +0000 Received: from CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::e452:1914:1054:de79]) by CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::e452:1914:1054:de79%7]) with mapi id 15.20.6565.016; Tue, 4 Jul 2023 16:39:38 +0000 Message-ID: Date: Tue, 4 Jul 2023 22:09:27 +0530 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH] vfio: do not coalesce DMA mappings Content-Language: en-US To: Thomas Monjalon , "Ding, Xuan" , "Burakov, Anatoly" Cc: "dev@dpdk.org" , "Yigit, Ferruh" , David Marchand , "Agarwal, Nikhil" , "He, Xingguang" , "Ling, WeiX" References: <20221230095853.1323616-1-nipun.gupta@amd.com> <1780903.TLkxdtWsSY@thomas> From: "Gupta, Nipun" In-Reply-To: <1780903.TLkxdtWsSY@thomas> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: PN2PR01CA0035.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c01:22::10) To CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8308:EE_|CY8PR12MB7684:EE_ X-MS-Office365-Filtering-Correlation-Id: bfa21423-7b67-4911-b8d1-08db7cad41a1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1Vk7bxmUwMXQWRHualIoKRnLOFP2MOR2P16YREgaTNfOC0i4I87hEnkPbzM70Er2fTPMRCJcC8V4ceZBABR7/ayb9NDyw3dqK/OutLdcCRboduk2nC76Hvb58RPUEFtgFeJjuXW+8+3riFS1RLDJJhFYDbL0wkO1Q+k0WCK8qNhK058K5bOUxZNmw05DiuNxSTiWGGQGumMYucfW5AHqu5ZfpBv69E5kaioa/Mi2xLqV2RCCKUlRhS26wlzgC+uIoUHhNacS/sQi4XbQuOzy8k2LNUSOPOh9gNyVxgaKi/xS42NMJ+HPZNWYIZa818kHGW1HIFM7I1Wk4eDWKaTSCt5TxFVVz/g/V6cckmw/aLzOKz1NI5CjumKLNpWT7/t5t47g/BHd8TEB0Rkp34sxHHF0TMa3WHLqQ4zygLnOA/HMuzc/p1Ct1ZV67gaJ37DAGLvsRDefPOdoAyqat4CE9r/gKMkeSST14kC/cgGk4VTC+DUC6/e7hZsR5kgCWzTAZ5S4kBVR+77scDfF5W3kn3JWJQb8YXzUxxPXwXo4t8sTzA4v6JoBNQ5IUhnGbs7ZDBV+WLGSpQMdOLV1+SsjQ4fGojZ6JOrVMBI7sARbsINLxBpduk8ycAMfJvPQitSSRyYkMk1WZWNSnm1UJ2jleDEW7Q+SRvZ+L10ZMcPZ4S61B9O3xPc00NFrpcIg2VmS X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR12MB8308.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(366004)(396003)(346002)(376002)(136003)(39860400002)(451199021)(6506007)(53546011)(6512007)(316002)(4326008)(66946007)(66476007)(66556008)(38100700002)(2616005)(83380400001)(186003)(26005)(478600001)(8676002)(110136005)(54906003)(8936002)(31686004)(31696002)(2906002)(6486002)(5660300002)(966005)(86362001)(41300700001)(6666004)(36756003)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OWNDbWx4RjhsVS9mZSt5dWhzN0ZkeldHOVZkYUxRd3FyMEhGV2FiTzlKVkNq?= =?utf-8?B?dkRyNEVrVVVnUVdLc2ZnNmpxNkZzelhZa21zZlRGdUpoQ3Z5ZldYTnU2ZXN4?= =?utf-8?B?M3BHdElSRDFVTm15MjBJb0h1eUwwbitxdk9hYyt1dEYzOW9HOWtiSlBFcnU3?= =?utf-8?B?bDJLVytwOVZ6UlZkV2xZb2tMNWhERFB0bW0xS3NwWHRnS2tCdnJjU2wrQUc5?= =?utf-8?B?d0xMdHNhMWpTTHRXOVhFbVJFcWl3dW1uUEpOL2ViTVl5dzkyZzhDekl2ejJa?= =?utf-8?B?ZUNZOW9WeVdraU5lWUMwV29DcXh1cEJyMDk3SXZOVjd4Y1U1VzR1aTZ3Q2c2?= =?utf-8?B?V2g3Zy9YSW5Xa2JwZXgrOFEvd0llbml2WmdEc2xBa3phK294SVpwTS8wd05F?= =?utf-8?B?RUxpekl4dXFLaWYwYVkzUks3Q0o0UzZlMjMrbW1ob0tmZFMwaFphSXVkMUht?= =?utf-8?B?T05mQmZ3YjdPSmU0cXRmbitBRitDL1BZRlRRWHJlM0laMnpKWDVydXh3b25t?= =?utf-8?B?VTU4b3dZUTF1WXpha3Y2V2RIbi84OGRTcGpQbjNuQWZRcEt5WTlDMDFVZjFq?= =?utf-8?B?bTZtcEx5V25RSnJXSGtJMnk1cmtYWFNkSWdXYWx4aEZTQ281dWZ1RUVBTysz?= =?utf-8?B?blpjbUlPNmZ0TUgwbHJhU0RPZ2V4YW16dWlDMjdvZFMvNzNaWU85N3VPQVhS?= =?utf-8?B?NWdTRTBaZElFelRSY3VkZzB4cVhtWGxvWXBmWTJiN25Xc2wrTkZSL3BHRC9E?= =?utf-8?B?MnZqNThPQU9sQVhZa2hsdnE4aWIrZkxSekhrUTUwa3RrQzVUcjFFSm5MMi82?= =?utf-8?B?L1BrSHZSalVIeFB6bjRzNmdoY1JjUXMzb0ozZnZjZ0RVaGNqdTNhQlRsNUNm?= =?utf-8?B?S1dZaVhzRC9Ub1paSDB3N2hOL0JYaTd5eDdxdXdxRm01bFVSa1k3NEwyOU5X?= =?utf-8?B?MkszeGloc0M1dmdtTWZPbHp3eFNXZTg3TjA4L3hLblJxRVNWcWlNNVE5MFZz?= =?utf-8?B?NmZVcXJBSW5WUEhCK2hRK3ZkZ3NDbzE3eEpZempaSE5xTFIwUXFIakRBK3FZ?= =?utf-8?B?cnFLZ1hNbkJRTXNwSDRWRzVyem9PQ1Bab1JhTm9nbE9kb2VGZE53VkgxYnQ4?= =?utf-8?B?eGNKRnZUd3E1cjNsUHFNanRCakExVVpzbUNMSUZRY0xkNHZrL3Nld3N2d2tX?= =?utf-8?B?VHY1ZDE2clRZdlpRcHNMV0lPeUUxTlUwbVFya0xOblBNdlRoejhiUmphN3do?= =?utf-8?B?QjZWUENtVk9HUGRTcEVJcXRhZTFIUVo5aWl0bW1yODJQR29ka013b1Fna3lD?= =?utf-8?B?M2Z4bnJGc2ZjVk11R0tHblQveUxXUUxOSDVjVmI3bmVmSFVNWmloc0JuZFdo?= =?utf-8?B?bjZaS0dXaWgzVkU3Mm5mdzcxSXhoZENpYTF1K0x1eGpyemFQRGE0YlR1ODRw?= =?utf-8?B?N0xvNDdwQ2JzaEl3WDRZWVdVQS9iWC83cnN3NlhQTGcyQTJGelhwRXhFNnU2?= =?utf-8?B?UnF3MmI4TkJZSE9qZUdzTEZiVHJubjZZWUQ4OU1JeVR5blZ1cm5iMFNPQUh4?= =?utf-8?B?cEM2bFBDYTJranQ4cFY1dWlCMHZwN3l1bk4wdlRjZ2lWOXJkRWlRYVRpZmF1?= =?utf-8?B?QWd5SVhpRWxaYkF4UFFMMGVpOC9oQjZlSE5wbkdSdmtWMENQdUpVZUdkZktN?= =?utf-8?B?YU14RlJyV2VYWXhpKzg0MGhGVUVZeFVGMVdDWjk4SVMyNG5rSlIvZExaS2xU?= =?utf-8?B?cTdFdUZLMUhlajNvTDhucDNKRGFQQzJOVUtkSkNCQU5JSzA0ckhpTlFMeThG?= =?utf-8?B?OWl6VmowR3hNSzhlL2V4UWl4L0M4NjhnSE9iV0JTZHBXR01oU2taa1F3RXBE?= =?utf-8?B?TFlFNVZLdWRUbG1SMGR0andWMWRxOVN5SlF3MUJDRVp6N21jN1UvdlE5MWVT?= =?utf-8?B?VzlGYnpSZXRSSlNzYmhacHR2SjVXVklpamZad3dwTEdBTzNZa05kY1pkcE84?= =?utf-8?B?dkN5b09xSkNwNDVjbFI2R2I5cHBvT0hXbnBDZWs0bDNzRDFSK2dCSmx6Z3p0?= =?utf-8?B?blVpWHF6T1JwTTV5YzlDdExzWGdoSWQxNGVRdlFsTm43dmpKUmppcjVGOHRt?= =?utf-8?Q?5sRmpDN/CHVSKpPNe3E2TvWEK?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: bfa21423-7b67-4911-b8d1-08db7cad41a1 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8308.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 16:39:37.8697 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: G9/AkmfIUDCTcVOoL6VGyYiVKHHq1s3lxr5NkF4cTVctcmGiyaXEm0dzf0QWoRgA X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7684 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 7/4/2023 7:39 PM, Thomas Monjalon wrote: > 04/07/2023 11:23, Gupta, Nipun: >> On 7/4/2023 1:36 PM, Ding, Xuan wrote: >>> From: Gupta, Nipun >>>> From: Ding, Xuan >>>>> From: Ding, Xuan >>>>>> From: Nipun Gupta >>>>>>> Hi Xuan, >>>>>>> >>>>>>> Thanks for pointing out the issue and figuring out the patch which >>>>>>> introduced this. If you have answers to below queries, please let me know: >>>>>>> >>>>>>> Is there any other test cases which tests "--no-huge" which pass? >>>>>> >>>>>> Yes, there are test cases adding "--no-huge" option to validate 4k >>>>>> page size in async vhost. >>>>>> Actually, the page size is decided by front-end, so I think this >>>>>> case can be removed. >>>>>> >>>>>> Previously, testpmd can start with "--no-huge" options (not sure if >>>>>> there are test cases). >>>>>> Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i >>>>>> >>>>>>> >>>>>>> Also, if we change the "-m" option to provide lower memory, does >>>>>>> the test pass? >>>>>> >>>>>> "-m" option is also added and does not work. >>>>>> >>>>>>> >>>>>>> When you mention too many pages exceed the capability of IOMMU, >>>>>>> you are referring to HW capability to create multiple pages? Here >>>>>>> it seems in case of 4K page size we need 256K pages which is limiting the >>>> capacity? >>>>>> >>>>>> Yes, this is the result of my initial debugging. >>>>>> The direct impact is that this kind of testpmd cases cannot start now. >>>>>> If this is expected, I think we can close this defect and ignore the "--no- >>>> huge" >>>>>> option when start. >>>>> >>>>> Any insights? Should we just ignore the "--no-huge" option and close this >>>> defect? >>>>> Now we did this as a workaround. Seems no one uses the "--no-huge" >>>>> option in testpmd now. >>>> >>>> VFIO supports dma_entry_limit as a module parameter, which has a default >>>> value of U16_MAX i.e. 64K, most likely which is limiting creation of 256K >>>> entries for 4K pages here. This can be modified while inserting vfio module: >>>> modprobe vfio_iommu_type1 dma_entry_limit=1000000 >>> >>> Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work. >>> The reason I think is vfio-pci is build-in in kernel driver (since 20.04) and it does not support dynamic insmod/rmmod. >>> >>> Does this command need to rmmod vfio first and then modprobe again? >>> >> >> If it is inserted as a module then you can remove using rmmod and then >> modprobe again with the dma_entry_limit parameter. Also note, >> vfio_iommu_type1 is the module which is limiting the entries to 64K, so >> this module needs to be inserted again providing the dma_entry_limit >> module param. >> >> In case the module is built-in you can provide via kernel command line >> parameter (ref: >> https://www.kernel.org/doc/html/v4.12/admin-guide/kernel-parameters.html). >> As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000" >> should be used in the bootargs to set the module parameters. >> >> FYI.. DPDK documentation also mentions the limitation at: >> https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html > > Yes the parameter is discussed in > https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html#vfio-memory-mapping-limits > but it does not mention we may need to decrease it with --no-huge. > Please could you add this to the documentation? sure! Ill send out a patch for this. > >