From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7A9DB42D91; Fri, 30 Jun 2023 03:46:58 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 07B634021F; Fri, 30 Jun 2023 03:46:58 +0200 (CEST) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2085.outbound.protection.outlook.com [40.107.92.85]) by mails.dpdk.org (Postfix) with ESMTP id 6E10D40150 for ; Fri, 30 Jun 2023 03:46:56 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RmMXMeOJ71/p4UJx29mzz7+nbsWFEcf0EifGYgvAzO3l2DIo7ULiXdKy+Ajapb7RaQpamhF9rW+Ck56VnzOp7wMolIkfzB4vl5dvNtnBA3q5LvA1OrxzU0Elu+BkZp7LCqOzTjqa4nOojtj+OfkNyEVFkBdn1Xd9LxhUFmKyiGQ2DxmjzVBlBh1md0vyKnIV7hJWXn6pLflqn7uj1xH3+8EXuwtIp7iJPQlYLIlEKH905IgdxUmXL8w5Fe+wr5ra6yyzsQuLoyOi59BQYa4SsTyv10cTjQe//DZGu2ip48W09/LbghD5koRTpUTeaiwKok/2AzNwiGMWYPZXBRQaeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BnPBUGJ10ytiY9nlYNJfH1d/jFvCUOs/UgqJ/T0iF1Q=; b=O5dYd2ahXceFfHuo/3FBDcbsd+QUrfdUYLH0AtXvHc3zEKaTNXpO8tiS2/26AZyq6va9RvWJAXAxa3RtZ2hsHLxBolO1RiHcmBB1JsVyn2YCfxoe8Ib0a2EEIIMGcgQqMKa6DlqfJ6fphuzTrVnABvpX27yqIBr4S3hr4Rw++nXGYxkH1wWUelQnhlsisPuFI4NRw1/fO8pus5uZE8BcE9oZZGBAuwbaErPRY+hG0yliqOQFGq1O6vmXA2/OMP/MicIm35JNKBZ4W19KUcb60TfLOG2dvxQkK7ROJs1hhg4mK9rc9GZMRo0LJnUn+XkwXVBGd/l0k+JFya43fujXIw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BnPBUGJ10ytiY9nlYNJfH1d/jFvCUOs/UgqJ/T0iF1Q=; b=1FhLsisaxjHYB+BpHVcmBi/KZN/EDuYus5dyixbCF+lu+NML1uuDU6Z9Qi+3M4t/tnHr1DkjmyCETMEdG9m7NscJQBMXIJWF/QxYro/p1K2hTWrI9sgcmUvIcg24PHETzwpfMizHDCEWJm/aZIEEldhRuaIEKQ8aosINSJNQxRU= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) by CH0PR12MB5219.namprd12.prod.outlook.com (2603:10b6:610:d2::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.15; Fri, 30 Jun 2023 01:46:51 +0000 Received: from CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::e452:1914:1054:de79]) by CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::e452:1914:1054:de79%7]) with mapi id 15.20.6521.024; Fri, 30 Jun 2023 01:46:51 +0000 Message-ID: Date: Fri, 30 Jun 2023 07:15:38 +0530 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH] vfio: do not coalesce DMA mappings To: "Ding, Xuan" , "dev@dpdk.org" , "thomas@monjalon.net" , "Burakov, Anatoly" , "ferruh.yigit@amd.com" Cc: "nikhil.agarwal@amd.com" , "He, Xingguang" , "Ling, WeiX" References: <20221230095853.1323616-1-nipun.gupta@amd.com> Content-Language: en-US From: Nipun Gupta In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: PN3PR01CA0022.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c01:97::23) To CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8308:EE_|CH0PR12MB5219:EE_ X-MS-Office365-Filtering-Correlation-Id: 5f5a74c1-687f-4dd4-ca00-08db790bdf96 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: f88wZ9EG0Smzi6MHVlS7LTpT7cTcd4Kzkfsweh7PhzXQI6vbgCpMO612kNVfS0+OJXEbYaoKqNc9sSpqgUdp+yfO36+9Tzym74vxeFnfq0JmcM+u+YhsGzOFJ/jQ/2cJ5swiYFfe0ZRBiTWaJxWVAgqjDKu7cKskjN6xvprq1o+OTp3m0A8wfbHD/AbOu3XgqJXJO1KjOa/5s9TV2KgEkpHau43WA+uaajEqTXnZ8tcOAepukv/U0wvJCvjDomnjD4cGI5zZvj6tAdIViEXyVVdQx7Ysm+nn2P84E7vbXj5Fk3HbuKHHZ8K5nwjZNXSIs/pdbTcdvMSQF+1x2IX2E43ZXYLJO2sGye5PLFua+7JfspKSY/eeI8EpLhBE9PqWAd8ZqfrDvLM7ydZFw8ATCoa3plcst8MsdKa+FhqFHrwVhOrYk/O3bvcXTzpKwbdoGTY47p3yQG8SWI+lIBrPtJTHIvP5gRP1/MrMgP+evkgiT3vX0DgZ8MyhDjfUeOPFL+2KG9cKYp4H3dcghXgLQxgexEB4U8vsbmWZbBuzgd0orzRg8YZNuaFS907igL0SMZWNWYVyV4QWb4ug/XXL3fl+WDcsEW62ySrDKY3bs+J0F+ewrUONoV67/d3gUkkPG/x+t5olk5dHzcmOD0Qo1g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR12MB8308.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(136003)(376002)(396003)(366004)(39860400002)(346002)(451199021)(6486002)(38100700002)(53546011)(6666004)(2616005)(83380400001)(6506007)(186003)(6512007)(966005)(31696002)(41300700001)(86362001)(110136005)(54906003)(316002)(36756003)(2906002)(66556008)(66476007)(66946007)(31686004)(4326008)(6636002)(44832011)(5660300002)(8936002)(478600001)(8676002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SjJxRzhiMU8rcUVBMzVjL2dINVBsWU1xcHNJdVZuVEMvMGtvTTQxYjRaR2Mr?= =?utf-8?B?RU15UWtaS1NoaXJBbEdCY0MyUWZrSjlldWptOEt1b1g0ZGY4ZGh2RWFnSSsw?= =?utf-8?B?UE5ETGJBSURjK0JCMVh1ZGdsTjEwVlRKV2dBVFFYQXk2VTJmUmdGSysxK2E5?= =?utf-8?B?bzBVbUlLWXovaHZVM0s1d0dqdkVKOGl4UStaaGhEWnljN09WVHpaUU5DSXpl?= =?utf-8?B?aFhSdVQ3c0tLVmxwck05VUMwUXNSL21ZMCtOWGNkZjA4Q3pyUXlGNUN5RkJN?= =?utf-8?B?YzJuUmZGVGxaYlprVUFYMmlRVDVnRmlkazIwaDFsNFQzUHprZGZCRkh6Sm42?= =?utf-8?B?aTBDZmd0K2t4Q1FJcXZtb21XeGpKV05KYVFKWGdjVk5zNU5zTnNna3FyT0k2?= =?utf-8?B?TzZzN0cvNEd0MFI0L3QyL0Q5UmVXNHNxODRDZnEydlBVaEpBbVY2Y0dnRlJF?= =?utf-8?B?R01INHBsTEd6a2dlNEFud1ErTkI3bHljeXB4a0FnMEJ5RTkwL1hGYmY0UG5X?= =?utf-8?B?OWxSc0lzU1kweHFxR252cm1QT0NwRE9Gd0gxa3lLdlVoVUdGYmh5akd3K2Jj?= =?utf-8?B?Wm1hKzlVZGVSYnBjNmV0SncrY1o1MEF2YTBDNUMvNHo3MlZ4TUpVSDFTZEhx?= =?utf-8?B?bU5nRXRGUng3V1Jtc1l4amFadE1VaTBTUEtoRzYwRStKdGY4aTkwdnk2SEVS?= =?utf-8?B?V0taeFMwbnM1M25OalRGUVpLRXJ0UGpIQ0xyOTRIaE53MlRKelVvRG1HYnlX?= =?utf-8?B?VDJVcmM5VTRWckZxTXVlUHMrVFdlbE5pVGZ4d0tHUUIxUUY4bEtza0RFVWFh?= =?utf-8?B?aGpVOTRnbGd3NENjaHJVa3lQN29TWnhuTFNaV3VPQitaZWJKc1Z0ZE5iMVlG?= =?utf-8?B?WFNPNmVJenNoVUtDbGhPNFcxc09KKzdQSW9IY0ZNNnRrNFk1TUpFV2RJVXR3?= =?utf-8?B?b0tGU0NTcFlvNmpBVUlSZXpWTDM1MlBMMzhlN2dLWDdlYWZQbFBtQlVXSy9r?= =?utf-8?B?dUZIZUxwODRDTloyNGFuOG1CeFlTb1dGSUtHTEZJM2Ivc1JtTktEVFI2VklX?= =?utf-8?B?R0RlUzU2bWpMQjNTUEl2UlY5bHBLbCs3UkVQOFBKTWJkY3dERk5rSGpiTzA4?= =?utf-8?B?STYwRitQSHFHZ0QzTlRSMktkbUcxSDE0Y1NOcjZwRFN2MUs0Q2hsZlYvTU9H?= =?utf-8?B?dUpiL1BSU1RjSks0REczWWpRZHdwUFFkSnpIczNWZm9DejhhQk91b0JPdkV5?= =?utf-8?B?Zjg4K3YvZE5DWmYxd3U0TzIvcHNzTVRHdHlnUGVMWm9BaVNMYkVic3cxNXRK?= =?utf-8?B?em5LOFMzd0dFZjRXc0dJdG9ibnlRNzd5Skx3UXpnbG1veDQwVzIydlp0YTE2?= =?utf-8?B?YVZZc25jaFNueDd6MUUzUDVROUY1M0NNcS8zOENnOUtlSHh5eEx4UDMxR0c5?= =?utf-8?B?bjN6R2dLUitQV1g4bXhkdEtPenlwSUtPdjRRUEJFeWNzSkEvYTh2MFpZb1dq?= =?utf-8?B?b0pFNzMvNXZCUnZKZGtIWVJjOGd1UE02ZUR6R0RWOXRnSnVmQVA0NEdKR1dN?= =?utf-8?B?UGFMdlVUcC93R2c3Uk84RjVxN3ZFeDM3cTV3bVpLR0NQYk5id255eG5ZYzdC?= =?utf-8?B?RU9XeGJoZFB3T01MZUtISzFQeTdIcGpQeUlyYmZTS1p2Ukl0VTI3bEF5UUFK?= =?utf-8?B?bEl0K2tpc1ZyVmx6RlAySDZCM0w3ZWhWYVRzNDJ5TThxdGZaOTNib0hiLzlV?= =?utf-8?B?UEZQMkcxMy9UbUhNdlJqU28zOG1HWDZnVjRXN3pGVDN2SXZuRUZ6U0l3VkZv?= =?utf-8?B?N01vbzgrT25lWlZ6UUNDSm83NU5nQ2tUb2FBMk5yWVN6MUNvUGQ0WDNBRHFD?= =?utf-8?B?MTdqTko2L3pmVWF1UVM1ZzhPdS9NdGgzRzlKMmR3TFN1WkN0OUYvNlJFNGFN?= =?utf-8?B?SHJhcVVBOE9sRmZsTXlrZTNnS294MDYzUjdDa1FFalE1cXZMVmx1OStGNlZq?= =?utf-8?B?QWs5M2ZOeXpoKzluUFd3RmJFQXh4SnFwSFlMQTFBcnRrd2dHbGFyTEozWEtZ?= =?utf-8?B?SS9PMGxoVk9Md0Q3OWJud1FRc3FzMzlHNkNXMDYxZ3gzQjN0RDRKUVU2NDNC?= =?utf-8?B?UkEzaW1MbTNQRG1wZDFLWHBlTmNCK1VpL2NFb0lYekFlVThRK2xkREpwQ0xo?= =?utf-8?Q?Pz+/4zytvSgLjADEvsDueGgmpFBYXLD+YyVPhg/CFX5L?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5f5a74c1-687f-4dd4-ca00-08db790bdf96 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8308.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Jun 2023 01:46:51.1082 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WNQKGLIU7ReYCp4AseampvMKKfkYS3v0UgtK8303bJbNID93km21V7rN3oxuEwnD X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5219 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Xuan, Thanks for pointing out the issue and figuring out the patch which introduced this. If you have answers to below queries, please let me know: Is there any other test cases which tests "--no-huge" which pass? Also, if we change the "-m" option to provide lower memory, does the test pass? When you mention too many pages exceed the capability of IOMMU, you are referring to HW capability to create multiple pages? Here it seems in case of 4K page size we need 256K pages which is limiting the capacity? Regards, Nipun On 6/29/2023 1:51 PM, Ding, Xuan wrote: > Hi Nipun, > > I'd like to appreciate your time reading this email. > > Our QA team found that since this commit "a399d7b5a994: do not coalesce DMA mappings" is introduced, > the dpdk testpmd start with "--no-huge" parameters will failed, and shows "EAL: Cannot set up DMA remapping, error 28 (No space left on device)". > So they reported it on dpdk Bugzilla: https://bugs.dpdk.org/show_bug.cgi?id=1235. > > I understand this feature is to keep consistent with the kernel and not allow memory segments be merged. > The side effect is the testpmd with "--no-huge" parameters will not be able to start because the too many pages will exceed the capability of IOMMU. > Is it expected? Should we remove the --no-huge" in our testcase? > > Regards, > Xuan > >> -----Original Message----- >> From: Nipun Gupta >> Sent: Friday, December 30, 2022 5:59 PM >> To: dev@dpdk.org; thomas@monjalon.net; Burakov, Anatoly >> ; ferruh.yigit@amd.com >> Cc: nikhil.agarwal@amd.com; Nipun Gupta >> Subject: [PATCH] vfio: do not coalesce DMA mappings >> >> At the cleanup time when dma unmap is done, linux kernel does not allow >> unmap of individual segments which were coalesced together while creating >> the DMA map for type1 IOMMU mappings. So, this change updates the >> mapping of the memory >> segments(hugepages) on a per-page basis. >> >> Signed-off-by: Nipun Gupta >> --- >> >> When hotplug of devices is used, multiple pages gets colaeced and a single >> mapping gets created for these pages (using APIs >> rte_memseg_contig_walk() and type1_map_contig(). On the cleanup time >> when the memory is released, the VFIO does not cleans up that memory and >> following error is observed in the eal for 2MB >> hugepages: >> EAL: Unexpected size 0 of DMA remapping cleared instead of 2097152 >> >> This is because VFIO does not clear the DMA (refer API >> vfio_dma_do_unmap() - >> https://elixir.bootlin.com/linux/latest/source/drivers/vfio/vfio_iommu_type1. >> c#L1330), >> where it checks the dma mapping where it checks for IOVA to free: >> https://elixir.bootlin.com/linux/latest/source/drivers/vfio/vfio_iommu_type1. >> c#L1418. >> >> Thus this change updates the mapping to be created individually instead of >> colaecing them. >> >> lib/eal/linux/eal_vfio.c | 29 ----------------------------- >> 1 file changed, 29 deletions(-) >> >> diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c index >> 549b86ae1d..56edccb0db 100644 >> --- a/lib/eal/linux/eal_vfio.c >> +++ b/lib/eal/linux/eal_vfio.c >> @@ -1369,19 +1369,6 @@ rte_vfio_get_group_num(const char *sysfs_base, >> return 1; >> } >> >> -static int >> -type1_map_contig(const struct rte_memseg_list *msl, const struct >> rte_memseg *ms, >> - size_t len, void *arg) >> -{ >> - int *vfio_container_fd = arg; >> - >> - if (msl->external) >> - return 0; >> - >> - return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, >> ms->iova, >> - len, 1); >> -} >> - >> static int >> type1_map(const struct rte_memseg_list *msl, const struct rte_memseg *ms, >> void *arg) >> @@ -1396,10 +1383,6 @@ type1_map(const struct rte_memseg_list *msl, >> const struct rte_memseg *ms, >> if (ms->iova == RTE_BAD_IOVA) >> return 0; >> >> - /* if IOVA mode is VA, we've already mapped the internal segments */ >> - if (!msl->external && rte_eal_iova_mode() == RTE_IOVA_VA) >> - return 0; >> - >> return vfio_type1_dma_mem_map(*vfio_container_fd, ms->addr_64, >> ms->iova, >> ms->len, 1); >> } >> @@ -1464,18 +1447,6 @@ vfio_type1_dma_mem_map(int vfio_container_fd, >> uint64_t vaddr, uint64_t iova, static int vfio_type1_dma_map(int >> vfio_container_fd) { >> - if (rte_eal_iova_mode() == RTE_IOVA_VA) { >> - /* with IOVA as VA mode, we can get away with mapping >> contiguous >> - * chunks rather than going page-by-page. >> - */ >> - int ret = rte_memseg_contig_walk(type1_map_contig, >> - &vfio_container_fd); >> - if (ret) >> - return ret; >> - /* we have to continue the walk because we've skipped the >> - * external segments during the config walk. >> - */ >> - } >> return rte_memseg_walk(type1_map, &vfio_container_fd); } >> >> -- >> 2.25.1 >