From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D717242AC9; Wed, 10 May 2023 14:59:26 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9CC52406B6; Wed, 10 May 2023 14:59:25 +0200 (CEST) Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2084.outbound.protection.outlook.com [40.107.212.84]) by mails.dpdk.org (Postfix) with ESMTP id C177340697 for ; Wed, 10 May 2023 14:59:23 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bOTnKai9I2BiLSEtQ/eV1h2i8FuNBiON03vV9vpmRz2ctFiyroTKaDBltLIWqc4YKSLkkR5J9ZeZPj7FIeQex0w5Xip5q3LnQBWgaRJBR6bNmaXNA5FXtu7uNKplBqQFb8kYGolt6mc3r9nsSJgGdU2evfVFzPOTCQEiOAePahDq2bTloFYPkvXiwETP3lHK9mjLYEfo1HhLtpqlYpwjBqjlMuT0q8gFnJa1FycGYc+hZXljhIHzYQALXLEgC6pK9086ltX7OtV6YVN+9MMO8JDZCGuQNYqRYWt1dsK07OtO1lGyVlGXab8pZWDbcXQ/lXzvdEr/O8YXPdUAljsTVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Nv6sbzfjh8Ml8CBddr6LVN15NQHWK6iP9VOTCtbp6ug=; b=TZA1t4oYkSpfqvIaWv0+8L9Wyx1CGWed67Ft7q590WgGSEUfYEL/3zUVwsPiClCZKujrMuGSHHYy9IGRejUm0LDgRyT616TQipT3krRH2Yw9SvOmWrre9IEN7LbtSi01AX5WflLBFQlGqF9jCHfLdK2DhsDImOD45gaRoU15/g9/zSsjZx4cbOwKW0zU0VjiBvjknotfnS449RFnoJYJNvNczk5oex/HzawT3FTpugda1pflevTcOkP6o+2lxbBRUfTkSs828ZeT3sSipT4FzY1fFZbRRP64T9CgNa4GoKeBxz55ICypGa0oedIhr4I2Hgs429kClr00AsPWoU8k5Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Nv6sbzfjh8Ml8CBddr6LVN15NQHWK6iP9VOTCtbp6ug=; b=epzpCelIZEtarreMuDj1XCndu1Lg8vfxHGpsP7mEM05Q8F9DfHgKCps5ycK8twgVKmrYpf8z6PetSqFspMX6wq7+aAs9u2aTTgjmODNE+qWNQFgGKVoD/+UNau8gmshO1AKkNSu1+WYFsGfgIbHy80M6H6gX/jM/qio8JdZdTOI= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) by SJ0PR12MB5407.namprd12.prod.outlook.com (2603:10b6:a03:3ac::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6387.19; Wed, 10 May 2023 12:59:21 +0000 Received: from CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::4e4f:3a93:b7bd:489c]) by CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::4e4f:3a93:b7bd:489c%5]) with mapi id 15.20.6363.033; Wed, 10 May 2023 12:59:21 +0000 Message-ID: <1cd5cf02-63a0-6314-d171-9d7771c94247@amd.com> Date: Wed, 10 May 2023 18:28:09 +0530 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: [PATCH v2] vfio: do not coalesce DMA mappings Content-Language: en-US To: David Marchand , "Burakov, Anatoly" Cc: "dev@dpdk.org" , "thomas@monjalon.net" , "Yigit, Ferruh" , "Agarwal, Nikhil" References: <20221230095853.1323616-1-nipun.gupta@amd.com> <20230104051936.2456411-1-nipun.gupta@amd.com> <0d356823-b198-ebd1-7a28-b678ae9edaf9@intel.com> <4e912b50-c17f-cfb4-c8e3-f3db4f96178c@amd.com> From: Nipun Gupta In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: PN3PR01CA0159.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c01:c8::20) To CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8308:EE_|SJ0PR12MB5407:EE_ X-MS-Office365-Filtering-Correlation-Id: d7eb9d65-29c0-4798-2064-08db51565eee X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3LtBpi6HcJ73Qi3xU8YO1R2YZiqxHKSxknMwaESVDpctbSXcKHmE/dXQ3NYgp0RuXtQzDyVv/375f9mnrOqoSsSGLDwNxp+KtRjBVWJvADeRFCqzxN7/TTbvZHu710wP3D29+Jt3teDAqLqfWY44scRqYKCVeXkVS5xtS2zu3Bvkl9JSnm9oeBBxjXWRGkEDqXy0Dr5ETA42/BmOlvLpoaPkv/Inl214YY1j7MtQ8Cpws/IlovQhY89yttqM8Vufrghu7FvqDfgwY/PZoo2mzFAfp//YwY1269bgcca36SHvcE//3s7UN8ZDIBC+HvgemaHglYaM4aN+z4JRnUF8gXiv8KIRUeostyS9Zs95Fu+8H+9zex+OdDZgWCJlrpNy3qST9rM1f+Cz2N5BgRHPmgZ4cnPL3FKfKIBjCo5e5g8B3dGqMyslNob7nCoNAFQTa30f+VEX8T0W0//iGJf8uEc26yDBp9EiEm+suyuDO5Vwo69C401Oy7p6odXRs2+BHfJBcQzo+RlQKxAzHF7idK759RybaEmcQQdq6Yk0tIeyD5TJgEdaSUL6zxGkEsb0esgmxuTYBT7kBxosiRoooeaztV1lVq6J9flo516D2O+8SX+dlZgk2OhZ5i/a/yGkS9pDkgdadbEuLbgJGAff9srsbGHP4/oGwNcwk9tVLXiJo5CY7XxJzOpvOkRl78KE X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR12MB8308.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(396003)(136003)(376002)(39860400002)(346002)(366004)(451199021)(31686004)(6666004)(966005)(66476007)(4326008)(478600001)(6486002)(66946007)(66556008)(316002)(110136005)(54906003)(86362001)(31696002)(36756003)(83380400001)(26005)(53546011)(6512007)(186003)(2616005)(6506007)(5660300002)(44832011)(2906002)(8936002)(8676002)(41300700001)(38100700002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QzdUOVlxTkpZbnFDL1lNS0pkVTFDT3dseFZ4ellBbmVmY3EramVGU0k1K1NH?= =?utf-8?B?V1NtSEl1RVpzZjNLYzl5THVGUUpRVHQycDRhS0ttYTF5UExmeVlvSlN3ankv?= =?utf-8?B?SWVERTQ5bXVtSzdxWmJ5UitSNlcrQ05HYy9aSWxuMmU4VnlIMXNmT3BnYk1Y?= =?utf-8?B?VlpPREEvQS9iQ2ZJMXZJWVYzWFQwKzRaUjdCZnIyVWo1UlNtRCtWdi9kSm5N?= =?utf-8?B?dTZHTlkrcXhTeHlsZkIzMkNhbkVmcFdkNWttQlBFRGV5YnlpNkJCaHFmdFc2?= =?utf-8?B?cWJDaWlwcys3NFZGOVJzYSs0NHRGZlFjV3oxRC9oZDl0cW9pd3FGdUZ2WnF0?= =?utf-8?B?UTNiblg4SGMrUWk2Q0lpQ282R1hGcEt4UnBIRGZpNFZBY3pnMDR4eWVGQ21B?= =?utf-8?B?b1c4ZjhsVWdBUEtseWdlaFJLeXcvN3B6Uk1aallRNTdPL0E2Z1BadDg1OTly?= =?utf-8?B?VTIxOXRxZmxxaGl4VHFXZWxBVDVSSjJSUkRqQlFGTXFRZHlSQ2hZblF4Q1Ri?= =?utf-8?B?dHRrYU5KMnduWG5PYkhGcHRuVURUZStXa1BuUVJWSHpUWTc0cGpFMkRSOFZj?= =?utf-8?B?ZWdyMFA5SEdsTUJHNEs3a2VZaFlOYUxLVXdwcXNyWEYwSE1NOFpuWk5idC9u?= =?utf-8?B?Rk9MRHJmeEg2MTZ0OVRWUXVMT25EMVFHY0RUeCtxbENPM3pFeG1sT08wcUtP?= =?utf-8?B?YStXczJJYWZJaU85UmZqSEdSd21nb1J1VnZ0MUg2eXZGUi9VcWtNbXE4a254?= =?utf-8?B?N0llWXlIemRrbUxXMC9wTFJEVG4yT2czWDlLK2tSaFVzTXY0VWppNFRBWnBt?= =?utf-8?B?enlFQ1J6K1gyUW40cmh0K1VTUlNZL1ZSTEF5ZFVqTjB5QTZockQ4aklHdm9y?= =?utf-8?B?aDNwYnZrc2dldmhPR0lHUVVTSHJKelJPaW9IY1BJbmt0YmtGVFdvTjZyMURJ?= =?utf-8?B?bnNpSGt0QU9vVXZrc2QrNFBvNCtCM1dYamk0c0p0ZHJ0dWhVUlMwWWNkSEhV?= =?utf-8?B?bkdJZ2oyd3FYSS9zUWZaa2RKL285aTE4NTdDNFU4ZnFzSEVJekFvR1FOeEdl?= =?utf-8?B?V2V0YVB5TVZUQnNkSDVOMlZOTFk4MXpacHFySDVsZkdNVHRlclM3ZStRbTds?= =?utf-8?B?bU9Ka3hucjVIaitFYXFHb2d3N0kwL2l5WmVxNGZUUFE0ZWxmNnYwVTNhRVpY?= =?utf-8?B?dXJkL1poeFR4N3NFTG0yeS9JQzBReGp4YmtZUGI5M1dkTGd2WDBOeVQ0eWlx?= =?utf-8?B?VHFGSHBFK3MraXNnanRBQTFGMlA2V255Nm9DQzRuWGc2SmJjUm5QZ3pZUjVh?= =?utf-8?B?L3B2dGZZUWZrZW01Tm9FTlBwYkg4UmxsaGJ1TzFzb0NjMm91aVA0VG9SeXFl?= =?utf-8?B?Nk5EMEJFbytOWG5Fc1FSY2dyZVF2SEJDSGpDWFUzTkRwOG1Sby9oaVpLNExt?= =?utf-8?B?RmZua3M4SDNaQWdGODRzN2huT1MyTVVTbldCQVUxYUlIb25BS2NEdGhBbzQ3?= =?utf-8?B?THJNUHBYazBFcHpYWStaQU1saGorWG1OakFPeGtHK3pvZnVqdDN1VXYzMnVi?= =?utf-8?B?WnpzelA1OHVHcG13RlpBbzlFRno3T29SWkN2anNqOTVEN29UYlNhMmVaREZh?= =?utf-8?B?K1A5L2ZNWEt6T3hDZkFWSVltWTVsTEw3SFozamZGZHlLQWVlOWtraWxXZUo0?= =?utf-8?B?QUNPQ3dzcHZ0MXFCdzhlQXoyb2FZSndmTkRhUG44R0I3U2haQU5GVS9RUmFu?= =?utf-8?B?NDdTMjFSekpmcldmN01PbEtHMWc5M3JjYmllZENyOGFKOFNINThqdUxpekVy?= =?utf-8?B?Uk5VWFA5WVc1eHhvOVN0Z0hvNFlIVkpUQ1B2RHdDSmhGWSthUVptWjlXVFo0?= =?utf-8?B?YTkxUHJZWkpuUGdzcFlWVHJGM2RRWFg3c1VBQm84WDNlVXk4SFBBVDhaWVpD?= =?utf-8?B?b25hMjFoZHdvb1RDNmVNeEtSdVRiZTNWa2RwaGY4bzNFTGRzd2ptRXF4V1lW?= =?utf-8?B?dnJHd1phRzVWU1BYWG5DamVXa3ZDNHFoRGFYcGxBOXZZWkhadmQzOHFQcith?= =?utf-8?B?WTBVUUZqdFFHUmFESnB0L2I2a3BJZlpiTnk2S1R4VzNSYm9NSlptanF5VlN4?= =?utf-8?Q?SPYs/S24Dt3sWsAKtaeILJ+tT?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: d7eb9d65-29c0-4798-2064-08db51565eee X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8308.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2023 12:59:20.8342 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vETN6K8k3iAygxCsFzUykAxEtLZhLql3YROi5XkXs5F8aCGxNAXPXSpUybVb67eX X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5407 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 4/24/2023 8:52 PM, David Marchand wrote: > > Hello Anatoly, > > On Wed, Apr 5, 2023 at 4:17 PM Burakov, Anatoly > wrote: >>>> Could you please provide some steps to reproduce the hotplug issue >>>> you're having? It would be great to have a test case for this patchset >>>> to put it in context. >>> >>> I am working on CDX bus >>> (http://patchwork.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/) and trying out some cases for plug/unplug. >>> >>> The test is as follows: >>> # Run testpmd application >>> ./dpdk-testpmd -c 0x3 -- -i --nb-cores=1 >>> >>> # Bind to VFIO >>> echo "vfio-cdx" > /sys/bus/cdx/devices/cdx-00\:00/driver_override >>> echo "cdx-00:00" > /sys/bus/cdx/drivers_probe >>> >>> # Plug a device >>> testpmd> port attach cdx:cdx-00:00 >>> >>> #quit testpmd >>> testpmd> quit >>> >>> This gave error at testpmd exit that memory cannot be freed. On >>> debugging I updated this code and seems it should be seen with any of >>> the device. >>> >>> I see similar test case (without quit) mentioned >>> https://doc.dpdk.org/dts/test_plans/hotplug_test_plan.html, but the >>> difference is that it is with igb_uio and issue is being observed with >>> VFIO. >>> >>> Please note the device/bus mentioned in the commands is not yet >>> upstreamed in DPDK, but patches would be sent out soon. >>> >>> Thanks, >>> Nipun >>> >> >> Thanks, I can reproduce this issue with regular devices too (run testpmd >> with no devices, bind a NIC to VFIO, attach it, then quit). You're >> correct in that since the initial mapping was done with mapping large >> contiguous zones (such as when mempools are created before attach), any >> subsequent freeing of memory will cause these errors to happen. >> >> I don't think this can be fixed by anything other than not doing the >> contiguous mapping thing, so provisionally, I think this patch should be >> accepted. I'll play around with it some more and get back to you :) > > Can we conclude on this topic? > It is best we merge this kind of change the sooner possible for a release. Hi Anatoly, Can you kindly update on this? Thanks, Nipun > > Thanks. > > > -- > David Marchand >