From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5C2E4428D4; Wed, 5 Apr 2023 16:17:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id EA88A41153; Wed, 5 Apr 2023 16:17:34 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id 6BDCC41133 for ; Wed, 5 Apr 2023 16:17:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680704252; x=1712240252; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=qL/MbROdjFhRC+XDYf3AfENqma+6pkrZ+JEWaI96OMU=; b=VTKXm3XMKBWDNFkxY9bjIHLiNsFc7hT3rET7Qa1f/SbmAmVdYIZfpnJe B+obxCXkDLDg+xrGvudImQb820u5p6tsXLCJ1wrlpL5rFxZNLbp0cUoJT QPcmqoIGUOD2bLsNo9FNEjh5FT1LO9jNuK3+McsEAVI/XWuvfIgVVwspt LPILiJ3LRZHBzjI+q7cVNrNjsuf9E2UU+rUSFV8bktuVwpJ/yidgsZqRf yPyyTAQE/3Yf6xNyOkke66i13rltJIA4bm4vc4BEE2LvZmeqFJZo1GbVA 5pInt+PvB9pUoQDniy2Bl11/GO2ViZz1feEURMeEa4g9Bi21hiMJ8wQpT A==; X-IronPort-AV: E=McAfee;i="6600,9927,10671"; a="370289924" X-IronPort-AV: E=Sophos;i="5.98,319,1673942400"; d="scan'208";a="370289924" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Apr 2023 07:16:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10671"; a="689307581" X-IronPort-AV: E=Sophos;i="5.98,319,1673942400"; d="scan'208";a="689307581" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by fmsmga007.fm.intel.com with ESMTP; 05 Apr 2023 07:16:37 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Wed, 5 Apr 2023 07:16:37 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21; Wed, 5 Apr 2023 07:16:37 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.21 via Frontend Transport; Wed, 5 Apr 2023 07:16:37 -0700 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (104.47.51.40) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.21; Wed, 5 Apr 2023 07:16:36 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ayaso/JnH36VFvS0oldwgy+n5doawXAIlfCySlaU6NG9WhuRDZAp5UcPwicGyOUjQxx/74ZC8IthjF3MSkDfSkEr56x69mjHPvqSILlm5ilombNMlsQWR8EdU6Md0FDC+77FhOYmy5adnt5nuNWadsbNSnIb0BjW4lWkd2tqYcpn1nYYPGFzTpEfsrFT4FL8TQcuPw6eotj9cSUKR8VD4ySrfR62p84hshTEQR6qzMeI4Ln8UzyshbBw/C4DqTyM6UEAzx0Imlbawi6KzPXQT1Y0X5G8qzeyw6WgR+m5ImzOEEqhohcAaKmZRpQDRF6YXBb9+ndJsgZSaho88HSH2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Y/Yk6Lb2NNG7zXK0ycG7GCjo9RBNkHz3fToQbQqmcbI=; b=PgUhOM4LaJQ3FNhdAQBXfWK8KbJlMLEbmBX4mOjrk99pGnOcozaaIKJxmX203oBB96gUCoCgRCO697WRvdADIO4NKYRWEs1v1JFu22AkGAKDPMnbUUg3b6biiJc+9ggAmreC3QHhEp25oz9LYB1IPHDu4rewkoo7pTaURtdiOWji5+uIsXWMqJfSfY+9ZBC+5giAxfhNDEGSI9NIPirHJkuBVtRn40EcTsncxw1Xcbz+syIRdNVjjDHyyyzfm3+/Oxv28075VIQ8u/5aq6dGSDVxM2KHh126jb9LZAkEyyBjkQzUvexCMAL0h5nZDdYmeD+O/dWE6Ciru6l/ftZGoQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) by SJ0PR11MB5678.namprd11.prod.outlook.com (2603:10b6:a03:3b8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.35; Wed, 5 Apr 2023 14:16:33 +0000 Received: from BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::8d57:e56e:c679:bdc3]) by BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::8d57:e56e:c679:bdc3%3]) with mapi id 15.20.6277.030; Wed, 5 Apr 2023 14:16:33 +0000 Message-ID: Date: Wed, 5 Apr 2023 15:16:29 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.9.0 Subject: Re: [PATCH v2] vfio: do not coalesce DMA mappings Content-Language: en-US To: Nipun Gupta , David Marchand CC: "dev@dpdk.org" , "thomas@monjalon.net" , "Yigit, Ferruh" , "Agarwal, Nikhil" References: <20221230095853.1323616-1-nipun.gupta@amd.com> <20230104051936.2456411-1-nipun.gupta@amd.com> <0d356823-b198-ebd1-7a28-b678ae9edaf9@intel.com> <4e912b50-c17f-cfb4-c8e3-f3db4f96178c@amd.com> From: "Burakov, Anatoly" In-Reply-To: <4e912b50-c17f-cfb4-c8e3-f3db4f96178c@amd.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: DB7PR02CA0002.eurprd02.prod.outlook.com (2603:10a6:10:52::15) To BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR11MB1251:EE_|SJ0PR11MB5678:EE_ X-MS-Office365-Filtering-Correlation-Id: 4c629867-21ce-4a1e-08f9-08db35e05bb3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: XUToaX9Jq5m0gSc4RVmLDnwppbQrTwJ2wkfHONB1G9wjinw+rv4nLWtz0C84nfmU6JZhLzlKbI2Z/xicgnCAGoNltpLzlzYtZuAFrdzU/Q3agqaBDXY5t+bcCICk1fZueaicizUClZqtP6Mnnm6F2Nyw3amzli3PkIfNHyRKFsQ+gKqCdUingTjdC8MBzvRPJjvyEsVX+VYGxtElC1XgKpwjgRPoG17/pcBrCAoD9ui5u+8NyHESSjGZbochdOR634ZynnylRqPofCbZ8Q1qr3CtnHWWgP9hUDnCZuOkCjreVaan3sFzQh2cWPOFP5F5LjKwDxnnrLWPppWclEBBhDnp157HJ1QLguTffTnicEq+i+iW8RDvChouR4SSKUgh3YLE64tQMhNIIJbSjNFG7wrpka3twsiMesms48NgZwLuKy4VuzM6ofcHMXTNaRtQV7GlHHWNw29pI3g2u0+UDuRMC7TLhgLm97G92B5+3d7eycuhGXnpgsdGaOV07G0kPoK3QDg8SgupWVfLzY2e+RARcQ8w3KSuC7dUbO1+ayXJeSKo1mmoJ1VzTXfuVmOa3zGQI00Zez3G8gjulW9ElmQCtf3DrSHYq2ghU+3U5PoeY8LrMCFoiQfZtph7vOIuIvFYYUD9uPqNGEZvUI+jqL44Yol3qg+GfFoOwbxhniYs1cX+jNKpCgq1hgWK70Zb X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR11MB1251.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(366004)(346002)(39860400002)(396003)(136003)(376002)(451199021)(31686004)(31696002)(6486002)(6666004)(478600001)(83380400001)(36756003)(86362001)(38100700002)(82960400001)(2616005)(966005)(54906003)(2906002)(110136005)(316002)(6506007)(186003)(53546011)(6512007)(26005)(5660300002)(8676002)(8936002)(4326008)(41300700001)(66946007)(66556008)(66476007)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TmNKK1NsbEppQ0FmdkZKNVlWaVhJVlI2R2puR1hkV2JpNlA3UHM4OElTNUQw?= =?utf-8?B?M0JxT2c4amVYVFZvb0tpNFJnanNJSW0wbnkwekhGZ2tkbDExeFFJaDc1b1N2?= =?utf-8?B?QmcvdHNXdTlLN3dtNStEQ29XVXQrT3NESG9KdGRZM1VHcC9rMTd3emdWRGc3?= =?utf-8?B?ejc1RFZ1ei9QYkJkb09IMFVPT0x3MWtRVmtBek80VDVMRTZNY1hOajRzaXVH?= =?utf-8?B?RlMxN3NoWDJaU2gxMmd1Y0JMM2k3NUFEOFI4Vk8yZHNvbVBkbWlmTW1NRVJk?= =?utf-8?B?TU5lbVV2RkZsanNXTE1NS3M3UVBRbjZiV3d1WVlKODJmMFNuKzlUU2JkZzN2?= =?utf-8?B?bmlmRWx3MzFQanVUWUhFZlNIT1NVS1ZodFBQa3ZIOFR2M1pKQXZjOUsyQU9u?= =?utf-8?B?cC9JbDJ1aXZNbURsZjlONDNHVEU2YWpKVWo5VGJTVm1FZnJ4U1p0TXJoTXdC?= =?utf-8?B?dXZLWGxxMGIzZXdDZmdsZWNPYTlMdXRWZFRKR0xUYXFqcUs3YmJPUFdTMkdK?= =?utf-8?B?YlhsTEE3S0l2aVkrQm00ZlFaYkxWSzRhZ2ZZUm00QldrNWcwdk42Qk4wQlpV?= =?utf-8?B?TXQ4VnpVUmdqbWpBVnN0UCtxMzJIa096MzRIQ3N5UzVIbFRwNEJIYTdyVWxV?= =?utf-8?B?Q0dQaGxKUWZ6K2NwTytJRWhiS0xmZFlkVmVhaWIyTkZRY0RSTkVYSy8yUTY5?= =?utf-8?B?WGVDT3NKdjl1dTlETnRjYXgyYnZ5YTZWc0I5RlJBNldYQTdnemVMaWFBbHh4?= =?utf-8?B?OTFQdzluZ1g0QW9DYVZjaHBhS3Y4N0NPWW82WXE5aTA1VjZIS2VrdSt4V2dO?= =?utf-8?B?UWFKVFQrcXBzakU2VGR3dU9qN0VxWmZOeDVGWWxhSTQyL054MmVhYlhqNlVX?= =?utf-8?B?ZmM2T3Z3UDhCK3l5TC9qNXEzVXZ6Vng0ZHBiOUQ4cDVzbVNWR2YxUlpmZHJG?= =?utf-8?B?bmlxc0pWNjNmV2FJV0lTeHBWS1NPSmE0eWNzdDQ4VUR0Ly9nd0Zkc3J0QWd4?= =?utf-8?B?MGg4QnJmK3dXYkY1NnlRditjbWQrUE5yQzNPcURtTkN1LzVZNVRubFdJUFI1?= =?utf-8?B?dzEzYy80ZDFKa2V1dlhmQVpnbU5pa0srZWszTG5LWEZBZUIyY1F5K1BEeEtR?= =?utf-8?B?eVhhMEh0aUNzVkZidVBtdmtmWUtpWFdiVTJBZWV6a29JZ0FIVnREUjlsTkFM?= =?utf-8?B?L3p1NTU4N3dyNDQyUXFzcUphMzY3dGJpL2JTMGdYdGs1amR2cXcvTTBlTGdp?= =?utf-8?B?TTZLeFQ3WjRFS24zZ3pMN3ljZmFPdEp2SmwrU0EreDRoeDRrUDBtTDRwU3lN?= =?utf-8?B?MGdRTUkxUitkcEdIeWsyUTZUdFplNUtPLzYyOWJLT2pmQ21mVmUvZHh3eEZV?= =?utf-8?B?MXhRSGkyYlZ4cG1yVVRvU2VQQ3phL3E3QnpOVnlOVWM1U2llYlBxSm9uNmxN?= =?utf-8?B?cXFZN3RsR1dYYkM3U0tKRHNIdE5DU3lDQVEzS2thMUJUWmlpNnFrSnRSNlhK?= =?utf-8?B?UTlrYnFheXdrR05lS1Vkdi9Fajh4dVpRcGR1QU92cnBHVkc2aEIycWVYd1Nz?= =?utf-8?B?cWdKVWFyOGhoK2pHL08ydmRSOUlpRm9MN2xTeE5pN1VMUStMRldRR0kvbFBj?= =?utf-8?B?d1AzOVY5azRocERqNmczQ2s1aklxN3lKalFtYnEzdXNZMXpGVXNZdExjUlZZ?= =?utf-8?B?MTdZdFdHaytlNm5ibGRMN0VlSkYvVlAzcW1FeVppVkNmdUgxMUJNK3VkNmdp?= =?utf-8?B?ZHM1bFRONGRTRURIMGZUdWRyemtQQW42TUI3MlJJU1J4YUtMb1VpVlZoRU1a?= =?utf-8?B?bW5HMjgxNWJJd2wrZ3JGWGc5MXc4dENMSWp0UkFwRjFHeVF3TERzS2dFVTNV?= =?utf-8?B?WDBKZUhWOG9TL3VpMzdGUU5vV3kyemtEY2RPQ044aTd1YzNOQU9CZ0JOVVQr?= =?utf-8?B?TXNwRFBRVTVzdlQrc2txc25RejlDWC8vdDRwM1NuMVhQN2JFcWJsTGlRQmEz?= =?utf-8?B?QkNCb3dKTldKVHc5L0hraVFZMmN3KzZ6NTlwUmRhOVBEbkNCK0xzZTlVRlk3?= =?utf-8?B?cVdNUTUzU0g3eFl4dW9Gc1ZLUzVzNk1EVjM3NU8wa1BMQmIzUkpHWjk1N2Ur?= =?utf-8?B?QlRvc0g1Z1FOYTJOcytZZm1rcGt6MEltb1VkVmpOSTNMMUpmZGZlR3VzblpI?= =?utf-8?B?K3c9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 4c629867-21ce-4a1e-08f9-08db35e05bb3 X-MS-Exchange-CrossTenant-AuthSource: BN6PR11MB1251.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2023 14:16:33.3548 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KCcVGT2fuobKntl+c+BJ8TOpbKY+TWU0Vtl3gUtpQin14QnunE1au3iP8iH+3SLw7b7Fhb0mYCPHg/2SdhTSsxuuDKuUwU88TFxeFmytZkI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5678 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 4/4/2023 5:32 PM, Nipun Gupta wrote: > > > On 4/4/2023 8:43 PM, Burakov, Anatoly wrote: >> Caution: This message originated from an External Source. Use proper >> caution when opening attachments, clicking links, or responding. >> >> >> On 2/7/2023 8:56 AM, Gupta, Nipun wrote: >>> [AMD Official Use Only - General] >>> >>> Hi David, >>> >>> I agree that change is not straightforward to review, but it should >>> not cause any functional issue as we are still creating all the >>> memory mappings, but one by one for each segment. >>> For hot plug case this causes issue as mentioned, that VFIO does not >>> allow unmap of the individual segments in case mapping was created of >>> a single coalesced segment. >>> >>> But yes, I am not sure why this code was added, which Anatoly may >>> have more understanding on. >> >> The motivation behind this code was that Linux allows limited amount of >> page mappings, so we were trying to save on those. However, since then >> there have been a few changes related to partial unmaps that may make it >> so that this code is not only no longer necessary, but is in fact >> actively harmful. I agree that this at least warrants a second look. >> >>> >>> Anatoly, >>> >>> Can you please provide your feedback on this change? >> >> The patch probably shouldn't include the mailmap changes :) > > Sure, will send a separate patch for it. > >> >> Could you please provide some steps to reproduce the hotplug issue >> you're having? It would be great to have a test case for this patchset >> to put it in context. > > I am working on CDX bus > (http://patchwork.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/) and trying out some cases for plug/unplug. > > The test is as follows: >   # Run testpmd application >   ./dpdk-testpmd -c 0x3 -- -i --nb-cores=1 > >   # Bind to VFIO >   echo "vfio-cdx" >  /sys/bus/cdx/devices/cdx-00\:00/driver_override >   echo "cdx-00:00" > /sys/bus/cdx/drivers_probe > >   # Plug a device >   testpmd> port attach cdx:cdx-00:00 > >   #quit testpmd >   testpmd> quit > > This gave error at testpmd exit that memory cannot be freed. On > debugging I updated this code and seems it should be seen with any of > the device. > > I see similar test case (without quit) mentioned > https://doc.dpdk.org/dts/test_plans/hotplug_test_plan.html, but the > difference is that it is with igb_uio and issue is being observed with > VFIO. > > Please note the device/bus mentioned in the commands is not yet > upstreamed in DPDK, but patches would be sent out soon. > > Thanks, > Nipun > Thanks, I can reproduce this issue with regular devices too (run testpmd with no devices, bind a NIC to VFIO, attach it, then quit). You're correct in that since the initial mapping was done with mapping large contiguous zones (such as when mempools are created before attach), any subsequent freeing of memory will cause these errors to happen. I don't think this can be fixed by anything other than not doing the contiguous mapping thing, so provisionally, I think this patch should be accepted. I'll play around with it some more and get back to you :) >> >> -- >> Thanks, >> Anatoly >> -- Thanks, Anatoly