From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DD72B428C6; Tue, 4 Apr 2023 18:32:35 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6935740EE3; Tue, 4 Apr 2023 18:32:35 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2057.outbound.protection.outlook.com [40.107.94.57]) by mails.dpdk.org (Postfix) with ESMTP id 0E5D840A7E for ; Tue, 4 Apr 2023 18:32:34 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K0XuAzuIkldQHvzlKktolMBgD/r3ODUIPoi4nRRqJgHCGileXn4GE+qjWV9k4WWFFlETjM7UU3393prDVs//FPRibVioy8VxVccfGYSPtUfe8hptUZp/OR0EU8reBQMaZv/nBMiiq4QHJwtnem+cVP2jSwQsKuc7PzfmM0/10gXP7zep/MYOq8rYFZv/rW9mITQatRjIh7HnZuxwWERkAzWHaU3+juKf2PwUWV85pDvx63HgKM8nxQ0b+3+7tiNOGXE1pgCTKMKuOdxS3htjg8mTdnTfD8R04GJ10Wb3JKhdbCC2j7kpAWjkDU21f/AoUJ6AJYAej6f3z0tS7EgYIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CpeUG4BeHP0SCYVLz7aTXG1uw+mU5hBHmhplZU+OTUc=; b=nQ4UgkOiK5SKKJKwfmibIJAWdLhzdXMq1leM1dPwhfvYIL1yXwaL0Xdl0LvrJIb9Yb55x8gm5Rm5VwG7iI12eXsYy4B8GTKwpFYRm1IFLuh7xf3u1b+z4Ab5F2wCbU4UwRjTSyLLwu/T/lLLsOflzoDz5I3y2pTtX51e7ZhO52mSehjvB4cPFR+f633WmNOGyd4Cd0tt17hwwCthIoYfYVsuVd6nuPjZZyBcLgSexV53zh66u5WL+863btlEQORiOXA53JGDLdYDQeQoy4uXoiEafwpMM3/c0fV0/GlzMX10JxVvW+keY26VZUHjh5eyOif9fmg5pTwQr9r5zyfK4Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CpeUG4BeHP0SCYVLz7aTXG1uw+mU5hBHmhplZU+OTUc=; b=wrcfRlsA4bjnHXQ4D10FnOinAOjDKooV1zcWyprqPYow4GPX6QVNM8+Xi6WXCum7uP/3so/n1hJMwEOz7360vZ83AQV5MIHMb9nWqvVj2G5J0vOr9r3SK4vAi4AtW+czE77RXkzssBTjHvMFWMyGDoKlL6ytYv6JP/lEkubJeC8= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from DM6PR12MB3082.namprd12.prod.outlook.com (2603:10b6:5:11b::12) by SA0PR12MB4541.namprd12.prod.outlook.com (2603:10b6:806:9e::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.35; Tue, 4 Apr 2023 16:32:31 +0000 Received: from DM6PR12MB3082.namprd12.prod.outlook.com ([fe80::72ca:e9ef:88e5:26d0]) by DM6PR12MB3082.namprd12.prod.outlook.com ([fe80::72ca:e9ef:88e5:26d0%4]) with mapi id 15.20.6254.033; Tue, 4 Apr 2023 16:32:31 +0000 Message-ID: <4e912b50-c17f-cfb4-c8e3-f3db4f96178c@amd.com> Date: Tue, 4 Apr 2023 22:02:18 +0530 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 Subject: Re: [PATCH v2] vfio: do not coalesce DMA mappings Content-Language: en-US To: "Burakov, Anatoly" , David Marchand Cc: "dev@dpdk.org" , "thomas@monjalon.net" , "Yigit, Ferruh" , "Agarwal, Nikhil" References: <20221230095853.1323616-1-nipun.gupta@amd.com> <20230104051936.2456411-1-nipun.gupta@amd.com> <0d356823-b198-ebd1-7a28-b678ae9edaf9@intel.com> From: Nipun Gupta In-Reply-To: <0d356823-b198-ebd1-7a28-b678ae9edaf9@intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: PN3PR01CA0045.INDPRD01.PROD.OUTLOOK.COM (2603:1096:c01:98::8) To DM6PR12MB3082.namprd12.prod.outlook.com (2603:10b6:5:11b::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3082:EE_|SA0PR12MB4541:EE_ X-MS-Office365-Filtering-Correlation-Id: 17b0a518-32c1-4713-548c-08db352a2fbc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tIzvOxYP6pCJY0bzyZFj/8CzjaHCYrLug4uycqwnKpV3onHbuf/jtb5pwLz3q/sUWv3EehFyOMTfDMFGFAN2hmM0fg7tjt2Dq2mXgsM0sXvUczXMLzIGLsxHe3cOoR+Ey4b3sb4YlKDf2BJKvyg2ubJVHXcI2zFO8zbIMKoPQ1i8DARlXaqd5OimCY0PfeqKViptyqc32/KuDzDCgNqGhjkKg0QjseiPYDe3R7p1k9SJGPg8pDGrCJu0ZCzolauGIb7lxERKr5eKuJJxcfR7csK1eOYgWuvfHkrGkhjSVj7nWxsv+Z0q57KSV6tLWYboRMqE9k+URDeulMfSNMkyGKyNCzGvRPAk2Sg9BxZ5M4mZOnJ6g91chxhggKu69hkMLRd4enIPlnwcEznvfnfNfWHZVcZPH81AL2E0WeQeQLlLIowY0k1armbRfsrj/WMxZsGoooEwRZW2YEP4bELPPGFaCDX3T/OKcf9Yb9oXeB+IVe/ANDa/02J9BVCAxJ3v4U+IxOl3uX9KIk1Hj9AyiU7AUs+eaiAIdnz0dX+XGyS0qWyNb2l6fiE2onRhzv+GQaGQzeafQN6JP2GTqlbzvEY9APfJ6fGEryxGfDJbODHwxJQmdaYNOxvLitoxXE5T7tVzz4v74DB8qLrqxagERqAzI/F90fnplJVVH1q1HKaoJURsfhg/X8ZIN2vSHlGO X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB3082.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(376002)(396003)(39860400002)(366004)(136003)(346002)(451199021)(8936002)(66946007)(66476007)(54906003)(316002)(41300700001)(110136005)(2906002)(478600001)(31686004)(66556008)(2616005)(6486002)(966005)(4326008)(6666004)(186003)(53546011)(44832011)(5660300002)(8676002)(6512007)(26005)(6506007)(83380400001)(86362001)(31696002)(38100700002)(36756003)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eWQvQ2p2KzFYWTFwZHVRRW1NWjVDTXVvTTFVOUF0R3g4dndWSVpSdDhEaUVZ?= =?utf-8?B?MkZubWErRm1lekNoTTF4eERSKzFTaGNoMS8zUWpEUTRBOFBDQk9CbG1RRkxn?= =?utf-8?B?WFBDTjBRbm9aWlZRSk80MHM5c2ZYWTI2N0ptY2xTdkpGWjRuY29rbVpTMlly?= =?utf-8?B?bnErTGpsZStVdnJsSHNnek1SL0xyK1IyTXAzSEpmMW0vT1A0WmFuTDZ6cmFn?= =?utf-8?B?OTJUcVRScXNteTdhWCt6dERuekRKaHhWak5NZzJoWkxyU3hFbk53SkNGK1Y5?= =?utf-8?B?Y01tYUpxVFNBeTJGbk9vT0VkbnB6YllLT0lwL2lCd0J5S2J2Unl0RVJxWjJC?= =?utf-8?B?K2JudGEvSFEzb0dsMkRmWWdGRHhyblVYSUlVZ3lBdVVPTkNZRWcrdEcrNUpC?= =?utf-8?B?T1gzNmcvN2pUWUNwL3Zxd3YyVE9SWDFKN3RxZG14bGRLS3M2dnd1aXlXOXM4?= =?utf-8?B?WUhvWFRXRC9hSmdYWkFNelRKdTJjVHhvTERGanNYZXQ3ZHQ3SUg3U09EMG1k?= =?utf-8?B?NGZ1NG14WXUycDJQTmdJdnErQ21rYUdnU3B4YytMRmdQaHhmK0FXd3AxY0dn?= =?utf-8?B?bmpmeUE5V1M0SmpEVWl0bWJjbFp4WGlnUEh1K1c1VWxScmtxK001ZVRoTHdT?= =?utf-8?B?ZWp3bGdoNUJvVVBwU0J6RGM2SUZad2RRcUdLRDVLVnI0MHVPdXlROHNNZ1k2?= =?utf-8?B?MHdndDYvK1RsNmRvM2dHWWk1YWNmNStBQXhCM3FSUHFsRHhIQXBXOUo3WFlL?= =?utf-8?B?MlVKT3ltMjhSZGg0UmNhK000WWFpTlRMeDhDVUJyQ21KR2JqakpSTkFzUUdJ?= =?utf-8?B?elBLck5rcnRnQUNYZzZwc3VTemUwcDFRU2NaTTVpQ1kzUm1lakMyRzVsdTNM?= =?utf-8?B?T0U0UlRxY1NaQkt1cW1TQjhkb3Urb2hqbDRnY3ZvR21hSWpFWHk2cnY1bk5o?= =?utf-8?B?WlpaeUk5MnRSSmE5UGNmSDVkS0JUZnVZU0sxNGl2eWZpQnJoKythZmkya1Ny?= =?utf-8?B?TDRqZndVb2hQelNjNmphSTMwMlEzdS9HaUQ2djJacTliSFdiTWZabUQ2K3dM?= =?utf-8?B?YjJoekRUcFczSUZoSUtsS01FaFlZbFJHVzlzeXMrQTROTnlGYTFvTUpZclB4?= =?utf-8?B?Y1pxZ0YzeGRubnpCOEt6M3d5S1dlcFFUeDFMVDRsbGZQaXQwN0VSNlpHWGhs?= =?utf-8?B?MlJWbmFOZi9CbjRPWkZMMkFONmROblQ1YUdCcE1GZjVuYlJZSUgvVy8xVDR0?= =?utf-8?B?R1hmQkw2OWFSQ0VPQkZ2YzNKdGN5MjVoSG4wSTlYUDNTaEgxc3dxWGtRUVhZ?= =?utf-8?B?VTBFcmJzOW40Y09hbmtXa3JHeFhLSER0amhxNXBNdFloTlJUQWRpNHB2Rjly?= =?utf-8?B?TU1ib2tiL2s5T25TamtFMzBtVExsVDViT2tKN2hhZjkxdmRua3ZRZEVsZEpI?= =?utf-8?B?K2lDeUt1YTc3T3dRODRoY1c1bXF6L0RuMmZqZWg3c3lpcE94S2lvLzAzQlFT?= =?utf-8?B?U1ViUm1DMkEzVStJYzQ5ZTB4SUJodWZ6ZkYxOWd1bEY4SXcwWTluYkl5ak9m?= =?utf-8?B?aTdNNDBUZmpzb3BKOTRPNDVKRmxha2tvS01SSjAzQ3A5MW0xVTBKbWtSZ0Q0?= =?utf-8?B?RTJTSm5Fd1VxMWF4N1JjNldkeDVobGovSW9HL3UyL1JvcFpjUmVuQXI4cUNB?= =?utf-8?B?R282N1U3dk01VnJwVDJYM1lLWk9vd2xHeDBLd3U0Tmg5bDBmRHBKVE8vUUZz?= =?utf-8?B?Yzl2UkhCcjMwQzRKcVJ6ZkNPTVYxcXdJdzJ1eTJUZzkwRXMrcXBTREgxSVBB?= =?utf-8?B?bDFZR1VYV1dBeTVCK3BCdFBuU0JKbndtUklmNGw5RUo2SFNjMW1YdW1uLys0?= =?utf-8?B?UUVQVE5yaWFDZHFOZXJYNEpjUXdwYUlrR0VHSWphNmVMM3hBL0VvRDM3OW5i?= =?utf-8?B?WkpBM2RFSVFKK2NMOE9mbXoxeEs3dVJaNGh1S2NGcVR0WTE2aDNWNHFGTS9R?= =?utf-8?B?cTVYUndaT0hEU0RtU0txZ2hNalY4TVdLWENkUEtZY2kybGpMeDA3am5ESlF2?= =?utf-8?B?RWZiRUFpMzAyQzY5Qmc2ekxvNVo0ejVKNjFreCtYV1RyNm9YTlFxcUtIUkxK?= =?utf-8?Q?BoQPXyq9o5o20a/kYILJOvQCb?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 17b0a518-32c1-4713-548c-08db352a2fbc X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3082.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Apr 2023 16:32:31.3106 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8tS9gmRg9wM76j0A/Wk7074fxEhOuTXipl8gTZLwx66WexYlcubPkgSkeEhe2rZJ X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4541 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 4/4/2023 8:43 PM, Burakov, Anatoly wrote: > Caution: This message originated from an External Source. Use proper > caution when opening attachments, clicking links, or responding. > > > On 2/7/2023 8:56 AM, Gupta, Nipun wrote: >> [AMD Official Use Only - General] >> >> Hi David, >> >> I agree that change is not straightforward to review, but it should >> not cause any functional issue as we are still creating all the memory >> mappings, but one by one for each segment. >> For hot plug case this causes issue as mentioned, that VFIO does not >> allow unmap of the individual segments in case mapping was created of >> a single coalesced segment. >> >> But yes, I am not sure why this code was added, which Anatoly may have >> more understanding on. > > The motivation behind this code was that Linux allows limited amount of > page mappings, so we were trying to save on those. However, since then > there have been a few changes related to partial unmaps that may make it > so that this code is not only no longer necessary, but is in fact > actively harmful. I agree that this at least warrants a second look. > >> >> Anatoly, >> >> Can you please provide your feedback on this change? > > The patch probably shouldn't include the mailmap changes :) Sure, will send a separate patch for it. > > Could you please provide some steps to reproduce the hotplug issue > you're having? It would be great to have a test case for this patchset > to put it in context. I am working on CDX bus (http://patchwork.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/) and trying out some cases for plug/unplug. The test is as follows: # Run testpmd application ./dpdk-testpmd -c 0x3 -- -i --nb-cores=1 # Bind to VFIO echo "vfio-cdx" > /sys/bus/cdx/devices/cdx-00\:00/driver_override echo "cdx-00:00" > /sys/bus/cdx/drivers_probe # Plug a device testpmd> port attach cdx:cdx-00:00 #quit testpmd testpmd> quit This gave error at testpmd exit that memory cannot be freed. On debugging I updated this code and seems it should be seen with any of the device. I see similar test case (without quit) mentioned https://doc.dpdk.org/dts/test_plans/hotplug_test_plan.html, but the difference is that it is with igb_uio and issue is being observed with VFIO. Please note the device/bus mentioned in the commands is not yet upstreamed in DPDK, but patches would be sent out soon. Thanks, Nipun > > -- > Thanks, > Anatoly >