From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <dev-bounces@dpdk.org> Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7F46442AD6; Thu, 11 May 2023 16:11:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1B5ED40DF6; Thu, 11 May 2023 16:11:50 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id F210F406B6 for <dev@dpdk.org>; Thu, 11 May 2023 16:11:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683814308; x=1715350308; h=message-id:date:subject:to:cc:references:from: in-reply-to:content-transfer-encoding:mime-version; bh=0utaxWUZYQVBI5KG3BhiX62MKqIyUTU6uS24iVu6Uyo=; b=VD5tOj4ZmniIVUvbuDkUGtn+S6UKQflKjhrF+9VZ+IBoeKI6J4OnTfoM dP0+9mvJ5VyIDA2RGK9BNzWx/nCUJnmUzieuTpRwiBzBpaROrKLKU3T7e ASa7u/7gzvzf8JhGeV3O8bxdsXB9w8atr/TewqVi+yIHmvA+uP8QmPE4h vHkABbpQZQoA+s79qVSAPVLtR2CZhcQDJfd80qNFCA4ct7Eb0vImCA5V1 /+KpT4WIJ5//5+tzaFM2/wg1uAPB1dCCR2KK57wJRGZ83xir/hium7kln 4TKDcBAE9yr7V/TZqMk0I/2xsLOFhCPt2lSRBNqIl0y1f5mT/xSzsV4Eq A==; X-IronPort-AV: E=McAfee;i="6600,9927,10707"; a="335014352" X-IronPort-AV: E=Sophos;i="5.99,266,1677571200"; d="scan'208";a="335014352" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 May 2023 07:09:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10707"; a="946194731" X-IronPort-AV: E=Sophos;i="5.99,266,1677571200"; d="scan'208";a="946194731" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga006.fm.intel.com with ESMTP; 11 May 2023 07:09:01 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Thu, 11 May 2023 07:09:01 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Thu, 11 May 2023 07:09:01 -0700 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (104.47.55.107) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Thu, 11 May 2023 07:09:00 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g5DENU+6f+xxdHRyEgaEi30eLEnv2JmiZoDQHGagI9P0ZhYENXxFvnxYAu3UILwS5rfG9t5oGOXDA9OdBzuV66n6nAXirCC4S5KbC4gpEHvIM7kr5S19CdDYtrpi9+6nHSVd6hjYX27e1HmCnZ2QjHtTPRB9qUbyqb+9zTsjLY64ugcBCbJVRuplJdOB1VrQGzEOrtD+F8IAAq67CF5ycmceKyq1ZFZre1e0isv1X5urs2ftFZxiSVDmyUgiYy3Xmq1moeAIheN1aPEDF1yexpR0Wa7nIiG3VTREPHLU3xgb2FHZuUoqu/xR7L0dE9vncNcPukvAFaGgTSMNERwpCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n+Y578+PmZrfIKno9kQYrgJAV0Lmi70wtjcYgkl0Md0=; b=QeGoLssDare4nxYG+ZfW6WcEJJ8paMG6Bar8aGcikrZdtuAC+YVHkFZmLu7zt0rfWXcPC+k9dpB2yXMZPOxMxLmMt3GUBIk5dboWU2uPB1mVFBWHzL6u2SXDtz0xlGDuwN1eBHBrW52fi6JpZ51QOXZEZA5fCdglnTQ8oYV9RtmyfkldACPiPVzgU/XZjPsvCJmklvrcAo2BeplyXiQjkVYIXP+8tKOj6fgdyz6ik27rg5XSAVl1i9rfV24dwsAEm6Uh9aXLm9Q2PE/jSKBGx/zZhMDAo7d9/8eioys618NaCu2jX0PRn6i7VtHlxPDIVAPUocAEknruRObK5UnjMQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) by SA1PR11MB7699.namprd11.prod.outlook.com (2603:10b6:806:338::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.30; Thu, 11 May 2023 14:08:58 +0000 Received: from BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::8d57:e56e:c679:bdc3]) by BN6PR11MB1251.namprd11.prod.outlook.com ([fe80::8d57:e56e:c679:bdc3%3]) with mapi id 15.20.6363.037; Thu, 11 May 2023 14:08:58 +0000 Message-ID: <b4030304-4cfd-ace1-402f-4016c8277d01@intel.com> Date: Thu, 11 May 2023 15:08:53 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0 Thunderbird/102.10.0 Subject: Re: [PATCH v2] vfio: do not coalesce DMA mappings Content-Language: en-US To: Nipun Gupta <nipun.gupta@amd.com>, David Marchand <david.marchand@redhat.com> CC: "dev@dpdk.org" <dev@dpdk.org>, "thomas@monjalon.net" <thomas@monjalon.net>, "Yigit, Ferruh" <Ferruh.Yigit@amd.com>, "Agarwal, Nikhil" <nikhil.agarwal@amd.com> References: <20221230095853.1323616-1-nipun.gupta@amd.com> <20230104051936.2456411-1-nipun.gupta@amd.com> <CAJFAV8zqSonhewH_RyOSMEPLG1HyyN3jVOe_nh_mJRvzTYLjaw@mail.gmail.com> <DM6PR12MB3082D6EA8637745C88EFB94FE8DB9@DM6PR12MB3082.namprd12.prod.outlook.com> <0d356823-b198-ebd1-7a28-b678ae9edaf9@intel.com> <4e912b50-c17f-cfb4-c8e3-f3db4f96178c@amd.com> <c82cc0a1-4545-2000-0027-15bc7df1b50e@intel.com> <CAJFAV8x5GEAVLBbH7ooFrsHCLLpPDhuZxya1_C46FFwvFdCpxQ@mail.gmail.com> <1cd5cf02-63a0-6314-d171-9d7771c94247@amd.com> From: "Burakov, Anatoly" <anatoly.burakov@intel.com> In-Reply-To: <1cd5cf02-63a0-6314-d171-9d7771c94247@amd.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO6P265CA0026.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:2ff::16) To BN6PR11MB1251.namprd11.prod.outlook.com (2603:10b6:404:48::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR11MB1251:EE_|SA1PR11MB7699:EE_ X-MS-Office365-Filtering-Correlation-Id: 768b7866-40f6-46a4-562e-08db522943b8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eZok1Vt8UmnKWzNtY4NYPnloh9oGUE3lMIbSduumnJfZGMbwEAqcT4AoxiHJIEzup/5jyKA04iCDvW+d3UyfIIWC9/Bh3bv2J5qIas+Ymk7WcWtDLwsYGi1z1x69pydh+14p0VbhMJ1zxujMOj2bzYN9IsLNgO/a2BAqWqxo1QIUtizBDsJl/QdFtE/x8Wc7XOtcYkkbPK6f2M6y7d4GN+g8kVxdlH3C5ZBRnEz1IozvlcL/KJspPOmGGEubds3duwdu6MICv4cfFkJfm1wc4P1Rc8LPmoUAkOu5E1y2wKtZrDJ9wEIK1oGDrmZgkl7XtduM1oGet857GdGlEKF/4/GVkXuK3QbyUC9sQRG+PepZLD7BXBLXVhqqX/HhXlgRaeQ//S/NSsFuaOS8Ku5q6HXpqnMp+PfoJyZYhM9slisW4x9jOLN5HuTZScrbVLdRMLk0bUrt8DbSqi9THm8mpF/XUDSBTtboFEycRU13jbbAGJZjMPQPKcmqHRtxCcqr4bW77lxH2P/Cte7oiaMTfQUv/5/AbFZzXz0YbUhoUeGDyLaR5HVkM+pricWrDW77l61X4nVVb4hbRYtlrk+iHEXsElSr5SUMWPORGFxUlMVexWi2L9AymCnKOXG0nAm2NrhUlICBoM3wLMZ7vCo6+CwSILHneNeEfPjk4o8HFes= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR11MB1251.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(376002)(39860400002)(396003)(136003)(346002)(366004)(451199021)(31686004)(966005)(4326008)(478600001)(66946007)(66556008)(66476007)(316002)(110136005)(6666004)(54906003)(6486002)(36756003)(31696002)(86362001)(83380400001)(26005)(6506007)(6512007)(53546011)(2616005)(5660300002)(8676002)(82960400001)(186003)(2906002)(8936002)(38100700002)(41300700001)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dmpGK0tUZVlkWmZ6OHMxdXdzMEl2MlU4OWM2aVdpc1UvMGl2QkJSTERUY2Na?= =?utf-8?B?VW9heDhSdGcxY3FyYXhJRDZNcG44N1I2M1AveFRMU1MvNFJWSWFPNXRTaEN2?= =?utf-8?B?aEpvenZJRlVwa0pPVGg4eGJEV2FBTUlhMG81UGtkeXo4eUJDcVUyNVpMZ2Nj?= =?utf-8?B?bjBwdU9IWFk4SHVtSVhuRVZHMHlHZHJNcC8vT1BkY0hzbzhoeTNud1NOWFdE?= =?utf-8?B?dDRHQlo3clQrWjk5SVZqMjdFSzFaQ3FsK0hWV2NDZUtYZlRUenZmQWFmZllS?= =?utf-8?B?aEpKbjFaUmkxd0Vka1VZd09xa0REdUM5bmhVa29acDJnbDBqNXQ3VjNYVGVs?= =?utf-8?B?M2puQnpZcHorQkhuOU4xM1l0SmI1RXNVQnA5Sk40aUJGVzMrL21xeUE4VCtx?= =?utf-8?B?VGc4NFR5bVRnQnByMzFzLzlDUGtXdXRLdEoxUHEvU05weWdYeHlwZHEyNXdK?= =?utf-8?B?eWpMNUpYcUVkblZOYUhHdi9EV1k2U1VDNmZncnZHNEZyM09qa0g3aW5YT2F3?= =?utf-8?B?Z0NZNnlRZEN2enlZZGJUekMvRytxWENpOTNqTXBzTkRHNzVFVDZxVitmMi9l?= =?utf-8?B?T3pmaDRJVDZySmgvT1E5OTN2OHp3aEpLMkUrZWhIQzA5N3N5bEErNG5HMVlO?= =?utf-8?B?dURmZVBjRk1uTm5OSFo0ZlFMa281bXNMck1DSEU2THRpZGZlT2lzTllGamor?= =?utf-8?B?U05yeUVzTWNWSGZxYjZPb2V0dVZLN0U4VGU0by9NcGlDSWRlcnNka2lRV1FP?= =?utf-8?B?VGVVcmdOZEZOcGUvdUhZNW1nSHFzRFJLRUZnbzZuUUlNQ1ZPTHNnM0lESDdN?= =?utf-8?B?MVFrc3M2dHZEU3pJYTBYNjgxTGxNUVZ5UGxFcjhlRzVXdTlOYUhnUmZDOU5x?= =?utf-8?B?bWpsMWJPT1hldCtkUDZoVUNwM2FyYjBObFRXaWlrWTVRUkRNd3ZxNlZaRFZ2?= =?utf-8?B?QmxXYWxHRjhuaXdtT2pCN2pVSmFMUUJ2YmJva24rQ2FRM0NhcmdkSDdwTDRS?= =?utf-8?B?WE5DR2hVWE9IcXhMNXhOTUxDSHhLMDVkSEwwNkZGckhRNjJLa05wejhiUk5i?= =?utf-8?B?WCs1ZUk0bTZxT0Z2VnFaLzcydDlTSkFvZ0RydTRjcEVicTBtZHExY1FGbWtN?= =?utf-8?B?YlhNREpZTTFlbUp4V3ZoLzJ1UVFmQlRUMFRZSEtjenpPOUFzRjBUYjlBMFB0?= =?utf-8?B?L3JERVFkbDFxc01XbE9IWlZ3bTRyNU5reUJMdE9sQWtuSm9XcjViM0E4TVow?= =?utf-8?B?cDF5NWhqYi9OQWxZeG13RmVmZVZNRVNsYjk4UFFGazYrZGJKdUlOZjh1MTF3?= =?utf-8?B?SGlzYVF3azRlUUFBaTRZbU96bUc1a2h2MDlxN3Z0cnd4ZjFqek42cWNjSzJF?= =?utf-8?B?dFh0MkZsSzQ0NmluTG9GdUxSdmpFd2xwcWc0L2hvVnYvbUFZaGZtMmxYSUlR?= =?utf-8?B?Q21FK0x2TDNaVldXcFp6UUtDWFA0SjlKQWVYWUplQzdJWHQ4Um1HU3NJZTVJ?= =?utf-8?B?c0J3OTBLZC9tL0lLbU12WXF0MFl0dVNxVm45a25wOGt6VTJTTDIrV0k1OGtM?= =?utf-8?B?NitieTBRNy9aaGxsZ0svWlBBeXgrVmp4U0EvSWpWWWN5S21ITTRzTXRGMWRF?= =?utf-8?B?dlFUQk9yYU1GUDZaK1R1U3BYMUVWY2xDZjljUXAwZTFyV2RkaWcxRGVTS3JP?= =?utf-8?B?S1BxUzVuRHNYc3R4UFpXUkNrTnZhVWErU2lyVXNLY2Ftc3hlQy9xL1NaOHRH?= =?utf-8?B?WFdEdi9zODFYa2FnNWdhNE9QTm9MYjROaUN0Rjh1azUycnR5ZkdKSGM2UjZy?= =?utf-8?B?SHgxc1VkNDltUjlIdjFMWkV3VmVmeERXOEhwTnVhTWpWZ2FaSVhVUnJlODBG?= =?utf-8?B?bzBkSDBiTUpXdHJOdzBVS3lXTXdMQjNVaHl1UERBZ1BpcTV5THFoWjlXR09l?= =?utf-8?B?M1YwdFFsdUoxbzJmNXUwNTJTUTh0ZVVzVzJWaEEyYklMQVYyNmVxOHNyY3pW?= =?utf-8?B?ZDRWVjdDVU5FeU9hR1VVZXliZHowTXNMLzFCSHptbDd5NEwwbGxta0Q4V0Jx?= =?utf-8?B?ZmFkSkdaanlYaGhBcmpaNExZQWs5amlaVFI4Q09mbHB5VmhoaWZ6YXk3QTRv?= =?utf-8?B?NjJXVEJ1SnU3MUxJT3FhbnQxcS9CVGFjc1pTejU5WkZaeGZvR3d4bkRBZWRw?= =?utf-8?B?eVE9PQ==?= X-MS-Exchange-CrossTenant-Network-Message-Id: 768b7866-40f6-46a4-562e-08db522943b8 X-MS-Exchange-CrossTenant-AuthSource: BN6PR11MB1251.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 May 2023 14:08:58.8241 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4Y8cWDG1nuy4XuWr8s6Lto5kY4frV6136F7uYqOdyKK0SVTQ244DZ6vMAed9hsPmRSEkRqwtB+kmO8hxCaw2zfoLWBBPZErU6FlhPZyStc4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB7699 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org On 5/10/2023 1:58 PM, Nipun Gupta wrote: > > > On 4/24/2023 8:52 PM, David Marchand wrote: >> >> Hello Anatoly, >> >> On Wed, Apr 5, 2023 at 4:17 PM Burakov, Anatoly >> <anatoly.burakov@intel.com> wrote: >>>>> Could you please provide some steps to reproduce the hotplug issue >>>>> you're having? It would be great to have a test case for this patchset >>>>> to put it in context. >>>> >>>> I am working on CDX bus >>>> (http://patchwork.dpdk.org/project/dpdk/patch/20230124140746.594066-2-nipun.gupta@amd.com/) and trying out some cases for plug/unplug. >>>> >>>> The test is as follows: >>>> # Run testpmd application >>>> ./dpdk-testpmd -c 0x3 -- -i --nb-cores=1 >>>> >>>> # Bind to VFIO >>>> echo "vfio-cdx" > /sys/bus/cdx/devices/cdx-00\:00/driver_override >>>> echo "cdx-00:00" > /sys/bus/cdx/drivers_probe >>>> >>>> # Plug a device >>>> testpmd> port attach cdx:cdx-00:00 >>>> >>>> #quit testpmd >>>> testpmd> quit >>>> >>>> This gave error at testpmd exit that memory cannot be freed. On >>>> debugging I updated this code and seems it should be seen with any of >>>> the device. >>>> >>>> I see similar test case (without quit) mentioned >>>> https://doc.dpdk.org/dts/test_plans/hotplug_test_plan.html, but the >>>> difference is that it is with igb_uio and issue is being observed with >>>> VFIO. >>>> >>>> Please note the device/bus mentioned in the commands is not yet >>>> upstreamed in DPDK, but patches would be sent out soon. >>>> >>>> Thanks, >>>> Nipun >>>> >>> >>> Thanks, I can reproduce this issue with regular devices too (run testpmd >>> with no devices, bind a NIC to VFIO, attach it, then quit). You're >>> correct in that since the initial mapping was done with mapping large >>> contiguous zones (such as when mempools are created before attach), any >>> subsequent freeing of memory will cause these errors to happen. >>> >>> I don't think this can be fixed by anything other than not doing the >>> contiguous mapping thing, so provisionally, I think this patch should be >>> accepted. I'll play around with it some more and get back to you :) >> >> Can we conclude on this topic? >> It is best we merge this kind of change the sooner possible for a >> release. > > Hi Anatoly, > Can you kindly update on this? > Hi all, apologies for late reply. Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com> -- Thanks, Anatoly