From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-SN1-obe.outbound.protection.outlook.com (mail-sn1nam01on0057.outbound.protection.outlook.com [104.47.32.57]) by dpdk.org (Postfix) with ESMTP id 0A4F11CD66 for ; Thu, 5 Apr 2018 21:00:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=YCWN4/uOUe56zdsYxcTNEmabmcbUAJ43n4M4T/iPIlo=; b=nB59RcPPYGyoHQd3AcaaBgbova1gNUkqfx6hg3Wr9eqN77GfaEyn+zVDABaSndXqT6vy6hhbeEGEAr/PwzTWyrD01+NjjAkPhq/9sLr3ruyvk2raZGR7q63RvlF2RVSZSWsXPyusNEJwHeieNtTgo+JmPNvaA02Ij7B9FCcqb7A= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Santosh.Shukla@cavium.com; Received: from [192.168.0.3] (27.6.209.157) by BY1PR0701MB1896.namprd07.prod.outlook.com (2a01:111:e400:51ab::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.631.10; Thu, 5 Apr 2018 19:00:16 +0000 To: Anatoly Burakov References: Cc: dev@dpdk.org From: santosh Message-ID: Date: Fri, 6 Apr 2018 00:29:57 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Content-Language: en-US X-Originating-IP: [27.6.209.157] X-ClientProxiedBy: BMXPR01CA0011.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:d::21) To BY1PR0701MB1896.namprd07.prod.outlook.com (2a01:111:e400:51ab::25) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 02bfa7e7-ca62-4018-6f76-08d59b27792c X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:BY1PR0701MB1896; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1896; 3:KR5HeqGCStVFf4Vo/jGiUC69zjxTWOuzgUvVKwKQ8xghhiq6UfvvgZht8fsyNYHVJeqFCIKvcsunT4tUkaygR/uHz+HhEb8saAz4JRjcEsV/DX7EKIZ/tpt3YhGhjukeslo5/fblsyHJ0o1QE5RJrqsC7Nzhxrq2lVCytf5EytC1MiJNfLZvQmf/s7pvnvUSo7WaL6i3/ky0+euC4lo+TZNSm1G4Ux3JZWrIbO8TTlqyZKJoUHfOoT0THOC/qgxN; 25:Yw9n8EtrtPqQo4H+7kLXbk6wZSmeGAiIRG8lLqbVllNTTnEcXvKpzDgJXsYUVk4ah64ZnpG1A1tIRAeUwwrzcOLJicsb8oLTJtgCQc9THthRXyHF0JdavGaTFQ4hQ4qmgxyDcOuRpe4bb9sc8jDOfCqO+aWysSHk0cGWEsGwXy0+yIIfsRA2ipUAccC0fptrp672vhWWXZGPZMHuQRLctq6u5XYPvc5/8ag6wgtGzQ1hdhk0dwd5kh9pUlTm9cgdpjOZDcaUFYtaxVcwxHZQ6dzCzMFk3zctjunqAN1p85Jutse83MCqXDcHqe7BQFe/aURbxtjySmk9fOvg6DrR3g==; 31:HyXw/0JZLBor3ZRYWHc7NlGXbEx6q2q/QbggC4YQ/k91QOddO4Tt2kpAA7zkw9x4ZGkcKojWipEOGFplzAgaKL2ziqAv7d98sBNae9gCDnDP318wkZV3/oZwMDozfN6vwXGY7jC/gxrUBpYdJuQtGopO9t/vAxg55chtg8R7P3tESuvztLgSGKllaGLRAICTO1ZF9iyqrSUIRn+Frpdr/XodZmcM59H6U8kG9StSBrI= X-MS-TrafficTypeDiagnostic: BY1PR0701MB1896: X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1896; 20:SkzvFj8XqbA/rXT/xwVy4SynyWUhl7k5DV4K4URMAHMd+VZsCdifiUMhXfZCoKVogLMVABuFM/ifQeXpYtFZwCpeK4VQcxZO+79fwhCkjhOwtM/64hQzif6Fs9brEqHC49TFlwV1Blcat4RvDZz4tzwfrXxstwEYsSCrU6U9sQdN2Zv5C/syY8NgbOyUaYBDmRcorS73ETwnwBsAFnfrcXmxhDlGtvVlKdNalgLspLi78lEYz5BTV1RWsqHS+VyPI60cgS67BHUb6qlulwgfeSZwz6ZljGdssp9CbLSzKV0RyFptNGFM5ywyoMvl7Fhn52dEGRK9s2w+FmREJz3pRkkCFLzy+kE0WX0HtgBSwob5H+SxybJBjN2CTXmrtJFzW+u7xc06z84hAiCYDuMNuDSz8O1OH8wmPrFmATmKVfFgSv/LdxK8EoK70wImFwc4R5o3HI2wsXiE5r5i4mgguIX4jgv4pXC3DRdal9eRT0qCVEYKmH6H96sujnPSVeGRBIQfdTKeBPfzrGDUAdZsG82qSd9dCCuoKVDT0kyoq/h6VpYXKl4+wC0r4ZyxxC1nhq3qKKrtS2zWq61rbCt+WbnDxzCHPCIl0xkCuZF5dMo= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(131327999870524)(17755550239193); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(10201501046)(3002001)(93006095)(3231221)(944501327)(52105095)(6041310)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123560045)(20161123562045)(6072148)(201708071742011); SRVR:BY1PR0701MB1896; BCL:0; PCL:0; RULEID:; SRVR:BY1PR0701MB1896; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1896; 4:QWqgPS7thXYbGtvN4fW5k4ADlbvt6XdaZWeH36ERNyJF9W1MFhh+pHuNWFx1rkjTMB2g8Y8Sdd0fgpWHihGvH4TK8w/7XN9SCGCMJfe2RTJ2PdR0o/4NvCmLicguGjmxofsim3QvaNlOfP6fvD2pn6xOqt7GWjJXfvr8Wb5RHY4EwdmZWcIx3tfNrN0ph2NooopewLfxxEcyrvtzQ9Q23NdofisZHgK9LIadGrbQqJT9jsrDo5lEsoXM3xH3R4cvQH8rDr34KV5V+zJ3OAIC1FIkY+qt8fxPsL/xfWofYDB1P0dlyI6T8Yt3jJ2OFaVMqG+D+oZgt6kyOhz2kKnLOHL6ZaD3xR/aUo/bee74gN0= X-Forefront-PRVS: 06339BAE63 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(39850400004)(346002)(396003)(366004)(39380400002)(376002)(189003)(199004)(7736002)(6916009)(478600001)(4326008)(230700001)(76176011)(8676002)(52146003)(6246003)(5009440100003)(3846002)(86152003)(52116002)(117156002)(81166006)(6116002)(23676004)(50466002)(6486002)(2486003)(53936002)(97736004)(64126003)(81156014)(229853002)(72206003)(59450400001)(31696002)(11346002)(105586002)(16526019)(65956001)(65806001)(36756003)(106356001)(476003)(25786009)(446003)(2906002)(47776003)(16576012)(316002)(58126008)(31686004)(386003)(26005)(2616005)(956004)(186003)(66066001)(65826007)(305945005)(6666003)(68736007)(486006)(8936002)(77096007)(5660300001)(42882007); DIR:OUT; SFP:1101; SCL:1; SRVR:BY1PR0701MB1896; H:[192.168.0.3]; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCWTFQUjA3MDFNQjE4OTY7MjM6SFN4NEJkRytSeXJpbW5CWFBNcng2bWsw?= =?utf-8?B?UEptS2ZWSkUzTURNN2lZTTFIY2JvekNWSC8rTXg5UWFlWllZeVZ5Y21icTdK?= =?utf-8?B?RGFIV1dkYVNoSmhrdnlzV2ZYMUlTdXkwU0dtRkhvd3liVld2TW1oWjRndGk4?= =?utf-8?B?VnNZbDgyRHY4VFZUWm1DWlR3c2NpWWtlRHd2SlRVK3ZReDh1NlpVVjVLQzAw?= =?utf-8?B?Q0JkYzBQSWVPMDlVMGFXcENWNnk4ZkI5bnVuenl6YnM3dzIxMkdJWHZmRVBG?= =?utf-8?B?eVI3dnpLVklteVdKK1ZyZ2M1Qnd2eGNwdHptRWZwaHNvZHB1MTZJaVdXUkhm?= =?utf-8?B?emNnNk0rVVMrMGprUjFKT3pNbGNGSEZyd21OMjFPcjNQaDlSdFBrWnlvWnRw?= =?utf-8?B?U3ZpMVJ5Nk1GRktaK2RlT1ZNbTlDTUc1MmhKblB2T21jQVM2TnFBRlk0SlRL?= =?utf-8?B?dG9EQUVIWXRodSt4bXJzek1NWm55YkN5ZmJ3Z0wybEJXSFl4aHVnaTR5N3hP?= =?utf-8?B?Nnl0eS92bEpTcnhhbU94RTN1TU00TUJjNndOalhiSTZvVHhqQVl1SHdNY0hD?= =?utf-8?B?eEVhYmlwTmNsSE92T01HLzloeDFFdnNKS2Z2L2JxTFJWTmlodnFwV2tWWWdX?= =?utf-8?B?bTB4S29sb0VPOFd2SU5QbVV1dEs4Tjg5dnNMQ3dOcFlpMHRXYlBYelluM01J?= =?utf-8?B?MzhORFRrdCtTd3huMHI0aWNBWVQzUi85U0Yzc2ZsRWx6S0tjWWUzbnQyRVRn?= =?utf-8?B?NG5PU2tGeGhFN056WmVSWDFyMVNFd1pNaldqakwya1ZkRzV2Y2dPeUNZRWFX?= =?utf-8?B?ZDJ6VVFPclh4d2s1SnF0a0QwZE9ncFJoWCtQbzRIUytwNVQvWm1YSnFoZnhJ?= =?utf-8?B?TENaNzh1TlBueGo2YXp5bGpGemVUbzFRNlY1cVZVT09nRDVYWUU4WnYrOUU4?= =?utf-8?B?TGJNVCtSRENzYXBlbzdnNWZ1RnFSbnhzRmpNS3M2R0xzWkFUWVZIYVdnUWNJ?= =?utf-8?B?VGkvZDA1U0pBUEtibWRNZHd6N0cweXJDVVltOVQvQkdYY1ZaYzRtR3N4Ly8y?= =?utf-8?B?L1IzdlZXWTEyQmNIT29GbEhlRGp3Nnp6d3R6V0FISkg4cmFZeE1oVFltdE9u?= =?utf-8?B?eGtROXo4K3VsT0FjM1ZYbm8yUzdtcTdiSlQxT01PNVI2NGh4aWg5blBIRkQ3?= =?utf-8?B?cFE0SjVxWExUR2VzeDBQRFQ4ODNYSkVPUDJ4SDllZkp1SGVwNkVVUnZiK1VU?= =?utf-8?B?dUY2cDF5VElqYWpKZURUY29NTWJPTm1tR0NLOHR2VzdoMzlkQW8zVjlNSWhw?= =?utf-8?B?dXVRZ2NQMmJZblJZU1JiSDRkU2RRYUsyWDRhWTlIbWdwUWo2d0piOUJjYWxa?= =?utf-8?B?K0tabzdwRnhuVnl6dGhlYzJtOThKSG50R29ZODdOR21yWUVpS3FGQTJXWnh4?= =?utf-8?B?SFI2b1h4dVhydjlFMG1uclRUQWF5cVFCc1FzUDZ2LzRpaVBuNUVDOGxQNjcr?= =?utf-8?B?VGttNWlLTmt5c0FLKzdFaC8zNzhieGJHZkRxR1hXL3pSV0N5dTNEMlJXbDRs?= =?utf-8?B?cHRZWG9HRDVIOFQvWkFBcGNFcXowZ1V5UVpxUDdCaGg4NklwcEIyQlhoVWF5?= =?utf-8?B?YUxNTjlSNUQ4SkRrWUpsY3NaZGhuYS9FZUU4ZlFnaHFzL2J5ZDBlV21RNERw?= =?utf-8?B?S1hyNHF4KzBUOE5EWDhzZHl2QkhVb1M4WE5iaXRRV0N4K0NDMldmcnIyS1d2?= =?utf-8?B?aU10OHk5ZFpuVjZxRGFLdDBxWEZNTUhZU1NTUEFmUGU3b2tHdWdEU0NMbW01?= =?utf-8?B?VnFraStmRGhkMGRqaUNTOUVINGZzdXNtQmk5citaYi82bVQySEVlUVp1M1Jm?= =?utf-8?B?Q09tZWVGTFpCWVVZY2ZGREtqaFdzRERHaHN5d2I5alV0MHVoWU1MQi9JRU1J?= =?utf-8?B?K3hlVUNCd0IxYXVQV3FRZHdta25Qd3E2MmZKRlJ6WFNIWmlhMStGYUdYeHBD?= =?utf-8?B?SWFxRXlxdkd0dTdrRDlmb2k3QXdoendGSDduMEE5ZEluY2JJSUMrMm5UcFpy?= =?utf-8?Q?C9D1kQ=3D?= X-Microsoft-Antispam-Message-Info: pd8oDMg4AZasDnm8TZWzR435kfQSTwK44z6WZ5XdNbIPln6BszY+NyMoP5GGvZGAFcuSx6OvYO6O7cI6Gk6sSbcsESmkRB7drmg8V9cf1AiD74FUNUiu9Bg8Zip1njABxKHNGcu+O5HbJVnE7m9mFG56FBjOmPFCribv2jPEXmKpgCynDgjEQU1eegfz+GpQ X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1896; 6:/4JwECveczK5E4eJnE/lzT2it+myonMV9UQKgZlsOyy3e3kyjuGZA7iccoSn9mc2aF2lWhE852vZa6Cux194gqmXD2t7oplLOdWFTNsSaGFelH+4VCjUXiGA6u/RR5bXVsaUpBMYG8BbvEfsG6Ywls8gdE5zDszCDAVBkEvq+REjpg3l7QzQBl3sflqr+YrcV6ilalAXIcXSHB9DxxXpad/lU96Aq21XFqFnn/aZosmVSQd5n19obtsMbPeIx+gOygETPOGR8QeTy6BlCAxUXgxCmeaRQ+54VJy6G7zX6o/JW7oLv+bLAGbikx6A3FlG/609i0MHXMK7z9KLNR5tJXthmPlOx8bHElTuWJTOMevhAFa2Kb1Q7/nnftTPWBfTOXrl/rM4s7AeVH+zra8I5QFYidH53NOfxeT/NtAMqj29wJuULPp1mKMv5alYqXbYsjyK+wMrNI4KiuXoDWR/HQ==; 5:OyK2F+58Q0k4o/tDc7A2MQuC/tf+YkezxI/lhzA+WiYCaj3pyk7HWGialQJzlwAbnglFCjAxgd824JxduLvepLcHcWTYXw/MvafSNkRmpUWsHCSzNROZkCexo3OLyUZWdhseEndZAyzItEiJGR/tFnThqGdaOVleCzo+zg3FBcQ=; 24:74U3y5ftQQwo5iUmUnkDo9aV3q+TQPyHx2XXJvQFpvkE0EwwhOnCrhYwheFikFB3sZjvne7SJ1xAfGGG6nhKqOnEAs6uAA0+lfxFSdUaCqU= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1896; 7:YVswh1NSOOUKZoheS1cQbYSP8xtzxtru6vNw3LqJe3axk0mG/+fzVsrT8yoK0IjeOn3g1RIukg7JfQ/dIpG2OAXJMJCW+mMSzE/5TwTqU7ycnNvxjLWnmS6B9GHDrElIHKL2T1WlPwiv/fIjznZdR3/S/z8hkcpehHxGHj0r2KP2DOlWJKAmu3yUFBY1vkUr8GxBwVZgQT2gpDf/zdSJiydyJwMxocD0wqtY3R7jHicyeaSqVqOvcdK46K/k8x5s X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2018 19:00:16.5371 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 02bfa7e7-ca62-4018-6f76-08d59b27792c X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0701MB1896 Subject: Re: [dpdk-dev] [PATCH v3 00/68] Memory Hotplug for DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2018 19:00:21 -0000 Hi Anatoly, On Wednesday 04 April 2018 04:51 AM, Anatoly Burakov wrote: > This patchset introduces dynamic memory allocation for DPDK (aka memory > hotplug). Based upon RFC submitted in December [1]. > > Dependencies (to be applied in specified order): > - IPC asynchronous request API patch [2] > - Function to return number of sockets [3] > - EAL IOVA fix [4] > > Deprecation notices relevant to this patchset: > - General outline of memory hotplug changes [5] > - EAL NUMA node count changes [6] > > The vast majority of changes are in the EAL and malloc, the external API > disruption is minimal: a new set of API's are added for contiguous memory > allocation for rte_memzone, and a few API additions in rte_memory due to > switch to memseg_lists as opposed to memsegs. Every other API change is > internal to EAL, and all of the memory allocation/freeing is handled > through rte_malloc, with no externally visible API changes. > > Quick outline of all changes done as part of this patchset: > > * Malloc heap adjusted to handle holes in address space > * Single memseg list replaced by multiple memseg lists > * VA space for hugepages is preallocated in advance > * Added alloc/free for pages happening as needed on rte_malloc/rte_free > * Added contiguous memory allocation API's for rte_memzone > * Added convenience API calls to walk over memsegs > * Integrated Pawel Wodkowski's patch for registering/unregistering memory > with VFIO [7] > * Callbacks for registering memory allocations > * Callbacks for allowing/disallowing allocations above specified limit > * Multiprocess support done via DPDK IPC introduced in 18.02 > > The biggest difference is a "memseg" now represents a single page (as opposed to > being a big contiguous block of pages). As a consequence, both memzones and > malloc elements are no longer guaranteed to be physically contiguous, unless > the user asks for it at reserve time. To preserve whatever functionality that > was dependent on previous behavior, a legacy memory option is also provided, > however it is expected (or perhaps vainly hoped) to be temporary solution. > > Why multiple memseg lists instead of one? Since memseg is a single page now, > the list of memsegs will get quite big, and we need to locate pages somehow > when we allocate and free them. We could of course just walk the list and > allocate one contiguous chunk of VA space for memsegs, but this > implementation uses separate lists instead in order to speed up many > operations with memseg lists. > > For v3, the following limitations are present: > - VFIO support is only smoke-tested (but is expected to work), VFIO support > with secondary processes is not tested; work is ongoing to validate VFIO > for all use cases > - FSLMC bus VFIO code is not yet integrated, work is in progress > > For testing, it is recommended to use the GitHub repository [8], as it will > have all of the dependencies already integrated. > > v3: > - Lots of compile fixes > - Fixes for multiprocess synchronization > - Introduced support for sPAPR IOMMU, courtesy of Gowrishankar @ IBM > - Fixes for mempool size calculation > - Added convenience memseg walk() API's > - Added alloc validation callback > > v2: - fixed deadlock at init > - reverted rte_panic changes at init, this is now handled inside IPC Tested-by: Santosh Shukla