From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-cys01nam02on0045.outbound.protection.outlook.com [104.47.37.45]) by dpdk.org (Postfix) with ESMTP id 09B1AA49C for ; Sat, 24 Mar 2018 07:01:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=RbihUNYMZZEoFV+D4Qu5wrP0z4psLIG27UChbEkk6ug=; b=XX9WAzoRr13kSCCfXfbXhgmqV3K4AHDSb9Ma74dfzKqMJON+iP1OmrJlOvBbdA20yufPjaPlEPJeb7AknPJDHSYF84MqnyQ6Km3J/yR3h+d50p0w+sXB5OINrghkZAB7WrJgFyizFmLgR6vj/srUtwi1Nj8sPgLtyx/tP3uwrSw= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Santosh.Shukla@cavium.com; Received: from [192.168.0.106] (103.76.56.167) by SN1PR0701MB1901.namprd07.prod.outlook.com (2a01:111:e400:517b::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.588.14; Sat, 24 Mar 2018 06:01:34 +0000 To: Anatoly Burakov , dev@dpdk.org Cc: Thomas Monjalon , Yuanhan Liu , Maxime Coquelin , Tiwei Bie , keith.wiles@intel.com, jianfeng.tan@intel.com, andras.kovacs@ericsson.com, laszlo.vadkeri@ericsson.com, benjamin.walker@intel.com, bruce.richardson@intel.com, konstantin.ananyev@intel.com, kuralamudhan.ramakrishnan@intel.com, louise.m.daly@intel.com, nelio.laranjeiro@6wind.com, yskoh@mellanox.com, pepperjo@japf.ch, jerin.jacob@caviumnetworks.com, hemant.agrawal@nxp.com, olivier.matz@6wind.com References: <495e60f4e02af8a344c0f817a60d1ee9b9322df4.1520428025.git.anatoly.burakov@intel.com> From: santosh Message-ID: <15b7a198-2963-30e1-9d31-abbe01dd62d1@caviumnetworks.com> Date: Sat, 24 Mar 2018 11:31:21 +0530 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <495e60f4e02af8a344c0f817a60d1ee9b9322df4.1520428025.git.anatoly.burakov@intel.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Content-Language: en-US X-Originating-IP: [103.76.56.167] X-ClientProxiedBy: SG2PR06CA0128.apcprd06.prod.outlook.com (2603:1096:1:1d::30) To SN1PR0701MB1901.namprd07.prod.outlook.com (2a01:111:e400:517b::27) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b00f93ac-8c38-461e-8bca-08d5914cb69f X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:SN1PR0701MB1901; X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB1901; 3:7Wqdlau6woAdlsgI2Sx2GJum54y7Ne4npt4O/EdBzVV/PMohi8usNH56VFmPfGu614gVPRC2yBWXfl0DryruZYl3VgIs7UdDrScshhAyDoUpGJj/kW4QeAxY6FR0D2DfdZNWeSQ2cyIYl2xY6XnhvMGZt9A2RaxHsCYjGswL/ejwWAqx2DvbPltXJwAuUyxOgSc9gWM/E9Rmtg1W33s3C7jOVQSPXDLJnsOlcHT1E+Um5f38ooF6PG97mXV4ecxK; 25:uZlOVunF15lc11zTGnBAR9ePTL0kcJ+Tl5X2rr0hkgrT38Kso5SJ1ICcYn612B8aFyXRwt25CNEDfxMHUD1a9vew1/1J+5nH4oqahYKEF3StqLRfBPiTlyJSJefBMzFN/AAUVYKr1ccY6D7eBbVak+gP8rDGGxMSY6jptS9P+13jIfphQw3GdUgSVHn8lPW1xGF2nxw1SclCptQCa+LNqHRycFhWiWHm1+NJTDE4b47RzyE4byZ6tIesP1svlKzCSQ/lfqjaI227kS1yIZwNVzyRWxXVrpt9gUhmw3bxaxvDqNIg5ItjB6+h9oOfBcsrSFsVfbhndqNqA4H0HFqWsw==; 31:wydeVR6dy6svtkFletJMBq2NyoWmVABc+Rj08osOMClMs+b9zSdv/BXLAFwUPtF9qdjrNgLvrSTnvwhnR1oxBsihQ99AS1ZKpBP958e4F0QUE2OtCwJw6If1kTLw+pLKSAae3GfflBFfwdA24GHJ0jeH0y+htfIpaNYISYbQ7NC9Q3of78WeSpGkyPLwbXpSTji4Mwsne0kkrk/i0BQdbfiJQCPn+GS5KpMq/jh2Z30= X-MS-TrafficTypeDiagnostic: SN1PR0701MB1901: X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB1901; 20:kBHXxuzQq4PTHYy1irhItKbWkdLvyDN2lkmIN8t3HQ9OU14CpiPPCJcH1cb5HOtwA9e5YWybhOTUngn5UxKNavKTXF7LxZ6+PFTVdKA29MDp9eQx5lshndgrfyOA5T3XAcImurK2EEFFjpyf4gaZrGdYCh0ND0V8b4pkvEnfuXlYBF4c4uxJ33QW9iROW0oYaKuQroi7XW64Iz/EBflr25HrTjS4x3R60BCi2kwmNJN/APVbAds0cF6Ys8Yflkz2Euh5hGaYqA71D/+rf5UHpkKyY6nP/KCtY3hFEJIWhzGlYXuHqUSZa4vjSyp4R/yyheAY/MdabtHfXjszoGdver/DAnDlgqfpBUhjl+bx2kaTtTnaRtkWfa7UGV9V0oOT4afQnUaZ167HGCQr9Wvfd4WvKbUcii/XApb4Ns8xmB2nYqRvN53dB113D4hCHvKOegNVRsR3kBg1Lf4ZFszJqEbi/buH0r4TM9Wm/2Hiv+UJHfKvMRygPlK35ofaJq4mr65oOxZ9e8srzocDKjn9tX4PEPsvUgnUu2DtxZI9ab0m/mfd6GVfFddviSJuVLraXX2zY5NXSd9Z3klAGvxhK9O1P8+2hmDpMQY8FdsJh08= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(788757137089)(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(3231221)(944501327)(52105095)(93006095)(10201501046)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123564045)(20161123558120)(20161123562045)(6072148)(201708071742011); SRVR:SN1PR0701MB1901; BCL:0; PCL:0; RULEID:; SRVR:SN1PR0701MB1901; X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB1901; 4:GwMR2locHkm/3zfdNmiKLFqEQc7VaDu8B0VxOTtyhmJIDquD6yJSyZVvvvqm+KSm28gyCkerveCfbIlWtmT7lfvW0RyLLVGQw2LoGrnOe+w2JioR+mTfTGgVXnxTdIqEFC2xKmtjb6xGxDSpPiHqYiHobhe/HkoPiGPiPOU8phzefPFV+faatIr3l/LKElWh47y2uqtLR6GQcg9/FndFl9cAQjz+UMxG2WLYM5Xm/VSqTq3jODJbztcKEr/CMXzoJUlpFXC9VWwU8P82N3tHM8t8Q0XIcrem5c6wrXzM2FC4dGNTuZeqBxBRK7zmdmX/JaHHVIfBR09OpwpRNrKQtpgtbdnR0yaFy9jmGmYFs6k= X-Forefront-PRVS: 0621E7E436 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(366004)(396003)(39860400002)(346002)(39380400002)(376002)(199004)(189003)(575784001)(3846002)(6116002)(5660300001)(386003)(59450400001)(68736007)(229853002)(64126003)(186003)(16526019)(26005)(106356001)(50466002)(77096007)(65956001)(6666003)(47776003)(65826007)(2616005)(956004)(6486002)(65806001)(42882007)(66066001)(230700001)(8936002)(76176011)(23676004)(316002)(36756003)(16576012)(81156014)(81166006)(11346002)(8676002)(54906003)(446003)(97736004)(105586002)(31686004)(8656006)(58126008)(53936002)(53376002)(31696002)(4326008)(25786009)(6246003)(7416002)(86152003)(6306002)(478600001)(52116002)(2486003)(52146003)(72206003)(305945005)(966005)(117156002)(2906002)(7736002); DIR:OUT; SFP:1101; SCL:1; SRVR:SN1PR0701MB1901; H:[192.168.0.106]; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtTTjFQUjA3MDFNQjE5MDE7MjM6bk9aM3BSTUk4Nmk0SzhtaFUvRHRTbHc3?= =?utf-8?B?d0h1TFg5YW5tWUlSbU84QUVObE5nUDd1M3BSeTNpUk5hY1VUell5dHF0Mk1J?= =?utf-8?B?L2QyTEZQTHNvN2xVaDRnNG9zak5jQzVjbW1nNEJ0TW9OU3M2K2p5d0FyaS9r?= =?utf-8?B?ZEZqbUg0eDFpNFE2WVZySVJIT2tCblFRU2h3WWtVakNIWFp2Y1pKcWdzdjEz?= =?utf-8?B?eWJLRXdZVVpYWUZtZ3dQRktZWFVCdkxaN0JlOXBscytiTENEdmFTV0pLR1g4?= =?utf-8?B?cm5hT3lQZmdQbEUxNVQzdytZNktUMUU1bk9Ga3c3QkJmcENtQ295NDBhZGJ6?= =?utf-8?B?VnZ5aHZsc3Vqb0JvYjgydWR3R1c4Y2FuUTY0a2U4MFkzSGk4cW5McER0TUdw?= =?utf-8?B?RjFmMXhtUVpVRWpHbk9TREJsYlR4TWJpaGRSZStPblB5M1p3WDQrM0pLeTgy?= =?utf-8?B?SEIxRVZuVUpBTG1wME40K3lvR2VlSjRaREt0Y2ozS205S3R3OUJ2aXdSMjFz?= =?utf-8?B?amp5Z2hWbHdlOSsyMkE2RENXM3VqWW1xZmx0cVFkR0hzczRNR1pGdmE3NW5D?= =?utf-8?B?STZRVjREd3RXTGxIdFBrUDZHL0xqdlhMT2pBWjBIS3c5OU9iM2sxU21YeG5a?= =?utf-8?B?NUo1eW1jVTJXbkpVVTE4NG1oVndPODR5eFJKbjdzTEZZRjZESlFzZS9oVERH?= =?utf-8?B?MjlzejNNSDE4TWFFL2lFZGJuNkdnMytwSkxiclN6N1VkdC9NcG1nSGEyYlpz?= =?utf-8?B?dXhLN2F1SndhMDVrL1cvWDE4R1VOTG11eHgzREJDbnlMMVErUDdtOWFMRHpI?= =?utf-8?B?UG96eHA3NTNzdDZmNGJJbkZMUmRDTVkzeUFsYytVS0VQc0x5L1Q4Q1g5NWxK?= =?utf-8?B?MlFmN3dhZFhrSWNndWFZMGk0UGVNM1YwR1BwQStNZzlnVGRuN0pOVnAzSXcr?= =?utf-8?B?NVRJNlhOdTEyT2R2amFxL1J2VEUyelVueEhmMzg0UXZkTlVyNFVHZGVkbDZ4?= =?utf-8?B?dXhUZ1RSMXI3Z3BjWVhWSmovd3pRdjJrRW01TXlCaUkzQlo4QXJ5RFN0UmUv?= =?utf-8?B?U1BtaWYrYjQ2dWY0d000ckQ2aGRMYWNYR3ZMV0VxSW04dXRoNnV5WHdLdHph?= =?utf-8?B?S25ON1d0RkxXUCtyUjB0elMrOGJuWHJOUENCN1VVS2NxaE9LODhlaFBjdjV2?= =?utf-8?B?eW9NWWVIK0JETFFjM1dJZG9ES3ZUakxkN2FXekNZSnhVNDJtM05OZ0d1cVly?= =?utf-8?B?UFAzQTdQL3oxSFBrN0lVNGs3WnV3dWNCOWhHbC9qQlVHMGEzNTJVbE9heU96?= =?utf-8?B?S3EyOUFzemxWVGlIeUlZZXI2amZGa2xQWHpiT1dJcE9OL255UEVkbnQyUEZ3?= =?utf-8?B?dVRnci8rQlhmTEs2RUhqbDR5eXVZc3NGY1Z5MmNtSW9tQm5SWTRNNHMxOCts?= =?utf-8?B?R0ZRUXBkMFk4ZkxQQkVIbU5NQU5aN1BUS1hQZVd1eDY4cHREdXRyZUwzRzZ6?= =?utf-8?B?bkRxb2hKTCtUaTZqU20xRWZCRm1SbEtRT3FPaW4zZG9VTStFSmdOaHhjZVdX?= =?utf-8?B?NmdmS1ZydU1DdWhMa3E0TU1PZmFWQktxdEQ2My9LZHZWWkEzNFhKcHRhVlBz?= =?utf-8?B?U01EVnd3S1pTVjQ5blp5S0RzalhtSElJQkZjUS9qUSt5VVF0c1hYUWExOG9l?= =?utf-8?B?bnRrYndNd01adFh6SzZHaU4zNEkvcUZSQmxBTVVzd2swZWc0cFVZN2RjM1o3?= =?utf-8?B?Z3oxeWd1MVA2ZVRVcW5IU3FCejNkM2Q2ekw3R0lPVnF6R1ZtWmZvcmswdndh?= =?utf-8?B?Tyt4aUp4cmFtS2NPMzRFY2JwTHNGUzVveHVMTUtySGZyWG1QZkVqK0c4bWwr?= =?utf-8?B?RzVwdUVaa3JuYnM2UFJqYk5PaTkweEwvSC9PRVA4Q3Q4N09uUkhIN2pRejRG?= =?utf-8?B?UGFsQVpRVEV5MDgwOUVBVGhSZ0hVVUZJU3pFd21LVEVpYjBtNDlkTXRxZENO?= =?utf-8?B?djh5R0h2eGMzV0JucmtpSlM4dXNZTWp5ckJoRy9pT3lyK3ZneUtCTmR0U3FD?= =?utf-8?B?c1hKbll0QmtxVWM2L20zTjdFU0liREVJRDJlU1poQk1UdXlOb0dxYmxDSXlr?= =?utf-8?Q?fjaRSpzUjleIxxQSA/wiLXDdc=3D?= X-Microsoft-Antispam-Message-Info: 3Nw1Z4OB3rDH+bEvA/qv+Xhhyx6J1zN6c2QGsqZ86kKZxrQgMiTZVASlYCJJCl9UUbxunK2rGm8zwj6NVij/7wvEc7Xeo+dFe2X1MrcO1S7reU8ExTm0szHsK99toV8nT9mXzZq7NB69POS6Lc1YHhMDGPeTmLFan6JNeA4DqIfiZiyxGYKkr+MBrPKhd2zO X-Microsoft-Exchange-Diagnostics: 1; SN1PR0701MB1901; 6:bYvJMYAbIgEMeX/wXxiDL2U/7ASV40o8wqovKRsg9iElf6kzXOIoxyUA79kK3l297JzT/tS+zXMbMxcmovN4vTuVpPwMdlWR/Ar3izRb7YtBP4e30c99JqUyDH+ErRQBAGaQAT7pnLGE0wDaLAmR8jzIpdFHtg3iAPsRP/PIdSS7CK29igqoXGBvHNikHkLAQ958FBBUYqrALb4aqowrWRhSslyXCLnolcYr8fIkjp3ksQNQcJ3tMxBrD6gfLPh4s6SfZPIb0hGdTbIN49lCEtU5c8+dhN1uSIwjvqHlxNaHhHhM4yyADct3IRWytRfEg0dMZuHprV6XcA66Xt6v8ktrSSTq/m7i3N3J36u3NL0=; 5:RKLu8wXpYvN14KLnedY3BJLsCTjNG0paqOBrISK4K3/1jTEuTEKb8gBko/nIeZRw/cXelnTG7p5Jw94ZJx7ANBnzQgPC221lCyZGnYadsxa7TkoOuTbFFOCHx6zNLOopNLrUMP7nQcNoMDn4AKfco3smHlUdG7egcrPqhNXrAOc=; 24:xkExsUY3GQ/LaZqibcY536yliL4I1ZXBSGD0Z1+A0AUkhBJJPr5X1aa9hDhVWB4q9FF1ZM2ibzbpfAmCfAdBFcc0/Tq0Btj+RvB2qn+nJMc=; 7:Sa2tXYFX7rXcAiURZgVncyzLBNMcCInv448MLMj8kwAtJ7Ao1hyftlaR7P5XYv8eh6HK6jifrrUizzxVwpxzxjQu5joG+fWfjCYyuWzSRUORecUcJuV4Hhtek/V9DIp26Thl+u/r8dJNnaQCQAbeN1YIJ6xY1d2Xy9Z50KYOu3gWDqmgsMj8WaVGb9HNJ97ImO2yx0Wr2TFzDPiBIFfEFbEu48orCM0cLcl964ExbxcQkD4Agw5pbmV6Wq7rJfij SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Mar 2018 06:01:34.0705 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b00f93ac-8c38-461e-8bca-08d5914cb69f X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR0701MB1901 Subject: Re: [dpdk-dev] [PATCH v2 13/41] eal: replace memseg with memseg lists X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Mar 2018 06:01:47 -0000 Hi Anatoly, On Wednesday 07 March 2018 10:26 PM, Anatoly Burakov wrote: > Before, we were aggregating multiple pages into one memseg, so the > number of memsegs was small. Now, each page gets its own memseg, > so the list of memsegs is huge. To accommodate the new memseg list > size and to keep the under-the-hood workings sane, the memseg list > is now not just a single list, but multiple lists. To be precise, > each hugepage size available on the system gets one or more memseg > lists, per socket. > > In order to support dynamic memory allocation, we reserve all > memory in advance. As in, we do an anonymous mmap() of the entire > maximum size of memory per hugepage size, per socket (which is > limited to either RTE_MAX_MEMSEG_PER_TYPE pages or > RTE_MAX_MEM_PER_TYPE gigabytes worth of memory, whichever is the > smaller one), split over multiple lists (which are limited to > either RTE_MAX_MEMSEG_PER_LIST memsegs or RTE_MAX_MEM_PER_LIST > gigabytes per list, whichever is the smaller one). > > So, for each hugepage size, we get (by default) up to 128G worth > of memory, per socket, split into chunks of up to 32G in size. > The address space is claimed at the start, in eal_common_memory.c. > The actual page allocation code is in eal_memalloc.c (Linux-only > for now), and largely consists of copied EAL memory init code. > > Pages in the list are also indexed by address. That is, for > non-legacy mode, in order to figure out where the page belongs, > one can simply look at base address for a memseg list. Similarly, > figuring out IOVA address of a memzone is a matter of finding the > right memseg list, getting offset and dividing by page size to get > the appropriate memseg. For legacy mode, old behavior of walking > the memseg list remains. > > Due to switch to fbarray and to avoid any intrusive changes, > secondary processes are not supported in this commit. Also, one > particular API call (dump physmem layout) no longer makes sense > and was removed, according to deprecation notice [1]. > > In legacy mode, nothing is preallocated, and all memsegs are in > a list like before, but each segment still resides in an appropriate > memseg list. > > The rest of the changes are really ripple effects from the memseg > change - heap changes, compile fixes, and rewrites to support > fbarray-backed memseg lists. > > [1] http://dpdk.org/dev/patchwork/patch/34002/ > > Signed-off-by: Anatoly Burakov > --- Thanks for good work!. Few observations: # Noticed performance regression for thunderx platform for l3fwd application, drops by 3%. git bisect shows this changeset is offending commit. I'm still investigating reason for perf-dip.. Would like to know - have you noticed any regression on x86 platform? # In next version, pl. make sure that individual patch builds successfully. Right now, Some patches are dependent on other, leads to build break, observed while git-bisecting. Few examples are: >>fa71cdef6963ed795fdd7e7f35085170bb300e39 >>1037fcd989176c5cc83db6223534205cac469765 >> befdec10759d30275a17a829919ee45228d91d3c >> 495e60f4e02af8a344c0f817a60d1ee9b9322df4 [above commits are from your github repo..] # Nits: Perhaps you may club all below comits into one single patch, as changes are identical... that way you'd reduce patch count by few less. 9a1e2a7bd9f6248c680ad3e444b6f173eb92d457 net/vmxnet3: use contiguous allocation for DMA memory 46388b194cd559b5cf7079e01b04bf67a99b64d7 net/virtio: use contiguous allocation for DMA memory a3d2eb10bd998ba3ae3a3d39adeaff38d2e53a9d net/qede: use contiguous allocation for DMA memory 6f16b23ef1f472db475edf05159dea5ae741dbf8 net/i40e: use contiguous allocation for DMA memory f9f7576eed35cb6aa50793810cdda43bcc0f4642 net/enic: use contiguous allocation for DMA memory 2af6c33009b8008da7028a351efed2932b1a13d0 net/ena: use contiguous allocation for DMA memory 18003e22bd7087e5e2e03543cb662d554f7bec52 net/cxgbe: use contiguous allocation for DMA memory 59f79182502dcb3634dfa3e7b918195829777460 net/bnx2x: use contiguous allocation for DMA memory f481a321e41da82ddfa00f5ddbcb42fc29e6ae76 net/avf: use contiguous allocation for DMA memory 5253e9b757c1855a296656d939f5c28e651fea69 crypto/qat: use contiguous allocation for DMA memory 297ab037b4c0d9d725aa6cfdd2c33f7cd9396899 ethdev: use contiguous allocation for DMA memory Thanks.