From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0048.outbound.protection.outlook.com [104.47.42.48]) by dpdk.org (Postfix) with ESMTP id BEC951B2AC for ; Fri, 17 Nov 2017 22:26:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multapplied.onmicrosoft.com; s=selector1-multapplied-net; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=F7WE0I0EeFQN85I61QuAn5msrYoVf8A23sDRQhOqb5Y=; b=gVwglvaEL+MnRVWQj95pC0A5VNhOHIxvHHQWBVDSrz9V1ouXO2x+L5fabUnE5Pj7Zxs/s7iZZRFeD5reHIm5+maK+zlkOuotD7IbJb3LrAKVuc5mD9mOw2D5cTEuuLesHFU1Gg4ZcK8VfsrrZiQxBIe5Ith1a6cP0b8gX5aOVj4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=ian.trick@multapplied.net; Received: from [10.100.0.160] (74.121.35.88) by DM5PR17MB1978.namprd17.prod.outlook.com (10.173.129.12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.239.5; Fri, 17 Nov 2017 21:26:14 +0000 To: "Dumitrescu, Cristian" , "users@dpdk.org" References: <3EB4FA525960D640B5BDFFD6A3D891267BAE4DD8@IRSMSX108.ger.corp.intel.com> <3968f1fb-0969-8cbc-3155-d5862d74560d@multapplied.net> <3EB4FA525960D640B5BDFFD6A3D891267BAE5326@IRSMSX108.ger.corp.intel.com> From: Ian Trick Message-ID: <3c8c9b45-92fc-defa-f52e-4bc809cd155e@multapplied.net> Date: Fri, 17 Nov 2017 13:26:11 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <3EB4FA525960D640B5BDFFD6A3D891267BAE5326@IRSMSX108.ger.corp.intel.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [74.121.35.88] X-ClientProxiedBy: MWHPR06CA0005.namprd06.prod.outlook.com (10.174.172.146) To DM5PR17MB1978.namprd17.prod.outlook.com (10.173.129.12) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e699b15b-66a7-4481-03b6-08d52e01d540 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(4534020)(4602075)(4603075)(4627115)(201702281549075)(2017052603258); SRVR:DM5PR17MB1978; X-Microsoft-Exchange-Diagnostics: 1; DM5PR17MB1978; 3:XutT4ueV2lH6q9HdBCbqxU2hdOklzha53miW6mcD7JL+B/SQ4mc5//GY3LgqCzfx0NJ8q8ifku0rF6Oa0OnboB3ihfPNeg58GP7/83aGGo5243Vik6LXeVfdHk23pOI1WkM/mhzXxbWtule7I3cwGhMsZmo40SWPymCa9LSzkJURIKJBSbzn6h8eEShIgxXw3h5JpHnO/MO3J2jKk6Drezy3F2gZqYGdiKfOz+yUonIpCW0Nen+wZM5fwcGJ1Y0o; 25:tCIVb7dPNoD3wmH+i1kPNJnhOkObRODF049ZHxq3UH1DjoQYg0HWrFycU/+zn6+5+1pox+kzcLKo/RWZww+Rev9hD++vncDwYqEIarm39Yjk07oV6sgpqbmtAhaVU54Qs+HkunILW+MOeIzobkcdTESLy80D7glPPDFb259Uv0AwUeY6ADalvDBa2gXHWM0wx+n9/m3IuaCd99hAKso24w71xdx9+usB7x98UyQIOucldjoVJl35IJeOyJDP+JRQFLAQ1imC26uC4gnwWaIqs/rdvdNXKDL9FYzduXacVhK2/EFAE6z5iTBKGvLwRAo/5lak1YIo13pjWRuqaFwkkw==; 31:Rwf3c9Wap2qKP8tceJ5tgXdqbapHC3DouWqq62m+lmbz51VB5ax2BEiLJSr7+88PpVxqf402OYpL9Zl6nc9/b4qyZCV+/AyY70QgVcVXn7ZjBSnDv6tgIKIiaCT6Qxhtfp/0LQ6liyRnK7CFS8fbt4EVPLe4ofMYz/ZgesTVZvNMnoCNfQ7HSgIYNZ+P4E6HXmE8v1nMdL++1e0bfWFoN3q4u6YoyDbWyyQkK4hg0Qo= X-MS-TrafficTypeDiagnostic: DM5PR17MB1978: X-Microsoft-Exchange-Diagnostics: 1; DM5PR17MB1978; 20:Pe00rgulaZaUrqe5OUrnWHywJWlEk7AqrlUVRQXhYyjJ/LHWNSe/F0VNoL4G78g9D35xMhgaINPNHTjUl7Y91xsQLiT/SN/Fv0e39CdbcPb52h0lI5IIDLhQh6K2eDqriFLHUsjwZ8ELCbw4NOVo8p5T63hVPvmJ7NZ5l4sLYezleM/cULCPAIgiJ7F01BC+7vH7UWXb6q8fw9um27V0OIunt4QdFMdeBi3J0uqCh5HxYf7N25iA9aGQUyuz4/wI; 4:ngSucYwv1xq3NZT5hid6KSosMvlXeLzPQlP6+zoNRkJsR3ApHEKlHNQC1q44eOR3Uq7SafqxsSd03K+9Ce/agLmBHmKyAgsSB2eZ6J/qV6Mzvl8nggvmqmHlbG7c4mvVGNIm/F0OY7p7R7n8zH5VjgWzjMMkOIX7aJcFLzLHdGtLFWGyDE3vjE6ZLEohCgVTzT4SCLVan444uv5ymWO0hqgtNYudtA0y+53fVfxyuonBlben4o09ykqZBeaedN17And6LZ6jUSQz8c9/JvfbC7PppMgRU/HdqJX8oAniN5dXHhMw66KuHFnb1iH+e7BW X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(100000703101)(100105400095)(3231022)(3002001)(10201501046)(93006095)(93001095)(6041248)(20161123558100)(20161123562025)(20161123564025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(2016111802025)(6043046)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:DM5PR17MB1978; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:DM5PR17MB1978; X-Forefront-PRVS: 049486C505 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6049001)(6009001)(39830400002)(346002)(376002)(24454002)(377424004)(13464003)(199003)(189002)(31686004)(86362001)(2501003)(6666003)(2950100002)(81166006)(81156014)(64126003)(230700001)(110136005)(8936002)(8676002)(7736002)(97736004)(31696002)(93886005)(189998001)(50466002)(305945005)(106356001)(25786009)(16576012)(105586002)(68736007)(33646002)(316002)(16526018)(65956001)(53936002)(6486002)(77096006)(65806001)(6246003)(50986999)(58126008)(47776003)(2906002)(229853002)(101416001)(36756003)(90366009)(66066001)(4001150100001)(83506002)(5660300001)(53546010)(3846002)(23676003)(65826007)(76176999)(54356999)(6116002)(478600001); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR17MB1978; H:[10.100.0.160]; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: multapplied.net does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtETTVQUjE3TUIxOTc4OzIzOkxoVFdpUWt4L0swMEdtcUNJRXNmdVE3a1N1?= =?utf-8?B?emxHUDg4ZTZQYm54SEtRNUxqWmVsK3E5VEs3Q0c3d3NEQlY2SDRjaFRSVWd6?= =?utf-8?B?cXJtYUhVSmdkV0FhRGpkTWh4dGNUNDY5djVDWGJXN0J3VGk3NnNvTkV2cVpy?= =?utf-8?B?ZnN4TGZRMUxtMFJKMWlaUzZiTlVzdzlvWVkrM2QwKzBWT3ZwU2hPOENCM2k5?= =?utf-8?B?VTVvM3p1OHYzUkNudVk3TG5wZWkzUy84eVNFOTQwSGI1TTgzZXZETUhsaG52?= =?utf-8?B?TjE5VGNqS0xIWlhNTlRVM0ZpRElWN3NUK2xIazdVVkJpZjJENU5IZzdOQXpX?= =?utf-8?B?TXBzSnNWMjluS0tkOEdCWFlud085bnlSWTd5Uy8rRjhyQ3FMMCtCRTExOWEy?= =?utf-8?B?d04yMzIybzU5UHhnYTAyS3JRRG8zc1BVNk5IL2hpQ01FR3c2UmZSRVFMR0Rm?= =?utf-8?B?a1Z2RXVFZE12Ums4NDd3RXVJV2NZbm55MTRsS2JvOWdmbE9SbEZkd3hyeENa?= =?utf-8?B?RlBHd2VNZy9JeWZuT1JNa2EvNklyNzNrSFZkLzVUdXFENnRGSFdndmppNXh2?= =?utf-8?B?T0Q0Uy8xMmd2cElUTXI3V25LcG5telRTTWdsYy9NbVBtWUFSOVAySVJiR3Nm?= =?utf-8?B?cG9uM0FTcmNDVTJSNThwamM4Q2lCQ3R0cWUyNTJsUmcvTXh3YXhVcExIOFlG?= =?utf-8?B?VFVxa3ZnSnc0V2NuaDgyRzZXNElpRUZPWGFRK0xrUFBYVEhwMEtWUTg2MlUx?= =?utf-8?B?K0lWV1dQQ2tHMHdqeCtjTXY5SDlkcml5Y2s4R1E1M2l1dUM4Q0l0VkFLU05m?= =?utf-8?B?WEhsLzNqUTExS0VWN2JlSWMxME1GMmxXUWVFNk9zOCtmRTVEMjlvSFZBa2hG?= =?utf-8?B?TmVzWFNKM2RaelNOVXE2THkyU1NzanF0ekF1MytjMjU5TFRtUkEzTDJTc0sx?= =?utf-8?B?WEQyaWNPWEsrQlhLQW14L0V6SEh2TDJqMXhLOXRMcXF4SEVybkxQaTRJWXRx?= =?utf-8?B?RytodWhGWmtUQUR4STh0OEVzMWJrSENQSVZiWXlsMVl6OU1VaVB5Z2dUV3A0?= =?utf-8?B?RmVUSGZqOUJUaFA2NG11SGc2NGJ1elRxYWdueHJBTXprL096L0tkbHFpMEtp?= =?utf-8?B?Szl4TFZvVTRqS2Y4a1JlRzhFMVZNdm5EbzFkN25iODRKYVMrczlkMGN3Z0l3?= =?utf-8?B?TmRURXdLTmo4bi9QWkVOQ3dNRFNCcU82Z3MzRGMvOEU4R2R4a0p6UzFBWEVt?= =?utf-8?B?RGFLV3NpbWRSYWlKOFJaSUlicVkrdm1KcGhIdk1EOVdRYVJybDlSK2tGVWd5?= =?utf-8?B?K1FUUUgyc3IwMStEVFhCUlJxbWpPajBBYVFmS09NVXB6SEdNaUplY2ZwWDdG?= =?utf-8?B?SW9Mcy9HQ0sxZ0plUHpKUktSc01kU0doMXdjVlV0Ti8rN1pMenpUTHlmcXRs?= =?utf-8?B?QXR2Mk1TNFZBT1Rjdzl6dVkyRXlOdVp5MElFandDUm53empubzZoWmYzVmMx?= =?utf-8?B?Vyt0NlRBNERLZVZGR2x2Z2lKbldNQkZRMWF4UXhpaXJLaURKZ0dzaFVldkI2?= =?utf-8?B?cnorNG9GM3lJUDQ4bzVsOUtpVk9oZjFCc3crOGZOcjRxUldRK2pYcEttUVVV?= =?utf-8?B?MCtwZHBEdXdOMFh2eW9nekMwdlBSZHZpS0M1US9wRC9JemIrVTBqOXpMMjRo?= =?utf-8?B?UWNPMC9ha0llU1Y0MSs4VzV2TGJsUHNwSHNDSVQ4ZmZOUzNBQ2lnV1VMV0to?= =?utf-8?B?N3pIOGpwaVNpdVkrNDhuaDgwWm5pMjJkNGdJOC9mOCtRN2cwZHY3ZU1Mdmxa?= =?utf-8?B?TzZLN0NxbHdhUElCNjBDd0d4WG1IR3VuWlpuLy9wVEVEbXJ0TUFFVlJFODN1?= =?utf-8?B?K1JHMlZEWGZ0RTA3UUlnM3lqV3JhMkxuZUh3VW1QN3JFTC9UZDByNWo4Q1pk?= =?utf-8?B?b2pHU3g3MTJWZXVrakZwSW5TekltTmZCVjUwWTNwNVFVSFJ3VjZDWGN1ZmtG?= =?utf-8?Q?Q8Ki9/?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR17MB1978; 6:RQYts8ZkmIT8nUraCriiI5dZ9mhqR5GDSCgSEbc3eQ16wn///z6YlylmB8H5neOLdZ6E1rmg+zCNkzKCr4SzC2ZforZ7er7c+XyOR1+GvC97Lx59pG3EaRvRAFvbJlRgGaOAuZHlm9pDeUi0XYsThE9DML/C9aRWIMzgZ9TPqPOvHcYfMcqD4MMY0+UdA6a5nDmzqkQDy24rlQll+V+ZKskjHoFvYBmTwaKNnvfEjh6Z7pcnNkKkhdC+eMGJBjGE3x+F72F9OqcTLza8skuiYnmZiFOm0I3rJbcPwGychIHqFUjYDUhzE5Nifzr/98qUyarYwMNli61NELWrQUbjmR9PVU2Y8apn0zMKOwTolds=; 5:CZVcgEDqD9JcOmvVY9Z11uKPH4FMS50Tc7c5JlopOOiMjkinFFgdUhBr7z37BzzYuevA8FAZ9QIzKieWMcJDRGGCjCygO0MTvdi9jzHhSnNykyxN9zNLCcmdslTONJO4vu4I+l8JzyUR6VamCm/7OW0uhO6ZYjBRoWkl5yse/OI=; 24:fK6Ijb9nz6/5RKfBR2Gad1YHkrSsXZYbGa9Tg1n53T1DEPx+ncxVKuqEdTgNSQFXiUQ+rRmmk5xo0Pp8svsh36Ri0zMyPjZoH6Y3O7KoBho=; 7:HvZjgK3bQb1WoS2PhNQbr77WW3lGqv51nROi3DzVkyXOLCuzW/0Zb8V9K0SwWw15YdKvTLgat/yjXXrhebqOtNdle+BVJ2mOKWwsixmpEVyCeXkrNcRg1iSb2r5oI7+9Wv0ZekH5J3BQGjBPRwxgVnEclag2Ju5Q0RNYSc14EC/m+e/FOM8GVLRKVSxxqgXmLzsBKn6MIpvix2+3hWLQypiUnPkOKr91bM0OtwS+jkdlch8y3QO3Xc4GfOEq824r SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: multapplied.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Nov 2017 21:26:14.8349 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e699b15b-66a7-4481-03b6-08d52e01d540 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 10c26ea1-9e95-414d-91d0-c44adf533c85 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR17MB1978 X-Mailman-Approved-At: Wed, 22 Nov 2017 09:48:16 +0100 Subject: Re: [dpdk-users] qos_sched in DPDK 17.11.0 fails to initialize mbuf pool X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Nov 2017 21:26:18 -0000 On 2017-11-17 10:34 AM, Dumitrescu, Cristian wrote: > > >> -----Original Message----- >> From: Ian Trick [mailto:ian.trick@multapplied.net] >> Sent: Friday, November 17, 2017 5:50 PM >> To: Dumitrescu, Cristian ; users@dpdk.org >> Subject: Re: qos_sched in DPDK 17.11.0 fails to initialize mbuf pool >> >> On 2017-11-17 04:19 AM, Dumitrescu, Cristian wrote: >>> >>> >>>> -----Original Message----- >>>> From: Ian Trick [mailto:ian.trick@multapplied.net] >>>> Sent: Friday, November 17, 2017 1:24 AM >>>> To: users@dpdk.org >>>> Cc: Dumitrescu, Cristian >>>> Subject: qos_sched in DPDK 17.11.0 fails to initialize mbuf pool >>>> >>>> Hi. I'm having an issue starting the qos_sched example program. >>>> >>>> # ./examples/qos_sched/build/qos_sched --no-huge -l 1,2,3 --vdev >>>> net_af_packet0,iface=eth1 -- --pfc "0,0,2,3" --cfg >>>> examples/qos_sched/profile_ov.cfg >>>> >>>> EAL: Detected 16 lcore(s) >>>> EAL: Probing VFIO support... >>>> EAL: Started without hugepages support, physical addresses not available >>>> EAL: PCI device 0000:08:00.0 on NUMA socket -1 >>>> EAL: Invalid NUMA socket, default to 0 >>>> EAL: probe driver: 8086:10d3 net_e1000_em >>>> PMD: Initializing pmd_af_packet for net_af_packet0 >>>> PMD: net_af_packet0: AF_PACKET MMAP parameters: >>>> PMD: net_af_packet0: block size 4096 >>>> PMD: net_af_packet0: block count 256 >>>> PMD: net_af_packet0: frame size 2048 >>>> PMD: net_af_packet0: frame count 512 >>>> PMD: net_af_packet0: creating AF_PACKET-backed ethdev on numa >> socket 0 >>>> EAL: Error - exiting with code: 1 >>>> Cause: Cannot init mbuf pool for socket 0 >>>> >>> >>> Personally I never used this application with --no-huge or with AF_PACKET, >> so I suggest you start from the configuration known to work (as detailed in >> the Sample App Guide) and then change/add one variable at a time to see >> which change triggers the mempool issue. >>> >>> This app needs large amounts of memory for the mempool, as traffic >> management is buffering lots of packets in lots of queues. Out typical tests >> are done with 4K pipes/output port (64K queues/output port) so we >> provision mempool to have 2M buffers for each output port. The size of the >> mempool is hardcoded in the application. >> >> Can I configure this to run with fewer queues or something so that it >> requires less memory. I thought running with profile_ov.cfg might have >> lower memory requirements since it includes: >>> number of pipes per subport = 32 >> compared to 4096 in the other configuration file. So I figured there >> would be fewer queues and buffers? But I only have 4GB available on the >> device I have if I want to test something that isn't AF_PACKET. >> > > Digging in the source code, I found that you can tweak the mempool size through this macro: > //file "main.h" > #define NB_MBUF (2*1024*1024) Oh right, I remember fiddling with that when trying to get it working --no-huge. Tweaking that worked in this case on a real interface in DPDK mode. Adding --no-huge makes it complain and not start up, so that might be what was happening in my original case. I think we're running with that option because we were having trouble using it under LXC. But I'll look into solving that. Thanks! > >>> >>>> >>>> This is version 17.11.0 from the repo. My RTE_TARGET is >>>> x86_64-native-linuxapp-clang. eth1 is a veth. I've tried running with >>>> `-m` and using a low value but the issue still happens. >>>> >>>> From what I can tell, rte_pktmbuf_pool_create() is failing and rte_errno >>>> is set to EINVAL. >>>> >>>> In librte_mempool/rte_mempool.c, the function >>>> rte_mempool_populate_virt() is succeeding this test and returning - >> EINVAL: >>>> >>>> if (RTE_ALIGN_CEIL(len, pg_sz) != len) >>>> return -EINVAL; >>>> >>>> In that context, len is mz->len, the length of a memzone passed by the >>>> caller, rte_mempool_populate_default(). Which got it here: >>>> >>>> mz = rte_memzone_reserve_aligned(mz_name, size, >>>> mp->socket_id, mz_flags, align); >>>> /* not enough memory, retry with the biggest zone we have */ >>>> if (mz == NULL) >>>> mz = rte_memzone_reserve_aligned(mz_name, 0, >>>> mp->socket_id, mz_flags, align); >>>> >>>> This fails the first call, and succeeds the second when it passes 0 as >>>> the size. memzone_reserve_aligned_thread_unsafe(), in >>>> librte_eal/common/eal_common_memzone.c, gets the length this way: >>>> >>>> requested_len = find_heap_max_free_elem(&socket_id, align); >>>> >>>> So the align value is 4096. But the value returned by >>>> find_heap_max_free_elem() isn't aligned to that -- I think? Since it >>>> fails the check later on. >>>> >>>> I'm not sure if this is a thing with my environment where I don't have >>>> enough memory? (Although I would have expected a different error for >>>> that.) Or I don't have the right program arguments? Or one of these >>>> functions isn't doing what it's supposed to?