From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from serv108.segi.ulg.ac.be (serv108.segi.ulg.ac.be [139.165.32.111]) by dpdk.org (Postfix) with ESMTP id 8DC7F4AC7 for ; Thu, 6 Oct 2016 13:02:13 +0200 (CEST) Received: from mbx12-zne.ulg.ac.be (serv470.segi.ulg.ac.be [139.165.32.199]) by serv108.segi.ulg.ac.be (Postfix) with ESMTP id 51A452011EC0; Thu, 6 Oct 2016 13:02:13 +0200 (CEST) Received: from localhost (localhost.localdomain [127.0.0.1]) by mbx12-zne.ulg.ac.be (Postfix) with ESMTP id 45A9F129E8B8; Thu, 6 Oct 2016 13:02:13 +0200 (CEST) Received: from mbx12-zne.ulg.ac.be ([127.0.0.1]) by localhost (mbx12-zne.ulg.ac.be [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 4DiwKgszA03P; Thu, 6 Oct 2016 13:02:13 +0200 (CEST) Received: from mbx12-zne.ulg.ac.be (mbx12-zne.ulg.ac.be [139.165.32.199]) by mbx12-zne.ulg.ac.be (Postfix) with ESMTP id 22ACD129EA41; Thu, 6 Oct 2016 13:02:13 +0200 (CEST) Date: Thu, 6 Oct 2016 13:02:12 +0200 (CEST) From: tom.barbette@ulg.ac.be To: Sergio Gonzalez Monroy Cc: Andriy Berestovskyy , Renata Saiakhova , users Message-ID: <377614611.32755165.1475751732537.JavaMail.zimbra@ulg.ac.be> In-Reply-To: <3f747784-4468-87bd-389c-9ed2d51e7c03@intel.com> References: <57F36199.5020100@oneaccess-net.com> <57F3787A.6060105@oneaccess-net.com> <57F388E5.3010405@oneaccess-net.com> <512920892.31118614.1475582525691.JavaMail.zimbra@ulg.ac.be> <3f747784-4468-87bd-389c-9ed2d51e7c03@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [139.165.223.24] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - GC53 (Linux)/8.0.9_GA_6191) Thread-Topic: rte_segments: hugepages are not in contiguous memory Thread-Index: 83Us6m7WAXEljhWCm7iscdP1510XFQ== Subject: Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Oct 2016 11:02:13 -0000 Hi, I had strange change in performances running the same test multiple time, u= sing the system for other things in between (among them, loading and unload= ing Netmap, which scatters the memory pretty well). It was a very simple fo= rwarding test taking packets on 4 interface and sending them back on the op= posite interface. The only difference between each run I could find was the= increasing memory scattering. Other strange performance issues included a = shift in the performance curve of throughput according to packet size. No explanation though... Tom ----- Mail original ----- De: "Sergio Gonzalez Monroy" =C3=80: "tom barbette" , "Andriy Berestovskyy" Cc: "Renata Saiakhova" , "users" Envoy=C3=A9: Mardi 4 Octobre 2016 16:09:29 Objet: Re: [dpdk-users] rte_segments: hugepages are not in contiguous memor= y Hi folks, In theory, there shouldn't be any performance difference between having=20 a mempool allocated from a single memseg (given the use the same number=20 of hugepages) versus multiple memsegs as it is all done on mempool creation/setup and each mbuf has its own=20 phys address. Tom, I cannot think of a reason why you would have higher memory access=20 for having scatter hugapages vs contig hugepages. Any details on the test you were running? Sergio On 04/10/2016 13:02, tom.barbette@ulg.ac.be wrote: > There is a noticeable performance drop with more scattering of the huge p= ages. > > I did not measure any difference accurately but I ended up rebooting my D= UT between each performance test because the pages get scattered with time = and re-launch of the DPDK application instead of the whole machine, because= the tests showed higher memory access cost each time I re-launched the app= lication. > > Tom > > ----- Mail original ----- > De: "Andriy Berestovskyy" > =C3=80: "Renata Saiakhova" > Cc: "Sergio Gonzalez Monroy", "users" > Envoy=C3=A9: Mardi 4 Octobre 2016 13:27:23 > Objet: Re: [dpdk-users] rte_segments: hugepages are not in contiguous=09m= emory > > Renata, > In theory 512 contiguous 2MB huge pages might get transparently > promoted to a single 1GB "superpage" and single TLB entry, but I am > not even sure if it is implemented in Linux... > > So, I do not think there will be any noticeable performance difference > between contiguous and non-contiguous 2MB huge pages. But you better > measure it to make sure ;) > > Regards, > Andriy > > On Tue, Oct 4, 2016 at 12:48 PM, Renata Saiakhova > wrote: >> Hi Andriy, >> >> thanks for your reply. I guess that contiguous memory is requested becau= se >> of the performance reasons. Do you know if I can expect a noticeable >> performance drop using non-contiguous memory? >> >> Renata >> >> >> On 10/04/2016 12:13 PM, Andriy Berestovskyy wrote: >>> Hi Renata, >>> DPDK supports non-contiguous memory pools, but >>> rte_pktmbuf_pool_create() uses rte_mempool_create_empty() with flags >>> set to zero, i.e. requests contiguous memory. >>> >>> As a workaround, in rte_pktmbuf_pool_create() try to pass >>> MEMPOOL_F_NO_PHYS_CONTIG flag as the last argument to >>> rte_mempool_create_empty(). >>> >>> Note that KNI and some PMDs in 16.07 still require contiguous memory >>> pools, so the trick might not work for your setup. For the KNI try the >>> DPDK's master branch which includes the commit by Ferruh Yigit: >>> >>> 8451269 kni: remove continuous memory restriction >>> >>> Regards, >>> Andriy >>> >>> >>> On Tue, Oct 4, 2016 at 11:38 AM, Renata Saiakhova >>> wrote: >>>> Hi Sergio, >>>> >>>> thank you for your quick answer. I also tried to allocate 1GB hugepage= , >>>> but >>>> seems kernel fails to allocate it: previously I've seen that >>>> HugePages_Total >>>> in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't k= now >>>> why). >>>> But anyway, if there is no way to control hugepage allocation in the >>>> sense >>>> they are in contiguous memory there is only way to accept it and adapt >>>> the >>>> code that it creates several pools which in total satisfy the requeste= d >>>> size. >>>> >>>> Renata >>>> >>>> >>>> On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote: >>>>> On 04/10/2016 09:00, Renata Saiakhova wrote: >>>>>> Hi all, >>>>>> >>>>>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux >>>>>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I >>>>>> pass a >>>>>> parameter in kernel command line to allocate 512 hugepages of 2 MB a= t >>>>>> boot >>>>>> time. They are successfully allocated. When an application with dpdk >>>>>> starts >>>>>> it calls rte_pktmbuf_pool_create() which in turns requests internall= y >>>>>> 649363712 bytes. Those bytes should be allocated from one of >>>>>> rte_memseg. >>>>>> rte_memsegs describes contiguous portions of memory (both physical a= nd >>>>>> virtual) built on hugepages. This allocation fails, because there ar= e >>>>>> no >>>>>> rte_memsegs of this size (or bigger). Further debugging shows that >>>>>> hugepages >>>>>> are allocated in non-contiguous physical memory and therefore >>>>>> rte_memsegs >>>>>> are built respecting gaps in physical memory. >>>>>> Below are the sizes of segments built on hugepages (in bytes) >>>>>> 2097152 >>>>>> 6291456 >>>>>> 2097152 >>>>>> 524288000 >>>>>> 2097152 >>>>>> 532676608 >>>>>> 2097152 >>>>>> 2097152 >>>>>> So there are 5 segments which includes only one hugepage! >>>>>> This behavior is completely different to what I observe with linux >>>>>> kernel >>>>>> 3.8 (used with the same application with dpdk) - where all hugepages >>>>>> are >>>>>> allocated in contiguous memory. >>>>>> Does anyone experience the same issue? Could it be some kernel optio= n >>>>>> which can do the magic? If not, and kernel can allocated hugepages i= n >>>>>> non-contiguous memory how dpdk is going to resolve it? >>>>>> >>>>> I don't think there is anything we can do to force the kernel to >>>>> pre-allocate contig hugepages on boot. If there was, we wouldn't need= to >>>>> do >>>>> all this mapping sorting and grouping we do on DPDK >>>>> as we would rely on the kernel giving us pre-allocated contig hugepag= es. >>>>> >>>>> If you have plenty of memory one possible work around would be to >>>>> increase >>>>> the number of default hugepages so we are likely to find more contigu= ous >>>>> ones. >>>>> >>>>> Is using 1GB hugepages a possibility in your case? >>>>> >>>>> Sergio >>>>> >>>>>> Thanks in advance, >>>>>> Renata >>>>>> >>>>> . >>>>> >