* [dpdk-users] rte_segments: hugepages are not in contiguous memory @ 2016-10-04 8:00 Renata Saiakhova 2016-10-04 8:27 ` Sergio Gonzalez Monroy 0 siblings, 1 reply; 9+ messages in thread From: Renata Saiakhova @ 2016-10-04 8:00 UTC (permalink / raw) To: users Hi all, I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I pass a parameter in kernel command line to allocate 512 hugepages of 2 MB at boot time. They are successfully allocated. When an application with dpdk starts it calls rte_pktmbuf_pool_create() which in turns requests internally 649363712 bytes. Those bytes should be allocated from one of rte_memseg. rte_memsegs describes contiguous portions of memory (both physical and virtual) built on hugepages. This allocation fails, because there are no rte_memsegs of this size (or bigger). Further debugging shows that hugepages are allocated in non-contiguous physical memory and therefore rte_memsegs are built respecting gaps in physical memory. Below are the sizes of segments built on hugepages (in bytes) 2097152 6291456 2097152 524288000 2097152 532676608 2097152 2097152 So there are 5 segments which includes only one hugepage! This behavior is completely different to what I observe with linux kernel 3.8 (used with the same application with dpdk) - where all hugepages are allocated in contiguous memory. Does anyone experience the same issue? Could it be some kernel option which can do the magic? If not, and kernel can allocated hugepages in non-contiguous memory how dpdk is going to resolve it? Thanks in advance, Renata ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory 2016-10-04 8:00 [dpdk-users] rte_segments: hugepages are not in contiguous memory Renata Saiakhova @ 2016-10-04 8:27 ` Sergio Gonzalez Monroy 2016-10-04 9:38 ` Renata Saiakhova 0 siblings, 1 reply; 9+ messages in thread From: Sergio Gonzalez Monroy @ 2016-10-04 8:27 UTC (permalink / raw) To: Renata Saiakhova, users On 04/10/2016 09:00, Renata Saiakhova wrote: > Hi all, > > I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux > kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I > pass a parameter in kernel command line to allocate 512 hugepages of 2 > MB at boot time. They are successfully allocated. When an application > with dpdk starts it calls rte_pktmbuf_pool_create() which in turns > requests internally 649363712 bytes. Those bytes should be allocated > from one of rte_memseg. rte_memsegs describes contiguous portions of > memory (both physical and virtual) built on hugepages. This allocation > fails, because there are no rte_memsegs of this size (or bigger). > Further debugging shows that hugepages are allocated in non-contiguous > physical memory and therefore rte_memsegs are built respecting gaps in > physical memory. > Below are the sizes of segments built on hugepages (in bytes) > 2097152 > 6291456 > 2097152 > 524288000 > 2097152 > 532676608 > 2097152 > 2097152 > So there are 5 segments which includes only one hugepage! > This behavior is completely different to what I observe with linux > kernel 3.8 (used with the same application with dpdk) - where all > hugepages are allocated in contiguous memory. > Does anyone experience the same issue? Could it be some kernel option > which can do the magic? If not, and kernel can allocated hugepages in > non-contiguous memory how dpdk is going to resolve it? > I don't think there is anything we can do to force the kernel to pre-allocate contig hugepages on boot. If there was, we wouldn't need to do all this mapping sorting and grouping we do on DPDK as we would rely on the kernel giving us pre-allocated contig hugepages. If you have plenty of memory one possible work around would be to increase the number of default hugepages so we are likely to find more contiguous ones. Is using 1GB hugepages a possibility in your case? Sergio > Thanks in advance, > Renata > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory 2016-10-04 8:27 ` Sergio Gonzalez Monroy @ 2016-10-04 9:38 ` Renata Saiakhova 2016-10-04 10:13 ` Andriy Berestovskyy 0 siblings, 1 reply; 9+ messages in thread From: Renata Saiakhova @ 2016-10-04 9:38 UTC (permalink / raw) To: Sergio Gonzalez Monroy, users Hi Sergio, thank you for your quick answer. I also tried to allocate 1GB hugepage, but seems kernel fails to allocate it: previously I've seen that HugePages_Total in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't know why). But anyway, if there is no way to control hugepage allocation in the sense they are in contiguous memory there is only way to accept it and adapt the code that it creates several pools which in total satisfy the requested size. Renata On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote: > On 04/10/2016 09:00, Renata Saiakhova wrote: >> Hi all, >> >> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux >> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I >> pass a parameter in kernel command line to allocate 512 hugepages of >> 2 MB at boot time. They are successfully allocated. When an >> application with dpdk starts it calls rte_pktmbuf_pool_create() which >> in turns requests internally 649363712 bytes. Those bytes should be >> allocated from one of rte_memseg. rte_memsegs describes contiguous >> portions of memory (both physical and virtual) built on hugepages. >> This allocation fails, because there are no rte_memsegs of this size >> (or bigger). Further debugging shows that hugepages are allocated in >> non-contiguous physical memory and therefore rte_memsegs are built >> respecting gaps in physical memory. >> Below are the sizes of segments built on hugepages (in bytes) >> 2097152 >> 6291456 >> 2097152 >> 524288000 >> 2097152 >> 532676608 >> 2097152 >> 2097152 >> So there are 5 segments which includes only one hugepage! >> This behavior is completely different to what I observe with linux >> kernel 3.8 (used with the same application with dpdk) - where all >> hugepages are allocated in contiguous memory. >> Does anyone experience the same issue? Could it be some kernel option >> which can do the magic? If not, and kernel can allocated hugepages in >> non-contiguous memory how dpdk is going to resolve it? >> > > I don't think there is anything we can do to force the kernel to > pre-allocate contig hugepages on boot. If there was, we wouldn't need > to do all this mapping sorting and grouping we do on DPDK > as we would rely on the kernel giving us pre-allocated contig hugepages. > > If you have plenty of memory one possible work around would be to > increase the number of default hugepages so we are likely to find more > contiguous ones. > > Is using 1GB hugepages a possibility in your case? > > Sergio > >> Thanks in advance, >> Renata >> > > . > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory 2016-10-04 9:38 ` Renata Saiakhova @ 2016-10-04 10:13 ` Andriy Berestovskyy 2016-10-04 10:48 ` Renata Saiakhova 0 siblings, 1 reply; 9+ messages in thread From: Andriy Berestovskyy @ 2016-10-04 10:13 UTC (permalink / raw) To: Renata Saiakhova; +Cc: Sergio Gonzalez Monroy, users Hi Renata, DPDK supports non-contiguous memory pools, but rte_pktmbuf_pool_create() uses rte_mempool_create_empty() with flags set to zero, i.e. requests contiguous memory. As a workaround, in rte_pktmbuf_pool_create() try to pass MEMPOOL_F_NO_PHYS_CONTIG flag as the last argument to rte_mempool_create_empty(). Note that KNI and some PMDs in 16.07 still require contiguous memory pools, so the trick might not work for your setup. For the KNI try the DPDK's master branch which includes the commit by Ferruh Yigit: 8451269 kni: remove continuous memory restriction Regards, Andriy On Tue, Oct 4, 2016 at 11:38 AM, Renata Saiakhova <Renata.Saiakhova@oneaccess-net.com> wrote: > Hi Sergio, > > thank you for your quick answer. I also tried to allocate 1GB hugepage, but > seems kernel fails to allocate it: previously I've seen that HugePages_Total > in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't know > why). > But anyway, if there is no way to control hugepage allocation in the sense > they are in contiguous memory there is only way to accept it and adapt the > code that it creates several pools which in total satisfy the requested > size. > > Renata > > > On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote: >> >> On 04/10/2016 09:00, Renata Saiakhova wrote: >>> >>> Hi all, >>> >>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux >>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I pass a >>> parameter in kernel command line to allocate 512 hugepages of 2 MB at boot >>> time. They are successfully allocated. When an application with dpdk starts >>> it calls rte_pktmbuf_pool_create() which in turns requests internally >>> 649363712 bytes. Those bytes should be allocated from one of rte_memseg. >>> rte_memsegs describes contiguous portions of memory (both physical and >>> virtual) built on hugepages. This allocation fails, because there are no >>> rte_memsegs of this size (or bigger). Further debugging shows that hugepages >>> are allocated in non-contiguous physical memory and therefore rte_memsegs >>> are built respecting gaps in physical memory. >>> Below are the sizes of segments built on hugepages (in bytes) >>> 2097152 >>> 6291456 >>> 2097152 >>> 524288000 >>> 2097152 >>> 532676608 >>> 2097152 >>> 2097152 >>> So there are 5 segments which includes only one hugepage! >>> This behavior is completely different to what I observe with linux kernel >>> 3.8 (used with the same application with dpdk) - where all hugepages are >>> allocated in contiguous memory. >>> Does anyone experience the same issue? Could it be some kernel option >>> which can do the magic? If not, and kernel can allocated hugepages in >>> non-contiguous memory how dpdk is going to resolve it? >>> >> >> I don't think there is anything we can do to force the kernel to >> pre-allocate contig hugepages on boot. If there was, we wouldn't need to do >> all this mapping sorting and grouping we do on DPDK >> as we would rely on the kernel giving us pre-allocated contig hugepages. >> >> If you have plenty of memory one possible work around would be to increase >> the number of default hugepages so we are likely to find more contiguous >> ones. >> >> Is using 1GB hugepages a possibility in your case? >> >> Sergio >> >>> Thanks in advance, >>> Renata >>> >> >> . >> > -- Andriy Berestovskyy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory 2016-10-04 10:13 ` Andriy Berestovskyy @ 2016-10-04 10:48 ` Renata Saiakhova 2016-10-04 11:27 ` Andriy Berestovskyy 0 siblings, 1 reply; 9+ messages in thread From: Renata Saiakhova @ 2016-10-04 10:48 UTC (permalink / raw) To: Andriy Berestovskyy; +Cc: Sergio Gonzalez Monroy, users Hi Andriy, thanks for your reply. I guess that contiguous memory is requested because of the performance reasons. Do you know if I can expect a noticeable performance drop using non-contiguous memory? Renata On 10/04/2016 12:13 PM, Andriy Berestovskyy wrote: > Hi Renata, > DPDK supports non-contiguous memory pools, but > rte_pktmbuf_pool_create() uses rte_mempool_create_empty() with flags > set to zero, i.e. requests contiguous memory. > > As a workaround, in rte_pktmbuf_pool_create() try to pass > MEMPOOL_F_NO_PHYS_CONTIG flag as the last argument to > rte_mempool_create_empty(). > > Note that KNI and some PMDs in 16.07 still require contiguous memory > pools, so the trick might not work for your setup. For the KNI try the > DPDK's master branch which includes the commit by Ferruh Yigit: > > 8451269 kni: remove continuous memory restriction > > Regards, > Andriy > > > On Tue, Oct 4, 2016 at 11:38 AM, Renata Saiakhova > <Renata.Saiakhova@oneaccess-net.com> wrote: >> Hi Sergio, >> >> thank you for your quick answer. I also tried to allocate 1GB hugepage, but >> seems kernel fails to allocate it: previously I've seen that HugePages_Total >> in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't know >> why). >> But anyway, if there is no way to control hugepage allocation in the sense >> they are in contiguous memory there is only way to accept it and adapt the >> code that it creates several pools which in total satisfy the requested >> size. >> >> Renata >> >> >> On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote: >>> On 04/10/2016 09:00, Renata Saiakhova wrote: >>>> Hi all, >>>> >>>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux >>>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I pass a >>>> parameter in kernel command line to allocate 512 hugepages of 2 MB at boot >>>> time. They are successfully allocated. When an application with dpdk starts >>>> it calls rte_pktmbuf_pool_create() which in turns requests internally >>>> 649363712 bytes. Those bytes should be allocated from one of rte_memseg. >>>> rte_memsegs describes contiguous portions of memory (both physical and >>>> virtual) built on hugepages. This allocation fails, because there are no >>>> rte_memsegs of this size (or bigger). Further debugging shows that hugepages >>>> are allocated in non-contiguous physical memory and therefore rte_memsegs >>>> are built respecting gaps in physical memory. >>>> Below are the sizes of segments built on hugepages (in bytes) >>>> 2097152 >>>> 6291456 >>>> 2097152 >>>> 524288000 >>>> 2097152 >>>> 532676608 >>>> 2097152 >>>> 2097152 >>>> So there are 5 segments which includes only one hugepage! >>>> This behavior is completely different to what I observe with linux kernel >>>> 3.8 (used with the same application with dpdk) - where all hugepages are >>>> allocated in contiguous memory. >>>> Does anyone experience the same issue? Could it be some kernel option >>>> which can do the magic? If not, and kernel can allocated hugepages in >>>> non-contiguous memory how dpdk is going to resolve it? >>>> >>> I don't think there is anything we can do to force the kernel to >>> pre-allocate contig hugepages on boot. If there was, we wouldn't need to do >>> all this mapping sorting and grouping we do on DPDK >>> as we would rely on the kernel giving us pre-allocated contig hugepages. >>> >>> If you have plenty of memory one possible work around would be to increase >>> the number of default hugepages so we are likely to find more contiguous >>> ones. >>> >>> Is using 1GB hugepages a possibility in your case? >>> >>> Sergio >>> >>>> Thanks in advance, >>>> Renata >>>> >>> . >>> > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory 2016-10-04 10:48 ` Renata Saiakhova @ 2016-10-04 11:27 ` Andriy Berestovskyy 2016-10-04 12:02 ` tom.barbette 0 siblings, 1 reply; 9+ messages in thread From: Andriy Berestovskyy @ 2016-10-04 11:27 UTC (permalink / raw) To: Renata Saiakhova; +Cc: Sergio Gonzalez Monroy, users Renata, In theory 512 contiguous 2MB huge pages might get transparently promoted to a single 1GB "superpage" and single TLB entry, but I am not even sure if it is implemented in Linux... So, I do not think there will be any noticeable performance difference between contiguous and non-contiguous 2MB huge pages. But you better measure it to make sure ;) Regards, Andriy On Tue, Oct 4, 2016 at 12:48 PM, Renata Saiakhova <Renata.Saiakhova@oneaccess-net.com> wrote: > Hi Andriy, > > thanks for your reply. I guess that contiguous memory is requested because > of the performance reasons. Do you know if I can expect a noticeable > performance drop using non-contiguous memory? > > Renata > > > On 10/04/2016 12:13 PM, Andriy Berestovskyy wrote: >> >> Hi Renata, >> DPDK supports non-contiguous memory pools, but >> rte_pktmbuf_pool_create() uses rte_mempool_create_empty() with flags >> set to zero, i.e. requests contiguous memory. >> >> As a workaround, in rte_pktmbuf_pool_create() try to pass >> MEMPOOL_F_NO_PHYS_CONTIG flag as the last argument to >> rte_mempool_create_empty(). >> >> Note that KNI and some PMDs in 16.07 still require contiguous memory >> pools, so the trick might not work for your setup. For the KNI try the >> DPDK's master branch which includes the commit by Ferruh Yigit: >> >> 8451269 kni: remove continuous memory restriction >> >> Regards, >> Andriy >> >> >> On Tue, Oct 4, 2016 at 11:38 AM, Renata Saiakhova >> <Renata.Saiakhova@oneaccess-net.com> wrote: >>> >>> Hi Sergio, >>> >>> thank you for your quick answer. I also tried to allocate 1GB hugepage, >>> but >>> seems kernel fails to allocate it: previously I've seen that >>> HugePages_Total >>> in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't know >>> why). >>> But anyway, if there is no way to control hugepage allocation in the >>> sense >>> they are in contiguous memory there is only way to accept it and adapt >>> the >>> code that it creates several pools which in total satisfy the requested >>> size. >>> >>> Renata >>> >>> >>> On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote: >>>> >>>> On 04/10/2016 09:00, Renata Saiakhova wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux >>>>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I >>>>> pass a >>>>> parameter in kernel command line to allocate 512 hugepages of 2 MB at >>>>> boot >>>>> time. They are successfully allocated. When an application with dpdk >>>>> starts >>>>> it calls rte_pktmbuf_pool_create() which in turns requests internally >>>>> 649363712 bytes. Those bytes should be allocated from one of >>>>> rte_memseg. >>>>> rte_memsegs describes contiguous portions of memory (both physical and >>>>> virtual) built on hugepages. This allocation fails, because there are >>>>> no >>>>> rte_memsegs of this size (or bigger). Further debugging shows that >>>>> hugepages >>>>> are allocated in non-contiguous physical memory and therefore >>>>> rte_memsegs >>>>> are built respecting gaps in physical memory. >>>>> Below are the sizes of segments built on hugepages (in bytes) >>>>> 2097152 >>>>> 6291456 >>>>> 2097152 >>>>> 524288000 >>>>> 2097152 >>>>> 532676608 >>>>> 2097152 >>>>> 2097152 >>>>> So there are 5 segments which includes only one hugepage! >>>>> This behavior is completely different to what I observe with linux >>>>> kernel >>>>> 3.8 (used with the same application with dpdk) - where all hugepages >>>>> are >>>>> allocated in contiguous memory. >>>>> Does anyone experience the same issue? Could it be some kernel option >>>>> which can do the magic? If not, and kernel can allocated hugepages in >>>>> non-contiguous memory how dpdk is going to resolve it? >>>>> >>>> I don't think there is anything we can do to force the kernel to >>>> pre-allocate contig hugepages on boot. If there was, we wouldn't need to >>>> do >>>> all this mapping sorting and grouping we do on DPDK >>>> as we would rely on the kernel giving us pre-allocated contig hugepages. >>>> >>>> If you have plenty of memory one possible work around would be to >>>> increase >>>> the number of default hugepages so we are likely to find more contiguous >>>> ones. >>>> >>>> Is using 1GB hugepages a possibility in your case? >>>> >>>> Sergio >>>> >>>>> Thanks in advance, >>>>> Renata >>>>> >>>> . >>>> >> >> > -- Andriy Berestovskyy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory 2016-10-04 11:27 ` Andriy Berestovskyy @ 2016-10-04 12:02 ` tom.barbette 2016-10-04 14:09 ` Sergio Gonzalez Monroy 0 siblings, 1 reply; 9+ messages in thread From: tom.barbette @ 2016-10-04 12:02 UTC (permalink / raw) To: Andriy Berestovskyy; +Cc: Renata Saiakhova, Sergio Gonzalez Monroy, users There is a noticeable performance drop with more scattering of the huge pages. I did not measure any difference accurately but I ended up rebooting my DUT between each performance test because the pages get scattered with time and re-launch of the DPDK application instead of the whole machine, because the tests showed higher memory access cost each time I re-launched the application. Tom ----- Mail original ----- De: "Andriy Berestovskyy" <aber@semihalf.com> À: "Renata Saiakhova" <Renata.Saiakhova@oneaccess-net.com> Cc: "Sergio Gonzalez Monroy" <sergio.gonzalez.monroy@intel.com>, "users" <users@dpdk.org> Envoyé: Mardi 4 Octobre 2016 13:27:23 Objet: Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory Renata, In theory 512 contiguous 2MB huge pages might get transparently promoted to a single 1GB "superpage" and single TLB entry, but I am not even sure if it is implemented in Linux... So, I do not think there will be any noticeable performance difference between contiguous and non-contiguous 2MB huge pages. But you better measure it to make sure ;) Regards, Andriy On Tue, Oct 4, 2016 at 12:48 PM, Renata Saiakhova <Renata.Saiakhova@oneaccess-net.com> wrote: > Hi Andriy, > > thanks for your reply. I guess that contiguous memory is requested because > of the performance reasons. Do you know if I can expect a noticeable > performance drop using non-contiguous memory? > > Renata > > > On 10/04/2016 12:13 PM, Andriy Berestovskyy wrote: >> >> Hi Renata, >> DPDK supports non-contiguous memory pools, but >> rte_pktmbuf_pool_create() uses rte_mempool_create_empty() with flags >> set to zero, i.e. requests contiguous memory. >> >> As a workaround, in rte_pktmbuf_pool_create() try to pass >> MEMPOOL_F_NO_PHYS_CONTIG flag as the last argument to >> rte_mempool_create_empty(). >> >> Note that KNI and some PMDs in 16.07 still require contiguous memory >> pools, so the trick might not work for your setup. For the KNI try the >> DPDK's master branch which includes the commit by Ferruh Yigit: >> >> 8451269 kni: remove continuous memory restriction >> >> Regards, >> Andriy >> >> >> On Tue, Oct 4, 2016 at 11:38 AM, Renata Saiakhova >> <Renata.Saiakhova@oneaccess-net.com> wrote: >>> >>> Hi Sergio, >>> >>> thank you for your quick answer. I also tried to allocate 1GB hugepage, >>> but >>> seems kernel fails to allocate it: previously I've seen that >>> HugePages_Total >>> in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't know >>> why). >>> But anyway, if there is no way to control hugepage allocation in the >>> sense >>> they are in contiguous memory there is only way to accept it and adapt >>> the >>> code that it creates several pools which in total satisfy the requested >>> size. >>> >>> Renata >>> >>> >>> On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote: >>>> >>>> On 04/10/2016 09:00, Renata Saiakhova wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux >>>>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I >>>>> pass a >>>>> parameter in kernel command line to allocate 512 hugepages of 2 MB at >>>>> boot >>>>> time. They are successfully allocated. When an application with dpdk >>>>> starts >>>>> it calls rte_pktmbuf_pool_create() which in turns requests internally >>>>> 649363712 bytes. Those bytes should be allocated from one of >>>>> rte_memseg. >>>>> rte_memsegs describes contiguous portions of memory (both physical and >>>>> virtual) built on hugepages. This allocation fails, because there are >>>>> no >>>>> rte_memsegs of this size (or bigger). Further debugging shows that >>>>> hugepages >>>>> are allocated in non-contiguous physical memory and therefore >>>>> rte_memsegs >>>>> are built respecting gaps in physical memory. >>>>> Below are the sizes of segments built on hugepages (in bytes) >>>>> 2097152 >>>>> 6291456 >>>>> 2097152 >>>>> 524288000 >>>>> 2097152 >>>>> 532676608 >>>>> 2097152 >>>>> 2097152 >>>>> So there are 5 segments which includes only one hugepage! >>>>> This behavior is completely different to what I observe with linux >>>>> kernel >>>>> 3.8 (used with the same application with dpdk) - where all hugepages >>>>> are >>>>> allocated in contiguous memory. >>>>> Does anyone experience the same issue? Could it be some kernel option >>>>> which can do the magic? If not, and kernel can allocated hugepages in >>>>> non-contiguous memory how dpdk is going to resolve it? >>>>> >>>> I don't think there is anything we can do to force the kernel to >>>> pre-allocate contig hugepages on boot. If there was, we wouldn't need to >>>> do >>>> all this mapping sorting and grouping we do on DPDK >>>> as we would rely on the kernel giving us pre-allocated contig hugepages. >>>> >>>> If you have plenty of memory one possible work around would be to >>>> increase >>>> the number of default hugepages so we are likely to find more contiguous >>>> ones. >>>> >>>> Is using 1GB hugepages a possibility in your case? >>>> >>>> Sergio >>>> >>>>> Thanks in advance, >>>>> Renata >>>>> >>>> . >>>> >> >> > -- Andriy Berestovskyy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory 2016-10-04 12:02 ` tom.barbette @ 2016-10-04 14:09 ` Sergio Gonzalez Monroy 2016-10-06 11:02 ` tom.barbette 0 siblings, 1 reply; 9+ messages in thread From: Sergio Gonzalez Monroy @ 2016-10-04 14:09 UTC (permalink / raw) To: tom.barbette, Andriy Berestovskyy; +Cc: Renata Saiakhova, users Hi folks, In theory, there shouldn't be any performance difference between having a mempool allocated from a single memseg (given the use the same number of hugepages) versus multiple memsegs as it is all done on mempool creation/setup and each mbuf has its own phys address. Tom, I cannot think of a reason why you would have higher memory access for having scatter hugapages vs contig hugepages. Any details on the test you were running? Sergio On 04/10/2016 13:02, tom.barbette@ulg.ac.be wrote: > There is a noticeable performance drop with more scattering of the huge pages. > > I did not measure any difference accurately but I ended up rebooting my DUT between each performance test because the pages get scattered with time and re-launch of the DPDK application instead of the whole machine, because the tests showed higher memory access cost each time I re-launched the application. > > Tom > > ----- Mail original ----- > De: "Andriy Berestovskyy"<aber@semihalf.com> > À: "Renata Saiakhova"<Renata.Saiakhova@oneaccess-net.com> > Cc: "Sergio Gonzalez Monroy"<sergio.gonzalez.monroy@intel.com>, "users"<users@dpdk.org> > Envoyé: Mardi 4 Octobre 2016 13:27:23 > Objet: Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory > > Renata, > In theory 512 contiguous 2MB huge pages might get transparently > promoted to a single 1GB "superpage" and single TLB entry, but I am > not even sure if it is implemented in Linux... > > So, I do not think there will be any noticeable performance difference > between contiguous and non-contiguous 2MB huge pages. But you better > measure it to make sure ;) > > Regards, > Andriy > > On Tue, Oct 4, 2016 at 12:48 PM, Renata Saiakhova > <Renata.Saiakhova@oneaccess-net.com> wrote: >> Hi Andriy, >> >> thanks for your reply. I guess that contiguous memory is requested because >> of the performance reasons. Do you know if I can expect a noticeable >> performance drop using non-contiguous memory? >> >> Renata >> >> >> On 10/04/2016 12:13 PM, Andriy Berestovskyy wrote: >>> Hi Renata, >>> DPDK supports non-contiguous memory pools, but >>> rte_pktmbuf_pool_create() uses rte_mempool_create_empty() with flags >>> set to zero, i.e. requests contiguous memory. >>> >>> As a workaround, in rte_pktmbuf_pool_create() try to pass >>> MEMPOOL_F_NO_PHYS_CONTIG flag as the last argument to >>> rte_mempool_create_empty(). >>> >>> Note that KNI and some PMDs in 16.07 still require contiguous memory >>> pools, so the trick might not work for your setup. For the KNI try the >>> DPDK's master branch which includes the commit by Ferruh Yigit: >>> >>> 8451269 kni: remove continuous memory restriction >>> >>> Regards, >>> Andriy >>> >>> >>> On Tue, Oct 4, 2016 at 11:38 AM, Renata Saiakhova >>> <Renata.Saiakhova@oneaccess-net.com> wrote: >>>> Hi Sergio, >>>> >>>> thank you for your quick answer. I also tried to allocate 1GB hugepage, >>>> but >>>> seems kernel fails to allocate it: previously I've seen that >>>> HugePages_Total >>>> in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't know >>>> why). >>>> But anyway, if there is no way to control hugepage allocation in the >>>> sense >>>> they are in contiguous memory there is only way to accept it and adapt >>>> the >>>> code that it creates several pools which in total satisfy the requested >>>> size. >>>> >>>> Renata >>>> >>>> >>>> On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote: >>>>> On 04/10/2016 09:00, Renata Saiakhova wrote: >>>>>> Hi all, >>>>>> >>>>>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux >>>>>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I >>>>>> pass a >>>>>> parameter in kernel command line to allocate 512 hugepages of 2 MB at >>>>>> boot >>>>>> time. They are successfully allocated. When an application with dpdk >>>>>> starts >>>>>> it calls rte_pktmbuf_pool_create() which in turns requests internally >>>>>> 649363712 bytes. Those bytes should be allocated from one of >>>>>> rte_memseg. >>>>>> rte_memsegs describes contiguous portions of memory (both physical and >>>>>> virtual) built on hugepages. This allocation fails, because there are >>>>>> no >>>>>> rte_memsegs of this size (or bigger). Further debugging shows that >>>>>> hugepages >>>>>> are allocated in non-contiguous physical memory and therefore >>>>>> rte_memsegs >>>>>> are built respecting gaps in physical memory. >>>>>> Below are the sizes of segments built on hugepages (in bytes) >>>>>> 2097152 >>>>>> 6291456 >>>>>> 2097152 >>>>>> 524288000 >>>>>> 2097152 >>>>>> 532676608 >>>>>> 2097152 >>>>>> 2097152 >>>>>> So there are 5 segments which includes only one hugepage! >>>>>> This behavior is completely different to what I observe with linux >>>>>> kernel >>>>>> 3.8 (used with the same application with dpdk) - where all hugepages >>>>>> are >>>>>> allocated in contiguous memory. >>>>>> Does anyone experience the same issue? Could it be some kernel option >>>>>> which can do the magic? If not, and kernel can allocated hugepages in >>>>>> non-contiguous memory how dpdk is going to resolve it? >>>>>> >>>>> I don't think there is anything we can do to force the kernel to >>>>> pre-allocate contig hugepages on boot. If there was, we wouldn't need to >>>>> do >>>>> all this mapping sorting and grouping we do on DPDK >>>>> as we would rely on the kernel giving us pre-allocated contig hugepages. >>>>> >>>>> If you have plenty of memory one possible work around would be to >>>>> increase >>>>> the number of default hugepages so we are likely to find more contiguous >>>>> ones. >>>>> >>>>> Is using 1GB hugepages a possibility in your case? >>>>> >>>>> Sergio >>>>> >>>>>> Thanks in advance, >>>>>> Renata >>>>>> >>>>> . >>>>> > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory 2016-10-04 14:09 ` Sergio Gonzalez Monroy @ 2016-10-06 11:02 ` tom.barbette 0 siblings, 0 replies; 9+ messages in thread From: tom.barbette @ 2016-10-06 11:02 UTC (permalink / raw) To: Sergio Gonzalez Monroy; +Cc: Andriy Berestovskyy, Renata Saiakhova, users Hi, I had strange change in performances running the same test multiple time, using the system for other things in between (among them, loading and unloading Netmap, which scatters the memory pretty well). It was a very simple forwarding test taking packets on 4 interface and sending them back on the opposite interface. The only difference between each run I could find was the increasing memory scattering. Other strange performance issues included a shift in the performance curve of throughput according to packet size. No explanation though... Tom ----- Mail original ----- De: "Sergio Gonzalez Monroy" <sergio.gonzalez.monroy@intel.com> À: "tom barbette" <tom.barbette@ulg.ac.be>, "Andriy Berestovskyy" <aber@semihalf.com> Cc: "Renata Saiakhova" <Renata.Saiakhova@oneaccess-net.com>, "users" <users@dpdk.org> Envoyé: Mardi 4 Octobre 2016 16:09:29 Objet: Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory Hi folks, In theory, there shouldn't be any performance difference between having a mempool allocated from a single memseg (given the use the same number of hugepages) versus multiple memsegs as it is all done on mempool creation/setup and each mbuf has its own phys address. Tom, I cannot think of a reason why you would have higher memory access for having scatter hugapages vs contig hugepages. Any details on the test you were running? Sergio On 04/10/2016 13:02, tom.barbette@ulg.ac.be wrote: > There is a noticeable performance drop with more scattering of the huge pages. > > I did not measure any difference accurately but I ended up rebooting my DUT between each performance test because the pages get scattered with time and re-launch of the DPDK application instead of the whole machine, because the tests showed higher memory access cost each time I re-launched the application. > > Tom > > ----- Mail original ----- > De: "Andriy Berestovskyy"<aber@semihalf.com> > À: "Renata Saiakhova"<Renata.Saiakhova@oneaccess-net.com> > Cc: "Sergio Gonzalez Monroy"<sergio.gonzalez.monroy@intel.com>, "users"<users@dpdk.org> > Envoyé: Mardi 4 Octobre 2016 13:27:23 > Objet: Re: [dpdk-users] rte_segments: hugepages are not in contiguous memory > > Renata, > In theory 512 contiguous 2MB huge pages might get transparently > promoted to a single 1GB "superpage" and single TLB entry, but I am > not even sure if it is implemented in Linux... > > So, I do not think there will be any noticeable performance difference > between contiguous and non-contiguous 2MB huge pages. But you better > measure it to make sure ;) > > Regards, > Andriy > > On Tue, Oct 4, 2016 at 12:48 PM, Renata Saiakhova > <Renata.Saiakhova@oneaccess-net.com> wrote: >> Hi Andriy, >> >> thanks for your reply. I guess that contiguous memory is requested because >> of the performance reasons. Do you know if I can expect a noticeable >> performance drop using non-contiguous memory? >> >> Renata >> >> >> On 10/04/2016 12:13 PM, Andriy Berestovskyy wrote: >>> Hi Renata, >>> DPDK supports non-contiguous memory pools, but >>> rte_pktmbuf_pool_create() uses rte_mempool_create_empty() with flags >>> set to zero, i.e. requests contiguous memory. >>> >>> As a workaround, in rte_pktmbuf_pool_create() try to pass >>> MEMPOOL_F_NO_PHYS_CONTIG flag as the last argument to >>> rte_mempool_create_empty(). >>> >>> Note that KNI and some PMDs in 16.07 still require contiguous memory >>> pools, so the trick might not work for your setup. For the KNI try the >>> DPDK's master branch which includes the commit by Ferruh Yigit: >>> >>> 8451269 kni: remove continuous memory restriction >>> >>> Regards, >>> Andriy >>> >>> >>> On Tue, Oct 4, 2016 at 11:38 AM, Renata Saiakhova >>> <Renata.Saiakhova@oneaccess-net.com> wrote: >>>> Hi Sergio, >>>> >>>> thank you for your quick answer. I also tried to allocate 1GB hugepage, >>>> but >>>> seems kernel fails to allocate it: previously I've seen that >>>> HugePages_Total >>>> in /proc/meminfo is set to 0, now - kernel hangs at boot time (don't know >>>> why). >>>> But anyway, if there is no way to control hugepage allocation in the >>>> sense >>>> they are in contiguous memory there is only way to accept it and adapt >>>> the >>>> code that it creates several pools which in total satisfy the requested >>>> size. >>>> >>>> Renata >>>> >>>> >>>> On 10/04/2016 10:27 AM, Sergio Gonzalez Monroy wrote: >>>>> On 04/10/2016 09:00, Renata Saiakhova wrote: >>>>>> Hi all, >>>>>> >>>>>> I'm using dpdk 16.04 (I tried 16.07 with the same results) and linux >>>>>> kernel 4.4.20 in a virtual machine (I'm using libvirt framework). I >>>>>> pass a >>>>>> parameter in kernel command line to allocate 512 hugepages of 2 MB at >>>>>> boot >>>>>> time. They are successfully allocated. When an application with dpdk >>>>>> starts >>>>>> it calls rte_pktmbuf_pool_create() which in turns requests internally >>>>>> 649363712 bytes. Those bytes should be allocated from one of >>>>>> rte_memseg. >>>>>> rte_memsegs describes contiguous portions of memory (both physical and >>>>>> virtual) built on hugepages. This allocation fails, because there are >>>>>> no >>>>>> rte_memsegs of this size (or bigger). Further debugging shows that >>>>>> hugepages >>>>>> are allocated in non-contiguous physical memory and therefore >>>>>> rte_memsegs >>>>>> are built respecting gaps in physical memory. >>>>>> Below are the sizes of segments built on hugepages (in bytes) >>>>>> 2097152 >>>>>> 6291456 >>>>>> 2097152 >>>>>> 524288000 >>>>>> 2097152 >>>>>> 532676608 >>>>>> 2097152 >>>>>> 2097152 >>>>>> So there are 5 segments which includes only one hugepage! >>>>>> This behavior is completely different to what I observe with linux >>>>>> kernel >>>>>> 3.8 (used with the same application with dpdk) - where all hugepages >>>>>> are >>>>>> allocated in contiguous memory. >>>>>> Does anyone experience the same issue? Could it be some kernel option >>>>>> which can do the magic? If not, and kernel can allocated hugepages in >>>>>> non-contiguous memory how dpdk is going to resolve it? >>>>>> >>>>> I don't think there is anything we can do to force the kernel to >>>>> pre-allocate contig hugepages on boot. If there was, we wouldn't need to >>>>> do >>>>> all this mapping sorting and grouping we do on DPDK >>>>> as we would rely on the kernel giving us pre-allocated contig hugepages. >>>>> >>>>> If you have plenty of memory one possible work around would be to >>>>> increase >>>>> the number of default hugepages so we are likely to find more contiguous >>>>> ones. >>>>> >>>>> Is using 1GB hugepages a possibility in your case? >>>>> >>>>> Sergio >>>>> >>>>>> Thanks in advance, >>>>>> Renata >>>>>> >>>>> . >>>>> > ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-10-06 11:02 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-10-04 8:00 [dpdk-users] rte_segments: hugepages are not in contiguous memory Renata Saiakhova 2016-10-04 8:27 ` Sergio Gonzalez Monroy 2016-10-04 9:38 ` Renata Saiakhova 2016-10-04 10:13 ` Andriy Berestovskyy 2016-10-04 10:48 ` Renata Saiakhova 2016-10-04 11:27 ` Andriy Berestovskyy 2016-10-04 12:02 ` tom.barbette 2016-10-04 14:09 ` Sergio Gonzalez Monroy 2016-10-06 11:02 ` tom.barbette
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).