re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

[jiangheng (G)]

[-- Attachment #1: Type: text/plain, Size: 3485 bytes --]

Hi
I've had the same error. It may be caused by this parameter:
https://github.com/DPDK/dpdk/blob/v22.11/config/rte_config.h#L33C9-L33C33
If you use 2MB hugepages, the max memory size is 2MB * 8192 = 16GB. This error will occur when you allocate more than 16GB of memory per numa.

But should not have this error if you use 512 MB of memory, can you show RTE_MAX_MEMSEG_PER_LIST in your dpdk?
Check whether the other sizes hugepage  is used, Theoretically, this error indicates that the rte_eal_init init fails and the program should exit.


发件人: Don Trotter <djtrotter@yahoo.com>
发送时间: 2023年9月8日 5:33
收件人: Stephen Hemminger <stephen@networkplumber.org>
抄送: users@dpdk.org
主题: Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

On Thursday, September 7, 2023 at 03:21:12 PM CDT, Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>> wrote:


On Thu, 7 Sep 2023 19:58:35 +0000 (UTC)

Don Trotter <djtrotter@yahoo.com<mailto:djtrotter@yahoo.com>> wrote:

>  To clarify, the log message when my application called rte_mempool_create() to create the "node_pool", and the call succeeded.
> Thanks,Don Trotter
>    On Thursday, September 7, 2023 at 01:54:08 PM CDT, Don Trotter <djtrotter@yahoo.com<mailto:djtrotter@yahoo.com>> wrote:
>
>  Hi,
> I have recently started using DPDK. I am working on a project on OcteonTX2 with DPDK 11.23.01. I am seeing this message from EAL during EAL init, but everything is working.
>     EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
> The system has 96GB of memory.  These 2 pools get created and everything works fine.
> mempool <mbuf_pool>@0x13fed3e00  flags=10  socket_id=0  pool=0x1140300000  iova=0x13fed3e00  nb_mem_chunks=1  size=65535  populated_size=65535  header_size=128  elt_size=10200  trailer_size=40  total_obj_size=10368  private_data_size=128  ops_index=0  ops_name: <cn9k_mempool_ops>  avg bytes/object=10368.558602
> mempool <node_pool>@0x1575d8180  flags=10  socket_id=-1  pool=0x19ffffd00  iova=0x1575d8180  nb_mem_chunks=2  size=134217727  populated_size=134217727  header_size=128  elt_size=256  trailer_size=0  total_obj_size=384  private_data_size=0  ops_index=3  ops_name: <ring_mp_mc>  avg bytes/object=384.000094
> You read that right. I’ve got 128*1024*1024 256 byte buffers created for data.
> I also see there is still heap left, although pretty low after.
> Heap id:0        Heap name:socket_0        Heap_size:55834574848,        Free_size:2403644544,        Alloc_size:53430930304,        Greatest_free_size:536870016,        Alloc_count:293,        Free_count:5,
> Linux free stats.
> tmp# free -h -w              total        used        free      shared     buffers       cache   availableMem:           95Gi        88Gi       6.5Gi       444Mi          0B       470Mi       599MiSwap:            0B          0B          0B
> Is there anything wrong with that EAL log? Is there a lurking problem?
> Thanks,Don Trotter

>
>

Did you setup hugepages?
How many and what size?
Is this a NUMA system?

Also 11.23.01 seems like a weird release number.
The DPDK release numbering scheme is year followed by month. I.e. 22.11 was released in November of 2022


# cat /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
176

Yes it is a NUMA system.

You are correct sir. The SDK is SDK11.23.01 (2023-01) and DPDK is 22.11.

Thanks, Don Trotter


[-- Attachment #2: Type: text/html, Size: 12034 bytes --]

Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

[Don Trotter]

[-- Attachment #1: Type: text/plain, Size: 4533 bytes --]

 On Thursday, September 7, 2023 at 09:26:34 PM CDT, jiangheng (G) <jiangheng14@huawei.com> wrote:
 
 
 
Hi
 
I've had the same error. It may be caused by this parameter:
 
https://github.com/DPDK/dpdk/blob/v22.11/config/rte_config.h#L33C9-L33C33
 
If you use 2MB hugepages, the max memory size is 2MB * 8192 = 16GB. This error will occur when you allocate more than 16GB of memory per numa.
 
  
 
But should not have this error if you use 512 MB of memory, can you show RTE_MAX_MEMSEG_PER_LIST in your dpdk?
 
Check whether the other sizes hugepage  is used, Theoretically, this error indicates that the rte_eal_init init fails and the program should exit.
   My initial statement was that the log  was seen "from EAL during EAL init", but that is not right. The call to rte_eal_init() is successful. Later I allocate the memory, using rte_mempool_create("node_pool", (128 * 1024 * 1024 - 1), 256, 256, 0, NULL, NULL, NULL, NULL, rte_socket_id(), 0). That is when I see the message. And the call to allocate the memory is successful, and I am processing packets and allocating nodes just fine. Here is the information from rte_config.h.
$ grep RTE_MAX_MEM  ./cn98xx-release-output/build/dpdk/config/rte_config.h#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#define RTE_MAX_MEM_MB_PER_LIST 32768#define RTE_MAX_MEMSEG_PER_TYPE 32768#define RTE_MAX_MEM_MB_PER_TYPE 65536#define RTE_MAX_MEMZONE 2560
 
  
 
发件人: Don Trotter <djtrotter@yahoo.com>
发送时间: 2023年9月8日 5:33
收件人: Stephen Hemminger <stephen@networkplumber.org>
抄送: users@dpdk.org
主题: Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
 
  
 
On Thursday, September 7, 2023 at 03:21:12 PM CDT, Stephen Hemminger <stephen@networkplumber.org> wrote:
 
  
 
  
 
On Thu, 7 Sep 2023 19:58:35 +0000 (UTC)
 

Don Trotter <djtrotter@yahoo.com> wrote:

>  To clarify, the log message when my application called rte_mempool_create() to create the "node_pool", and the call succeeded.
> Thanks,Don Trotter
>    On Thursday, September 7, 2023 at 01:54:08 PM CDT, Don Trotter <djtrotter@yahoo.com> wrote: 
> 
>  Hi,
> I have recently started using DPDK. I am working on a project on OcteonTX2 with DPDK 11.23.01. I am seeing this message from EAL during EAL init, but everything is working.
>     EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
> The system has 96GB of memory.  These 2 pools get created and everything works fine.
> mempool <mbuf_pool>@0x13fed3e00  flags=10  socket_id=0  pool=0x1140300000  iova=0x13fed3e00  nb_mem_chunks=1  size=65535  populated_size=65535  header_size=128  elt_size=10200  trailer_size=40  total_obj_size=10368  private_data_size=128  ops_index=0  ops_name: <cn9k_mempool_ops>  avg bytes/object=10368.558602
> mempool <node_pool>@0x1575d8180  flags=10  socket_id=-1  pool=0x19ffffd00  iova=0x1575d8180  nb_mem_chunks=2  size=134217727  populated_size=134217727  header_size=128  elt_size=256  trailer_size=0  total_obj_size=384  private_data_size=0  ops_index=3  ops_name: <ring_mp_mc>  avg bytes/object=384.000094
> You read that right. I’ve got 128*1024*1024 256 byte buffers created for data.
> I also see there is still heap left, although pretty low after.
> Heap id:0        Heap name:socket_0        Heap_size:55834574848,        Free_size:2403644544,        Alloc_size:53430930304,        Greatest_free_size:536870016,        Alloc_count:293,        Free_count:5,
> Linux free stats.
> tmp# free -h -w              total        used        free      shared     buffers       cache   availableMem:           95Gi        88Gi       6.5Gi       444Mi          0B       470Mi       599MiSwap:            0B          0B          0B
> Is there anything wrong with that EAL log? Is there a lurking problem?
> Thanks,Don Trotter
 

>
> 

Did you setup hugepages?
How many and what size?
Is this a NUMA system?

Also 11.23.01 seems like a weird release number.
The DPDK release numbering scheme is year followed by month. I.e. 22.11 was released in November of 2022
 
  
 
  
 
# cat /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
 
176
 
  
 
Yes it is a NUMA system.
 
  
 
You are correct sir. The SDK is SDK11.23.01 (2023-01) and DPDK is 22.11.
 
  
 
Thanks, Don Trotter
 
  

   

[-- Attachment #2: Type: text/html, Size: 12267 bytes --]

Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

[Stephen Hemminger]

On Fri, 8 Sep 2023 14:31:22 +0000 (UTC)
Don Trotter <djtrotter@yahoo.com> wrote:

>  On Thursday, September 7, 2023 at 09:26:34 PM CDT, jiangheng (G) <jiangheng14@huawei.com> wrote:
>  
>  
>  
> Hi
>  
> I've had the same error. It may be caused by this parameter:
>  
> https://github.com/DPDK/dpdk/blob/v22.11/config/rte_config.h#L33C9-L33C33
>  
> If you use 2MB hugepages, the max memory size is 2MB * 8192 = 16GB. This error will occur when you allocate more than 16GB of memory per numa.
>  
>   
>  
> But should not have this error if you use 512 MB of memory, can you show RTE_MAX_MEMSEG_PER_LIST in your dpdk?
>  
> Check whether the other sizes hugepage  is used, Theoretically, this error indicates that the rte_eal_init init fails and the program should exit.
>    My initial statement was that the log  was seen "from EAL during EAL init", but that is not right. The call to rte_eal_init() is successful. Later I allocate the memory, using rte_mempool_create("node_pool", (128 * 1024 * 1024 - 1), 256, 256, 0, NULL, NULL, NULL, NULL, rte_socket_id(), 0). That is when I see the message. And the call to allocate the memory is successful, and I am processing packets and allocating nodes just fine. Here is the information from rte_config.h.
> $ grep RTE_MAX_MEM  ./cn98xx-release-output/build/dpdk/config/rte_config.h#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#define RTE_MAX_MEM_MB_PER_LIST 32768#define RTE_MAX_MEMSEG_PER_TYPE 32768#define RTE_MAX_MEM_MB_PER_TYPE 65536#define RTE_MAX_MEMZONE 2560
>  
>   
>  
> 发件人: Don Trotter <djtrotter@yahoo.com>
> 发送时间: 2023年9月8日 5:33
> 收件人: Stephen Hemminger <stephen@networkplumber.org>
> 抄送: users@dpdk.org
> 主题: Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
>  
>   
>  
> On Thursday, September 7, 2023 at 03:21:12 PM CDT, Stephen Hemminger <stephen@networkplumber.org> wrote:
>  
>   
>  
>   
>  
> On Thu, 7 Sep 2023 19:58:35 +0000 (UTC)
>  
> 
> Don Trotter <djtrotter@yahoo.com> wrote:
> 
> >  To clarify, the log message when my application called rte_mempool_create() to create the "node_pool", and the call succeeded.
> > Thanks,Don Trotter
> >    On Thursday, September 7, 2023 at 01:54:08 PM CDT, Don Trotter <djtrotter@yahoo.com> wrote: 
> > 
> >  Hi,
> > I have recently started using DPDK. I am working on a project on OcteonTX2 with DPDK 11.23.01. I am seeing this message from EAL during EAL init, but everything is working.
> >     EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
> > The system has 96GB of memory.  These 2 pools get created and everything works fine.
> > mempool <mbuf_pool>@0x13fed3e00  flags=10  socket_id=0  pool=0x1140300000  iova=0x13fed3e00  nb_mem_chunks=1  size=65535  populated_size=65535  header_size=128  elt_size=10200  trailer_size=40  total_obj_size=10368  private_data_size=128  ops_index=0  ops_name: <cn9k_mempool_ops>  avg bytes/object=10368.558602
> > mempool <node_pool>@0x1575d8180  flags=10  socket_id=-1  pool=0x19ffffd00  iova=0x1575d8180  nb_mem_chunks=2  size=134217727  populated_size=134217727  header_size=128  elt_size=256  trailer_size=0  total_obj_size=384  private_data_size=0  ops_index=3  ops_name: <ring_mp_mc>  avg bytes/object=384.000094
> > You read that right. I’ve got 128*1024*1024 256 byte buffers created for data.
> > I also see there is still heap left, although pretty low after.
> > Heap id:0        Heap name:socket_0        Heap_size:55834574848,        Free_size:2403644544,        Alloc_size:53430930304,        Greatest_free_size:536870016,        Alloc_count:293,        Free_count:5,
> > Linux free stats.
> > tmp# free -h -w              total        used        free      shared     buffers       cache   availableMem:           95Gi        88Gi       6.5Gi       444Mi          0B       470Mi       599MiSwap:            0B          0B          0B
> > Is there anything wrong with that EAL log? Is there a lurking problem?
> > Thanks,Don Trotter  
>  
> 
> >
> >   
> 
> Did you setup hugepages?
> How many and what size?
> Is this a NUMA system?
> 
> Also 11.23.01 seems like a weird release number.
> The DPDK release numbering scheme is year followed by month. I.e. 22.11 was released in November of 2022
>  
>   
>  
>   
>  
> # cat /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
>  
> 176
>  
>   
>  
> Yes it is a NUMA system.
>  
>   
>  
> You are correct sir. The SDK is SDK11.23.01 (2023-01) and DPDK is 22.11.
>  
>   
>  
> Thanks, Don Trotter
>  
>   
> 
>    

You may need to add hugepages on each NUMA node.

Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

[Don Trotter]

[-- Attachment #1: Type: text/plain, Size: 2727 bytes --]

 On Thursday, September 7, 2023 at 03:21:12 PM CDT, Stephen Hemminger <stephen@networkplumber.org> wrote:

On Thu, 7 Sep 2023 19:58:35 +0000 (UTC)
Don Trotter <djtrotter@yahoo.com> wrote:

>  To clarify, the log message when my application called rte_mempool_create() to create the "node_pool", and the call succeeded.
> Thanks,Don Trotter
>    On Thursday, September 7, 2023 at 01:54:08 PM CDT, Don Trotter <djtrotter@yahoo.com> wrote: 
> 
>  Hi,
> I have recently started using DPDK. I am working on a project on OcteonTX2 with DPDK 11.23.01. I am seeing this message from EAL during EAL init, but everything is working.
>     EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
> The system has 96GB of memory.  These 2 pools get created and everything works fine.
> mempool <mbuf_pool>@0x13fed3e00  flags=10  socket_id=0  pool=0x1140300000  iova=0x13fed3e00  nb_mem_chunks=1  size=65535  populated_size=65535  header_size=128  elt_size=10200  trailer_size=40  total_obj_size=10368  private_data_size=128  ops_index=0  ops_name: <cn9k_mempool_ops>  avg bytes/object=10368.558602
> mempool <node_pool>@0x1575d8180  flags=10  socket_id=-1  pool=0x19ffffd00  iova=0x1575d8180  nb_mem_chunks=2  size=134217727  populated_size=134217727  header_size=128  elt_size=256  trailer_size=0  total_obj_size=384  private_data_size=0  ops_index=3  ops_name: <ring_mp_mc>  avg bytes/object=384.000094
> You read that right. I’ve got 128*1024*1024 256 byte buffers created for data.
> I also see there is still heap left, although pretty low after.
> Heap id:0        Heap name:socket_0        Heap_size:55834574848,        Free_size:2403644544,        Alloc_size:53430930304,        Greatest_free_size:536870016,        Alloc_count:293,        Free_count:5,
> Linux free stats.
> tmp# free -h -w              total        used        free      shared     buffers       cache   availableMem:           95Gi        88Gi       6.5Gi       444Mi          0B       470Mi       599MiSwap:            0B          0B          0B
> Is there anything wrong with that EAL log? Is there a lurking problem?
> Thanks,Don Trotter
>
> 

Did you setup hugepages?
How many and what size?
Is this a NUMA system?

Also 11.23.01 seems like a weird release number.
The DPDK release numbering scheme is year followed by month. I.e. 22.11 was released in November of 2022

# cat /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages176
Yes it is a NUMA system.
You are correct sir. The SDK is SDK11.23.01 (2023-01) and DPDK is 22.11.   
Thanks, Don Trotter
  

[-- Attachment #2: Type: text/html, Size: 6781 bytes --]

Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

[Stephen Hemminger]

On Thu, 7 Sep 2023 19:58:35 +0000 (UTC)
Don Trotter <djtrotter@yahoo.com> wrote:

>  To clarify, the log message when my application called rte_mempool_create() to create the "node_pool", and the call succeeded.
> Thanks,Don Trotter
>     On Thursday, September 7, 2023 at 01:54:08 PM CDT, Don Trotter <djtrotter@yahoo.com> wrote:  
>  
>  Hi,
> I have recently started using DPDK. I am working on a project on OcteonTX2 with DPDK 11.23.01. I am seeing this message from EAL during EAL init, but everything is working.
>     EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
> The system has 96GB of memory.  These 2 pools get created and everything works fine.
> mempool <mbuf_pool>@0x13fed3e00  flags=10  socket_id=0  pool=0x1140300000  iova=0x13fed3e00  nb_mem_chunks=1  size=65535  populated_size=65535  header_size=128  elt_size=10200  trailer_size=40  total_obj_size=10368  private_data_size=128  ops_index=0  ops_name: <cn9k_mempool_ops>  avg bytes/object=10368.558602
> mempool <node_pool>@0x1575d8180  flags=10  socket_id=-1  pool=0x19ffffd00  iova=0x1575d8180  nb_mem_chunks=2  size=134217727  populated_size=134217727  header_size=128  elt_size=256  trailer_size=0  total_obj_size=384  private_data_size=0  ops_index=3  ops_name: <ring_mp_mc>  avg bytes/object=384.000094
> You read that right. I’ve got 128*1024*1024 256 byte buffers created for data.
> I also see there is still heap left, although pretty low after.
> Heap id:0        Heap name:socket_0        Heap_size:55834574848,        Free_size:2403644544,        Alloc_size:53430930304,        Greatest_free_size:536870016,        Alloc_count:293,        Free_count:5,
> Linux free stats.
> tmp# free -h -w              total        used        free      shared     buffers       cache   availableMem:           95Gi        88Gi       6.5Gi       444Mi          0B       470Mi       599MiSwap:            0B          0B          0B
> Is there anything wrong with that EAL log? Is there a lurking problem?
> Thanks,Don Trotter
> 
>   

Did you setup hugepages?
How many and what size?
Is this a NUMA system?

Also 11.23.01 seems like a weird release number.
The DPDK release numbering scheme is year followed by month. I.e. 22.11 was released in November of 2022

Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

[Don Trotter]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]

 To clarify, the log message when my application called rte_mempool_create() to create the "node_pool", and the call succeeded.
Thanks,Don Trotter
    On Thursday, September 7, 2023 at 01:54:08 PM CDT, Don Trotter <djtrotter@yahoo.com> wrote:  
 
 Hi,
I have recently started using DPDK. I am working on a project on OcteonTX2 with DPDK 11.23.01. I am seeing this message from EAL during EAL init, but everything is working.
    EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
The system has 96GB of memory.  These 2 pools get created and everything works fine.
mempool <mbuf_pool>@0x13fed3e00  flags=10  socket_id=0  pool=0x1140300000  iova=0x13fed3e00  nb_mem_chunks=1  size=65535  populated_size=65535  header_size=128  elt_size=10200  trailer_size=40  total_obj_size=10368  private_data_size=128  ops_index=0  ops_name: <cn9k_mempool_ops>  avg bytes/object=10368.558602
mempool <node_pool>@0x1575d8180  flags=10  socket_id=-1  pool=0x19ffffd00  iova=0x1575d8180  nb_mem_chunks=2  size=134217727  populated_size=134217727  header_size=128  elt_size=256  trailer_size=0  total_obj_size=384  private_data_size=0  ops_index=3  ops_name: <ring_mp_mc>  avg bytes/object=384.000094
You read that right. I’ve got 128*1024*1024 256 byte buffers created for data.
I also see there is still heap left, although pretty low after.
Heap id:0        Heap name:socket_0        Heap_size:55834574848,        Free_size:2403644544,        Alloc_size:53430930304,        Greatest_free_size:536870016,        Alloc_count:293,        Free_count:5,
Linux free stats.
tmp# free -h -w              total        used        free      shared     buffers       cache   availableMem:           95Gi        88Gi       6.5Gi       444Mi          0B       470Mi       599MiSwap:            0B          0B          0B
Is there anything wrong with that EAL log? Is there a lurking problem?
Thanks,Don Trotter

  

[-- Attachment #2: Type: text/html, Size: 4495 bytes --]

EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list

[Don Trotter]

[-- Attachment #1: Type: text/plain, Size: 1807 bytes --]

Hi,
I have recently started using DPDK. I am working on a project on OcteonTX2 with DPDK 11.23.01. I am seeing this message from EAL during EAL init, but everything is working.
    EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list
The system has 96GB of memory.  These 2 pools get created and everything works fine.
mempool <mbuf_pool>@0x13fed3e00  flags=10  socket_id=0  pool=0x1140300000  iova=0x13fed3e00  nb_mem_chunks=1  size=65535  populated_size=65535  header_size=128  elt_size=10200  trailer_size=40  total_obj_size=10368  private_data_size=128  ops_index=0  ops_name: <cn9k_mempool_ops>  avg bytes/object=10368.558602
mempool <node_pool>@0x1575d8180  flags=10  socket_id=-1  pool=0x19ffffd00  iova=0x1575d8180  nb_mem_chunks=2  size=134217727  populated_size=134217727  header_size=128  elt_size=256  trailer_size=0  total_obj_size=384  private_data_size=0  ops_index=3  ops_name: <ring_mp_mc>  avg bytes/object=384.000094
You read that right. I’ve got 128*1024*1024 256 byte buffers created for data.
I also see there is still heap left, although pretty low after.
Heap id:0        Heap name:socket_0        Heap_size:55834574848,        Free_size:2403644544,        Alloc_size:53430930304,        Greatest_free_size:536870016,        Alloc_count:293,        Free_count:5,
Linux free stats.
tmp# free -h -w              total        used        free      shared     buffers       cache   availableMem:           95Gi        88Gi       6.5Gi       444Mi          0B       470Mi       599MiSwap:            0B          0B          0B
Is there anything wrong with that EAL log? Is there a lurking problem?
Thanks,Don Trotter


[-- Attachment #2: Type: text/html, Size: 3398 bytes --]