From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E162B4254B for ; Fri, 8 Sep 2023 23:03:05 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 617F94028A; Fri, 8 Sep 2023 23:03:05 +0200 (CEST) Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by mails.dpdk.org (Postfix) with ESMTP id 02EFF40285 for ; Fri, 8 Sep 2023 23:03:03 +0200 (CEST) Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-565403bda57so1824987a12.3 for ; Fri, 08 Sep 2023 14:03:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1694206983; x=1694811783; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=ONP/zVSLc+WQI9DCP9j6pUDK8H7lkrZU1qdwN0i75AU=; b=xGaG7fKN4T/3V8+hkzzBJPC0n8OTDdOhDGpJ+pc/y/sBI+Dk8UeX2Rh5R+4DKrb05y Mk4kZYBkc3AGf+l9Wcy+JWE8M557VcD7BqRDzzfMSCH6cudxfjNQ5KBh99tAf/LfGoan Y4lxHaK657Eyqew0Z15r5b9NOHpjuBJalIhHzUZpTWpeRlwExDc5o5JaB0H0wS+MnTEh cgq0Bn+p0wBh2vfsW1RCfjXZKR0IvklCrQWdM+jny4H5PmclvOQ8xWUGHmC/ksmnX3HV /J77XQqHLobiyHDLuiBnZ1W1pA8KQKqNDU/WVy737y2C94pUaADMEDjhJen80vyaLxP2 jbWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694206983; x=1694811783; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ONP/zVSLc+WQI9DCP9j6pUDK8H7lkrZU1qdwN0i75AU=; b=bC+64rkFDuMN0gJ4iApMwHF4QaN01X4KsbZvrnd2CyMc/D392LVRH45Ro/Lq2gu20h Xsntc5Spn/RwTLpiHyAPAnE0ibuIPrzB/4Sc/MAthMC4WcLcLc6b5uTOzkv0o+7sAcAY 76oDhA/06w1Vr3/ufuocOIS8prkpUlhARJccqmYidRdqTaGAPyIz7H4DlUNq8IvFznYH vy5nUSVBj7H3xsdGtxpfPO8rd3alLMACwu+T7fjNZLCtf8IXJIUDURmjH4UScIXnu/Nv QEnyb0QEzY0LxZGDlONjFPcE5HcTxa128NYxi6MqSYS57ksFYz9uOUWqYH5NoKJiyfmt e1Pw== X-Gm-Message-State: AOJu0Yxug488ht58OFZ1AB54iIslq2mfKiv5zDK6XLM9xVpC0bh9R5bY 4LJGBG5NFAYruW+dzcgcN1++QQ== X-Google-Smtp-Source: AGHT+IGT0ymt+ScT/Ea0rwnz7FVcsNZtIQHulK2hTYRCwVYhSwt9QlIiMWC9J9n9GQ7IpvDcxrWJFA== X-Received: by 2002:a17:90b:1958:b0:26b:698f:dda7 with SMTP id nk24-20020a17090b195800b0026b698fdda7mr3812237pjb.32.1694206982984; Fri, 08 Sep 2023 14:03:02 -0700 (PDT) Received: from hermes.local (204-195-112-131.wavecable.com. [204.195.112.131]) by smtp.gmail.com with ESMTPSA id gq1-20020a17090b104100b002639c4f81cesm3457381pjb.3.2023.09.08.14.03.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Sep 2023 14:03:02 -0700 (PDT) Date: Fri, 8 Sep 2023 14:03:01 -0700 From: Stephen Hemminger To: Don Trotter Cc: "jiangheng (G)" , "users@dpdk.org" Subject: Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find suitable memseg_list Message-ID: <20230908140301.19b2bda2@hermes.local> In-Reply-To: <1466053359.5447492.1694183482188@mail.yahoo.com> References: <22952d8e22994649b2c8f1b22997e0ce@huawei.com> <1466053359.5447492.1694183482188@mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Fri, 8 Sep 2023 14:31:22 +0000 (UTC) Don Trotter wrote: > On Thursday, September 7, 2023 at 09:26:34 PM CDT, jiangheng (G) wrote: > =20 > =20 > =20 > Hi > =20 > I've had the same error. It may be caused by this parameter: > =20 > https://github.com/DPDK/dpdk/blob/v22.11/config/rte_config.h#L33C9-L33C33 > =20 > If you use 2MB hugepages, the max memory size is 2MB * 8192 =3D 16GB. Thi= s error will occur when you allocate more than 16GB of memory per numa. > =20 > =C2=A0 > =20 > But should not have this error if you use 512 MB of memory, can you show = RTE_MAX_MEMSEG_PER_LIST in your dpdk? > =20 > Check whether the other sizes hugepage =C2=A0is used, Theoretically, this= error indicates that the rte_eal_init init fails and the program should ex= it. > =C2=A0My initial statement was that the log=C2=A0 was seen "from EAL du= ring EAL init", but that is not right. The call to rte_eal_init() is succes= sful. Later I allocate the memory, using=C2=A0rte_mempool_create("node_pool= ",=C2=A0(128 * 1024 * 1024 - 1), 256, 256, 0, NULL, NULL, NULL, NULL, rte_s= ocket_id(), 0). That is when I see the message. And the call to allocate th= e memory is successful, and I am processing packets and allocating nodes ju= st fine. Here is the information from rte_config.h. > $ grep RTE_MAX_MEM=C2=A0 ./cn98xx-release-output/build/dpdk/config/rte_co= nfig.h#define RTE_MAX_MEMSEG_LISTS 128#define RTE_MAX_MEMSEG_PER_LIST 8192#= define RTE_MAX_MEM_MB_PER_LIST 32768#define RTE_MAX_MEMSEG_PER_TYPE 32768#d= efine RTE_MAX_MEM_MB_PER_TYPE 65536#define RTE_MAX_MEMZONE 2560 > =20 > =C2=A0 > =20 > =E5=8F=91=E4=BB=B6=E4=BA=BA: Don Trotter > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2023=E5=B9=B49=E6=9C=888=E6=97=A5 5= :33 > =E6=94=B6=E4=BB=B6=E4=BA=BA: Stephen Hemminger > =E6=8A=84=E9=80=81: users@dpdk.org > =E4=B8=BB=E9=A2=98: Re: EAL: eal_memalloc_alloc_seg_bulk(): couldn't find= suitable memseg_list > =20 > =C2=A0 > =20 > On Thursday, September 7, 2023 at 03:21:12 PM CDT, Stephen Hemminger wrote: > =20 > =C2=A0 > =20 > =C2=A0 > =20 > On Thu, 7 Sep 2023 19:58:35 +0000 (UTC) > =20 >=20 > Don Trotter wrote: >=20 > >=C2=A0 To clarify, the log message when my application called=C2=A0rte_m= empool_create() to create the "node_pool", and the call succeeded. > > Thanks,Don Trotter > >=C2=A0 =C2=A0 On Thursday, September 7, 2023 at 01:54:08 PM CDT, Don Tro= tter wrote:=C2=A0 > >=C2=A0 > >=C2=A0 Hi, > > I have recently started using DPDK. I am working on a project on Octeon= TX2 with DPDK 11.23.01. I am seeing this message from EAL during EAL init, = but everything is working. > > =C2=A0 =C2=A0=C2=A0EAL: eal_memalloc_alloc_seg_bulk(): couldn't find su= itable memseg_list > > The system has 96GB of memory.=C2=A0 These 2 pools get created and ever= ything works fine. > > mempool @0x13fed3e00=C2=A0 flags=3D10=C2=A0 socket_id=3D0=C2= =A0 pool=3D0x1140300000=C2=A0 iova=3D0x13fed3e00=C2=A0 nb_mem_chunks=3D1=C2= =A0 size=3D65535=C2=A0 populated_size=3D65535=C2=A0 header_size=3D128=C2=A0= elt_size=3D10200=C2=A0 trailer_size=3D40=C2=A0 total_obj_size=3D10368=C2= =A0 private_data_size=3D128=C2=A0 ops_index=3D0=C2=A0 ops_name: =C2=A0 avg bytes/object=3D10368.558602 > > mempool @0x1575d8180=C2=A0 flags=3D10=C2=A0 socket_id=3D-1= =C2=A0 pool=3D0x19ffffd00=C2=A0 iova=3D0x1575d8180=C2=A0 nb_mem_chunks=3D2= =C2=A0 size=3D134217727=C2=A0 populated_size=3D134217727=C2=A0 header_size= =3D128=C2=A0 elt_size=3D256=C2=A0 trailer_size=3D0=C2=A0 total_obj_size=3D3= 84=C2=A0 private_data_size=3D0=C2=A0 ops_index=3D3=C2=A0 ops_name: =C2=A0 avg bytes/object=3D384.000094 > > You read that right. I=E2=80=99ve got 128*1024*1024 256 byte buffers cr= eated for data. > > I also see there is still heap left, although pretty low after. > > Heap id:0=C2=A0 =C2=A0 =C2=A0 =C2=A0 Heap name:socket_0=C2=A0 =C2=A0 = =C2=A0 =C2=A0 Heap_size:55834574848,=C2=A0 =C2=A0 =C2=A0 =C2=A0 Free_size:2= 403644544,=C2=A0 =C2=A0 =C2=A0 =C2=A0 Alloc_size:53430930304,=C2=A0 =C2=A0 = =C2=A0 =C2=A0 Greatest_free_size:536870016,=C2=A0 =C2=A0 =C2=A0 =C2=A0 Allo= c_count:293,=C2=A0 =C2=A0 =C2=A0 =C2=A0 Free_count:5, > > Linux free stats. > > tmp# free -h -w=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 total= =C2=A0 =C2=A0 =C2=A0 =C2=A0 used=C2=A0 =C2=A0 =C2=A0 =C2=A0 free=C2=A0 =C2= =A0 =C2=A0 shared=C2=A0 =C2=A0 =C2=A0buffers=C2=A0 =C2=A0 =C2=A0 =C2=A0cach= e=C2=A0 =C2=A0availableMem:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A095Gi=C2= =A0 =C2=A0 =C2=A0 =C2=A0 88Gi=C2=A0 =C2=A0 =C2=A0 =C2=A06.5Gi=C2=A0 =C2=A0 = =C2=A0 =C2=A0444Mi=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0B=C2=A0 =C2=A0 =C2=A0= =C2=A0470Mi=C2=A0 =C2=A0 =C2=A0 =C2=A0599MiSwap:=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 0B=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0B=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 0B > > Is there anything wrong with that EAL log? Is there a lurking problem? > > Thanks,Don Trotter =20 > =20 >=20 > > > >=C2=A0 =20 >=20 > Did you setup hugepages? > How many and what size? > Is this a NUMA system? >=20 > Also 11.23.01 seems like a weird release number. > The DPDK release numbering scheme is year followed by month. I.e. 22.11 w= as released in November of 2022 > =20 > =C2=A0 > =20 > =C2=A0 > =20 > # cat /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages > =20 > 176 > =20 > =C2=A0 > =20 > Yes it is a NUMA system. > =20 > =C2=A0 > =20 > You are correct sir. The SDK is SDK11.23.01 (2023-01) and DPDK is 22.11. > =20 > =C2=A0 > =20 > Thanks, Don Trotter > =20 > =C2=A0 >=20 > =20 You may need to add hugepages on each NUMA node.