From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 92F35A04FD; Tue, 27 Dec 2022 10:00:28 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4344C410FC; Tue, 27 Dec 2022 10:00:28 +0100 (CET) Received: from loongson.cn (mail.loongson.cn [114.242.206.163]) by mails.dpdk.org (Postfix) with ESMTP id ACE7940E2D for ; Tue, 27 Dec 2022 10:00:26 +0100 (CET) Received: from loongson.cn (unknown [10.20.42.19]) by gateway (Coremail) with SMTP id _____8Axy+ontKpjRtAIAA--.19409S3; Tue, 27 Dec 2022 17:00:24 +0800 (CST) Received: from localhost.localdomain (unknown [10.20.42.19]) by localhost.localdomain (Coremail) with SMTP id AQAAf8BxsOQjtKpjM9YNAA--.47032S3; Tue, 27 Dec 2022 17:00:22 +0800 (CST) Subject: Re: [PATCH] malloc: enhance NUMA affinity heuristic To: David Marchand Cc: dev@dpdk.org, olivier.matz@6wind.com, ferruh.yigit@amd.com, kaisenx.you@intel.com, Anatoly Burakov , Bruce Richardson References: <20221221104858.296530-1-david.marchand@redhat.com> From: zhoumin Message-ID: <0572b450-609d-0053-6fe3-beab118e7020@loongson.cn> Date: Tue, 27 Dec 2022 17:00:19 +0800 User-Agent: Mozilla/5.0 (X11; Linux mips64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-CM-TRANSID: AQAAf8BxsOQjtKpjM9YNAA--.47032S3 X-CM-SenderInfo: 52kr3ztlq6z05rqj20fqof0/ X-Coremail-Antispam: 1Uk129KBjvJXoWxXw13uF15ur15tw1Uuw4Durg_yoWrWr4Upw 40gF18GFWkKry8W3Wrt3W0q34rX347Ga15XrWjv3WkAr1DG342qr4YqF1qgFsrAr48tryr ZF1UJw12vFyYkaDanT9S1TB71UUUUjUqnTZGkaVYY2UrUUUUj1kv1TuYvTs0mT0YCTnIWj qI5I8CrVACY4xI64kE6c02F40Ex7xfYxn0WfASr-VFAUDa7-sFnT9fnUUIcSsGvfJTRUUU bqxYFVCjjxCrM7AC8VAFwI0_Jr0_Gr1l1xkIjI8I6I8E6xAIw20EY4v20xvaj40_Wr0E3s 1l1IIY67AEw4v_Jr0_Jr4l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxSw2x7M28EF7xv wVC0I7IYx2IY67AKxVW8JVW5JwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxVW8JVWxJwA2z4 x0Y4vEx4A2jsIE14v26r4UJVWxJr1l84ACjcxK6I8E87Iv6xkF7I0E14v26r4UJVWxJr1l n4kS14v26r1Y6r17M2AIxVAIcxkEcVAq07x20xvEncxIr21l57IF6xkI12xvs2x26I8E6x ACxx1l5I8CrVACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r126r1DMcIj6I8E 87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IY64vIr41lc7I2V7IY0V AS07AlzVAYIcxG8wCY1x0262kKe7AKxVWUAVWUtwCF04k20xvY0x0EwIxGrwCFx2IqxVCF s4IE7xkEbVWUJVW8JwCFI7km07C267AKxVWUXVWUAwC20s026c02F40E14v26r1j6r18MI 8I3I0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_JF0_Jw1lIxkGc2Ij64vIr41l IxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Jr0_Gr1lIx AIcVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2 jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU82ZX5UUUUU== X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi David, First of all, I sincerely apologize for the late reply. I had checked this issue carefully and had some useful findings. On Wed, Dec 21, 2022 at 22:57 PM, David Marchand wrote: > Hello Min, > > On Wed, Dec 21, 2022 at 11:49 AM David Marchand > wrote: >> Trying to allocate memory on the first detected numa node has less >> chance to find some memory actually available rather than on the main >> lcore numa node (especially when the DPDK application is started only >> on one numa node). >> >> Signed-off-by: David Marchand > I see a failure in the loongarch CI. > > Running binary with > argv[]:'/home/zhoumin/dpdk/build/app/test/dpdk-test' > '--file-prefix=eal_flags_c_opt_autotest' '--proc-type=secondary' > '--lcores' '0-1,2@(5-7),(3-5)@(0,2),(0,6),7' > Error - process did not run ok with valid corelist value > Test Failed > > The logs don't give the full picture (though it is not LoongArch CI fault). > > I tried to read back on past mail exchanges about the loongarch > server, but I did not find the info. > I suspect cores 5 to 7 belong to different numa nodes, can you confirm? The cores 5 to 7 belong to the same numa node (NUMA node1) on the Loongson-3C5000LL CPU on which LoongArch DPDK CI runs. > > I'll post a new revision to account for this case. > The LoongArch DPDK CI uses the core 0-7 to run all the DPDK unit tests by adding the arg '-l 0-7' in the meson test args. In the above test case, the arg '--lcores' '0-1,2@(5-7),(3-5)@(0,2),(0,6),7' will make the lcore 0 and 6 to run on the core 0 or 6. The logs of EAL will make it more clear when I set the log level of EAL to debug as follows: EAL: Main lcore 0 is ready (tid=fff3ee18f0;cpuset=[0,6]) EAL: lcore 1 is ready (tid=fff2de4cf0;cpuset=[1]) EAL: lcore 2 is ready (tid=fff25e0cf0;cpuset=[5,6,7]) EAL: lcore 5 is ready (tid=fff0dd4cf0;cpuset=[0,2]) EAL: lcore 4 is ready (tid=fff15d8cf0;cpuset=[0,2]) EAL: lcore 3 is ready (tid=fff1ddccf0;cpuset=[0,2]) EAL: lcore 7 is ready (tid=ffdb7f8cf0;cpuset=[7]) EAL: lcore 6 is ready (tid=ffdbffccf0;cpuset=[0,6]) However, The cores 0 and 6 belong to different numa nodes on the Loongson-3C5000LL CPU. The core 0 belongs to NUMA node 0 and the core 6 belongs to NUMA node 1 as follows: $ lscpu Architecture:        loongarch64 Byte Order:          Little Endian CPU(s):              32 On-line CPU(s) list: 0-31 Thread(s) per core:  1 Core(s) per socket:  4 Socket(s):           8 NUMA node(s):        8 ... NUMA node0 CPU(s):   0-3 NUMA node1 CPU(s):   4-7 NUMA node2 CPU(s):   8-11 NUMA node3 CPU(s):   12-15 NUMA node4 CPU(s):   16-19 NUMA node5 CPU(s):   20-23 NUMA node6 CPU(s):   24-27 NUMA node7 CPU(s):   28-31 ... So the socket_id for the lcore 0 and 6 will be set to -1 which can be seen from the thread_update_affinity(). Meanwhile, I print out the socket_id for the lcore 0 to RTE_MAX_LCORE - 1 as follows: lcore_config[*].socket_id: -1 0 1 0 0 0 -1 1 2 2 2 2 3 3 3 3 4 4 4 4 5 5 5 5 6 6 6 6 7 7 7 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 In this test case, the modified malloc_get_numa_socket() will return -1 which caused a memory allocation failure. Whether it is acceptable in DPDK that the socket_id for a lcore is -1? If it's ok, maybe we can check the socket_id of main lcore before using it, such as: diff --git a/lib/eal/common/malloc_heap.c b/lib/eal/common/malloc_heap.c index d7c410b786..3ee19aee15 100644 --- a/lib/eal/common/malloc_heap.c +++ b/lib/eal/common/malloc_heap.c @@ -717,6 +717,10 @@ malloc_get_numa_socket(void)                         return socket_id;         } +       socket_id = rte_lcore_to_socket_id(rte_get_main_lcore()); +       if (socket_id != (unsigned int)SOCKET_ID_ANY) +               return socket_id; +         return rte_socket_id_by_idx(0);  }