From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 27EC3423A1 for ; Tue, 10 Jan 2023 11:17:14 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 22C3340691; Tue, 10 Jan 2023 11:17:14 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id 62ABD40689 for ; Tue, 10 Jan 2023 11:17:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673345831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=b3aJXOtkaDSSRj0vjpmY7sZSWLbWVymdW+APjePQWq0=; b=KFixskp5xDtKF+SGF1EVqQXIRyBeF/2ymuAM2Er1vL2L0yB8EPcY2ZpyBCclsp2OflmrID nlw8IgAKDo8WXnHu2Q8JiQnPejcX9C+5ym2noUhFRv8O4Oy34r/cqtohqYPV8H/Vjvf9Bv tMMfXkkwG1mKqYRaifmdG5EJrFiK7t8= Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-255-6RcHy2CdMP2IRVkeAiDnZw-1; Tue, 10 Jan 2023 05:17:10 -0500 X-MC-Unique: 6RcHy2CdMP2IRVkeAiDnZw-1 Received: by mail-pl1-f197.google.com with SMTP id u2-20020a17090341c200b00192bc565119so8180058ple.16 for ; Tue, 10 Jan 2023 02:17:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=b3aJXOtkaDSSRj0vjpmY7sZSWLbWVymdW+APjePQWq0=; b=EqSx7tPLacczlNU4Dg1z3Sd0dghCFp2GlD6xr9AZ1FiaJl7WOabjN9D++XCNqcTIJl zVk+O8kdfadFPk10xnPTSrJFzAI6BT55dw1/aR2joTmPpkFjLRCIOhSXfluXslN88HT2 dfNnoFlzmsIvSKKipsOzpFj/30rZGE/09Z/U5vzcpvjOnTAomZE4s1/GqmSaR4Bj9E6M 64F5QSK0V8pZB0A6Ot/fVO6M90f7M6sW4U8Cc3B7seUAmeJqiGcP98kUizbuTTMFkEJw mQ9gogVSqJV+x+DRfs8CPRETy7VIXenP4klElO0xiFyu0tk1ocZxXHtun2W4TmCYebIs fS2Q== X-Gm-Message-State: AFqh2ko23Qfv0LHrgB2clrJGbsfc0Z8Eycop1QGoC7N+ZVkMqZ48lXL2 nuiT226vHOd5l7uaXTe1RGH1qvlFozJAxVLVpH2S387PmiiZFNDmCn+G3n4iNVbhZ2kIH63zZs5 nIaTqB+vrKnuke/AXUwf/dvM= X-Received: by 2002:a63:1d46:0:b0:478:e150:e028 with SMTP id d6-20020a631d46000000b00478e150e028mr4754811pgm.284.1673345829830; Tue, 10 Jan 2023 02:17:09 -0800 (PST) X-Google-Smtp-Source: AMrXdXttTApT/J27bnnupMEQVVxKjSwuQY9sCJkcCTyejt6mXV3N36wEafNrM0Dv3hW7yakO8Z0u94BFFQnSo7mLy3s= X-Received: by 2002:a63:1d46:0:b0:478:e150:e028 with SMTP id d6-20020a631d46000000b00478e150e028mr4754797pgm.284.1673345829471; Tue, 10 Jan 2023 02:17:09 -0800 (PST) MIME-Version: 1.0 References: <20221117065726.277672-1-kaisenx.you@intel.com> <3ad04278-59c0-0c60-5c8c-9e57f33bb0de@amd.com> In-Reply-To: From: David Marchand Date: Tue, 10 Jan 2023 11:16:58 +0100 Message-ID: Subject: Re: [PATCH] net/iavf:fix slow memory allocation To: "You, KaisenX" Cc: Ferruh Yigit , "dev@dpdk.org" , "Burakov, Anatoly" , "stable@dpdk.org" , "Yang, Qiming" , "Zhou, YidingX" , "Wu, Jingjing" , "Xing, Beilei" , "Zhang, Qi Z" , Luca Boccassi , "Mcnamara, John" , Kevin Traynor X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hello, On Tue, Dec 27, 2022 at 7:06 AM You, KaisenX wrote: > > > > > > > I tried to play a bit with a E810 nic on a dual numa and I > > > > > > > can't see anything wrong for now. > > > > > > > Can you provide a simple and small reproducer of your issue? > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > This is my environment: > > > > > > Enter "lscpu" on the command line: > > > > > > NUMA: > > > > > > NUMA node(s): 2 > > > > > > NUMA node0 CPU(S) : 0-27,56-83 > > > > > > NUMA node1 CPU(S) : 28-55,84-111 > > > > > > > > > > > > List the steps to reproduce the issue: > > > > > > > > > > > > 1. create vf and blind to dpdk > > > > > > echo 1 > /sys/bus/pci/devices/0000\:ca\:00.0/sriov_ numvfs > > > > > > ./usertools/dpdk-devbind. py -b vfio-pci 0000:ca:01.0 2. launch > > > > > > testpmd ./x86_ 64-native-linuxapp-clang/app/dpdk-testpmd -l > > > > > > 28-48 -n 4 -a 0000:ca:01.0 --file-prefix=dpdk_ 525342_ > > > > > > 20221104042659 -- -i > > > > > > --rxq=256 --txq=256 > > > > > > --total-num-mbufs=500000 > > > > > > > > > > > > Parameter Description: > > > > > > "-l 28-48":The range of parameter values after "-l" must be on > > > > > > "NUMA > > > > > node1 CPU(S)" > > > > > > "0000:ca:01.0":inset on node1 > > > > > - Back to your topic. > > > > > Can you try this simple hack: > > > > > > > > > > diff --git a/lib/eal/common/eal_common_thread.c > > > > > b/lib/eal/common/eal_common_thread.c > > > > > index c5d8b4327d..92160c7fa6 100644 > > > > > --- a/lib/eal/common/eal_common_thread.c > > > > > +++ b/lib/eal/common/eal_common_thread.c > > > > > @@ -253,6 +253,7 @@ static void *ctrl_thread_init(void *arg) > > > > > void *routine_arg = params->arg; > > > > > > > > > > __rte_thread_init(rte_lcore_id(), cpuset); > > > > > + RTE_PER_LCORE(_socket_id) = SOCKET_ID_ANY; > > > > > params->ret = pthread_setaffinity_np(pthread_self(), > > > sizeof(*cpuset), > > > > > cpuset); > > > > > if (params->ret != 0) { > > > > > > > > > Thanks for your advice. > > > > > > > > But this issue still exists after I tried. > > > > > > Ok, I think I understand what is wrong... but I am still guessing as I > > > am not sure what your "issue" is. > > > Can you have a try with: > > > https://patchwork.dpdk.org/project/dpdk/patch/20221221104858.296530- > > 1- > > > david.marchand@redhat.com/ > > > > > > Thanks. > > > > > I think this issue is similar to the description in the patch you gave me. > > > > when the DPDK application is started only on one numa node, Interrupt > > thread find memory on another numa node. This leads to a whole set of > > memory allocation/release operations every time when "rte_malloc" is called. > > This is the root cause of this issue. > > > > This issue can be solved after I tried. > > Thanks for your advice. > > After further testing in a different environment, we found the issue still > existed in your last patch. After troubleshooting, it is found that in the > "malloc_get_numa_socket()" API, if the return value of "rte_socket_id()" > is "SOCKET_ID_ANY" (- 1), the API will return > "rte_lcore_to_socket_id (rte_get_main_lcore())"; > Otherwise, "malloc_get_numa_socket()" API will directly return > "the return value of rte_socket_id()",in this case, the issue cannot be solved. > > And the return value of "rte_socket_id()" is modified by the solution you > suggested in your last email (RTE_PER_LCORE (_socket_id)=SOCKET_ ID_ ANY;). > Therefore, I think merging your two suggestions together could completely solve this issue. > > Can you please update your accordingly? Please try the last revision and report back. https://patchwork.dpdk.org/project/dpdk/list/?series=26362 Thanks. -- David Marchand