From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AB94E45A03 for ; Tue, 24 Sep 2024 16:40:53 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 27E9640274; Tue, 24 Sep 2024 16:40:53 +0200 (CEST) Received: from sonic304-21.consmr.mail.ne1.yahoo.com (sonic304-21.consmr.mail.ne1.yahoo.com [66.163.191.147]) by mails.dpdk.org (Postfix) with ESMTP id 6D725400D5 for ; Tue, 24 Sep 2024 16:40:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727188850; bh=SblAQleRJhWkFsabgpwPs6lKh7SO1JcEjNV70af5NcQ=; h=Date:From:To:In-Reply-To:References:Subject:From:Subject:Reply-To; b=AKREKUISGTYO5Si1mbPOmvqkvElYw5xbyfLf0bXYUlKE3z+X5by+gUbE+DSUk6jZmZBe0wc+R9GR3WrIaAaqXHhjEdXHYG1JDZsb/N998+JI3T1+XSpmehEX0lRSb5o7f5Zm0xBbo28EOuL62tYuzfhhEk4ytoI11mWdbOOobypAmlJIsTirrWO66R7qVPhNu+X3B/oaYO3YTH/1xoCc/vgsdxuWggvs5CYw1Zlm6No0KhoDAMd620yoBvDp5JsPLA5CsaAnCBZboVgVkrJavGUYdse9xBE7VZZXbcqB+5vjq+ZvrevD+rllErVxoC88J3qzoqF6Cscp/dnrKfL1IA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727188850; bh=qRCGFdDz1Tmc+UzokqgQwpDODtpixA2r1sU5ThiDEOp=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=rHH2wVPzO1MF4tpSqofeopGInhwVbstziNfH9vDmy3EHP6eQkBWO28t3ypSGJPmr5idonfHSensc+Rcyp3nnJLgZ1OW5F3673Ad/8GnFTnLI37qDTcFleS1t8UPBfdeGC/IpytMCcY8Q4SxTEQdRNifrYK/WvjJKNWQgJgjxx98EJWmpw5olIB7ey5uTfW+kEJxtzIiJbGjUNZsQbWLWgAS0aOTjwWPxxSELBs7yf2SScRK7uvEkHWf/xb612rQExq1VNpdWWLdl3GoYj8FopWk4sAbTZTFK9e5Ce3qPqIqNiTSqaij3h4CfhIOuVK2Gi2Dwjrz6q+RkiWraMY+Uqw== X-YMail-OSG: 1GaXUWgVM1n5s0UjnZI_5g1cYI6NBUSdxGBIrsrn5Yb7PAXJ_RxFSGKJ6sCizww aZIkgn8lv4Ajsir6c3BnIXY.MI_uHFuGiasX7.UcSHqOaOOS9EdffpMzTv5mCLdRIYNxGajMVJB. gIOCEd1p1A9H7Q1bnvnS58KXu61L.GZFm4ljAr9IxOFh_fY20sH7JzJPvH1FwyJE6JkZ0XAsO3JB 2KFdJfIQrL2EkipbYhYCiOtFatiOnPvLwMx_5G8M8LqhRAr9wOURP3UVR2wIGdR9Hlt_huzuOsm2 3qPU8z3hU0Sn0KIMpkJZYb0ds5hIxzjSgdfhSKTNcSaI2J5nA66OmxA0FDKF8N8KjH43eW._j42Z a5a8DgmL0q3zP70E8Z6kexDXYi75Ma40yrLd9hg.Y_hHdbjneNJwf2IqAmTxuahhHmdOXmGBvc0Q oFHNJDb2W4rCQ5Qsib1UWDtWeDFrsW6Iisi2m9yijtHKgc53RtTehjJfFeHQQ8gRnZitIr9VjmrC 2mwMR9Lg9YvKoDXG.eLFf2WSolUFmFBH3nfGmgSTsR3oc7GkQ3sDEs43xVLVxHp6KMEkMmj8I8SU GgH1dW.qXnKr53bGZKB0KUyX_53F2ebHzmz2z_ckneKfVtVIez_qeRzTHAYzI8a8CDttJYUffpEC 3oSiLON7GNCJlwaqEwqCRoA3n3q.zJ540tCHqn63ztgkYmmWZXhzBMqcGLtpdL.rk.Rh8eL6J3gG A8lYCNRpK69u1fnO0uG5BslOuyQvmKZMGG_AWkFjfd_tetVGvJwA.ROUI8ACDUShPF3efqy_fPgx Jw5J0EqM9SxY.aeLFsDRjBzQ3efFbmCtp_SsR1jEb2_FzqlF6ArO2Md05DmixjLvxEHB7xlG_Ul2 BZpc192A3uncDhH.ulS7lHescWtShs4CDj_rkO8nGnk1fmZBNairf2gixI3UGwG6dKFAotgaotr3 ODXo7uoYFdg0BJrdmZQHS_IYlul2_sCOWvRIkJ8z.ofMP0lhWvo4DUtMmi_jobGpa9BiBcvjcStE gbrB0mISJwzzB55uq0soN.JwjkG9x7WiDovykfUDmz7UH.jBDvDiqhZGnNL9RkbO7vyvF.iiKE_Q .deirGA1ZlpbojvXSk4_QwrDRPtb0BTxM3sbvrafriEQiqjYRLuRvBuOlZZ8tjHJrDPK3E8TRGg8 8w9N5PEoZbzHUiAiwVQGm9gc3vLgTBTZbH5H5ogsG835PMA0BwsFYQrS6Sx5dgM8bbrpnFy8KHvr eX4g2avI1nuyOmfAQZBRHBcP9hmD7aMEe0jLWg.J_ndRXn28ZV1Jl83Gk3FnzMPVsp6HUbY9SfjN fbrPG1169DR3.HaMzfv4fvW6aQVb7IEmRHg4.DbDwaI3j.Un2L4h9DdBDvqGxwXJNqzFh0jhHe.o AdQgB3WX8m2HkizNbd3XdhhkxEbeKo7b2N14yny.u6CxKh04OOUJ8SkDnCkuLboyK7Ugiq5uTSvM s.2Z8Lsq1U5CR.tSVzMDuqhtcFBSELBy649JIOn_.sGdY_k7wHq0ryEfK4nc_ZcYng7YvPviMA1V M2TEkhpnerPgBb2jmadNIa4pm0uYHJ6gcqRv1i3XbUKbHJIl_pQHGcxOp0zFyJLp2kZ79Ojf5WhN f94G7Car71.vGECxE9pJXNi44REWdFzn9frPb732WE_CjnmxvMrTz.wdkSgNBOxwqTRFtkKJpxda WnJ68tkHAB7uAHc9BworrV4Bc8_h4JQoC0jKO9kV38Re_JHUfTi5egah2k13mOGmoYfEzZl9255n 3fmjBpkEyfhW2nKIMz8VGXBZlMJ9.hP.4hXmTp2OrJVKn3VTiA6Erak_b_hO9wGDLBjyzanA20pe aRiOwmpmVAHHO1RpvCgcR1j6ZdAH9BG5IcNdKFbrElh20rYtf3r59LVcJfZRzVMoGFmKqSiOkOg4 QUnB6XyJ2BoEctdD4GFPNYEq20anlT5_fwb2pJQpO65BluboYBtWlCC4Xk6hEB77xxNRsRuX._8l H6CC80BQtC9MFKmMsCCf6Tg8IoO0_gFcrwOXJZLHqdygDCOlY_AvvmzzM8OihmkzRCVSn4FLV3lg vTBzEdTCTRQog.aS4ppYMfTz.QFXatvfXQWPYNlksvP4uEhfullicYCkxp3jae0xFI.frt13l4Y_ 0Quzg29L8_7gxKJ751czbKI005sz4TgngdV3IrW8zJwP0Ua00endjOYdIrBMNYDpCHVNNAj9rXPs sDNrbpIk_VLfhbl5COsZa9MBdj_5RxVS1ywb_MNvPktnMzA-- X-Sonic-MF: X-Sonic-ID: 0be34637-05ca-4e57-a322-efcf30b8a337 Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.ne1.yahoo.com with HTTP; Tue, 24 Sep 2024 14:40:50 +0000 Date: Tue, 24 Sep 2024 14:40:49 +0000 (UTC) From: amit sehas To: Nishant Verma , "users@dpdk.org" Message-ID: <2042269904.11975457.1727188849962@mail.yahoo.com> In-Reply-To: References: <1987164393.11670398.1727125003663.ref@mail.yahoo.com> <1987164393.11670398.1727125003663@mail.yahoo.com> <1299564509.11731667.1727133474900@mail.yahoo.com> <2025533199.11789856.1727143607670@mail.yahoo.com> Subject: Re: core performance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.22645 YMailNorrin X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Thanks for your response, and thanks for your input on the set_priority,=C2= =A0 The best guess we have at this point is that this is not a dpdk performance= issue. This is an issue with some threads running into more context switch= es than the others and hence not getting the same slice of the CPU. We are = certain that this is not a dpdk performance issue, the code is uniformly slow in one thread versus the other and the threads are doing = a very large amount of work including accessing databases. The threads in q= uestion are not really doing packet processing as much as other work. So this is certainly not a dpdk performance issue. This is an issue of kern= el threads not being scheduled properly or in the worse case the cores runn= ing on different frequency (which is quite unlikely no the AWS Xeons we are= running this on). If you are asking for the dpdk config files to check for dpdk related perfo= rmance issue then we are quite certain the issue is not with dpdk performan= ce ... regards On Tuesday, September 24, 2024 at 06:23:39 AM PDT, Nishant Verma wrote:=20 I assume you are using any variant of linux. So execute command "lscpu" and= provide the=C2=A0output. Also share your command or config file that provides application to know wh= ich core to use and how many memory channels and ports. Thanks. Regards, Nishant Verma On Mon, Sep 23, 2024 at 10:06=E2=80=AFPM amit sehas wrote= : > Thanks for your response, i am not sure i understand your question ... we= have our product that utilizes dpdk ... the commands are just our server c= ommands and parameters ... and the lscpu is the hyperthreaded 8 thread Xeon= instance in AWS ... >=20 > regards >=20 >=20 >=20 >=20 >=20 >=20 > On Monday, September 23, 2024 at 06:14:16 PM PDT, Nishant Verma wrote:=20 >=20 >=20 >=20 >=20 >=20 > Can you share output of lscpu and command you are using to execute the ap= p? >=20 > . >=20 > Regards, > Nishant Verma >=20 >=20 > On Mon, Sep 23, 2024 at 7:17=E2=80=AFPM amit sehas wrot= e: >> Thanks for the responses, this is on AWS, which is utilizing Xeon with h= yperthreading. Not utilizing hyperthreading is not an option. >>=20 >> After trying a few things i am narrowing down on the following approach: >>=20 >> only for the critical threads we could utilize:=C2=A0 rte_thread_set_pri= ority to=C2=A0RTE_THREAD_PRIORITY_REALTIME_CRITICAL >>=20 >> however this API requires a rte_thread_t parameter, if we utilize rte_ea= l_remote_launch, we are not provided with this parameter. >> I am searching through the code to see if there is an API where i can ob= tain the rte_thread_t for the current thread that was launched with rte_eal= _remote_launch. >>=20 >> regards >>=20 >>=20 >>=20 >>=20 >>=20 >>=20 >> On Monday, September 23, 2024 at 03:18:11 PM PDT, Nishant Verma wrote:=20 >>=20 >>=20 >>=20 >>=20 >>=20 >> Also make sure all core you are using are physical core not the logical = core.=C2=A0 >> Secondly, check your core isolation options and apply them accordingly. >>=20 >>=20 >> . >>=20 >> Regards, >> Nishant Verma >>=20 >>=20 >> On Mon, Sep 23, 2024 at 6:04=E2=80=AFPM Wisam Jaddo = wrote: >>> Hello Amit, >>>=20 >>>> -----Original Message----- >>>> From: amit sehas >>>> Sent: Monday, September 23, 2024 11:57 PM >>>> To: users@dpdk.org >>>> Subject: core performance >>>>=20 >>>> We are seeing different dpdk threads (launched via=C2=A0rte_eal_remote= _launch()), >>>> demonstrate very different performance. >>>>=20 >>>>=20 >>>>=20 >>>> After placing counters all over the code, we realize that some threads= are >>>> uniformly slow, in other words there is no application level issue tha= t is >>>> throttling one thread over the other. We come to the conculsion that e= ither >>>> the Cores on which they are running are not at the same frequency whic= h >>>> seems doubtful or the threads are not getting a chance to execute on t= he cores >>>> uniformly. >>>>=20 >>>>=20 >>>>=20 >>>> It seems that isolcpus has been deprecated in recent versions of linux= . >>>>=20 >>>>=20 >>>>=20 >>>> What is the recommended approach to prevent the kernel from utilizing = some >>>> CPU threads, for anything other than the threads that are launched on = them. >>>=20 >>> If you are wishing to run each thread on separate core, try to use rte_= eal_mp_remote_launch() >>> instead of rte_eal_remote_launch(), make sure that your CPU is isolated= , and you are passing correct >>> Cores that were isolated to your app using -c, -l. >>>=20 >>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> Is there some API in dpdk which also helps us determine which CPU core= the >>>> thread is pinned to? >>>>=20 >>>> I did not find any code in dpdk which actually performed pinning of a = thread to >>>> a CPU core. >>>>=20 >>>>=20 >>>>=20 >>>> In our case it is more or less certain that the different threads are = simply not >>>> getting the same CPU core time, as a result some are demonstrating hig= her >>>> throughput than the others ... >>>>=20 >>>>=20 >>>>=20 >>>> how do we fix this? >>>=20 >>> BRs, >>> Wisam Jaddo >>>=20 >>=20 >>=20 >=20 >=20