From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E46C045A3A for ; Thu, 26 Sep 2024 18:56:09 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 619224067E; Thu, 26 Sep 2024 18:56:09 +0200 (CEST) Received: from sonic315-20.consmr.mail.ne1.yahoo.com (sonic315-20.consmr.mail.ne1.yahoo.com [66.163.190.146]) by mails.dpdk.org (Postfix) with ESMTP id 368B540672 for ; Thu, 26 Sep 2024 18:56:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727369766; bh=uyQNHlQJkrTTPKUCfu59gsIC5v/5pLQLIlp1km7CD5M=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To; b=Xdhx925K7F47xNGDWp+4Yfadtb21zaIWlM+jeLCA1ZDi+h+lSQCk127X2YPukCAfyXSICRIwgAKiq/aIrGF4Yx0l+oQTMcmsMJ+xGeARyBiTEJ2z2HOLtS3nYCjkWiA9OF6ERwmvEmebBcWNyPAfkv8Z218qIZdwE/ziNMxAdi7h19c8LHOeTLhVwS38h0bkNy6WkFoa8ufa3WlONLhHqePpAHUrxo5iVe9DMyP5yiRnGMW7n7/kyzjZpuVXIpgoiVhP5EONrgN8wEKcNdFLXzL5RQAAoLH7jOcIatgz8+tq2mqI6spdGBRF2fFecqbHsf11rcvdxSshH2UjELMVGw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727369766; bh=+hmgYONZW1mr2/CqtQa0OnI3BmDuV/zwyQdzAH1zmHa=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=hKUd/HPDo1YetM4V9do04dYpGajzzKtnixuvQIfX+7kp7K+3s/dPX/DjO/1wcnOFqF0EACSg5AkPy3mDTOaIuYOHqrqL2KRXqThF0PnvZWdbGBNI5Q4Hbj76XawU++W9InaVuX+b6tGsr98L9YICjHdiVHrG1f+/Clr4v/c4xeb8Z3zqgPkpJrT6llsJt+DV/D39X0qXOenTKptkE9U4TVvrs/yWJxt1rrMWXvWK9wEdj5g+TQxFCsKryJWUnqKCz7zM+eNsSzxHvNZpE75VH6siaMeLeEJsDXZVRK5nueFohEMPaa79UpcwR4K/RxGiA7/V0Gmku/MKATQ6qEB+2g== X-YMail-OSG: z.K1y1EVM1nEC3m8OkIMK5yxY9kzyxy8.h2f3Pa8fqkRL_MwF9wCyEA1jScp59S pw0Nl_6ipg32cFhxJYZm3HPrRnADYZaJaJ9N_II1QF1G9aSBDY_IIPWAgBpRyP9HE5p96Ua0tjCt dZx5jz4NgzF_lTw0AWg5QAMsSEoWzt7dl_VtKDyHTG9AOlB8C55Q7PpPpysl6u3MVaWDqdvpcY9i eQ2OvtCDDEnIlK8Zjdnw0V0W8qGSWWhHCggvU_gNRvNizUYVVojJ4astoEgvYH09kwiIrv5Cdy24 XNdJk2nxD1MGjiFJg1xUA_x6JmebZ12mSJUi3m_d4reDPhJIqPleByNg0kU5SMzm2YLsz71s6.zT YfgY5QyN9xBoZyj.CWS09Gn1KlMUo8zXzwTK3sLIPuc.s88_rBcCjH5UAbw.DR0H2O7.KnBKxYA7 .OVmxEfCg3KoVn_ta7FanTWqdz5rp2.kExP0Y2lSiiTv5eS3HUe7V8XZjD0CveI4LbaSWYiqK1NU 0NfQu2iWUufZF5tbFNd9dtqwUVHsjrA5Qp76XSOU7ha836xZ.ZblcaM514UDILoQMzbFhMYeCaHU ZfwVh1Twsa7Nv_o1kutZR_3TxmdRWUcFa6nyrzCKTv1TP3hDDEhJtHAByr2qKXlIMfKQx2CSOo7K 0vyUnmC08k4JdOqsl03hx6eMZSKL.ei642hc_txeghPLkfSMlnAPsK.Dq9dqxVkKzs.XNnBIPTcM QbX5paeNkzwq3REmqP6odmHqwO63PYv1KMLAlPbzwofmTPfMIHgt3OxCya1JFTAhMn9TsjCqUTQm RswvBzCBORx_0CbsLmdocYBQJirG9g9Ev6KIKi7AL7.XDJMjORHa3ZZXm2pNT0vBbyayGuAN402E BPUJXPOQ3hp.c1FBqLPVBIhszdLyBWf1deGpErA3C5_F5YURk3H2ee3E3MoEN7un9K0OIo6d4ONC oBH6M5ABjZT63BMi8Y.q1RNatsHGknN9LjjHRKUB6iJNTbwM1.5FY6yh4NAqkPHXew1FF0gv4irn b6X3XJ5dC3M5M8TJsq7kupMZGCvgkb_aNzrlV16IVIx0sVo3pI2X5pWH9Hk.7Q7WQIyBAKsu3a4. wNNMuAjRCQrI8FmP04fA1tmJLkeJP2gSapeyrKNZnX87lkCIFglG2nb5fv3ag0bv5Wlj4.ApCNq0 W.u5RgfzeaXwX_ED_EmJomGTJJA7Zxh77MTScKjiTpFA.aYIXTh7zQA8kbTNkci.sIh6I_zO9xCR bJkez3.YvBFIZDabEirVSvGvKALpSft_0I5NaGYibLUEWvUAy2ZyKocAjsEZUTWacrvfVwWi.mps ivPHXUqWMEPnariXC3LGjKeAZTt0yBeIMYelVxa9RUqXFEJ1MoFL0FvQ4wnCQHnJkBES0HCuJK_5 D1BW5tU.g..DuH9DbjYPkMU158r12FWUG9twv24PisLplNHeBf.jBvUyP2U6GPs7..j.IRjvzDOH mzXYZ9CBLSrIPmWfv_DknaKNv1X8qi3AKpQ8SOcB35IGz6A9GT81FmXlTl7HV7V0w9urRxqck_kp TI3.23iybINrjFLY30c4yjy.8FJeIWNdAUja67eoVCCbywYsxMI1ZmX9eepoiXTaQowXDBPiU.GB gjSQYFjgHY4nUHkSklYPJh6WeApMtime5X_eWmx2pEzYcdkvbtuaFuGAzVroFxVac8oWqkyb2ENi QMSNcWl2HpKkl0YBfSYF4H22uxSyBZU_v4i3B3OgtkBG8F2x2yeNsx3Kt3dCG4l6G.1hxS4YzpXp di0HdvwSUwSfyuCcyTHLIM8oYOX65RSHaHKlmmKMFPtd41pEa65St4NpBxZADvtft.bZi2KE92O. AxXKxqg5NrhHUk3uICubOPihV6_zS143jOkbFmy0BU.4ISqrI2ppK36CtpLugRZs1iVmly1F.33X kkpvuLMzNBnMlWQg2mUKczd5cnvdbi8bOmS8USUAMWq8HMmItqxGuhKuHUItw3l_7o70LHto9CAs 0nu0DDwO4wGw_mvOMRRrNugQIUoEhxmeEc1SbDUiMHUGrFKzTuYyECIt9IgaCLFb8CAGeFOf8J5h b312iaPVZL5R519bpThR8PE8NPxWswVjHsi75AAvtAVDD6p3t_x5InkC94jinE22GTNi1ztDFc70 .8SGSASr6nuO0eyJ6iJ9P0CSza_MoRALfpyX1EyWW0IMGAcMKe_J7TxluFRJaIZw34isXulalp7Q CmY_n73Jdf5UYf_4HWbuU8RPY4iPuyf_ZzChSBcxTqX8- X-Sonic-MF: X-Sonic-ID: c718dc0d-67eb-4024-b29f-b39e687f37bb Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.ne1.yahoo.com with HTTP; Thu, 26 Sep 2024 16:56:06 +0000 Date: Thu, 26 Sep 2024 16:56:04 +0000 (UTC) From: amit sehas To: Stephen Hemminger Cc: Nishant Verma , "users@dpdk.org" Message-ID: <47151973.13053687.1727369764729@mail.yahoo.com> In-Reply-To: <810098753.12902128.1727353971978@mail.yahoo.com> References: <1987164393.11670398.1727125003663.ref@mail.yahoo.com> <1987164393.11670398.1727125003663@mail.yahoo.com> <1299564509.11731667.1727133474900@mail.yahoo.com> <2025533199.11789856.1727143607670@mail.yahoo.com> <2042269904.11975457.1727188849962@mail.yahoo.com> <20240924093813.29a01783@fedora> <26643152.12164440.1727210825368@mail.yahoo.com> <810098753.12902128.1727353971978@mail.yahoo.com> Subject: Re: core performance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.22645 YMailNorrin X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Belos is the lscpu that was requested, it appears to suggest an 8 vCPU thre= ad setup ... if am reading it correctly: $ lscpu Architecture:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0x86_64 =C2=A0 CPU op-mode(s):=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A032-bit, 64-bit =C2=A0 Address sizes:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 46 bits physical, 4= 8 bits virtual =C2=A0 Byte Order:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Little En= dian CPU(s):=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A08 =C2=A0 On-line CPU(s) list:=C2=A0 =C2=A0 0-7 Vendor ID:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 GenuineIn= tel =C2=A0 BIOS Vendor ID:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Intel(R) Corporatio= n =C2=A0 Model name:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Intel(R) = Xeon(R) Platinum 8259CL CPU @ 2.50GHz =C2=A0 =C2=A0 BIOS Model name:=C2=A0 =C2=A0 =C2=A0 Intel(R) Xeon(R) Platinu= m 8259CL CPU @ 2.50GHz =C2=A0 =C2=A0 CPU family:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A06 =C2=A0 =C2=A0 Model:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 85 =C2=A0 =C2=A0 Thread(s) per core:=C2=A0 =C2=A02 =C2=A0 =C2=A0 Core(s) per socket:=C2=A0 =C2=A04 =C2=A0 =C2=A0 Socket(s):=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1 =C2=A0 =C2=A0 Stepping:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A07 =C2=A0 =C2=A0 BogoMIPS:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A04999= .99 =C2=A0 =C2=A0 Flags:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 cl =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 flush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp l= m constant_tsc re =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 p_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_kn= own_freq pni pclm =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 ulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popc= nt tsc_deadline_t =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 imer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dn= owprefetch invpci =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 d_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms= invpcid mpx avx5 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 12f avx512dq rdseed adx smap clflushopt clwb avx512cd avx= 512bw avx512vl xs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 aveopt xsavec xgetbv1 xsaves ida arat pku ospke Virtualization features: =C2=A0 Hypervisor vendor:=C2=A0 =C2=A0 =C2=A0 KVM =C2=A0 Virtualization type:=C2=A0 =C2=A0 full Caches (sum of all): =C2=A0 L1d:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 128 KiB (4 instances) =C2=A0 L1i:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 128 KiB (4 instances) =C2=A0 L2:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A04 MiB (4 instances) =C2=A0 L3:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A035.8 MiB (1 instance) NUMA: =C2=A0 NUMA node(s):=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01 =C2=A0 NUMA node0 CPU(s):=C2=A0 =C2=A0 =C2=A0 0-7 Vulnerabilities: =C2=A0 Gather data sampling:=C2=A0 =C2=A0Unknown: Dependent on hypervisor s= tatus =C2=A0 Itlb multihit:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 KVM: Mitigation: VM= X unsupported =C2=A0 L1tf:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0Mitigation; PTE Inversion =C2=A0 Mds:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host stat= e unkn =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 own =C2=A0 Meltdown:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Miti= gation; PTI =C2=A0 Mmio stale data:=C2=A0 =C2=A0 =C2=A0 =C2=A0 Vulnerable: Clear CPU bu= ffers attempted, no microcode; SMT Host state unkn =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 own =C2=A0 Reg file data sampling: Not affected =C2=A0 Retbleed:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Vuln= erable =C2=A0 Spec rstack overflow:=C2=A0 =C2=A0Not affected =C2=A0 Spec store bypass:=C2=A0 =C2=A0 =C2=A0 Vulnerable =C2=A0 Spectre v1:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Mitigatio= n; usercopy/swapgs barriers and __user pointer sanitization =C2=A0 Spectre v2:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Mitigatio= n; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affec =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 ted; BHI Retpoline =C2=A0 Srbds:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Not affected =C2=A0 Tsx async abort:=C2=A0 =C2=A0 =C2=A0 =C2=A0 Not affected On Thursday, September 26, 2024 at 05:32:52 AM PDT, amit sehas wrote:=20 Simply reordering the launch of different threads brings back a lot of the = lost performance, this is a clear evidence that some CPU threads are more p= redisposed to context switches than the others. This is a thread scheduling issue at the CPU level as we have expected. In = a previous exchange someone has suggested that utilizing=C2=A0rte_thread_se= t_priority to=C2=A0RTE_THREAD_PRIORITY_REALTIME_CRITICAL is not a good idea= =C2=A0 we should be able to prioritize some threads over the other threads ... sin= ce we are utilizing rte_eal_remote_launch, one would think that such a func= tonality should be a part of the library ... any ideas folks? regards On Tuesday, September 24, 2024 at 01:47:05 PM PDT, amit sehas wrote:=20 Thanks for the suggestions, so this is a database server which is doing lot= s of stuff, not every thread is heavily involved in dpdk packet processing.= As a result the guidelines for attaining the most dpdk performance are app= licable to only a few threads. In this particular issue we are specificially looking at CPU scheduling of = threads that are primarily heavily processing database queries. These threa= ds, from our measurements, are not being uniformly scheduled on the CPU ... This is our primary concern, since we utilized rte_eal_remote_launch to spa= wn the threads, we are wondering if there are any options in this API that = will allow us to more uniformly allocate the CPU to threads that are critic= al... regards On Tuesday, September 24, 2024 at 09:38:16 AM PDT, Stephen Hemminger wrote:=20 On Tue, 24 Sep 2024 14:40:49 +0000 (UTC) amit sehas wrote: > Thanks for your response, and thanks for your input on the set_priority,= =C2=A0 >=20 > The best guess we have at this point is that this is not a dpdk performan= ce issue. This is an issue with some threads running into more context swit= ches than the others and hence not getting the same slice of the CPU. We ar= e certain that this is not a dpdk performance issue, the code > is uniformly slow in one thread versus the other and the threads are doin= g a very large amount of work including accessing databases. The threads in= question are not really doing packet processing as much as other work. >=20 > So this is certainly not a dpdk performance issue. This is an issue of ke= rnel threads not being scheduled properly or in the worse case the cores ru= nning on different frequency (which is quite unlikely no the AWS Xeons we a= re running this on). >=20 > If you are asking for the dpdk config files to check for dpdk related per= formance issue then we are quite certain the issue is not with dpdk perform= ance ... > On Mon, Sep 23, 2024 at 10:06=E2=80=AFPM amit sehas wro= te: > > Thanks for your response, i am not sure i understand your question ... = we have our product that utilizes dpdk ... the commands are just our server= commands and parameters ... and the lscpu is the hyperthreaded 8 thread Xe= on instance in AWS ... The rules of getting performance in DPDK: =C2=A0 - use DPDK threads (pinned) for datapath =C2=A0 - use isolated CPU's for those DPDK threads =C2=A0 - do not do any system calls =C2=A0 - avoid floating point You can use tracing tools like strace or BPF to see what the thread is doin= g.