From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 57F3C45A38 for ; Thu, 26 Sep 2024 14:32:59 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CFD964028B; Thu, 26 Sep 2024 14:32:58 +0200 (CEST) Received: from sonic308-56.consmr.mail.ne1.yahoo.com (sonic308-56.consmr.mail.ne1.yahoo.com [66.163.187.31]) by mails.dpdk.org (Postfix) with ESMTP id 37DE94025D for ; Thu, 26 Sep 2024 14:32:57 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727353976; bh=WxmpM/JPBcW2+M6vFjDvjXQ6W8dVbJqCD/FZj3suMuM=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To; b=B1bZmxWY+OdXrWxTDmNWHLVT99WaGtaFvWhXT+iXLRqeXKlrofsPO3x6mEHB2yLvmxmuv12OABv2D05OHJc4ikw+mo3tNOtDF6/dJwTqUiPZKCsBJNoAcsZ3gnhEs0CfFhsnP1mg7oM356p5+IhEna/9RIHGJ7LwsQhyFXGLaPS0aGSh4K4YqLHCSXUxd2LG3MNLb3PnK4RXQrwlHApU7+gIhBe7mcH2Lq6HdESAly1S6q9IIz2eu5uFsLltZh4K1Zdx5s4z8EaIP8Zqy2n/JaOsS8vCqtchA5o258t7y+J7HdDz04B9ctHWl7kqG4FLZX4LKK44EEyF/AwT55bv5w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727353976; bh=EhHPlyF4GlIlmvEYu50Gu5hBANtBmfVxEdlHjUrbuAB=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=MV35eLqd1ZPRf/2fODPotRPtPVgJpmIKsUnDPM/6WK8GiRxmXAaSwfpJIce41M7BhN6l79YRE4U6nneKgy7CitBPuXSOE//7jUlA7aOopeIu4ccLFKCAoB8jb0HFpo0fyn3DDMDvhgWQth8+JDxc7saMABLe4yueOQp0h+IYM8zZ6QrB1+/cc2g40FcUxKyz7o7Omj757rg6JNY5/+olrpx/5dcC25O5SnTMNYVBtb/+UT5X+JL7je8Fq12gROLWVezDerbZYaS1+G3WPkvRvRzuAfpMJJwCUsT+m/9AhbPcDIKvC45U/8O2scNbtyh6Vbj6OWnQIjszNJzB+K9v5g== X-YMail-OSG: hYN4M6QVM1mDFBmmfCNRw2Fzj08Jnx2jYNLG4pMw40OKqTshwtXBlAAra2lOnWI 4JzB1U0z2ojhNniE9R0AZ7P5_62_uyjxnAFnSOY_pTLCsS3mQ53h9_rb47nAx50mJeOJ3_5B9DrS ZhVexnpt1.TTUK3KqsgGzjfLTmY5uJycKXHR1JRsgCqe3XyFhfZ842StK2WvicyWKEVetNGTBq56 wutYcsTA3GBEbhnnuNd8IiJXvsT9TVMf8JPrILFLBhnGNDCSc.y_HnPDDKfdjN77GhPXiCmFlVBY EqNY6.AGN17fX3sSIQo8EUFtd0yM3kcfg4PmVcyHMe44GHecn_MceGJzhNtGaPPDaYguJX3tRk_q BQUPSC_HL.ByQMBxcwUph405R._qSz6R6NsNt4lq9teFbj77j_BMCpY0HSS8jiuGn0SU0OGAEmU_ 1ormktKK2Ox_ruWdLpGim5gN5E6zFoc9NVq6SGfQWlAoUwO5IOvNZchHYlQYE2L1qDrYZtp0jGOq fRtiIuhPeVPYwhSR3.Roe3xUjRXgxVOycCJ4dwv.l72Q.vFPsaOVmpM5JZcX0NAq6HQ5WHaGe0sd 22ACEDaJps7mqgMxpJ9aviQhLjiEu_oStQkerRXWHgku6xE8dzw8LsWw9WYcjyK8XR2f5bRXlawS AHmUlfbdXyV72BmErxgoR8FQBkQKkJ0nu3d1VLHtvs1s3.4juqHj7mn.BFXF_zEoFo5qUZ_40wpL GtSFn3Y_TpJAtG6kqG3Zn3DGaFsadi9yXkkOdfWbjNh.j6sq8yhlT4c0bmwulxd4N9580ILEOBHS rNfX7onqoLJ.ucYhmUuncueC6eVh.._HhIDA_H7_wK8B2wxAi_.OeiireSLRn73cMgyV7xEAsf1G nGMJFTu1C22Aw1CQelC46aYqcOE1ptfOINQBbh..eB_IL_EVYaE1vxlV6mCQ3i_BwAVi1w_41QNE 1s4Qj3tqlS5TisYAETzDOj00IdT9YCk2o_ty7M20kNoU.nD2rgz92l4K5cyPgjdR.zVNrGedpkOd QBkQKZCWZM0iwCjN7JAoahtnWyXLJFd6uabXLZvRhH46fuQYYdD00HxnFk0wiCJj7V51Oj02kGNZ rXRWfv6YAWneeWUdtR7RVtCKWWrbKFHJTcBqkB_FGTr4RX.kFBxqc9vb9voa9_ZcRdjh1yKVBH7e a72FFYAdKCHcHCAkbsfUD_cJKZNjomweIlL5X3h5NFHGdi9tJ.MU9o_.PelGkkBfqPgcXubJ.DNq tG0mXZdUZzRF5ifEDpgWJx5oPaFVNG6FIZxv0eQaBqHz9bDCLfpS7cICTHYw4XLBk89YV6WJxTW9 VDMK6mHMTam7VLJ2v2FR_QFsq3guxrGDlDZa9J8dEwUJms9.P7VjjGndh9Yom7thIrasOQ4bdvnV vR5QAsr1jeDdh6c.OxNRSfd4TwtkIW3Ih_Ri046prkuJQzbDSzzrO.XwK4SHG5hH9.krYeZeBOgv ybBaOlMI3ookOJ2L02JpILAYQbZAdGRPDPCKqVs0Rnc.J1i_3UpGQmiRQ3kCIAfpEOi_DHlcWM7_ o0S0z5q99fwCgcCWU48NrfrjEZN0MUYac0p9Hoyj2oWCy2lPzRJCha3VADE7xQEZrTPhVTg9xfZt dki8f4.07ZTAeAaxRpEN.0sw7gS5BzO2sSra4fdDlntQh7fWdqDuoPVqjvNN48JHfrOt0sA4rgp4 xBJpl_BOmz_5zryehUDjNB_mtfj8YPYv3aLg0hKFqXRgJxC3aBEvhCEYqgX_RwhJhrz7XjppIYYE 1bcqg_Wcq8CIo.PkfII9cTIcBh99k2Pf3o5KXoTdDUl6dspSdNl4qrotpZbxJUI0zwJ3wSCs61nY MjpuWJapu2RetnLHuYkSbjksrxgdZaKzUAlGtbBImG5ne3yquIg7Lfdfpkcw0PxaG4qdD5JoW.tq gkaNN5ihCGAhlacfHLnd5TQs.0yK5a7d1FYy51v7N3VHvjP0lLgU_5Mspf4SkZRijFXVSRiXCYAN STCcBuSKzzFHuFEUTe.oNK2pOoVQAmZLTlIpOXe9NK5uKWBxEhZfUpmRsVhUgQlAs7.6ftVoHfPP .tiC.GgnEiZwtdrol_D2XqPLIsbZWh7Pk3CNGh466tOyEDyzCvolBPXPmI0OIEvP98yceI1S.89l FLTXd7ore_MZbA1sno53rEkc9NWdpr90wEeBOo_q4EO0nLLHFbLg7xan_8lvq73zrlZl7D6en6tS sXJQOSlA_YYnpZniNkbg3IDPqBTx3Q5_m4eLGf2_yQ4ePRqWQwhe4oGZ7hsksFdXrLer2wEZD X-Sonic-MF: X-Sonic-ID: 347fb894-6c50-4114-84a4-2f86121ffa06 Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.ne1.yahoo.com with HTTP; Thu, 26 Sep 2024 12:32:56 +0000 Date: Thu, 26 Sep 2024 12:32:51 +0000 (UTC) From: amit sehas To: Stephen Hemminger Cc: Nishant Verma , "users@dpdk.org" Message-ID: <810098753.12902128.1727353971978@mail.yahoo.com> In-Reply-To: <26643152.12164440.1727210825368@mail.yahoo.com> References: <1987164393.11670398.1727125003663.ref@mail.yahoo.com> <1987164393.11670398.1727125003663@mail.yahoo.com> <1299564509.11731667.1727133474900@mail.yahoo.com> <2025533199.11789856.1727143607670@mail.yahoo.com> <2042269904.11975457.1727188849962@mail.yahoo.com> <20240924093813.29a01783@fedora> <26643152.12164440.1727210825368@mail.yahoo.com> Subject: Re: core performance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.22645 YMailNorrin X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Simply reordering the launch of different threads brings back a lot of the = lost performance, this is a clear evidence that some CPU threads are more p= redisposed to context switches than the others. This is a thread scheduling issue at the CPU level as we have expected. In = a previous exchange someone has suggested that utilizing=C2=A0rte_thread_se= t_priority to=C2=A0RTE_THREAD_PRIORITY_REALTIME_CRITICAL is not a good idea= =C2=A0 we should be able to prioritize some threads over the other threads ... sin= ce we are utilizing rte_eal_remote_launch, one would think that such a func= tonality should be a part of the library ... any ideas folks? regards On Tuesday, September 24, 2024 at 01:47:05 PM PDT, amit sehas wrote:=20 Thanks for the suggestions, so this is a database server which is doing lot= s of stuff, not every thread is heavily involved in dpdk packet processing.= As a result the guidelines for attaining the most dpdk performance are app= licable to only a few threads. In this particular issue we are specificially looking at CPU scheduling of = threads that are primarily heavily processing database queries. These threa= ds, from our measurements, are not being uniformly scheduled on the CPU ... This is our primary concern, since we utilized rte_eal_remote_launch to spa= wn the threads, we are wondering if there are any options in this API that = will allow us to more uniformly allocate the CPU to threads that are critic= al... regards On Tuesday, September 24, 2024 at 09:38:16 AM PDT, Stephen Hemminger wrote:=20 On Tue, 24 Sep 2024 14:40:49 +0000 (UTC) amit sehas wrote: > Thanks for your response, and thanks for your input on the set_priority,= =C2=A0 >=20 > The best guess we have at this point is that this is not a dpdk performan= ce issue. This is an issue with some threads running into more context swit= ches than the others and hence not getting the same slice of the CPU. We ar= e certain that this is not a dpdk performance issue, the code > is uniformly slow in one thread versus the other and the threads are doin= g a very large amount of work including accessing databases. The threads in= question are not really doing packet processing as much as other work. >=20 > So this is certainly not a dpdk performance issue. This is an issue of ke= rnel threads not being scheduled properly or in the worse case the cores ru= nning on different frequency (which is quite unlikely no the AWS Xeons we a= re running this on). >=20 > If you are asking for the dpdk config files to check for dpdk related per= formance issue then we are quite certain the issue is not with dpdk perform= ance ... > On Mon, Sep 23, 2024 at 10:06=E2=80=AFPM amit sehas wro= te: > > Thanks for your response, i am not sure i understand your question ... = we have our product that utilizes dpdk ... the commands are just our server= commands and parameters ... and the lscpu is the hyperthreaded 8 thread Xe= on instance in AWS ... The rules of getting performance in DPDK: =C2=A0 - use DPDK threads (pinned) for datapath =C2=A0 - use isolated CPU's for those DPDK threads =C2=A0 - do not do any system calls =C2=A0 - avoid floating point You can use tracing tools like strace or BPF to see what the thread is doin= g.