From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DB5E645A1E for ; Tue, 24 Sep 2024 22:47:12 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CB3AB40274; Tue, 24 Sep 2024 22:47:12 +0200 (CEST) Received: from sonic316-21.consmr.mail.ne1.yahoo.com (sonic316-21.consmr.mail.ne1.yahoo.com [66.163.187.147]) by mails.dpdk.org (Postfix) with ESMTP id 3456D40270 for ; Tue, 24 Sep 2024 22:47:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727210830; bh=exCjaWp1iHLqQCcyzomEF/yfodVNgAMihZJqGlqCP90=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To; b=Ov/qLVmLu8MtUp2uOpnhsgAVnIFtJKwp6hayUOEch7y7AxHOtLrPQQO5XRlVEpcCalD/QNp3RkIp1fpSttaBEI+g9FOkEcBDOoLYIQrAxa7r3Um3zb5o7bo0cTREdMDN3NX76CPIV9mbgX3QtxY/fVE2J04jPNlCo1PeO4T58gq6QYJeAPzQyGpCFzqWOq6ndgrv/27DMau3qZzaRNYKcRZprdOje4TSXsONbbt6tyHU+0f3kVXLu2Ei4WtUBGKB2isPcNmEteA7/Lzhx62VSwg5QrXI4dAWp9krxBp2k15/pkfYj4VHsgfiT4+MJmQ58dcYdXzVLvHmrZAfytnGCg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727210830; bh=jYKIbIGKoJvodTnijqM5wgYuMelfZSZ3YUCPiRzmQm2=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=raFoZkNNMVPPkRHeylk7AK3xX6g1aAHZ1z3774b8YoRTVopYwTCzIwVwpizczZh5N7RgaYrAMa0VATNXGpW0nhti3ghg09SGrB4G14s0YXnQQ/eXjHjLFwuTQ6ooh8rgCvSxOnFB4XxxHnCeghwBOZ+vZGWWlKEEx6aQcIIvBeR3vCL5pXxGpW7TEKyKb3BiDMPJVL4zl22qH5E91fyuTbUwkTUIDroQfZbdG4W0635K9xrgcngb9CEw3lm6UdHRU0njqt0DyFvuL6in57VfaYq7sBf2yfFx5T/JHRcEr0ZFOFa45vFzMtA5pjFSUulzLO+elLi62hyjgIBt8nOnow== X-YMail-OSG: gZbbAMsVM1mlsRucZpB4Pb7o31DTUtc6LjN1ePppCJkEXNmK_cUvqK2yTdCWKPQ InJYxM64gGaF_bY1pwhUmZXriWXiFAEvL54cZOFqxD1oTAVFENTC9Yfdx3ImJ71jn3qglUMV5wIG NI8lw1ql2jfJ.k72NoZApMua4l86UemGF7kMLaMN6jP7fr.u5D31L3Neu2InW53sRUrnI3lh9E4c FRNGnWpR7xJlKJ9m6ptPoo025iJkddNbbV0fmoZ_.xCNPBQ8l7HezcUONrzsFj7yka2q6JeIbeHe 1rxPmEnHhoax592JWYrL5YqerhJSIO.t0sS1oy99dBXkRTPY8vGrtWlErv49DIW3i7MsKtI7kQDx HMuvM0sVV0j6G.CQ5ywM11AnNlYNvX8GRHbgq8y_CD82UgwPJAIUS0MegV9dYJSah6E97vTLnxpz j1GoO5BveKt.r_uLnsOdPWjUifNmVomOs7_5VtCD1cU09H45YbH4ogwi62GB8mQKd7IFjgUyc0sP JFr4zkZ3i3L6S_1B0pWfjdFSraFjK.FEZuJ0hhvam7aR8BCsR14jUR5p9EKjrpaj6Nj_E4jY7MM2 67PwLE_M8T38HYF4GL0kdPy3Z1Y6_i35ZiKVyG_uayDfNi1Gp_Q33kQJVgg50pwJOIctxOkGPTay CA6vDvS1QSKMHriNTMQi80V3ooC9OBk7bTgOorC3SiuDSlvIX6VmfZcPekLmhygl97iiU36H5a_R QBeJdUTqYjw9ynY8622bALfaXKpkYuI5GH8qBOTbCvMNwMsLVsfwjti42AcZEIFjUdxdkcerEP1T tPyE7myqGlxIQznbNIOaeJh4uyfR5hCM8rdXOlcUk1qQRmMnQqnIj8dSkii9G3u_zkS.7vPkrW7Z C7hMyLoaAQGPf7UewMS3fXlR6fSl8z8NBD2imEZHByf9no4o4z6c5wucogbZk2f9Bm_W6NaWIfsB lmevH_HDwQlUJHHONcsBD7lZCNjejgU_T0j0KjWEV8cst0gWam0mChSgkXYKhzLvBaZQcDbAENNp AQaYboIcNcsylURFmTvVsOdpq0QY8M1.AuaoO9yzUisy9oDfZr61uXNOghL0hQ4lyXJNSAbiX5Ix lzMqSIWPRKfMwH7mBX6uGnt0f_0MwYlAeEQhm34ZAJ8HuYqaz7AjoJTMg4O6dUa0rVgmXHWvCXj9 Ui8kPn1cEMLvesRxM7rTGbh3PRt2uN1YPd7Ru6JM0_BE4NNPPFs45LhX2sXStlNPzz9wbahElbkE oUDnDBietjDAkYm98f.TkQnarTmul2cPTjhoy6abK5LPRRDcbC5fiVHmsPk_8cxE2crcFx3mujxX Ms9aydxpoROzxWR6BpCkgCc0vptVqOG8pwQ6L4CW7CqdDGsvTgMaaYLjVyQ1TVur9wYAhftN0MPu YjYzSj8N_qWoQ.7KUTW2FoIH0TBUY1uWlYCuGVXoMoLVP6K2.APM7Rf6b1NDCfOZrDOkAFkiLiuH jlYBn3Q0cF5hUCK._Tpp4x4S22wgZ_ObFe_Wbnb8oZVDY6cMPIJKkt1TX14fWV1FunvHA7O90zmM DtD3JrBEN3EMVOUg_uDRtpLdUFBNExyUHQKJ_Et0akXa.VSZhHgzXz6tOF8Wmnjt83UmfUTnwDsF UVtvNTN1dv2HZCZhpCuCRGzE2G7A1nh6kLWVo.UqtVvKa6Yq2_D8tRnfn3ib8vTEKqWXt_3F26tK MeTuHpznuckSs68dsN0aIfA9rd7abzLeqDc.ezyntUNfsy1OYqkl5mmadkif.UYm0WI5eLTWlgX8 S2GlmpGuFa2HKMSZnQzKrm.6gCX9ZiCa1aHOmeihAB6A4gtQoTwsASOsvnapdNPAtTQpTOP_kcV4 8UH_3.5gOl2dx7rYA8iGF9gODUkxpZi3l0WkPVcEbHWZvQ9RDsJtxYu.uJhZvX0RgScGv_0MOakY rRfpbtoAE7ScPdcwmmqisH8YJoQcJkn5TiscxVJ4nOuu8282UmghaEjLq.Hvllk6pb2iIxw7oEro y80cets9U3yJWgXI3G6ozSY8pT_o.H5ZQm1iHaDh9YMdcnDryIRp1EKfZIRZ6MlqkMQEPjT6BViX kGcM0_paEnLVreGH2TE6HdjxN.O2ti3VSciQydtYiVJMxvC7qzM.wnu4l4EOfi1X_ao.JdIYM6jw e7zvz5W7VOBxCkuwjarcs7Gq0XgYp8BRjzDJfiYS3K7Z1TBO65.JeHpijtVKeiS0msgxn9C1ims2 KX6RGSRaOJLiKPkIiPXmwbR.1T1C8R3XHb_QXu_IYbfY- X-Sonic-MF: X-Sonic-ID: 9261faee-6fd0-4144-ba13-b998811d5462 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.ne1.yahoo.com with HTTP; Tue, 24 Sep 2024 20:47:10 +0000 Date: Tue, 24 Sep 2024 20:47:05 +0000 (UTC) From: amit sehas To: Stephen Hemminger Cc: Nishant Verma , "users@dpdk.org" Message-ID: <26643152.12164440.1727210825368@mail.yahoo.com> In-Reply-To: <20240924093813.29a01783@fedora> References: <1987164393.11670398.1727125003663.ref@mail.yahoo.com> <1987164393.11670398.1727125003663@mail.yahoo.com> <1299564509.11731667.1727133474900@mail.yahoo.com> <2025533199.11789856.1727143607670@mail.yahoo.com> <2042269904.11975457.1727188849962@mail.yahoo.com> <20240924093813.29a01783@fedora> Subject: Re: core performance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.22645 YMailNorrin X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Thanks for the suggestions, so this is a database server which is doing lot= s of stuff, not every thread is heavily involved in dpdk packet processing.= As a result the guidelines for attaining the most dpdk performance are app= licable to only a few threads. In this particular issue we are specificially looking at CPU scheduling of = threads that are primarily heavily processing database queries. These threa= ds, from our measurements, are not being uniformly scheduled on the CPU ... This is our primary concern, since we utilized rte_eal_remote_launch to spa= wn the threads, we are wondering if there are any options in this API that = will allow us to more uniformly allocate the CPU to threads that are critic= al... regards On Tuesday, September 24, 2024 at 09:38:16 AM PDT, Stephen Hemminger wrote:=20 On Tue, 24 Sep 2024 14:40:49 +0000 (UTC) amit sehas wrote: > Thanks for your response, and thanks for your input on the set_priority,= =C2=A0 >=20 > The best guess we have at this point is that this is not a dpdk performan= ce issue. This is an issue with some threads running into more context swit= ches than the others and hence not getting the same slice of the CPU. We ar= e certain that this is not a dpdk performance issue, the code > is uniformly slow in one thread versus the other and the threads are doin= g a very large amount of work including accessing databases. The threads in= question are not really doing packet processing as much as other work. >=20 > So this is certainly not a dpdk performance issue. This is an issue of ke= rnel threads not being scheduled properly or in the worse case the cores ru= nning on different frequency (which is quite unlikely no the AWS Xeons we a= re running this on). >=20 > If you are asking for the dpdk config files to check for dpdk related per= formance issue then we are quite certain the issue is not with dpdk perform= ance ... > On Mon, Sep 23, 2024 at 10:06=E2=80=AFPM amit sehas wro= te: > > Thanks for your response, i am not sure i understand your question ... = we have our product that utilizes dpdk ... the commands are just our server= commands and parameters ... and the lscpu is the hyperthreaded 8 thread Xe= on instance in AWS ... The rules of getting performance in DPDK: =C2=A0 - use DPDK threads (pinned) for datapath =C2=A0 - use isolated CPU's for those DPDK threads =C2=A0 - do not do any system calls =C2=A0 - avoid floating point You can use tracing tools like strace or BPF to see what the thread is doin= g.