From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DF7EF45A3C for ; Fri, 27 Sep 2024 05:13:54 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C3D9E400EF; Fri, 27 Sep 2024 05:13:54 +0200 (CEST) Received: from sonic310-25.consmr.mail.ne1.yahoo.com (sonic310-25.consmr.mail.ne1.yahoo.com [66.163.186.206]) by mails.dpdk.org (Postfix) with ESMTP id 6E0004003C for ; Fri, 27 Sep 2024 05:13:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727406832; bh=NFtPiE/bucnC71+4kJffDLVI21tXzOdKCqlqg4IpBnE=; h=Date:From:To:Cc:In-Reply-To:References:Subject:From:Subject:Reply-To; b=bl5P/jiKng1aAJD2HMpLpeC8By+7dD33UFdPGynoR/QeIMBaxhKHkLJJWPxncKkadkN1/OmplCIuWutVdCytEpQu1fOuoifDbGxfZaXV5X5uuZW1bQktA/vZ1hb9mddf8MTQHilZGdVNpqFXQVu6bV/Ci0mDgtQyy+FxZ/1RFT35iqOp5b6Gt/91J56+YGmJv3YAu+Hzf3jH/w2u95JVp71pKFjC2hfFFsd8Qddt8KYSJ83ZsgkYh0GQedSJBgkEDemekUypRYKLNYnECp59Lt5Jd/0/LZPt8bIJdqAVHe38zXDBVYvZ7VrWUhF07Om9LGqPwxT9THOyNcadYC+uyg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1727406832; bh=UHeOMXgPKHFRaXZtND0I6aQFa29eEjLr++iSjKy5nUT=; h=X-Sonic-MF:Date:From:To:Subject:From:Subject; b=mAuVRl+K45JA3hsrLNDBmQTecXWewPTUNJoh6y7dsOGNLBz0LPw1fwrBx//mQu+mcQq75JfJKy7r23esXaueB4zn90j9q8SlUA/ChCLf8eVr6KdPOTfgV+g7AMccIAuCwGw7JotNR5+2dZSWIXre80sUNj4mlKD/ZIjnoU2T8r8pMml8lI9bA3TEgq7YCX+U3W9QA7eqROHN/UlehJ4tr9xNUTzK8eGwgb8AmFcAq6L9fL45MBjBC2gNsDKav8Ezq0rp2MFOtC6IqE1QSHG8RpnrQqizSqW0qQ2B++Q3NJ7VYMA6vHGN71lV1Wc+zpcJZ4JsL7zFpOaUJjdI6GQ3ig== X-YMail-OSG: L5D4Qb8VM1liVFrZqP_Ez8BX.211sZ2K8hQfQke61nZDdJPh4Ig6d.wySU.m50T PMAoma4fRfMkeW_02zAE4j.pXsoxVd.hmZh118V3JREnDyxGDpIymeBl5l.GicwyIARI4zTk9xOm tbrVQsJbWVDN6kLaE0BqMiVpVL9hyF0GzWQvlRSgLsAOy9l71XabQ8_xdazDTz6T8EMb16OrRVKZ tKNpgr_7sVdo3Uk.JbVSR46zL14hJjmisnVukpAGbnnoX0jt8Ne5qEehYSmJnjh_myxz0p1n03eu q3rv9XJx6rJ6rbhkRG3LKgNQISDRaojBeB2Qb9fGi1jNeNEkxKkg5o87y.G0OKkJji0JcDTRusvE alSfjlh_AITWYcleVoEHt97UB9VxhVNBPU7J6oosU8AlQCRJxFTZYLw7IyBGbwzRF3T8MNfsrIvd Eizb6LRgGwKkVBf2w67IdKAlwAiFGN.AN4B7uooUvGNOU58x9MWtDPGdd_GmCaVsNWsqcrelL.gw M4n3OlBG.51trcwknfW5wh3lgK.7FltH0tN7MBvAX7YDLRknx3QnQsX8J678Dwgn4Sz5Xa9pkmEg ggHc17R4fyERvI8knXrOhzwTOYCWxz0xxH3Cr5QpztSMzTyZFjTNFPHzVYpDHV.FGmSg0izau6eh 8Alrzr5IDjqa6Xk_9ueGMomiSQz6W06Femqp9Ein8GGl6iV2MTSjh1rZS60_ceBCrRFDn7Sb4BO9 RO1PucBqeYSo53WieH_qi1CI3vXS4dbTuHfWK0lVgQ.hdSS7HhuZXXAVpH8dl1JRyofv64BQBStR BfoS2rJh4fYq2iG8n6aa8Azi_dX_uynM0cHgCK0lWwKJ_zqs1y2lCHh2xIcZWEVjvOBfHjPOP1NJ WuL3bXkzNi4_nmQ4IADlYRNMNlRcPC2R6eU.JKg2YQ19jvsKutKF4Y.LCkIRECJ6Zlju4KLPAZql EKsqDP_bRWzmLrDtlcojWoqjDX2nMNujnkUoY_kUpISCRrRoMwx7Rh0ezuTFpTrEXsvX4XB46rvR 3PunnV_EHIt.QSwwH_xPQ184y_pokHd5DC_7tiYngCKWsV6lG4mPBaDsaiT8ZmV_6QBGcHYbtvGo TlCh4ztMGWSMLRgwIIPuWWf86X..f6I1SpbZFtGxw43dYn5y1lkJqqLz718V9uHCUK3nVTEWjMAY L6L.tOocVV9Bl38o4.cmAy_egTcfCenRPqq84ZLojNwkSDFN9bKIvgcI_VQayXNWynRtQ3yiZwoL ijVM7QkiBxt6m.OsimsQsxyvapn8XFr6CdY7g.XTvt8Qd8W.KfFCn4muiMqXaw03X3YcUAJUmbVi ZCBlDuMt.3Svd61ULV7VkfTNZeBwR_1LMjEDU0clx3JIfFbzKqQAMDPVJ16HjvFC64GVTUAtfRgp CX6gL7BNTcpZz07SRQlg6Ese6ZbYzKJXBbljMfI.mdO.c5duYAYq9oMb7EL9Aan4kTeMe_x5vhB0 O1VG.4B7CnwGpiYms5HDuXWzzlSWNkU5offY9ghJe6oDqqAZOCK3Zti16Lp9NbGNe7TRTgHFwSSa d7BUdGPEWbl9dJfKleuYTRjiozqklNlMfOGDPEQoWFoUh7r0K6Lio7899xdBGdPEi7AclulbdN9F 1xNLsHrR7epzRHfQxjZZgxQ5GZ9kp8znJBAUFp_3v0UTOmSUXoYRYtr213xL0zn0_hDFtGHhArao 0D71KIyNm4hF.zF02o0NX0LkULV.9ziUik0tkk6imLvCusnduxUoiAXNw4rrViWAM.rfiZAKFFJQ K7umrcUqm0OjqsJxZi.rZyoSiz_XJzpRtojUBvxy_MYoxAhp3W0MinHG.N4DJF2Ip_KDdptGUbky WbDvq66oMURVtimBVily8ul5Xf58Rrp_pzeeJNDqCM3JkSOBRhiwUx2xTYlI9n6TtGdpLnaaFVXY RdPxaoMkppF_cPjnrlLvhEMmTV6bCZ_X0kzaVoTqG7bujIWten26p782PQbuf_yOCbHKAohJ7DaB QeK0BsTkATl5YCEfFXSdfV0PiEhm.CBGgIa.ngDtc9byF_Jr_R7z6l1BM9ZuWqA5pN3LX_n15AVd JRfSUP7fTT9yNzGHEAa4.ZrgHiPa5Ew9tgdVKN5fnzI_u7qkroOvMbpBCM_ZGwMyQr7euTXBM9CV tJYAhfTb5bwJRzlMIzXJxuJFxPznKcgtq8Wk0kKXiuERv7XhGGYuDE2DRY40NIt3hmRg8PpFQZ6s hUz_HgTQnQSz9iSuqOA4HEQJnx8UiM9SpMddk1EhIY1Zp X-Sonic-MF: X-Sonic-ID: 41637cd7-44b0-4bbd-bc0b-aca1b86c6cd7 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.ne1.yahoo.com with HTTP; Fri, 27 Sep 2024 03:13:52 +0000 Date: Fri, 27 Sep 2024 03:13:48 +0000 (UTC) From: amit sehas To: Stephen Hemminger Cc: Nishant Verma , "users@dpdk.org" Message-ID: <894778529.13304006.1727406828202@mail.yahoo.com> In-Reply-To: <20240926200327.6f74ae2d@hermes.local> References: <1987164393.11670398.1727125003663.ref@mail.yahoo.com> <1987164393.11670398.1727125003663@mail.yahoo.com> <1299564509.11731667.1727133474900@mail.yahoo.com> <2025533199.11789856.1727143607670@mail.yahoo.com> <2042269904.11975457.1727188849962@mail.yahoo.com> <20240924093813.29a01783@fedora> <26643152.12164440.1727210825368@mail.yahoo.com> <810098753.12902128.1727353971978@mail.yahoo.com> <47151973.13053687.1727369764729@mail.yahoo.com> <610700496.13049973.1727370197636@mail.yahoo.com> <20240926200327.6f74ae2d@hermes.local> Subject: Re: core performance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Mailer: WebService/1.1.22645 YMailNorrin X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Thanks for the suggestion, i didnt even know about cpu_layout.py ... i will= definitely try it.... i made some more measurements and so far this is the= hypothesis: 1) 8 hyperthreads are not the same as 8 CPUs, the scale up is not linear. 2) the vCPU cache allocation per logical CPU thread is also important, if 2= threads are running the same code on the same physical core but 2 differen= t logical cores then we will not have cores competing with each other. 3) try to run dissimilar code on the logical cores that run on the same phy= sical core ... the cpu map is deinitely worth figuring out ... regards On Thursday, September 26, 2024 at 08:03:30 PM PDT, Stephen Hemminger wrote:=20 On Thu, 26 Sep 2024 17:03:17 +0000 (UTC) amit sehas wrote: > If there is a way to determine: >=20 > vCPU thread utilization numbers over a period of time, such as a few hour= s >=20 > or which processes are consuming the most CPU >=20 > top always indicates that the server is consuming the most CPU. >=20 > Now i am begining to wonder if 8 vCPU threads really are capable of runni= ng 6 high intensity threads or only 4 such threads? Dont know >=20 > Also tried to utilize=C2=A0pthread_setschedparam() explicitly on some of = the threads, it made no difference to the performance. But if we do it on m= ore than 1-2 threads then it hangs the whole system. >=20 > This is primarily a matter of CPU scheduling, and if we restirct context = switching on even 2 critical threads we have a win. >=20 >=20 Some other recommendations. =C2=A0 - avoid CPU 0 you can't isolate it, and it has other stuff that has = to run there =C2=A0 =C2=A0 if you have main thread that sleeps, and worker threads that = poll, then =C2=A0 =C2=A0 go ahead and put main on cpu 0. =C2=A0 - don't put two active polling cores on shared hyper-thread. =C2=A0 =C2=A0 You can used DPDK's cpu_layout.py script to show this. For example: $ ./usertools/cpu_layout.py=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Core and Socket Information (as reported by '/sys/devices/system/cpu') =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D cores =3D=C2=A0 [0, 1, 2, 3] sockets =3D=C2=A0 [0] =C2=A0 =C2=A0 =C2=A0 Socket 0=C2=A0 =C2=A0=20 =C2=A0 =C2=A0 =C2=A0 --------=C2=A0 =C2=A0=20 Core 0 [0, 4]=C2=A0 =C2=A0 =C2=A0=20 Core 1 [1, 5]=C2=A0 =C2=A0 =C2=A0=20 Core 2 [2, 6]=C2=A0 =C2=A0 =C2=A0=20 Core 3 [3, 7]=20 On this system, don't poll on cores 0 and 4 (system activity). Use lcore 1, 2, 3=20