From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9A189A0544; Mon, 10 Oct 2022 19:38:58 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3300140143; Mon, 10 Oct 2022 19:38:58 +0200 (CEST) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by mails.dpdk.org (Postfix) with ESMTP id 4312240041 for ; Mon, 10 Oct 2022 19:38:57 +0200 (CEST) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id E170917DA0 for ; Mon, 10 Oct 2022 19:38:56 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id E02B217EA1; Mon, 10 Oct 2022 19:38:56 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hermod.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-2.5 required=5.0 tests=ALL_TRUSTED, AWL, NICE_REPLY_A autolearn=disabled version=3.4.6 X-Spam-Score: -2.5 Received: from [192.168.1.59] (h-62-63-215-114.A163.priv.bahnhof.se [62.63.215.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 1FAB217C4A; Mon, 10 Oct 2022 19:38:52 +0200 (CEST) Message-ID: Date: Mon, 10 Oct 2022 19:38:51 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Subject: Re: [PATCH v7 0/4] Add lcore poll busyness telemetry Content-Language: en-US To: =?UTF-8?Q?Morten_Br=c3=b8rup?= , =?UTF-8?Q?Mattias_R=c3=b6nnblom?= , Kevin Laatz , dev@dpdk.org Cc: anatoly.burakov@intel.com, Bruce Richardson , Konstantin Ananyev , Conor Walsh , David Hunt , Nicolas Chautru , Fan Zhang , Ashish Gupta , Akhil Goyal , Chengwen Feng , Ray Kinsella , Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , Jerin Jacob , Sachin Saxena , Hemant Agrawal , Ori Kam , Honnappa Nagarahalli , David Marchand References: <24c49429394294cfbf0d9c506b205029bac77c8b.1657890378.git.anatoly.burakov@intel.com> <20220914092929.1159773-1-kevin.laatz@intel.com> <3ec04d77-43e2-6eed-53b5-723ca5058911@intel.com> <98CBD80474FA8B44BF855DF32C47DC35D873A0@smartserver.smartshare.dk> <1be242de-8aef-5d8b-9ffa-f5db460fc6ef@ericsson.com> <98CBD80474FA8B44BF855DF32C47DC35D873CE@smartserver.smartshare.dk> From: =?UTF-8?Q?Mattias_R=c3=b6nnblom?= In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D873CE@smartserver.smartshare.dk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2022-10-10 17:22, Morten Brørup wrote: >> From: Mattias Rönnblom [mailto:mattias.ronnblom@ericsson.com] >> Sent: Thursday, 6 October 2022 17.27 >> >> On 2022-10-06 15:25, Morten Brørup wrote: >>>> From: Kevin Laatz [mailto:kevin.laatz@intel.com] >>>> Sent: Wednesday, 5 October 2022 15.45 >>>> >>>> On 14/09/2022 10:29, Kevin Laatz wrote: >>>>> Currently, there is no way to measure lcore polling busyness in a >>>> passive >>>>> way, without any modifications to the application. This patchset >> adds >>>> a new >>>>> EAL API that will be able to passively track core polling busyness. >>>> As part >>>>> of the set, new telemetry endpoints are added to read the generate >>>> metrics. >>>>> >>>>> --- >>>> >>>> Based on the feedback in the discussions on this patchset, we have >>>> decided to revoke the submission of this patchset for the 22.11 >>>> release. >>>> >>>> We will re-evaluate the design with the aim to provide a more >>>> acceptable >>>> solution in a future release. >>> >>> Good call. Thank you! >>> >>> I suggest having an open discussion about requirements/expectations >> for such a solution, before you implement any code. >>> >>> We haven't found the golden solution for our application, but we have >> discussed it quite a lot internally. Here are some of our thoughts: >>> >>> The application must feed the library with information about how much >> work it is doing. >>> >>> E.g. A pipeline stage that polls the NIC for N ingress packets could >> feed the busyness library with values such as: >>> - "no work": zero packets received, >>> - "25 % utilization": less than N packets received (in this >> example: 8 of max 32 packets = 25 %), or >>> - "100% utilization, possibly more work to do": all N packets >> received (more packets could be ready in the queue, but we don't know). >>> >> >> If some lcore's NIC RX queue always, for every poll operation, produces >> 8 packets out of a max burst of 32, I would argue that lcore is 100% >> busy. With always something to do, it doesn't have a single cycle to >> spare. > > I would argue that if I have four cores each only processing 25 % of the packets, one core would suffice instead. Or, the application could schedule the function at 1/4 of the frequency it does now (e.g. call the function once every 40 microseconds instead of once every 10 microseconds). > Do you mean "only processing packets 25% of the time"? If yes, being able to replace four core @ 25% utilization with one core @ 100% might be a reasonable first guess. I'm not sure how it relates to what I wrote, though. > However, the business does not scale linearly with the number of packets processed - which an intended benefit of bursting. > Sure, there's usually a non-linear relationship between the system capacity used and the resulting CPU utilization. It can be both in the manner you describe below, with the per-packet processing latency reduced at higher rates, or the other way around. For example, NIC RX LLC stashing may cause a lot of LLC evictions, and generally the application might have a larger working set size during high load, so there may be forces working in the other direction as well. It seems to me "busyness" telemetry value should just be lcore thread CPU utilization (in total, or with some per-module breakdown). If you want to know how much of the system's capacity is used, you need help from an application-specific agent, equipped with a model of how CPU utilization and capacity relates. Such a heuristic could take other factors into account as well, e.g. the average queue sizes, packet rates, packet sizes etc. In my experience, for high touch applications (i.e., those that spends thousands of cycles per packet), CPU utilization is a pretty decent approximation on how much of the system's capacity is used. > Here are some real life numbers from our in-house profiler library in a production environment, which says that polling the NIC for packets takes on average: > > 104 cycles when the NIC returns 0 packets, > 529 cycles when the NIC returns 1 packet, > 679 cycles when the NIC returns 8 packets, and > 1275 cycles when the NIC returns a full burst of 32 packets. > > (This includes some overhead from our application, so you will see other numbers in your application.) > >> >> It seems to me that you basically have two options, if you do >> application-level "busyness" reporting. >> >> Either the application >> a) reports when a section of useful work begins, and when it ends, as >> two separate function calls. >> b) after having taken a time stamp, and having completed a section of >> code which turned out to be something useful, it reports back to the >> busyness module with one function call, containing the busy cycles >> spent. >> >> In a), the two calls could be to the same function, with a boolean >> argument informing the busyness module if this is the beginning of a >> busy or an idle period. In such case, just pass "num_pkts_dequeued > 0" >> to the call. > > Our profiler library has a start()and an end() function, and an end_and_start() function for when a section directly follows the preceding section (to only take one timestamp instead of two). > I like the idea of a end_and_start() (except for the name, maybe). >> >> What you would like is a solution which avoid ping-pong between idle >> and >> busy states (with the resulting time stamping and computations) in >> scenarios where a lcore thread mix sources of work which often have >> items available, with sources that do not (e.g., packets in a RX queue >> versus reassembly timeouts in a core-local timer wheel). It would be >> better in that situation, to attribute the timer wheel poll cycles as >> busy cycles. >> >> Another crucial aspect is that you want the API to be simple, and code >> changes to be minimal. >> >> It's unclear to me if you need to account for both idle and busy >> cycles, >> or only busy cycles, and assume all other cycles are idle. The will be >> for a traditional 1:1 EAL thread <-> CPU core mapping, but not if the >> "--lcores" parameter is used to create floating EAL threads, and EAL >> threads which share the same core, and thus may not be able to use 100% >> of the TSC cycles. >> >>> A pipeline stage that services a QoS scheduler could additionally >> feed the library with values such as: >>> - "100% utilization, definitely more work to do": stopped >> processing due to some "max work per call" limitation. >>> - "waiting, no work until [DELAY] ns": current timeslot has been >> filled, waiting for the next timeslot to start. >>> >>> It is important to note that any pipeline stage processing packets >> (or some other objects!) might process a different maximum number of >> objects than the ingress pipeline stage. What I mean is: The number N >> might not be the same for all pipeline stages. >>> >>> >>> The information should be collected per lcore or thread, also to >> prevent cache trashing. >>> >>> Additionally, it could be collected per pipeline stage too, making >> the collection two-dimensional. This would essentially make it a >> profiling library, where you - in addition to seeing how much time is >> spent working - also can see which work the time is spent on. >>> >> >> If you introduce subcategories of "busy", like "busy-with-X", and >> "busy-with-Y", the book keeping will be more expensive, since you will >> transit between states even for 100% busy lcores (which in principle >> you >> never, or at least very rarely, need to do if you have only busy and >> idle as states). >> >> If your application is organized as DPDK services, you will get this >> already today, on a DPDK service level. >> >> If you have your application organized as a pipeline, and you use an >> event device as a scheduler between the stages, that event device has a >> good opportunity to do this kind of bookkeeping. DSW, for example, >> keeps >> track of the average processing latency for events, and how many events >> of various types have been processed. >> > > Lots of good input, Mattias. Let's see what others suggest. :-) > >>> As mentioned during the previous discussions, APIs should be provided >> to make the collected information machine readable, so the application >> can use it for power management and other purposes. >>> >>> One of the simple things I would like to be able to extract from such >> a library is CPU Utilization (percentage) per lcore. > >>> And since I want the CPU Utilization to be shown for multiple the >> time intervals (usually 1, 5 or 15 minutes; but perhaps also 1 second >> or 1 millisecond) the output data should be exposed as a counter type, >> so my "loadavg application" can calculate the rate by subtracting the >> previously obtained value from the current value and divide the >> difference by the time interval. >>> >> >> I agree. In addition, you also want the "raw data" (lcore busy cycles) >> so you can do you own sampling, at your own favorite-length intervals. >> >>> -Morten >>> >