From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by dpdk.org (Postfix) with ESMTP id F0C3F3238 for ; Thu, 15 Nov 2018 06:46:20 +0100 (CET) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 72F684000F for ; Thu, 15 Nov 2018 06:46:20 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 5FC9D40014; Thu, 15 Nov 2018 06:46:20 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-0.9 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.1 X-Spam-Score: -0.9 Received: from [192.168.1.59] (host-90-232-59-122.mobileonline.telia.com [90.232.59.122]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 927A14000F; Thu, 15 Nov 2018 06:46:17 +0100 (CET) To: Venky Venkatesh , "dev@dpdk.org" References: <27A03E76-DED0-435F-B02F-24A7A7B1BCC9@contoso.com> <779258cb-490f-0111-94ce-bc87d1502ed0@lysator.liu.se> <0AD526BD-FC54-4128-829D-6D5EE8BEAFC6@paloaltonetworks.com> From: =?UTF-8?Q?Mattias_R=c3=b6nnblom?= Message-ID: Date: Thu, 15 Nov 2018 06:46:17 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <0AD526BD-FC54-4128-829D-6D5EE8BEAFC6@paloaltonetworks.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV using ClamSMTP Subject: Re: [dpdk-dev] Application used for DSW event_dev performance testing X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Nov 2018 05:46:21 -0000 On 2018-11-14 22:56, Venky Venkatesh wrote: > Mattias, > Thanks for the prompt response. Appreciate your situation of not being able to share the proprietary code. More answers inline as [VV]: > --Venky > > On 11/14/18, 11:41 AM, "Mattias Rönnblom" wrote: > > On 2018-11-14 20:16, Venky Venkatesh wrote: > > Hi, > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2018-2DSeptember_111344.html&d=DwIDaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=w2W5SR0mU5u5mz008DZNCsexDN1Lr9bpL7ZGKuD0Zd4&m=H4I6cuKi4kKoypKWz8mjDoXLGgkSNurKbKXrq4qJs5A&s=AD0KG106hPreSKeTQMRzDPwnEfBR9oD6dtjpL2Plt4c&e= mentions that there is a sample application where “worker cores can sustain 300-400 million event/s. With a pipeline > > with 1000 clock cycles of work per stage, the average event device > > overhead is somewhere 50-150 clock cycles/event”. Is this sample application code available? > > > It's proprietary code, although it's also been tested by some of our > partners. > > The primary reason for it not being contributed to DPDK is because it's > a fair amount of work to do so. I would refer to it as an eventdev > pipeline simulator, rather than a sample app. > > > We have written a similar simple sample application where 1 core keeps enqueuing (as NEW/ATOMIC) and n-cores dequeue (and RELEASE) and do no other work. But we are not seeing anything close in terms of performance. Also we are seeing some counter intuitive behaviors such as a burst of 32 is worse than burst of 1. We surely have something wrong and would thus compare against a good application that you have written. Could you pls share it? > > > > Is this enqueue or dequeue burst? How large is n? Is this explicit release? > [VV]: Yes both are burst of 32. I tried n=4-7. It is explicit RELEASE. > If you want good scheduler throughput, don't do explicit release. With other event devices, and heavy-weight pipelines, there might be a point to do so because the released event's flow could potentially be scheduled on other cores. However, on DSW migration won't happen until the application has finished processing its burst, at the earliest. DSW does buffer on enqueue, so large enqueue bursts doesn't improve performance much. They should not decrease performance, unless you go above the configured max burst. > What do you set nb_events_limit to? Good DSW performance much depends on > the average burst size on the event rings, which in turn is dependent on > the number of in-flight events. On really high core-count systems you > might also want to increase DSW_MAX_PORT_OPS_PER_BG_TASK, since it > effectively puts a limit on the maximum number of events buffered on the > output buffers. > [VV]: struct rte_event_dev_config config = { > .nb_event_queues = 2, > .nb_event_ports = 5, > .nb_events_limit = 4096, > .nb_event_queue_flows = 1024, > .nb_event_port_dequeue_depth = 128, > .nb_event_port_enqueue_depth = 128, > }; > struct rte_event_port_conf p_conf = { > .dequeue_depth = 64, > .enqueue_depth = 64, > .new_event_threshold = 1024, "new_event_threshold" effectively puts a limit on the number of inflight events. You should increase this to something close to "nb_events_limit". > .disable_implicit_release = 0, > }; > struct rte_event_queue_conf q_conf = { > .schedule_type = RTE_SCHED_TYPE_ATOMIC, > .priority = RTE_EVENT_DEV_PRIORITY_NORMAL, > .nb_atomic_flows = 1024, > .nb_atomic_order_sequences = 1024, > }; > > In the pipeline simulator all cores produce events initially, and then > recycles events when the number of in-flight events reach a certain > threshold (50% of nb_events_limit). A single lcore won't be able to fill > the pipeline, if you have zero-work stages. > [VV]: I have a single NEW event enqueue thread(0) and a bunch of “dequeue and RELEASE” threads (1-4) – simple case. I have a stats print thread(5) as well. If the 1 enqueue thread is unable to fill the pipeline, what counter would indicate that? I see the contrary effect -- I am tracking the number of times enqueue fails and that number is large. > > There's no counter for failed enqueues, although maybe there should be. "dev_credits_on_load" can be seen as an estimate of how many events are currently inflight in the scheduler. If this number is close to your "new_event_threshold", the pipeline is busy. If it's low, in the couple-of-hundreds range, your pipeline is likely not-so-busy (even idle) because not enough events are being fed into it. You can obviously detect failed NEW enqueues in the application as well. I'm not sure exactly how much one core can produce, and it obviously depends on what kind of core, but it's certainly a lot lower than "300-400 millions events/s". Maybe something like 40-50 Mevents/s. What is your flow id distribution? As in, how many flow ids are you actively using in the events are you feeding the different queues/pipeline stages. > Even though I can't send you the simulator code at this point, I'm happy > to assist you in any DSW-related endeavors. > [VV]: My program is a simple enough program (nothing proprietary) that I can share. Can I unicast it to you for a quick recommendation? > Sure, although I prefer to have any discussions on the mailing list, so other users can learn from your experiences. Btw, you really need to get a proper mail user agent, or configure the one you have to quote messages as per normal convention.