Re: [dpdk-dev] Application used for DSW event_dev performance testing

["Mattias Rönnblom" <mattias.ronnblom@ericsson.com>]

On 2018-11-14 22:56, Venky Venkatesh wrote:
> Mattias,
> Thanks for the prompt response. Appreciate your situation of not being able to share the proprietary code. More answers inline as [VV]:
> --Venky
> 
> On 11/14/18, 11:41 AM, "Mattias Rönnblom" <hofors@lysator.liu.se> wrote:
> 
>      On 2018-11-14 20:16, Venky Venkatesh wrote:
>      > Hi,
>      >
>      > https://urldefense.proofpoint.com/v2/url?u=https-3A__mails.dpdk.org_archives_dev_2018-2DSeptember_111344.html&d=DwIDaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=w2W5SR0mU5u5mz008DZNCsexDN1Lr9bpL7ZGKuD0Zd4&m=H4I6cuKi4kKoypKWz8mjDoXLGgkSNurKbKXrq4qJs5A&s=AD0KG106hPreSKeTQMRzDPwnEfBR9oD6dtjpL2Plt4c&e= mentions that there is a sample application where “worker cores can sustain 300-400 million event/s. With a pipeline
>      > with 1000 clock cycles of work per stage, the average event device
>      > overhead is somewhere 50-150 clock cycles/event”. Is this sample application code available?
>      >
>      It's proprietary code, although it's also been tested by some of our
>      partners.
>      
>      The primary reason for it not being contributed to DPDK is because it's
>      a fair amount of work to do so. I would refer to it as an eventdev
>      pipeline simulator, rather than a sample app.
>      
>      > We have written a similar simple sample application where 1 core keeps enqueuing (as NEW/ATOMIC) and n-cores dequeue (and RELEASE) and do no other work. But we are not seeing anything close in terms of performance. Also we are seeing some counter intuitive behaviors such as a burst of 32 is worse than burst of 1. We surely have something wrong and would thus compare against a good application that you have written. Could you pls share it?
>      >
>      
>      Is this enqueue or dequeue burst? How large is n? Is this explicit release?
>   [VV]: Yes both are burst of 32. I tried n=4-7. It is explicit RELEASE.
>     

If you want good scheduler throughput, don't do explicit release. With 
other event devices, and heavy-weight pipelines, there might be a point 
to do so because the released event's flow could potentially be 
scheduled on other cores. However, on DSW migration won't happen until 
the application has finished processing its burst, at the earliest.

DSW does buffer on enqueue, so large enqueue bursts doesn't improve 
performance much. They should not decrease performance, unless you go 
above the configured max burst.

>      What do you set nb_events_limit to? Good DSW performance much depends on
>      the average burst size on the event rings, which in turn is dependent on
>      the number of in-flight events. On really high core-count systems you
>      might also want to increase DSW_MAX_PORT_OPS_PER_BG_TASK, since it
>      effectively puts a limit on the maximum number of events buffered on the
>      output buffers.
> [VV]:         struct rte_event_dev_config config = {
>                          .nb_event_queues = 2,
>                          .nb_event_ports = 5,
>                          .nb_events_limit  = 4096,
>                          .nb_event_queue_flows = 1024,
>                          .nb_event_port_dequeue_depth = 128,
>                          .nb_event_port_enqueue_depth = 128,
>          }; >          struct rte_event_port_conf p_conf = {
>                          .dequeue_depth = 64,
>                          .enqueue_depth = 64,
>                          .new_event_threshold = 1024,

"new_event_threshold" effectively puts a limit on the number of inflight 
events. You should increase this to something close to "nb_events_limit".

>                          .disable_implicit_release = 0,
>          };
>          struct rte_event_queue_conf q_conf = {
>                          .schedule_type = RTE_SCHED_TYPE_ATOMIC,
>                          .priority = RTE_EVENT_DEV_PRIORITY_NORMAL,
>                          .nb_atomic_flows = 1024,
>                          .nb_atomic_order_sequences = 1024,
>          };

>      
>      In the pipeline simulator all cores produce events initially, and then
>      recycles events when the number of in-flight events reach a certain
>      threshold (50% of nb_events_limit). A single lcore won't be able to fill
>      the pipeline, if you have zero-work stages.
> [VV]: I have a single NEW event enqueue thread(0) and a bunch of “dequeue and RELEASE” threads (1-4) – simple case. I have a stats print thread(5) as well. If the 1 enqueue thread is unable to fill the pipeline, what counter would indicate that? I see the contrary effect -- I am tracking the number of times enqueue fails and that number is large.
> 
>      
There's no counter for failed enqueues, although maybe there should be. 
"dev_credits_on_load" can be seen as an estimate of how many events are 
currently inflight in the scheduler. If this number is close to your 
"new_event_threshold", the pipeline is busy. If it's low, in the 
couple-of-hundreds range, your pipeline is likely not-so-busy (even 
idle) because not enough events are being fed into it.

You can obviously detect failed NEW enqueues in the application as well.

I'm not sure exactly how much one core can produce, and it obviously 
depends on what kind of core, but it's certainly a lot lower than 
"300-400 millions events/s". Maybe something like 40-50 Mevents/s.

What is your flow id distribution? As in, how many flow ids are you 
actively using in the events are you feeding the different 
queues/pipeline stages.

>      Even though I can't send you the simulator code at this point, I'm happy
>      to assist you in any DSW-related endeavors.
> [VV]: My program is a simple enough program (nothing proprietary) that I can share. Can I unicast it to you for a quick recommendation?
> 

Sure, although I prefer to have any discussions on the mailing list, so 
other users can learn from your experiences.

Btw, you really need to get a proper mail user agent, or configure the 
one you have to quote messages as per normal convention.