From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id C5788A0555; Wed, 19 Feb 2020 13:05:57 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 104921B13C; Wed, 19 Feb 2020 13:05:57 +0100 (CET) Received: from mail-ua1-f42.google.com (mail-ua1-f42.google.com [209.85.222.42]) by dpdk.org (Postfix) with ESMTP id E5C163B5 for ; Wed, 19 Feb 2020 13:05:55 +0100 (CET) Received: by mail-ua1-f42.google.com with SMTP id h32so86963uah.4 for ; Wed, 19 Feb 2020 04:05:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dtnOonQaMA30QXhJKp6+XI20heHqu+HTsT/tW4Pc7tA=; b=OG+gYVTVuFUoKPxMfIX691cyTQ+WG0hYigbb4LdvYC8vvyYWXDL1a9iAWMoA8emD4z S8eauppZt0rxbEA+kaXN3N8tWVdJUCNpalUKSzWRRmnofnBQ390YAbJMnzLYGMiVKgIT PmiPvgL4Jn3ffOVZnFJ53BnWa+ejSNh86tH0ZrUJF4Wj+ONpV7kBvM6W71hKBk+VR+7S nuvPaZDqP1DNZh2Nn8qI3fHAmIKqc163q36yAzux+lbG/SF2tCF3gO/eFk0nTO/vPXnu g9lhf1hQ4KpqSv21tvcIJoGVbVSld2U7YrBnvHgzMXMkPYYbzizxNAq/nHKagnmHPNXa fvLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dtnOonQaMA30QXhJKp6+XI20heHqu+HTsT/tW4Pc7tA=; b=X81uFwp8x1CppXMbbHJaP8/pdBZE9HfLFZOP0RMVaP79zpN2hAoEnXqNTLDsYWwnUj kJVygNJqNCqQuuopaJUVelITCvS2pKc7myFcI0TDXB6a2EqjaVyK4+fVprgCXJz94lyQ wb0vhOEiaqbXCZhTuDt0+uexHSL+6GrSPNwNZEVPav3TYbuJ8Up/9/saeSyj8q31ekBp unFnwTAelIPEj3lnCpsHHnCN4tU7ozrDk3PEg7nLdQwKiOVYqxrhihCw2CKQu4ANqHeO vZpX0mUOQnqC2DRkePfP1jQgjKw9Qvh8sl9/HZB9jKx08JKIv/lDqOaMmmbUmR1de4P/ mY4A== X-Gm-Message-State: APjAAAWrWlnbPUhQI7sJkhdXRgmP7KVnSp9AvrIInKfVaJp7N1zG9wgH E6ui52fSj672KnXXNyTAYFmNFeF9RRREiLhuOK8= X-Google-Smtp-Source: APXvYqyN5FnwtNtDjIFAoRv0uAal9FMl4AenthmnpL2IrZrdB9DgXez+IHd0MngDyYQb4H1hV8D7a6gvNXy0/n/ZM/E= X-Received: by 2002:ab0:555e:: with SMTP id u30mr12962558uaa.39.1582113954962; Wed, 19 Feb 2020 04:05:54 -0800 (PST) MIME-Version: 1.0 References: <20200219105328.GF22738@platinum> In-Reply-To: <20200219105328.GF22738@platinum> From: Victor Huertas Date: Wed, 19 Feb 2020 13:05:43 +0100 Message-ID: To: Olivier Matz Cc: James Huang , cristian.dumitrescu@intel.com, dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] Fwd: Fwd: high latency detected in IP pipeline example X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Oliver, Thanks for your answer. I think that the most appropriate maintainer to answer to this issue is Cristian as it is the maintainer of ip_pipeline. In order to tell you how to reproduce the problem, you should go back to the DPDK v17.11 and run the ip_pipeline app having a *.cfg configuration file where you put N (where N is more than 3) pipelines in a row. No matter if the pipelines apply the default entry of the tables they may have. The important thing is that the packet crosses all the pipelines, not the process that they receive. That is, the point is that several f_run() functions (one for each pipeline) falls into the same thread which is associated to an unique logical core (the f_runs are executed one after the other) and the rte_mbufs are read and written into/from software queues. If you need a particular configuration that you can test quickly I should remount this environment in my lab and test it again.This would take a while from my part. I have evolved the pipelines a little bit from the original app and I cannot provide you with these pipelines. The only thing I can assure you is that the mechanism of "connecting" the pipelines depending on the logical core where you want to run them is untouched from the original app. This latency issue is something that worries me quite a lot because I need to justify to my bosses the use of DPDK as a key library to improve performance of the application we are developing in my company (please, understand that I cannot tell you more). I decided to base the application on the ip_pipeline app becuase it offered me the opportunity to mix pipelines including built-in tables and built-in f_run with customized pipelines with custom f_run function. I saw a very attractive point in this--> flexibility in conforming our own packet processing models by contatenating well defined pipelines. However I found myself with the mentoined latency issue. I hope this allows you to understand better where I am now. Regards, El mi=C3=A9., 19 feb. 2020 a las 11:53, Olivier Matz () escribi=C3=B3: > Hi Victor, > > I have no experience with ip_pipeline. I can at least say that this > latency is much higher that what you should get. > > My initial thought was that you were using several pthreads bound to the > same core, but from what I read in your first mail, this is not the > case. > > Do you have a simple way to reproduce your issue with the original > example app? > > > Olivier > > On Wed, Feb 19, 2020 at 11:37:21AM +0100, Victor Huertas wrote: > > Hi , > > > > I put some maintainers as destination that could provide some extra > > information on this issue. > > I hope they can shed some light on this. > > > > Regards > > > > El mi=C3=A9., 19 feb. 2020 a las 9:29, Victor Huertas () > > escribi=C3=B3: > > > > > OK James, > > > Thanks for sharing your own experience. > > > What I would need right now is to know from maintainers if this laten= cy > > > behaviour is something inherent in DPDK in the particular case we ar= e > > > talking about. Furthermore, I would also appreciate it if some > maintainer > > > could tell us if there is some workaround or special configuration th= at > > > completely mitigate this latency. I guess that there is one mitigatio= n > > > mechanism, which is the approach that the new ip_pipeline app example > > > exposes: if two or more pipelines are in the same core the "connectio= n" > > > between them is not a software queue but a "direct table connection". > > > > > > This proposed approach has a big impact on my application and I would > like > > > to know if there is other mitigation approach taking into account the > "old" > > > version of ip_pipeline example. > > > > > > Thanks for your attention > > > > > > > > > El mar., 18 feb. 2020 a las 23:09, James Huang () > > > escribi=C3=B3: > > > > > >> No. I didn't notice the RTT bouncing symptoms. > > >> In high throughput scenario, if multiple pipelines runs in a single > cpu > > >> core, it does increase the latency. > > >> > > >> > > >> Regards, > > >> James Huang > > >> > > >> > > >> On Tue, Feb 18, 2020 at 1:50 AM Victor Huertas > > >> wrote: > > >> > > >>> Dear James, > > >>> > > >>> I have done two different tests with the following configuration: > > >>> [PIPELINE 0 MASTER core =3D0] > > >>> [PIPELINE 1 core=3D1] --- SWQ1--->[PIPELINE 2 core=3D1] -----SWQ2--= --> > > >>> [PIPELINE 3 core=3D1] > > >>> > > >>> The first test (sending a single ping to cross all the pipelines to > > >>> measure RTT) has been done by setting the burst_write to 32 in SWQ1 > and > > >>> SWQ2. NOTE: All the times we use rte_ring_enqueue_burst in the > pipelines 1 > > >>> and 2 we set the number of packets to write to 1. > > >>> > > >>> The result of this first test is as shown subsquently: > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D343 ttl=3D63 time=3D59.8 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D344 ttl=3D63 time=3D59.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D345 ttl=3D63 time=3D59.2 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D346 ttl=3D63 time=3D59.0 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D347 ttl=3D63 time=3D59.0 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D348 ttl=3D63 time=3D59.2 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D349 ttl=3D63 time=3D59.3 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D350 ttl=3D63 time=3D59.1 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D351 ttl=3D63 time=3D58.9 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D352 ttl=3D63 time=3D58.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D353 ttl=3D63 time=3D58.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D354 ttl=3D63 time=3D58.0 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D355 ttl=3D63 time=3D58.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D356 ttl=3D63 time=3D57.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D357 ttl=3D63 time=3D56.9 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D358 ttl=3D63 time=3D57.2 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D359 ttl=3D63 time=3D57.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D360 ttl=3D63 time=3D57.3 ms > > >>> > > >>> As you can see, the RTT is quite high and the range of values is > more or > > >>> less stable. > > >>> > > >>> The second test is the same as the first one but setting burst_writ= e > to > > >>> 1 for all SWQs. The result is this one: > > >>> > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D131 ttl=3D63 time=3D10.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D132 ttl=3D63 time=3D10.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D133 ttl=3D63 time=3D10.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D134 ttl=3D63 time=3D10.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D135 ttl=3D63 time=3D10.8 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D136 ttl=3D63 time=3D10.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D137 ttl=3D63 time=3D10.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D138 ttl=3D63 time=3D10.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D139 ttl=3D63 time=3D10.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D140 ttl=3D63 time=3D10.2 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D141 ttl=3D63 time=3D10.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D142 ttl=3D63 time=3D10.9 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D143 ttl=3D63 time=3D11.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D144 ttl=3D63 time=3D11.3 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D145 ttl=3D63 time=3D11.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D146 ttl=3D63 time=3D11.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D147 ttl=3D63 time=3D11.0 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D148 ttl=3D63 time=3D11.3 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D149 ttl=3D63 time=3D12.0 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D150 ttl=3D63 time=3D12.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D151 ttl=3D63 time=3D12.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D152 ttl=3D63 time=3D12.3 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D153 ttl=3D63 time=3D12.8 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D154 ttl=3D63 time=3D12.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D155 ttl=3D63 time=3D12.8 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D156 ttl=3D63 time=3D12.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D157 ttl=3D63 time=3D12.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D158 ttl=3D63 time=3D12.9 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D159 ttl=3D63 time=3D13.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D160 ttl=3D63 time=3D13.8 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D161 ttl=3D63 time=3D13.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D162 ttl=3D63 time=3D13.3 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D163 ttl=3D63 time=3D13.3 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D164 ttl=3D63 time=3D13.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D165 ttl=3D63 time=3D13.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D166 ttl=3D63 time=3D13.8 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D167 ttl=3D63 time=3D14.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D168 ttl=3D63 time=3D14.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D169 ttl=3D63 time=3D14.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D170 ttl=3D63 time=3D14.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D171 ttl=3D63 time=3D14.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D172 ttl=3D63 time=3D14.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D173 ttl=3D63 time=3D14.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D174 ttl=3D63 time=3D14.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D175 ttl=3D63 time=3D15.1 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D176 ttl=3D63 time=3D15.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D177 ttl=3D63 time=3D16.0 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D178 ttl=3D63 time=3D16.9 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D179 ttl=3D63 time=3D17.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D180 ttl=3D63 time=3D17.6 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D181 ttl=3D63 time=3D17.9 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D182 ttl=3D63 time=3D17.9 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D183 ttl=3D63 time=3D18.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D184 ttl=3D63 time=3D18.9 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D185 ttl=3D63 time=3D19.8 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D186 ttl=3D63 time=3D19.8 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D187 ttl=3D63 time=3D10.7 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D188 ttl=3D63 time=3D10.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D189 ttl=3D63 time=3D10.4 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D190 ttl=3D63 time=3D10.3 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D191 ttl=3D63 time=3D10.5 ms > > >>> 64 bytes from 192.168.0.101: icmp_seq=3D192 ttl=3D63 time=3D10.7 ms > > >>> As you mentioned, the delay has decreased a lot but it is still > > >>> considerably high (in a normal router this delay is less than 1 ms)= . > > >>> A second strange behaviour is seen in the evolution of the RTT > detected. > > >>> It begins in 10 ms and goes increasing little by litttle to reach a > peak of > > >>> 20 ms aprox and then it suddely comes back to 10 ms again to > increase again > > >>> till 20 ms. > > >>> > > >>> Is this the behaviour you have in your case when the burst_write is > set > > >>> to 1? > > >>> > > >>> Regards, > > >>> > > >>> El mar., 18 feb. 2020 a las 8:18, James Huang (= ) > > >>> escribi=C3=B3: > > >>> > > >>>> No. We didn't see noticable throughput difference in our test. > > >>>> > > >>>> On Mon., Feb. 17, 2020, 11:04 p.m. Victor Huertas < > vhuertas@gmail.com> > > >>>> wrote: > > >>>> > > >>>>> Thanks James for your quick answer. > > >>>>> I guess that this configuration modification implies that the > packets > > >>>>> must be written one by one in the sw ring. Did you notice loose o= f > > >>>>> performance (in throughput) in your aplicaci=C3=B3n because of th= at? > > >>>>> > > >>>>> Regards > > >>>>> > > >>>>> El mar., 18 feb. 2020 0:10, James Huang > escribi=C3=B3: > > >>>>> > > >>>>>> Yes, I experienced similar issue in my application. In a short > > >>>>>> answer, set the swqs write burst value to 1 may reduce the laten= cy > > >>>>>> significantly. The default write burst value is 32. > > >>>>>> > > >>>>>> On Mon., Feb. 17, 2020, 8:41 a.m. Victor Huertas < > vhuertas@gmail.com> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Hi all, > > >>>>>>> > > >>>>>>> I am developing my own DPDK application basing it in the > dpdk-stable > > >>>>>>> ip_pipeline example. > > >>>>>>> At this moment I am using the 17.11 LTS version of DPDK and I a= mb > > >>>>>>> observing > > >>>>>>> some extrange behaviour. Maybe it is an old issue that can be > solved > > >>>>>>> quickly so I would appreciate it if some expert can shade a > light on > > >>>>>>> this. > > >>>>>>> > > >>>>>>> The ip_pipeline example allows you to develop Pipelines that > perform > > >>>>>>> specific packet processing functions (ROUTING, FLOW_CLASSIFYING= , > > >>>>>>> etc...). > > >>>>>>> The thing is that I am extending some of this pipelines with my > own. > > >>>>>>> However I want to take advantage of the built-in ip_pipeline > > >>>>>>> capability of > > >>>>>>> arbitrarily assigning the logical core where the pipeline > (f_run() > > >>>>>>> function) must be executed so that i can adapt the packet > processing > > >>>>>>> power > > >>>>>>> to the amount of the number of cores available. > > >>>>>>> Taking this into account I have observed something strange. I > show > > >>>>>>> you this > > >>>>>>> simple example below. > > >>>>>>> > > >>>>>>> Case 1: > > >>>>>>> [PIPELINE 0 MASTER core =3D0] > > >>>>>>> [PIPELINE 1 core=3D1] --- SWQ1--->[PIPELINE 2 core=3D2] > -----SWQ2----> > > >>>>>>> [PIPELINE 3 core=3D3] > > >>>>>>> > > >>>>>>> Case 2: > > >>>>>>> [PIPELINE 0 MASTER core =3D0] > > >>>>>>> [PIPELINE 1 core=3D1] --- SWQ1--->[PIPELINE 2 core=3D1] > -----SWQ2----> > > >>>>>>> [PIPELINE 3 core=3D1] > > >>>>>>> > > >>>>>>> I send a ping between two hosts connected at both sides of the > > >>>>>>> pipeline > > >>>>>>> model which allows these pings to cross all the pipelines (from > 1 to > > >>>>>>> 3). > > >>>>>>> What I observe in Case 1 (each pipeline has its own thread in > > >>>>>>> different > > >>>>>>> core) is that the reported RTT is less than 1 ms, whereas in > Case 2 > > >>>>>>> (all > > >>>>>>> pipelines except MASTER are run in the same thread) is 20 ms. > > >>>>>>> Furthermore, > > >>>>>>> in Case 2, if I increase a lot (hundreds of Mbps) the packet ra= te > > >>>>>>> this RTT > > >>>>>>> decreases to 3 or 4 ms. > > >>>>>>> > > >>>>>>> Has somebody observed this behaviour in the past? Can it be > solved > > >>>>>>> somehow? > > >>>>>>> > > >>>>>>> Thanks a lot for your attention > > >>>>>>> -- > > >>>>>>> Victor > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>>> Victor > > >>>>>>> > > >>>>>> > > >>> > > >>> -- > > >>> Victor > > >>> > > >> > > > > > > -- > > > Victor > > > > > > > > > -- > > Victor > --=20 Victor