From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <3chas3@gmail.com> Received: from mail-it0-f52.google.com (mail-it0-f52.google.com [209.85.214.52]) by dpdk.org (Postfix) with ESMTP id 0272C231E; Wed, 29 Aug 2018 16:30:46 +0200 (CEST) Received: by mail-it0-f52.google.com with SMTP id x79-v6so5241307ita.1; Wed, 29 Aug 2018 07:30:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:sender:from:date:message-id :subject:to:cc; bh=cwoQGmAS0HqAl1WnPffoxANx++DkR/fZN/ly5u2wS+c=; b=ftYtJqYlbn+5WMy1/ITrElC+lQq9n5SgJdo/5DnDGiMTuiK+IJqfelTA5n1sNKt7Lq A/8pszQw8Q0Vde9yu+0yPeOgv1ZV/n/xkHxhcc3IQXVwO6gqS9I7qIeXNf1B+TVcjx2f OOGjOgHCzSE9VAC2nvnxvzTj4kfaAn1xEeEln9DnrUOIzgjZTJPOPRnZlwJkT1C2W2+D uAdfuLJK/0NzST/zH3U/Y+1cwwEAVAR6STW9mxq3RaXyk2I/FyA2u/yUSthoZ5bBgFdu YispigSxwM6TBuhoN/KuqcjRreRa9q6AI4NpWZLyS5E7DbC5LrbSGu8//VEgMtvnnuhP ZSNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:sender:from :date:message-id:subject:to:cc; bh=cwoQGmAS0HqAl1WnPffoxANx++DkR/fZN/ly5u2wS+c=; b=cm0FBK0BkSOXbmTcKFca2QHDNp5Pc3RvF2ox3vDBdTekPnL09khdJ5ia2IUg3aGzMT utj6pDd6Vz+o5xxOJy8vTnP+a2+jwJ97nSbBKLcKWlwagwCEuUb1d6o1MDSJqewuA2G+ fBYhLEq5RhdJ0xk2vAZi+WamkEnVzoIkkI/sOmMYuc9hJ9lfyZRQqUonut057KKsJneI qM0jvdu1Jbq7S2S7ZYV8bf8blX/dkhUkPyreRzBtu02Sldu36Fmvi5oAzTmy4wx571MI ghrEoDU/4fWinwIeThEuH4Z/okP7f1IgarW2vVHSgTTeIzDuT6DpPOYLtFh7m9CFLymh QkCw== X-Gm-Message-State: APzg51BSJUUO0jyeAx8UbNhnmgiEm4f8mMIDIRwGVU75h97UWir2ZRry kT0PjGqdNW8Znk2nIjugzzJu1Lvvq5TZYhh69XE= X-Google-Smtp-Source: ANB0VdYWyzGIEJU9M1NkBRnq+YLPWx51x/OMRt097O2NPm5dIxwGQSNCGbRvO69xi15uR+UEhQFwBQx+rSU9n4Ki1u4= X-Received: by 2002:a24:6b0d:: with SMTP id v13-v6mr5794737itc.16.1535553046317; Wed, 29 Aug 2018 07:30:46 -0700 (PDT) MIME-Version: 1.0 References: <20180816125202.15980-1-bluca@debian.org> <20180816133208.26566-1-bluca@debian.org> <1534933159.5764.107.camel@debian.org> <20180822174316.GA29821@roosta> In-Reply-To: Sender: chasmosaurus@gmail.com X-Google-Sender-Delegation: chasmosaurus@gmail.com From: Chas Williams <3chas3@gmail.com> Date: Wed, 29 Aug 2018 10:30:34 -0400 X-Google-Sender-Auth: AjKbKy5_FIJzTXTVvvXJjEFVyOE Message-ID: To: Matan Azrad Cc: Eric Kinzie , bluca@debian.org, dev@dpdk.org, Declan Doherty , Chas Williams , stable@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH v4] net/bonding: per-slave intermediate rx ring X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 14:30:47 -0000 On Tue, Aug 28, 2018 at 5:51 AM Matan Azrad wrote: > > > From: Chas Williams > >On Mon, Aug 27, 2018 at 11:30 AM Matan Azrad > wrote: > > >>>Because rings are generally quite efficient. > >> > >>But you are using a ring in addition to regular array management, it > must hurt performance of the bonding PMD > >>(means the bonding itself - not the slaves PMDs which are called from > the bonding) > > > >It adds latency. > > And by that hurts the application performance because it takes more CPU > time in the bonding PMD. > No, as I said before it takes _less_ CPU time in the bonding PMD because we use a more optimal read from the slaves. > > >It increases performance because we spend less CPU time reading from the > PMDs. > > So, it's hack in the bonding PMD to improve some slaves code performance > but hurt the bonding code performance, > Over all the performance we gain for those slaves improves the application > performance only when working with those slaves. > But may hurt the application performance when working with other slaves. > What is your evidence that is hurts bonding performance? Your argument is purely theoretical. I could easily argue than even for non-vectorized PMDs there is a performance gain because we spend less time switching between PMDs. If you are going to read from a PMD you should attempt to read as much as possible. It's expensive to read the cards registers and perform the queue manipulations. > > > This means we have more CPU to use for > >post processing (i.e. routing). > > > > >>>Bonding is in a middle ground between application and PMD. > >>Yes. > >>>What bonding is doing, may not improve all applications. > >>Yes, but it can be solved using some bonding modes. > >>> If using a ring to buffer the vectorized receive routines, improves > your particular application, > >>>that's great. > >>It may be not great and even bad for some other PMDs which are not > vectororized. > >> > >>> However, I don't think I can say that it would help all > >>>applications. As you point out, there is overhead associated with > >>>a ring. > >>Yes. > >>>Bonding's receive burst isn't especially efficient (in mode 4). > >> > >>Why? > >> > >>It makes a copy of the slaves, has a fair bit of stack usage, > >>needs to check the slave status, and needs to examine each > >>packet to see if it is a slow protocol packet. So each > >>packet is essentially read twice. The fast queue code for mode 4 > >>avoids some of this (and probably ignores checking collecting > >>incorrectly). If you find a slow protocol packet, you need to > >>chop it out of the array with memmove due to overlap. > > > >Agree. > >So to improve the bonding performance you need to optimize the aboves > problems. > >There is no connection to the ring. > > > >And as I have described numerous times, these problems > >can't be easily fixed and preserve the existing API. > > Sometimes we need to work harder to see a gain for all. > We should not apply a patch because it is easy and show a gain for > specific scenarios. > > >>> Bonding benefits from being able to read as much as possible (within > limits of > >>>course, large reads would blow out caches) from each slave. > >> > >>The slaves PMDs can benefits in the same way. > >> > >>>It can't return all that data though because applications tend to use > the > >>>burst size that would be efficient for a typical PMD. > >> > >>What is the preferred burst size of the bonding? Maybe the application > should use it when they are using bonding. > >> > >>The preferred burst size for bonding would be the sum of all the > >>slaves ideal read size. However, that's not likely to be simple > >>since most applications decide early the size for the read/write > >>burst operations. > >> > >>>An alternative might be to ask bonding applications to simply issue > larger reads for > >>>certain modes. That's probably not as easy as it sounds given the > >>>way that the burst length effects multiplexing. > >> > >>Can you explain it more? > >> > >>A longer burst size on one PMD will tend to favor that PMD > >>over others. It will fill your internal queues with more > >>of its packets. > > > >Agree, it's about fairness. > > > >> > >>>Another solution might be just alternatively poll the individual > >>>slaves on each rx burst. But that means you need to poll at a > >>>faster rate. Depending on your application, you might not be > >>>able to do that. > > > >>Again, can you be more precise in the above explanation? > >> > >>If the application knows that there are two slaves backing > >>a bonding interface, the application could just read twice > >>from the bonding interface, knowing that the bonding > >>interface is going to alternate between the slaves. But > >>this requires the application to know things about the bonding > >>PMD, like the number of slaves. > > > >Why should the application poll twice? > >Poll slave 0, than process it's packets, poll slave 1 than process its > packets... > >What is the problem? > > > >Because let's say that each slave is 10G and you are using > > > >link aggregation with two slaves. If you have tuned your > > > >application on the assumption that a PMD is approximately > >10G, then you are going to be under polling the bonding PMD. > >For the above to work, you need to ensure that the application > >is polling sufficiently to keep up. > > But each poll will be shorter, no slaves loops, only one slave call. > But you still need to poll the bonding PMD N times as fast where N is the number of slaves. The "scheduler" in the application may not be aware of that. That was (apparently) the point of the way the current bonding PMD. It hides everything from the application. > > >>> We can avoid this scheduling overhead by just > >>>doing the extra reads in bonding and buffering in a ring. > >>> > >>>Since bonding is going to be buffering everything in a ring, > >> > >>? I don't familiar with it. For now I don't think we need a ring. > >> > >>We have some empirical testing that shows performance improvements. > >>How do you explain this? > > > >You are using a hack in bonding which hurt the bonding but improves the > vectorized PMD you use. > > > >>Rings aren't inefficient. > > > >Only when there is a need to use it. > > > >I believe we have identified a need. > > > > > >>There's significant overhead of calling the rx burst routine on any PMD. > >>You need to get the PMD data structures into local memory. > > > >Yes. > > > >>Reading as much as possible makes sense. > > > >Agree. > > > >> Could you generally do this for all PMDs? Yes, but the ring adds > latency. Latency > >>that isn't an issue for the bonding PMD because of all the > >>other inefficiencies (for mode 4). > > > >Enlarging the bonding latency by that way makes some vectorized slave > PMDs happy and makes the bonding worst > >for other slaves PMDs - this is not a good idea to put it upstream. > > > >Improving the other problems in the bonding(reduce the bonding latency) > will do the job for you and for others. > > > >Latency is not the issue with bonding. The issue with bonding > >is the overhead associated with making an rx burst call. We > >add latency (via rings) to make part of the bonding driver more > >efficient. > > > >Again, I suspect it even helps the non-vectorized PMDs. > >Calling a PMD's rx burst routine is costly and we are switching between > >PMD's inside the bonding driver. Bonding is halfway between an > >application and a PMD. What we are doing in the bonding PMD is > >what an application would typically do. But instead of forcing > >all the bonding users to do this, we are doing it for them. > > Not agree. > The overhead in bonding may be critical for some other PMDs\scenario. > The overhead is bonding is undesirable. It's not easy to address because of the initial design goals. > > To summarize: > We are not agree. > > Bottom line in my side, > It is not a good Idea to add overhead in the bonding PMD which is a > generic PMD just to get gain for some specific scenarios in some specific > PMDs while for other scenarios\PMDs it is bad. > You are making some assumptions which are simply not valid. BOnding is _not_ a generic PMD. As discussed above, bonding is somewhere between application and PMD. The choices made for bonding where to make it easy to integrate into an existing application with the application having to know anything about bonding. > Matan. > > > > >