[dpdk-dev] Why IP_PIPELINE is faster than L2FWD

[dpdk-dev] Why IP_PIPELINE is faster than L2FWD

[Royce Niu]

Hi all,

I tested default L2FWD and IP_PIPELINE (pass-through). The throughput of
IP_PIPELINE is higher immensely.

There are only two virtual NICs in KVM. The experiment is just moving
packet from vNIC0  to vNIC1. I think the function is so simple. Why L2FWD
is much slower?

How can I improve L2FWD, to make L2FWD faster?

Thanks!

-- 
Regards,

Royce

Re: [dpdk-dev] Why IP_PIPELINE is faster than L2FWD

[Bruce Richardson]

On Thu, Dec 22, 2016 at 12:18:12AM +0800, Royce Niu wrote:
> Hi all,
> 
> I tested default L2FWD and IP_PIPELINE (pass-through). The throughput of
> IP_PIPELINE is higher immensely.
> 
> There are only two virtual NICs in KVM. The experiment is just moving
> packet from vNIC0  to vNIC1. I think the function is so simple. Why L2FWD
> is much slower?
> 
> How can I improve L2FWD, to make L2FWD faster?
> 
Is IP_PIPELINE in passthrough mode modifying the packets? L2FWD swaps
the mac addresses on each packet as it processes them, which can slow it
down. L2FWD is also more an example of how the APIs work than anything
else. For fastest possible port-to-port forwarding, testpmd should give
the highest performance.

/Bruce

Re: [dpdk-dev] Why IP_PIPELINE is faster than L2FWD

[Royce Niu]

But, actually, L3FWD of IP_PIPELINE is also faster than stock L2FWD, which
also modifies mac addr. How can explain this?

Actually, I want to know why IP_PIPELINE is much faster and I can learn
from IP_PIPELINE and make our own program.

But, the documentation of that is not detailed enough. if it is possible,
could you tell me where is the key to boost? Thanks!

On Thu, Dec 22, 2016 at 7:15 PM, Bruce Richardson <
bruce.richardson@intel.com> wrote:

> On Thu, Dec 22, 2016 at 12:18:12AM +0800, Royce Niu wrote:
> > Hi all,
> >
> > I tested default L2FWD and IP_PIPELINE (pass-through). The throughput of
> > IP_PIPELINE is higher immensely.
> >
> > There are only two virtual NICs in KVM. The experiment is just moving
> > packet from vNIC0  to vNIC1. I think the function is so simple. Why L2FWD
> > is much slower?
> >
> > How can I improve L2FWD, to make L2FWD faster?
> >
> Is IP_PIPELINE in passthrough mode modifying the packets? L2FWD swaps
> the mac addresses on each packet as it processes them, which can slow it
> down. L2FWD is also more an example of how the APIs work than anything
> else. For fastest possible port-to-port forwarding, testpmd should give
> the highest performance.
>
> /Bruce
>



-- 
Regards,

Royce

Re: [dpdk-dev] Why IP_PIPELINE is faster than L2FWD

[Bruce Richardson]

On Thu, Dec 22, 2016 at 08:48:50PM +0800, Royce Niu wrote:
> But, actually, L3FWD of IP_PIPELINE is also faster than stock L2FWD, which
> also modifies mac addr. How can explain this?
> 
> Actually, I want to know why IP_PIPELINE is much faster and I can learn
> from IP_PIPELINE and make our own program.
> 
> But, the documentation of that is not detailed enough. if it is possible,
> could you tell me where is the key to boost? Thanks!
>

Adding Cristian as IP Pipeline maintainer.

A lot of tuning work went into IP Pipeline and the table and port
libraries it uses, so I'm not sure that there is just one or two key
changes which give it such good performance. L2 forward just hasn't had
the same level of tuning and, while performing well, is also simplified
to make it understandable as an example. Contrast the code in l2fwd
against equivalent vector code in l3fwd-lpm* files e.g. l3fwd_lpm_sse.h.
The latter is very high performing, the former is more readable.

Regards,
/Bruce

> On Thu, Dec 22, 2016 at 7:15 PM, Bruce Richardson <
> bruce.richardson@intel.com> wrote:
> 
> > On Thu, Dec 22, 2016 at 12:18:12AM +0800, Royce Niu wrote:
> > > Hi all,
> > >
> > > I tested default L2FWD and IP_PIPELINE (pass-through). The throughput of
> > > IP_PIPELINE is higher immensely.
> > >
> > > There are only two virtual NICs in KVM. The experiment is just moving
> > > packet from vNIC0  to vNIC1. I think the function is so simple. Why L2FWD
> > > is much slower?
> > >
> > > How can I improve L2FWD, to make L2FWD faster?
> > >
> > Is IP_PIPELINE in passthrough mode modifying the packets? L2FWD swaps
> > the mac addresses on each packet as it processes them, which can slow it
> > down. L2FWD is also more an example of how the APIs work than anything
> > else. For fastest possible port-to-port forwarding, testpmd should give
> > the highest performance.
> >
> > /Bruce
> >
> 
> 
> 
> -- 
> Regards,
> 
> Royce

Re: [dpdk-dev] Why IP_PIPELINE is faster than L2FWD

[Royce Niu]

Dear Bruce,

Thanks for your kind explanation.

I will try to follow your suggestion and see the source code.

On Thu, Dec 22, 2016 at 9:25 PM, Bruce Richardson <
bruce.richardson@intel.com> wrote:

> On Thu, Dec 22, 2016 at 08:48:50PM +0800, Royce Niu wrote:
> > But, actually, L3FWD of IP_PIPELINE is also faster than stock L2FWD,
> which
> > also modifies mac addr. How can explain this?
> >
> > Actually, I want to know why IP_PIPELINE is much faster and I can learn
> > from IP_PIPELINE and make our own program.
> >
> > But, the documentation of that is not detailed enough. if it is possible,
> > could you tell me where is the key to boost? Thanks!
> >
>
> Adding Cristian as IP Pipeline maintainer.
>
> A lot of tuning work went into IP Pipeline and the table and port
> libraries it uses, so I'm not sure that there is just one or two key
> changes which give it such good performance. L2 forward just hasn't had
> the same level of tuning and, while performing well, is also simplified
> to make it understandable as an example. Contrast the code in l2fwd
> against equivalent vector code in l3fwd-lpm* files e.g. l3fwd_lpm_sse.h.
> The latter is very high performing, the former is more readable.
>
> Regards,
> /Bruce
>
> > On Thu, Dec 22, 2016 at 7:15 PM, Bruce Richardson <
> > bruce.richardson@intel.com> wrote:
> >
> > > On Thu, Dec 22, 2016 at 12:18:12AM +0800, Royce Niu wrote:
> > > > Hi all,
> > > >
> > > > I tested default L2FWD and IP_PIPELINE (pass-through). The
> throughput of
> > > > IP_PIPELINE is higher immensely.
> > > >
> > > > There are only two virtual NICs in KVM. The experiment is just moving
> > > > packet from vNIC0  to vNIC1. I think the function is so simple. Why
> L2FWD
> > > > is much slower?
> > > >
> > > > How can I improve L2FWD, to make L2FWD faster?
> > > >
> > > Is IP_PIPELINE in passthrough mode modifying the packets? L2FWD swaps
> > > the mac addresses on each packet as it processes them, which can slow
> it
> > > down. L2FWD is also more an example of how the APIs work than anything
> > > else. For fastest possible port-to-port forwarding, testpmd should give
> > > the highest performance.
> > >
> > > /Bruce
> > >
> >
> >
> >
> > --
> > Regards,
> >
> > Royce
>



-- 
Regards,

Royce

Re: [dpdk-dev] Why IP_PIPELINE is faster than L2FWD

[Xu, Qian Q]

As far as I know, L2FWD only uses 1 core for all RX/TX, for all queues, but for ip_pipeline, you may use more cores. 
A simple question, are you using 1core in ip_pipeline or l3fwd test? 

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Royce Niu
> Sent: Thursday, December 22, 2016 9:36 PM
> To: Richardson, Bruce <bruce.richardson@intel.com>
> Cc: Royce Niu <royceniu@gmail.com>; dev@dpdk.org; Dumitrescu, Cristian
> <cristian.dumitrescu@intel.com>
> Subject: Re: [dpdk-dev] Why IP_PIPELINE is faster than L2FWD
> 
> Dear Bruce,
> 
> Thanks for your kind explanation.
> 
> I will try to follow your suggestion and see the source code.
> 
> On Thu, Dec 22, 2016 at 9:25 PM, Bruce Richardson <
> bruce.richardson@intel.com> wrote:
> 
> > On Thu, Dec 22, 2016 at 08:48:50PM +0800, Royce Niu wrote:
> > > But, actually, L3FWD of IP_PIPELINE is also faster than stock L2FWD,
> > which
> > > also modifies mac addr. How can explain this?
> > >
> > > Actually, I want to know why IP_PIPELINE is much faster and I can
> > > learn from IP_PIPELINE and make our own program.
> > >
> > > But, the documentation of that is not detailed enough. if it is
> > > possible, could you tell me where is the key to boost? Thanks!
> > >
> >
> > Adding Cristian as IP Pipeline maintainer.
> >
> > A lot of tuning work went into IP Pipeline and the table and port
> > libraries it uses, so I'm not sure that there is just one or two key
> > changes which give it such good performance. L2 forward just hasn't
> > had the same level of tuning and, while performing well, is also
> > simplified to make it understandable as an example. Contrast the code
> > in l2fwd against equivalent vector code in l3fwd-lpm* files e.g.
> l3fwd_lpm_sse.h.
> > The latter is very high performing, the former is more readable.
> >
> > Regards,
> > /Bruce
> >
> > > On Thu, Dec 22, 2016 at 7:15 PM, Bruce Richardson <
> > > bruce.richardson@intel.com> wrote:
> > >
> > > > On Thu, Dec 22, 2016 at 12:18:12AM +0800, Royce Niu wrote:
> > > > > Hi all,
> > > > >
> > > > > I tested default L2FWD and IP_PIPELINE (pass-through). The
> > throughput of
> > > > > IP_PIPELINE is higher immensely.
> > > > >
> > > > > There are only two virtual NICs in KVM. The experiment is just
> > > > > moving packet from vNIC0  to vNIC1. I think the function is so
> > > > > simple. Why
> > L2FWD
> > > > > is much slower?
> > > > >
> > > > > How can I improve L2FWD, to make L2FWD faster?
> > > > >
> > > > Is IP_PIPELINE in passthrough mode modifying the packets? L2FWD
> > > > swaps the mac addresses on each packet as it processes them, which
> > > > can slow
> > it
> > > > down. L2FWD is also more an example of how the APIs work than
> > > > anything else. For fastest possible port-to-port forwarding,
> > > > testpmd should give the highest performance.
> > > >
> > > > /Bruce
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Royce
> >
> 
> 
> 
> --
> Regards,
> 
> Royce

Re: [dpdk-dev] Why IP_PIPELINE is faster than L2FWD

[Royce Niu]

Yes. One core is assigned.

On Fri, 23 Dec 2016 at 9:34 AM, Xu, Qian Q <qian.q.xu@intel.com> wrote:

> As far as I know, L2FWD only uses 1 core for all RX/TX, for all queues,
> but for ip_pipeline, you may use more cores.
>
> A simple question, are you using 1core in ip_pipeline or l3fwd test?
>
>
>
> > -----Original Message-----
>
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Royce Niu
>
> > Sent: Thursday, December 22, 2016 9:36 PM
>
> > To: Richardson, Bruce <bruce.richardson@intel.com>
>
> > Cc: Royce Niu <royceniu@gmail.com>; dev@dpdk.org; Dumitrescu, Cristian
>
> > <cristian.dumitrescu@intel.com>
>
> > Subject: Re: [dpdk-dev] Why IP_PIPELINE is faster than L2FWD
>
> >
>
> > Dear Bruce,
>
> >
>
> > Thanks for your kind explanation.
>
> >
>
> > I will try to follow your suggestion and see the source code.
>
> >
>
> > On Thu, Dec 22, 2016 at 9:25 PM, Bruce Richardson <
>
> > bruce.richardson@intel.com> wrote:
>
> >
>
> > > On Thu, Dec 22, 2016 at 08:48:50PM +0800, Royce Niu wrote:
>
> > > > But, actually, L3FWD of IP_PIPELINE is also faster than stock L2FWD,
>
> > > which
>
> > > > also modifies mac addr. How can explain this?
>
> > > >
>
> > > > Actually, I want to know why IP_PIPELINE is much faster and I can
>
> > > > learn from IP_PIPELINE and make our own program.
>
> > > >
>
> > > > But, the documentation of that is not detailed enough. if it is
>
> > > > possible, could you tell me where is the key to boost? Thanks!
>
> > > >
>
> > >
>
> > > Adding Cristian as IP Pipeline maintainer.
>
> > >
>
> > > A lot of tuning work went into IP Pipeline and the table and port
>
> > > libraries it uses, so I'm not sure that there is just one or two key
>
> > > changes which give it such good performance. L2 forward just hasn't
>
> > > had the same level of tuning and, while performing well, is also
>
> > > simplified to make it understandable as an example. Contrast the code
>
> > > in l2fwd against equivalent vector code in l3fwd-lpm* files e.g.
>
> > l3fwd_lpm_sse.h.
>
> > > The latter is very high performing, the former is more readable.
>
> > >
>
> > > Regards,
>
> > > /Bruce
>
> > >
>
> > > > On Thu, Dec 22, 2016 at 7:15 PM, Bruce Richardson <
>
> > > > bruce.richardson@intel.com> wrote:
>
> > > >
>
> > > > > On Thu, Dec 22, 2016 at 12:18:12AM +0800, Royce Niu wrote:
>
> > > > > > Hi all,
>
> > > > > >
>
> > > > > > I tested default L2FWD and IP_PIPELINE (pass-through). The
>
> > > throughput of
>
> > > > > > IP_PIPELINE is higher immensely.
>
> > > > > >
>
> > > > > > There are only two virtual NICs in KVM. The experiment is just
>
> > > > > > moving packet from vNIC0  to vNIC1. I think the function is so
>
> > > > > > simple. Why
>
> > > L2FWD
>
> > > > > > is much slower?
>
> > > > > >
>
> > > > > > How can I improve L2FWD, to make L2FWD faster?
>
> > > > > >
>
> > > > > Is IP_PIPELINE in passthrough mode modifying the packets? L2FWD
>
> > > > > swaps the mac addresses on each packet as it processes them, which
>
> > > > > can slow
>
> > > it
>
> > > > > down. L2FWD is also more an example of how the APIs work than
>
> > > > > anything else. For fastest possible port-to-port forwarding,
>
> > > > > testpmd should give the highest performance.
>
> > > > >
>
> > > > > /Bruce
>
> > > > >
>
> > > >
>
> > > >
>
> > > >
>
> > > > --
>
> > > > Regards,
>
> > > >
>
> > > > Royce
>
> > >
>
> >
>
> >
>
> >
>
> > --
>
> > Regards,
>
> >
>
> > Royce
>
>