[dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[Hanoch Haim (hhaim)]

Hi,

We (trex traffic generator project) upgrade DPDK version from 16.07 to 17.02arc-3 and we experienced a performance degradation on the following NICS:

XL710  : 10-15%
ixgbe   :  8% in one case
mlx5    : 8% 2 case
X710    :  no impact (same driver as XL710)
VIC       : no impact

It might be related to DPDK infra change as it affects more than one driver (maybe mbuf?).
Wanted to know if this is expected before investing more into this. The Y axis numbers in all the following charts (from Grafana) are MPPS/core which means how many cycles per packet the CPU invest. Bigger MPPS/core means less CPU cycles to send packet.

Labeles:

trex-08 (XL710)
trex-09 (X710)
trex-07 (mlx5)
kiwi02 (ixgbe)

Grafanna charts can be seen here: (we can't share pointers to Grafanna)
https://snag.gy/wGYp3c.jpg

More information on the tests and setups can be found in this document
https://trex-tgn.cisco.com/trex/doc/trex_analytics.html


link to the project
https://github.com/cisco-system-traffic-generator/trex-core

Thanks,
Hanoh

Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hanoch Haim (hhaim)
> Sent: Tuesday, February 14, 2017 11:45 AM
> To: dev@dpdk.org
> Cc: Ido Barnea (ibarnea) <ibarnea@cisco.com>; Hanoch Haim (hhaim)
> <hhaim@cisco.com>
> Subject: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%
> 
> Hi,
> 
> We (trex traffic generator project) upgrade DPDK version from 16.07 to
> 17.02arc-3 and we experienced a performance degradation on the following
> NICS:
> 
> XL710  : 10-15%
> ixgbe   :  8% in one case
> mlx5    : 8% 2 case
> X710    :  no impact (same driver as XL710)
> VIC       : no impact
> 
> It might be related to DPDK infra change as it affects more than one
> driver (maybe mbuf?).
> Wanted to know if this is expected before investing more into this. The Y
> axis numbers in all the following charts (from Grafana) are MPPS/core
> which means how many cycles per packet the CPU invest. Bigger MPPS/core
> means less CPU cycles to send packet.

Hi,

Thanks for the update. From a quick check with the Intel test team they haven't seen this. They would have flagged it if they had. Maybe someone from Mellanox/Netronome could confirm as well.

Could you do a git-bisect to identify the change that caused this?

John

Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[Hanoch Haim (hhaim)]

Hi John, thank you for the fast response. 
I assume that Intel tests are more like rx->tx tests. 
In our case we are doing mostly tx, which is more similar to dpdk-pkt-gen 
The cases that we cached the mbuf was affected the most. 
We expect to see the same issue with a simple DPDK application

Thanks,
Hanoh

-----Original Message-----
From: Mcnamara, John [mailto:john.mcnamara@intel.com] 
Sent: Tuesday, February 14, 2017 2:19 PM
To: Hanoch Haim (hhaim); dev@dpdk.org
Cc: Ido Barnea (ibarnea)
Subject: RE: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hanoch Haim 
> (hhaim)
> Sent: Tuesday, February 14, 2017 11:45 AM
> To: dev@dpdk.org
> Cc: Ido Barnea (ibarnea) <ibarnea@cisco.com>; Hanoch Haim (hhaim) 
> <hhaim@cisco.com>
> Subject: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%
> 
> Hi,
> 
> We (trex traffic generator project) upgrade DPDK version from 16.07 to
> 17.02arc-3 and we experienced a performance degradation on the 
> following
> NICS:
> 
> XL710  : 10-15%
> ixgbe   :  8% in one case
> mlx5    : 8% 2 case
> X710    :  no impact (same driver as XL710)
> VIC       : no impact
> 
> It might be related to DPDK infra change as it affects more than one 
> driver (maybe mbuf?).
> Wanted to know if this is expected before investing more into this. 
> The Y axis numbers in all the following charts (from Grafana) are 
> MPPS/core which means how many cycles per packet the CPU invest. 
> Bigger MPPS/core means less CPU cycles to send packet.

Hi,

Thanks for the update. From a quick check with the Intel test team they haven't seen this. They would have flagged it if they had. Maybe someone from Mellanox/Netronome could confirm as well.

Could you do a git-bisect to identify the change that caused this?

John

Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[Thomas Monjalon]

2017-02-14 12:19, Mcnamara, John:
> From: Hanoch Haim (hhaim)
> > We (trex traffic generator project) upgrade DPDK version from 16.07 to
> > 17.02arc-3 and we experienced a performance degradation on the following
> > NICS:
[...]
> 
> Could you do a git-bisect to identify the change that caused this?

+1 for the bisect.
Please could you try to isolate the change causing the regression?

Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[De Lara Guarch, Pablo]

Hi,

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas Monjalon
> Sent: Tuesday, February 14, 2017 1:05 PM
> To: Hanoch Haim (hhaim)
> Cc: dev@dpdk.org; Mcnamara, John; Ido Barnea (ibarnea)
> Subject: Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%
> 
> 2017-02-14 12:19, Mcnamara, John:
> > From: Hanoch Haim (hhaim)
> > > We (trex traffic generator project) upgrade DPDK version from 16.07 to
> > > 17.02arc-3 and we experienced a performance degradation on the
> following
> > > NICS:
> [...]
> >
> > Could you do a git-bisect to identify the change that caused this?
> 
> +1 for the bisect.
> Please could you try to isolate the change causing the regression?

I reproduced the regression for i40e driver (3-4% degradation, not 10%). The commit that caused it was the following:
5ea0942129a4 ("net/i40e: add packet type metadata in vector Rx")

This commit was introduced in 16.11, and not in 17.02. This caused some performance degradation
in the i40e vector RX, due to the addition of packet type parsing in the PMD.

Thanks,
Pablo

Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[De Lara Guarch, Pablo]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of De Lara Guarch,
> Pablo
> Sent: Tuesday, February 14, 2017 1:28 PM
> To: Thomas Monjalon; Hanoch Haim (hhaim)
> Cc: dev@dpdk.org; Mcnamara, John; Ido Barnea (ibarnea)
> Subject: Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%
> 
> Hi,
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas
> Monjalon
> > Sent: Tuesday, February 14, 2017 1:05 PM
> > To: Hanoch Haim (hhaim)
> > Cc: dev@dpdk.org; Mcnamara, John; Ido Barnea (ibarnea)
> > Subject: Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of
> ~10%
> >
> > 2017-02-14 12:19, Mcnamara, John:
> > > From: Hanoch Haim (hhaim)
> > > > We (trex traffic generator project) upgrade DPDK version from 16.07
> to
> > > > 17.02arc-3 and we experienced a performance degradation on the
> > following
> > > > NICS:
> > [...]
> > >
> > > Could you do a git-bisect to identify the change that caused this?
> >
> > +1 for the bisect.
> > Please could you try to isolate the change causing the regression?
> 
> I reproduced the regression for i40e driver (3-4% degradation, not 10%).
> The commit that caused it was the following:
> 5ea0942129a4 ("net/i40e: add packet type metadata in vector Rx")
> 
> This commit was introduced in 16.11, and not in 17.02. This caused some
> performance degradation
> in the i40e vector RX, due to the addition of packet type parsing in the
> PMD.
> 
> Thanks,
> Pablo

Sorry, I was talking about RX and not TX, so this is probably not valid then.

Pablo

Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[Hanoch Haim (hhaim)]

Hi Thomas and John,

We found that the performance degradation is not related to DPDK 17.02 for Intel XL710/ixgbe.  Now It is back to normal. The fix was to add a one unrelated line to a specific file. This probably changed the instructions trace. This line is not related to DP or CP. (We still don't fully understand that).
Mlx5 small degradation wasn't solved yet. 

Thanks,
Hanoh

-----Original Message-----
From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] 
Sent: Tuesday, February 14, 2017 3:05 PM
To: Hanoch Haim (hhaim)
Cc: dev@dpdk.org; Mcnamara, John; Ido Barnea (ibarnea)
Subject: Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

2017-02-14 12:19, Mcnamara, John:
> From: Hanoch Haim (hhaim)
> > We (trex traffic generator project) upgrade DPDK version from 16.07 to
> > 17.02arc-3 and we experienced a performance degradation on the following
> > NICS:
[...]
> 
> Could you do a git-bisect to identify the change that caused this?

+1 for the bisect.
Please could you try to isolate the change causing the regression?

Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[Gudimetla, Leela Sankar]

Hi,

"Existing thread subject: DPDK 17.02 RC-3 performance degradation of ~10%"

We are in the process of upgrading from dpdk 16.11 to 17.02. We are also seeing performance degradation in our packet processing application for about ~3% with 17.02.
When we browsed through mailing lists, we found a similar thread as mentioned in this email subject from Cisco T-Rex. But the thread did not point to any resolution instead they could fix it in their software.
Before starting to figure out which commit of dpdk (git bisect) is causing our application to degrade (which is tedious and slow), we are wondering that if the resolution for the T-Rex is shared in broad sense like which dpdk api etc.
That would really help.

Thanks & Regards,
Leela sankar Gudimetla

Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

[Gudimetla, Leela Sankar]

Hi,

I am resending it again since I did not see any activity with below original email sent out.
Sorry for the spam if it is a duplicate.


Thanks & Regards,
Leela sankar Gudimetla

From: Gudimetla, Leela Sankar
Sent: Wednesday, June 21, 2017 2:24 PM
To: 'dev@dpdk.org'
Cc: 'hhaim@cisco.com'
Subject: Re: [dpdk-dev] DPDK 17.02 RC-3 performance degradation of ~10%

Hi,

"Existing thread subject: DPDK 17.02 RC-3 performance degradation of ~10%"

We are in the process of upgrading from dpdk 16.11 to 17.02. We are also seeing performance degradation in our packet processing application for about ~3% with 17.02.
When we browsed through mailing lists, we found a similar thread as mentioned in this email subject from Cisco T-Rex. But the thread did not point to any resolution instead they could fix it in their software.
Before starting to figure out which commit of dpdk (git bisect) is causing our application to degrade (which is tedious and slow), we are wondering that if the resolution for the T-Rex is shared in broad sense like which dpdk api etc.
That would really help.

Thanks & Regards,
Leela sankar Gudimetla