[dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[Martin Weiser]

Hi,

with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES
ports without reason. Even directly after starting test-pmd the error
counter immediately is 1 without even a single packet being sent to the
device:

./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
...
testpmd> show port stats all

  ######################## NIC statistics for port 0  ########################
  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1
  RX-nombuf:  0
  TX-packets: 0          TX-errors: 0          TX-bytes:  0
  ############################################################################

  ######################## NIC statistics for port 1  ########################
  RX-packets: 0          RX-missed: 0          RX-bytes:  0
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 1
  RX-nombuf:  0
  TX-packets: 0          TX-errors: 0          TX-bytes:  0
  ############################################################################


When packet forwarding is started the ports perform normally and
properly forward all packets but a huge number of ierrors is counted:

testpmd> start
...
testpmd> show port stats all

  ######################## NIC statistics for port 0  ########################
  RX-packets: 9011857    RX-missed: 0          RX-bytes:  5020932992
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9011753
  RX-nombuf:  0
  TX-packets: 9026250    TX-errors: 0          TX-bytes:  2922375542
  ############################################################################

  ######################## NIC statistics for port 1  ########################
  RX-packets: 9026250    RX-missed: 0          RX-bytes:  2922375542
  RX-badcrc:  0          RX-badlen: 0          RX-errors: 9026138
  RX-nombuf:  0
  TX-packets: 9011857    TX-errors: 0          TX-bytes:  5020932992
  ############################################################################


When running the exact same test with DPDK version 2.0 no ierrors are
reported.
Is anyone else seeing strange ierrors being reported for Intel Niantic
cards with DPDK 2.1?

Best regards,
Martin

Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[Andriy Berestovskyy]

Yes Marcin,
The issue was discussed here:
http://dpdk.org/ml/archives/dev/2015-September/023229.html

You can either fix the ierrors in ixgbe_dev_stats_get() or implement a
workaround in your app getting the extended statistics and counting
out some of extended counters from the ierrors.

Here is an example:
https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306

Regards,
Andriy

On Wed, Oct 21, 2015 at 10:38 AM, Martin Weiser
<martin.weiser@allegro-packets.com> wrote:
> Hi,
>
> with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES
> ports without reason. Even directly after starting test-pmd the error
> counter immediately is 1 without even a single packet being sent to the
> device:
>
> ./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
> ...
> testpmd> show port stats all
>
>   ######################## NIC statistics for port 0  ########################
>   RX-packets: 0          RX-missed: 0          RX-bytes:  0
>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 1
>   RX-nombuf:  0
>   TX-packets: 0          TX-errors: 0          TX-bytes:  0
>   ############################################################################
>
>   ######################## NIC statistics for port 1  ########################
>   RX-packets: 0          RX-missed: 0          RX-bytes:  0
>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 1
>   RX-nombuf:  0
>   TX-packets: 0          TX-errors: 0          TX-bytes:  0
>   ############################################################################
>
>
> When packet forwarding is started the ports perform normally and
> properly forward all packets but a huge number of ierrors is counted:
>
> testpmd> start
> ...
> testpmd> show port stats all
>
>   ######################## NIC statistics for port 0  ########################
>   RX-packets: 9011857    RX-missed: 0          RX-bytes:  5020932992
>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 9011753
>   RX-nombuf:  0
>   TX-packets: 9026250    TX-errors: 0          TX-bytes:  2922375542
>   ############################################################################
>
>   ######################## NIC statistics for port 1  ########################
>   RX-packets: 9026250    RX-missed: 0          RX-bytes:  2922375542
>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 9026138
>   RX-nombuf:  0
>   TX-packets: 9011857    TX-errors: 0          TX-bytes:  5020932992
>   ############################################################################
>
>
> When running the exact same test with DPDK version 2.0 no ierrors are
> reported.
> Is anyone else seeing strange ierrors being reported for Intel Niantic
> cards with DPDK 2.1?
>
> Best regards,
> Martin
>



-- 
Andriy Berestovskyy

Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[Martin Weiser]

Hi Andriy,

thank you for pointing this discussion out to me. I somehow missed it.
Unfortunately it looks like the discussion stopped after Maryam made a
good proposal so I will vote in on that and hopefully get things started
again.

Best regards,
Martin



On 21.10.15 17:53, Andriy Berestovskyy wrote:
> Yes Marcin,
> The issue was discussed here:
> http://dpdk.org/ml/archives/dev/2015-September/023229.html
>
> You can either fix the ierrors in ixgbe_dev_stats_get() or implement a
> workaround in your app getting the extended statistics and counting
> out some of extended counters from the ierrors.
>
> Here is an example:
> https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306
>
> Regards,
> Andriy
>
> On Wed, Oct 21, 2015 at 10:38 AM, Martin Weiser
> <martin.weiser@allegro-packets.com> wrote:
>> Hi,
>>
>> with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES
>> ports without reason. Even directly after starting test-pmd the error
>> counter immediately is 1 without even a single packet being sent to the
>> device:
>>
>> ./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
>> ...
>> testpmd> show port stats all
>>
>>   ######################## NIC statistics for port 0  ########################
>>   RX-packets: 0          RX-missed: 0          RX-bytes:  0
>>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 1
>>   RX-nombuf:  0
>>   TX-packets: 0          TX-errors: 0          TX-bytes:  0
>>   ############################################################################
>>
>>   ######################## NIC statistics for port 1  ########################
>>   RX-packets: 0          RX-missed: 0          RX-bytes:  0
>>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 1
>>   RX-nombuf:  0
>>   TX-packets: 0          TX-errors: 0          TX-bytes:  0
>>   ############################################################################
>>
>>
>> When packet forwarding is started the ports perform normally and
>> properly forward all packets but a huge number of ierrors is counted:
>>
>> testpmd> start
>> ...
>> testpmd> show port stats all
>>
>>   ######################## NIC statistics for port 0  ########################
>>   RX-packets: 9011857    RX-missed: 0          RX-bytes:  5020932992
>>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 9011753
>>   RX-nombuf:  0
>>   TX-packets: 9026250    TX-errors: 0          TX-bytes:  2922375542
>>   ############################################################################
>>
>>   ######################## NIC statistics for port 1  ########################
>>   RX-packets: 9026250    RX-missed: 0          RX-bytes:  2922375542
>>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 9026138
>>   RX-nombuf:  0
>>   TX-packets: 9011857    TX-errors: 0          TX-bytes:  5020932992
>>   ############################################################################
>>
>>
>> When running the exact same test with DPDK version 2.0 no ierrors are
>> reported.
>> Is anyone else seeing strange ierrors being reported for Intel Niantic
>> cards with DPDK 2.1?
>>
>> Best regards,
>> Martin
>>
>
>

Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[Andriy Berestovskyy]

Hi Martin,
We agreed on the main point: it's an issue. IMO the implementation
details are up to Maryam.

There have been few patches, so I guess it will be fixed in 2.2.

Andriy


On Thu, Oct 22, 2015 at 9:46 AM, Martin Weiser
<martin.weiser@allegro-packets.com> wrote:
> Hi Andriy,
>
> thank you for pointing this discussion out to me. I somehow missed it.
> Unfortunately it looks like the discussion stopped after Maryam made a
> good proposal so I will vote in on that and hopefully get things started
> again.
>
> Best regards,
> Martin
>
>
>
> On 21.10.15 17:53, Andriy Berestovskyy wrote:
>> Yes Marcin,
>> The issue was discussed here:
>> http://dpdk.org/ml/archives/dev/2015-September/023229.html
>>
>> You can either fix the ierrors in ixgbe_dev_stats_get() or implement a
>> workaround in your app getting the extended statistics and counting
>> out some of extended counters from the ierrors.
>>
>> Here is an example:
>> https://github.com/Juniper/contrail-vrouter/commit/72f6ca05ac81d0ca5e7eb93c6ffe7a93648c2b00#diff-99c1f65a00658c7d38b3d1b64cb5fd93R1306
>>
>> Regards,
>> Andriy
>>
>> On Wed, Oct 21, 2015 at 10:38 AM, Martin Weiser
>> <martin.weiser@allegro-packets.com> wrote:
>>> Hi,
>>>
>>> with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES
>>> ports without reason. Even directly after starting test-pmd the error
>>> counter immediately is 1 without even a single packet being sent to the
>>> device:
>>>
>>> ./testpmd -c 0xfe -n 4 -- --portmask 0x3 --interactive
>>> ...
>>> testpmd> show port stats all
>>>
>>>   ######################## NIC statistics for port 0  ########################
>>>   RX-packets: 0          RX-missed: 0          RX-bytes:  0
>>>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 1
>>>   RX-nombuf:  0
>>>   TX-packets: 0          TX-errors: 0          TX-bytes:  0
>>>   ############################################################################
>>>
>>>   ######################## NIC statistics for port 1  ########################
>>>   RX-packets: 0          RX-missed: 0          RX-bytes:  0
>>>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 1
>>>   RX-nombuf:  0
>>>   TX-packets: 0          TX-errors: 0          TX-bytes:  0
>>>   ############################################################################
>>>
>>>
>>> When packet forwarding is started the ports perform normally and
>>> properly forward all packets but a huge number of ierrors is counted:
>>>
>>> testpmd> start
>>> ...
>>> testpmd> show port stats all
>>>
>>>   ######################## NIC statistics for port 0  ########################
>>>   RX-packets: 9011857    RX-missed: 0          RX-bytes:  5020932992
>>>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 9011753
>>>   RX-nombuf:  0
>>>   TX-packets: 9026250    TX-errors: 0          TX-bytes:  2922375542
>>>   ############################################################################
>>>
>>>   ######################## NIC statistics for port 1  ########################
>>>   RX-packets: 9026250    RX-missed: 0          RX-bytes:  2922375542
>>>   RX-badcrc:  0          RX-badlen: 0          RX-errors: 9026138
>>>   RX-nombuf:  0
>>>   TX-packets: 9011857    TX-errors: 0          TX-bytes:  5020932992
>>>   ############################################################################
>>>
>>>
>>> When running the exact same test with DPDK version 2.0 no ierrors are
>>> reported.
>>> Is anyone else seeing strange ierrors being reported for Intel Niantic
>>> cards with DPDK 2.1?
>>>
>>> Best regards,
>>> Martin
>>>
>>
>>
>
>



-- 
Andriy Berestovskyy

Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[Van Haaren, Harry]

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Martin Weiser
> Sent: Wednesday, October 21, 2015 9:38 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

> with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES
> ports without reason. Even directly after starting test-pmd the error
> counter immediately is 1 without even a single packet being sent to the
> device

Hi Martin,

To dig up an old thread - I've recently experienced similar issues that the rx error count increases on 82599, so I dug into the sources and realized that the MAC fault counters were being added to the rx error count.

The following patch removes the MAC local and remote fault counts from rx errors, and this has fixed the issue here:
http://dpdk.org/dev/patchwork/patch/8525/

If this issue is still open perhaps try the patch and see if it improves the situation.

On a related note, I'm currently implementing extended statistics for ixgbe, which showed me that the MAC faults were the root cause of the rx increase in my case.

-Harry

Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[Martin Weiser]

Hi Harry,

first of all thank you for your effort.
I have done some test with the current master (which includes you
patch). It improves the situation but does not fix it entirely. The
rx-error which showed up immediately after starting the interface is
gone since this was probably caused by mac_remote_errors. But we still
see a huge number of rx-errors although all packets are received
properly and when looking at the extended stats those come from the
rx_l3_l4_xsum_error counter.
In our setup we are dealing with lots of UDP traffic which does have the
UDP checksum set to 0 (which to my knowledge is allowed for UDP). This
traffic seems to cause those rx_l3_l4_xsum_errors. When doing the same
test with other NICs (e.g. XL710) no rx-errors are accounted.
For the generic stats interface I would prefer only packets that could
not be received to be accounted in the rx-error counter regardless of
the actual NIC. What do you think?

Regards,
Martin


On 02.11.15 18:32, Van Haaren, Harry wrote:
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Martin Weiser
>> Sent: Wednesday, October 21, 2015 9:38 AM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1
>> with DPDK 2.1 we are seeing the ierrors counter increasing for 82599ES
>> ports without reason. Even directly after starting test-pmd the error
>> counter immediately is 1 without even a single packet being sent to the
>> device
> Hi Martin,
>
> To dig up an old thread - I've recently experienced similar issues that the rx error count increases on 82599, so I dug into the sources and realized that the MAC fault counters were being added to the rx error count.
>
> The following patch removes the MAC local and remote fault counts from rx errors, and this has fixed the issue here:
> http://dpdk.org/dev/patchwork/patch/8525/
>
> If this issue is still open perhaps try the patch and see if it improves the situation.
>
> On a related note, I'm currently implementing extended statistics for ixgbe, which showed me that the MAC faults were the root cause of the rx increase in my case.
>
> -Harry

[dpdk-dev] [PATCH] ixgbe: remove l3_l4_xsum_errors from rx errors

[Harry van Haaren]

This patch removes l3_l4_xsum_errors from rx errors.

The reason to remove it is that UDP packets have an optional checksum, and
when not calculated the checksum field should be set to zero. When the
checksum is not calculated (zero-ed out), the hardware still counts a valid
UDP packet as an l3_l4_xsum_error.

This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"

The solution is to remove l3_l4_xsum_errors from rx_errors, as discussed on
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/

Suggested-by: Martin Weiser <martin.weiser@allegro-packets.com>
Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 25966ef..9ed5b49 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -2480,7 +2480,6 @@ ixgbe_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 	                  total_missed_rx +
 	                  hw_stats->illerrc +
 	                  hw_stats->errbc +
-	                  hw_stats->xec +
 	                  hw_stats->rfc +
 	                  hw_stats->fccrc +
 	                  hw_stats->fclast;
-- 
1.9.1

Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[Van Haaren, Harry]

> From: Martin Weiser [mailto:martin.weiser@allegro-packets.com]
> Subject: Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

> The
> rx-error which showed up immediately after starting the interface is
> gone since this was probably caused by mac_remote_errors. 

Improvement - that's good.

> But we still
> see a huge number of rx-errors although all packets are received
> properly and when looking at the extended stats those come from the
> rx_l3_l4_xsum_error counter.

That is useful information, good to know that statistic is the root cause.

> In our setup we are dealing with lots of UDP traffic which does have the
> UDP checksum set to 0 (which to my knowledge is allowed for UDP).

Yes indeed checksum calculation for UDP is optional, and should be set to zero when not performed.

> This
> traffic seems to cause those rx_l3_l4_xsum_errors.
> When doing the same
> test with other NICs (e.g. XL710) no rx-errors are accounted.

So this is a ixgbe bug, and listed in the errata, item 43 "Integrity Error Reported for IPv4/UDP Packets with Zero Checksum" in http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/82599-10-gbe-controller-spec-update.pdf 

> For the generic stats interface I would prefer only packets that could
> not be received to be accounted in the rx-error counter regardless of
> the actual NIC. What do you think?

Agreed. I've sent a patch that removes "xec", the register name for l3_l4_xsum_errors:
http://dpdk.org/dev/patchwork/patch/8678/

Would you test it please? -Harry

Re: [dpdk-dev] [PATCH] ixgbe: remove l3_l4_xsum_errors from rx errors

[Thomas Monjalon]

2015-11-04 15:50, Harry van Haaren:
> This patch removes l3_l4_xsum_errors from rx errors.
> 
> The reason to remove it is that UDP packets have an optional checksum, and
> when not calculated the checksum field should be set to zero. When the
> checksum is not calculated (zero-ed out), the hardware still counts a valid
> UDP packet as an l3_l4_xsum_error.
> 
> This hardware issue is documented in 82599 errata, titled:
> "Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"

Please Harry, could you add a Fixes tag to track the origin of the bug?

[dpdk-dev] [PATCH v2] ixgbe: remove l3_l4_xsum_errors from rx errors

[Harry van Haaren]

This patch removes l3_l4_xsum_errors from rx errors.

The reason to remove it is that UDP packets have an optional checksum, and
when not calculated the checksum field should be set to zero. When the
checksum is not calculated (zero-ed out), the hardware still counts a valid
UDP packet as an l3_l4_xsum_error.

This hardware issue is documented in 82599 errata, titled:
"Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"

The solution is to remove l3_l4_xsum_errors from rx_errors, as discussed on
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/

Fixes: f6bf669b9900 ("ixgbe: account more Rx errors")

Suggested-by: Martin Weiser <martin.weiser@allegro-packets.com>
Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
---

v2: Add fixes line

 drivers/net/ixgbe/ixgbe_ethdev.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c
index 25966ef..9ed5b49 100644
--- a/drivers/net/ixgbe/ixgbe_ethdev.c
+++ b/drivers/net/ixgbe/ixgbe_ethdev.c
@@ -2480,7 +2480,6 @@ ixgbe_dev_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats)
 	                  total_missed_rx +
 	                  hw_stats->illerrc +
 	                  hw_stats->errbc +
-	                  hw_stats->xec +
 	                  hw_stats->rfc +
 	                  hw_stats->fccrc +
 	                  hw_stats->fclast;
-- 
1.9.1

Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1

[Martin Weiser]

On 04.11.15 16:54, Van Haaren, Harry wrote:
>> From: Martin Weiser [mailto:martin.weiser@allegro-packets.com]
>> Subject: Re: [dpdk-dev] ixgbe: ierrors counter spuriously increasing in DPDK 2.1
>> The
>> rx-error which showed up immediately after starting the interface is
>> gone since this was probably caused by mac_remote_errors. 
> Improvement - that's good.
>
>> But we still
>> see a huge number of rx-errors although all packets are received
>> properly and when looking at the extended stats those come from the
>> rx_l3_l4_xsum_error counter.
> That is useful information, good to know that statistic is the root cause.
>
>> In our setup we are dealing with lots of UDP traffic which does have the
>> UDP checksum set to 0 (which to my knowledge is allowed for UDP).
> Yes indeed checksum calculation for UDP is optional, and should be set to zero when not performed.
>
>> This
>> traffic seems to cause those rx_l3_l4_xsum_errors.
>> When doing the same
>> test with other NICs (e.g. XL710) no rx-errors are accounted.
> So this is a ixgbe bug, and listed in the errata, item 43 "Integrity Error Reported for IPv4/UDP Packets with Zero Checksum" in http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/82599-10-gbe-controller-spec-update.pdf 
>
>> For the generic stats interface I would prefer only packets that could
>> not be received to be accounted in the rx-error counter regardless of
>> the actual NIC. What do you think?
> Agreed. I've sent a patch that removes "xec", the register name for l3_l4_xsum_errors:
> http://dpdk.org/dev/patchwork/patch/8678/
>
> Would you test it please? -Harry
I have tested the patch and now everything looks good.

Best regards, Martin

Re: [dpdk-dev] [PATCH v2] ixgbe: remove l3_l4_xsum_errors from rx errors

[Thomas Monjalon]

2015-11-04 16:14, Harry van Haaren:
> This patch removes l3_l4_xsum_errors from rx errors.
> 
> The reason to remove it is that UDP packets have an optional checksum, and
> when not calculated the checksum field should be set to zero. When the
> checksum is not calculated (zero-ed out), the hardware still counts a valid
> UDP packet as an l3_l4_xsum_error.
> 
> This hardware issue is documented in 82599 errata, titled:
> "Integrity Error Reported for IPv4/UDP Packets with Zero Checksum"
> 
> The solution is to remove l3_l4_xsum_errors from rx_errors, as discussed on
> http://thread.gmane.org/gmane.comp.networking.dpdk.devel/25590/
> 
> Fixes: f6bf669b9900 ("ixgbe: account more Rx errors")
> 
> Suggested-by: Martin Weiser <martin.weiser@allegro-packets.com>
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>

Applied, thanks