[dpdk-users] rx-missed errors with the fortville nic

[dpdk-users] rx-missed errors with the fortville nic

[Dorsett, Michal]

I would like your help in investigating a packet loss problem we have with an XL710 NIC split in 4x10 configuration and the X710:

Issue:
While injecting traffic to one of the NICs ports we noticed that not all packets are received by the application - rx_missed statistic is incrementing.

HW Environment:

-        Dell power edge r630 (BIOS 2.4.3 - release 17/Jan/2017)

-        One XL710 NIC-QDA2 connected to a 4x10G splitter (FW 5.0 API 1.5 NVM 05.00.05 eetrack 800029265, probe driver: 8086:1583 rte_i40e_pmd)

o   NIC is set to 4x10G

SW Environment:

-        DPDK version 16.07 - the issue is reproduced on version 17.08 as well.

-        4 cores (2,4,6,8 on socket 0) "connected" to all 4 ports via a dedicated queue in the following configuration example:

o   Core 4 -> queue (id=0) per  port (0,1,2,3)

o   Core 6 -> queue (id=1) per port (0,1,2,3)

o   Core 8 -> queue (id=2) per  port (0,1,2,3)

o   Core 10 -> queue (id=3) per port (0,1,2,3)


Traffic profile:

-        Frame size : 500 bytes

-        Rate : 500000 FPS

-        encapsulation : ETH-IPv6-UDP

Experiment #1 with "testpmd" application:

Load: ~2403846 FPS.

sudo ./testpmd -c 0x1555555555 -n 4 -- -i --nb-cores=16 --total-num-mbufs=262144 --rxd=1024 --rxq=8 --txq=8 --numa --port-topology=chained

After starting,  "show port stats" returned  rx_missed counter > 0.

Sample of port stats :
######################## NIC statistics for port 1  ########################
  RX-packets: 18393849   RX-missed: 112105508  RX-bytes:  67562507568
  RX-errors: 2
  RX-nombuf:  0
  TX-packets: 42         TX-errors: 0          TX-bytes:  3387

  Throughput (since last show)
  Rx-pps:      2152353
  Tx-pps:            0
  ############################################################################

Sample output when starting "testpmd":

testpmd> start
io packet forwarding - ports=4 - cores=16 - streams=32 - NUMA support enabled, MP over anonymous pages disabled
Logical Core 2 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01
Logical Core 4 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01
 RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01
Logical Core 6 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=4 (socket 0) -> TX P=1/Q=4 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=5 (socket 0) -> TX P=1/Q=5 (socket 0) peer=02:00:00:00:00:01
Logical Core 8 (socket 0) forwards packets on 2 streams:
  RX P=0/Q=6 (socket 0) -> TX P=1/Q=6 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=7 (socket 0) -> TX P=1/Q=7 (socket 0) peer=02:00:00:00:00:01
Logical Core 10 (socket 0) forwards packets on 2 streams:
  RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
Logical Core 12 (socket 0) forwards packets on 2 streams:
  RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00
Logical Core 14 (socket 0) forwards packets on 2 streams:
  RX P=1/Q=4 (socket 0) -> TX P=0/Q=4 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=5 (socket 0) -> TX P=0/Q=5 (socket 0) peer=02:00:00:00:00:00
Logical Core 16 (socket 0) forwards packets on 2 streams:
  RX P=1/Q=6 (socket 0) -> TX P=0/Q=6 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=7 (socket 0) -> TX P=0/Q=7 (socket 0) peer=02:00:00:00:00:00
Logical Core 18 (socket 0) forwards packets on 2 streams:
  RX P=2/Q=0 (socket 0) -> TX P=3/Q=0 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=1 (socket 0) -> TX P=3/Q=1 (socket 0) peer=02:00:00:00:00:03
Logical Core 20 (socket 0) forwards packets on 2 streams:
  RX P=2/Q=2 (socket 0) -> TX P=3/Q=2 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=3 (socket 0) -> TX P=3/Q=3 (socket 0) peer=02:00:00:00:00:03
Logical Core 22 (socket 0) forwards packets on 2 streams:
  RX P=2/Q=4 (socket 0) -> TX P=3/Q=4 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=5 (socket 0) -> TX P=3/Q=5 (socket 0) peer=02:00:00:00:00:03
Logical Core 24 (socket 0) forwards packets on 2 streams:
  RX P=2/Q=6 (socket 0) -> TX P=3/Q=6 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=7 (socket 0) -> TX P=3/Q=7 (socket 0) peer=02:00:00:00:00:03
Logical Core 26 (socket 0) forwards packets on 2 streams:
  RX P=3/Q=0 (socket 0) -> TX P=2/Q=0 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=1 (socket 0) -> TX P=2/Q=1 (socket 0) peer=02:00:00:00:00:02
Logical Core 28 (socket 0) forwards packets on 2 streams:
  RX P=3/Q=2 (socket 0) -> TX P=2/Q=2 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=3 (socket 0) -> TX P=2/Q=3 (socket 0) peer=02:00:00:00:00:02
Logical Core 30 (socket 0) forwards packets on 2 streams:
  RX P=3/Q=4 (socket 0) -> TX P=2/Q=4 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=5 (socket 0) -> TX P=2/Q=5 (socket 0) peer=02:00:00:00:00:02
Logical Core 32 (socket 0) forwards packets on 2 streams:
  RX P=3/Q=6 (socket 0) -> TX P=2/Q=6 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=7 (socket 0) -> TX P=2/Q=7 (socket 0) peer=02:00:00:00:00:02

Experiment #2 with "testpmd" application:

The following configuration (2403846 FPS) generates no rx missed:

sudo ./testpmd -c 0x15555 -n 4 -- -i --nb-cores=8 --total-num-mbufs=262144 --rxd=1024 --rxq=8 --txq=8 --numa --port-topology=chained

testpmd> start
io packet forwarding - ports=4 - cores=8 - streams=32 - NUMA support enabled, MP over anonymous pages disabled
Logical Core 2 (socket 0) forwards packets on 4 streams:
  RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=1 (socket 0) -> TX P=1/Q=1 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=2 (socket 0) -> TX P=1/Q=2 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=3 (socket 0) -> TX P=1/Q=3 (socket 0) peer=02:00:00:00:00:01
Logical Core 4 (socket 0) forwards packets on 4 streams:
  RX P=0/Q=4 (socket 0) -> TX P=1/Q=4 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=5 (socket 0) -> TX P=1/Q=5 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=6 (socket 0) -> TX P=1/Q=6 (socket 0) peer=02:00:00:00:00:01
  RX P=0/Q=7 (socket 0) -> TX P=1/Q=7 (socket 0) peer=02:00:00:00:00:01
Logical Core 6 (socket 0) forwards packets on 4 streams:
  RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=1 (socket 0) -> TX P=0/Q=1 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=2 (socket 0) -> TX P=0/Q=2 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=3 (socket 0) -> TX P=0/Q=3 (socket 0) peer=02:00:00:00:00:00
Logical Core 8 (socket 0) forwards packets on 4 streams:
  RX P=1/Q=4 (socket 0) -> TX P=0/Q=4 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=5 (socket 0) -> TX P=0/Q=5 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=6 (socket 0) -> TX P=0/Q=6 (socket 0) peer=02:00:00:00:00:00
  RX P=1/Q=7 (socket 0) -> TX P=0/Q=7 (socket 0) peer=02:00:00:00:00:00
Logical Core 10 (socket 0) forwards packets on 4 streams:
  RX P=2/Q=0 (socket 0) -> TX P=3/Q=0 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=1 (socket 0) -> TX P=3/Q=1 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=2 (socket 0) -> TX P=3/Q=2 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=3 (socket 0) -> TX P=3/Q=3 (socket 0) peer=02:00:00:00:00:03
Logical Core 12 (socket 0) forwards packets on 4 streams:
  RX P=2/Q=4 (socket 0) -> TX P=3/Q=4 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=5 (socket 0) -> TX P=3/Q=5 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=6 (socket 0) -> TX P=3/Q=6 (socket 0) peer=02:00:00:00:00:03
  RX P=2/Q=7 (socket 0) -> TX P=3/Q=7 (socket 0) peer=02:00:00:00:00:03
Logical Core 14 (socket 0) forwards packets on 4 streams:
  RX P=3/Q=0 (socket 0) -> TX P=2/Q=0 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=1 (socket 0) -> TX P=2/Q=1 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=2 (socket 0) -> TX P=2/Q=2 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=3 (socket 0) -> TX P=2/Q=3 (socket 0) peer=02:00:00:00:00:02
Logical Core 16 (socket 0) forwards packets on 4 streams:
  RX P=3/Q=4 (socket 0) -> TX P=2/Q=4 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=5 (socket 0) -> TX P=2/Q=5 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=6 (socket 0) -> TX P=2/Q=6 (socket 0) peer=02:00:00:00:00:02
  RX P=3/Q=7 (socket 0) -> TX P=2/Q=7 (socket 0) peer=02:00:00:00:00:02


testpmd> show port stats 1

  ######################## NIC statistics for port 1  ########################
  RX-packets: 211295261  RX-missed: 0          RX-bytes:  104802384960
  RX-errors: 0
  RX-nombuf:  0
  TX-packets: 53         TX-errors: 0          TX-bytes:  4047

  Throughput (since last show)
  Rx-pps:      2403872
  Tx-pps:            0
  ############################################################################


Note that we experience this issue with traffic as low as 1.5Gbps.

This is a show-stopper issue for us. Any assistance will be much appreciated.

Thanks,

Michal Dorsett
Developer, Strategic IP Group
Desk: +972 962 4350
Mobile: +972 50 771 6689
Verint Cyber Intelligence
www.verint.com<http://www.verint.com/>



This electronic message may contain proprietary and confidential information of Verint Systems Inc., its affiliates and/or subsidiaries. The information is intended to be for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient (or authorized to receive this e-mail for the intended recipient), you may not use, copy, disclose or distribute to anyone this message or any information contained in this message. If you have received this electronic message in error, please notify us by replying to this e-mail.

Re: [dpdk-users] rx-missed errors with the fortville nic

[Vincent JARDIN]

Le 30/10/2017 à 03:29, Dorsett, Michal a écrit :
> a packet loss problem we have with an XL710 NIC split in 4x10 configuration and the X710

Michal,

I do confirm that i40e is known to have some packet drop issues, even at 
low traffic rate. I am not aware of public information explaining issues 
and solutions on it.

best regards,
   Vincent

Re: [dpdk-users] rx-missed errors with the fortville nic

[Dorsett, Michal]

Thank you, Vincent, for your very candid response. It is very much appreciated.

However, to the DPDK community as a whole, and the i40e pmd team specifically, I am quite puzzled by this state of affairs.

This exact issue is detailed in http://dpdk.org/ml/archives/dev/2016-August/045416.html, to which there doesn't seem to be a reply.

Can someone provide a resolution?

Thanks,

Michal


-----Original Message-----
From: Vincent JARDIN [mailto:vincent.jardin@6wind.com]
Sent: Tuesday, October 31, 2017 3:25 AM
To: Dorsett, Michal <Michal.Dorsett@verint.com>; users@dpdk.org
Cc: Barlev, Eddie <Eddie.Barlev@verint.com>; Maimon, Royi <Royi.Maimon@verint.com>; Heijmans, Merav <Merav.Heijmans@verint.com>
Subject: Re: [dpdk-users] rx-missed errors with the fortville nic

Le 30/10/2017 à 03:29, Dorsett, Michal a écrit :
> a packet loss problem we have with an XL710 NIC split in 4x10
> configuration and the X710

Michal,

I do confirm that i40e is known to have some packet drop issues, even at low traffic rate. I am not aware of public information explaining issues and solutions on it.

best regards,
   Vincent


This electronic message may contain proprietary and confidential information of Verint Systems Inc., its affiliates and/or subsidiaries. The information is intended to be for the use of the individual(s) or entity(ies) named above. If you are not the intended recipient (or authorized to receive this e-mail for the intended recipient), you may not use, copy, disclose or distribute to anyone this message or any information contained in this message. If you have received this electronic message in error, please notify us by replying to this e-mail.