DPDK usage discussions
 help / color / mirror / Atom feed
From: Francesco Montorsi <francesco.montorsi@infovista.com>
To: "Yan,
	Xiaoping (NSB - CN/Hangzhou)" <xiaoping.yan@nokia-sbell.com>,
	Gerry Wan <gerryw@stanford.edu>, Asaf Penso <asafp@nvidia.com>,
	Slava Ovsiienko <viacheslavo@nvidia.com>,
	Matan Azrad <matan@nvidia.com>,
	Raslan Darawsheh <rasland@nvidia.com>
Cc: Martin Weiser <martin.weiser@allegro-packets.com>,
	David Marchand <david.marchand@redhat.com>,
	"users@dpdk.org" <users@dpdk.org>
Subject: RE: mlx5 VF packet lost between rx_port_unicast_packets and rx_good_packets
Date: Wed, 10 Nov 2021 23:53:43 +0000
Message-ID: <AS8PR03MB67747E3EA2CDB3D84126509491939@AS8PR03MB6774.eurprd03.prod.outlook.com> (raw)

[-- Attachment #1: Type: text/plain, Size: 16740 bytes --]

Hi all,
I hit the exact same problem reported by Yan.
I’m using:

  *   2 Mellanox CX5 MT28800 installed on 2 different servers, connected together
  *   Device FW (as reported by DPDK): 16.31.1014
  *   DPDK 19.11.5 (from 6WindGate actually)

I sent roughly 360M packets from one server to the other using “testpmd” (in –forward-mode=txonly).
My DPDK application on the other server is reporting the following xstats counters (:

CounterName                      PORT0        PORT1        TOTAL
rx_good_packets:              76727920,           0,    76727920
tx_good_packets:                     0,           0,           0
rx_good_bytes:              4910586880,           0,  4910586880
tx_good_bytes:                       0,           0,           0
rx_missed_errors:                    0,           0,           0
rx_errors:                           0,           0,           0
tx_errors:                           0,           0,           0
rx_mbuf_allocation_errors:           0,           0,           0
rx_q0packets:                        0,           0,           0
rx_q0bytes:                          0,           0,           0
rx_q0errors:                         0,           0,           0
rx_q1packets:                        0,           0,           0
rx_q1bytes:                          0,           0,           0
rx_q1errors:                         0,           0,           0
rx_q2packets:                        0,           0,           0
rx_q2bytes:                          0,           0,           0
rx_q2errors:                         0,           0,           0
rx_q3packets:                        0,           0,           0
rx_q3bytes:                          0,           0,           0
rx_q3errors:                         0,           0,           0
rx_q4packets:                        0,           0,           0
rx_q4bytes:                          0,           0,           0
rx_q4errors:                         0,           0,           0
rx_q5packets:                 76727920,           0,    76727920
rx_q5bytes:                 4910586880,           0,  4910586880
rx_q5errors:                         0,           0,           0
rx_q6packets:                        0,           0,           0
rx_q6bytes:                          0,           0,           0
rx_q6errors:                         0,           0,           0
rx_q7packets:                        0,           0,           0
rx_q7bytes:                          0,           0,           0
rx_q7errors:                         0,           0,           0
rx_q8packets:                        0,           0,           0
rx_q8bytes:                          0,           0,           0
rx_q8errors:                         0,           0,           0
rx_q9packets:                        0,           0,           0
rx_q9bytes:                          0,           0,           0
rx_q9errors:                         0,           0,           0
rx_q10packets:                       0,           0,           0
rx_q10bytes:                         0,           0,           0
rx_q10errors:                        0,           0,           0
rx_q11packets:                       0,           0,           0
rx_q11bytes:                         0,           0,           0
rx_q11errors:                        0,           0,           0
tx_q0packets:                        0,           0,           0
tx_q0bytes:                          0,           0,           0
rx_wqe_err:                          0,           0,           0
rx_port_unicast_packets:     360316064,           0,   360316064
rx_port_unicast_bytes:     23060228096,           0, 23060228096
tx_port_unicast_packets:             0,           0,           0
tx_port_unicast_bytes:               0,           0,           0
rx_port_multicast_packets:           0,           0,           0
rx_port_multicast_bytes:             0,           0,           0
tx_port_multicast_packets:           0,           0,           0
tx_port_multicast_bytes:             0,           0,           0
rx_port_broadcast_packets:           0,           0,           0
rx_port_broadcast_bytes:             0,           0,           0
tx_port_broadcast_packets:           0,           0,           0
tx_port_broadcast_bytes:             0,           0,           0
tx_packets_phy:                      0,           0,           0
rx_packets_phy:                      0,           0,           0
rx_crc_errors_phy:                   0,           0,           0
tx_bytes_phy:                        0,           0,           0
rx_bytes_phy:                        0,           0,           0
rx_in_range_len_errors_phy           0,           0,           0
rx_symbol_err_phy:                   0,           0,           0
rx_discards_phy:                     0,           0,           0
tx_discards_phy:                     0,           0,           0
tx_errors_phy:                       0,           0,           0
rx_out_of_buffer:                    0,           0,           0

So rx_good_packets is roughly 76M pkts, while rx_port_unicast_packets has counted correctly all 360M pkts sent by testpmd.
Of course my application layer has been able to dequeue from the DPDK port only 76M pkts so the remaining (rx_port_unicast_packets- rx_good_packets) got lost but are not reported in the “imissed” counter or “ierrors” counter of rte_eth_stats…

It would be not so easy for me to test against latest DPDK… also from what Yan has reported, the issue is still there in DPDK stable 20.11.3…

@Mellanox maintainers: any update on this issue? Is there a workaround to get the dropped packet back into the rte_eth_stats counters?


Thanks

Francesco Montorsi


From: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com>
Sent: Friday, October 29, 2021 2:44 AM
To: Gerry Wan <gerryw@stanford.edu>; Asaf Penso <asafp@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>; Matan Azrad <matan@nvidia.com>; Raslan Darawsheh <rasland@nvidia.com>
Cc: Martin Weiser <martin.weiser@allegro-packets.com>; David Marchand <david.marchand@redhat.com>; users@dpdk.org
Subject: RE: mlx5 VF packet lost between rx_port_unicast_packets and rx_good_packets

Hi,

Yes, it’s always zero…
It seems dpdk-stable-20.11-3 already include this patch.
[xiaopiya@fedora30 dpdk-stable-20.11.3]$ patch -p1 < ./4-4-net-mlx5-fix-imissed-statistics.diff
patching file drivers/net/mlx5/linux/mlx5_os.c
Reversed (or previously applied) patch detected!  Assume -R? [n] n
Apply anyway? [n] ^C
[xiaopiya@fedora30 dpdk-stable-20.11.3]$ grep -r "mlx5_queue_counter_id_prepare" ./
./drivers/net/mlx5/linux/mlx5_os.c:mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
./drivers/net/mlx5/linux/mlx5_os.c:             mlx5_queue_counter_id_prepare(eth_dev);
./drivers/net/mlx5/linux/mlx5_os.c.rej:+mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
./drivers/net/mlx5/linux/mlx5_os.c.rej:+                mlx5_queue_counter_id_prepare(eth_dev);
./4-4-net-mlx5-fix-imissed-statistics.diff:+mlx5_queue_counter_id_prepare(struct rte_eth_dev *dev)
./4-4-net-mlx5-fix-imissed-statistics.diff:+            mlx5_queue_counter_id_prepare(eth_dev);

Thank you.

Best regards
Yan Xiaoping

From: Gerry Wan <gerryw@stanford.edu<mailto:gerryw@stanford.edu>>
Sent: 2021年10月28日 9:58
To: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com<mailto:xiaoping.yan@nokia-sbell.com>>
Cc: Martin Weiser <martin.weiser@allegro-packets.com<mailto:martin.weiser@allegro-packets.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>; Asaf Penso <asafp@nvidia.com<mailto:asafp@nvidia.com>>; Slava Ovsiienko <viacheslavo@nvidia.com<mailto:viacheslavo@nvidia.com>>; Matan Azrad <matan@nvidia.com<mailto:matan@nvidia.com>>; Raslan Darawsheh <rasland@nvidia.com<mailto:rasland@nvidia.com>>; users@dpdk.org<mailto:users@dpdk.org>
Subject: Re: mlx5 VF packet lost between rx_port_unicast_packets and rx_good_packets

Are the rx_missed_errors/rx_out_of_buffer counters always showing 0 no matter how fast you push your generator?

I had a similar issue with missing counters on DPDK 20.11 that was fixed on 21.05 by applying this patch:
http://patchwork.dpdk.org/project/dpdk/patch/1614249901-307665-5-git-send-email-matan@nvidia.com/

Potentially relevant thread:
https://inbox.dpdk.org/users/CAAcwi38rs2Vk9MKhRGS3kAK+=dYAnDdECT7f+Ts-f13cANYB+Q@mail.gmail.com/

On Wed, Oct 27, 2021 at 6:39 PM Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com<mailto:xiaoping.yan@nokia-sbell.com>> wrote:
Hi,

I checked the counter from PF with ethtool -S, there is no counter named 'rx_prio0_buf_discard'
Anyway, I checked all counters from ethtool output, there is not any counter reflects the dropped packets.

Any suggestion from mlx maintainer? @Matan Azrad @Asaf Penso @Slava Ovsiienko @Raslan Darawsheh

Thank you.


Best regards
Yan Xiaoping

-----Original Message-----
From: Martin Weiser <martin.weiser@allegro-packets.com<mailto:martin.weiser@allegro-packets.com>>
Sent: 2021年10月27日 15:54
To: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com<mailto:xiaoping.yan@nokia-sbell.com>>; David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>
Cc: Asaf Penso <asafp@nvidia.com<mailto:asafp@nvidia.com>>; users@dpdk.org<mailto:users@dpdk.org>; Slava Ovsiienko <viacheslavo@nvidia.com<mailto:viacheslavo@nvidia.com>>; Matan Azrad <matan@nvidia.com<mailto:matan@nvidia.com>>; Raslan Darawsheh <rasland@nvidia.com<mailto:rasland@nvidia.com>>
Subject: Re: mlx5 VF packet lost between rx_port_unicast_packets and rx_good_packets

Hi,

you may want to check the counter 'rx_prio0_buf_discard' with ethtool (which is not available in DPDK xstats as it seems that this counter is global for the card and not available per port).
I opened a ticket a while ago regarding this issue:
https://bugs.dpdk.org/show_bug.cgi?id=749

Best regards,
Martin


Am 27.10.21 um 08:17 schrieb Yan, Xiaoping (NSB - CN/Hangzhou):
> Hi,
>
> I tried with dpdk 20.11-3 downloaded from
> https://fast.dpdk.org/rel/dpdk-20.11.3.tar.xz
> Problem still exist:
> 1. there is packet loss with 2mpps (small packet), 2. no counter for
> the dropped packet in NIC.
>
> traffic generator stats: sends 41990938, receives back 41986105, lost
> 4833 testpmd fwd stats: RX-packets: 41986110, TX-packets: 41986110
> port xstats: rx_unicast_packets: 41990938  (all packets reached to the NIC port), rx_good_packets: 41986111 (some is lost), but there is not any counter of the lost packet.
>
> Here is the log:
> [root@up-0 /]# dpdk-testpmd -l "2,3" --legacy-mem --socket-mem
> "5000,0" -a 0000:03:06.7  -- -i --nb-cores=1 --portmask=0x1 --rxd=512
> --txd=512
> EAL: Detected 28 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Detected static linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: No available hugepages reported in hugepages-2048kB
> EAL: Probing VFIO support...
> EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:03:06.7
> (socket 0)
> mlx5_pci: cannot bind mlx5 socket: Read-only file system
> mlx5_pci: Cannot initialize socket: Read-only file system
> EAL: No legacy callbacks, legacy socket not created Interactive-mode
> selected
> testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176,
> socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
>
> Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.
>
> Configuring Port 0 (socket 0)
> Port 0: 7A:9A:8A:A6:86:93
> Checking link statuses...
> Done
> testpmd> port stop 0
> Stopping ports...
> Checking link statuses...
> Done
> testpmd> vlan set filter on 0
> testpmd> rx_vlan add 767 0
> testpmd> port start 0
> Port 0: 7A:9A:8A:A6:86:93
> Checking link statuses...
> Done
> testpmd> set fwd 5tswap
> Set 5tswap packet forwarding mode
> testpmd> start
> 5tswap packet forwarding - ports=1 - cores=1 - streams=1 - NUMA
> support enabled, MP allocation mode: native Logical Core 3 (socket 0) forwards packets on 1 streams:
>    RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0)
> peer=02:00:00:00:00:00
>
>    5tswap packet forwarding packets/burst=32
>    nb forwarding cores=1 - nb forwarding ports=1
>    port 0: RX queue number: 1 Tx queue number: 1
>      Rx offloads=0x200 Tx offloads=0x0
>      RX queue: 0
>        RX desc=512 - RX free threshold=64
>        RX threshold registers: pthresh=0 hthresh=0  wthresh=0
>        RX Offloads=0x200
>      TX queue: 0
>        TX desc=512 - TX free threshold=0
>        TX threshold registers: pthresh=0 hthresh=0  wthresh=0
>        TX offloads=0x0 - TX RS bit threshold=0
>
> testpmd> show fwd stats all
>
>    ---------------------- Forward statistics for port 0  ----------------------
>    RX-packets: 41986110       RX-dropped: 0             RX-total: 41986110
>    TX-packets: 41986110       TX-dropped: 0             TX-total: 41986110
>
> ----------------------------------------------------------------------
> ------
>
> testpmd> show port xstats 0
> ###### NIC extended statistics for port 0
> rx_good_packets: 41986111
> tx_good_packets: 41986111
> rx_good_bytes: 3106973594
> tx_good_bytes: 3106973594
> rx_missed_errors: 0
> rx_errors: 0
> tx_errors: 0
> rx_mbuf_allocation_errors: 0
> rx_q0_packets: 41986111
> rx_q0_bytes: 3106973594
> rx_q0_errors: 0
> tx_q0_packets: 41986111
> tx_q0_bytes: 3106973594
> rx_wqe_errors: 0
> rx_unicast_packets: 41990938
> rx_unicast_bytes: 3107329412
> tx_unicast_packets: 41986111
> tx_unicast_bytes: 3106973594
> rx_multicast_packets: 1
> rx_multicast_bytes: 114
> tx_multicast_packets: 0
> tx_multicast_bytes: 0
> rx_broadcast_packets: 5
> rx_broadcast_bytes: 1710
> tx_broadcast_packets: 0
> tx_broadcast_bytes: 0
> tx_phy_packets: 0
> rx_phy_packets: 0
> rx_phy_crc_errors: 0
> tx_phy_bytes: 0
> rx_phy_bytes: 0
> rx_phy_in_range_len_errors: 0
> rx_phy_symbol_errors: 0
> rx_phy_discard_packets: 0
> tx_phy_discard_packets: 0
> tx_phy_errors: 0
> rx_out_of_buffer: 0
> tx_pp_missed_interrupt_errors: 0
> tx_pp_rearm_queue_errors: 0
> tx_pp_clock_queue_errors: 0
> tx_pp_timestamp_past_errors: 0
> tx_pp_timestamp_future_errors: 0
> tx_pp_jitter: 0
> tx_pp_wander: 0
> tx_pp_sync_lost: 0
> testpmd> q
> Command not found
> testpmd> exit
> Command not found
> testpmd> quit
> Telling cores to stop...
> Waiting for lcores to finish...
>
>    ---------------------- Forward statistics for port 0  ----------------------
>    RX-packets: 41986112       RX-dropped: 0             RX-total: 41986112
>    TX-packets: 41986112       TX-dropped: 0             TX-total: 41986112
>
> ----------------------------------------------------------------------
> ------
>
> Best regards
> Yan Xiaoping
>
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com<mailto:david.marchand@redhat.com>>
> Sent: 2021年10月18日 18:45
> To: Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com<mailto:xiaoping.yan@nokia-sbell.com>>
> Cc: Asaf Penso <asafp@nvidia.com<mailto:asafp@nvidia.com>>; users@dpdk.org<mailto:users@dpdk.org>; Slava Ovsiienko
> <viacheslavo@nvidia.com<mailto:viacheslavo@nvidia.com>>; Matan Azrad <matan@nvidia.com<mailto:matan@nvidia.com>>; Raslan
> Darawsheh <rasland@nvidia.com<mailto:rasland@nvidia.com>>
> Subject: Re: mlx5 VF packet lost between rx_port_unicast_packets and
> rx_good_packets
>
> On Mon, Oct 18, 2021 at 11:28 AM Yan, Xiaoping (NSB - CN/Hangzhou) <xiaoping.yan@nokia-sbell.com<mailto:xiaoping.yan@nokia-sbell.com>> wrote:
>> I have cloned dpdk code from github
>>
>> [xiaopiya@fedora30 dpdk]$ git remote -v origin
>> https://github.com/DPDK/dpdk.git (fetch) origin
>> https://github.com/DPDK/dpdk.git (push)
>>
>> which tag should I use?
>>
>> Or do I have to download 20.11.3 from git.dpdk.org<http://git.dpdk.org>?
>>
>> Sorry, I don’t know the relation between https://github.com/DPDK and  git.dpdk.org<http://git.dpdk.org>?
> Github DPDK/dpdk repo is a replication of the main repo hosted on dpdk.org<http://dpdk.org> servers.
>
> The official git repos and releases tarballs are on dpdk.org<http://dpdk.org> servers.
> The list of official releases tarballs is at:
> http://core.dpdk.org/download/ The main repo git is at:
> https://git.dpdk.org/dpdk/ The LTS/stable releases repo git is at:
> https://git.dpdk.org/dpdk-stable/
>
>
> --
> David Marchand
>

[-- Attachment #2: Type: text/html, Size: 52041 bytes --]

             reply	other threads:[~2021-11-15 17:38 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-10 23:53 Francesco Montorsi [this message]
     [not found] ` <DM8PR12MB5494181EE59EA4018F71E57DCD949@DM8PR12MB5494.namprd12.prod.outlook.com>
2021-11-11 13:25   ` Francesco Montorsi
  -- strict thread matches above, loose matches on Subject: below --
2021-07-05 10:07 [dpdk-users] " Yan, Xiaoping (NSB - CN/Hangzhou)
2021-07-13 12:35 ` Asaf Penso
2021-07-26  4:52   ` Yan, Xiaoping (NSB - CN/Hangzhou)
     [not found]     ` <DM8PR12MB54940E42337767B960E6BD28CDA69@DM8PR12MB5494.namprd12.prod.outlook.com>
     [not found]       ` <16515cb8ebbb4e7d833914040fa5943f@nokia-sbell.com>
     [not found]         ` <df15fbaac9a644fa93c55e1cfbc9226b@nokia-sbell.com>
     [not found]           ` <DM8PR12MB5494A1A84DA28E47CCB7496ECDA99@DM8PR12MB5494.namprd12.prod.outlook.com>
     [not found]             ` <22e029bf2ae6407e910cb2dcf11d49ce@nokia-sbell.com>
     [not found]               ` <e8e7cf781dd444fa9648facfc824d4c9@nokia-sbell.com>
     [not found]                 ` <4b0dd266b53541c7bc4964c29e24a0e6@nokia-sbell.com>
2021-09-30  8:05                   ` Yan, Xiaoping (NSB - CN/Hangzhou)
2021-10-14  6:55                     ` Asaf Penso
2021-10-14  9:33                       ` Yan, Xiaoping (NSB - CN/Hangzhou)
2021-10-14  9:50                         ` Asaf Penso
2021-10-14 10:15                           ` Yan, Xiaoping (NSB - CN/Hangzhou)
2021-10-14 11:48                             ` Asaf Penso
2021-10-18  9:28                               ` Yan, Xiaoping (NSB - CN/Hangzhou)
2021-10-18 10:45                                 ` David Marchand
2021-10-27  6:17                                   ` Yan, Xiaoping (NSB - CN/Hangzhou)
2021-10-27  7:26                                     ` David Marchand
2021-10-27  7:54                                     ` Martin Weiser
2021-10-28  1:39                                       ` Yan, Xiaoping (NSB - CN/Hangzhou)
2021-10-28  1:58                                         ` Gerry Wan
2021-10-29  0:44                                           ` Yan, Xiaoping (NSB - CN/Hangzhou)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AS8PR03MB67747E3EA2CDB3D84126509491939@AS8PR03MB6774.eurprd03.prod.outlook.com \
    --to=francesco.montorsi@infovista.com \
    --cc=asafp@nvidia.com \
    --cc=david.marchand@redhat.com \
    --cc=gerryw@stanford.edu \
    --cc=martin.weiser@allegro-packets.com \
    --cc=matan@nvidia.com \
    --cc=rasland@nvidia.com \
    --cc=users@dpdk.org \
    --cc=viacheslavo@nvidia.com \
    --cc=xiaoping.yan@nokia-sbell.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK usage discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git