mlx5: imissed versus prio0_buf_discards

mlx5: imissed versus prio0_buf_discards

[Tony Hart]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

I have a simple DPDK app that receives packets via RSS from a CX7 (400G).
The app uses 16 queues across 16 cores.  What I see is dropped packets even
at only 50Mpps.

Looking at rte_eth_port_xstats() I see  rx_prio0_buf_discard_packets
matches the number of packets dropped however the imissed counter (from
rte_eth_port_stats) is 0.  Indeed when I look at the rx_queue depths from
each thread in the app they barely reach 30 entries (I'm using the default
number of queue descs).

What is the difference between rx_prio0_buf_discards and imissed counters,
why would rx_prio0_buf_discards increase but not imissed?

many thanks,
tony

fyi: this is using DPKD 24.07 and the HWS RTE FLOW Api to setup the RSS
flow.  Firmware is 28.41

[-- Attachment #2: Type: text/html, Size: 1111 bytes --]

RE: mlx5: imissed versus prio0_buf_discards

[Dariusz Sosnowski]

Hi Tony,

I apologize for the late response.

> From: Tony Hart <tony.hart@domainhart.com> 
> Sent: Saturday, October 12, 2024 17:09
> To: users@dpdk.org
> Subject: mlx5: imissed versus prio0_buf_discards
> 
> External email: Use caution opening links or attachments 
> 
> 
> 
> I have a simple DPDK app that receives packets via RSS from a CX7 (400G).  The app uses 16 queues across 16 cores.  What I see is dropped packets even at only 50Mpps.
> 
> Looking at rte_eth_port_xstats() I see  rx_prio0_buf_discard_packets matches the number of packets dropped however the imissed counter (from rte_eth_port_stats) is 0.  Indeed when I look at the rx_queue depths from each thread in the app they barely reach 30 entries (I'm using the default number of queue descs).
> 
> What is the difference between rx_prio0_buf_discards and imissed counters, why would rx_prio0_buf_discards increase but not imissed?

Both counters measure packet drops, but at different levels:

- imissed - Measures drops caused by lack of free descriptors in the Rx queue.
  This indicates that SW cannot keep up with current packet rate.
- rx_prio0_buf_discards - Measures drops caused by lack of free space in NIC's Rx buffer.
  This indicates that HW cannot keep up with current packet rate.

What kind of traffic are you generating?
What kind of flow tables and rules do you create?
In your application, do you see that packets are roughly equally distributed across all 16 Rx queues?

> 
> many thanks,
> tony
> 
> fyi: this is using DPKD 24.07 and the HWS RTE FLOW Api to setup the RSS flow.  Firmware is 28.41

Best regards,
Dariusz Sosnowski

Re: mlx5: imissed versus prio0_buf_discards

[Tony Hart]

[-- Attachment #1: Type: text/plain, Size: 10059 bytes --]

Hi Dariusz,
Thanks for the reply.

I'm using 68  byte UDP packets with diverse SIP,DIPs, the traffic is
balanced between the cores (I included some logging below)

The flows and groups I have are

group 0: eth => jump group 2

group 2:  eth / ipv4 => rss to sw queues

group 2: eth => rss to sw queues (lower priority)

group 4: eth => random N => to sw queue 0

group 4: eth => drop (lower priority)


Group 4 is not actually referenced from anywhere, I assume this does not
affect performance.



2024-10-28 10:25:25  INFO: <0>  stats, ipackets:799741213,
opackets:799741213
2024-10-28 10:25:25  INFO: <0> flow 0x188195140 grp:pri 0:001 end end =>
jump jump 2/end end
2024-10-28 10:25:25  INFO: <0> flow 0x188174500 grp:pri 2:548 eth eth/ip4
ip4/end end => count count/rss rss 0-31/end end (799741213, 51183437632)
2024-10-28 10:25:25  INFO: <0> flow 0x1883d73c0 grp:pri 2:549 eth eth/end
end => count count/rss rss 0-31/end end (0, 0)
2024-10-28 10:25:25  INFO: <0> flow 0x1883b67c0 grp:pri 4:001 random
random/end end => rss rss 0-31/end end
2024-10-28 10:25:25  INFO: <0> flow 0x1883d7140 grp:pri 4:549 end end =>
drop drop/end end
2024-10-28 10:25:35  INFO: <0> flow 0x188195140 grp:pri 0:001 end end =>
jump jump 2/end end
2024-10-28 10:25:35  INFO: <0> flow 0x188174500 grp:pri 2:548 eth eth/ip4
ip4/end end => count count/rss rss 0-31/end end (799741213, 51183437632)
2024-10-28 10:25:35  INFO: <0> flow 0x1883d73c0 grp:pri 2:549 eth eth/end
end => count count/rss rss 0-31/end end (0, 0)
2024-10-28 10:25:35  INFO: <0> flow 0x1883b67c0 grp:pri 4:001 random
random/end end => rss rss 0-31/end end
2024-10-28 10:25:35  INFO: <0> flow 0x1883d7140 grp:pri 4:549 end end =>
drop drop/end end
2024-10-28 10:25:40  INFO: <0> xstats, rx_good_packets:799741213,
tx_good_packets:799741213, rx_good_bytes:51183437632,
tx_good_bytes:51183437632, rx_q0_packets:24987083, rx_q0_bytes:1599173312,
rx_q1_packets:24988480, rx_q1_bytes:1599262720, rx_q2_packets:24995304,
rx_q2_bytes:1599699456, rx_q3_packets:24998711, rx_q3_bytes:1599917504,
rx_q4_packets:24989350, rx_q4_bytes:1599318400, rx_q5_packets:24991546,
rx_q5_bytes:1599458944, rx_q6_packets:24991647, rx_q6_bytes:1599465408,
rx_q7_packets:24995441, rx_q7_bytes:1599708224, rx_q8_packets:24989564,
rx_q8_bytes:1599332096, rx_q9_packets:24990980, rx_q9_bytes:1599422720,
rx_q10_packets:24996265, rx_q10_bytes:1599760960, rx_q11_packets:24996320,
rx_q11_bytes:1599764480, rx_q12_packets:24987707, rx_q12_bytes:1599213248,
rx_q13_packets:24983936, rx_q13_bytes:1598971904, rx_q14_packets:24994621,
rx_q14_bytes:1599655744, rx_q15_packets:24991660, rx_q15_bytes:1599466240,
tx_q0_packets:24987083, tx_q0_bytes:1599173312, tx_q1_packets:24988480,
tx_q1_bytes:1599262720, tx_q2_packets:24995304, tx_q2_bytes:1599699456,
tx_q3_packets:24998711, tx_q3_bytes:1599917504, tx_q4_packets:24989350,
tx_q4_bytes:1599318400, tx_q5_packets:24991546, tx_q5_bytes:1599458944,
tx_q6_packets:24991647, tx_q6_bytes:1599465408, tx_q7_packets:24995441,
tx_q7_bytes:1599708224, tx_q8_packets:24989564, tx_q8_bytes:1599332096,
tx_q9_packets:24990980, tx_q9_bytes:1599422720, tx_q10_packets:24996265,
tx_q10_bytes:1599760960, tx_q11_packets:24996320, tx_q11_bytes:1599764480,
tx_q12_packets:24987707, tx_q12_bytes:1599213248, tx_q13_packets:24983936,
tx_q13_bytes:1598971904, tx_q14_packets:24994621, tx_q14_bytes:1599655744,
tx_q15_packets:24991660, tx_q15_bytes:1599466240,
rx_unicast_bytes:51183437632, rx_unicast_packets:799741213,
tx_unicast_bytes:51183437632, tx_unicast_packets:799741213,
tx_phy_packets:799741213, rx_phy_packets:1500000000,
rx_prio0_buf_discard_packets:700258787, tx_phy_bytes:54382402484,
rx_phy_bytes:102000000000
2024-10-28 10:25:40  INFO: <0>  stats, ipackets:799741213,
opackets:799741213
^C2024-10-28 10:25:42  INFO: forwarder13 exiting on core 21 n_rx_pkts:
24987707, n_sample_pkts: 0, max_rx_burst: 36, max_queue_depth: 30
2024-10-28 10:25:42  INFO: forwarder5 exiting on core 13 n_rx_pkts:
24989350, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 30
2024-10-28 10:25:42  INFO: forwarder8 exiting on core 16 n_rx_pkts:
24995441, n_sample_pkts: 0, max_rx_burst: 36, max_queue_depth: 36
2024-10-28 10:25:42  INFO: forwarder14 exiting on core 22 n_rx_pkts:
24983936, n_sample_pkts: 0, max_rx_burst: 36, max_queue_depth: 30
2024-10-28 10:25:42  INFO: forwarder19 exiting on core 27 n_rx_pkts:
24995349, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder4 exiting on core 12 n_rx_pkts:
24998711, n_sample_pkts: 0, max_rx_burst: 33, max_queue_depth: 33
2024-10-28 10:25:42  INFO: forwarder7 exiting on core 15 n_rx_pkts:
24991647, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder1 exiting on core 9 n_rx_pkts:
24987083, n_sample_pkts: 0, max_rx_burst: 32, max_queue_depth: 30
2024-10-28 10:25:42  INFO: forwarder11 exiting on core 19 n_rx_pkts:
24996265, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder15 exiting on core 23 n_rx_pkts:
24994621, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder10 exiting on core 18 n_rx_pkts:
24990980, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 32
2024-10-28 10:25:42  INFO: forwarder3 exiting on core 11 n_rx_pkts:
24995304, n_sample_pkts: 0, max_rx_burst: 25, max_queue_depth: 25
2024-10-28 10:25:42  INFO: forwarder2 exiting on core 10 n_rx_pkts:
24988480, n_sample_pkts: 0, max_rx_burst: 32, max_queue_depth: 30
2024-10-28 10:25:42  INFO: forwarder6 exiting on core 14 n_rx_pkts:
24991546, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 30
2024-10-28 10:25:42  INFO: forwarder12 exiting on core 20 n_rx_pkts:
24996320, n_sample_pkts: 0, max_rx_burst: 28, max_queue_depth: 28
2024-10-28 10:25:42  INFO: forwarder9 exiting on core 17 n_rx_pkts:
24989564, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder17 exiting on core 25 n_rx_pkts:
24986227, n_sample_pkts: 0, max_rx_burst: 36, max_queue_depth: 32
2024-10-28 10:25:42  INFO: forwarder20 exiting on core 28 n_rx_pkts:
24994610, n_sample_pkts: 0, max_rx_burst: 37, max_queue_depth: 35
2024-10-28 10:25:42  INFO: forwarder16 exiting on core 24 n_rx_pkts:
24991660, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder26 exiting on core 34 n_rx_pkts:
24994289, n_sample_pkts: 0, max_rx_burst: 29, max_queue_depth: 24
2024-10-28 10:25:42  INFO: forwarder23 exiting on core 31 n_rx_pkts:
24986582, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder22 exiting on core 30 n_rx_pkts:
24995067, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder21 exiting on core 29 n_rx_pkts:
24994793, n_sample_pkts: 0, max_rx_burst: 29, max_queue_depth: 23
2024-10-28 10:25:42  INFO: forwarder31 exiting on core 39 n_rx_pkts:
24993460, n_sample_pkts: 0, max_rx_burst: 26, max_queue_depth: 22
2024-10-28 10:25:42  INFO: forwarder29 exiting on core 37 n_rx_pkts:
24987219, n_sample_pkts: 0, max_rx_burst: 30, max_queue_depth: 30
2024-10-28 10:25:42  INFO: forwarder24 exiting on core 32 n_rx_pkts:
24986863, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder32 exiting on core 40 n_rx_pkts:
24991661, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 34
2024-10-28 10:25:42  INFO: forwarder18 exiting on core 26 n_rx_pkts:
24991276, n_sample_pkts: 0, max_rx_burst: 38, max_queue_depth: 36
2024-10-28 10:25:42  INFO: forwarder30 exiting on core 38 n_rx_pkts:
24989407, n_sample_pkts: 0, max_rx_burst: 30, max_queue_depth: 30
2024-10-28 10:25:42  INFO: forwarder25 exiting on core 33 n_rx_pkts:
24994537, n_sample_pkts: 0, max_rx_burst: 26, max_queue_depth: 22
2024-10-28 10:25:42  INFO: forwarder28 exiting on core 36 n_rx_pkts:
24996627, n_sample_pkts: 0, max_rx_burst: 26, max_queue_depth: 22
2024-10-28 10:25:42  INFO: forwarder27 exiting on core 35 n_rx_pkts:
24994631, n_sample_pkts: 0, max_rx_burst: 34, max_queue_depth: 32

On Fri, Oct 25, 2024 at 8:03 AM Dariusz Sosnowski <dsosnowski@nvidia.com>
wrote:

> Hi Tony,
>
> I apologize for the late response.
>
> > From: Tony Hart <tony.hart@domainhart.com>
> > Sent: Saturday, October 12, 2024 17:09
> > To: users@dpdk.org
> > Subject: mlx5: imissed versus prio0_buf_discards
> >
> > External email: Use caution opening links or attachments
> >
> >
> >
> > I have a simple DPDK app that receives packets via RSS from a CX7
> (400G).  The app uses 16 queues across 16 cores.  What I see is dropped
> packets even at only 50Mpps.
> >
> > Looking at rte_eth_port_xstats() I see  rx_prio0_buf_discard_packets
> matches the number of packets dropped however the imissed counter (from
> rte_eth_port_stats) is 0.  Indeed when I look at the rx_queue depths from
> each thread in the app they barely reach 30 entries (I'm using the default
> number of queue descs).
> >
> > What is the difference between rx_prio0_buf_discards and imissed
> counters, why would rx_prio0_buf_discards increase but not imissed?
>
> Both counters measure packet drops, but at different levels:
>
> - imissed - Measures drops caused by lack of free descriptors in the Rx
> queue.
>   This indicates that SW cannot keep up with current packet rate.
> - rx_prio0_buf_discards - Measures drops caused by lack of free space in
> NIC's Rx buffer.
>   This indicates that HW cannot keep up with current packet rate.
>
> What kind of traffic are you generating?
> What kind of flow tables and rules do you create?
> In your application, do you see that packets are roughly equally
> distributed across all 16 Rx queues?
>
> >
> > many thanks,
> > tony
> >
> > fyi: this is using DPKD 24.07 and the HWS RTE FLOW Api to setup the RSS
> flow.  Firmware is 28.41
>
> Best regards,
> Dariusz Sosnowski
>


-- 
tony

[-- Attachment #2: Type: text/html, Size: 11460 bytes --]