Re: [dpdk-users] MLX ConnectX-4 Discarding packets

[Thomas Monjalon <thomas@monjalon.net>]

Great, thanks for the update!


12/09/2021 11:32, Filip Janiszewski:
> Alright, nailed it down to a wrong preferred PCIe device in the BIOS
> configuration, it has not been changed after the NIC have been moved to
> another PCIe slot.
> 
> Now the EPYC is going really great, getting 100Gbps rate easily.
> 
> Thank
> 
> Il 9/11/21 4:34 PM, Filip Janiszewski ha scritto:
> > I wanted just to add, while running the same exact testpmd on the other
> > machine I won't get a single miss with the same patter traffic:
> > 
> > .
> > testpmd> stop
> > Telling cores to stop...
> > Waiting for lcores to finish...
> > 
> >   ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 0/Queue= 0
> > -------
> >   RX-packets: 61711939       TX-packets: 0              TX-dropped: 0
> > 
> > 
> >   ------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 0/Queue= 1
> > -------
> >   RX-packets: 62889424       TX-packets: 0              TX-dropped: 0
> > 
> > 
> >   ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 0/Queue= 2
> > -------
> >   RX-packets: 61914199       TX-packets: 0              TX-dropped: 0
> > 
> > 
> >   ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 0/Queue= 3
> > -------
> >   RX-packets: 63484438       TX-packets: 0              TX-dropped: 0
> > 
> > 
> >   ---------------------- Forward statistics for port 0
> > ----------------------
> >   RX-packets: 250000000      RX-dropped: 0             RX-total: 250000000
> >   TX-packets: 0              TX-dropped: 0             TX-total: 0
> > 
> > ----------------------------------------------------------------------------
> > 
> >   +++++++++++++++ Accumulated forward statistics for all
> > ports+++++++++++++++
> >   RX-packets: 250000000      RX-dropped: 0             RX-total: 250000000
> >   TX-packets: 0              TX-dropped: 0             TX-total: 0
> > 
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > .
> > 
> > In the lab I've the EPYC connected directly to the Xeon using a 100GbE
> > link, both same RHL8.4 and same DPDK 21.02, running:
> > 
> > .
> > ./dpdk-testpmd -l 21-31 -n 8 -w 81:00.1  -- -i --rxq=4 --txq=4
> > --burst=64 --forward-mode=rxonly --rss-ip --total-num-mbufs=4194304
> > --nb-cores=4
> > .
> > 
> > and sending from the other end with pktgen, the EPYC loss tons of
> > packets (see my previous email), the Xeon don't loss anything.
> > 
> > *Confusion!*
> > 
> > Il 9/11/21 4:19 PM, Filip Janiszewski ha scritto:
> >> Thanks,
> >>
> >> I knew that document and we've implemented many of those settings/rules,
> >> but perhaps there's one crucial I've forgot? Wonder which one.
> >>
> >> Anyway, increasing the amount of queues impinge the performance, while
> >> sending 250M packets over a 100GbE link to an Intel 810-cqda2 NIC
> >> mounted on the EPYC Milan server, i see:
> >>
> >> .
> >> 1 queue, 30Gbps, ~45Mpps, 64B frame = imiss: 54,590,111
> >> 2 queue, 30Gbps, ~45Mpps, 64B frame = imiss: 79,394,138
> >> 4 queue, 30Gbps, ~45Mpps, 64B frame = imiss: 87,414,030
> >> .
> >>
> >> With DPDK 21.02 on RHL8.4. I can't observe this situation while
> >> capturing from my Intel server where increasing the queues leads to
> >> better performance (while with the test input set I drop with one queue,
> >> I do not drop anymore with 2 on the Intel server.)
> >>
> >> A customer with a brand new EPYC Milan server in his lab observed as
> >> well this scenario which is a bit of a worry, but again it might be some
> >> config/compilation issue we need do deal with?
> >>
> >> BTW, the same issue can be reproduced with testpmd, using 4 queues and
> >> the same input data set (250M of 64bytes frame at 30Gbps):
> >>
> >> .
> >> testpmd> stop
> >> Telling cores to stop...
> >> Waiting for lcores to finish...
> >>
> >>   ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 0/Queue= 0
> >> -------
> >>   RX-packets: 41762999       TX-packets: 0              TX-dropped: 0
> >>
> >>
> >>   ------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 0/Queue= 1
> >> -------
> >>   RX-packets: 40152306       TX-packets: 0              TX-dropped: 0
> >>
> >>
> >>   ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 0/Queue= 2
> >> -------
> >>   RX-packets: 41153402       TX-packets: 0              TX-dropped: 0
> >>
> >>
> >>   ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 0/Queue= 3
> >> -------
> >>   RX-packets: 38341370       TX-packets: 0              TX-dropped: 0
> >>
> >>
> >>   ---------------------- Forward statistics for port 0
> >> ----------------------
> >>   RX-packets: 161410077      RX-dropped: 88589923      RX-total: 250000000
> >>   TX-packets: 0              TX-dropped: 0             TX-total: 0
> >>
> >> ----------------------------------------------------------------------------
> >> .
> >>
> >> .
> >> testpmd> show port xstats 0
> >> ###### NIC extended statistics for port 0
> >> rx_good_packets: 161410081
> >> tx_good_packets: 0
> >> rx_good_bytes: 9684605284
> >> tx_good_bytes: 0
> >> rx_missed_errors: 88589923
> >> .
> >>
> >> Can't figure out what's wrong here..
> >>
> >>
> >> Il 9/11/21 12:20 PM, Steffen Weise ha scritto:
> >>> Hi Filip,
> >>>
> >>> i have not seen the same issues.
> >>> Are you aware of this tuning guide? I applied it and had no issues with
> >>> intel 100G NIC.
> >>>
> >>> HPC Tuning Guide for AMD EPYC Processors
> >>> http://developer.amd.com/wp-content/resources/56420.pdf
> >>> <http://developer.amd.com/wp-content/resources/56420.pdf>
> >>>
> >>> Hope it helps.
> >>>
> >>> Cheers,
> >>> Steffen Weise
> >>>
> >>>
> >>>> Am 11.09.2021 um 10:56 schrieb Filip Janiszewski
> >>>> <contact@filipjaniszewski.com>:
> >>>>
> >>>> I ran more tests,
> >>>>
> >>>> This AMD server is a bit confusing, I can tune it to capture 28Mpps (64
> >>>> bytes frame) from one single core, so I would assume that using one more
> >>>> core will at least increase a bit the capture capabilities, but it's
> >>>> not, 1% more speed and it drops regardless of how many queues are
> >>>> configured - I've not observed this situation on the Intel server, where
> >>>> adding more queues/cores scale to higher throughput.
> >>>>
> >>>> This issue have been verified now with both Mellanox and Intel (810
> >>>> series, 100GbE) NICs.
> >>>>
> >>>> Anybody encountered anything similar?
> >>>>
> >>>> Thanks
> >>>>
> >>>> Il 9/10/21 3:34 PM, Filip Janiszewski ha scritto:
> >>>>> Hi,
> >>>>>
> >>>>> I've switched a 100Gbe MLX ConnectX-4 card from an Intel Xeon server to
> >>>>> an AMD EPYC server (running 75F3 CPU, 256GiB of RAM and PCIe4 lanes),
> >>>>> and using the same capture software we can't get any faster than 10Gbps,
> >>>>> when exceeding that speed regardless of the amount of queues configured
> >>>>> the rx_discards_phy counter starts to raise and packets are lost in huge
> >>>>> amounts.
> >>>>>
> >>>>> On the Xeon machine, I was able to get easily to 50Gbps with 4 queues.
> >>>>>
> >>>>> Is there any specific DPDK configuration that we might want to setup for
> >>>>> those AMD servers? The software is DPDK based so I wonder if some build
> >>>>> option is missing somewhere.
> >>>>>
> >>>>> What else I might want to look for to investigate this issue?
> >>>>>
> >>>>> Thanks