DPDK usage discussions
 help / color / mirror / Atom feed
From: Raslan Darawsheh <rasland@mellanox.com>
To: Tom Barbette <barbette@kth.se>, Yan Lei <l.yan@epfl.ch>,
	Wisam Monther <wisamm@mellanox.com>,
	Thomas Monjalon <thomas@monjalon.net>
Cc: "users@dpdk.org" <users@dpdk.org>
Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec
Date: Tue, 21 Apr 2020 12:30:23 +0000	[thread overview]
Message-ID: <AM0PR05MB670713AC7CC0003011D81E7CC2D50@AM0PR05MB6707.eurprd05.prod.outlook.com> (raw)
In-Reply-To: <50ddf9ff-654b-347c-bb76-41595ad16f9b@kth.se>

Hi Tom,

Can you send an example for an rte_flow rule that you are trying ?
I guess since you are using RSS it might affect more the performance what flows are being used.

Kindest regards,
Raslan Darawsheh

> -----Original Message-----
> From: users <users-bounces@dpdk.org> On Behalf Of Tom Barbette
> Sent: Tuesday, April 21, 2020 12:00 PM
> To: Yan Lei <l.yan@epfl.ch>; Wisam Monther <wisamm@mellanox.com>;
> Thomas Monjalon <thomas@monjalon.net>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than
> 4K per sec
> 
> Interesting! No I did not try the flow_perf, it was from our own
> application.
> 
> I'm actually taking that number from the installation time of a single
> rule, that have RSS action which is probably more costly. So this and
> that may bring down the performance.
> 
> Tom
> 
> Le 20/04/2020 à 15:48, Yan Lei a écrit :
> >
> > Hi Tom,
> >
> > I guess "SW steering" refers to the "direct verbs/rules"
> >
> (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail
> s.dpdk.org%2Farchives%2Fdev%2F2019-
> February%2F125303.html&amp;data=02%7C01%7Crasland%40mellanox.com
> %7C5a9698dca303410e1cf208d7e5d251d7%7Ca652971c7d2e4d9ba6a4d14925
> 6f461b%7C0%7C0%7C637230563788809406&amp;sdata=AFzmKniGXDV5yCnd
> 3qQIRQNUQ0YCxS0jXv2b2n6QW0Q%3D&amp;reserved=0). group=0
> > is still the same old (pre dpdk 19.05) slow implementation of flow
> > insertion. But just my guess.
> >
> > How did you measure the flow insertion rate? Did you use the patch they
> > mentioned earlier in the thread? With that patch I got 330K with
> > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
> > --group=1 --ether --ipv4 --udp --queue --flows-count=1000000.
> >
> > Cheers,
> > Lei
> >
> > ------------------------------------------------------------------------
> > *From:* Tom Barbette <barbette@kth.se>
> > *Sent:* Monday, April 20, 2020 2:24 PM
> > *To:* Wisam Monther; Thomas Monjalon; Yan Lei
> > *Cc:* users@dpdk.org
> > *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
> > than 4K per sec
> > Hi,
> >
> > Le 19/04/2020 à 16:07, Wisam Monther a écrit :
> >> Hey Yan,
> >>
> >> For FW steering there is HW limitation on number of flows that can be
> added on it.
> >> Which is table 0, and I think it's 65536
> >>
> >> But to get Millions of rules use --group=1 which is SW steering.
> > What do you mean by SW steering?
> >
> > Using group 1 we had better performance but only around 10K. I think the
> > whole process lacks an update feature instead of delete+create, and the
> > ability to batch rules.
> >
> >> Moreover make sure you have enough memory in the app to have good
> insertion rate.
> >>
> >> If you have enough 1G huge pages then it's ok.
> >> If you are working with 2M pages your command should be like this:
> >>
> >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096
> -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000
> >>
> >> BRs,
> >> Wisam Jaddo
> >>
> >>> -----Original Message-----
> >>> From: Thomas Monjalon <thomas@monjalon.net>
> >>> Sent: Sunday, April 19, 2020 4:58 PM
> >>> To: Yan Lei <l.yan@epfl.ch>
> >>> Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com>
> >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
> than 4K
> >>> per sec
> >>>
> >>> +Cc Wisam
> >>>
> >>> 16/04/2020 17:32, Yan Lei:
> >>>> Hi Thomas,
> >>>>
> >>>>
> >>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3.
> >>>>
> >>>>
> >>>> TL;DR
> >>>>
> >>>>
> >>>> 1. I was only able to generate 3K rules per second.
> >>>>
> >>>> 2. The maximum number of distinct rules the NIC can support seems to
> be
> >>> 65536.
> >>>>
> >>>>
> >>>> How can I increase the insertion rate? Any firmware/driver config I
> need to
> >>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch
> defaults to
> >>> generate 4 million distinct flows though...
> >>>>
> >>>>
> >>>> Thanks in advance!
> >>>>
> >>>>
> >>>>
> >>>> Initially, running
> >>>>
> >>>>
> >>>> ```
> >>>>
> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
> >>>> --ether --ipv4 --udp --queue --flows-count=1000000
> >>>>
> >>>> ```
> >>>>
> >>>>
> >>>> failed after a few seconds and it gave
> >>>>
> >>>>
> >>>> ```
> >>>> Flow can't be created 1 message: hardware refuses to create flow
> >>>> EAL: Error - exiting with code: 1
> >>>>    Cause: error in creating flow
> >>>> ```
> >>>>
> >>>>
> >>>> Then I added a small debug patch (attached) and it showed that the
> error
> >>> happens when creating the 65536th flow rule.
> >
> > The first table is indeed limited to something around that number. But
> > performance are already degrading before that point. Even with OFED 5
> > and the firmware that comes with it.
> >
> >>>>
> >>>>
> >>>> ```
> >>>> Flow can't be created 1 message: hardware refuses to create flow
> >>>> EAL: Error - exiting with code: 1
> >>>>    Cause: error in creating flow,flows generated: 65536 ```
> >>>>
> >>>>
> >>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I
> >>> changed the outer ip mask to 0xffff, the above command runs fine.
> >>>>
> >>>>
> >>>> To see how many rules I can generate per second. I ran (with the outer
> >>>> ip mask 0xffff)
> >>>>
> >>>>
> >>>> ```
> >>>>
> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
> >>>> --ether --ipv4 --udp --queue --flows-count=65536
> >>>>
> >>>> ```
> >>>>
> >>>>
> >>>> and it gives
> >>>>
> >>>>
> >>>> ```
> >>>>
> >>>> :: Total flow insertion rate -> 3.015922 K/Sec
> >>>> :: The time for creating 65536 in flows 21.730005 seconds
> >>>> :: EAGAIN counter = 0
> >>>> ```
> >>>> So 3 rules per sec. Which is close to what I observed before.
> >>>>
> >>>> ```
> >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
> >>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives
> >>>>
> >>>> ```
> >>>> :: Total flow insertion rate -> 0.949381 K/Sec
> >>>> :: The time for creating 100000 in flows 105.331842 seconds
> >>>> :: EAGAIN counter = 0
> >>>> ```
> >>>> Have no idea why it's only 1k/sec in this case...
> >>>>
> >>>> Thanks and cheers,
> >>>> Lei
> >>>>
> >>>>
> >>>> ________________________________
> >>>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei
> >>>> <l.yan@epfl.ch>
> >>>> Sent: Tuesday, April 14, 2020 1:20 PM
> >>>> To: Thomas Monjalon
> >>>> Cc: users@dpdk.org
> >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
> >>>> than 4K per sec
> >>>>
> >>>> Hi Thomas,
> >>>>
> >>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3).
> >>>>
> >>>> Cheers,
> >>>> Lei
> >>>> ________________________________
> >>>> From: Thomas Monjalon <thomas@monjalon.net>
> >>>> Sent: Tuesday, April 14, 2020 12:12:28 PM
> >>>> To: Yan Lei
> >>>> Cc: users@dpdk.org
> >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
> >>>> than 4K per sec
> >>>>
> >>>> Hi,
> >>>>
> >>>> 10/04/2020 20:11, Yan Lei:
> >>>>> I am doing some study that requires inserting more than 1 million
> >>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a
> ConnectX-5
> >>> NIC.
> >>>>>
> >>>>> But I only managed to create around 3.3K rules per second.
> >>>>> Below is the code I used to measure the insertion rate:
> >>>>
> >>>> Please could you review this new application designed for such
> measure?
> >>>>
> >>>>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
> >>>>
> >>>
> hes.dpdk.org%2Fpatch%2F68058%2F&amp;data=02%7C01%7Cwisamm%40m
> >>> ellanox.c
> >>>>
> >>>
> om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d14
> 9
> >>> 256f46
> >>>>
> >>>
> 1b%7C0%7C0%7C637229014854391590&amp;sdata=dxxEmm4DWoMPeNGy
> M
> >>> FaYgk%2BjSE
> >>>> %2FwVKLnYAwQ7QhjKGc%3D&amp;reserved=0
> >>>>
> >>>> Any feedback about the above patch is welcome. Feel free to try and
> review
> >>> it.
> >>>
> >>>
> >>>
> >>
> >
> >
> > Tom

  reply	other threads:[~2020-04-21 12:30 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-10 18:11 Yan Lei
2020-04-14 10:12 ` Thomas Monjalon
2020-04-14 11:20   ` Yan Lei
2020-04-16 15:32     ` Yan Lei
2020-04-19 13:57       ` Thomas Monjalon
2020-04-19 14:07         ` Wisam Monther
2020-04-20 12:24           ` Tom Barbette
2020-04-20 13:48             ` Yan Lei
2020-04-21  8:59               ` Tom Barbette
2020-04-21 12:30                 ` Raslan Darawsheh [this message]
2020-04-24 10:12                   ` Tom Barbette
2020-04-24 12:40                     ` Yan Lei

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AM0PR05MB670713AC7CC0003011D81E7CC2D50@AM0PR05MB6707.eurprd05.prod.outlook.com \
    --to=rasland@mellanox.com \
    --cc=barbette@kth.se \
    --cc=l.yan@epfl.ch \
    --cc=thomas@monjalon.net \
    --cc=users@dpdk.org \
    --cc=wisamm@mellanox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).