DPDK usage discussions
 help / color / mirror / Atom feed
From: Wisam Monther <wisamm@mellanox.com>
To: Thomas Monjalon <thomas@monjalon.net>, Yan Lei <l.yan@epfl.ch>
Cc: "users@dpdk.org" <users@dpdk.org>
Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec
Date: Sun, 19 Apr 2020 14:07:57 +0000	[thread overview]
Message-ID: <AM0PR05MB6610A4740DB9AA55F80C9391A9D70@AM0PR05MB6610.eurprd05.prod.outlook.com> (raw)
In-Reply-To: <6171086.9CP3fYhb5E@thomas>

Hey Yan,

For FW steering there is HW limitation on number of flows that can be added on it.
Which is table 0, and I think it's 65536

But to get Millions of rules use --group=1 which is SW steering.

Moreover make sure you have enough memory in the app to have good insertion rate.

If you have enough 1G huge pages then it's ok.
If you are working with 2M pages your command should be like this:

sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000

BRs,
Wisam Jaddo

>-----Original Message-----
>From: Thomas Monjalon <thomas@monjalon.net>
>Sent: Sunday, April 19, 2020 4:58 PM
>To: Yan Lei <l.yan@epfl.ch>
>Cc: users@dpdk.org; Wisam Monther <wisamm@mellanox.com>
>Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K
>per sec
>
>+Cc Wisam
>
>16/04/2020 17:32, Yan Lei:
>> Hi Thomas,
>>
>>
>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3.
>>
>>
>> TL;DR
>>
>>
>> 1. I was only able to generate 3K rules per second.
>>
>> 2. The maximum number of distinct rules the NIC can support seems to be
>65536.
>>
>>
>> How can I increase the insertion rate? Any firmware/driver config I need to
>tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch defaults to
>generate 4 million distinct flows though...
>>
>>
>> Thanks in advance!
>>
>>
>>
>> Initially, running
>>
>>
>> ```
>>
>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>> --ether --ipv4 --udp --queue --flows-count=1000000
>>
>> ```
>>
>>
>> failed after a few seconds and it gave
>>
>>
>> ```
>> Flow can't be created 1 message: hardware refuses to create flow
>> EAL: Error - exiting with code: 1
>>   Cause: error in creating flow
>> ```
>>
>>
>> Then I added a small debug patch (attached) and it showed that the error
>happens when creating the 65536th flow rule.
>>
>>
>> ```
>> Flow can't be created 1 message: hardware refuses to create flow
>> EAL: Error - exiting with code: 1
>>   Cause: error in creating flow,flows generated: 65536 ```
>>
>>
>> My guess is that the NIC can only accept 65536 concurrent rules. Once I
>changed the outer ip mask to 0xffff, the above command runs fine.
>>
>>
>> To see how many rules I can generate per second. I ran (with the outer
>> ip mask 0xffff)
>>
>>
>> ```
>>
>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>> --ether --ipv4 --udp --queue --flows-count=65536
>>
>> ```
>>
>>
>> and it gives
>>
>>
>> ```
>>
>> :: Total flow insertion rate -> 3.015922 K/Sec
>> :: The time for creating 65536 in flows 21.730005 seconds
>> :: EAGAIN counter = 0
>> ```
>> So 3 rules per sec. Which is close to what I observed before.
>>
>> ```
>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress
>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives
>>
>> ```
>> :: Total flow insertion rate -> 0.949381 K/Sec
>> :: The time for creating 100000 in flows 105.331842 seconds
>> :: EAGAIN counter = 0
>> ```
>> Have no idea why it's only 1k/sec in this case...
>>
>> Thanks and cheers,
>> Lei
>>
>>
>> ________________________________
>> From: users <users-bounces@dpdk.org> on behalf of Yan Lei
>> <l.yan@epfl.ch>
>> Sent: Tuesday, April 14, 2020 1:20 PM
>> To: Thomas Monjalon
>> Cc: users@dpdk.org
>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
>> than 4K per sec
>>
>> Hi Thomas,
>>
>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3).
>>
>> Cheers,
>> Lei
>> ________________________________
>> From: Thomas Monjalon <thomas@monjalon.net>
>> Sent: Tuesday, April 14, 2020 12:12:28 PM
>> To: Yan Lei
>> Cc: users@dpdk.org
>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less
>> than 4K per sec
>>
>> Hi,
>>
>> 10/04/2020 20:11, Yan Lei:
>> > I am doing some study that requires inserting more than 1 million
>> > flow rules per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5
>NIC.
>> >
>> > But I only managed to create around 3.3K rules per second.
>> > Below is the code I used to measure the insertion rate:
>>
>> Please could you review this new application designed for such measure?
>>
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
>>
>hes.dpdk.org%2Fpatch%2F68058%2F&amp;data=02%7C01%7Cwisamm%40m
>ellanox.c
>>
>om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d149
>256f46
>>
>1b%7C0%7C0%7C637229014854391590&amp;sdata=dxxEmm4DWoMPeNGyM
>FaYgk%2BjSE
>> %2FwVKLnYAwQ7QhjKGc%3D&amp;reserved=0
>>
>> Any feedback about the above patch is welcome. Feel free to try and review
>it.
>
>
>


  reply	other threads:[~2020-04-19 14:08 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-10 18:11 Yan Lei
2020-04-14 10:12 ` Thomas Monjalon
2020-04-14 11:20   ` Yan Lei
2020-04-16 15:32     ` Yan Lei
2020-04-19 13:57       ` Thomas Monjalon
2020-04-19 14:07         ` Wisam Monther [this message]
2020-04-20 12:24           ` Tom Barbette
2020-04-20 13:48             ` Yan Lei
2020-04-21  8:59               ` Tom Barbette
2020-04-21 12:30                 ` Raslan Darawsheh
2020-04-24 10:12                   ` Tom Barbette
2020-04-24 12:40                     ` Yan Lei

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=AM0PR05MB6610A4740DB9AA55F80C9391A9D70@AM0PR05MB6610.eurprd05.prod.outlook.com \
    --to=wisamm@mellanox.com \
    --cc=l.yan@epfl.ch \
    --cc=thomas@monjalon.net \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).