From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 943E4A0598 for ; Fri, 10 Apr 2020 20:11:18 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CB4991C0CB; Fri, 10 Apr 2020 20:11:17 +0200 (CEST) Received: from smtp5.epfl.ch (smtp5.epfl.ch [128.178.224.8]) by dpdk.org (Postfix) with ESMTP id 157001C069 for ; Fri, 10 Apr 2020 20:11:17 +0200 (CEST) Received: (qmail 8740 invoked by uid 107); 10 Apr 2020 18:11:16 -0000 Received: from ax-snat-224-179.epfl.ch (HELO ewa08.intranet.epfl.ch) (192.168.224.179) (TLS, AES256-GCM-SHA384 cipher) by mail.epfl.ch (AngelmatoPhylax SMTP proxy) with ESMTPS; Fri, 10 Apr 2020 20:11:16 +0200 X-EPFL-Auth: X7KAKIDABXmAEe3AeDQNllfoDYFSUQjkrvDAFUEUVusH9h6gjss= Received: from ewa09.intranet.epfl.ch (128.178.224.180) by ewa08.intranet.epfl.ch (128.178.224.179) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Fri, 10 Apr 2020 20:11:16 +0200 Received: from ewa09.intranet.epfl.ch ([fe80::796d:b166:b908:8a7a]) by ewa09.intranet.epfl.ch ([fe80::796d:b166:b908:8a7a%3]) with mapi id 15.01.1913.007; Fri, 10 Apr 2020 20:11:15 +0200 From: Yan Lei To: "users@dpdk.org" Thread-Topic: [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec Thread-Index: AQHWD18w6DXuHcd1MEWNCNEXOo4dZA== Date: Fri, 10 Apr 2020 18:11:15 +0000 Message-ID: <2cb8c79c6e0a4829996f7a3b56386e89@epfl.ch> Accept-Language: en-US, fr-CH Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.179.253.71] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than 4K per sec X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi, I am doing some study that requires inserting more than 1 million flow rule= s per second to the NIC. And I runs DPDK 19.11 on a ConnectX-5 NIC. But I only managed to create around 3.3K rules per second. Below is the cod= e I used to measure the insertion rate: uint16_t mask =3D UINT16_MAX; uint64_t timer_start =3D rte_get_tsc_cycles(); for (int udp =3D 0; udp < num_rules; udp++) // just a simple wrapper of rte_flow_validate() & rte_flow_create() // (Removing validation seems to have little impact on performance) // Each rule basically assigns udp packets with specific dst port value= to a RX queue // 1st arg =3D> NIC port // 2nd arg =3D> Priority (This doesn't matter to insertion rate accord= ing to my observation) // 3rd arg =3D> dst udp port spec // 4th arg =3D> dst udp port mask // 5th arg =3D> queue index generate_dst_udp_flow(0, 1, udp % UINT16_MAX, mask, udp % 12); uint64_t timer_val =3D rte_get_tsc_cycles() - timer_start; printf("[BENCH] Create %d udp flow takes %ld us\n", num_rules, timer_val * 1000000 / rte_get_tsc_hz()); With 60000 rules I got [BENCH] Create 60000 udp flow takes 17821419 us. S= o 300us for one insertion.... Which is too slow... According to the mlx5 PMD manual (http://doc.dpdk.org/guides-19.11/nics/mlx= 5.html), insertion rate should be much higher: "Flow insertion rate of more then million flows per second, when using Dire= ct Rules." And this has been introduced since DPDK 19.05 (See the release note http://= doc.dpdk.org/guides-19.05/rel_notes/release_19_05.html#new-features and th= e patch note here https://mails.dpdk.org/archives/dev/2019-February/125303.= html). Did I miss anything? How can I get the promised 1 million flows per sec? My setup is as following: - CPU: E5-2697 v3 (14 cores, SMT disabled, CPU frequency fixed @ 2.6 GHz) - NIC: Mellanox MCX515A-CCAT (installed on PCIe Gen3 x16) - DPDK: 19.11 - OFED: 4.7-3.2.9.0 with upstream libs (I also tried standalone RDMA-CORE:= v28.0 instead of the one in OFED but got similar results) - Kernel: 4.15 - OS: Ubuntu 18.04 - Firmware: 16.26.1040 The firmware/driver/dpdk are tuned in the same way as here (http://fast.dpd= k.org/doc/perf/DPDK_19_11_Mellanox_NIC_performance_report.pdf). Your feedbacks will be much appreciated. Thanks, Lei