From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by dpdk.org (Postfix) with ESMTP id 15F115B2E for ; Fri, 1 Mar 2019 03:57:16 +0100 (CET) Received: by mail-lf1-f41.google.com with SMTP id h10so16861076lfc.12 for ; Thu, 28 Feb 2019 18:57:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=oQOUEGbzktewexNyYrSgTuBCTCzxzZQNmoVU23pG5wA=; b=bmlQ3a7Z/6wmP2CGtnzDmTHTtoEmWpc+yA/9NKlhvvPYDu11AYPQKxBXYhn/P+ksTj 4VvbOeq8YNL9NDfPgIj6E0LmWLgc/xIC4x8pQa90ieZ5s7lBCOkO9PwKaaiq97+Vav9I 1BfC6Ofgt8086P3+FCAQzQlMBK4jqm5Affd/XQa0YImhad0Vyy2DBz/QAovgk9hXKrr4 gPx8ZDObSYbGKsWC520IGcpIfwkNLmFsnLZS2d6rVxUNIeuxEBzGNQx8t/WwR1R2Rt99 7x2z6kAchwhN48On0OberP2TBdCSLtU6vvVOhLdUBNaEnTKWJjviI2ytzq2Ja+f+0XyC psEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=oQOUEGbzktewexNyYrSgTuBCTCzxzZQNmoVU23pG5wA=; b=Gnzm2eKH8/6fegTSlLD9eErv4a+kdgfZXADaWXWFt+IZX9zf6kKFJkC1Hh+Xpesgne Wq31C45jZT1ezbn7C9t9uGledvJbzetqv4We1ptgVVk/4MDnNGmjdIkvlsQS/eCH6pp6 ojRqwCq2nr3RtWNFOy9r10y1Mnc3YXerXuF5OgPeQcTm5V2fyYglF1WOaJ16LwjBXFUh m44/6ogO4E+fkM1mlxsCU0lVMCpNi2na3xRyF5hXflhviP7dCFsfA0NiSQzxhhXbRDOC N4tE/s1DzujHT6oBPcj2NhN6QFb50vfHis4p6PmcHrQ3BMAHD6INrCBiskWNBeib9EHJ 2hHw== X-Gm-Message-State: APjAAAV+quqhDh1GaWojctu3VzZpfMt25rlX5MEk65Qo2GJ2IL3+MPIu W8GIZEtl9LZYGQUfAKUFypY4cZ346hEKS0sYg/k= X-Google-Smtp-Source: APXvYqzVS0bbjHW0hXj+hyJ2ybaAAp6I5UhnGjQ1IWcUsSDT29MqY1NXbJYV05OgR22OThOQ/HYBZn3OZtHdubrHI5s= X-Received: by 2002:a19:c207:: with SMTP id l7mr1511941lfc.114.1551409035501; Thu, 28 Feb 2019 18:57:15 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Arvind Narayanan Date: Thu, 28 Feb 2019 20:57:04 -0600 Message-ID: To: Cliff Burdick Cc: users Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] rte_flow / hw-offloading is degrading performance when testing @ 100G X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Mar 2019 02:57:16 -0000 On Thu, Feb 28, 2019, 8:23 PM Cliff Burdick wrote: > What size packets are you using? I've only steered to 2 rx queues by IP > dst match, and was able to hit 100Gbps. That's with a 4KB jumboframe. > 64 bytes. Agreed this is small, what seems interesting is l3fwd is able to handle 64B but rte_flow suffers (a lot) - suggesting offloading is expensive?! I'm doing something similar, steering to different queues based off dst_ip. However, my tests have around 80 rules, each rule steering to one of the 20 rx_queues. I have a one-to-one rx_queue-to-core_id mapping. Arvind > On Thu, Feb 28, 2019, 17:42 Arvind Narayanan > wrote: > >> Hi, >> >> I am using DPDK 18.11 on Ubuntu 18.04, with Mellanox Connect X-5 100G >> EN (MLNX_OFED_LINUX-4.5-1.0.1.0-ubuntu18.04-x86_64). >> Packet generator: t-rex 2.49 running on another machine. >> >> I am able to achieve 100G line rate with l3fwd application (fr sz 64B) >> using the parameters suggested in their performance report. >> ( >> https://fast.dpdk.org/doc/perf/DPDK_18_11_Mellanox_NIC_performance_report.pdf >> ) >> >> However, as soon as I install rte_flow rules to steer packets to >> different queues and/or use rte_flow's mark action, the throughput >> reduces to ~41G. I also modified DPDK's flow_filtering example >> application, and am getting the same reduced throughput of around 41G >> out of 100G. But without rte_flow, it goes to 100G. >> >> I didn't change any OS/Kernel parameters to test l3fwd or the >> application that uses rte_flow. I also ensure the application is >> numa-aware and use 20 cores to handle 100G traffic. >> >> Upon further investigation (using Mellanox NIC counters), the drop in >> throughput is due to mbuf allocation errors. >> >> Is such performance degradation normal when performing hw-acceleration >> using rte_flow? >> Has anyone tested throughput performance using rte_flow @ 100G? >> >> Its surprising to see hardware offloading is degrading the >> performance, unless I am doing something wrong. >> >> Thanks, >> Arvind >> >