From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 11440A0527 for ; Wed, 8 Jul 2020 19:23:15 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C30F61DB89; Wed, 8 Jul 2020 19:23:14 +0200 (CEST) Received: from dfw-mailout20.raytheon.com (dfw-mailout20.raytheon.com [199.46.199.221]) by dpdk.org (Postfix) with ESMTP id 58BCB1DA87 for ; Wed, 8 Jul 2020 19:23:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raytheon.com; h=from : to : subject : date : message-id : content-type : content-transfer-encoding : mime-version; s=dkim2017; bh=JubdFUIupK8w5OakBSk6pUon72OjTlH18VXNHJzKxCA=; b=QnZvywDZJfymz3yVYeqOIAP6Bml8rjdR1eSse1ibkloCh0JVclSTxa27KxbAN77atP3R RIaCzLVVRUVU6ZBMkAqeL/fgDcRO8yrIL9PM48fbSCmaKWmiYJLNU6phICoGfxDxHRa4 2Lk1wyIGbl7cMdL3H87sfMXKzGyMFRJGZgdY3soTJmdYOFPfdbEnqrWombUr+iATVp5V 2SM1EaWmiZyZ4K7v6JLZBw7M6ve+XY5tC4Yyin/S3FOf3yOVg3qw5jEyD4EJ+rgNwSzF 1vTaS3gSxa7deLOgPYoeEDAjTReWMnfa9bgI61LQNUDdEjmPL7NsjBwrckHNKoa6AMIJ TQ== Received: from tx-mailout10.rtnmail.ray.com (tx-mailout10.rtnmail.ray.com [138.126.127.234]) by dfw-mailout20.ext.ray.com (8.16.0.27/8.16.0.27) with ESMTPS id 068HNBfe007617 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 8 Jul 2020 17:23:11 GMT Received: from USG02-CY1-obe.outbound.protection.office365.us ([23.103.199.180]) by tx-mailout10.rtnmail.ray.com (8.16.0.27/8.16.0.27) with ESMTPS id 068HN6Up046120 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Wed, 8 Jul 2020 17:23:11 GMT ARC-Seal: i=1; a=rsa-sha256; s=arcselector5401; d=microsoft.com; cv=none; b=t9pD+yjDuTStBCbZoYmtRs2gT+eyPVfy3NWzYZuXx/7HKefKO8txFkurmR0gq6CwvkK+T6sIfWspxpgfKXjhKZFSTQ2wbyVDQ8toiyxQmnhX9qLGQnYOg75OdYkh1D0SFUmUT7Ccr/t9CKsp6bDmHN9tWsk3yV3fcvF2j3pfJr+MdAiRxLW6AEr4OTZDvt5f4U1SCN0bc9xgSkjMcw/QUkC6bXhFvKWUJ2Hm7f76sdTiyBww+SFblZVvDSSdZFlgQtJP8IgwyEsr0XgHBz50LbUHwNIvYVo9Tdi1busQW7qk9LebZ/MgfItuNUMZ4+NxAHpa9+X0PUuf4Q9k7MuQPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector5401; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JubdFUIupK8w5OakBSk6pUon72OjTlH18VXNHJzKxCA=; b=QdewcvqyiMbhNs0EAf8P60d66B+OLIzlnwAOrJH13Uj+o9d7oYLEHpMg3dF+f1hgvFFros/TtHyRTtPibJDGHdSf6luakmF65ZMGZdzMmr4hws9YkFruiDNfRr9BltW9wNQ6oClnnezea5fNQ9hHzwYWgn6D9KNqOkm322EBri4BXIF1XSEzyxff/NzNzMkjkmPFucPHQaJILrue5SXMOPo5cSP66aNUEToLNsytT3aPBZ9we4beBbKwzWLuLLOKm4L/TyPf0uYXdbFz01Pwkk69USqLvkaifiMMbVVOnLO4l8EC72oGxvx0hZojNMaQGAree38terVokv1RETEq8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=raytheon.com; dmarc=pass action=none header.from=raytheon.com; dkim=pass header.d=raytheon.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raytheon.onmicrosoft.com; s=selector1-raytheon-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JubdFUIupK8w5OakBSk6pUon72OjTlH18VXNHJzKxCA=; b=zFN+NHAPmjH9MZ4MvbYifbTIi44VQgkICRohvyxc3rRiBF0JIO9mgTc97/y+Mdptib36pBGhvVxa1HlKzBrWMjOglXpbIPe88Fafs1Iujb8tiN1kyEFtmlQhENOaTz3faycA50M5gHcTwxscqWWoE77aRDRY+2YYiHrlGdS1fVM= Received: from DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM (52.145.10.7) by DM3P110MB0234.NAMP110.PROD.OUTLOOK.COM (23.103.33.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3131.26; Wed, 8 Jul 2020 17:23:02 +0000 Received: from DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM ([fe80::acdf:5f25:31d2:f1f]) by DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM ([fe80::acdf:5f25:31d2:f1f%4]) with mapi id 15.20.3131.037; Wed, 8 Jul 2020 17:23:02 +0000 From: Bev SCHWARTZ To: "users@dpdk.org" Thread-Topic: Significant performance degradation when using tx buffers rather than rte_eth_tx_burst Thread-Index: AQHWVUo5xfzvTyaToU2KoY92P+U7iQ== Date: Wed, 8 Jul 2020 17:23:01 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=raytheon.com; x-originating-ip: [128.89.58.140] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fd2dc12e-9be0-4f14-2d00-08d823639166 x-ms-traffictypediagnostic: DM3P110MB0234: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:1417; x-forefront-prvs: 04583CED1A x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(366004)(186003)(66476007)(76116006)(66946007)(66556008)(64756008)(9686003)(55016002)(2906002)(5660300002)(52536014)(6916009)(71200400001)(26005)(6506007)(86362001)(8936002)(8676002)(66446008)(498600001)(33656002)(7696005); DIR:OUT; SFP:1102; x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: raytheon.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: fd2dc12e-9be0-4f14-2d00-08d823639166 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jul 2020 17:23:01.8323 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: e0237e42-92d1-4357-bd3b-ce855a2b3eba X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM3P110MB0234 X-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-07-08_15:2020-07-08,2020-07-08 signatures=0 X-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-07-08_15:2020-07-08,2020-07-08 signatures=0 X-DMZ-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 lowpriorityscore=0 mlxlogscore=660 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2004280000 definitions=main-2007080110 X-DMZ-Spam-Reason: mlx Subject: [dpdk-users] Significant performance degradation when using tx buffers rather than rte_eth_tx_burst X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" I am writing a bridge using DPDK, where I have traffic read from one port t= ransmitted to the other. Here is the core of the program, based on basicfw= d.c.=0A= =0A= while (!force_quit) {=0A= nb_rx =3D rte_eth_rx_burst(rx_port, rx_queue, bufs, BURST_SIZE);=0A= for (i =3D 0; i < nb_rx; i++) {=0A= /* inspect packet */=0A= }=0A= nb_tx =3D rte_eth_tx_burst(tx_port, tx_queue, bufs, nb_rx);=0A= for (i =3D nb_tx; i < nb_rx; i++) {=0A= rte_pktmbuf_free(bufs[i]);=0A= }=0A= }=0A= =0A= (A bunch of error checking and such left out for brevity.)=0A= =0A= This worked great, I got bandwidth equivalent to using a Linux Bridge.=0A= =0A= I then tried using tx buffers instead. (Initialization code left out for b= revity.) Here is the new loop.=0A= =0A= while (!force_quit) {=0A= nb_rx =3D rte_eth_rx_burst(rx_port, rx_queue, bufs, BURST_SIZE);=0A= for (i =3D 0; i < nb_rx; i++) {=0A= /* inspect packet */=0A= rte_eth_tx_buffer(tx_port, tx_queue, tx_buffer, bufs[i]);=0A= }=0A= rte_eth_tx_buffer_flush(tx_port, tx_queue, tx_buffer);=0A= }=0A= =0A= (Once again, error checking left out for brevity.)=0A= =0A= I am running this on 8 cores, each core has its own loop. (tx_buffer is cr= eated for each core.)=0A= =0A= If I have well balanced traffic across the cores, then my performance goes = down, about 5% or so. If I have unbalanced traffic such as all traffic com= ing from a single flow, my performance goes down 80% from about 10 gbs to 2= gbs.=0A= =0A= I want to stress that the ONLY thing that changed in this code is changing = how I transmit packets. Everything else is the same.=0A= =0A= Any idea why this would cause such a degradation in bit rate?=0A= =0A= -Bev=