From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 53156A0540 for ; Mon, 13 Jul 2020 19:18:59 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 059FF1D543; Mon, 13 Jul 2020 19:18:59 +0200 (CEST) Received: from dfw-mailout10.raytheon.com (dfw-mailout10.raytheon.com [199.46.199.220]) by dpdk.org (Postfix) with ESMTP id 31C411D538 for ; Mon, 13 Jul 2020 19:18:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raytheon.com; h=from : to : cc : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version : subject; s=dkim2017; bh=iFb+xlJ00J9ZJ9TqCC7sExMQYUEDMgV159Wc2fu5yB0=; b=WrTJcv5pldLtbJ2Krjwk/oDMo5BN2LOV7Mh1oxT7D4mVbhaQdfIBTwIRDWUUi9dgYLdE 1xxhTVdsbudd1fk2ZFctKp++9QTCNmjJetmgVqlJ5XeVDmekHtW+5FTtnxfPFxxZeZBg eDnLNlDJiKjEH6DFFQq3P993TATF7KdzTwXzTTklNuubqp1/zTgcGR4rFwZpzXyD3yho Gnu3V5NQTS79wDqMqR2Jaup1Tc6F4OiB36NTGaCzUcHGwa0Im/kCBdpcB6h5UOl9GQev FhxhsTgSa1/EsRqD+klomXDjdZSH4yRgm63k85XxippfgwmBwj3vXNn7iGqJaoPreI97 LA== Received: from tx-mailout10.rtnmail.ray.com (tx-mailout10.rtnmail.ray.com [138.126.127.234]) by dfw-mailout10.ext.ray.com (8.16.0.27/8.16.0.27) with ESMTPS id 06DHItb5007227 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 13 Jul 2020 17:18:55 GMT Received: from USG02-CY1-obe.outbound.protection.office365.us ([23.103.199.175]) by tx-mailout10.rtnmail.ray.com (8.16.0.27/8.16.0.27) with ESMTPS id 06DHIs0M005058 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 13 Jul 2020 17:18:55 GMT ARC-Seal: i=1; a=rsa-sha256; s=arcselector5401; d=microsoft.com; cv=none; b=x4D3nXEfbWtdogKzz1b0zML/NAP9yu4DQTg/KnRBi0ebkdnj03tGqH4qfxk4SXVK1Y2vYiGs/r1Ot06Cd2UFRPelgrMsP3aSsaSCR6/2vDm7F9sLMsI0ZBEurTx1yeBokT4UWgnkMEqiIerD+4oehB8OkhBHMwft0dkVhW7Uj/lQ/C/unvbGBUGjhuoPpb+ZTTkNmlnxBXN4sJbp/OlcM8vss/bblVbE9WTPzF8wea0RFPfI1a9v+iNbUPE9k1WZJZ+D3zk8aRgQqbuA8Kh0zheQTlpvoX3ipO4UdjkLBMbsl4h3oeMzsMJqmND7FNSY4i1sYsVzZoWqxUwYrgZAHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector5401; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iFb+xlJ00J9ZJ9TqCC7sExMQYUEDMgV159Wc2fu5yB0=; b=zXhhPEUb1INN2Gw619Rlk8G8kPbaoRXVin5k2OsByEVjG0L7MxdNxRJmmOYOshps/gi1hfwO6hs6TCCZVjktD1qhGIRDfGUnq1WBn5eLdchNoVe0Fni/CIKoR3llHby1DcPlhL6tnNPQZO6Zud6IJUlWA9wHn3pQBg9bpBJm42EosDmi+Y/hNEwhDOZhM8y+TQ6xpWWGdwqQdH7z8A92yRedFp7JbhU/VP0g6VaMMvEmwBjobdNv9BS4RPiUp860DsXU+T9q+r7SGFUuprWffS2sKIFUfcUPpzs2d/HsqVSctPLu2xceKZQM0nh0miU4hoDuHPghu0LVTQK8eZInoQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=raytheon.com; dmarc=pass action=none header.from=raytheon.com; dkim=pass header.d=raytheon.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raytheon.onmicrosoft.com; s=selector1-raytheon-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iFb+xlJ00J9ZJ9TqCC7sExMQYUEDMgV159Wc2fu5yB0=; b=oyRbAgZbPX/9dpKue9fW52uQZ9XDQQNOJf6cApuUiEvXOVtGidbBM2jg/8oOKQvwDm775nM/qCFFHiBWJy0rljwxYm8yBldmSw1vdiacw0XS02D1jX6HdSDEd4RhGr+nFT69oDsVtdkHXldu66/kXtlubiXfUGx0QHwz3REEh74= Received: from DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM (52.145.10.7) by DM3P110MB0251.NAMP110.PROD.OUTLOOK.COM (23.103.33.154) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.31; Mon, 13 Jul 2020 17:18:49 +0000 Received: from DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM ([fe80::54ba:abc3:36c0:76f9]) by DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM ([fe80::54ba:abc3:36c0:76f9%6]) with mapi id 15.20.3153.032; Mon, 13 Jul 2020 17:18:49 +0000 From: Bev SCHWARTZ To: Manish Kumar , Suraj R Gupta CC: "users@dpdk.org" Thread-Topic: [External] Re: [dpdk-users] Significant performance degradation when using tx buffers rather than rte_eth_tx_burst Thread-Index: AQHWVUo5xfzvTyaToU2KoY92P+U7iaj+JfgAgAbuFYCAALLDDg== Date: Mon, 13 Jul 2020 17:18:49 +0000 Message-ID: References: , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=raytheon.com; x-originating-ip: [128.89.58.220] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e9d21fc0-409a-4d37-ba73-08d82750cefd x-ms-traffictypediagnostic: DM3P110MB0251: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:1751; x-forefront-prvs: 04631F8F77 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(366004)(71200400001)(52536014)(26005)(6506007)(83380400001)(53546011)(186003)(110136005)(55016002)(2906002)(9686003)(4326008)(76116006)(66446008)(66476007)(66556008)(64756008)(8676002)(66946007)(8936002)(498600001)(33656002)(5660300002)(86362001)(7696005); DIR:OUT; SFP:1102; x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: raytheon.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM3P110MB0524.NAMP110.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: e9d21fc0-409a-4d37-ba73-08d82750cefd X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Jul 2020 17:18:49.4188 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: e0237e42-92d1-4357-bd3b-ce855a2b3eba X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM3P110MB0251 X-CC: manish.jangid08@gmail.com, surajrgupta@iith.ac.in, users@dpdk.org X-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-07-13_15:2020-07-13,2020-07-13 signatures=0 X-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-07-13_15:2020-07-13,2020-07-13 signatures=0 X-DMZ-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 lowpriorityscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-2006250000 definitions=main-2007130123 X-DMZ-Spam-Reason: mlx Subject: Re: [dpdk-users] Significant performance degradation when using tx buffers rather than rte_eth_tx_burst X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" I am writing a bridge program. Originally, I based my implementation on sk= eleton/basicfwd.c. I next wanted to support using multi-core, so I found l= 2fwd.c as a simple model for tying queues to cores. However, l2fwd.c uses = rte_eth_tx_buffer. Not understanding enough about dpdk, I switched over to= using rte_eth_tx_buffer because I wrongly thought that it had to be used w= ith multi-core. I have changed my code back to using rte_eth_tx_burst, and that has solved = my problem. However, on very unbalanced traffic, using rte_eth_tx_buffer c= aused an 80% performance degradation. That seems rather extreme for such a= small change, so I was inquiring to see if people understood why. And giv= en this degradation, I'm surprised that l2fwd uses rte_eth_tx_buffer instea= d of rte_eth_tx_burst. -Bev ________________________________________ From: Manish Kumar Sent: Monday, July 13, 2020 2:32 AM To: Suraj R Gupta Cc: Bev SCHWARTZ; users@dpdk.org Subject: [External] Re: [dpdk-users] Significant performance degradation wh= en using tx buffers rather than rte_eth_tx_burst I agree with Suraj on the same. @Bev : Were you trying to use rte_eth_tx_bu= ffer function as part of just an experiment ? As per your email you already= got performance with the rte_eth_tx_burst function. Regards Manish On Wed, Jul 8, 2020 at 1:42 PM Suraj R Gupta > wrote: Hi bev, If my understanding is right, rte_eth_tx_burst transmits output packets immediately with a specified number of packets. While, 'rte_eth_tx_buffer' buffers the packet in the queue of the port, the packets would be transmitted only when buffer is or rte_eth_tx_buffer_flush is called. Since you are buffering packets one by one and then you are calling flush, this may have contributed to the delay. Thanks and Regards Suraj R Gupta On Wed, Jul 8, 2020 at 10:53 PM Bev SCHWARTZ > wrote: > I am writing a bridge using DPDK, where I have traffic read from one port > transmitted to the other. Here is the core of the program, based on > basicfwd.c. > > while (!force_quit) { > nb_rx =3D rte_eth_rx_burst(rx_port, rx_queue, bufs, BURST_SIZE); > for (i =3D 0; i < nb_rx; i++) { > /* inspect packet */ > } > nb_tx =3D rte_eth_tx_burst(tx_port, tx_queue, bufs, nb_rx); > for (i =3D nb_tx; i < nb_rx; i++) { > rte_pktmbuf_free(bufs[i]); > } > } > > (A bunch of error checking and such left out for brevity.) > > This worked great, I got bandwidth equivalent to using a Linux Bridge. > > I then tried using tx buffers instead. (Initialization code left out for > brevity.) Here is the new loop. > > while (!force_quit) { > nb_rx =3D rte_eth_rx_burst(rx_port, rx_queue, bufs, BURST_SIZE); > for (i =3D 0; i < nb_rx; i++) { > /* inspect packet */ > rte_eth_tx_buffer(tx_port, tx_queue, tx_buffer, bufs[i]); > } > rte_eth_tx_buffer_flush(tx_port, tx_queue, tx_buffer); > } > > (Once again, error checking left out for brevity.) > > I am running this on 8 cores, each core has its own loop. (tx_buffer is > created for each core.) > > If I have well balanced traffic across the cores, then my performance goe= s > down, about 5% or so. If I have unbalanced traffic such as all traffic > coming from a single flow, my performance goes down 80% from about 10 gbs > to 2gbs. > > I want to stress that the ONLY thing that changed in this code is changin= g > how I transmit packets. Everything else is the same. > > Any idea why this would cause such a degradation in bit rate? > > -Bev -- Thanks and Regards Suraj R Gupta -- Thanks Manish Kumar