From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from COL004-OMC4S17.hotmail.com (col004-omc4s17.hotmail.com [65.55.34.219]) by dpdk.org (Postfix) with ESMTP id 5558538EB for ; Mon, 30 Jan 2017 17:02:16 +0100 (CET) Received: from EUR01-HE1-obe.outbound.protection.outlook.com ([65.55.34.199]) by COL004-OMC4S17.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Mon, 30 Jan 2017 08:02:15 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=OBHbJxMYc2F4nz3Ku6cbTqN0sosg4QJGCNAS9Wg26R4=; b=nGvciHmRN5mdTm/xoOWY0QjEx6fqcEFr3z/TBkDHf+rzCOsQkYTFYcCXdnqlraQpeRoVu9pRSYITAl3CJBmsGwHKdXgSwbI5/sVEZbXwYK14J7d3XpVtKmww8Y+OoLP6Bd2OcOV30UxJ7cI3dWaDPwZI645ODbUq2Wd0ROYITmfGZ/ln6uvdCV5O0f4HXHRr65MdzRwLDlt3JcCprzjgoYcZcd/8V60Y0ActvZqy5p4bWCgsF72/v825jktrzuqdjQOhdR243XN3i3BdH+DS5efzYUdEfu8Vrc6VdzFMaG8ViYLoxQXerAS79wqGwlpq5/KduWztHcLqF345wYkaBg== Received: from HE1EUR01FT051.eop-EUR01.prod.protection.outlook.com (10.152.0.57) by HE1EUR01HT041.eop-EUR01.prod.protection.outlook.com (10.152.1.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.803.8; Mon, 30 Jan 2017 16:02:11 +0000 Received: from DB6PR1001MB1416.EURPRD10.PROD.OUTLOOK.COM (10.152.0.58) by HE1EUR01FT051.mail.protection.outlook.com (10.152.1.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.874.2 via Frontend Transport; Mon, 30 Jan 2017 16:02:11 +0000 Received: from DB6PR1001MB1416.EURPRD10.PROD.OUTLOOK.COM ([10.171.78.136]) by DB6PR1001MB1416.EURPRD10.PROD.OUTLOOK.COM ([10.171.78.136]) with mapi id 15.01.0874.019; Mon, 30 Jan 2017 16:02:11 +0000 From: Peter Keereweer To: "Wiles, Keith" CC: "users@dpdk.org" Thread-Topic: [dpdk-users] What to do after rte_eth_tx_burst: free or send again remaining packets? Thread-Index: AQHSeZ89QhXZzOqpSEWE4fljFDCT1aFOTeesgAC074CAAiJJtA== Date: Mon, 30 Jan 2017 16:02:11 +0000 Message-ID: References: , In-Reply-To: Accept-Language: nl-NL, en-US Content-Language: nl-NL X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=hotmail.com; x-incomingtopheadermarker: OriginalChecksum:85B78430D9CE2450ED3869135883EDE06C5ECFE7E292B9555150E65F51D37979; UpperCasedChecksum:3BAF474B9F6D2DE42C1B7F044C0F833ECA548F89C44EE8EA213B671DB4D2D8E2; SizeAsReceived:8083; Count:40 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [6LrSMjsf8SSF0CFnhtvC/Pghb0olA6nK] x-incomingheadercount: 40 x-eopattributedmessage: 0 x-microsoft-exchange-diagnostics: 1; HE1EUR01HT041; 7:M8ppMVEdsTyV+T3kww6Mqtr9Rg8EOFkozyOYINwMgubzseyCw0upQoFiHwMaX4D7yqdKBYc0CSnUWiPaBadAYtai6hbBMh1VhMPYQdckUXFbAfluJ99HINaKI+R47sEaozjwNSGJtRQEzkA+p/IsSercRDBypLS2w7nT+UJv3LXXMcVbHBIotTprPh/cFVsmAhqgWisR8EEpv7oE61fNY4eqBffj8E5Afu34RUfJdgOy9fjKUJ3Fc1lqj9pNU+eN8BcoCcbUdLlQiKZMacp6UaLb3FXtB3DlLQK0hWsA5NkHc6BYcxh/VbkIgdjq0qbvnGjzN3fllleELUI1HQNKQzwgT3f9YbVfk5RPR3jRJ5vIA3fhYaJ8LmTNILd0rvkOqml0XpkS2BNqhiMguOQfV0nNTx8YbidiXd5DWuMu3ATTPbAjwUJ3yocq8pFOQQDwZuXc3sFF3HjPc9jj365EEQ== x-forefront-antispam-report: EFV:NLI; SFV:NSPM; SFS:(10019020)(98900005); DIR:OUT; SFP:1102; SCL:1; SRVR:HE1EUR01HT041; H:DB6PR1001MB1416.EURPRD10.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; x-ms-office365-filtering-correlation-id: 78938834-fc5e-4076-e36f-08d4492959a9 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(1601124038)(5061506344)(1603103113)(1601125047)(1701031023); SRVR:HE1EUR01HT041; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(444111334)(444112120)(432015012)(82015046); SRVR:HE1EUR01HT041; BCL:0; PCL:0; RULEID:; SRVR:HE1EUR01HT041; x-forefront-prvs: 0203C93D51 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Jan 2017 16:02:11.1564 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1EUR01HT041 X-OriginalArrivalTime: 30 Jan 2017 16:02:15.0278 (UTC) FILETIME=[3993C8E0:01D27B12] Subject: Re: [dpdk-users] What to do after rte_eth_tx_burst: free or send again remaining packets? X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jan 2017 16:02:17 -0000 Hi Keith, Thanks a lot for your response! Based on your information I have tested dif= ferent burst sizes in the Load Balancer application (an let the TX ring siz= e unchanged). One can configure the read / write burst sizes of the NIC and= the software queues as a command line option. The default value of all bur= st size is equal to 144. If I configure all read/write burst sizes as 32, e= very packet will be transmitted by the TX core and no packets are dropped. = But is this a valid solution? It seems to work, but it feels a little bit s= trange to decrease the burst size from 144 to 32. Another solution is implementing a while loop (like in _send_burst_fast in = pktgen), so every packet will be transmitted. This solution seems to work t= oo, but the same question here, is this a valid solution? The strange feeli= ng about this solution is that basically the same happens in the ixgbe driv= er code (ixgbe_rxtx.c): uint16_t ixgbe_xmit_pkts_simple(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) { uint16_t nb_tx; /* Try to transmit at least chunks of TX_MAX_BURST pkts */ if (likely(nb_pkts <=3D RTE_PMD_IXGBE_TX_MAX_BURST)) return tx_xmit_pkts(tx_queue, tx_pkts, nb_pkts); /* transmit more than the max burst, in chunks of TX_MAX_BURST */ nb_tx =3D 0; while (nb_pkts) { uint16_t ret, n; n =3D (uint16_t)RTE_MIN(nb_pkts, RTE_PMD_IXGBE_TX_MAX_BURST); ret =3D tx_xmit_pkts(tx_queue, &(tx_pkts[nb_tx]), n); nb_tx =3D (uint16_t)(nb_tx + ret); nb_pkts =3D (uint16_t)(nb_pkts - ret); if (ret < n) break; } return nb_tx; } To be honest, I don't know whether this piece of code is called if I use th= e rte_eth_tx_burst, but I expect something similar happens when rte_eth_tx_= burst uses another transmitting function in the ixgbe driver code. This whi= le loop in the ixgbe driver code is exactly doing same as using a while loo= p in combination with rte_eth_tx_burst. But if I don't use a while loop in = combination with the rte_eth_tx_burst (and a burst size of 144) it's not wo= rking (many packets are dropped), but if I implement this while loop it see= ms to work... I hope you can help me again with finding the best solution to solve this p= roblem! Peter Van: Wiles, Keith Verzonden: zaterdag 28 januari 2017 23:43 Aan: Peter Keereweer CC: users@dpdk.org Onderwerp: Re: [dpdk-users] What to do after rte_eth_tx_burst: free or send= again remaining packets? =A0 =20 > On Jan 28, 2017, at 1:57 PM, Peter Keereweer = wrote: >=20 > Hi! >=20 > Currently I'am running some tests with the Load Balancer Sample Applicati= on. I'm testing the Load Balancer Sample Application by sending packets wit= h pktgen. > I have a setup of 2 servers with each server containing a Intel 10Gbe 825= 99 NIC (connected to each other). I have configured the Load Balancer appli= cation to use 1 core for RX, 1 worker core and 1 TX core. The TX core sends= all packets back to the pktgen application. >=20 > With the pktgen I send 1024 UDP packets to the Load Balancer. Every packe= t processed by the worker core will be printed to the screen (I added this = code by myself). If I send 1024 UDP packets, 1008 ( =3D 7 x 144) packets wi= ll be printed to the screen. This is=A0 correct, because the RX core reads= packets with a burst size of 144. So if I send 1024 packets, I expect 1008= packets back in the pktgen application. But surprisingly I only receive 22= 4 packets instead of 1008 packets. After some research I found that that= =A0 224 packets is not just a random number, its 7 x 32 (=3D 224). So if th= e RX reads 7 x 144 packets, I get back 7 x 32 packets. After digging into t= he code from the Load Balancer application I found in 'runtime.c' in the 'a= pp_lcore_io_tx' function this code : >=20 > n_pkts =3D rte_eth_tx_burst( >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 port, >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 0, >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 lp->tx.mbuf_out[port].array, >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 (uint16_t) n_mbufs); >=20 > ... >=20 > if (unlikely(n_pkts < n_mbufs)) { >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 uint32_t k; >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 for (k =3D n_pkts; k < n_mbufs; k ++) { >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 struct rte_mbuf *pkt_to_fr= ee =3D lp->tx.mbuf_out[port].array[k]; >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 rte_pktmbuf_free(pkt_to_fr= ee); >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0 } >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 } >=20 > What I understand from this code is that n_mbufs 'packets' are send with = 'rte_eth_tx_burst' function. This function returns n_pkts, the number of pa= ckets that are actually send. If the actual number of packets send is small= er then n_mbufs (packets ready for=A0 send given to the rte_eth_tx_burst) = then all remaining packets, which are not send, are freed. In de the Load B= alancer application, n_mbufs is equal to 144. But in my case 'rte_eth_tx_bu= rst' returns the value 32, and not 144. So 32 packets are actually send=A0 = and the remaining packets (144 - 32 =3D 112) are freed. This is the reason= why I get 224 (7 x 32) packets back instead of 1008 (=3D 7 x 144). >=20 > But the question is: why are the remaining packets freed instead of tryin= g to send them again? If I look into the 'pktgen.c', there is a function '_= send_burst_fast' where all remaining packets are trying to be send again (i= n a while loop until they are all=A0 send) instead of freeing them (see co= de below) : >=20 > static __inline__ void > _send_burst_fast(port_info_t *info, uint16_t qid) > { >=A0=A0=A0=A0=A0=A0=A0=A0 struct mbuf_table=A0=A0 *mtab =3D &info->q[qid].t= x_mbufs; >=A0=A0=A0=A0=A0=A0=A0=A0 struct rte_mbuf **pkts; >=A0=A0=A0=A0=A0=A0=A0=A0 uint32_t ret, cnt; >=20 >=A0=A0=A0=A0=A0=A0=A0=A0 cnt =3D mtab->len; >=A0=A0=A0=A0=A0=A0=A0=A0 mtab->len =3D 0; >=20 >=A0=A0=A0=A0=A0=A0=A0=A0 pkts=A0=A0=A0 =3D mtab->m_table; >=20 >=A0=A0=A0=A0=A0=A0=A0=A0 if (rte_atomic32_read(&info->port_flags) & PROCES= S_TX_TAP_PKTS) { >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 while (cnt > 0) { >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 r= et =3D rte_eth_tx_burst(info->pid, qid, pkts, cnt); >=20 >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 p= ktgen_do_tx_tap(info, pkts, ret); >=20 >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 p= kts +=3D ret; >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 c= nt -=3D ret; >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 } >=A0=A0=A0=A0=A0=A0=A0=A0 } else { >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 while(cnt > 0) { >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 r= et =3D rte_eth_tx_burst(info->pid, qid, pkts, cnt); >=20 >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 p= kts +=3D ret; >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 c= nt -=3D ret; >=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 } >=A0=A0=A0=A0=A0=A0=A0=A0 } > }=20 >=20 > Why is this while loop (sending packets until they have all been send) no= t implemented in the 'app_lcore_io_tx' function in the Load Balancer applic= ation? That would make sense right? It looks like that the Load Balancer ap= plication makes an assumption that=A0 if not all packets have been send, t= he remaining packets failed during the sending proces and should be freed. The size of the TX ring on the hardware is limited in size, but you can adj= ust that size. In pktgen I attempt to send all packets requested to be sent= , but in the load balancer the developer decided to just drop the packets t= hat are not sent as the TX hardware ring or even a SW ring is full. This n= ormally means the core is sending packets faster then the HW ring on the NI= C can send the packets. It was just a choice of the developer to drop the packets instead of trying= again until the packets array is empty. One possible way to fix this is to= increase the size of the TX ring 2-4 time larger then the RX ring. This st= ill does not truly solve the problem it just moves it to the RX ring. The = NIC if is does not have a valid RX descriptor and a place to DMA the packet= into memory it gets dropped at the wire. BTW increasing the TX ring size a= lso means the these packets will not returned to the free pool and you can= exhaust the packet pool. The packets are stuck on the TX ring as done beca= use the threshold to reclaim the done packets is too high. Say you have 1024 ring size and the high watermark for flushing the done of= f the ring is 900 packets. Then if the packet pool is only 512 packets then= when you send 512 packets they will all be on the TX done queue and now yo= u are in a deadlock not being able to send a packet as they are all on the= TX done ring. This normally does not happen as the ring sizes or normally = much smaller then the number of TX packets or even RX packets. In pktgen I attempt to send all of the packets requested as it does not mak= e any sense for the user to ask to send 10000 packets and pktgen only send = some number less as the core sending the packets can over run the TX queue = at some point. I hope that helps. >=20 > I hope someone can help me with this questions. Thank you in advance!! >=20 > Peter Regards, Keith =