From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by dpdk.org (Postfix) with ESMTP id 689822BD5 for ; Mon, 19 Nov 2018 14:49:09 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Nov 2018 05:49:08 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,252,1539673200"; d="scan'208";a="107504725" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by fmsmga004.fm.intel.com with ESMTP; 19 Nov 2018 05:49:08 -0800 Received: from fmsmsx151.amr.corp.intel.com (10.18.125.4) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 19 Nov 2018 05:49:08 -0800 Received: from fmsmsx118.amr.corp.intel.com ([169.254.1.160]) by FMSMSX151.amr.corp.intel.com ([169.254.7.81]) with mapi id 14.03.0415.000; Mon, 19 Nov 2018 05:49:08 -0800 From: "Wiles, Keith" To: Kyle Larose CC: Harsh Patel , "users@dpdk.org" Thread-Topic: [dpdk-users] Query on handling packets Thread-Index: AQHUdzydRRkBFdv4fkKjO7j2RJcyb6VGGaEAgACGp4CAAAyagIABE0+AgAFRlQCAAnP3gIACAjmAgAC+mQCAAZQ9gIAAFqaAgARlHYCAAMRLgIACmg8A Date: Mon, 19 Nov 2018 13:49:07 +0000 Message-ID: <485F0372-7486-473B-ACDA-F42A2D86EF03@intel.com> References: <71CBA720-633D-4CFE-805C-606DAAEDD356@intel.com> <3C60E59D-36AD-4382-8CC3-89D4EEB0140D@intel.com> <76959924-D9DB-4C58-BB05-E33107AD98AC@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.254.186.128] Content-Type: text/plain; charset="us-ascii" Content-ID: <696885F2EFF3764987B0A9333C9E227F@intel.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-users] Query on handling packets X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 13:49:10 -0000 > On Nov 17, 2018, at 4:05 PM, Kyle Larose wrote: >=20 > On Sat, Nov 17, 2018 at 5:22 AM Harsh Patel wr= ote: >>=20 >> Hello, >> Thanks a lot for going through the code and providing us with so much >> information. >> We removed all the memcpy/malloc from the data path as you suggested and > ... >> After removing this, we are able to see a performance gain but not as go= od >> as raw socket. >>=20 >=20 > You're using an unordered_map to map your buffer pointers back to the > mbufs. While it may not do a memcpy all the time, It will likely end > up doing a malloc arbitrarily when you insert or remove entries from > the map. If it needs to resize the table, it'll be even worse. You may > want to consider using librte_hash: > https://doc.dpdk.org/api/rte__hash_8h.html instead. Or, even better, > see if you can design the system to avoid needing to do a lookup like > this. Can you return a handle with the mbuf pointer and the data > together? >=20 > You're also using floating point math where it's unnecessary (the > timing check). Just multiply the numerator by 1000000 prior to doing > the division. I doubt you'll overflow a uint64_t with that. It's not > as efficient as integer math, though I'm not sure offhand it'd cause a > major perf problem. >=20 > One final thing: using a raw socket, the kernel will take over > transmitting and receiving to the NIC itself. that means it is free to > use multiple CPUs for the rx and tx. I notice that you only have one > rx/tx queue, meaning at most one CPU can send and receive packets. > When running your performance test with the raw socket, you may want > to see how busy the system is doing packet sends and receives. Is it > using more than one CPU's worth of processing? Is it using less, but > when combined with your main application's usage, the overall system > is still using more than one? Along with the floating point math, I would remove all floating point math = and use the rte_rdtsc() function to use cycles. Using something like: uint64_t cur_tsc, next_tsc, timo =3D (rte_timer_get_hz() / 16); /* One 16th= of a second use 2/4/8/16/32 power of two numbers to make the math simple d= ivide */ cur_tsc =3D rte_rdtsc(); next_tsc =3D cur_tsc + timo; /* Now next_tsc the next time to flush */ while(1) { cur_tsc =3D rte_rdtsc(); if (cur_tsc >=3D next_tsc) { flush(); next_tsc +=3D timo; } /* Do other stuff */ } For the m_bufPktMap I would use the rte_hash or do not use a hash at all by= grabbing the buffer address and subtract the mbuf =3D (struct rte_mbuf *)RTE_PTR_SUB(buf, sizeof(struct rte_mbuf) + RTE_= MAX_HEADROOM); DpdkNetDevice:Write(uint8_t *buffer, size_t length) { struct rte_mbuf *pkt; uint64_t cur_tsc; pkt =3D (struct rte_mbuf *)RTE_PTR_SUB(buffer, sizeof(struct rte_mbuf) + R= TE_MAX_HEADROOM); /* No need to test pkt, but buffer maybe tested to make sure it is not nul= l above the math above */ pkt->pk_len =3D length; pkt->data_len =3D length; rte_eth_tx_buffer(m_portId, 0, m_txBuffer, pkt); cur_tsc =3D rte_rdtsc(); /* next_tsc is a private variable */ if (cur_tsc >=3D next_tsc) { rte_eth_tx_buffer_flush(m_portId, 0, m_txBuffer); /* hardcoded the queue = id, should be fixed */ next_tsc =3D cur_tsc + timo; /* timo is a fixed number of cycles to wait = */ } return length; } DpdkNetDevice::Read() { struct rte_mbuf *pkt; if (m_rxBuffer->length =3D=3D 0) { m_rxBuffer->next =3D 0; m_rxBuffer->length =3D rte_eth_rx_burst(m_portId, 0, m_rxBuffer->pmts, MA= X_PKT_BURST); if (m_rxBuffer->length =3D=3D 0) return std::make_pair(NULL, -1); } pkt =3D m_rxBuffer->pkts[m_rxBuffer->next++]; /* do not use rte_pktmbuf_read() as it does a copy for the complete packet= */ return std:make_pair(rte_pktmbuf_mtod(pkt, char *), pkt->pkt_len); } void DpdkNetDevice::FreeBuf(uint8_t *buf) { struct rte_mbuf *pkt; if (!buf) return; pkt =3D (struct rte_mbuf *)RTE_PKT_SUB(buf, sizeof(rte_mbuf) + RTE_MAX_HEA= DROOM); rte_pktmbuf_free(pkt); } When your code is done with the buffer, then convert the buffer address bac= k to a rte_mbuf pointer and call rte_pktmbuf_free(pkt); This should elimina= te the copy and floating point code. Converting my C code to C++ priceless = :-) Hopefully the buffer address passed is the original buffer address and has = not be adjusted. Regards, Keith