From: "Wiles, Keith" <keith.wiles@intel.com>
To: Kyle Larose <eomereadig@gmail.com>
Cc: Harsh Patel <thadodaharsh10@gmail.com>,
"users@dpdk.org" <users@dpdk.org>
Subject: Re: [dpdk-users] Query on handling packets
Date: Mon, 19 Nov 2018 13:49:07 +0000 [thread overview]
Message-ID: <485F0372-7486-473B-ACDA-F42A2D86EF03@intel.com> (raw)
In-Reply-To: <CAMFWN9nik1F5L=Ffy3s43eg=C2QEUzMjzmc-edLQRxXTRznczQ@mail.gmail.com>
> On Nov 17, 2018, at 4:05 PM, Kyle Larose <eomereadig@gmail.com> wrote:
>
> On Sat, Nov 17, 2018 at 5:22 AM Harsh Patel <thadodaharsh10@gmail.com> wrote:
>>
>> Hello,
>> Thanks a lot for going through the code and providing us with so much
>> information.
>> We removed all the memcpy/malloc from the data path as you suggested and
> ...
>> After removing this, we are able to see a performance gain but not as good
>> as raw socket.
>>
>
> You're using an unordered_map to map your buffer pointers back to the
> mbufs. While it may not do a memcpy all the time, It will likely end
> up doing a malloc arbitrarily when you insert or remove entries from
> the map. If it needs to resize the table, it'll be even worse. You may
> want to consider using librte_hash:
> https://doc.dpdk.org/api/rte__hash_8h.html instead. Or, even better,
> see if you can design the system to avoid needing to do a lookup like
> this. Can you return a handle with the mbuf pointer and the data
> together?
>
> You're also using floating point math where it's unnecessary (the
> timing check). Just multiply the numerator by 1000000 prior to doing
> the division. I doubt you'll overflow a uint64_t with that. It's not
> as efficient as integer math, though I'm not sure offhand it'd cause a
> major perf problem.
>
> One final thing: using a raw socket, the kernel will take over
> transmitting and receiving to the NIC itself. that means it is free to
> use multiple CPUs for the rx and tx. I notice that you only have one
> rx/tx queue, meaning at most one CPU can send and receive packets.
> When running your performance test with the raw socket, you may want
> to see how busy the system is doing packet sends and receives. Is it
> using more than one CPU's worth of processing? Is it using less, but
> when combined with your main application's usage, the overall system
> is still using more than one?
Along with the floating point math, I would remove all floating point math and use the rte_rdtsc() function to use cycles. Using something like:
uint64_t cur_tsc, next_tsc, timo = (rte_timer_get_hz() / 16); /* One 16th of a second use 2/4/8/16/32 power of two numbers to make the math simple divide */
cur_tsc = rte_rdtsc();
next_tsc = cur_tsc + timo; /* Now next_tsc the next time to flush */
while(1) {
cur_tsc = rte_rdtsc();
if (cur_tsc >= next_tsc) {
flush();
next_tsc += timo;
}
/* Do other stuff */
}
For the m_bufPktMap I would use the rte_hash or do not use a hash at all by grabbing the buffer address and subtract the
mbuf = (struct rte_mbuf *)RTE_PTR_SUB(buf, sizeof(struct rte_mbuf) + RTE_MAX_HEADROOM);
DpdkNetDevice:Write(uint8_t *buffer, size_t length)
{
struct rte_mbuf *pkt;
uint64_t cur_tsc;
pkt = (struct rte_mbuf *)RTE_PTR_SUB(buffer, sizeof(struct rte_mbuf) + RTE_MAX_HEADROOM);
/* No need to test pkt, but buffer maybe tested to make sure it is not null above the math above */
pkt->pk_len = length;
pkt->data_len = length;
rte_eth_tx_buffer(m_portId, 0, m_txBuffer, pkt);
cur_tsc = rte_rdtsc();
/* next_tsc is a private variable */
if (cur_tsc >= next_tsc) {
rte_eth_tx_buffer_flush(m_portId, 0, m_txBuffer); /* hardcoded the queue id, should be fixed */
next_tsc = cur_tsc + timo; /* timo is a fixed number of cycles to wait */
}
return length;
}
DpdkNetDevice::Read()
{
struct rte_mbuf *pkt;
if (m_rxBuffer->length == 0) {
m_rxBuffer->next = 0;
m_rxBuffer->length = rte_eth_rx_burst(m_portId, 0, m_rxBuffer->pmts, MAX_PKT_BURST);
if (m_rxBuffer->length == 0)
return std::make_pair(NULL, -1);
}
pkt = m_rxBuffer->pkts[m_rxBuffer->next++];
/* do not use rte_pktmbuf_read() as it does a copy for the complete packet */
return std:make_pair(rte_pktmbuf_mtod(pkt, char *), pkt->pkt_len);
}
void
DpdkNetDevice::FreeBuf(uint8_t *buf)
{
struct rte_mbuf *pkt;
if (!buf)
return;
pkt = (struct rte_mbuf *)RTE_PKT_SUB(buf, sizeof(rte_mbuf) + RTE_MAX_HEADROOM);
rte_pktmbuf_free(pkt);
}
When your code is done with the buffer, then convert the buffer address back to a rte_mbuf pointer and call rte_pktmbuf_free(pkt); This should eliminate the copy and floating point code. Converting my C code to C++ priceless :-)
Hopefully the buffer address passed is the original buffer address and has not be adjusted.
Regards,
Keith
next prev parent reply other threads:[~2018-11-19 13:49 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-08 8:24 Harsh Patel
2018-11-08 8:56 ` Wiles, Keith
2018-11-08 16:58 ` Harsh Patel
2018-11-08 17:43 ` Wiles, Keith
2018-11-09 10:09 ` Harsh Patel
2018-11-09 21:26 ` Wiles, Keith
2018-11-10 6:17 ` Wiles, Keith
2018-11-11 19:45 ` Harsh Patel
2018-11-13 2:25 ` Harsh Patel
2018-11-13 13:47 ` Wiles, Keith
2018-11-14 13:54 ` Harsh Patel
2018-11-14 15:02 ` Wiles, Keith
2018-11-14 15:04 ` Wiles, Keith
2018-11-14 15:15 ` Wiles, Keith
2018-11-17 10:22 ` Harsh Patel
2018-11-17 22:05 ` Kyle Larose
2018-11-19 13:49 ` Wiles, Keith [this message]
2018-11-22 15:54 ` Harsh Patel
2018-11-24 15:43 ` Wiles, Keith
2018-11-24 15:48 ` Wiles, Keith
2018-11-24 16:01 ` Wiles, Keith
2018-11-25 4:35 ` Stephen Hemminger
2018-11-30 9:02 ` Harsh Patel
2018-11-30 10:24 ` Harsh Patel
2018-11-30 15:54 ` Wiles, Keith
2018-12-03 9:37 ` Harsh Patel
2018-12-14 17:41 ` Harsh Patel
2018-12-14 18:06 ` Wiles, Keith
[not found] ` <CAA0iYrHyLtO3XLXMq-aeVhgJhns0+ErfuhEeDSNDi4cFVBcZmw@mail.gmail.com>
2018-12-30 0:19 ` Wiles, Keith
2018-12-30 0:30 ` Wiles, Keith
2019-01-03 18:12 ` Harsh Patel
2019-01-03 22:43 ` Wiles, Keith
2019-01-04 5:57 ` Harsh Patel
2019-01-16 13:55 ` Harsh Patel
2019-01-30 23:36 ` Harsh Patel
2019-01-31 16:58 ` Wiles, Keith
2019-02-05 6:37 ` Harsh Patel
2019-02-05 13:03 ` Wiles, Keith
2019-02-05 14:00 ` Harsh Patel
2019-02-05 14:12 ` Wiles, Keith
2019-02-05 14:22 ` Harsh Patel
2019-02-05 14:27 ` Wiles, Keith
2019-02-05 14:33 ` Harsh Patel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=485F0372-7486-473B-ACDA-F42A2D86EF03@intel.com \
--to=keith.wiles@intel.com \
--cc=eomereadig@gmail.com \
--cc=thadodaharsh10@gmail.com \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).