From: "Wang, Zhihong" <zhihong.wang@intel.com>
To: Luke Gorrie <luke@snabb.co>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
"snabb-devel@googlegroups.com" <snabb-devel@googlegroups.com>
Subject: Re: [dpdk-dev] [PATCH 0/4] DPDK memcpy optimization
Date: Tue, 27 Jan 2015 07:19:35 +0000 [thread overview]
Message-ID: <F60F360A2500CD45ACDB1D700268892D0E76129D@SHSMSX101.ccr.corp.intel.com> (raw)
In-Reply-To: <CAA2XHbdOUBJwgMMJF7xG2Rh+sPkDfxKp8JkTe6+3zgn-WC7TdQ@mail.gmail.com>
Hey Luke,
Thanks for the excellent questions!
The following script will launch the memcpy test in DPDK:
echo -e 'memcpy_autotest\nmemcpy_perf_autotest\nquit\n' | ./x86_64-native-linuxapp-gcc/app/test -c 4 -n 4 -- -i
Thanks for sharing the object code, I think it’s the Sandy Bridge version though.
The rte_memcpy for Haswell is quite simple too, this is a decision based on arch difference: Haswell has significant improvements in memory hierarchy.
The Sandy Bridge unaligned memcpy is large in size but it has better performance because converting unaligned loads into aligned ones is crucial for in cache memcpy on Sandy Bridge.
The rep instruction is still not fast enough yet, but I can’t say much about it since I haven’t investigated it thoroughly.
To my understanding memcpy optimization is all about trade-offs according to use cases and this one is for DPDK scenario (Small size, in cache: you may find quite a few with only 6 bytes or so), you can refer to the rfc for this patch.
It’s not likely that one could make one that’re optimal for all scenarios.
But I agree with the author of glibc memcpy on this: A program with too many memcpys is a program with design flaw.
Thanks
Zhihong (John)
From: lukego@gmail.com [mailto:lukego@gmail.com] On Behalf Of Luke Gorrie
Sent: Monday, January 26, 2015 4:03 PM
To: Wang, Zhihong
Cc: dev@dpdk.org; snabb-devel@googlegroups.com
Subject: Re: [dpdk-dev] [PATCH 0/4] DPDK memcpy optimization
On 26 January 2015 at 02:30, Wang, Zhihong <zhihong.wang@intel.com<mailto:zhihong.wang@intel.com>> wrote:
Hi Luke,
I’m very glad that you’re interested in this work. ☺
Great :).
I never published any performance data, and haven’t run cachebench.
We use test_memcpy_perf.c in DPDK to do the test mainly, because it’s the environment that DPDK runs. You can also find the performance comparison there with glibc.
It can be launched in <target>/app/test: memcpy_perf_autotest.
Could you give me a command-line example to run this please? (Sorry if this should be obvious.)
Finally, inline can bring benefits based on practice, constant value unrolling for example, and for DPDK we need all possible optimization.
Do we need to think about code size and potential instruction cache thrashing?
For me one call to rte_memcpy compiles to 3520 instructions<https://gist.github.com/lukego/8b17a07246d999331b04> in 20KB of object code. That's more than half the size of the Haswell instruction cache (32KB) per call.
glibc 2.20's memcpy_avx_unaligned<https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S;h=9f033f54568c3e5b6d9de9b3ba75f5be41070b92;hb=HEAD> is only 909 bytes shared/total and also seems to have basically excellent performance on Haswell.
So I am concerned about the code size of rte_memcpy, especially when inlined, and meta-concerned about the nonlinear impact of nested inlined functions on both compile time and object code size.
There is another issue that I am concerned about:
The Intel Optimization Guide suggests that rep movs is very efficient starting in Ivy Bridge. In practice though it seems to be much slower than using vector instructions, even though it is faster than it used to be in Sandy Bridge. Is that true?
This could have a substantial impact on off-the-shelf memcpy. glibc 2.20's memcpy uses movs for sizes >= 2048 and that is where performance takes a dive for me (in microbenchmarks). GCC will also emit inline string move instructions for certain constant-size memcpy calls at certain optimization levels.
So I feel like I haven't yet found the right memcpy for me. and we haven't even started to look at the interesting parts like cache-coherence behaviour when sharing data between cores (vhost) and whether streaming load/store can be used to defend the state of cache lines between cores.
Do I make any sense? What do I miss?
Cheers,
-Luke
next prev parent reply other threads:[~2015-01-27 7:21 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-01-19 1:53 zhihong.wang
2015-01-19 1:53 ` [dpdk-dev] [PATCH 1/4] app/test: Disabled VTA for memcpy test in app/test/Makefile zhihong.wang
2015-01-19 1:53 ` [dpdk-dev] [PATCH 2/4] app/test: Removed unnecessary test cases in test_memcpy.c zhihong.wang
2015-01-19 1:53 ` [dpdk-dev] [PATCH 3/4] app/test: Extended test coverage in test_memcpy_perf.c zhihong.wang
2015-01-19 1:53 ` [dpdk-dev] [PATCH 4/4] lib/librte_eal: Optimized memcpy in arch/x86/rte_memcpy.h for both SSE and AVX platforms zhihong.wang
2015-01-20 17:15 ` Stephen Hemminger
2015-01-20 19:16 ` Neil Horman
2015-01-21 3:18 ` Wang, Zhihong
2015-01-25 20:02 ` Jim Thompson
2015-01-26 14:43 ` Wodkowski, PawelX
2015-01-27 5:12 ` Wang, Zhihong
2015-01-19 13:02 ` [dpdk-dev] [PATCH 0/4] DPDK memcpy optimization Neil Horman
2015-01-20 3:01 ` Wang, Zhihong
2015-01-20 15:11 ` Neil Horman
2015-01-20 16:14 ` Bruce Richardson
2015-01-21 3:44 ` Wang, Zhihong
2015-01-21 11:40 ` Bruce Richardson
2015-01-21 12:02 ` Ananyev, Konstantin
2015-01-21 12:38 ` Neil Horman
2015-01-23 3:26 ` Wang, Zhihong
2015-01-21 12:36 ` Marc Sune
2015-01-21 13:02 ` Bruce Richardson
2015-01-21 13:21 ` Marc Sune
2015-01-21 13:26 ` Bruce Richardson
2015-01-21 19:49 ` Stephen Hemminger
2015-01-21 20:54 ` Neil Horman
2015-01-21 21:25 ` Jim Thompson
2015-01-22 0:53 ` Stephen Hemminger
2015-01-22 9:06 ` Luke Gorrie
2015-01-22 13:29 ` Jay Rolette
2015-01-22 18:27 ` Luke Gorrie
2015-01-22 19:36 ` Jay Rolette
2015-01-22 18:21 ` EDMISON, Kelvin (Kelvin)
2015-01-27 8:22 ` Wang, Zhihong
2015-01-28 21:48 ` EDMISON, Kelvin (Kelvin)
2015-01-29 1:53 ` Wang, Zhihong
2015-01-23 6:52 ` Wang, Zhihong
2015-01-26 18:29 ` Ananyev, Konstantin
2015-01-27 1:42 ` Wang, Zhihong
2015-01-27 11:30 ` Ananyev, Konstantin
2015-01-27 12:19 ` Ananyev, Konstantin
2015-01-28 2:06 ` Wang, Zhihong
2015-01-25 14:50 ` Luke Gorrie
2015-01-26 1:30 ` Wang, Zhihong
2015-01-26 8:03 ` Luke Gorrie
2015-01-27 7:19 ` Wang, Zhihong [this message]
2015-01-27 13:57 ` [dpdk-dev] [snabb-devel] " Luke Gorrie
2015-01-29 3:42 ` [dpdk-dev] " Fu, JingguoX
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=F60F360A2500CD45ACDB1D700268892D0E76129D@SHSMSX101.ccr.corp.intel.com \
--to=zhihong.wang@intel.com \
--cc=dev@dpdk.org \
--cc=luke@snabb.co \
--cc=snabb-devel@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).