From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-we0-f170.google.com (mail-we0-f170.google.com [74.125.82.170]) by dpdk.org (Postfix) with ESMTP id 72CC0255 for ; Mon, 26 Jan 2015 09:03:12 +0100 (CET) Received: by mail-we0-f170.google.com with SMTP id w55so2107187wes.1 for ; Mon, 26 Jan 2015 00:03:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=+0YLSHlGhW3OwGZyXwNGGfRRAnfN8ojWWO/dvST9u3E=; b=EQzvRcIGk0+fXIVcCpKoZ1Puag0VGK7NtoEcPhLAnSy9VDaFACMKT6iNIXFDmGRsWH PWc9bQrxavi0CJRq7vMWozOshQ0PrNctfnSIHE1cZSDM6CNZuLlgux7Oy+75DyTAiclg Y6r0cAB8/XhHcXgOEd+cSzUxZEjZTCCc7sXFXtBPEl2gKIngKcnKJHIk+QVIXwrVRfnS 9EKtwIf/US51I9/uLCiKJagHQwTfKoXPUwrMEFJbZ+lT6cMRXQjmS/ciK35MGkEk3fWl UWwVnrLoSysvcbLZaesZwJKw12mFlapTMxKiu30AgIeI17gmDGZb1CAEnBOjVs797bYc Qsgg== MIME-Version: 1.0 X-Received: by 10.181.12.36 with SMTP id en4mr2909249wid.55.1422259392269; Mon, 26 Jan 2015 00:03:12 -0800 (PST) Sender: lukego@gmail.com Received: by 10.27.6.134 with HTTP; Mon, 26 Jan 2015 00:03:12 -0800 (PST) In-Reply-To: References: <1421632414-10027-1-git-send-email-zhihong.wang@intel.com> Date: Mon, 26 Jan 2015 09:03:12 +0100 X-Google-Sender-Auth: JZvAvO_PBBTfUeEgBrsFwP_-lro Message-ID: From: Luke Gorrie To: "Wang, Zhihong" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" , "snabb-devel@googlegroups.com" Subject: Re: [dpdk-dev] [PATCH 0/4] DPDK memcpy optimization X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jan 2015 08:03:12 -0000 On 26 January 2015 at 02:30, Wang, Zhihong wrote: > Hi Luke, > > > > I=E2=80=99m very glad that you=E2=80=99re interested in this work. J > Great :). I never published any performance data, and haven=E2=80=99t run cachebenc= h. > > We use test_memcpy_perf.c in DPDK to do the test mainly, because it=E2=80= =99s the > environment that DPDK runs. You can also find the performance comparison > there with glibc. > > It can be launched in /app/test: memcpy_perf_autotest. > Could you give me a command-line example to run this please? (Sorry if this should be obvious.) > Finally, inline can bring benefits based on practice, constant value > unrolling for example, and for DPDK we need all possible optimization. > Do we need to think about code size and potential instruction cache thrashing? For me one call to rte_memcpy compiles to 3520 instructions in 20KB of object code. That's more than half the size of the Haswell instruction cache (32KB) per call. glibc 2.20's memcpy_avx_unaligned is only 909 bytes shared/total and also seems to have basically excellent performance on Haswell. So I am concerned about the code size of rte_memcpy, especially when inlined, and meta-concerned about the nonlinear impact of nested inlined functions on both compile time and object code size. There is another issue that I am concerned about: The Intel Optimization Guide suggests that rep movs is very efficient starting in Ivy Bridge. In practice though it seems to be much slower than using vector instructions, even though it is faster than it used to be in Sandy Bridge. Is that true? This could have a substantial impact on off-the-shelf memcpy. glibc 2.20's memcpy uses movs for sizes >=3D 2048 and that is where performance takes a dive for me (in microbenchmarks). GCC will also emit inline string move instructions for certain constant-size memcpy calls at certain optimization levels. So I feel like I haven't yet found the right memcpy for me. and we haven't even started to look at the interesting parts like cache-coherence behaviour when sharing data between cores (vhost) and whether streaming load/store can be used to defend the state of cache lines between cores. Do I make any sense? What do I miss? Cheers, -Luke