From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B6434A0545; Wed, 10 Aug 2022 13:47:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5496D4068E; Wed, 10 Aug 2022 13:47:47 +0200 (CEST) Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by mails.dpdk.org (Postfix) with ESMTP id 8BF1C4067C for ; Wed, 10 Aug 2022 13:47:45 +0200 (CEST) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id B08E591D3 for ; Wed, 10 Aug 2022 13:47:44 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 9F2FB925D; Wed, 10 Aug 2022 13:47:44 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hermod.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.6 required=5.0 tests=ALL_TRUSTED, AWL, NICE_REPLY_A, T_SCC_BODY_TEXT_LINE autolearn=disabled version=3.4.6 X-Spam-Score: -1.6 Received: from [192.168.1.59] (unknown [62.63.215.114]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 53660925C; Wed, 10 Aug 2022 13:47:43 +0200 (CEST) Message-ID: Date: Wed, 10 Aug 2022 13:47:42 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: [RFC v2] non-temporal memcpy Content-Language: en-US To: =?UTF-8?Q?Morten_Br=c3=b8rup?= , dev@dpdk.org, Bruce Richardson , Konstantin Ananyev Cc: Jan Viktorin , Ruifeng Wang , David Christensen , Stanislaw Kardach References: <98CBD80474FA8B44BF855DF32C47DC35D871D4@smartserver.smartshare.dk> <9ac934d2-ad05-6ec9-3bb6-63986d68d5d3@lysator.liu.se> <98CBD80474FA8B44BF855DF32C47DC35D87247@smartserver.smartshare.dk> <44172db2-5f03-e58e-f72c-76eac1cd192c@lysator.liu.se> <98CBD80474FA8B44BF855DF32C47DC35D8724B@smartserver.smartshare.dk> From: =?UTF-8?Q?Mattias_R=c3=b6nnblom?= In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D8724B@smartserver.smartshare.dk> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2022-08-09 17:00, Morten Brørup wrote: >> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se] >> Sent: Tuesday, 9 August 2022 14.05 >> >> On 2022-08-09 11:46, Morten Brørup wrote: >>>> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se] >>>> Sent: Sunday, 7 August 2022 22.25 >>>> >>>> On 2022-07-19 17:26, Morten Brørup wrote: >>>>> This RFC proposes a set of functions optimized for non-temporal >>>> memory copy. >>>>> >>>>> At this stage, I am asking for feedback on the concept. >>>>> >>>>> Applications sometimes data to another memory location, which is >> only >>>> used >>>>> much later. >>>>> In this case, it is inefficient to pollute the data cache with the >>>> copied >>>>> data. >>>>> >>>>> An example use case (originating from a real life application): >>>>> Copying filtered packets, or the first part of them, into a capture >>>> buffer >>>>> for offline analysis. >>>>> >>>>> The purpose of these functions is to achieve a performance gain by >>>> not >>>>> polluting the cache when copying data. >>>>> Although the throughput may be improved by further optimization, I >> do >>>> not >>>>> consider througput optimization relevant initially. >>>>> >>>>> The x86 non-temporal load instructions have 16 byte alignment >>>>> requirements [1], while ARM non-temporal load instructions are >>>> available with >>>>> 4 byte alignment requirements [2]. >>>>> Both platforms offer non-temporal store instructions with 4 byte >>>> alignment >>>>> requirements. >>>>> >>>> >>>> I don't think memcpy() functions should have alignment requirements. >>>> That's not very practical, and violates the principle of least >>>> surprise. >>> >>> I didn't make the CPUs with these alignment requirements. >>> >>> However, I will offer optimized performance in a generic NT memcpy() >> function in the cases where the individual alignment requirements of >> various CPUs happen to be met. >>> >>>> >>>> Use normal memcpy() for the unaligned parts, and for the whole thing >>>> for >>>> small sizes (at least on x86). >>>> >>> >>> I'm not going to plunge into some advanced vector programming, so I'm >> working on an implementation where misalignment is handled by using a >> bounce buffer (allocated on the stack, which is probably cache hot >> anyway). >>> >>> >> >> I don't know for the NT load + NT store case, but for regular load + NT >> store, this is trivial. The implementation I've used is 36 >> straight-forward lines of code. > > Is that implementation available for inspiration anywhere? > #define NT_THRESHOLD (2 * CACHE_LINE_SIZE) void nt_memcpy(void *__restrict dst, const void * __restrict src, size_t n) { if (n < NT_THRESHOLD) { memcpy(dst, src, n); return; } size_t n_unaligned = CACHE_LINE_SIZE - (uintptr_t)dst % CACHE_LINE_SIZE; if (n_unaligned > n) n_unaligned = n; memcpy(dst, src, n_unaligned); dst += n_unaligned; src += n_unaligned; n -= n_unaligned; size_t num_lines = n / CACHE_LINE_SIZE; size_t i; for (i = 0; i < num_lines; i++) { size_t j; for (j = 0; j < CACHE_LINE_SIZE / sizeof(__m128i); j++) { __m128i blk = _mm_loadu_si128((const __m128i *)src); /* non-temporal store */ _mm_stream_si128((__m128i *)dst, blk); src += sizeof(__m128i); dst += sizeof(__m128i); } n -= CACHE_LINE_SIZE; } if (num_lines > 0) _mm_sfence(); memcpy(dst, src, n); } (This was written as a part of a benchmark exercise, and hasn't been properly tested.) Use this for inspiration, or I can DPDK-ify this and make it a proper patch/RFC. I would try to add support for NT load as well, and make both NT load and store depend on flags parameter. The above threshold setting is completely arbitrary. What you should keep in mind when thinking about the threshold, is that it might well be worth to suffer a little lower performance of NT store + sfence (compared to regular store), since you will benefit from not trashing the cache. For example, back-to-back copying of 1500 bytes buffers with this copying routine is much slower than regular memcpy() (measured in the core cycles spent in the copying), but nevertheless in a real-world application it may still improve the overall performance, since the packet copies doesn't evict useful data from the various caches. I know for sure that certain applications do benefit.