RE: [RFC v2] non-temporal memcpy

[Konstantin Ananyev <konstantin.ananyev@huawei.com>]

> > > > >>>> Hi Morten,
> > > > >>>>
> > > > >>>>> This RFC proposes a set of functions optimized for non-
> > temporal
> > > > >>>> memory copy.
> > > > >>>>>
> > > > >>>>> At this stage, I am asking for feedback on the concept.
> > > > >>>>>
> > > > >>>>> Applications sometimes data to another memory location, which
> > is
> > > > >> only
> > > > >>>> used
> > > > >>>>> much later.
> > > > >>>>> In this case, it is inefficient to pollute the data cache
> > with
> > > > the
> > > > >>>> copied
> > > > >>>>> data.
> > > > >>>>>
> > > > >>>>> An example use case (originating from a real life
> > application):
> > > > >>>>> Copying filtered packets, or the first part of them, into a
> > > > capture
> > > > >>>> buffer
> > > > >>>>> for offline analysis.
> > > > >>>>>
> > > > >>>>> The purpose of these functions is to achieve a performance
> > gain
> > > > by
> > > > >>>> not
> > > > >>>>> polluting the cache when copying data.
> > > > >>>>> Although the throughput may be improved by further
> > optimization,
> > > > I
> > > > >> do
> > > > >>>> not
> > > > >>>>> consider througput optimization relevant initially.
> > > > >>>>>
> > > > >>>>> The x86 non-temporal load instructions have 16 byte alignment
> > > > >>>>> requirements [1], while ARM non-temporal load instructions
> > are
> > > > >>>> available with
> > > > >>>>> 4 byte alignment requirements [2].
> > > > >>>>> Both platforms offer non-temporal store instructions with 4
> > byte
> > > > >>>> alignment
> > > > >>>>> requirements.
> > > > >>>>>
> > > > >>>>> In addition to the primary function without any alignment
> > > > >>>> requirements, we
> > > > >>>>> also provide functions for respectivly 16 and 4 byte aligned
> > > > access
> > > > >>>> for
> > > > >>>>> performance purposes.
> > > > >>>>>
> > > > >>>>> The function names resemble standard C library function
> > names,
> > > > but
> > > > >>>> their
> > > > >>>>> signatures are intentionally different. No need to drag
> > legacy
> > > > into
> > > > >>>> it.
> > > > >>>>>
> > > > >>>>> NB: Don't comment on spaces for indentation; a patch will
> > follow
> > > > >> DPDK
> > > > >>>> coding
> > > > >>>>> style and use TAB.
> > > > >>>>
> > > > >>>>
> > > > >>>> I think there were discussions in other direction - remove
> > > > >> rte_memcpy()
> > > > >>>> completely and use memcpy() instead...
> > > > >>>
> > > > >>> Yes, the highly optimized rte_memcpy() implementation of
> > memcpy()
> > > > has
> > > > >> become obsolete, now that modern compilers provide an efficient
> > > > >> memcpy() implementation.
> > > > >>>
> > > > >>> It's an excellent reference, because we should learn from it,
> > and
> > > > >> avoid introducing similar mistakes with non-temporal memcpy.
> > > > >>>
> > > > >>>> But if we have a good use case for that, then I am positive in
> > > > >>>> principle.
> > > > >>>
> > > > >>> The standard C library doesn't offer non-temporal memcpy(), so
> > we
> > > > >> need to implement it ourselves.
> > > > >>>
> > > > >>>> Though I think we need a clear use-case within dpdk for it
> > > > >>>> to demonstrate perfomance gain.
> > > > >>>
> > > > >>> The performance gain is to avoid polluting the data cache. DPDK
> > > > >> example applications, like l3fwd, are probably too primitive to
> > > > measure
> > > > >> any benefit in this regard.
> > > > >>>
> > > > >>>> Probably copying packets within pdump lib, or examples/dma. or
> > ...
> > > > >>>
> > > > >>> Good point - the new functions should be used somewhere within
> > > > DPDK.
> > > > >> For this purpose, I will look into modifying rte_pktmbuf_copy(),
> > > > which
> > > > >> is used by pdump_copy(), to use non-temporal copying of the
> > packet
> > > > >> data.
> > > > >>>
> > > > >>>> Another thought - do we really need a separate inline function
> > for
> > > > >> each
> > > > >>>> flavour?
> > > > >>>> Might be just one non-inline rte_memcpy_nt(dst, src, size,
> > flags),
> > > > >>>> where flags could be combination of NT_SRC, NT_DST, and keep
> > > > >> alignment
> > > > >>>> detection/decisions to particular implementation?
> > > > >>>
> > > > >>> Thank you for the feedback, Konstantin.
> > > > >>>
> > > > >>> My answer to this suggestion gets a little longwinded...
> > > > >>>
> > > > >>> Looking at the DPDK pcapng library, it copies a 4 byte aligned
> > > > >> metadata structure sized 28 byte. So it can do with 4 byte
> > aligned
> > > > >> functions.
> > > > >>>
> > > > >>> Our application can capture packets starting at the IP header,
> > > > which
> > > > >> is offset by 14 byte (Ethernet header size) from the packet
> > buffer,
> > > > so
> > > > >> it requires 2 byte alignment. And thus, requiring 4 byte
> > alignment
> > > > is
> > > > >> not acceptable.
> > > > >>>
> > > > >>> Our application uses 16 byte alignment in the capture buffer
> > area,
> > > > >> and can benefit from 16 byte aligned functions. Furthermore, x86
> > > > >> processors require 16 byte alignment for non-temporal load
> > > > >> instructions, so I think a 16 byte aligned non-temporal memcpy
> > > > function
> > > > >> should be offered.
> > > > >>
> > > > >>
> > > > >> Yes, x86 needs 16B alignment for NT load/stores
> > > > >> But that's supposed to be arch specific limitation,
> > > > >> that we probably want to hide, no?
> > > > >
> > > > > Agree.
> > > > >
> > > > >> Inside the function can check alignment of both src and dst
> > > > >> and decide should it use NT load/store instructions or just
> > > > >> do normal copy.
> > > > >
> > > > > Yes, I'm experimenting with the x86 inline function shown below.
> > And
> > > > hopefully, with some "extern inline" or other magic, I can hide the
> > > > different implementations in the arch specific headers, and only
> > expose
> > > > the function declaration of rte_memcpy_nt() in the common header.
> > > > >
> > > > > I'm currently working on the x86 implementation - when I'm
> > satisfied
> > > > with that, I'll look into how to hide the implementations in the
> > arch
> > > > specific header files, and only expose the common function
> > declaration
> > > > in the generic header file also used for documentation. I works for
> > > > rte_memcpy(), so I can probably find the way to do it there.
> > > > >
> > > > > /*
> > > > >   * Non-Temporal Memory Operations Flags.
> > > > >   */
> > > > >
> > > > > #define RTE_MEMOPS_F_LENA_MASK  (UINT64_C(0xFE) << 0)   /**
> > Length
> > > > alignment mask. */
> > > > > #define RTE_MEMOPS_F_LEN2A      (UINT64_C(2) << 0)      /**
> > Length is
> > > > 2 byte aligned. */
> > > > > #define RTE_MEMOPS_F_LEN4A      (UINT64_C(4) << 0)      /**
> > Length is
> > > > 4 byte aligned. */
> > > > > #define RTE_MEMOPS_F_LEN8A      (UINT64_C(8) << 0)      /**
> > Length is
> > > > 8 byte aligned. */
> > > > > #define RTE_MEMOPS_F_LEN16A     (UINT64_C(16) << 0)     /**
> > Length is
> > > > 16 byte aligned. */
> > > > > #define RTE_MEMOPS_F_LEN32A     (UINT64_C(32) << 0)     /**
> > Length is
> > > > 32 byte aligned. */
> > > > > #define RTE_MEMOPS_F_LEN64A     (UINT64_C(64) << 0)     /**
> > Length is
> > > > 64 byte aligned. */
> > > > > #define RTE_MEMOPS_F_LEN128A    (UINT64_C(128) << 0)    /**
> > Length is
> > > > 128 byte aligned. */
> > > > >
> > > > > #define RTE_MEMOPS_F_DSTA_MASK  (UINT64_C(0xFE) << 8)   /**
> > > > Destination address alignment mask. */
> > > > > #define RTE_MEMOPS_F_DST2A      (UINT64_C(2) << 8)      /**
> > > > Destination address is 2 byte aligned. */
> > > > > #define RTE_MEMOPS_F_DST4A      (UINT64_C(4) << 8)      /**
> > > > Destination address is 4 byte aligned. */
> > > > > #define RTE_MEMOPS_F_DST8A      (UINT64_C(8) << 8)      /**
> > > > Destination address is 8 byte aligned. */
> > > > > #define RTE_MEMOPS_F_DST16A     (UINT64_C(16) << 8)     /**
> > > > Destination address is 16 byte aligned. */
> > > > > #define RTE_MEMOPS_F_DST32A     (UINT64_C(32) << 8)     /**
> > > > Destination address is 32 byte aligned. */
> > > > > #define RTE_MEMOPS_F_DST64A     (UINT64_C(64) << 8)     /**
> > > > Destination address is 64 byte aligned. */
> > > > > #define RTE_MEMOPS_F_DST128A    (UINT64_C(128) << 8)    /**
> > > > Destination address is 128 byte aligned. */
> > > > >
> > > > > #define RTE_MEMOPS_F_SRCA_MASK  (UINT64_C(0xFE) << 16)  /**
> > Source
> > > > address alignment mask. */
> > > > > #define RTE_MEMOPS_F_SRC2A      (UINT64_C(2) << 16)     /**
> > Source
> > > > address is 2 byte aligned. */
> > > > > #define RTE_MEMOPS_F_SRC4A      (UINT64_C(4) << 16)     /**
> > Source
> > > > address is 4 byte aligned. */
> > > > > #define RTE_MEMOPS_F_SRC8A      (UINT64_C(8) << 16)     /**
> > Source
> > > > address is 8 byte aligned. */
> > > > > #define RTE_MEMOPS_F_SRC16A     (UINT64_C(16) << 16)    /**
> > Source
> > > > address is 16 byte aligned. */
> > > > > #define RTE_MEMOPS_F_SRC32A     (UINT64_C(32) << 16)    /**
> > Source
> > > > address is 32 byte aligned. */
> > > > > #define RTE_MEMOPS_F_SRC64A     (UINT64_C(64) << 16)    /**
> > Source
> > > > address is 64 byte aligned. */
> > > > > #define RTE_MEMOPS_F_SRC128A    (UINT64_C(128) << 16)   /**
> > Source
> > > > address is 128 byte aligned. */
> > > > >
> > > > > /**
> > > > >   * @warning
> > > > >   * @b EXPERIMENTAL: this API may change without prior notice.
> > > > >   *
> > > > >   * Non-temporal memory copy.
> > > > >   * The memory areas must not overlap.
> > > > >   *
> > > > >   * @note
> > > > >   * If the destination and/or length is unaligned, some copied
> > bytes
> > > > will be
> > > > >   * stored in the destination memory area using temporal access.
> > > > >   *
> > > > >   * @param dst
> > > > >   *   Pointer to the non-temporal destination memory area.
> > > > >   * @param src
> > > > >   *   Pointer to the non-temporal source memory area.
> > > > >   * @param len
> > > > >   *   Number of bytes to copy.
> > > > >   * @param flags
> > > > >   *   Hints for memory access.
> > > > >   *   Any of the RTE_MEMOPS_F_LENnA, RTE_MEMOPS_F_DSTnA,
> > > > RTE_MEMOPS_F_SRCnA flags.
> > > > >   */
> > > > > __rte_experimental
> > > > > static __rte_always_inline
> > > > > __attribute__((__nonnull__(1, 2), __access__(write_only, 1, 3),
> > > > __access__(read_only, 2, 3)))
> > > > > void rte_memcpy_nt(void * __rte_restrict dst, const void *
> > > > __rte_restrict src, size_t len,
> > > > >          const uint64_t flags)
> > > > > {
> > > > >      if (__builtin_constant_p(flags) ?
> > > > >              ((flags & RTE_MEMOPS_F_LENA_MASK) >=
> > RTE_MEMOPS_F_LEN16A
> > > > &&
> > > > >              (flags & RTE_MEMOPS_F_DSTA_MASK) >=
> > RTE_MEMOPS_F_DST16A)
> > > > :
> > > > >              !(((uintptr_t)dst | len) & (16 - 1))) {
> > > > >          if (__builtin_constant_p(flags) ?
> > > > >                  (flags & RTE_MEMOPS_F_SRCA_MASK) >=
> > > > RTE_MEMOPS_F_SRC16A :
> > > > >                  !((uintptr_t)src & (16 - 1)))
> > > > >              rte_memcpy_nt16a(dst, src, len/*, flags*/);
> > > > >          else
> > > > >              rte_memcpy_nt16dla(dst, src, len/*, flags*/);
> > > > >      }
> > > > >      else if (__builtin_constant_p(flags) ? (
> > > > >              (flags & RTE_MEMOPS_F_LENA_MASK) >=
> > RTE_MEMOPS_F_LEN4A
> > > > &&
> > > > >              (flags & RTE_MEMOPS_F_DSTA_MASK) >=
> > RTE_MEMOPS_F_DST4A
> > > > &&
> > > > >              (flags & RTE_MEMOPS_F_SRCA_MASK) >=
> > RTE_MEMOPS_F_SRC4A)
> > > > :
> > > > >              !(((uintptr_t)dst | (uintptr_t)src | len) & (4 -
> > 1))) {
> > > > >          rte_memcpy_nt4a(dst, src, len/*, flags*/);
> > > > >      }
> > > > >      else
> > > > >          rte_memcpy_nt_unaligned(dst, src, len/*, flags*/);
> > > > > }
> > > >
> > > >
> > > > Do we really need to expose all these dozen flags?
> > > > My thought at least about x86 implementaion was about something
> > more
> > > > simple like:
> > > > void rte_memcpy_nt(void * __rte_restrict dst,
> > > > 	const void * __rte_restrict src, size_t len,
> > > > 	const uint64_t flags)
> > > > {
> > > >
> > > > 	if (flags == (SRC_NT | DST_NT) && ((dst | src) & 0xf) == 0) {
> > > > 		_do_nt_src_nt_dst_nt(...);
> > > > 	} else if (flags == DST_NT && (dst & 0xf) == 0) {
> > > > 		_do_src_na_dst_nt(...);
> > > > 	} else if (flags == SRC_NT && (src & 0xf) == 0) {
> > > > 		_do_src_nt_dst_na(...);
> > > > 	} else
> > > > 		memcpy(dst, src, len);
> > > > }
> > >
> > > The combination of flags, inline and __builtin_constant_p() allows
> > the compiler to produce zero-overhead code. Without it, the
> > > resulting code will contain a bunch of run-time bitmask comparisons
> > and branches to determine the ultimate copy function.
> >
> > I think it is unavoidable, unless your intention for this function to
> > trust flags without checking actual addresses.
> 
> The intention is to trust the flags. Fast path functions should trust that parameters passed are valid and conform to specified
> requirements.
> 
> We can always throw in a few RTE_ASSERTs - they are omitted unless compiled for debug, and thus have zero cost in production.
> 
> >
> >  On x86
> > > there are not only 16 byte, but also 4 byte alignment variants of
> > non-temporal store. The beauty of it will be more obvious when the
> > > patch is ready.
> >
> > Ok, I will just wait for final version then :)
> >
> > >
> > > And in my current working version (not the code provided here), the
> > flags are hints, so using them will be optional. The function
> > > headers will look roughly like this:
> > >
> > > static inline void rte_memcpy_nt_ex(
> > > 		void * dst, const void * src,
> > > 		size_t len, uint64_t flags);
> > >
> > > static inline void rte_memcpy_nt(
> > > 		void * dst, const void * src,
> > > 		size_t len)
> > > {
> > > 	rte_memcpy_nt_ex(dst, src, len, 0);
> > > }
> > >
> > > I might add an _ex postfix variant of the mbuf packet non-temporal
> > copy function too, but I'm not working on that function yet, so I
> > > don't yet know if it makes sense or not.
> > >
> > > My concept for build time alignment hints can also be copied into an
> > rte_memcpy_ex() function for improved performance. But I
> > > don't want my patch to expand too much outside its initial scope, so
> > I will not modify rte_memcpy() with this patch.
> > >
> > > Alternatively, I could provide rte_memcpy_ex(d,s,l,flags) instead of
> > rte_memcpy_nt[_ex](), and use the flags to indicate non-
> > > temporal source and destination.
> > >
> > > This is only a question about which API the community prefers. I will
> > not change the implementation of rte_memcpy() with this patch -
> > > it's another job to do that.
> > >
> > > >
> > > > >
> > > > >
> > > > >>
> > > > >>
> > > > >>> While working on these funtions, I experimented with an
> > > > >> rte_memcpy_nt() taking flags, which is also my personal
> > preference,
> > > > but
> > > > >> haven't succeed yet. Especially when copying a 16 byte aligned
> > > > >> structure of only 16 byte, the overhead of the function call +
> > > > >> comparing the flags + the copy loop overhead is significant,
> > > > compared
> > > > >> to inline code consisting of only one pair of "movntdqa
> > > > (%rsi),%xmm0;
> > > > >> movntdq %xmm0,(%rdi)" instructions.
> > > > >>>
> > > > >>> Remember that a non-inlined rte_memcpy_nt() will be called with
> > > > very
> > > > >> varying size, due to the typical mix of small and big packets,
> > so
> > > > >> branch prediction will not help.
> > > > >>>
> > > > >>> This RFC does not yet show the rte_memcpy_nt() function
> > handling
> > > > >> unaligned load/store, but it is more complex than the aligned
> > > > >> functions. So I think the aligned variants are warranted - for
> > > > >> performance reasons.
> > > > >>>
> > > > >>> Some of the need for exposing individual functions for
> > different
> > > > >> alignment stems from the compiler being unable to determine the
> > > > >> alignment of the source and destination pointers at build time.
> > So
> > > > we
> > > > >> need to help the compiler with this at build time, and thus the
> > need
> > > > >> for inlining the function. If we expose a bunch of small inline
> > > > >> functions or a big inline function with flags seems to be a
> > matter
> > > > of
> > > > >> taste.
> > > > >>>
> > > > >>> Thinking about it, you are probably right that exposing a
> > single
> > > > >> function with flags is better for documentation purposes and
> > easier
> > > > for
> > > > >> other architectures to implement. But it still needs to be
> > inline,
> > > > for
> > > > >> the reasons described above.
> > > > >>
> > > > >>
> > > > >> Ok, my initial thought was that main use-case for it would be
> > > > copying
> > > > >> of
> > > > >> big chunks of data, but from your description it might not be
> > the
> > > > case.
> > > > >
> > > > > This is for quickly copying relatively small pieces of data
> > > > synchronously without polluting the CPUs data cache, e.g. just
> > before
> > > > passing on a packet to an Ethernet PMD for transmission.
> > > > >
> > > > > Big chunks of data should be copied asynchronously by DMA.
> > > > >
> > > > >> Yes, for just 16/32B copy function call overhead might be way
> > too
> > > > >> high...
> > > > >> As another alternative - would memcpy_nt_bulk() help somehow?
> > > > >> It can do copying for the several src/dst pairs at once and
> > > > >> that might help to amortize cost of function call.
> > > > >
> > > > > In many cases, memcpy_nt() will replace memcpy() inside loops, so
> > it
> > > > should be just as easy to use as memcpy(). E.g. look at
> > > > rte_pktmbuf_copy()... Building a memcopy array to pass to
> > > > memcpy_nt_bulk() from rte_pktmbuf_copy() would require a
> > significant
> > > > rewrite of rte_pktmbuf_copy(), compared to just replacing
> > rte_memcpy()
> > > > with rte_memcpy_nt(). And this is just one function using memcpy().
> > > >
> > > > Actually, one question I have for such small data-transfer
> > > > (16B per packet) - do you still see some noticable perfomance
> > > > improvement for such scenario?
> > >
> > > Copying 16 byte from each packet in a burst of 32 packets would
> > otherwise pollute 64 cache lines = 4 KB cache. With typically 64 KB L1
> > > cache, I think it makes a difference.
> >
> > I understand the intention behind, my question was - it is really
> > measurable?
> > Something like: using pktmbuf_copy_nt(len=16) over using
> > pktmbuf_copy(len=16)
> > on workload X gives Y% thoughtput improvement?
> 
> If the application is complex enough, and needs some of those 4 KB cache otherwise wasted, there will be a significant throughput
> improvement; otherwise probably not.
> 
> I have a general problem with this type of question: I hate that throughput is the only KPI (Key Performance Indicator) getting any
> attention on the mailing list! Other KPIs, such as latency and resource conservation, are just as important in many real life use cases.

Well, I suppose that sort of expected question for the patch that introduces performance optimization:
what is the benefit we expect to get and is it worth the effort?
Throughput or latency improvement seems like an obvious choice here.
About resource conservation - if the patch aims to improve cache consumption, then on some cache-bound
workloads it should result in throughput improvement, correct? 

> 
> Here's a number for you: 6.25 % reduction in L1 data cache consumption. (Assuming 64 KB L1 cache with 64 byte cache lines and
> application burst length of 32 packets.)

I understand that it should reduce cache eviction rate.
The thing is that non-temporal stores are not free also: they consume WC buffers and some memory-bus bandwidth.
AFAIK, for 16B non-consecutive NT stores, it means that only 25% of WC buffers capacity will be used,
and in theory it might lead to extra memory pressure and worse performance in general.
In fact, IA manuals explicitly recommend to avoid partial cach-line writes whenever possible.
Now, I don't know what would be more expensive in that case: re-fill extra cache-lines,
or extra partial write memory transactions.
That's why I asked for some performance numbers here.
 
> >
> > >
> > > > Another question - who will do 'sfence' after the copying?
> > > > Would it be inside memcpy_nt (seems quite costly), or would
> > > > it be another API function for that: memcpy_nt_flush() or so?
> > >
> > > Outside. Only the developer knows when it is required, so it wouldn't
> > make any sense to add the cost inside memcpy_nt().
> > >
> > > I don't think we should add a flush function; it would just be
> > another name for an already existing function. Referring to the
> > required
> > > operation in the memcpy_nt() function documentation should suffice.
> > >
> > > >
> > > > >>
> > > > >>
> > > > >>>
> > > > >>>>
> > > > >>>>
> > > > >>>>> [1] https://www.intel.com/content/www/us/en/docs/intrinsics-
> > > > >>>> guide/index.html#text=_mm_stream_load
> > > > >>>>> [2] https://developer.arm.com/documentation/100076/0100/A64-
> > > > >>>> Instruction-Set-Reference/A64-Floating-point-
> > Instructions/LDNP--
> > > > >> SIMD-
> > > > >>>> and-FP-
> > > > >>>>>
> > > > >>>>> V2:
> > > > >>>>> - Only copy from non-temporal source to non-temporal
> > destination.
> > > > >>>>>      I.e. remove the two variants with only source and/or
> > > > >> destination
> > > > >>>> being
> > > > >>>>>      non-temporal.
> > > > >>>>> - Do not require alignment.
> > > > >>>>>      Instead, offer additional 4 and 16 byte aligned
> > functions
> > > > for
> > > > >>>> performance
> > > > >>>>>      purposes.
> > > > >>>>> - Implemented two of the functions for x86.
> > > > >>>>> - Remove memset function.
> > > > >>>>>
> > > > >>>>> Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
> > > > >>>>> ---
> > > > >>>>>
> > > > >>>>> /**
> > > > >>>>>     * @warning
> > > > >>>>>     * @b EXPERIMENTAL: this API may change without prior
> > notice.
> > > > >>>>>     *
> > > > >>>>>     * Copy data from non-temporal source to non-temporal
> > > > >> destination.
> > > > >>>>>     *
> > > > >>>>>     * @param dst
> > > > >>>>>     *   Pointer to the non-temporal destination of the data.
> > > > >>>>>     *   Should be 4 byte aligned, for optimal performance.
> > > > >>>>>     * @param src
> > > > >>>>>     *   Pointer to the non-temporal source data.
> > > > >>>>>     *   No alignment requirements.
> > > > >>>>>     * @param len
> > > > >>>>>     *   Number of bytes to copy.
> > > > >>>>>     *   Should be be divisible by 4, for optimal performance.
> > > > >>>>>     */
> > > > >>>>> __rte_experimental
> > > > >>>>> static __rte_always_inline
> > > > >>>>> __attribute__((__nonnull__(1, 2), __access__(write_only, 1,
> > 3),
> > > > >>>> __access__(read_only, 2, 3)))
> > > > >>>>> void rte_memcpy_nt(void * __rte_restrict dst, const void *
> > > > >>>> __rte_restrict src, size_t len)
> > > > >>>>> /* Implementation T.B.D. */
> > > > >>>>>
> > > > >>>>> /**
> > > > >>>>>     * @warning
> > > > >>>>>     * @b EXPERIMENTAL: this API may change without prior
> > notice.
> > > > >>>>>     *
> > > > >>>>>     * Copy data in blocks of 16 byte from aligned non-
> > temporal
> > > > >> source
> > > > >>>>>     * to aligned non-temporal destination.
> > > > >>>>>     *
> > > > >>>>>     * @param dst
> > > > >>>>>     *   Pointer to the non-temporal destination of the data.
> > > > >>>>>     *   Must be 16 byte aligned.
> > > > >>>>>     * @param src
> > > > >>>>>     *   Pointer to the non-temporal source data.
> > > > >>>>>     *   Must be 16 byte aligned.
> > > > >>>>>     * @param len
> > > > >>>>>     *   Number of bytes to copy.
> > > > >>>>>     *   Must be divisible by 16.
> > > > >>>>>     */
> > > > >>>>> __rte_experimental
> > > > >>>>> static __rte_always_inline
> > > > >>>>> __attribute__((__nonnull__(1, 2), __access__(write_only, 1,
> > 3),
> > > > >>>> __access__(read_only, 2, 3)))
> > > > >>>>> void rte_memcpy_nt16a(void * __rte_restrict dst, const void *
> > > > >>>> __rte_restrict src, size_t len)
> > > > >>>>> {
> > > > >>>>>        const void * const  end = RTE_PTR_ADD(src, len);
> > > > >>>>>
> > > > >>>>>        RTE_ASSERT(rte_is_aligned(dst, sizeof(__m128i)));
> > > > >>>>>        RTE_ASSERT(rte_is_aligned(src, sizeof(__m128i)));
> > > > >>>>>        RTE_ASSERT(rte_is_aligned(len, sizeof(__m128i)));
> > > > >>>>>
> > > > >>>>>        /* Copy large portion of data. */
> > > > >>>>>        while (RTE_PTR_DIFF(end, src) >= 4 * sizeof(__m128i))
> > {
> > > > >>>>>            register __m128i    xmm0, xmm1, xmm2, xmm3;
> > > > >>>>>
> > > > >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> > > > const
> > > > >>>> pointer as parameter. */
> > > > >>>>> #pragma GCC diagnostic push
> > > > >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> > > > >>>>>            xmm0 = _mm_stream_load_si128(RTE_PTR_ADD(src, 0 *
> > > > >>>> sizeof(__m128i)));
> > > > >>>>>            xmm1 = _mm_stream_load_si128(RTE_PTR_ADD(src, 1 *
> > > > >>>> sizeof(__m128i)));
> > > > >>>>>            xmm2 = _mm_stream_load_si128(RTE_PTR_ADD(src, 2 *
> > > > >>>> sizeof(__m128i)));
> > > > >>>>>            xmm3 = _mm_stream_load_si128(RTE_PTR_ADD(src, 3 *
> > > > >>>> sizeof(__m128i)));
> > > > >>>>> #pragma GCC diagnostic pop
> > > > >>>>>            _mm_stream_si128(RTE_PTR_ADD(dst, 0 *
> > > > sizeof(__m128i)),
> > > > >>>> xmm0);
> > > > >>>>>            _mm_stream_si128(RTE_PTR_ADD(dst, 1 *
> > > > sizeof(__m128i)),
> > > > >>>> xmm1);
> > > > >>>>>            _mm_stream_si128(RTE_PTR_ADD(dst, 2 *
> > > > sizeof(__m128i)),
> > > > >>>> xmm2);
> > > > >>>>>            _mm_stream_si128(RTE_PTR_ADD(dst, 3 *
> > > > sizeof(__m128i)),
> > > > >>>> xmm3);
> > > > >>>>>            src = RTE_PTR_ADD(src, 4 * sizeof(__m128i));
> > > > >>>>>            dst = RTE_PTR_ADD(dst, 4 * sizeof(__m128i));
> > > > >>>>>        }
> > > > >>>>>
> > > > >>>>>        /* Copy remaining data. */
> > > > >>>>>        while (src != end) {
> > > > >>>>>            register __m128i    xmm;
> > > > >>>>>
> > > > >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> > > > const
> > > > >>>> pointer as parameter. */
> > > > >>>>> #pragma GCC diagnostic push
> > > > >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> > > > >>>>>            xmm = _mm_stream_load_si128(src);
> > > > >>>>> #pragma GCC diagnostic pop
> > > > >>>>>            _mm_stream_si128(dst, xmm);
> > > > >>>>>            src = RTE_PTR_ADD(src, sizeof(__m128i));
> > > > >>>>>            dst = RTE_PTR_ADD(dst, sizeof(__m128i));
> > > > >>>>>        }
> > > > >>>>> }
> > > > >>>>>
> > > > >>>>> /**
> > > > >>>>>     * @warning
> > > > >>>>>     * @b EXPERIMENTAL: this API may change without prior
> > notice.
> > > > >>>>>     *
> > > > >>>>>     * Copy data in blocks of 4 byte from aligned non-temporal
> > > > source
> > > > >>>>>     * to aligned non-temporal destination.
> > > > >>>>>     *
> > > > >>>>>     * @param dst
> > > > >>>>>     *   Pointer to the non-temporal destination of the data.
> > > > >>>>>     *   Must be 4 byte aligned.
> > > > >>>>>     * @param src
> > > > >>>>>     *   Pointer to the non-temporal source data.
> > > > >>>>>     *   Must be 4 byte aligned.
> > > > >>>>>     * @param len
> > > > >>>>>     *   Number of bytes to copy.
> > > > >>>>>     *   Must be divisible by 4.
> > > > >>>>>     */
> > > > >>>>> __rte_experimental
> > > > >>>>> static __rte_always_inline
> > > > >>>>> __attribute__((__nonnull__(1, 2), __access__(write_only, 1,
> > 3),
> > > > >>>> __access__(read_only, 2, 3)))
> > > > >>>>> void rte_memcpy_nt4a(void * __rte_restrict dst, const void *
> > > > >>>> __rte_restrict src, size_t len)
> > > > >>>>> {
> > > > >>>>>        int32_t             buf[sizeof(__m128i) /
> > sizeof(int32_t)]
> > > > >>>> __rte_aligned(sizeof(__m128i));
> > > > >>>>>        /** Address of source data, rounded down to achieve
> > > > >> alignment.
> > > > >>>> */
> > > > >>>>>        const void *        srca = RTE_PTR_ALIGN_FLOOR(src,
> > > > >>>> sizeof(__m128i));
> > > > >>>>>        /** Address of end of source data, rounded down to
> > achieve
> > > > >>>> alignment. */
> > > > >>>>>        const void * const  srcenda =
> > > > >>>> RTE_PTR_ALIGN_FLOOR(RTE_PTR_ADD(src, len), sizeof(__m128i));
> > > > >>>>>        const int           offset =  RTE_PTR_DIFF(src, srca)
> > /
> > > > >>>> sizeof(int32_t);
> > > > >>>>>        register __m128i    xmm0;
> > > > >>>>>
> > > > >>>>>        RTE_ASSERT(rte_is_aligned(dst, sizeof(int32_t)));
> > > > >>>>>        RTE_ASSERT(rte_is_aligned(src, sizeof(int32_t)));
> > > > >>>>>        RTE_ASSERT(rte_is_aligned(len, sizeof(int32_t)));
> > > > >>>>>
> > > > >>>>>        if (unlikely(len == 0)) return;
> > > > >>>>>
> > > > >>>>>        /* Copy first, non-__m128i aligned, part of source
> > data.
> > > > */
> > > > >>>>>        if (offset) {
> > > > >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> > > > const
> > > > >>>> pointer as parameter. */
> > > > >>>>> #pragma GCC diagnostic push
> > > > >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> > > > >>>>>            xmm0 = _mm_stream_load_si128(srca);
> > > > >>>>>            _mm_store_si128((void *)buf, xmm0);
> > > > >>>>> #pragma GCC diagnostic pop
> > > > >>>>>            switch (offset) {
> > > > >>>>>                case 1:
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> > > > >>>> sizeof(int32_t)), buf[1]);
> > > > >>>>>                    if (unlikely(len == 1 * sizeof(int32_t)))
> > > > return;
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> > > > >>>> sizeof(int32_t)), buf[2]);
> > > > >>>>>                    if (unlikely(len == 2 * sizeof(int32_t)))
> > > > return;
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 2 *
> > > > >>>> sizeof(int32_t)), buf[3]);
> > > > >>>>>                    break;
> > > > >>>>>                case 2:
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> > > > >>>> sizeof(int32_t)), buf[2]);
> > > > >>>>>                    if (unlikely(len == 1 * sizeof(int32_t)))
> > > > return;
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> > > > >>>> sizeof(int32_t)), buf[3]);
> > > > >>>>>                    break;
> > > > >>>>>                case 3:
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> > > > >>>> sizeof(int32_t)), buf[3]);
> > > > >>>>>                    break;
> > > > >>>>>            }
> > > > >>>>>            srca = RTE_PTR_ADD(srca, (4 - offset) *
> > > > sizeof(int32_t));
> > > > >>>>>            dst = RTE_PTR_ADD(dst, (4 - offset) *
> > > > sizeof(int32_t));
> > > > >>>>>        }
> > > > >>>>>
> > > > >>>>>        /* Copy middle, __m128i aligned, part of source data.
> > */
> > > > >>>>>        while (srca != srcenda) {
> > > > >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> > > > const
> > > > >>>> pointer as parameter. */
> > > > >>>>> #pragma GCC diagnostic push
> > > > >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> > > > >>>>>            xmm0 = _mm_stream_load_si128(srca);
> > > > >>>>> #pragma GCC diagnostic pop
> > > > >>>>>            _mm_store_si128((void *)buf, xmm0);
> > > > >>>>>            _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> > sizeof(int32_t)),
> > > > >>>> buf[0]);
> > > > >>>>>            _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> > sizeof(int32_t)),
> > > > >>>> buf[1]);
> > > > >>>>>            _mm_stream_si32(RTE_PTR_ADD(dst, 2 *
> > sizeof(int32_t)),
> > > > >>>> buf[2]);
> > > > >>>>>            _mm_stream_si32(RTE_PTR_ADD(dst, 3 *
> > sizeof(int32_t)),
> > > > >>>> buf[3]);
> > > > >>>>>            srca = RTE_PTR_ADD(srca, sizeof(__m128i));
> > > > >>>>>            dst = RTE_PTR_ADD(dst, 4 * sizeof(int32_t));
> > > > >>>>>        }
> > > > >>>>>
> > > > >>>>>        /* Copy last, non-__m128i aligned, part of source
> > data. */
> > > > >>>>>        if (RTE_PTR_DIFF(srca, src) != 4) {
> > > > >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> > > > const
> > > > >>>> pointer as parameter. */
> > > > >>>>> #pragma GCC diagnostic push
> > > > >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> > > > >>>>>            xmm0 = _mm_stream_load_si128(srca);
> > > > >>>>>            _mm_store_si128((void *)buf, xmm0);
> > > > >>>>> #pragma GCC diagnostic pop
> > > > >>>>>            switch (offset) {
> > > > >>>>>                case 1:
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> > > > >>>> sizeof(int32_t)), buf[0]);
> > > > >>>>>                    break;
> > > > >>>>>                case 2:
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> > > > >>>> sizeof(int32_t)), buf[0]);
> > > > >>>>>                    if (unlikely(RTE_PTR_DIFF(srca, src) == 1
> > *
> > > > >>>> sizeof(int32_t))) return;
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> > > > >>>> sizeof(int32_t)), buf[1]);
> > > > >>>>>                    break;
> > > > >>>>>                case 3:
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> > > > >>>> sizeof(int32_t)), buf[0]);
> > > > >>>>>                    if (unlikely(RTE_PTR_DIFF(srca, src) == 1
> > *
> > > > >>>> sizeof(int32_t))) return;
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> > > > >>>> sizeof(int32_t)), buf[1]);
> > > > >>>>>                    if (unlikely(RTE_PTR_DIFF(srca, src) == 2
> > *
> > > > >>>> sizeof(int32_t))) return;
> > > > >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 2 *
> > > > >>>> sizeof(int32_t)), buf[2]);
> > > > >>>>>                    break;
> > > > >>>>>            }
> > > > >>>>>        }
> > > > >>>>> }
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >
> > > >