RE: [RFC v2] non-temporal memcpy

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> Sent: Friday, 29 July 2022 12.00
> 
> 24/07/2022 23:18, Morten Brørup пишет:
> >> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> >> Sent: Sunday, 24 July 2022 15.35
> >>
> >> 22/07/2022 11:44, Morten Brørup пишет:
> >>>> From: Konstantin Ananyev [mailto:konstantin.v.ananyev@yandex.ru]
> >>>> Sent: Friday, 22 July 2022 01.20
> >>>>
> >>>> Hi Morten,
> >>>>
> >>>>> This RFC proposes a set of functions optimized for non-temporal
> >>>> memory copy.
> >>>>>
> >>>>> At this stage, I am asking for feedback on the concept.
> >>>>>
> >>>>> Applications sometimes data to another memory location, which is
> >> only
> >>>> used
> >>>>> much later.
> >>>>> In this case, it is inefficient to pollute the data cache with
> the
> >>>> copied
> >>>>> data.
> >>>>>
> >>>>> An example use case (originating from a real life application):
> >>>>> Copying filtered packets, or the first part of them, into a
> capture
> >>>> buffer
> >>>>> for offline analysis.
> >>>>>
> >>>>> The purpose of these functions is to achieve a performance gain
> by
> >>>> not
> >>>>> polluting the cache when copying data.
> >>>>> Although the throughput may be improved by further optimization,
> I
> >> do
> >>>> not
> >>>>> consider througput optimization relevant initially.
> >>>>>
> >>>>> The x86 non-temporal load instructions have 16 byte alignment
> >>>>> requirements [1], while ARM non-temporal load instructions are
> >>>> available with
> >>>>> 4 byte alignment requirements [2].
> >>>>> Both platforms offer non-temporal store instructions with 4 byte
> >>>> alignment
> >>>>> requirements.
> >>>>>
> >>>>> In addition to the primary function without any alignment
> >>>> requirements, we
> >>>>> also provide functions for respectivly 16 and 4 byte aligned
> access
> >>>> for
> >>>>> performance purposes.
> >>>>>
> >>>>> The function names resemble standard C library function names,
> but
> >>>> their
> >>>>> signatures are intentionally different. No need to drag legacy
> into
> >>>> it.
> >>>>>
> >>>>> NB: Don't comment on spaces for indentation; a patch will follow
> >> DPDK
> >>>> coding
> >>>>> style and use TAB.
> >>>>
> >>>>
> >>>> I think there were discussions in other direction - remove
> >> rte_memcpy()
> >>>> completely and use memcpy() instead...
> >>>
> >>> Yes, the highly optimized rte_memcpy() implementation of memcpy()
> has
> >> become obsolete, now that modern compilers provide an efficient
> >> memcpy() implementation.
> >>>
> >>> It's an excellent reference, because we should learn from it, and
> >> avoid introducing similar mistakes with non-temporal memcpy.
> >>>
> >>>> But if we have a good use case for that, then I am positive in
> >>>> principle.
> >>>
> >>> The standard C library doesn't offer non-temporal memcpy(), so we
> >> need to implement it ourselves.
> >>>
> >>>> Though I think we need a clear use-case within dpdk for it
> >>>> to demonstrate perfomance gain.
> >>>
> >>> The performance gain is to avoid polluting the data cache. DPDK
> >> example applications, like l3fwd, are probably too primitive to
> measure
> >> any benefit in this regard.
> >>>
> >>>> Probably copying packets within pdump lib, or examples/dma. or ...
> >>>
> >>> Good point - the new functions should be used somewhere within
> DPDK.
> >> For this purpose, I will look into modifying rte_pktmbuf_copy(),
> which
> >> is used by pdump_copy(), to use non-temporal copying of the packet
> >> data.
> >>>
> >>>> Another thought - do we really need a separate inline function for
> >> each
> >>>> flavour?
> >>>> Might be just one non-inline rte_memcpy_nt(dst, src, size, flags),
> >>>> where flags could be combination of NT_SRC, NT_DST, and keep
> >> alignment
> >>>> detection/decisions to particular implementation?
> >>>
> >>> Thank you for the feedback, Konstantin.
> >>>
> >>> My answer to this suggestion gets a little longwinded...
> >>>
> >>> Looking at the DPDK pcapng library, it copies a 4 byte aligned
> >> metadata structure sized 28 byte. So it can do with 4 byte aligned
> >> functions.
> >>>
> >>> Our application can capture packets starting at the IP header,
> which
> >> is offset by 14 byte (Ethernet header size) from the packet buffer,
> so
> >> it requires 2 byte alignment. And thus, requiring 4 byte alignment
> is
> >> not acceptable.
> >>>
> >>> Our application uses 16 byte alignment in the capture buffer area,
> >> and can benefit from 16 byte aligned functions. Furthermore, x86
> >> processors require 16 byte alignment for non-temporal load
> >> instructions, so I think a 16 byte aligned non-temporal memcpy
> function
> >> should be offered.
> >>
> >>
> >> Yes, x86 needs 16B alignment for NT load/stores
> >> But that's supposed to be arch specific limitation,
> >> that we probably want to hide, no?
> >
> > Agree.
> >
> >> Inside the function can check alignment of both src and dst
> >> and decide should it use NT load/store instructions or just
> >> do normal copy.
> >
> > Yes, I'm experimenting with the x86 inline function shown below. And
> hopefully, with some "extern inline" or other magic, I can hide the
> different implementations in the arch specific headers, and only expose
> the function declaration of rte_memcpy_nt() in the common header.
> >
> > I'm currently working on the x86 implementation - when I'm satisfied
> with that, I'll look into how to hide the implementations in the arch
> specific header files, and only expose the common function declaration
> in the generic header file also used for documentation. I works for
> rte_memcpy(), so I can probably find the way to do it there.
> >
> > /*
> >   * Non-Temporal Memory Operations Flags.
> >   */
> >
> > #define RTE_MEMOPS_F_LENA_MASK  (UINT64_C(0xFE) << 0)   /** Length
> alignment mask. */
> > #define RTE_MEMOPS_F_LEN2A      (UINT64_C(2) << 0)      /** Length is
> 2 byte aligned. */
> > #define RTE_MEMOPS_F_LEN4A      (UINT64_C(4) << 0)      /** Length is
> 4 byte aligned. */
> > #define RTE_MEMOPS_F_LEN8A      (UINT64_C(8) << 0)      /** Length is
> 8 byte aligned. */
> > #define RTE_MEMOPS_F_LEN16A     (UINT64_C(16) << 0)     /** Length is
> 16 byte aligned. */
> > #define RTE_MEMOPS_F_LEN32A     (UINT64_C(32) << 0)     /** Length is
> 32 byte aligned. */
> > #define RTE_MEMOPS_F_LEN64A     (UINT64_C(64) << 0)     /** Length is
> 64 byte aligned. */
> > #define RTE_MEMOPS_F_LEN128A    (UINT64_C(128) << 0)    /** Length is
> 128 byte aligned. */
> >
> > #define RTE_MEMOPS_F_DSTA_MASK  (UINT64_C(0xFE) << 8)   /**
> Destination address alignment mask. */
> > #define RTE_MEMOPS_F_DST2A      (UINT64_C(2) << 8)      /**
> Destination address is 2 byte aligned. */
> > #define RTE_MEMOPS_F_DST4A      (UINT64_C(4) << 8)      /**
> Destination address is 4 byte aligned. */
> > #define RTE_MEMOPS_F_DST8A      (UINT64_C(8) << 8)      /**
> Destination address is 8 byte aligned. */
> > #define RTE_MEMOPS_F_DST16A     (UINT64_C(16) << 8)     /**
> Destination address is 16 byte aligned. */
> > #define RTE_MEMOPS_F_DST32A     (UINT64_C(32) << 8)     /**
> Destination address is 32 byte aligned. */
> > #define RTE_MEMOPS_F_DST64A     (UINT64_C(64) << 8)     /**
> Destination address is 64 byte aligned. */
> > #define RTE_MEMOPS_F_DST128A    (UINT64_C(128) << 8)    /**
> Destination address is 128 byte aligned. */
> >
> > #define RTE_MEMOPS_F_SRCA_MASK  (UINT64_C(0xFE) << 16)  /** Source
> address alignment mask. */
> > #define RTE_MEMOPS_F_SRC2A      (UINT64_C(2) << 16)     /** Source
> address is 2 byte aligned. */
> > #define RTE_MEMOPS_F_SRC4A      (UINT64_C(4) << 16)     /** Source
> address is 4 byte aligned. */
> > #define RTE_MEMOPS_F_SRC8A      (UINT64_C(8) << 16)     /** Source
> address is 8 byte aligned. */
> > #define RTE_MEMOPS_F_SRC16A     (UINT64_C(16) << 16)    /** Source
> address is 16 byte aligned. */
> > #define RTE_MEMOPS_F_SRC32A     (UINT64_C(32) << 16)    /** Source
> address is 32 byte aligned. */
> > #define RTE_MEMOPS_F_SRC64A     (UINT64_C(64) << 16)    /** Source
> address is 64 byte aligned. */
> > #define RTE_MEMOPS_F_SRC128A    (UINT64_C(128) << 16)   /** Source
> address is 128 byte aligned. */
> >
> > /**
> >   * @warning
> >   * @b EXPERIMENTAL: this API may change without prior notice.
> >   *
> >   * Non-temporal memory copy.
> >   * The memory areas must not overlap.
> >   *
> >   * @note
> >   * If the destination and/or length is unaligned, some copied bytes
> will be
> >   * stored in the destination memory area using temporal access.
> >   *
> >   * @param dst
> >   *   Pointer to the non-temporal destination memory area.
> >   * @param src
> >   *   Pointer to the non-temporal source memory area.
> >   * @param len
> >   *   Number of bytes to copy.
> >   * @param flags
> >   *   Hints for memory access.
> >   *   Any of the RTE_MEMOPS_F_LENnA, RTE_MEMOPS_F_DSTnA,
> RTE_MEMOPS_F_SRCnA flags.
> >   */
> > __rte_experimental
> > static __rte_always_inline
> > __attribute__((__nonnull__(1, 2), __access__(write_only, 1, 3),
> __access__(read_only, 2, 3)))
> > void rte_memcpy_nt(void * __rte_restrict dst, const void *
> __rte_restrict src, size_t len,
> >          const uint64_t flags)
> > {
> >      if (__builtin_constant_p(flags) ?
> >              ((flags & RTE_MEMOPS_F_LENA_MASK) >= RTE_MEMOPS_F_LEN16A
> &&
> >              (flags & RTE_MEMOPS_F_DSTA_MASK) >= RTE_MEMOPS_F_DST16A)
> :
> >              !(((uintptr_t)dst | len) & (16 - 1))) {
> >          if (__builtin_constant_p(flags) ?
> >                  (flags & RTE_MEMOPS_F_SRCA_MASK) >=
> RTE_MEMOPS_F_SRC16A :
> >                  !((uintptr_t)src & (16 - 1)))
> >              rte_memcpy_nt16a(dst, src, len/*, flags*/);
> >          else
> >              rte_memcpy_nt16dla(dst, src, len/*, flags*/);
> >      }
> >      else if (__builtin_constant_p(flags) ? (
> >              (flags & RTE_MEMOPS_F_LENA_MASK) >= RTE_MEMOPS_F_LEN4A
> &&
> >              (flags & RTE_MEMOPS_F_DSTA_MASK) >= RTE_MEMOPS_F_DST4A
> &&
> >              (flags & RTE_MEMOPS_F_SRCA_MASK) >= RTE_MEMOPS_F_SRC4A)
> :
> >              !(((uintptr_t)dst | (uintptr_t)src | len) & (4 - 1))) {
> >          rte_memcpy_nt4a(dst, src, len/*, flags*/);
> >      }
> >      else
> >          rte_memcpy_nt_unaligned(dst, src, len/*, flags*/);
> > }
> 
> 
> Do we really need to expose all these dozen flags?
> My thought at least about x86 implementaion was about something more
> simple like:
> void rte_memcpy_nt(void * __rte_restrict dst,
> 	const void * __rte_restrict src, size_t len,
> 	const uint64_t flags)
> {
> 
> 	if (flags == (SRC_NT | DST_NT) && ((dst | src) & 0xf) == 0) {
> 		_do_nt_src_nt_dst_nt(...);
> 	} else if (flags == DST_NT && (dst & 0xf) == 0) {
> 		_do_src_na_dst_nt(...);
> 	} else if (flags == SRC_NT && (src & 0xf) == 0) {
> 		_do_src_nt_dst_na(...);
> 	} else
> 		memcpy(dst, src, len);
> }

The combination of flags, inline and __builtin_constant_p() allows the compiler to produce zero-overhead code. Without it, the resulting code will contain a bunch of run-time bitmask comparisons and branches to determine the ultimate copy function. On x86 there are not only 16 byte, but also 4 byte alignment variants of non-temporal store. The beauty of it will be more obvious when the patch is ready.

And in my current working version (not the code provided here), the flags are hints, so using them will be optional. The function headers will look roughly like this:

static inline void rte_memcpy_nt_ex(
		void * dst, const void * src,
		size_t len, uint64_t flags);

static inline void rte_memcpy_nt(
		void * dst, const void * src,
		size_t len) 
{
	rte_memcpy_nt_ex(dst, src, len, 0);
}

I might add an _ex postfix variant of the mbuf packet non-temporal copy function too, but I'm not working on that function yet, so I don't yet know if it makes sense or not.

My concept for build time alignment hints can also be copied into an rte_memcpy_ex() function for improved performance. But I don't want my patch to expand too much outside its initial scope, so I will not modify rte_memcpy() with this patch.

Alternatively, I could provide rte_memcpy_ex(d,s,l,flags) instead of rte_memcpy_nt[_ex](), and use the flags to indicate non-temporal source and destination.

This is only a question about which API the community prefers. I will not change the implementation of rte_memcpy() with this patch - it's another job to do that.

> 
> >
> >
> >>
> >>
> >>> While working on these funtions, I experimented with an
> >> rte_memcpy_nt() taking flags, which is also my personal preference,
> but
> >> haven't succeed yet. Especially when copying a 16 byte aligned
> >> structure of only 16 byte, the overhead of the function call +
> >> comparing the flags + the copy loop overhead is significant,
> compared
> >> to inline code consisting of only one pair of "movntdqa
> (%rsi),%xmm0;
> >> movntdq %xmm0,(%rdi)" instructions.
> >>>
> >>> Remember that a non-inlined rte_memcpy_nt() will be called with
> very
> >> varying size, due to the typical mix of small and big packets, so
> >> branch prediction will not help.
> >>>
> >>> This RFC does not yet show the rte_memcpy_nt() function handling
> >> unaligned load/store, but it is more complex than the aligned
> >> functions. So I think the aligned variants are warranted - for
> >> performance reasons.
> >>>
> >>> Some of the need for exposing individual functions for different
> >> alignment stems from the compiler being unable to determine the
> >> alignment of the source and destination pointers at build time. So
> we
> >> need to help the compiler with this at build time, and thus the need
> >> for inlining the function. If we expose a bunch of small inline
> >> functions or a big inline function with flags seems to be a matter
> of
> >> taste.
> >>>
> >>> Thinking about it, you are probably right that exposing a single
> >> function with flags is better for documentation purposes and easier
> for
> >> other architectures to implement. But it still needs to be inline,
> for
> >> the reasons described above.
> >>
> >>
> >> Ok, my initial thought was that main use-case for it would be
> copying
> >> of
> >> big chunks of data, but from your description it might not be the
> case.
> >
> > This is for quickly copying relatively small pieces of data
> synchronously without polluting the CPUs data cache, e.g. just before
> passing on a packet to an Ethernet PMD for transmission.
> >
> > Big chunks of data should be copied asynchronously by DMA.
> >
> >> Yes, for just 16/32B copy function call overhead might be way too
> >> high...
> >> As another alternative - would memcpy_nt_bulk() help somehow?
> >> It can do copying for the several src/dst pairs at once and
> >> that might help to amortize cost of function call.
> >
> > In many cases, memcpy_nt() will replace memcpy() inside loops, so it
> should be just as easy to use as memcpy(). E.g. look at
> rte_pktmbuf_copy()... Building a memcopy array to pass to
> memcpy_nt_bulk() from rte_pktmbuf_copy() would require a significant
> rewrite of rte_pktmbuf_copy(), compared to just replacing rte_memcpy()
> with rte_memcpy_nt(). And this is just one function using memcpy().
> 
> Actually, one question I have for such small data-transfer
> (16B per packet) - do you still see some noticable perfomance
> improvement for such scenario?

Copying 16 byte from each packet in a burst of 32 packets would otherwise pollute 64 cache lines = 4 KB cache. With typically 64 KB L1 cache, I think it makes a difference.

> Another question - who will do 'sfence' after the copying?
> Would it be inside memcpy_nt (seems quite costly), or would
> it be another API function for that: memcpy_nt_flush() or so?

Outside. Only the developer knows when it is required, so it wouldn't make any sense to add the cost inside memcpy_nt().

I don't think we should add a flush function; it would just be another name for an already existing function. Referring to the required operation in the memcpy_nt() function documentation should suffice.

> 
> >>
> >>
> >>>
> >>>>
> >>>>
> >>>>> [1] https://www.intel.com/content/www/us/en/docs/intrinsics-
> >>>> guide/index.html#text=_mm_stream_load
> >>>>> [2] https://developer.arm.com/documentation/100076/0100/A64-
> >>>> Instruction-Set-Reference/A64-Floating-point-Instructions/LDNP--
> >> SIMD-
> >>>> and-FP-
> >>>>>
> >>>>> V2:
> >>>>> - Only copy from non-temporal source to non-temporal destination.
> >>>>>      I.e. remove the two variants with only source and/or
> >> destination
> >>>> being
> >>>>>      non-temporal.
> >>>>> - Do not require alignment.
> >>>>>      Instead, offer additional 4 and 16 byte aligned functions
> for
> >>>> performance
> >>>>>      purposes.
> >>>>> - Implemented two of the functions for x86.
> >>>>> - Remove memset function.
> >>>>>
> >>>>> Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
> >>>>> ---
> >>>>>
> >>>>> /**
> >>>>>     * @warning
> >>>>>     * @b EXPERIMENTAL: this API may change without prior notice.
> >>>>>     *
> >>>>>     * Copy data from non-temporal source to non-temporal
> >> destination.
> >>>>>     *
> >>>>>     * @param dst
> >>>>>     *   Pointer to the non-temporal destination of the data.
> >>>>>     *   Should be 4 byte aligned, for optimal performance.
> >>>>>     * @param src
> >>>>>     *   Pointer to the non-temporal source data.
> >>>>>     *   No alignment requirements.
> >>>>>     * @param len
> >>>>>     *   Number of bytes to copy.
> >>>>>     *   Should be be divisible by 4, for optimal performance.
> >>>>>     */
> >>>>> __rte_experimental
> >>>>> static __rte_always_inline
> >>>>> __attribute__((__nonnull__(1, 2), __access__(write_only, 1, 3),
> >>>> __access__(read_only, 2, 3)))
> >>>>> void rte_memcpy_nt(void * __rte_restrict dst, const void *
> >>>> __rte_restrict src, size_t len)
> >>>>> /* Implementation T.B.D. */
> >>>>>
> >>>>> /**
> >>>>>     * @warning
> >>>>>     * @b EXPERIMENTAL: this API may change without prior notice.
> >>>>>     *
> >>>>>     * Copy data in blocks of 16 byte from aligned non-temporal
> >> source
> >>>>>     * to aligned non-temporal destination.
> >>>>>     *
> >>>>>     * @param dst
> >>>>>     *   Pointer to the non-temporal destination of the data.
> >>>>>     *   Must be 16 byte aligned.
> >>>>>     * @param src
> >>>>>     *   Pointer to the non-temporal source data.
> >>>>>     *   Must be 16 byte aligned.
> >>>>>     * @param len
> >>>>>     *   Number of bytes to copy.
> >>>>>     *   Must be divisible by 16.
> >>>>>     */
> >>>>> __rte_experimental
> >>>>> static __rte_always_inline
> >>>>> __attribute__((__nonnull__(1, 2), __access__(write_only, 1, 3),
> >>>> __access__(read_only, 2, 3)))
> >>>>> void rte_memcpy_nt16a(void * __rte_restrict dst, const void *
> >>>> __rte_restrict src, size_t len)
> >>>>> {
> >>>>>        const void * const  end = RTE_PTR_ADD(src, len);
> >>>>>
> >>>>>        RTE_ASSERT(rte_is_aligned(dst, sizeof(__m128i)));
> >>>>>        RTE_ASSERT(rte_is_aligned(src, sizeof(__m128i)));
> >>>>>        RTE_ASSERT(rte_is_aligned(len, sizeof(__m128i)));
> >>>>>
> >>>>>        /* Copy large portion of data. */
> >>>>>        while (RTE_PTR_DIFF(end, src) >= 4 * sizeof(__m128i)) {
> >>>>>            register __m128i    xmm0, xmm1, xmm2, xmm3;
> >>>>>
> >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> const
> >>>> pointer as parameter. */
> >>>>> #pragma GCC diagnostic push
> >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> >>>>>            xmm0 = _mm_stream_load_si128(RTE_PTR_ADD(src, 0 *
> >>>> sizeof(__m128i)));
> >>>>>            xmm1 = _mm_stream_load_si128(RTE_PTR_ADD(src, 1 *
> >>>> sizeof(__m128i)));
> >>>>>            xmm2 = _mm_stream_load_si128(RTE_PTR_ADD(src, 2 *
> >>>> sizeof(__m128i)));
> >>>>>            xmm3 = _mm_stream_load_si128(RTE_PTR_ADD(src, 3 *
> >>>> sizeof(__m128i)));
> >>>>> #pragma GCC diagnostic pop
> >>>>>            _mm_stream_si128(RTE_PTR_ADD(dst, 0 *
> sizeof(__m128i)),
> >>>> xmm0);
> >>>>>            _mm_stream_si128(RTE_PTR_ADD(dst, 1 *
> sizeof(__m128i)),
> >>>> xmm1);
> >>>>>            _mm_stream_si128(RTE_PTR_ADD(dst, 2 *
> sizeof(__m128i)),
> >>>> xmm2);
> >>>>>            _mm_stream_si128(RTE_PTR_ADD(dst, 3 *
> sizeof(__m128i)),
> >>>> xmm3);
> >>>>>            src = RTE_PTR_ADD(src, 4 * sizeof(__m128i));
> >>>>>            dst = RTE_PTR_ADD(dst, 4 * sizeof(__m128i));
> >>>>>        }
> >>>>>
> >>>>>        /* Copy remaining data. */
> >>>>>        while (src != end) {
> >>>>>            register __m128i    xmm;
> >>>>>
> >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> const
> >>>> pointer as parameter. */
> >>>>> #pragma GCC diagnostic push
> >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> >>>>>            xmm = _mm_stream_load_si128(src);
> >>>>> #pragma GCC diagnostic pop
> >>>>>            _mm_stream_si128(dst, xmm);
> >>>>>            src = RTE_PTR_ADD(src, sizeof(__m128i));
> >>>>>            dst = RTE_PTR_ADD(dst, sizeof(__m128i));
> >>>>>        }
> >>>>> }
> >>>>>
> >>>>> /**
> >>>>>     * @warning
> >>>>>     * @b EXPERIMENTAL: this API may change without prior notice.
> >>>>>     *
> >>>>>     * Copy data in blocks of 4 byte from aligned non-temporal
> source
> >>>>>     * to aligned non-temporal destination.
> >>>>>     *
> >>>>>     * @param dst
> >>>>>     *   Pointer to the non-temporal destination of the data.
> >>>>>     *   Must be 4 byte aligned.
> >>>>>     * @param src
> >>>>>     *   Pointer to the non-temporal source data.
> >>>>>     *   Must be 4 byte aligned.
> >>>>>     * @param len
> >>>>>     *   Number of bytes to copy.
> >>>>>     *   Must be divisible by 4.
> >>>>>     */
> >>>>> __rte_experimental
> >>>>> static __rte_always_inline
> >>>>> __attribute__((__nonnull__(1, 2), __access__(write_only, 1, 3),
> >>>> __access__(read_only, 2, 3)))
> >>>>> void rte_memcpy_nt4a(void * __rte_restrict dst, const void *
> >>>> __rte_restrict src, size_t len)
> >>>>> {
> >>>>>        int32_t             buf[sizeof(__m128i) / sizeof(int32_t)]
> >>>> __rte_aligned(sizeof(__m128i));
> >>>>>        /** Address of source data, rounded down to achieve
> >> alignment.
> >>>> */
> >>>>>        const void *        srca = RTE_PTR_ALIGN_FLOOR(src,
> >>>> sizeof(__m128i));
> >>>>>        /** Address of end of source data, rounded down to achieve
> >>>> alignment. */
> >>>>>        const void * const  srcenda =
> >>>> RTE_PTR_ALIGN_FLOOR(RTE_PTR_ADD(src, len), sizeof(__m128i));
> >>>>>        const int           offset =  RTE_PTR_DIFF(src, srca) /
> >>>> sizeof(int32_t);
> >>>>>        register __m128i    xmm0;
> >>>>>
> >>>>>        RTE_ASSERT(rte_is_aligned(dst, sizeof(int32_t)));
> >>>>>        RTE_ASSERT(rte_is_aligned(src, sizeof(int32_t)));
> >>>>>        RTE_ASSERT(rte_is_aligned(len, sizeof(int32_t)));
> >>>>>
> >>>>>        if (unlikely(len == 0)) return;
> >>>>>
> >>>>>        /* Copy first, non-__m128i aligned, part of source data.
> */
> >>>>>        if (offset) {
> >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> const
> >>>> pointer as parameter. */
> >>>>> #pragma GCC diagnostic push
> >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> >>>>>            xmm0 = _mm_stream_load_si128(srca);
> >>>>>            _mm_store_si128((void *)buf, xmm0);
> >>>>> #pragma GCC diagnostic pop
> >>>>>            switch (offset) {
> >>>>>                case 1:
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> >>>> sizeof(int32_t)), buf[1]);
> >>>>>                    if (unlikely(len == 1 * sizeof(int32_t)))
> return;
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> >>>> sizeof(int32_t)), buf[2]);
> >>>>>                    if (unlikely(len == 2 * sizeof(int32_t)))
> return;
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 2 *
> >>>> sizeof(int32_t)), buf[3]);
> >>>>>                    break;
> >>>>>                case 2:
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> >>>> sizeof(int32_t)), buf[2]);
> >>>>>                    if (unlikely(len == 1 * sizeof(int32_t)))
> return;
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> >>>> sizeof(int32_t)), buf[3]);
> >>>>>                    break;
> >>>>>                case 3:
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> >>>> sizeof(int32_t)), buf[3]);
> >>>>>                    break;
> >>>>>            }
> >>>>>            srca = RTE_PTR_ADD(srca, (4 - offset) *
> sizeof(int32_t));
> >>>>>            dst = RTE_PTR_ADD(dst, (4 - offset) *
> sizeof(int32_t));
> >>>>>        }
> >>>>>
> >>>>>        /* Copy middle, __m128i aligned, part of source data. */
> >>>>>        while (srca != srcenda) {
> >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> const
> >>>> pointer as parameter. */
> >>>>> #pragma GCC diagnostic push
> >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> >>>>>            xmm0 = _mm_stream_load_si128(srca);
> >>>>> #pragma GCC diagnostic pop
> >>>>>            _mm_store_si128((void *)buf, xmm0);
> >>>>>            _mm_stream_si32(RTE_PTR_ADD(dst, 0 * sizeof(int32_t)),
> >>>> buf[0]);
> >>>>>            _mm_stream_si32(RTE_PTR_ADD(dst, 1 * sizeof(int32_t)),
> >>>> buf[1]);
> >>>>>            _mm_stream_si32(RTE_PTR_ADD(dst, 2 * sizeof(int32_t)),
> >>>> buf[2]);
> >>>>>            _mm_stream_si32(RTE_PTR_ADD(dst, 3 * sizeof(int32_t)),
> >>>> buf[3]);
> >>>>>            srca = RTE_PTR_ADD(srca, sizeof(__m128i));
> >>>>>            dst = RTE_PTR_ADD(dst, 4 * sizeof(int32_t));
> >>>>>        }
> >>>>>
> >>>>>        /* Copy last, non-__m128i aligned, part of source data. */
> >>>>>        if (RTE_PTR_DIFF(srca, src) != 4) {
> >>>>> /* Note: Workaround for _mm_stream_load_si128() not taking a
> const
> >>>> pointer as parameter. */
> >>>>> #pragma GCC diagnostic push
> >>>>> #pragma GCC diagnostic ignored "-Wdiscarded-qualifiers"
> >>>>>            xmm0 = _mm_stream_load_si128(srca);
> >>>>>            _mm_store_si128((void *)buf, xmm0);
> >>>>> #pragma GCC diagnostic pop
> >>>>>            switch (offset) {
> >>>>>                case 1:
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> >>>> sizeof(int32_t)), buf[0]);
> >>>>>                    break;
> >>>>>                case 2:
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> >>>> sizeof(int32_t)), buf[0]);
> >>>>>                    if (unlikely(RTE_PTR_DIFF(srca, src) == 1 *
> >>>> sizeof(int32_t))) return;
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> >>>> sizeof(int32_t)), buf[1]);
> >>>>>                    break;
> >>>>>                case 3:
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 0 *
> >>>> sizeof(int32_t)), buf[0]);
> >>>>>                    if (unlikely(RTE_PTR_DIFF(srca, src) == 1 *
> >>>> sizeof(int32_t))) return;
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 1 *
> >>>> sizeof(int32_t)), buf[1]);
> >>>>>                    if (unlikely(RTE_PTR_DIFF(srca, src) == 2 *
> >>>> sizeof(int32_t))) return;
> >>>>>                    _mm_stream_si32(RTE_PTR_ADD(dst, 2 *
> >>>> sizeof(int32_t)), buf[2]);
> >>>>>                    break;
> >>>>>            }
> >>>>>        }
> >>>>> }
> >>>>>
> >>>>
> >>>
> >>
> >
>