RE: [RFC v2] non-temporal memcpy

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Mattias Rönnblom [mailto:hofors@lysator.liu.se]
> Sent: Sunday, 7 August 2022 22.20
> 
> On 2022-07-29 22:26, Morten Brørup wrote:
> > +TO: @Honnappa, we need input from ARM
> >
> >> From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
> >> Sent: Friday, 29 July 2022 21.49
> >>>
> >>>> From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
> >>>> Sent: Friday, 29 July 2022 14.14
> >>>>
> >>>>
> >>>> Sorry, missed that part.
> >>>>
> >>>>>
> >>>>>> Another question - who will do 'sfence' after the copying?
> >>>>>> Would it be inside memcpy_nt (seems quite costly), or would
> >>>>>> it be another API function for that: memcpy_nt_flush() or so?
> >>>>>
> >>>>> Outside. Only the developer knows when it is required, so it
> >> wouldn't
> >>>> make any sense to add the cost inside memcpy_nt().
> >>>>>
> >>>>> I don't think we should add a flush function; it would just be
> >>>> another name for an already existing function. Referring to the
> >>>> required
> >>>>> operation in the memcpy_nt() function documentation should
> >> suffice.
> >>>>>
> >>>>
> >>>> Ok, but again wouldn't it be arch specific?
> >>>> AFAIK for x86 it needs to boil down to sfence, for other
> >> architectures
> >>>> - I don't know.
> >>>> If you think there already is some generic one (rte_wmb?) that
> >> would
> >>>> always produce
> >>>> correct instructions - sure let's use it.
> >>>>
> >>>
> >>> DPDK has generic functions to wrap architecture specific stuff like
> >> memory barriers.
> >>>
> >>> Because they are non-temporal stores, I suspect that rte_mb() is
> >> required before reading the data from the location it was copied to.
> >>> Ensuring that STORE operations are ordered (rte_wmb) might not
> >> suffice. However, I'm not a CPU expert, so I will seek advice from
> >>> more qualified people in the community on this.
> >>
> >> I think for IA sfence is enough, see citation below,
> >> for other architectures - no idea.
> >> What I am trying to say - it needs to be the *same* function on all
> >> archs we support.
> >
> > Now I get it: rte_wmb() might be appropriate on x86, but if any other
> architecture requires something else, we should add a new common
> function for flushing, e.g. rte_memcpy_nt_flush().
> >
> 
> rte_wmb() not being enough also my understanding. NT stores are weakly
> ordered on x86, and requires a sfence to be ordered with non-NT stores.
> 
> Unfortunately, this per-memcpy sfence instruction make even 1500-byte
> sized copy operations much slower - at least in micro benchmarks.

I agree that calling rte_mb() from each NT memcpy would be counterproductive on small/medium copy operations.

Which is why rte_mb() is not called by the NT memcpy function itself, but by the application. This requirement will be part of the function's documentation.

The application must call rte_mb() before it accesses the copy.

Alternatively, the application can call rte_mb() after a burst of NT memcopies.

> 
> >>
> >> IA SW optimization manual:
> >> 9.4.2 Streaming Store Usage Models
> >> The two primary usage domains for streaming store are coherent
> requests
> >> and non-coherent requests.
> >> 9.4.2.1 Coherent Requests
> >> Coherent requests are normal loads and stores to system memory,
> which
> >> may also hit cache lines
> >> present in another processor in a multiprocessor environment. With
> >> coherent requests, a streaming store
> >> can be used in the same way as a regular store that has been mapped
> >> with a WC memory type (PAT or
> >> MTRR). An SFENCE instruction must be used within a producer-consumer
> >> usage model in order to ensure
> >> coherency and visibility of data between processors.
> >> Within a single-processor system, the CPU can also re-read the same
> >> memory location and be assured of
> >> coherence (that is, a single, consistent view of this memory
> location).
> >> The same is true for a multiprocessor
> >> (MP) system, assuming an accepted MP software producer-consumer
> >> synchronization policy is
> >> employed.
> >>
> >
> > With this reference, I am convinced that you are right about the
> SFENCE. This puts a checkmark on this item on my TODO list for the
> patch. Thank you, Konstantin!
> >
> > Any ARM CPU experts on the mailing list seeing this, not on vacation?
> @Honnappa, I'm looking at you. :-)
> >
> > Summing up, the question is:
> >
> > After a bunch of *non-temporal* stores (STNP instruction) on ARM
> architecture, does calling rte_wmb() suffice to ensure the data is
> visible across the system?
> >