Re: [dpdk-dev] [PATCH] atomic: clarify use of memory barriers

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

Hi Oliver,

>- optimize some code to avoid a real memory barrier when not required    (timers, virtio, ...)

That seems like a good thing to me.

> - make the code more readable to distinguish between the 2 kinds of memory barrier.

That part seems a bit misleading to me.
rte_compiler_barier() - is a barrier just for a compiler, not for real cpu. 
It only guarantees that the compiler wouldn't reorder instructions across it while emitting the code.

Looking at Intel Memory Ordering rules (Intel System PG, section 8.2):

1) Reads may be reordered with older writes to different locations but not with older writes to the same location.

So with the following fragment of code:
int a;
extern int *x, *y;
L0:
*y = 0;
rte_compiler_barrier();
L1:
 a = *x;

There is no guarantee that store at L0 will always be finished before load at L1.
Which means to me that rte_smp_mb() can't be identical to compiler_barrier, but should be real 'mfence' instruction instead.  

2) Writes to memory are not reordered with other writes, with the following exceptions:
   ...
   streaming stores (writes) executed with the non-temporal move instructions (MOVNTI, MOVNTQ, MOVNTDQ, MOVNTPS, and MOVNTPD); 
   ...

So with the following fragment of code:
extern int *x;
extern  __128i a, *p;
L0: 
_mm_stream_si128( p, a);
rte_compiler_barrier();
L1:
*x = 0;

There is no guarantee that store at L0 will always be finished before store at L1.
Which means to me that rte_smp_wmb() can't be identical to compiler_barrier, but should be real 'sfence' instruction instead.  

The only replacement that seems safe to me is:
#define	rte_smp_rmb() rte_compiler_barrier()

But now, there seems a confusion: everyone has to remember that smp_mb() and smp_wmb() are 'real' fences, while smp_rmb() is not.
That's why my suggestion was to simply keep using compiler_barrier() for all cases, when we don't need real fence.

Thanks
Konstantin

-----Original Message-----
From: Olivier MATZ [mailto:olivier.matz@6wind.com] 
Sent: Tuesday, May 20, 2014 1:13 PM
To: Ananyev, Konstantin; dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH] atomic: clarify use of memory barriers

Hi Konstantin,

Thank you for your review and feedback.

On 05/20/2014 12:05 PM, Ananyev, Konstantin wrote:
>> Note that on x86 CPUs, memory barriers between different cores can be guaranteed by a simple compiler barrier.
>
> I don't think this is totally correct.
> Yes, for  Intel cpus in many cases memory barrier could be avoided due to nearly strict memory ordering.
> Though there are few cases where reordering is possible and when fence instructions would be needed.

I tried to mimic the behavior of linux that differentiates *mb() from
smp_*mb(), but I did too fast. In linux, we have [1]:

   smp_mb() = mb() = asm volatile("mfence":::"memory")
   smp_rmb() = compiler_barrier()
   smp_wmb() = compiler_barrier()

At least this should fixed in the patch. By the way, just for reference,
the idea of the patch came from a discussion we had on the list [2].

> For me:
> +#define	rte_smp_rmb() rte_compiler_barrier()
> Seems a bit misleading, as there is no real fence.
> So I suggest we keep rte_compiler_barrier() naming and usage.

The objectives of the patch (which was probably not explained very
clearly in the commit log) were:
- make the code more readable to distinguish between the 2 kinds of
   memory barrier.
- optimize some code to avoid a real memory barrier when not required
   (timers, virtio, ...)

Having a compiler barrier in place of a memory barrier in the code
does not really help to understand what the developper wanted to do.
In the current code we can see that the use of rte_compiler_barrier()
is ambiguous, as it need a comment to clarify the situation:

	rte_compiler_barrier();   /* rmb */

Don't you think we could fix the patch but keep its logic?

Regards,
Olivier

[1] http://lxr.free-electrons.com/source/arch/x86/include/asm/barrier.h#L81
[2] http://dpdk.org/ml/archives/dev/2014-March/001741.html