[dpdk-dev] [PATCH] kni: fix kni rx fifo producer synchronization

[dpdk-dev] [PATCH] kni: fix kni rx fifo producer synchronization

[Kiran Kumar]

With existing code in kni_fifo_put, rx_q values are not being updated
before updating fifo_write. While reading rx_q in kni_net_rx_normal,
This is causing the sync issue on other core. So adding a write
barrier to make sure the values being synced before updating fifo_write.

Fixes: 3fc5ca2f6352 ("kni: initial import")

Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
---
 lib/librte_kni/rte_kni_fifo.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/librte_kni/rte_kni_fifo.h b/lib/librte_kni/rte_kni_fifo.h
index ac26a8c..4d6b33e 100644
--- a/lib/librte_kni/rte_kni_fifo.h
+++ b/lib/librte_kni/rte_kni_fifo.h
@@ -39,6 +39,7 @@ kni_fifo_put(struct rte_kni_fifo *fifo, void **data, unsigned num)
 		fifo->buffer[fifo_write] = data[i];
 		fifo_write = new_write;
 	}
+	rte_smp_wmb();
 	fifo->write = fifo_write;
 	return i;
 }
-- 
2.7.4

Re: [dpdk-dev] [PATCH] kni: fix kni rx fifo producer synchronization

[Ferruh Yigit]

On 8/9/2018 11:23 AM, Kiran Kumar wrote:
> With existing code in kni_fifo_put, rx_q values are not being updated
> before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> This is causing the sync issue on other core. So adding a write
> barrier to make sure the values being synced before updating fifo_write.
> 
> Fixes: 3fc5ca2f6352 ("kni: initial import")
> 
> Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> ---
>  lib/librte_kni/rte_kni_fifo.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/lib/librte_kni/rte_kni_fifo.h b/lib/librte_kni/rte_kni_fifo.h
> index ac26a8c..4d6b33e 100644
> --- a/lib/librte_kni/rte_kni_fifo.h
> +++ b/lib/librte_kni/rte_kni_fifo.h
> @@ -39,6 +39,7 @@ kni_fifo_put(struct rte_kni_fifo *fifo, void **data, unsigned num)
>  		fifo->buffer[fifo_write] = data[i];
>  		fifo_write = new_write;
>  	}
> +	rte_smp_wmb();
>  	fifo->write = fifo_write;
>  	return i;

For Intel this is just a compiler barrier so no issue but not sure if a memory
barrier is required here,

Related code block is:

|-          for (i = 0; i < num; i++) {
||                  new_write = (new_write + 1) & (fifo->len - 1);
||
||                  if (new_write == fifo_read)
||                          break;
||                  fifo->buffer[fifo_write] = data[i];
||                  fifo_write = new_write;
||          }
|           fifo->write = fifo_write;

"fifo_write" is updated in the loop, so there is a dependency to it for
"fifo->write". Can memory writes be reordered when there is a dependency?

Cc'ed a few more people for comment.

Re: [dpdk-dev] [PATCH] kni: fix kni rx fifo producer synchronization

[Jerin Jacob]

-----Original Message-----
> Date: Thu, 9 Aug 2018 12:30:57 +0100
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> CC: dev@dpdk.org, Jerin Jacob <jerin.jacob@caviumnetworks.com>, Konstantin
>  Ananyev <konstantin.ananyev@intel.com>, Bruce Richardson
>  <bruce.richardson@intel.com>, Santosh Shukla
>  <santosh.shukla@caviumnetworks.com>
> Subject: Re: [dpdk-dev] [PATCH] kni: fix kni rx fifo producer
>  synchronization
> User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
>  Thunderbird/52.9.1
> 
> 
> On 8/9/2018 11:23 AM, Kiran Kumar wrote:
> > With existing code in kni_fifo_put, rx_q values are not being updated
> > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > This is causing the sync issue on other core. So adding a write
> > barrier to make sure the values being synced before updating fifo_write.
> >
> > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >
> > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> > ---
> >  lib/librte_kni/rte_kni_fifo.h | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/lib/librte_kni/rte_kni_fifo.h b/lib/librte_kni/rte_kni_fifo.h
> > index ac26a8c..4d6b33e 100644
> > --- a/lib/librte_kni/rte_kni_fifo.h
> > +++ b/lib/librte_kni/rte_kni_fifo.h
> > @@ -39,6 +39,7 @@ kni_fifo_put(struct rte_kni_fifo *fifo, void **data, unsigned num)
> >               fifo->buffer[fifo_write] = data[i];
> >               fifo_write = new_write;
> >       }
> > +     rte_smp_wmb();
> >       fifo->write = fifo_write;
> >       return i;
> 
> For Intel this is just a compiler barrier so no issue but not sure if a memory
> barrier is required here,
> 
> Related code block is:
> 
> |-          for (i = 0; i < num; i++) {
> ||                  new_write = (new_write + 1) & (fifo->len - 1);
> ||
> ||                  if (new_write == fifo_read)
> ||                          break;
> ||                  fifo->buffer[fifo_write] = data[i];
> ||                  fifo_write = new_write;
> ||          }
> |           fifo->write = fifo_write;
> 
> "fifo_write" is updated in the loop, so there is a dependency to it for
> "fifo->write". Can memory writes be reordered when there is a dependency?

>From CPU0 compiler instruction generation perspective, It will take care of the
dependency. In weakly ordered machine, it does NOT grandees CPU1 sees
fifo->write as final write.

So, IMO, This patch make sense. We are able to reproduce the problem on
arm64 machine with certain traffic.

> 
> Cc'ed a few more people for comment.

Re: [dpdk-dev] [PATCH] kni: fix kni rx fifo producer synchronization

[Ferruh Yigit]

On 8/16/2018 7:52 AM, Jerin Jacob wrote:
> -----Original Message-----
>> Date: Thu, 9 Aug 2018 12:30:57 +0100
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
>> CC: dev@dpdk.org, Jerin Jacob <jerin.jacob@caviumnetworks.com>, Konstantin
>>  Ananyev <konstantin.ananyev@intel.com>, Bruce Richardson
>>  <bruce.richardson@intel.com>, Santosh Shukla
>>  <santosh.shukla@caviumnetworks.com>
>> Subject: Re: [dpdk-dev] [PATCH] kni: fix kni rx fifo producer
>>  synchronization
>> User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
>>  Thunderbird/52.9.1
>>
>>
>> On 8/9/2018 11:23 AM, Kiran Kumar wrote:
>>> With existing code in kni_fifo_put, rx_q values are not being updated
>>> before updating fifo_write. While reading rx_q in kni_net_rx_normal,
>>> This is causing the sync issue on other core. So adding a write
>>> barrier to make sure the values being synced before updating fifo_write.
>>>
>>> Fixes: 3fc5ca2f6352 ("kni: initial import")
>>>
>>> Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
>>> ---
>>>  lib/librte_kni/rte_kni_fifo.h | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/lib/librte_kni/rte_kni_fifo.h b/lib/librte_kni/rte_kni_fifo.h
>>> index ac26a8c..4d6b33e 100644
>>> --- a/lib/librte_kni/rte_kni_fifo.h
>>> +++ b/lib/librte_kni/rte_kni_fifo.h
>>> @@ -39,6 +39,7 @@ kni_fifo_put(struct rte_kni_fifo *fifo, void **data, unsigned num)
>>>               fifo->buffer[fifo_write] = data[i];
>>>               fifo_write = new_write;
>>>       }
>>> +     rte_smp_wmb();
>>>       fifo->write = fifo_write;
>>>       return i;
>>
>> For Intel this is just a compiler barrier so no issue but not sure if a memory
>> barrier is required here,
>>
>> Related code block is:
>>
>> |-          for (i = 0; i < num; i++) {
>> ||                  new_write = (new_write + 1) & (fifo->len - 1);
>> ||
>> ||                  if (new_write == fifo_read)
>> ||                          break;
>> ||                  fifo->buffer[fifo_write] = data[i];
>> ||                  fifo_write = new_write;
>> ||          }
>> |           fifo->write = fifo_write;
>>
>> "fifo_write" is updated in the loop, so there is a dependency to it for
>> "fifo->write". Can memory writes be reordered when there is a dependency?
> 
> From CPU0 compiler instruction generation perspective, It will take care of the
> dependency. In weakly ordered machine, it does NOT grandees CPU1 sees
> fifo->write as final write.
> 
> So, IMO, This patch make sense. We are able to reproduce the problem on
> arm64 machine with certain traffic.

OK, thanks for clarification.

> 
>>
>> Cc'ed a few more people for comment.

Re: [dpdk-dev] [PATCH] kni: fix kni rx fifo producer synchronization

[Jerin Jacob]

-----Original Message-----
> Date: Thu,  9 Aug 2018 15:53:53 +0530
> From: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> To: ferruh.yigit@intel.com
> CC: dev@dpdk.org, Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> Subject: [dpdk-dev]  [PATCH] kni: fix kni rx fifo producer synchronization
> X-Mailer: git-send-email 2.7.4
> 
> External Email
> 
> With existing code in kni_fifo_put, rx_q values are not being updated
> before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> This is causing the sync issue on other core. So adding a write
> barrier to make sure the values being synced before updating fifo_write.
> 
> Fixes: 3fc5ca2f6352 ("kni: initial import")
> 

Please fix following check patch issue.
Wrong headline lowercase:
	kni: fix kni rx fifo producer synchronization

and Cc stable@dpdk.org


With above change:

Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>


> Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> ---
>  lib/librte_kni/rte_kni_fifo.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/lib/librte_kni/rte_kni_fifo.h b/lib/librte_kni/rte_kni_fifo.h
> index ac26a8c..4d6b33e 100644
> --- a/lib/librte_kni/rte_kni_fifo.h
> +++ b/lib/librte_kni/rte_kni_fifo.h
> @@ -39,6 +39,7 @@ kni_fifo_put(struct rte_kni_fifo *fifo, void **data, unsigned num)
>                 fifo->buffer[fifo_write] = data[i];
>                 fifo_write = new_write;
>         }
> +       rte_smp_wmb();
>         fifo->write = fifo_write;
>         return i;
>  }
> --
> 2.7.4
> 

[dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Kiran Kumar]

With existing code in kni_fifo_put, rx_q values are not being updated
before updating fifo_write. While reading rx_q in kni_net_rx_normal,
This is causing the sync issue on other core. So adding a write
barrier to make sure the values being synced before updating fifo_write.

Fixes: 3fc5ca2f6352 ("kni: initial import")

Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
---
 v2 changes:
	- Changed rx in headline
 lib/librte_kni/rte_kni_fifo.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/librte_kni/rte_kni_fifo.h b/lib/librte_kni/rte_kni_fifo.h
index ac26a8c..4d6b33e 100644
--- a/lib/librte_kni/rte_kni_fifo.h
+++ b/lib/librte_kni/rte_kni_fifo.h
@@ -39,6 +39,7 @@ kni_fifo_put(struct rte_kni_fifo *fifo, void **data, unsigned num)
 		fifo->buffer[fifo_write] = data[i];
 		fifo_write = new_write;
 	}
+	rte_smp_wmb();
 	fifo->write = fifo_write;
 	return i;
 }
--
2.7.4

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Ferruh Yigit]

On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> With existing code in kni_fifo_put, rx_q values are not being updated
> before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> This is causing the sync issue on other core. So adding a write
> barrier to make sure the values being synced before updating fifo_write.
> 
> Fixes: 3fc5ca2f6352 ("kni: initial import")
> 
> Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>

Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Gavin Hu]

This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Ferruh Yigit
> Sent: Monday, August 27, 2018 10:08 PM
> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com>;
> jerin.jacob@caviumnetworks.com
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
>
> On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > With existing code in kni_fifo_put, rx_q values are not being updated
> > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > This is causing the sync issue on other core. So adding a write
> > barrier to make sure the values being synced before updating fifo_write.
> >
> > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >
> > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Kokkilagadda, Kiran]

In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.



________________________________
From: Gavin Hu <Gavin.Hu@arm.com>
Sent: Monday, August 27, 2018 9:10 PM
To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
Cc: dev@dpdk.org; Honnappa Nagarahalli
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

External Email

This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Ferruh Yigit
> Sent: Monday, August 27, 2018 10:08 PM
> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com>;
> jerin.jacob@caviumnetworks.com
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
>
> On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > With existing code in kni_fifo_put, rx_q values are not being updated
> > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > This is causing the sync issue on other core. So adding a write
> > barrier to make sure the values being synced before updating fifo_write.
> >
> > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >
> > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Kokkilagadda, Kiran]

I need to add the same write barrier change in kernel side kni_fifo_put. I will add it and will send v3.


________________________________
From: Kokkilagadda, Kiran
Sent: Tuesday, August 28, 2018 4:13:59 PM
To: Gavin Hu; Ferruh Yigit; Jacob, Jerin
Cc: dev@dpdk.org; Honnappa Nagarahalli
Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization


In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.



________________________________
From: Gavin Hu <Gavin.Hu@arm.com>
Sent: Monday, August 27, 2018 9:10 PM
To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
Cc: dev@dpdk.org; Honnappa Nagarahalli
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

External Email

This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Ferruh Yigit
> Sent: Monday, August 27, 2018 10:08 PM
> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com>;
> jerin.jacob@caviumnetworks.com
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
>
> On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > With existing code in kni_fifo_put, rx_q values are not being updated
> > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > This is causing the sync issue on other core. So adding a write
> > barrier to make sure the values being synced before updating fifo_write.
> >
> > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >
> > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com>
> > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Gavin Hu]

Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get "too new" data and get lost of the old data.

From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>
Sent: Tuesday, August 28, 2018 6:44 PM
To: Gavin Hu <Gavin.Hu@arm.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization


In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.



________________________________
From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
Sent: Monday, August 27, 2018 9:10 PM
To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

External Email

This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> Sent: Monday, August 27, 2018 10:08 PM
> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
> jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
>
> On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > With existing code in kni_fifo_put, rx_q values are not being updated
> > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > This is causing the sync issue on other core. So adding a write
> > barrier to make sure the values being synced before updating fifo_write.
> >
> > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >
> > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
> > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
>
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Honnappa Nagarahalli]

I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
IMO, we should solve this using c11 atomics. This will also help remove the use of 'volatile' from 'rte_kni_fifo' structure.

If you want us to put together a patch with this idea, please let us know.

Thank you,
Honnappa

From: Gavin Hu
Sent: Tuesday, August 28, 2018 2:31 PM
To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get "too new" data and get lost of the old data.

From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
Sent: Tuesday, August 28, 2018 6:44 PM
To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization


In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.



________________________________
From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
Sent: Monday, August 27, 2018 9:10 PM
To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

External Email

This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> Sent: Monday, August 27, 2018 10:08 PM
> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
> jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
>
> On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > With existing code in kni_fifo_put, rx_q values are not being updated
> > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > This is causing the sync issue on other core. So adding a write
> > barrier to make sure the values being synced before updating fifo_write.
> >
> > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >
> > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
> > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
>
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Kokkilagadda, Kiran]

Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.


________________________________
From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
Sent: Wednesday, August 29, 2018 10:29 AM
To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization


External Email

I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.

IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.



If you want us to put together a patch with this idea, please let us know.



Thank you,

Honnappa



From: Gavin Hu
Sent: Tuesday, August 28, 2018 2:31 PM
To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization



Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.

We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.



From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
Sent: Tuesday, August 28, 2018 6:44 PM
To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization



In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.





________________________________

From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
Sent: Monday, August 27, 2018 9:10 PM
To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization



External Email

This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> Sent: Monday, August 27, 2018 10:08 PM
> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
> jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
>
> On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > With existing code in kni_fifo_put, rx_q values are not being updated
> > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > This is causing the sync issue on other core. So adding a write
> > barrier to make sure the values being synced before updating fifo_write.
> >
> > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >
> > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
> > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
>
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Ola Liljedahl]

Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?

-- Ola

From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
Date: Wednesday, 29 August 2018 at 07:50
To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization


Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.

________________________________
From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
Sent: Wednesday, August 29, 2018 10:29 AM
To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization


External Email

I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.

IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.



If you want us to put together a patch with this idea, please let us know.



Thank you,

Honnappa



From: Gavin Hu
Sent: Tuesday, August 28, 2018 2:31 PM
To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization



Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.

We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.



From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
Sent: Tuesday, August 28, 2018 6:44 PM
To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization



In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.





________________________________

From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
Sent: Monday, August 27, 2018 9:10 PM
To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization



External Email

This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.

> -----Original Message-----
> From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> Sent: Monday, August 27, 2018 10:08 PM
> To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
> jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
>
> On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > With existing code in kni_fifo_put, rx_q values are not being updated
> > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > This is causing the sync issue on other core. So adding a write
> > barrier to make sure the values being synced before updating fifo_write.
> >
> > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >
> > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
> > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
>
> Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Jerin Jacob]

-----Original Message-----
> Date: Wed, 29 Aug 2018 07:34:34 +0000
> From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
>  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
>  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>  <Jerin.JacobKollanukkaran@cavium.com>
> CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve Capper
>  <Steve.Capper@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>  synchronization
> user-agent: Microsoft-MacOutlook/10.10.0.180812
> 
> Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?

What would be the change in interface? Is it removing the volatile for
C11 case, Then you can use anonymous union OR #define to keep the size 
and offset of the element intact.

struct rte_kni_fifo { 
#ifndef RTE_C11...
        volatile unsigned write;     /**< Next position to be written*/
        volatile unsigned read;      /**< Next position to be read */
#else
        unsigned write;     /**< Next position to be written*/
        unsigned read;      /**< Next position to be read */
#endif
        unsigned len;                /**< Circular buffer length */
        unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
        void *volatile buffer[];     /**< The buffer contains mbuf
pointers */
};

Anonymous union example:
https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461

You can check the ABI breakage by devtools/validate-abi.sh

> 
> -- Ola
> 
> From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
> Date: Wednesday, 29 August 2018 at 07:50
> To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
> Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> 
> 
> Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.
> 
> ________________________________
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Wednesday, August 29, 2018 10:29 AM
> To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
> Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> 
> 
> External Email
> 
> I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
> 
> IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
> 
> 
> 
> If you want us to put together a patch with this idea, please let us know.
> 
> 
> 
> Thank you,
> 
> Honnappa
> 
> 
> 
> From: Gavin Hu
> Sent: Tuesday, August 28, 2018 2:31 PM
> To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
> Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> 
> 
> 
> Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
> 
> We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.
> 
> 
> 
> From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
> Sent: Tuesday, August 28, 2018 6:44 PM
> To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> 
> 
> 
> In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.
> 
> 
> 
> 
> 
> ________________________________
> 
> From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
> Sent: Monday, August 27, 2018 9:10 PM
> To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> 
> 
> 
> External Email
> 
> This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.
> 
> > -----Original Message-----
> > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> > Sent: Monday, August 27, 2018 10:08 PM
> > To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
> > jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> > synchronization
> >
> > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > > With existing code in kni_fifo_put, rx_q values are not being updated
> > > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > > This is causing the sync issue on other core. So adding a write
> > > barrier to make sure the values being synced before updating fifo_write.
> > >
> > > Fixes: 3fc5ca2f6352 ("kni: initial import")
> > >
> > > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
> > > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
> >
> > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Ola Liljedahl]

There was a mention of rte_ring which is a different data structure. But perhaps I misunderstood why this was mentioned and the idea was only to use the C11 memory model as is also used in rte_ring nowadays.

But why would we have different code for x86 and for other architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC __atomic builtins), the code generated for x86 will be the same. __atomic_load(__ATOMIC_ACQUIRE) and __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and stores on x86?

-- Ola

On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com> wrote:

    -----Original Message-----
    > Date: Wed, 29 Aug 2018 07:34:34 +0000
    > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
    > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
    >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
    >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
    >  <Jerin.JacobKollanukkaran@cavium.com>
    > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve Capper
    >  <Steve.Capper@arm.com>
    > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
    >  synchronization
    > user-agent: Microsoft-MacOutlook/10.10.0.180812
    > 
    > Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?
    
    What would be the change in interface? Is it removing the volatile for
    C11 case, Then you can use anonymous union OR #define to keep the size 
    and offset of the element intact.
    
    struct rte_kni_fifo { 
    #ifndef RTE_C11...
            volatile unsigned write;     /**< Next position to be written*/
            volatile unsigned read;      /**< Next position to be read */
    #else
            unsigned write;     /**< Next position to be written*/
            unsigned read;      /**< Next position to be read */
    #endif
            unsigned len;                /**< Circular buffer length */
            unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
            void *volatile buffer[];     /**< The buffer contains mbuf
    pointers */
    };
    
    Anonymous union example:
    https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
    
    You can check the ABI breakage by devtools/validate-abi.sh
    
    > 
    > -- Ola
    > 
    > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
    > Date: Wednesday, 29 August 2018 at 07:50
    > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
    > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
    > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
    > 
    > 
    > Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.
    > 
    > ________________________________
    > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
    > Sent: Wednesday, August 29, 2018 10:29 AM
    > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
    > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
    > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
    > 
    > 
    > External Email
    > 
    > I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
    > 
    > IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
    > 
    > 
    > 
    > If you want us to put together a patch with this idea, please let us know.
    > 
    > 
    > 
    > Thank you,
    > 
    > Honnappa
    > 
    > 
    > 
    > From: Gavin Hu
    > Sent: Tuesday, August 28, 2018 2:31 PM
    > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
    > Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
    > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
    > 
    > 
    > 
    > Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
    > 
    > We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.
    > 
    > 
    > 
    > From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
    > Sent: Tuesday, August 28, 2018 6:44 PM
    > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
    > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
    > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
    > 
    > 
    > 
    > In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.
    > 
    > 
    > 
    > 
    > 
    > ________________________________
    > 
    > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
    > Sent: Monday, August 27, 2018 9:10 PM
    > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
    > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
    > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
    > 
    > 
    > 
    > External Email
    > 
    > This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.
    > 
    > > -----Original Message-----
    > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
    > > Sent: Monday, August 27, 2018 10:08 PM
    > > To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
    > > jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
    > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
    > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
    > > synchronization
    > >
    > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
    > > > With existing code in kni_fifo_put, rx_q values are not being updated
    > > > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
    > > > This is causing the sync issue on other core. So adding a write
    > > > barrier to make sure the values being synced before updating fifo_write.
    > > >
    > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
    > > >
    > > > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
    > > > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
    > >
    > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
    > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
    

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Jerin Jacob]

-----Original Message-----
> Date: Wed, 29 Aug 2018 08:47:56 +0000
> From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
>  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
>  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>, nd
>  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>  synchronization
> user-agent: Microsoft-MacOutlook/10.10.0.180812
> 
> 
> There was a mention of rte_ring which is a different data structure. But perhaps I misunderstood why this was mentioned and the idea was only to use the C11 memory model as is also used in rte_ring nowadays.
> 
> But why would we have different code for x86 and for other architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC __atomic builtins), the code generated for x86 will be the same. __atomic_load(__ATOMIC_ACQUIRE) and __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and stores on x86?

# One reason was __atomic builtins  primitives were implemented in gcc 4.7 and x86 would
like to support < gcc 4.7 and ICC compiler.
# The theme was no change in the existing code for x86.I am not sure about the code generation for x86 with __atomic builtins,
I let x86 maintainers to comments on this.


> 
> -- Ola
> 
> On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com> wrote:
> 
>     -----Original Message-----
>     > Date: Wed, 29 Aug 2018 07:34:34 +0000
>     > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
>     > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
>     >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
>     >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>     >  <Jerin.JacobKollanukkaran@cavium.com>
>     > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve Capper
>     >  <Steve.Capper@arm.com>
>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>     >  synchronization
>     > user-agent: Microsoft-MacOutlook/10.10.0.180812
>     >
>     > Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?
> 
>     What would be the change in interface? Is it removing the volatile for
>     C11 case, Then you can use anonymous union OR #define to keep the size
>     and offset of the element intact.
> 
>     struct rte_kni_fifo {
>     #ifndef RTE_C11...
>             volatile unsigned write;     /**< Next position to be written*/
>             volatile unsigned read;      /**< Next position to be read */
>     #else
>             unsigned write;     /**< Next position to be written*/
>             unsigned read;      /**< Next position to be read */
>     #endif
>             unsigned len;                /**< Circular buffer length */
>             unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
>             void *volatile buffer[];     /**< The buffer contains mbuf
>     pointers */
>     };
> 
>     Anonymous union example:
>     https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
> 
>     You can check the ABI breakage by devtools/validate-abi.sh
> 
>     >
>     > -- Ola
>     >
>     > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
>     > Date: Wednesday, 29 August 2018 at 07:50
>     > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
>     > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     > Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.
>     >
>     > ________________________________
>     > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>     > Sent: Wednesday, August 29, 2018 10:29 AM
>     > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
>     > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     > External Email
>     >
>     > I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
>     >
>     > IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
>     >
>     >
>     >
>     > If you want us to put together a patch with this idea, please let us know.
>     >
>     >
>     >
>     > Thank you,
>     >
>     > Honnappa
>     >
>     >
>     >
>     > From: Gavin Hu
>     > Sent: Tuesday, August 28, 2018 2:31 PM
>     > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
>     > Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     >
>     > Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
>     >
>     > We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.
>     >
>     >
>     >
>     > From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
>     > Sent: Tuesday, August 28, 2018 6:44 PM
>     > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     >
>     > In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.
>     >
>     >
>     >
>     >
>     >
>     > ________________________________
>     >
>     > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
>     > Sent: Monday, August 27, 2018 9:10 PM
>     > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     >
>     > External Email
>     >
>     > This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.
>     >
>     > > -----Original Message-----
>     > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
>     > > Sent: Monday, August 27, 2018 10:08 PM
>     > > To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
>     > > jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
>     > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
>     > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>     > > synchronization
>     > >
>     > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
>     > > > With existing code in kni_fifo_put, rx_q values are not being updated
>     > > > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
>     > > > This is causing the sync issue on other core. So adding a write
>     > > > barrier to make sure the values being synced before updating fifo_write.
>     > > >
>     > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
>     > > >
>     > > > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
>     > > > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
>     > >
>     > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
>     > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> 
> 

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Honnappa Nagarahalli]

Hi Jerin,
	Is there any reason for having 'RTE_RING_USE_C11_MEM_MODEL', which is specific to rte_ring? I do not see a need for choosing only some algorithms to work with C11 model. I suggest that we change this to 'RTE_USE_C11_MEM_MODEL' so that it can apply to all libraries/algorithms.

Thank you,
Honnappa

-----Original Message-----
From: Jerin Jacob <jerin.jacob@caviumnetworks.com> 
Sent: Wednesday, August 29, 2018 3:58 AM
To: Ola Liljedahl <Ola.Liljedahl@arm.com>
Cc: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu <Gavin.Hu@arm.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>; dev@dpdk.org; nd <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>
Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

-----Original Message-----
> Date: Wed, 29 Aug 2018 08:47:56 +0000
> From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa  
> Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu 
> <Gavin.Hu@arm.com>,  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>, 
> nd  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer  
> synchronization
> user-agent: Microsoft-MacOutlook/10.10.0.180812
> 
> 
> There was a mention of rte_ring which is a different data structure. But perhaps I misunderstood why this was mentioned and the idea was only to use the C11 memory model as is also used in rte_ring nowadays.
> 
> But why would we have different code for x86 and for other architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC __atomic builtins), the code generated for x86 will be the same. __atomic_load(__ATOMIC_ACQUIRE) and __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and stores on x86?

# One reason was __atomic builtins  primitives were implemented in gcc 4.7 and x86 would like to support < gcc 4.7 and ICC compiler.
# The theme was no change in the existing code for x86.I am not sure about the code generation for x86 with __atomic builtins, I let x86 maintainers to comments on this.


> 
> -- Ola
> 
> On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com> wrote:
> 
>     -----Original Message-----
>     > Date: Wed, 29 Aug 2018 07:34:34 +0000
>     > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
>     > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
>     >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
>     >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>     >  <Jerin.JacobKollanukkaran@cavium.com>
>     > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve Capper
>     >  <Steve.Capper@arm.com>
>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>     >  synchronization
>     > user-agent: Microsoft-MacOutlook/10.10.0.180812
>     >
>     > Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?
> 
>     What would be the change in interface? Is it removing the volatile for
>     C11 case, Then you can use anonymous union OR #define to keep the size
>     and offset of the element intact.
> 
>     struct rte_kni_fifo {
>     #ifndef RTE_C11...
>             volatile unsigned write;     /**< Next position to be written*/
>             volatile unsigned read;      /**< Next position to be read */
>     #else
>             unsigned write;     /**< Next position to be written*/
>             unsigned read;      /**< Next position to be read */
>     #endif
>             unsigned len;                /**< Circular buffer length */
>             unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
>             void *volatile buffer[];     /**< The buffer contains mbuf
>     pointers */
>     };
> 
>     Anonymous union example:
>     https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
> 
>     You can check the ABI breakage by devtools/validate-abi.sh
> 
>     >
>     > -- Ola
>     >
>     > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
>     > Date: Wednesday, 29 August 2018 at 07:50
>     > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
>     > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     > Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.
>     >
>     > ________________________________
>     > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>     > Sent: Wednesday, August 29, 2018 10:29 AM
>     > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
>     > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     > External Email
>     >
>     > I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
>     >
>     > IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
>     >
>     >
>     >
>     > If you want us to put together a patch with this idea, please let us know.
>     >
>     >
>     >
>     > Thank you,
>     >
>     > Honnappa
>     >
>     >
>     >
>     > From: Gavin Hu
>     > Sent: Tuesday, August 28, 2018 2:31 PM
>     > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
>     > Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     >
>     > Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
>     >
>     > We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.
>     >
>     >
>     >
>     > From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
>     > Sent: Tuesday, August 28, 2018 6:44 PM
>     > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     >
>     > In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.
>     >
>     >
>     >
>     >
>     >
>     > ________________________________
>     >
>     > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
>     > Sent: Monday, August 27, 2018 9:10 PM
>     > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>     >
>     >
>     >
>     > External Email
>     >
>     > This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.
>     >
>     > > -----Original Message-----
>     > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
>     > > Sent: Monday, August 27, 2018 10:08 PM
>     > > To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
>     > > jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
>     > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
>     > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>     > > synchronization
>     > >
>     > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
>     > > > With existing code in kni_fifo_put, rx_q values are not being updated
>     > > > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
>     > > > This is causing the sync issue on other core. So adding a write
>     > > > barrier to make sure the values being synced before updating fifo_write.
>     > > >
>     > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
>     > > >
>     > > > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
>     > > > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
>     > >
>     > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
>     > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> 
> 

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Jerin Jacob]

-----Original Message-----
> Date: Thu, 13 Sep 2018 17:40:53 +0000
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, Ola Liljedahl
>  <Ola.Liljedahl@arm.com>
> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu (Arm
>  Technology China)" <Gavin.Hu@arm.com>, Ferruh Yigit
>  <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>, nd
>  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>, "Phil Yang (Arm
>  Technology China)" <Phil.Yang@arm.com>
> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>  synchronization
> 
> 
> Hi Jerin,
>         Is there any reason for having 'RTE_RING_USE_C11_MEM_MODEL', which is specific to rte_ring? I do not see a need for choosing only some algorithms to work with C11 model. I suggest that we change this to 'RTE_USE_C11_MEM_MODEL' so that it can apply to all libraries/algorithms.


Yes. Makes sense to me to keep only single config option.

> 
> Thank you,
> Honnappa
> 
> -----Original Message-----
> From: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Sent: Wednesday, August 29, 2018 3:58 AM
> To: Ola Liljedahl <Ola.Liljedahl@arm.com>
> Cc: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu <Gavin.Hu@arm.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>; dev@dpdk.org; nd <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> 
> -----Original Message-----
> > Date: Wed, 29 Aug 2018 08:47:56 +0000
> > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
> > Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu
> > <Gavin.Hu@arm.com>,  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>,
> > nd  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>
> > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> > synchronization
> > user-agent: Microsoft-MacOutlook/10.10.0.180812
> >
> >
> > There was a mention of rte_ring which is a different data structure. But perhaps I misunderstood why this was mentioned and the idea was only to use the C11 memory model as is also used in rte_ring nowadays.
> >
> > But why would we have different code for x86 and for other architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC __atomic builtins), the code generated for x86 will be the same. __atomic_load(__ATOMIC_ACQUIRE) and __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and stores on x86?
> 
> # One reason was __atomic builtins  primitives were implemented in gcc 4.7 and x86 would like to support < gcc 4.7 and ICC compiler.
> # The theme was no change in the existing code for x86.I am not sure about the code generation for x86 with __atomic builtins, I let x86 maintainers to comments on this.
> 
> 
> >
> > -- Ola
> >
> > On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com> wrote:
> >
> >     -----Original Message-----
> >     > Date: Wed, 29 Aug 2018 07:34:34 +0000
> >     > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> >     > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
> >     >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
> >     >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >     >  <Jerin.JacobKollanukkaran@cavium.com>
> >     > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve Capper
> >     >  <Steve.Capper@arm.com>
> >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >     >  synchronization
> >     > user-agent: Microsoft-MacOutlook/10.10.0.180812
> >     >
> >     > Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?
> >
> >     What would be the change in interface? Is it removing the volatile for
> >     C11 case, Then you can use anonymous union OR #define to keep the size
> >     and offset of the element intact.
> >
> >     struct rte_kni_fifo {
> >     #ifndef RTE_C11...
> >             volatile unsigned write;     /**< Next position to be written*/
> >             volatile unsigned read;      /**< Next position to be read */
> >     #else
> >             unsigned write;     /**< Next position to be written*/
> >             unsigned read;      /**< Next position to be read */
> >     #endif
> >             unsigned len;                /**< Circular buffer length */
> >             unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
> >             void *volatile buffer[];     /**< The buffer contains mbuf
> >     pointers */
> >     };
> >
> >     Anonymous union example:
> >     https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
> >
> >     You can check the ABI breakage by devtools/validate-abi.sh
> >
> >     >
> >     > -- Ola
> >     >
> >     > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
> >     > Date: Wednesday, 29 August 2018 at 07:50
> >     > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
> >     > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
> >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     > Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.
> >     >
> >     > ________________________________
> >     > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> >     > Sent: Wednesday, August 29, 2018 10:29 AM
> >     > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
> >     > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
> >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     > External Email
> >     >
> >     > I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
> >     >
> >     > IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
> >     >
> >     >
> >     >
> >     > If you want us to put together a patch with this idea, please let us know.
> >     >
> >     >
> >     >
> >     > Thank you,
> >     >
> >     > Honnappa
> >     >
> >     >
> >     >
> >     > From: Gavin Hu
> >     > Sent: Tuesday, August 28, 2018 2:31 PM
> >     > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
> >     > Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
> >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     >
> >     > Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
> >     >
> >     > We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.
> >     >
> >     >
> >     >
> >     > From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
> >     > Sent: Tuesday, August 28, 2018 6:44 PM
> >     > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
> >     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
> >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     >
> >     > In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > ________________________________
> >     >
> >     > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
> >     > Sent: Monday, August 27, 2018 9:10 PM
> >     > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
> >     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
> >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     >
> >     > External Email
> >     >
> >     > This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.
> >     >
> >     > > -----Original Message-----
> >     > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> >     > > Sent: Monday, August 27, 2018 10:08 PM
> >     > > To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
> >     > > jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> >     > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> >     > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >     > > synchronization
> >     > >
> >     > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> >     > > > With existing code in kni_fifo_put, rx_q values are not being updated
> >     > > > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> >     > > > This is causing the sync issue on other core. So adding a write
> >     > > > barrier to make sure the values being synced before updating fifo_write.
> >     > > >
> >     > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >     > > >
> >     > > > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
> >     > > > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
> >     > >
> >     > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> >     > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> >
> >

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Honnappa Nagarahalli]

-----Original Message-----
> Date: Thu, 13 Sep 2018 17:40:53 +0000
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, Ola Liljedahl  
> <Ola.Liljedahl@arm.com>
> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu 
> (Arm  Technology China)" <Gavin.Hu@arm.com>, Ferruh Yigit  
> <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>, 
> nd  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>, "Phil Yang (Arm  
> Technology China)" <Phil.Yang@arm.com>
> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer  
> synchronization
> 
> 
> Hi Jerin,
>         Is there any reason for having 'RTE_RING_USE_C11_MEM_MODEL', which is specific to rte_ring? I do not see a need for choosing only some algorithms to work with C11 model. I suggest that we change this to 'RTE_USE_C11_MEM_MODEL' so that it can apply to all libraries/algorithms.


Yes. Makes sense to me to keep only single config option.

rte_ring has 2 sets of algorithms for Arm architecture, one with C11 memory model and the other with barriers. Going forward (for ex: for KNI), I think we should support C11 memory model only and skip the barriers.

Also, do you see any issues in making C11 memory model default for Arm architecture?

> 
> Thank you,
> Honnappa
> 
> -----Original Message-----
> From: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> Sent: Wednesday, August 29, 2018 3:58 AM
> To: Ola Liljedahl <Ola.Liljedahl@arm.com>
> Cc: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Honnappa 
> Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu 
> <Gavin.Hu@arm.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, 
> Jerin <Jerin.JacobKollanukkaran@cavium.com>; dev@dpdk.org; nd 
> <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer 
> synchronization
> 
> -----Original Message-----
> > Date: Wed, 29 Aug 2018 08:47:56 +0000
> > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa 
> > Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu 
> > <Gavin.Hu@arm.com>,  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" 
> > <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper 
> > <Steve.Capper@arm.com>
> > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer 
> > synchronization
> > user-agent: Microsoft-MacOutlook/10.10.0.180812
> >
> >
> > There was a mention of rte_ring which is a different data structure. But perhaps I misunderstood why this was mentioned and the idea was only to use the C11 memory model as is also used in rte_ring nowadays.
> >
> > But why would we have different code for x86 and for other architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC __atomic builtins), the code generated for x86 will be the same. __atomic_load(__ATOMIC_ACQUIRE) and __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and stores on x86?
> 
> # One reason was __atomic builtins  primitives were implemented in gcc 4.7 and x86 would like to support < gcc 4.7 and ICC compiler.
> # The theme was no change in the existing code for x86.I am not sure about the code generation for x86 with __atomic builtins, I let x86 maintainers to comments on this.
> 
> 
> >
> > -- Ola
> >
> > On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com> wrote:
> >
> >     -----Original Message-----
> >     > Date: Wed, 29 Aug 2018 07:34:34 +0000
> >     > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> >     > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
> >     >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
> >     >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >     >  <Jerin.JacobKollanukkaran@cavium.com>
> >     > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve Capper
> >     >  <Steve.Capper@arm.com>
> >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >     >  synchronization
> >     > user-agent: Microsoft-MacOutlook/10.10.0.180812
> >     >
> >     > Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?
> >
> >     What would be the change in interface? Is it removing the volatile for
> >     C11 case, Then you can use anonymous union OR #define to keep the size
> >     and offset of the element intact.
> >
> >     struct rte_kni_fifo {
> >     #ifndef RTE_C11...
> >             volatile unsigned write;     /**< Next position to be written*/
> >             volatile unsigned read;      /**< Next position to be read */
> >     #else
> >             unsigned write;     /**< Next position to be written*/
> >             unsigned read;      /**< Next position to be read */
> >     #endif
> >             unsigned len;                /**< Circular buffer length */
> >             unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
> >             void *volatile buffer[];     /**< The buffer contains mbuf
> >     pointers */
> >     };
> >
> >     Anonymous union example:
> >     https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
> >
> >     You can check the ABI breakage by devtools/validate-abi.sh
> >
> >     >
> >     > -- Ola
> >     >
> >     > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
> >     > Date: Wednesday, 29 August 2018 at 07:50
> >     > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
> >     > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
> >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     > Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.
> >     >
> >     > ________________________________
> >     > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> >     > Sent: Wednesday, August 29, 2018 10:29 AM
> >     > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
> >     > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
> >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     > External Email
> >     >
> >     > I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
> >     >
> >     > IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
> >     >
> >     >
> >     >
> >     > If you want us to put together a patch with this idea, please let us know.
> >     >
> >     >
> >     >
> >     > Thank you,
> >     >
> >     > Honnappa
> >     >
> >     >
> >     >
> >     > From: Gavin Hu
> >     > Sent: Tuesday, August 28, 2018 2:31 PM
> >     > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
> >     > Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
> >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     >
> >     > Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
> >     >
> >     > We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.
> >     >
> >     >
> >     >
> >     > From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
> >     > Sent: Tuesday, August 28, 2018 6:44 PM
> >     > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
> >     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
> >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     >
> >     > In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.
> >     >
> >     >
> >     >
> >     >
> >     >
> >     > ________________________________
> >     >
> >     > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
> >     > Sent: Monday, August 27, 2018 9:10 PM
> >     > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
> >     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
> >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> >     >
> >     >
> >     >
> >     > External Email
> >     >
> >     > This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.
> >     >
> >     > > -----Original Message-----
> >     > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> >     > > Sent: Monday, August 27, 2018 10:08 PM
> >     > > To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
> >     > > jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> >     > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> >     > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >     > > synchronization
> >     > >
> >     > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> >     > > > With existing code in kni_fifo_put, rx_q values are not being updated
> >     > > > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> >     > > > This is causing the sync issue on other core. So adding a write
> >     > > > barrier to make sure the values being synced before updating fifo_write.
> >     > > >
> >     > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >     > > >
> >     > > > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
> >     > > > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
> >     > >
> >     > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> >     > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> >
> >

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Jerin Jacob]

-----Original Message-----
> Date: Thu, 13 Sep 2018 23:45:31 +0000
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: Ola Liljedahl <Ola.Liljedahl@arm.com>, "Kokkilagadda, Kiran"
>  <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu (Arm Technology China)"
>  <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>, nd
>  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>, "Phil Yang (Arm
>  Technology China)" <Phil.Yang@arm.com>
> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>  synchronization
> 
> External Email
> 
> -----Original Message-----
> > Date: Thu, 13 Sep 2018 17:40:53 +0000
> > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, Ola Liljedahl
> > <Ola.Liljedahl@arm.com>
> > CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu
> > (Arm  Technology China)" <Gavin.Hu@arm.com>, Ferruh Yigit
> > <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>,
> > nd  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>, "Phil Yang (Arm
> > Technology China)" <Phil.Yang@arm.com>
> > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> > synchronization
> >
> >
> > Hi Jerin,
> >         Is there any reason for having 'RTE_RING_USE_C11_MEM_MODEL', which is specific to rte_ring? I do not see a need for choosing only some algorithms to work with C11 model. I suggest that we change this to 'RTE_USE_C11_MEM_MODEL' so that it can apply to all libraries/algorithms.
> 
> 
> Yes. Makes sense to me to keep only single config option.
> 
> rte_ring has 2 sets of algorithms for Arm architecture, one with C11 memory model and the other with barriers. Going forward (for ex: for KNI), I think we should support C11 memory model only and skip the barriers.

IMO, Both should be supported and set N as in the config/common_base.
Based on architecture or micro architecture the performance can vary.
So keeping both options and allowing to override to arch/micro arch
specific config file makes sense to me.(like existing model, as smp_*
ops are compiler NOP for x86)
 
> Also, do you see any issues in making C11 memory model default for Arm architecture?

It is already set default Y to arm64. see config/common_armv8a_linuxapp.

And it is possible for micro architecture to override, see
config/defconfig_arm64-thunderx-linuxapp-gcc


> 
> >
> > Thank you,
> > Honnappa
> >
> > -----Original Message-----
> > From: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > Sent: Wednesday, August 29, 2018 3:58 AM
> > To: Ola Liljedahl <Ola.Liljedahl@arm.com>
> > Cc: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Honnappa
> > Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu
> > <Gavin.Hu@arm.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob,
> > Jerin <Jerin.JacobKollanukkaran@cavium.com>; dev@dpdk.org; nd
> > <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>
> > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> > synchronization
> >
> > -----Original Message-----
> > > Date: Wed, 29 Aug 2018 08:47:56 +0000
> > > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> > > To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> > > CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
> > > Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu
> > > <Gavin.Hu@arm.com>,  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> > >  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org"
> > > <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper
> > > <Steve.Capper@arm.com>
> > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> > > synchronization
> > > user-agent: Microsoft-MacOutlook/10.10.0.180812
> > >
> > >
> > > There was a mention of rte_ring which is a different data structure. But perhaps I misunderstood why this was mentioned and the idea was only to use the C11 memory model as is also used in rte_ring nowadays.
> > >
> > > But why would we have different code for x86 and for other architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC __atomic builtins), the code generated for x86 will be the same. __atomic_load(__ATOMIC_ACQUIRE) and __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and stores on x86?
> >
> > # One reason was __atomic builtins  primitives were implemented in gcc 4.7 and x86 would like to support < gcc 4.7 and ICC compiler.
> > # The theme was no change in the existing code for x86.I am not sure about the code generation for x86 with __atomic builtins, I let x86 maintainers to comments on this.
> >
> >
> > >
> > > -- Ola
> > >
> > > On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com> wrote:
> > >
> > >     -----Original Message-----
> > >     > Date: Wed, 29 Aug 2018 07:34:34 +0000
> > >     > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> > >     > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
> > >     >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
> > >     >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> > >     >  <Jerin.JacobKollanukkaran@cavium.com>
> > >     > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve Capper
> > >     >  <Steve.Capper@arm.com>
> > >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> > >     >  synchronization
> > >     > user-agent: Microsoft-MacOutlook/10.10.0.180812
> > >     >
> > >     > Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?
> > >
> > >     What would be the change in interface? Is it removing the volatile for
> > >     C11 case, Then you can use anonymous union OR #define to keep the size
> > >     and offset of the element intact.
> > >
> > >     struct rte_kni_fifo {
> > >     #ifndef RTE_C11...
> > >             volatile unsigned write;     /**< Next position to be written*/
> > >             volatile unsigned read;      /**< Next position to be read */
> > >     #else
> > >             unsigned write;     /**< Next position to be written*/
> > >             unsigned read;      /**< Next position to be read */
> > >     #endif
> > >             unsigned len;                /**< Circular buffer length */
> > >             unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
> > >             void *volatile buffer[];     /**< The buffer contains mbuf
> > >     pointers */
> > >     };
> > >
> > >     Anonymous union example:
> > >     https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
> > >
> > >     You can check the ABI breakage by devtools/validate-abi.sh
> > >
> > >     >
> > >     > -- Ola
> > >     >
> > >     > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
> > >     > Date: Wednesday, 29 August 2018 at 07:50
> > >     > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
> > >     > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
> > >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> > >     >
> > >     >
> > >     > Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.
> > >     >
> > >     > ________________________________
> > >     > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> > >     > Sent: Wednesday, August 29, 2018 10:29 AM
> > >     > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
> > >     > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
> > >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> > >     >
> > >     >
> > >     > External Email
> > >     >
> > >     > I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
> > >     >
> > >     > IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
> > >     >
> > >     >
> > >     >
> > >     > If you want us to put together a patch with this idea, please let us know.
> > >     >
> > >     >
> > >     >
> > >     > Thank you,
> > >     >
> > >     > Honnappa
> > >     >
> > >     >
> > >     >
> > >     > From: Gavin Hu
> > >     > Sent: Tuesday, August 28, 2018 2:31 PM
> > >     > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
> > >     > Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
> > >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> > >     >
> > >     >
> > >     >
> > >     > Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
> > >     >
> > >     > We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.
> > >     >
> > >     >
> > >     >
> > >     > From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
> > >     > Sent: Tuesday, August 28, 2018 6:44 PM
> > >     > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
> > >     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
> > >     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> > >     >
> > >     >
> > >     >
> > >     > In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.
> > >     >
> > >     >
> > >     >
> > >     >
> > >     >
> > >     > ________________________________
> > >     >
> > >     > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
> > >     > Sent: Monday, August 27, 2018 9:10 PM
> > >     > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
> > >     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
> > >     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
> > >     >
> > >     >
> > >     >
> > >     > External Email
> > >     >
> > >     > This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.
> > >     >
> > >     > > -----Original Message-----
> > >     > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> > >     > > Sent: Monday, August 27, 2018 10:08 PM
> > >     > > To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
> > >     > > jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> > >     > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> > >     > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> > >     > > synchronization
> > >     > >
> > >     > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> > >     > > > With existing code in kni_fifo_put, rx_q values are not being updated
> > >     > > > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
> > >     > > > This is causing the sync issue on other core. So adding a write
> > >     > > > barrier to make sure the values being synced before updating fifo_write.
> > >     > > >
> > >     > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
> > >     > > >
> > >     > > > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
> > >     > > > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
> > >     > >
> > >     > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> > >     > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
> > >
> > >

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Ferruh Yigit]

On 9/14/2018 3:45 AM, Jerin Jacob wrote:
> -----Original Message-----
>> Date: Thu, 13 Sep 2018 23:45:31 +0000
>> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>> CC: Ola Liljedahl <Ola.Liljedahl@arm.com>, "Kokkilagadda, Kiran"
>>  <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu (Arm Technology China)"
>>  <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>, nd
>>  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>, "Phil Yang (Arm
>>  Technology China)" <Phil.Yang@arm.com>
>> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>>  synchronization
>>
>> External Email
>>
>> -----Original Message-----
>>> Date: Thu, 13 Sep 2018 17:40:53 +0000
>>> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>>> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, Ola Liljedahl
>>> <Ola.Liljedahl@arm.com>
>>> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu
>>> (Arm  Technology China)" <Gavin.Hu@arm.com>, Ferruh Yigit
>>> <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>>>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org" <dev@dpdk.org>,
>>> nd  <nd@arm.com>, Steve Capper <Steve.Capper@arm.com>, "Phil Yang (Arm
>>> Technology China)" <Phil.Yang@arm.com>
>>> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>>> synchronization
>>>
>>>
>>> Hi Jerin,
>>>         Is there any reason for having 'RTE_RING_USE_C11_MEM_MODEL', which is specific to rte_ring? I do not see a need for choosing only some algorithms to work with C11 model. I suggest that we change this to 'RTE_USE_C11_MEM_MODEL' so that it can apply to all libraries/algorithms.
>>
>>
>> Yes. Makes sense to me to keep only single config option.
>>
>> rte_ring has 2 sets of algorithms for Arm architecture, one with C11 memory model and the other with barriers. Going forward (for ex: for KNI), I think we should support C11 memory model only and skip the barriers.
> 
> IMO, Both should be supported and set N as in the config/common_base.
> Based on architecture or micro architecture the performance can vary.
> So keeping both options and allowing to override to arch/micro arch
> specific config file makes sense to me.(like existing model, as smp_*
> ops are compiler NOP for x86)

Hi Jerin, Honnappa,  Kiran,

Will there be a new version for this release?

I can see two options:
1- Add read/write barriers for both library and kernel parts.
2- Use c11 atomics
  2a- change existing RTE_RING_USE_C11_MEM_MODEL to RTE_USE_C11_MEM_MODEL
  2b- Use RTE_USE_C11_MEM_MODEL to implement c11 atomic for arm and ppc

2) seems agreed on, but is it clear who will work on it?

And 1) looks easier to implement, if 2) won't make time for release can we
fallback to this one?

Thanks,
ferruh

>  
>> Also, do you see any issues in making C11 memory model default for Arm architecture?
> 
> It is already set default Y to arm64. see config/common_armv8a_linuxapp.
> 
> And it is possible for micro architecture to override, see
> config/defconfig_arm64-thunderx-linuxapp-gcc
> 
> 
>>
>>>
>>> Thank you,
>>> Honnappa
>>>
>>> -----Original Message-----
>>> From: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>>> Sent: Wednesday, August 29, 2018 3:58 AM
>>> To: Ola Liljedahl <Ola.Liljedahl@arm.com>
>>> Cc: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Honnappa
>>> Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu
>>> <Gavin.Hu@arm.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob,
>>> Jerin <Jerin.JacobKollanukkaran@cavium.com>; dev@dpdk.org; nd
>>> <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>
>>> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>>> synchronization
>>>
>>> -----Original Message-----
>>>> Date: Wed, 29 Aug 2018 08:47:56 +0000
>>>> From: Ola Liljedahl <Ola.Liljedahl@arm.com>
>>>> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
>>>> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
>>>> Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu
>>>> <Gavin.Hu@arm.com>,  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>>>>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org"
>>>> <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper
>>>> <Steve.Capper@arm.com>
>>>> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>>>> synchronization
>>>> user-agent: Microsoft-MacOutlook/10.10.0.180812
>>>>
>>>>
>>>> There was a mention of rte_ring which is a different data structure. But perhaps I misunderstood why this was mentioned and the idea was only to use the C11 memory model as is also used in rte_ring nowadays.
>>>>
>>>> But why would we have different code for x86 and for other architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC __atomic builtins), the code generated for x86 will be the same. __atomic_load(__ATOMIC_ACQUIRE) and __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and stores on x86?
>>>
>>> # One reason was __atomic builtins  primitives were implemented in gcc 4.7 and x86 would like to support < gcc 4.7 and ICC compiler.
>>> # The theme was no change in the existing code for x86.I am not sure about the code generation for x86 with __atomic builtins, I let x86 maintainers to comments on this.
>>>
>>>
>>>>
>>>> -- Ola
>>>>
>>>> On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com> wrote:
>>>>
>>>>     -----Original Message-----
>>>>     > Date: Wed, 29 Aug 2018 07:34:34 +0000
>>>>     > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
>>>>     > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
>>>>     >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>,
>>>>     >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
>>>>     >  <Jerin.JacobKollanukkaran@cavium.com>
>>>>     > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve Capper
>>>>     >  <Steve.Capper@arm.com>
>>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>>>>     >  synchronization
>>>>     > user-agent: Microsoft-MacOutlook/10.10.0.180812
>>>>     >
>>>>     > Is the rte_kni kernel/user binary interface subject to backwards compatibility requirements? Or can we change it for a new DPDK release?
>>>>
>>>>     What would be the change in interface? Is it removing the volatile for
>>>>     C11 case, Then you can use anonymous union OR #define to keep the size
>>>>     and offset of the element intact.
>>>>
>>>>     struct rte_kni_fifo {
>>>>     #ifndef RTE_C11...
>>>>             volatile unsigned write;     /**< Next position to be written*/
>>>>             volatile unsigned read;      /**< Next position to be read */
>>>>     #else
>>>>             unsigned write;     /**< Next position to be written*/
>>>>             unsigned read;      /**< Next position to be read */
>>>>     #endif
>>>>             unsigned len;                /**< Circular buffer length */
>>>>             unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
>>>>             void *volatile buffer[];     /**< The buffer contains mbuf
>>>>     pointers */
>>>>     };
>>>>
>>>>     Anonymous union example:
>>>>     https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
>>>>
>>>>     You can check the ABI breakage by devtools/validate-abi.sh
>>>>
>>>>     >
>>>>     > -- Ola
>>>>     >
>>>>     > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
>>>>     > Date: Wednesday, 29 August 2018 at 07:50
>>>>     > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob, Jerin" <Jerin.JacobKollanukkaran@cavium.com>
>>>>     > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
>>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>>>>     >
>>>>     >
>>>>     > Agreed. Please go a head and make the changes. You need to make same change in kernel side also. And please use c11 ring (see rte_ring) mechanism so that it won't impact other platforms like intel. We need this change just for arm and ppc.
>>>>     >
>>>>     > ________________________________
>>>>     > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
>>>>     > Sent: Wednesday, August 29, 2018 10:29 AM
>>>>     > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
>>>>     > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
>>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>>>>     >
>>>>     >
>>>>     > External Email
>>>>     >
>>>>     > I agree with Gavin here. Store to fifo->write and fifo->read can get hoisted resulting in accessing invalid buffer array entries or over writing of the buffer array entries.
>>>>     >
>>>>     > IMO, we should solve this using c11 atomics. This will also help remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
>>>>     >
>>>>     >
>>>>     >
>>>>     > If you want us to put together a patch with this idea, please let us know.
>>>>     >
>>>>     >
>>>>     >
>>>>     > Thank you,
>>>>     >
>>>>     > Honnappa
>>>>     >
>>>>     >
>>>>     >
>>>>     > From: Gavin Hu
>>>>     > Sent: Tuesday, August 28, 2018 2:31 PM
>>>>     > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>
>>>>     > Cc: dev@dpdk.org; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
>>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>>>>     >
>>>>     >
>>>>     >
>>>>     > Assuming reader and writer may execute on different CPU's, this become standard multithreaded programming.
>>>>     >
>>>>     > We are concerned about that update the reader pointer too early(weak ordering may reorder it before reading from the slots), that means the slots are released and may immediately overwritten by the writer then you get “too new” data and get lost of the old data.
>>>>     >
>>>>     >
>>>>     >
>>>>     > From: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
>>>>     > Sent: Tuesday, August 28, 2018 6:44 PM
>>>>     > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>; Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@cavium.com>>
>>>>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
>>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>>>>     >
>>>>     >
>>>>     >
>>>>     > In this instance there won't be any problem, as until the value of fifo->write changes, this loop won't get executed. As of now we didn't see any issue with it and for performance reasons, we don't want to keep read barrier.
>>>>     >
>>>>     >
>>>>     >
>>>>     >
>>>>     >
>>>>     > ________________________________
>>>>     >
>>>>     > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
>>>>     > Sent: Monday, August 27, 2018 9:10 PM
>>>>     > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
>>>>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
>>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization
>>>>     >
>>>>     >
>>>>     >
>>>>     > External Email
>>>>     >
>>>>     > This fix is not complete, kni_fifo_get requires a read fence also, otherwise it probably gets stale data on a weak ordering platform.
>>>>     >
>>>>     > > -----Original Message-----
>>>>     > > From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> On Behalf Of Ferruh Yigit
>>>>     > > Sent: Monday, August 27, 2018 10:08 PM
>>>>     > > To: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>;
>>>>     > > jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
>>>>     > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
>>>>     > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
>>>>     > > synchronization
>>>>     > >
>>>>     > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
>>>>     > > > With existing code in kni_fifo_put, rx_q values are not being updated
>>>>     > > > before updating fifo_write. While reading rx_q in kni_net_rx_normal,
>>>>     > > > This is causing the sync issue on other core. So adding a write
>>>>     > > > barrier to make sure the values being synced before updating fifo_write.
>>>>     > > >
>>>>     > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
>>>>     > > >
>>>>     > > > Signed-off-by: Kiran Kumar <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetworks.com>>
>>>>     > > > Acked-by: Jerin Jacob <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>>
>>>>     > >
>>>>     > > Acked-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
>>>>     > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
>>>>
>>>>

Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Honnappa Nagarahalli]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> Sent: Tuesday, September 18, 2018 10:54 AM
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; Kokkilagadda, Kiran
> <Kiran.Kokkilagadda@cavium.com>
> Cc: Ola Liljedahl <Ola.Liljedahl@arm.com>; Gavin Hu (Arm Technology China)
> <Gavin.Hu@arm.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>;
> dev@dpdk.org; nd <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>;
> Phil Yang (Arm Technology China) <Phil.Yang@arm.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Konstantin Ananyev
> <konstantin.ananyev@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> 
> On 9/14/2018 3:45 AM, Jerin Jacob wrote:
> > -----Original Message-----
> >> Date: Thu, 13 Sep 2018 23:45:31 +0000
> >> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> >> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> >> CC: Ola Liljedahl <Ola.Liljedahl@arm.com>, "Kokkilagadda, Kiran"
> >>  <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu (Arm Technology China)"
> >>  <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,
> Jerin"
> >>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org"
> >> <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper
> >> <Steve.Capper@arm.com>, "Phil Yang (Arm  Technology China)"
> >> <Phil.Yang@arm.com>
> >> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >> synchronization
> >>
> >> External Email
> >>
> >> -----Original Message-----
> >>> Date: Thu, 13 Sep 2018 17:40:53 +0000
> >>> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> >>> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, Ola Liljedahl
> >>> <Ola.Liljedahl@arm.com>
> >>> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu
> >>> (Arm  Technology China)" <Gavin.Hu@arm.com>, Ferruh Yigit
> >>> <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >>>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org"
> >>> <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper
> >>> <Steve.Capper@arm.com>, "Phil Yang (Arm Technology China)"
> >>> <Phil.Yang@arm.com>
> >>> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>> synchronization
> >>>
> >>>
> >>> Hi Jerin,
> >>>         Is there any reason for having 'RTE_RING_USE_C11_MEM_MODEL',
> which is specific to rte_ring? I do not see a need for choosing only some
> algorithms to work with C11 model. I suggest that we change this to
> 'RTE_USE_C11_MEM_MODEL' so that it can apply to all libraries/algorithms.
> >>
> >>
> >> Yes. Makes sense to me to keep only single config option.
> >>
> >> rte_ring has 2 sets of algorithms for Arm architecture, one with C11
> memory model and the other with barriers. Going forward (for ex: for KNI), I
> think we should support C11 memory model only and skip the barriers.
> >
> > IMO, Both should be supported and set N as in the config/common_base.
> > Based on architecture or micro architecture the performance can vary.
> > So keeping both options and allowing to override to arch/micro arch
> > specific config file makes sense to me.(like existing model, as smp_*
> > ops are compiler NOP for x86)
> 
> Hi Jerin, Honnappa,  Kiran,
> 
> Will there be a new version for this release?
> 
> I can see two options:
> 1- Add read/write barriers for both library and kernel parts.
> 2- Use c11 atomics
>   2a- change existing RTE_RING_USE_C11_MEM_MODEL to
> RTE_USE_C11_MEM_MODEL
>   2b- Use RTE_USE_C11_MEM_MODEL to implement c11 atomic for arm and
> ppc
> 
> 2) seems agreed on, but is it clear who will work on it?

Sorry for the late reply. We have implemented 2), currently undergoing internal review. We will get this out today. We will work through the community reviews quickly after that.

> 
> And 1) looks easier to implement, if 2) won't make time for release can we
> fallback to this one?
> 
> Thanks,
> ferruh
> 
> >
> >> Also, do you see any issues in making C11 memory model default for Arm
> architecture?
> >
> > It is already set default Y to arm64. see config/common_armv8a_linuxapp.
> >
> > And it is possible for micro architecture to override, see
> > config/defconfig_arm64-thunderx-linuxapp-gcc
> >
> >
> >>
> >>>
> >>> Thank you,
> >>> Honnappa
> >>>
> >>> -----Original Message-----
> >>> From: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> >>> Sent: Wednesday, August 29, 2018 3:58 AM
> >>> To: Ola Liljedahl <Ola.Liljedahl@arm.com>
> >>> Cc: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Honnappa
> >>> Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu
> >>> <Gavin.Hu@arm.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob,
> >>> Jerin <Jerin.JacobKollanukkaran@cavium.com>; dev@dpdk.org; nd
> >>> <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>
> >>> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>> synchronization
> >>>
> >>> -----Original Message-----
> >>>> Date: Wed, 29 Aug 2018 08:47:56 +0000
> >>>> From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> >>>> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> >>>> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
> >>>> Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu
> >>>> <Gavin.Hu@arm.com>,  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,
> Jerin"
> >>>>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org"
> >>>> <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper
> >>>> <Steve.Capper@arm.com>
> >>>> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>>> synchronization
> >>>> user-agent: Microsoft-MacOutlook/10.10.0.180812
> >>>>
> >>>>
> >>>> There was a mention of rte_ring which is a different data structure. But
> perhaps I misunderstood why this was mentioned and the idea was only to
> use the C11 memory model as is also used in rte_ring nowadays.
> >>>>
> >>>> But why would we have different code for x86 and for other
> architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC
> __atomic builtins), the code generated for x86 will be the same.
> __atomic_load(__ATOMIC_ACQUIRE) and
> __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and
> stores on x86?
> >>>
> >>> # One reason was __atomic builtins  primitives were implemented in gcc
> 4.7 and x86 would like to support < gcc 4.7 and ICC compiler.
> >>> # The theme was no change in the existing code for x86.I am not sure
> about the code generation for x86 with __atomic builtins, I let x86
> maintainers to comments on this.
> >>>
> >>>
> >>>>
> >>>> -- Ola
> >>>>
> >>>> On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com>
> wrote:
> >>>>
> >>>>     -----Original Message-----
> >>>>     > Date: Wed, 29 Aug 2018 07:34:34 +0000
> >>>>     > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> >>>>     > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>,
> Honnappa
> >>>>     >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu
> <Gavin.Hu@arm.com>,
> >>>>     >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >>>>     >  <Jerin.JacobKollanukkaran@cavium.com>
> >>>>     > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve
> Capper
> >>>>     >  <Steve.Capper@arm.com>
> >>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>>>     >  synchronization
> >>>>     > user-agent: Microsoft-MacOutlook/10.10.0.180812
> >>>>     >
> >>>>     > Is the rte_kni kernel/user binary interface subject to backwards
> compatibility requirements? Or can we change it for a new DPDK release?
> >>>>
> >>>>     What would be the change in interface? Is it removing the volatile for
> >>>>     C11 case, Then you can use anonymous union OR #define to keep the
> size
> >>>>     and offset of the element intact.
> >>>>
> >>>>     struct rte_kni_fifo {
> >>>>     #ifndef RTE_C11...
> >>>>             volatile unsigned write;     /**< Next position to be written*/
> >>>>             volatile unsigned read;      /**< Next position to be read */
> >>>>     #else
> >>>>             unsigned write;     /**< Next position to be written*/
> >>>>             unsigned read;      /**< Next position to be read */
> >>>>     #endif
> >>>>             unsigned len;                /**< Circular buffer length */
> >>>>             unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
> >>>>             void *volatile buffer[];     /**< The buffer contains mbuf
> >>>>     pointers */
> >>>>     };
> >>>>
> >>>>     Anonymous union example:
> >>>>     https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
> >>>>
> >>>>     You can check the ABI breakage by devtools/validate-abi.sh
> >>>>
> >>>>     >
> >>>>     > -- Ola
> >>>>     >
> >>>>     > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
> >>>>     > Date: Wednesday, 29 August 2018 at 07:50
> >>>>     > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>,
> Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,
> Jerin" <Jerin.JacobKollanukkaran@cavium.com>
> >>>>     > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola
> Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
> >>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     > Agreed. Please go a head and make the changes. You need to make
> same change in kernel side also. And please use c11 ring (see rte_ring)
> mechanism so that it won't impact other platforms like intel. We need this
> change just for arm and ppc.
> >>>>     >
> >>>>     > ________________________________
> >>>>     > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> >>>>     > Sent: Wednesday, August 29, 2018 10:29 AM
> >>>>     > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
> >>>>     > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
> >>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     > External Email
> >>>>     >
> >>>>     > I agree with Gavin here. Store to fifo->write and fifo->read can get
> hoisted resulting in accessing invalid buffer array entries or over writing of the
> buffer array entries.
> >>>>     >
> >>>>     > IMO, we should solve this using c11 atomics. This will also help
> remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > If you want us to put together a patch with this idea, please let us
> know.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > Thank you,
> >>>>     >
> >>>>     > Honnappa
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > From: Gavin Hu
> >>>>     > Sent: Tuesday, August 28, 2018 2:31 PM
> >>>>     > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh
> Yigit <ferruh.yigit@intel.com>; Jacob, Jerin
> <Jerin.JacobKollanukkaran@cavium.com>
> >>>>     > Cc: dev@dpdk.org; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl
> <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
> >>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > Assuming reader and writer may execute on different CPU's, this
> become standard multithreaded programming.
> >>>>     >
> >>>>     > We are concerned about that update the reader pointer too
> early(weak ordering may reorder it before reading from the slots), that means
> the slots are released and may immediately overwritten by the writer then
> you get “too new” data and get lost of the old data.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > From: Kokkilagadda, Kiran
> <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
> >>>>     > Sent: Tuesday, August 28, 2018 6:44 PM
> >>>>     > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>;
> Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob,
> Jerin
> <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@ca
> vium.com>>
> >>>>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
> >>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > In this instance there won't be any problem, as until the value of
> fifo->write changes, this loop won't get executed. As of now we didn't see any
> issue with it and for performance reasons, we don't want to keep read barrier.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > ________________________________
> >>>>     >
> >>>>     > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
> >>>>     > Sent: Monday, August 27, 2018 9:10 PM
> >>>>     > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
> >>>>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
> >>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > External Email
> >>>>     >
> >>>>     > This fix is not complete, kni_fifo_get requires a read fence also,
> otherwise it probably gets stale data on a weak ordering platform.
> >>>>     >
> >>>>     > > -----Original Message-----
> >>>>     > > From: dev <dev-bounces@dpdk.org<mailto:dev-
> bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> >>>>     > > Sent: Monday, August 27, 2018 10:08 PM
> >>>>     > > To: Kiran Kumar
> <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetwor
> ks.com>>;
> >>>>     > >
> jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> >>>>     > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> >>>>     > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>>>     > > synchronization
> >>>>     > >
> >>>>     > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> >>>>     > > > With existing code in kni_fifo_put, rx_q values are not being
> updated
> >>>>     > > > before updating fifo_write. While reading rx_q in
> kni_net_rx_normal,
> >>>>     > > > This is causing the sync issue on other core. So adding a write
> >>>>     > > > barrier to make sure the values being synced before updating
> fifo_write.
> >>>>     > > >
> >>>>     > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >>>>     > > >
> >>>>     > > > Signed-off-by: Kiran Kumar
> <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetwor
> ks.com>>
> >>>>     > > > Acked-by: Jerin Jacob
> <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com
> >>
> >>>>     > >
> >>>>     > > Acked-by: Ferruh Yigit
> <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> >>>>     > IMPORTANT NOTICE: The contents of this email and any
> attachments are confidential and may also be privileged. If you are not the
> intended recipient, please notify the sender immediately and do not disclose
> the contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
> >>>>
> >>>>