Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer synchronization

[Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>]

> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> Sent: Tuesday, September 18, 2018 10:54 AM
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; Kokkilagadda, Kiran
> <Kiran.Kokkilagadda@cavium.com>
> Cc: Ola Liljedahl <Ola.Liljedahl@arm.com>; Gavin Hu (Arm Technology China)
> <Gavin.Hu@arm.com>; Jacob, Jerin <Jerin.JacobKollanukkaran@cavium.com>;
> dev@dpdk.org; nd <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>;
> Phil Yang (Arm Technology China) <Phil.Yang@arm.com>; Bruce Richardson
> <bruce.richardson@intel.com>; Konstantin Ananyev
> <konstantin.ananyev@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> 
> On 9/14/2018 3:45 AM, Jerin Jacob wrote:
> > -----Original Message-----
> >> Date: Thu, 13 Sep 2018 23:45:31 +0000
> >> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> >> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> >> CC: Ola Liljedahl <Ola.Liljedahl@arm.com>, "Kokkilagadda, Kiran"
> >>  <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu (Arm Technology China)"
> >>  <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,
> Jerin"
> >>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org"
> >> <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper
> >> <Steve.Capper@arm.com>, "Phil Yang (Arm  Technology China)"
> >> <Phil.Yang@arm.com>
> >> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >> synchronization
> >>
> >> External Email
> >>
> >> -----Original Message-----
> >>> Date: Thu, 13 Sep 2018 17:40:53 +0000
> >>> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> >>> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>, Ola Liljedahl
> >>> <Ola.Liljedahl@arm.com>
> >>> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, "Gavin Hu
> >>> (Arm  Technology China)" <Gavin.Hu@arm.com>, Ferruh Yigit
> >>> <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >>>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org"
> >>> <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper
> >>> <Steve.Capper@arm.com>, "Phil Yang (Arm Technology China)"
> >>> <Phil.Yang@arm.com>
> >>> Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>> synchronization
> >>>
> >>>
> >>> Hi Jerin,
> >>>         Is there any reason for having 'RTE_RING_USE_C11_MEM_MODEL',
> which is specific to rte_ring? I do not see a need for choosing only some
> algorithms to work with C11 model. I suggest that we change this to
> 'RTE_USE_C11_MEM_MODEL' so that it can apply to all libraries/algorithms.
> >>
> >>
> >> Yes. Makes sense to me to keep only single config option.
> >>
> >> rte_ring has 2 sets of algorithms for Arm architecture, one with C11
> memory model and the other with barriers. Going forward (for ex: for KNI), I
> think we should support C11 memory model only and skip the barriers.
> >
> > IMO, Both should be supported and set N as in the config/common_base.
> > Based on architecture or micro architecture the performance can vary.
> > So keeping both options and allowing to override to arch/micro arch
> > specific config file makes sense to me.(like existing model, as smp_*
> > ops are compiler NOP for x86)
> 
> Hi Jerin, Honnappa,  Kiran,
> 
> Will there be a new version for this release?
> 
> I can see two options:
> 1- Add read/write barriers for both library and kernel parts.
> 2- Use c11 atomics
>   2a- change existing RTE_RING_USE_C11_MEM_MODEL to
> RTE_USE_C11_MEM_MODEL
>   2b- Use RTE_USE_C11_MEM_MODEL to implement c11 atomic for arm and
> ppc
> 
> 2) seems agreed on, but is it clear who will work on it?

Sorry for the late reply. We have implemented 2), currently undergoing internal review. We will get this out today. We will work through the community reviews quickly after that.

> 
> And 1) looks easier to implement, if 2) won't make time for release can we
> fallback to this one?
> 
> Thanks,
> ferruh
> 
> >
> >> Also, do you see any issues in making C11 memory model default for Arm
> architecture?
> >
> > It is already set default Y to arm64. see config/common_armv8a_linuxapp.
> >
> > And it is possible for micro architecture to override, see
> > config/defconfig_arm64-thunderx-linuxapp-gcc
> >
> >
> >>
> >>>
> >>> Thank you,
> >>> Honnappa
> >>>
> >>> -----Original Message-----
> >>> From: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> >>> Sent: Wednesday, August 29, 2018 3:58 AM
> >>> To: Ola Liljedahl <Ola.Liljedahl@arm.com>
> >>> Cc: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Honnappa
> >>> Nagarahalli <Honnappa.Nagarahalli@arm.com>; Gavin Hu
> >>> <Gavin.Hu@arm.com>; Ferruh Yigit <ferruh.yigit@intel.com>; Jacob,
> >>> Jerin <Jerin.JacobKollanukkaran@cavium.com>; dev@dpdk.org; nd
> >>> <nd@arm.com>; Steve Capper <Steve.Capper@arm.com>
> >>> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>> synchronization
> >>>
> >>> -----Original Message-----
> >>>> Date: Wed, 29 Aug 2018 08:47:56 +0000
> >>>> From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> >>>> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> >>>> CC: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>, Honnappa
> >>>> Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu
> >>>> <Gavin.Hu@arm.com>,  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,
> Jerin"
> >>>>  <Jerin.JacobKollanukkaran@cavium.com>, "dev@dpdk.org"
> >>>> <dev@dpdk.org>, nd  <nd@arm.com>, Steve Capper
> >>>> <Steve.Capper@arm.com>
> >>>> Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>>> synchronization
> >>>> user-agent: Microsoft-MacOutlook/10.10.0.180812
> >>>>
> >>>>
> >>>> There was a mention of rte_ring which is a different data structure. But
> perhaps I misunderstood why this was mentioned and the idea was only to
> use the C11 memory model as is also used in rte_ring nowadays.
> >>>>
> >>>> But why would we have different code for x86 and for other
> architectures (ARM, Power)? If we use the C11 memory model (and e.g. GCC
> __atomic builtins), the code generated for x86 will be the same.
> __atomic_load(__ATOMIC_ACQUIRE) and
> __atomic_store(__ATOMIC_RELEASE) should translate to plain loads and
> stores on x86?
> >>>
> >>> # One reason was __atomic builtins  primitives were implemented in gcc
> 4.7 and x86 would like to support < gcc 4.7 and ICC compiler.
> >>> # The theme was no change in the existing code for x86.I am not sure
> about the code generation for x86 with __atomic builtins, I let x86
> maintainers to comments on this.
> >>>
> >>>
> >>>>
> >>>> -- Ola
> >>>>
> >>>> On 29/08/2018, 10:28, "Jerin Jacob" <jerin.jacob@caviumnetworks.com>
> wrote:
> >>>>
> >>>>     -----Original Message-----
> >>>>     > Date: Wed, 29 Aug 2018 07:34:34 +0000
> >>>>     > From: Ola Liljedahl <Ola.Liljedahl@arm.com>
> >>>>     > To: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>,
> Honnappa
> >>>>     >  Nagarahalli <Honnappa.Nagarahalli@arm.com>, Gavin Hu
> <Gavin.Hu@arm.com>,
> >>>>     >  Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,  Jerin"
> >>>>     >  <Jerin.JacobKollanukkaran@cavium.com>
> >>>>     > CC: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Steve
> Capper
> >>>>     >  <Steve.Capper@arm.com>
> >>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>>>     >  synchronization
> >>>>     > user-agent: Microsoft-MacOutlook/10.10.0.180812
> >>>>     >
> >>>>     > Is the rte_kni kernel/user binary interface subject to backwards
> compatibility requirements? Or can we change it for a new DPDK release?
> >>>>
> >>>>     What would be the change in interface? Is it removing the volatile for
> >>>>     C11 case, Then you can use anonymous union OR #define to keep the
> size
> >>>>     and offset of the element intact.
> >>>>
> >>>>     struct rte_kni_fifo {
> >>>>     #ifndef RTE_C11...
> >>>>             volatile unsigned write;     /**< Next position to be written*/
> >>>>             volatile unsigned read;      /**< Next position to be read */
> >>>>     #else
> >>>>             unsigned write;     /**< Next position to be written*/
> >>>>             unsigned read;      /**< Next position to be read */
> >>>>     #endif
> >>>>             unsigned len;                /**< Circular buffer length */
> >>>>             unsigned elem_size;          /**< Pointer size - for 32/64 bitOS */
> >>>>             void *volatile buffer[];     /**< The buffer contains mbuf
> >>>>     pointers */
> >>>>     };
> >>>>
> >>>>     Anonymous union example:
> >>>>     https://git.dpdk.org/dpdk/tree/lib/librte_mbuf/rte_mbuf.h#n461
> >>>>
> >>>>     You can check the ABI breakage by devtools/validate-abi.sh
> >>>>
> >>>>     >
> >>>>     > -- Ola
> >>>>     >
> >>>>     > From: "Kokkilagadda, Kiran" <Kiran.Kokkilagadda@cavium.com>
> >>>>     > Date: Wednesday, 29 August 2018 at 07:50
> >>>>     > To: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>,
> Gavin Hu <Gavin.Hu@arm.com>, Ferruh Yigit <ferruh.yigit@intel.com>, "Jacob,
> Jerin" <Jerin.JacobKollanukkaran@cavium.com>
> >>>>     > Cc: "dev@dpdk.org" <dev@dpdk.org>, nd <nd@arm.com>, Ola
> Liljedahl <Ola.Liljedahl@arm.com>, Steve Capper <Steve.Capper@arm.com>
> >>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     > Agreed. Please go a head and make the changes. You need to make
> same change in kernel side also. And please use c11 ring (see rte_ring)
> mechanism so that it won't impact other platforms like intel. We need this
> change just for arm and ppc.
> >>>>     >
> >>>>     > ________________________________
> >>>>     > From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> >>>>     > Sent: Wednesday, August 29, 2018 10:29 AM
> >>>>     > To: Gavin Hu; Kokkilagadda, Kiran; Ferruh Yigit; Jacob, Jerin
> >>>>     > Cc: dev@dpdk.org; nd; Ola Liljedahl; Steve Capper
> >>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     > External Email
> >>>>     >
> >>>>     > I agree with Gavin here. Store to fifo->write and fifo->read can get
> hoisted resulting in accessing invalid buffer array entries or over writing of the
> buffer array entries.
> >>>>     >
> >>>>     > IMO, we should solve this using c11 atomics. This will also help
> remove the use of ‘volatile’ from ‘rte_kni_fifo’ structure.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > If you want us to put together a patch with this idea, please let us
> know.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > Thank you,
> >>>>     >
> >>>>     > Honnappa
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > From: Gavin Hu
> >>>>     > Sent: Tuesday, August 28, 2018 2:31 PM
> >>>>     > To: Kokkilagadda, Kiran <Kiran.Kokkilagadda@cavium.com>; Ferruh
> Yigit <ferruh.yigit@intel.com>; Jacob, Jerin
> <Jerin.JacobKollanukkaran@cavium.com>
> >>>>     > Cc: dev@dpdk.org; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; nd <nd@arm.com>; Ola Liljedahl
> <Ola.Liljedahl@arm.com>; Steve Capper <Steve.Capper@arm.com>
> >>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > Assuming reader and writer may execute on different CPU's, this
> become standard multithreaded programming.
> >>>>     >
> >>>>     > We are concerned about that update the reader pointer too
> early(weak ordering may reorder it before reading from the slots), that means
> the slots are released and may immediately overwritten by the writer then
> you get “too new” data and get lost of the old data.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > From: Kokkilagadda, Kiran
> <Kiran.Kokkilagadda@cavium.com<mailto:Kiran.Kokkilagadda@cavium.com>>
> >>>>     > Sent: Tuesday, August 28, 2018 6:44 PM
> >>>>     > To: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>;
> Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>; Jacob,
> Jerin
> <Jerin.JacobKollanukkaran@cavium.com<mailto:Jerin.JacobKollanukkaran@ca
> vium.com>>
> >>>>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com<mailto:Honnappa.Nagarahalli@arm.com>>
> >>>>     > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > In this instance there won't be any problem, as until the value of
> fifo->write changes, this loop won't get executed. As of now we didn't see any
> issue with it and for performance reasons, we don't want to keep read barrier.
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > ________________________________
> >>>>     >
> >>>>     > From: Gavin Hu <Gavin.Hu@arm.com<mailto:Gavin.Hu@arm.com>>
> >>>>     > Sent: Monday, August 27, 2018 9:10 PM
> >>>>     > To: Ferruh Yigit; Kokkilagadda, Kiran; Jacob, Jerin
> >>>>     > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Honnappa Nagarahalli
> >>>>     > Subject: RE: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> synchronization
> >>>>     >
> >>>>     >
> >>>>     >
> >>>>     > External Email
> >>>>     >
> >>>>     > This fix is not complete, kni_fifo_get requires a read fence also,
> otherwise it probably gets stale data on a weak ordering platform.
> >>>>     >
> >>>>     > > -----Original Message-----
> >>>>     > > From: dev <dev-bounces@dpdk.org<mailto:dev-
> bounces@dpdk.org>> On Behalf Of Ferruh Yigit
> >>>>     > > Sent: Monday, August 27, 2018 10:08 PM
> >>>>     > > To: Kiran Kumar
> <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetwor
> ks.com>>;
> >>>>     > >
> jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com>
> >>>>     > > Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> >>>>     > > Subject: Re: [dpdk-dev] [PATCH v2] kni: fix kni Rx fifo producer
> >>>>     > > synchronization
> >>>>     > >
> >>>>     > > On 8/16/2018 10:55 AM, Kiran Kumar wrote:
> >>>>     > > > With existing code in kni_fifo_put, rx_q values are not being
> updated
> >>>>     > > > before updating fifo_write. While reading rx_q in
> kni_net_rx_normal,
> >>>>     > > > This is causing the sync issue on other core. So adding a write
> >>>>     > > > barrier to make sure the values being synced before updating
> fifo_write.
> >>>>     > > >
> >>>>     > > > Fixes: 3fc5ca2f6352 ("kni: initial import")
> >>>>     > > >
> >>>>     > > > Signed-off-by: Kiran Kumar
> <kkokkilagadda@caviumnetworks.com<mailto:kkokkilagadda@caviumnetwor
> ks.com>>
> >>>>     > > > Acked-by: Jerin Jacob
> <jerin.jacob@caviumnetworks.com<mailto:jerin.jacob@caviumnetworks.com
> >>
> >>>>     > >
> >>>>     > > Acked-by: Ferruh Yigit
> <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> >>>>     > IMPORTANT NOTICE: The contents of this email and any
> attachments are confidential and may also be privileged. If you are not the
> intended recipient, please notify the sender immediately and do not disclose
> the contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
> >>>>
> >>>>