From: "Michael S. Tsirkin" <mst@redhat.com>
To: Shahaf Shuler <shahafs@mellanox.com>
Cc: Ilya Maximets <i.maximets@samsung.com>,
"dev@dpdk.org" <dev@dpdk.org>,
Maxime Coquelin <maxime.coquelin@redhat.com>,
Xiao Wang <xiao.w.wang@intel.com>,
"jfreimann@redhat.com" <jfreimann@redhat.com>,
Tiwei Bie <tiwei.bie@intel.com>,
Zhihong Wang <zhihong.wang@intel.com>,
Jason Wang <jasowang@redhat.com>,
"xiaolong.ye@intel.com" <xiaolong.ye@intel.com>,
"alejandro.lucero@netronome.com" <alejandro.lucero@netronome.com>,
Daniel Marcovitch <danielm@mellanox.com>
Subject: Re: [dpdk-dev] [PATCH v2] net/virtio: add platform memory ordering feature support
Date: Tue, 12 Feb 2019 12:50:57 -0500 [thread overview]
Message-ID: <20190212123732-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <DB7PR05MB44264D757FE9E008794327B8C3810@DB7PR05MB4426.eurprd05.prod.outlook.com>
On Tue, Jan 15, 2019 at 08:55:50AM +0000, Shahaf Shuler wrote:
> Tuesday, January 15, 2019 10:29 AM, Ilya Maximets:
> > Subject: Re: [dpdk-dev] [PATCH v2] net/virtio: add platform memory
> > ordering feature support
> >
> > On 15.01.2019 9:33, Shahaf Shuler wrote:
> > > Thursday, January 10, 2019 10:37 PM, Shahaf Shuler:
> > >> Subject: RE: [dpdk-dev] [PATCH v2] net/virtio: add platform memory
> > >> ordering feature support
> > >>
> > >> Wednesday, January 9, 2019 5:50 PM, Michael S. Tsirkin:
> > >>> alejandro.lucero@netronome.com; Daniel Marcovitch
> > >>> <danielm@mellanox.com>
> > >>> Subject: Re: [dpdk-dev] [PATCH v2] net/virtio: add platform memory
> > >>> ordering feature support
> > >>>
> > >>> On Wed, Jan 09, 2019 at 05:34:38PM +0300, Ilya Maximets wrote:
> > >>>> virtio_mb() is really heavy. I'd like to avoid it somehow, but I
> > >>>> don't know how to do this yet.
> > >>>
> > >>> Linux driver doesn't avoid it either.
> > >>
> > >> I understand v3 was merged but still would like to continue the
> > >> discuss and make sure all is clear and agreed.
> > >>
> > >> Form patch [1] description it is very clear why we need the
> > >> rte_smp_mb() barrier.
> > >> However I am not sure why this barrier is interoperate into rte_mb in
> > >> case of vDPA. In vDPA case, both read of the user ring and write of
> > >> the avail index are for local cached memory.
> > >> The only write which is to uncachable memory (device memory) is the
> > >> notify itself.
> > >>
> > >> As I mentioned, there is a need to have a store fence before doing
> > >> the notify, but from different reasons. So vDPA use case and need Is
> > >> a bit different than what presented in [1].
> > >
> > > Any answer?
> > > It is pity if we add redundant barriers which will degrade the driver
> > performance.
> >
> > Sorry for late responses. Have a lot of work with OVS right now.
> >
> > Regarding your question.
> > Current code looks like this:
> >
> > 1. Update ring.
> > 2. virtio_wmb()
> > 3. Update idx.
> > 4. virtio_mb()
> > 5. read flags.
> > 6. notify.
> >
> > virtio_mb() is here to avoid reordering of steps 3 and 5.
> > i.e. we need a full barrier to ensure the order between store (idx update)
> > and load (reading the flags). Otherwise we could miss the notification.
> > We can't avoid the barrier here, because even x86 does not guarantee the
> > ordering of the local load with earlier local store.
>
> This is clear. You need the rte_smp_mb() here. My question is why you need the rte_mb() in case of vDPA?
> As you said, all accesses are local.
Are you asking why the different barriers? Or as you asking why is a barrier
needed at all?
The barriers themselves are clearly needed.
But in my opinion some dpdk barrier implementations are sub-optimal and too
strong. For example on intel: the big question is whether anyone does
any non-temporals. In absence of these, and with non-cacheable mappings
on x86 wmb and rmb should be a nop, and mb should be a locked instruction.
It might make sense to add rte_dma_rmb/wmb/mb.
> Pasting you commit code:
> /*
> * Per virtio_ring.h in Linux.
> * For virtio_pci on SMP, we don't need to order with respect to MMIO
> * accesses through relaxed memory I/O windows, so smp_mb() et al are
> * sufficient.
> *
> * For using virtio to talk to real devices (eg. vDPA) we do need real
> * barriers.
> */
> static inline void
> virtio_mb(uint8_t weak_barriers)
> {
> if (weak_barriers)
> rte_smp_mb();
> else
> rte_mb();
> }
>
> >
> > >
> > >>
> > >> [1]
> > >>
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpa
> > >>
> > tches.dpdk.org%2Fpatch%2F49545%2F&data=02%7C01%7Cshahafs%40
> > mellan
> > >>
> > ox.com%7C01907f1b2e0e4002cb7508d67ac38a98%7Ca652971c7d2e4d9ba6a4
> > d1492
> > >>
> > 56f461b%7C0%7C0%7C636831377591864200&sdata=TSpc%2Fzyq2aq0N3
> > %2Bh4o
> > >> ro4std8ut%2FQU6%2BOeMDeeaQdsM%3D&reserved=0
> > >>
> > >>>
> > >>> --
> > >>> MST
> > >
> > >
next prev parent reply other threads:[~2019-02-12 17:51 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CGME20181214153817eucas1p19a41cdd791879252e1f3a5d77c427845@eucas1p1.samsung.com>
2018-12-14 15:38 ` [dpdk-dev] [PATCH] " Ilya Maximets
2018-12-14 17:00 ` Michael S. Tsirkin
2018-12-14 17:23 ` Ilya Maximets
[not found] ` <CGME20181226163717eucas1p15276eb45e35abe2c9cf3e7c1e0050823@eucas1p1.samsung.com>
2018-12-26 16:37 ` [dpdk-dev] [PATCH v2] " Ilya Maximets
2018-12-27 10:07 ` Shahaf Shuler
2019-01-09 14:34 ` Ilya Maximets
2019-01-09 15:50 ` Michael S. Tsirkin
2019-01-10 20:36 ` Shahaf Shuler
2019-01-15 6:33 ` Shahaf Shuler
2019-01-15 8:29 ` Ilya Maximets
2019-01-15 8:55 ` Shahaf Shuler
2019-01-15 10:23 ` Ilya Maximets
2019-02-12 17:50 ` Michael S. Tsirkin [this message]
[not found] ` <CGME20190109145021eucas1p1bfe194ffafaaaa5df62243c92b2ed6cd@eucas1p1.samsung.com>
2019-01-09 14:50 ` [dpdk-dev] [PATCH v3 0/3] Missing barriers and VIRTIO_F_ORDER_PLATFORM Ilya Maximets
[not found] ` <CGME20190109145027eucas1p2437215de0df4c691eb84d4e84bfc71e5@eucas1p2.samsung.com>
2019-01-09 14:50 ` [dpdk-dev] [PATCH v3 1/3] net/virtio: add missing barrier before reading the flags Ilya Maximets
2019-01-10 14:31 ` Maxime Coquelin
[not found] ` <CGME20190109145034eucas1p2183e275e316b87917b96fa184fc7d7cb@eucas1p2.samsung.com>
2019-01-09 14:50 ` [dpdk-dev] [PATCH v3 2/3] net/virtio: update memory ordering comment for vq notify Ilya Maximets
2019-01-10 8:19 ` Gavin Hu (Arm Technology China)
2019-01-10 9:18 ` Maxime Coquelin
2019-01-10 9:55 ` Ilya Maximets
2019-01-10 14:56 ` Michael S. Tsirkin
2019-01-10 14:31 ` Maxime Coquelin
[not found] ` <CGME20190109145040eucas1p2d9afc678ef94986544bde07b77373e6f@eucas1p2.samsung.com>
2019-01-09 14:50 ` [dpdk-dev] [PATCH v3 3/3] net/virtio: add platform memory ordering feature support Ilya Maximets
2019-01-10 14:31 ` Maxime Coquelin
2019-01-09 14:55 ` [dpdk-dev] [PATCH v3 0/3] Missing barriers and VIRTIO_F_ORDER_PLATFORM Michael S. Tsirkin
2019-01-09 15:24 ` Ilya Maximets
2019-01-09 16:53 ` Ferruh Yigit
2019-01-10 15:19 ` Maxime Coquelin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190212123732-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=alejandro.lucero@netronome.com \
--cc=danielm@mellanox.com \
--cc=dev@dpdk.org \
--cc=i.maximets@samsung.com \
--cc=jasowang@redhat.com \
--cc=jfreimann@redhat.com \
--cc=maxime.coquelin@redhat.com \
--cc=shahafs@mellanox.com \
--cc=tiwei.bie@intel.com \
--cc=xiao.w.wang@intel.com \
--cc=xiaolong.ye@intel.com \
--cc=zhihong.wang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).