From: Gavin Hu <Gavin.Hu@arm.com>
To: Jerin Jacob <jerinjacobk@gmail.com>
Cc: dpdk-dev <dev@dpdk.org>, nd <nd@arm.com>,
David Marchand <david.marchand@redhat.com>,
"thomas@monjalon.net" <thomas@monjalon.net>,
"rasland@mellanox.com" <rasland@mellanox.com>,
"maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>,
"tiwei.bie@intel.com" <tiwei.bie@intel.com>,
"hemant.agrawal@nxp.com" <hemant.agrawal@nxp.com>,
"jerinj@marvell.com" <jerinj@marvell.com>,
Pavan Nikhilesh <pbhagavatula@marvell.com>,
Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>,
Ruifeng Wang <Ruifeng.Wang@arm.com>,
Phil Yang <Phil.Yang@arm.com>, Joyce Kong <Joyce.Kong@arm.com>,
Steve Capper <Steve.Capper@arm.com>, nd <nd@arm.com>
Subject: Re: [dpdk-dev] [PATCH v2 1/3] eal/arm64: relax the io barrier for aarch64
Date: Mon, 23 Dec 2019 09:14:05 +0000 [thread overview]
Message-ID: <VI1PR08MB5376D17A59C4A1495A11AF508F2E0@VI1PR08MB5376.eurprd08.prod.outlook.com> (raw)
In-Reply-To: <CALBAE1Oc00oBCtuxqCNQ7-UTsOxwjtrQfE+YFBNGkpk=MGabPA@mail.gmail.com>
Hi Jerin,
I think we are on the same page with regard to the problem, and the situations, thanks for illuminating the historical background of the two barriers.
About the solution, I added inline comments.
> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Friday, December 20, 2019 2:56 PM
> To: Gavin Hu <Gavin.Hu@arm.com>
> Cc: dpdk-dev <dev@dpdk.org>; nd <nd@arm.com>; David Marchand
> <david.marchand@redhat.com>; thomas@monjalon.net;
> rasland@mellanox.com; maxime.coquelin@redhat.com;
> tiwei.bie@intel.com; hemant.agrawal@nxp.com; jerinj@marvell.com;
> Pavan Nikhilesh <pbhagavatula@marvell.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; Ruifeng Wang
> <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>; Joyce Kong
> <Joyce.Kong@arm.com>; Steve Capper <Steve.Capper@arm.com>
> Subject: Re: [dpdk-dev] [PATCH v2 1/3] eal/arm64: relax the io barrier for
> aarch64
>
> On Fri, Dec 20, 2019 at 12:02 PM Gavin Hu <Gavin.Hu@arm.com> wrote:
> >
> > Hi Jerin,
>
> Hi Gavin,
>
>
> > > > > > >
> > > > > > > The peripheral coherence order for a memory-mapped peripheral
> > > > > signifies the
> > > > > > > order in which accesses arrive at the endpoint. For a read or a
> write
> > > > > RW1
> > > > > > > and a read or a write RW2 to the same peripheral, then RW1 will
> > > appear
> > > > > in
> > > > > > > the peripheral coherence order for the peripheral before RW2 if
> > > either
> > > > > of
> > > > > > > the following cases apply:
> > > > > > > 1. RW1 and RW2 are accesses using Non-cacheable or Device
> > > attributes
> > > > > and
> > > > > > > RW1 is Ordered-before RW2.
> > > > > > > 2. RW1 and RW2 are accesses using Device-nGnRE or Device-
> > > nGnRnE
> > > > > attributes
> > > > > > > and RW1 appears in program order before RW2.
> > > > > >
> > > > > >
> > > > > > This is true if RW1 and RW2 addresses are device memory. i.e the
> > > > > > registers in the PCI bar address.
> > > > > > If RW1 is DDR address which is been used by the controller(say NIC
> > > > > > ring descriptor) then there will be an issue.
> > > > > > For example Intel i40e driver, the admin queue update in Host DDR
> > > > > > memory and it updates the doorbell.
> > > > > > In such a case, this patch will create an issue. Correct? Have you
> > > > > > checked this patch with ARM64 + XL710 controllers?
> > > >
> > > > This patch relaxes the rte_io_*mb barriers for pure PCI device memory
> > > accesses.
> > >
> > > Yes. This would break cases for mixed access fro i40e drivers.
> > >
> > > >
> > > > For mixed accesses of DDR and PCI device memory,
> rte_smp_*mb(DMB
> > > ISH) is not sufficient.
> > > > But rte_cio_*mb(DMB OSH) is sufficient and can be used.
> > >
> > > Yes. Let me share a bit of history.
> > >
> > > 1) There are a lot of drivers(initially developed in x86) that have
> > > mixed access and don't have any barriers as x86 does not need it.
> > > 2) rte_io introduced to fix that
> > > 3) Item (2) introduced the performance issues in the fast path as an
> > > optimization rte_cio_* introduced.
> > Exactly, this patch is to mitigate the performance issues introduced by
> rte_io('dsb' is too much and unnecessary here).
> > Rte_cio instead is definitely required for mixed access.
> > >
> > > So in the current of the scheme of things, we have APIs to FIX
> > > portability issue(rte_io) and performance issue(rte_cio).
> > > IMO, we may not need any change in infra code now. If you think, the
> > > documentation is missing then we can enhance it.
> > > If we make infra change then again drivers needs to be updated and
> tested.
> > No changes for rte_cio, the semantics, and definitions of rte_io does not
> change either, if limited the scope to PCI, which is the case in DPDK
> context(?).
> > The change lies only in the implementation, right?
> >
> > Just looked at the link you shared and found i40 driver is missing
> rte_cio_*mb in i40e_asq_send_command, but the old rte_io_*mb rescued.
> > Will submit a new patch in this series to used rte_cio together with new
> relaxed rte_io and do more tests.
> >
> > Yes, this is a big change, also a big optimization, for aarch64, in our tests it
> has very positive results.
>
> It will be optimization only when if we are changing in the fast path.
> In the slow path, it does not matter.
> I think, the First step should be to use rte_cio_* wherever it is
> coherent memory used in _fast path_. I think, Almost every driver
> fixed that.
>
> I am not against this patch(changing the slow path to use rte_cio*
> from rte_io* and virtio changes associated with that).
> If you are taking that patch, pay attention to all the drivers in the
> tree which is using rte_io* for mixed access in slowpath.
I see 30+ drivers has calling rte_io* directly or indirectly through rte_write/read*.
It is hard for me to figure out all the mixed accesses in these drivers, and as you said, it makes no sense to change the _slow path_.
How about we keep the old rte_io as is, and introduce 'fast path' version of rte_io for new code use?
Then in future, we may merge the two?
Another reason about this proposal is maybe there is rte_io calling in the fast path, but they are not mixed accesses and rte_cio is not suitable.
Any thoughts?
>
> > But as the case in i40e, we must pay attention to where rte_cio was
> missing but rescued by old rte_io(but not by new rte_io).
> >
> >
next prev parent reply other threads:[~2019-12-23 9:14 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-22 15:27 [dpdk-dev] [PATCH v1 0/3] relax io barrier for aarch64 and use smp barriers for virtual pci memory Gavin Hu
2019-10-22 15:27 ` [dpdk-dev] [PATCH v1 1/3] eal/arm64: relax the io barrier for aarch64 Gavin Hu
2019-10-22 15:27 ` [dpdk-dev] [PATCH v1 2/3] net/virtio: virtual PCI requires smp barriers Gavin Hu
2019-10-22 15:27 ` [dpdk-dev] [PATCH v1 3/3] crypto/virtio: " Gavin Hu
2019-10-23 8:22 ` [dpdk-dev] [PATCH v1 0/3] relax io barrier for aarch64 and use smp barriers for virtual pci memory Maxime Coquelin
2019-11-07 1:13 ` Gavin Hu (Arm Technology China)
2019-12-20 3:09 ` [dpdk-dev] [PATCH v2 " Gavin Hu
2019-12-20 3:09 ` [dpdk-dev] [PATCH v2 1/3] eal/arm64: relax the io barrier for aarch64 Gavin Hu
2019-12-20 3:33 ` Jerin Jacob
2019-12-20 3:38 ` Jerin Jacob
2019-12-20 4:19 ` Gavin Hu
2019-12-20 4:34 ` Jerin Jacob
2019-12-20 6:32 ` Gavin Hu
2019-12-20 6:55 ` Jerin Jacob
2019-12-23 9:14 ` Gavin Hu [this message]
2019-12-23 9:19 ` Jerin Jacob
2019-12-23 10:16 ` Gavin Hu
2020-01-02 9:51 ` Jerin Jacob
2020-01-03 6:30 ` Gavin Hu
2020-01-03 7:34 ` Jerin Jacob
2020-01-03 9:12 ` Gavin Hu
2019-12-20 3:09 ` [dpdk-dev] [PATCH v2 2/3] net/virtio: virtual PCI requires smp barriers Gavin Hu
2019-12-20 8:17 ` Tiwei Bie
2019-12-20 10:19 ` Gavin Hu
2019-12-20 3:09 ` [dpdk-dev] [PATCH v2 3/3] crypto/virtio: " Gavin Hu
2020-02-08 13:48 ` [dpdk-dev] [PATCH v3] net/i40e: relaxed barrier in the tx fastpath Gavin Hu
2020-02-11 2:11 ` Ye Xiaolong
2020-02-12 6:02 ` Gavin Hu
2020-02-15 8:25 ` Jerin Jacob
2020-02-12 5:56 ` [dpdk-dev] [PATCH v4] " Gavin Hu
2020-02-15 15:16 ` Ye Xiaolong
2020-02-16 9:51 ` Thomas Monjalon
2020-02-16 16:38 ` Ye Xiaolong
2020-02-16 17:36 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=VI1PR08MB5376D17A59C4A1495A11AF508F2E0@VI1PR08MB5376.eurprd08.prod.outlook.com \
--to=gavin.hu@arm.com \
--cc=Honnappa.Nagarahalli@arm.com \
--cc=Joyce.Kong@arm.com \
--cc=Phil.Yang@arm.com \
--cc=Ruifeng.Wang@arm.com \
--cc=Steve.Capper@arm.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=hemant.agrawal@nxp.com \
--cc=jerinj@marvell.com \
--cc=jerinjacobk@gmail.com \
--cc=maxime.coquelin@redhat.com \
--cc=nd@arm.com \
--cc=pbhagavatula@marvell.com \
--cc=rasland@mellanox.com \
--cc=thomas@monjalon.net \
--cc=tiwei.bie@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).