From: Andrew Rybchenko <arybchenko@solarflare.com>
To: Gavin Hu <Gavin.Hu@arm.com>, "dev@dpdk.org" <dev@dpdk.org>
Cc: nd <nd@arm.com>,
"david.marchand@redhat.com" <david.marchand@redhat.com>,
"thomas@monjalon.net" <thomas@monjalon.net>,
"rasland@mellanox.com" <rasland@mellanox.com>,
"drc@linux.vnet.ibm.com" <drc@linux.vnet.ibm.com>,
"bruce.richardson@intel.com" <bruce.richardson@intel.com>,
"konstantin.ananyev@intel.com" <konstantin.ananyev@intel.com>,
"matan@mellanox.com" <matan@mellanox.com>,
"shahafs@mellanox.com" <shahafs@mellanox.com>,
"viacheslavo@mellanox.com" <viacheslavo@mellanox.com>,
"jerinj@marvell.com" <jerinj@marvell.com>,
Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>,
Ruifeng Wang <Ruifeng.Wang@arm.com>,
Phil Yang <Phil.Yang@arm.com>, Joyce Kong <Joyce.Kong@arm.com>,
Steve Capper <Steve.Capper@arm.com>
Subject: Re: [dpdk-dev] [PATCH RFC v2 0/7] introduce new barrier class and use it for mlx5 PMD
Date: Mon, 13 Apr 2020 12:51:39 +0300 [thread overview]
Message-ID: <a83b6301-c564-f26c-6ff5-afda941a523a@solarflare.com> (raw)
In-Reply-To: <VI1PR08MB5376C1F8C7464B792F9894248FDF0@VI1PR08MB5376.eurprd08.prod.outlook.com>
On 4/11/20 6:46 AM, Gavin Hu wrote:
> Hi Andrew,
>
>> -----Original Message-----
>> From: Andrew Rybchenko <arybchenko@solarflare.com>
>> Sent: Saturday, April 11, 2020 1:21 AM
>> To: Gavin Hu <Gavin.Hu@arm.com>; dev@dpdk.org
>> Cc: nd <nd@arm.com>; david.marchand@redhat.com;
>> thomas@monjalon.net; rasland@mellanox.com; drc@linux.vnet.ibm.com;
>> bruce.richardson@intel.com; konstantin.ananyev@intel.com;
>> matan@mellanox.com; shahafs@mellanox.com; viacheslavo@mellanox.com;
>> jerinj@marvell.com; Honnappa Nagarahalli
>> <Honnappa.Nagarahalli@arm.com>; Ruifeng Wang
>> <Ruifeng.Wang@arm.com>; Phil Yang <Phil.Yang@arm.com>; Joyce Kong
>> <Joyce.Kong@arm.com>; Steve Capper <Steve.Capper@arm.com>
>> Subject: Re: [dpdk-dev] [PATCH RFC v2 0/7] introduce new barrier class and
>> use it for mlx5 PMD
>>
>> On 4/10/20 7:41 PM, Gavin Hu wrote:
>>> To order writes to various memory types, 'sfence' is required for x86,
>>> and 'dmb oshst' is required for aarch64.
>>>
>>> But within DPDK, there is no abstracted barriers covers this
>>> combination: sfence(x86)/dmb(aarch64).
>>>
>>> So introduce a new barrier class - rte_dma_*mb for this combination,
>>>
>>> Doorbell rings are typical use cases of this new barrier class, which
>>> requires something ready in the memory before letting HW aware.
>>>
>>> As a note, rte_io_wmb and rte_cio_wmb are compiler barriers for x86,
>> while
>>> rte_wmb is 'dsb' for aarch64.
>>
>> As far as I can see rte_cio_wmb() is exactly definition of the barrier
>> to be used for doorbells. Am I missing something?
>
> I understand rte_cio_wmb is for DMA buffers, for examples, descriptors, work queues, located in the host memory, but shared between CPU and IO device.
> rte_io_wmb is for MMIO regions.
> We are missing the barriers for various memory types, eg. Doorbell cases.
When the patch series is applied, we'll have 5 types of memory
barriers: regular, smp, cio, io, dma. Do we really need so
many? May be we need a table in description which could
help to make the right choice. I.e. type of access on both
axis and type of barrier to use on intersection.
> There is an implication in the definition of rte_cio_wmb, it can not be used for non-coherent MMIO region(WC?)
> http://code.dpdk.org/dpdk/v20.02/source/lib/librte_eal/common/include/generic/rte_atomic.h#L124
>> May be it is just a bug in rte_cio_wmb() on x86?
> rte_cio_wmb is ok for doorbells on aarch64, but looking through the kernel code, 'sfence' is required for various/mixed memory types.
> DPDK mlx5 PMD uses rte_cio_wmb widely and wisely, it orders sequences of writes to host memory that shared by IO device.
> Strengthening rte_cio_wmb may hurt performance, so a new barrier class is introduced to optimize for aarch64, in the fast path only, while not impacting x86.
> http://code.dpdk.org/dpdk/v20.02/source/drivers/net/mlx5/mlx5_rxtx.c#L1087
May be my problem that I don't fully understand real-life
usecases when cio should be used in accordance with its
current definition. Does it make sense without doorbell?
Does HW polling via DMA?
Thanks for explanations,
Andrew.
>>
>>> In the joint preliminary testing between Arm and Ampere, 8%~13%
>>> performance boost was measured.
>>>
>>> As there is no functionality changes, it will not impact x86.
>>>
>>> Gavin Hu (6):
>>> eal: introduce new class of barriers for DMA use cases
>>> net/mlx5: dmb for immediate doorbell ring on aarch64
>>> net/mlx5: relax barrier to order UAR writes on aarch64
>>> net/mlx5: relax barrier for aarch64
>>> net/mlx5: add descriptive comment for a barrier
>>> doc: clarify one configuration in mlx5 guide
>>>
>>> Phil Yang (1):
>>> net/mlx5: relax ordering for multi-packet RQ buffer refcnt
>>>
>>> doc/guides/nics/mlx5.rst | 6 ++--
>>> drivers/net/mlx5/mlx5_rxq.c | 2 +-
>>> drivers/net/mlx5/mlx5_rxtx.c | 16 ++++++-----
>>> drivers/net/mlx5/mlx5_rxtx.h | 14 ++++++----
>>> lib/librte_eal/arm/include/rte_atomic_32.h | 6 ++++
>>> lib/librte_eal/arm/include/rte_atomic_64.h | 6 ++++
>>> lib/librte_eal/include/generic/rte_atomic.h | 31 +++++++++++++++++++++
>>> lib/librte_eal/ppc/include/rte_atomic.h | 6 ++++
>>> lib/librte_eal/x86/include/rte_atomic.h | 6 ++++
>>> 9 files changed, 78 insertions(+), 15 deletions(-)
>>>
>
next prev parent reply other threads:[~2020-04-13 9:52 UTC|newest]
Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-02-13 12:38 [dpdk-dev] [PATCH RFC v1 0/6] barrier fix and optimization for mlx5 on aarch64 Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 1/6] net/mlx5: relax the barrier for UAR write Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 2/6] net/mlx5: use cio barrier before the BF WQE Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 3/6] net/mlx5: add missing barrier Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 4/6] net/mlx5: add descriptive comment for a barrier Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 5/6] net/mlx5: non-cacheable mapping defaulted for aarch64 Gavin Hu
2020-02-13 12:38 ` [dpdk-dev] [PATCH RFC v1 6/6] net/mlx5: relaxed ordering for multi-packet RQ buffer refcnt Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 0/7] introduce new barrier class and use it for mlx5 PMD Gavin Hu
2020-04-10 17:20 ` Andrew Rybchenko
2020-04-11 3:46 ` Gavin Hu
2020-04-13 9:51 ` Andrew Rybchenko [this message]
2020-04-13 16:46 ` Gavin Hu
2020-05-11 18:06 ` [dpdk-dev] [RFC] eal: adjust barriers for IO on Armv8-a Honnappa Nagarahalli
2020-05-12 6:18 ` Ruifeng Wang
2020-05-12 6:42 ` Jerin Jacob
2020-05-12 8:02 ` Ruifeng Wang
2020-05-12 8:28 ` Jerin Jacob
2020-05-12 21:44 ` Honnappa Nagarahalli
2020-05-13 14:49 ` Jerin Jacob
2020-05-14 1:02 ` Honnappa Nagarahalli
2020-06-27 19:12 ` [dpdk-dev] [PATCH v2] " Honnappa Nagarahalli
2020-06-27 19:25 ` Honnappa Nagarahalli
2020-06-30 5:13 ` Jerin Jacob
2020-07-03 18:57 ` [dpdk-dev] [PATCH v3 1/3] " Honnappa Nagarahalli
2020-07-03 18:57 ` [dpdk-dev] [PATCH v3 2/3] doc: update armv8-a IO barrier changes Honnappa Nagarahalli
2020-07-05 0:57 ` Jerin Jacob
2020-07-03 18:57 ` [dpdk-dev] [PATCH v3 3/3] doc: update deprecation of CIO barrier APIs Honnappa Nagarahalli
2020-07-05 0:57 ` Jerin Jacob
2020-07-07 20:19 ` Ajit Khaparde
2020-07-08 11:05 ` Ananyev, Konstantin
2020-07-06 23:43 ` [dpdk-dev] [PATCH v4 1/3] eal: adjust barriers for IO on Armv8-a Honnappa Nagarahalli
2020-07-06 23:43 ` [dpdk-dev] [PATCH v4 2/3] doc: update armv8-a IO barrier changes Honnappa Nagarahalli
2020-07-07 8:36 ` David Marchand
2020-07-07 18:37 ` Honnappa Nagarahalli
2020-07-06 23:43 ` [dpdk-dev] [PATCH v4 3/3] doc: update deprecation of CIO barrier APIs Honnappa Nagarahalli
2020-07-07 8:39 ` David Marchand
2020-07-07 20:14 ` David Christensen
2020-07-08 11:49 ` David Marchand
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 1/7] eal: introduce new class of barriers for DMA use cases Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 2/7] net/mlx5: dmb for immediate doorbell ring on aarch64 Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 3/7] net/mlx5: relax barrier to order UAR writes " Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 4/7] net/mlx5: relax barrier for aarch64 Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 5/7] net/mlx5: add descriptive comment for a barrier Gavin Hu
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 6/7] net/mlx5: relax ordering for multi-packet RQ buffer refcnt Gavin Hu
2020-06-23 8:26 ` [dpdk-dev] [PATCH v3] net/mlx5: relaxed " Phil Yang
2020-07-13 3:02 ` Phil Yang
2020-07-20 23:21 ` Alexander Kozyrev
2020-07-21 1:55 ` Phil Yang
2020-07-21 3:58 ` Alexander Kozyrev
2020-07-21 4:03 ` Honnappa Nagarahalli
2020-07-21 4:11 ` Alexander Kozyrev
2020-07-22 12:06 ` Phil Yang
2020-07-23 4:47 ` Honnappa Nagarahalli
2020-07-23 6:11 ` Phil Yang
2020-07-23 16:53 ` Alexander Kozyrev
2020-07-27 14:52 ` Phil Yang
2020-08-06 2:43 ` Alexander Kozyrev
2020-08-11 5:20 ` Honnappa Nagarahalli
2020-09-02 21:52 ` Alexander Kozyrev
2020-09-03 2:55 ` Phil Yang
2020-09-09 13:29 ` Alexander Kozyrev
2020-09-10 1:34 ` Honnappa Nagarahalli
2020-09-03 2:53 ` [dpdk-dev] [PATCH v4] " Phil Yang
2020-09-10 1:30 ` Honnappa Nagarahalli
2020-09-10 1:36 ` Alexander Kozyrev
2020-09-29 15:22 ` Phil Yang
2020-09-30 12:44 ` Slava Ovsiienko
2020-09-30 12:52 ` Raslan Darawsheh
2020-09-30 13:57 ` Raslan Darawsheh
2020-04-10 16:41 ` [dpdk-dev] [PATCH RFC v2 7/7] doc: clarify one configuration in mlx5 guide Gavin Hu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a83b6301-c564-f26c-6ff5-afda941a523a@solarflare.com \
--to=arybchenko@solarflare.com \
--cc=Gavin.Hu@arm.com \
--cc=Honnappa.Nagarahalli@arm.com \
--cc=Joyce.Kong@arm.com \
--cc=Phil.Yang@arm.com \
--cc=Ruifeng.Wang@arm.com \
--cc=Steve.Capper@arm.com \
--cc=bruce.richardson@intel.com \
--cc=david.marchand@redhat.com \
--cc=dev@dpdk.org \
--cc=drc@linux.vnet.ibm.com \
--cc=jerinj@marvell.com \
--cc=konstantin.ananyev@intel.com \
--cc=matan@mellanox.com \
--cc=nd@arm.com \
--cc=rasland@mellanox.com \
--cc=shahafs@mellanox.com \
--cc=thomas@monjalon.net \
--cc=viacheslavo@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).