From: Andrew Rybchenko <arybchenko@solarflare.com>
To: Yongseok Koh <yskoh@mellanox.com>, Thomas Monjalon <thomas@monjalon.net>
Cc: "Jianbo Liu" <jianbo.liu@arm.com>, "dev@dpdk.org" <dev@dpdk.org>,
"Adrien Mazarguil" <adrien.mazarguil@6wind.com>,
"Nélio Laranjeiro" <nelio.laranjeiro@6wind.com>,
"jerin.jacob@caviumnetworks.com" <jerin.jacob@caviumnetworks.com>,
"konstantin.ananyev@intel.com" <konstantin.ananyev@intel.com>,
"bruce.richardson@intel.com" <bruce.richardson@intel.com>,
"Chao Zhu" <chaozhu@linux.vnet.ibm.com>
Subject: Re: [dpdk-dev] [PATCH v2 1/8] eal: introduce DMA memory barriers
Date: Thu, 18 Jan 2018 14:56:03 +0300 [thread overview]
Message-ID: <efbc451f-3416-1a98-b994-87f20848e351@solarflare.com> (raw)
In-Reply-To: <C08EC0A2-C39A-424D-9269-10FCEAE73BED@mellanox.com>
On 01/17/2018 09:39 PM, Yongseok Koh wrote:
>> On Jan 17, 2018, at 5:46 AM, Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 16/01/2018 10:10, Jianbo Liu:
>>> The 01/16/2018 10:49, Andrew Rybchenko wrote:
>>>> On 01/16/2018 04:10 AM, Yongseok Koh wrote:
>>>>> This commit introduces rte_dma_wmb() and rte_dma_rmb(), in order to
>>>>> guarantee the ordering of coherent shared memory between the CPU and a DMA
>>>>> capable device.
>>>>>
>>>>> Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
>>>>> ---
>>>>> lib/librte_eal/common/include/generic/rte_atomic.h | 18 ++++++++++++++++++
>>>>> 1 file changed, 18 insertions(+)
>>>>>
>>>>> diff --git a/lib/librte_eal/common/include/generic/rte_atomic.h b/lib/librte_eal/common/include/generic/rte_atomic.h
>>>>> index 16af5ca57..2e0503ce6 100644
>>>>> --- a/lib/librte_eal/common/include/generic/rte_atomic.h
>>>>> +++ b/lib/librte_eal/common/include/generic/rte_atomic.h
>>>>> @@ -98,6 +98,24 @@ static inline void rte_io_wmb(void);
>>>>> */
>>>>> static inline void rte_io_rmb(void);
>>>>> +/**
>>>>> + * Write memory barrier for coherent memory between lcore and IO device
>>>>> + *
>>>>> + * Guarantees that the STORE operations on coherent memory that
>>>>> + * precede the rte_dma_wmb() call are visible to I/O device before the
>>>>> + * STORE operations that follow it.
>>>>> + */
>>>>> +static inline void rte_dma_wmb(void);
>>>>> +
>>>>> +/**
>>>>> + * Read memory barrier for coherent memory between lcore and IO device
>>>>> + *
>>>>> + * Guarantees that the LOAD operations on coherent memory updated by
>>>>> + * IO device that precede the rte_dma_rmb() call are visible to CPU
>>>>> + * before the LOAD operations that follow it.
>>>>> + */
>>>>> +static inline void rte_dma_rmb(void);
>>>>> +
>>>>> #endif /* __DOXYGEN__ */
>>>>> /**
>>>> I'm not an ARMv8 expert so, my comments could be a bit ignorant.
>>>> I'd like to understand the difference between io and added here dma
>>>> barriers.
>>>> The difference should be clearly explained. Otherwise we'll constantly hit
>>>> on incorrect choice of barrier type.
>>>> Also I don't understand why "dma" name is chosen taking into account
>>>> that definition is bound to coherent memory between lcore and IO device.
>>> A good explanation can be found here.
>>>
>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.kernel.org%2Fpub%2Fscm%2Flinux%2Fkernel%2Fgit%2Ftorvalds%2Flinux.git%2Fcommit%2F%3Fid%3D1077fa36f23e259858caf6f269a47393a5aff523&data=02%7C01%7Cyskoh%40mellanox.com%7C7b526265cbf1449f3db208d55db0c55d%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636517936183877836&sdata=2%2Fi8Gs2n%2Fnbe9%2FJ3GWr22ndPmQVmvM2Xh12r3j1ZWlg%3D&reserved=0
>> I agree that something more is needed to explain when to use rte_io_*.
>> The only difference between rte_io_* and rte_dma_* is "on coherent memory".
> Okay will add more explanation and send out v3 soon. But, please note that
> there's no concrete theory when to use which barrier. Actually, it is mostly
> for ARMv8 because it provides more options for barriers. For other archs, as you
> can see in the patches, there's no difference from IO barriers.
Absence of concrete theory does not make choice of the memory barrier
easier.
I would say it complicates it significantly. I think it is a minimal
requirement for
the patchset to explain why a new type should be defined instead of just
fixing of the rte_io_* barriers on ARMv8. What's the different? Which
criteria
should be checked/taken into account to make the right choice?
As far as I can see igb_uio and uio_pci_generic do coherent DMA mapping.
It is not that easy with VFIO since in theory it could be non-coherent if
snooping is not supported by IOMMU. Don't know if it is real.
If so, it makes barrier choice UIO driver-dependent. Sounds bad.
next prev parent reply other threads:[~2018-01-18 11:56 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-27 4:28 [dpdk-dev] [PATCH 1/2] eal/arm64: modify I/O device " Yongseok Koh
2017-12-27 4:28 ` [dpdk-dev] [PATCH 2/2] net/mlx5: fix synchonization on polling Rx completions Yongseok Koh
2018-01-04 12:58 ` [dpdk-dev] [PATCH 1/2] eal/arm64: modify I/O device memory barriers Jerin Jacob
2018-01-08 1:55 ` Jianbo Liu
2018-01-16 0:42 ` Yongseok Koh
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 0/8] introduce DMA " Yongseok Koh
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 1/8] eal: " Yongseok Koh
2018-01-16 2:47 ` Jianbo Liu
2018-01-16 7:49 ` Andrew Rybchenko
2018-01-16 9:10 ` Jianbo Liu
2018-01-17 13:46 ` Thomas Monjalon
2018-01-17 18:39 ` Yongseok Koh
2018-01-18 11:56 ` Andrew Rybchenko [this message]
2018-01-18 18:14 ` Yongseok Koh
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 2/8] eal/x86: define " Yongseok Koh
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 3/8] eal/ppc64: define DMA device " Yongseok Koh
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 4/8] eal/armv7: define DMA " Yongseok Koh
2018-01-16 2:48 ` Jianbo Liu
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 5/8] eal/arm64: " Yongseok Koh
2018-01-16 2:50 ` Jianbo Liu
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 6/8] net/mlx5: remove unnecessary memory barrier Yongseok Koh
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 7/8] net/mlx5: replace IO memory barrier with DMA " Yongseok Koh
2018-01-16 1:10 ` [dpdk-dev] [PATCH v2 8/8] net/mlx5: fix synchonization on polling Rx completions Yongseok Koh
2018-01-16 3:53 ` Jianbo Liu
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 0/8] introduce DMA memory barriers Yongseok Koh
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 1/8] eal: " Yongseok Koh
2018-01-19 7:16 ` Andrew Rybchenko
2018-01-22 18:29 ` Yongseok Koh
2018-01-22 20:59 ` Thomas Monjalon
2018-01-23 4:35 ` Jerin Jacob
2018-01-25 19:08 ` Yongseok Koh
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 2/8] eal/x86: define " Yongseok Koh
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 3/8] eal/ppc64: " Yongseok Koh
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 4/8] eal/armv7: " Yongseok Koh
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 5/8] eal/arm64: " Yongseok Koh
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 6/8] net/mlx5: remove unnecessary memory barrier Yongseok Koh
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 7/8] net/mlx5: replace IO memory barrier with DMA " Yongseok Koh
2018-01-19 0:44 ` [dpdk-dev] [PATCH v3 8/8] net/mlx5: fix synchonization on polling Rx completions Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 0/9] introduce coherent I/O memory barriers Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 1/9] eal: add Doxygen grouping for " Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 2/9] eal: introduce coherent I/O " Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 3/9] eal/x86: define " Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 4/9] eal/ppc64: " Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 5/9] eal/armv7: " Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 6/9] eal/arm64: " Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 7/9] net/mlx5: remove unnecessary memory barrier Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 8/9] net/mlx5: replace I/O memory barrier with coherent version Yongseok Koh
2018-01-25 21:02 ` [dpdk-dev] [PATCH v4 9/9] net/mlx5: fix synchronization on polling Rx completions Yongseok Koh
2018-01-28 7:32 ` [dpdk-dev] [PATCH v4 0/9] introduce coherent I/O memory barriers Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=efbc451f-3416-1a98-b994-87f20848e351@solarflare.com \
--to=arybchenko@solarflare.com \
--cc=adrien.mazarguil@6wind.com \
--cc=bruce.richardson@intel.com \
--cc=chaozhu@linux.vnet.ibm.com \
--cc=dev@dpdk.org \
--cc=jerin.jacob@caviumnetworks.com \
--cc=jianbo.liu@arm.com \
--cc=konstantin.ananyev@intel.com \
--cc=nelio.laranjeiro@6wind.com \
--cc=thomas@monjalon.net \
--cc=yskoh@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).