From: Ilya Maximets <i.maximets@ovn.org>
To: Bruce Richardson <bruce.richardson@intel.com>
Cc: i.maximets@ovn.org, "Pai G, Sunil" <sunil.pai.g@intel.com>,
"Stokes, Ian" <ian.stokes@intel.com>,
"Hu, Jiayu" <jiayu.hu@intel.com>,
"Ferriter, Cian" <cian.ferriter@intel.com>,
"Van Haaren, Harry" <harry.van.haaren@intel.com>,
"Maxime Coquelin (maxime.coquelin@redhat.com)"
<maxime.coquelin@redhat.com>,
"ovs-dev@openvswitch.org" <ovs-dev@openvswitch.org>,
"dev@dpdk.org" <dev@dpdk.org>,
"Mcnamara, John" <john.mcnamara@intel.com>,
"O'Driscoll, Tim" <tim.odriscoll@intel.com>,
"Finn, Emma" <emma.finn@intel.com>
Subject: Re: OVS DPDK DMA-Dev library/Design Discussion
Date: Wed, 30 Mar 2022 13:41:34 +0200 [thread overview]
Message-ID: <ea799746-090c-c7a8-ed37-78f8f88e0b8c@ovn.org> (raw)
In-Reply-To: <YkQ7Mz6l0JIHW8Gh@bricha3-MOBL.ger.corp.intel.com>
On 3/30/22 13:12, Bruce Richardson wrote:
> On Wed, Mar 30, 2022 at 12:52:15PM +0200, Ilya Maximets wrote:
>> On 3/30/22 12:41, Ilya Maximets wrote:
>>> Forking the thread to discuss a memory consistency/ordering model.
>>>
>>> AFAICT, dmadev can be anything from part of a CPU to a completely
>>> separate PCI device. However, I don't see any memory ordering being
>>> enforced or even described in the dmadev API or documentation.
>>> Please, point me to the correct documentation, if I somehow missed it.
>>>
>>> We have a DMA device (A) and a CPU core (B) writing respectively
>>> the data and the descriptor info. CPU core (C) is reading the
>>> descriptor and the data it points too.
>>>
>>> A few things about that process:
>>>
>>> 1. There is no memory barrier between writes A and B (Did I miss
>>> them?). Meaning that those operations can be seen by C in a
>>> different order regardless of barriers issued by C and regardless
>>> of the nature of devices A and B.
>>>
>>> 2. Even if there is a write barrier between A and B, there is
>>> no guarantee that C will see these writes in the same order
>>> as C doesn't use real memory barriers because vhost advertises
>>
>> s/advertises/does not advertise/
>>
>>> VIRTIO_F_ORDER_PLATFORM.
>>>
>>> So, I'm getting to conclusion that there is a missing write barrier
>>> on the vhost side and vhost itself must not advertise the
>>
>> s/must not/must/
>>
>> Sorry, I wrote things backwards. :)
>>
>>> VIRTIO_F_ORDER_PLATFORM, so the virtio driver can use actual memory
>>> barriers.
>>>
>>> Would like to hear some thoughts on that topic. Is it a real issue?
>>> Is it an issue considering all possible CPU architectures and DMA
>>> HW variants?
>>>
>
> In terms of ordering of operations using dmadev:
>
> * Some DMA HW will perform all operations strictly in order e.g. Intel
> IOAT, while other hardware may not guarantee order of operations/do
> things in parallel e.g. Intel DSA. Therefore the dmadev API provides the
> fence operation which allows the order to be enforced. The fence can be
> thought of as a full memory barrier, meaning no jobs after the barrier can
> be started until all those before it have completed. Obviously, for HW
> where order is always enforced, this will be a no-op, but for hardware that
> parallelizes, we want to reduce the fences to get best performance.
>
> * For synchronization between DMA devices and CPUs, where a CPU can only
> write after a DMA copy has been done, the CPU must wait for the dma
> completion to guarantee ordering. Once the completion has been returned
> the completed operation is globally visible to all cores.
Thanks for explanation! Some questions though:
In our case one CPU waits for completion and another CPU is actually using
the data. IOW, "CPU must wait" is a bit ambiguous. Which CPU must wait?
Or should it be "Once the completion is visible on any core, the completed
operation is globally visible to all cores." ?
And the main question:
Are these synchronization claims documented somewhere?
Best regards, Ilya Maximets.
next prev parent reply other threads:[~2022-03-30 11:41 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-24 15:36 Stokes, Ian
2022-03-28 18:19 ` Pai G, Sunil
2022-03-29 12:51 ` Morten Brørup
2022-03-29 13:01 ` Van Haaren, Harry
2022-03-29 14:44 ` Morten Brørup
2022-03-29 16:24 ` Maxime Coquelin
2022-03-29 16:45 ` Morten Brørup
2022-03-29 17:03 ` Bruce Richardson
2022-03-29 17:13 ` Morten Brørup
2022-03-29 17:45 ` Ilya Maximets
2022-03-29 18:46 ` Morten Brørup
2022-03-30 2:02 ` Hu, Jiayu
2022-03-30 9:25 ` Maxime Coquelin
2022-03-30 10:20 ` Bruce Richardson
2022-03-30 14:27 ` Hu, Jiayu
2022-03-29 17:46 ` Van Haaren, Harry
2022-03-29 19:59 ` Morten Brørup
2022-03-30 9:01 ` Van Haaren, Harry
2022-04-07 14:04 ` Van Haaren, Harry
2022-04-07 14:25 ` Maxime Coquelin
2022-04-07 14:39 ` Ilya Maximets
2022-04-07 14:42 ` Van Haaren, Harry
2022-04-07 15:01 ` Ilya Maximets
2022-04-07 15:46 ` Maxime Coquelin
2022-04-07 16:04 ` Bruce Richardson
2022-04-08 7:13 ` Hu, Jiayu
2022-04-08 8:21 ` Morten Brørup
2022-04-08 9:57 ` Ilya Maximets
2022-04-20 15:39 ` Mcnamara, John
2022-04-20 16:41 ` Mcnamara, John
2022-04-25 21:46 ` Ilya Maximets
2022-04-27 14:55 ` Mcnamara, John
2022-04-27 20:34 ` Bruce Richardson
2022-04-28 12:59 ` Ilya Maximets
2022-04-28 13:55 ` Bruce Richardson
2022-05-03 19:38 ` Van Haaren, Harry
2022-05-10 14:39 ` Van Haaren, Harry
2022-05-24 12:12 ` Ilya Maximets
2022-03-30 10:41 ` Ilya Maximets
2022-03-30 10:52 ` Ilya Maximets
2022-03-30 11:12 ` Bruce Richardson
2022-03-30 11:41 ` Ilya Maximets [this message]
2022-03-30 14:09 ` Bruce Richardson
2022-04-05 11:29 ` Ilya Maximets
2022-04-05 12:07 ` Bruce Richardson
2022-04-08 6:29 ` Pai G, Sunil
2022-05-13 8:52 ` fengchengwen
2022-05-13 9:10 ` Bruce Richardson
2022-05-13 9:48 ` fengchengwen
2022-05-13 10:34 ` Bruce Richardson
2022-05-16 9:04 ` Morten Brørup
2022-05-16 22:31 ` [EXT] " Radha Chintakuntla
-- strict thread matches above, loose matches on Subject: below --
2022-04-25 15:19 Mcnamara, John
2022-04-21 14:57 Mcnamara, John
[not found] <DM6PR11MB3227AC0014F321EB901BE385FC199@DM6PR11MB3227.namprd11.prod.outlook.com>
2022-04-21 11:51 ` Mcnamara, John
[not found] <DM8PR11MB5605B4A5DBD79FFDB4B1C3B2BD0A9@DM8PR11MB5605.namprd11.prod.outlook.com>
2022-03-21 18:23 ` Pai G, Sunil
2022-03-15 15:48 Stokes, Ian
2022-03-15 13:17 Stokes, Ian
2022-03-15 11:15 Stokes, Ian
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ea799746-090c-c7a8-ed37-78f8f88e0b8c@ovn.org \
--to=i.maximets@ovn.org \
--cc=bruce.richardson@intel.com \
--cc=cian.ferriter@intel.com \
--cc=dev@dpdk.org \
--cc=emma.finn@intel.com \
--cc=harry.van.haaren@intel.com \
--cc=ian.stokes@intel.com \
--cc=jiayu.hu@intel.com \
--cc=john.mcnamara@intel.com \
--cc=maxime.coquelin@redhat.com \
--cc=ovs-dev@openvswitch.org \
--cc=sunil.pai.g@intel.com \
--cc=tim.odriscoll@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).