From: "Morten Brørup" <mb@smartsharesystems.com>
To: "Maxime Coquelin" <maxime.coquelin@redhat.com>,
"Van Haaren, Harry" <harry.van.haaren@intel.com>,
"Pai G, Sunil" <sunil.pai.g@intel.com>,
"Stokes, Ian" <ian.stokes@intel.com>,
"Hu, Jiayu" <jiayu.hu@intel.com>,
"Ferriter, Cian" <cian.ferriter@intel.com>,
"Ilya Maximets" <i.maximets@ovn.org>, <ovs-dev@openvswitch.org>,
<dev@dpdk.org>
Cc: "Mcnamara, John" <john.mcnamara@intel.com>,
"O'Driscoll, Tim" <tim.odriscoll@intel.com>,
"Finn, Emma" <emma.finn@intel.com>
Subject: RE: OVS DPDK DMA-Dev library/Design Discussion
Date: Tue, 29 Mar 2022 18:45:19 +0200 [thread overview]
Message-ID: <98CBD80474FA8B44BF855DF32C47DC35D86F7E@smartserver.smartshare.dk> (raw)
In-Reply-To: <7968dd0b-8647-8d7b-786f-dc876bcbf3f0@redhat.com>
> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> Sent: Tuesday, 29 March 2022 18.24
>
> Hi Morten,
>
> On 3/29/22 16:44, Morten Brørup wrote:
> >> From: Van Haaren, Harry [mailto:harry.van.haaren@intel.com]
> >> Sent: Tuesday, 29 March 2022 15.02
> >>
> >>> From: Morten Brørup <mb@smartsharesystems.com>
> >>> Sent: Tuesday, March 29, 2022 1:51 PM
> >>>
> >>> Having thought more about it, I think that a completely different
> architectural approach is required:
> >>>
> >>> Many of the DPDK Ethernet PMDs implement a variety of RX and TX
> packet burst functions, each optimized for different CPU vector
> instruction sets. The availability of a DMA engine should be treated
> the same way. So I suggest that PMDs copying packet contents, e.g.
> memif, pcap, vmxnet3, should implement DMA optimized RX and TX packet
> burst functions.
> >>>
> >>> Similarly for the DPDK vhost library.
> >>>
> >>> In such an architecture, it would be the application's job to
> allocate DMA channels and assign them to the specific PMDs that should
> use them. But the actual use of the DMA channels would move down below
> the application and into the DPDK PMDs and libraries.
> >>>
> >>>
> >>> Med venlig hilsen / Kind regards,
> >>> -Morten Brørup
> >>
> >> Hi Morten,
> >>
> >> That's *exactly* how this architecture is designed & implemented.
> >> 1. The DMA configuration and initialization is up to the application
> (OVS).
> >> 2. The VHost library is passed the DMA-dev ID, and its new async
> rx/tx APIs, and uses the DMA device to accelerate the copy.
> >>
> >> Looking forward to talking on the call that just started. Regards, -
> Harry
> >>
> >
> > OK, thanks - as I said on the call, I haven't looked at the patches.
> >
> > Then, I suppose that the TX completions can be handled in the TX
> function, and the RX completions can be handled in the RX function,
> just like the Ethdev PMDs handle packet descriptors:
> >
> > TX_Burst(tx_packet_array):
> > 1. Clean up descriptors processed by the NIC chip. --> Process TX
> DMA channel completions. (Effectively, the 2nd pipeline stage.)
> > 2. Pass on the tx_packet_array to the NIC chip descriptors. --> Pass
> on the tx_packet_array to the TX DMA channel. (Effectively, the 1st
> pipeline stage.)
>
> The problem is Tx function might not be called again, so enqueued
> packets in 2. may never be completed from a Virtio point of view. IOW,
> the packets will be copied to the Virtio descriptors buffers, but the
> descriptors will not be made available to the Virtio driver.
In that case, the application needs to call TX_Burst() periodically with an empty array, for completion purposes.
Or some sort of TX_Keepalive() function can be added to the DPDK library, to handle DMA completion. It might even handle multiple DMA channels, if convenient - and if possible without locking or other weird complexity.
Here is another idea, inspired by a presentation at one of the DPDK Userspace conferences. It may be wishful thinking, though:
Add an additional transaction to each DMA burst; a special transaction containing the memory write operation that makes the descriptors available to the Virtio driver.
> >
> > RX_burst(rx_packet_array):
> > 1. Pass on the finished NIC chip RX descriptors to the
> rx_packet_array. --> Process RX DMA channel completions. (Effectively,
> the 2nd pipeline stage.)
> > 2. Replenish NIC chip RX descriptors. --> Start RX DMA channel for
> any waiting packets. (Effectively, the 1nd pipeline stage.)
> >
> > PMD_init():
> > - Prepare NIC chip RX descriptors. (In other words: Replenish NIC
> chip RX descriptors. = RX pipeline stage 1.)
> >
> > PS: Rearranged the email, so we can avoid top posting.
> >
>
next prev parent reply other threads:[~2022-03-29 16:45 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-24 15:36 Stokes, Ian
2022-03-28 18:19 ` Pai G, Sunil
2022-03-29 12:51 ` Morten Brørup
2022-03-29 13:01 ` Van Haaren, Harry
2022-03-29 14:44 ` Morten Brørup
2022-03-29 16:24 ` Maxime Coquelin
2022-03-29 16:45 ` Morten Brørup [this message]
2022-03-29 17:03 ` Bruce Richardson
2022-03-29 17:13 ` Morten Brørup
2022-03-29 17:45 ` Ilya Maximets
2022-03-29 18:46 ` Morten Brørup
2022-03-30 2:02 ` Hu, Jiayu
2022-03-30 9:25 ` Maxime Coquelin
2022-03-30 10:20 ` Bruce Richardson
2022-03-30 14:27 ` Hu, Jiayu
2022-03-29 17:46 ` Van Haaren, Harry
2022-03-29 19:59 ` Morten Brørup
2022-03-30 9:01 ` Van Haaren, Harry
2022-04-07 14:04 ` Van Haaren, Harry
2022-04-07 14:25 ` Maxime Coquelin
2022-04-07 14:39 ` Ilya Maximets
2022-04-07 14:42 ` Van Haaren, Harry
2022-04-07 15:01 ` Ilya Maximets
2022-04-07 15:46 ` Maxime Coquelin
2022-04-07 16:04 ` Bruce Richardson
2022-04-08 7:13 ` Hu, Jiayu
2022-04-08 8:21 ` Morten Brørup
2022-04-08 9:57 ` Ilya Maximets
2022-04-20 15:39 ` Mcnamara, John
2022-04-20 16:41 ` Mcnamara, John
2022-04-25 21:46 ` Ilya Maximets
2022-04-27 14:55 ` Mcnamara, John
2022-04-27 20:34 ` Bruce Richardson
2022-04-28 12:59 ` Ilya Maximets
2022-04-28 13:55 ` Bruce Richardson
2022-05-03 19:38 ` Van Haaren, Harry
2022-05-10 14:39 ` Van Haaren, Harry
2022-05-24 12:12 ` Ilya Maximets
2022-03-30 10:41 ` Ilya Maximets
2022-03-30 10:52 ` Ilya Maximets
2022-03-30 11:12 ` Bruce Richardson
2022-03-30 11:41 ` Ilya Maximets
2022-03-30 14:09 ` Bruce Richardson
2022-04-05 11:29 ` Ilya Maximets
2022-04-05 12:07 ` Bruce Richardson
2022-04-08 6:29 ` Pai G, Sunil
2022-05-13 8:52 ` fengchengwen
2022-05-13 9:10 ` Bruce Richardson
2022-05-13 9:48 ` fengchengwen
2022-05-13 10:34 ` Bruce Richardson
2022-05-16 9:04 ` Morten Brørup
2022-05-16 22:31 ` [EXT] " Radha Chintakuntla
-- strict thread matches above, loose matches on Subject: below --
2022-04-25 15:19 Mcnamara, John
2022-04-21 14:57 Mcnamara, John
[not found] <DM6PR11MB3227AC0014F321EB901BE385FC199@DM6PR11MB3227.namprd11.prod.outlook.com>
2022-04-21 11:51 ` Mcnamara, John
[not found] <DM8PR11MB5605B4A5DBD79FFDB4B1C3B2BD0A9@DM8PR11MB5605.namprd11.prod.outlook.com>
2022-03-21 18:23 ` Pai G, Sunil
2022-03-15 15:48 Stokes, Ian
2022-03-15 13:17 Stokes, Ian
2022-03-15 11:15 Stokes, Ian
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=98CBD80474FA8B44BF855DF32C47DC35D86F7E@smartserver.smartshare.dk \
--to=mb@smartsharesystems.com \
--cc=cian.ferriter@intel.com \
--cc=dev@dpdk.org \
--cc=emma.finn@intel.com \
--cc=harry.van.haaren@intel.com \
--cc=i.maximets@ovn.org \
--cc=ian.stokes@intel.com \
--cc=jiayu.hu@intel.com \
--cc=john.mcnamara@intel.com \
--cc=maxime.coquelin@redhat.com \
--cc=ovs-dev@openvswitch.org \
--cc=sunil.pai.g@intel.com \
--cc=tim.odriscoll@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).