[dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Fu, Patrick]

Background
====================================
DPDK vhost library implements a user-space VirtIO net backend allowing host applications to directly communicate with VirtIO front-end in VMs and containers. However, every vhost enqueue/dequeue operation requires to copy packet buffers between guest and host memory. The overhead of copying large bulk of data makes the vhost backend become the I/O bottleneck. DMA engines, including un-core DMA accelerator, like Crystal Beach DMA (CBDMA) and Data Streaming Accelerator (DSA), and discrete card general purpose DMA, are extremely efficient in data movement within system memory. Therefore, we propose a set of asynchronous DMA data movement API in vhost library for DMA acceleration. With offloading packet copies in vhost data-path from the CPU to the DMA engine, which can not only accelerate data transfers, but also save precious CPU core resources.

New API Overview
====================================
The proposed APIs in the vhost library support various DMA engines to accelerate data transfers in the data-path. For the higher performance, DMA engines work in an asynchronous manner, where DMA data transfers and CPU computations are executed in parallel. The proposed API consists of control path API and data path API. The control path API includes Registration API and DMA operation callback, and the data path API includes asynchronous API. To remove the dependency of vendor specific DMA engines, the DMA operation callback provides generic DMA data transfer abstractions. To support asynchronous DMA data movement, the new async API provides asynchronous ring operation semantic in data-path. To enable/disable DMA acceleration for virtqueues, users need to use registration API is to register/unregister DMA callback implementations to the vhost library and bind DMA channels to virtqueues. The DMA channels used by virtqueues are provided by DPDK applications, which is backed by  virtual or physical DMA devices.
The proposed APIs are consisted of 3 sub-sets:
1. DMA Registration APIs
2. DMA Operation Callbacks
3. Async Data APIs

DMA Registration APIs
==================================== 
DMA acceleration is per queue basis. DPDK applications need to explicitly decide whether a virtqueue needs DMA acceleration and which DMA channel to use. In addition, a DMA channel is dedicated to a virtqueue and a DMA channel cannot be bound to multiple virtqueues at the same time. To enable DMA acceleration for a virtqueue, DPDK applications need to implement DMA operation callbacks for a specific DMA type (e.g. CBDMA) first, then register the callbacks to the vhost library and bind a DMA channel to a virtqueue, and finally use the new async API to perform data-path operations on the virtqueue.
The definitions of registration API are shown below:
int rte_vhost_async_channel_register(int vid, uint16_t queue_id,
					struct rte_vdma_device_ops *ops);

int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id);

The "rte_vhost_async_channel_register" is to register implemented DMA operation callbacks to the vhost library and bind a DMA channel to a virtqueue. DPDK applications must implement the corresponding DMA operation callbacks for various DMA engines. To enable DMA acceleration for a virtqueue, DPDK applications need to explicitly call "rte_vhost_async_channel_register" for the virtqueue.  The "ops" points to the implementation of callbacks. 
The "rte_vhost_async_channel_unregister" unregisters DMA operation callbacks and unbind the DMA channel from the virtqueue. If a virtqueue does not bind to a DMA channel, it will use SW data-path without DMA acceleration.

DMA Operation Callbacks
==================================== 
The definitions of DMA operation callback are shown below:
struct iovec {	/** this is kernel uapi structure */
	void *iov_base;	/** buffer address */
	size_t iov_len;	/** buffer length */
};

struct iov_iter {	
	size_t iov_offset;
	size_t count;		/** total bytes of a packet */
	struct iovec *iov;	/** array of data buffers */
	unsigned long nr_segs;	/** number of iovec structures */
	uintptr_t usr_data;	/** app specific memory handler*/
};

struct dma_trans_desc {
	struct iov_iter *src; /** source memory iov_iter*/
	struct iov_iter *dst; /** destination memory iov_iter*/
};

struct dma_trans_status {
	uintptr_t src_usr_data; /** trans completed memory handler*/
	uintptr_t dst_usr_data; /** trans completed memory handler*/
};

struct rte_vhost_async_channel_ops {
	/** Instruct a DMA channel to perform copies for a batch of packets */
	int (*transfer_data)( struct dma_trans_desc *descs,
				 uint16_t count);

        	/** check copy-completed packets from a DMA channel */
	int (*check_completed_copies)( struct dma_trans_status *usr_data,
					uint16_t max_packets);
};

The first callback "transfer_data" is to submit a batch of packet copies to a DMA channel. As a packet's source or destination buffer can be a vector of buffers or a single data stream, we use "struct dma_trans_desc" to construct the source and destination buffer of packet.  Copying a packet is to move data from source iov_iter structure to destination iov_iter structure. The "count" is the number of packets to do copy. 
The second callback "check_completed_copies" queries the completion status of the DMA. An "usr_data" member variable is embedded in "iov_iter" structure, which serves as a unique identifier of the memory region described by "iov_iter". As the source/destination buffer can be scatter-gather, the DMA channel may perform its copies out-of-order. When all copies of an iov_iter are completed by the DMA channel, the "check_completed_copies" should return the associated "usr_data" by "dma_trans_status" structure. 

Async Data APIs
==================================== 
The definitions of new enqueue API are shown below:
uint16_t rte_vhost_submit_enqueue_burst(int vid, uint16_t queue_id, struct rte_mbuf **pkts, uint16_t count);

uint16_t rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id, struct rte_mbuf **pkts, uint16_t count);

The "rte_vhost_submit_enqueue_burst" is to enqueue a batch of packets to a virtqueue with giving ownership of enqueue packets to the vhost library. DPDK applications cannot reuse the enqueued packets until they get back the ownership. For a virtqueue enabled DMA acceleration by the "rte_vhost_async_channel_register", the "rte_vhost_submit_enqueue_burst" will use the bound DMA channel to perform packet copies; moreover, the function is non-blocking, which just submits packet copies to the DMA channel but without waiting for completion. For a virtqueue without enabling DMA acceleration, the "rte_vhost_submit_enqueue_burst" will use SW data-path, where the CPU performs packet copies. It worth noticing that DPDK applications cannot directly reuse enqueued packet buffers by "rte_vhost_submit_enqueue_burst", even if it uses SW data-path.

The "rte_vhost_poll_enqueue_completed" returns ownership for the packets whose copies are all completed currently, either by the DMA channel or the CPU. It is a non-blocking function, which will not wait for DMA copies completion. After getting back the ownership of packets enqueued by "rte_vhost_submit_enqueue_burst", DPDK applications can further process the packet buffers, e.g. free pktmbufs.

Sample Work Flow
==================================== 
Some DMA engines, like CBDMA, need to use physical addresses and do not support I/O page fault. In addition, some guests may want to avoid memory swapping out. For these cases, we can pin guest memory by setting a new flag "RTE_VHOST_USER_DMA_COPY" in rte_vhost_driver_register(). Here is an example of how to use CBDMA to accelerate vhost enqueue operation:
Step1: Implement DMA operation callbacks for CBDMA via IOAT PMD
Step2: call rte_vhost_driver_register with flag "RTE_VHOST_USER_DMA_COPY" (pin guest memory)
Step3: call rte_vhost_async_channel_register to register DMA channel
Step4: call rte_vhost_submit_enqueue_burst to enqueue packets
Step5: call rte_vhost_poll_enqueue_completed get back the ownership of the packets whose copies are completed
Step6: call rte_pktmbuf_free to free packet mbuf

Signed-off-by: Patrick Fu <patrick.fu@intel.com>
Signed-off-by: Jiayu Hu <jiayu.hu@intel.com> 

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Jerin Jacob]

On Fri, Apr 17, 2020 at 12:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
>
> Background
> ====================================
> DPDK vhost library implements a user-space VirtIO net backend allowing host applications to directly communicate with VirtIO front-end in VMs and containers. However, every vhost enqueue/dequeue operation requires to copy packet buffers between guest and host memory. The overhead of copying large bulk of data makes the vhost backend become the I/O bottleneck. DMA engines, including un-core DMA accelerator, like Crystal Beach DMA (CBDMA) and Data Streaming Accelerator (DSA), and discrete card general purpose DMA, are extremely efficient in data movement within system memory. Therefore, we propose a set of asynchronous DMA data movement API in vhost library for DMA acceleration. With offloading packet copies in vhost data-path from the CPU to the DMA engine, which can not only accelerate data transfers, but also save precious CPU core resources.
>
> New API Overview
> ====================================
> The proposed APIs in the vhost library support various DMA engines to accelerate data transfers in the data-path. For the higher performance, DMA engines work in an asynchronous manner, where DMA data transfers and CPU computations are executed in parallel. The proposed API consists of control path API and data path API. The control path API includes Registration API and DMA operation callback, and the data path API includes asynchronous API. To remove the dependency of vendor specific DMA engines, the DMA operation callback provides generic DMA data transfer abstractions. To support asynchronous DMA data movement, the new async API provides asynchronous ring operation semantic in data-path. To enable/disable DMA acceleration for virtqueues, users need to use registration API is to register/unregister DMA callback implementations to the vhost library and bind DMA channels to virtqueues. The DMA channels used by virtqueues are provided by DPDK applications, which is backed by  virtual or physical DMA devices.
> The proposed APIs are consisted of 3 sub-sets:
> 1. DMA Registration APIs
> 2. DMA Operation Callbacks
> 3. Async Data APIs
>
> DMA Registration APIs
> ====================================
> DMA acceleration is per queue basis. DPDK applications need to explicitly decide whether a virtqueue needs DMA acceleration and which DMA channel to use. In addition, a DMA channel is dedicated to a virtqueue and a DMA channel cannot be bound to multiple virtqueues at the same time. To enable DMA acceleration for a virtqueue, DPDK applications need to implement DMA operation callbacks for a specific DMA type (e.g. CBDMA) first, then register the callbacks to the vhost library and bind a DMA channel to a virtqueue, and finally use the new async API to perform data-path operations on the virtqueue.
> The definitions of registration API are shown below:
> int rte_vhost_async_channel_register(int vid, uint16_t queue_id,
>                                         struct rte_vdma_device_ops *ops);
>
> int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id);

We already have multiple DMA implementation over raw dev.
Why not make a new dmadev class for DMA acceleration and use it by
virtio and any other clients?

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Fu, Patrick]

Hi Jerin,

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Friday, April 17, 2020 4:02 PM
> To: Fu, Patrick <patrick.fu@intel.com>
> Cc: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>; Ye,
> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
> Zhihong <zhihong.wang@intel.com>; Liang, Cunming
> <cunming.liang@intel.com>
> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost
> with DMA Engines
> 
> On Fri, Apr 17, 2020 at 12:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
> >
> > Background
> > ====================================
> > DPDK vhost library implements a user-space VirtIO net backend allowing
> host applications to directly communicate with VirtIO front-end in VMs and
> containers. However, every vhost enqueue/dequeue operation requires to
> copy packet buffers between guest and host memory. The overhead of
> copying large bulk of data makes the vhost backend become the I/O
> bottleneck. DMA engines, including un-core DMA accelerator, like Crystal
> Beach DMA (CBDMA) and Data Streaming Accelerator (DSA), and discrete
> card general purpose DMA, are extremely efficient in data movement within
> system memory. Therefore, we propose a set of asynchronous DMA data
> movement API in vhost library for DMA acceleration. With offloading packet
> copies in vhost data-path from the CPU to the DMA engine, which can not
> only accelerate data transfers, but also save precious CPU core resources.
> >
> > New API Overview
> > ====================================
> > The proposed APIs in the vhost library support various DMA engines to
> accelerate data transfers in the data-path. For the higher performance, DMA
> engines work in an asynchronous manner, where DMA data transfers and
> CPU computations are executed in parallel. The proposed API consists of
> control path API and data path API. The control path API includes
> Registration API and DMA operation callback, and the data path API includes
> asynchronous API. To remove the dependency of vendor specific DMA
> engines, the DMA operation callback provides generic DMA data transfer
> abstractions. To support asynchronous DMA data movement, the new async
> API provides asynchronous ring operation semantic in data-path. To
> enable/disable DMA acceleration for virtqueues, users need to use
> registration API is to register/unregister DMA callback implementations to
> the vhost library and bind DMA channels to virtqueues. The DMA channels
> used by virtqueues are provided by DPDK applications, which is backed by
> virtual or physical DMA devices.
> > The proposed APIs are consisted of 3 sub-sets:
> > 1. DMA Registration APIs
> > 2. DMA Operation Callbacks
> > 3. Async Data APIs
> >
> > DMA Registration APIs
> > ====================================
> > DMA acceleration is per queue basis. DPDK applications need to explicitly
> decide whether a virtqueue needs DMA acceleration and which DMA channel
> to use. In addition, a DMA channel is dedicated to a virtqueue and a DMA
> channel cannot be bound to multiple virtqueues at the same time. To enable
> DMA acceleration for a virtqueue, DPDK applications need to implement
> DMA operation callbacks for a specific DMA type (e.g. CBDMA) first, then
> register the callbacks to the vhost library and bind a DMA channel to a
> virtqueue, and finally use the new async API to perform data-path operations
> on the virtqueue.
> > The definitions of registration API are shown below:
> > int rte_vhost_async_channel_register(int vid, uint16_t queue_id,
> >                                         struct rte_vdma_device_ops
> > *ops);
> >
> > int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id);
> 
> We already have multiple DMA implementation over raw dev.
> Why not make a new dmadev class for DMA acceleration and use it by virtio
> and any other clients?

I believe it doesn't conflict. The purpose of this RFC is to create an async data path in vhost-user and provide a way for applications to work with this new path. dmadev is another topic which could be discussed separately. If we do have the dmadev available in the future, this vhost async data path could certainly be backed by the new dma abstraction without major interface change.

Thanks,

Patrick

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Maxime Coquelin]

On 4/17/20 10:29 AM, Fu, Patrick wrote:
> Hi Jerin,
> 
>> -----Original Message-----
>> From: Jerin Jacob <jerinjacobk@gmail.com>
>> Sent: Friday, April 17, 2020 4:02 PM
>> To: Fu, Patrick <patrick.fu@intel.com>
>> Cc: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>; Ye,
>> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
>> Zhihong <zhihong.wang@intel.com>; Liang, Cunming
>> <cunming.liang@intel.com>
>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost
>> with DMA Engines
>>
>> On Fri, Apr 17, 2020 at 12:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
>>>
>>> Background
>>> ====================================
>>> DPDK vhost library implements a user-space VirtIO net backend allowing
>> host applications to directly communicate with VirtIO front-end in VMs and
>> containers. However, every vhost enqueue/dequeue operation requires to
>> copy packet buffers between guest and host memory. The overhead of
>> copying large bulk of data makes the vhost backend become the I/O
>> bottleneck. DMA engines, including un-core DMA accelerator, like Crystal
>> Beach DMA (CBDMA) and Data Streaming Accelerator (DSA), and discrete
>> card general purpose DMA, are extremely efficient in data movement within
>> system memory. Therefore, we propose a set of asynchronous DMA data
>> movement API in vhost library for DMA acceleration. With offloading packet
>> copies in vhost data-path from the CPU to the DMA engine, which can not
>> only accelerate data transfers, but also save precious CPU core resources.
>>>
>>> New API Overview
>>> ====================================
>>> The proposed APIs in the vhost library support various DMA engines to
>> accelerate data transfers in the data-path. For the higher performance, DMA
>> engines work in an asynchronous manner, where DMA data transfers and
>> CPU computations are executed in parallel. The proposed API consists of
>> control path API and data path API. The control path API includes
>> Registration API and DMA operation callback, and the data path API includes
>> asynchronous API. To remove the dependency of vendor specific DMA
>> engines, the DMA operation callback provides generic DMA data transfer
>> abstractions. To support asynchronous DMA data movement, the new async
>> API provides asynchronous ring operation semantic in data-path. To
>> enable/disable DMA acceleration for virtqueues, users need to use
>> registration API is to register/unregister DMA callback implementations to
>> the vhost library and bind DMA channels to virtqueues. The DMA channels
>> used by virtqueues are provided by DPDK applications, which is backed by
>> virtual or physical DMA devices.
>>> The proposed APIs are consisted of 3 sub-sets:
>>> 1. DMA Registration APIs
>>> 2. DMA Operation Callbacks
>>> 3. Async Data APIs
>>>
>>> DMA Registration APIs
>>> ====================================
>>> DMA acceleration is per queue basis. DPDK applications need to explicitly
>> decide whether a virtqueue needs DMA acceleration and which DMA channel
>> to use. In addition, a DMA channel is dedicated to a virtqueue and a DMA
>> channel cannot be bound to multiple virtqueues at the same time. To enable
>> DMA acceleration for a virtqueue, DPDK applications need to implement
>> DMA operation callbacks for a specific DMA type (e.g. CBDMA) first, then
>> register the callbacks to the vhost library and bind a DMA channel to a
>> virtqueue, and finally use the new async API to perform data-path operations
>> on the virtqueue.
>>> The definitions of registration API are shown below:
>>> int rte_vhost_async_channel_register(int vid, uint16_t queue_id,
>>>                                         struct rte_vdma_device_ops
>>> *ops);
>>>
>>> int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id);
>>
>> We already have multiple DMA implementation over raw dev.
>> Why not make a new dmadev class for DMA acceleration and use it by virtio
>> and any other clients?
> 
> I believe it doesn't conflict. The purpose of this RFC is to create an async data path in vhost-user and provide a way for applications to work with this new path. dmadev is another topic which could be discussed separately. If we do have the dmadev available in the future, this vhost async data path could certainly be backed by the new dma abstraction without major interface change.

Maybe that one advantage of a dmadev class is that it would be easier
and more transparent for the application to consume.

The application would register some DMA devices, pass them to the Vhost
library, and then rte_vhost_submit_enqueue_burst and
rte_vhost_poll_enqueue_completed would call the dmadev callbacks
directly.

Do you think that could work?

Thanks,
Maxime

> Thanks,
> 
> Patrick
> 

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Fu, Patrick]

> -----Original Message-----
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> Sent: Friday, April 17, 2020 4:40 PM
> To: Fu, Patrick <patrick.fu@intel.com>; Jerin Jacob <jerinjacobk@gmail.com>
> Cc: dev@dpdk.org; Ye, Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu
> <jiayu.hu@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>; Liang,
> Cunming <cunming.liang@intel.com>
> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost
> with DMA Engines
> 
> 
> 
> On 4/17/20 10:29 AM, Fu, Patrick wrote:
> > Hi Jerin,
> >
> >> -----Original Message-----
> >> From: Jerin Jacob <jerinjacobk@gmail.com>
> >> Sent: Friday, April 17, 2020 4:02 PM
> >> To: Fu, Patrick <patrick.fu@intel.com>
> >> Cc: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>;
> Ye,
> >> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>;
> >> Wang, Zhihong <zhihong.wang@intel.com>; Liang, Cunming
> >> <cunming.liang@intel.com>
> >> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK
> >> vHost with DMA Engines
> >>
> >> On Fri, Apr 17, 2020 at 12:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
> >>>
> >>> Background
> >>> ====================================
> >>> DPDK vhost library implements a user-space VirtIO net backend
> >>> allowing
> >> host applications to directly communicate with VirtIO front-end in
> >> VMs and containers. However, every vhost enqueue/dequeue operation
> >> requires to copy packet buffers between guest and host memory. The
> >> overhead of copying large bulk of data makes the vhost backend become
> >> the I/O bottleneck. DMA engines, including un-core DMA accelerator,
> >> like Crystal Beach DMA (CBDMA) and Data Streaming Accelerator (DSA),
> >> and discrete card general purpose DMA, are extremely efficient in
> >> data movement within system memory. Therefore, we propose a set of
> >> asynchronous DMA data movement API in vhost library for DMA
> >> acceleration. With offloading packet copies in vhost data-path from
> >> the CPU to the DMA engine, which can not only accelerate data transfers,
> but also save precious CPU core resources.
> >>>
> >>> New API Overview
> >>> ====================================
> >>> The proposed APIs in the vhost library support various DMA engines
> >>> to
> >> accelerate data transfers in the data-path. For the higher
> >> performance, DMA engines work in an asynchronous manner, where
> DMA
> >> data transfers and CPU computations are executed in parallel. The
> >> proposed API consists of control path API and data path API. The
> >> control path API includes Registration API and DMA operation
> >> callback, and the data path API includes asynchronous API. To remove
> >> the dependency of vendor specific DMA engines, the DMA operation
> >> callback provides generic DMA data transfer abstractions. To support
> >> asynchronous DMA data movement, the new async API provides
> >> asynchronous ring operation semantic in data-path. To enable/disable
> >> DMA acceleration for virtqueues, users need to use registration API
> >> is to register/unregister DMA callback implementations to the vhost
> >> library and bind DMA channels to virtqueues. The DMA channels used by
> >> virtqueues are provided by DPDK applications, which is backed by virtual
> or physical DMA devices.
> >>> The proposed APIs are consisted of 3 sub-sets:
> >>> 1. DMA Registration APIs
> >>> 2. DMA Operation Callbacks
> >>> 3. Async Data APIs
> >>>
> >>> DMA Registration APIs
> >>> ====================================
> >>> DMA acceleration is per queue basis. DPDK applications need to
> >>> explicitly
> >> decide whether a virtqueue needs DMA acceleration and which DMA
> >> channel to use. In addition, a DMA channel is dedicated to a
> >> virtqueue and a DMA channel cannot be bound to multiple virtqueues at
> >> the same time. To enable DMA acceleration for a virtqueue, DPDK
> >> applications need to implement DMA operation callbacks for a specific
> >> DMA type (e.g. CBDMA) first, then register the callbacks to the vhost
> >> library and bind a DMA channel to a virtqueue, and finally use the
> >> new async API to perform data-path operations on the virtqueue.
> >>> The definitions of registration API are shown below:
> >>> int rte_vhost_async_channel_register(int vid, uint16_t queue_id,
> >>>                                         struct rte_vdma_device_ops
> >>> *ops);
> >>>
> >>> int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id);
> >>
> >> We already have multiple DMA implementation over raw dev.
> >> Why not make a new dmadev class for DMA acceleration and use it by
> >> virtio and any other clients?
> >
> > I believe it doesn't conflict. The purpose of this RFC is to create an async
> data path in vhost-user and provide a way for applications to work with this
> new path. dmadev is another topic which could be discussed separately. If
> we do have the dmadev available in the future, this vhost async data path
> could certainly be backed by the new dma abstraction without major
> interface change.
> 
> Maybe that one advantage of a dmadev class is that it would be easier and
> more transparent for the application to consume.
> 
> The application would register some DMA devices, pass them to the Vhost
> library, and then rte_vhost_submit_enqueue_burst and
> rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
> 
> Do you think that could work?
> 
Yes, this is a workable model. As I said in previous reply, I have no objection to make the dmadev. However, what we currently want to do is creating the async data path for vhost, and we actually have no preference to the underlying DMA device model. I believe our current design of the API proto type /data structures are quite common for various DMA acceleration solutions and there is no blocker for any new DMA device to adapt to these APIs or extend to a new one. 

Thanks,

Patrick

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Jerin Jacob]

On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Maxime Coquelin <maxime.coquelin@redhat.com>
> > Sent: Friday, April 17, 2020 4:40 PM
> > To: Fu, Patrick <patrick.fu@intel.com>; Jerin Jacob <jerinjacobk@gmail.com>
> > Cc: dev@dpdk.org; Ye, Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu
> > <jiayu.hu@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>; Liang,
> > Cunming <cunming.liang@intel.com>
> > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost
> > with DMA Engines
> >
> >
> >
> > On 4/17/20 10:29 AM, Fu, Patrick wrote:
> > > Hi Jerin,
> > >
> > >> -----Original Message-----
> > >> From: Jerin Jacob <jerinjacobk@gmail.com>
> > >> Sent: Friday, April 17, 2020 4:02 PM
> > >> To: Fu, Patrick <patrick.fu@intel.com>
> > >> Cc: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>;
> > Ye,
> > >> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>;
> > >> Wang, Zhihong <zhihong.wang@intel.com>; Liang, Cunming
> > >> <cunming.liang@intel.com>
> > >> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK
> > >> vHost with DMA Engines
> > >>
> > >> On Fri, Apr 17, 2020 at 12:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
> > >>>
> > >>> Background
> > >>> ====================================
> > >>> DPDK vhost library implements a user-space VirtIO net backend
> > >>> allowing
> > >> host applications to directly communicate with VirtIO front-end in
> > >> VMs and containers. However, every vhost enqueue/dequeue operation
> > >> requires to copy packet buffers between guest and host memory. The
> > >> overhead of copying large bulk of data makes the vhost backend become
> > >> the I/O bottleneck. DMA engines, including un-core DMA accelerator,
> > >> like Crystal Beach DMA (CBDMA) and Data Streaming Accelerator (DSA),
> > >> and discrete card general purpose DMA, are extremely efficient in
> > >> data movement within system memory. Therefore, we propose a set of
> > >> asynchronous DMA data movement API in vhost library for DMA
> > >> acceleration. With offloading packet copies in vhost data-path from
> > >> the CPU to the DMA engine, which can not only accelerate data transfers,
> > but also save precious CPU core resources.
> > >>>
> > >>> New API Overview
> > >>> ====================================
> > >>> The proposed APIs in the vhost library support various DMA engines
> > >>> to
> > >> accelerate data transfers in the data-path. For the higher
> > >> performance, DMA engines work in an asynchronous manner, where
> > DMA
> > >> data transfers and CPU computations are executed in parallel. The
> > >> proposed API consists of control path API and data path API. The
> > >> control path API includes Registration API and DMA operation
> > >> callback, and the data path API includes asynchronous API. To remove
> > >> the dependency of vendor specific DMA engines, the DMA operation
> > >> callback provides generic DMA data transfer abstractions. To support
> > >> asynchronous DMA data movement, the new async API provides
> > >> asynchronous ring operation semantic in data-path. To enable/disable
> > >> DMA acceleration for virtqueues, users need to use registration API
> > >> is to register/unregister DMA callback implementations to the vhost
> > >> library and bind DMA channels to virtqueues. The DMA channels used by
> > >> virtqueues are provided by DPDK applications, which is backed by virtual
> > or physical DMA devices.
> > >>> The proposed APIs are consisted of 3 sub-sets:
> > >>> 1. DMA Registration APIs
> > >>> 2. DMA Operation Callbacks
> > >>> 3. Async Data APIs
> > >>>
> > >>> DMA Registration APIs
> > >>> ====================================
> > >>> DMA acceleration is per queue basis. DPDK applications need to
> > >>> explicitly
> > >> decide whether a virtqueue needs DMA acceleration and which DMA
> > >> channel to use. In addition, a DMA channel is dedicated to a
> > >> virtqueue and a DMA channel cannot be bound to multiple virtqueues at
> > >> the same time. To enable DMA acceleration for a virtqueue, DPDK
> > >> applications need to implement DMA operation callbacks for a specific
> > >> DMA type (e.g. CBDMA) first, then register the callbacks to the vhost
> > >> library and bind a DMA channel to a virtqueue, and finally use the
> > >> new async API to perform data-path operations on the virtqueue.
> > >>> The definitions of registration API are shown below:
> > >>> int rte_vhost_async_channel_register(int vid, uint16_t queue_id,
> > >>>                                         struct rte_vdma_device_ops
> > >>> *ops);
> > >>>
> > >>> int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id);
> > >>
> > >> We already have multiple DMA implementation over raw dev.
> > >> Why not make a new dmadev class for DMA acceleration and use it by
> > >> virtio and any other clients?
> > >
> > > I believe it doesn't conflict. The purpose of this RFC is to create an async
> > data path in vhost-user and provide a way for applications to work with this
> > new path. dmadev is another topic which could be discussed separately. If
> > we do have the dmadev available in the future, this vhost async data path
> > could certainly be backed by the new dma abstraction without major
> > interface change.
> >
> > Maybe that one advantage of a dmadev class is that it would be easier and
> > more transparent for the application to consume.
> >
> > The application would register some DMA devices, pass them to the Vhost
> > library, and then rte_vhost_submit_enqueue_burst and
> > rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
> >
> > Do you think that could work?
> >
> Yes, this is a workable model. As I said in previous reply, I have no objection to make the dmadev. However, what we currently want to do is creating the async data path for vhost, and we actually have no preference to the underlying DMA device model. I believe our current design of the API proto type /data structures are quite common for various DMA acceleration solutions and there is no blocker for any new DMA device to adapt to these APIs or extend to a new one.

IMO, as a driver writer,  we should not be writing TWO DMA driver. One
for vhost and other one for rawdev.
If vhost is the first consumer of DMA needed then I think, it make
sense to add dmadev first.
The rawdev DMA driver to dmadev DMA driver conversion will be the
driver owner job.
I think, it makes sense to define the dmadev API and then costume by
virtio to avoid integration issues.



>
> Thanks,
>
> Patrick
>

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Liang, Cunming]

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Friday, April 17, 2020 5:55 PM
> To: Fu, Patrick <patrick.fu@intel.com>
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; dev@dpdk.org; Ye,
> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
> Zhihong <zhihong.wang@intel.com>; Liang, Cunming <cunming.liang@intel.com>
> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
> DMA Engines
> 
> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
> >
> >
[...]
> > > >
> > > > I believe it doesn't conflict. The purpose of this RFC is to
> > > > create an async
> > > data path in vhost-user and provide a way for applications to work
> > > with this new path. dmadev is another topic which could be discussed
> > > separately. If we do have the dmadev available in the future, this
> > > vhost async data path could certainly be backed by the new dma
> > > abstraction without major interface change.
> > >
> > > Maybe that one advantage of a dmadev class is that it would be
> > > easier and more transparent for the application to consume.
> > >
> > > The application would register some DMA devices, pass them to the
> > > Vhost library, and then rte_vhost_submit_enqueue_burst and
> > > rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
> > >
> > > Do you think that could work?
> > >
> > Yes, this is a workable model. As I said in previous reply, I have no objection to
> make the dmadev. However, what we currently want to do is creating the async
> data path for vhost, and we actually have no preference to the underlying DMA
> device model. I believe our current design of the API proto type /data structures
> are quite common for various DMA acceleration solutions and there is no blocker
> for any new DMA device to adapt to these APIs or extend to a new one.
> 
> IMO, as a driver writer,  we should not be writing TWO DMA driver. One for vhost
> and other one for rawdev.
It's the most simplest case if statically 1:1 mapping driver (e.g. {port, queue}) to a vhost session {vid, qid}. However, it's not enough scalable to integrate device model with vhost library. There're a few intentions belong to app logic rather than driver, e.g. 1:N load balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.

It was not asking to writing two drivers. Each driver remains to offer provider for its own device class, which is independent. App provides the intension (adapter) to associate various device capability to vhost session.

> If vhost is the first consumer of DMA needed then I think, it make sense to add
> dmadev first.
On the other hand, it's risky to define 'dmadev' according to vhost's flavor before not getting aware of any other candidates. Comparing with kern Async TX DMA API (async_memcpy), RFC is very much focus on S/G buffer but not a async_memcpy.

> The rawdev DMA driver to dmadev DMA driver conversion will be the driver owner
> job.
It's true when it's necessary. Even that is the case, it's better for vhost to be independent with any device model, moreover vhost usage doesn't have broad enough coverage for a new device class.

> I think, it makes sense to define the dmadev API and then costume by virtio to
> avoid integration issues.
Vhost is a library but not a app. We'd better avoid to intro either overkill integration logic or extra device model dependence.

Thanks,
Steve

> 
> 
> 
> >
> > Thanks,
> >
> > Patrick
> >

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Jerin Jacob]

On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming <cunming.liang@intel.com> wrote:
>
>
>
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk@gmail.com>
> > Sent: Friday, April 17, 2020 5:55 PM
> > To: Fu, Patrick <patrick.fu@intel.com>
> > Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; dev@dpdk.org; Ye,
> > Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
> > Zhihong <zhihong.wang@intel.com>; Liang, Cunming <cunming.liang@intel.com>
> > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
> > DMA Engines
> >
> > On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
> > >
> > >
> [...]
> > > > >
> > > > > I believe it doesn't conflict. The purpose of this RFC is to
> > > > > create an async
> > > > data path in vhost-user and provide a way for applications to work
> > > > with this new path. dmadev is another topic which could be discussed
> > > > separately. If we do have the dmadev available in the future, this
> > > > vhost async data path could certainly be backed by the new dma
> > > > abstraction without major interface change.
> > > >
> > > > Maybe that one advantage of a dmadev class is that it would be
> > > > easier and more transparent for the application to consume.
> > > >
> > > > The application would register some DMA devices, pass them to the
> > > > Vhost library, and then rte_vhost_submit_enqueue_burst and
> > > > rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
> > > >
> > > > Do you think that could work?
> > > >
> > > Yes, this is a workable model. As I said in previous reply, I have no objection to
> > make the dmadev. However, what we currently want to do is creating the async
> > data path for vhost, and we actually have no preference to the underlying DMA
> > device model. I believe our current design of the API proto type /data structures
> > are quite common for various DMA acceleration solutions and there is no blocker
> > for any new DMA device to adapt to these APIs or extend to a new one.
> >
> > IMO, as a driver writer,  we should not be writing TWO DMA driver. One for vhost
> > and other one for rawdev.
> It's the most simplest case if statically 1:1 mapping driver (e.g. {port, queue}) to a vhost session {vid, qid}. However, it's not enough scalable to integrate device model with vhost library. There're a few intentions belong to app logic rather than driver, e.g. 1:N load balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.


Before moving to reply to comments, Which DMA engine you are planning
to integrate with vHOST?
Is is ioat? if not ioat(drivers/raw/ioat/), How do you think, how we
can integrate this IOAT DMA engine to vHOST as a use case?



>
> It was not asking to writing two drivers. Each driver remains to offer provider for its own device class, which is independent. App provides the intension (adapter) to associate various device capability to vhost session.
>
> > If vhost is the first consumer of DMA needed then I think, it make sense to add
> > dmadev first.
> On the other hand, it's risky to define 'dmadev' according to vhost's flavor before not getting aware of any other candidates. Comparing with kern Async TX DMA API (async_memcpy), RFC is very much focus on S/G buffer but not a async_memcpy.
>
> > The rawdev DMA driver to dmadev DMA driver conversion will be the driver owner
> > job.
> It's true when it's necessary. Even that is the case, it's better for vhost to be independent with any device model, moreover vhost usage doesn't have broad enough coverage for a new device class.
>
> > I think, it makes sense to define the dmadev API and then costume by virtio to
> > avoid integration issues.
> Vhost is a library but not a app. We'd better avoid to intro either overkill integration logic or extra device model dependence.
>
> Thanks,
> Steve
>
> >
> >
> >
> > >
> > > Thanks,
> > >
> > > Patrick
> > >

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Maxime Coquelin]

On 4/20/20 1:13 PM, Jerin Jacob wrote:
> On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming <cunming.liang@intel.com> wrote:
>>
>>
>>
>>> -----Original Message-----
>>> From: Jerin Jacob <jerinjacobk@gmail.com>
>>> Sent: Friday, April 17, 2020 5:55 PM
>>> To: Fu, Patrick <patrick.fu@intel.com>
>>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; dev@dpdk.org; Ye,
>>> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
>>> Zhihong <zhihong.wang@intel.com>; Liang, Cunming <cunming.liang@intel.com>
>>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
>>> DMA Engines
>>>
>>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
>>>>
>>>>
>> [...]
>>>>>>
>>>>>> I believe it doesn't conflict. The purpose of this RFC is to
>>>>>> create an async
>>>>> data path in vhost-user and provide a way for applications to work
>>>>> with this new path. dmadev is another topic which could be discussed
>>>>> separately. If we do have the dmadev available in the future, this
>>>>> vhost async data path could certainly be backed by the new dma
>>>>> abstraction without major interface change.
>>>>>
>>>>> Maybe that one advantage of a dmadev class is that it would be
>>>>> easier and more transparent for the application to consume.
>>>>>
>>>>> The application would register some DMA devices, pass them to the
>>>>> Vhost library, and then rte_vhost_submit_enqueue_burst and
>>>>> rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
>>>>>
>>>>> Do you think that could work?
>>>>>
>>>> Yes, this is a workable model. As I said in previous reply, I have no objection to
>>> make the dmadev. However, what we currently want to do is creating the async
>>> data path for vhost, and we actually have no preference to the underlying DMA
>>> device model. I believe our current design of the API proto type /data structures
>>> are quite common for various DMA acceleration solutions and there is no blocker
>>> for any new DMA device to adapt to these APIs or extend to a new one.
>>>
>>> IMO, as a driver writer,  we should not be writing TWO DMA driver. One for vhost
>>> and other one for rawdev.
>> It's the most simplest case if statically 1:1 mapping driver (e.g. {port, queue}) to a vhost session {vid, qid}. However, it's not enough scalable to integrate device model with vhost library. There're a few intentions belong to app logic rather than driver, e.g. 1:N load balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.
> 
> 
> Before moving to reply to comments, Which DMA engine you are planning
> to integrate with vHOST?
> Is is ioat? if not ioat(drivers/raw/ioat/), How do you think, how we
> can integrate this IOAT DMA engine to vHOST as a use case?
> 

I guess it could be done in the vhost example.


> 
>>
>> It was not asking to writing two drivers. Each driver remains to offer provider for its own device class, which is independent. App provides the intension (adapter) to associate various device capability to vhost session.
>>
>>> If vhost is the first consumer of DMA needed then I think, it make sense to add
>>> dmadev first.
>> On the other hand, it's risky to define 'dmadev' according to vhost's flavor before not getting aware of any other candidates. Comparing with kern Async TX DMA API (async_memcpy), RFC is very much focus on S/G buffer but not a async_memcpy.
>>
>>> The rawdev DMA driver to dmadev DMA driver conversion will be the driver owner
>>> job.
>> It's true when it's necessary. Even that is the case, it's better for vhost to be independent with any device model, moreover vhost usage doesn't have broad enough coverage for a new device class.
>>
>>> I think, it makes sense to define the dmadev API and then costume by virtio to
>>> avoid integration issues.
>> Vhost is a library but not a app. We'd better avoid to intro either overkill integration logic or extra device model dependence.
>>
>> Thanks,
>> Steve
>>
>>>
>>>
>>>
>>>>
>>>> Thanks,
>>>>
>>>> Patrick
>>>>
> 

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Maxime Coquelin]

On 4/20/20 9:59 AM, Liang, Cunming wrote:
> 
> 
>> -----Original Message-----
>> From: Jerin Jacob <jerinjacobk@gmail.com>
>> Sent: Friday, April 17, 2020 5:55 PM
>> To: Fu, Patrick <patrick.fu@intel.com>
>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; dev@dpdk.org; Ye,
>> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
>> Zhihong <zhihong.wang@intel.com>; Liang, Cunming <cunming.liang@intel.com>
>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
>> DMA Engines
>>
>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
>>>
>>>
> [...]
>>>>>
>>>>> I believe it doesn't conflict. The purpose of this RFC is to
>>>>> create an async
>>>> data path in vhost-user and provide a way for applications to work
>>>> with this new path. dmadev is another topic which could be discussed
>>>> separately. If we do have the dmadev available in the future, this
>>>> vhost async data path could certainly be backed by the new dma
>>>> abstraction without major interface change.
>>>>
>>>> Maybe that one advantage of a dmadev class is that it would be
>>>> easier and more transparent for the application to consume.
>>>>
>>>> The application would register some DMA devices, pass them to the
>>>> Vhost library, and then rte_vhost_submit_enqueue_burst and
>>>> rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
>>>>
>>>> Do you think that could work?
>>>>
>>> Yes, this is a workable model. As I said in previous reply, I have no objection to
>> make the dmadev. However, what we currently want to do is creating the async
>> data path for vhost, and we actually have no preference to the underlying DMA
>> device model. I believe our current design of the API proto type /data structures
>> are quite common for various DMA acceleration solutions and there is no blocker
>> for any new DMA device to adapt to these APIs or extend to a new one.
>>
>> IMO, as a driver writer,  we should not be writing TWO DMA driver. One for vhost
>> and other one for rawdev.
> It's the most simplest case if statically 1:1 mapping driver (e.g. {port, queue}) to a vhost session {vid, qid}. However, it's not enough scalable to integrate device model with vhost library. There're a few intentions belong to app logic rather than driver, e.g. 1:N load balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.
> 
> It was not asking to writing two drivers. Each driver remains to offer provider for its own device class, which is independent. App provides the intension (adapter) to associate various device capability to vhost session.
> 
>> If vhost is the first consumer of DMA needed then I think, it make sense to add
>> dmadev first.
> On the other hand, it's risky to define 'dmadev' according to vhost's flavor before not getting aware of any other candidates. Comparing with kern Async TX DMA API (async_memcpy), RFC is very much focus on S/G buffer but not a async_memcpy.
> 
>> The rawdev DMA driver to dmadev DMA driver conversion will be the driver owner
>> job.
> It's true when it's necessary. Even that is the case, it's better for vhost to be independent with any device model, moreover vhost usage doesn't have broad enough coverage for a new device class.
> 
>> I think, it makes sense to define the dmadev API and then costume by virtio to
>> avoid integration issues.
> Vhost is a library but not a app. We'd better avoid to intro either overkill integration logic or extra device model dependence.

Thinking at it again, I support Patrick and Steve approach. It will be
more flexible than adding dmadev support into the Vhost library
directly.

For example, this new API could be used to have a cleaner integration of
Tx zero-copy, which is kind of hacky for now.

Thanks,
Maxime

> Thanks,
> Steve
> 
>>
>>
>>
>>>
>>> Thanks,
>>>
>>> Patrick
>>>

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Jerin Jacob]

On Mon, Apr 20, 2020 at 5:14 PM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
>
>
> On 4/20/20 1:13 PM, Jerin Jacob wrote:
> > On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming <cunming.liang@intel.com> wrote:
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: Jerin Jacob <jerinjacobk@gmail.com>
> >>> Sent: Friday, April 17, 2020 5:55 PM
> >>> To: Fu, Patrick <patrick.fu@intel.com>
> >>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; dev@dpdk.org; Ye,
> >>> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
> >>> Zhihong <zhihong.wang@intel.com>; Liang, Cunming <cunming.liang@intel.com>
> >>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
> >>> DMA Engines
> >>>
> >>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
> >>>>
> >>>>
> >> [...]
> >>>>>>
> >>>>>> I believe it doesn't conflict. The purpose of this RFC is to
> >>>>>> create an async
> >>>>> data path in vhost-user and provide a way for applications to work
> >>>>> with this new path. dmadev is another topic which could be discussed
> >>>>> separately. If we do have the dmadev available in the future, this
> >>>>> vhost async data path could certainly be backed by the new dma
> >>>>> abstraction without major interface change.
> >>>>>
> >>>>> Maybe that one advantage of a dmadev class is that it would be
> >>>>> easier and more transparent for the application to consume.
> >>>>>
> >>>>> The application would register some DMA devices, pass them to the
> >>>>> Vhost library, and then rte_vhost_submit_enqueue_burst and
> >>>>> rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
> >>>>>
> >>>>> Do you think that could work?
> >>>>>
> >>>> Yes, this is a workable model. As I said in previous reply, I have no objection to
> >>> make the dmadev. However, what we currently want to do is creating the async
> >>> data path for vhost, and we actually have no preference to the underlying DMA
> >>> device model. I believe our current design of the API proto type /data structures
> >>> are quite common for various DMA acceleration solutions and there is no blocker
> >>> for any new DMA device to adapt to these APIs or extend to a new one.
> >>>
> >>> IMO, as a driver writer,  we should not be writing TWO DMA driver. One for vhost
> >>> and other one for rawdev.
> >> It's the most simplest case if statically 1:1 mapping driver (e.g. {port, queue}) to a vhost session {vid, qid}. However, it's not enough scalable to integrate device model with vhost library. There're a few intentions belong to app logic rather than driver, e.g. 1:N load balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.
> >
> >
> > Before moving to reply to comments, Which DMA engine you are planning
> > to integrate with vHOST?
> > Is is ioat? if not ioat(drivers/raw/ioat/), How do you think, how we
> > can integrate this IOAT DMA engine to vHOST as a use case?
> >
>
> I guess it could be done in the vhost example.


Could not see any reference to DMA in  examples/vhost*

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Maxime Coquelin]

On 4/20/20 2:08 PM, Jerin Jacob wrote:
> On Mon, Apr 20, 2020 at 5:14 PM Maxime Coquelin
> <maxime.coquelin@redhat.com> wrote:
>>
>>
>>
>> On 4/20/20 1:13 PM, Jerin Jacob wrote:
>>> On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming <cunming.liang@intel.com> wrote:
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Jerin Jacob <jerinjacobk@gmail.com>
>>>>> Sent: Friday, April 17, 2020 5:55 PM
>>>>> To: Fu, Patrick <patrick.fu@intel.com>
>>>>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; dev@dpdk.org; Ye,
>>>>> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
>>>>> Zhihong <zhihong.wang@intel.com>; Liang, Cunming <cunming.liang@intel.com>
>>>>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
>>>>> DMA Engines
>>>>>
>>>>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
>>>>>>
>>>>>>
>>>> [...]
>>>>>>>>
>>>>>>>> I believe it doesn't conflict. The purpose of this RFC is to
>>>>>>>> create an async
>>>>>>> data path in vhost-user and provide a way for applications to work
>>>>>>> with this new path. dmadev is another topic which could be discussed
>>>>>>> separately. If we do have the dmadev available in the future, this
>>>>>>> vhost async data path could certainly be backed by the new dma
>>>>>>> abstraction without major interface change.
>>>>>>>
>>>>>>> Maybe that one advantage of a dmadev class is that it would be
>>>>>>> easier and more transparent for the application to consume.
>>>>>>>
>>>>>>> The application would register some DMA devices, pass them to the
>>>>>>> Vhost library, and then rte_vhost_submit_enqueue_burst and
>>>>>>> rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
>>>>>>>
>>>>>>> Do you think that could work?
>>>>>>>
>>>>>> Yes, this is a workable model. As I said in previous reply, I have no objection to
>>>>> make the dmadev. However, what we currently want to do is creating the async
>>>>> data path for vhost, and we actually have no preference to the underlying DMA
>>>>> device model. I believe our current design of the API proto type /data structures
>>>>> are quite common for various DMA acceleration solutions and there is no blocker
>>>>> for any new DMA device to adapt to these APIs or extend to a new one.
>>>>>
>>>>> IMO, as a driver writer,  we should not be writing TWO DMA driver. One for vhost
>>>>> and other one for rawdev.
>>>> It's the most simplest case if statically 1:1 mapping driver (e.g. {port, queue}) to a vhost session {vid, qid}. However, it's not enough scalable to integrate device model with vhost library. There're a few intentions belong to app logic rather than driver, e.g. 1:N load balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.
>>>
>>>
>>> Before moving to reply to comments, Which DMA engine you are planning
>>> to integrate with vHOST?
>>> Is is ioat? if not ioat(drivers/raw/ioat/), How do you think, how we
>>> can integrate this IOAT DMA engine to vHOST as a use case?
>>>
>>
>> I guess it could be done in the vhost example.
> 
> 
> Could not see any reference to DMA in  examples/vhost*
> 

That's because we are discussing the API to introduce DMA support in
this exact mail thread, nothing has been merged yet.

Maxime

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Jerin Jacob]

On Mon, Apr 20, 2020 at 5:40 PM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
>
>
> On 4/20/20 2:08 PM, Jerin Jacob wrote:
> > On Mon, Apr 20, 2020 at 5:14 PM Maxime Coquelin
> > <maxime.coquelin@redhat.com> wrote:
> >>
> >>
> >>
> >> On 4/20/20 1:13 PM, Jerin Jacob wrote:
> >>> On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming <cunming.liang@intel.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Jerin Jacob <jerinjacobk@gmail.com>
> >>>>> Sent: Friday, April 17, 2020 5:55 PM
> >>>>> To: Fu, Patrick <patrick.fu@intel.com>
> >>>>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; dev@dpdk.org; Ye,
> >>>>> Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
> >>>>> Zhihong <zhihong.wang@intel.com>; Liang, Cunming <cunming.liang@intel.com>
> >>>>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
> >>>>> DMA Engines
> >>>>>
> >>>>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com> wrote:
> >>>>>>
> >>>>>>
> >>>> [...]
> >>>>>>>>
> >>>>>>>> I believe it doesn't conflict. The purpose of this RFC is to
> >>>>>>>> create an async
> >>>>>>> data path in vhost-user and provide a way for applications to work
> >>>>>>> with this new path. dmadev is another topic which could be discussed
> >>>>>>> separately. If we do have the dmadev available in the future, this
> >>>>>>> vhost async data path could certainly be backed by the new dma
> >>>>>>> abstraction without major interface change.
> >>>>>>>
> >>>>>>> Maybe that one advantage of a dmadev class is that it would be
> >>>>>>> easier and more transparent for the application to consume.
> >>>>>>>
> >>>>>>> The application would register some DMA devices, pass them to the
> >>>>>>> Vhost library, and then rte_vhost_submit_enqueue_burst and
> >>>>>>> rte_vhost_poll_enqueue_completed would call the dmadev callbacks directly.
> >>>>>>>
> >>>>>>> Do you think that could work?
> >>>>>>>
> >>>>>> Yes, this is a workable model. As I said in previous reply, I have no objection to
> >>>>> make the dmadev. However, what we currently want to do is creating the async
> >>>>> data path for vhost, and we actually have no preference to the underlying DMA
> >>>>> device model. I believe our current design of the API proto type /data structures
> >>>>> are quite common for various DMA acceleration solutions and there is no blocker
> >>>>> for any new DMA device to adapt to these APIs or extend to a new one.
> >>>>>
> >>>>> IMO, as a driver writer,  we should not be writing TWO DMA driver. One for vhost
> >>>>> and other one for rawdev.
> >>>> It's the most simplest case if statically 1:1 mapping driver (e.g. {port, queue}) to a vhost session {vid, qid}. However, it's not enough scalable to integrate device model with vhost library. There're a few intentions belong to app logic rather than driver, e.g. 1:N load balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.
> >>>
> >>>
> >>> Before moving to reply to comments, Which DMA engine you are planning
> >>> to integrate with vHOST?
> >>> Is is ioat? if not ioat(drivers/raw/ioat/), How do you think, how we
> >>> can integrate this IOAT DMA engine to vHOST as a use case?
> >>>
> >>
> >> I guess it could be done in the vhost example.
> >
> >
> > Could not see any reference to DMA in  examples/vhost*
> >
>
> That's because we are discussing the API to introduce DMA support in
> this exact mail thread, nothing has been merged yet.

Some confusion here. Original question was,
# This is an RFC for DMA support in vHOST
# What is the underneath DMA engine planned for hooking to vHOST async
API as a "implementation" for this RFC?
# If it ioat, How does the integration work with ioat exiting
rawdriver and new API?
# if it not ioat, What it takes to add support ioat based DMA engine
to vHOST aysnc API

>
> Maxime
>

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Fu, Patrick]

Hi Jerin

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Monday, April 20, 2020 8:15 PM
> To: Maxime Coquelin <maxime.coquelin@redhat.com>
> Cc: Liang, Cunming <cunming.liang@intel.com>; Fu, Patrick
> <patrick.fu@intel.com>; dev@dpdk.org; Ye, Xiaolong
> <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, Zhihong
> <zhihong.wang@intel.com>
> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost
> with DMA Engines
> 
> On Mon, Apr 20, 2020 at 5:40 PM Maxime Coquelin
> <maxime.coquelin@redhat.com> wrote:
> >
> >
> >
> > On 4/20/20 2:08 PM, Jerin Jacob wrote:
> > > On Mon, Apr 20, 2020 at 5:14 PM Maxime Coquelin
> > > <maxime.coquelin@redhat.com> wrote:
> > >>
> > >>
> > >>
> > >> On 4/20/20 1:13 PM, Jerin Jacob wrote:
> > >>> On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming
> <cunming.liang@intel.com> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>>> -----Original Message-----
> > >>>>> From: Jerin Jacob <jerinjacobk@gmail.com>
> > >>>>> Sent: Friday, April 17, 2020 5:55 PM
> > >>>>> To: Fu, Patrick <patrick.fu@intel.com>
> > >>>>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>;
> dev@dpdk.org;
> > >>>>> Ye, Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu
> > >>>>> <jiayu.hu@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>;
> > >>>>> Liang, Cunming <cunming.liang@intel.com>
> > >>>>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for
> > >>>>> DPDK vHost with DMA Engines
> > >>>>>
> > >>>>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com>
> wrote:
> > >>>>>>
> > >>>>>>
> > >>>> [...]
> > >>>>>>>>
> > >>>>>>>> I believe it doesn't conflict. The purpose of this RFC is to
> > >>>>>>>> create an async
> > >>>>>>> data path in vhost-user and provide a way for applications to
> > >>>>>>> work with this new path. dmadev is another topic which could
> > >>>>>>> be discussed separately. If we do have the dmadev available in
> > >>>>>>> the future, this vhost async data path could certainly be
> > >>>>>>> backed by the new dma abstraction without major interface
> change.
> > >>>>>>>
> > >>>>>>> Maybe that one advantage of a dmadev class is that it would be
> > >>>>>>> easier and more transparent for the application to consume.
> > >>>>>>>
> > >>>>>>> The application would register some DMA devices, pass them to
> > >>>>>>> the Vhost library, and then rte_vhost_submit_enqueue_burst and
> > >>>>>>> rte_vhost_poll_enqueue_completed would call the dmadev
> callbacks directly.
> > >>>>>>>
> > >>>>>>> Do you think that could work?
> > >>>>>>>
> > >>>>>> Yes, this is a workable model. As I said in previous reply, I
> > >>>>>> have no objection to
> > >>>>> make the dmadev. However, what we currently want to do is
> > >>>>> creating the async data path for vhost, and we actually have no
> > >>>>> preference to the underlying DMA device model. I believe our
> > >>>>> current design of the API proto type /data structures are quite
> > >>>>> common for various DMA acceleration solutions and there is no
> blocker for any new DMA device to adapt to these APIs or extend to a new
> one.
> > >>>>>
> > >>>>> IMO, as a driver writer,  we should not be writing TWO DMA
> > >>>>> driver. One for vhost and other one for rawdev.
> > >>>> It's the most simplest case if statically 1:1 mapping driver (e.g. {port,
> queue}) to a vhost session {vid, qid}. However, it's not enough scalable to
> integrate device model with vhost library. There're a few intentions belong to
> app logic rather than driver, e.g. 1:N load balancing, various device type
> usages (e.g. vhost zcopy via ethdev) and etc.
> > >>>
> > >>>
> > >>> Before moving to reply to comments, Which DMA engine you are
> > >>> planning to integrate with vHOST?
> > >>> Is is ioat? if not ioat(drivers/raw/ioat/), How do you think, how
> > >>> we can integrate this IOAT DMA engine to vHOST as a use case?
> > >>>
> > >>
> > >> I guess it could be done in the vhost example.
> > >
> > >
> > > Could not see any reference to DMA in  examples/vhost*
> > >
> >
> > That's because we are discussing the API to introduce DMA support in
> > this exact mail thread, nothing has been merged yet.
> 
> Some confusion here. Original question was, # This is an RFC for DMA
> support in vHOST # What is the underneath DMA engine planned for hooking
> to vHOST async API as a "implementation" for this RFC?
> # If it ioat, How does the integration work with ioat exiting rawdriver and
> new API?
> # if it not ioat, What it takes to add support ioat based DMA engine to vHOST
> aysnc API
> 
It most likely that IOAT could be leveraged as the first demonstration on the async DMA acceleration for vHOST. However, this is neither a limitation nor do we design this RFC specifically for IOAT.
With current RFC design, we will need applications to implement callbacks (which will call into the IOAT pmd in IOAT case) that can work with vHost async path.

Thanks,

Patrick

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Jerin Jacob]

On Tue, Apr 21, 2020 at 8:14 AM Fu, Patrick <patrick.fu@intel.com> wrote:
>
> Hi Jerin

Hi Patrick,

>
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk@gmail.com>
> > Sent: Monday, April 20, 2020 8:15 PM
> > To: Maxime Coquelin <maxime.coquelin@redhat.com>
> > Cc: Liang, Cunming <cunming.liang@intel.com>; Fu, Patrick
> > <patrick.fu@intel.com>; dev@dpdk.org; Ye, Xiaolong
> > <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, Zhihong
> > <zhihong.wang@intel.com>
> > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost
> > with DMA Engines
> >
> > On Mon, Apr 20, 2020 at 5:40 PM Maxime Coquelin
> > <maxime.coquelin@redhat.com> wrote:
> > >
> > >
> > >
> > > On 4/20/20 2:08 PM, Jerin Jacob wrote:
> > > > On Mon, Apr 20, 2020 at 5:14 PM Maxime Coquelin
> > > > <maxime.coquelin@redhat.com> wrote:
> > > >>
> > > >>
> > > >>
> > > >> On 4/20/20 1:13 PM, Jerin Jacob wrote:
> > > >>> On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming
> > <cunming.liang@intel.com> wrote:
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> -----Original Message-----
> > > >>>>> From: Jerin Jacob <jerinjacobk@gmail.com>
> > > >>>>> Sent: Friday, April 17, 2020 5:55 PM
> > > >>>>> To: Fu, Patrick <patrick.fu@intel.com>
> > > >>>>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>;
> > dev@dpdk.org;
> > > >>>>> Ye, Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu
> > > >>>>> <jiayu.hu@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>;
> > > >>>>> Liang, Cunming <cunming.liang@intel.com>
> > > >>>>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for
> > > >>>>> DPDK vHost with DMA Engines
> > > >>>>>
> > > >>>>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick <patrick.fu@intel.com>
> > wrote:
> > > >>>>>>
> > > >>>>>>
> > > >>>> [...]
> > > >>>>>>>>
> > > >>>>>>>> I believe it doesn't conflict. The purpose of this RFC is to
> > > >>>>>>>> create an async
> > > >>>>>>> data path in vhost-user and provide a way for applications to
> > > >>>>>>> work with this new path. dmadev is another topic which could
> > > >>>>>>> be discussed separately. If we do have the dmadev available in
> > > >>>>>>> the future, this vhost async data path could certainly be
> > > >>>>>>> backed by the new dma abstraction without major interface
> > change.
> > > >>>>>>>
> > > >>>>>>> Maybe that one advantage of a dmadev class is that it would be
> > > >>>>>>> easier and more transparent for the application to consume.
> > > >>>>>>>
> > > >>>>>>> The application would register some DMA devices, pass them to
> > > >>>>>>> the Vhost library, and then rte_vhost_submit_enqueue_burst and
> > > >>>>>>> rte_vhost_poll_enqueue_completed would call the dmadev
> > callbacks directly.
> > > >>>>>>>
> > > >>>>>>> Do you think that could work?
> > > >>>>>>>
> > > >>>>>> Yes, this is a workable model. As I said in previous reply, I
> > > >>>>>> have no objection to
> > > >>>>> make the dmadev. However, what we currently want to do is
> > > >>>>> creating the async data path for vhost, and we actually have no
> > > >>>>> preference to the underlying DMA device model. I believe our
> > > >>>>> current design of the API proto type /data structures are quite
> > > >>>>> common for various DMA acceleration solutions and there is no
> > blocker for any new DMA device to adapt to these APIs or extend to a new
> > one.
> > > >>>>>
> > > >>>>> IMO, as a driver writer,  we should not be writing TWO DMA
> > > >>>>> driver. One for vhost and other one for rawdev.
> > > >>>> It's the most simplest case if statically 1:1 mapping driver (e.g. {port,
> > queue}) to a vhost session {vid, qid}. However, it's not enough scalable to
> > integrate device model with vhost library. There're a few intentions belong to
> > app logic rather than driver, e.g. 1:N load balancing, various device type
> > usages (e.g. vhost zcopy via ethdev) and etc.
> > > >>>
> > > >>>
> > > >>> Before moving to reply to comments, Which DMA engine you are
> > > >>> planning to integrate with vHOST?
> > > >>> Is is ioat? if not ioat(drivers/raw/ioat/), How do you think, how
> > > >>> we can integrate this IOAT DMA engine to vHOST as a use case?
> > > >>>
> > > >>
> > > >> I guess it could be done in the vhost example.
> > > >
> > > >
> > > > Could not see any reference to DMA in  examples/vhost*
> > > >
> > >
> > > That's because we are discussing the API to introduce DMA support in
> > > this exact mail thread, nothing has been merged yet.
> >
> > Some confusion here. Original question was, # This is an RFC for DMA
> > support in vHOST # What is the underneath DMA engine planned for hooking
> > to vHOST async API as a "implementation" for this RFC?
> > # If it ioat, How does the integration work with ioat exiting rawdriver and
> > new API?
> > # if it not ioat, What it takes to add support ioat based DMA engine to vHOST
> > aysnc API
> >
> It most likely that IOAT could be leveraged as the first demonstration on the async DMA acceleration for vHOST. However, this is neither a limitation nor do we design this RFC specifically for IOAT.
> With current RFC design, we will need applications to implement callbacks (which will call into the IOAT pmd in IOAT case) that can work with vHost async path.

Then it would be calling some PMD specific APIs for dpaa2_qdma,
octeontx2_dma, ioat and there will issue with integrating  DMA
consumer as vHOST and another consumer together.
The correct approach is to create a new class for dma like Linux and
vHOST consume as a client so that integration aspects are intact.





>
> Thanks,
>
> Patrick
>
>

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Liang, Cunming]

Hi Jerin,

> -----Original Message-----
> From: Jerin Jacob <jerinjacobk@gmail.com>
> Sent: Tuesday, April 21, 2020 2:04 PM
> To: Fu, Patrick <patrick.fu@intel.com>
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; Liang, Cunming
> <cunming.liang@intel.com>; dev@dpdk.org; Ye, Xiaolong
> <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, Zhihong
> <zhihong.wang@intel.com>
> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
> DMA Engines
> 
> On Tue, Apr 21, 2020 at 8:14 AM Fu, Patrick <patrick.fu@intel.com> wrote:
> >
> > Hi Jerin
> 
> Hi Patrick,
> 
> >
> > > -----Original Message-----
> > > From: Jerin Jacob <jerinjacobk@gmail.com>
> > > Sent: Monday, April 20, 2020 8:15 PM
> > > To: Maxime Coquelin <maxime.coquelin@redhat.com>
> > > Cc: Liang, Cunming <cunming.liang@intel.com>; Fu, Patrick
> > > <patrick.fu@intel.com>; dev@dpdk.org; Ye, Xiaolong
> > > <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
> > > Zhihong <zhihong.wang@intel.com>
> > > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK
> > > vHost with DMA Engines
> > >
> > > On Mon, Apr 20, 2020 at 5:40 PM Maxime Coquelin
> > > <maxime.coquelin@redhat.com> wrote:
> > > >
> > > >
> > > >
> > > > On 4/20/20 2:08 PM, Jerin Jacob wrote:
> > > > > On Mon, Apr 20, 2020 at 5:14 PM Maxime Coquelin
> > > > > <maxime.coquelin@redhat.com> wrote:
> > > > >>
> > > > >>
> > > > >>
> > > > >> On 4/20/20 1:13 PM, Jerin Jacob wrote:
> > > > >>> On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming
> > > <cunming.liang@intel.com> wrote:
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>> -----Original Message-----
> > > > >>>>> From: Jerin Jacob <jerinjacobk@gmail.com>
> > > > >>>>> Sent: Friday, April 17, 2020 5:55 PM
> > > > >>>>> To: Fu, Patrick <patrick.fu@intel.com>
> > > > >>>>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>;
> > > dev@dpdk.org;
> > > > >>>>> Ye, Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu
> > > > >>>>> <jiayu.hu@intel.com>; Wang, Zhihong
> > > > >>>>> <zhihong.wang@intel.com>; Liang, Cunming
> > > > >>>>> <cunming.liang@intel.com>
> > > > >>>>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for
> > > > >>>>> DPDK vHost with DMA Engines
> > > > >>>>>
> > > > >>>>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick
> > > > >>>>> <patrick.fu@intel.com>
> > > wrote:
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>> [...]
> > > > >>>>>>>>
> > > > >>>>>>>> I believe it doesn't conflict. The purpose of this RFC is
> > > > >>>>>>>> to create an async
> > > > >>>>>>> data path in vhost-user and provide a way for applications
> > > > >>>>>>> to work with this new path. dmadev is another topic which
> > > > >>>>>>> could be discussed separately. If we do have the dmadev
> > > > >>>>>>> available in the future, this vhost async data path could
> > > > >>>>>>> certainly be backed by the new dma abstraction without
> > > > >>>>>>> major interface
> > > change.
> > > > >>>>>>>
> > > > >>>>>>> Maybe that one advantage of a dmadev class is that it
> > > > >>>>>>> would be easier and more transparent for the application to
> consume.
> > > > >>>>>>>
> > > > >>>>>>> The application would register some DMA devices, pass them
> > > > >>>>>>> to the Vhost library, and then
> > > > >>>>>>> rte_vhost_submit_enqueue_burst and
> > > > >>>>>>> rte_vhost_poll_enqueue_completed would call the dmadev
> > > callbacks directly.
> > > > >>>>>>>
> > > > >>>>>>> Do you think that could work?
> > > > >>>>>>>
> > > > >>>>>> Yes, this is a workable model. As I said in previous reply,
> > > > >>>>>> I have no objection to
> > > > >>>>> make the dmadev. However, what we currently want to do is
> > > > >>>>> creating the async data path for vhost, and we actually have
> > > > >>>>> no preference to the underlying DMA device model. I believe
> > > > >>>>> our current design of the API proto type /data structures
> > > > >>>>> are quite common for various DMA acceleration solutions and
> > > > >>>>> there is no
> > > blocker for any new DMA device to adapt to these APIs or extend to a
> > > new one.
> > > > >>>>>
> > > > >>>>> IMO, as a driver writer,  we should not be writing TWO DMA
> > > > >>>>> driver. One for vhost and other one for rawdev.
> > > > >>>> It's the most simplest case if statically 1:1 mapping driver
> > > > >>>> (e.g. {port,
> > > queue}) to a vhost session {vid, qid}. However, it's not enough
> > > scalable to integrate device model with vhost library. There're a
> > > few intentions belong to app logic rather than driver, e.g. 1:N load
> > > balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.
> > > > >>>
> > > > >>>
> > > > >>> Before moving to reply to comments, Which DMA engine you are
> > > > >>> planning to integrate with vHOST?
> > > > >>> Is is ioat? if not ioat(drivers/raw/ioat/), How do you think,
> > > > >>> how we can integrate this IOAT DMA engine to vHOST as a use case?
> > > > >>>
> > > > >>
> > > > >> I guess it could be done in the vhost example.
> > > > >
> > > > >
> > > > > Could not see any reference to DMA in  examples/vhost*
> > > > >
> > > >
> > > > That's because we are discussing the API to introduce DMA support
> > > > in this exact mail thread, nothing has been merged yet.
> > >
> > > Some confusion here. Original question was, # This is an RFC for DMA
> > > support in vHOST # What is the underneath DMA engine planned for
> > > hooking to vHOST async API as a "implementation" for this RFC?
> > > # If it ioat, How does the integration work with ioat exiting
> > > rawdriver and new API?
> > > # if it not ioat, What it takes to add support ioat based DMA engine
> > > to vHOST aysnc API
> > >
> > It most likely that IOAT could be leveraged as the first demonstration on the
> async DMA acceleration for vHOST. However, this is neither a limitation nor do we
> design this RFC specifically for IOAT.
> > With current RFC design, we will need applications to implement callbacks
> (which will call into the IOAT pmd in IOAT case) that can work with vHost async
> path.
> 
> Then it would be calling some PMD specific APIs for dpaa2_qdma, octeontx2_dma,
> ioat and there will issue with integrating  DMA consumer as vHOST and another
> consumer together.
The main effort is to intro async-mode API for vhost allowing external hooks for raw buffer (VM and/or host app) access regardless of virtio ring layout.
It never forces ops to leverage DMA device, think about rxtx_callback of ethdev. That's pretty much app's or helper library's flavor of the ops.
If not comfortable to demo ops with dma, that's fine for us to focus on CPU only as a hook provider in the sample, and omit 'w/ DMA engine' from RFC.

> The correct approach is to create a new class for dma like Linux and vHOST
> consume as a client so that integration aspects are intact.
I'm curious what's the usages when dpaa2_qdma, octeontx2_dma, ioat being introduce into raw device, why these usage don’t have excuse, but you believe vhost has.
They exists for a while as a raw device. If it's do necessary, they've already built a class... 
As you said, vhost is one of the client to consume but not owns device class, moreover these raw device is not the only 'server' to vhost.
For your concern, it worth a separate conversation but not limited for vhost case.

Thanks,
Steve
> 
> 
> 
> 
> 
> >
> > Thanks,
> >
> > Patrick
> >
> >

Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines

[Jerin Jacob]

On Tue, Apr 21, 2020 at 2:00 PM Liang, Cunming <cunming.liang@intel.com> wrote:
>
> Hi Jerin,
>
> > -----Original Message-----
> > From: Jerin Jacob <jerinjacobk@gmail.com>
> > Sent: Tuesday, April 21, 2020 2:04 PM
> > To: Fu, Patrick <patrick.fu@intel.com>
> > Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; Liang, Cunming
> > <cunming.liang@intel.com>; dev@dpdk.org; Ye, Xiaolong
> > <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, Zhihong
> > <zhihong.wang@intel.com>
> > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with
> > DMA Engines
> >
> > On Tue, Apr 21, 2020 at 8:14 AM Fu, Patrick <patrick.fu@intel.com> wrote:
> > >
> > > Hi Jerin
> >
> > Hi Patrick,
> >
> > >
> > > > -----Original Message-----
> > > > From: Jerin Jacob <jerinjacobk@gmail.com>
> > > > Sent: Monday, April 20, 2020 8:15 PM
> > > > To: Maxime Coquelin <maxime.coquelin@redhat.com>
> > > > Cc: Liang, Cunming <cunming.liang@intel.com>; Fu, Patrick
> > > > <patrick.fu@intel.com>; dev@dpdk.org; Ye, Xiaolong
> > > > <xiaolong.ye@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang,
> > > > Zhihong <zhihong.wang@intel.com>
> > > > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK
> > > > vHost with DMA Engines
> > > >
> > > > On Mon, Apr 20, 2020 at 5:40 PM Maxime Coquelin
> > > > <maxime.coquelin@redhat.com> wrote:
> > > > >
> > > > >
> > > > >
> > > > > On 4/20/20 2:08 PM, Jerin Jacob wrote:
> > > > > > On Mon, Apr 20, 2020 at 5:14 PM Maxime Coquelin
> > > > > > <maxime.coquelin@redhat.com> wrote:
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On 4/20/20 1:13 PM, Jerin Jacob wrote:
> > > > > >>> On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming
> > > > <cunming.liang@intel.com> wrote:
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>> -----Original Message-----
> > > > > >>>>> From: Jerin Jacob <jerinjacobk@gmail.com>
> > > > > >>>>> Sent: Friday, April 17, 2020 5:55 PM
> > > > > >>>>> To: Fu, Patrick <patrick.fu@intel.com>
> > > > > >>>>> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>;
> > > > dev@dpdk.org;
> > > > > >>>>> Ye, Xiaolong <xiaolong.ye@intel.com>; Hu, Jiayu
> > > > > >>>>> <jiayu.hu@intel.com>; Wang, Zhihong
> > > > > >>>>> <zhihong.wang@intel.com>; Liang, Cunming
> > > > > >>>>> <cunming.liang@intel.com>
> > > > > >>>>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for
> > > > > >>>>> DPDK vHost with DMA Engines
> > > > > >>>>>
> > > > > >>>>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick
> > > > > >>>>> <patrick.fu@intel.com>
> > > > wrote:
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>> [...]
> > > > > >>>>>>>>
> > > > > >>>>>>>> I believe it doesn't conflict. The purpose of this RFC is
> > > > > >>>>>>>> to create an async
> > > > > >>>>>>> data path in vhost-user and provide a way for applications
> > > > > >>>>>>> to work with this new path. dmadev is another topic which
> > > > > >>>>>>> could be discussed separately. If we do have the dmadev
> > > > > >>>>>>> available in the future, this vhost async data path could
> > > > > >>>>>>> certainly be backed by the new dma abstraction without
> > > > > >>>>>>> major interface
> > > > change.
> > > > > >>>>>>>
> > > > > >>>>>>> Maybe that one advantage of a dmadev class is that it
> > > > > >>>>>>> would be easier and more transparent for the application to
> > consume.
> > > > > >>>>>>>
> > > > > >>>>>>> The application would register some DMA devices, pass them
> > > > > >>>>>>> to the Vhost library, and then
> > > > > >>>>>>> rte_vhost_submit_enqueue_burst and
> > > > > >>>>>>> rte_vhost_poll_enqueue_completed would call the dmadev
> > > > callbacks directly.
> > > > > >>>>>>>
> > > > > >>>>>>> Do you think that could work?
> > > > > >>>>>>>
> > > > > >>>>>> Yes, this is a workable model. As I said in previous reply,
> > > > > >>>>>> I have no objection to
> > > > > >>>>> make the dmadev. However, what we currently want to do is
> > > > > >>>>> creating the async data path for vhost, and we actually have
> > > > > >>>>> no preference to the underlying DMA device model. I believe
> > > > > >>>>> our current design of the API proto type /data structures
> > > > > >>>>> are quite common for various DMA acceleration solutions and
> > > > > >>>>> there is no
> > > > blocker for any new DMA device to adapt to these APIs or extend to a
> > > > new one.
> > > > > >>>>>
> > > > > >>>>> IMO, as a driver writer,  we should not be writing TWO DMA
> > > > > >>>>> driver. One for vhost and other one for rawdev.
> > > > > >>>> It's the most simplest case if statically 1:1 mapping driver
> > > > > >>>> (e.g. {port,
> > > > queue}) to a vhost session {vid, qid}. However, it's not enough
> > > > scalable to integrate device model with vhost library. There're a
> > > > few intentions belong to app logic rather than driver, e.g. 1:N load
> > > > balancing, various device type usages (e.g. vhost zcopy via ethdev) and etc.
> > > > > >>>
> > > > > >>>
> > > > > >>> Before moving to reply to comments, Which DMA engine you are
> > > > > >>> planning to integrate with vHOST?
> > > > > >>> Is is ioat? if not ioat(drivers/raw/ioat/), How do you think,
> > > > > >>> how we can integrate this IOAT DMA engine to vHOST as a use case?
> > > > > >>>
> > > > > >>
> > > > > >> I guess it could be done in the vhost example.
> > > > > >
> > > > > >
> > > > > > Could not see any reference to DMA in  examples/vhost*
> > > > > >
> > > > >
> > > > > That's because we are discussing the API to introduce DMA support
> > > > > in this exact mail thread, nothing has been merged yet.
> > > >
> > > > Some confusion here. Original question was, # This is an RFC for DMA
> > > > support in vHOST # What is the underneath DMA engine planned for
> > > > hooking to vHOST async API as a "implementation" for this RFC?
> > > > # If it ioat, How does the integration work with ioat exiting
> > > > rawdriver and new API?
> > > > # if it not ioat, What it takes to add support ioat based DMA engine
> > > > to vHOST aysnc API
> > > >
> > > It most likely that IOAT could be leveraged as the first demonstration on the
> > async DMA acceleration for vHOST. However, this is neither a limitation nor do we
> > design this RFC specifically for IOAT.
> > > With current RFC design, we will need applications to implement callbacks
> > (which will call into the IOAT pmd in IOAT case) that can work with vHost async
> > path.
> >
> > Then it would be calling some PMD specific APIs for dpaa2_qdma, octeontx2_dma,
> > ioat and there will issue with integrating  DMA consumer as vHOST and another
> > consumer together.
> The main effort is to intro async-mode API for vhost allowing external hooks for raw buffer (VM and/or host app) access regardless of virtio ring layout.
> It never forces ops to leverage DMA device, think about rxtx_callback of ethdev. That's pretty much app's or helper library's flavor of the ops.
> If not comfortable to demo ops with dma, that's fine for us to focus on CPU only as a hook provider in the sample, and omit 'w/ DMA engine' from RFC.

See below,

> > The correct approach is to create a new class for dma like Linux and vHOST
> > consume as a client so that integration aspects are intact.
> I'm curious what's the usages when dpaa2_qdma, octeontx2_dma, ioat being introduce into raw device, why these usage don’t have excuse, but you believe vhost has.

This is the first time, a DPDK subsystem tries to use DMA. A subsystem
to subsystem communication should not be through private PMD API.

> They exists for a while as a raw device. If it's do necessary, they've already built a class...
> As you said, vhost is one of the client to consume but not owns device class, moreover these raw device is not the only 'server' to vhost.
> For your concern, it worth a separate conversation but not limited for vhost case.

Yes. It is not limited to vhost and any DPDK subsystem uses the DMA HW
it should be through normative DPDK API.

What you are proposing is the purpose of the demo it is OK. In the
real-world, it can not work if they're more than one DMA consumer
in the system until unless if we standardize the DMA usage in
client/server model.

Semantically,
DMA API should allow
- Allow registering DMA engines, it can be ioat, dpaa2_qdma,
octeontx2_dma or CPU as the fallback
- The client(vhost API) can request for session
- And use the DMA service using created session/channel.

If you do this, you don't need any API in vhost, it can use DMA API
from dmadev and platform can
choose the correct DMA engine instead of the application selects the DMA engine.

I would like to stop here. If Virtio and DPDK maintainers are OK with
the RFC scheme then no issues from the side
as I don't have time.







>
> Thanks,
> Steve
> >
> >
> >
> >
> >
> > >
> > > Thanks,
> > >
> > > Patrick
> > >
> > >