From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 74532A058A; Fri, 17 Apr 2020 11:55:09 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CCAEC1DBD2; Fri, 17 Apr 2020 11:55:08 +0200 (CEST) Received: from mail-il1-f181.google.com (mail-il1-f181.google.com [209.85.166.181]) by dpdk.org (Postfix) with ESMTP id 0B7EB1DBC9 for ; Fri, 17 Apr 2020 11:55:07 +0200 (CEST) Received: by mail-il1-f181.google.com with SMTP id t8so1585468ilj.3 for ; Fri, 17 Apr 2020 02:55:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=VoSPT+K+W4w5zUVHqlNk4KYohl0FmQ9wFC1zKmSm2kc=; b=WxgLrBD+ztvdzbRUacXl5R2kvcFqYoCfhGlFYeZEz1+aYrPc3K/8sx4CKPtFAQWB25 SsrOES5TaacoLcZ1jjmXgXYEekRbUl/ZeEdpViEuUAHfMbIYCutIStW/kzSZHlBQjAqw 4ZPV8SD+WkrGNt7XAdc5JCdO3dqBCDob8dNBRDoY0JlP02Gi+IOCAGfDpEfWVqksa6EH MQn4Yoscbdk29BDADsGDlzpdfykP23RU352ayXraBLlo1OdLAlsIgHSjL3qO6keDTs02 3niugXDLf5PELY5kwfrPFuWNGQGXfCPQOlcWXJmQXY9xSjEW/OvUsishi83UXRcVthdz SJUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=VoSPT+K+W4w5zUVHqlNk4KYohl0FmQ9wFC1zKmSm2kc=; b=JFF/Hy7Fg6jjBWKfF1xdkPbkzAl7Q2rOZ/XGM5q1jdBzJfSzqUJyT7ZN8GiwVlF6IS MLg517hYz2HOpHs723tUsCin4by6oucyWl8RUmruzUKxHJx0dZp4rLB/89G2/tBekGAA F+qDsFwVVrOGQX5dKR7fJA7MZDIO9pAUk9+VY/2Ccr4O6tdOr9a6vHSLAcksuHZ86UmT U0uns2SHm/ngyiN1iZAS6elxqs5xogKsFNHgsepfxcUdt2w3xRGR4IxXgL0aQJwLTzQz ZO0Y/pRERs2IdideQdYjtnBC+wmEYiZHiNYAzJmAmjoyViV/aKmsCY1YA2IpRp19DsCN 94Uw== X-Gm-Message-State: AGi0PuYrkBEEzv88QcU611mNsQgrpSRyIjxmwCzYBpN9AyE5uKfY1VmI 8N/nCMC7XLlhjWorTcxGPfHLcTEQu1FYZfoy7YM= X-Google-Smtp-Source: APiQypJuFeuaa0u4bCfk1plHaM6q0Bow11hNstG5dpseMzmICar3u4sIpU2YwY6oVLv5qmudeZyxY0L3yJoPqLBnkh8= X-Received: by 2002:a92:48cb:: with SMTP id j72mr2162339ilg.162.1587117307184; Fri, 17 Apr 2020 02:55:07 -0700 (PDT) MIME-Version: 1.0 References: <89B17B9B05A1964E8D40D6090018F28151277ADF@SHSMSX107.ccr.corp.intel.com> <89B17B9B05A1964E8D40D6090018F28151277C5C@SHSMSX107.ccr.corp.intel.com> <8e4691c5-3e58-fb5d-f1f5-6b9c994c3949@redhat.com> <89B17B9B05A1964E8D40D6090018F28151277D4F@SHSMSX107.ccr.corp.intel.com> In-Reply-To: <89B17B9B05A1964E8D40D6090018F28151277D4F@SHSMSX107.ccr.corp.intel.com> From: Jerin Jacob Date: Fri, 17 Apr 2020 15:24:51 +0530 Message-ID: To: "Fu, Patrick" Cc: Maxime Coquelin , "dev@dpdk.org" , "Ye, Xiaolong" , "Hu, Jiayu" , "Wang, Zhihong" , "Liang, Cunming" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick wrote: > > > > > -----Original Message----- > > From: Maxime Coquelin > > Sent: Friday, April 17, 2020 4:40 PM > > To: Fu, Patrick ; Jerin Jacob > > Cc: dev@dpdk.org; Ye, Xiaolong ; Hu, Jiayu > > ; Wang, Zhihong ; Liang, > > Cunming > > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost > > with DMA Engines > > > > > > > > On 4/17/20 10:29 AM, Fu, Patrick wrote: > > > Hi Jerin, > > > > > >> -----Original Message----- > > >> From: Jerin Jacob > > >> Sent: Friday, April 17, 2020 4:02 PM > > >> To: Fu, Patrick > > >> Cc: dev@dpdk.org; Maxime Coquelin ; > > Ye, > > >> Xiaolong ; Hu, Jiayu ; > > >> Wang, Zhihong ; Liang, Cunming > > >> > > >> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK > > >> vHost with DMA Engines > > >> > > >> On Fri, Apr 17, 2020 at 12:56 PM Fu, Patrick = wrote: > > >>> > > >>> Background > > >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > >>> DPDK vhost library implements a user-space VirtIO net backend > > >>> allowing > > >> host applications to directly communicate with VirtIO front-end in > > >> VMs and containers. However, every vhost enqueue/dequeue operation > > >> requires to copy packet buffers between guest and host memory. The > > >> overhead of copying large bulk of data makes the vhost backend becom= e > > >> the I/O bottleneck. DMA engines, including un-core DMA accelerator, > > >> like Crystal Beach DMA (CBDMA) and Data Streaming Accelerator (DSA), > > >> and discrete card general purpose DMA, are extremely efficient in > > >> data movement within system memory. Therefore, we propose a set of > > >> asynchronous DMA data movement API in vhost library for DMA > > >> acceleration. With offloading packet copies in vhost data-path from > > >> the CPU to the DMA engine, which can not only accelerate data transf= ers, > > but also save precious CPU core resources. > > >>> > > >>> New API Overview > > >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > >>> The proposed APIs in the vhost library support various DMA engines > > >>> to > > >> accelerate data transfers in the data-path. For the higher > > >> performance, DMA engines work in an asynchronous manner, where > > DMA > > >> data transfers and CPU computations are executed in parallel. The > > >> proposed API consists of control path API and data path API. The > > >> control path API includes Registration API and DMA operation > > >> callback, and the data path API includes asynchronous API. To remove > > >> the dependency of vendor specific DMA engines, the DMA operation > > >> callback provides generic DMA data transfer abstractions. To support > > >> asynchronous DMA data movement, the new async API provides > > >> asynchronous ring operation semantic in data-path. To enable/disable > > >> DMA acceleration for virtqueues, users need to use registration API > > >> is to register/unregister DMA callback implementations to the vhost > > >> library and bind DMA channels to virtqueues. The DMA channels used b= y > > >> virtqueues are provided by DPDK applications, which is backed by vir= tual > > or physical DMA devices. > > >>> The proposed APIs are consisted of 3 sub-sets: > > >>> 1. DMA Registration APIs > > >>> 2. DMA Operation Callbacks > > >>> 3. Async Data APIs > > >>> > > >>> DMA Registration APIs > > >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > >>> DMA acceleration is per queue basis. DPDK applications need to > > >>> explicitly > > >> decide whether a virtqueue needs DMA acceleration and which DMA > > >> channel to use. In addition, a DMA channel is dedicated to a > > >> virtqueue and a DMA channel cannot be bound to multiple virtqueues a= t > > >> the same time. To enable DMA acceleration for a virtqueue, DPDK > > >> applications need to implement DMA operation callbacks for a specifi= c > > >> DMA type (e.g. CBDMA) first, then register the callbacks to the vhos= t > > >> library and bind a DMA channel to a virtqueue, and finally use the > > >> new async API to perform data-path operations on the virtqueue. > > >>> The definitions of registration API are shown below: > > >>> int rte_vhost_async_channel_register(int vid, uint16_t queue_id, > > >>> struct rte_vdma_device_ops > > >>> *ops); > > >>> > > >>> int rte_vhost_async_channel_unregister(int vid, uint16_t queue_id); > > >> > > >> We already have multiple DMA implementation over raw dev. > > >> Why not make a new dmadev class for DMA acceleration and use it by > > >> virtio and any other clients? > > > > > > I believe it doesn't conflict. The purpose of this RFC is to create a= n async > > data path in vhost-user and provide a way for applications to work with= this > > new path. dmadev is another topic which could be discussed separately. = If > > we do have the dmadev available in the future, this vhost async data pa= th > > could certainly be backed by the new dma abstraction without major > > interface change. > > > > Maybe that one advantage of a dmadev class is that it would be easier a= nd > > more transparent for the application to consume. > > > > The application would register some DMA devices, pass them to the Vhost > > library, and then rte_vhost_submit_enqueue_burst and > > rte_vhost_poll_enqueue_completed would call the dmadev callbacks direct= ly. > > > > Do you think that could work? > > > Yes, this is a workable model. As I said in previous reply, I have no obj= ection to make the dmadev. However, what we currently want to do is creatin= g the async data path for vhost, and we actually have no preference to the = underlying DMA device model. I believe our current design of the API proto = type /data structures are quite common for various DMA acceleration solutio= ns and there is no blocker for any new DMA device to adapt to these APIs or= extend to a new one. IMO, as a driver writer, we should not be writing TWO DMA driver. One for vhost and other one for rawdev. If vhost is the first consumer of DMA needed then I think, it make sense to add dmadev first. The rawdev DMA driver to dmadev DMA driver conversion will be the driver owner job. I think, it makes sense to define the dmadev API and then costume by virtio to avoid integration issues. > > Thanks, > > Patrick >