From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5AA79A0588; Tue, 21 Apr 2020 11:06:05 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 25A9B1D5E1; Tue, 21 Apr 2020 11:06:04 +0200 (CEST) Received: from mail-io1-f68.google.com (mail-io1-f68.google.com [209.85.166.68]) by dpdk.org (Postfix) with ESMTP id BC6921D150 for ; Tue, 21 Apr 2020 11:06:02 +0200 (CEST) Received: by mail-io1-f68.google.com with SMTP id o127so14259867iof.0 for ; Tue, 21 Apr 2020 02:06:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=VBGDMLNVz7CzeErl/E0gugHyvaFNyUilBk8ADuPwKvE=; b=cTZYUuu0zrWAQsuinvSFrWUCW64eDwGrHZw5glF9IZlOulP9O1hfE83E+abEPUnnth MTe28915Wn4jr94tK/YH48GyCk+fnRIYNHsTH0OJbpw1fZcmKMDuA/n+p9UvvJ0BlF1g catSdwdM1z4Mb0urARBZzPaKXrEVAMlYDaKO4oPiS2VBYHfbhaPARYM/ZufEr8RUFZdJ PkOAoQI7A06U+frN0ORI261XcGc7TtVg57W4IYKkn2t52BSu0pBV+iVCOPsEluZiAkCA 78OYnpg8eUVr4sZ3KasXl4Wnf9rAz1BR3xE0MKeozx8/g1kv3Lf0tSmoZ5vaJvh33Vii D8Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=VBGDMLNVz7CzeErl/E0gugHyvaFNyUilBk8ADuPwKvE=; b=Y2tZAPE6FwKToImraQB/YlTPnTbRhvyO1WhxlLwXLQ64eBTtI5k3NwPpZcPuV8U9ZE 4z3sGS1RZFMlot3meZsOcGRw4ywswPqj6gRxGOtg+N8gM0h0H4m+A5sc93Gu8AC32C3C zK3HECXjfNiiMUqa+fhxGBXHMjaPyGKeXNVd34xIsWgmwU5TnHTFx1eyTdHLuJ87gdEp SfM+k8mzczunF3rogthN4TD7qukvDS9ISNVQAC2mT/yFK0JT2QvNtVsKjYfBILVQFJ4V L76mdD9b9IUTvEE+T5pj/8pSDLeKOsczmB5XqpsFJtlfvBCgndpJWwXRtpZ41s8eyyLu 5mcg== X-Gm-Message-State: AGi0PuZtwNNQ3/Ln4XyzFVwrjVAYsWpvefXYpN278pEyriFwqNZsvK2V pgxLbKIaSlRH8BTnVTrtJOO/gdbQoMfaM9t//l4= X-Google-Smtp-Source: APiQypKPs5PpfNcBDkuxA37vvgvJ3WPUrY4Bqdtp3g3KATigCvqUdZIVZy2FPpyrxR7Zw+83Iv6aQNFfyeCSyzKqvjg= X-Received: by 2002:a05:6638:5ad:: with SMTP id b13mr18938596jar.113.1587459961933; Tue, 21 Apr 2020 02:06:01 -0700 (PDT) MIME-Version: 1.0 References: <89B17B9B05A1964E8D40D6090018F28151277ADF@SHSMSX107.ccr.corp.intel.com> <89B17B9B05A1964E8D40D6090018F28151277C5C@SHSMSX107.ccr.corp.intel.com> <8e4691c5-3e58-fb5d-f1f5-6b9c994c3949@redhat.com> <89B17B9B05A1964E8D40D6090018F28151277D4F@SHSMSX107.ccr.corp.intel.com> <0aaa4b51-f705-1dc4-db60-00e9084db2ea@redhat.com> <89B17B9B05A1964E8D40D6090018F2815127AE8D@SHSMSX107.ccr.corp.intel.com> In-Reply-To: From: Jerin Jacob Date: Tue, 21 Apr 2020 14:35:45 +0530 Message-ID: To: "Liang, Cunming" Cc: "Fu, Patrick" , Maxime Coquelin , "dev@dpdk.org" , "Ye, Xiaolong" , "Hu, Jiayu" , "Wang, Zhihong" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost with DMA Engines X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Tue, Apr 21, 2020 at 2:00 PM Liang, Cunming wr= ote: > > Hi Jerin, > > > -----Original Message----- > > From: Jerin Jacob > > Sent: Tuesday, April 21, 2020 2:04 PM > > To: Fu, Patrick > > Cc: Maxime Coquelin ; Liang, Cunming > > ; dev@dpdk.org; Ye, Xiaolong > > ; Hu, Jiayu ; Wang, Zhihong > > > > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK vHost= with > > DMA Engines > > > > On Tue, Apr 21, 2020 at 8:14 AM Fu, Patrick wrot= e: > > > > > > Hi Jerin > > > > Hi Patrick, > > > > > > > > > -----Original Message----- > > > > From: Jerin Jacob > > > > Sent: Monday, April 20, 2020 8:15 PM > > > > To: Maxime Coquelin > > > > Cc: Liang, Cunming ; Fu, Patrick > > > > ; dev@dpdk.org; Ye, Xiaolong > > > > ; Hu, Jiayu ; Wang, > > > > Zhihong > > > > Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement for DPDK > > > > vHost with DMA Engines > > > > > > > > On Mon, Apr 20, 2020 at 5:40 PM Maxime Coquelin > > > > wrote: > > > > > > > > > > > > > > > > > > > > On 4/20/20 2:08 PM, Jerin Jacob wrote: > > > > > > On Mon, Apr 20, 2020 at 5:14 PM Maxime Coquelin > > > > > > wrote: > > > > > >> > > > > > >> > > > > > >> > > > > > >> On 4/20/20 1:13 PM, Jerin Jacob wrote: > > > > > >>> On Mon, Apr 20, 2020 at 1:29 PM Liang, Cunming > > > > wrote: > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>>> -----Original Message----- > > > > > >>>>> From: Jerin Jacob > > > > > >>>>> Sent: Friday, April 17, 2020 5:55 PM > > > > > >>>>> To: Fu, Patrick > > > > > >>>>> Cc: Maxime Coquelin ; > > > > dev@dpdk.org; > > > > > >>>>> Ye, Xiaolong ; Hu, Jiayu > > > > > >>>>> ; Wang, Zhihong > > > > > >>>>> ; Liang, Cunming > > > > > >>>>> > > > > > >>>>> Subject: Re: [dpdk-dev] [RFC] Accelerating Data Movement fo= r > > > > > >>>>> DPDK vHost with DMA Engines > > > > > >>>>> > > > > > >>>>> On Fri, Apr 17, 2020 at 2:56 PM Fu, Patrick > > > > > >>>>> > > > > wrote: > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>> [...] > > > > > >>>>>>>> > > > > > >>>>>>>> I believe it doesn't conflict. The purpose of this RFC i= s > > > > > >>>>>>>> to create an async > > > > > >>>>>>> data path in vhost-user and provide a way for application= s > > > > > >>>>>>> to work with this new path. dmadev is another topic which > > > > > >>>>>>> could be discussed separately. If we do have the dmadev > > > > > >>>>>>> available in the future, this vhost async data path could > > > > > >>>>>>> certainly be backed by the new dma abstraction without > > > > > >>>>>>> major interface > > > > change. > > > > > >>>>>>> > > > > > >>>>>>> Maybe that one advantage of a dmadev class is that it > > > > > >>>>>>> would be easier and more transparent for the application = to > > consume. > > > > > >>>>>>> > > > > > >>>>>>> The application would register some DMA devices, pass the= m > > > > > >>>>>>> to the Vhost library, and then > > > > > >>>>>>> rte_vhost_submit_enqueue_burst and > > > > > >>>>>>> rte_vhost_poll_enqueue_completed would call the dmadev > > > > callbacks directly. > > > > > >>>>>>> > > > > > >>>>>>> Do you think that could work? > > > > > >>>>>>> > > > > > >>>>>> Yes, this is a workable model. As I said in previous reply= , > > > > > >>>>>> I have no objection to > > > > > >>>>> make the dmadev. However, what we currently want to do is > > > > > >>>>> creating the async data path for vhost, and we actually hav= e > > > > > >>>>> no preference to the underlying DMA device model. I believe > > > > > >>>>> our current design of the API proto type /data structures > > > > > >>>>> are quite common for various DMA acceleration solutions and > > > > > >>>>> there is no > > > > blocker for any new DMA device to adapt to these APIs or extend to = a > > > > new one. > > > > > >>>>> > > > > > >>>>> IMO, as a driver writer, we should not be writing TWO DMA > > > > > >>>>> driver. One for vhost and other one for rawdev. > > > > > >>>> It's the most simplest case if statically 1:1 mapping driver > > > > > >>>> (e.g. {port, > > > > queue}) to a vhost session {vid, qid}. However, it's not enough > > > > scalable to integrate device model with vhost library. There're a > > > > few intentions belong to app logic rather than driver, e.g. 1:N loa= d > > > > balancing, various device type usages (e.g. vhost zcopy via ethdev)= and etc. > > > > > >>> > > > > > >>> > > > > > >>> Before moving to reply to comments, Which DMA engine you are > > > > > >>> planning to integrate with vHOST? > > > > > >>> Is is ioat? if not ioat(drivers/raw/ioat/), How do you think, > > > > > >>> how we can integrate this IOAT DMA engine to vHOST as a use c= ase? > > > > > >>> > > > > > >> > > > > > >> I guess it could be done in the vhost example. > > > > > > > > > > > > > > > > > > Could not see any reference to DMA in examples/vhost* > > > > > > > > > > > > > > > > That's because we are discussing the API to introduce DMA support > > > > > in this exact mail thread, nothing has been merged yet. > > > > > > > > Some confusion here. Original question was, # This is an RFC for DM= A > > > > support in vHOST # What is the underneath DMA engine planned for > > > > hooking to vHOST async API as a "implementation" for this RFC? > > > > # If it ioat, How does the integration work with ioat exiting > > > > rawdriver and new API? > > > > # if it not ioat, What it takes to add support ioat based DMA engin= e > > > > to vHOST aysnc API > > > > > > > It most likely that IOAT could be leveraged as the first demonstratio= n on the > > async DMA acceleration for vHOST. However, this is neither a limitation= nor do we > > design this RFC specifically for IOAT. > > > With current RFC design, we will need applications to implement callb= acks > > (which will call into the IOAT pmd in IOAT case) that can work with vHo= st async > > path. > > > > Then it would be calling some PMD specific APIs for dpaa2_qdma, octeont= x2_dma, > > ioat and there will issue with integrating DMA consumer as vHOST and a= nother > > consumer together. > The main effort is to intro async-mode API for vhost allowing external ho= oks for raw buffer (VM and/or host app) access regardless of virtio ring la= yout. > It never forces ops to leverage DMA device, think about rxtx_callback of = ethdev. That's pretty much app's or helper library's flavor of the ops. > If not comfortable to demo ops with dma, that's fine for us to focus on C= PU only as a hook provider in the sample, and omit 'w/ DMA engine' from RFC= . See below, > > The correct approach is to create a new class for dma like Linux and vH= OST > > consume as a client so that integration aspects are intact. > I'm curious what's the usages when dpaa2_qdma, octeontx2_dma, ioat being = introduce into raw device, why these usage don=E2=80=99t have excuse, but y= ou believe vhost has. This is the first time, a DPDK subsystem tries to use DMA. A subsystem to subsystem communication should not be through private PMD API. > They exists for a while as a raw device. If it's do necessary, they've al= ready built a class... > As you said, vhost is one of the client to consume but not owns device cl= ass, moreover these raw device is not the only 'server' to vhost. > For your concern, it worth a separate conversation but not limited for vh= ost case. Yes. It is not limited to vhost and any DPDK subsystem uses the DMA HW it should be through normative DPDK API. What you are proposing is the purpose of the demo it is OK. In the real-world, it can not work if they're more than one DMA consumer in the system until unless if we standardize the DMA usage in client/server model. Semantically, DMA API should allow - Allow registering DMA engines, it can be ioat, dpaa2_qdma, octeontx2_dma or CPU as the fallback - The client(vhost API) can request for session - And use the DMA service using created session/channel. If you do this, you don't need any API in vhost, it can use DMA API from dmadev and platform can choose the correct DMA engine instead of the application selects the DMA en= gine. I would like to stop here. If Virtio and DPDK maintainers are OK with the RFC scheme then no issues from the side as I don't have time. > > Thanks, > Steve > > > > > > > > > > > > > > > > Thanks, > > > > > > Patrick > > > > > >