From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A0F5DA0583; Thu, 19 Mar 2020 08:33:46 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8DEEF1515; Thu, 19 Mar 2020 08:33:45 +0100 (CET) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by dpdk.org (Postfix) with ESMTP id AB7BAF94 for ; Thu, 19 Mar 2020 08:33:43 +0100 (CET) IronPort-SDR: iUpxa5wk5O0E+pBN419tsEVC8L1L5GBX5ewDDgLgmOXvG/cjrFRAJtR0dxGvsUepMyUB/xQnVd u2nenNrUNJxQ== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Mar 2020 00:33:42 -0700 IronPort-SDR: 99X3qfa5Qay6KnwzMtg0BfTlsncNuXP9UBqX+u3EWOQX5odOZ0LdPmMSpoIcX1wGiCVAiCECT3 IInJpO1BsLrg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.70,570,1574150400"; d="scan'208";a="444468270" Received: from fmsmsx108.amr.corp.intel.com ([10.18.124.206]) by fmsmga005.fm.intel.com with ESMTP; 19 Mar 2020 00:33:42 -0700 Received: from shsmsx602.ccr.corp.intel.com (10.109.6.142) by FMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 19 Mar 2020 00:33:42 -0700 Received: from shsmsx602.ccr.corp.intel.com (10.109.6.142) by SHSMSX602.ccr.corp.intel.com (10.109.6.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 19 Mar 2020 15:33:39 +0800 Received: from shsmsx602.ccr.corp.intel.com ([10.109.6.142]) by SHSMSX602.ccr.corp.intel.com ([10.109.6.142]) with mapi id 15.01.1713.004; Thu, 19 Mar 2020 15:33:39 +0800 From: "Hu, Jiayu" To: Maxime Coquelin , "dev@dpdk.org" CC: "Ye, Xiaolong" , "Wang, Zhihong" Thread-Topic: [PATCH 0/4] Support DMA-accelerated Tx operations for vhost-user PMD Thread-Index: AQHV/AXRSj5F1TraekipaLGRDPhWjKhMBbGAgAN5u9A= Date: Thu, 19 Mar 2020 07:33:39 +0000 Message-ID: <33221483053a41e8bd8d4bd0cb582634@intel.com> References: <1584436885-18651-1-git-send-email-jiayu.hu@intel.com> <370798e0-b006-4a33-d8d9-1aea7bf4af49@redhat.com> In-Reply-To: <370798e0-b006-4a33-d8d9-1aea7bf4af49@redhat.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows x-originating-ip: [10.239.127.36] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH 0/4] Support DMA-accelerated Tx operations for vhost-user PMD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Maxime, Thanks for your comments. Replies are inline. > -----Original Message----- > From: Maxime Coquelin > Sent: Tuesday, March 17, 2020 5:54 PM > To: Hu, Jiayu ; dev@dpdk.org > Cc: Ye, Xiaolong ; Wang, Zhihong > > Subject: Re: [PATCH 0/4] Support DMA-accelerated Tx operations for vhost- > user PMD >=20 > Hi Jiayu, >=20 > On 3/17/20 10:21 AM, Jiayu Hu wrote: > > In vhost-user PMD's Tx operations, where data movement is heavily > involved, > > performing large memory copies usually takes up a major part of CPU > cycles > > and becomes the hot spot. To offload expensive memory operations from > the > > CPU, this patch set proposes to leverage DMA engines, e.g., I/OAT, a DM= A > > engine in the Intel's processor, to accelerate large copies for vhost-u= ser. > > > > Large copies are offloaded from the CPU to the DMA in an asynchronous > > manner. The CPU just submits copy jobs to the DMA but without waiting > > for its copy completion. Thus, there is no CPU intervention during data > > transfer; we can save precious CPU cycles and improve the overall > > throughput for vhost-user PMD based applications, like OVS. During > > packet transmission, it offloads large copies to the DMA and performs > > small copies by the CPU, due to startup overheads associated with the D= MA. > > > > vhost-user PMD is able to support various DMA engines, but it just > > supports I/OAT devices currently. In addition, I/OAT acceleration is on= ly > > enabled for Tx operations of split rings. Users can explicitly assign a > > I/OAT device to a queue by the parameter 'dmas'. However, one I/OAT > device > > can only be used by one queue, and a queue can use one I/OAT device at = a > > time. > > > > We measure the performance in testpmd. With 1024 bytes packets, > compared > > with the original SW data path, DMA-enabled vhost-user PMD can improve > > the throughput around 20%~30% in the VM2VM and PVP cases. > Furthermore, > > with larger packets, the throughput improvement will be higher. >=20 >=20 > I'm not sure it should be done like that for several reasons. >=20 > First, it seems really complex for the user to get the command line > right. There is no mention in the doc patch on how to bind the DMAs to > the DPDK application. Are all the DMAs on the system capable of doing > it? DMA engines in Intel CPU are able to move data within system memory. Currently, we have I/OAT and we will have DSA in the future. > I think it should be made transparent to the user, who should not have > to specify the DMA device address in command line. The user should just > pass a devarg specifying he wants to use DMAs, if available. How do you think of replacing DMA address with specific DMA capabilities, l= ike "dmas=3D[txq0@DMACOPY]". As I/OAT only supports data movement, we can just provide basic DMA copy ability now. But when there are more DMA device= s, we can add capabilities in devargs later. >=20 > Second, it looks too much vendor-specific. IMHO, we should have a DMA > framework, so that the driver can request DMA channels based on > capabilities. We only have one DMA engine, I/OAT, in DPDK, and it is implemented as a rawdev. IMO, it will be very hard to provide a generic DMA abstraction currently. In addition, I/OAT specific API is called inside vhost-user PMD, we can replace these function calls when we have a DMA framework in the future. Users are unaware of the changes. Does it make sense to you? >=20 > Also, I don't think implementing ring processing in the Vhost PMD is > welcome, Vhost PMD should just be a wrapper for the Vhost library. Doing > that in Vhost PMD causes code duplication, and will be a maintenance > burden on the long run. >=20 > As IOAT is a kind of acceleration, why not implement it through the vDPA > framework? vDPA framework should be extended to support this kind of > acceleration which requires some CPU processing, as opposed to full > offload of the ring processing as it only supports today. The main reason of implementing data path in vhost PMD is to avoid impactin= g SW data path in vhost library. Even if we implement it as an instance of vD= PA, we also have to implement data path in a new vdev PMD, as DMA just accelera= tes memory copy and all ring operations have to be done by the CPU. There is st= ill the code duplication issue. Thanks, Jiayu >=20 > Kind regards, > Maxime >=20 > > Jiayu Hu (4): > > vhost: populate guest memory for DMA-accelerated vhost-user > > net/vhost: setup vrings for DMA-accelerated datapath > > net/vhost: leverage DMA engines to accelerate Tx operations > > doc: add I/OAT acceleration support for vhost-user PMD > > > > doc/guides/nics/vhost.rst | 14 + > > drivers/Makefile | 2 +- > > drivers/net/vhost/Makefile | 6 +- > > drivers/net/vhost/internal.h | 160 +++++++ > > drivers/net/vhost/meson.build | 5 +- > > drivers/net/vhost/rte_eth_vhost.c | 308 +++++++++++--- > > drivers/net/vhost/virtio_net.c | 861 > ++++++++++++++++++++++++++++++++++++++ > > drivers/net/vhost/virtio_net.h | 288 +++++++++++++ > > lib/librte_vhost/rte_vhost.h | 1 + > > lib/librte_vhost/socket.c | 20 + > > lib/librte_vhost/vhost.h | 2 + > > lib/librte_vhost/vhost_user.c | 3 +- > > 12 files changed, 1597 insertions(+), 73 deletions(-) > > create mode 100644 drivers/net/vhost/internal.h > > create mode 100644 drivers/net/vhost/virtio_net.c > > create mode 100644 drivers/net/vhost/virtio_net.h > >