From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 88F50A034E; Wed, 9 Feb 2022 05:49:03 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 7510F410FD; Wed, 9 Feb 2022 05:49:03 +0100 (CET) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 7C14A410F3 for ; Wed, 9 Feb 2022 05:49:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1644382142; x=1675918142; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1GJoQF7AVMAs09TMhpRmrvbg7rFgpVhau50RrRHZmRY=; b=hl58kY1Li26EBwQtttQxuWqBOYlBiIfBOXjhJnNwgc8BglcBXsnLUaP5 uO7fjOpcpza4oZGrKr5VXUgUpdQpB7A5GlYyztScReONHifwLGyGZwFM+ sSSw3P928ax96mZ4dz9xdon0Ltdd/boV3pyYKTQ1q2zKh6hUIsYF5jVDi 5dcWZxdjro6y5/2AcTY0T6oKAqKtJmza69isrd3cKdBrt9x/vyx/rmdol wW+rGouY3ELXyY6sA729qaAYRITRxHTIr9GeqrG16w/q9e0JpTXjB+MUh tfta1UaD4eIUf4WvSF42icnNq6EqbN7kro1Ajrxj6dDvLk5ikmR9pYVjL A==; X-IronPort-AV: E=McAfee;i="6200,9189,10252"; a="309861704" X-IronPort-AV: E=Sophos;i="5.88,354,1635231600"; d="scan'208";a="309861704" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Feb 2022 20:49:01 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,354,1635231600"; d="scan'208";a="540926759" Received: from npgdpdkvirtiojiayuhu117.sh.intel.com ([10.67.119.202]) by orsmga008.jf.intel.com with ESMTP; 08 Feb 2022 20:48:59 -0800 From: Jiayu Hu To: dev@dpdk.org Cc: maxime.coquelin@redhat.com, i.maximets@ovn.org, chenbo.xia@intel.com, xuan.ding@intel.com, cheng1.jiang@intel.com, liangma@liangbit.com, Jiayu Hu Subject: [PATCH v4 0/1] integrate dmadev in vhost Date: Wed, 9 Feb 2022 07:51:44 -0500 Message-Id: <20220209125145.1918050-1-jiayu.hu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220208104031.1885640-2-jiayu.hu@intel.com> References: <20220208104031.1885640-2-jiayu.hu@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Since dmadev is introduced in 21.11, to avoid the overhead of vhost DMA abstraction layer and simplify application logics, this patch integrates dmadev in vhost. To enable the flexibility of using DMA devices in different function modules, not limited in vhost, vhost doesn't manage DMA devices. Applications, like OVS, need to manage and configure DMA devices and tell vhost what DMA device to use in every dataplane function call. In addition, vhost supports M:N mapping between vrings and DMA virtual channels. Specifically, one vring can use multiple different DMA channels and one DMA channel can be shared by multiple vrings at the same time. The reason of enabling one vring to use multiple DMA channels is that it's possible that more than one dataplane threads enqueue packets to the same vring with their own DMA virtual channels. Besides, the number of DMA devices is limited. For the purpose of scaling, it's necessary to support sharing DMA channels among vrings. As only enqueue path is enabled DMA acceleration, the new dataplane functions are like: 1). rte_vhost_submit_enqueue_burst(vid, queue_id, pkts, count, dma_id, dma_vchan): Get descriptors and submit copies to DMA virtual channel for the packets that need to be send to VM. 2). rte_vhost_poll_enqueue_completed(vid, queue_id, pkts, count, dma_id, dma_vchan): Check completed DMA copies from the given DMA virtual channel and write back corresponding descriptors to vring. OVS needs to call rte_vhost_poll_enqueue_completed to clean in-flight copies on previous call and it can be called inside rxq_recv function, so that it doesn't require big change in OVS datapath. For example: netdev_dpdk_vhost_rxq_recv() { ... qid = rxq->queue_id * VIRTIO_QNUM + VIRTIO_RXQ; rte_vhost_poll_enqueue_completed(vid, qid, ...); } Change log ========== v3 -> v4: - fix coding style issues - optmize and shorter log - optimize doc v2 -> v3: - remove SW fallback - remove middle-packet dma submit - refactor rte_async_dma_configure() and remove poll_factor - introduce vhost_async_dma_transfer_one() - rename rte_vhost_iov_iter and rte_vhost_iovec and place them in vhost.h - refactor LOG format - print error log for rte_dma_copy() failure with avoiding log flood - avoid log flood for rte_dma_completed() failure - fix some typo and update comment and doc v1 -> v2: - add SW fallback if rte_dma_copy() reports error - print error if rte_dma_completed() reports error - add poll_factor while call rte_dma_completed() for scatter-gaher packets - use trylock instead of lock in rte_vhost_poll_enqueue_completed() - check if dma_id and vchan_id valid - input dma_id in rte_vhost_async_dma_configure() - remove useless code, brace and hardcode in vhost example - redefine MAX_VHOST_DEVICE to RTE_MAX_VHOST_DEVICE - update doc and comments rfc -> v1: - remove useless code - support dynamic DMA vchannel ring size (rte_vhost_async_dma_configure) - fix several bugs - fix typo and coding style issues - replace "while" with "for" - update programmer guide - support share dma among vhost in vhost example - remove "--dma-type" in vhost example Jiayu Hu (1): vhost: integrate dmadev in asynchronous data-path doc/guides/prog_guide/vhost_lib.rst | 100 +++++----- examples/vhost/Makefile | 2 +- examples/vhost/ioat.c | 218 ---------------------- examples/vhost/ioat.h | 63 ------- examples/vhost/main.c | 253 ++++++++++++++++++++----- examples/vhost/main.h | 11 ++ examples/vhost/meson.build | 6 +- lib/vhost/meson.build | 2 +- lib/vhost/rte_vhost.h | 2 + lib/vhost/rte_vhost_async.h | 145 ++++----------- lib/vhost/version.map | 3 + lib/vhost/vhost.c | 121 ++++++++---- lib/vhost/vhost.h | 85 ++++++++- lib/vhost/virtio_net.c | 277 ++++++++++++++++++++++------ 14 files changed, 693 insertions(+), 595 deletions(-) delete mode 100644 examples/vhost/ioat.c delete mode 100644 examples/vhost/ioat.h -- 2.25.1