From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 034AEA0553; Fri, 10 Jun 2022 17:35:54 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E9D39410F2; Fri, 10 Jun 2022 17:35:53 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id ECE0140689; Fri, 10 Jun 2022 17:35:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654875351; x=1686411351; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=t7aEiPchwp9ML5i79nYcdP8TJKXujf6cI8W3AplGlkQ=; b=BS6/JRpOCjoRyJWqGdvppP+Fr+6JWmWdJdudLVnPM7PbwKHGqiu46n/r atcS1E/LmxCvO+tHnadRySmqmR5FDC7d/2Vd/N1gcMLFwpznAi9QDeFes 3uf7OM5yreTmo1YcFeMB+rwfOV6HauuofvvN1/LzuucUjtE5umeTyyaU1 BY/VN31CoPoyugpBkECghfVPHSHQBg9Q87hzbKdsw/t9g0+4HMuas0VNO bYHRGEwix8+RSu+S3Qa+6Rri7+HCGfbGgN8gLEtep7/ubQn41oQLfehIG +4HaBy/0YEFaCk83GbxHSDxZ4zelOZZ/52L6V9uaOqbfgzasswtO4Bylu A==; X-IronPort-AV: E=McAfee;i="6400,9594,10374"; a="278796028" X-IronPort-AV: E=Sophos;i="5.91,290,1647327600"; d="scan'208";a="278796028" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2022 08:35:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,290,1647327600"; d="scan'208";a="610761212" Received: from silpixa00401385.ir.intel.com (HELO silpixa00401385.ger.corp.intel.com.) ([10.237.223.181]) by orsmga008.jf.intel.com with ESMTP; 10 Jun 2022 08:35:44 -0700 From: Bruce Richardson To: dev@dpdk.org Cc: Maxime Coquelin , Chenbo Xia , Bruce Richardson , stable@dpdk.org Subject: [PATCH v3 1/4] doc/howto: rework section on virtio-user as exception path Date: Fri, 10 Jun 2022 16:35:32 +0100 Message-Id: <20220610153535.659076-2-bruce.richardson@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220610153535.659076-1-bruce.richardson@intel.com> References: <20220527162659.129022-1-bruce.richardson@intel.com> <20220610153535.659076-1-bruce.richardson@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This patch extensively reworks the howto guide on using virtio-user for exception packets. Changes include: * rename "exceptional path" to "exception path" * remove references to uio and just reference vfio-pci * simplify testpmd command-lines, giving a basic usage example first before adding on detail about checksum or TSO parameters * give a complete working example showing traffic flowing through the whole system from a testpmd loopback using the created TAP netdev * replace use of "ifconfig" with Linux standard "ip" command * other general rewording. CC: stable@dpdk.org Signed-off-by: Bruce Richardson --- V3: * fix error reported by Chenbo on review. * add stable on CC, since this rework could be applicable for older releases too, if desired for backport. --- .../howto/virtio_user_as_exceptional_path.rst | 159 +++++++++++------- 1 file changed, 100 insertions(+), 59 deletions(-) diff --git a/doc/guides/howto/virtio_user_as_exceptional_path.rst b/doc/guides/howto/virtio_user_as_exceptional_path.rst index ec021af399..3d3d1b1219 100644 --- a/doc/guides/howto/virtio_user_as_exceptional_path.rst +++ b/doc/guides/howto/virtio_user_as_exceptional_path.rst @@ -3,116 +3,157 @@ .. _virtio_user_as_exceptional_path: -Virtio_user as Exceptional Path -=============================== +Virtio_user as Exception Path +============================= -The virtual device, virtio-user, was originally introduced with vhost-user -backend, as a high performance solution for IPC (Inter-Process Communication) +.. note:: + + This solution is only applicable to Linux systems. + +The virtual device, virtio-user, was originally introduced with the vhost-user +backend as a high performance solution for IPC (Inter-Process Communication) and user space container networking. -Virtio_user with vhost-kernel backend is a solution for exceptional path, -such as KNI which exchanges packets with kernel networking stack. This -solution is very promising in: +Beyond this originally intended use, virtio-user can be used in conjunction with the vhost-kernel +backend as a solution for dealing with exception path packets which need to be injected into the +Linux kernel for processing there. +In this regard, virtio-user and vhost in kernel space are an alternative to DPDK KNI for +transferring packets between a DPDK packet processing application and the kernel stack. + +This solution has a number of advantages over alternatives such as KNI: * Maintenance All kernel modules needed by this solution, vhost and vhost-net (kernel), - are upstreamed and extensively used kernel module. + are upstreamed and extensively used. * Features - vhost-net is born to be a networking solution, which has lots of networking - related features, like multi queue, tso, multi-seg mbuf, etc. + vhost-net is designed to be a networking solution, and, as such, has lots of networking + related features, such as multi queue support, TSO, multi-segment buffer support, etc. * Performance - similar to KNI, this solution would use one or more kthreads to - send/receive packets to/from user space DPDK applications, which has little - impact on user space polling thread (except that it might enter into kernel - space to wake up those kthreads if necessary). + similar to KNI, this solution would uses one or more kthreads to + send/receive packets to/from user space DPDK applications, which minimises the impact + on the polling DPDK threads. -The overview of an application using virtio-user as exceptional path is shown +The overview of an application using virtio-user as exception path is shown in :numref:`figure_virtio_user_as_exceptional_path`. .. _figure_virtio_user_as_exceptional_path: .. figure:: img/virtio_user_as_exceptional_path.* - Overview of a DPDK app using virtio-user as exceptional path + Overview of a DPDK app using virtio-user as exception path + +Example Usage With Testpmd +--------------------------- -Sample Usage ------------- +.. note:: + + These instruction assume that the vhost/vhost-net kernel modules are available and have already + been loaded into the running kernel. + It also assumes that the DPDK virtio driver has not been disabled in the DPDK build. -As a prerequisite, the vhost/vhost-net kernel CONFIG should be chosen before -compiling the kernel and those kernel modules should be inserted. +To run a simple test of virtio-user as exception path using testpmd: -#. Compile DPDK and bind a physical NIC to igb_uio/uio_pci_generic/vfio-pci. +#. Compile DPDK and bind a NIC to vfio-pci as documented in :ref:`linux_gsg_linux_drivers`. - This physical NIC is for communicating with outside. + This physical NIC is for communicating with the outside world, + and serves as a packet source in this example. -#. Run testpmd. +#. Run testpmd to forward packets from NIC to kernel, + passing in a suitable list of logical cores to run on (``-l`` parameter), + and optionally the PCI address of the physical NIC to use (``-a`` parameter). + The virtio-user device for interfacing to the kernel is specified via a ``-vdev`` argument, + taking the parameters described below. .. code-block:: console - $(testpmd) -l 2-3 -n 4 \ - --vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024 \ - -- -i --tx-offloads=0x0000002c --enable-lro \ - --txd=1024 --rxd=1024 + /path/to/dpdk-testpmd -l -a \ + --vdev=virtio_user0,path=/dev/vhost-net,queues=1,queue_size=1024 - This command runs testpmd with two ports, one physical NIC to communicate - with outside, and one virtio-user to communicate with kernel. + * ``path`` -* ``--enable-lro`` + The path to the kernel vhost-net device. - This is used to negotiate VIRTIO_NET_F_GUEST_TSO4 and - VIRTIO_NET_F_GUEST_TSO6 feature so that large packets from kernel can be - transmitted to DPDK application and further TSOed by physical NIC. + * ``queue_size`` -* ``queue_size`` + 256 by default. To avoid shortage of descriptors, we can increase it to 1024. - 256 by default. To avoid shortage of descriptors, we can increase it to 1024. + * ``queues`` -* ``queues`` + Number of virt-queues. Each queue will be served by a kthread. - Number of multi-queues. Each queue will be served by a kthread. For example: +#. Once testpmd is running, a new network interface - called ``tap0`` by default - + will be present on the system. + This should be configured with an IP address and then enabled for use: .. code-block:: console - $(testpmd) -l 2-3 -n 4 \ - --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 \ - -- -i --tx-offloads=0x0000002c --enable-lro \ - --txq=2 --rxq=2 --txd=1024 --rxd=1024 + ip addr add 192.168.1.1/24 dev tap0 + ip link set dev tap0 up -#. Enable Rx checksum offloads in testpmd: +#. To observe packet forwarding through the kernel, + a second testpmd instance can be run on the system, + taking packets from the kernel using an ``af_packet`` socket on the ``tap0`` interface. - .. code-block:: console + .. code-block:: console - (testpmd) port stop 0 - (testpmd) port config 0 rx_offload tcp_cksum on - (testpmd) port config 0 rx_offload udp_cksum on - (testpmd) port start 0 + /path/to/dpdk-testpmd -l --vdev=net_af_packet0,iface=tap0 --in-memory --no-pci -#. Start testpmd: + When running this instance, + we can use ``--in-memory`` flag to avoid hugepage naming conflicts with the previous instance, + and we also use ``--no-pci`` flag to only use the ``af_packet`` interface for all traffic forwarding. - .. code-block:: console +#. Running traffic into the system through the NIC should see that traffic returned back again, + having been forwarded through both testpmd instances. + This can be confirmed by checking the testpmd statistics on testpmd exit. - (testpmd) start +For more advanced use of virtio-user with testpmd in this scenario, +some other more advanced options may also be used. +For example: -#. Configure IP address and start tap: +* ``--tx-offloads=0x02c`` + + This testpmd option enables TX offloads for UDP and TCP checksum on transmit, + as well as TCP TSO support. + The list of the offload flag values can be seen in header `rte_ethdev.h + `_. + +* ``--enable-lro`` + + This testpmd option is used to negotiate VIRTIO_NET_F_GUEST_TSO4 and + VIRTIO_NET_F_GUEST_TSO6 feature so that large packets from the kernel can be + transmitted to the DPDK application and further TSOed by physical NIC. + If unsupported by the physical NIC, errors may be reported by testpmd with this option. + + +* Enabling Rx checksum offloads for physical port: + + Within testpmd, you can enable and disable offloads on a per-port basis, + rather than enabling them for both ports. + For the physical NIC, it may be desirable to enable checksum offload on packet RX. + This may be done as below, if testpmd is run with ``-i`` flag for interactive mode. .. code-block:: console - ifconfig tap0 1.1.1.1/24 up + testpmd> port stop 0 + testpmd> port config 0 rx_offload tcp_cksum on + testpmd> port config 0 rx_offload udp_cksum on + testpmd> port start 0 -.. note:: +* Multiple queue support - The tap device will be named tap0, tap1, etc, by kernel. + Better performance may be achieved by using multiple queues, + so that multiple kernel threads are handling the traffic on the kernel side. + For example, to use 2 queues on both NIC and virtio ports, + while also enabling TX offloads and LRO support: -Then, all traffic from physical NIC can be forwarded into kernel stack, and all -traffic on the tap0 can be sent out from physical NIC. + .. code-block:: console -Limitations ------------ + /path/to/dpdk-testpmd --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 -- \ + -i --tx-offloads=0x002c --enable-lro --txq=2 --rxq=2 --txd=1024 --rxd=1024 -This solution is only available on Linux systems. -- 2.34.1