[PATCH v3 1/4] doc/howto: rework section on virtio-user as exception path

[PATCH v3 1/4] doc/howto: rework section on virtio-user as exception path

[Bruce Richardson]

This patch extensively reworks the howto guide on using virtio-user for
exception packets. Changes include:

* rename "exceptional path" to "exception path"
* remove references to uio and just reference vfio-pci
* simplify testpmd command-lines, giving a basic usage example first
  before adding on detail about checksum or TSO parameters
* give a complete working example showing traffic flowing through the
  whole system from a testpmd loopback using the created TAP netdev
* replace use of "ifconfig" with Linux standard "ip" command
* other general rewording.

CC: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
V3:
* fix error reported by Chenbo on review.
* add stable on CC, since this rework could be applicable for older
  releases too, if desired for backport.
---
 .../howto/virtio_user_as_exceptional_path.rst | 159 +++++++++++-------
 1 file changed, 100 insertions(+), 59 deletions(-)

diff --git a/doc/guides/howto/virtio_user_as_exceptional_path.rst b/doc/guides/howto/virtio_user_as_exceptional_path.rst
index ec021af399..3d3d1b1219 100644
--- a/doc/guides/howto/virtio_user_as_exceptional_path.rst
+++ b/doc/guides/howto/virtio_user_as_exceptional_path.rst
@@ -3,116 +3,157 @@

 .. _virtio_user_as_exceptional_path:

-Virtio_user as Exceptional Path
-===============================
+Virtio_user as Exception Path
+=============================

-The virtual device, virtio-user, was originally introduced with vhost-user
-backend, as a high performance solution for IPC (Inter-Process Communication)
+.. note::
+
+   This solution is only applicable to Linux systems.
+
+The virtual device, virtio-user, was originally introduced with the vhost-user
+backend as a high performance solution for IPC (Inter-Process Communication)
 and user space container networking.

-Virtio_user with vhost-kernel backend is a solution for exceptional path,
-such as KNI which exchanges packets with kernel networking stack. This
-solution is very promising in:
+Beyond this originally intended use, virtio-user can be used in conjunction with the vhost-kernel
+backend as a solution for dealing with exception path packets which need to be injected into the
+Linux kernel for processing there.
+In this regard, virtio-user and vhost in kernel space are an alternative to DPDK KNI for
+transferring packets between a DPDK packet processing application and the kernel stack.
+
+This solution has a number of advantages over alternatives such as KNI:

 *   Maintenance

     All kernel modules needed by this solution, vhost and vhost-net (kernel),
-    are upstreamed and extensively used kernel module.
+    are upstreamed and extensively used.

 *   Features

-    vhost-net is born to be a networking solution, which has lots of networking
-    related features, like multi queue, tso, multi-seg mbuf, etc.
+    vhost-net is designed to be a networking solution, and, as such, has lots of networking
+    related features, such as multi queue support, TSO, multi-segment buffer support, etc.

 *   Performance

-    similar to KNI, this solution would use one or more kthreads to
-    send/receive packets to/from user space DPDK applications, which has little
-    impact on user space polling thread (except that it might enter into kernel
-    space to wake up those kthreads if necessary).
+    similar to KNI, this solution would uses one or more kthreads to
+    send/receive packets to/from user space DPDK applications, which minimises the impact
+    on the polling DPDK threads.

-The overview of an application using virtio-user as exceptional path is shown
+The overview of an application using virtio-user as exception path is shown
 in :numref:`figure_virtio_user_as_exceptional_path`.

 .. _figure_virtio_user_as_exceptional_path:

 .. figure:: img/virtio_user_as_exceptional_path.*

-   Overview of a DPDK app using virtio-user as exceptional path
+   Overview of a DPDK app using virtio-user as exception path
+

+Example Usage With Testpmd
+---------------------------

-Sample Usage
-------------
+.. note::
+
+   These instruction assume that the vhost/vhost-net kernel modules are available and have already
+   been loaded into the running kernel.
+   It also assumes that the DPDK virtio driver has not been disabled in the DPDK build.

-As a prerequisite, the vhost/vhost-net kernel CONFIG should be chosen before
-compiling the kernel and those kernel modules should be inserted.
+To run a simple test of virtio-user as exception path using testpmd:

-#.  Compile DPDK and bind a physical NIC to igb_uio/uio_pci_generic/vfio-pci.
+#.  Compile DPDK and bind a NIC to vfio-pci as documented in :ref:`linux_gsg_linux_drivers`.

-    This physical NIC is for communicating with outside.
+    This physical NIC is for communicating with the outside world,
+    and serves as a packet source in this example.

-#.  Run testpmd.
+#.  Run testpmd to forward packets from NIC to kernel,
+    passing in a suitable list of logical cores to run on  (``-l`` parameter),
+    and optionally the PCI address of the physical NIC to use (``-a`` parameter).
+    The virtio-user device for interfacing to the kernel is specified via a ``-vdev`` argument,
+    taking the parameters described below.

     .. code-block:: console

-        $(testpmd) -l 2-3 -n 4 \
-		--vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024 \
-		-- -i --tx-offloads=0x0000002c --enable-lro \
-		--txd=1024 --rxd=1024
+        /path/to/dpdk-testpmd -l <cores> -a <pci BDF> \
+            --vdev=virtio_user0,path=/dev/vhost-net,queues=1,queue_size=1024

-    This command runs testpmd with two ports, one physical NIC to communicate
-    with outside, and one virtio-user to communicate with kernel.
+    * ``path``

-* ``--enable-lro``
+       The path to the kernel vhost-net device.

-    This is used to negotiate VIRTIO_NET_F_GUEST_TSO4 and
-    VIRTIO_NET_F_GUEST_TSO6 feature so that large packets from kernel can be
-    transmitted to DPDK application and further TSOed by physical NIC.
+    * ``queue_size``

-* ``queue_size``
+       256 by default. To avoid shortage of descriptors, we can increase it to 1024.

-    256 by default. To avoid shortage of descriptors, we can increase it to 1024.
+    * ``queues``

-* ``queues``
+       Number of virt-queues. Each queue will be served by a kthread.

-    Number of multi-queues. Each queue will be served by a kthread. For example:
+#. Once testpmd is running, a new network interface - called ``tap0`` by default -
+   will be present on the system.
+   This should be configured with an IP address and then enabled for use:

     .. code-block:: console

-        $(testpmd) -l 2-3 -n 4 \
-		--vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 \
-		-- -i --tx-offloads=0x0000002c --enable-lro \
-		--txq=2 --rxq=2 --txd=1024 --rxd=1024
+        ip addr add 192.168.1.1/24 dev tap0
+        ip link set dev tap0 up

-#. Enable Rx checksum offloads in testpmd:
+#. To observe packet forwarding through the kernel,
+   a second testpmd instance can be run on the system,
+   taking packets from the kernel using an ``af_packet`` socket on the ``tap0`` interface.

-    .. code-block:: console
+   .. code-block:: console

-        (testpmd) port stop 0
-        (testpmd) port config 0 rx_offload tcp_cksum on
-        (testpmd) port config 0 rx_offload udp_cksum on
-        (testpmd) port start 0
+       /path/to/dpdk-testpmd -l <cores> --vdev=net_af_packet0,iface=tap0 --in-memory --no-pci

-#. Start testpmd:
+   When running this instance,
+   we can use ``--in-memory`` flag to avoid hugepage naming conflicts with the previous instance,
+   and we also use ``--no-pci`` flag to only use the ``af_packet`` interface for all traffic forwarding.

-    .. code-block:: console
+#. Running traffic into the system through the NIC should see that traffic returned back again,
+   having been forwarded through both testpmd instances.
+   This can be confirmed by checking the testpmd statistics on testpmd exit.

-        (testpmd) start
+For more advanced use of virtio-user with testpmd in this scenario,
+some other more advanced options may also be used.
+For example:

-#.  Configure IP address and start tap:
+* ``--tx-offloads=0x02c``
+
+    This testpmd option enables TX offloads for UDP and TCP checksum on transmit,
+    as well as TCP TSO support.
+    The list of the offload flag values can be seen in header `rte_ethdev.h
+    <https://doc.dpdk.org/api/rte__ethdev_8h.html>`_.
+
+* ``--enable-lro``
+
+    This testpmd option is used to negotiate VIRTIO_NET_F_GUEST_TSO4 and
+    VIRTIO_NET_F_GUEST_TSO6 feature so that large packets from the kernel can be
+    transmitted to the DPDK application and further TSOed by physical NIC.
+    If unsupported by the physical NIC, errors may be reported by testpmd with this option.
+
+
+* Enabling Rx checksum offloads for physical port:
+
+   Within testpmd, you can enable and disable offloads on a per-port basis,
+   rather than enabling them for both ports.
+   For the physical NIC, it may be desirable to enable checksum offload on packet RX.
+   This may be done as below, if testpmd is run with ``-i`` flag for interactive mode.

     .. code-block:: console

-        ifconfig tap0 1.1.1.1/24 up
+        testpmd> port stop 0
+        testpmd> port config 0 rx_offload tcp_cksum on
+        testpmd> port config 0 rx_offload udp_cksum on
+        testpmd> port start 0

-.. note::
+* Multiple queue support

-    The tap device will be named tap0, tap1, etc, by kernel.
+    Better performance may be achieved by using multiple queues,
+    so that multiple kernel threads are handling the traffic on the kernel side.
+    For example, to use 2 queues on both NIC and virtio ports,
+    while also enabling TX offloads and LRO support:

-Then, all traffic from physical NIC can be forwarded into kernel stack, and all
-traffic on the tap0 can be sent out from physical NIC.
+    .. code-block:: console

-Limitations
------------
+        /path/to/dpdk-testpmd --vdev=virtio_user0,path=/dev/vhost-net,queues=2,queue_size=1024 -- \
+            -i --tx-offloads=0x002c --enable-lro --txq=2 --rxq=2 --txd=1024 --rxd=1024

-This solution is only available on Linux systems.
--
2.34.1

RE: [PATCH v3 1/4] doc/howto: rework section on virtio-user as exception path

[Xia, Chenbo]

Hi Bruce,

> -----Original Message-----
> From: Richardson, Bruce <bruce.richardson@intel.com>
> Sent: Friday, June 10, 2022 11:36 PM
> To: dev@dpdk.org
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>; Xia, Chenbo
> <chenbo.xia@intel.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> stable@dpdk.org
> Subject: [PATCH v3 1/4] doc/howto: rework section on virtio-user as
> exception path
> 
> This patch extensively reworks the howto guide on using virtio-user for
> exception packets. Changes include:
> 
> * rename "exceptional path" to "exception path"
> * remove references to uio and just reference vfio-pci
> * simplify testpmd command-lines, giving a basic usage example first
>   before adding on detail about checksum or TSO parameters
> * give a complete working example showing traffic flowing through the
>   whole system from a testpmd loopback using the created TAP netdev
> * replace use of "ifconfig" with Linux standard "ip" command
> * other general rewording.
> 
> CC: stable@dpdk.org

Not sure you want to backport or not as there's no fixes tag.

For this patch:

Acked-by: Chenbo Xia <chenbo.xia@intel.com>

> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> ---
> V3:
> * fix error reported by Chenbo on review.
> * add stable on CC, since this rework could be applicable for older
>   releases too, if desired for backport.
> ---
>  .../howto/virtio_user_as_exceptional_path.rst | 159 +++++++++++-------
>  1 file changed, 100 insertions(+), 59 deletions(-)
> 
> diff --git a/doc/guides/howto/virtio_user_as_exceptional_path.rst
> b/doc/guides/howto/virtio_user_as_exceptional_path.rst
> index ec021af399..3d3d1b1219 100644
> --- a/doc/guides/howto/virtio_user_as_exceptional_path.rst
> +++ b/doc/guides/howto/virtio_user_as_exceptional_path.rst
> @@ -3,116 +3,157 @@
> 
>  .. _virtio_user_as_exceptional_path:
> 
> -Virtio_user as Exceptional Path
> -===============================
> +Virtio_user as Exception Path
> +=============================
> 
> -The virtual device, virtio-user, was originally introduced with vhost-
> user
> -backend, as a high performance solution for IPC (Inter-Process
> Communication)
> +.. note::
> +
> +   This solution is only applicable to Linux systems.
> +
> +The virtual device, virtio-user, was originally introduced with the
> vhost-user
> +backend as a high performance solution for IPC (Inter-Process
> Communication)
>  and user space container networking.
> 
> -Virtio_user with vhost-kernel backend is a solution for exceptional path,
> -such as KNI which exchanges packets with kernel networking stack. This
> -solution is very promising in:
> +Beyond this originally intended use, virtio-user can be used in
> conjunction with the vhost-kernel
> +backend as a solution for dealing with exception path packets which need
> to be injected into the
> +Linux kernel for processing there.
> +In this regard, virtio-user and vhost in kernel space are an alternative
> to DPDK KNI for
> +transferring packets between a DPDK packet processing application and the
> kernel stack.
> +
> +This solution has a number of advantages over alternatives such as KNI:
> 
>  *   Maintenance
> 
>      All kernel modules needed by this solution, vhost and vhost-net
> (kernel),
> -    are upstreamed and extensively used kernel module.
> +    are upstreamed and extensively used.
> 
>  *   Features
> 
> -    vhost-net is born to be a networking solution, which has lots of
> networking
> -    related features, like multi queue, tso, multi-seg mbuf, etc.
> +    vhost-net is designed to be a networking solution, and, as such, has
> lots of networking
> +    related features, such as multi queue support, TSO, multi-segment
> buffer support, etc.
> 
>  *   Performance
> 
> -    similar to KNI, this solution would use one or more kthreads to
> -    send/receive packets to/from user space DPDK applications, which has
> little
> -    impact on user space polling thread (except that it might enter into
> kernel
> -    space to wake up those kthreads if necessary).
> +    similar to KNI, this solution would uses one or more kthreads to
> +    send/receive packets to/from user space DPDK applications, which
> minimises the impact
> +    on the polling DPDK threads.
> 
> -The overview of an application using virtio-user as exceptional path is
> shown
> +The overview of an application using virtio-user as exception path is
> shown
>  in :numref:`figure_virtio_user_as_exceptional_path`.
> 
>  .. _figure_virtio_user_as_exceptional_path:
> 
>  .. figure:: img/virtio_user_as_exceptional_path.*
> 
> -   Overview of a DPDK app using virtio-user as exceptional path
> +   Overview of a DPDK app using virtio-user as exception path
> +
> 
> +Example Usage With Testpmd
> +---------------------------
> 
> -Sample Usage
> -------------
> +.. note::
> +
> +   These instruction assume that the vhost/vhost-net kernel modules are
> available and have already
> +   been loaded into the running kernel.
> +   It also assumes that the DPDK virtio driver has not been disabled in
> the DPDK build.
> 
> -As a prerequisite, the vhost/vhost-net kernel CONFIG should be chosen
> before
> -compiling the kernel and those kernel modules should be inserted.
> +To run a simple test of virtio-user as exception path using testpmd:
> 
> -#.  Compile DPDK and bind a physical NIC to igb_uio/uio_pci_generic/vfio-
> pci.
> +#.  Compile DPDK and bind a NIC to vfio-pci as documented
> in :ref:`linux_gsg_linux_drivers`.
> 
> -    This physical NIC is for communicating with outside.
> +    This physical NIC is for communicating with the outside world,
> +    and serves as a packet source in this example.
> 
> -#.  Run testpmd.
> +#.  Run testpmd to forward packets from NIC to kernel,
> +    passing in a suitable list of logical cores to run on  (``-l``
> parameter),
> +    and optionally the PCI address of the physical NIC to use (``-a``
> parameter).
> +    The virtio-user device for interfacing to the kernel is specified via
> a ``-vdev`` argument,
> +    taking the parameters described below.
> 
>      .. code-block:: console
> 
> -        $(testpmd) -l 2-3 -n 4 \
> -		--vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024 \
> -		-- -i --tx-offloads=0x0000002c --enable-lro \
> -		--txd=1024 --rxd=1024
> +        /path/to/dpdk-testpmd -l <cores> -a <pci BDF> \
> +            --vdev=virtio_user0,path=/dev/vhost-
> net,queues=1,queue_size=1024
> 
> -    This command runs testpmd with two ports, one physical NIC to
> communicate
> -    with outside, and one virtio-user to communicate with kernel.
> +    * ``path``
> 
> -* ``--enable-lro``
> +       The path to the kernel vhost-net device.
> 
> -    This is used to negotiate VIRTIO_NET_F_GUEST_TSO4 and
> -    VIRTIO_NET_F_GUEST_TSO6 feature so that large packets from kernel can
> be
> -    transmitted to DPDK application and further TSOed by physical NIC.
> +    * ``queue_size``
> 
> -* ``queue_size``
> +       256 by default. To avoid shortage of descriptors, we can increase
> it to 1024.
> 
> -    256 by default. To avoid shortage of descriptors, we can increase it
> to 1024.
> +    * ``queues``
> 
> -* ``queues``
> +       Number of virt-queues. Each queue will be served by a kthread.
> 
> -    Number of multi-queues. Each queue will be served by a kthread. For
> example:
> +#. Once testpmd is running, a new network interface - called ``tap0`` by
> default -
> +   will be present on the system.
> +   This should be configured with an IP address and then enabled for use:
> 
>      .. code-block:: console
> 
> -        $(testpmd) -l 2-3 -n 4 \
> -		--vdev=virtio_user0,path=/dev/vhost-
> net,queues=2,queue_size=1024 \
> -		-- -i --tx-offloads=0x0000002c --enable-lro \
> -		--txq=2 --rxq=2 --txd=1024 --rxd=1024
> +        ip addr add 192.168.1.1/24 dev tap0
> +        ip link set dev tap0 up
> 
> -#. Enable Rx checksum offloads in testpmd:
> +#. To observe packet forwarding through the kernel,
> +   a second testpmd instance can be run on the system,
> +   taking packets from the kernel using an ``af_packet`` socket on the
> ``tap0`` interface.
> 
> -    .. code-block:: console
> +   .. code-block:: console
> 
> -        (testpmd) port stop 0
> -        (testpmd) port config 0 rx_offload tcp_cksum on
> -        (testpmd) port config 0 rx_offload udp_cksum on
> -        (testpmd) port start 0
> +       /path/to/dpdk-testpmd -l <cores> --vdev=net_af_packet0,iface=tap0
> --in-memory --no-pci
> 
> -#. Start testpmd:
> +   When running this instance,
> +   we can use ``--in-memory`` flag to avoid hugepage naming conflicts
> with the previous instance,
> +   and we also use ``--no-pci`` flag to only use the ``af_packet``
> interface for all traffic forwarding.
> 
> -    .. code-block:: console
> +#. Running traffic into the system through the NIC should see that
> traffic returned back again,
> +   having been forwarded through both testpmd instances.
> +   This can be confirmed by checking the testpmd statistics on testpmd
> exit.
> 
> -        (testpmd) start
> +For more advanced use of virtio-user with testpmd in this scenario,
> +some other more advanced options may also be used.
> +For example:
> 
> -#.  Configure IP address and start tap:
> +* ``--tx-offloads=0x02c``
> +
> +    This testpmd option enables TX offloads for UDP and TCP checksum on
> transmit,
> +    as well as TCP TSO support.
> +    The list of the offload flag values can be seen in header
> `rte_ethdev.h
> +    <https://doc.dpdk.org/api/rte__ethdev_8h.html>`_.
> +
> +* ``--enable-lro``
> +
> +    This testpmd option is used to negotiate VIRTIO_NET_F_GUEST_TSO4 and
> +    VIRTIO_NET_F_GUEST_TSO6 feature so that large packets from the kernel
> can be
> +    transmitted to the DPDK application and further TSOed by physical NIC.
> +    If unsupported by the physical NIC, errors may be reported by testpmd
> with this option.
> +
> +
> +* Enabling Rx checksum offloads for physical port:
> +
> +   Within testpmd, you can enable and disable offloads on a per-port
> basis,
> +   rather than enabling them for both ports.
> +   For the physical NIC, it may be desirable to enable checksum offload
> on packet RX.
> +   This may be done as below, if testpmd is run with ``-i`` flag for
> interactive mode.
> 
>      .. code-block:: console
> 
> -        ifconfig tap0 1.1.1.1/24 up
> +        testpmd> port stop 0
> +        testpmd> port config 0 rx_offload tcp_cksum on
> +        testpmd> port config 0 rx_offload udp_cksum on
> +        testpmd> port start 0
> 
> -.. note::
> +* Multiple queue support
> 
> -    The tap device will be named tap0, tap1, etc, by kernel.
> +    Better performance may be achieved by using multiple queues,
> +    so that multiple kernel threads are handling the traffic on the
> kernel side.
> +    For example, to use 2 queues on both NIC and virtio ports,
> +    while also enabling TX offloads and LRO support:
> 
> -Then, all traffic from physical NIC can be forwarded into kernel stack,
> and all
> -traffic on the tap0 can be sent out from physical NIC.
> +    .. code-block:: console
> 
> -Limitations
> ------------
> +        /path/to/dpdk-testpmd --vdev=virtio_user0,path=/dev/vhost-
> net,queues=2,queue_size=1024 -- \
> +            -i --tx-offloads=0x002c --enable-lro --txq=2 --rxq=2 --
> txd=1024 --rxd=1024
> 
> -This solution is only available on Linux systems.
> --
> 2.34.1

Re: [PATCH v3 1/4] doc/howto: rework section on virtio-user as exception path

[Maxime Coquelin]

On 6/10/22 17:35, Bruce Richardson wrote:
> This patch extensively reworks the howto guide on using virtio-user for
> exception packets. Changes include:
> 
> * rename "exceptional path" to "exception path"
> * remove references to uio and just reference vfio-pci
> * simplify testpmd command-lines, giving a basic usage example first
>    before adding on detail about checksum or TSO parameters
> * give a complete working example showing traffic flowing through the
>    whole system from a testpmd loopback using the created TAP netdev
> * replace use of "ifconfig" with Linux standard "ip" command
> * other general rewording.
> 
> CC: stable@dpdk.org
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> ---
> V3:
> * fix error reported by Chenbo on review.
> * add stable on CC, since this rework could be applicable for older
>    releases too, if desired for backport.
> ---
>   .../howto/virtio_user_as_exceptional_path.rst | 159 +++++++++++-------
>   1 file changed, 100 insertions(+), 59 deletions(-)
> 

Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [PATCH v3 1/4] doc/howto: rework section on virtio-user as exception path

[Thomas Monjalon]

10/06/2022 17:35, Bruce Richardson:
> This patch extensively reworks the howto guide on using virtio-user for
> exception packets. Changes include:
> 
> * rename "exceptional path" to "exception path"
> * remove references to uio and just reference vfio-pci
> * simplify testpmd command-lines, giving a basic usage example first
>   before adding on detail about checksum or TSO parameters
> * give a complete working example showing traffic flowing through the
>   whole system from a testpmd loopback using the created TAP netdev
> * replace use of "ifconfig" with Linux standard "ip" command
> * other general rewording.
> 
> CC: stable@dpdk.org

I don't think we should keep this for backport.
This is an enhancement.
Anyway new developments are probably not looking in old branches.

Re: [PATCH v3 1/4] doc/howto: rework section on virtio-user as exception path

[Bruce Richardson]

On Mon, Jul 11, 2022 at 03:10:56PM +0200, Thomas Monjalon wrote:
> 10/06/2022 17:35, Bruce Richardson:
> > This patch extensively reworks the howto guide on using virtio-user for
> > exception packets. Changes include:
> > 
> > * rename "exceptional path" to "exception path"
> > * remove references to uio and just reference vfio-pci
> > * simplify testpmd command-lines, giving a basic usage example first
> >   before adding on detail about checksum or TSO parameters
> > * give a complete working example showing traffic flowing through the
> >   whole system from a testpmd loopback using the created TAP netdev
> > * replace use of "ifconfig" with Linux standard "ip" command
> > * other general rewording.
> > 
> > CC: stable@dpdk.org
> 
> I don't think we should keep this for backport.
> This is an enhancement.
> Anyway new developments are probably not looking in old branches.
> 
Sure, that's fine with me.