From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mogw1109.ocn.ad.jp (mogw1109.ocn.ad.jp [153.149.229.10]) by dpdk.org (Postfix) with ESMTP id E9BA55F38 for ; Mon, 18 Jun 2018 08:21:55 +0200 (CEST) Received: from mf-smf-ucb024c2 (mf-smf-ucb024c2.ocn.ad.jp [153.153.66.161]) by mogw1109.ocn.ad.jp (Postfix) with ESMTP id 76ADB128023A; Mon, 18 Jun 2018 15:21:54 +0900 (JST) Received: from ntt.pod01.mv-mta-ucb025 ([153.149.142.99]) by mf-smf-ucb024c2 with ESMTP id UnXyfnxVpQNRUUnY2fnLQ5; Mon, 18 Jun 2018 15:21:54 +0900 Received: from smtp.ocn.ne.jp ([153.149.227.133]) by ntt.pod01.mv-mta-ucb025 with id 06Mu1y00D2tKTyH016Mu1C; Mon, 18 Jun 2018 06:21:54 +0000 Received: from localhost.localdomain (p5164-ipngn8501marunouchi.tokyo.ocn.ne.jp [153.214.228.164]) by smtp.ocn.ne.jp (Postfix) with ESMTPA; Mon, 18 Jun 2018 15:21:54 +0900 (JST) From: ogawa.yasufumi@lab.ntt.co.jp To: ferruh.yigit@intel.com, spp@dpdk.org, nakamura.hiroyuki@po.ntt-tx.co.jp Cc: Yasufumi Ogawa Date: Mon, 18 Jun 2018 15:21:31 +0900 Message-Id: <20180618062138.13263-9-ogawa.yasufumi@lab.ntt.co.jp> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180618062138.13263-1-ogawa.yasufumi@lab.ntt.co.jp> References: <20180615070615.15594-1-ogawa.yasufumi@lab.ntt.co.jp> <20180618062138.13263-1-ogawa.yasufumi@lab.ntt.co.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [spp] [PATCH v2 08/15] docs: add manual for app container launchers X-BeenThere: spp@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Soft Patch Panel List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 06:21:57 -0000 From: Yasufumi Ogawa App container launcher is a python script for launching DPDK app on a container. It composes docker command and app command with given options to launch the container. Signed-off-by: Yasufumi Ogawa --- docs/guides/tools/sppc/app_launcher.rst | 1058 +++++++++++++++++++++++ 1 file changed, 1058 insertions(+) create mode 100644 docs/guides/tools/sppc/app_launcher.rst diff --git a/docs/guides/tools/sppc/app_launcher.rst b/docs/guides/tools/sppc/app_launcher.rst new file mode 100644 index 0000000..02b46c0 --- /dev/null +++ b/docs/guides/tools/sppc/app_launcher.rst @@ -0,0 +1,1058 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017-2018 Nippon Telegraph and Telephone Corporation + +.. _spp_container_app_launcher: + +App Container Launchers +======================= + +App container launcher is a python script for running SPP or DPDK +application on a container. +As described by name, for instance, ``pktgen.py`` launches pktgen-dpdk +inside a container. + + +.. code-block:: console + + $ tree app/ + app/ + ... + ├── helloworld.py + ├── l2fwd.py + ├── l3fwd.py + ├── load-balancer.py + ├── pktgen.py + ├── spp-nfv.py + ├── spp-primary.py + ├── spp-vm.py + └── testpmd.py + + +.. note:: + + As described in + :ref:`sppc_gs_build_docker_imgs` + section, you had better to use Ubuntu 16.04 with + ``--dist-ver`` option because SPP container is not stable for running + on the latest version. + + Please notice that examples in this section does not use ``dist-ver`` + options explicitly for simplicity. + + +.. _sppc_appl_setup: + +Setup +----- + +You should define ``SPP_CTRL_IP`` environment variable to SPP controller +be accessed from other SPP processes inside containers. +SPP controller is a CLI tool for accepting user's commands. + +You cannot use ``127.0.0.1`` or ``localhost`` for ``SPP_CTRL_IP`` +because SPP processes try to find SPP controller inside each of +containers and fail to. +From inside of the container, SPP processes should be known IP address +other than ``127.0.0.1`` or ``localhost`` +of host on which SPP controller running. + +SPP controller should be launched before other SPP processes. + +.. code-block:: console + + $ cd /path/to/spp + $ python src/spp.py + + +.. _sppc_appl_spp_primary: + +SPP Primary Container +--------------------- + +SPP primary process is launched from ``app/spp-primary.py`` as an +app container. +It manages resources on host from inside the container. +``app/spp-primary.py`` calls ``docker run`` with +``-v`` option to mount hugepages and other devices in the container +to share them between host and containers. + +SPP primary process is launched as foreground or background mode. +You can show statistics of packet forwarding if you launch it with +two cores and in foreground mode. +In this case, SPP primary uses one for resource management and +another one for showing statistics. +If you need to minimize the usage of cores, or are not interested in +the statistics, +you should give just one core and run in background mode. +If you run SPP primary in foreground mode with one core, +it shows log messages which is also referred in syslog. + +Here is an example for launching SPP primary with core list 0-1 in +foreground mode. +You should give portmask opiton ``-p`` because SPP primary requires +at least one port, or failed to launch. +This example is assumed that host machine has two or more +physical ports. + +.. code-block:: console + + $ cd /path/to/spp/tools/sppc + $ python app/spp-primary -l 0-1 -p 0x03 -fg + +It is another example with one core and two ports in background mode. + +.. code-block:: console + + $ python app/spp-primary -l 0 -p 0x03 + +SPP primary is able to run with virtual devices instead of +physical NICs for a case +you do not have dedicated NICs for DPDK. +SPP container supports two types of virtual device with options. + +* ``--dev-tap-ids`` or ``-dt``: Add TAP devices +* ``--dev-vhost-ids`` or ``-dv``: Add vhost devices + +.. code-block:: console + + $ python app/spp-primary -l 0 -dt 1,2 -p 0x03 + + + +If you need to inspect a docker command without launching +a container, use ``--dry-run`` option. +It composes docker command and just display it without running the +docker command. + +You refer all of options with ``-h`` option. +Options of app container scripts are categorized four types. +First one is EAL option, for example ``-l``, ``-c`` or ``-m``. +Second one is app container option which is a common set of options for +app containers connected with SPP. So, containers of SPP processes do +not have app container option. +Third one is application specific option. In this case, +``-n``, ``-p`` or ``-ip``. +Final one is container option, for example ``--dist-name`` or +``--ci``. +EAL options and container options are common for all of app container +scripts. +On the other hand, application specific options are different each other. + +.. code-block:: console + + $ python app/spp-primary.py -h + usage: spp-primary.py [-h] [-l CORE_LIST] [-c CORE_MASK] [-m MEM] + [--socket-mem SOCKET_MEM] + [--nof-memchan NOF_MEMCHAN] + [-n NOF_RING] [-p PORT_MASK] [-dv DEV_VHOST_IDS] + [-dt DEV_TAP_IDS] [-ip CTRL_IP] + [--ctrl-port CTRL_PORT] + [--dist-name DIST_NAME] [--dist-ver DIST_VER] + [-ci CONTAINER_IMAGE] [-fg] [--dry-run] + + Launcher for spp-primary application container + + optional arguments: + -h, --help show this help message and exit + -l CORE_LIST, --core-list CORE_LIST + Core list + -c CORE_MASK, --core-mask CORE_MASK + Core mask + -m MEM, --mem MEM Memory size (default is 1024) + --socket-mem SOCKET_MEM + Memory size + --nof-memchan NOF_MEMCHAN + Number of memory channels (default is 4) + -n NOF_RING, --nof-ring NOF_RING + Maximum number of Ring PMD + -p PORT_MASK, --port-mask PORT_MASK + Port mask + -dv DEV_VHOST_IDS, --dev-vhost-ids DEV_VHOST_IDS + vhost device IDs + -dt DEV_TAP_IDS, --dev-tap-ids DEV_TAP_IDS + TAP device IDs + -ip CTRL_IP, --ctrl-ip CTRL_IP + IP address of SPP controller + --ctrl-port CTRL_PORT + Port of SPP controller + --dist-name DIST_NAME + Name of Linux distribution + --dist-ver DIST_VER Version of Linux distribution + -ci CONTAINER_IMAGE, --container-image CONTAINER_IMAGE + Name of container image + -fg, --foreground Run container as foreground mode + --dry-run Only print matrix, do not run, and exit + + +.. _sppc_appl_spp_secondary: + +SPP Secondary Container +----------------------- + +In SPP, there are three types of secondary process, ``spp_nfv``, +``spp_vm`` and ``spp_vf``. +However, SPP container does not support ``spp_vf`` currently. + +``spp_nfv`` is usually used for forwarding packets on host and +``spp_vm`` is used for insidea a virtual machine. +However, in a context of container, the usage is just a bit different. +Both of secondary processes are launched as app containers, +but ``spp_vm`` container has virtio interfaces while ``spp_nfv`` +container does not have it. +It means that ``spp_nfv`` behaves as a path between NICs or containers, +but ``spp_vm`` is a client of ``spp_nfv`` and simply forwarding from rx +to tx ports. You do not need to use ``spp_vm`` without some cases for +using virtual machines. + +``spp-nfv.py`` launches ``spp_nfv`` as an app container and requires +options for secondary ID and core list (or core mask). + +.. code-block:: console + + $ cd /path/to/spp/tools/sppc + $ python app/spp-nfv.py -i 1 -l 2-3 + +Refer help for all of options and usges. +It shows only application specific options for simplicity. + + +.. code-block:: console + + $ python app/spp-nfv.py -h + usage: spp-nfv.py [-h] [-l CORE_LIST] [-c CORE_MASK] [-m MEM] + [--socket-mem SOCKET_MEM] [--nof-memchan NOF_MEMCHAN] + [-i SEC_ID] [-ip CTRL_IP] [--ctrl-port CTRL_PORT] + [--dist-name DIST_NAME] [--dist-ver DIST_VER] + [-ci CONTAINER_IMAGE] [-fg] [--dry-run] + + Launcher for spp-nfv application container + + optional arguments: + ... + -i SEC_ID, --sec-id SEC_ID + Secondary ID + -ip CTRL_IP, --ctrl-ip CTRL_IP + IP address of SPP controller + --ctrl-port CTRL_PORT + Port of SPP controller + ... + + +For launching ``spp_vm`` for vhost interface, it is required to assign +a vhost device ID from ``app/spp-nfv.py`` and launch ``app/spp-vm.py`` +with same ID.. + +.. code-block:: console + + # Add vhost 1 from spp-nfv.py of sec 1 + spp > sec 1;add vhost 1 + +Launch a secondary container with device ID 1. + +.. code-block:: console + + $ python app/spp-vm.py -i 2 -l 4-5 -d 1 + + +.. _sppc_appl_l2fwd: + +L2fwd Container +--------------- + +``app/l2fwd.py`` is a launcher script for DPDK ``l2fwd`` sample +application. +It launches ``l2fwd`` on a container with specified +vhost interfaces. + +This is an example for launching with two cores (6-7th cores) with +``-l`` and two vhost interfaces with ``-d``. +``l2fwd`` requires ``--port-mask`` or ``-p`` option and the number of +ports should be even number. + +.. code-block:: console + + $ cd /path/to/spp/tools/sppc + $ python app/l2fwd.py -l 6-7 -d 1,2 -p 0x03 -fg + ... + +Refer help for all of options and usges. +It includes app container options, for example ``-d`` for vhost devices +and ``-nq`` for the number of queues of virtio, because ``l2fwd`` is not +a SPP process. +It shows options without of EAL and container for simplicity. + +.. code-block:: console + + $ python app/l2fwd.py -h + usage: l2fwd.py [-h] [-l CORE_LIST] [-c CORE_MASK] [-m MEM] + [--socket-mem SOCKET_MEM] [--nof-memchan NOF_MEMCHAN] + [-d DEV_IDS] [-nq NOF_QUEUES] [--no-privileged] + [-p PORT_MASK] + [--dist-name DIST_NAME] [--dist-ver DIST_VER] + [-ci CONTAINER_IMAGE] [-fg] [--dry-run] + + Launcher for l2fwd application container + + optional arguments: + ... + -d DEV_IDS, --dev-ids DEV_IDS + two or more even vhost device IDs + -nq NOF_QUEUES, --nof-queues NOF_QUEUES + Number of queues of virtio (default is 1) + --no-privileged Disable docker's privileged mode if it's needed + -p PORT_MASK, --port-mask PORT_MASK + Port mask + ... + + +.. _sppc_appl_l3fwd: + +L3fwd Container +--------------- + +`L3fwd +`_ +application is a simple example of packet processing +using the DPDK. +Differed from l2fwd, the forwarding decision is made based on +information read from input packet. +This application provides LPM (longest prefix match) or +EM (exact match) methods for packet classification. + +``app/l3fwd.py`` launches l3fwd on a container. +As ``l3fwd`` application, this python script takes several options +other than EAL for port configurations and classification methods. +The mandatory options for the application are ``-p`` for portmask +and ``--config`` for rx as a set of combination of +``(port, queue, locre)``. + +Here is an example for launching l3fwd app container with two vhost +interfaces and printed log messages. +There are two rx ports. ``(0,0,1)`` is for queue of port 0 for which +lcore 1 is assigned, and ``(1,0,2)`` is for port 1. +In this case, you should add ``-nq`` option because the number of both +of rx and tx queues are two while the default number of virtio device +is one. +The number of tx queues, is two in this case, is decided to be the same +value as the number of lcores. +In ``--vdev`` option setup in the script, the number of queues is +defined as ``virtio_...,queues=2,...``. + +.. code-block:: console + + $ cd /path/to/spp/tools/sppc + $ python app/l3fwd.py -l 1-2 -nq 2 -d 1,2 \ + -p 0x03 --config="(0,0,1),(1,0,2)" -fg + sudo docker run \ + -it \ + ... + --vdev virtio_user1,queues=2,path=/var/run/usvhost1 \ + --vdev virtio_user2,queues=2,path=/var/run/usvhost2 \ + --file-prefix spp-l3fwd-container1 \ + -- \ + -p 0x03 \ + --config "(0,0,8),(1,0,9)" \ + --parse-ptype ipv4 + EAL: Detected 16 lcore(s) + EAL: Auto-detected process type: PRIMARY + EAL: Multi-process socket /var/run/.spp-l3fwd-container1_unix + EAL: Probing VFIO support... + soft parse-ptype is enabled + LPM or EM none selected, default LPM on + Initializing port 0 ... Creating queues: nb_rxq=1 nb_txq=2... + LPM: Adding route 0x01010100 / 24 (0) + LPM: Adding route 0x02010100 / 24 (1) + LPM: Adding route IPV6 / 48 (0) + LPM: Adding route IPV6 / 48 (1) + txq=8,0,0 txq=9,1,0 + Initializing port 1 ... Creating queues: nb_rxq=1 nb_txq=2... + + Initializing rx queues on lcore 8 ... rxq=0,0,0 + Initializing rx queues on lcore 9 ... rxq=1,0,0 + ... + +You can increase lcores more than the number of ports, for instance, +four lcores for two ports. +However, remaining 3rd and 4th lcores do nothing and require +``-nq 4`` for tx queues. + +Default classification rule is ``LPM`` and the routing table is defined +in ``dpdk/examples/l3fwd/l3fwd_lpm.c`` as below. + +.. code-block:: c + + static struct ipv4_l3fwd_lpm_route ipv4_l3fwd_lpm_route_array[] = { + {IPv4(1, 1, 1, 0), 24, 0}, + {IPv4(2, 1, 1, 0), 24, 1}, + {IPv4(3, 1, 1, 0), 24, 2}, + {IPv4(4, 1, 1, 0), 24, 3}, + {IPv4(5, 1, 1, 0), 24, 4}, + {IPv4(6, 1, 1, 0), 24, 5}, + {IPv4(7, 1, 1, 0), 24, 6}, + {IPv4(8, 1, 1, 0), 24, 7}, + }; + + +Refer help for all of options and usges. +It shows options without of EAL and container for simplicity. + +.. code-block:: console + + $ python app/l3fwd.py -h + usage: l3fwd.py [-h] [-l CORE_LIST] [-c CORE_MASK] [-m MEM] + [--socket-mem SOCKET_MEM] [--nof-memchan NOF_MEMCHAN] + [-d DEV_IDS] [-nq NOF_QUEUES] [--no-privileged] + [-p PORT_MASK] [--config CONFIG] [-P] [-E] [-L] + [-dst [ETH_DEST [ETH_DEST ...]]] [--enable-jumbo] + [--max-pkt-len MAX_PKT_LEN] [--no-numa] + [--hash-entry-num] [--ipv6] [--parse-ptype PARSE_PTYPE] + [--dist-name DIST_NAME] [--dist-ver DIST_VER] + [-ci CONTAINER_IMAGE] [-fg] [--dry-run] + + Launcher for l3fwd application container + + optional arguments: + ... + -d DEV_IDS, --dev-ids DEV_IDS + two or more even vhost device IDs + -nq NOF_QUEUES, --nof-queues NOF_QUEUES + Number of queues of virtio (default is 1) + --no-privileged Disable docker's privileged mode if it's needed + -p PORT_MASK, --port-mask PORT_MASK + (Mandatory) Port mask + --config CONFIG (Mandatory) Define set of port, queue, lcore for + ports + -P, --promiscous Set all ports to promiscous mode (default is None) + -E, --exact-match Enable exact match (default is None) + -L, --longest-prefix-match + Enable longest prefix match (default is None) + -dst [ETH_DEST [ETH_DEST ...]], --eth-dest [ETH_DEST [ETH_DEST ...]] + Ethernet dest for port X (X,MM:MM:MM:MM:MM:MM) + --enable-jumbo Enable jumbo frames, [--enable-jumbo [--max-pkt-len + PKTLEN]] + --max-pkt-len MAX_PKT_LEN + Max packet length (64-9600) if jumbo is enabled. + --no-numa Disable NUMA awareness (default is None) + --hash-entry-num Specify the hash entry number in hexadecimal + (default is None) + --ipv6 Specify the hash entry number in hexadecimal + (default is None) + --parse-ptype PARSE_PTYPE + Set analyze packet type, ipv4 or ipv6 (default is + ipv4) + ... + + +.. _sppc_appl_testpmd: + +Testpmd Container +----------------- + +``testpmd.py`` is a launcher script for DPDK's +`testpmd +`_ +application. + +It launches ``testpmd`` inside a container with specified +vhost interfaces. + +This is an example for launching with three cores (6-8th cores) +and two vhost interfaces. +This example is for launching ``testpmd`` in interactive mode. + +.. code-block:: console + + $ cd /path/to/spp/tools/sppc + $ python app/testpmd.py -l 6-8 -d 1,2 -fg -i + sudo docker run \ + ... + -- \ + --interactive + ... + Checking link statuses... + Done + testpmd> + +Testpmd has many useful options. Please refer to +`Running the Application +`_ +section for instructions. + +.. note:: + ``testpmd.py`` does not support all of options of testpmd currently. + You can find all of options with ``-h`` option, but some of them + is not implemented. If you run testpmd with not supported option, + It just prints an error message to mention. + + .. code-block:: console + + $ python app/testpmd.py -l 1,2 -d 1,2 --port-topology=chained + Error: '--port-topology' is not supported yet + + +Refer help for all of options and usges. +It shows options without of EAL and container. + +.. code-block:: console + + $ python app/testpmd.py -h + usage: testpmd.py [-h] [-l CORE_LIST] [-c CORE_MASK] [-m MEM] + [--socket-mem SOCKET_MEM] [--nof-memchan NOF_MEMCHAN] + [-d DEV_IDS] [-nq NOF_QUEUES] [--no-privileged] + [--pci] [-i] [-a] [--tx-first] + [--stats-period STATS_PERIOD] + [--nb-cores NB_CORES] [--coremask COREMASK] + [--portmask PORTMASK] [--no-numa] + [--port-numa-config PORT_NUMA_CONFIG] + [--ring-numa-config RING_NUMA_CONFIG] + [--socket-num SOCKET_NUM] [--mbuf-size MBUF_SIZE] + [--total-num-mbufs TOTAL_NUM_MBUFS] + [--max-pkt-len MAX_PKT_LEN] + [--eth-peers-configfile ETH_PEERS_CONFIGFILE] + [--eth-peer ETH_PEER] [--pkt-filter-mode PKT_FILTER_MODE] + [--pkt-filter-report-hash PKT_FILTER_REPORT_HASH] + [--pkt-filter-size PKT_FILTER_SIZE] + [--pkt-filter-flexbytes-offset PKT_FILTER_FLEXBYTES_OFFSET] + [--pkt-filter-drop-queue PKT_FILTER_DROP_QUEUE] + [--disable-crc-strip] [--enable-lro] [--enable-rx-cksum] + [--enable-scatter] [--enable-hw-vlan] + [--enable-hw-vlan-filter] [--enable-hw-vlan-strip] + [--enable-hw-vlan-extend] [--enable-drop-en] + [--disable-rss] [--port-topology PORT_TOPOLOGY] + [--forward-mode FORWARD_MODE] [--rss-ip] [--rss-udp] + [--rxq RXQ] [--rxd RXD] [--txq TXQ] [--txd TXD] + [--burst BURST] [--mbcache MBCACHE] [--rxpt RXPT] + [--rxht RXHT] [--rxfreet RXFREET] [--rxwt RXWT] + [--txpt TXPT] [--txht TXHT] [--txwt TXWT] + [--txfreet TXFREET] [--txrst TXRST] + [--rx-queue-stats-mapping RX_QUEUE_STATS_MAPPING] + [--tx-queue-stats-mapping TX_QUEUE_STATS_MAPPING] + [--no-flush-rx] [--txpkts TXPKTS] [--disable-link-check] + [--no-lsc-interrupt] [--no-rmv-interrupt] + [--bitrate-stats [BITRATE_STATS [BITRATE_STATS ...]]] + [--print-event PRINT_EVENT] [--mask-event MASK_EVENT] + [--flow-isolate-all] [--tx-offloads TX_OFFLOADS] + [--hot-plug] [--vxlan-gpe-port VXLAN_GPE_PORT] + [--mlockall] [--no-mlockall] [--dist-name DIST_NAME] + [--dist-ver DIST_VER] [-ci CONTAINER_IMAGE] [-fg] + [--dry-run] + + Launcher for testpmd application container + + optional arguments: + ... + -d DEV_IDS, --dev-ids DEV_IDS + two or more even vhost device IDs + -nq NOF_QUEUES, --nof-queues NOF_QUEUES + Number of queues of virtio (default is 1) + --no-privileged Disable docker's privileged mode if it's needed + --pci Enable PCI (default is None) + -i, --interactive Run in interactive mode (default is None) + -a, --auto-start Start forwarding on initialization (default is None) + --tx-first Start forwarding, after sending a burst of packets + first + --stats-period STATS_PERIOD + Period of displaying stats, if interactive is + disabled + --nb-cores NB_CORES Number of forwarding cores + --coremask COREMASK Hexadecimal bitmask of the cores, do not include + master lcore + --portmask PORTMASK Hexadecimal bitmask of the ports + --no-numa Disable NUMA-aware allocation of RX/TX rings and RX + mbuf + --port-numa-config PORT_NUMA_CONFIG + Specify port allocation as + (port,socket)[,(port,socket)] + --ring-numa-config RING_NUMA_CONFIG + Specify ring allocation as + (port,flag,socket)[,(port,flag,socket)] + --socket-num SOCKET_NUM + Socket from which all memory is allocated in NUMA + mode + --mbuf-size MBUF_SIZE + Size of mbufs used to N (< 65536) bytes (default is + 2048) + --total-num-mbufs TOTAL_NUM_MBUFS + Number of mbufs allocated in mbuf pools, N > 1024. + --max-pkt-len MAX_PKT_LEN + Maximum packet size to N (>= 64) bytes (default is + 1518) + --eth-peers-configfile ETH_PEERS_CONFIGFILE + Config file of Ether addrs of the peer ports + --eth-peer ETH_PEER Set MAC addr of port N as 'N,XX:XX:XX:XX:XX:XX' + --pkt-filter-mode PKT_FILTER_MODE + Flow Director mode, 'none'(default), 'signature' or + 'perfect' + --pkt-filter-report-hash PKT_FILTER_REPORT_HASH + Flow Director hash match mode, 'none', + 'match'(default) or 'always' + --pkt-filter-size PKT_FILTER_SIZE + Flow Director memory size ('64K', '128K', '256K'). + The default is 64K. + --pkt-filter-flexbytes-offset PKT_FILTER_FLEXBYTES_OFFSET + Flexbytes offset (0-32, default is 0x6) defined in + words counted from the first byte of the dest MAC + address + --pkt-filter-drop-queue PKT_FILTER_DROP_QUEUE + Set the drop-queue (default is 127) + --disable-crc-strip Disable hardware CRC stripping + --enable-lro Enable large receive offload + --enable-rx-cksum Enable hardware RX checksum offload + --enable-scatter Enable scatter (multi-segment) RX + --enable-hw-vlan Enable hardware vlan (default is None) + --enable-hw-vlan-filter + Enable hardware VLAN filter + --enable-hw-vlan-strip + Enable hardware VLAN strip + --enable-hw-vlan-extend + Enable hardware VLAN extend + --enable-drop-en Enable per-queue packet drop if no descriptors + --disable-rss Disable RSS (Receive Side Scaling + --port-topology PORT_TOPOLOGY + Port topology, 'paired' (the default) or 'chained' + --forward-mode FORWARD_MODE + Forwarding mode, 'io' (default), 'mac', 'mac_swap', + 'flowgen', 'rxonly', 'txonly', 'csum', 'icmpecho', + 'ieee1588', 'tm' + --rss-ip Set RSS functions for IPv4/IPv6 only + --rss-udp Set RSS functions for IPv4/IPv6 and UDP + --rxq RXQ Number of RX queues per port, 1-65535 (default is 1) + --rxd RXD Number of descriptors in the RX rings + (default is 128) + --txq TXQ Number of TX queues per port, 1-65535 (default is 1) + --txd TXD Number of descriptors in the TX rings + (default is 512) + --burst BURST Number of packets per burst, 1-512 (default is 32) + --mbcache MBCACHE Cache of mbuf memory pools, 0-512 (default is 16) + --rxpt RXPT Prefetch threshold register of RX rings + (default is 8) + --rxht RXHT Host threshold register of RX rings (default is 8) + --rxfreet RXFREET Free threshold of RX descriptors,0-'rxd' (default is + 0) + --rxwt RXWT Write-back threshold register of RX rings + (default is 4) + --txpt TXPT Prefetch threshold register of TX rings (default is + 36) + --txht TXHT Host threshold register of TX rings (default is 0) + --txwt TXWT Write-back threshold register of TX rings (default + is 0) + --txfreet TXFREET Free threshold of RX descriptors, 0-'txd' (default + is 0) + --txrst TXRST Transmit RS bit threshold of TX rings, 0-'txd' + (default is 0) + --rx-queue-stats-mapping RX_QUEUE_STATS_MAPPING + RX queues statistics counters mapping 0-15 as + '(port,queue,mapping)[,(port,queue,mapping)]' + --tx-queue-stats-mapping TX_QUEUE_STATS_MAPPING + TX queues statistics counters mapping 0-15 as + '(port,queue,mapping)[,(port,queue,mapping)]' + --no-flush-rx Don’t flush the RX streams before starting + forwarding, Used mainly with the PCAP PMD + --txpkts TXPKTS TX segment sizes or total packet length, Valid for + tx-only and flowgen + --disable-link-check Disable check on link status when starting/stopping + ports + --no-lsc-interrupt Disable LSC interrupts for all ports + --no-rmv-interrupt Disable RMV interrupts for all ports + --bitrate-stats [BITRATE_STATS [BITRATE_STATS ...]] + Logical core N to perform bitrate calculation + --print-event PRINT_EVENT + Enable printing the occurrence of the designated + event, + --mask-event MASK_EVENT + Disable printing the occurrence of the designated + event, + --flow-isolate-all Providing this parameter requests flow API isolated + mode on all ports at initialization time + --tx-offloads TX_OFFLOADS + Hexadecimal bitmask of TX queue offloads (default + is 0) + --hot-plug Enable device event monitor machenism for hotplug + --vxlan-gpe-port VXLAN_GPE_PORT + UDP port number of tunnel VXLAN-GPE (default is + 4790) + --mlockall Enable locking all memory + --no-mlockall Disable locking all memory + ... + + +.. _sppc_appl_pktgen: + +Pktgen-dpdk Container +--------------------- + +``pktgen.py`` is a launcher script for +`pktgen-dpdk +`_. +Pktgen is a software based traffic generator powered by the DPDK +fast packet processing framework. +It is not only high-performance for generating 10GB traffic with +64 byte frames, but also very configurable to handle packets with +UDP, TCP, ARP, ICMP, GRE, MPLS and Queue-in-Queue. +It also supports +`Lua +`_ +for detailed configurations. + +This ``pktgen.py`` script launches ``pktgen`` app container +with specified vhost interfaces. +Here is an example for launching with seven lcores (8-14th) +and three vhost interfaces. + +.. code-block:: console + + $ cd /path/to/spp/tools/sppc + $ python app/pktgen.py -l 8-14 -d 1-3 -fg --dist-ver 16.04 + sudo docker run \ + ... + sppc/pktgen-ubuntu:16.04 \ + /root/dpdk/../pktgen-dpdk/app/x86_64-native-linuxapp-gcc/pktgen \ + -l 8-14 \ + ... + -- \ + -m [9:10].0,[11:12].1,[13:14].2 + ... + +You notice that given lcores ``-l 8-14`` are assigned appropriately. +Lcore 8 is used as master and remaining six lcores are use to worker +threads for three ports as ``-m [9:10].0,[11:12].1,[13:14].2`` equally. +If the number of given lcores is larger than required, +remained lcores are simply not used. + +Calculation of core assignment of ``pktgen.py`` currently is supporting +up to four lcores for each of ports. +If you assign fire or more lcores to a port, ``pktgen.py`` terminates +to launch app container. +It is because a usecase more than four lcores is rare and +calculation is to be complicated. + +.. code-block:: console + + # Assign five lcores for a slave is failed to launch + $ python app/pktgen.py -l 6-11 -d 1 + Error: Too many cores for calculation for port assignment! + Please consider to use '--matrix' for assigning directly + +Here are other examples of lcore assignment of ``pktgen.py`` to help +your understanding. + +**1. Three lcores for two ports** + +Assign one lcore to master and two lcores two slaves for two ports. + +.. code-block:: console + + $ python app/pktgen.py -l 6-8 -d 1,2 + ... + -m 7.0,8.1 \ + + +**2. Seven lcores for three ports** + +Assign one lcore for master and each of two lcores to +three slaves for three ports. + +.. code-block:: console + + $ python app/pktgen.py -l 6-12 -d 1,2,3 + ... + -m [7:8].0,[9:10].1,[11:12].2 \ + + +**3. Seven lcores for two ports** + +Assign one lcore for master and each of three lcores to +two slaves for two ports. +In this case, each of three lcores cannot be assigned rx and tx port +equally, so given two lcores to rx and one core to tx. + +.. code-block:: console + + $ python app/pktgen.py -l 6-12 -d 1,2 + ... + -m [7-8:9].0,[10-11:12].1 \ + + +Refer help for all of options and usges. +It shows options without of EAL and container for simplicity. + +.. code-block:: console + + $ python app/pktgen.py -h + usage: pktgen.py [-h] [-l CORE_LIST] [-c CORE_MASK] [-m MEM] + [--socket-mem SOCKET_MEM] [--nof-memchan NOF_MEMCHAN] + [-d DEV_IDS] [-nq NOF_QUEUES] [--no-privileged] + [--matrix MATRIX] [--log-level LOG_LEVEL] + [--dist-name DIST_NAME] [--dist-ver DIST_VER] + [-ci CONTAINER_IMAGE] [-fg] [--dry-run] + + Launcher for pktgen-dpdk application container + + optional arguments: + ... + -d DEV_IDS, --dev-ids DEV_IDS + two or more even vhost device IDs + -nq NOF_QUEUES, --nof-queues NOF_QUEUES + Number of queues of virtio (default is 1) + --no-privileged Disable docker's privileged mode if it's needed + -s PCAP_FILE, --pcap-file PCAP_FILE + PCAP packet flow file of port, defined as + 'N:filename' + -f SCRIPT_FILE, --script-file SCRIPT_FILE + Pktgen script (.pkt) to or a Lua script (.lua) + -lf LOG_FILE, --log-file LOG_FILE + Filename to write a log, as '-l' of pktgen + -P, --promiscuous Enable PROMISCUOUS mode on all ports + -G, --sock-default Enable socket support using default server values + of localhost:0x5606 + -g SOCK_ADDRESS, --sock-address SOCK_ADDRESS + Same as -G but with an optional IP address and port + number + -T, --term-color Enable color terminal output in VT100 + -N, --numa Enable NUMA support + --matrix MATRIX Matrix of cores and port as '-m' of pktgen, such as + [1:2].0 or 1.0 + ... + + +.. _sppc_appl_load_balancer: + +Load-Balancer Container +----------------------- + +`Load-Balancer +`_ +is an application distributes packet I/O task with several worker +lcores to share IP addressing. + +There are three types of lcore roles in this application, rx, tx and +worker lcores. Rx lcores retrieve packets from NICs and Tx lcores +send it to the destinations. +Worker lcores intermediate them, receive packets from rx lcores, +classify by looking up the address and send it to each of destination +tx lcores. +Each of lcores has a set of references of lcore ID and queue +as described in `Application Configuration +`_. + +``load-balancer.py`` expects four mandatory options. + + * -rx: "(PORT, QUEUE, LCORE), ...", list of NIC RX ports and + queues handled by the I/O RX lcores. This parameter also implicitly + defines the list of I/O RX lcores. + * -tx: "(PORT, LCORE), ...", list of NIC TX ports handled by + the I/O TX lcores. This parameter also implicitly defines the list + of I/O TX lcores. + * -w: The list of the worker lcores. + * --lpm: "IP / PREFIX => PORT", list of LPM rules used by the worker + lcores for packet forwarding. + +Here is an example for one rx, one tx and two worker on lcores 8-10. +Both of rx and rx is assinged to the same lcore 8. +It receives packets from port 0 and forwards it port 0 or 1. +The destination port is defined as ``--lpm`` option. + +.. code-block:: console + + $ cd /path/to/spp/tools/sppc + $ python app/load-balancer.py -fg -l 8-10 -d 1,2 \ + -rx "(0,0,8)" -tx "(0,8),(1,8)" -w 9,10 \ + --lpm "1.0.0.0/24=>0; 1.0.1.0/24=>1;" + +If you are succeeded to launch the app container, +it shows details of rx, tx, worker lcores and LPM rules +, and starts forwarding. + +.. code-block:: console + + ... + Checking link statusdone + Port0 Link Up - speed 10000Mbps - full-duplex + Port1 Link Up - speed 10000Mbps - full-duplex + Initialization completed. + NIC RX ports: 0 (0 ) ; + I/O lcore 8 (socket 0): RX ports (0, 0) ; Output rings 0x7f9af7347... + Worker lcore 9 (socket 0) ID 0: Input rings 0x7f9af7347880 ; + Worker lcore 10 (socket 0) ID 1: Input rings 0x7f9af7345680 ; + + NIC TX ports: 0 1 ; + I/O lcore 8 (socket 0): Input rings per TX port 0 (0x7f9af7343480 ... + Worker lcore 9 (socket 0) ID 0: + Output rings per TX port 0 (0x7f9af7343480) 1 (0x7f9af7341280) ; + Worker lcore 10 (socket 0) ID 1: + Output rings per TX port 0 (0x7f9af733f080) 1 (0x7f9af733ce80) ; + LPM rules: + 0: 1.0.0.0/24 => 0; + 1: 1.0.1.0/24 => 1; + Ring sizes: NIC RX = 1024; Worker in = 1024; Worker out = 1024; NIC TX... + Burst sizes: I/O RX (rd = 144, wr = 144); Worker (rd = 144, wr = 144);... + Logical core 9 (worker 0) main loop. + Logical core 10 (worker 1) main loop. + Logical core 8 (I/O) main loop. + + +To stop forwarding, you need to terminate the application +but might not able to with *Ctrl-C*. +In this case, you can use ``docker kill`` command to terminate it. +Find the name of container on which ``load_balancer`` is running +and kill it. + +.. code-block:: console + + $ docker ps + CONTAINER ID IMAGE ... NAMES + 80ce3711b85e sppc/dpdk-ubuntu:16.04 ... competent_galileo # kill it + 281aa8f236ef sppc/spp-ubuntu:16.04 ... youthful_mcnulty + $ docker kill competent_galileo + + +.. note:: + + You shold care about the number of worker lcores. If you add lcore 11 + and assign it for third worker thread, + it is failed to lauhch the application. + + .. code-block:: console + + ... + EAL: Probing VFIO support... + Incorrect value for --w argument (-8) + + load_balancer -- + + Application manadatory parameters: + --rx "(PORT, QUEUE, LCORE), ..." : List of NIC RX ports and queues + handled by the I/O RX lcores + ... + + + The reason is the number of lcore is considered as invalid in + ``parse_arg_w()`` as below. + ``n_tuples`` is the number of lcores and it should be + `2^n`, or returned with error code. + + .. code-block:: c + + // Defined in dpdk/examples/load_balancer/config.c + static int + parse_arg_w(const char *arg) + { + const char *p = arg; + uint32_t n_tuples; + ... + if ((n_tuples & (n_tuples - 1)) != 0) { + return -8; + } + ... + + +Here are other examples. + +**1. Separate rx and tx lcores** + +Use four lcores 8-11 for rx, tx and two worker threads. +The number of ports is same as the previous example. +You notice that rx and tx have different lcore number, 8 and 9. + +.. code-block:: console + + $ python app/load-balancer.py -fg -l 8-11 -d 1,2 \ + -rx "(0,0,8)" \ + -tx "(0,9),(1,9)" \ + -w 10,11 \ + --lpm "1.0.0.0/24=>0; 1.0.1.0/24=>1;" + +**2. Assign multiple queues for rx** + +To classify for three destination ports, use one rx lcore, +three tx lcores and four worker lcores. +In this case, rx has two queues and using ``-nq 2``. +You should start queue ID from 0 and to be in serial as `0,1,2,...`, +or failed to launch. + +.. code-block:: console + + $ python app/load-balancer.py -fg -l 8-13 -d 1,2,3 -nq 2 \ + -rx "(0,0,8),(0,1,8)" \ + -tx "(0,9),(1,9),(2,9)" \ + -w 10,11,12,13 \ + --lpm "1.0.0.0/24=>0; 1.0.1.0/24=>1; 1.0.2.0/24=>2;" + + +``load-balancer.py`` supports all of options other than mandatories. +Refer options and usages by ``load-balancer.py -h``. + +.. code-block:: console + + $ python app/load-balancer.py -h + usage: load-balancer.py [-h] [-l CORE_LIST] [-c CORE_MASK] [-m MEM] + [--socket-mem SOCKET_MEM] + [--nof-memchan NOF_MEMCHAN] + [-d DEV_IDS] [-nq NOF_QUEUES] [--no-privileged] + [-rx RX_PORTS] [-tx TX_PORTS] [-w WORKER_LCORES] + [-rsz RING_SIZES] [-bsz BURST_SIZES] [--lpm LPM] + [--pos-lb POS_LB] [--dist-name DIST_NAME] + [--dist-ver DIST_VER] [-ci CONTAINER_IMAGE] + [-fg] [--dry-run] + + Launcher for load-balancer application container + + optional arguments: + ... + -d DEV_IDS, --dev-ids DEV_IDS + two or more even vhost device IDs + -nq NOF_QUEUES, --nof-queues NOF_QUEUES + Number of queues of virtio (default is 1) + --no-privileged Disable docker's privileged mode if it's needed + -rx RX_PORTS, --rx-ports RX_PORTS + List of rx ports and queues handled by the I/O rx + lcores + -tx TX_PORTS, --tx-ports TX_PORTS + List of tx ports and queues handled by the I/O tx + lcores + -w WORKER_LCORES, --worker-lcores WORKER_LCORES + List of worker lcores + -rsz RING_SIZES, --ring-sizes RING_SIZES + Ring sizes of 'rx_read,rx_send,w_send,tx_written' + -bsz BURST_SIZES, --burst-sizes BURST_SIZES + Burst sizes of rx, worker or tx + --lpm LPM List of LPM rules + --pos-lb POS_LB Position of the 1-byte field used for identify + worker + ... + + +.. _sppc_appl_helloworld: + +Helloworld Container +-------------------- + +The `helloworld +`_ +sample application is an example of the simplest DPDK application +that can be written. + +Unlike from the other applications, it does not work as a network +function actually. +This app container script ``helloworld.py`` is intended to be used +as a template for an user defined app container script. +You can use it as a template for developing your app container script. +An instruction for developing app container script is described in +:ref:`sppc_howto_define_appc`. + +Helloworld app container has no application specific options. There are +only EAL and app container options. +You should give ``-l`` and ``-d`` options for the simplest app +container. +Helloworld application does not use vhost and ``-d`` options is not +required for the app, but required to setup continer itself. + +.. code-block:: console + + $ cd /path/to/spp/tools/sppc + $ python app/helloworld.py -l 4-6 -d 1 -fg + ... -- 2.17.1