From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mogw0232.ocn.ad.jp (mogw0232.ocn.ad.jp [114.147.58.38]) by dpdk.org (Postfix) with ESMTP id 1A1F01B29B for ; Mon, 5 Feb 2018 03:38:56 +0100 (CET) Received: from mf-smf-ucb002.ocn.ad.jp (mf-smf-ucb002.ocn.ad.jp [153.149.227.4]) by mogw0232.ocn.ad.jp (Postfix) with ESMTP id 755B420C288; Mon, 5 Feb 2018 11:38:54 +0900 (JST) Received: from mf-smf-ucb002.ocn.ad.jp (mf-smf-ucb002 [153.149.227.4]) by mf-smf-ucb002.ocn.ad.jp (Postfix) with ESMTP id 573A5C0676; Mon, 5 Feb 2018 11:38:54 +0900 (JST) Received: from ntt.pod01.mv-mta-ucb024 (mv-mta-ucb024.ocn.ad.jp [153.149.142.98]) by mf-smf-ucb002.ocn.ad.jp (Switch-3.3.4/Switch-3.3.4) with ESMTP id w152cqrX026302; Mon, 5 Feb 2018 11:38:54 +0900 Received: from smtp.ocn.ne.jp ([153.149.227.134]) by ntt.pod01.mv-mta-ucb024 with id 6qet1x0072ud8JZ01qetvN; Mon, 05 Feb 2018 02:38:53 +0000 Received: from localhost.localdomain (sp49-97-109-142.msc.spmode.ne.jp [49.97.109.142]) by smtp.ocn.ne.jp (Postfix) with ESMTPA; Mon, 5 Feb 2018 11:38:53 +0900 (JST) From: ogawa.yasufumi@lab.ntt.co.jp To: ferruh.yigit@intel.com, spp@dpdk.org, nakamura.hiroyuki@po.ntt-tx.co.jp Cc: Yasufumi Ogawa Date: Mon, 5 Feb 2018 11:38:58 +0900 Message-Id: <20180205023921.7775-5-ogawa.yasufumi@lab.ntt.co.jp> X-Mailer: git-send-email 2.13.1 In-Reply-To: <20180205023921.7775-1-ogawa.yasufumi@lab.ntt.co.jp> References: <20180205023921.7775-1-ogawa.yasufumi@lab.ntt.co.jp> Subject: [spp] [PATCH 05/28] docs: add spp_vf gsg section X-BeenThere: spp@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Soft Patch Panel List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Feb 2018 02:38:57 -0000 From: Yasufumi Ogawa Signed-off-by: Yasufumi Ogawa --- docs/guides/spp_vf/gsg/build.rst | 296 +++++++++++++++++++++++++++++++++++ docs/guides/spp_vf/gsg/howto_use.rst | 158 +++++++++++++++++++ docs/guides/spp_vf/gsg/index.rst | 6 + docs/guides/spp_vf/index.rst | 4 +- 4 files changed, 463 insertions(+), 1 deletion(-) create mode 100644 docs/guides/spp_vf/gsg/build.rst create mode 100644 docs/guides/spp_vf/gsg/howto_use.rst diff --git a/docs/guides/spp_vf/gsg/build.rst b/docs/guides/spp_vf/gsg/build.rst new file mode 100644 index 0000000..f363430 --- /dev/null +++ b/docs/guides/spp_vf/gsg/build.rst @@ -0,0 +1,296 @@ +.. BSD LICENSE + Copyright(c) 2010-2014 Intel Corporation. All rights reserved. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +Build +===== + +Environment +----------- + +* Ubuntu 16.04 +* qemu-kvm 2.7 or later +* DPDK v17.11 or later + + +Edit Config +----------- + +Uncomment user and group in ``/etc/libvirt/qemu.conf``. + +.. code-block:: console + + # /etc/libvirt/qemu.conf + + user = "root" + group = "root" + +Change ``KVM_HUGEPAGES`` from 0 to 1 in ``/etc/default/qemu-kvm``. + +.. code-block:: console + + # /etc/default/qemu-kvm + + KVM_HUGEPAGES=1 + +Change grub config for hugepages and isolcpus. + +.. code-block:: c + + # /etc/default/grub + + GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=2,4,6,8,10,12-18,20,22,24,26-42,44,46 hugepagesz=1G hugepages=36 default_hugepagesz=1G" + +For hugepages, isolcpus, refer to the dpdk documentation below. + +* `Use of Hugepages in the Linux Environment `_ +* `Using Linux Core Isolation to Reduce Context Switches `_ +* `Linux boot command line `_ + +You need to run ``update-grub`` and reboot to activate grub config. + +.. code-block:: console + + $ sudo upadte-grub + $ sudo reboot + + +You can check hugepage settings as following. + +.. code-block:: console + + $ cat /proc/meminfo | grep -i huge + AnonHugePages: 2048 kB + HugePages_Total: 36 # /etc/default/grub + HugePages_Free: 36 + HugePages_Rsvd: 0 + HugePages_Surp: 0 + Hugepagesize: 1048576 kB # /etc/default/grub + + $ mount | grep -i huge + cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb,nsroot=/) + hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) + hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,relatime,mode=775,gid=117) + hugetlb on /run/lxcfs/controllers/hugetlb type cgroup (rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb,nsroot=/) + +Finally, you unmount default hugepage. + +.. code-block:: console + + $ sudo unmount /dev/hugepages + + +Install DPDK +------------ + +Install DPDK in any directory. This is a simple instruction and please +refer +`Getting Started Guide for Linux +`_ +for details. + +.. code-block:: console + + $ cd /path/to/any_dir + $ git clone http://dpdk.org/git/dpdk + $ cd dpdk + $ git checkout [TAG_NAME(e.g. v17.05)] + $ export RTE_SDK=`pwd` + $ export RTE_TARGET=x86_64-native-linuxapp-gcc + $ make T=x86_64-native-linuxapp-gcc install + + +Install SPP +----------- + +Clone SPP in any directory and compile it. + +.. code-block:: console + + $ cd /path/to/spp_home/ + $ git clone https://github.com/ntt-ns/Soft-Patch-Panel.git + export SPP_HOME=/path/to/spp_home/Soft-Patch-Panel + $ cd $SPP_HOME + $ make + +Setup for DPDK +-------------- + +Load igb_uio module. + +.. code-block:: console + + $ sudo modprobe uio + $ sudo insmod $RTE_SDK/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko + $ lsmod | grep uio + igb_uio 16384 0 # igb_uio is loaded + uio 20480 1 igb_uio + +Then, bind it with PCI_Number. + +.. code-block:: console + + $ $RTE_SDK/usertools/dpdk-devbind.py --status + # check your device for PCI_Number + + $ sudo $RTE_SDK/usertools/dpdk-devbind.py --bind=igb_uio [PCI_Number] + +virsh setup +----------- + +Edit VM configuration with virsh. + +.. code-block:: console + + $ virsh edit [VM_NAME] + +.. code-block:: xml + + + spp-vm1 + d90f5420-861a-4479-8559-62d7a1545cb9 + 4194304 + 4194304 + + + + 4 + + hvm + + + + + + + + + destroy + restart + restart + + /usr/local/bin/qemu-system-x86_64 + + + + +
+ + + + + +
+ + +
+ + + +
+ + + + + +
+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +Trouble Shooting Guide +---------------------- + +You might encounter a permission error for ``tmp/sockN`` because +of appamor. +In this case, you should try it. + +.. code-block:: console + + $ sudo ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/usr.lib.libvirt.virt-aa-helper + $ sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd + $ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper + $ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd + $ sudo service apparmor reload + $ sudo service apparmor restart + $ sudo service libvirt-bin restart + +Or, you remove appamor. + +.. code-block:: console + + $ sudo apt-get remove apparmor + +If you use CentOS, not Ubuntu, confirm that SELinux doesn't prevent +for permission. +SELinux should be disabled in this case. + +.. code-block:: console + + # /etc/selinux/config + SELINUX=disabled + +Check your SELinux configuration. + +.. code-block:: console + + $ getenforce + Disabled diff --git a/docs/guides/spp_vf/gsg/howto_use.rst b/docs/guides/spp_vf/gsg/howto_use.rst new file mode 100644 index 0000000..1db3ebf --- /dev/null +++ b/docs/guides/spp_vf/gsg/howto_use.rst @@ -0,0 +1,158 @@ +.. BSD LICENSE + Copyright(c) 2010-2014 Intel Corporation. All rights reserved. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +How to Use +========== + +SPP_VF +------ + +``SPP_VF`` is a SR-IOV like network functionality for NFV. + +.. image:: images/spp_vf_overview.svg + :height: 550 em + :width: 550 em + +Environment +----------- + +* Ubuntu 16.04 +* qemu-kvm 2.7 or later +* DPDK v17.11 or later + +Launch SPP +---------- + +Before launching spp, you need to setup DPDK, virsh, etc. described at +:doc:`build`. + +SPP Controller +~~~~~~~~~~~~~~ + +First, run SPP Controller with port numbers for spp primary and secondary. + +.. code-block:: console + + $ python ./src/spp_vf.py -p 5555 -s 6666 + + +SPP Primary +~~~~~~~~~~~ + +SPP primary allocates and manages resources for secondary processes. +You need to run SPP primary before secondary processes. + +SPP primary has two kinds of options for DPDK and spp. +Before ``--`` are for DPDK is, and after ``--`` are for spp. + +See +`DPDK documentation `_ +about options for DPDK. + +Options of spp primary are: + + * -p : port mask + * -n : number of rings + * -s : IPv4 address and port for spp primary + +Then, spp primary can be launched like this. + +.. code-block:: console + + $ sudo ./src/primary/x86_64-native-linuxapp-gcc/spp_primary \ + -c 0x02 -n 4 --socket-mem 512,512 \ + --huge-dir=/run/hugepages/kvm \ + --proc-type=primary \ + -- -p 0x03 -n 9 -s 127.0.0.1:5555 + +SPP Secondary +~~~~~~~~~~~~~ + +spp secondary processes(``spp_vf``) can be launched with two kinds of +options, like primary process. + +Like primary process, ``spp_vf`` has two kinds of options. One is for +DPDK, the other is ``spp_vf``. + +``spp_vf`` specific options are: + + * --client-id : client id + * -s : IPv4 address and port for spp secondary + * --vhost-client : vhost-user client enable setting + +``spp_vf`` can be launched like this. + +.. code-block:: console + + $ sudo ./src/vf/x86_64-native-linuxapp-gcc/spp_vf \ + -c 0x3ffd -n 4 --proc-type=secondary \ + -- --client-id 1 -s 127.0.0.1:6666 --vhost-client + +If ``--vhost-client`` option is specified, then ``vhost-user`` act as +the client, otherwise the server. +For reconnect feature from SPP to VM, ``--vhost-client`` option can be +used. This reconnect features requires QEMU 2.7 (or later). +See also `DPDK documentation _. + +VM +-- + +Launch VMs with ``virsh`` command. + +.. code-block:: console + + $ virsh start [VM] + + +Additional Network Configurations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To enable processes running on the VM to communicate through spp, +it is required additional network configurations on host and guest VMs. + +Guest VMs +""""""""" + +.. code-block:: console + + # Interface for vhost + $ sudo ifconfig [IF_NAME] inet [IP_ADDR] netmask [NETMASK] up + + # Disable offload for vhost interface + $ sudo ethtool -K [IF_NAME] tx off + +Host2 +""""" + +.. code-block:: console + + # Disable offload for VM interface + $ ethtool -K [IF_NAME] tx off diff --git a/docs/guides/spp_vf/gsg/index.rst b/docs/guides/spp_vf/gsg/index.rst index f8b1783..c9028fa 100644 --- a/docs/guides/spp_vf/gsg/index.rst +++ b/docs/guides/spp_vf/gsg/index.rst @@ -30,3 +30,9 @@ Getting Started =============== + +.. toctree:: + :maxdepth: 2 + + build + howto_use diff --git a/docs/guides/spp_vf/index.rst b/docs/guides/spp_vf/index.rst index 83d2daa..e4419df 100644 --- a/docs/guides/spp_vf/index.rst +++ b/docs/guides/spp_vf/index.rst @@ -36,6 +36,8 @@ SPP VF :numbered: overview - getting_started/index + design + gsg/index use_cases/index commands/index + explain/index -- 2.13.1