From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail04.ics.ntt-tx.co.jp (mail05.ics.ntt-tx.co.jp [210.232.35.69]) by dpdk.org (Postfix) with ESMTP id A63E61B01B for ; Tue, 16 Jan 2018 06:16:43 +0100 (CET) Received: from gwchk03.silk.ntt-tx.co.jp (gwchk03.silk.ntt-tx.co.jp [10.107.0.111]) by mail04.ics.ntt-tx.co.jp (unknown) with ESMTP id w0G5GgWp007967 for unknown; Tue, 16 Jan 2018 14:16:42 +0900 Received: (from root@localhost) by gwchk03.silk.ntt-tx.co.jp (unknown) id w0G5Gghc025463 for unknown; Tue, 16 Jan 2018 14:16:42 +0900 Received: from gwchk.silk.ntt-tx.co.jp [10.107.0.110] by gwchk03.silk.ntt-tx.co.jp with ESMTP id QAA25461; Tue, 16 Jan 2018 14:16:42 +0900 Received: from imss03.silk.ntt-tx.co.jp (localhost [127.0.0.1]) by imss03.silk.ntt-tx.co.jp (unknown) with ESMTP id w0G5GgTZ009150 for unknown; Tue, 16 Jan 2018 14:16:42 +0900 Received: from mgate02.silk.ntt-tx.co.jp (smtp02.silk.ntt-tx.co.jp [10.107.0.37]) by imss03.silk.ntt-tx.co.jp (unknown) with ESMTP id w0G5GgbY009146 for unknown; Tue, 16 Jan 2018 14:16:42 +0900 Message-Id: <201801160516.w0G5GgbY009146@imss03.silk.ntt-tx.co.jp> Received: from localhost by mgate02.silk.ntt-tx.co.jp (unknown) id w0G5Gfwt026680 ; Tue, 16 Jan 2018 14:16:42 +0900 From: x-fn-spp@sl.ntt-tx.co.jp To: spp@dpdk.org Date: Tue, 16 Jan 2018 14:16:22 +0900 X-Mailer: git-send-email 1.9.1 In-Reply-To: <3e13a243-6c3f-d849-f2f4-67732e5a44cb@intel.com> References: <3e13a243-6c3f-d849-f2f4-67732e5a44cb@intel.com> X-TM-AS-MML: No Subject: [spp] [PATCH 11/30] doc: add sample usage X-BeenThere: spp@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Soft Patch Panel List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jan 2018 05:16:44 -0000 From: Hiroyuki Nakamura Signed-off-by: Naoki Takada --- docs/spp_vf/how_to_use.md | 2 +- docs/spp_vf/sample_usage.md | 212 ++++++++++++++++++++++++++++++++ docs/spp_vf/setup.md | 291 -------------------------------------------- docs/spp_vf/setup_guide.md | 291 ++++++++++++++++++++++++++++++++++++++++++++ docs/spp_vf/spp_vf.md | 6 +- 5 files changed, 509 insertions(+), 293 deletions(-) create mode 100644 docs/spp_vf/sample_usage.md delete mode 100644 docs/spp_vf/setup.md create mode 100644 docs/spp_vf/setup_guide.md diff --git a/docs/spp_vf/how_to_use.md b/docs/spp_vf/how_to_use.md index ca9cf15..6b5019c 100644 --- a/docs/spp_vf/how_to_use.md +++ b/docs/spp_vf/how_to_use.md @@ -14,7 +14,7 @@ SPP_VF is a SR-IOV like network functionality for NFV. ## Launch SPP -Before launching spp, you need to setup described as [setup guide](setup.md). +Before launching spp, you need to setup described as [setup guide](setup_guide.md). ### SPP Controller diff --git a/docs/spp_vf/sample_usage.md b/docs/spp_vf/sample_usage.md new file mode 100644 index 0000000..2157378 --- /dev/null +++ b/docs/spp_vf/sample_usage.md @@ -0,0 +1,212 @@ +# SPP_VF Sample Usage + +This sample demonstrates an use-case of MAC address classification of +spp_vf. +It enables to access to VMs with ssh from a remote node. + +Before trying this sample, install SPP by following +[setup guide](setup_guide.md). + +## Testing Steps + +In this section, you configure spp_vf and client applications for +this use-case. + + +### Setup SPP + +First, launch spp controller and primary process. + + ```sh + $ pwd + /path/to/Soft-Patch-Panel + + # SPP controller + $ python ./src/spp.py -p 5555 -s 6666 + + # SPP primary + $ sudo ./src/primary/x86_64-native-linuxapp-gcc/spp_primary \ + -c 0x02 -n 4 --socket-mem 512,512 --huge-dir=/run/hugepages/kvm \ + --proc-type=primary \ + -- \ + -p 0x03 -n 8 -s 127.0.0.1:5555 + ``` + +TODO(yasufum) add description for sec. + + ```sh + # start nc for secondary 1 + $ while true; do nc -l 11111; done + + # start secondary 1 + $ sudo ./src/vf/x86_64-native-linuxapp-gcc/spp_vf \ + -c 0x00fd -n 4 --proc-type=secondary \ + -- \ + --process-id 1 \ + --config $SPRINT_REVIEW_HOME/spp_vf1_without_cmtab.json \ + -s 127.0.0.1:11111 + + # start nc for secondary 2 + $ while true; do nc -l 11112; done + + # start secondary 2 + $ sudo ./src/vf/x86_64-native-linuxapp-gcc/spp_vf \ + -c 0x3f01 -n 4 --proc-type=secondary \ + -- \ + --process-id 2 \ + --config $SPRINT_REVIEW_HOME/spp_vf2_without_cmtab.json \ + -s 127.0.0.1:11112 + ``` + +### Setup network configuration for VMs. + +Start two VMs. + + ```sh + $ virsh start spp-vm1 + $ virsh start spp-vm2 + ``` + +Login to spp-vm1 for network configuration. +To not ask for unknown keys while login VMs, +set `-oStrictHostKeyChecking=no` option for ssh. + + ```sh + $ ssh -oStrictHostKeyChecking=no ntt@192.168.122.31 + ``` + +Up interfaces for vhost and register them to arp table inside spp-vm1. +In addition, you have to disable TCP offload function, or ssh is faled +after configuration is done. + + ```sh + # up interfaces + $ sudo ifconfig ens4 inet 192.168.240.21 netmask 255.255.255.0 up + $ sudo ifconfig ens5 inet 192.168.250.22 netmask 255.255.255.0 up + + # register to arp table + $ sudo arp -s 192.168.240.11 a0:36:9f:78:86:78 -i ens4 + $ sudo arp -s 192.168.250.13 a0:36:9f:6c:ed:bc -i ens5 + + # diable TCP offload + $ sudo ethtool -K ens4 tx off + $ sudo ethtool -K ens5 tx off + ``` + +Configurations for spp-vm2 is same as spp-vm1. + + ```sh + $ ssh -oStrictHostKeyChecking=no ntt@192.168.122.32 + + # up interfaces + $ sudo ifconfig ens4 inet 192.168.240.31 netmask 255.255.255.0 up + $ sudo ifconfig ens5 inet 192.168.250.32 netmask 255.255.255.0 up + + # register to arp table + $ sudo arp -s 192.168.240.11 a0:36:9f:78:86:78 -i ens4 + $ sudo arp -s 192.168.250.13 a0:36:9f:6c:ed:bc -i ens5 + + # diable TCP offload + $ sudo ethtool -K ens4 tx off + $ sudo ethtool -K ens5 tx off + ``` + +Check the configuration by trying ssh from remote machine that +connection is accepted but discarded in spp secondary. +If you do ssh for VM1, you find a messages from spp secondary for +discarding packets. + +## Test Application + +TODO(yasufum) json-based steps are deprecated. + +### Register MAC address to Classifier + +Send a request for getting each of process IDs with nc command. +TODO(yasufum) for what? + + ```sh + { + "commands": [ + { + "command": "process" + } + ] + } + ``` + +Register MAC addresses to classifier. + + ```sh + { + "commands": [ + { + "command": "classifier_table", + "type": "mac", + "value": "52:54:00:12:34:56", + "port": "ring0" + }, + { + "command": "classifier_table", + "type": "mac", + "value": "52:54:00:12:34:58", + "port": "ring1" + }, + { + "command": "flush" + } + ] + } + ``` + + ```sh + { + "commands": [ + { + "command": "classifier_table", + "type": "mac", + "value": "52:54:00:12:34:57", + "port": "ring4" + }, + { + "command": "flush" + } + ] + } + ``` + +### Login to VMs + +Now, you can login VMs. + + ```sh + # spp-vm1 via NIC0 + $ ssh ntt@192.168.240.21 + + # spp-vm1 via NIC1 + $ ssh ntt@192.168.250.22 + + # spp-vm2 via NIC0 + $ ssh ntt@192.168.240.31 + + # spp-vm2 via NIC1 + $ ssh ntt@192.168.250.32 + ``` + +If you unregister the addresses, send request as following. + + ```sh + { + "commands": [ + { + "command": "classifier_table", + "type": "mac", + "value": "52:54:00:12:34:58", + "port": "unuse" + }, + { + "command": "flush" + } + ] + } + ``` diff --git a/docs/spp_vf/setup.md b/docs/spp_vf/setup.md deleted file mode 100644 index 450e68a..0000000 --- a/docs/spp_vf/setup.md +++ /dev/null @@ -1,291 +0,0 @@ -# Setup Guide - -## Environment - -* Ubuntu 16.04 -* qemu-kvm -* DPDK v17.05 - -## Setting - -### Host - -#### Edit Config - -Uncomment user and group in `/etc/libvirt/qemu.conf`. - -```sh -# /etc/libvirt/qemu.conf - -user = "root" -group = "root" -``` - -Change `KVM_HUGEPAGES` from 0 to 1 in `/etc/default/qemu-kvm`. - -```sh -# /etc/default/qemu-kvm - -KVM_HUGEPAGES=1 -``` - -Change grub config for hugepages and isolcpus. - -```sh -# /etc/default/grub - -GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=2,4,6,8,10,12-18,20,22,24,26-42,44,46 hugepagesz=1G hugepages=36 default_hugepagesz=1G" -``` - -You need to run `update-grub` and reboot to activate grub config. - -```sh -$ sudo upadte-grub -$ sudo reboot -``` - -You can check hugepage settings as following. - -```sh -$ cat /proc/meminfo | grep -i huge -AnonHugePages: 2048 kB -HugePages_Total: 36 # /etc/default/grub -HugePages_Free: 36 -HugePages_Rsvd: 0 -HugePages_Surp: 0 -Hugepagesize: 1048576 kB # /etc/default/grub - -$ mount | grep -i huge -cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb,nsroot=/) -hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) -hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,relatime,mode=775,gid=117) -hugetlb on /run/lxcfs/controllers/hugetlb type cgroup (rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb,nsroot=/) -``` - -Finally, you unmount default hugepage. - -```sh -$ sudo unmount /dev/hugepages -``` - -#### Install jasson - -Network configuration is defined in JSON and `spp_vf` reads config from -the file while launching. -[jasson](http://www.digip.org/jansson/) is a JSON library written in C. - -It is required to use `json_path` feature of `jasson` for `spp_vf`. -It has develped under `json_path` branch and you need to checkout and compile -it manually. - -```sh -$ git clone https://github.com/rogerz/jansson -$ cd jansson -$ sudo git checkout json_path -Branch json_path set up to track remote branch json_path from origin. Switched to a new branch 'json_path' -``` - -This setup guide expects that `jasson` is placed as `/opt/jasson`. - -```sh -$ sudo mkdir -p /opt/jansson -$ sudo mv jansson /opt/jansson -``` - -Compile it as following. - -```sh -$ cd /opt/jansson/jansson -$ sudo autoreconf -i -$ sudo ./configure -$ sudo make -$ sudo make install -$ sudo ldconfig -``` - -Then, confirm that header files of jasson are generated in `/usr/local/include`. - -```sh -$ ls -al /usr/local/include -total 24 -drwxr-xr-x 2 root root 4096 Jul 28 16:45 . -drwxr-xr-x 10 root root 4096 May 27 10:23 .. --rw-r--r-- 1 root root 1183 Jul 28 16:45 jansson_config.h --rw-r--r-- 1 root root 9499 Jul 28 16:45 jansson.h -``` - -#### Install DPDK - -Install DPDK v17.05 in any directory. This is a simple instruction and please refer -[Getting Started Guide for Linux](http://dpdk.org/doc/guides/linux_gsg/index.html) -for details. - -```sh -$ cd /path/to/any_dir -$ git clone http://dpdk.org/git/dpdk -$ cd dpdk -$ git checkout v17.05 -$ export RTE_SDK=`pwd` -$ export RTE_TARGET=x86_64-native-linuxapp-gcc -$ make T=x86_64-native-linuxapp-gcc install -``` - -#### Install SPP - -Clone SPP in any directory and compile it. - -```sh -$ cd /path/to/spp_home/ -$ git clone https://github.com/ntt-ns/Soft-Patch-Panel.git -export SPP_HOME=/path/to/spp_home/Soft-Patch-Panel -$ cd $SPP_HOME -$ make -``` - -#### Setup for DPDK - -Load igb_uio module. - -```sh -$ sudo modprobe uio -$ sudo insmod $RTE_SDK/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko -$ lsmod | grep uio -igb_uio 16384 0 # igb_uio is loaded -uio 20480 1 igb_uio -``` - -Then, bind it with PCI_Number. -```sh -$ $RTE_SDK/usertools/dpdk-devbind.py --status -# check your device for PCI_Number - -$ sudo $RTE_SDK/usertools/dpdk-devbind.py --bind=igb_uio [PCI_Number] -``` - -#### virsh setup - -Edit VM configuration with virsh. - -```sh -$ virsh edit [VM_NAME] -``` - -```xml - - spp-vm1 - d90f5420-861a-4479-8559-62d7a1545cb9 - 4194304 - 4194304 - - - - 4 - - hvm - - - - - - - - - destroy - restart - restart - - /usr/local/bin/qemu-system-x86_64 - - - - -
- - - - - -
- - -
- - - -
- - - - - -
- - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - -``` - -### Trouble Shooting - -You might encounter a permission error for `tmp/sockN` because of appamor. -In this case, you should try it. - -```sh -$ sudo ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/usr.lib.libvirt.virt-aa-helper -$ sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd -$ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper -$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd -$ sudo service apparmor reload -$ sudo service apparmor restart -$ sudo service libvirt-bin restart -``` - -Or, you remove appamor. - -```sh -$ sudo apt-get remove apparmor -``` - -If you use CentOS, not Ubuntu, confirm that SELinux doesn't prevent for permission. -SELinux should be disabled in this case. - -```sh -# /etc/selinux/config -SELINUX=disabled -``` - -Check your SELinux configuration. - -```sh -$ getenforce -Disabled -``` diff --git a/docs/spp_vf/setup_guide.md b/docs/spp_vf/setup_guide.md new file mode 100644 index 0000000..450e68a --- /dev/null +++ b/docs/spp_vf/setup_guide.md @@ -0,0 +1,291 @@ +# Setup Guide + +## Environment + +* Ubuntu 16.04 +* qemu-kvm +* DPDK v17.05 + +## Setting + +### Host + +#### Edit Config + +Uncomment user and group in `/etc/libvirt/qemu.conf`. + +```sh +# /etc/libvirt/qemu.conf + +user = "root" +group = "root" +``` + +Change `KVM_HUGEPAGES` from 0 to 1 in `/etc/default/qemu-kvm`. + +```sh +# /etc/default/qemu-kvm + +KVM_HUGEPAGES=1 +``` + +Change grub config for hugepages and isolcpus. + +```sh +# /etc/default/grub + +GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=2,4,6,8,10,12-18,20,22,24,26-42,44,46 hugepagesz=1G hugepages=36 default_hugepagesz=1G" +``` + +You need to run `update-grub` and reboot to activate grub config. + +```sh +$ sudo upadte-grub +$ sudo reboot +``` + +You can check hugepage settings as following. + +```sh +$ cat /proc/meminfo | grep -i huge +AnonHugePages: 2048 kB +HugePages_Total: 36 # /etc/default/grub +HugePages_Free: 36 +HugePages_Rsvd: 0 +HugePages_Surp: 0 +Hugepagesize: 1048576 kB # /etc/default/grub + +$ mount | grep -i huge +cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb,nsroot=/) +hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) +hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,relatime,mode=775,gid=117) +hugetlb on /run/lxcfs/controllers/hugetlb type cgroup (rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb,nsroot=/) +``` + +Finally, you unmount default hugepage. + +```sh +$ sudo unmount /dev/hugepages +``` + +#### Install jasson + +Network configuration is defined in JSON and `spp_vf` reads config from +the file while launching. +[jasson](http://www.digip.org/jansson/) is a JSON library written in C. + +It is required to use `json_path` feature of `jasson` for `spp_vf`. +It has develped under `json_path` branch and you need to checkout and compile +it manually. + +```sh +$ git clone https://github.com/rogerz/jansson +$ cd jansson +$ sudo git checkout json_path +Branch json_path set up to track remote branch json_path from origin. Switched to a new branch 'json_path' +``` + +This setup guide expects that `jasson` is placed as `/opt/jasson`. + +```sh +$ sudo mkdir -p /opt/jansson +$ sudo mv jansson /opt/jansson +``` + +Compile it as following. + +```sh +$ cd /opt/jansson/jansson +$ sudo autoreconf -i +$ sudo ./configure +$ sudo make +$ sudo make install +$ sudo ldconfig +``` + +Then, confirm that header files of jasson are generated in `/usr/local/include`. + +```sh +$ ls -al /usr/local/include +total 24 +drwxr-xr-x 2 root root 4096 Jul 28 16:45 . +drwxr-xr-x 10 root root 4096 May 27 10:23 .. +-rw-r--r-- 1 root root 1183 Jul 28 16:45 jansson_config.h +-rw-r--r-- 1 root root 9499 Jul 28 16:45 jansson.h +``` + +#### Install DPDK + +Install DPDK v17.05 in any directory. This is a simple instruction and please refer +[Getting Started Guide for Linux](http://dpdk.org/doc/guides/linux_gsg/index.html) +for details. + +```sh +$ cd /path/to/any_dir +$ git clone http://dpdk.org/git/dpdk +$ cd dpdk +$ git checkout v17.05 +$ export RTE_SDK=`pwd` +$ export RTE_TARGET=x86_64-native-linuxapp-gcc +$ make T=x86_64-native-linuxapp-gcc install +``` + +#### Install SPP + +Clone SPP in any directory and compile it. + +```sh +$ cd /path/to/spp_home/ +$ git clone https://github.com/ntt-ns/Soft-Patch-Panel.git +export SPP_HOME=/path/to/spp_home/Soft-Patch-Panel +$ cd $SPP_HOME +$ make +``` + +#### Setup for DPDK + +Load igb_uio module. + +```sh +$ sudo modprobe uio +$ sudo insmod $RTE_SDK/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko +$ lsmod | grep uio +igb_uio 16384 0 # igb_uio is loaded +uio 20480 1 igb_uio +``` + +Then, bind it with PCI_Number. +```sh +$ $RTE_SDK/usertools/dpdk-devbind.py --status +# check your device for PCI_Number + +$ sudo $RTE_SDK/usertools/dpdk-devbind.py --bind=igb_uio [PCI_Number] +``` + +#### virsh setup + +Edit VM configuration with virsh. + +```sh +$ virsh edit [VM_NAME] +``` + +```xml + + spp-vm1 + d90f5420-861a-4479-8559-62d7a1545cb9 + 4194304 + 4194304 + + + + 4 + + hvm + + + + + + + + + destroy + restart + restart + + /usr/local/bin/qemu-system-x86_64 + + + + +
+ + + + + +
+ + +
+ + + +
+ + + + + +
+ + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + +``` + +### Trouble Shooting + +You might encounter a permission error for `tmp/sockN` because of appamor. +In this case, you should try it. + +```sh +$ sudo ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/usr.lib.libvirt.virt-aa-helper +$ sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd +$ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper +$ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd +$ sudo service apparmor reload +$ sudo service apparmor restart +$ sudo service libvirt-bin restart +``` + +Or, you remove appamor. + +```sh +$ sudo apt-get remove apparmor +``` + +If you use CentOS, not Ubuntu, confirm that SELinux doesn't prevent for permission. +SELinux should be disabled in this case. + +```sh +# /etc/selinux/config +SELINUX=disabled +``` + +Check your SELinux configuration. + +```sh +$ getenforce +Disabled +``` diff --git a/docs/spp_vf/spp_vf.md b/docs/spp_vf/spp_vf.md index 786e712..99c914f 100644 --- a/docs/spp_vf/spp_vf.md +++ b/docs/spp_vf/spp_vf.md @@ -29,12 +29,16 @@ by looking MAC address in the packet.. ## Build the Application -See [setup guide](setup.md). +See [setup guide](setup_guide.md). ## Running the Application See [how to use](how_to_use.md). +## Sample Usage + +See [sample usage](sample_usage.md). + ## Explanation The following sections provide some explanation of the code. -- 1.9.1