[dpdk-dev] [PATCH] doc: introduce openwrt how-to guide

[dpdk-dev] [PATCH] doc: introduce openwrt how-to guide

[Xiaolong Ye]

This doc describes how to enable DPDK on openwrt in both virtual and
physical x86 environment.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 doc/guides/howto/index.rst   |   1 +
 doc/guides/howto/openwrt.rst | 180 +++++++++++++++++++++++++++++++++++
 2 files changed, 181 insertions(+)
 create mode 100644 doc/guides/howto/openwrt.rst

diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a4c131652..5a97ea508 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -19,3 +19,4 @@ HowTo Guides
     packet_capture_framework
     telemetry
     debug_troubleshoot
+    openwrt
diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
new file mode 100644
index 000000000..62be3a031
--- /dev/null
+++ b/doc/guides/howto/openwrt.rst
@@ -0,0 +1,180 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Enable DPDK on openwrt
+======================
+
+This document describes how to enable Data Plane Development Kit(DPDK) on
+Openwrt in both virtual and physical x86 environment.
+
+Introduction
+------------
+
+The OpenWrt project is a Linux operating system targeting embedded devices.
+Instead of trying to create a single, static firmware, OpenWrt provides a fully
+writable filesystem with package management. This frees user from the
+application selection and configuration provided by the vendor and allows user
+to customize the device through the use of packages to suit any application. For
+developers, OpenWrt is the framework to build and application without having to
+build a complete firmware around it, for users this means the ability for full
+customization, to use the device in ways never envisioned.
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
+unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
+
+Build OpenWrt
+-------------
+
+You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
+fully customize your own OpenWrt, it is highly recommended to build it through
+the source code, you can clone the OpenWrt source code by:
+
+.. code-block:: console
+
+	git clone https://git.openwrt.org/openwrt/openwrt.git
+
+OpenWrt configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+* Select ``x86`` in ``Target System``
+* Select ``x86_64`` in ``Subtarget``
+* Select ``Build the OpenWrt SDK`` for cross-compilation environment
+* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
+			   -> ``ToolChain Options``
+			   -> ``C Library implementation``
+
+Kernel configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Below configurations need to be enabled:
+
+* CONFIG_UIO=y
+* CONFIG_HUGETLBFS=y
+* CONFIG_HUGETLB_PAGE=y
+* CONFIG_PAGE_MONITOR=y
+
+Build steps
+~~~~~~~~~~~
+
+1. Run ``./scripts/feeds update -a`` to obtain all the latest package definitions
+defined in feeds.conf / feeds.conf.default
+
+2. Run ``./scripts/feeds install -a`` to install symlinks for all obtained
+packages into package/feeds/
+
+3. Run ``make menuconfig`` to select preferred configuration mentioned above for
+the toolchain, target system & firmware packages.
+
+3. Run ``make kernel_menuconfig`` to select preferred kernel configurations.
+
+4. Run ``make`` to build your firmware. This will download all sources, build
+the cross-compile toolchain and then cross-compile the Linux kernel & all
+chosen applications for your target system.
+
+After build is done, you can find the images and sdk in ``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
+
+DPDK Cross Compilation for OpenWrt
+----------------------------------
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+NUMA is required to run dpdk in x86.
+
+.. note::
+
+   For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2,
+   otherwise the compilation will fail with errors.
+
+.. code-block:: console
+
+	git clone https://github.com/numactl/numactl.git
+	cd numactl
+	git checkout v2.0.13 -b v2.0.13
+	./autogen.sh
+	autoconf -i
+	export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
+	./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
+	make install
+
+The numa header files and lib file is generated in the include and lib folder respectively under <OpenWrt SDK toolchain dir>.
+
+Build DPDK
+~~~~~~~~~~
+
+.. code-block:: console
+
+	export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
+	export RTE_SDK=`pwd`
+	export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
+	make config T=x86_64-native-linuxapp-gcc
+	make -j 100 CROSS=x86_64-openwrt-linux-gnu-
+
+Running DPDK application on OpenWrt
+-----------------------------------
+
+Virtual machine
+~~~~~~~~~~~~~~~
+
+* Extract boot image
+
+.. code-block:: console
+
+	gzip -d openwrt-x86-64-combined-ext4.img.gz
+
+* Launch Qemu
+
+.. code-block:: console
+
+	qemu-system-x86_64 \
+	        -cpu host \
+	        -smp 8 \
+	        -enable-kvm \
+	        -M q35 \
+	        -m 2048M \
+	        -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
+	        -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
+	        -device ide-hd,drive=d0,bus=ide.0 \
+	        -net nic,vlan=0 \
+	        -net nic,vlan=1 \
+	        -net user,vlan=1 \
+	        -display none \
+
+
+Physical machine
+~~~~~~~~~~~~~~~~
+
+Installation
+
+If you are using Windows PC, you can use some free and opensource raw disk image writer program such as
+``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image (openwrt-x86-64-combined-ext4.img) to a USB
+flash driver or USB SDcard with SDcard or a Sata hard drivre or SSD from your PC.
+
+If you are using Linux, you can use old dd tool to write OpenWrt image to the drive you want to write the
+image on.
+
+.. code-block:: console
+
+	dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
+
+Where sdX is name of the drive. (You can find it though ``fdisk -l``)
+
+Running DPDK
+~~~~~~~~~~~~
+
+* Setup dpdk environment
+
+  * Scp built numa libraries (including soft link) to /usr/lib64
+  * Setup hugepages
+  * insmod igb_uio.ko (scp the built igb_uio.ko first)
+  * Bind the NIC to igb_uio.ko
+
+* Launch testpmd
+
+  * scp built testpmd to qemu
+  * ./testpmd -c 0xf -- -i
+
+ More detailed info about how to run a DPDK application refer to ``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
-- 
2.17.1

Re: [dpdk-dev] [PATCH] doc: introduce openwrt how-to guide

[Bruce Richardson]

On Fri, Nov 29, 2019 at 04:19:11PM +0800, Xiaolong Ye wrote:
> This doc describes how to enable DPDK on openwrt in both virtual and
> physical x86 environment.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---

At this point, I don't think it's a good idea to be adding more
instructions for building DPDK using make - it will only make the future
switchover to using meson harder. Can you update the doc to describe the
process using meson for the DPDK build.

/Bruce

Re: [dpdk-dev] [PATCH] doc: introduce openwrt how-to guide

[Ye Xiaolong]

On 11/29, Bruce Richardson wrote:
>On Fri, Nov 29, 2019 at 04:19:11PM +0800, Xiaolong Ye wrote:
>> This doc describes how to enable DPDK on openwrt in both virtual and
>> physical x86 environment.
>> 
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> ---
>
>At this point, I don't think it's a good idea to be adding more
>instructions for building DPDK using make - it will only make the future
>switchover to using meson harder. Can you update the doc to describe the
>process using meson for the DPDK build.

Good point, I'll try the meson build and update the doc accordingly.

Thanks,
Xiaolong

>
>/Bruce

Re: [dpdk-dev] [PATCH] doc: introduce openwrt how-to guide

[Stephen Hemminger]

On Fri, 29 Nov 2019 16:19:11 +0800
Xiaolong Ye <xiaolong.ye@intel.com> wrote:

> This doc describes how to enable DPDK on openwrt in both virtual and
> physical x86 environment.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
>  doc/guides/howto/index.rst   |   1 +
>  doc/guides/howto/openwrt.rst | 180 +++++++++++++++++++++++++++++++++++
>  2 files changed, 181 insertions(+)
>  create mode 100644 doc/guides/howto/openwrt.rst
> 
> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
> index a4c131652..5a97ea508 100644
> --- a/doc/guides/howto/index.rst
> +++ b/doc/guides/howto/index.rst
> @@ -19,3 +19,4 @@ HowTo Guides
>      packet_capture_framework
>      telemetry
>      debug_troubleshoot
> +    openwrt
> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
> new file mode 100644
> index 000000000..62be3a031
> --- /dev/null
> +++ b/doc/guides/howto/openwrt.rst
> @@ -0,0 +1,180 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2019 Intel Corporation.
> +
> +Enable DPDK on openwrt
> +======================
> +
> +This document describes how to enable Data Plane Development Kit(DPDK) on
> +Openwrt in both virtual and physical x86 environment.
> +
> +Introduction
> +------------
> +
> +The OpenWrt project is a Linux operating system targeting embedded devices.
> +Instead of trying to create a single, static firmware, OpenWrt provides a fully
> +writable filesystem with package management. This frees user from the
> +application selection and configuration provided by the vendor and allows user
> +to customize the device through the use of packages to suit any application. For
> +developers, OpenWrt is the framework to build and application without having to
> +build a complete firmware around it, for users this means the ability for full
> +customization, to use the device in ways never envisioned.
> +
> +Pre-requisites
> +~~~~~~~~~~~~~~
> +
> +You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
> +unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
> +
> +Build OpenWrt
> +-------------
> +
> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
> +fully customize your own OpenWrt, it is highly recommended to build it through
> +the source code, you can clone the OpenWrt source code by:
> +
> +.. code-block:: console
> +
> +	git clone https://git.openwrt.org/openwrt/openwrt.git
> +
> +OpenWrt configuration
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +* Select ``x86`` in ``Target System``
> +* Select ``x86_64`` in ``Subtarget``
> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
> +			   -> ``ToolChain Options``
> +			   -> ``C Library implementation``
> +
> +Kernel configuration
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Below configurations need to be enabled:
> +
> +* CONFIG_UIO=y
> +* CONFIG_HUGETLBFS=y
> +* CONFIG_HUGETLB_PAGE=y
> +* CONFIG_PAGE_MONITOR=y
> +
> +Build steps
> +~~~~~~~~~~~
> +
> +1. Run ``./scripts/feeds update -a`` to obtain all the latest package definitions
> +defined in feeds.conf / feeds.conf.default
> +
> +2. Run ``./scripts/feeds install -a`` to install symlinks for all obtained
> +packages into package/feeds/
> +
> +3. Run ``make menuconfig`` to select preferred configuration mentioned above for
> +the toolchain, target system & firmware packages.
> +
> +3. Run ``make kernel_menuconfig`` to select preferred kernel configurations.
> +
> +4. Run ``make`` to build your firmware. This will download all sources, build
> +the cross-compile toolchain and then cross-compile the Linux kernel & all
> +chosen applications for your target system.
> +
> +After build is done, you can find the images and sdk in ``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
> +
> +DPDK Cross Compilation for OpenWrt
> +----------------------------------
> +
> +Pre-requisites
> +~~~~~~~~~~~~~~
> +
> +NUMA is required to run dpdk in x86.
> +
> +.. note::
> +
> +   For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2,
> +   otherwise the compilation will fail with errors.
> +
> +.. code-block:: console
> +
> +	git clone https://github.com/numactl/numactl.git
> +	cd numactl
> +	git checkout v2.0.13 -b v2.0.13
> +	./autogen.sh
> +	autoconf -i
> +	export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
> +	./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
> +	make install
> +
> +The numa header files and lib file is generated in the include and lib folder respectively under <OpenWrt SDK toolchain dir>.
> +
> +Build DPDK
> +~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +	export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
> +	export RTE_SDK=`pwd`
> +	export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
> +	make config T=x86_64-native-linuxapp-gcc
> +	make -j 100 CROSS=x86_64-openwrt-linux-gnu-
> +
> +Running DPDK application on OpenWrt
> +-----------------------------------
> +
> +Virtual machine
> +~~~~~~~~~~~~~~~
> +
> +* Extract boot image
> +
> +.. code-block:: console
> +
> +	gzip -d openwrt-x86-64-combined-ext4.img.gz
> +
> +* Launch Qemu
> +
> +.. code-block:: console
> +
> +	qemu-system-x86_64 \
> +	        -cpu host \
> +	        -smp 8 \
> +	        -enable-kvm \
> +	        -M q35 \
> +	        -m 2048M \
> +	        -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
> +	        -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
> +	        -device ide-hd,drive=d0,bus=ide.0 \
> +	        -net nic,vlan=0 \
> +	        -net nic,vlan=1 \
> +	        -net user,vlan=1 \
> +	        -display none \
> +
> +
> +Physical machine
> +~~~~~~~~~~~~~~~~
> +
> +Installation
> +
> +If you are using Windows PC, you can use some free and opensource raw disk image writer program such as
> +``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image (openwrt-x86-64-combined-ext4.img) to a USB
> +flash driver or USB SDcard with SDcard or a Sata hard drivre or SSD from your PC.
> +
> +If you are using Linux, you can use old dd tool to write OpenWrt image to the drive you want to write the
> +image on.
> +
> +.. code-block:: console
> +
> +	dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
> +
> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
> +
> +Running DPDK
> +~~~~~~~~~~~~
> +
> +* Setup dpdk environment
> +
> +  * Scp built numa libraries (including soft link) to /usr/lib64
> +  * Setup hugepages
> +  * insmod igb_uio.ko (scp the built igb_uio.ko first)
> +  * Bind the NIC to igb_uio.ko
> +
> +* Launch testpmd
> +
> +  * scp built testpmd to qemu
> +  * ./testpmd -c 0xf -- -i
> +
> + More detailed info about how to run a DPDK application refer to ``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.

It is good to have more documentation and examples.
My one concern is that a lot of this overlaps existing documents and is likely
to get out of date.  Could you use links instead to describe how
to install OpenWrt and how to run testpmd.

Re: [dpdk-dev] [PATCH] doc: introduce openwrt how-to guide

[Ye Xiaolong]

On 11/29, Stephen Hemminger wrote:
>On Fri, 29 Nov 2019 16:19:11 +0800
>Xiaolong Ye <xiaolong.ye@intel.com> wrote:
>
>> This doc describes how to enable DPDK on openwrt in both virtual and
>> physical x86 environment.
>> 
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> ---
>>  doc/guides/howto/index.rst   |   1 +
>>  doc/guides/howto/openwrt.rst | 180 +++++++++++++++++++++++++++++++++++
>>  2 files changed, 181 insertions(+)
>>  create mode 100644 doc/guides/howto/openwrt.rst
>> 
>> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
>> index a4c131652..5a97ea508 100644
>> --- a/doc/guides/howto/index.rst
>> +++ b/doc/guides/howto/index.rst
>> @@ -19,3 +19,4 @@ HowTo Guides
>>      packet_capture_framework
>>      telemetry
>>      debug_troubleshoot
>> +    openwrt
>> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
>> new file mode 100644
>> index 000000000..62be3a031
>> --- /dev/null
>> +++ b/doc/guides/howto/openwrt.rst
>> @@ -0,0 +1,180 @@
>> +..  SPDX-License-Identifier: BSD-3-Clause
>> +    Copyright(c) 2019 Intel Corporation.
>> +
>> +Enable DPDK on openwrt
>> +======================
>> +
>> +This document describes how to enable Data Plane Development Kit(DPDK) on
>> +Openwrt in both virtual and physical x86 environment.
>> +
>> +Introduction
>> +------------
>> +
>> +The OpenWrt project is a Linux operating system targeting embedded devices.
>> +Instead of trying to create a single, static firmware, OpenWrt provides a fully
>> +writable filesystem with package management. This frees user from the
>> +application selection and configuration provided by the vendor and allows user
>> +to customize the device through the use of packages to suit any application. For
>> +developers, OpenWrt is the framework to build and application without having to
>> +build a complete firmware around it, for users this means the ability for full
>> +customization, to use the device in ways never envisioned.
>> +
>> +Pre-requisites
>> +~~~~~~~~~~~~~~
>> +
>> +You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
>> +unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
>> +
>> +Build OpenWrt
>> +-------------
>> +
>> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
>> +fully customize your own OpenWrt, it is highly recommended to build it through
>> +the source code, you can clone the OpenWrt source code by:
>> +
>> +.. code-block:: console
>> +
>> +	git clone https://git.openwrt.org/openwrt/openwrt.git
>> +
>> +OpenWrt configuration
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +* Select ``x86`` in ``Target System``
>> +* Select ``x86_64`` in ``Subtarget``
>> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
>> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
>> +			   -> ``ToolChain Options``
>> +			   -> ``C Library implementation``
>> +
>> +Kernel configuration
>> +~~~~~~~~~~~~~~~~~~~~
>> +
>> +Below configurations need to be enabled:
>> +
>> +* CONFIG_UIO=y
>> +* CONFIG_HUGETLBFS=y
>> +* CONFIG_HUGETLB_PAGE=y
>> +* CONFIG_PAGE_MONITOR=y
>> +
>> +Build steps
>> +~~~~~~~~~~~
>> +
>> +1. Run ``./scripts/feeds update -a`` to obtain all the latest package definitions
>> +defined in feeds.conf / feeds.conf.default
>> +
>> +2. Run ``./scripts/feeds install -a`` to install symlinks for all obtained
>> +packages into package/feeds/
>> +
>> +3. Run ``make menuconfig`` to select preferred configuration mentioned above for
>> +the toolchain, target system & firmware packages.
>> +
>> +3. Run ``make kernel_menuconfig`` to select preferred kernel configurations.
>> +
>> +4. Run ``make`` to build your firmware. This will download all sources, build
>> +the cross-compile toolchain and then cross-compile the Linux kernel & all
>> +chosen applications for your target system.
>> +
>> +After build is done, you can find the images and sdk in ``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
>> +
>> +DPDK Cross Compilation for OpenWrt
>> +----------------------------------
>> +
>> +Pre-requisites
>> +~~~~~~~~~~~~~~
>> +
>> +NUMA is required to run dpdk in x86.
>> +
>> +.. note::
>> +
>> +   For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2,
>> +   otherwise the compilation will fail with errors.
>> +
>> +.. code-block:: console
>> +
>> +	git clone https://github.com/numactl/numactl.git
>> +	cd numactl
>> +	git checkout v2.0.13 -b v2.0.13
>> +	./autogen.sh
>> +	autoconf -i
>> +	export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
>> +	./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
>> +	make install
>> +
>> +The numa header files and lib file is generated in the include and lib folder respectively under <OpenWrt SDK toolchain dir>.
>> +
>> +Build DPDK
>> +~~~~~~~~~~
>> +
>> +.. code-block:: console
>> +
>> +	export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
>> +	export RTE_SDK=`pwd`
>> +	export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
>> +	make config T=x86_64-native-linuxapp-gcc
>> +	make -j 100 CROSS=x86_64-openwrt-linux-gnu-
>> +
>> +Running DPDK application on OpenWrt
>> +-----------------------------------
>> +
>> +Virtual machine
>> +~~~~~~~~~~~~~~~
>> +
>> +* Extract boot image
>> +
>> +.. code-block:: console
>> +
>> +	gzip -d openwrt-x86-64-combined-ext4.img.gz
>> +
>> +* Launch Qemu
>> +
>> +.. code-block:: console
>> +
>> +	qemu-system-x86_64 \
>> +	        -cpu host \
>> +	        -smp 8 \
>> +	        -enable-kvm \
>> +	        -M q35 \
>> +	        -m 2048M \
>> +	        -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
>> +	        -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
>> +	        -device ide-hd,drive=d0,bus=ide.0 \
>> +	        -net nic,vlan=0 \
>> +	        -net nic,vlan=1 \
>> +	        -net user,vlan=1 \
>> +	        -display none \
>> +
>> +
>> +Physical machine
>> +~~~~~~~~~~~~~~~~
>> +
>> +Installation
>> +
>> +If you are using Windows PC, you can use some free and opensource raw disk image writer program such as
>> +``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image (openwrt-x86-64-combined-ext4.img) to a USB
>> +flash driver or USB SDcard with SDcard or a Sata hard drivre or SSD from your PC.
>> +
>> +If you are using Linux, you can use old dd tool to write OpenWrt image to the drive you want to write the
>> +image on.
>> +
>> +.. code-block:: console
>> +
>> +	dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
>> +
>> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
>> +
>> +Running DPDK
>> +~~~~~~~~~~~~
>> +
>> +* Setup dpdk environment
>> +
>> +  * Scp built numa libraries (including soft link) to /usr/lib64
>> +  * Setup hugepages
>> +  * insmod igb_uio.ko (scp the built igb_uio.ko first)
>> +  * Bind the NIC to igb_uio.ko
>> +
>> +* Launch testpmd
>> +
>> +  * scp built testpmd to qemu
>> +  * ./testpmd -c 0xf -- -i
>> +
>> + More detailed info about how to run a DPDK application refer to ``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
>
>It is good to have more documentation and examples.
>My one concern is that a lot of this overlaps existing documents and is likely
>to get out of date.  Could you use links instead to describe how
>to install OpenWrt and how to run testpmd.

Make sense, I'll use links for those steps.

Thanks,
Xiaolong

[dpdk-dev] [PATCH v2] doc: introduce openwrt how-to guide

[Xiaolong Ye]

This doc describes how to enable DPDK on openwrt in both virtual and
physical x86 environment.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---

v2 changes:

1. add meson build steps for dpdk
2. replace steps about build openwrt and running dpdk application with links

 doc/guides/howto/index.rst   |   1 +
 doc/guides/howto/openwrt.rst | 181 +++++++++++++++++++++++++++++++++++
 2 files changed, 182 insertions(+)
 create mode 100644 doc/guides/howto/openwrt.rst

diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a4c131652..5a97ea508 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -19,3 +19,4 @@ HowTo Guides
     packet_capture_framework
     telemetry
     debug_troubleshoot
+    openwrt
diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
new file mode 100644
index 000000000..b5ad66466
--- /dev/null
+++ b/doc/guides/howto/openwrt.rst
@@ -0,0 +1,181 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Enable DPDK on openwrt
+======================
+
+This document describes how to enable Data Plane Development Kit(DPDK) on
+Openwrt in both virtual and physical x86 environment.
+
+Introduction
+------------
+
+The OpenWrt project is a Linux operating system targeting embedded devices.
+Instead of trying to create a single, static firmware, OpenWrt provides a fully
+writable filesystem with package management. This frees user from the
+application selection and configuration provided by the vendor and allows user
+to customize the device through the use of packages to suit any application. For
+developers, OpenWrt is the framework to build and application without having to
+build a complete firmware around it, for users this means the ability for full
+customization, to use the device in ways never envisioned.
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
+unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
+
+Build OpenWrt
+-------------
+
+You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
+fully customize your own OpenWrt, it is highly recommended to build it through
+the source code, you can clone the OpenWrt source code by:
+
+.. code-block:: console
+
+	git clone https://git.openwrt.org/openwrt/openwrt.git
+
+OpenWrt configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+* Select ``x86`` in ``Target System``
+* Select ``x86_64`` in ``Subtarget``
+* Select ``Build the OpenWrt SDK`` for cross-compilation environment
+* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
+			   -> ``ToolChain Options``
+			   -> ``C Library implementation``
+
+Kernel configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Below configurations need to be enabled:
+
+* CONFIG_UIO=y
+* CONFIG_HUGETLBFS=y
+* CONFIG_HUGETLB_PAGE=y
+* CONFIG_PAGE_MONITOR=y
+
+Build steps
+~~~~~~~~~~~
+
+For detailed OpenWrt build steps, please refer to guide in its official site.
+
+`OpenWrt build guide
+<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
+
+After build is done, you can find the images and sdk in ``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
+
+DPDK Cross Compilation for OpenWrt
+----------------------------------
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+NUMA is required to run dpdk in x86.
+
+.. note::
+
+	For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2,
+otherwise the compilation will fail with errors.
+
+.. code-block:: console
+
+	git clone https://github.com/numactl/numactl.git
+	cd numactl
+	git checkout v2.0.13 -b v2.0.13
+	./autogen.sh
+	autoconf -i
+	export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
+	./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
+	make install
+
+The numa header files and lib file is generated in the include and lib folder respectively under <OpenWrt SDK toolchain dir>.
+
+Build DPDK
+~~~~~~~~~~
+
+* meson build
+
+To cross compile with meson build, you need to write customized cross file first.
+
+.. code-block:: console
+
+	[binaries]
+	c = 'x86_64-openwrt-linux-gcc'
+	cpp = 'x86_64-openwrt-linux-cpp'
+	ar = 'x86_64-openwrt-linux-ar'
+	strip = 'x86_64-openwrt-linux-strip'
+
+	meson builddir --cross-file openwrt-cross
+	ninja -C builddir
+
+.. note::
+
+	For compiling the igb_uio with the kernel version used in target machine, you need to explicitly specify kernel_dir in meson_options.txt.
+
+* make
+
+.. code-block:: console
+
+	export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
+	export RTE_SDK=`pwd`
+	export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
+	make config T=x86_64-native-linuxapp-gcc
+	make -j 100 CROSS=x86_64-openwrt-linux-gnu-
+
+Running DPDK application on OpenWrt
+-----------------------------------
+
+Virtual machine
+~~~~~~~~~~~~~~~
+
+* Extract boot image
+
+.. code-block:: console
+
+	gzip -d openwrt-x86-64-combined-ext4.img.gz
+
+* Launch Qemu
+
+.. code-block:: console
+
+	qemu-system-x86_64 \
+	        -cpu host \
+	        -smp 8 \
+	        -enable-kvm \
+	        -M q35 \
+	        -m 2048M \
+	        -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
+	        -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
+	        -device ide-hd,drive=d0,bus=ide.0 \
+	        -net nic,vlan=0 \
+	        -net nic,vlan=1 \
+	        -net user,vlan=1 \
+	        -display none \
+
+
+Physical machine
+~~~~~~~~~~~~~~~~
+
+If you are using Windows PC, you can use some free and opensource raw disk image writer program such as
+``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image (openwrt-x86-64-combined-ext4.img) to a USB
+flash driver or USB SDcard with SDcard or a Sata hard drivre or SSD from your PC.
+
+If you are using Linux, you can use old dd tool to write OpenWrt image to the drive you want to write the
+image on.
+
+.. code-block:: console
+
+	dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
+
+Where sdX is name of the drive. (You can find it though ``fdisk -l``)
+
+Running DPDK
+~~~~~~~~~~~~
+
+More detailed info about how to run a DPDK application please refer to ``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
+
+.. note::
+
+	You need to install pre-built numa libraries (including soft link) to /usr/lib64 in OpenWrt.
-- 
2.17.1

Re: [dpdk-dev] [PATCH v2] doc: introduce openwrt how-to guide

[Jerin Jacob]

On Mon, Dec 2, 2019 at 3:38 PM Xiaolong Ye <xiaolong.ye@intel.com> wrote:
>
> This doc describes how to enable DPDK on openwrt in both virtual and
> physical x86 environment.


Could you split the documentation into generic and x86 specific? So
other architectures
can add the architecture-specific details instead of duplicating.


>
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
>
> v2 changes:
>
> 1. add meson build steps for dpdk
> 2. replace steps about build openwrt and running dpdk application with links
>
>  doc/guides/howto/index.rst   |   1 +
>  doc/guides/howto/openwrt.rst | 181 +++++++++++++++++++++++++++++++++++
>  2 files changed, 182 insertions(+)
>  create mode 100644 doc/guides/howto/openwrt.rst
>
> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
> index a4c131652..5a97ea508 100644
> --- a/doc/guides/howto/index.rst
> +++ b/doc/guides/howto/index.rst
> @@ -19,3 +19,4 @@ HowTo Guides
>      packet_capture_framework
>      telemetry
>      debug_troubleshoot
> +    openwrt
> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
> new file mode 100644
> index 000000000..b5ad66466
> --- /dev/null
> +++ b/doc/guides/howto/openwrt.rst
> @@ -0,0 +1,181 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2019 Intel Corporation.
> +
> +Enable DPDK on openwrt
> +======================
> +
> +This document describes how to enable Data Plane Development Kit(DPDK) on
> +Openwrt in both virtual and physical x86 environment.
> +
> +Introduction
> +------------
> +
> +The OpenWrt project is a Linux operating system targeting embedded devices.
> +Instead of trying to create a single, static firmware, OpenWrt provides a fully
> +writable filesystem with package management. This frees user from the
> +application selection and configuration provided by the vendor and allows user
> +to customize the device through the use of packages to suit any application. For
> +developers, OpenWrt is the framework to build and application without having to
> +build a complete firmware around it, for users this means the ability for full
> +customization, to use the device in ways never envisioned.
> +
> +Pre-requisites
> +~~~~~~~~~~~~~~
> +
> +You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
> +unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
> +
> +Build OpenWrt
> +-------------
> +
> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
> +fully customize your own OpenWrt, it is highly recommended to build it through
> +the source code, you can clone the OpenWrt source code by:
> +
> +.. code-block:: console
> +
> +       git clone https://git.openwrt.org/openwrt/openwrt.git
> +
> +OpenWrt configuration
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +* Select ``x86`` in ``Target System``
> +* Select ``x86_64`` in ``Subtarget``
> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
> +                          -> ``ToolChain Options``
> +                          -> ``C Library implementation``
> +
> +Kernel configuration
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Below configurations need to be enabled:
> +
> +* CONFIG_UIO=y
> +* CONFIG_HUGETLBFS=y
> +* CONFIG_HUGETLB_PAGE=y
> +* CONFIG_PAGE_MONITOR=y
> +
> +Build steps
> +~~~~~~~~~~~
> +
> +For detailed OpenWrt build steps, please refer to guide in its official site.
> +
> +`OpenWrt build guide
> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
> +
> +After build is done, you can find the images and sdk in ``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
> +
> +DPDK Cross Compilation for OpenWrt
> +----------------------------------
> +
> +Pre-requisites
> +~~~~~~~~~~~~~~
> +
> +NUMA is required to run dpdk in x86.
> +
> +.. note::
> +
> +       For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2,
> +otherwise the compilation will fail with errors.
> +
> +.. code-block:: console
> +
> +       git clone https://github.com/numactl/numactl.git
> +       cd numactl
> +       git checkout v2.0.13 -b v2.0.13
> +       ./autogen.sh
> +       autoconf -i
> +       export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
> +       ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
> +       make install
> +
> +The numa header files and lib file is generated in the include and lib folder respectively under <OpenWrt SDK toolchain dir>.
> +
> +Build DPDK
> +~~~~~~~~~~
> +
> +* meson build
> +
> +To cross compile with meson build, you need to write customized cross file first.
> +
> +.. code-block:: console
> +
> +       [binaries]
> +       c = 'x86_64-openwrt-linux-gcc'
> +       cpp = 'x86_64-openwrt-linux-cpp'
> +       ar = 'x86_64-openwrt-linux-ar'
> +       strip = 'x86_64-openwrt-linux-strip'
> +
> +       meson builddir --cross-file openwrt-cross
> +       ninja -C builddir
> +
> +.. note::
> +
> +       For compiling the igb_uio with the kernel version used in target machine, you need to explicitly specify kernel_dir in meson_options.txt.
> +
> +* make
> +
> +.. code-block:: console
> +
> +       export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
> +       export RTE_SDK=`pwd`
> +       export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
> +       make config T=x86_64-native-linuxapp-gcc
> +       make -j 100 CROSS=x86_64-openwrt-linux-gnu-
> +
> +Running DPDK application on OpenWrt
> +-----------------------------------
> +
> +Virtual machine
> +~~~~~~~~~~~~~~~
> +
> +* Extract boot image
> +
> +.. code-block:: console
> +
> +       gzip -d openwrt-x86-64-combined-ext4.img.gz
> +
> +* Launch Qemu
> +
> +.. code-block:: console
> +
> +       qemu-system-x86_64 \
> +               -cpu host \
> +               -smp 8 \
> +               -enable-kvm \
> +               -M q35 \
> +               -m 2048M \
> +               -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
> +               -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
> +               -device ide-hd,drive=d0,bus=ide.0 \
> +               -net nic,vlan=0 \
> +               -net nic,vlan=1 \
> +               -net user,vlan=1 \
> +               -display none \
> +
> +
> +Physical machine
> +~~~~~~~~~~~~~~~~
> +
> +If you are using Windows PC, you can use some free and opensource raw disk image writer program such as
> +``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image (openwrt-x86-64-combined-ext4.img) to a USB
> +flash driver or USB SDcard with SDcard or a Sata hard drivre or SSD from your PC.
> +
> +If you are using Linux, you can use old dd tool to write OpenWrt image to the drive you want to write the
> +image on.
> +
> +.. code-block:: console
> +
> +       dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
> +
> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
> +
> +Running DPDK
> +~~~~~~~~~~~~
> +
> +More detailed info about how to run a DPDK application please refer to ``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
> +
> +.. note::
> +
> +       You need to install pre-built numa libraries (including soft link) to /usr/lib64 in OpenWrt.
> --
> 2.17.1
>

Re: [dpdk-dev] [PATCH v2] doc: introduce openwrt how-to guide

[Ye Xiaolong]

On 12/02, Jerin Jacob wrote:
>On Mon, Dec 2, 2019 at 3:38 PM Xiaolong Ye <xiaolong.ye@intel.com> wrote:
>>
>> This doc describes how to enable DPDK on openwrt in both virtual and
>> physical x86 environment.
>
>
>Could you split the documentation into generic and x86 specific? So
>other architectures
>can add the architecture-specific details instead of duplicating.

Actually most of the steps are generic, expect that user need to specify different
targets when running `make menuconfig` for building openwrt. I'll emphasize that
it related section.

Thanks,
Xiaolong

>
>
>>
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> ---
>>
>> v2 changes:
>>
>> 1. add meson build steps for dpdk
>> 2. replace steps about build openwrt and running dpdk application with links
>>
>>  doc/guides/howto/index.rst   |   1 +
>>  doc/guides/howto/openwrt.rst | 181 +++++++++++++++++++++++++++++++++++
>>  2 files changed, 182 insertions(+)
>>  create mode 100644 doc/guides/howto/openwrt.rst
>>
>> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
>> index a4c131652..5a97ea508 100644
>> --- a/doc/guides/howto/index.rst
>> +++ b/doc/guides/howto/index.rst
>> @@ -19,3 +19,4 @@ HowTo Guides
>>      packet_capture_framework
>>      telemetry
>>      debug_troubleshoot
>> +    openwrt
>> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
>> new file mode 100644
>> index 000000000..b5ad66466
>> --- /dev/null
>> +++ b/doc/guides/howto/openwrt.rst
>> @@ -0,0 +1,181 @@
>> +..  SPDX-License-Identifier: BSD-3-Clause
>> +    Copyright(c) 2019 Intel Corporation.
>> +
>> +Enable DPDK on openwrt
>> +======================
>> +
>> +This document describes how to enable Data Plane Development Kit(DPDK) on
>> +Openwrt in both virtual and physical x86 environment.
>> +
>> +Introduction
>> +------------
>> +
>> +The OpenWrt project is a Linux operating system targeting embedded devices.
>> +Instead of trying to create a single, static firmware, OpenWrt provides a fully
>> +writable filesystem with package management. This frees user from the
>> +application selection and configuration provided by the vendor and allows user
>> +to customize the device through the use of packages to suit any application. For
>> +developers, OpenWrt is the framework to build and application without having to
>> +build a complete firmware around it, for users this means the ability for full
>> +customization, to use the device in ways never envisioned.
>> +
>> +Pre-requisites
>> +~~~~~~~~~~~~~~
>> +
>> +You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
>> +unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
>> +
>> +Build OpenWrt
>> +-------------
>> +
>> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
>> +fully customize your own OpenWrt, it is highly recommended to build it through
>> +the source code, you can clone the OpenWrt source code by:
>> +
>> +.. code-block:: console
>> +
>> +       git clone https://git.openwrt.org/openwrt/openwrt.git
>> +
>> +OpenWrt configuration
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +* Select ``x86`` in ``Target System``
>> +* Select ``x86_64`` in ``Subtarget``
>> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
>> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
>> +                          -> ``ToolChain Options``
>> +                          -> ``C Library implementation``
>> +
>> +Kernel configuration
>> +~~~~~~~~~~~~~~~~~~~~
>> +
>> +Below configurations need to be enabled:
>> +
>> +* CONFIG_UIO=y
>> +* CONFIG_HUGETLBFS=y
>> +* CONFIG_HUGETLB_PAGE=y
>> +* CONFIG_PAGE_MONITOR=y
>> +
>> +Build steps
>> +~~~~~~~~~~~
>> +
>> +For detailed OpenWrt build steps, please refer to guide in its official site.
>> +
>> +`OpenWrt build guide
>> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
>> +
>> +After build is done, you can find the images and sdk in ``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
>> +
>> +DPDK Cross Compilation for OpenWrt
>> +----------------------------------
>> +
>> +Pre-requisites
>> +~~~~~~~~~~~~~~
>> +
>> +NUMA is required to run dpdk in x86.
>> +
>> +.. note::
>> +
>> +       For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2,
>> +otherwise the compilation will fail with errors.
>> +
>> +.. code-block:: console
>> +
>> +       git clone https://github.com/numactl/numactl.git
>> +       cd numactl
>> +       git checkout v2.0.13 -b v2.0.13
>> +       ./autogen.sh
>> +       autoconf -i
>> +       export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
>> +       ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
>> +       make install
>> +
>> +The numa header files and lib file is generated in the include and lib folder respectively under <OpenWrt SDK toolchain dir>.
>> +
>> +Build DPDK
>> +~~~~~~~~~~
>> +
>> +* meson build
>> +
>> +To cross compile with meson build, you need to write customized cross file first.
>> +
>> +.. code-block:: console
>> +
>> +       [binaries]
>> +       c = 'x86_64-openwrt-linux-gcc'
>> +       cpp = 'x86_64-openwrt-linux-cpp'
>> +       ar = 'x86_64-openwrt-linux-ar'
>> +       strip = 'x86_64-openwrt-linux-strip'
>> +
>> +       meson builddir --cross-file openwrt-cross
>> +       ninja -C builddir
>> +
>> +.. note::
>> +
>> +       For compiling the igb_uio with the kernel version used in target machine, you need to explicitly specify kernel_dir in meson_options.txt.
>> +
>> +* make
>> +
>> +.. code-block:: console
>> +
>> +       export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
>> +       export RTE_SDK=`pwd`
>> +       export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
>> +       make config T=x86_64-native-linuxapp-gcc
>> +       make -j 100 CROSS=x86_64-openwrt-linux-gnu-
>> +
>> +Running DPDK application on OpenWrt
>> +-----------------------------------
>> +
>> +Virtual machine
>> +~~~~~~~~~~~~~~~
>> +
>> +* Extract boot image
>> +
>> +.. code-block:: console
>> +
>> +       gzip -d openwrt-x86-64-combined-ext4.img.gz
>> +
>> +* Launch Qemu
>> +
>> +.. code-block:: console
>> +
>> +       qemu-system-x86_64 \
>> +               -cpu host \
>> +               -smp 8 \
>> +               -enable-kvm \
>> +               -M q35 \
>> +               -m 2048M \
>> +               -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
>> +               -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
>> +               -device ide-hd,drive=d0,bus=ide.0 \
>> +               -net nic,vlan=0 \
>> +               -net nic,vlan=1 \
>> +               -net user,vlan=1 \
>> +               -display none \
>> +
>> +
>> +Physical machine
>> +~~~~~~~~~~~~~~~~
>> +
>> +If you are using Windows PC, you can use some free and opensource raw disk image writer program such as
>> +``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image (openwrt-x86-64-combined-ext4.img) to a USB
>> +flash driver or USB SDcard with SDcard or a Sata hard drivre or SSD from your PC.
>> +
>> +If you are using Linux, you can use old dd tool to write OpenWrt image to the drive you want to write the
>> +image on.
>> +
>> +.. code-block:: console
>> +
>> +       dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
>> +
>> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
>> +
>> +Running DPDK
>> +~~~~~~~~~~~~
>> +
>> +More detailed info about how to run a DPDK application please refer to ``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
>> +
>> +.. note::
>> +
>> +       You need to install pre-built numa libraries (including soft link) to /usr/lib64 in OpenWrt.
>> --
>> 2.17.1
>>

[dpdk-dev] [PATCH v3] doc: introduce openwrt how-to guide

[Xiaolong Ye]

This doc describes how to enable DPDK on openwrt in both virtual and
physical x86 environment.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---

V3 changes:

1. emphasize target select in `OpenWrt configuration` section

v2 changes:

1. add meson build steps for dpdk
2. replace steps about build openwrt and running dpdk application with links

 doc/guides/howto/index.rst   |   1 +
 doc/guides/howto/openwrt.rst | 184 +++++++++++++++++++++++++++++++++++
 2 files changed, 185 insertions(+)
 create mode 100644 doc/guides/howto/openwrt.rst

diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a4c131652..5a97ea508 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -19,3 +19,4 @@ HowTo Guides
     packet_capture_framework
     telemetry
     debug_troubleshoot
+    openwrt
diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
new file mode 100644
index 000000000..703e30497
--- /dev/null
+++ b/doc/guides/howto/openwrt.rst
@@ -0,0 +1,184 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Enable DPDK on openwrt
+======================
+
+This document describes how to enable Data Plane Development Kit(DPDK) on
+Openwrt in both virtual and physical machine.
+
+Introduction
+------------
+
+The OpenWrt project is a Linux operating system targeting embedded devices.
+Instead of trying to create a single, static firmware, OpenWrt provides a fully
+writable filesystem with package management. This frees user from the
+application selection and configuration provided by the vendor and allows user
+to customize the device through the use of packages to suit any application. For
+developers, OpenWrt is the framework to build and application without having to
+build a complete firmware around it, for users this means the ability for full
+customization, to use the device in ways never envisioned.
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
+unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
+
+Build OpenWrt
+-------------
+
+You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
+fully customize your own OpenWrt, it is highly recommended to build it through
+the source code, you can clone the OpenWrt source code by:
+
+.. code-block:: console
+
+	git clone https://git.openwrt.org/openwrt/openwrt.git
+
+OpenWrt configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+You need to specify the Traget System & Subtarget through OpenWrt configuration,
+take x86_64 for example, you need to:
+
+* Select ``x86`` in ``Target System``
+* Select ``x86_64`` in ``Subtarget``
+* Select ``Build the OpenWrt SDK`` for cross-compilation environment
+* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
+			   -> ``ToolChain Options``
+			   -> ``C Library implementation``
+
+Kernel configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Below configurations need to be enabled:
+
+* CONFIG_UIO=y
+* CONFIG_HUGETLBFS=y
+* CONFIG_HUGETLB_PAGE=y
+* CONFIG_PAGE_MONITOR=y
+
+Build steps
+~~~~~~~~~~~
+
+For detailed OpenWrt build steps, please refer to guide in its official site.
+
+`OpenWrt build guide
+<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
+
+After build is done, you can find the images and sdk in ``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
+
+DPDK Cross Compilation for OpenWrt
+----------------------------------
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+NUMA is required to run dpdk in x86.
+
+.. note::
+
+	For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2,
+otherwise the compilation will fail with errors.
+
+.. code-block:: console
+
+	git clone https://github.com/numactl/numactl.git
+	cd numactl
+	git checkout v2.0.13 -b v2.0.13
+	./autogen.sh
+	autoconf -i
+	export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
+	./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
+	make install
+
+The numa header files and lib file is generated in the include and lib folder respectively under <OpenWrt SDK toolchain dir>.
+
+Build DPDK
+~~~~~~~~~~
+
+* meson build
+
+To cross compile with meson build, you need to write customized cross file first.
+
+.. code-block:: console
+
+	[binaries]
+	c = 'x86_64-openwrt-linux-gcc'
+	cpp = 'x86_64-openwrt-linux-cpp'
+	ar = 'x86_64-openwrt-linux-ar'
+	strip = 'x86_64-openwrt-linux-strip'
+
+	meson builddir --cross-file openwrt-cross
+	ninja -C builddir
+
+.. note::
+
+	For compiling the igb_uio with the kernel version used in target machine, you need to explicitly specify kernel_dir in meson_options.txt.
+
+* make
+
+.. code-block:: console
+
+	export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
+	export RTE_SDK=`pwd`
+	export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
+	make config T=x86_64-native-linuxapp-gcc
+	make -j 100 CROSS=x86_64-openwrt-linux-gnu-
+
+Running DPDK application on OpenWrt
+-----------------------------------
+
+Virtual machine
+~~~~~~~~~~~~~~~
+
+* Extract boot image
+
+.. code-block:: console
+
+	gzip -d openwrt-x86-64-combined-ext4.img.gz
+
+* Launch Qemu
+
+.. code-block:: console
+
+	qemu-system-x86_64 \
+	        -cpu host \
+	        -smp 8 \
+	        -enable-kvm \
+	        -M q35 \
+	        -m 2048M \
+	        -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
+	        -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
+	        -device ide-hd,drive=d0,bus=ide.0 \
+	        -net nic,vlan=0 \
+	        -net nic,vlan=1 \
+	        -net user,vlan=1 \
+	        -display none \
+
+
+Physical machine
+~~~~~~~~~~~~~~~~
+
+If you are using Windows PC, you can use some free and opensource raw disk image writer program such as
+``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image (openwrt-x86-64-combined-ext4.img) to a USB
+flash driver or USB SDcard with SDcard or a Sata hard drivre or SSD from your PC.
+
+If you are using Linux, you can use old dd tool to write OpenWrt image to the drive you want to write the
+image on.
+
+.. code-block:: console
+
+	dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
+
+Where sdX is name of the drive. (You can find it though ``fdisk -l``)
+
+Running DPDK
+~~~~~~~~~~~~
+
+More detailed info about how to run a DPDK application please refer to ``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
+
+.. note::
+
+	You need to install pre-built numa libraries (including soft link) to /usr/lib64 in OpenWrt.
-- 
2.17.1

[dpdk-dev] [PATCH v4] doc: introduce openwrt how-to guide

[Xiaolong Ye]

This doc describes how to enable DPDK on openwrt in both virtual and
physical x86 environment.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---

V4 changes:

1. add release notes

V3 changes:

1. emphasize target select in `OpenWrt configuration` section

V2 changes:

1. add meson build steps for dpdk
2. replace steps about build openwrt and running dpdk application with
links

 doc/guides/howto/index.rst             |   1 +
 doc/guides/howto/openwrt.rst           | 184 +++++++++++++++++++++++++
 doc/guides/rel_notes/release_20_02.rst |   4 +
 3 files changed, 189 insertions(+)
 create mode 100644 doc/guides/howto/openwrt.rst

diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a4c131652..5a97ea508 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -19,3 +19,4 @@ HowTo Guides
     packet_capture_framework
     telemetry
     debug_troubleshoot
+    openwrt
diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
new file mode 100644
index 000000000..05097c62d
--- /dev/null
+++ b/doc/guides/howto/openwrt.rst
@@ -0,0 +1,184 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Enable DPDK on openwrt
+======================
+
+This document describes how to enable Data Plane Development Kit(DPDK) on
+OpenWrt in both virtual and physical machine.
+
+Introduction
+------------
+
+The OpenWrt project is a Linux operating system targeting embedded devices.
+Instead of trying to create a single, static firmware, OpenWrt provides a fully
+writable filesystem with package management. This frees user from the
+application selection and configuration provided by the vendor and allows user
+to customize the device through the use of packages to suit any application. For
+developers, OpenWrt is the framework to build and application without having to
+build a complete firmware around it, for users this means the ability for full
+customization, to use the device in ways never envisioned.
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
+unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
+
+Build OpenWrt
+-------------
+
+You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
+fully customize your own OpenWrt, it is highly recommended to build it through
+the source code, you can clone the OpenWrt source code by:
+
+.. code-block:: console
+
+	git clone https://git.openwrt.org/openwrt/openwrt.git
+
+OpenWrt configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+You need to specify the Traget System & Subtarget through OpenWrt configuration,
+take x86_64 for example, you need to:
+
+* Select ``x86`` in ``Target System``
+* Select ``x86_64`` in ``Subtarget``
+* Select ``Build the OpenWrt SDK`` for cross-compilation environment
+* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
+			   -> ``ToolChain Options``
+			   -> ``C Library implementation``
+
+Kernel configuration
+~~~~~~~~~~~~~~~~~~~~
+
+Below configurations need to be enabled:
+
+* CONFIG_UIO=y
+* CONFIG_HUGETLBFS=y
+* CONFIG_HUGETLB_PAGE=y
+* CONFIG_PAGE_MONITOR=y
+
+Build steps
+~~~~~~~~~~~
+
+For detailed OpenWrt build steps, please refer to guide in its official site.
+
+`OpenWrt build guide
+<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
+
+After build is done, you can find the images and sdk in ``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
+
+DPDK Cross Compilation for OpenWrt
+----------------------------------
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+NUMA is required to run dpdk in x86.
+
+.. note::
+
+	For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2,
+otherwise the compilation will fail with errors.
+
+.. code-block:: console
+
+	git clone https://github.com/numactl/numactl.git
+	cd numactl
+	git checkout v2.0.13 -b v2.0.13
+	./autogen.sh
+	autoconf -i
+	export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
+	./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
+	make install
+
+The numa header files and lib file is generated in the include and lib folder respectively under <OpenWrt SDK toolchain dir>.
+
+Build DPDK
+~~~~~~~~~~
+
+* meson build
+
+To cross compile with meson build, you need to write customized cross file first.
+
+.. code-block:: console
+
+	[binaries]
+	c = 'x86_64-openwrt-linux-gcc'
+	cpp = 'x86_64-openwrt-linux-cpp'
+	ar = 'x86_64-openwrt-linux-ar'
+	strip = 'x86_64-openwrt-linux-strip'
+
+	meson builddir --cross-file openwrt-cross
+	ninja -C builddir
+
+.. note::
+
+	For compiling the igb_uio with the kernel version used in target machine, you need to explicitly specify kernel_dir in meson_options.txt.
+
+* make
+
+.. code-block:: console
+
+	export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
+	export RTE_SDK=`pwd`
+	export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
+	make config T=x86_64-native-linuxapp-gcc
+	make -j 100 CROSS=x86_64-openwrt-linux-gnu-
+
+Running DPDK application on OpenWrt
+-----------------------------------
+
+Virtual machine
+~~~~~~~~~~~~~~~
+
+* Extract boot image
+
+.. code-block:: console
+
+	gzip -d openwrt-x86-64-combined-ext4.img.gz
+
+* Launch Qemu
+
+.. code-block:: console
+
+	qemu-system-x86_64 \
+	        -cpu host \
+	        -smp 8 \
+	        -enable-kvm \
+	        -M q35 \
+	        -m 2048M \
+	        -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
+	        -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
+	        -device ide-hd,drive=d0,bus=ide.0 \
+	        -net nic,vlan=0 \
+	        -net nic,vlan=1 \
+	        -net user,vlan=1 \
+	        -display none \
+
+
+Physical machine
+~~~~~~~~~~~~~~~~
+
+If you are using Windows PC, you can use some free and opensource raw disk image writer program such as
+``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image (openwrt-x86-64-combined-ext4.img) to a USB
+flash driver or USB SDcard with SDcard or a Sata hard drivre or SSD from your PC.
+
+If you are using Linux, you can use old dd tool to write OpenWrt image to the drive you want to write the
+image on.
+
+.. code-block:: console
+
+	dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
+
+Where sdX is name of the drive. (You can find it though ``fdisk -l``)
+
+Running DPDK
+~~~~~~~~~~~~
+
+More detailed info about how to run a DPDK application please refer to ``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
+
+.. note::
+
+	You need to install pre-built numa libraries (including soft link) to /usr/lib64 in OpenWrt.
diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
index 0eaa45a76..78390fdb2 100644
--- a/doc/guides/rel_notes/release_20_02.rst
+++ b/doc/guides/rel_notes/release_20_02.rst
@@ -56,6 +56,10 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **Added OpenWrt howto guide.**
+
+  Added document describes how to enable DPDK on OpenWrt in both virtual and
+  physical machine.
 
 Removed Items
 -------------
-- 
2.17.1

Re: [dpdk-dev] [PATCH v4] doc: introduce openwrt how-to guide

[Zhang, Xiao]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Xiaolong Ye
> Sent: Monday, December 16, 2019 11:01 AM
> To: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>
> Cc: dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Stephen
> Hemminger <stephen@networkplumber.org>; Ye, Xiaolong
> <xiaolong.ye@intel.com>
> Subject: [dpdk-dev] [PATCH v4] doc: introduce openwrt how-to guide
> 
> This doc describes how to enable DPDK on openwrt in both virtual and physical
> x86 environment.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>

Acked-by: Xiao Zhang <xiao.zhang@intel.com>

Re: [dpdk-dev] [PATCH v4] doc: introduce openwrt how-to guide

[Mcnamara, John]

Hi,

Overall looks good. Some notes below:

> +
> +.. note::
> +
> +	For compiling the NUMA lib, run libtool --version to ensure the
> +libtool version >= 2.2, otherwise the compilation will fail with errors.


This continuation line isn't indented and raised a build error:

    $ make doc-guides-html                                
    sphinx processing guides-html...
    /work/dpdk_docs/doc/guides/howto/openwrt.rst:80: WARNING: 
    Explicit markup ends without a blank line; unexpected unindent.

Also you should used spaces not tabs for indentation in the docs.

> +If you are using Windows PC, you can use some free and opensource raw
> +disk image writer program such as
> +``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image
> +(openwrt-x86-64-combined-ext4.img) to a USB flash driver or USB SDcard
> with SDcard or a Sata hard drivre or SSD from your PC.

Typo: drive

I'll send you a few other suggested minor corrections.

John

Re: [dpdk-dev] [PATCH v4] doc: introduce openwrt how-to guide

[Ye Xiaolong]

Hi, John

Thanks for your review.

On 01/17, Mcnamara, John wrote:
>Hi,
>
>Overall looks good. Some notes below:
>
>> +
>> +.. note::
>> +
>> +For compiling the NUMA lib, run libtool --version to ensure the
>> +libtool version >= 2.2, otherwise the compilation will fail with errors.
>
>
>This continuation line isn't indented and raised a build error:
>
>    $ make doc-guides-html
>    sphinx processing guides-html...
>    /work/dpdk_docs/doc/guides/howto/openwrt.rst:80: WARNING:
>    Explicit markup ends without a blank line; unexpected unindent.
>
>Also you should used spaces not tabs for indentation in the docs.

Got it.

>
>> +If you are using Windows PC, you can use some free and opensource raw
>> +disk image writer program such as
>> +``Win32 Disk Imager`` and ``Etcher`` to write OpenWrt image
>> +(openwrt-x86-64-combined-ext4.img) to a USB flash driver or USB SDcard
>> with SDcard or a Sata hard drivre or SSD from your PC.
>
>Typo: drive
>
>I'll send you a few other suggested minor corrections.

I'll send a new version after I correct all the suggestions.

Thanks,
Xiaolong

>
>John
>

[dpdk-dev] [PATCH v5] doc: introduce openwrt how-to guide

[Xiaolong Ye]

This doc describes how to enable DPDK on Openwrt in both virtual and
physical x86 environment.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---

V5 changes:

1. improve the doc's grammar and wording according to John's
suggestions.

V4 changes:

1. add release notes

V3 changes:

1. emphasize target select in `OpenWrt configuration` section

V2 changes:

1. add meson build steps for dpdk
2. replace steps about build openwrt and running dpdk application with
links

 doc/guides/howto/index.rst             |   1 +
 doc/guides/howto/openwrt.rst           | 184 +++++++++++++++++++++++++
 doc/guides/rel_notes/release_20_02.rst |   4 +
 3 files changed, 189 insertions(+)
 create mode 100644 doc/guides/howto/openwrt.rst

diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a4c131652..5a97ea508 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -19,3 +19,4 @@ HowTo Guides
     packet_capture_framework
     telemetry
     debug_troubleshoot
+    openwrt
diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
new file mode 100644
index 000000000..fceab1b91
--- /dev/null
+++ b/doc/guides/howto/openwrt.rst
@@ -0,0 +1,184 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Enable DPDK on openwrt
+======================
+
+This document describes how to enable Data Plane Development Kit (DPDK) on
+Openwrt in both a virtual and physical x86 environment.
+
+Introduction
+------------
+
+The OpenWrt project is a Linux operating system targeting embedded devices.
+Instead of trying to create a single, static firmware, OpenWrt provides a fully
+writable filesystem with package management. This frees the user from the
+application selection and configuration provided by the vendor and allows users
+to customize the device through the use of packages to suit any application. For
+developers OpenWrt is the framework to build an application without having to
+build a complete firmware around it. For users is offers full customization
+to use the device in ways never envisioned.
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
+unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
+
+Build OpenWrt
+-------------
+
+You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
+fully customize your own OpenWrt, it is highly recommended to build it from
+the source code. You can clone the OpenWrt source code as follows:
+
+.. code-block:: console
+
+    git clone https://git.openwrt.org/openwrt/openwrt.git
+
+OpenWrt configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+* Select ``x86`` in ``Target System``
+* Select ``x86_64`` in ``Subtarget``
+* Select ``Build the OpenWrt SDK`` for cross-compilation environment
+* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
+  then ``ToolChain Options`` and ``C Library implementation``
+
+Kernel configuration
+~~~~~~~~~~~~~~~~~~~~
+
+The following configurations should be enabled:
+
+* ``CONFIG_UIO=y``
+* ``CONFIG_HUGETLBFS=y``
+* ``CONFIG_HUGETLB_PAGE=y``
+* ``CONFIG_PAGE_MONITOR=y``
+
+Build steps
+~~~~~~~~~~~
+
+For detailed OpenWrt build steps, please refer to the
+`OpenWrt build guide
+<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
+
+After the build is completed, you can find the images and sdk in
+``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
+
+
+DPDK Cross Compilation for OpenWrt
+----------------------------------
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+NUMA is required to run DPDK in x86.
+
+.. note::
+
+    For compiling the NUMA lib, run ``libtool --version`` to ensure the libtool
+    version >= 2.2, otherwise the compilation will fail with errors.
+
+.. code-block:: console
+
+    git clone https://github.com/numactl/numactl.git
+    cd numactl
+    git checkout v2.0.13 -b v2.0.13
+    ./autogen.sh
+    autoconf -i
+    export PATH=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
+    ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
+    make install
+
+The numa header files and lib file is generated in the include and lib folder
+respectively under <OpenWrt SDK toolchain dir>.
+
+Build DPDK
+~~~~~~~~~~
+
+To cross compile with meson build, you need to write a customized cross file
+first.
+
+.. code-block:: console
+
+    [binaries]
+    c = 'x86_64-openwrt-linux-gcc'
+    cpp = 'x86_64-openwrt-linux-cpp'
+    ar = 'x86_64-openwrt-linux-ar'
+    strip = 'x86_64-openwrt-linux-strip'
+
+    meson builddir --cross-file openwrt-cross
+    ninja -C builddir
+
+.. note::
+
+    For compiling the igb_uio with the kernel version used in target machine,
+    you need to explicitly specify kernel_dir in meson_options.txt.
+
+To cross compile with make:
+
+.. code-block:: console
+
+    export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
+    export RTE_SDK=`pwd`
+    export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
+    make config T=x86_64-native-linuxapp-gcc
+    make -j 100 CROSS=x86_64-openwrt-linux-gnu-
+
+Running DPDK application on OpenWrt
+-----------------------------------
+
+Virtual machine
+~~~~~~~~~~~~~~~
+
+* Extract the boot image
+
+.. code-block:: console
+
+    gzip -d openwrt-x86-64-combined-ext4.img.gz
+
+* Launch Qemu
+
+.. code-block:: console
+
+    qemu-system-x86_64 \
+            -cpu host \
+            -smp 8 \
+            -enable-kvm \
+            -M q35 \
+            -m 2048M \
+            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
+            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
+            -device ide-hd,drive=d0,bus=ide.0 \
+            -net nic,vlan=0 \
+            -net nic,vlan=1 \
+            -net user,vlan=1 \
+            -display none \
+
+
+Physical machine
+~~~~~~~~~~~~~~~~
+
+If you are using a Windows PC, you can use an image writer application such as
+``Win32 Disk Imager`` and ``Etcher`` to write the OpenWrt image
+(openwrt-x86-64-combined-ext4.img) to a USB flash driver or SDcard.
+
+If you are using Linux, you can use the ``dd`` tool to write the OpenWrt image
+to the drive you want to write the image on.
+
+.. code-block:: console
+
+    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
+
+Where sdX is name of the drive. (You can find it though ``fdisk -l``)
+
+Running DPDK
+~~~~~~~~~~~~
+
+More detailed info about how to run a DPDK application please refer to
+``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
+
+.. note::
+
+    You need to install pre-built numa libraries (including soft link)
+    to /usr/lib64 in OpenWrt.
diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
index 08f571268..9e1e0dd89 100644
--- a/doc/guides/rel_notes/release_20_02.rst
+++ b/doc/guides/rel_notes/release_20_02.rst
@@ -116,6 +116,10 @@ New Features
 
   Added support for ESP rte_flow patterns to the testpmd application.
 
+* **Added OpenWrt howto guide.**
+
+  Added document describes how to enable DPDK on OpenWrt in both virtual and
+  physical machine.
 
 Removed Items
 -------------
-- 
2.17.1

Re: [dpdk-dev] [PATCH v5] doc: introduce openwrt how-to guide

[Mcnamara, John]

> -----Original Message-----
> From: Ye, Xiaolong <xiaolong.ye@intel.com>
> Sent: Saturday, January 18, 2020 5:49 AM
> To: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>
> Cc: dev@dpdk.org; Richardson, Bruce <bruce.richardson@intel.com>; Stephen
> Hemminger <stephen@networkplumber.org>; Ye, Xiaolong
> <xiaolong.ye@intel.com>
> Subject: [PATCH v5] doc: introduce openwrt how-to guide
> 
> This doc describes how to enable DPDK on Openwrt in both virtual and
> physical x86 environment.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>

Acked-by: John McNamara <john.mcnamara@intel.com>

Re: [dpdk-dev] [PATCH v5] doc: introduce openwrt how-to guide

[Thomas Monjalon]

18/01/2020 06:48, Xiaolong Ye:
> This doc describes how to enable DPDK on Openwrt in both virtual and
> physical x86 environment.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
> --- /dev/null
> +++ b/doc/guides/howto/openwrt.rst
> +Enable DPDK on openwrt
> +======================
> +
> +This document describes how to enable Data Plane Development Kit (DPDK) on
> +Openwrt in both a virtual and physical x86 environment.

Please take care of the uppercases in OpenWrt.

> +
> +Introduction
> +------------
> +
> +The OpenWrt project is a Linux operating system targeting embedded devices.
> +Instead of trying to create a single, static firmware, OpenWrt provides a fully
> +writable filesystem with package management. This frees the user from the
> +application selection and configuration provided by the vendor and allows users
> +to customize the device through the use of packages to suit any application. For
> +developers OpenWrt is the framework to build an application without having to
> +build a complete firmware around it. For users is offers full customization
> +to use the device in ways never envisioned.

Why doing marketing for OpenWrt in DPDK doc?
It is a copy paste from https://openwrt.org/
I think it is enough to say OpenWrt is a source-based router OS with package management.

> +
> +Pre-requisites
> +~~~~~~~~~~~~~~
> +
> +You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
> +unzip, gawk, getopt, subversion, libz-dev and libc headers installed.

These are prerequisites for building, so should be in section below.
Maybe we can just provide a link to the official doc instead.

> +
> +Build OpenWrt
> +-------------
> +
> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To

Please take the habit of splitting lines after a punctuation.

> +fully customize your own OpenWrt, it is highly recommended to build it from
> +the source code. You can clone the OpenWrt source code as follows:
> +
> +.. code-block:: console
> +
> +    git clone https://git.openwrt.org/openwrt/openwrt.git
> +
> +OpenWrt configuration
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +* Select ``x86`` in ``Target System``
> +* Select ``x86_64`` in ``Subtarget``
> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
> +  then ``ToolChain Options`` and ``C Library implementation``
> +
> +Kernel configuration
> +~~~~~~~~~~~~~~~~~~~~
> +
> +The following configurations should be enabled:
> +
> +* ``CONFIG_UIO=y``

No, UIO is not recommended anymore.

> +* ``CONFIG_HUGETLBFS=y``
> +* ``CONFIG_HUGETLB_PAGE=y``
> +* ``CONFIG_PAGE_MONITOR=y``

Why PAGE_MONITOR?

> +
> +Build steps
> +~~~~~~~~~~~
> +
> +For detailed OpenWrt build steps, please refer to the
> +`OpenWrt build guide
> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
> +
> +After the build is completed, you can find the images and sdk in
> +``<OpenWrt Root>/bin/targets/x86/64-glibc/``.

s/sdk/SDK/

[...]
> +    [binaries]
> +    c = 'x86_64-openwrt-linux-gcc'
> +    cpp = 'x86_64-openwrt-linux-cpp'
> +    ar = 'x86_64-openwrt-linux-ar'
> +    strip = 'x86_64-openwrt-linux-strip'
> +
> +    meson builddir --cross-file openwrt-cross

This is really a good example why meson should allow to override the toolchain prefix.
Please work with meson on this topic.

[...]
> +To cross compile with make:
> +
> +.. code-block:: console
> +
> +    export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
> +    export RTE_SDK=`pwd`
> +    export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
> +    make config T=x86_64-native-linuxapp-gcc
> +    make -j 100 CROSS=x86_64-openwrt-linux-gnu-

The make build system should not be documented as it is going to be deprecated.

> +
> +Running DPDK application on OpenWrt
> +-----------------------------------
> +
> +Virtual machine
> +~~~~~~~~~~~~~~~
> +
> +* Extract the boot image
> +
> +.. code-block:: console
> +
> +    gzip -d openwrt-x86-64-combined-ext4.img.gz
> +
> +* Launch Qemu
> +
> +.. code-block:: console
> +
> +    qemu-system-x86_64 \
> +            -cpu host \
> +            -smp 8 \
> +            -enable-kvm \
> +            -M q35 \
> +            -m 2048M \
> +            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
> +            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
> +            -device ide-hd,drive=d0,bus=ide.0 \
> +            -net nic,vlan=0 \
> +            -net nic,vlan=1 \
> +            -net user,vlan=1 \
> +            -display none \
> +
> +
> +Physical machine
> +~~~~~~~~~~~~~~~~
> +
> +If you are using a Windows PC, you can use an image writer application such as
> +``Win32 Disk Imager`` and ``Etcher`` to write the OpenWrt image
> +(openwrt-x86-64-combined-ext4.img) to a USB flash driver or SDcard.

If you just built a DPDK for Linux, you are not running Windows,
except if you are using a Linux server for compilation.
Anyway, we should not try to document such procedure in my opinion.
I'm sure you can refer to the OpenWrt documentation for flashing the image.

> +
> +If you are using Linux, you can use the ``dd`` tool to write the OpenWrt image
> +to the drive you want to write the image on.
> +
> +.. code-block:: console
> +
> +    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
> +
> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
> +
> +Running DPDK
> +~~~~~~~~~~~~
> +
> +More detailed info about how to run a DPDK application please refer to
> +``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
> +
> +.. note::
> +
> +    You need to install pre-built numa libraries (including soft link)

s/numa/NUMA/

> +    to /usr/lib64 in OpenWrt.

libnuma is not packaged in OpenWrt?


> --- a/doc/guides/rel_notes/release_20_02.rst
> +++ b/doc/guides/rel_notes/release_20_02.rst
> @@ -116,6 +116,10 @@ New Features
>  
>    Added support for ESP rte_flow patterns to the testpmd application.
>  
> +* **Added OpenWrt howto guide.**
> +
> +  Added document describes how to enable DPDK on OpenWrt in both virtual and
> +  physical machine.
>  
>  Removed Items
>  -------------

You miss a doubled blank line before this title.

Re: [dpdk-dev] [PATCH v5] doc: introduce openwrt how-to guide

[Ye Xiaolong]

Hi, Thomas

Thanks a lot for your detailed review.

On 02/16, Thomas Monjalon wrote:
>18/01/2020 06:48, Xiaolong Ye:
>> This doc describes how to enable DPDK on Openwrt in both virtual and
>> physical x86 environment.
>> 
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> ---
>> --- /dev/null
>> +++ b/doc/guides/howto/openwrt.rst
>> +Enable DPDK on openwrt
>> +======================
>> +
>> +This document describes how to enable Data Plane Development Kit (DPDK) on
>> +Openwrt in both a virtual and physical x86 environment.
>
>Please take care of the uppercases in OpenWrt.

Got it, will stick to OpenWrt for all occurrences.

>
>> +
>> +Introduction
>> +------------
>> +
>> +The OpenWrt project is a Linux operating system targeting embedded devices.
>> +Instead of trying to create a single, static firmware, OpenWrt provides a fully
>> +writable filesystem with package management. This frees the user from the
>> +application selection and configuration provided by the vendor and allows users
>> +to customize the device through the use of packages to suit any application. For
>> +developers OpenWrt is the framework to build an application without having to
>> +build a complete firmware around it. For users is offers full customization
>> +to use the device in ways never envisioned.
>
>Why doing marketing for OpenWrt in DPDK doc?
>It is a copy paste from https://openwrt.org/
>I think it is enough to say OpenWrt is a source-based router OS with package management.

Just want to gave some basic intro about OpenWrt, but it seems too much.
Will adopt your suggestion in next version.

>
>> +
>> +Pre-requisites
>> +~~~~~~~~~~~~~~
>> +
>> +You need gcc, binutils, bzip2, flex, python3.5+, perl, make, find, grep, diff,
>> +unzip, gawk, getopt, subversion, libz-dev and libc headers installed.
>
>These are prerequisites for building, so should be in section below.
>Maybe we can just provide a link to the official doc instead.
>

Make sense, will provide the link in next version.

>> +
>> +Build OpenWrt
>> +-------------
>> +
>> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases. To
>
>Please take the habit of splitting lines after a punctuation.
>

Will keep this in mind.

>> +fully customize your own OpenWrt, it is highly recommended to build it from
>> +the source code. You can clone the OpenWrt source code as follows:
>> +
>> +.. code-block:: console
>> +
>> +    git clone https://git.openwrt.org/openwrt/openwrt.git
>> +
>> +OpenWrt configuration
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +* Select ``x86`` in ``Target System``
>> +* Select ``x86_64`` in ``Subtarget``
>> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
>> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
>> +  then ``ToolChain Options`` and ``C Library implementation``
>> +
>> +Kernel configuration
>> +~~~~~~~~~~~~~~~~~~~~
>> +
>> +The following configurations should be enabled:
>> +
>> +* ``CONFIG_UIO=y``
>
>No, UIO is not recommended anymore.

Got it, will recommend vfio instead.

>
>> +* ``CONFIG_HUGETLBFS=y``
>> +* ``CONFIG_HUGETLB_PAGE=y``
>> +* ``CONFIG_PAGE_MONITOR=y``
>
>Why PAGE_MONITOR?

Good catch, should be CONFIG_PROC_PAGE_MONITOR.

>
>> +
>> +Build steps
>> +~~~~~~~~~~~
>> +
>> +For detailed OpenWrt build steps, please refer to the
>> +`OpenWrt build guide
>> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
>> +
>> +After the build is completed, you can find the images and sdk in
>> +``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
>
>s/sdk/SDK/

Got it.

>
>[...]
>> +    [binaries]
>> +    c = 'x86_64-openwrt-linux-gcc'
>> +    cpp = 'x86_64-openwrt-linux-cpp'
>> +    ar = 'x86_64-openwrt-linux-ar'
>> +    strip = 'x86_64-openwrt-linux-strip'
>> +
>> +    meson builddir --cross-file openwrt-cross
>
>This is really a good example why meson should allow to override the toolchain prefix.

Not sure whether I get your point or not, do you mean meson should support
something like "cross = 'x86_64-openwrt-linux-'" in config file (just like the
CROSS_COMPILE for make build system),  so we don't need to override the build
tools one by one?

>Please work with meson on this topic.

Sure, I guess time is not enough to do it in this release, will plan for
next one.

>
>[...]
>> +To cross compile with make:
>> +
>> +.. code-block:: console
>> +
>> +    export STAGING_DIR=<OpenWrt sdk>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir
>> +    export RTE_SDK=`pwd`
>> +    export RTE_KERNELDIR=<OpenWrt Root>/build_dir/target-x86_64_glibc/linux-x86_64/linux-4.19.81/
>> +    make config T=x86_64-native-linuxapp-gcc
>> +    make -j 100 CROSS=x86_64-openwrt-linux-gnu-
>
>The make build system should not be documented as it is going to be deprecated.

Got it.

>
>> +
>> +Running DPDK application on OpenWrt
>> +-----------------------------------
>> +
>> +Virtual machine
>> +~~~~~~~~~~~~~~~
>> +
>> +* Extract the boot image
>> +
>> +.. code-block:: console
>> +
>> +    gzip -d openwrt-x86-64-combined-ext4.img.gz
>> +
>> +* Launch Qemu
>> +
>> +.. code-block:: console
>> +
>> +    qemu-system-x86_64 \
>> +            -cpu host \
>> +            -smp 8 \
>> +            -enable-kvm \
>> +            -M q35 \
>> +            -m 2048M \
>> +            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
>> +            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
>> +            -device ide-hd,drive=d0,bus=ide.0 \
>> +            -net nic,vlan=0 \
>> +            -net nic,vlan=1 \
>> +            -net user,vlan=1 \
>> +            -display none \
>> +
>> +
>> +Physical machine
>> +~~~~~~~~~~~~~~~~
>> +
>> +If you are using a Windows PC, you can use an image writer application such as
>> +``Win32 Disk Imager`` and ``Etcher`` to write the OpenWrt image
>> +(openwrt-x86-64-combined-ext4.img) to a USB flash driver or SDcard.
>
>If you just built a DPDK for Linux, you are not running Windows,
>except if you are using a Linux server for compilation.
>Anyway, we should not try to document such procedure in my opinion.
>I'm sure you can refer to the OpenWrt documentation for flashing the image.

Ok, I'll stay focus on Linux for this doc.

>
>> +
>> +If you are using Linux, you can use the ``dd`` tool to write the OpenWrt image
>> +to the drive you want to write the image on.
>> +
>> +.. code-block:: console
>> +
>> +    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
>> +
>> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
>> +
>> +Running DPDK
>> +~~~~~~~~~~~~
>> +
>> +More detailed info about how to run a DPDK application please refer to
>> +``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
>> +
>> +.. note::
>> +
>> +    You need to install pre-built numa libraries (including soft link)
>
>s/numa/NUMA/

Got it.

>
>> +    to /usr/lib64 in OpenWrt.
>
>libnuma is not packaged in OpenWrt?
>

Unfortunately not, I guess the reason maybe OpenWrt is mainly focused on low-end
embedded systems which don't have numa support. 

>
>> --- a/doc/guides/rel_notes/release_20_02.rst
>> +++ b/doc/guides/rel_notes/release_20_02.rst
>> @@ -116,6 +116,10 @@ New Features
>>  
>>    Added support for ESP rte_flow patterns to the testpmd application.
>>  
>> +* **Added OpenWrt howto guide.**
>> +
>> +  Added document describes how to enable DPDK on OpenWrt in both virtual and
>> +  physical machine.
>>  
>>  Removed Items
>>  -------------
>
>You miss a doubled blank line before this title.

Will keep 2 blank lines before "Removed Items".

Thanks,
Xiaolong

Re: [dpdk-dev] [PATCH v5] doc: introduce openwrt how-to guide

[Thomas Monjalon]

16/02/2020 18:29, Ye Xiaolong:
> On 02/16, Thomas Monjalon wrote:
> >18/01/2020 06:48, Xiaolong Ye:
> >[...]
> >> +    [binaries]
> >> +    c = 'x86_64-openwrt-linux-gcc'
> >> +    cpp = 'x86_64-openwrt-linux-cpp'
> >> +    ar = 'x86_64-openwrt-linux-ar'
> >> +    strip = 'x86_64-openwrt-linux-strip'
> >> +
> >> +    meson builddir --cross-file openwrt-cross
> >
> >This is really a good example why meson should allow to override the toolchain prefix.
> 
> Not sure whether I get your point or not, do you mean meson should support
> something like "cross = 'x86_64-openwrt-linux-'" in config file (just like the
> CROSS_COMPILE for make build system),  so we don't need to override the build
> tools one by one?

Yes, I think meson should support CROSS_COMPILE environment variable
or a similar config option. I don't see the point of creating a new
config file for each toolchain prefix.


> >> +    to /usr/lib64 in OpenWrt.
> >
> >libnuma is not packaged in OpenWrt?
> 
> Unfortunately not, I guess the reason maybe OpenWrt is mainly focused on low-end
> embedded systems which don't have numa support. 

I would be nice to work on it in OpenWrt project then.

Re: [dpdk-dev] [PATCH v5] doc: introduce openwrt how-to guide

[Ye Xiaolong]

On 02/16, Thomas Monjalon wrote:
>16/02/2020 18:29, Ye Xiaolong:
>> On 02/16, Thomas Monjalon wrote:
>> >18/01/2020 06:48, Xiaolong Ye:
>> >[...]
>> >> +    [binaries]
>> >> +    c = 'x86_64-openwrt-linux-gcc'
>> >> +    cpp = 'x86_64-openwrt-linux-cpp'
>> >> +    ar = 'x86_64-openwrt-linux-ar'
>> >> +    strip = 'x86_64-openwrt-linux-strip'
>> >> +
>> >> +    meson builddir --cross-file openwrt-cross
>> >
>> >This is really a good example why meson should allow to override the toolchain prefix.
>> 
>> Not sure whether I get your point or not, do you mean meson should support
>> something like "cross = 'x86_64-openwrt-linux-'" in config file (just like the
>> CROSS_COMPILE for make build system),  so we don't need to override the build
>> tools one by one?
>
>Yes, I think meson should support CROSS_COMPILE environment variable
>or a similar config option. I don't see the point of creating a new
>config file for each toolchain prefix.

Totally agree, I would try to work on this to make it happen.

>
>
>> >> +    to /usr/lib64 in OpenWrt.
>> >
>> >libnuma is not packaged in OpenWrt?
>> 
>> Unfortunately not, I guess the reason maybe OpenWrt is mainly focused on low-end
>> embedded systems which don't have numa support. 
>
>I would be nice to work on it in OpenWrt project then.

Make sense, I would submit a request to OpenWrt project.

Thanks,
Xiaolong

>
>
>

[dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Xiaolong Ye]

This doc describes how to enable DPDK on Openwrt in both virtual and
physical x86 environment.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
Acked-by: John McNamara <john.mcnamara@intel.com>
---

V6 changes:

1. addressed review comments raised by Thomas

V5 changes:

1. improve the doc's grammar and wording according to John's
suggestions.

V4 changes:

1. add release notes

V3 changes:

1. emphasize target select in `OpenWrt configuration` section

V2 changes:

1. add meson build steps for dpdk
2. replace steps about build openwrt and running dpdk application with
links


 doc/guides/howto/index.rst             |   1 +
 doc/guides/howto/openwrt.rst           | 163 +++++++++++++++++++++++++
 doc/guides/rel_notes/release_20_02.rst |   5 +
 3 files changed, 169 insertions(+)
 create mode 100644 doc/guides/howto/openwrt.rst

diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
index a4c131652..5a97ea508 100644
--- a/doc/guides/howto/index.rst
+++ b/doc/guides/howto/index.rst
@@ -19,3 +19,4 @@ HowTo Guides
     packet_capture_framework
     telemetry
     debug_troubleshoot
+    openwrt
diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
new file mode 100644
index 000000000..6081f057b
--- /dev/null
+++ b/doc/guides/howto/openwrt.rst
@@ -0,0 +1,163 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2019 Intel Corporation.
+
+Enable DPDK on OpenWrt
+======================
+
+This document describes how to enable Data Plane Development Kit (DPDK) on
+OpenWrt in both a virtual and physical x86 environment.
+
+Introduction
+------------
+
+The OpenWrt project is a well-known source-based router OS which provides a
+fully writable filesystem with package management.
+
+Build OpenWrt
+-------------
+
+You can obtain OpenWrt image through https://downloads.openwrt.org/releases.
+To fully customize your own OpenWrt, it is highly recommended to build it from
+the source code. You can clone the OpenWrt source code as follows:
+
+.. code-block:: console
+
+    git clone https://git.openwrt.org/openwrt/openwrt.git
+
+OpenWrt configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+* Select ``x86`` in ``Target System``
+* Select ``x86_64`` in ``Subtarget``
+* Select ``Build the OpenWrt SDK`` for cross-compilation environment
+* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
+  then ``ToolChain Options`` and ``C Library implementation``
+
+Kernel configuration
+~~~~~~~~~~~~~~~~~~~~
+
+The following configurations should be enabled:
+
+* ``CONFIG_VFIO_IOMMU_TYPE1=y``
+* ``CONFIG_VFIO_VIRQFD=y``
+* ``CONFIG_VFIO=y``
+* ``CONFIG_VFIO_NOIOMMU=y``
+* ``CONFIG_VFIO_PCI=y``
+* ``CONFIG_VFIO_PCI_MMAP=y``
+* ``CONFIG_HUGETLBFS=y``
+* ``CONFIG_HUGETLB_PAGE=y``
+* ``CONFIG_PROC_PAGE_MONITOR=y``
+
+Build steps
+~~~~~~~~~~~
+
+For detailed OpenWrt build steps and prerequisites, please refer to the
+`OpenWrt build guide
+<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
+
+After the build is completed, you can find the images and SDK in
+``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
+
+
+DPDK Cross Compilation for OpenWrt
+----------------------------------
+
+Pre-requisites
+~~~~~~~~~~~~~~
+
+NUMA is required to run DPDK in x86.
+
+.. note::
+
+    For compiling the NUMA lib, run ``libtool --version`` to ensure the libtool
+    version >= 2.2, otherwise the compilation will fail with errors.
+
+.. code-block:: console
+
+    git clone https://github.com/numactl/numactl.git
+    cd numactl
+    git checkout v2.0.13 -b v2.0.13
+    ./autogen.sh
+    autoconf -i
+    export PATH=<OpenWrt SDK>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
+    ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
+    make install
+
+The numa header files and lib file is generated in the include and lib folder
+respectively under <OpenWrt SDK toolchain dir>.
+
+Build DPDK
+~~~~~~~~~~
+
+To cross compile with meson build, you need to write a customized cross file
+first.
+
+.. code-block:: console
+
+    [binaries]
+    c = 'x86_64-openwrt-linux-gcc'
+    cpp = 'x86_64-openwrt-linux-cpp'
+    ar = 'x86_64-openwrt-linux-ar'
+    strip = 'x86_64-openwrt-linux-strip'
+
+    meson builddir --cross-file openwrt-cross
+    ninja -C builddir
+
+.. note::
+
+    For compiling the igb_uio with the kernel version used in target machine,
+    you need to explicitly specify kernel_dir in meson_options.txt.
+
+Running DPDK application on OpenWrt
+-----------------------------------
+
+Virtual machine
+~~~~~~~~~~~~~~~
+
+* Extract the boot image
+
+.. code-block:: console
+
+    gzip -d openwrt-x86-64-combined-ext4.img.gz
+
+* Launch Qemu
+
+.. code-block:: console
+
+    qemu-system-x86_64 \
+            -cpu host \
+            -smp 8 \
+            -enable-kvm \
+            -M q35 \
+            -m 2048M \
+            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
+            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
+            -device ide-hd,drive=d0,bus=ide.0 \
+            -net nic,vlan=0 \
+            -net nic,vlan=1 \
+            -net user,vlan=1 \
+            -display none \
+
+
+Physical machine
+~~~~~~~~~~~~~~~~
+
+You can use the ``dd`` tool to write the OpenWrt image to the drive you
+want to write the image on.
+
+.. code-block:: console
+
+    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
+
+Where sdX is name of the drive. (You can find it though ``fdisk -l``)
+
+Running DPDK
+~~~~~~~~~~~~
+
+More detailed info about how to run a DPDK application please refer to
+``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
+
+.. note::
+
+    You need to install pre-built NUMA libraries (including soft link)
+    to /usr/lib64 in OpenWrt.
diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
index 78dab7cfc..25df894a9 100644
--- a/doc/guides/rel_notes/release_20_02.rst
+++ b/doc/guides/rel_notes/release_20_02.rst
@@ -229,6 +229,11 @@ New Features
   Enhanced the compression performance tool by adding a cycle-count mode
   which can be used to help measure and tune hardware and software PMDs.
 
+* **Added OpenWrt howto guide.**
+
+  Added document describes how to enable DPDK on OpenWrt in both virtual and
+  physical machine.
+
 
 Removed Items
 -------------
-- 
2.17.1

Re: [dpdk-dev] [PATCH v5] doc: introduce openwrt how-to guide

[Dmitry Kozlyuk]

> On 02/16, Thomas Monjalon wrote:
> >16/02/2020 18:29, Ye Xiaolong:  
> >> On 02/16, Thomas Monjalon wrote:  
> >> >18/01/2020 06:48, Xiaolong Ye:
> >> >[...]  
> >> >> +    [binaries]
> >> >> +    c = 'x86_64-openwrt-linux-gcc'
> >> >> +    cpp = 'x86_64-openwrt-linux-cpp'
> >> >> +    ar = 'x86_64-openwrt-linux-ar'
> >> >> +    strip = 'x86_64-openwrt-linux-strip'
> >> >> +
> >> >> +    meson builddir --cross-file openwrt-cross  
> >> >
> >> >This is really a good example why meson should allow to override the toolchain prefix.  
> >> 
> >> Not sure whether I get your point or not, do you mean meson should support
> >> something like "cross = 'x86_64-openwrt-linux-'" in config file (just like the
> >> CROSS_COMPILE for make build system),  so we don't need to override the build
> >> tools one by one?  
> >
> >Yes, I think meson should support CROSS_COMPILE environment variable
> >or a similar config option. I don't see the point of creating a new
> >config file for each toolchain prefix.  
> 
> Totally agree, I would try to work on this to make it happen.

You might be interested in this PR and its discussion:
https://github.com/mesonbuild/meson/pull/5859

-- 
Dmitry Kozlyuk

Re: [dpdk-dev] [PATCH v5] doc: introduce openwrt how-to guide

[Ye Xiaolong]

On 02/17, Dmitry Kozlyuk wrote:
>> On 02/16, Thomas Monjalon wrote:
>> >16/02/2020 18:29, Ye Xiaolong:  
>> >> On 02/16, Thomas Monjalon wrote:  
>> >> >18/01/2020 06:48, Xiaolong Ye:
>> >> >[...]  
>> >> >> +    [binaries]
>> >> >> +    c = 'x86_64-openwrt-linux-gcc'
>> >> >> +    cpp = 'x86_64-openwrt-linux-cpp'
>> >> >> +    ar = 'x86_64-openwrt-linux-ar'
>> >> >> +    strip = 'x86_64-openwrt-linux-strip'
>> >> >> +
>> >> >> +    meson builddir --cross-file openwrt-cross  
>> >> >
>> >> >This is really a good example why meson should allow to override the toolchain prefix.  
>> >> 
>> >> Not sure whether I get your point or not, do you mean meson should support
>> >> something like "cross = 'x86_64-openwrt-linux-'" in config file (just like the
>> >> CROSS_COMPILE for make build system),  so we don't need to override the build
>> >> tools one by one?  
>> >
>> >Yes, I think meson should support CROSS_COMPILE environment variable
>> >or a similar config option. I don't see the point of creating a new
>> >config file for each toolchain prefix.  
>> 
>> Totally agree, I would try to work on this to make it happen.
>
>You might be interested in this PR and its discussion:
>https://github.com/mesonbuild/meson/pull/5859

Thanks for the info, I'll look into it.


Thanks,
Xiaolong
>
>-- 
>Dmitry Kozlyuk

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Ray Kinsella]

On 17/02/2020 02:47, Xiaolong Ye wrote:
> This doc describes how to enable DPDK on Openwrt in both virtual and
> physical x86 environment.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> Acked-by: John McNamara <john.mcnamara@intel.com>
> ---
> 
> V6 changes:
> 
> 1. addressed review comments raised by Thomas
> 
> V5 changes:
> 
> 1. improve the doc's grammar and wording according to John's
> suggestions.
> 
> V4 changes:
> 
> 1. add release notes
> 
> V3 changes:
> 
> 1. emphasize target select in `OpenWrt configuration` section
> 
> V2 changes:
> 
> 1. add meson build steps for dpdk
> 2. replace steps about build openwrt and running dpdk application with
> links
> 
> 
>  doc/guides/howto/index.rst             |   1 +
>  doc/guides/howto/openwrt.rst           | 163 +++++++++++++++++++++++++
>  doc/guides/rel_notes/release_20_02.rst |   5 +
>  3 files changed, 169 insertions(+)
>  create mode 100644 doc/guides/howto/openwrt.rst
> 
> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
> index a4c131652..5a97ea508 100644
> --- a/doc/guides/howto/index.rst
> +++ b/doc/guides/howto/index.rst
> @@ -19,3 +19,4 @@ HowTo Guides
>      packet_capture_framework
>      telemetry
>      debug_troubleshoot
> +    openwrt
> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
> new file mode 100644
> index 000000000..6081f057b
> --- /dev/null
> +++ b/doc/guides/howto/openwrt.rst
> @@ -0,0 +1,163 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2019 Intel Corporation.
> +
> +Enable DPDK on OpenWrt
> +======================
> +
> +This document describes how to enable Data Plane Development Kit (DPDK) on
> +OpenWrt in both a virtual and physical x86 environment.
> +
> +Introduction
> +------------
> +
> +The OpenWrt project is a well-known source-based router OS which provides a
> +fully writable filesystem with package management.
> +
> +Build OpenWrt
> +-------------
> +
> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases.
> +To fully customize your own OpenWrt, it is highly recommended to build it from
> +the source code. You can clone the OpenWrt source code as follows:
> +
> +.. code-block:: console
> +
> +    git clone https://git.openwrt.org/openwrt/openwrt.git
> +
> +OpenWrt configuration
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +* Select ``x86`` in ``Target System``
> +* Select ``x86_64`` in ``Subtarget``
> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
> +  then ``ToolChain Options`` and ``C Library implementation``
> +
> +Kernel configuration
> +~~~~~~~~~~~~~~~~~~~~
> +
> +The following configurations should be enabled:
> +
> +* ``CONFIG_VFIO_IOMMU_TYPE1=y``
> +* ``CONFIG_VFIO_VIRQFD=y``
> +* ``CONFIG_VFIO=y``
> +* ``CONFIG_VFIO_NOIOMMU=y``
> +* ``CONFIG_VFIO_PCI=y``
> +* ``CONFIG_VFIO_PCI_MMAP=y``
> +* ``CONFIG_HUGETLBFS=y``
> +* ``CONFIG_HUGETLB_PAGE=y``
> +* ``CONFIG_PROC_PAGE_MONITOR=y``
> +

My 2c is that this is only ok as a short term solution.
However we really ought to be aiming for upstream support in OpenWRT, right?

For instance, why would enabling VFIO in OpenWRT necessitate a kernel rebuild.
We have had Intel IOMMU's for more than 10 years, why wouldn't this be enabled by default?
I imagine hugepages have been around a similar period.

> +Build steps
> +~~~~~~~~~~~
> +
> +For detailed OpenWrt build steps and prerequisites, please refer to the
> +`OpenWrt build guide
> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
> +
> +After the build is completed, you can find the images and SDK in
> +``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
> +
> +
> +DPDK Cross Compilation for OpenWrt
> +----------------------------------

> +Pre-requisites
> +~~~~~~~~~~~~~~
> +
> +NUMA is required to run DPDK in x86.
> +
> +.. note::
> +
> +    For compiling the NUMA lib, run ``libtool --version`` to ensure the libtool
> +    version >= 2.2, otherwise the compilation will fail with errors.
> +
> +.. code-block:: console
> +
> +    git clone https://github.com/numactl/numactl.git
> +    cd numactl
> +    git checkout v2.0.13 -b v2.0.13
> +    ./autogen.sh
> +    autoconf -i
> +    export PATH=<OpenWrt SDK>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
> +    ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
> +    make install
> +
> +The numa header files and lib file is generated in the include and lib folder
> +respectively under <OpenWrt SDK toolchain dir>.

Why not an upstream numactl package in OpenWRT,
that just installs the shared libraries and development headers instead of these gymnastics? 

> +Build DPDK
> +~~~~~~~~~~
> +
> +To cross compile with meson build, you need to write a customized cross file
> +first.
> +
> +.. code-block:: console
> +
> +    [binaries]
> +    c = 'x86_64-openwrt-linux-gcc'
> +    cpp = 'x86_64-openwrt-linux-cpp'
> +    ar = 'x86_64-openwrt-linux-ar'
> +    strip = 'x86_64-openwrt-linux-strip'
> +
> +    meson builddir --cross-file openwrt-cross
> +    ninja -C builddir
> +
> +.. note::
> +
> +    For compiling the igb_uio with the kernel version used in target machine,
> +    you need to explicitly specify kernel_dir in meson_options.txt.

As above, why not have an upstream DPDK package in OpenWRT,
that just installs the shared libraries and development headers instead of these gymnastics? 

> +Running DPDK application on OpenWrt
> +-----------------------------------
> +
> +Virtual machine
> +~~~~~~~~~~~~~~~
> +
> +* Extract the boot image
> +
> +.. code-block:: console
> +
> +    gzip -d openwrt-x86-64-combined-ext4.img.gz
> +
> +* Launch Qemu
> +
> +.. code-block:: console
> +
> +    qemu-system-x86_64 \
> +            -cpu host \
> +            -smp 8 \
> +            -enable-kvm \
> +            -M q35 \
> +            -m 2048M \
> +            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
> +            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
> +            -device ide-hd,drive=d0,bus=ide.0 \
> +            -net nic,vlan=0 \
> +            -net nic,vlan=1 \
> +            -net user,vlan=1 \
> +            -display none \
> +
> +
> +Physical machine
> +~~~~~~~~~~~~~~~~
> +
> +You can use the ``dd`` tool to write the OpenWrt image to the drive you
> +want to write the image on.
> +
> +.. code-block:: console
> +
> +    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
> +
> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
> +
> +Running DPDK
> +~~~~~~~~~~~~
> +
> +More detailed info about how to run a DPDK application please refer to
> +``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
> +
> +.. note::
> +
> +    You need to install pre-built NUMA libraries (including soft link)
> +    to /usr/lib64 in OpenWrt.
> diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
> index 78dab7cfc..25df894a9 100644
> --- a/doc/guides/rel_notes/release_20_02.rst
> +++ b/doc/guides/rel_notes/release_20_02.rst
> @@ -229,6 +229,11 @@ New Features
>    Enhanced the compression performance tool by adding a cycle-count mode
>    which can be used to help measure and tune hardware and software PMDs.
>  
> +* **Added OpenWrt howto guide.**
> +
> +  Added document describes how to enable DPDK on OpenWrt in both virtual and
> +  physical machine.
> +
>  
>  Removed Items
>  -------------
> 

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Ye Xiaolong]

Hi, Ray

On 02/17, Ray Kinsella wrote:
>On 17/02/2020 02:47, Xiaolong Ye wrote:
>> This doc describes how to enable DPDK on Openwrt in both virtual and
>> physical x86 environment.
>> 
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> Acked-by: John McNamara <john.mcnamara@intel.com>
>> ---
>> 
>> V6 changes:
>> 
>> 1. addressed review comments raised by Thomas
>> 
>> V5 changes:
>> 
>> 1. improve the doc's grammar and wording according to John's
>> suggestions.
>> 
>> V4 changes:
>> 
>> 1. add release notes
>> 
>> V3 changes:
>> 
>> 1. emphasize target select in `OpenWrt configuration` section
>> 
>> V2 changes:
>> 
>> 1. add meson build steps for dpdk
>> 2. replace steps about build openwrt and running dpdk application with
>> links
>> 
>> 
>>  doc/guides/howto/index.rst             |   1 +
>>  doc/guides/howto/openwrt.rst           | 163 +++++++++++++++++++++++++
>>  doc/guides/rel_notes/release_20_02.rst |   5 +
>>  3 files changed, 169 insertions(+)
>>  create mode 100644 doc/guides/howto/openwrt.rst
>> 
>> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
>> index a4c131652..5a97ea508 100644
>> --- a/doc/guides/howto/index.rst
>> +++ b/doc/guides/howto/index.rst
>> @@ -19,3 +19,4 @@ HowTo Guides
>>      packet_capture_framework
>>      telemetry
>>      debug_troubleshoot
>> +    openwrt
>> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
>> new file mode 100644
>> index 000000000..6081f057b
>> --- /dev/null
>> +++ b/doc/guides/howto/openwrt.rst
>> @@ -0,0 +1,163 @@
>> +..  SPDX-License-Identifier: BSD-3-Clause
>> +    Copyright(c) 2019 Intel Corporation.
>> +
>> +Enable DPDK on OpenWrt
>> +======================
>> +
>> +This document describes how to enable Data Plane Development Kit (DPDK) on
>> +OpenWrt in both a virtual and physical x86 environment.
>> +
>> +Introduction
>> +------------
>> +
>> +The OpenWrt project is a well-known source-based router OS which provides a
>> +fully writable filesystem with package management.
>> +
>> +Build OpenWrt
>> +-------------
>> +
>> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases.
>> +To fully customize your own OpenWrt, it is highly recommended to build it from
>> +the source code. You can clone the OpenWrt source code as follows:
>> +
>> +.. code-block:: console
>> +
>> +    git clone https://git.openwrt.org/openwrt/openwrt.git
>> +
>> +OpenWrt configuration
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +* Select ``x86`` in ``Target System``
>> +* Select ``x86_64`` in ``Subtarget``
>> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
>> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
>> +  then ``ToolChain Options`` and ``C Library implementation``
>> +
>> +Kernel configuration
>> +~~~~~~~~~~~~~~~~~~~~
>> +
>> +The following configurations should be enabled:
>> +
>> +* ``CONFIG_VFIO_IOMMU_TYPE1=y``
>> +* ``CONFIG_VFIO_VIRQFD=y``
>> +* ``CONFIG_VFIO=y``
>> +* ``CONFIG_VFIO_NOIOMMU=y``
>> +* ``CONFIG_VFIO_PCI=y``
>> +* ``CONFIG_VFIO_PCI_MMAP=y``
>> +* ``CONFIG_HUGETLBFS=y``
>> +* ``CONFIG_HUGETLB_PAGE=y``
>> +* ``CONFIG_PROC_PAGE_MONITOR=y``
>> +
>
>My 2c is that this is only ok as a short term solution.
>However we really ought to be aiming for upstream support in OpenWRT, right?

Thanks for the input, I agree that the perfect way would be getting everything
(kernel support, numa lib) ready in OpenWrt, and we can just install DPDK as
shared libraries in OpenWrt. This could be our next step.

while currently, this is the only way of enabling DPDK on OpenWrt by what
we have in hands, we can update the guide once Openwrt side support is ready.

Thanks,
Xiaolong

>
>For instance, why would enabling VFIO in OpenWRT necessitate a kernel rebuild.
>We have had Intel IOMMU's for more than 10 years, why wouldn't this be enabled by default?
>I imagine hugepages have been around a similar period.
>
>> +Build steps
>> +~~~~~~~~~~~
>> +
>> +For detailed OpenWrt build steps and prerequisites, please refer to the
>> +`OpenWrt build guide
>> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
>> +
>> +After the build is completed, you can find the images and SDK in
>> +``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
>> +
>> +
>> +DPDK Cross Compilation for OpenWrt
>> +----------------------------------
>
>> +Pre-requisites
>> +~~~~~~~~~~~~~~
>> +
>> +NUMA is required to run DPDK in x86.
>> +
>> +.. note::
>> +
>> +    For compiling the NUMA lib, run ``libtool --version`` to ensure the libtool
>> +    version >= 2.2, otherwise the compilation will fail with errors.
>> +
>> +.. code-block:: console
>> +
>> +    git clone https://github.com/numactl/numactl.git
>> +    cd numactl
>> +    git checkout v2.0.13 -b v2.0.13
>> +    ./autogen.sh
>> +    autoconf -i
>> +    export PATH=<OpenWrt SDK>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
>> +    ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
>> +    make install
>> +
>> +The numa header files and lib file is generated in the include and lib folder
>> +respectively under <OpenWrt SDK toolchain dir>.
>
>Why not an upstream numactl package in OpenWRT,
>that just installs the shared libraries and development headers instead of these gymnastics? 
>
>> +Build DPDK
>> +~~~~~~~~~~
>> +
>> +To cross compile with meson build, you need to write a customized cross file
>> +first.
>> +
>> +.. code-block:: console
>> +
>> +    [binaries]
>> +    c = 'x86_64-openwrt-linux-gcc'
>> +    cpp = 'x86_64-openwrt-linux-cpp'
>> +    ar = 'x86_64-openwrt-linux-ar'
>> +    strip = 'x86_64-openwrt-linux-strip'
>> +
>> +    meson builddir --cross-file openwrt-cross
>> +    ninja -C builddir
>> +
>> +.. note::
>> +
>> +    For compiling the igb_uio with the kernel version used in target machine,
>> +    you need to explicitly specify kernel_dir in meson_options.txt.
>
>As above, why not have an upstream DPDK package in OpenWRT,
>that just installs the shared libraries and development headers instead of these gymnastics? 
>
>> +Running DPDK application on OpenWrt
>> +-----------------------------------
>> +
>> +Virtual machine
>> +~~~~~~~~~~~~~~~
>> +
>> +* Extract the boot image
>> +
>> +.. code-block:: console
>> +
>> +    gzip -d openwrt-x86-64-combined-ext4.img.gz
>> +
>> +* Launch Qemu
>> +
>> +.. code-block:: console
>> +
>> +    qemu-system-x86_64 \
>> +            -cpu host \
>> +            -smp 8 \
>> +            -enable-kvm \
>> +            -M q35 \
>> +            -m 2048M \
>> +            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
>> +            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
>> +            -device ide-hd,drive=d0,bus=ide.0 \
>> +            -net nic,vlan=0 \
>> +            -net nic,vlan=1 \
>> +            -net user,vlan=1 \
>> +            -display none \
>> +
>> +
>> +Physical machine
>> +~~~~~~~~~~~~~~~~
>> +
>> +You can use the ``dd`` tool to write the OpenWrt image to the drive you
>> +want to write the image on.
>> +
>> +.. code-block:: console
>> +
>> +    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
>> +
>> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
>> +
>> +Running DPDK
>> +~~~~~~~~~~~~
>> +
>> +More detailed info about how to run a DPDK application please refer to
>> +``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
>> +
>> +.. note::
>> +
>> +    You need to install pre-built NUMA libraries (including soft link)
>> +    to /usr/lib64 in OpenWrt.
>> diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
>> index 78dab7cfc..25df894a9 100644
>> --- a/doc/guides/rel_notes/release_20_02.rst
>> +++ b/doc/guides/rel_notes/release_20_02.rst
>> @@ -229,6 +229,11 @@ New Features
>>    Enhanced the compression performance tool by adding a cycle-count mode
>>    which can be used to help measure and tune hardware and software PMDs.
>>  
>> +* **Added OpenWrt howto guide.**
>> +
>> +  Added document describes how to enable DPDK on OpenWrt in both virtual and
>> +  physical machine.
>> +
>>  
>>  Removed Items
>>  -------------
>> 
>

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Ray Kinsella]

On 17/02/2020 15:44, Ye Xiaolong wrote:
> Hi, Ray
> 
> On 02/17, Ray Kinsella wrote:
>> On 17/02/2020 02:47, Xiaolong Ye wrote:
>>> This doc describes how to enable DPDK on Openwrt in both virtual and
>>> physical x86 environment.
>>>
>>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>>> Acked-by: John McNamara <john.mcnamara@intel.com>
>>> ---
>>>
>>> V6 changes:
>>>
>>> 1. addressed review comments raised by Thomas
>>>
>>> V5 changes:
>>>
>>> 1. improve the doc's grammar and wording according to John's
>>> suggestions.
>>>
>>> V4 changes:
>>>
>>> 1. add release notes
>>>
>>> V3 changes:
>>>
>>> 1. emphasize target select in `OpenWrt configuration` section
>>>
>>> V2 changes:
>>>
>>> 1. add meson build steps for dpdk
>>> 2. replace steps about build openwrt and running dpdk application with
>>> links
>>>
>>>
>>>  doc/guides/howto/index.rst             |   1 +
>>>  doc/guides/howto/openwrt.rst           | 163 +++++++++++++++++++++++++
>>>  doc/guides/rel_notes/release_20_02.rst |   5 +
>>>  3 files changed, 169 insertions(+)
>>>  create mode 100644 doc/guides/howto/openwrt.rst
>>>
>>> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
>>> index a4c131652..5a97ea508 100644
>>> --- a/doc/guides/howto/index.rst
>>> +++ b/doc/guides/howto/index.rst
>>> @@ -19,3 +19,4 @@ HowTo Guides
>>>      packet_capture_framework
>>>      telemetry
>>>      debug_troubleshoot
>>> +    openwrt
>>> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
>>> new file mode 100644
>>> index 000000000..6081f057b
>>> --- /dev/null
>>> +++ b/doc/guides/howto/openwrt.rst
>>> @@ -0,0 +1,163 @@
>>> +..  SPDX-License-Identifier: BSD-3-Clause
>>> +    Copyright(c) 2019 Intel Corporation.
>>> +
>>> +Enable DPDK on OpenWrt
>>> +======================
>>> +
>>> +This document describes how to enable Data Plane Development Kit (DPDK) on
>>> +OpenWrt in both a virtual and physical x86 environment.
>>> +
>>> +Introduction
>>> +------------
>>> +
>>> +The OpenWrt project is a well-known source-based router OS which provides a
>>> +fully writable filesystem with package management.
>>> +
>>> +Build OpenWrt
>>> +-------------
>>> +
>>> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases.
>>> +To fully customize your own OpenWrt, it is highly recommended to build it from
>>> +the source code. You can clone the OpenWrt source code as follows:
>>> +
>>> +.. code-block:: console
>>> +
>>> +    git clone https://git.openwrt.org/openwrt/openwrt.git
>>> +
>>> +OpenWrt configuration
>>> +~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +* Select ``x86`` in ``Target System``
>>> +* Select ``x86_64`` in ``Subtarget``
>>> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
>>> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
>>> +  then ``ToolChain Options`` and ``C Library implementation``
>>> +
>>> +Kernel configuration
>>> +~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +The following configurations should be enabled:
>>> +
>>> +* ``CONFIG_VFIO_IOMMU_TYPE1=y``
>>> +* ``CONFIG_VFIO_VIRQFD=y``
>>> +* ``CONFIG_VFIO=y``
>>> +* ``CONFIG_VFIO_NOIOMMU=y``
>>> +* ``CONFIG_VFIO_PCI=y``
>>> +* ``CONFIG_VFIO_PCI_MMAP=y``
>>> +* ``CONFIG_HUGETLBFS=y``
>>> +* ``CONFIG_HUGETLB_PAGE=y``
>>> +* ``CONFIG_PROC_PAGE_MONITOR=y``
>>> +
>>
>> My 2c is that this is only ok as a short term solution.
>> However we really ought to be aiming for upstream support in OpenWRT, right?
> 
> Thanks for the input, I agree that the perfect way would be getting everything
> (kernel support, numa lib) ready in OpenWrt, and we can just install DPDK as
> shared libraries in OpenWrt. This could be our next step.
> 
> while currently, this is the only way of enabling DPDK on OpenWrt by what
> we have in hands, we can update the guide once Openwrt side support is ready.
> 
> Thanks,
> Xiaolong

Thanks Xiaolong,

So to be clear.
I see that documentation like this is brittle and has a limited lifespan.
I can almost guarantee the next version of DPDK or OpenWRT will break it.
(and how would we know?)

Much better to do this in the right place - OpenWRT. 

> 
>>
>> For instance, why would enabling VFIO in OpenWRT necessitate a kernel rebuild.
>> We have had Intel IOMMU's for more than 10 years, why wouldn't this be enabled by default?
>> I imagine hugepages have been around a similar period.
>>
>>> +Build steps
>>> +~~~~~~~~~~~
>>> +
>>> +For detailed OpenWrt build steps and prerequisites, please refer to the
>>> +`OpenWrt build guide
>>> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
>>> +
>>> +After the build is completed, you can find the images and SDK in
>>> +``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
>>> +
>>> +
>>> +DPDK Cross Compilation for OpenWrt
>>> +----------------------------------
>>
>>> +Pre-requisites
>>> +~~~~~~~~~~~~~~
>>> +
>>> +NUMA is required to run DPDK in x86.
>>> +
>>> +.. note::
>>> +
>>> +    For compiling the NUMA lib, run ``libtool --version`` to ensure the libtool
>>> +    version >= 2.2, otherwise the compilation will fail with errors.
>>> +
>>> +.. code-block:: console
>>> +
>>> +    git clone https://github.com/numactl/numactl.git
>>> +    cd numactl
>>> +    git checkout v2.0.13 -b v2.0.13
>>> +    ./autogen.sh
>>> +    autoconf -i
>>> +    export PATH=<OpenWrt SDK>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
>>> +    ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
>>> +    make install
>>> +
>>> +The numa header files and lib file is generated in the include and lib folder
>>> +respectively under <OpenWrt SDK toolchain dir>.
>>
>> Why not an upstream numactl package in OpenWRT,
>> that just installs the shared libraries and development headers instead of these gymnastics? 
>>
>>> +Build DPDK
>>> +~~~~~~~~~~
>>> +
>>> +To cross compile with meson build, you need to write a customized cross file
>>> +first.
>>> +
>>> +.. code-block:: console
>>> +
>>> +    [binaries]
>>> +    c = 'x86_64-openwrt-linux-gcc'
>>> +    cpp = 'x86_64-openwrt-linux-cpp'
>>> +    ar = 'x86_64-openwrt-linux-ar'
>>> +    strip = 'x86_64-openwrt-linux-strip'
>>> +
>>> +    meson builddir --cross-file openwrt-cross
>>> +    ninja -C builddir
>>> +
>>> +.. note::
>>> +
>>> +    For compiling the igb_uio with the kernel version used in target machine,
>>> +    you need to explicitly specify kernel_dir in meson_options.txt.
>>
>> As above, why not have an upstream DPDK package in OpenWRT,
>> that just installs the shared libraries and development headers instead of these gymnastics? 
>>
>>> +Running DPDK application on OpenWrt
>>> +-----------------------------------
>>> +
>>> +Virtual machine
>>> +~~~~~~~~~~~~~~~
>>> +
>>> +* Extract the boot image
>>> +
>>> +.. code-block:: console
>>> +
>>> +    gzip -d openwrt-x86-64-combined-ext4.img.gz
>>> +
>>> +* Launch Qemu
>>> +
>>> +.. code-block:: console
>>> +
>>> +    qemu-system-x86_64 \
>>> +            -cpu host \
>>> +            -smp 8 \
>>> +            -enable-kvm \
>>> +            -M q35 \
>>> +            -m 2048M \
>>> +            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
>>> +            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
>>> +            -device ide-hd,drive=d0,bus=ide.0 \
>>> +            -net nic,vlan=0 \
>>> +            -net nic,vlan=1 \
>>> +            -net user,vlan=1 \
>>> +            -display none \
>>> +
>>> +
>>> +Physical machine
>>> +~~~~~~~~~~~~~~~~
>>> +
>>> +You can use the ``dd`` tool to write the OpenWrt image to the drive you
>>> +want to write the image on.
>>> +
>>> +.. code-block:: console
>>> +
>>> +    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
>>> +
>>> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
>>> +
>>> +Running DPDK
>>> +~~~~~~~~~~~~
>>> +
>>> +More detailed info about how to run a DPDK application please refer to
>>> +``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
>>> +
>>> +.. note::
>>> +
>>> +    You need to install pre-built NUMA libraries (including soft link)
>>> +    to /usr/lib64 in OpenWrt.
>>> diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
>>> index 78dab7cfc..25df894a9 100644
>>> --- a/doc/guides/rel_notes/release_20_02.rst
>>> +++ b/doc/guides/rel_notes/release_20_02.rst
>>> @@ -229,6 +229,11 @@ New Features
>>>    Enhanced the compression performance tool by adding a cycle-count mode
>>>    which can be used to help measure and tune hardware and software PMDs.
>>>  
>>> +* **Added OpenWrt howto guide.**
>>> +
>>> +  Added document describes how to enable DPDK on OpenWrt in both virtual and
>>> +  physical machine.
>>> +
>>>  
>>>  Removed Items
>>>  -------------
>>>
>>

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Mcnamara, John]

> -----Original Message-----
> From: Ray Kinsella <mdr@ashroe.eu>
> Sent: Monday, February 17, 2020 3:49 PM
> To: Ye, Xiaolong <xiaolong.ye@intel.com>
> Cc: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
> <marko.kovacevic@intel.com>; dev@dpdk.org; Richardson, Bruce
> <bruce.richardson@intel.com>; Stephen Hemminger
> <stephen@networkplumber.org>; Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide
> 
> 
> ..
> 
> So to be clear.
> I see that documentation like this is brittle and has a limited lifespan.
> I can almost guarantee the next version of DPDK or OpenWRT will break it.
> (and how would we know?)
> 
> Much better to do this in the right place - OpenWRT.

Hi Ray,

These comments are a little late, we have been through 6 revisions.

I suggest putting the doc in as it is, since it useful, and revise it to make it more update proof in the next release.

John

Acked-by: John McNamara <john.mcnamara@intel.com>

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Ray Kinsella]

On 18/02/2020 08:26, Mcnamara, John wrote:
> 
> 
>> -----Original Message-----
>> From: Ray Kinsella <mdr@ashroe.eu>
>> Sent: Monday, February 17, 2020 3:49 PM
>> To: Ye, Xiaolong <xiaolong.ye@intel.com>
>> Cc: Mcnamara, John <john.mcnamara@intel.com>; Kovacevic, Marko
>> <marko.kovacevic@intel.com>; dev@dpdk.org; Richardson, Bruce
>> <bruce.richardson@intel.com>; Stephen Hemminger
>> <stephen@networkplumber.org>; Thomas Monjalon <thomas@monjalon.net>
>> Subject: Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide
>>
>>
>> ..
>>
>> So to be clear.
>> I see that documentation like this is brittle and has a limited lifespan.
>> I can almost guarantee the next version of DPDK or OpenWRT will break it.
>> (and how would we know?)
>>
>> Much better to do this in the right place - OpenWRT.
> 
> Hi Ray,
> 
> These comments are a little late, we have been through 6 revisions.

Understood.

> I suggest putting the doc in as it is, since it useful, and revise it to make it more update proof in the next release.

Agreed - feedback was late. 
I am registering my protest, that this cannot be called 'done' - further work is required to make this sustainable.

> 
> John
> 
> Acked-by: John McNamara <john.mcnamara@intel.com>
> 

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Thomas Monjalon]

18/02/2020 10:12, Ray Kinsella:
> On 18/02/2020 08:26, Mcnamara, John wrote:
> > From: Ray Kinsella <mdr@ashroe.eu>
> >> So to be clear.
> >> I see that documentation like this is brittle and has a limited lifespan.
> >> I can almost guarantee the next version of DPDK or OpenWRT will break it.
> >> (and how would we know?)
> >>
> >> Much better to do this in the right place - OpenWRT.
> > 
> > Hi Ray,
> > 
> > These comments are a little late, we have been through 6 revisions.
> 
> Understood.
> 
> > I suggest putting the doc in as it is, since it useful, and revise it to make it more update proof in the next release.
> 
> Agreed - feedback was late. 
> I am registering my protest, that this cannot be called 'done' - further work is required to make this sustainable.

I agree the most important is to work on DPDK integration.
Would be nice to have DPDK integrated as a package in OpenWrt
and in Buildroot as well.
And please make sure meson allows to choose any toolchain easily.

Thanks

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Ye Xiaolong]

On 02/18, Thomas Monjalon wrote:
>18/02/2020 10:12, Ray Kinsella:
>> On 18/02/2020 08:26, Mcnamara, John wrote:
>> > From: Ray Kinsella <mdr@ashroe.eu>
>> >> So to be clear.
>> >> I see that documentation like this is brittle and has a limited lifespan.
>> >> I can almost guarantee the next version of DPDK or OpenWRT will break it.
>> >> (and how would we know?)
>> >>
>> >> Much better to do this in the right place - OpenWRT.
>> > 
>> > Hi Ray,
>> > 
>> > These comments are a little late, we have been through 6 revisions.
>> 
>> Understood.
>> 
>> > I suggest putting the doc in as it is, since it useful, and revise it to make it more update proof in the next release.
>> 
>> Agreed - feedback was late. 
>> I am registering my protest, that this cannot be called 'done' - further work is required to make this sustainable.
>
>I agree the most important is to work on DPDK integration.
>Would be nice to have DPDK integrated as a package in OpenWrt
>and in Buildroot as well.

Yes, agree this would be a better and clean solution.

>And please make sure meson allows to choose any toolchain easily.

It's in my todo list now. :)

Thanks,
Xiaolong
>
>Thanks
>
>

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Ray Kinsella]

On 18/02/2020 09:49, Ye Xiaolong wrote:
> On 02/18, Thomas Monjalon wrote:
>> 18/02/2020 10:12, Ray Kinsella:
>>> On 18/02/2020 08:26, Mcnamara, John wrote:
>>>> From: Ray Kinsella <mdr@ashroe.eu>
>>>>> So to be clear.
>>>>> I see that documentation like this is brittle and has a limited lifespan.
>>>>> I can almost guarantee the next version of DPDK or OpenWRT will break it.
>>>>> (and how would we know?)
>>>>>
>>>>> Much better to do this in the right place - OpenWRT.
>>>>
>>>> Hi Ray,
>>>>
>>>> These comments are a little late, we have been through 6 revisions.
>>>
>>> Understood.
>>>
>>>> I suggest putting the doc in as it is, since it useful, and revise it to make it more update proof in the next release.
>>>
>>> Agreed - feedback was late. 
>>> I am registering my protest, that this cannot be called 'done' - further work is required to make this sustainable.
>>
>> I agree the most important is to work on DPDK integration.
>> Would be nice to have DPDK integrated as a package in OpenWrt
>> and in Buildroot as well.
> 
> Yes, agree this would be a better and clean solution.
> 
>> And please make sure meson allows to choose any toolchain easily.
> 
> It's in my todo list now. :)
> 
> Thanks,
> Xiaolong
>>
>> Thanks
>>
>>

perfect, thank you.

Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Thomas Monjalon]

17/02/2020 03:47, Xiaolong Ye:
> This doc describes how to enable DPDK on Openwrt in both virtual and
> physical x86 environment.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> Acked-by: John McNamara <john.mcnamara@intel.com>

Applied, thanks

I hope we'll be able to drop this how-to soon,
in favor of a DPDK package well integrated in OpenWrt :-)