Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Ye Xiaolong <xiaolong.ye@intel.com>]

Hi, Ray

On 02/17, Ray Kinsella wrote:
>On 17/02/2020 02:47, Xiaolong Ye wrote:
>> This doc describes how to enable DPDK on Openwrt in both virtual and
>> physical x86 environment.
>> 
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> Acked-by: John McNamara <john.mcnamara@intel.com>
>> ---
>> 
>> V6 changes:
>> 
>> 1. addressed review comments raised by Thomas
>> 
>> V5 changes:
>> 
>> 1. improve the doc's grammar and wording according to John's
>> suggestions.
>> 
>> V4 changes:
>> 
>> 1. add release notes
>> 
>> V3 changes:
>> 
>> 1. emphasize target select in `OpenWrt configuration` section
>> 
>> V2 changes:
>> 
>> 1. add meson build steps for dpdk
>> 2. replace steps about build openwrt and running dpdk application with
>> links
>> 
>> 
>>  doc/guides/howto/index.rst             |   1 +
>>  doc/guides/howto/openwrt.rst           | 163 +++++++++++++++++++++++++
>>  doc/guides/rel_notes/release_20_02.rst |   5 +
>>  3 files changed, 169 insertions(+)
>>  create mode 100644 doc/guides/howto/openwrt.rst
>> 
>> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
>> index a4c131652..5a97ea508 100644
>> --- a/doc/guides/howto/index.rst
>> +++ b/doc/guides/howto/index.rst
>> @@ -19,3 +19,4 @@ HowTo Guides
>>      packet_capture_framework
>>      telemetry
>>      debug_troubleshoot
>> +    openwrt
>> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
>> new file mode 100644
>> index 000000000..6081f057b
>> --- /dev/null
>> +++ b/doc/guides/howto/openwrt.rst
>> @@ -0,0 +1,163 @@
>> +..  SPDX-License-Identifier: BSD-3-Clause
>> +    Copyright(c) 2019 Intel Corporation.
>> +
>> +Enable DPDK on OpenWrt
>> +======================
>> +
>> +This document describes how to enable Data Plane Development Kit (DPDK) on
>> +OpenWrt in both a virtual and physical x86 environment.
>> +
>> +Introduction
>> +------------
>> +
>> +The OpenWrt project is a well-known source-based router OS which provides a
>> +fully writable filesystem with package management.
>> +
>> +Build OpenWrt
>> +-------------
>> +
>> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases.
>> +To fully customize your own OpenWrt, it is highly recommended to build it from
>> +the source code. You can clone the OpenWrt source code as follows:
>> +
>> +.. code-block:: console
>> +
>> +    git clone https://git.openwrt.org/openwrt/openwrt.git
>> +
>> +OpenWrt configuration
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +* Select ``x86`` in ``Target System``
>> +* Select ``x86_64`` in ``Subtarget``
>> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
>> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
>> +  then ``ToolChain Options`` and ``C Library implementation``
>> +
>> +Kernel configuration
>> +~~~~~~~~~~~~~~~~~~~~
>> +
>> +The following configurations should be enabled:
>> +
>> +* ``CONFIG_VFIO_IOMMU_TYPE1=y``
>> +* ``CONFIG_VFIO_VIRQFD=y``
>> +* ``CONFIG_VFIO=y``
>> +* ``CONFIG_VFIO_NOIOMMU=y``
>> +* ``CONFIG_VFIO_PCI=y``
>> +* ``CONFIG_VFIO_PCI_MMAP=y``
>> +* ``CONFIG_HUGETLBFS=y``
>> +* ``CONFIG_HUGETLB_PAGE=y``
>> +* ``CONFIG_PROC_PAGE_MONITOR=y``
>> +
>
>My 2c is that this is only ok as a short term solution.
>However we really ought to be aiming for upstream support in OpenWRT, right?

Thanks for the input, I agree that the perfect way would be getting everything
(kernel support, numa lib) ready in OpenWrt, and we can just install DPDK as
shared libraries in OpenWrt. This could be our next step.

while currently, this is the only way of enabling DPDK on OpenWrt by what
we have in hands, we can update the guide once Openwrt side support is ready.

Thanks,
Xiaolong

>
>For instance, why would enabling VFIO in OpenWRT necessitate a kernel rebuild.
>We have had Intel IOMMU's for more than 10 years, why wouldn't this be enabled by default?
>I imagine hugepages have been around a similar period.
>
>> +Build steps
>> +~~~~~~~~~~~
>> +
>> +For detailed OpenWrt build steps and prerequisites, please refer to the
>> +`OpenWrt build guide
>> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
>> +
>> +After the build is completed, you can find the images and SDK in
>> +``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
>> +
>> +
>> +DPDK Cross Compilation for OpenWrt
>> +----------------------------------
>
>> +Pre-requisites
>> +~~~~~~~~~~~~~~
>> +
>> +NUMA is required to run DPDK in x86.
>> +
>> +.. note::
>> +
>> +    For compiling the NUMA lib, run ``libtool --version`` to ensure the libtool
>> +    version >= 2.2, otherwise the compilation will fail with errors.
>> +
>> +.. code-block:: console
>> +
>> +    git clone https://github.com/numactl/numactl.git
>> +    cd numactl
>> +    git checkout v2.0.13 -b v2.0.13
>> +    ./autogen.sh
>> +    autoconf -i
>> +    export PATH=<OpenWrt SDK>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
>> +    ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
>> +    make install
>> +
>> +The numa header files and lib file is generated in the include and lib folder
>> +respectively under <OpenWrt SDK toolchain dir>.
>
>Why not an upstream numactl package in OpenWRT,
>that just installs the shared libraries and development headers instead of these gymnastics? 
>
>> +Build DPDK
>> +~~~~~~~~~~
>> +
>> +To cross compile with meson build, you need to write a customized cross file
>> +first.
>> +
>> +.. code-block:: console
>> +
>> +    [binaries]
>> +    c = 'x86_64-openwrt-linux-gcc'
>> +    cpp = 'x86_64-openwrt-linux-cpp'
>> +    ar = 'x86_64-openwrt-linux-ar'
>> +    strip = 'x86_64-openwrt-linux-strip'
>> +
>> +    meson builddir --cross-file openwrt-cross
>> +    ninja -C builddir
>> +
>> +.. note::
>> +
>> +    For compiling the igb_uio with the kernel version used in target machine,
>> +    you need to explicitly specify kernel_dir in meson_options.txt.
>
>As above, why not have an upstream DPDK package in OpenWRT,
>that just installs the shared libraries and development headers instead of these gymnastics? 
>
>> +Running DPDK application on OpenWrt
>> +-----------------------------------
>> +
>> +Virtual machine
>> +~~~~~~~~~~~~~~~
>> +
>> +* Extract the boot image
>> +
>> +.. code-block:: console
>> +
>> +    gzip -d openwrt-x86-64-combined-ext4.img.gz
>> +
>> +* Launch Qemu
>> +
>> +.. code-block:: console
>> +
>> +    qemu-system-x86_64 \
>> +            -cpu host \
>> +            -smp 8 \
>> +            -enable-kvm \
>> +            -M q35 \
>> +            -m 2048M \
>> +            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
>> +            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
>> +            -device ide-hd,drive=d0,bus=ide.0 \
>> +            -net nic,vlan=0 \
>> +            -net nic,vlan=1 \
>> +            -net user,vlan=1 \
>> +            -display none \
>> +
>> +
>> +Physical machine
>> +~~~~~~~~~~~~~~~~
>> +
>> +You can use the ``dd`` tool to write the OpenWrt image to the drive you
>> +want to write the image on.
>> +
>> +.. code-block:: console
>> +
>> +    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
>> +
>> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
>> +
>> +Running DPDK
>> +~~~~~~~~~~~~
>> +
>> +More detailed info about how to run a DPDK application please refer to
>> +``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
>> +
>> +.. note::
>> +
>> +    You need to install pre-built NUMA libraries (including soft link)
>> +    to /usr/lib64 in OpenWrt.
>> diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
>> index 78dab7cfc..25df894a9 100644
>> --- a/doc/guides/rel_notes/release_20_02.rst
>> +++ b/doc/guides/rel_notes/release_20_02.rst
>> @@ -229,6 +229,11 @@ New Features
>>    Enhanced the compression performance tool by adding a cycle-count mode
>>    which can be used to help measure and tune hardware and software PMDs.
>>  
>> +* **Added OpenWrt howto guide.**
>> +
>> +  Added document describes how to enable DPDK on OpenWrt in both virtual and
>> +  physical machine.
>> +
>>  
>>  Removed Items
>>  -------------
>> 
>