Re: [dpdk-dev] [PATCH v6] doc: introduce openwrt how-to guide

[Ray Kinsella <mdr@ashroe.eu>]

On 17/02/2020 15:44, Ye Xiaolong wrote:
> Hi, Ray
> 
> On 02/17, Ray Kinsella wrote:
>> On 17/02/2020 02:47, Xiaolong Ye wrote:
>>> This doc describes how to enable DPDK on Openwrt in both virtual and
>>> physical x86 environment.
>>>
>>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>>> Acked-by: John McNamara <john.mcnamara@intel.com>
>>> ---
>>>
>>> V6 changes:
>>>
>>> 1. addressed review comments raised by Thomas
>>>
>>> V5 changes:
>>>
>>> 1. improve the doc's grammar and wording according to John's
>>> suggestions.
>>>
>>> V4 changes:
>>>
>>> 1. add release notes
>>>
>>> V3 changes:
>>>
>>> 1. emphasize target select in `OpenWrt configuration` section
>>>
>>> V2 changes:
>>>
>>> 1. add meson build steps for dpdk
>>> 2. replace steps about build openwrt and running dpdk application with
>>> links
>>>
>>>
>>>  doc/guides/howto/index.rst             |   1 +
>>>  doc/guides/howto/openwrt.rst           | 163 +++++++++++++++++++++++++
>>>  doc/guides/rel_notes/release_20_02.rst |   5 +
>>>  3 files changed, 169 insertions(+)
>>>  create mode 100644 doc/guides/howto/openwrt.rst
>>>
>>> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst
>>> index a4c131652..5a97ea508 100644
>>> --- a/doc/guides/howto/index.rst
>>> +++ b/doc/guides/howto/index.rst
>>> @@ -19,3 +19,4 @@ HowTo Guides
>>>      packet_capture_framework
>>>      telemetry
>>>      debug_troubleshoot
>>> +    openwrt
>>> diff --git a/doc/guides/howto/openwrt.rst b/doc/guides/howto/openwrt.rst
>>> new file mode 100644
>>> index 000000000..6081f057b
>>> --- /dev/null
>>> +++ b/doc/guides/howto/openwrt.rst
>>> @@ -0,0 +1,163 @@
>>> +..  SPDX-License-Identifier: BSD-3-Clause
>>> +    Copyright(c) 2019 Intel Corporation.
>>> +
>>> +Enable DPDK on OpenWrt
>>> +======================
>>> +
>>> +This document describes how to enable Data Plane Development Kit (DPDK) on
>>> +OpenWrt in both a virtual and physical x86 environment.
>>> +
>>> +Introduction
>>> +------------
>>> +
>>> +The OpenWrt project is a well-known source-based router OS which provides a
>>> +fully writable filesystem with package management.
>>> +
>>> +Build OpenWrt
>>> +-------------
>>> +
>>> +You can obtain OpenWrt image through https://downloads.openwrt.org/releases.
>>> +To fully customize your own OpenWrt, it is highly recommended to build it from
>>> +the source code. You can clone the OpenWrt source code as follows:
>>> +
>>> +.. code-block:: console
>>> +
>>> +    git clone https://git.openwrt.org/openwrt/openwrt.git
>>> +
>>> +OpenWrt configuration
>>> +~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +* Select ``x86`` in ``Target System``
>>> +* Select ``x86_64`` in ``Subtarget``
>>> +* Select ``Build the OpenWrt SDK`` for cross-compilation environment
>>> +* Select ``Use glibc`` in ``Advanced configuration options (for developers)``
>>> +  then ``ToolChain Options`` and ``C Library implementation``
>>> +
>>> +Kernel configuration
>>> +~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +The following configurations should be enabled:
>>> +
>>> +* ``CONFIG_VFIO_IOMMU_TYPE1=y``
>>> +* ``CONFIG_VFIO_VIRQFD=y``
>>> +* ``CONFIG_VFIO=y``
>>> +* ``CONFIG_VFIO_NOIOMMU=y``
>>> +* ``CONFIG_VFIO_PCI=y``
>>> +* ``CONFIG_VFIO_PCI_MMAP=y``
>>> +* ``CONFIG_HUGETLBFS=y``
>>> +* ``CONFIG_HUGETLB_PAGE=y``
>>> +* ``CONFIG_PROC_PAGE_MONITOR=y``
>>> +
>>
>> My 2c is that this is only ok as a short term solution.
>> However we really ought to be aiming for upstream support in OpenWRT, right?
> 
> Thanks for the input, I agree that the perfect way would be getting everything
> (kernel support, numa lib) ready in OpenWrt, and we can just install DPDK as
> shared libraries in OpenWrt. This could be our next step.
> 
> while currently, this is the only way of enabling DPDK on OpenWrt by what
> we have in hands, we can update the guide once Openwrt side support is ready.
> 
> Thanks,
> Xiaolong

Thanks Xiaolong,

So to be clear.
I see that documentation like this is brittle and has a limited lifespan.
I can almost guarantee the next version of DPDK or OpenWRT will break it.
(and how would we know?)

Much better to do this in the right place - OpenWRT. 

> 
>>
>> For instance, why would enabling VFIO in OpenWRT necessitate a kernel rebuild.
>> We have had Intel IOMMU's for more than 10 years, why wouldn't this be enabled by default?
>> I imagine hugepages have been around a similar period.
>>
>>> +Build steps
>>> +~~~~~~~~~~~
>>> +
>>> +For detailed OpenWrt build steps and prerequisites, please refer to the
>>> +`OpenWrt build guide
>>> +<https://openwrt.org/docs/guide-developer/build-system/use-buildsystem>`_.
>>> +
>>> +After the build is completed, you can find the images and SDK in
>>> +``<OpenWrt Root>/bin/targets/x86/64-glibc/``.
>>> +
>>> +
>>> +DPDK Cross Compilation for OpenWrt
>>> +----------------------------------
>>
>>> +Pre-requisites
>>> +~~~~~~~~~~~~~~
>>> +
>>> +NUMA is required to run DPDK in x86.
>>> +
>>> +.. note::
>>> +
>>> +    For compiling the NUMA lib, run ``libtool --version`` to ensure the libtool
>>> +    version >= 2.2, otherwise the compilation will fail with errors.
>>> +
>>> +.. code-block:: console
>>> +
>>> +    git clone https://github.com/numactl/numactl.git
>>> +    cd numactl
>>> +    git checkout v2.0.13 -b v2.0.13
>>> +    ./autogen.sh
>>> +    autoconf -i
>>> +    export PATH=<OpenWrt SDK>/glibc/openwrt-sdk-x86-64_gcc-8.3.0_glibc.Linux-x86_64/staging_dir/toolchain-x86_64_gcc-8.3.0_glibc/bin/:$PATH
>>> +    ./configure CC=x86_64-openwrt-linux-gnu-gcc --prefix=<OpenWrt SDK toolchain dir>
>>> +    make install
>>> +
>>> +The numa header files and lib file is generated in the include and lib folder
>>> +respectively under <OpenWrt SDK toolchain dir>.
>>
>> Why not an upstream numactl package in OpenWRT,
>> that just installs the shared libraries and development headers instead of these gymnastics? 
>>
>>> +Build DPDK
>>> +~~~~~~~~~~
>>> +
>>> +To cross compile with meson build, you need to write a customized cross file
>>> +first.
>>> +
>>> +.. code-block:: console
>>> +
>>> +    [binaries]
>>> +    c = 'x86_64-openwrt-linux-gcc'
>>> +    cpp = 'x86_64-openwrt-linux-cpp'
>>> +    ar = 'x86_64-openwrt-linux-ar'
>>> +    strip = 'x86_64-openwrt-linux-strip'
>>> +
>>> +    meson builddir --cross-file openwrt-cross
>>> +    ninja -C builddir
>>> +
>>> +.. note::
>>> +
>>> +    For compiling the igb_uio with the kernel version used in target machine,
>>> +    you need to explicitly specify kernel_dir in meson_options.txt.
>>
>> As above, why not have an upstream DPDK package in OpenWRT,
>> that just installs the shared libraries and development headers instead of these gymnastics? 
>>
>>> +Running DPDK application on OpenWrt
>>> +-----------------------------------
>>> +
>>> +Virtual machine
>>> +~~~~~~~~~~~~~~~
>>> +
>>> +* Extract the boot image
>>> +
>>> +.. code-block:: console
>>> +
>>> +    gzip -d openwrt-x86-64-combined-ext4.img.gz
>>> +
>>> +* Launch Qemu
>>> +
>>> +.. code-block:: console
>>> +
>>> +    qemu-system-x86_64 \
>>> +            -cpu host \
>>> +            -smp 8 \
>>> +            -enable-kvm \
>>> +            -M q35 \
>>> +            -m 2048M \
>>> +            -object memory-backend-file,id=mem,size=2048M,mem-path=/tmp/hugepages,share=on \
>>> +            -drive file=<Your OpenWrt images folder>/openwrt-x86-64-combined-ext4.img,id=d0,if=none,bus=0,unit=0 \
>>> +            -device ide-hd,drive=d0,bus=ide.0 \
>>> +            -net nic,vlan=0 \
>>> +            -net nic,vlan=1 \
>>> +            -net user,vlan=1 \
>>> +            -display none \
>>> +
>>> +
>>> +Physical machine
>>> +~~~~~~~~~~~~~~~~
>>> +
>>> +You can use the ``dd`` tool to write the OpenWrt image to the drive you
>>> +want to write the image on.
>>> +
>>> +.. code-block:: console
>>> +
>>> +    dd if=openwrt-18.06.1-x86-64-combined-squashfs.img of=/dev/sdX
>>> +
>>> +Where sdX is name of the drive. (You can find it though ``fdisk -l``)
>>> +
>>> +Running DPDK
>>> +~~~~~~~~~~~~
>>> +
>>> +More detailed info about how to run a DPDK application please refer to
>>> +``Running DPDK Applications`` section of :ref:`the DPDK documentation <linux_gsg>`.
>>> +
>>> +.. note::
>>> +
>>> +    You need to install pre-built NUMA libraries (including soft link)
>>> +    to /usr/lib64 in OpenWrt.
>>> diff --git a/doc/guides/rel_notes/release_20_02.rst b/doc/guides/rel_notes/release_20_02.rst
>>> index 78dab7cfc..25df894a9 100644
>>> --- a/doc/guides/rel_notes/release_20_02.rst
>>> +++ b/doc/guides/rel_notes/release_20_02.rst
>>> @@ -229,6 +229,11 @@ New Features
>>>    Enhanced the compression performance tool by adding a cycle-count mode
>>>    which can be used to help measure and tune hardware and software PMDs.
>>>  
>>> +* **Added OpenWrt howto guide.**
>>> +
>>> +  Added document describes how to enable DPDK on OpenWrt in both virtual and
>>> +  physical machine.
>>> +
>>>  
>>>  Removed Items
>>>  -------------
>>>
>>