Re: [dpdk-dev] [PATCH v3 2/2] ci: Introduce travis builds for github repositories

[Aaron Conole <aconole@redhat.com>]

Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> writes:

> Hi Michael/Aaron,
> 	Thanks for adding the Arm build. I have few comments inline.
>
>> -----Original Message-----
>> From: Michael Santana <msantana@redhat.com>
>> Sent: Friday, February 1, 2019 10:48 AM
>> To: dev@dpdk.org
>> Cc: Aaron Conole <aconole@redhat.com>; Bruce Richardson
>> <bruce.richardson@intel.com>; Honnappa Nagarahalli
>> <Honnappa.Nagarahalli@arm.com>; thomas@monjalon.net
>> Subject: [PATCH v3 2/2] ci: Introduce travis builds for github repositories
>> 
>> GitHub is a service used by developers to store repositories.  GitHub provides
>> service integrations that allow 3rd party services to access developer
>> repositories and perform actions.  One of these services is Travis-CI, a simple
>> continuous integration platform.
>> 
>> This is a simple initial implementation of a travis build for the DPDK project.  It
>> doesn't require any changes from individual developers to enable, but will
>> allow those developers who opt-in to GitHub and the travis service to get
>> automatic builds for every push they make.
>> 
>> Additionally, the travis service will send an email to the test-report list
>> informing anyone interested in the automated build (including a result).
>> 
>> Signed-off-by: Aaron Conole <aconole@redhat.com>
>> Signed-off-by: Michael Santana <msantana@redhat.com>
>> ---
>> v3:
>>   - Renamed ambiguous variable names and comments, including the variable
>>     KERNEL to DISABLE_KERNEL_MODULES and comment 'source for python' to
>>     'Repo for python'
>>   - Removed duplicate file meson_cross_aarch64_gcc.txt. Used
>>     arm64_armv8_linuxapp_gcc file instead
>> 
>>  .ci/linux-build.sh                  |  88 +++++++++++++++
>>  .ci/linux-setup.sh                  |  31 ++++++
>>  .travis.yml                         | 159 ++++++++++++++++++++++++++++
>>  MAINTAINERS                         |   7 ++
>>  doc/guides/contributing/patches.rst |   4 +
>>  5 files changed, 289 insertions(+)
>>  create mode 100755 .ci/linux-build.sh
>>  create mode 100755 .ci/linux-setup.sh
>>  create mode 100644 .travis.yml
>> 
>> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh new file mode 100755 index
>> 000000000..7aa90822c
>> --- /dev/null
>> +++ b/.ci/linux-build.sh
>> @@ -0,0 +1,88 @@
>> +#!/bin/bash
>> +
>> +# check for whether we're clang or gcc
>> +# setup the right options depending on the environment variables # run
>> +the build
>> +
>> +# Just used for the 'classic' configuration system (ie: make)
>> +set_conf() {
>> +    echo "[BUILT WITH $2 SET TO $3]"
>> +    c="$1/.config"
>> +    shift
>> +
>> +    if grep -q "$1" "$c"; then
>> +        sed -i "s:^$1=.*$:$1=$2:g" $c
>> +    else
>> +        echo $1=$2 >> "$c"
>> +    fi
>> +}
>> +
>> +BUILD_ARCH="x86_64-native-linuxapp-"
>> +
>> +if [ "${ARM64}" == "1" ]; then
> Would be good to change the variable names from ARM64 to 'AARCH64...'

Since we need to spin a change anyway, we'll change it.

>> +    # convert the arch specifier
>> +    BUILD_ARCH="arm64-armv8a-linuxapp-"
>> +    ARM64_TOOL="linaro-arm-tool"
>> +    export PATH=$PATH:$(pwd)/${ARM64_TOOL}/bin
>> +fi
>> +
>> +
>> +if [ "${NINJABUILD}" == "1" ]; then
>> +    OPTS=""
>> +
>> +    DEF_LIB="static"
>> +    if [ "${SHARED}" == "1" ]; then
>> +        DEF_LIB="shared"
>> +    fi
>> +
>> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
>> +        OPTS="-Denable_kmods=false"
>> +    fi
>> +
>> +    if [ "${ARM64}" == "1" ]; then
>> +        OPTS="${OPTS} --cross-file config/arm/arm64_armv8_linuxapp_gcc"
>> +    fi
>> +
>> +    OPTS="$OPTS --default-library=$DEF_LIB"
>> +    meson build --werror -Dexamples=all ${OPTS}
>> +    ninja -C build
>> +else
>> +    EXTRA_OPTS=""
>> +
>> +    make config T="${BUILD_ARCH}${CC}"
>> +
>> +    set_conf build CONFIG_RTE_KNI_KMOD n
>> +    set_conf build CONFIG_RTE_EAL_IGB_UIO n
>> +
>> +    if dpkg --list | grep -q zlib1g ; then
>> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB y
>> +    fi
>> +
>> +    if dpkg --list | grep -q libpcap-dev ; then
>> +        set_conf build CONFIG_RTE_PORT_PCAP y
>> +    fi
>> +
>> +    if [ "${SHARED}" == "1" ]; then
>> +        set_conf build CONFIG_RTE_BUILD_SHARED_LIB y
>> +    fi
>> +
>> +    if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then
>> +        echo Unsupported kernel builds at the moment
>> +    fi
>> +
>> +    if [ "${ARM64}" == "1" ]; then
>> +        EXTRA_OPTS="CROSS=aarch64-linux-gnu-"
>> +
>> +        # need to turn off these extras
>> +        set_conf build CONFIG_RTE_PORT_PCAP n
>> +        set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB n
>> +
>> +        # convert the CC/CXX variables
>> +        export CC=aarch64-linux-gnu-${CC}
>> +        export CXX=aarch64-linux-gnu-${CXX}
>> +        export AR=aarch64-linux-gnu-ar
>> +        export STRIP=aarch64-linux-gnu-strip
>> +    fi
>> +
>> +    make all ${EXTRA_OPTS}
>> +fi
>> diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh new file mode 100755 index
>> 000000000..7d6478ef9
>> --- /dev/null
>> +++ b/.ci/linux-setup.sh
>> @@ -0,0 +1,31 @@
>> +#!/bin/bash
>> +
>> +python3.5 -m pip install --upgrade meson --user
>> +
>> +echo "ARM64 is [ ${ARM64} ]"
>> +
>> +if [ "${ARM64}" == "1" ]; then
>> +    # need to build & install libnuma
>> +    # This will only be minimal support for now.
>> +
>> ARM64_TOOL_URL='https://releases.linaro.org/components/toolchain/binari
>> es/latest-7/aarch64-linux-gnu/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-
>> linux-gnu.tar.xz'
>> +    ARM64_TOOL="linaro-arm-tool"
>> +    NUMA_GIT_URL="https://github.com/numactl/numactl.git"
>> +
>> +    wget -O "${ARM64_TOOL}.tar.xz" "${ARM64_TOOL_URL}"
>> +    tar -xf "${ARM64_TOOL}.tar.xz"
>> +    mv gcc-linaro* "${ARM64_TOOL}"
>> +    export PATH=$PATH:$(pwd)/${ARM64_TOOL}/bin
>> +    git clone "${NUMA_GIT_URL}"
>> +    cd numactl
>> +    git checkout v2.0.11
>> +    ./autogen.sh
>> +    autoconf -i
>> +    mkdir numa_bin
>> +    ./configure --host=aarch64-linux-gnu CC=aarch64-linux-gnu-gcc \
>> +                --prefix=$(pwd)/numa_bin
>> +    make install # install numa
>> +    cd ..
>> +    cp numactl/numa_bin/include/numa*.h "${ARM64_TOOL}/aarch64-linux-
>> gnu/libc/usr/include/"
>> +    cp numactl/numa_bin/lib/libnuma.* "${ARM64_TOOL}/aarch64-linux-
>> gnu/lib64/"
>> +    cp numactl/numa_bin/lib/libnuma.* "${ARM64_TOOL}/lib/"
>> +fi
>> diff --git a/.travis.yml b/.travis.yml
>> new file mode 100644
>> index 000000000..afd63aa33
>> --- /dev/null
>> +++ b/.travis.yml
>> @@ -0,0 +1,159 @@
>> +language: c
>> +compiler:
>> +  - gcc
>> +  - clang
>> +
>> +os:
>> +  - linux
>> +
>> +addons:
>> +  apt:
>> +    sources:
>> +      - deadsnakes  #Repo for python 3.5
>> +      - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +    packages:
>> +      - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +
>> +before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
>> +
>> +sudo: false
>> +
>> +env:
>> +  - SHARED=1
>> +  - DISABLE_KERNEL_MODULES=1
>> +  - SHARED=1 DISABLE_KERNEL_MODULES=1
>> +  - NINJABUILD=1
>> +  - NINJABUILD=1 SHARED=1
>> +  - NINJABUILD=1 DISABLE_KERNEL_MODULES=1
>> +  - NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
> My understanding is we need to list 'ARM64=1' and 'ARM64=1 NINJABUILD=1' here.

We don't.  We have a build explicitly in the matrix section below to
cover it.

>> +
>> +matrix:
>> +  include:
>> +  - env: SHARED=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: SHARED=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: DISABLE_KERNEL_MODULES=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: clang
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: ARM64=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), libtool,
>> +python3.5, python3-pip]
>> +  - env: ARM64=1 NINJABUILD=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [linux-headers-$(uname -r), libtool, python3.5,
>> +python3-pip, ninja-build]
>> +  - env: NINJABUILD=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: NINJABUILD=1 SHARED=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: NINJABUILD=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +  - env: NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1
>> +    compiler: gcc
>> +    addons:
>> +      apt:
>> +        sources:
>> +          - deadsnakes  #Repo for python 3.5
>> +          - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2'
>> +        packages:
>> +          - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4]
>> +          - [libnuma-dev, linux-headers-$(uname -r), python3.5,
>> +python3-pip, ninja-build]
>> +
>> +
>> +script: ./.ci/${TRAVIS_OS_NAME}-build.sh
>> +
>> +notifications:
>> +  email:
>> +    recipients:
>> +      - test-report@dpdk.org
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 835d8a201..eed6f69d3 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -119,6 +119,13 @@ F: config/rte_config.h
>>  F: buildtools/gen-pmdinfo-cfile.sh
>>  F: buildtools/symlink-drivers-solibs.sh
>> 
>> +Public CI
>> +M: Aaron Conole <aconole@redhat.com>
>> +M: Michael Santana <msantana@redhat.com>
>> +F: .travis.yml
>> +F: .ci/
>> +F: meson_cross_aarch64_gcc.txt
> Not required

Oops.  Okay we will fix it.

>> +
>>  ABI versioning
>>  M: Neil Horman <nhorman@tuxdriver.com>
>>  F: lib/librte_compat/
>> diff --git a/doc/guides/contributing/patches.rst
>> b/doc/guides/contributing/patches.rst
>> index a64bb0368..49e930cbb 100644
>> --- a/doc/guides/contributing/patches.rst
>> +++ b/doc/guides/contributing/patches.rst
>> @@ -32,6 +32,10 @@ The mailing list for DPDK development is
>> `dev@dpdk.org <http://mails.dpdk.org/ar  Contributors will need to `register
>> for the mailing list <http://mails.dpdk.org/listinfo/dev>`_ in order to submit
>> patches.
>>  It is also worth registering for the DPDK `Patchwork
>> <http://patches.dpdk.org/project/dpdk/list/>`_
>> 
>> +If you are using the GitHub service, you can link your repository to
>> +the ``travis-ci.org`` build service.  When you push patches to your
>> +GitHub repository, the travis service will automatically build your changes.
>> +
>>  The development process requires some familiarity with the ``git`` version
>> control system.
>>  Refer to the `Pro Git Book <http://www.git-scm.com/book/>`_ for further
>> information.
>> 
>> --
>> 2.20.1