From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id D2B8E5B20 for ; Wed, 27 Feb 2019 15:35:24 +0100 (CET) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id AB7677BDCE; Wed, 27 Feb 2019 14:35:23 +0000 (UTC) Received: from dhcp-25.97.bos.redhat.com (unknown [10.18.25.61]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A94F96136E; Wed, 27 Feb 2019 14:35:22 +0000 (UTC) From: Aaron Conole To: Thomas Monjalon Cc: Michael Santana , dev@dpdk.org, Bruce Richardson , Honnappa Nagarahalli References: <20190206221308.22349-1-msantana@redhat.com> <20190207220114.8020-1-msantana@redhat.com> <20190207220114.8020-3-msantana@redhat.com> <2766343.Bn91GvS4A1@xps> Date: Wed, 27 Feb 2019 09:35:21 -0500 In-Reply-To: <2766343.Bn91GvS4A1@xps> (Thomas Monjalon's message of "Wed, 27 Feb 2019 14:56:28 +0100") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Wed, 27 Feb 2019 14:35:24 +0000 (UTC) Subject: Re: [dpdk-dev] [PATCH v5 2/2] ci: Introduce travis builds for github repositories X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Feb 2019 14:35:25 -0000 Thomas Monjalon writes: > Hi, > > 07/02/2019 23:01, Michael Santana: >> GitHub is a service used by developers to store repositories. GitHub >> provides service integrations that allow 3rd party services to access >> developer repositories and perform actions. One of these services is >> Travis-CI, a simple continuous integration platform. >> >> This is a simple initial implementation of a travis build for the DPDK >> project. It doesn't require any changes from individual developers to >> enable, but will allow those developers who opt-in to GitHub and the >> travis service to get automatic builds for every push they make. >> >> Additionally, the travis service will send an email to the test-report >> list informing anyone interested in the automated build (including a >> result). >> >> Signed-off-by: Aaron Conole >> Signed-off-by: Michael Santana >> Acked-by: Bruce Richardson >> Reviewed-by: Honnappa Nagarahalli > > I think it is a good idea to provide some recipes for some well known CI. > > I have some concerns about the organization of what is generic/specific > and about the automation of reports. > Detailed comments below: > >> --- /dev/null >> +++ b/.ci/linux-build.sh >> +# Just used for the 'classic' configuration system (ie: make) > > I am not sure about supporting the legacy system in a new CI. For now, documentation all says that the legacy system is the supported system. I think it's appropriate to continue to support it until such time as it is eliminated. Otherwise, when an end user builds we don't know what to say about support - IE: if they have problems with the classic system for whatever reason, do we tell them "sorry, we cannot help"? The patches.rst mentions that all patches must pass with the Makefile system, and the contributing/documentation.rst calls it the "standard DPDK build system." If you want to change those things to reflect something different please do, and we can drop all of the stuff related to it, but until that time we won't. >> +set_conf() { >> + echo "[BUILT WITH $2 SET TO $3]" > > In order to improve readability, I suggest to document the parameters > on the first line. Example: > > set_conf() { # > >> + c="$1/.config" >> + shift >> + >> + if grep -q "$1" "$c"; then >> + sed -i "s:^$1=.*$:$1=$2:g" $c > > You don't need the "g" as there is only one option per line. > >> + else >> + echo $1=$2 >> "$c" > > The else branch is useless because .config includes all options. Okay. >> + fi >> +} >> + >> +BUILD_ARCH="x86_64-native-linuxapp-" > > We could have some native Arm compilation. Sure. Is this just commentary? Do you suggest a change here? This is a default, and will be adjusted later by other parameters. >> + >> +if [ "${AARCH64}" == "1" ]; then >> + # convert the arch specifier >> + BUILD_ARCH="arm64-armv8a-linuxapp-" >> + AARCH64_TOOL="linaro-arm-tool" > > What is this directory? It looks really specific to Travis. It's specific to the AARCH64 toolchain that was pulled in as part of linux-prepare.sh - do you think something should change? >> + export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin >> +fi >> + >> + >> +if [ "${NINJABUILD}" == "1" ]; then >> + OPTS="" >> + >> + DEF_LIB="static" >> + if [ "${SHARED}" == "1" ]; then >> + DEF_LIB="shared" >> + fi >> + >> + if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then >> + OPTS="-Denable_kmods=false" >> + fi > > Isn't it possible to directly provide the meson options in travis.yml > instead of doing a translation with new option names? Yes and no. It would be possible, but we try to support both build systems and they have different options afaict. So we need something common. Maybe we missed something? >> + >> + if [ "${AARCH64}" == "1" ]; then >> + OPTS="${OPTS} --cross-file config/arm/arm64_armv8_linuxapp_gcc" >> + fi >> + >> + OPTS="$OPTS --default-library=$DEF_LIB" >> + meson build --werror -Dexamples=all ${OPTS} >> + ninja -C build >> +else >> + EXTRA_OPTS="" >> + >> + make config T="${BUILD_ARCH}${CC}" >> + >> + set_conf build CONFIG_RTE_KNI_KMOD n >> + set_conf build CONFIG_RTE_EAL_IGB_UIO n > > Why these options are fixed? There was a problem with the Travis system when trying to build some of the kernel modules, but I don't remember the details. Maybe Michael does. >> + >> + if dpkg --list | grep -q zlib1g ; then >> + set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB y >> + fi > > This is Debian-specific. Will address this in v6. >> + >> + if dpkg --list | grep -q libpcap-dev ; then >> + set_conf build CONFIG_RTE_PORT_PCAP y >> + fi >> + >> + if [ "${SHARED}" == "1" ]; then >> + set_conf build CONFIG_RTE_BUILD_SHARED_LIB y >> + fi >> + >> + if [ "${DISABLE_KERNEL_MODULES}" == "1" ]; then >> + echo Unsupported kernel builds at the moment >> + fi >> + >> + if [ "${AARCH64}" == "1" ]; then >> + EXTRA_OPTS="CROSS=aarch64-linux-gnu-" >> + >> + # need to turn off these extras >> + set_conf build CONFIG_RTE_PORT_PCAP n >> + set_conf build CONFIG_RTE_LIBRTE_PMD_ZLIB n >> + >> + # convert the CC/CXX variables >> + export CC=aarch64-linux-gnu-${CC} >> + export CXX=aarch64-linux-gnu-${CXX} >> + export AR=aarch64-linux-gnu-ar >> + export STRIP=aarch64-linux-gnu-strip >> + fi >> + >> + make all ${EXTRA_OPTS} >> +fi >> diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh >> new file mode 100755 >> index 000000000..8dfd5aa49 >> --- /dev/null >> +++ b/.ci/linux-setup.sh >> @@ -0,0 +1,31 @@ >> +#!/bin/bash >> + >> +python3.5 -m pip install --upgrade meson --user > > Which distributions have python3.5? > It looks very specific. I agree, could probably just be python3 we need to check and see if we can just use that. But, we did need the upgrade. Travis environment comes up with ubuntu 14.04, which includes python3.4, and the requisite version of meson needs python3.5 or higher. >> + >> +echo "AARCH64 is [ ${AARCH64} ]" >> + >> +if [ "${AARCH64}" == "1" ]; then >> + # need to build & install libnuma > > Why is it needed? linbnuma is optional. > I think this file can be dropped or renamed to > something like install-libnuma-for-cross-arm.sh We needed this because it broke the meson build when cross compiling, IIRC. I will investigate it further to be sure. >> + # This will only be minimal support for now. >> + >> AARCH64_TOOL_URL='https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/gcc-linaro-7.4.1-2019.02-x86_64_aarch64-linux-gnu.tar.xz' >> + AARCH64_TOOL="linaro-arm-tool" >> + NUMA_GIT_URL="https://github.com/numactl/numactl.git" >> + >> + wget -O "${AARCH64_TOOL}.tar.xz" "${AARCH64_TOOL_URL}" >> + tar -xf "${AARCH64_TOOL}.tar.xz" >> + mv gcc-linaro* "${AARCH64_TOOL}" >> + export PATH=$PATH:$(pwd)/${AARCH64_TOOL}/bin >> + git clone "${NUMA_GIT_URL}" >> + cd numactl >> + git checkout v2.0.11 >> + ./autogen.sh >> + autoconf -i >> + mkdir numa_bin >> + ./configure --host=aarch64-linux-gnu CC=aarch64-linux-gnu-gcc \ >> + --prefix=$(pwd)/numa_bin >> + make install # install numa >> + cd .. >> + cp numactl/numa_bin/include/numa*.h >> "${AARCH64_TOOL}/aarch64-linux-gnu/libc/usr/include/" >> + cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/aarch64-linux-gnu/lib64/" >> + cp numactl/numa_bin/lib/libnuma.* "${AARCH64_TOOL}/lib/" >> +fi >> diff --git a/.travis.yml b/.travis.yml >> new file mode 100644 >> index 000000000..0faa35770 >> --- /dev/null >> +++ b/.travis.yml >> @@ -0,0 +1,159 @@ >> +language: c >> +compiler: >> + - gcc >> + - clang >> + >> +os: >> + - linux >> + >> +addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, ninja-build] >> + >> +before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh >> + >> +sudo: false >> + >> +env: >> + - SHARED=1 >> + - DISABLE_KERNEL_MODULES=1 >> + - SHARED=1 DISABLE_KERNEL_MODULES=1 >> + - NINJABUILD=1 >> + - NINJABUILD=1 SHARED=1 >> + - NINJABUILD=1 DISABLE_KERNEL_MODULES=1 >> + - NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1 >> + >> +matrix: >> + include: >> + - env: SHARED=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: DISABLE_KERNEL_MODULES=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: SHARED=1 DISABLE_KERNEL_MODULES=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: SHARED=1 >> + compiler: clang >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: DISABLE_KERNEL_MODULES=1 >> + compiler: clang >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: SHARED=1 DISABLE_KERNEL_MODULES=1 >> + compiler: clang >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: AARCH64=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), libtool, python3.5, python3-pip] >> + - env: AARCH64=1 NINJABUILD=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [linux-headers-$(uname -r), libtool, python3.5, python3-pip, ninja-build] >> + - env: NINJABUILD=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: NINJABUILD=1 SHARED=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: NINJABUILD=1 DISABLE_KERNEL_MODULES=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + - env: NINJABUILD=1 SHARED=1 DISABLE_KERNEL_MODULES=1 >> + compiler: gcc >> + addons: >> + apt: >> + sources: >> + - deadsnakes #Repo for python 3.5 >> + - sourceline: 'ppa:mstipicevic/ninja-build-1-7-2' >> + packages: >> + - [libbsd-dev, libpcap-dev, libcrypto++-dev, libjansson4] >> + - [libnuma-dev, linux-headers-$(uname -r), python3.5, python3-pip, >> ninja-build] >> + >> + >> +script: ./.ci/${TRAVIS_OS_NAME}-build.sh >> + >> +notifications: >> + email: >> + recipients: >> + - test-report@dpdk.org > > The idea of this mailing list is to receive reports about > the upstream development. When doing a private development, > reports should not be sent. How can it be disabled? There are a few variables related to controlling this that we can investigate so that we only alert on the main repository, if you want. We could also just remove it. Not a problem either way.