This series tries to make the travis configuration a bit easier to read, then drops jobs that are not essential or functional duplicates in the hope we can give some oxygen to the robot who seems to lag quite a bit and can't catch up. -- David Marchand David Marchand (4): ci: remove unnecessary dependency on linux headers ci: fix Travis config warnings ci: use an explicit list of Travis jobs ci: reorganise Travis jobs .travis.yml | 139 ++++++++++++++++++++-------------------------------- 1 file changed, 54 insertions(+), 85 deletions(-) -- 2.23.0
Following removal of kmod compilation, we don't need to install linux-headers anymore. Fixes: ea860973592b ("ci: remove redundant configs disabling kmods") Signed-off-by: David Marchand <david.marchand@redhat.com> --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 1d43f16ae..671447ee8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -17,7 +17,7 @@ addons: apt: update: true packages: &required_packages - - [libnuma-dev, linux-headers-$(uname -r), python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] aarch64_packages: &aarch64_packages - *required_packages -- 2.23.0
Reading https://config.travis-ci.com/ and using https://config.travis-ci.com/explore to check changes, we can cleanup some warnings reported by the config validation options in Travis. Signed-off-by: David Marchand <david.marchand@redhat.com> --- .travis.yml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.travis.yml b/.travis.yml index 671447ee8..7c1b2ff1b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -19,22 +19,22 @@ addons: packages: &required_packages - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] -aarch64_packages: &aarch64_packages +_aarch64_packages: &aarch64_packages - *required_packages - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu] -extra_packages: &extra_packages +_extra_packages: &extra_packages - *required_packages - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4] -libabigail_build_packages: &libabigail_build_packages +_libabigail_build_packages: &libabigail_build_packages - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev] -build_32b_packages: &build_32b_packages +_build_32b_packages: &build_32b_packages - *required_packages - [gcc-multilib] -doc_packages: &doc_packages +_doc_packages: &doc_packages - [doxygen, graphviz, python3-sphinx] before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh @@ -44,7 +44,7 @@ env: - DEF_LIB="shared" - DEF_LIB="shared" RUN_TESTS=1 -matrix: +jobs: include: - env: DEF_LIB="static" BUILD_32BIT=1 compiler: gcc -- 2.23.0
Maintaining the .travis.yml requires some knowledge of how Travis computes the jobs list (combination of os: arch: compiler: etc...). Let's switch to an explicit list to find all jobs at a glance. To enhance readability, jobs have been sorted per arch/compiler with comments to isolate blocks. Setting required_packages for aarch64 native jobs is unnecessary, the global addons: values are the same. This commit does not change the jobs list (21 jobs in total). Signed-off-by: David Marchand <david.marchand@redhat.com> --- .travis.yml | 102 ++++++++++++++++++++++++++-------------------------- 1 file changed, 52 insertions(+), 50 deletions(-) diff --git a/.travis.yml b/.travis.yml index 7c1b2ff1b..c174d5f30 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,17 +1,14 @@ +# default values for all jobs language: c cache: ccache: true directories: - libabigail - reference -compiler: - - gcc - - clang dist: bionic -os: - - linux +os: linux addons: apt: @@ -38,98 +35,105 @@ _doc_packages: &doc_packages - [doxygen, graphviz, python3-sphinx] before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh - -env: - - DEF_LIB="static" - - DEF_LIB="shared" - - DEF_LIB="shared" RUN_TESTS=1 +script: ./.ci/${TRAVIS_OS_NAME}-build.sh jobs: include: - - env: DEF_LIB="static" BUILD_32BIT=1 + # x86_64 gcc jobs + - env: DEF_LIB="static" + arch: amd64 compiler: gcc - addons: - apt: - packages: - - *build_32b_packages - - env: DEF_LIB="static" AARCH64=1 + - env: DEF_LIB="shared" + arch: amd64 compiler: gcc - addons: - apt: - packages: - - *aarch64_packages - - env: DEF_LIB="shared" AARCH64=1 + - env: DEF_LIB="shared" RUN_TESTS=1 + arch: amd64 compiler: gcc - addons: - apt: - packages: - - *aarch64_packages - env: DEF_LIB="static" EXTRA_PACKAGES=1 + arch: amd64 compiler: gcc addons: apt: packages: - *extra_packages - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1 + arch: amd64 compiler: gcc addons: apt: packages: - *extra_packages - *doc_packages - - env: DEF_LIB="static" EXTRA_PACKAGES=1 + - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1 + arch: amd64 compiler: gcc addons: apt: packages: - *extra_packages + - *libabigail_build_packages + # x86_64 clang jobs + - env: DEF_LIB="static" + arch: amd64 + compiler: clang + - env: DEF_LIB="shared" + arch: amd64 + compiler: clang + - env: DEF_LIB="shared" RUN_TESTS=1 + arch: amd64 + compiler: clang - env: DEF_LIB="static" EXTRA_PACKAGES=1 + arch: amd64 compiler: clang addons: apt: packages: - *extra_packages - env: DEF_LIB="shared" EXTRA_PACKAGES=1 + arch: amd64 compiler: clang addons: apt: packages: - *extra_packages - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1 + arch: amd64 compiler: clang addons: apt: packages: - *extra_packages - *doc_packages - - env: DEF_LIB="static" - arch: arm64 + # x86_64 cross-compiling 32-bits jobs + - env: DEF_LIB="static" BUILD_32BIT=1 + arch: amd64 compiler: gcc addons: apt: packages: - - *required_packages - - env: DEF_LIB="shared" - arch: arm64 + - *build_32b_packages + # x86_64 cross-compiling aarch64 jobs + - env: DEF_LIB="static" AARCH64=1 + arch: amd64 compiler: gcc addons: apt: packages: - - *required_packages - - env: DEF_LIB="static" - arch: arm64 - compiler: clang + - *aarch64_packages + - env: DEF_LIB="shared" AARCH64=1 + arch: amd64 + compiler: gcc addons: apt: packages: - - *required_packages + - *aarch64_packages + # aarch64 gcc jobs + - env: DEF_LIB="static" + arch: arm64 + compiler: gcc - env: DEF_LIB="shared" arch: arm64 - compiler: clang - addons: - apt: - packages: - - *required_packages + compiler: gcc - env: DEF_LIB="shared" BUILD_DOCS=1 arch: arm64 compiler: gcc @@ -138,13 +142,6 @@ jobs: packages: - *required_packages - *doc_packages - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1 - compiler: gcc - addons: - apt: - packages: - - *extra_packages - - *libabigail_build_packages - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1 arch: arm64 compiler: gcc @@ -153,5 +150,10 @@ jobs: packages: - *extra_packages - *libabigail_build_packages - -script: ./.ci/${TRAVIS_OS_NAME}-build.sh + # aarch64 clang jobs + - env: DEF_LIB="static" + arch: arm64 + compiler: clang + - env: DEF_LIB="shared" + arch: arm64 + compiler: clang -- 2.23.0
Let's prune the jobs list to limit the amount of time spent by the robot in Travis. Since meson enables automatically the relevant components, there is not much gain in testing with extra_packages vs required_packages only. For a given arch/compiler/env combination, compilation is first tested in all jobs that run tests or build the docs or run the ABI checks. In the same context, for jobs that accumulates running tests, building the docs etc..., those steps are independent and can be split to save some cpu on Travis. With this, we go down from 21 to 15 jobs. Note: this patch requires a flush of the existing caches in Travis. Signed-off-by: David Marchand <david.marchand@redhat.com> --- .travis.yml | 51 +++++++++------------------------------------------ 1 file changed, 9 insertions(+), 42 deletions(-) diff --git a/.travis.yml b/.travis.yml index c174d5f30..b64a81bd0 100644 --- a/.travis.yml +++ b/.travis.yml @@ -15,15 +15,12 @@ addons: update: true packages: &required_packages - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4] _aarch64_packages: &aarch64_packages - *required_packages - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu] -_extra_packages: &extra_packages - - *required_packages - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4] - _libabigail_build_packages: &libabigail_build_packages - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev] @@ -43,66 +40,39 @@ jobs: - env: DEF_LIB="static" arch: amd64 compiler: gcc - - env: DEF_LIB="shared" - arch: amd64 - compiler: gcc - env: DEF_LIB="shared" RUN_TESTS=1 arch: amd64 compiler: gcc - - env: DEF_LIB="static" EXTRA_PACKAGES=1 - arch: amd64 - compiler: gcc - addons: - apt: - packages: - - *extra_packages - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1 + - env: DEF_LIB="shared" BUILD_DOCS=1 arch: amd64 compiler: gcc addons: apt: packages: - - *extra_packages + - *required_packages - *doc_packages - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1 + - env: DEF_LIB="shared" ABI_CHECKS=1 arch: amd64 compiler: gcc addons: apt: packages: - - *extra_packages + - *required_packages - *libabigail_build_packages # x86_64 clang jobs - env: DEF_LIB="static" arch: amd64 compiler: clang - - env: DEF_LIB="shared" - arch: amd64 - compiler: clang - env: DEF_LIB="shared" RUN_TESTS=1 arch: amd64 compiler: clang - - env: DEF_LIB="static" EXTRA_PACKAGES=1 - arch: amd64 - compiler: clang - addons: - apt: - packages: - - *extra_packages - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 - arch: amd64 - compiler: clang - addons: - apt: - packages: - - *extra_packages - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1 + - env: DEF_LIB="shared" BUILD_DOCS=1 arch: amd64 compiler: clang addons: apt: packages: - - *extra_packages + - *required_packages - *doc_packages # x86_64 cross-compiling 32-bits jobs - env: DEF_LIB="static" BUILD_32BIT=1 @@ -131,9 +101,6 @@ jobs: - env: DEF_LIB="static" arch: arm64 compiler: gcc - - env: DEF_LIB="shared" - arch: arm64 - compiler: gcc - env: DEF_LIB="shared" BUILD_DOCS=1 arch: arm64 compiler: gcc @@ -142,13 +109,13 @@ jobs: packages: - *required_packages - *doc_packages - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1 + - env: DEF_LIB="shared" ABI_CHECKS=1 arch: arm64 compiler: gcc addons: apt: packages: - - *extra_packages + - *required_packages - *libabigail_build_packages # aarch64 clang jobs - env: DEF_LIB="static" -- 2.23.0
David Marchand <david.marchand@redhat.com> writes: > Let's prune the jobs list to limit the amount of time spent by the robot > in Travis. > > Since meson enables automatically the relevant components, there is not > much gain in testing with extra_packages vs required_packages only. > > For a given arch/compiler/env combination, compilation is first tested > in all jobs that run tests or build the docs or run the ABI checks. > In the same context, for jobs that accumulates running tests, building > the docs etc..., those steps are independent and can be split to save > some cpu on Travis. > > With this, we go down from 21 to 15 jobs. > > Note: this patch requires a flush of the existing caches in Travis. > > Signed-off-by: David Marchand <david.marchand@redhat.com> > --- In general, I think the idea with required vs. extra was to have a build that did the minimum required, and one that did all the packages (to allow a minimum vs. full DPDK). At least, that's from http://mails.dpdk.org/archives/dev/2019-January/124007.html Not sure if that's still something anyone cares about. > .travis.yml | 51 +++++++++------------------------------------------ > 1 file changed, 9 insertions(+), 42 deletions(-) > > diff --git a/.travis.yml b/.travis.yml > index c174d5f30..b64a81bd0 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -15,15 +15,12 @@ addons: > update: true > packages: &required_packages > - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build] > + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4] > > _aarch64_packages: &aarch64_packages > - *required_packages > - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu] > > -_extra_packages: &extra_packages > - - *required_packages > - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4] > - > _libabigail_build_packages: &libabigail_build_packages > - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev] > > @@ -43,66 +40,39 @@ jobs: > - env: DEF_LIB="static" > arch: amd64 > compiler: gcc > - - env: DEF_LIB="shared" > - arch: amd64 > - compiler: gcc > - env: DEF_LIB="shared" RUN_TESTS=1 > arch: amd64 > compiler: gcc > - - env: DEF_LIB="static" EXTRA_PACKAGES=1 > - arch: amd64 > - compiler: gcc > - addons: > - apt: > - packages: > - - *extra_packages > - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1 > + - env: DEF_LIB="shared" BUILD_DOCS=1 > arch: amd64 > compiler: gcc > addons: > apt: > packages: > - - *extra_packages > + - *required_packages > - *doc_packages > - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1 > + - env: DEF_LIB="shared" ABI_CHECKS=1 > arch: amd64 > compiler: gcc > addons: > apt: > packages: > - - *extra_packages > + - *required_packages > - *libabigail_build_packages > # x86_64 clang jobs > - env: DEF_LIB="static" > arch: amd64 > compiler: clang > - - env: DEF_LIB="shared" > - arch: amd64 > - compiler: clang > - env: DEF_LIB="shared" RUN_TESTS=1 > arch: amd64 > compiler: clang > - - env: DEF_LIB="static" EXTRA_PACKAGES=1 > - arch: amd64 > - compiler: clang > - addons: > - apt: > - packages: > - - *extra_packages > - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 > - arch: amd64 > - compiler: clang > - addons: > - apt: > - packages: > - - *extra_packages > - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1 > + - env: DEF_LIB="shared" BUILD_DOCS=1 > arch: amd64 > compiler: clang > addons: > apt: > packages: > - - *extra_packages > + - *required_packages > - *doc_packages > # x86_64 cross-compiling 32-bits jobs > - env: DEF_LIB="static" BUILD_32BIT=1 > @@ -131,9 +101,6 @@ jobs: > - env: DEF_LIB="static" > arch: arm64 > compiler: gcc > - - env: DEF_LIB="shared" > - arch: arm64 > - compiler: gcc > - env: DEF_LIB="shared" BUILD_DOCS=1 > arch: arm64 > compiler: gcc > @@ -142,13 +109,13 @@ jobs: > packages: > - *required_packages > - *doc_packages > - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1 > + - env: DEF_LIB="shared" ABI_CHECKS=1 > arch: arm64 > compiler: gcc > addons: > apt: > packages: > - - *extra_packages > + - *required_packages > - *libabigail_build_packages > # aarch64 clang jobs > - env: DEF_LIB="static"
On 2020-02-19 11:41, David Marchand wrote:
> Let's prune the jobs list to limit the amount of time spent by the
> robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> .travis.yml | 51 +++++++++------------------------------------------
> 1 file changed, 9 insertions(+), 42 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
> update: true
> packages: &required_packages
> - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip,
> ninja-build]
> + - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
>
> _aarch64_packages: &aarch64_packages
> - *required_packages
> - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross,
> pkg-config-aarch64-linux-gnu]
>
> -_extra_packages: &extra_packages
> - - *required_packages
> - - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
> -
> _libabigail_build_packages: &libabigail_build_packages
> - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>
> @@ -43,66 +40,39 @@ jobs:
> - env: DEF_LIB="static"
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: gcc
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: gcc
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: gcc
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: amd64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # x86_64 clang jobs
> - env: DEF_LIB="static"
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="shared"
> - arch: amd64
> - compiler: clang
> - env: DEF_LIB="shared" RUN_TESTS=1
> arch: amd64
> compiler: clang
> - - env: DEF_LIB="static" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> - arch: amd64
> - compiler: clang
> - addons:
> - apt:
> - packages:
> - - *extra_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> + - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: amd64
> compiler: clang
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *doc_packages
> # x86_64 cross-compiling 32-bits jobs
> - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
> - env: DEF_LIB="static"
> arch: arm64
> compiler: gcc
> - - env: DEF_LIB="shared"
> - arch: arm64
> - compiler: gcc
> - env: DEF_LIB="shared" BUILD_DOCS=1
> arch: arm64
> compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
> packages:
> - *required_packages
> - *doc_packages
> - - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> + - env: DEF_LIB="shared" ABI_CHECKS=1
> arch: arm64
> compiler: gcc
> addons:
> apt:
> packages:
> - - *extra_packages
> + - *required_packages
> - *libabigail_build_packages
> # aarch64 clang jobs
> - env: DEF_LIB="static"
Much easier to follow, I had no issues adding ppc64le jobs.
All my travis jobs ran fine.
LGTM (the series)
19/02/2020 22:39, Aaron Conole: > David Marchand <david.marchand@redhat.com> writes: > > > Let's prune the jobs list to limit the amount of time spent by the robot > > in Travis. > > > > Since meson enables automatically the relevant components, there is not > > much gain in testing with extra_packages vs required_packages only. > > > > For a given arch/compiler/env combination, compilation is first tested > > in all jobs that run tests or build the docs or run the ABI checks. > > In the same context, for jobs that accumulates running tests, building > > the docs etc..., those steps are independent and can be split to save > > some cpu on Travis. > > > > With this, we go down from 21 to 15 jobs. > > > > Note: this patch requires a flush of the existing caches in Travis. > > > > Signed-off-by: David Marchand <david.marchand@redhat.com> > > --- > > In general, I think the idea with required vs. extra was to have a build > that did the minimum required, and one that did all the packages (to > allow a minimum vs. full DPDK). > > At least, that's from > http://mails.dpdk.org/archives/dev/2019-January/124007.html I think the benefit of a minimum build is to have a quick report, and easy to setup. > Not sure if that's still something anyone cares about. Given that Travis knows how to satisfy the dependencies, and that we must wait for all jobs to finish, I don't see any benefit of a minimal setup.
19/02/2020 20:41, David Marchand:
> Following removal of kmod compilation, we don't need to install
> linux-headers anymore.
>
> Fixes: ea860973592b ("ci: remove redundant configs disabling kmods")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
19/02/2020 20:41, David Marchand:
> Reading https://config.travis-ci.com/ and using
> https://config.travis-ci.com/explore to check changes, we can cleanup
> some warnings reported by the config validation options in Travis.
For documentation purpose, would be good to describe issues in the commit log.
19/02/2020 20:41, David Marchand:
> Maintaining the .travis.yml requires some knowledge of how Travis
> computes the jobs list (combination of os: arch: compiler: etc...).
> Let's switch to an explicit list to find all jobs at a glance.
>
> To enhance readability, jobs have been sorted per arch/compiler with
> comments to isolate blocks.
>
> Setting required_packages for aarch64 native jobs is unnecessary,
> the global addons: values are the same.
>
> This commit does not change the jobs list (21 jobs in total).
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Thank you for thinking to poor humans like me with no Travis-fu skill.
19/02/2020 20:41, David Marchand:
> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
A small step for DPDK but a big save for the earth ;)
On Thu, Feb 20, 2020 at 12:03 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 19/02/2020 20:41, David Marchand:
> > Reading https://config.travis-ci.com/ and using
> > https://config.travis-ci.com/explore to check changes, we can cleanup
> > some warnings reported by the config validation options in Travis.
>
> For documentation purpose, would be good to describe issues in the commit log.
Ok, will do.
--
David Marchand
On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote: > > 19/02/2020 22:39, Aaron Conole: > > David Marchand <david.marchand@redhat.com> writes: > > > > > Let's prune the jobs list to limit the amount of time spent by the robot > > > in Travis. > > > > > > Since meson enables automatically the relevant components, there is not > > > much gain in testing with extra_packages vs required_packages only. > > > > > > For a given arch/compiler/env combination, compilation is first tested > > > in all jobs that run tests or build the docs or run the ABI checks. > > > In the same context, for jobs that accumulates running tests, building > > > the docs etc..., those steps are independent and can be split to save > > > some cpu on Travis. > > > > > > With this, we go down from 21 to 15 jobs. > > > > > > Note: this patch requires a flush of the existing caches in Travis. > > > > > > Signed-off-by: David Marchand <david.marchand@redhat.com> > > > --- > > > > In general, I think the idea with required vs. extra was to have a build > > that did the minimum required, and one that did all the packages (to > > allow a minimum vs. full DPDK). > > > > At least, that's from > > http://mails.dpdk.org/archives/dev/2019-January/124007.html > > I think the benefit of a minimum build is to have a quick report, > and easy to setup. Yes, Travis serves as a first gate when submitting patches. But since Travis is best effort/free, we can't have a full coverage. > > Not sure if that's still something anyone cares about. > > Given that Travis knows how to satisfy the dependencies, > and that we must wait for all jobs to finish, > I don't see any benefit of a minimal setup. This minimal setup also tests that dpdk dependencies are correct. If a change makes something rely on libX and libX is in the packages always installed in Travis, the missing dependency would not get caught. But here, this adds too many jobs. UNH, Intel and other CIs should step in and fill this kind of gap. -- David Marchand
David Marchand <david.marchand@redhat.com> writes: > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote: >> >> 19/02/2020 22:39, Aaron Conole: >> > David Marchand <david.marchand@redhat.com> writes: >> > >> > > Let's prune the jobs list to limit the amount of time spent by the robot >> > > in Travis. >> > > >> > > Since meson enables automatically the relevant components, there is not >> > > much gain in testing with extra_packages vs required_packages only. >> > > >> > > For a given arch/compiler/env combination, compilation is first tested >> > > in all jobs that run tests or build the docs or run the ABI checks. >> > > In the same context, for jobs that accumulates running tests, building >> > > the docs etc..., those steps are independent and can be split to save >> > > some cpu on Travis. >> > > >> > > With this, we go down from 21 to 15 jobs. >> > > >> > > Note: this patch requires a flush of the existing caches in Travis. >> > > >> > > Signed-off-by: David Marchand <david.marchand@redhat.com> >> > > --- >> > >> > In general, I think the idea with required vs. extra was to have a build >> > that did the minimum required, and one that did all the packages (to >> > allow a minimum vs. full DPDK). >> > >> > At least, that's from >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html >> >> I think the benefit of a minimum build is to have a quick report, >> and easy to setup. > > Yes, Travis serves as a first gate when submitting patches. > But since Travis is best effort/free, we can't have a full coverage. > > >> > Not sure if that's still something anyone cares about. >> >> Given that Travis knows how to satisfy the dependencies, >> and that we must wait for all jobs to finish, >> I don't see any benefit of a minimal setup. > > This minimal setup also tests that dpdk dependencies are correct. > If a change makes something rely on libX and libX is in the packages > always installed in Travis, the missing dependency would not get > caught. > > But here, this adds too many jobs. > > UNH, Intel and other CIs should step in and fill this kind of gap. Okay, makes sense to me. Are one of these CI providers offering to cover this? Also, Acked-by: Aaron Conole <aconole@redhat.com> > > -- > David Marchand
David Marchand <david.marchand@redhat.com> writes:
> Maintaining the .travis.yml requires some knowledge of how Travis
> computes the jobs list (combination of os: arch: compiler: etc...).
> Let's switch to an explicit list to find all jobs at a glance.
>
> To enhance readability, jobs have been sorted per arch/compiler with
> comments to isolate blocks.
>
> Setting required_packages for aarch64 native jobs is unnecessary,
> the global addons: values are the same.
>
> This commit does not change the jobs list (21 jobs in total).
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Acked-by: Aaron Conole <aconole@redhat.com>
David Marchand <david.marchand@redhat.com> writes:
> Reading https://config.travis-ci.com/ and using
> https://config.travis-ci.com/explore to check changes, we can cleanup
> some warnings reported by the config validation options in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Acked-by: Aaron Conole <aconole@redhat.com>
Echoing what Thomas wrote, please include the issues warnings.
David Marchand <david.marchand@redhat.com> writes:
> Following removal of kmod compilation, we don't need to install
> linux-headers anymore.
>
> Fixes: ea860973592b ("ci: remove redundant configs disabling kmods")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
Acked-by: Aaron Conole <aconole@redhat.com>
On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >> 19/02/2020 22:39, Aaron Conole:
> >> > David Marchand <david.marchand@redhat.com> writes:
> >> >
> >> > > Let's prune the jobs list to limit the amount of time spent by the robot
> >> > > in Travis.
> >> > >
> >> > > Since meson enables automatically the relevant components, there is not
> >> > > much gain in testing with extra_packages vs required_packages only.
> >> > >
> >> > > For a given arch/compiler/env combination, compilation is first tested
> >> > > in all jobs that run tests or build the docs or run the ABI checks.
> >> > > In the same context, for jobs that accumulates running tests, building
> >> > > the docs etc..., those steps are independent and can be split to save
> >> > > some cpu on Travis.
> >> > >
> >> > > With this, we go down from 21 to 15 jobs.
> >> > >
> >> > > Note: this patch requires a flush of the existing caches in Travis.
> >> > >
> >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >> > > ---
> >> >
> >> > In general, I think the idea with required vs. extra was to have a build
> >> > that did the minimum required, and one that did all the packages (to
> >> > allow a minimum vs. full DPDK).
> >> >
> >> > At least, that's from
> >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
> >>
> >> I think the benefit of a minimum build is to have a quick report,
> >> and easy to setup.
> >
> > Yes, Travis serves as a first gate when submitting patches.
> > But since Travis is best effort/free, we can't have a full coverage.
> >
> >
> >> > Not sure if that's still something anyone cares about.
> >>
> >> Given that Travis knows how to satisfy the dependencies,
> >> and that we must wait for all jobs to finish,
> >> I don't see any benefit of a minimal setup.
> >
> > This minimal setup also tests that dpdk dependencies are correct.
> > If a change makes something rely on libX and libX is in the packages
> > always installed in Travis, the missing dependency would not get
> > caught.
> >
> > But here, this adds too many jobs.
> >
> > UNH, Intel and other CIs should step in and fill this kind of gap.
>
> Okay, makes sense to me. Are one of these CI providers offering to
> cover this?
Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org.
For the CI guys, which packages are installed on the systems/vms that
do compilation tests?
Is it possible to have a summary of the different setups?
Thanks.
--
David Marchand
On Thu, Feb 20, 2020 at 12:03 PM Thomas Monjalon <thomas@monjalon.net> wrote: > > 19/02/2020 20:41, David Marchand: > > Reading https://config.travis-ci.com/ and using > > https://config.travis-ci.com/explore to check changes, we can cleanup > > some warnings reported by the config validation options in Travis. > > For documentation purpose, would be good to describe issues in the commit log. Since I only have this commitlog to update, I intend to push the series with the following addition: Example on a job in master: https://travis-ci.com/DPDK/dpdk/builds/149537002/config Build config validation: root: deprecated key doc_packages (anchor on a non-private key) root: deprecated key build_32b_packages (anchor on a non-private key) root: deprecated key libabigail_build_packages (anchor on a non-private key) root: deprecated key extra_packages (anchor on a non-private key) root: deprecated key aarch64_packages (anchor on a non-private key) root: key matrix is an alias for jobs, using jobs For the "(anchor on a non-private key)" warnings, the Travis documentation [1] recommends prefixing private keys with _ (Travis schema is available at [2]). The use of the "matrix" key is allowed, but it is just an alias to the "jobs" key. 1: https://docs.travis-ci.com/user/build-config-yaml#private-keys-as-yaml-anchors-and-aliases-and-external-tooling 2: https://github.com/travis-ci/travis-yml/blob/730a77f4028b91293a55c8c1c52fc72e64379dde/schema.json -- David Marchand
[-- Attachment #1: Type: text/plain, Size: 3853 bytes --] On Thu, Feb 20, 2020 at 11:02 AM David Marchand <david.marchand@redhat.com> wrote: > > On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote: > > > > David Marchand <david.marchand@redhat.com> writes: > > > > > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote: > > >> > > >> 19/02/2020 22:39, Aaron Conole: > > >> > David Marchand <david.marchand@redhat.com> writes: > > >> > > > >> > > Let's prune the jobs list to limit the amount of time spent by the robot > > >> > > in Travis. > > >> > > > > >> > > Since meson enables automatically the relevant components, there is not > > >> > > much gain in testing with extra_packages vs required_packages only. > > >> > > > > >> > > For a given arch/compiler/env combination, compilation is first tested > > >> > > in all jobs that run tests or build the docs or run the ABI checks. > > >> > > In the same context, for jobs that accumulates running tests, building > > >> > > the docs etc..., those steps are independent and can be split to save > > >> > > some cpu on Travis. > > >> > > > > >> > > With this, we go down from 21 to 15 jobs. > > >> > > > > >> > > Note: this patch requires a flush of the existing caches in Travis. > > >> > > > > >> > > Signed-off-by: David Marchand <david.marchand@redhat.com> > > >> > > --- > > >> > > > >> > In general, I think the idea with required vs. extra was to have a build > > >> > that did the minimum required, and one that did all the packages (to > > >> > allow a minimum vs. full DPDK). > > >> > > > >> > At least, that's from > > >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html > > >> > > >> I think the benefit of a minimum build is to have a quick report, > > >> and easy to setup. > > > > > > Yes, Travis serves as a first gate when submitting patches. > > > But since Travis is best effort/free, we can't have a full coverage. > > > > > > > > >> > Not sure if that's still something anyone cares about. > > >> > > >> Given that Travis knows how to satisfy the dependencies, > > >> and that we must wait for all jobs to finish, > > >> I don't see any benefit of a minimal setup. > > > > > > This minimal setup also tests that dpdk dependencies are correct. > > > If a change makes something rely on libX and libX is in the packages > > > always installed in Travis, the missing dependency would not get > > > caught. > > > > > > But here, this adds too many jobs. > > > > > > UNH, Intel and other CIs should step in and fill this kind of gap. > > > > Okay, makes sense to me. Are one of these CI providers offering to > > cover this? > > Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org. > > For the CI guys, which packages are installed on the systems/vms that > do compilation tests? For our compile tests, we compile with pretty much all dpdk dependencies that can easily be installed from the distro's repository (the dependencies we installed are based on the meson output). The thought process was to test compiling as much of dpdk as possible (with constraints mentioned before). Here are the dependencies we install on some of the setups: Ubuntu 18.04: make gcc ninja-build pkg-config libnuma-dev librdmacm-dev librdmacm1 rdma-core libelf-dev libmnl-dev libpcap-dev libcrypto++-dev libjansson-dev (meson installed with pip) CentOS 8 and Fedora 31: diffutils gcc make ninja-build meson pkg-config numactl-devel librdmacm rdma-core-devel elfutils-libelf-devel libmnl-devel libpcap-devel jansson-devel openSuse Leap 15: gcc make ninja pkg-config libnuma-devel librdmacm1 rdma-core-devel libelf-devel libmnl-devel libpcap-devel libjansson-devel (meson installed with pip) > Is it possible to have a summary of the different setups? > I've attached an overview/summary of what we do at the UNH-IOL DPDK Community Lab. > > Thanks. > > -- > David Marchand > [-- Attachment #2: DPDK Lab Summary February 2020.pdf --] [-- Type: application/pdf, Size: 846111 bytes --]
David Marchand <david.marchand@redhat.com> writes:
> On Thu, Feb 20, 2020 at 12:03 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 19/02/2020 20:41, David Marchand:
>> > Reading https://config.travis-ci.com/ and using
>> > https://config.travis-ci.com/explore to check changes, we can cleanup
>> > some warnings reported by the config validation options in Travis.
>>
>> For documentation purpose, would be good to describe issues in the commit log.
>
> Since I only have this commitlog to update, I intend to push the
> series with the following addition:
>
> Example on a job in master:
> https://travis-ci.com/DPDK/dpdk/builds/149537002/config
>
> Build config validation:
> root: deprecated key doc_packages (anchor on a non-private key)
> root: deprecated key build_32b_packages (anchor on a non-private key)
> root: deprecated key libabigail_build_packages (anchor on a non-private key)
> root: deprecated key extra_packages (anchor on a non-private key)
> root: deprecated key aarch64_packages (anchor on a non-private key)
> root: key matrix is an alias for jobs, using jobs
>
> For the "(anchor on a non-private key)" warnings, the Travis
> documentation [1] recommends prefixing private keys with _ (Travis
> schema is available at [2]).
>
> The use of the "matrix" key is allowed, but it is just an alias to the
> "jobs" key.
>
> 1:
> https://docs.travis-ci.com/user/build-config-yaml#private-keys-as-yaml-anchors-and-aliases-and-external-tooling
> 2:
> https://github.com/travis-ci/travis-yml/blob/730a77f4028b91293a55c8c1c52fc72e64379dde/schema.json
LGTM
On Wed, Feb 19, 2020 at 8:41 PM David Marchand
<david.marchand@redhat.com> wrote:
> This series tries to make the travis configuration a bit easier to read,
> then drops jobs that are not essential or functional duplicates in the
> hope we can give some oxygen to the robot who seems to lag quite a bit
> and can't catch up.
Travis cache for master flushed.
Series applied.
--
David Marchand
20/02/2020 22:01, Aaron Conole:
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Thu, Feb 20, 2020 at 12:03 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >> 19/02/2020 20:41, David Marchand:
> >> > Reading https://config.travis-ci.com/ and using
> >> > https://config.travis-ci.com/explore to check changes, we can cleanup
> >> > some warnings reported by the config validation options in Travis.
> >>
> >> For documentation purpose, would be good to describe issues in the commit log.
> >
> > Since I only have this commitlog to update, I intend to push the
> > series with the following addition:
> >
> > Example on a job in master:
> > https://travis-ci.com/DPDK/dpdk/builds/149537002/config
> >
> > Build config validation:
> > root: deprecated key doc_packages (anchor on a non-private key)
> > root: deprecated key build_32b_packages (anchor on a non-private key)
> > root: deprecated key libabigail_build_packages (anchor on a non-private key)
> > root: deprecated key extra_packages (anchor on a non-private key)
> > root: deprecated key aarch64_packages (anchor on a non-private key)
> > root: key matrix is an alias for jobs, using jobs
> >
> > For the "(anchor on a non-private key)" warnings, the Travis
> > documentation [1] recommends prefixing private keys with _ (Travis
> > schema is available at [2]).
> >
> > The use of the "matrix" key is allowed, but it is just an alias to the
> > "jobs" key.
> >
> > 1:
> > https://docs.travis-ci.com/user/build-config-yaml#private-keys-as-yaml-anchors-and-aliases-and-external-tooling
> > 2:
> > https://github.com/travis-ci/travis-yml/blob/730a77f4028b91293a55c8c1c52fc72e64379dde/schema.json
>
> LGTM
Perfect, thanks