[dpdk-dev] [PATCH 0/4] Reorganise Travis jobs

[dpdk-dev] [PATCH 0/4] Reorganise Travis jobs

[David Marchand]

This series tries to make the travis configuration a bit easier to read,
then drops jobs that are not essential or functional duplicates in the
hope we can give some oxygen to the robot who seems to lag quite a bit
and can't catch up.

-- 
David Marchand

David Marchand (4):
  ci: remove unnecessary dependency on linux headers
  ci: fix Travis config warnings
  ci: use an explicit list of Travis jobs
  ci: reorganise Travis jobs

 .travis.yml | 139 ++++++++++++++++++++--------------------------------
 1 file changed, 54 insertions(+), 85 deletions(-)

-- 
2.23.0

[dpdk-dev] [PATCH 1/4] ci: remove unnecessary dependency on Linux headers

[David Marchand]

Following removal of kmod compilation, we don't need to install
linux-headers anymore.

Fixes: ea860973592b ("ci: remove redundant configs disabling kmods")

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 1d43f16ae..671447ee8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -17,7 +17,7 @@ addons:
   apt:
     update: true
     packages: &required_packages
-      - [libnuma-dev, linux-headers-$(uname -r), python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
 
 aarch64_packages: &aarch64_packages
   - *required_packages
-- 
2.23.0

Re: [dpdk-dev] [PATCH 1/4] ci: remove unnecessary dependency on Linux headers

[Thomas Monjalon]

19/02/2020 20:41, David Marchand:
> Following removal of kmod compilation, we don't need to install
> linux-headers anymore.
> 
> Fixes: ea860973592b ("ci: remove redundant configs disabling kmods")
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>

Acked-by: Thomas Monjalon <thomas@monjalon.net>

Re: [dpdk-dev] [PATCH 1/4] ci: remove unnecessary dependency on Linux headers

[Aaron Conole]

David Marchand <david.marchand@redhat.com> writes:

> Following removal of kmod compilation, we don't need to install
> linux-headers anymore.
>
> Fixes: ea860973592b ("ci: remove redundant configs disabling kmods")
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

Acked-by: Aaron Conole <aconole@redhat.com>

[dpdk-dev] [PATCH 2/4] ci: fix Travis config warnings

[David Marchand]

Reading https://config.travis-ci.com/ and using
https://config.travis-ci.com/explore to check changes, we can cleanup
some warnings reported by the config validation options in Travis.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .travis.yml | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 671447ee8..7c1b2ff1b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -19,22 +19,22 @@ addons:
     packages: &required_packages
       - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
 
-aarch64_packages: &aarch64_packages
+_aarch64_packages: &aarch64_packages
   - *required_packages
   - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
 
-extra_packages: &extra_packages
+_extra_packages: &extra_packages
   - *required_packages
   - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
 
-libabigail_build_packages: &libabigail_build_packages
+_libabigail_build_packages: &libabigail_build_packages
   - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
 
-build_32b_packages: &build_32b_packages
+_build_32b_packages: &build_32b_packages
   - *required_packages
   - [gcc-multilib]
 
-doc_packages: &doc_packages
+_doc_packages: &doc_packages
   - [doxygen, graphviz, python3-sphinx]
 
 before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
@@ -44,7 +44,7 @@ env:
   - DEF_LIB="shared"
   - DEF_LIB="shared" RUN_TESTS=1
 
-matrix:
+jobs:
   include:
   - env: DEF_LIB="static" BUILD_32BIT=1
     compiler: gcc
-- 
2.23.0

Re: [dpdk-dev] [PATCH 2/4] ci: fix Travis config warnings

[Thomas Monjalon]

19/02/2020 20:41, David Marchand:
> Reading https://config.travis-ci.com/ and using
> https://config.travis-ci.com/explore to check changes, we can cleanup
> some warnings reported by the config validation options in Travis.

For documentation purpose, would be good to describe issues in the commit log.

Re: [dpdk-dev] [PATCH 2/4] ci: fix Travis config warnings

[David Marchand]

On Thu, Feb 20, 2020 at 12:03 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 19/02/2020 20:41, David Marchand:
> > Reading https://config.travis-ci.com/ and using
> > https://config.travis-ci.com/explore to check changes, we can cleanup
> > some warnings reported by the config validation options in Travis.
>
> For documentation purpose, would be good to describe issues in the commit log.

Ok, will do.


-- 
David Marchand

Re: [dpdk-dev] [PATCH 2/4] ci: fix Travis config warnings

[David Marchand]

On Thu, Feb 20, 2020 at 12:03 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 19/02/2020 20:41, David Marchand:
> > Reading https://config.travis-ci.com/ and using
> > https://config.travis-ci.com/explore to check changes, we can cleanup
> > some warnings reported by the config validation options in Travis.
>
> For documentation purpose, would be good to describe issues in the commit log.

Since I only have this commitlog to update, I intend to push the
series with the following addition:

Example on a job in master:
https://travis-ci.com/DPDK/dpdk/builds/149537002/config

Build config validation:
root: deprecated key doc_packages (anchor on a non-private key)
root: deprecated key build_32b_packages (anchor on a non-private key)
root: deprecated key libabigail_build_packages (anchor on a non-private key)
root: deprecated key extra_packages (anchor on a non-private key)
root: deprecated key aarch64_packages (anchor on a non-private key)
root: key matrix is an alias for jobs, using jobs

For the "(anchor on a non-private key)" warnings, the Travis
documentation [1] recommends prefixing private keys with _ (Travis
schema is available at [2]).

The use of the "matrix" key is allowed, but it is just an alias to the
"jobs" key.

1: https://docs.travis-ci.com/user/build-config-yaml#private-keys-as-yaml-anchors-and-aliases-and-external-tooling
2: https://github.com/travis-ci/travis-yml/blob/730a77f4028b91293a55c8c1c52fc72e64379dde/schema.json


-- 
David Marchand

Re: [dpdk-dev] [PATCH 2/4] ci: fix Travis config warnings

[Aaron Conole]

David Marchand <david.marchand@redhat.com> writes:

> On Thu, Feb 20, 2020 at 12:03 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 19/02/2020 20:41, David Marchand:
>> > Reading https://config.travis-ci.com/ and using
>> > https://config.travis-ci.com/explore to check changes, we can cleanup
>> > some warnings reported by the config validation options in Travis.
>>
>> For documentation purpose, would be good to describe issues in the commit log.
>
> Since I only have this commitlog to update, I intend to push the
> series with the following addition:
>
> Example on a job in master:
> https://travis-ci.com/DPDK/dpdk/builds/149537002/config
>
> Build config validation:
> root: deprecated key doc_packages (anchor on a non-private key)
> root: deprecated key build_32b_packages (anchor on a non-private key)
> root: deprecated key libabigail_build_packages (anchor on a non-private key)
> root: deprecated key extra_packages (anchor on a non-private key)
> root: deprecated key aarch64_packages (anchor on a non-private key)
> root: key matrix is an alias for jobs, using jobs
>
> For the "(anchor on a non-private key)" warnings, the Travis
> documentation [1] recommends prefixing private keys with _ (Travis
> schema is available at [2]).
>
> The use of the "matrix" key is allowed, but it is just an alias to the
> "jobs" key.
>
> 1:
> https://docs.travis-ci.com/user/build-config-yaml#private-keys-as-yaml-anchors-and-aliases-and-external-tooling
> 2:
> https://github.com/travis-ci/travis-yml/blob/730a77f4028b91293a55c8c1c52fc72e64379dde/schema.json

LGTM

Re: [dpdk-dev] [PATCH 2/4] ci: fix Travis config warnings

[Thomas Monjalon]

20/02/2020 22:01, Aaron Conole:
> David Marchand <david.marchand@redhat.com> writes:
> 
> > On Thu, Feb 20, 2020 at 12:03 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >> 19/02/2020 20:41, David Marchand:
> >> > Reading https://config.travis-ci.com/ and using
> >> > https://config.travis-ci.com/explore to check changes, we can cleanup
> >> > some warnings reported by the config validation options in Travis.
> >>
> >> For documentation purpose, would be good to describe issues in the commit log.
> >
> > Since I only have this commitlog to update, I intend to push the
> > series with the following addition:
> >
> > Example on a job in master:
> > https://travis-ci.com/DPDK/dpdk/builds/149537002/config
> >
> > Build config validation:
> > root: deprecated key doc_packages (anchor on a non-private key)
> > root: deprecated key build_32b_packages (anchor on a non-private key)
> > root: deprecated key libabigail_build_packages (anchor on a non-private key)
> > root: deprecated key extra_packages (anchor on a non-private key)
> > root: deprecated key aarch64_packages (anchor on a non-private key)
> > root: key matrix is an alias for jobs, using jobs
> >
> > For the "(anchor on a non-private key)" warnings, the Travis
> > documentation [1] recommends prefixing private keys with _ (Travis
> > schema is available at [2]).
> >
> > The use of the "matrix" key is allowed, but it is just an alias to the
> > "jobs" key.
> >
> > 1:
> > https://docs.travis-ci.com/user/build-config-yaml#private-keys-as-yaml-anchors-and-aliases-and-external-tooling
> > 2:
> > https://github.com/travis-ci/travis-yml/blob/730a77f4028b91293a55c8c1c52fc72e64379dde/schema.json
> 
> LGTM

Perfect, thanks

Re: [dpdk-dev] [PATCH 2/4] ci: fix Travis config warnings

[Aaron Conole]

David Marchand <david.marchand@redhat.com> writes:

> Reading https://config.travis-ci.com/ and using
> https://config.travis-ci.com/explore to check changes, we can cleanup
> some warnings reported by the config validation options in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

Acked-by: Aaron Conole <aconole@redhat.com>

Echoing what Thomas wrote, please include the issues warnings.

[dpdk-dev] [PATCH 3/4] ci: use an explicit list of Travis jobs

[David Marchand]

Maintaining the .travis.yml requires some knowledge of how Travis
computes the jobs list (combination of os: arch: compiler: etc...).
Let's switch to an explicit list to find all jobs at a glance.

To enhance readability, jobs have been sorted per arch/compiler with
comments to isolate blocks.

Setting required_packages for aarch64 native jobs is unnecessary,
the global addons: values are the same.

This commit does not change the jobs list (21 jobs in total).

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .travis.yml | 102 ++++++++++++++++++++++++++--------------------------
 1 file changed, 52 insertions(+), 50 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 7c1b2ff1b..c174d5f30 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,17 +1,14 @@
+# default values for all jobs
 language: c
 cache:
   ccache: true
   directories:
     - libabigail
     - reference
-compiler:
-  - gcc
-  - clang
 
 dist: bionic
 
-os:
-  - linux
+os: linux
 
 addons:
   apt:
@@ -38,98 +35,105 @@ _doc_packages: &doc_packages
   - [doxygen, graphviz, python3-sphinx]
 
 before_install: ./.ci/${TRAVIS_OS_NAME}-setup.sh
-
-env:
-  - DEF_LIB="static"
-  - DEF_LIB="shared"
-  - DEF_LIB="shared" RUN_TESTS=1
+script: ./.ci/${TRAVIS_OS_NAME}-build.sh
 
 jobs:
   include:
-  - env: DEF_LIB="static" BUILD_32BIT=1
+  # x86_64 gcc jobs
+  - env: DEF_LIB="static"
+    arch: amd64
     compiler: gcc
-    addons:
-      apt:
-        packages:
-          - *build_32b_packages
-  - env: DEF_LIB="static" AARCH64=1
+  - env: DEF_LIB="shared"
+    arch: amd64
     compiler: gcc
-    addons:
-      apt:
-        packages:
-          - *aarch64_packages
-  - env: DEF_LIB="shared" AARCH64=1
+  - env: DEF_LIB="shared" RUN_TESTS=1
+    arch: amd64
     compiler: gcc
-    addons:
-      apt:
-        packages:
-          - *aarch64_packages
   - env: DEF_LIB="static" EXTRA_PACKAGES=1
+    arch: amd64
     compiler: gcc
     addons:
       apt:
         packages:
           - *extra_packages
   - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+    arch: amd64
     compiler: gcc
     addons:
       apt:
         packages:
           - *extra_packages
           - *doc_packages
-  - env: DEF_LIB="static" EXTRA_PACKAGES=1
+  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+    arch: amd64
     compiler: gcc
     addons:
       apt:
         packages:
           - *extra_packages
+          - *libabigail_build_packages
+  # x86_64 clang jobs
+  - env: DEF_LIB="static"
+    arch: amd64
+    compiler: clang
+  - env: DEF_LIB="shared"
+    arch: amd64
+    compiler: clang
+  - env: DEF_LIB="shared" RUN_TESTS=1
+    arch: amd64
+    compiler: clang
   - env: DEF_LIB="static" EXTRA_PACKAGES=1
+    arch: amd64
     compiler: clang
     addons:
       apt:
         packages:
           - *extra_packages
   - env: DEF_LIB="shared" EXTRA_PACKAGES=1
+    arch: amd64
     compiler: clang
     addons:
       apt:
         packages:
           - *extra_packages
   - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+    arch: amd64
     compiler: clang
     addons:
       apt:
         packages:
           - *extra_packages
           - *doc_packages
-  - env: DEF_LIB="static"
-    arch: arm64
+  # x86_64 cross-compiling 32-bits jobs
+  - env: DEF_LIB="static" BUILD_32BIT=1
+    arch: amd64
     compiler: gcc
     addons:
       apt:
         packages:
-          - *required_packages
-  - env: DEF_LIB="shared"
-    arch: arm64
+          - *build_32b_packages
+  # x86_64 cross-compiling aarch64 jobs
+  - env: DEF_LIB="static" AARCH64=1
+    arch: amd64
     compiler: gcc
     addons:
       apt:
         packages:
-          - *required_packages
-  - env: DEF_LIB="static"
-    arch: arm64
-    compiler: clang
+          - *aarch64_packages
+  - env: DEF_LIB="shared" AARCH64=1
+    arch: amd64
+    compiler: gcc
     addons:
       apt:
         packages:
-          - *required_packages
+          - *aarch64_packages
+  # aarch64 gcc jobs
+  - env: DEF_LIB="static"
+    arch: arm64
+    compiler: gcc
   - env: DEF_LIB="shared"
     arch: arm64
-    compiler: clang
-    addons:
-      apt:
-        packages:
-          - *required_packages
+    compiler: gcc
   - env: DEF_LIB="shared" BUILD_DOCS=1
     arch: arm64
     compiler: gcc
@@ -138,13 +142,6 @@ jobs:
         packages:
           - *required_packages
           - *doc_packages
-  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
-    compiler: gcc
-    addons:
-      apt:
-        packages:
-          - *extra_packages
-          - *libabigail_build_packages
   - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
     arch: arm64
     compiler: gcc
@@ -153,5 +150,10 @@ jobs:
         packages:
           - *extra_packages
           - *libabigail_build_packages
-
-script: ./.ci/${TRAVIS_OS_NAME}-build.sh
+  # aarch64 clang jobs
+  - env: DEF_LIB="static"
+    arch: arm64
+    compiler: clang
+  - env: DEF_LIB="shared"
+    arch: arm64
+    compiler: clang
-- 
2.23.0

Re: [dpdk-dev] [PATCH 3/4] ci: use an explicit list of Travis jobs

[Thomas Monjalon]

19/02/2020 20:41, David Marchand:
> Maintaining the .travis.yml requires some knowledge of how Travis
> computes the jobs list (combination of os: arch: compiler: etc...).
> Let's switch to an explicit list to find all jobs at a glance.
> 
> To enhance readability, jobs have been sorted per arch/compiler with
> comments to isolate blocks.
> 
> Setting required_packages for aarch64 native jobs is unnecessary,
> the global addons: values are the same.
> 
> This commit does not change the jobs list (21 jobs in total).
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>

Acked-by: Thomas Monjalon <thomas@monjalon.net>

Thank you for thinking to poor humans like me with no Travis-fu skill.

Re: [dpdk-dev] [PATCH 3/4] ci: use an explicit list of Travis jobs

[Aaron Conole]

David Marchand <david.marchand@redhat.com> writes:

> Maintaining the .travis.yml requires some knowledge of how Travis
> computes the jobs list (combination of os: arch: compiler: etc...).
> Let's switch to an explicit list to find all jobs at a glance.
>
> To enhance readability, jobs have been sorted per arch/compiler with
> comments to isolate blocks.
>
> Setting required_packages for aarch64 native jobs is unnecessary,
> the global addons: values are the same.
>
> This commit does not change the jobs list (21 jobs in total).
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

Acked-by: Aaron Conole <aconole@redhat.com>

[dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs

[David Marchand]

Let's prune the jobs list to limit the amount of time spent by the robot
in Travis.

Since meson enables automatically the relevant components, there is not
much gain in testing with extra_packages vs required_packages only.

For a given arch/compiler/env combination, compilation is first tested
in all jobs that run tests or build the docs or run the ABI checks.
In the same context, for jobs that accumulates running tests, building
the docs etc..., those steps are independent and can be split to save
some cpu on Travis.

With this, we go down from 21 to 15 jobs.

Note: this patch requires a flush of the existing caches in Travis.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .travis.yml | 51 +++++++++------------------------------------------
 1 file changed, 9 insertions(+), 42 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index c174d5f30..b64a81bd0 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -15,15 +15,12 @@ addons:
     update: true
     packages: &required_packages
       - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
+      - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
 
 _aarch64_packages: &aarch64_packages
   - *required_packages
   - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
 
-_extra_packages: &extra_packages
-  - *required_packages
-  - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
-
 _libabigail_build_packages: &libabigail_build_packages
   - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
 
@@ -43,66 +40,39 @@ jobs:
   - env: DEF_LIB="static"
     arch: amd64
     compiler: gcc
-  - env: DEF_LIB="shared"
-    arch: amd64
-    compiler: gcc
   - env: DEF_LIB="shared" RUN_TESTS=1
     arch: amd64
     compiler: gcc
-  - env: DEF_LIB="static" EXTRA_PACKAGES=1
-    arch: amd64
-    compiler: gcc
-    addons:
-      apt:
-        packages:
-          - *extra_packages
-  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+  - env: DEF_LIB="shared" BUILD_DOCS=1
     arch: amd64
     compiler: gcc
     addons:
       apt:
         packages:
-          - *extra_packages
+          - *required_packages
           - *doc_packages
-  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+  - env: DEF_LIB="shared" ABI_CHECKS=1
     arch: amd64
     compiler: gcc
     addons:
       apt:
         packages:
-          - *extra_packages
+          - *required_packages
           - *libabigail_build_packages
   # x86_64 clang jobs
   - env: DEF_LIB="static"
     arch: amd64
     compiler: clang
-  - env: DEF_LIB="shared"
-    arch: amd64
-    compiler: clang
   - env: DEF_LIB="shared" RUN_TESTS=1
     arch: amd64
     compiler: clang
-  - env: DEF_LIB="static" EXTRA_PACKAGES=1
-    arch: amd64
-    compiler: clang
-    addons:
-      apt:
-        packages:
-          - *extra_packages
-  - env: DEF_LIB="shared" EXTRA_PACKAGES=1
-    arch: amd64
-    compiler: clang
-    addons:
-      apt:
-        packages:
-          - *extra_packages
-  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
+  - env: DEF_LIB="shared" BUILD_DOCS=1
     arch: amd64
     compiler: clang
     addons:
       apt:
         packages:
-          - *extra_packages
+          - *required_packages
           - *doc_packages
   # x86_64 cross-compiling 32-bits jobs
   - env: DEF_LIB="static" BUILD_32BIT=1
@@ -131,9 +101,6 @@ jobs:
   - env: DEF_LIB="static"
     arch: arm64
     compiler: gcc
-  - env: DEF_LIB="shared"
-    arch: arm64
-    compiler: gcc
   - env: DEF_LIB="shared" BUILD_DOCS=1
     arch: arm64
     compiler: gcc
@@ -142,13 +109,13 @@ jobs:
         packages:
           - *required_packages
           - *doc_packages
-  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
+  - env: DEF_LIB="shared" ABI_CHECKS=1
     arch: arm64
     compiler: gcc
     addons:
       apt:
         packages:
-          - *extra_packages
+          - *required_packages
           - *libabigail_build_packages
   # aarch64 clang jobs
   - env: DEF_LIB="static"
-- 
2.23.0

Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs

[Aaron Conole]

David Marchand <david.marchand@redhat.com> writes:

> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
>
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
>
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
>
> With this, we go down from 21 to 15 jobs.
>
> Note: this patch requires a flush of the existing caches in Travis.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

In general, I think the idea with required vs. extra was to have a build
that did the minimum required, and one that did all the packages (to
allow a minimum vs. full DPDK).

At least, that's from
http://mails.dpdk.org/archives/dev/2019-January/124007.html

Not sure if that's still something anyone cares about.

>  .travis.yml | 51 +++++++++------------------------------------------
>  1 file changed, 9 insertions(+), 42 deletions(-)
>
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
>      update: true
>      packages: &required_packages
>        - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip, ninja-build]
> +      - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
>
>  _aarch64_packages: &aarch64_packages
>    - *required_packages
>    - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross, pkg-config-aarch64-linux-gnu]
>  
> -_extra_packages: &extra_packages
> -  - *required_packages
> -  - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev, libfdt-dev, libjansson4]
> -
>  _libabigail_build_packages: &libabigail_build_packages
>    - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
>  
> @@ -43,66 +40,39 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="shared"
> -    arch: amd64
> -    compiler: gcc
>    - env: DEF_LIB="shared" RUN_TESTS=1
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="static" EXTRA_PACKAGES=1
> -    arch: amd64
> -    compiler: gcc
> -    addons:
> -      apt:
> -        packages:
> -          - *extra_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=1
>      arch: amd64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
> -          - *extra_packages
> +          - *required_packages
>            - *doc_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> +  - env: DEF_LIB="shared" ABI_CHECKS=1
>      arch: amd64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
> -          - *extra_packages
> +          - *required_packages
>            - *libabigail_build_packages
>    # x86_64 clang jobs
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="shared"
> -    arch: amd64
> -    compiler: clang
>    - env: DEF_LIB="shared" RUN_TESTS=1
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="static" EXTRA_PACKAGES=1
> -    arch: amd64
> -    compiler: clang
> -    addons:
> -      apt:
> -        packages:
> -          - *extra_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> -    arch: amd64
> -    compiler: clang
> -    addons:
> -      apt:
> -        packages:
> -          - *extra_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=1
>      arch: amd64
>      compiler: clang
>      addons:
>        apt:
>          packages:
> -          - *extra_packages
> +          - *required_packages
>            - *doc_packages
>    # x86_64 cross-compiling 32-bits jobs
>    - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: arm64
>      compiler: gcc
> -  - env: DEF_LIB="shared"
> -    arch: arm64
> -    compiler: gcc
>    - env: DEF_LIB="shared" BUILD_DOCS=1
>      arch: arm64
>      compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
>          packages:
>            - *required_packages
>            - *doc_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> +  - env: DEF_LIB="shared" ABI_CHECKS=1
>      arch: arm64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
> -          - *extra_packages
> +          - *required_packages
>            - *libabigail_build_packages

>    # aarch64 clang jobs
>    - env: DEF_LIB="static"

Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs

[Thomas Monjalon]

19/02/2020 22:39, Aaron Conole:
> David Marchand <david.marchand@redhat.com> writes:
> 
> > Let's prune the jobs list to limit the amount of time spent by the robot
> > in Travis.
> >
> > Since meson enables automatically the relevant components, there is not
> > much gain in testing with extra_packages vs required_packages only.
> >
> > For a given arch/compiler/env combination, compilation is first tested
> > in all jobs that run tests or build the docs or run the ABI checks.
> > In the same context, for jobs that accumulates running tests, building
> > the docs etc..., those steps are independent and can be split to save
> > some cpu on Travis.
> >
> > With this, we go down from 21 to 15 jobs.
> >
> > Note: this patch requires a flush of the existing caches in Travis.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> 
> In general, I think the idea with required vs. extra was to have a build
> that did the minimum required, and one that did all the packages (to
> allow a minimum vs. full DPDK).
> 
> At least, that's from
> http://mails.dpdk.org/archives/dev/2019-January/124007.html

I think the benefit of a minimum build is to have a quick report,
and easy to setup.

> Not sure if that's still something anyone cares about.

Given that Travis knows how to satisfy the dependencies,
and that we must wait for all jobs to finish,
I don't see any benefit of a minimal setup.

Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs

[David Marchand]

On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 19/02/2020 22:39, Aaron Conole:
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > Let's prune the jobs list to limit the amount of time spent by the robot
> > > in Travis.
> > >
> > > Since meson enables automatically the relevant components, there is not
> > > much gain in testing with extra_packages vs required_packages only.
> > >
> > > For a given arch/compiler/env combination, compilation is first tested
> > > in all jobs that run tests or build the docs or run the ABI checks.
> > > In the same context, for jobs that accumulates running tests, building
> > > the docs etc..., those steps are independent and can be split to save
> > > some cpu on Travis.
> > >
> > > With this, we go down from 21 to 15 jobs.
> > >
> > > Note: this patch requires a flush of the existing caches in Travis.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> >
> > In general, I think the idea with required vs. extra was to have a build
> > that did the minimum required, and one that did all the packages (to
> > allow a minimum vs. full DPDK).
> >
> > At least, that's from
> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
>
> I think the benefit of a minimum build is to have a quick report,
> and easy to setup.

Yes, Travis serves as a first gate when submitting patches.
But since Travis is best effort/free, we can't have a full coverage.


> > Not sure if that's still something anyone cares about.
>
> Given that Travis knows how to satisfy the dependencies,
> and that we must wait for all jobs to finish,
> I don't see any benefit of a minimal setup.

This minimal setup also tests that dpdk dependencies are correct.
If a change makes something rely on libX and libX is in the packages
always installed in Travis, the missing dependency would not get
caught.

But here, this adds too many jobs.

UNH, Intel and other CIs should step in and fill this kind of gap.


--
David Marchand

Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs

[Aaron Conole]

David Marchand <david.marchand@redhat.com> writes:

> On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 19/02/2020 22:39, Aaron Conole:
>> > David Marchand <david.marchand@redhat.com> writes:
>> >
>> > > Let's prune the jobs list to limit the amount of time spent by the robot
>> > > in Travis.
>> > >
>> > > Since meson enables automatically the relevant components, there is not
>> > > much gain in testing with extra_packages vs required_packages only.
>> > >
>> > > For a given arch/compiler/env combination, compilation is first tested
>> > > in all jobs that run tests or build the docs or run the ABI checks.
>> > > In the same context, for jobs that accumulates running tests, building
>> > > the docs etc..., those steps are independent and can be split to save
>> > > some cpu on Travis.
>> > >
>> > > With this, we go down from 21 to 15 jobs.
>> > >
>> > > Note: this patch requires a flush of the existing caches in Travis.
>> > >
>> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
>> > > ---
>> >
>> > In general, I think the idea with required vs. extra was to have a build
>> > that did the minimum required, and one that did all the packages (to
>> > allow a minimum vs. full DPDK).
>> >
>> > At least, that's from
>> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
>>
>> I think the benefit of a minimum build is to have a quick report,
>> and easy to setup.
>
> Yes, Travis serves as a first gate when submitting patches.
> But since Travis is best effort/free, we can't have a full coverage.
>
>
>> > Not sure if that's still something anyone cares about.
>>
>> Given that Travis knows how to satisfy the dependencies,
>> and that we must wait for all jobs to finish,
>> I don't see any benefit of a minimal setup.
>
> This minimal setup also tests that dpdk dependencies are correct.
> If a change makes something rely on libX and libX is in the packages
> always installed in Travis, the missing dependency would not get
> caught.
>
> But here, this adds too many jobs.
>
> UNH, Intel and other CIs should step in and fill this kind of gap.

Okay, makes sense to me.  Are one of these CI providers offering to
cover this?

Also,

Acked-by: Aaron Conole <aconole@redhat.com>

>
> --
> David Marchand

Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs

[David Marchand]

On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote:
>
> David Marchand <david.marchand@redhat.com> writes:
>
> > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >>
> >> 19/02/2020 22:39, Aaron Conole:
> >> > David Marchand <david.marchand@redhat.com> writes:
> >> >
> >> > > Let's prune the jobs list to limit the amount of time spent by the robot
> >> > > in Travis.
> >> > >
> >> > > Since meson enables automatically the relevant components, there is not
> >> > > much gain in testing with extra_packages vs required_packages only.
> >> > >
> >> > > For a given arch/compiler/env combination, compilation is first tested
> >> > > in all jobs that run tests or build the docs or run the ABI checks.
> >> > > In the same context, for jobs that accumulates running tests, building
> >> > > the docs etc..., those steps are independent and can be split to save
> >> > > some cpu on Travis.
> >> > >
> >> > > With this, we go down from 21 to 15 jobs.
> >> > >
> >> > > Note: this patch requires a flush of the existing caches in Travis.
> >> > >
> >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> >> > > ---
> >> >
> >> > In general, I think the idea with required vs. extra was to have a build
> >> > that did the minimum required, and one that did all the packages (to
> >> > allow a minimum vs. full DPDK).
> >> >
> >> > At least, that's from
> >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
> >>
> >> I think the benefit of a minimum build is to have a quick report,
> >> and easy to setup.
> >
> > Yes, Travis serves as a first gate when submitting patches.
> > But since Travis is best effort/free, we can't have a full coverage.
> >
> >
> >> > Not sure if that's still something anyone cares about.
> >>
> >> Given that Travis knows how to satisfy the dependencies,
> >> and that we must wait for all jobs to finish,
> >> I don't see any benefit of a minimal setup.
> >
> > This minimal setup also tests that dpdk dependencies are correct.
> > If a change makes something rely on libX and libX is in the packages
> > always installed in Travis, the missing dependency would not get
> > caught.
> >
> > But here, this adds too many jobs.
> >
> > UNH, Intel and other CIs should step in and fill this kind of gap.
>
> Okay, makes sense to me.  Are one of these CI providers offering to
> cover this?

Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org.

For the CI guys, which packages are installed on the systems/vms that
do compilation tests?
Is it possible to have a summary of the different setups?


Thanks.

--
David Marchand

Re: [dpdk-dev] [dpdk-ci] [PATCH 4/4] ci: reorganise Travis jobs

[Jeremy Plsek]

[-- Attachment #1: Type: text/plain, Size: 3853 bytes --]

On Thu, Feb 20, 2020 at 11:02 AM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Thu, Feb 20, 2020 at 3:35 PM Aaron Conole <aconole@redhat.com> wrote:
> >
> > David Marchand <david.marchand@redhat.com> writes:
> >
> > > On Thu, Feb 20, 2020 at 11:42 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >>
> > >> 19/02/2020 22:39, Aaron Conole:
> > >> > David Marchand <david.marchand@redhat.com> writes:
> > >> >
> > >> > > Let's prune the jobs list to limit the amount of time spent by the robot
> > >> > > in Travis.
> > >> > >
> > >> > > Since meson enables automatically the relevant components, there is not
> > >> > > much gain in testing with extra_packages vs required_packages only.
> > >> > >
> > >> > > For a given arch/compiler/env combination, compilation is first tested
> > >> > > in all jobs that run tests or build the docs or run the ABI checks.
> > >> > > In the same context, for jobs that accumulates running tests, building
> > >> > > the docs etc..., those steps are independent and can be split to save
> > >> > > some cpu on Travis.
> > >> > >
> > >> > > With this, we go down from 21 to 15 jobs.
> > >> > >
> > >> > > Note: this patch requires a flush of the existing caches in Travis.
> > >> > >
> > >> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > >> > > ---
> > >> >
> > >> > In general, I think the idea with required vs. extra was to have a build
> > >> > that did the minimum required, and one that did all the packages (to
> > >> > allow a minimum vs. full DPDK).
> > >> >
> > >> > At least, that's from
> > >> > http://mails.dpdk.org/archives/dev/2019-January/124007.html
> > >>
> > >> I think the benefit of a minimum build is to have a quick report,
> > >> and easy to setup.
> > >
> > > Yes, Travis serves as a first gate when submitting patches.
> > > But since Travis is best effort/free, we can't have a full coverage.
> > >
> > >
> > >> > Not sure if that's still something anyone cares about.
> > >>
> > >> Given that Travis knows how to satisfy the dependencies,
> > >> and that we must wait for all jobs to finish,
> > >> I don't see any benefit of a minimal setup.
> > >
> > > This minimal setup also tests that dpdk dependencies are correct.
> > > If a change makes something rely on libX and libX is in the packages
> > > always installed in Travis, the missing dependency would not get
> > > caught.
> > >
> > > But here, this adds too many jobs.
> > >
> > > UNH, Intel and other CIs should step in and fill this kind of gap.
> >
> > Okay, makes sense to me.  Are one of these CI providers offering to
> > cover this?
>
> Maybe it is already covered, the best is to ask, so sending to ci@dpdk.org.
>
> For the CI guys, which packages are installed on the systems/vms that
> do compilation tests?

For our compile tests, we compile with pretty much all dpdk
dependencies that can easily be installed from the distro's repository
(the dependencies we installed are based on the meson output). The
thought process was to test compiling as much of dpdk as possible
(with constraints mentioned before).

Here are the dependencies we install on some of the setups:

Ubuntu 18.04: make gcc ninja-build pkg-config libnuma-dev
librdmacm-dev librdmacm1 rdma-core libelf-dev libmnl-dev libpcap-dev
libcrypto++-dev libjansson-dev
(meson installed with pip)

CentOS 8 and Fedora 31: diffutils gcc make ninja-build meson
pkg-config numactl-devel librdmacm rdma-core-devel
elfutils-libelf-devel libmnl-devel libpcap-devel jansson-devel

openSuse Leap 15: gcc make ninja pkg-config libnuma-devel librdmacm1
rdma-core-devel libelf-devel libmnl-devel libpcap-devel
libjansson-devel
(meson installed with pip)

> Is it possible to have a summary of the different setups?
>
I've attached an overview/summary of what we do at the UNH-IOL DPDK
Community Lab.
>
> Thanks.

>
> --
> David Marchand
>

[-- Attachment #2: DPDK Lab Summary February 2020.pdf --]
[-- Type: application/pdf, Size: 846111 bytes --]

Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs

[dwilder]

On 2020-02-19 11:41, David Marchand wrote:
> Let's prune the jobs list to limit the amount of time spent by the 
> robot
> in Travis.
> 
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
> 
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
> 
> With this, we go down from 21 to 15 jobs.
> 
> Note: this patch requires a flush of the existing caches in Travis.
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
>  .travis.yml | 51 +++++++++------------------------------------------
>  1 file changed, 9 insertions(+), 42 deletions(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index c174d5f30..b64a81bd0 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -15,15 +15,12 @@ addons:
>      update: true
>      packages: &required_packages
>        - [libnuma-dev, python3-setuptools, python3-wheel, python3-pip,
> ninja-build]
> +      - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
> 
>  _aarch64_packages: &aarch64_packages
>    - *required_packages
>    - [gcc-aarch64-linux-gnu, libc6-dev-arm64-cross,
> pkg-config-aarch64-linux-gnu]
> 
> -_extra_packages: &extra_packages
> -  - *required_packages
> -  - [libbsd-dev, libpcap-dev, libibverbs-dev, libcrypto++-dev,
> libfdt-dev, libjansson4]
> -
>  _libabigail_build_packages: &libabigail_build_packages
>    - [autoconf, automake, libtool, pkg-config, libxml2-dev, libdw-dev]
> 
> @@ -43,66 +40,39 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="shared"
> -    arch: amd64
> -    compiler: gcc
>    - env: DEF_LIB="shared" RUN_TESTS=1
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="static" EXTRA_PACKAGES=1
> -    arch: amd64
> -    compiler: gcc
> -    addons:
> -      apt:
> -        packages:
> -          - *extra_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=1
>      arch: amd64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
> -          - *extra_packages
> +          - *required_packages
>            - *doc_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> +  - env: DEF_LIB="shared" ABI_CHECKS=1
>      arch: amd64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
> -          - *extra_packages
> +          - *required_packages
>            - *libabigail_build_packages
>    # x86_64 clang jobs
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="shared"
> -    arch: amd64
> -    compiler: clang
>    - env: DEF_LIB="shared" RUN_TESTS=1
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="static" EXTRA_PACKAGES=1
> -    arch: amd64
> -    compiler: clang
> -    addons:
> -      apt:
> -        packages:
> -          - *extra_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1
> -    arch: amd64
> -    compiler: clang
> -    addons:
> -      apt:
> -        packages:
> -          - *extra_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 RUN_TESTS=1 BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=1
>      arch: amd64
>      compiler: clang
>      addons:
>        apt:
>          packages:
> -          - *extra_packages
> +          - *required_packages
>            - *doc_packages
>    # x86_64 cross-compiling 32-bits jobs
>    - env: DEF_LIB="static" BUILD_32BIT=1
> @@ -131,9 +101,6 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: arm64
>      compiler: gcc
> -  - env: DEF_LIB="shared"
> -    arch: arm64
> -    compiler: gcc
>    - env: DEF_LIB="shared" BUILD_DOCS=1
>      arch: arm64
>      compiler: gcc
> @@ -142,13 +109,13 @@ jobs:
>          packages:
>            - *required_packages
>            - *doc_packages
> -  - env: DEF_LIB="shared" EXTRA_PACKAGES=1 ABI_CHECKS=1
> +  - env: DEF_LIB="shared" ABI_CHECKS=1
>      arch: arm64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
> -          - *extra_packages
> +          - *required_packages
>            - *libabigail_build_packages
>    # aarch64 clang jobs
>    - env: DEF_LIB="static"

Much easier to follow,  I had no issues adding ppc64le jobs.
All my travis jobs ran fine.

LGTM  (the series)

Re: [dpdk-dev] [PATCH 4/4] ci: reorganise Travis jobs

[Thomas Monjalon]

19/02/2020 20:41, David Marchand:
> Let's prune the jobs list to limit the amount of time spent by the robot
> in Travis.
> 
> Since meson enables automatically the relevant components, there is not
> much gain in testing with extra_packages vs required_packages only.
> 
> For a given arch/compiler/env combination, compilation is first tested
> in all jobs that run tests or build the docs or run the ABI checks.
> In the same context, for jobs that accumulates running tests, building
> the docs etc..., those steps are independent and can be split to save
> some cpu on Travis.
> 
> With this, we go down from 21 to 15 jobs.
> 
> Note: this patch requires a flush of the existing caches in Travis.
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>

Acked-by: Thomas Monjalon <thomas@monjalon.net>

A small step for DPDK but a big save for the earth ;)

Re: [dpdk-dev] [PATCH 0/4] Reorganise Travis jobs

[David Marchand]

On Wed, Feb 19, 2020 at 8:41 PM David Marchand
<david.marchand@redhat.com> wrote:
> This series tries to make the travis configuration a bit easier to read,
> then drops jobs that are not essential or functional duplicates in the
> hope we can give some oxygen to the robot who seems to lag quite a bit
> and can't catch up.

Travis cache for master flushed.
Series applied.


--
David Marchand