DPDK patches and discussions
 help / color / mirror / Atom feed
From: Aaron Conole <aconole@redhat.com>
To: David Marchand <david.marchand@redhat.com>
Cc: dev@dpdk.org, Michael Santana <maicolgabriel@hotmail.com>,
	Thomas Monjalon <thomas@monjalon.net>
Subject: Re: [dpdk-dev] [PATCH v2 1/2] ci: hook to GitHub Actions
Date: Mon, 14 Dec 2020 09:12:31 -0500
Message-ID: <f7tmtygiii8.fsf@dhcp-25.97.bos.redhat.com> (raw)
In-Reply-To: <20201204173622.12053-1-david.marchand@redhat.com> (David Marchand's message of "Fri, 4 Dec 2020 18:36:21 +0100")

David Marchand <david.marchand@redhat.com> writes:

> With the recent changes in terms of free access to the Travis CI, let's
> offer an alternative with GitHub Actions.
> Running jobs on ARM is not supported unless using external runners, so
> this commit only adds builds for x86_64 and cross compiling for i386 and
> aarch64.
>
> Differences with the Travis CI integration:
> - Error logs are not dumped to the console when something goes wrong.
>   Instead, they are gathered in a "catch-all" step and attached as
>   artifacts.
> - A cache entry is stored once and for all, but if no cache is found you
>   can inherit from the default branch cache. The cache is 5GB large, for
>   the whole git repository.
> - The maximum retention of logs and artifacts is 3 months.
> - /home/runner is world writable, so a workaround has been added for
>   starting dpdk processes.
> - Ilya, working on OVS GHA support, noticed that jobs can run with
>   processors that don't have the same capabilities. For DPDK, this
>   impacts the ccache content since everything was built with
>   -march=native so far, and we will end up with binaries that can't run
>   in a later build. The problem has not been seen in Travis CI (?) but
>   it is safer to use a fixed "-Dmachine=default" in any case.

That's because the build machine and test machine are the same, but I
think GHA uses a different model, and will spawn a new environment for
the steps.  I'm not 100% sure, because it's all supposed to be a
black-box.

> - Scheduling jobs is part of the configuration and takes the form of a
>   crontab. A build is scheduled every Monday at 0:00 (UTC) to provide a
>   default ccache for the week (useful for the ovsrobot).
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

Acked-by: Aaron Conole <aconole@redhat.com>

> Changelog since v1:
> - changed shell variables value in CI scripts and Travis configuration
>   (s/=[^\$]*1/=\1true), this makes it easier for GHA,
> - forced compilation as 'default' to avoid random unit tests issues in
>   GHA,
> - scheduled a run per week on Monday at 0:00 UTC,
> - updated the ccache key:
>   - no need to depend on the default-library parameter since this
>     parameter only impacts the linking of dpdk binaries,
>   - the week when the cache is generated is added so that jobs in
>     other branches can benefit from a recent cache (mimicking what we had
>     for the robot in Travis),
> - realigned documentation generation with what is done in Travis:
>   generating the doc in all jobs was a waste of resources,
>
> ---
>  .ci/linux-build.sh          |  17 +++---
>  .github/workflows/build.yml | 100 ++++++++++++++++++++++++++++++++++++
>  .travis.yml                 |  24 ++++-----
>  MAINTAINERS                 |   1 +
>  4 files changed, 123 insertions(+), 19 deletions(-)
>  create mode 100644 .github/workflows/build.yml
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index d079801d78..ee8d07f865 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -12,7 +12,9 @@ on_error() {
>          fi
>      done
>  }
> -trap on_error EXIT
> +# We capture the error logs as artifacts in Github Actions, no need to dump
> +# them via a EXIT handler.
> +[ -n "$GITHUB_WORKFLOW" ] || trap on_error EXIT
>  
>  install_libabigail() {
>      version=$1
> @@ -28,16 +30,16 @@ install_libabigail() {
>      rm ${version}.tar.gz
>  }
>  
> -if [ "$AARCH64" = "1" ]; then
> +if [ "$AARCH64" = "true" ]; then
>      # convert the arch specifier
>      OPTS="$OPTS --cross-file config/arm/arm64_armv8_linux_gcc"
>  fi
>  
> -if [ "$BUILD_DOCS" = "1" ]; then
> +if [ "$BUILD_DOCS" = "true" ]; then
>      OPTS="$OPTS -Denable_docs=true"
>  fi
>  
> -if [ "$BUILD_32BIT" = "1" ]; then
> +if [ "$BUILD_32BIT" = "true" ]; then
>      OPTS="$OPTS -Dc_args=-m32 -Dc_link_args=-m32"
>      export PKG_CONFIG_LIBDIR="/usr/lib32/pkgconfig"
>  fi
> @@ -48,16 +50,17 @@ else
>      OPTS="$OPTS -Dexamples=all"
>  fi
>  
> +OPTS="$OPTS -Dmachine=default"
>  OPTS="$OPTS --default-library=$DEF_LIB"
>  OPTS="$OPTS --buildtype=debugoptimized"
>  meson build --werror $OPTS
>  ninja -C build
>  
> -if [ "$AARCH64" != "1" ]; then
> +if [ "$AARCH64" != "true" ]; then
>      devtools/test-null.sh
>  fi
>  
> -if [ "$ABI_CHECKS" = "1" ]; then
> +if [ "$ABI_CHECKS" = "true" ]; then
>      LIBABIGAIL_VERSION=${LIBABIGAIL_VERSION:-libabigail-1.6}
>  
>      if [ "$(cat libabigail/VERSION 2>/dev/null)" != "$LIBABIGAIL_VERSION" ]; then
> @@ -95,6 +98,6 @@ if [ "$ABI_CHECKS" = "1" ]; then
>      devtools/check-abi.sh reference install ${ABI_CHECKS_WARN_ONLY:-}
>  fi
>  
> -if [ "$RUN_TESTS" = "1" ]; then
> +if [ "$RUN_TESTS" = "true" ]; then
>      sudo meson test -C build --suite fast-tests -t 3
>  fi
> diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
> new file mode 100644
> index 0000000000..bef6e52372
> --- /dev/null
> +++ b/.github/workflows/build.yml
> @@ -0,0 +1,100 @@
> +name: build
> +
> +on:
> +  push:
> +  schedule:
> +    - cron: '0 0 * * 1'
> +
> +defaults:
> +  run:
> +    shell: bash --noprofile --norc -exo pipefail {0}
> +
> +jobs:
> +  build:
> +    name: ${{ join(matrix.config.*, '-') }}
> +    runs-on: ${{ matrix.config.os }}
> +    env:
> +      AARCH64: ${{ matrix.config.cross == 'aarch64' }}
> +      BUILD_32BIT: ${{ matrix.config.cross == 'i386' }}
> +      BUILD_DOCS: ${{ contains(matrix.config.checks, 'doc') }}
> +      CC: ccache ${{ matrix.config.compiler }}
> +      DEF_LIB: ${{ matrix.config.library }}
> +      RUN_TESTS: ${{ contains(matrix.config.checks, 'tests') }}
> +
> +    strategy:
> +      fail-fast: false
> +      matrix:
> +        config:
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: shared
> +            checks: doc+tests
> +          - os: ubuntu-18.04
> +            compiler: clang
> +            library: static
> +          - os: ubuntu-18.04
> +            compiler: clang
> +            library: shared
> +            checks: doc+tests
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +            cross: i386
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: static
> +            cross: aarch64
> +          - os: ubuntu-18.04
> +            compiler: gcc
> +            library: shared
> +            cross: aarch64
> +
> +    steps:
> +    - name: Checkout sources
> +      uses: actions/checkout@v2
> +    - name: Generate cache keys
> +      id: get_ref_keys
> +      run: |
> +        echo -n '::set-output name=ccache::'
> +        echo 'ccache-${{ matrix.config.os }}-${{ matrix.config.compiler }}-${{ matrix.config.cross }}-'$(date -u +%Y-w%W)
> +    - name: Retrieve ccache cache
> +      uses: actions/cache@v2
> +      with:
> +        path: ~/.ccache
> +        key: ${{ steps.get_ref_keys.outputs.ccache }}-${{ github.ref }}
> +        restore-keys: |
> +          ${{ steps.get_ref_keys.outputs.ccache }}-refs/heads/main
> +    - name: Install packages
> +      run: sudo apt install -y ccache libnuma-dev python3-setuptools
> +        python3-wheel python3-pip ninja-build libbsd-dev libpcap-dev
> +        libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
> +    - name: Install i386 cross compiling packages
> +      if: env.BUILD_32BIT == 'true'
> +      run: sudo apt install -y gcc-multilib
> +    - name: Install aarch64 cross compiling packages
> +      if: env.AARCH64 == 'true'
> +      run: sudo apt install -y gcc-aarch64-linux-gnu libc6-dev-arm64-cross
> +        pkg-config-aarch64-linux-gnu
> +    - name: Install doc generation packages
> +      if: env.BUILD_DOCS == 'true'
> +      run: sudo apt install -y doxygen graphviz python3-sphinx
> +        python3-sphinx-rtd-theme
> +    - name: Run setup
> +      run: |
> +        .ci/linux-setup.sh
> +        # Workaround on $HOME permissions as EAL checks them for plugin loading
> +        chmod o-w $HOME
> +    - name: Build and test
> +      run: .ci/linux-build.sh
> +    - name: Upload logs on failure
> +      if: failure()
> +      uses: actions/upload-artifact@v2
> +      with:
> +        name: meson-logs-${{ join(matrix.config.*, '-') }}
> +        path: |
> +          build/meson-logs/testlog.txt
> +          build/.ninja_log
> +          build/meson-logs/meson-log.txt
> diff --git a/.travis.yml b/.travis.yml
> index 5e12db23b5..d655e286c3 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -34,10 +34,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: amd64
>      compiler: gcc
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -49,10 +49,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: amd64
>      compiler: clang
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: amd64
>      compiler: clang
>      addons:
> @@ -61,7 +61,7 @@ jobs:
>            - *required_packages
>            - *doc_packages
>    # x86_64 cross-compiling 32-bits jobs
> -  - env: DEF_LIB="static" BUILD_32BIT=1
> +  - env: DEF_LIB="static" BUILD_32BIT=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -69,14 +69,14 @@ jobs:
>          packages:
>            - *build_32b_packages
>    # x86_64 cross-compiling aarch64 jobs
> -  - env: DEF_LIB="static" AARCH64=1
> +  - env: DEF_LIB="static" AARCH64=true
>      arch: amd64
>      compiler: gcc
>      addons:
>        apt:
>          packages:
>            - *aarch64_packages
> -  - env: DEF_LIB="shared" AARCH64=1
> +  - env: DEF_LIB="shared" AARCH64=true
>      arch: amd64
>      compiler: gcc
>      addons:
> @@ -87,16 +87,16 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: arm64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: arm64
>      compiler: gcc
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      dist: focal
>      arch: arm64-graviton2
>      virt: vm
>      group: edge
>      compiler: gcc
> -  - env: DEF_LIB="shared" BUILD_DOCS=1
> +  - env: DEF_LIB="shared" BUILD_DOCS=true
>      arch: arm64
>      compiler: gcc
>      addons:
> @@ -108,10 +108,10 @@ jobs:
>    - env: DEF_LIB="static"
>      arch: arm64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      arch: arm64
>      compiler: clang
> -  - env: DEF_LIB="shared" RUN_TESTS=1
> +  - env: DEF_LIB="shared" RUN_TESTS=true
>      dist: focal
>      arch: arm64-graviton2
>      virt: vm
> diff --git a/MAINTAINERS b/MAINTAINERS
> index eafe9f8c46..f45c8c1b13 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -109,6 +109,7 @@ Public CI
>  M: Aaron Conole <aconole@redhat.com>
>  M: Michael Santana <maicolgabriel@hotmail.com>
>  F: .travis.yml
> +F: .github/workflows/build.yml
>  F: .ci/
>  
>  ABI Policy & Versioning


  parent reply	other threads:[~2020-12-14 14:12 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-24 21:57 [dpdk-dev] [PATCH] ci: hook to Github Actions David Marchand
2020-11-25 13:44 ` Aaron Conole
2020-11-25 14:31   ` David Marchand
2020-11-26  4:46     ` Honnappa Nagarahalli
2020-11-26  8:06       ` David Marchand
2020-11-26 17:01         ` Honnappa Nagarahalli
2020-12-08 14:08           ` David Marchand
2020-12-04 17:36 ` [dpdk-dev] [PATCH v2 1/2] ci: hook to GitHub Actions David Marchand
2020-12-04 17:36   ` [dpdk-dev] [PATCH v2 2/2] ci: enable v21 ABI checks David Marchand
2020-12-14 14:13     ` Aaron Conole
2020-12-11 20:07   ` [dpdk-dev] [PATCH v2 1/2] ci: hook to GitHub Actions Ferruh Yigit
2020-12-14 10:44     ` Thomas Monjalon
2020-12-14 14:12   ` Aaron Conole [this message]
2020-12-14 16:17   ` David Marchand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f7tmtygiii8.fsf@dhcp-25.97.bos.redhat.com \
    --to=aconole@redhat.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=maicolgabriel@hotmail.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git