DPDK patches and discussions
 help / color / mirror / Atom feed
* [PATCH] ci: add Fedora 35 container in GHA
@ 2021-11-16 12:31 David Marchand
  2021-11-16 14:44 ` Aaron Conole
                   ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: David Marchand @ 2021-11-16 12:31 UTC (permalink / raw)
  To: dev; +Cc: thomas, ferruh.yigit, Aaron Conole, Michael Santana

Build DPDK with Fedora 35 containers.
Differences with the Ubuntu GHA vm images:
- tasks are run as root in containers, no need for sudo,
- compiler must be explicitly installed: clang is not in container images,
- GHA artifacts can't contain a ':' in their name, and must be filtered,

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .ci/linux-build.sh          |  3 ++
 .ci/linux-setup.sh          |  3 ++
 .github/workflows/build.yml | 67 +++++++++++++++++++++++++++++++++++--
 3 files changed, 71 insertions(+), 2 deletions(-)

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index ef0bd099be..e34a74ab64 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -1,5 +1,8 @@
 #!/bin/sh -xe
 
+# Builds are run as root in containers, no need for sudo
+[ "$(id -u)" != '0' ] || alias sudo=
+
 on_error() {
     if [ $? = 0 ]; then
         exit
diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh
index ca35b4b38d..f5ef6b1b98 100755
--- a/.ci/linux-setup.sh
+++ b/.ci/linux-setup.sh
@@ -1,5 +1,8 @@
 #!/bin/sh -xe
 
+# Builds are run as root in containers, no need for sudo
+[ "$(id -u)" != '0' ] || alias sudo=
+
 # need to install as 'root' since some of the unit tests won't run without it
 sudo python3 -m pip install --upgrade 'meson==0.49.2'
 
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 4151cafee7..8429ffb721 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -10,7 +10,7 @@ defaults:
     shell: bash --noprofile --norc -exo pipefail {0}
 
 jobs:
-  build:
+  ubuntu-vms:
     name: ${{ join(matrix.config.*, '-') }}
     runs-on: ${{ matrix.config.os }}
     env:
@@ -140,7 +140,70 @@ jobs:
       with:
         name: meson-logs-${{ join(matrix.config.*, '-') }}
         path: |
+          build/.ninja_log
+          build/gdb.log
+          build/meson-logs/meson-log.txt
           build/meson-logs/testlog.txt
+
+  rpm-containers:
+    name: ${{ join(matrix.config.*, '-') }}
+    runs-on: ubuntu-latest
+    container:
+      image: ${{ matrix.config.image }}
+    env:
+      CC: ccache ${{ matrix.config.compiler }}
+      DEF_LIB: ${{ matrix.config.library }}
+
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - image: fedora:35
+            compiler: gcc
+            library: static
+          - image: fedora:35
+            compiler: gcc
+            library: shared
+          - image: fedora:35
+            compiler: clang
+            library: static
+          - image: fedora:35
+            compiler: clang
+            library: shared
+
+    steps:
+    - name: Checkout sources
+      uses: actions/checkout@v2
+    - name: Generate various keys
+      id: get_keys
+      run: |
+        echo -n '::set-output name=ccache::'
+        echo 'ccache-${{ matrix.config.image }}-${{ matrix.config.compiler }}-'$(date -u +%Y-w%W)
+        echo -n '::set-output name=logs::'
+        echo 'meson-logs-${{ join(matrix.config.*, '-') }}' | tr -d ':'
+    - name: Retrieve ccache cache
+      uses: actions/cache@v2
+      with:
+        path: ~/.ccache
+        key: ${{ steps.get_keys.outputs.ccache }}-${{ github.ref }}
+        restore-keys: |
+          ${{ steps.get_keys.outputs.ccache }}-refs/heads/main
+    - name: Update
+      run: dnf update -y || true
+    - name: Install packages
+      run: dnf install -y ccache numactl-devel python3-setuptools
+        python3-wheel python3-pip python3-pyelftools ninja-build
+        libatomic libbpf-devel libfdt-devel libpcap-devel openssl-devel
+        rdma-core-devel zlib-devel ${{ matrix.config.compiler }}
+    - name: Run setup
+      run: .ci/linux-setup.sh
+    - name: Build
+      run: .ci/linux-build.sh
+    - name: Upload logs on failure
+      if: failure()
+      uses: actions/upload-artifact@v2
+      with:
+        name: ${{ steps.get_keys.outputs.logs }}
+        path: |
           build/.ninja_log
           build/meson-logs/meson-log.txt
-          build/gdb.log
-- 
2.23.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] ci: add Fedora 35 container in GHA
  2021-11-16 12:31 [PATCH] ci: add Fedora 35 container in GHA David Marchand
@ 2021-11-16 14:44 ` Aaron Conole
  2021-11-17 17:29   ` David Marchand
  2022-04-01 18:02 ` [PATCH v2] " David Marchand
  2022-04-04 14:24 ` [PATCH v3] " David Marchand
  2 siblings, 1 reply; 9+ messages in thread
From: Aaron Conole @ 2021-11-16 14:44 UTC (permalink / raw)
  To: David Marchand; +Cc: dev, thomas, ferruh.yigit, Michael Santana

David Marchand <david.marchand@redhat.com> writes:

> Build DPDK with Fedora 35 containers.
> Differences with the Ubuntu GHA vm images:
> - tasks are run as root in containers, no need for sudo,
> - compiler must be explicitly installed: clang is not in container images,
> - GHA artifacts can't contain a ':' in their name, and must be filtered,
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---

This is awesome!

Acked-by: Aaron Conole <aconole@redhat.com>


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] ci: add Fedora 35 container in GHA
  2021-11-16 14:44 ` Aaron Conole
@ 2021-11-17 17:29   ` David Marchand
  2021-11-18 14:17     ` Aaron Conole
  0 siblings, 1 reply; 9+ messages in thread
From: David Marchand @ 2021-11-17 17:29 UTC (permalink / raw)
  To: Aaron Conole; +Cc: dev, Thomas Monjalon, Yigit, Ferruh, Michael Santana

On Tue, Nov 16, 2021 at 3:44 PM Aaron Conole <aconole@redhat.com> wrote:
> > Build DPDK with Fedora 35 containers.
> > Differences with the Ubuntu GHA vm images:
> > - tasks are run as root in containers, no need for sudo,
> > - compiler must be explicitly installed: clang is not in container images,
> > - GHA artifacts can't contain a ':' in their name, and must be filtered,
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Aaron Conole <aconole@redhat.com>

Unfortunately, while testing it a bit more (having it in my tree), I
got some random failures when pulling container images.
I suspect docker limits image pulls... but I am not sure yet.
I'll have to investigate this.


--
David Marchand


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] ci: add Fedora 35 container in GHA
  2021-11-17 17:29   ` David Marchand
@ 2021-11-18 14:17     ` Aaron Conole
  0 siblings, 0 replies; 9+ messages in thread
From: Aaron Conole @ 2021-11-18 14:17 UTC (permalink / raw)
  To: David Marchand; +Cc: dev, Thomas Monjalon, Yigit, Ferruh, Michael Santana

David Marchand <david.marchand@redhat.com> writes:

> On Tue, Nov 16, 2021 at 3:44 PM Aaron Conole <aconole@redhat.com> wrote:
>> > Build DPDK with Fedora 35 containers.
>> > Differences with the Ubuntu GHA vm images:
>> > - tasks are run as root in containers, no need for sudo,
>> > - compiler must be explicitly installed: clang is not in container images,
>> > - GHA artifacts can't contain a ':' in their name, and must be filtered,
>> >
>> > Signed-off-by: David Marchand <david.marchand@redhat.com>
>> Acked-by: Aaron Conole <aconole@redhat.com>
>
> Unfortunately, while testing it a bit more (having it in my tree), I
> got some random failures when pulling container images.
> I suspect docker limits image pulls... but I am not sure yet.
> I'll have to investigate this.

I do see that there were warnings even with the github robot's run - but
those builds eventually went through.  I guess if it isn't stable, it
makes sense to not deploy.

One alternative is to create a separate "workflow" for github action
that can be called "fedora" and that will be recorded differently by the
robot and then if there's a failure at least it will be a different
"test."  This probably isn't preferable, though - since it creates
another test in the UI.

>
> --
> David Marchand


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2] ci: add Fedora 35 container in GHA
  2021-11-16 12:31 [PATCH] ci: add Fedora 35 container in GHA David Marchand
  2021-11-16 14:44 ` Aaron Conole
@ 2022-04-01 18:02 ` David Marchand
  2022-04-02  8:01   ` David Marchand
  2022-04-04 14:24 ` [PATCH v3] " David Marchand
  2 siblings, 1 reply; 9+ messages in thread
From: David Marchand @ 2022-04-01 18:02 UTC (permalink / raw)
  To: dev; +Cc: Aaron Conole, Michael Santana

Build DPDK with Fedora 35 containers.

GHA container support does not allow caching images and docker hub
seems to limit image pulls.
On the other hand, the Fedora project hub does not seem to limit them,
so prefer this hub.
Nevertheless, let's try to be good citizens and cache (once a day) a
prepared image for subsequent builds.

Differences with the Ubuntu GHA vm images:
- tasks are run as root in containers, no need for sudo,
- compiler must be explicitly installed,
- GHA artifacts can't contain a ':' in their name, and must be filtered,

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since v1:
- switched to Fedora project hub,
- cached container image,

---
 .ci/linux-build.sh          |   3 +
 .ci/linux-setup.sh          |   3 +
 .github/workflows/build.yml | 106 +++++++++++++++++++++++++++++++++++-
 3 files changed, 110 insertions(+), 2 deletions(-)

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index 69940dec8b..774a1441bf 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -1,5 +1,8 @@
 #!/bin/sh -xe
 
+# Builds are run as root in containers, no need for sudo
+[ "$(id -u)" != '0' ] || alias sudo=
+
 on_error() {
     if [ $? = 0 ]; then
         exit
diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh
index ca35b4b38d..f5ef6b1b98 100755
--- a/.ci/linux-setup.sh
+++ b/.ci/linux-setup.sh
@@ -1,5 +1,8 @@
 #!/bin/sh -xe
 
+# Builds are run as root in containers, no need for sudo
+[ "$(id -u)" != '0' ] || alias sudo=
+
 # need to install as 'root' since some of the unit tests won't run without it
 sudo python3 -m pip install --upgrade 'meson==0.49.2'
 
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 02819aa5de..b01868ad76 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -10,7 +10,7 @@ defaults:
     shell: bash --noprofile --norc -exo pipefail {0}
 
 jobs:
-  build:
+  ubuntu-vms:
     name: ${{ join(matrix.config.*, '-') }}
     runs-on: ${{ matrix.config.os }}
     env:
@@ -145,7 +145,109 @@ jobs:
       with:
         name: meson-logs-${{ join(matrix.config.*, '-') }}
         path: |
+          build/.ninja_log
+          build/gdb.log
+          build/meson-logs/meson-log.txt
           build/meson-logs/testlog.txt
+
+  rpm-containers:
+    name: ${{ join(matrix.config.*, '-') }}
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - image: fedora:35
+            compiler: gcc
+            library: static
+          - image: fedora:35
+            compiler: gcc
+            library: shared
+          - image: fedora:35
+            compiler: clang
+            library: static
+          - image: fedora:35
+            compiler: clang
+            library: shared
+
+    steps:
+    - name: Checkout sources
+      uses: actions/checkout@v2
+    - name: Generate various keys
+      id: get_keys
+      run: |
+        echo -n '::set-output name=ccache::'
+        echo 'ccache-${{ matrix.config.image }}-${{ matrix.config.compiler }}-'$(date -u +%Y-w%W)
+        echo -n '::set-output name=image::'
+        echo 'image-${{ matrix.config.image }}-'$(date -u +%Y-%m-%d)
+        echo -n '::set-output name=logs::'
+        echo 'meson-logs-${{ join(matrix.config.*, '-') }}' | tr -d ':'
+    - name: Retrieve ccache cache
+      uses: actions/cache@v2
+      with:
+        path: ~/.ccache
+        key: ${{ steps.get_keys.outputs.ccache }}-${{ github.ref }}
+        restore-keys: |
+          ${{ steps.get_keys.outputs.ccache }}-refs/heads/main
+    - name: Retrieve image cache
+      id: image_cache
+      uses: actions/cache@v2
+      with:
+        path: ~/.image
+        key: ${{ steps.get_keys.outputs.image }}
+    - name: Prepare working directory
+      run: |
+        mkdir -p ~/.ccache
+        mkdir -p ~/.image
+        > ~/env
+        echo CC=ccache ${{ matrix.config.compiler }} >> ~/env
+        echo DEF_LIB=${{ matrix.config.library }} >> ~/env
+    - name: Pull and prepare a fresh image
+      if: steps.image_cache.outputs.cache-hit != 'true'
+      run: |
+        docker pull registry.fedoraproject.org/${{ matrix.config.image }}
+        docker run \
+          --mount type=bind,src=$HOME/.ccache,dst=/root/.ccache \
+          --mount type=bind,src=$(pwd),dst=/root/dpdk \
+          --env-file ~/env -w /root/dpdk -d -i --name dpdk \
+          registry.fedoraproject.org/${{ matrix.config.image }} \
+          bash -li
+    - name: Load the cached image
+      if: steps.image_cache.outputs.cache-hit == 'true'
+      run: |
+        docker load -i ~/.image/${{ matrix.config.image }}.tar
+        docker run \
+          --mount type=bind,src=$HOME/.ccache,dst=/root/.ccache \
+          --mount type=bind,src=$(pwd),dst=/root/dpdk \
+          --env-file ~/env -w /root/dpdk -d -i --name dpdk \
+          local:dpdk \
+          bash -li
+    - name: Update
+      run: docker exec -i dpdk dnf update -y || true
+    - name: Install packages
+      run: docker exec -i dpdk dnf install -y ccache numactl-devel
+        python3-setuptools python3-wheel python3-pip python3-pyelftools
+        ninja-build libatomic libbpf-devel libfdt-devel libpcap-devel
+        openssl-devel rdma-core-devel zlib-devel
+    - name: Save image in cache
+      if: steps.image_cache.outputs.cache-hit != 'true'
+      run: |
+        docker commit dpdk local:dpdk
+        docker save -o ~/.image/${{ matrix.config.image }}.tar local:dpdk
+    - name: Install compiler
+      run: docker exec -i dpdk dnf install -y ${{ matrix.config.compiler }}
+    - name: Run setup
+      run: docker exec -i dpdk .ci/linux-setup.sh
+    - name: Build
+      run: docker exec -i dpdk .ci/linux-build.sh
+    - name: Stop image
+      run: docker kill dpdk
+    - name: Upload logs on failure
+      if: failure()
+      uses: actions/upload-artifact@v2
+      with:
+        name: ${{ steps.get_keys.outputs.logs }}
+        path: |
           build/.ninja_log
           build/meson-logs/meson-log.txt
-          build/gdb.log
-- 
2.23.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2] ci: add Fedora 35 container in GHA
  2022-04-01 18:02 ` [PATCH v2] " David Marchand
@ 2022-04-02  8:01   ` David Marchand
  0 siblings, 0 replies; 9+ messages in thread
From: David Marchand @ 2022-04-02  8:01 UTC (permalink / raw)
  To: dev; +Cc: Aaron Conole, Michael Santana

On Fri, Apr 1, 2022 at 8:03 PM David Marchand <david.marchand@redhat.com> wrote:
>
> Build DPDK with Fedora 35 containers.
>
> GHA container support does not allow caching images and docker hub
> seems to limit image pulls.
> On the other hand, the Fedora project hub does not seem to limit them,
> so prefer this hub.
> Nevertheless, let's try to be good citizens and cache (once a day) a
> prepared image for subsequent builds.

We could move to a pipeline approach, with a first stage job preparing
the image and pushing to the cache, and the second stage pulling the
image from the cache and starting compilations.
Food for thoughts...

>
> Differences with the Ubuntu GHA vm images:
> - tasks are run as root in containers, no need for sudo,
> - compiler must be explicitly installed,
> - GHA artifacts can't contain a ':' in their name, and must be filtered,
>


-- 
David Marchand


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v3] ci: add Fedora 35 container in GHA
  2021-11-16 12:31 [PATCH] ci: add Fedora 35 container in GHA David Marchand
  2021-11-16 14:44 ` Aaron Conole
  2022-04-01 18:02 ` [PATCH v2] " David Marchand
@ 2022-04-04 14:24 ` David Marchand
  2022-04-04 16:06   ` Aaron Conole
  2 siblings, 1 reply; 9+ messages in thread
From: David Marchand @ 2022-04-04 14:24 UTC (permalink / raw)
  To: dev; +Cc: rjarry, Aaron Conole, Michael Santana

Build DPDK with Fedora 35 containers.

GHA container support does not allow caching images and docker hub
seems to limit image pulls.
On the other hand, the Fedora project hub does not seem to limit them,
so prefer this hub.
Nevertheless, let's try to be good citizens and cache (once a day) a
prepared image for subsequent builds.
This preparation is done in a first prepare-container-images job.
The rpm-container-builds job then depends on it with a 'needs:' tag.

Differences with builds in Ubuntu GHA vm images:
- tasks are run as root in containers, no need for sudo,
- compiler must be explicitly installed,
- GHA artifacts can't contain a ':' in their name, and must be filtered,
- environment variables are not inherited and must be passed explicitly,

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since v2:
- used a two stage pipeline so that container image generation
  is done only once,
- changed image name local:dpdk to dpdk-local (closer to conventions),

Changes since v1:
- switched to Fedora project hub,
- cached container image,

---
 .ci/linux-build.sh          |   3 +
 .ci/linux-setup.sh          |   3 +
 .github/workflows/build.yml | 142 +++++++++++++++++++++++++++++++++++-
 3 files changed, 146 insertions(+), 2 deletions(-)

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index 69940dec8b..774a1441bf 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -1,5 +1,8 @@
 #!/bin/sh -xe
 
+# Builds are run as root in containers, no need for sudo
+[ "$(id -u)" != '0' ] || alias sudo=
+
 on_error() {
     if [ $? = 0 ]; then
         exit
diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh
index ca35b4b38d..f5ef6b1b98 100755
--- a/.ci/linux-setup.sh
+++ b/.ci/linux-setup.sh
@@ -1,5 +1,8 @@
 #!/bin/sh -xe
 
+# Builds are run as root in containers, no need for sudo
+[ "$(id -u)" != '0' ] || alias sudo=
+
 # need to install as 'root' since some of the unit tests won't run without it
 sudo python3 -m pip install --upgrade 'meson==0.49.2'
 
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 02819aa5de..22daaabb91 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -10,7 +10,7 @@ defaults:
     shell: bash --noprofile --norc -exo pipefail {0}
 
 jobs:
-  build:
+  ubuntu-vm-builds:
     name: ${{ join(matrix.config.*, '-') }}
     runs-on: ${{ matrix.config.os }}
     env:
@@ -145,7 +145,145 @@ jobs:
       with:
         name: meson-logs-${{ join(matrix.config.*, '-') }}
         path: |
+          build/.ninja_log
+          build/gdb.log
+          build/meson-logs/meson-log.txt
           build/meson-logs/testlog.txt
+
+  prepare-container-images:
+    name: ${{ join(matrix.config.*, '-') }}
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - image: fedora:35
+
+    steps:
+    - name: Generate various keys
+      id: get_keys
+      run: |
+        echo -n '::set-output name=image::'
+        echo 'image-${{ matrix.config.image }}-'$(date -u +%Y-%m-%d)
+    - name: Retrieve image cache
+      id: image_cache
+      uses: actions/cache@v2
+      with:
+        path: ~/.image
+        key: ${{ steps.get_keys.outputs.image }}
+    - name: Pull and prepare a fresh image
+      if: steps.image_cache.outputs.cache-hit != 'true'
+      run: |
+        docker pull registry.fedoraproject.org/${{ matrix.config.image }}
+        docker run -d -i --rm --name dpdk \
+          registry.fedoraproject.org/${{ matrix.config.image }} \
+          bash -li
+    - name: Update
+      if: steps.image_cache.outputs.cache-hit != 'true'
+      run: docker exec -i dpdk dnf update -y
+    - name: Install packages
+      if: steps.image_cache.outputs.cache-hit != 'true'
+      run: docker exec -i dpdk dnf install -y ccache numactl-devel
+        python3-setuptools python3-wheel python3-pip python3-pyelftools
+        ninja-build libatomic libbpf-devel libfdt-devel libpcap-devel
+        openssl-devel rdma-core-devel zlib-devel
+    - name: Save image in cache
+      if: steps.image_cache.outputs.cache-hit != 'true'
+      run: |
+        docker commit dpdk dpdk-local
+        mkdir -p ~/.image
+        docker save -o ~/.image/${{ matrix.config.image }}.tar dpdk-local
+    - name: Stop image
+      if: steps.image_cache.outputs.cache-hit != 'true'
+      run: docker kill dpdk
+
+  rpm-container-builds:
+    needs: prepare-container-images
+    name: ${{ join(matrix.config.*, '-') }}
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - image: fedora:35
+            compiler: gcc
+            library: static
+          - image: fedora:35
+            compiler: gcc
+            library: shared
+          - image: fedora:35
+            compiler: clang
+            library: static
+          - image: fedora:35
+            compiler: clang
+            library: shared
+
+    steps:
+    - name: Checkout sources
+      uses: actions/checkout@v2
+    - name: Generate various keys
+      id: get_keys
+      run: |
+        echo -n '::set-output name=ccache::'
+        echo 'ccache-${{ matrix.config.image }}-${{ matrix.config.compiler }}-'$(date -u +%Y-w%W)
+        echo -n '::set-output name=image::'
+        echo 'image-${{ matrix.config.image }}-'$(date -u +%Y-%m-%d)
+        echo -n '::set-output name=logs::'
+        echo 'meson-logs-${{ join(matrix.config.*, '-') }}' | tr -d ':'
+    - name: Retrieve image cache
+      id: image_cache
+      uses: actions/cache@v2
+      with:
+        path: ~/.image
+        key: ${{ steps.get_keys.outputs.image }}
+    - name: Fail if no image (not supposed to happen)
+      if: steps.image_cache.outputs.cache-hit != 'true'
+      run: |
+        echo 'Image ${{ matrix.config.image }} is not cached.'
+        false
+    - name: Retrieve ccache cache
+      uses: actions/cache@v2
+      with:
+        path: ~/.ccache
+        key: ${{ steps.get_keys.outputs.ccache }}-${{ github.ref }}
+        restore-keys: |
+          ${{ steps.get_keys.outputs.ccache }}-refs/heads/main
+    - name: Prepare working directory
+      run: |
+        mkdir -p ~/.ccache
+        > ~/env
+        echo CC=ccache ${{ matrix.config.compiler }} >> ~/env
+        echo DEF_LIB=${{ matrix.config.library }} >> ~/env
+    - name: Load the cached image
+      run: |
+        docker load -i ~/.image/${{ matrix.config.image }}.tar
+        docker run -d -i --rm --name dpdk \
+          --mount type=bind,src=$HOME/.ccache,dst=/root/.ccache \
+          --mount type=bind,src=$(pwd),dst=/root/dpdk \
+          --env-file ~/env \
+          -w /root/dpdk \
+          dpdk-local
+          bash -li
+    - name: Update
+      run: docker exec -i dpdk dnf update -y || true
+    - name: Install packages
+      run: docker exec -i dpdk dnf install -y ccache numactl-devel
+        python3-setuptools python3-wheel python3-pip python3-pyelftools
+        ninja-build libatomic libbpf-devel libfdt-devel libpcap-devel
+        openssl-devel rdma-core-devel zlib-devel ${{ matrix.config.compiler }}
+    - name: Run setup
+      run: docker exec -i dpdk .ci/linux-setup.sh
+    - name: Build
+      run: docker exec -i dpdk .ci/linux-build.sh
+    - name: Stop image
+      run: docker kill dpdk
+    - name: Upload logs on failure
+      if: failure()
+      uses: actions/upload-artifact@v2
+      with:
+        name: ${{ steps.get_keys.outputs.logs }}
+        path: |
           build/.ninja_log
           build/meson-logs/meson-log.txt
-          build/gdb.log
-- 
2.23.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v3] ci: add Fedora 35 container in GHA
  2022-04-04 14:24 ` [PATCH v3] " David Marchand
@ 2022-04-04 16:06   ` Aaron Conole
  2022-04-06 18:58     ` David Marchand
  0 siblings, 1 reply; 9+ messages in thread
From: Aaron Conole @ 2022-04-04 16:06 UTC (permalink / raw)
  To: David Marchand; +Cc: dev, rjarry, Michael Santana

David Marchand <david.marchand@redhat.com> writes:

> Build DPDK with Fedora 35 containers.
>
> GHA container support does not allow caching images and docker hub
> seems to limit image pulls.
> On the other hand, the Fedora project hub does not seem to limit them,
> so prefer this hub.
> Nevertheless, let's try to be good citizens and cache (once a day) a
> prepared image for subsequent builds.
> This preparation is done in a first prepare-container-images job.
> The rpm-container-builds job then depends on it with a 'needs:' tag.
>
> Differences with builds in Ubuntu GHA vm images:
> - tasks are run as root in containers, no need for sudo,
> - compiler must be explicitly installed,
> - GHA artifacts can't contain a ':' in their name, and must be filtered,
> - environment variables are not inherited and must be passed explicitly,
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changes since v2:
> - used a two stage pipeline so that container image generation
>   is done only once,
> - changed image name local:dpdk to dpdk-local (closer to conventions),
>
> Changes since v1:
> - switched to Fedora project hub,
> - cached container image,
>
> ---

This is fantastic!

Acked-by: Aaron Conole <aconole@redhat.com>


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v3] ci: add Fedora 35 container in GHA
  2022-04-04 16:06   ` Aaron Conole
@ 2022-04-06 18:58     ` David Marchand
  0 siblings, 0 replies; 9+ messages in thread
From: David Marchand @ 2022-04-06 18:58 UTC (permalink / raw)
  To: David Marchand; +Cc: dev, Aaron Conole, Robin Jarry, Michael Santana

On Mon, Apr 4, 2022 at 6:06 PM Aaron Conole <aconole@redhat.com> wrote:
> David Marchand <david.marchand@redhat.com> writes:
>
> > Build DPDK with Fedora 35 containers.
> >
> > GHA container support does not allow caching images and docker hub
> > seems to limit image pulls.
> > On the other hand, the Fedora project hub does not seem to limit them,
> > so prefer this hub.
> > Nevertheless, let's try to be good citizens and cache (once a day) a
> > prepared image for subsequent builds.
> > This preparation is done in a first prepare-container-images job.
> > The rpm-container-builds job then depends on it with a 'needs:' tag.
> >
> > Differences with builds in Ubuntu GHA vm images:
> > - tasks are run as root in containers, no need for sudo,
> > - compiler must be explicitly installed,
> > - GHA artifacts can't contain a ':' in their name, and must be filtered,
> > - environment variables are not inherited and must be passed explicitly,
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Aaron Conole <aconole@redhat.com>

Applied, thanks.


-- 
David Marchand


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2022-04-06 18:59 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-16 12:31 [PATCH] ci: add Fedora 35 container in GHA David Marchand
2021-11-16 14:44 ` Aaron Conole
2021-11-17 17:29   ` David Marchand
2021-11-18 14:17     ` Aaron Conole
2022-04-01 18:02 ` [PATCH v2] " David Marchand
2022-04-02  8:01   ` David Marchand
2022-04-04 14:24 ` [PATCH v3] " David Marchand
2022-04-04 16:06   ` Aaron Conole
2022-04-06 18:58     ` David Marchand

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).