patches for DPDK stable branches
 help / color / mirror / Atom feed
* [dpdk-stable] [PATCH v1 1/1] ci: enable DPDK GHA for arm64 with self-hosted runners
       [not found] <20211013080341.29561-1-serena.he@arm.com>
@ 2021-10-13  8:03 ` Serena He
  2021-10-13 11:32   ` [dpdk-stable] [dpdk-dev] " Michael Santana
  0 siblings, 1 reply; 3+ messages in thread
From: Serena He @ 2021-10-13  8:03 UTC (permalink / raw)
  To: aconole, maicolgabriel, david.marchand, thomas
  Cc: dev, nd, honnappa.nagarahalli, ruifeng.wang, Dean.Arnold,
	Serena He, stable

CI jobs are triggered only for repos installed with given GHApp and runners

Cc: stable@dpdk.org

Signed-off-by: Serena He <serena.he@arm.com>

---
 .github/workflows/build-arm64.yml | 118 ++++++++++++++++++++++++++++++
 1 file changed, 118 insertions(+)
 create mode 100644 .github/workflows/build-arm64.yml

diff --git a/.github/workflows/build-arm64.yml b/.github/workflows/build-arm64.yml
new file mode 100644
index 0000000000..570563f7c8
--- /dev/null
+++ b/.github/workflows/build-arm64.yml
@@ -0,0 +1,118 @@
+name: build-arm64
+
+on:
+  push:
+  schedule:
+    - cron: '0 0 * * 1'
+
+defaults:
+  run:
+    shell: bash --noprofile --norc -exo pipefail {0}
+
+jobs:
+  build:
+    # Here, runners for arm64 are accessed by installed GitHub APP, thus will not be available by fork.
+    # you can change the following 'if' and 'runs-on' if you have your own runners installed.
+    # or request to get your repo on the whitelist to use GitHub APP and delete this 'if'.
+    if: ${{ github.repository == 'DPDK/dpdk' || github.repository == 'ovsrobot/dpdk' }}
+    name: ${{ join(matrix.config.*, '-') }}
+    runs-on: ${{ matrix.config.os }}
+    env:
+      ABI_CHECKS: ${{ contains(matrix.config.checks, 'abi') }}
+      BUILD_DOCS: ${{ contains(matrix.config.checks, 'doc') }}
+      CL: ${{ matrix.config.compiler == 'clang' }}
+      CC: ccache ${{ matrix.config.compiler }}
+      DEF_LIB: ${{ matrix.config.library }}
+      LIBABIGAIL_VERSION: libabigail-1.8
+      REF_GIT_TAG: none
+
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - os: [self-hosted,arm-ubuntu-20.04]
+            compiler: gcc
+            library: static
+          - os: [self-hosted,arm-ubuntu-20.04]
+            compiler: gcc
+            library: shared
+            checks: doc+tests
+          - os: [self-hosted,arm-ubuntu-20.04]
+            compiler: clang
+            library: static
+          - os: [self-hosted,arm-ubuntu-20.04]
+            compiler: clang
+            library: shared
+            checks: doc+tests
+
+    steps:
+    - name: Checkout sources
+      uses: actions/checkout@v2
+    - name: Generate cache keys
+      id: get_ref_keys
+      run: |
+        echo -n '::set-output name=ccache::'
+        echo 'ccache-${{ matrix.config.os }}-${{ matrix.config.compiler }}-${{ matrix.config.cross }}-'$(date -u +%Y-w%W)
+        echo -n '::set-output name=libabigail::'
+        echo 'libabigail-${{ matrix.config.os }}'
+        echo -n '::set-output name=abi::'
+        echo 'abi-${{ matrix.config.os }}-${{ matrix.config.compiler }}-${{ matrix.config.cross }}-${{ env.LIBABIGAIL_VERSION }}-${{ env.REF_GIT_TAG }}'
+    - name: Retrieve ccache cache
+      uses: actions/cache@v2
+      with:
+        path: ~/.ccache
+        key: ${{ steps.get_ref_keys.outputs.ccache }}-${{ github.ref }}
+        restore-keys: |
+          ${{ steps.get_ref_keys.outputs.ccache }}-refs/heads/main
+    - name: Retrieve libabigail cache
+      id: libabigail-cache
+      uses: actions/cache@v2
+      if: env.ABI_CHECKS == 'true'
+      with:
+        path: libabigail
+        key: ${{ steps.get_ref_keys.outputs.libabigail }}
+    - name: Retrieve ABI reference cache
+      uses: actions/cache@v2
+      if: env.ABI_CHECKS == 'true'
+      with:
+        path: reference
+        key: ${{ steps.get_ref_keys.outputs.abi }}
+    - name: Update APT cache
+      run: sudo apt update || true
+    - name: Install packages
+      run: sudo apt install -y ccache libnuma-dev python3-setuptools
+        python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
+        libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
+        libarchive-dev zlib1g-dev pkgconf
+    - name: Install libabigail build dependencies if no cache is available
+      if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
+      run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
+          libdw-dev
+
+    - name: Install test tools packages
+      run: sudo apt install -y gdb
+    - name: Install doc generation packages
+      if: env.BUILD_DOCS == 'true'
+      run: sudo apt install -y doxygen graphviz python3-sphinx
+        python3-sphinx-rtd-theme
+    - name: Run setup
+      run: |
+        .ci/linux-setup.sh
+        # Workaround on $HOME permissions as EAL checks them for plugin loading
+        chmod o-w $HOME
+    - name: Install clang
+      if: env.CL == 'true'
+      run: sudo apt install -y clang
+    - name: Build and test
+      run: .ci/linux-build.sh
+    - name: Upload logs on failure
+      if: failure()
+      uses: actions/upload-artifact@v2
+      with:
+        name: meson-logs-${{ join(matrix.config.*, '-') }}
+        path: |
+          build/meson-logs/testlog.txt
+          build/.ninja_log
+          build/meson-logs/meson-log.txt
+          build/gdb.log
+
-- 
2.17.1


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [dpdk-stable] [dpdk-dev] [PATCH v1 1/1] ci: enable DPDK GHA for arm64 with self-hosted runners
  2021-10-13  8:03 ` [dpdk-stable] [PATCH v1 1/1] ci: enable DPDK GHA for arm64 with self-hosted runners Serena He
@ 2021-10-13 11:32   ` Michael Santana
  2021-10-14  7:04     ` Serena He
  0 siblings, 1 reply; 3+ messages in thread
From: Michael Santana @ 2021-10-13 11:32 UTC (permalink / raw)
  To: Serena He, aconole, maicolgabriel, david.marchand, thomas
  Cc: dev, nd, honnappa.nagarahalli, ruifeng.wang, Dean.Arnold, stable



On 10/13/21 4:03 AM, Serena He wrote:
> CI jobs are triggered only for repos installed with given GHApp and runners
> 
> Cc: stable@dpdk.org
> 
> Signed-off-by: Serena He <serena.he@arm.com>
> 
> ---
>   .github/workflows/build-arm64.yml | 118 ++++++++++++++++++++++++++++++
>   1 file changed, 118 insertions(+)
>   create mode 100644 .github/workflows/build-arm64.yml
> 
> diff --git a/.github/workflows/build-arm64.yml b/.github/workflows/build-arm64.yml
> new file mode 100644
> index 0000000000..570563f7c8
> --- /dev/null
> +++ b/.github/workflows/build-arm64.yml
Adding a new workflow should work on our 0-day-bot. We now support 
having multiple workflows so this looks good
> @@ -0,0 +1,118 @@
> +name: build-arm64
> +
> +on:
> +  push:
> +  schedule:
> +    - cron: '0 0 * * 1'
nit: Please add a comment for when this is scheduled so we dont have to 
do cron math :)
> +
> +defaults:
> +  run:
> +    shell: bash --noprofile --norc -exo pipefail {0}
> +
> +jobs:
> +  build:
> +    # Here, runners for arm64 are accessed by installed GitHub APP, thus will not be available by fork.
> +    # you can change the following 'if' and 'runs-on' if you have your own runners installed.
> +    # or request to get your repo on the whitelist to use GitHub APP and delete this 'if'.
I think I understand. I think you mean s/GitHub APP/GitHub/ . otherwise 
I dont know what that is. From my understanding you had to request 
special arm-based runners from github

Are DPDK/dpdk and ovsrobot/dpdk whitelisted to use the arm-based runners?

Maybe there was a thread about this in the past that I missed, but where 
and how do you get these arm-based runners from github?
> +    if: ${{ github.repository == 'DPDK/dpdk' || github.repository == 'ovsrobot/dpdk' }}
> +    name: ${{ join(matrix.config.*, '-') }}
> +    runs-on: ${{ matrix.config.os }}
> +    env:
> +      ABI_CHECKS: ${{ contains(matrix.config.checks, 'abi') }}
> +      BUILD_DOCS: ${{ contains(matrix.config.checks, 'doc') }}
> +      CL: ${{ matrix.config.compiler == 'clang' }}
> +      CC: ccache ${{ matrix.config.compiler }}
> +      DEF_LIB: ${{ matrix.config.library }}
> +      LIBABIGAIL_VERSION: libabigail-1.8
> +      REF_GIT_TAG: none
> +
> +    strategy:
> +      fail-fast: false
> +      matrix:
> +        config:
> +          - os: [self-hosted,arm-ubuntu-20.04]
> +            compiler: gcc
> +            library: static
> +          - os: [self-hosted,arm-ubuntu-20.04]
> +            compiler: gcc
> +            library: shared
> +            checks: doc+tests
> +          - os: [self-hosted,arm-ubuntu-20.04]
> +            compiler: clang
> +            library: static
> +          - os: [self-hosted,arm-ubuntu-20.04]
> +            compiler: clang
> +            library: shared
> +            checks: doc+tests
> +
> +    steps:
> +    - name: Checkout sources
> +      uses: actions/checkout@v2
> +    - name: Generate cache keys
> +      id: get_ref_keys
> +      run: |
> +        echo -n '::set-output name=ccache::'
> +        echo 'ccache-${{ matrix.config.os }}-${{ matrix.config.compiler }}-${{ matrix.config.cross }}-'$(date -u +%Y-w%W)
> +        echo -n '::set-output name=libabigail::'
> +        echo 'libabigail-${{ matrix.config.os }}'
> +        echo -n '::set-output name=abi::'
> +        echo 'abi-${{ matrix.config.os }}-${{ matrix.config.compiler }}-${{ matrix.config.cross }}-${{ env.LIBABIGAIL_VERSION }}-${{ env.REF_GIT_TAG }}'
> +    - name: Retrieve ccache cache
> +      uses: actions/cache@v2
> +      with:
> +        path: ~/.ccache
> +        key: ${{ steps.get_ref_keys.outputs.ccache }}-${{ github.ref }}
> +        restore-keys: |
> +          ${{ steps.get_ref_keys.outputs.ccache }}-refs/heads/main
> +    - name: Retrieve libabigail cache
> +      id: libabigail-cache
> +      uses: actions/cache@v2
> +      if: env.ABI_CHECKS == 'true'
> +      with:
> +        path: libabigail
> +        key: ${{ steps.get_ref_keys.outputs.libabigail }}
> +    - name: Retrieve ABI reference cache
> +      uses: actions/cache@v2
> +      if: env.ABI_CHECKS == 'true'
> +      with:
> +        path: reference
> +        key: ${{ steps.get_ref_keys.outputs.abi }}
> +    - name: Update APT cache
> +      run: sudo apt update || true
> +    - name: Install packages
> +      run: sudo apt install -y ccache libnuma-dev python3-setuptools
> +        python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
> +        libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
> +        libarchive-dev zlib1g-dev pkgconf
> +    - name: Install libabigail build dependencies if no cache is available
> +      if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-hit != 'true'
> +      run: sudo apt install -y autoconf automake libtool pkg-config libxml2-dev
> +          libdw-dev
Lots of caching stuff. All of it needed?
> +
> +    - name: Install test tools packages
> +      run: sudo apt install -y gdb
> +    - name: Install doc generation packages
> +      if: env.BUILD_DOCS == 'true'
> +      run: sudo apt install -y doxygen graphviz python3-sphinx
> +        python3-sphinx-rtd-theme
> +    - name: Run setup
> +      run: |
> +        .ci/linux-setup.sh
> +        # Workaround on $HOME permissions as EAL checks them for plugin loading
> +        chmod o-w $HOME
> +    - name: Install clang
> +      if: env.CL == 'true'
> +      run: sudo apt install -y clang
> +    - name: Build and test
> +      run: .ci/linux-build.sh
> +    - name: Upload logs on failure
> +      if: failure()
> +      uses: actions/upload-artifact@v2
> +      with:
> +        name: meson-logs-${{ join(matrix.config.*, '-') }}
> +        path: |
> +          build/meson-logs/testlog.txt
> +          build/.ninja_log
> +          build/meson-logs/meson-log.txt
> +          build/gdb.log
LGTM!
> +
> 


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [dpdk-stable] [dpdk-dev] [PATCH v1 1/1] ci: enable DPDK GHA for arm64 with self-hosted runners
  2021-10-13 11:32   ` [dpdk-stable] [dpdk-dev] " Michael Santana
@ 2021-10-14  7:04     ` Serena He
  0 siblings, 0 replies; 3+ messages in thread
From: Serena He @ 2021-10-14  7:04 UTC (permalink / raw)
  To: Michael Santana
  Cc: dev, nd, Honnappa Nagarahalli, Ruifeng Wang, Dean Arnold, stable,
	aconole, maicolgabriel, david.marchand, thomas

Hi Michael, thanks for the feedback, and here are some comments below.

> On 10/13/21 4:03 AM, Serena He wrote:
> > CI jobs are triggered only for repos installed with given GHApp and
> > runners
> >
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Serena He <serena.he@arm.com>
> >
> > ---
> >   .github/workflows/build-arm64.yml | 118
> ++++++++++++++++++++++++++++++
> >   1 file changed, 118 insertions(+)
> >   create mode 100644 .github/workflows/build-arm64.yml
> >
> > diff --git a/.github/workflows/build-arm64.yml
> > b/.github/workflows/build-arm64.yml
> > new file mode 100644
> > index 0000000000..570563f7c8
> > --- /dev/null
> > +++ b/.github/workflows/build-arm64.yml
> Adding a new workflow should work on our 0-day-bot. We now support
> having multiple workflows so this looks good

Great!

> > @@ -0,0 +1,118 @@
> > +name: build-arm64
> > +
> > +on:
> > +  push:
> > +  schedule:
> > +    - cron: '0 0 * * 1'
> nit: Please add a comment for when this is scheduled so we dont have to do
> cron math :)

Sure, I will add that. 

> > +
> > +defaults:
> > +  run:
> > +    shell: bash --noprofile --norc -exo pipefail {0}
> > +
> > +jobs:
> > +  build:
> > +    # Here, runners for arm64 are accessed by installed GitHub APP, thus
> will not be available by fork.
> > +    # you can change the following 'if' and 'runs-on' if you have your own
> runners installed.
> > +    # or request to get your repo on the whitelist to use GitHub APP and
> delete this 'if'.
> I think I understand. I think you mean s/GitHub APP/GitHub/ . otherwise I
> dont know what that is. From my understanding you had to request special
> arm-based runners from github
> 
> Are DPDK/dpdk and ovsrobot/dpdk whitelisted to use the arm-based
> runners?
> 
> Maybe there was a thread about this in the past that I missed, but where and
> how do you get these arm-based runners from github?

GitHub APPs are integrations with the GitHub APIs and this one provided here will send requests for arm-based runners from AWS cloud. Document will be provided along with the APP to make better understanding after release. DPDK/dpdk and ovsrobot/dpdk are both whitelisted to use.

> > +    if: ${{ github.repository == 'DPDK/dpdk' || github.repository ==
> 'ovsrobot/dpdk' }}
[...]
> > +    - name: Generate cache keys
> > +      id: get_ref_keys
> > +      run: |
> > +        echo -n '::set-output name=ccache::'
> > +        echo 'ccache-${{ matrix.config.os }}-${{ matrix.config.compiler }}-
> ${{ matrix.config.cross }}-'$(date -u +%Y-w%W)
> > +        echo -n '::set-output name=libabigail::'
> > +        echo 'libabigail-${{ matrix.config.os }}'
> > +        echo -n '::set-output name=abi::'
> > +        echo 'abi-${{ matrix.config.os }}-${{ matrix.config.compiler }}-
> ${{ matrix.config.cross }}-${{ env.LIBABIGAIL_VERSION }}-
> ${{ env.REF_GIT_TAG }}'
> > +    - name: Retrieve ccache cache
> > +      uses: actions/cache@v2
> > +      with:
> > +        path: ~/.ccache
> > +        key: ${{ steps.get_ref_keys.outputs.ccache }}-${{ github.ref }}
> > +        restore-keys: |
> > +          ${{ steps.get_ref_keys.outputs.ccache }}-refs/heads/main
> > +    - name: Retrieve libabigail cache
> > +      id: libabigail-cache
> > +      uses: actions/cache@v2
> > +      if: env.ABI_CHECKS == 'true'
> > +      with:
> > +        path: libabigail
> > +        key: ${{ steps.get_ref_keys.outputs.libabigail }}
> > +    - name: Retrieve ABI reference cache
> > +      uses: actions/cache@v2
> > +      if: env.ABI_CHECKS == 'true'
> > +      with:
> > +        path: reference
> > +        key: ${{ steps.get_ref_keys.outputs.abi }}
> > +    - name: Update APT cache
> > +      run: sudo apt update || true
> > +    - name: Install packages
> > +      run: sudo apt install -y ccache libnuma-dev python3-setuptools
> > +        python3-wheel python3-pip python3-pyelftools ninja-build libbsd-dev
> > +        libpcap-dev libibverbs-dev libcrypto++-dev libfdt-dev libjansson-dev
> > +        libarchive-dev zlib1g-dev pkgconf
> > +    - name: Install libabigail build dependencies if no cache is available
> > +      if: env.ABI_CHECKS == 'true' && steps.libabigail-cache.outputs.cache-
> hit != 'true'
> > +      run: sudo apt install -y autoconf automake libtool pkg-config libxml2-
> dev
> > +          libdw-dev
> Lots of caching stuff. All of it needed?

All these caching stuff is in consistence with the build workflow on GitHub provided runners, so should be needed.


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-10-14  7:05 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20211013080341.29561-1-serena.he@arm.com>
2021-10-13  8:03 ` [dpdk-stable] [PATCH v1 1/1] ci: enable DPDK GHA for arm64 with self-hosted runners Serena He
2021-10-13 11:32   ` [dpdk-stable] [dpdk-dev] " Michael Santana
2021-10-14  7:04     ` Serena He

patches for DPDK stable branches

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.dpdk.org/stable/0 stable/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 stable stable/ http://inbox.dpdk.org/stable \
		stable@dpdk.org
	public-inbox-index stable

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.stable


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git