From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D693AA0507; Mon, 4 Apr 2022 16:24:48 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6C6DC4281A; Mon, 4 Apr 2022 16:24:48 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mails.dpdk.org (Postfix) with ESMTP id E41144068C for ; Mon, 4 Apr 2022 16:24:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649082286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VeXxOuOWRPt9qOV0+fBErzmVySQGmWd05Obx+Tz9/C0=; b=CbpzqYvFUk8LtIs9nPhp4qVnQWnnWpnnxqMDpVYv7/bzgOkUoDK6eO7TbLl2PukDXj25Lu xgQnNgekITb4U/V3emrRCrOScSHepszjQIzVUGiiRNZV+65RIRASSjDYl3UuxkNrbUf6T7 be1ZBO5UjYHHOL23zVCou/D00hUhEX8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-225-MTnMPCdwOSe5RQUIGsLogQ-1; Mon, 04 Apr 2022 10:24:45 -0400 X-MC-Unique: MTnMPCdwOSe5RQUIGsLogQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 077513C14CC3; Mon, 4 Apr 2022 14:24:45 +0000 (UTC) Received: from dmarchan.remote.csb (unknown [10.40.195.27]) by smtp.corp.redhat.com (Postfix) with ESMTP id 112DE4122AC0; Mon, 4 Apr 2022 14:24:43 +0000 (UTC) From: David Marchand To: dev@dpdk.org Cc: rjarry@redhat.com, Aaron Conole , Michael Santana Subject: [PATCH v3] ci: add Fedora 35 container in GHA Date: Mon, 4 Apr 2022 16:24:39 +0200 Message-Id: <20220404142439.15299-1-david.marchand@redhat.com> In-Reply-To: <20211116123102.19237-1-david.marchand@redhat.com> References: <20211116123102.19237-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david.marchand@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Build DPDK with Fedora 35 containers. GHA container support does not allow caching images and docker hub seems to limit image pulls. On the other hand, the Fedora project hub does not seem to limit them, so prefer this hub. Nevertheless, let's try to be good citizens and cache (once a day) a prepared image for subsequent builds. This preparation is done in a first prepare-container-images job. The rpm-container-builds job then depends on it with a 'needs:' tag. Differences with builds in Ubuntu GHA vm images: - tasks are run as root in containers, no need for sudo, - compiler must be explicitly installed, - GHA artifacts can't contain a ':' in their name, and must be filtered, - environment variables are not inherited and must be passed explicitly, Signed-off-by: David Marchand --- Changes since v2: - used a two stage pipeline so that container image generation is done only once, - changed image name local:dpdk to dpdk-local (closer to conventions), Changes since v1: - switched to Fedora project hub, - cached container image, --- .ci/linux-build.sh | 3 + .ci/linux-setup.sh | 3 + .github/workflows/build.yml | 142 +++++++++++++++++++++++++++++++++++- 3 files changed, 146 insertions(+), 2 deletions(-) diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh index 69940dec8b..774a1441bf 100755 --- a/.ci/linux-build.sh +++ b/.ci/linux-build.sh @@ -1,5 +1,8 @@ #!/bin/sh -xe +# Builds are run as root in containers, no need for sudo +[ "$(id -u)" != '0' ] || alias sudo= + on_error() { if [ $? = 0 ]; then exit diff --git a/.ci/linux-setup.sh b/.ci/linux-setup.sh index ca35b4b38d..f5ef6b1b98 100755 --- a/.ci/linux-setup.sh +++ b/.ci/linux-setup.sh @@ -1,5 +1,8 @@ #!/bin/sh -xe +# Builds are run as root in containers, no need for sudo +[ "$(id -u)" != '0' ] || alias sudo= + # need to install as 'root' since some of the unit tests won't run without it sudo python3 -m pip install --upgrade 'meson==0.49.2' diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 02819aa5de..22daaabb91 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -10,7 +10,7 @@ defaults: shell: bash --noprofile --norc -exo pipefail {0} jobs: - build: + ubuntu-vm-builds: name: ${{ join(matrix.config.*, '-') }} runs-on: ${{ matrix.config.os }} env: @@ -145,7 +145,145 @@ jobs: with: name: meson-logs-${{ join(matrix.config.*, '-') }} path: | + build/.ninja_log + build/gdb.log + build/meson-logs/meson-log.txt build/meson-logs/testlog.txt + + prepare-container-images: + name: ${{ join(matrix.config.*, '-') }} + runs-on: ubuntu-latest + + strategy: + fail-fast: false + matrix: + config: + - image: fedora:35 + + steps: + - name: Generate various keys + id: get_keys + run: | + echo -n '::set-output name=image::' + echo 'image-${{ matrix.config.image }}-'$(date -u +%Y-%m-%d) + - name: Retrieve image cache + id: image_cache + uses: actions/cache@v2 + with: + path: ~/.image + key: ${{ steps.get_keys.outputs.image }} + - name: Pull and prepare a fresh image + if: steps.image_cache.outputs.cache-hit != 'true' + run: | + docker pull registry.fedoraproject.org/${{ matrix.config.image }} + docker run -d -i --rm --name dpdk \ + registry.fedoraproject.org/${{ matrix.config.image }} \ + bash -li + - name: Update + if: steps.image_cache.outputs.cache-hit != 'true' + run: docker exec -i dpdk dnf update -y + - name: Install packages + if: steps.image_cache.outputs.cache-hit != 'true' + run: docker exec -i dpdk dnf install -y ccache numactl-devel + python3-setuptools python3-wheel python3-pip python3-pyelftools + ninja-build libatomic libbpf-devel libfdt-devel libpcap-devel + openssl-devel rdma-core-devel zlib-devel + - name: Save image in cache + if: steps.image_cache.outputs.cache-hit != 'true' + run: | + docker commit dpdk dpdk-local + mkdir -p ~/.image + docker save -o ~/.image/${{ matrix.config.image }}.tar dpdk-local + - name: Stop image + if: steps.image_cache.outputs.cache-hit != 'true' + run: docker kill dpdk + + rpm-container-builds: + needs: prepare-container-images + name: ${{ join(matrix.config.*, '-') }} + runs-on: ubuntu-latest + + strategy: + fail-fast: false + matrix: + config: + - image: fedora:35 + compiler: gcc + library: static + - image: fedora:35 + compiler: gcc + library: shared + - image: fedora:35 + compiler: clang + library: static + - image: fedora:35 + compiler: clang + library: shared + + steps: + - name: Checkout sources + uses: actions/checkout@v2 + - name: Generate various keys + id: get_keys + run: | + echo -n '::set-output name=ccache::' + echo 'ccache-${{ matrix.config.image }}-${{ matrix.config.compiler }}-'$(date -u +%Y-w%W) + echo -n '::set-output name=image::' + echo 'image-${{ matrix.config.image }}-'$(date -u +%Y-%m-%d) + echo -n '::set-output name=logs::' + echo 'meson-logs-${{ join(matrix.config.*, '-') }}' | tr -d ':' + - name: Retrieve image cache + id: image_cache + uses: actions/cache@v2 + with: + path: ~/.image + key: ${{ steps.get_keys.outputs.image }} + - name: Fail if no image (not supposed to happen) + if: steps.image_cache.outputs.cache-hit != 'true' + run: | + echo 'Image ${{ matrix.config.image }} is not cached.' + false + - name: Retrieve ccache cache + uses: actions/cache@v2 + with: + path: ~/.ccache + key: ${{ steps.get_keys.outputs.ccache }}-${{ github.ref }} + restore-keys: | + ${{ steps.get_keys.outputs.ccache }}-refs/heads/main + - name: Prepare working directory + run: | + mkdir -p ~/.ccache + > ~/env + echo CC=ccache ${{ matrix.config.compiler }} >> ~/env + echo DEF_LIB=${{ matrix.config.library }} >> ~/env + - name: Load the cached image + run: | + docker load -i ~/.image/${{ matrix.config.image }}.tar + docker run -d -i --rm --name dpdk \ + --mount type=bind,src=$HOME/.ccache,dst=/root/.ccache \ + --mount type=bind,src=$(pwd),dst=/root/dpdk \ + --env-file ~/env \ + -w /root/dpdk \ + dpdk-local + bash -li + - name: Update + run: docker exec -i dpdk dnf update -y || true + - name: Install packages + run: docker exec -i dpdk dnf install -y ccache numactl-devel + python3-setuptools python3-wheel python3-pip python3-pyelftools + ninja-build libatomic libbpf-devel libfdt-devel libpcap-devel + openssl-devel rdma-core-devel zlib-devel ${{ matrix.config.compiler }} + - name: Run setup + run: docker exec -i dpdk .ci/linux-setup.sh + - name: Build + run: docker exec -i dpdk .ci/linux-build.sh + - name: Stop image + run: docker kill dpdk + - name: Upload logs on failure + if: failure() + uses: actions/upload-artifact@v2 + with: + name: ${{ steps.get_keys.outputs.logs }} + path: | build/.ninja_log build/meson-logs/meson-log.txt - build/gdb.log -- 2.23.0