From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 51322A04B5 for ; Wed, 4 Dec 2019 16:02:07 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 483B01BF74; Wed, 4 Dec 2019 16:02:07 +0100 (CET) Received: from mail-lf1-f54.google.com (mail-lf1-f54.google.com [209.85.167.54]) by dpdk.org (Postfix) with ESMTP id 48D981BE9D for ; Wed, 4 Dec 2019 16:02:06 +0100 (CET) Received: by mail-lf1-f54.google.com with SMTP id r14so6399782lfm.5 for ; Wed, 04 Dec 2019 07:02:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IC4nYPwEQquSI7kILPQ0gQG+A683AvDQ7KXbcJYBrac=; b=gkV2SHoUuBa7DlDc3HMKHr3DGWmDVQ4hb4GzjPGOVm5yPPabNXTJ6DKD4ZaZMwnP3S JkGeLDok+hPAgHnsm9vIAKIkQcpqB1OG9tdd5++RsuTsqojeuOK1HIteV1VqybUPoWIh lTVSwYjYku3+LHEXyCA1vl5ka2Jya+YQTMieM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IC4nYPwEQquSI7kILPQ0gQG+A683AvDQ7KXbcJYBrac=; b=CTVMlA0F26awRW/PDeKipFp9If641cEEknLZNUpqem0/n265FWVyP8cw9T2rs3P0CJ Z/J3k1CJ9hMn7Ka9PoKzsF9Sfrm5CVM6B3rbDIqBd/2BDFGAEwOWEvr5WAFAWQka3lwe lKlihNiy/6Gz6tnEc5+k9pv2qeZeTYjZjjSihwlsghEhcZdY3z0kvyJnelNfArO5Ej37 HolL2pe2lnKjDBCgU5ink42y+WEttrlx/VSfJ1UWbRwaiWrvsCcCBjVrorgbGDxmgrtH XuqIkZdk4lAAU9+8J2Usf8L0DNLpCLLXjNTnyvoNXLQCjhW5YKWjjjKA7GsG2CAiOxkc dXRQ== X-Gm-Message-State: APjAAAUbG7BHFq6nlO+0M3L9PsnTNHjt6yokrxMaiWFS/9P6WXtYNojH D01VZIFIDAVReeD2hmSX6W4cfsSG13CeOBHEN1EASw== X-Google-Smtp-Source: APXvYqy/w5512dFWJwCYYxza6wAHiTskHjpUzVgd2SRQVVM2n7VM1YzCEEKE2+3KimxFH+YdUsnarbZJctpqkIl8Kr0= X-Received: by 2002:a19:c382:: with SMTP id t124mr2186634lff.124.1575471725761; Wed, 04 Dec 2019 07:02:05 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Jeremy Plsek Date: Wed, 4 Dec 2019 10:01:29 -0500 Message-ID: To: David Marchand Cc: ci@dpdk.org, Bruce Richardson , Aaron Conole Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-ci] Meson failing on master X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" On Wed, Dec 4, 2019 at 6:09 AM David Marchand wrote: > > Hello Jeremy, > > I noticed that meson is failing for a patch on master. > Example: https://lab.dpdk.org/results/dashboard/patchsets/8804/ > > The artifact gives the non-obvious (to me) error log: > > Build targets in project: 810 > Option default_library is: shared [default: static] > Found ninja-1.8.2 at /usr/bin/ninja > ninja: error: manifest 'build.ninja' still dirty after 100 tries > > [1] Error running command. > > Talked to Bruce, and he suspects something on the filesystem level. > Is this issue happening on a system with the overlayfs stuff you mentioned? > Hi David, This started happening last week as well (https://lab.dpdk.org/results/dashboard/patchsets/8711/), so I disabled it temporarily. I enabled it yesterday, and it was fine for a few patch sets (https://lab.dpdk.org/results/dashboard/patchsets/8803/), but, as you mentioned, it broke again. I've disabled it again for future tests until it can be resolved. This is not related to overlayfs work I've been working on (the overlayfs work was for the bare metal machines). These tests run in a container, using podman, with a mount to the jenkins workspace. The containers run on a no-so-special Ubuntu 18.04 VM. I've noticed similar issues for other CI (https://github.com/ninja-build/ninja/issues/1599, https://github.com/neovim/neovim/issues/10213), which, for the most part, wasn't really resolved or gave a solution, and may not be related to our issue. Hmmm... I was able to reproduce it with the tarball that "failed" most recently (ps 8804), but it passed with the older "failed" tarball (ps 8711). The difference between them that I noticed, was that when untarring, it (ps 8804) would complain that the date was in the future. $ date Wed Dec 4 14:28:18 UTC 2019 $ ls -l --full-time ... -rw-rw-r-- 1 jenkins-local jenkins-local 12 2019-12-04 15:03:12.000000000 +0000 VERSION I tried to just update the files to the current time... $ find . -exec touch {} + And now it compiles... Hmmm... Yes, it seems that causes it? Updating the date fixed it? Interesting bug. So there are a few issues to look into: 1. Why the dates in the tarball are wrong in the first place (my issue. or just add a dirty fix to the compile script to just update the date after extracting) 2. Why the date of the file matters so much in meson/ninja (I understand that build systems typically use the date to check whether to recompile a file, but it should be more graceful about handling this type of in-the-future problem) I'll see if I can make a simple example and maybe make a bug report. (At the minimum meson/ninja should fail gracefully rather than retry 100 times.) > > -- > David Marchand > -- Jeremy Plsek UNH InterOperability Laboratory