From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 770CFA0547 for ; Wed, 19 May 2021 18:06:49 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 46AF240041; Wed, 19 May 2021 18:06:49 +0200 (CEST) Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by mails.dpdk.org (Postfix) with ESMTP id 6E38040041 for ; Wed, 19 May 2021 18:06:47 +0200 (CEST) Received: by mail-ej1-f44.google.com with SMTP id i7so2927087ejc.5 for ; Wed, 19 May 2021 09:06:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dRBD92CTEug038eK58aTAQkgRxNVBqtd7lsijqYwOsA=; b=AwSUMSmkgJ3wLbdYtafHT6gdjLnezA2fgKCtxCLtMTG1AO8AOTiaW8YbfM0R1dDFbB OSGqz6xuP5+hnh6W0v3YytJd5eoY4pfEU9caVLm8Bzj78GOxBge7PaefPIMq9mvd32BX F80UskNVk5PY3VtplEx4GjDuxSwCgcgCCBvFs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dRBD92CTEug038eK58aTAQkgRxNVBqtd7lsijqYwOsA=; b=H4IvfHFS/JEi8sWE0D5UmQVvstTX0FW6/06hZ84Gmvi8LbOMHG/beQLD/jFcBWU1Pl +Y47prdk8sg/Nj2OcnHzeqoet/F2NhIIUlY+OOtT0BSfWL0N0Y2QeXjsDzU7Ken7VpAm l3S7uFXiYQ6Y6k2WLCH05XTjld281MrtYo4R4XQb4jQoZm0z+R08hY6H5KjU/+OwkykL +04AuFIVBdqIjY7/v6XrDZI4coyXu4Hw4SJT6+SK722sJWKo9lW6J4jNSFKf4v6+LQOz ZCiJdHhb/wQyGyzrKzy6G94z8c+zAdKbKzF67VOmJ/LdbPKckKl9n5W5niuxWH96sV8d 206A== X-Gm-Message-State: AOAM533ijY8OZXKWbgCGv8S3c7Zkohr5b9uxzu5U73IUdubYzs/wjEk+ SVbxsdkfXEX+5D8BG801QkaLCQEJkGAr8ZGcOAcyNQ== X-Google-Smtp-Source: ABdhPJzrcsSrn5ife0FQ9KWcQKgMGCn6qXjA1FrcUAxsfiLhAQRPy+CDOC94Ggo+tGe6ZwUmLuVllS0M4jpj/QKjBIE= X-Received: by 2002:a17:906:3e89:: with SMTP id a9mr13313233ejj.405.1621440407089; Wed, 19 May 2021 09:06:47 -0700 (PDT) MIME-Version: 1.0 References: <1645051.GgosbX4935@thomas> In-Reply-To: From: Lincoln Lavoie Date: Wed, 19 May 2021 12:06:35 -0400 Message-ID: To: Aaron Conole Cc: Thomas Monjalon , ci@dpdk.org, dpdklab , David Marchand Content-Type: multipart/alternative; boundary="0000000000007d51ef05c2b1037e" Subject: Re: [dpdk-ci] UNH CI failing X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --0000000000007d51ef05c2b1037e Content-Type: text/plain; charset="UTF-8" I'm continuing to hunt. I did confirm all of those systems are syncing time from the same master, using chronyd, and look to be configured correctly. I'm not sure what would cause the time to jump like that, unless maybe the "master", which is our IPA server, synced it's time and that caused some sort of ripple to the downstream other systems. The patches have been rerun, with most results coming in, https://lab.dpdk.org/results/dashboard/patchsets/17130/, and are passing without issue. The only failures I see in recent patches are on "bugfix for Kunpeng SVE compile" (https://lab.dpdk.org/results/dashboard/patchsets/17135/), with unit tests failing on two OSes. And the intermittent Dynamic Config failure. We think we have tracked this down to a patch we sent into DTS, and that was merged into DTS, but was then reverted a few days ago ( https://git.dpdk.org/tools/dts/commit/?id=90f460df240b3020191916b15705abe208a14694). I've asked Lijuan why it was reverted. Cheers, Lincoln On Wed, May 19, 2021 at 11:55 AM Aaron Conole wrote: > Lincoln Lavoie writes: > > > As far as I can tell, it looks like one of the clocks fell out of sync > on the container runners, which caused the > > builds to fail. > > I think that's also been causing some failures with the alarm_test and > cycles_test unit tests. If the time source is making adjustments to > time, we will probably fail these tests as well. > > > Also, as far I can tell from an initial look, it impacted the two > patches Thomas cited. Two patches that are > > running now (i.e. they don't have a full set of results yet, look like > they are running on. So, it was a transient > > issue. Obviously we need to track down its root cause. I suspect > something happened with NTP, which should > > be keeping the runners and bare metal systems synced. I'm looking into > that now. > > > > For the patches with the failed jobs, we will queue those for rerun > today. > > > > Cheers, > > Lincoln > > > > On Wed, May 19, 2021 at 9:05 AM Aaron Conole wrote: > > > > Thomas Monjalon writes: > > > > > It seems the IOL CI is failing today: > > > > > > > > > https://patches.dpdk.org/project/dpdk/patch/1621406749-15536-1-git-send-email-changpeng.liu@intel.com/ > > > > > > > > https://patches.dpdk.org/project/dpdk/patch/20210519032745.707639-1-stevex.yang@intel.com/ > > > > > > > > That's especially embarassing for closing the release. > > > > I don't see any useful logs in the failures. > > > > What changed? > > -- *Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu https://www.iol.unh.edu +1-603-674-2755 (m) --0000000000007d51ef05c2b1037e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I&#= 39;m continuing to hunt. I did confirm all of those systems are syncing tim= e from the same master, using chronyd, and look to be configured correctly.= =C2=A0 I'm not sure what would cause the time to jump like that, unless= maybe the "master", which is our IPA server, synced it's tim= e and that caused some sort of ripple to the downstream other systems.

The patches have been rerun, = with most results coming in,=C2=A0https://lab.dpdk.org/results/dashboard/patchsets= /17130/, and are passing without issue.

The only failures I see in recent patches=C2=A0are on = "bugfix for Kunpeng SVE compile" (https://lab.dpdk.org/results/dashboard= /patchsets/17135/), with unit tests failing=C2=A0on two OSes.

And the intermittent=C2=A0Dynamic = Config failure.=C2=A0 We think we have tracked this down to a patch we sent= into DTS, and that was merged into DTS, but was then reverted a few days a= go (https://git.dpdk.org/tools/dts/commit/?id=3D90f46= 0df240b3020191916b15705abe208a14694).=C2=A0 I've asked Lijuan why i= t was reverted.=C2=A0

Che= ers,
Lincoln

On Wed, May 19, 2021 at 11:55 AM Aaron Conole <aconole@redhat.com> wrote:
Lincoln Lavoie <lylavoie@iol.unh.edu<= /a>> writes:

> As far as I can tell, it looks like one of the clocks fell out of sync= on the container runners, which caused the
> builds to fail.

I think that's also been causing some failures with the alarm_test and<= br> cycles_test unit tests.=C2=A0 If the time source is making adjustments to time, we will probably fail these tests as well.

> Also, as far I can tell from an initial look, it impacted the two patc= hes Thomas cited.=C2=A0 Two patches that are
> running now (i.e. they don't have a full set of results yet, look = like they are running on.=C2=A0 So, it was a transient
> issue.=C2=A0 Obviously we need to track down its root cause. I suspect= something happened with NTP, which should
> be keeping the runners and bare metal systems synced.=C2=A0 I'm lo= oking into that now.
>
> For the patches with the failed jobs, we will queue those for rerun to= day.
>
> Cheers,
> Lincoln
>
> On Wed, May 19, 2021 at 9:05 AM Aaron Conole <aconole@redhat.com> wrote:
>
>=C2=A0 Thomas Monjalon <thomas@monjalon.net> writes:
>
>=C2=A0 > It seems the IOL CI is failing today:
>=C2=A0 >
>=C2=A0 >
>=C2=A0 https://patches.dpdk.org/project/dpdk/patch/1621406749-15536-1= -git-send-email-changpeng.liu@intel.com/
>=C2=A0
>=C2=A0 >
>=C2=A0 https://patches.dpdk.org/project/dpdk/patch/20210519032745.707639-1-stevex= .yang@intel.com/
>=C2=A0
>=C2=A0 >
>=C2=A0 > That's especially embarassing for closing the release.<= br> >
>=C2=A0 I don't see any useful logs in the failures.
>
>=C2=A0 What changed?



--
Lincoln Lavoie
Prin= cipal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, = Durham, NH 03824
+1-603-674-= 2755 (m)

--0000000000007d51ef05c2b1037e--