From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6B17FA0524 for ; Wed, 2 Jun 2021 14:55:55 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A89D4069F; Wed, 2 Jun 2021 14:55:55 +0200 (CEST) Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53]) by mails.dpdk.org (Postfix) with ESMTP id 97C1440689 for ; Wed, 2 Jun 2021 14:55:53 +0200 (CEST) Received: by mail-ej1-f53.google.com with SMTP id k7so3640398ejv.12 for ; Wed, 02 Jun 2021 05:55:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Rewv+4L/EgJq0TWN572+WNW/fd9ajSmoxAEOCXwL0sU=; b=WQ6hxAlhL+QIR/0zQ8ulw51SLGGeqTF3mgq6/wKeIKDu+rc3CHor2n20ylkG3PGojQ VoQqR/5hiDPkG5Wjbu4RREMtZ9H0nx8u26d+kW6lb8hztJpCLjb3R8FjKZ0XbTJaASr4 982gJGCAVy7ZsW6btTnVbPTd6Wd81DmtJaWh8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Rewv+4L/EgJq0TWN572+WNW/fd9ajSmoxAEOCXwL0sU=; b=J7MMVOq/C1H3qNeS0VCABloq0xt19VJGROgBas3Swp0zOYU8epmVnmbkSMu13HcL0y yQD9NkCLeyYqM2YPBBMaOk1kuzob3aLC3uwWgnZUSpv6m6CyQ4T7KBcCSZo0h9fh/o8l 2Lz52STcVDC+MF9kgDyRfdauFiviC4pKry7F12qa6TOyPwQ9TbcL4bDZyp8N++RkU6f3 3odXyw8bqi2V9+24y+b+JD4qCeWX0XcAPSOl3FzYyfKQRKBO32eGT5Bdi1Y5aIYdYeQl dgbHK+wj9ETdiQycpDI8yy3ZoMbNZbpqgvRV6hy6hdnMPVMHlNZOxGxQmuGKzXoGW1Wg IUAg== X-Gm-Message-State: AOAM530SV+dZFOsbUd2Wd44iUTp8K2erwNh+eIgF+sKQQE4+kphngmY5 bf7B42P9+j6A9ZPPP4erylPw63sdo6aPWBULUvXPqxjfKowcTg== X-Google-Smtp-Source: ABdhPJzew4HjTdhe1YmC7wBoCpPky3/d+IwaxYY0hdl7tkWSaEvvsqxqWj+Om0Qq9CSc+3hyRSLBsRoTm14XtS7jqJQ= X-Received: by 2002:a17:906:d04f:: with SMTP id bo15mr33236025ejb.405.1622638553237; Wed, 02 Jun 2021 05:55:53 -0700 (PDT) MIME-Version: 1.0 References: <101415929.9lNPY92Spv@thomas> In-Reply-To: <101415929.9lNPY92Spv@thomas> From: Lincoln Lavoie Date: Wed, 2 Jun 2021 08:55:42 -0400 Message-ID: To: Thomas Monjalon Cc: ci@dpdk.org, aconole@redhat.com Content-Type: multipart/alternative; boundary="00000000000090a56005c3c7fa0a" Subject: Re: [dpdk-ci] CI reliability X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" --00000000000090a56005c3c7fa0a Content-Type: text/plain; charset="UTF-8" Hi Thomas, The unit tests that fail are nearly always the same specific unit test. Aaron addressed some of these in a patch that has yet to be applied to DPDK (cycles_autotest and test_alarm). The other one that we still consistently fail is func_reentrancy_autotest. It seems like that unit test case can pass in one run and fail in the next. We have not been able to determine a root cause for it yet. Maybe that is something the devs could help look into. Other failures have been caused by DTS. As part of the plan, we've been trying to upgrade the DTS deployments on the system, so as the other changes are made, we can easily pull those in. However, pulling in the new DTS version has also pulled in bugs that exist in that version. For example, on the stats test suite, it was changed to not skip the test_xstats_check_vf when no VMs are configured on the system, so when the overall test suite was being run, it failed on the bare metal where there are VMs configured right now. Every time the lab has to upgrade DTS, we run the risk of introducing these types of failures, which then take time to debug and fix. Cheers, Lincoln On Wed, Jun 2, 2021 at 3:27 AM Thomas Monjalon wrote: > I see a lot of failures in the CI, especially unit tests run in UNH IOL. > It seems to fail for several weeks but did not investigate more. > What is the cause and what is the plan? > Should we rely on CI results? > > > -- *Lincoln Lavoie* Principal Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 lylavoie@iol.unh.edu https://www.iol.unh.edu +1-603-674-2755 (m) --00000000000090a56005c3c7fa0a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi = Thomas,

The unit tests th= at fail are nearly always the same specific unit test. Aaron addressed some= of these in a patch that has yet to be applied=C2=A0to DPDK (cycles_autote= st and test_alarm).=C2=A0 The other one that we still consistently fail is= =C2=A0func_reentrancy_autotest.=C2=A0 It seems like that unit test case can= pass in one run and fail in the next.=C2=A0 We have not been able to deter= mine a root cause for it yet.=C2=A0 Maybe that is something=C2=A0the devs c= ould help look into.

Othe= r failures have been caused by DTS.=C2=A0 As part of the plan, we've be= en trying to upgrade the DTS deployments on the system, so as the other cha= nges are made, we can easily pull those in.=C2=A0 However, pulling in the n= ew DTS version has also pulled in bugs that exist in that version.=C2=A0 Fo= r example, on the stats test suite, it was changed to not skip the=C2=A0tes= t_xstats_check_vf when no VMs are configured on the system, so when the ove= rall test suite was being run, it failed on the bare metal where there are = VMs configured right now.=C2=A0 Every time the lab has to upgrade DTS, we r= un the risk of introducing these types of failures, which then take time to= debug and fix.

Cheers,Lincoln

On Wed, Jun 2, 2021 at 3:27 AM Thomas Monjalon <thomas@monjalon.net> wrote:
I see a lot of failures = in the CI, especially unit tests run in UNH IOL.
It seems to fail for several weeks but did not investigate more.
What is the cause and what is the plan?
Should we rely on CI results?




--
Lincoln Lavoie
Prin= cipal Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, = Durham, NH 03824
+1-603-674-= 2755 (m)

--00000000000090a56005c3c7fa0a--