From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ci-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id 9F76C42215;
	Thu, 31 Aug 2023 21:37:47 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 8833E4028D;
	Thu, 31 Aug 2023 21:37:47 +0200 (CEST)
Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com
 [209.85.128.182])
 by mails.dpdk.org (Postfix) with ESMTP id 192034028A
 for <ci@dpdk.org>; Thu, 31 Aug 2023 21:37:46 +0200 (CEST)
Received: by mail-yw1-f182.google.com with SMTP id
 00721157ae682-59234aaca15so14058017b3.3
 for <ci@dpdk.org>; Thu, 31 Aug 2023 12:37:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=iol.unh.edu; s=unh-iol; t=1693510665; x=1694115465; darn=dpdk.org;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:from:to:cc:subject:date:message-id:reply-to;
 bh=UwzEx2DcAp189RQSSAwBTjGWn+yWxVCM/0dc6RCgjwg=;
 b=hS60fI71CCoc7Umk6uhVxBWNL7Jia8MHxQyCS4Uvs5xfmid/MSVhmlOhgbGArC9tWk
 whuvckuzp5LkQYB/C2ESyLe0e0W4craT36BgeOipcUyNsDHTsn4KgZxv0JbxpStf9ktj
 VyFXelm6GJDJ2sqy/wX/qWBktwr089a3KxOR0=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20221208; t=1693510665; x=1694115465;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
 :reply-to;
 bh=UwzEx2DcAp189RQSSAwBTjGWn+yWxVCM/0dc6RCgjwg=;
 b=N1MPzNX30ZKHxsyyJ/R9EIz9AHXJ4CZB2s6u090NR8/MHTx7LjG7vp2nGbvPmWB1xt
 /V1DwaURko9Y+pOQrk8a4BuzYIqJWN6p3rr1B54JFTpZqwOhY2lsZOHw+SMHDtTu/7UI
 hmLhhmE7skwqmCoocFl8czcVs7HHC7UcqmtGun055mXvlWxz/Ou3+KNZ6tHF2c+Ybpfe
 ovRvmvsUiFEgFzqpNuVgSE7P9HgE7q2JXOl7Opviy8WrTMaLneu9UPPXINc9CBGFtHy0
 N++54lpVGLbPERL9WKNNL7L0p1YFlpEVd0phFmLelqnobh3new19+IQNvDuJJ5Uxy5eN
 yhTg==
X-Gm-Message-State: AOJu0YwIgAhkJPnz6yKGifq/IOoCCnWNR48O3zkaiWCrospGCjBhjbvz
 MJS7CzII/Ce/7ik1b1E1uFqQdz8YHBI+sGByrQ3ViW1/iPX3DzKYP6mFCw==
X-Google-Smtp-Source: AGHT+IFYCeRlaErI3t9EY1gYnIzd62S6oWAncmqxLbvw30pnuALGIuBpxRLl+iI8u3SciIe2E34Mo+92ocqn4pP2gyw=
X-Received: by 2002:a81:928d:0:b0:573:cacd:306 with SMTP id
 j135-20020a81928d000000b00573cacd0306mr480455ywg.16.1693510665091; Thu, 31
 Aug 2023 12:37:45 -0700 (PDT)
MIME-Version: 1.0
References: <CAC-YWqiQfH4Rx-Et1jGHhGK9i47d0AArKy-B2P77iYbbM+Lpig@mail.gmail.com>
 <CAC-YWqg9LaOXGo9DT0oz0Co6oSqb2QVZTeWrQ2i=Em=R0X-AYQ@mail.gmail.com>
 <CAC-YWqjbaiVOWqb1-1kmp2Ci+VLACVkUygyMb83HfoNHur9NhQ@mail.gmail.com>
 <7734826a-840d-d0d9-e7a5-91951223398c@oktetlabs.ru>
 <CAC-YWqgvgKRgffwWY3mqWbC8o-LJ_o0BkRuuBxDSqg8Pnj0h1Q@mail.gmail.com>
 <9d920676-485d-3b4d-ca20-2b5ea3a5b606@oktetlabs.ru>
 <CAC-YWqhth9q5QDVhtfDoXkfQGwg4b3w2Q0d5EzhYZMFEhhzE6Q@mail.gmail.com>
 <873c7972-3e5a-9e82-9449-4d12b2c96032@oktetlabs.ru>
 <cc758ad7-2f40-7c6c-ffab-c574011ba770@oktetlabs.ru>
 <CAC-YWqj28X-YhLzM-rChuKMdaoTrDS6fN68YxzS+rb1N-i5AkQ@mail.gmail.com>
 <6c9eea95-7d53-c82c-bced-823e4e9db62a@oktetlabs.ru>
 <d62aa33b-cae9-ed71-3100-32be2a87d13f@oktetlabs.ru>
 <CAC-YWqgqOFqXHm9N5VxzkjtdfhoF8ZgqPQNA1rFFbdNpu6D1BQ@mail.gmail.com>
 <74cec43e-43d5-eb4c-caa2-8ebada2680c1@oktetlabs.ru>
 <6f77b754-4b12-4e64-e25d-3037af475410@oktetlabs.ru>
 <CAC-YWqg608zsEMdkKyVi+DGG6VJhUT9dBKRs5kXWorUmw1qrow@mail.gmail.com>
 <172ce67d-087f-ac0d-9840-a76cba2a10c0@oktetlabs.ru>
In-Reply-To: <172ce67d-087f-ac0d-9840-a76cba2a10c0@oktetlabs.ru>
From: Adam Hassick <ahassick@iol.unh.edu>
Date: Thu, 31 Aug 2023 15:38:16 -0400
Message-ID: <CAC-YWqgrTuaeWSDn9iJSdYpR_Y4akwfAhuoCQQt2rkfB_eB4_g@mail.gmail.com>
Subject: Re: Setting up DPDK PMD Test Suite
To: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Cc: Patrick Robb <probb@iol.unh.edu>,
 Konstantin Ushakov <Konstantin.Ushakov@oktetlabs.ru>, ci@dpdk.org
Content-Type: multipart/alternative; boundary="0000000000009dd73806043d2dac"
X-BeenThere: ci@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK CI discussions <ci.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/ci>,
 <mailto:ci-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/ci/>
List-Post: <mailto:ci@dpdk.org>
List-Help: <mailto:ci-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/ci>,
 <mailto:ci-request@dpdk.org?subject=subscribe>
Errors-To: ci-bounces@dpdk.org

--0000000000009dd73806043d2dac
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Andrew,

I have one additional question as well: Does the test engine support
running tests on two ARMv8 test agents?

1. We'll sort out warnings this week. Thanks for heads up.
>

Great. Let me know when that's fixed.

Support for old LTS branches was dropped some time ago, but in the future
> it is definitely possible to keep it for new LTS branches. I think 22.11 =
is
> supported, but I'm not sure about older LTS releases.
>

Good to know.


> 2. You can add command-line option --sanity to run tests marked with
> TEST_HARNESS_SANITY requirement (see dpdk-ethdev-ts/scripts/run.sh and gr=
ep
> TEST_HARNESS_SANITY dpdk-ethdev-ts to see which tests are marked). Yes,
> there is a space for terminology improvement here. We'll do it.
>
> Also it takes a lot of time because of failures and tests which wait for
> some timeout.
>

That makes sense to me. We'll use the time to complete tests on virtio or
the Intel devices as a reference for how long the tests really take to
complete.
We will explore the possibility of periodically running the sanity tests
for patches.


> The test harness can provide coverage reports based on gcov, but I'm not
> sure what you mean by a "dial" to control test coverage. Provided reports
> are rather for human to analyze.
>

The general idea is to have some kind of parameter on the test suite, which
could be an integer ranging from zero to ten, that controls how many tests
are run based on how important the test is.

Similar to how some command line interfaces provide a verbosity level
parameter (some number of "-v" arguments) to control the importance of the
information in the log.
The verbosity level zero only prints very important log messages, while ten
prints everything.

In much the same manner as above, this "dial" parameter controls what tests
are run and with what parameters based on how important those tests and
test parameter combinations are.
Coverage Level zero tells the suite to run a very basic set of important
tests, with minimal parameterization. This mode would take only ~5-10
minutes to run.
In contrast, Coverage Level ten includes all the edge cases, every
combination of test parameters, everything the test suite can do, which
takes the normal several hours to run.
The values 1 - 9 are between those two extremes, allowing the user to get a
gradient of test coverage in the results and to limit the running time.

Then we could, for example, run the "run.sh" with a level of 2 or 3 for
incoming patches that need quick results, and with a level of 10 for the
less often run periodic tests performed on main or LTS branches.


> 3. Yes, really many tests on Mellanox CX5 NICs report unexpected testing
> results. Unfortunately it is time consuming to fill in expectations
> database since it is necessary to analyze testing results and classify if
> it is a bug or just acceptable behaviour aspect.
>
> Bublik allows to compare results of two runs. It is useful for human, but
> still not good for automation.
>
> I have local patch for mlx5 driver which reports Tx ring size maximum. It
> makes pass rate higher. It is a problem for test harness that mlx5 does n=
ot
> report limits right now.
>
> Pass rate on Intel X710 is about 92% on my test rig. Pass rate on virtio
> net is 99% right now and could be done 100% easily (just one thing to fix
> in expectations).
>
> I think logs storage setup is essential for logs analysis. Of course, you
> can request HTML logs when you run tests (--log-html=3Dhtml) or generate
> after run using dpdk-ethdev-ts/scripts/html-log.sh and open index.html in=
 a
> browser, but logs storage makes it more convenient.
>

We are interested in setting up Bublik, potentially as an externally-facing
component, once we have our process of running the test suite stabilized.
Once we are able to run the test suite again, I'll see what the pass rate
is on our other hardware.
Good to know that it isn't an issue with our dev testbed causing the high
fail rate.

For Intel hardware, we have an XL710 and an Intel E810-C in our development
testbed. Although they are slightly different devices, ideally the pass
rate will be identical or similar. I have yet to set up a VM pair for
virtio, but we will soon.

Latest version of test-environment has examples of our CGI scripts which we
> use for log storage (see tools/log_server/README.md).
>
> Also all bits for Jenkins setup are available. See
> dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
> ts-rigs-sample.
>

Jenkins integration, setting up production rig configurations, and
permanent log storage will be our next steps once I am able to run the
tests again.
Unless there is an easy way to have meson not pass "-Werror" into GCC. Then
I would be able to run the test suite.

Thanks,
Adam



>
> On 8/29/23 17:02, Adam Hassick wrote:
>
> Hi Andrew,
>
> That fix seems to have resolved the issue, thanks for the quick turnaroun=
d
> time on that patch.
> Now that we have the RCF timeout issue resolved, there are a few other
> questions and issues that we have about the tests themselves.
>
> 1. The test suite fails to build with a couple warnings.
>
> Below is the stderr log from compilation:
>
> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>> -fdiagnostics-color=3Dalways -pipe -D_FILE_OFFSET_BITS=3D64 -Wall -Winva=
lid-pch
>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>> ../../lib/dpdk_pmd_ts.c: In function
>> =E2=80=98test_create_traffic_generator_params=E2=80=99:
>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and n=
o
>> format arguments [-Werror=3Dformat-security]
>> 5577 |     rc =3D te_kvpair_add(result, buf, mode);
>> |     ^~
>> cc1: all warnings being treated as errors
>> ninja: build stopped: subcommand failed.
>> ninja: Entering directory `.'
>> FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
>> cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
>> -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
>> -fdiagnostics-color=3Dalways -pipe -D_FILE_OFFSET_BITS=3D64 -Wall -Winva=
lid-pch
>> -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o '
>> lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
>> ../../lib/dpdk_pmd_ts.c: In function
>> =E2=80=98test_create_traffic_generator_params=E2=80=99:
>> ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and n=
o
>> format arguments [-Werror=3Dformat-security]
>> 5577 |     rc =3D te_kvpair_add(result, buf, mode);
>> |     ^~
>> cc1: all warnings being treated as errors
>>
>
> This error wasn't occurring last week, which was the last time I ran the
> tests.
> The TE host and the DUT have GCC v9.4.0 installed, and the tester has GCC
> v11.4.0 installed, if this information is helpful.
>
> 2. On the Mellanox CX5s, there are over 6,000 tests run, which
> collectively take around 9 hours. Is it possible, and would it make sense=
,
> to lower the test coverage and have the test suite run faster?
>
> For some context, we run immediate testing on incoming patches for DPDK
> main and development branches, as well as periodic test runs on the main,
> stable, and LTS branches.
> For us to consider including this test suite as part of our immediate
> testing on patches, we would have to reduce the test coverage to the most
> important tests.
> This is primarily to reduce the testing time to, for example, less than 3=
0
> minutes. Testing on patches can't take too long because the lab can recei=
ve
> numerous patches each day, which each require individual testing runs.
>
> At what frequency we run these tests, and on what, still needs to be
> discussed with the DPDK community, but it would be nice to know if the te=
st
> suite had a "dial" to control the testing coverage.
>
> 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300 o=
f
> ~6,600 tests passed. Is there anything we can do to diagnose these test
> failures?
>
> Thanks,
> Adam
>
>
> On Tue, Aug 29, 2023 at 8:07=E2=80=AFAM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru> wrote:
>
>> Hi Adam,
>>
>> I've pushed the fix in main branch and a new tag v1.18.1. It should solv=
e
>> the problem with IPv6 address from DNS.
>>
>> Andrew.
>>
>> On 8/29/23 00:05, Andrew Rybchenko wrote:
>>
>> Hi Adam,
>>
>> > Does the test engine prefer to use IPv6 over IPv4 for initiating the
>> RCF connection to the test bed hosts? And if so, is there a way to force=
 it
>> to use IPv4?
>>
>> Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your case=
,
>> I guess it is the root cause of the problem.
>> Of course, it is TE problem since I see really weird code in
>> lib/comm_net_engine/comm_net_engine.c line 135.
>>
>> I've pushed fix to the branch user/arybchik/fix_ipv4_only in
>> ts-factory/test-environment repository. Please, try.
>>
>> It is late night fix with minimal testing and no review. I'll pass it
>> through review process tomorrow and
>> hopefully it will be released in one-two days.
>>
>> Andrew.
>>
>> On 8/28/23 18:02, Adam Hassick wrote:
>>
>> Hi Andrew,
>>
>> We have yet to notice a distinct pattern with the failures. Sometimes,
>> the RCF will start and connect without issue a few times in a row before
>> failing to connect again. Once the issue begins to occur, neither reboot=
ing
>> all of the hosts (test engine VM, tester, IUT) or deleting all of the bu=
ild
>> directories (suites, agents, inst) and rebooting the hosts afterward
>> resolves the issue. When it begins working again seems very arbitrary to=
 us.
>>
>> I do usually try to terminate the test engine with Ctrl+C, but when it
>> hangs while trying to start RCF, that does not work.
>>
>> Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF
>> connection to the test bed hosts? And if so, is there a way to force it =
to
>> use IPv4?
>>
>>  - Adam
>>
>> On Fri, Aug 25, 2023 at 1:35=E2=80=AFPM Andrew Rybchenko <
>> andrew.rybchenko@oktetlabs.ru> wrote:
>>
>>> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>
>>> Done. It works fine for me without any issues.
>>>
>>> Have you noticed any pattern when it works or does not work?
>>> May be it is a problem of not clean state after termination?
>>> Does it work fine the first time after DUTs reboot?
>>> How do you terminate testing? It should be done using Ctrl+C in termina=
l
>>> where you execute run.sh command.
>>>  In this case it should shutdown gracefully and close all test agents
>>> and engine applications.
>>>
>>> (I'm trying to understand why you've seen many test agent processes. It
>>> should not happen.)
>>>
>>> Andrew.
>>>
>>> On 8/25/23 17:41, Andrew Rybchenko wrote:
>>>
>>> On 8/25/23 17:06, Adam Hassick wrote:
>>>
>>> Hi Andrew,
>>>
>>> Two of our systems (the Test Engine runner and the DUT host) are runnin=
g
>>> Ubuntu 20.04 LTS, however this morning I noticed that the tester system
>>> (the one having issues) is running Ubuntu 22.04 LTS.
>>> This could be the source of the problem. I encountered a dependency
>>> issue trying to run the Test Engine on 22.04 LTS, so I downgraded the
>>> system. Since the tester is also the host having connection issues, I w=
ill
>>> try downgrading that system to 20.04, and see if that changes anything.
>>>
>>>
>>> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu
>>> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
>>> Right now Debian 11 is used for test engine in nightly regressions.
>>>
>>> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.
>>>
>>> I did try passing in the "--vg-rcf" argument to the run.sh script of th=
e
>>> test suite after installing valgrind, but there was no additional outpu=
t
>>> that I saw.
>>>
>>>
>>> Sorry, I should valgrind output should be in valgrind.te_rcf (direction
>>> where you run test engine).
>>>
>>>
>>> I will try pulling in the changes you've pushed up, and will see if tha=
t
>>> fixes anything.
>>>
>>> Thanks,
>>> Adam
>>>
>>> On Fri, Aug 25, 2023 at 9:57=E2=80=AFAM Andrew Rybchenko <
>>> andrew.rybchenko@oktetlabs.ru> wrote:
>>>
>>>> Hello Adam,
>>>>
>>>> On 8/24/23 23:54, Andrew Rybchenko wrote:
>>>>
>>>> I'd like to try to repeat the problem locally. Which Linux distro is
>>>> running on test engine and agents?
>>>>
>>>> In fact I know one problem with Debian 12 and Fedora 38 and we have
>>>> patch in review to fix it, however, the behaviour is different in
>>>> this case, so it is unlike the same problem.
>>>>
>>>>
>>>> I've just published a new tag which fixes known test engine side
>>>> problems on Debian 12 and Fedora 38.
>>>>
>>>>
>>>> One more idea is to install valgrind on the test engine host and
>>>> run with option --vg-rcf to check if something weird is happening.
>>>>
>>>> What I don't understand right now is why I see just one failed attempt
>>>> to connect in your log.txt and then Logger shutdown after 9 minutes.
>>>>
>>>> Andrew.
>>>>
>>>> On 8/24/23 23:29, Adam Hassick wrote:
>>>>
>>>>  > Is there any firewall in the network or on test hosts which could
>>>> block incoming TCP connection to the port 23571
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where
>>>> you run test engine?
>>>>
>>>> Our test engine host and the testbed are on the same subnet. The
>>>> connection does work sometimes.
>>>>
>>>>  > If behaviour the same on the next try and you see that test agent i=
s
>>>> kept running, could you check using
>>>>  >
>>>>  > # netstat -tnlp
>>>>  >
>>>>  > that Test Agent is listening on the port and try to establish TCP
>>>> connection from test agent using
>>>>  >
>>>>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>  >
>>>>  > and check if TCP connection could be established.
>>>>
>>>> I was able to replicate the same behavior again, where it hangs while
>>>> RCF is trying to start.
>>>> Running this command, I see this in the output:
>>>>
>>>> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
>>>> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>>>>  18599/ta
>>>>
>>>> So it seems like it is listening on the correct port.
>>>> Additionally, I was able to connect to the Tester machine from our Tes=
t
>>>> Engine host using telnet. It printed the PID of the process once the
>>>> connection was opened.
>>>>
>>>> I tried running the "ta" application manually on the command line, and
>>>> it didn't print anything at all.
>>>> Maybe the issue is something on the Test Engine side.
>>>>
>>>> On Thu, Aug 24, 2023 at 2:35=E2=80=AFPM Andrew Rybchenko <
>>>> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>     Hi Adam,
>>>>
>>>>      > On the tester host (which appears to be the Peer agent), there
>>>>     are four processes that I see running, which look like the test
>>>>     agent processes.
>>>>
>>>>     Before the next try I'd recommend to kill these processes.
>>>>
>>>>     Is there any firewall in the network or on test hosts which could
>>>>     block incoming TCP connection to the port 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>>>>     where you run test engine?
>>>>
>>>>     If behaviour the same on the next try and you see that test agent
>>>> is
>>>>     kept running, could you check using
>>>>
>>>>     # netstat -tnlp
>>>>
>>>>     that Test Agent is listening on the port and try to establish TCP
>>>>     connection from test agent using
>>>>
>>>>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>     and check if TCP connection could be established.
>>>>
>>>>     Another idea is to login Tester under root as testing does, get
>>>>     start TA command from the log and try it by hands without -n and
>>>>     remove extra escaping.
>>>>
>>>>     # sudo PATH=3D${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>>>>
>>>> LD_LIBRARY_PATH=3D${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x8=
6_root_76872_1692885663_1
>>>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
>>>> host=3Diol-dts-tester.dpdklab.iol.unh.edu:
>>>> port=3D23571:user=3Droot:key=3D/opt/tsf/keys/id_ed25519:ssh_port=3D22:=
copy_timeout=3D15:kill_timeout=3D15:sudo=3D:shell=3D
>>>>
>>>>     Hopefully in this case test agent directory remains in the /tmp an=
d
>>>>     you don't need to copy it as testing does.
>>>>     May be output could shed some light on what's going on.
>>>>
>>>>     Andrew.
>>>>
>>>>     On 8/24/23 17:30, Adam Hassick wrote:
>>>>
>>>>     Hi Andrew,
>>>>
>>>>     This is the output that I see in the terminal when this failure
>>>>     occurs, after the test agent binaries build and the test engine
>>>>     starts:
>>>>
>>>>     Platform default build - pass
>>>>     Simple RCF consistency check succeeded
>>>>     --->>> Starting Logger...done
>>>>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>>>>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>>>>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>>>>
>>>>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>>>>     processes. I let it hang for around 9 minutes.
>>>>
>>>>     On the tester host (which appears to be the Peer agent), there are
>>>>     four processes that I see running, which look like the test agent
>>>>     processes.
>>>>
>>>>     ta.Peer is an empty file. I've attached the log.txt from this run.
>>>>
>>>>      - Adam
>>>>
>>>>     On Thu, Aug 24, 2023 at 4:22=E2=80=AFAM Andrew Rybchenko
>>>>     <andrew.rybchenko@oktetlabs.ru
>>>>     <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>         Hi Adam,
>>>>
>>>>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>>>>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>>>>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>>>>         says that copy_timeout is in seconds and implementation in
>>>>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>>>>         Theoretically select() could be interrupted by signal, but I
>>>>         think it is unlikely here.
>>>>
>>>>         I'm not sure that I understand what do you mean by RCF
>>>>         connection timeout. Does it happen on TE startup when RCF
>>>>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>>>>         does it happen when tests are in progress, e.g. in the middle
>>>>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>>>>         likely either host with test agent dies or test agent itself
>>>>         crashes. It would be easier for me if classify it if you share
>>>>         text log (log.txt, full or just corresponding fragment with
>>>>         some context). Also content of ta.DPDK or ta.Peer file
>>>>         depending on which agent has problems could shed some light.
>>>>         Corresponding files contain stdout/stderr of test agents.
>>>>
>>>>         Andrew.
>>>>
>>>>         On 8/23/23 17:45, Adam Hassick wrote:
>>>>
>>>>         Hi Andrew,
>>>>
>>>>         I've set up a test rig repository here, and have created
>>>>         configurations for our development testbed based off of the
>>>>         examples.
>>>>         We've been able to get the test suite to run manually on
>>>>         Mellanox CX5 devices once.
>>>>         However, we are running into an issue where, when RCF starts,
>>>>         the RCF connection times out very frequently. We aren't sure
>>>>         why this is the case.
>>>>         It works sometimes, but most of the time when we try to run
>>>>         the test engine, it encounters this issue.
>>>>         I've tried changing the RCF port by setting
>>>>         "TE_RCF_PORT=3D<some port number>" and rebooting the testbed
>>>>         machines. Neither seems to fix the issue.
>>>>
>>>>         It also seems like the timeout takes far longer than 60
>>>>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=3D60"
>>>>         before I try to run the test suite.
>>>>         I assume the unit for this variable is seconds?
>>>>
>>>>         Thanks,
>>>>         Adam
>>>>
>>>>         On Mon, Aug 21, 2023 at 10:19=E2=80=AFAM Adam Hassick
>>>>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>> wrote:
>>>>
>>>>             Hi Andrew,
>>>>
>>>>             Thanks, I've cloned the example repository and will start
>>>>             setting up a configuration for our development testbed
>>>>             today. I'll let you know if I run into any difficulties
>>>>             or have any questions.
>>>>
>>>>              - Adam
>>>>
>>>>             On Sun, Aug 20, 2023 at 4:40=E2=80=AFAM Andrew Rybchenko
>>>>             <andrew.rybchenko@oktetlabs.ru
>>>>             <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 I've published
>>>>                 https://github.com/ts-factory/ts-rigs-sample
>>>>                 <https://github.com/ts-factory/ts-rigs-sample>
>>>> <https://github.com/ts-factory/ts-rigs-sample>.
>>>>                 Hopefully it will help to define your test rigs and
>>>>                 successfully run some tests manually. Feel free to
>>>>                 ask any questions and I'll answer here and try to
>>>>                 update documentation.
>>>>
>>>>                 Meanwhile I'll prepare missing bits for steps (2) and
>>>>                 (3).
>>>>                 Hopefully everything is in place for step (4), but we
>>>>                 need to make steps (2) and (3) first.
>>>>
>>>>                 Andrew.
>>>>
>>>>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>>>>
>>>>                 Hi Adam,
>>>>
>>>>                 > I've conferred with the rest of the team, and we
>>>>                 think it would be best to move forward with mainly
>>>>                 option B.
>>>>
>>>>                 OK, I'll provide the sample on Monday for you. It is
>>>>                 almost ready right now, but I need to double-check
>>>>                 it before publishing.
>>>>
>>>>                 Regards,
>>>>                 Andrew.
>>>>
>>>>                 On 8/17/23 20:03, Adam Hassick wrote:
>>>>
>>>>                 Hi Andrew,
>>>>
>>>>                 I'm adding the CI mailing list to this
>>>>                 conversation. Others in the community might find
>>>>                 this conversation valuable.
>>>>
>>>>                 We do want to run testing on a regular basis. The
>>>>                 Jenkins integration will be very useful for us, as
>>>>                 most of our CI is orchestrated by Jenkins.
>>>>                 I've conferred with the rest of the team, and we
>>>>                 think it would be best to move forward with mainly
>>>>                 option B.
>>>>                 If you would like to know anything about our
>>>>                 testbeds that would help you with creating an
>>>>                 example ts-rigs repo, I'd be happy to answer any
>>>>                 questions you have.
>>>>
>>>>                 We have multiple test rigs (we call these
>>>>                 "DUT-tester pairs") that we run our existing
>>>>                 hardware testing on, with differing network
>>>>                 hardware and CPU architecture. I figured this might
>>>>                 be an important detail.
>>>>
>>>>                 Thanks,
>>>>                 Adam
>>>>
>>>>                 On Thu, Aug 17, 2023 at 11:44=E2=80=AFAM Andrew Rybche=
nko
>>>>                 <andrew.rybchenko@oktetlabs.ru
>>>>                 <mailto:andrew.rybchenko@oktetlabs.ru>
>>>> <andrew.rybchenko@oktetlabs.ru>> wrote:
>>>>
>>>>                     Greatings Adam,
>>>>
>>>>                     I'm happy to hear that you're trying to bring
>>>>                     it up.
>>>>
>>>>                     As I understand the final goal is to run it on
>>>>                     regular basis. So, we need to make it properly
>>>>                     from the very beginning.
>>>>                     Bring up of all features consists of 4 steps:
>>>>
>>>>                     1. Create site-specific repository (we call it
>>>>                     ts-rigs) which contains information about test
>>>>                     rigs and other site-specific information like
>>>>                     where to send mails, where to store logs etc.
>>>>                     It is required for manual execution as well,
>>>>                     since test rigs description is essential. I'll
>>>>                     return to the topic below.
>>>>
>>>>                     2. Setup logs storage for automated runs.
>>>>                     Basically it is a disk space plus apache2 web
>>>>                     server with few CGI scripts which help a lot to
>>>>                     save disk space.
>>>>
>>>>                     3. Setup Bublik web application which provides
>>>>                     web interface to view testing results. Same as
>>>>                     https://ts-factory.io/bublik
>>>>                     <https://ts-factory.io/bublik>
>>>> <https://ts-factory.io/bublik>
>>>>
>>>>                     4. Setup Jenkins to run tests on regularly,
>>>>                     save logs in log storage (2) and import it to
>>>>                     bublik (3).
>>>>
>>>>                     Last few month we spent on our homework to make
>>>>                     it simpler to bring up automated execution
>>>>                     using Jenkins -
>>>>                     https://github.com/ts-factory/te-jenkins
>>>>                     <https://github.com/ts-factory/te-jenkins>
>>>> <https://github.com/ts-factory/te-jenkins>
>>>>                     Corresponding bits in dpdk-ethdev-ts will be
>>>>                     available tomorrow.
>>>>
>>>>                     Let's return to the step (1).
>>>>
>>>>                     Unfortunately there is no publicly available
>>>>                     example of the ts-rigs repository since
>>>>                     sensitive site-specific information is located
>>>>                     there. But I'm ready to help you to create it
>>>>                     for UNH. I see two options here:
>>>>
>>>>                     (A) I'll ask questions and based on your
>>>>                     answers will create the first draft with my
>>>>                     comments.
>>>>
>>>>                     (B) I'll make a template/example ts-rigs repo,
>>>>                     publish it and you'll create UNH ts-rigs based
>>>>                     on it.
>>>>
>>>>                     Of course, I'll help to debug and finally bring
>>>>                     it up in any case.
>>>>
>>>>                     (A) is a bit simpler for me and you, but (B) is
>>>>                     a bit more generic and will help other
>>>>                     potential users to bring it up.
>>>>                     We can combine (A)+(B). I.e. start from (A).
>>>>                     What do you think?
>>>>
>>>>                     Thanks,
>>>>                     Andrew.
>>>>
>>>>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>>>>
>>>>                     Greetings Adam,
>>>>
>>>>
>>>>                     Thanks for contacting us. I copy Andrew who
>>>>                     would be happy to help
>>>>
>>>>                     Thanks,
>>>>                     Konstantin
>>>>
>>>>                     On 16 Aug 2023, at 21:50, Adam Hassick
>>>>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>>>>                     <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu> wrote:
>>>>
>>>>                     =EF=BB=BF
>>>>                     Greetings Konstantin,
>>>>
>>>>                     I am in the process of setting up the DPDK
>>>>                     Poll Mode Driver test suite as an addition to
>>>>                     our testing coverage for DPDK at the UNH lab.
>>>>
>>>>                     I have some questions about how to set the
>>>>                     test suite arguments.
>>>>
>>>>                     I have been able to configure the Test Engine
>>>>                     to connect to the hosts in the testbed. The
>>>>                     RCF, Configurator, and Tester all begin to
>>>>                     run, however the prelude of the test suite
>>>>                     fails to run.
>>>>
>>>>
>>>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>>>>
>>>>                     The documentation mentions that there are
>>>>                     several test parameters for the test suite,
>>>>                     like for the IUT test link MAC, etc. These
>>>>                     seem like they would need to be set somewhere
>>>>                     to run many of the tests.
>>>>
>>>>                     I see in the Test Engine documentation, there
>>>>                     are instructions on how to create new
>>>>                     parameters for test suites in the Tester
>>>>                     configuration, but there is nothing in the
>>>>                     user guide or in the Tester guide for how to
>>>>                     set the arguments for the parameters when
>>>>                     running the test suite that I can find. I'm
>>>>                     not sure if I need to write my own Tester
>>>>                     config, or if I should be setting these in
>>>>                     some other way.
>>>>
>>>>                     How should these values be set?
>>>>
>>>>                     I'm also not sure what environment
>>>>                     variables/arguments are strictly necessary or
>>>>                     which are optional.
>>>>
>>>>                     Regards,
>>>>                     Adam
>>>>
>>>>                     --                     *Adam Hassick*
>>>>                     Senior Developer
>>>>                     UNH InterOperability Lab
>>>>                     ahassick@iol.unh.edu
>>>>                     <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>                     iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>                     +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>                 --                 *Adam Hassick*
>>>>                 Senior Developer
>>>>                 UNH InterOperability Lab
>>>>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>                 iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>                 +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>             --             *Adam Hassick*
>>>>             Senior Developer
>>>>             UNH InterOperability Lab
>>>>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>             iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>             +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>         --         *Adam Hassick*
>>>>         Senior Developer
>>>>         UNH InterOperability Lab
>>>>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>         iol.unh.edu <https://www.iol.unh.edu/>
>>>> <https://www.iol.unh.edu/>
>>>>         +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>>     --     *Adam Hassick*
>>>>     Senior Developer
>>>>     UNH InterOperability Lab
>>>>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>>     +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Adam Hassick*
>>>> Senior Developer
>>>> UNH InterOperability Lab
>>>> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
>>>> <ahassick@iol.unh.edu>
>>>> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>>>> +1 (603) 475-8248
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> *Adam Hassick*
>>> Senior Developer
>>> UNH InterOperability Lab
>>> ahassick@iol.unh.edu
>>> iol.unh.edu <https://www.iol.unh.edu/>
>>> +1 (603) 475-8248
>>>
>>>
>>>
>>>
>>
>> --
>> *Adam Hassick*
>> Senior Developer
>> UNH InterOperability Lab
>> ahassick@iol.unh.edu
>> iol.unh.edu <https://www.iol.unh.edu/>
>> +1 (603) 475-8248
>>
>>
>>
>>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu
> iol.unh.edu <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>

--0000000000009dd73806043d2dac
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Andrew,<br></div><div><br>I have one additional qu=
estion as well: Does the test engine support running tests on two ARMv8 tes=
t agents?</div><div><br></div><div><blockquote class=3D"gmail_quote" style=
=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding=
-left:1ex">1. We&#39;ll sort out warnings this week.
      Thanks for heads up.<br></blockquote><div><br></div><div>Great. Let m=
e know when that&#39;s fixed.</div></div><div><br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex"><div>Support for old LTS branches was drop=
ped some time ago, but in the
      future it is definitely possible to keep it for new LTS branches.
      I think 22.11 is supported, but I&#39;m not sure about older LTS
      releases.</div></blockquote><div><br></div><div>Good to know.<br><div=
>=C2=A0<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.=
8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
      2. You can add command-line option --sanity to run tests marked
      with TEST_HARNESS_SANITY requirement (see
      dpdk-ethdev-ts/scripts/run.sh and grep TEST_HARNESS_SANITY
      dpdk-ethdev-ts to see which tests are marked). Yes, there is a
      space for terminology improvement here. We&#39;ll do it.<br><br>
      Also it takes a lot of time because of failures and tests which
      wait for some timeout.<br></blockquote>
      </div><div><br></div><div>That makes sense to me. We&#39;ll use the t=
ime to complete tests on virtio or the Intel devices as a reference for how=
 long the tests really take to complete.<br></div><div>We will explore the =
possibility of periodically running the sanity tests for patches.<br></div>=
<div>=C2=A0<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p=
x 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
      The test harness can provide coverage reports based on gcov, but
      I&#39;m not sure what you mean by a &quot;dial&quot; to control test =
coverage.
      Provided reports are rather for human to analyze.<br></blockquote></d=
iv><div><br></div><div>The general idea is to have some kind of parameter o=
n the test suite, which could be an integer ranging from zero to ten, that =
controls how many tests are run based on how important the test is.<br><br>=
</div><div>Similar to how some command line interfaces provide a verbosity =
level=20
parameter (some number of &quot;-v&quot; arguments) to control the importan=
ce of=20
the information in the log.<br></div>The verbosity level zero only prints v=
ery important log messages, while ten prints everything.<br></div><div><br>=
In much the same manner as above, this &quot;dial&quot; parameter controls =
what tests are run and with what parameters based on how important those te=
sts and test parameter combinations are.<br>Coverage Level zero tells the s=
uite to run a very basic set of important tests, with minimal parameterizat=
ion. This mode would take only ~5-10 minutes to run.<br>In contrast, Covera=
ge Level ten includes all the edge cases, every combination of test paramet=
ers, everything the test suite can do, which takes the normal several hours=
 to run.<br>The values 1 - 9 are between those two extremes, allowing the u=
ser to get a gradient of test coverage in the results and to limit the runn=
ing time.<br><br></div>Then we could, for example, run the &quot;run.sh&quo=
t; with a level of 2 or 3 for incoming patches that need quick results, and=
 with a level of 10 for the less often run periodic tests performed on main=
 or LTS branches.<br><div><div><div>=C2=A0</div><blockquote class=3D"gmail_=
quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,=
204);padding-left:1ex"><div><div>
      3. Yes, really many tests on Mellanox CX5 NICs report unexpected
      testing results. Unfortunately it is time consuming to fill in
      expectations database since it is necessary to analyze testing
      results and classify if it is a bug or just acceptable behaviour
      aspect.<br>
      <br>
      Bublik allows to compare results of two runs. It is useful for
      human, but still not good for automation.<br>
      <br>
      I have local patch for mlx5 driver which reports Tx ring size
      maximum. It makes pass rate higher. It is a problem for test
      harness that mlx5 does not report limits right now.<br>
      <br>
      Pass rate on Intel X710 is about 92% on my test rig. Pass rate on
      virtio net is 99% right now and could be done 100% easily (just
      one thing to fix in expectations).<br>
      <br>
      I think logs storage setup is essential for logs analysis. Of
      course, you can request HTML logs when you run tests
      (--log-html=3Dhtml) or generate after run using
      dpdk-ethdev-ts/scripts/html-log.sh and open index.html in a
      browser, but logs storage makes it more convenient.<br></div></div></=
blockquote><div><br>We are interested in setting up Bublik, potentially as =
an externally-facing component, once we have our process of running the tes=
t suite stabilized.</div><div>Once we are able to run the test suite again,=
 I&#39;ll see what the pass rate is on our other hardware.<br>Good to know =
that it isn&#39;t an issue with our dev testbed causing the high fail rate.=
</div></div><div><div><br></div><div>For Intel hardware, we have an XL710 a=
nd an Intel E810-C in our development testbed. Although they are slightly d=
ifferent devices, ideally the pass rate will be identical or similar. I hav=
e yet to set up a VM pair for virtio, but we will soon.<br></div><div><br><=
blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l=
eft:1px solid rgb(204,204,204);padding-left:1ex">
      Latest version of test-environment has examples of our CGI scripts
      which we use for log storage (see tools/log_server/README.md).<br><br=
>
      Also all bits for Jenkins setup are available. See
      dpdk-ethdev-ts/jenkins/README.md and examples of jenkins files in
      ts-rigs-sample.<br></blockquote>
      </div><div><br></div><div>Jenkins integration, setting up production =
rig configurations, and permanent log storage will be our next steps once I=
 am able to run the tests again.<br></div><div>Unless there is an easy way =
to have meson not pass &quot;-Werror&quot; into GCC. Then I would be able t=
o run the test suite.<br></div><div><br></div><div>Thanks,<br></div><div>Ad=
am<br></div><div><br></div><div>=C2=A0</div><blockquote class=3D"gmail_quot=
e" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)=
;padding-left:1ex"><div><div>
      <br>
      On 8/29/23 17:02, Adam Hassick wrote:<br>
    </div>
    <blockquote type=3D"cite">
     =20
      <div dir=3D"ltr">
        <div>
          <div>
            <div>Hi Andrew,<br>
              <br>
            </div>
            That fix seems to have resolved the issue, thanks for the
            quick turnaround time on that patch.<br>
          </div>
          <div>Now that we have the RCF timeout issue resolved, there
            are a few other questions and issues that we have about the
            tests themselves.</div>
          <br>
        </div>
        <div>1. The test suite fails to build with a couple warnings.<br>
        </div>
        <div><br>
        </div>
        <div>Below is the stderr log from compilation:<br>
        </div>
        <br>
        <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex=
;border-left:1px solid rgb(204,204,204);padding-left:1ex">FAILED:
          <a href=3D"mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o" t=
arget=3D"_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><br>
          cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
          -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
          -fdiagnostics-color=3Dalways -pipe -D_FILE_OFFSET_BITS=3D64 -Wall
          -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
          -fPIC -MD -MQ &#39;<a href=3D"mailto:lib/76b5a35@@ts_dpdk_pmd@sta=
/dpdk_pmd_ts.c.o" target=3D"_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_t=
s.c.o</a>&#39;
          -MF &#39;<a href=3D"mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_=
ts.c.o.d" target=3D"_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d<=
/a>&#39; -o
          &#39;<a href=3D"mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c=
.o" target=3D"_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>&#39;=
 -c
          ../../lib/dpdk_pmd_ts.c<br>
          ../../lib/dpdk_pmd_ts.c: In function
          =E2=80=98test_create_traffic_generator_params=E2=80=99:<br>
          ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
          literal and no format arguments [-Werror=3Dformat-security]<br>
          5577 | =C2=A0 =C2=A0 rc =3D te_kvpair_add(result, buf, mode);<br>
          | =C2=A0 =C2=A0 ^~<br>
          cc1: all warnings being treated as errors<br>
          ninja: build stopped: subcommand failed.<br>
          ninja: Entering directory `.&#39;<br>
          FAILED: <a href=3D"mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_t=
s.c.o" target=3D"_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a><b=
r>
          cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib
          -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include
          -fdiagnostics-color=3Dalways -pipe -D_FILE_OFFSET_BITS=3D64 -Wall
          -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W
          -fPIC -MD -MQ &#39;<a href=3D"mailto:lib/76b5a35@@ts_dpdk_pmd@sta=
/dpdk_pmd_ts.c.o" target=3D"_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_t=
s.c.o</a>&#39;
          -MF &#39;<a href=3D"mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_=
ts.c.o.d" target=3D"_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d<=
/a>&#39; -o
          &#39;<a href=3D"mailto:lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c=
.o" target=3D"_blank">lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o</a>&#39;=
 -c
          ../../lib/dpdk_pmd_ts.c<br>
          ../../lib/dpdk_pmd_ts.c: In function
          =E2=80=98test_create_traffic_generator_params=E2=80=99:<br>
          ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string
          literal and no format arguments [-Werror=3Dformat-security]<br>
          5577 | =C2=A0 =C2=A0 rc =3D te_kvpair_add(result, buf, mode);<br>
          | =C2=A0 =C2=A0 ^~<br>
          cc1: all warnings being treated as errors<br>
        </blockquote>
        <div>
          <div>
            <div><br>
            </div>
            <div>This error wasn&#39;t occurring last week, which was the
              last time I ran the tests.<br>
            </div>
            <div>The TE host and the DUT have GCC v9.4.0 installed, and
              the tester has GCC v11.4.0 installed, if this information
              is helpful.<br>
            </div>
            <div><br>
            </div>
            <div>2. On the Mellanox CX5s, there are over 6,000 tests
              run, which collectively take around 9 hours. Is it
              possible, and would it make sense, to lower the test
              coverage and have the test suite run faster?<br>
              <br>
            </div>
            <div>For some context, we run immediate testing on incoming
              patches for DPDK main and development branches, as well as
              periodic test runs on the main, stable, and LTS branches.<br>
            </div>
            <div>For us to consider including this test suite as part of
              our immediate testing on patches, we would have to reduce
              the test coverage to the most important tests.<br>
              This is primarily to reduce the testing time to, for
              example, less than 30 minutes. Testing on patches can&#39;t
              take too long because the lab can receive numerous patches
              each day, which each require individual testing runs.<br>
              <br>
            </div>
            <div>At what frequency we run these tests, and on what,
              still needs to be discussed with the DPDK community, but
              it would be nice to know if the test suite had a &quot;dial&q=
uot; to
              control the testing coverage.<br>
            </div>
            <div><br>
            </div>
            <div>3. We see a lot of test failures on our Mellanox CX5
              NICs. Around 2,300 of ~6,600 tests passed. Is there
              anything we can do to diagnose these test failures?<br>
            </div>
            <div><br>
            </div>
            <div>Thanks,<br>
            </div>
            <div>Adam<br>
            </div>
            <div><br>
            </div>
          </div>
        </div>
      </div>
      <br>
      <div class=3D"gmail_quote">
        <div dir=3D"ltr" class=3D"gmail_attr">On Tue, Aug 29, 2023 at
          8:07=E2=80=AFAM Andrew Rybchenko &lt;<a href=3D"mailto:andrew.ryb=
chenko@oktetlabs.ru" target=3D"_blank">andrew.rybchenko@oktetlabs.ru</a>&gt=
;
          wrote:<br>
        </div>
        <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex=
;border-left:1px solid rgb(204,204,204);padding-left:1ex">
          <div>
            <div>Hi Adam,<br>
              <br>
              I&#39;ve pushed the fix in main branch and a new tag v1.18.1.
              It should solve the problem with IPv6 address from DNS.<br>
              <br>
              Andrew.<br>
              <br>
              On 8/29/23 00:05, Andrew Rybchenko wrote:<br>
            </div>
            <blockquote type=3D"cite">
              <div>Hi Adam,<br>
                <br>
                &gt; Does the test engine prefer to use IPv6 over IPv4
                for initiating the RCF connection to the test bed hosts?
                And if so, is there a way to force it to use IPv4?<br>
                <br>
                Brilliant idea. If DNS returns both IPv4 and IPv6
                addresses in your case, I guess it is the root cause of
                the problem.<br>
                Of course, it is TE problem since I see really weird
                code in lib/comm_net_engine/comm_net_engine.c line 135.<br>
                <br>
                I&#39;ve pushed fix to the branch
                user/arybchik/fix_ipv4_only in
                ts-factory/test-environment repository. Please, try.<br>
                <br>
                It is late night fix with minimal testing and no review.
                I&#39;ll pass it through review process tomorrow and<br>
                hopefully it will be released in one-two days.<br>
                <br>
                Andrew.<br>
                <br>
                On 8/28/23 18:02, Adam Hassick wrote:<br>
              </div>
              <blockquote type=3D"cite">
                <div dir=3D"ltr">
                  <div>
                    <div>
                      <div>Hi Andrew,<br>
                        <br>
                      </div>
                      We have yet to notice a distinct pattern with the
                      failures. Sometimes, the RCF will start and
                      connect without issue a few times in a row before
                      failing to connect again. Once the issue begins to
                      occur, neither rebooting all of the hosts (test
                      engine VM, tester, IUT) or deleting all of the
                      build directories (suites, agents, inst) and
                      rebooting the hosts afterward resolves the issue.
                      When it begins working again seems very arbitrary
                      to us.<br>
                      <br>
                    </div>
                    <div>I do usually try to terminate the test engine
                      with Ctrl+C, but when it hangs while trying to
                      start RCF, that does not work.<br>
                    </div>
                    <div><br>
                    </div>
                    <div>Does the test engine prefer to use IPv6 over
                      IPv4 for initiating the RCF connection to the test
                      bed hosts? And if so, is there a way to force it
                      to use IPv4?<br>
                      <br>
                    </div>
                    <div>=C2=A0- Adam<br>
                    </div>
                  </div>
                </div>
                <br>
                <div class=3D"gmail_quote">
                  <div dir=3D"ltr" class=3D"gmail_attr">On Fri, Aug 25, 202=
3
                    at 1:35=E2=80=AFPM Andrew Rybchenko &lt;<a href=3D"mail=
to:andrew.rybchenko@oktetlabs.ru" target=3D"_blank">andrew.rybchenko@oktetl=
abs.ru</a>&gt;
                    wrote:<br>
                  </div>
                  <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px=
 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                    <div>
                      <div>&gt; I&#39;ll double-check test engine on Ubuntu
                        20.04 and Ubuntu 22.04.<br>
                        <br>
                        Done. It works fine for me without any issues.<br>
                        <br>
                        Have you noticed any pattern when it works or
                        does not work?<br>
                        May be it is a problem of not clean state after
                        termination?<br>
                        Does it work fine the first time after DUTs
                        reboot?<br>
                        How do you terminate testing? It should be done
                        using Ctrl+C in terminal where you execute
                        run.sh command.<br>
                        =C2=A0In this case it should shutdown gracefully an=
d
                        close all test agents and engine applications.<br>
                        <br>
                        (I&#39;m trying to understand why you&#39;ve seen m=
any
                        test agent processes. It should not happen.)<br>
                        <br>
                        Andrew.<br>
                        <br>
                        On 8/25/23 17:41, Andrew Rybchenko wrote:<br>
                      </div>
                      <blockquote type=3D"cite">
                        <div>On 8/25/23 17:06, Adam Hassick wrote:<br>
                        </div>
                        <blockquote type=3D"cite">
                          <div dir=3D"ltr">
                            <div>
                              <div>Hi Andrew,<br>
                                <br>
                              </div>
                              Two of our systems (the Test Engine runner
                              and the DUT host) are running Ubuntu 20.04
                              LTS, however this morning I noticed that
                              the tester system (the one having issues)
                              is running Ubuntu 22.04 LTS.<br>
                            </div>
                            <div>This could be the source of the
                              problem. I encountered a dependency issue
                              trying to run the Test Engine on 22.04
                              LTS, so I downgraded the system. Since the
                              tester is also the host having connection
                              issues, I will try downgrading that system
                              to 20.04, and see if that changes
                              anything.<br>
                            </div>
                          </div>
                        </blockquote>
                        <br>
                        Unlikely, but who knows. We run tests (DUTs) on
                        Ubuntu 20.04, Ubuntu 22.04, Ubuntu 22.10, Ubuntu
                        23.04, Debian 11 and Fedora 38 every night.<br>
                        Right now Debian 11 is used for test engine in
                        nightly regressions.<br>
                        <br>
                        I&#39;ll double-check test engine on Ubuntu 20.04
                        and Ubuntu 22.04.<br>
                        <br>
                        <blockquote type=3D"cite">
                          <div dir=3D"ltr">
                            <div>I did try passing in the &quot;--vg-rcf&qu=
ot;
                              argument to the run.sh script of the test
                              suite after installing valgrind, but there
                              was no additional output that I saw.<br>
                            </div>
                          </div>
                        </blockquote>
                        <br>
                        Sorry, I should valgrind output should be in
                        valgrind.te_rcf (direction where you run test
                        engine).<br>
                        <br>
                        <blockquote type=3D"cite">
                          <div dir=3D"ltr">
                            <div><br>
                            </div>
                            <div>I will try pulling in the changes
                              you&#39;ve pushed up, and will see if that
                              fixes anything.<br>
                              <br>
                            </div>
                            <div>Thanks,<br>
                            </div>
                            <div>Adam<br>
                            </div>
                          </div>
                          <br>
                          <div class=3D"gmail_quote">
                            <div dir=3D"ltr" class=3D"gmail_attr">On Fri,
                              Aug 25, 2023 at 9:57=E2=80=AFAM Andrew Rybche=
nko
                              &lt;<a href=3D"mailto:andrew.rybchenko@oktetl=
abs.ru" target=3D"_blank">andrew.rybchenko@oktetlabs.ru</a>&gt;
                              wrote:<br>
                            </div>
                            <blockquote class=3D"gmail_quote" style=3D"marg=
in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e=
x">
                              <div>
                                <div>Hello Adam, <br>
                                  <br>
                                  On 8/24/23 23:54, Andrew Rybchenko
                                  wrote:<br>
                                </div>
                                <blockquote type=3D"cite">I&#39;d like to t=
ry
                                  to repeat the problem locally. Which
                                  Linux distro is running on test engine
                                  and agents? <br>
                                  <br>
                                  In fact I know one problem with Debian
                                  12 and Fedora 38 and we have <br>
                                  patch in review to fix it, however,
                                  the behaviour is different in <br>
                                  this case, so it is unlike the same
                                  problem. <br>
                                </blockquote>
                                <br>
                                I&#39;ve just published a new tag which
                                fixes known test engine side problems on
                                Debian 12 and Fedora 38.<br>
                                <br>
                                <blockquote type=3D"cite"> <br>
                                  One more idea is to install valgrind
                                  on the test engine host and <br>
                                  run with option --vg-rcf to check if
                                  something weird is happening. <br>
                                  <br>
                                  What I don&#39;t understand right now is
                                  why I see just one failed attempt <br>
                                  to connect in your log.txt and then
                                  Logger shutdown after 9 minutes. <br>
                                  <br>
                                  Andrew. <br>
                                  <br>
                                  On 8/24/23 23:29, Adam Hassick wrote:
                                  <br>
                                  <blockquote type=3D"cite">=C2=A0&gt; Is t=
here
                                    any firewall in the network or on
                                    test hosts which could block
                                    incoming TCP connection to the port
                                    23571 <a href=3D"http://iol-dts-tester.=
dpdklab.iol.unh.edu:23571" target=3D"_blank">&lt;http://iol-dts-tester.dpdk=
lab.iol.unh.edu:23571&gt;</a>
                                    from the host where you run test
                                    engine? <br>
                                    <br>
                                    Our test engine host and the testbed
                                    are on the same subnet. The
                                    connection does work sometimes. <br>
                                    <br>
                                    =C2=A0&gt; If behaviour the same on the
                                    next try and you see that test agent
                                    is kept running, could you check
                                    using <br>
                                    =C2=A0&gt; <br>
                                    =C2=A0&gt; # netstat -tnlp <br>
                                    =C2=A0&gt; <br>
                                    =C2=A0&gt; that Test Agent is listening
                                    on the port and try to establish TCP
                                    connection from test agent using <br>
                                    =C2=A0&gt; <br>
                                    =C2=A0&gt; $ telnet <a href=3D"http://i=
ol-dts-tester.dpdklab.iol.unh.edu" target=3D"_blank">iol-dts-tester.dpdklab=
.iol.unh.edu</a>
                                    <a href=3D"http://iol-dts-tester.dpdkla=
b.iol.unh.edu:23571" target=3D"_blank">&lt;http://iol-dts-tester.dpdklab.io=
l.unh.edu:23571&gt;</a>
                                    23571 <a href=3D"http://iol-dts-tester.=
dpdklab.iol.unh.edu:23571" target=3D"_blank">&lt;http://iol-dts-tester.dpdk=
lab.iol.unh.edu:23571&gt;</a>
                                    <br>
                                    =C2=A0&gt; <br>
                                    =C2=A0&gt; and check if TCP connection
                                    could be established. <br>
                                    <br>
                                    I was able to replicate the same
                                    behavior again, where it hangs while
                                    RCF is trying to start. <br>
                                    Running this command, I see this in
                                    the output: <br>
                                    <br>
                                    tcp =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0=
 =C2=A0 =C2=A00 <a href=3D"http://0.0.0.0:23571" target=3D"_blank">0.0.0.0:=
23571</a>
                                    <a href=3D"http://0.0.0.0:23571" target=
=3D"_blank">&lt;http://0.0.0.0:23571&gt;</a>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                                    0.0.0.0:* =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 LISTEN =C2=A0 =C2=A0
                                    =C2=A018599/ta <br>
                                    <br>
                                    So it seems like it is listening on
                                    the correct port. <br>
                                    Additionally, I was able to connect
                                    to the Tester machine from our Test
                                    Engine host using telnet. It printed
                                    the PID of the process once the
                                    connection was opened. <br>
                                    <br>
                                    I tried running the &quot;ta&quot; appl=
ication
                                    manually on the command line, and it
                                    didn&#39;t print anything at all. <br>
                                    Maybe the issue is something on the
                                    Test Engine side. <br>
                                    <br>
                                    On Thu, Aug 24, 2023 at 2:35=E2=80=AFPM
                                    Andrew Rybchenko &lt;<a href=3D"mailto:=
andrew.rybchenko@oktetlabs.ru" target=3D"_blank">andrew.rybchenko@oktetlabs=
.ru</a>
                                    <a href=3D"mailto:andrew.rybchenko@okte=
tlabs.ru" target=3D"_blank">&lt;mailto:andrew.rybchenko@oktetlabs.ru&gt;</a=
>&gt;
                                    wrote: <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 Hi Adam, <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0=C2=A0 &gt; On the te=
ster host (which
                                    appears to be the Peer agent), there
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 are four processes t=
hat I see
                                    running, which look like the test <br>
                                    =C2=A0=C2=A0=C2=A0 agent processes. <br=
>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 Before the next try =
I&#39;d
                                    recommend to kill these processes. <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 Is there any firewal=
l in the
                                    network or on test hosts which could
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 block incoming TCP c=
onnection to
                                    the port 23571 <br>
                                    =C2=A0=C2=A0=C2=A0 <a href=3D"http://io=
l-dts-tester.dpdklab.iol.unh.edu:23571" target=3D"_blank">&lt;http://iol-dt=
s-tester.dpdklab.iol.unh.edu:23571&gt;</a>
                                    from the host <br>
                                    =C2=A0=C2=A0=C2=A0 where you run test e=
ngine? <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 If behaviour the sam=
e on the
                                    next try and you see that test agent
                                    is <br>
                                    =C2=A0=C2=A0=C2=A0 kept running, could =
you check
                                    using <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 # netstat -tnlp <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 that Test Agent is l=
istening on
                                    the port and try to establish TCP <br>
                                    =C2=A0=C2=A0=C2=A0 connection from test=
 agent using
                                    <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 $ telnet <a href=3D"=
http://iol-dts-tester.dpdklab.iol.unh.edu" target=3D"_blank">iol-dts-tester=
.dpdklab.iol.unh.edu</a>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 <a href=3D"http://io=
l-dts-tester.dpdklab.iol.unh.edu:23571" target=3D"_blank">&lt;http://iol-dt=
s-tester.dpdklab.iol.unh.edu:23571&gt;</a>
                                    23571 <br>
                                    =C2=A0=C2=A0=C2=A0 <a href=3D"http://io=
l-dts-tester.dpdklab.iol.unh.edu:23571" target=3D"_blank">&lt;http://iol-dt=
s-tester.dpdklab.iol.unh.edu:23571&gt;</a>
                                    <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 and check if TCP con=
nection
                                    could be established. <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 Another idea is to l=
ogin Tester
                                    under root as testing does, get <br>
                                    =C2=A0=C2=A0=C2=A0 start TA command fro=
m the log
                                    and try it by hands without -n and <br>
                                    =C2=A0=C2=A0=C2=A0 remove extra escapin=
g. <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 # sudo
                                    PATH=3D${PATH}:/tmp/linux_x86_root_7687=
2_1692885663_1
                                    <br>
                                    =C2=A0=C2=A0=C2=A0
LD_LIBRARY_PATH=3D${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_roo=
t_76872_1692885663_1
/tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=3Diol-dts-tester.dpdklab.iol.unh.edu:port=3D23571:user=3Droot:key=3D/o=
pt/tsf/keys/id_ed25519:ssh_port=3D22:copy_timeout=3D15:kill_timeout=3D15:su=
do=3D:shell=3D<br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 Hopefully in this ca=
se test
                                    agent directory remains in the /tmp
                                    and <br>
                                    =C2=A0=C2=A0=C2=A0 you don&#39;t need t=
o copy it as
                                    testing does. <br>
                                    =C2=A0=C2=A0=C2=A0 May be output could =
shed some
                                    light on what&#39;s going on. <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 Andrew. <br>
                                    <br>
                                    =C2=A0=C2=A0=C2=A0 On 8/24/23 17:30, Ad=
am Hassick
                                    wrote: <br>
                                    <blockquote type=3D"cite">=C2=A0=C2=A0=
=C2=A0 Hi
                                      Andrew, <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 This is the output=
 that I see
                                      in the terminal when this failure
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 occurs, after the =
test agent
                                      binaries build and the test engine
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 starts: <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 Platform default b=
uild - pass
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 Simple RCF consist=
ency check
                                      succeeded <br>
                                      =C2=A0=C2=A0=C2=A0 ---&gt;&gt;&gt; St=
arting
                                      Logger...done <br>
                                      =C2=A0=C2=A0=C2=A0 ---&gt;&gt;&gt; St=
arting
                                      RCF...rcf_net_engine_connect():
                                      Connection timed <br>
                                      =C2=A0=C2=A0=C2=A0 out <a href=3D"htt=
p://iol-dts-tester.dpdklab.iol.unh.edu:23571" target=3D"_blank">iol-dts-tes=
ter.dpdklab.iol.unh.edu:23571</a>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 <a href=3D"http://=
iol-dts-tester.dpdklab.iol.unh.edu:23571" target=3D"_blank">&lt;http://iol-=
dts-tester.dpdklab.iol.unh.edu:23571&gt;</a>
                                      <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 Then, it hangs her=
e until I
                                      kill the &quot;te_rcf&quot; and &quot=
;te_tee&quot; <br>
                                      =C2=A0=C2=A0=C2=A0 processes. I let i=
t hang for
                                      around 9 minutes. <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 On the tester host=
 (which
                                      appears to be the Peer agent),
                                      there are <br>
                                      =C2=A0=C2=A0=C2=A0 four processes tha=
t I see
                                      running, which look like the test
                                      agent <br>
                                      =C2=A0=C2=A0=C2=A0 processes. <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 ta.Peer is an empt=
y file. I&#39;ve
                                      attached the log.txt from this
                                      run. <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 =C2=A0- Adam <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 On Thu, Aug 24, 20=
23 at
                                      4:22=E2=80=AFAM Andrew Rybchenko <br>
                                      =C2=A0=C2=A0=C2=A0 &lt;<a href=3D"mai=
lto:andrew.rybchenko@oktetlabs.ru" target=3D"_blank">andrew.rybchenko@oktet=
labs.ru</a>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 <a href=3D"mailto:=
andrew.rybchenko@oktetlabs.ru" target=3D"_blank">&lt;mailto:andrew.rybchenk=
o@oktetlabs.ru&gt;</a>&gt;
                                      wrote: <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 Hi Adam, <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 Yes, TE_RCFUNIX_TIMEOUT is
                                      in seconds. I&#39;ve double-checked <=
br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 that it goes to
                                      &#39;copy_timeout&#39; in
                                      ts-conf/rcf.conf. <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 Description in in
                                      doc/sphinx/pages/group_te_engine_rcf.=
rst
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 says that copy_timeout is
                                      in seconds and implementation in <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 lib/rcfunix/rcfunix.c
                                      passes the value to select()
                                      tv_sec. <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 Theoretically select()
                                      could be interrupted by signal,
                                      but I <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 think it is unlikely here.
                                      <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 I&#39;m not sure that I
                                      understand what do you mean by RCF
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 connection timeout. Does
                                      it happen on TE startup when RCF <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 starts test agents. If so,
                                      TE_RCFUNIX_TIMEOUT could help. Or
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 does it happen when tests
                                      are in progress, e.g. in the
                                      middle <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 of a test. If so,
                                      TE_RCFUNIX_TIMEOUT is unrelated
                                      and most <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 likely either host with
                                      test agent dies or test agent
                                      itself <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 crashes. It would be
                                      easier for me if classify it if
                                      you share <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 text log (log.txt, full or
                                      just corresponding fragment with <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 some context). Also
                                      content of ta.DPDK or ta.Peer file
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 depending on which agent
                                      has problems could shed some
                                      light. <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 Corresponding files
                                      contain stdout/stderr of test
                                      agents. <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 Andrew. <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 On 8/23/23 17:45, Adam
                                      Hassick wrote: <br>
                                      <blockquote type=3D"cite">=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi
                                        Andrew, <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 I&#39;ve set up a test rig
                                        repository here, and have
                                        created <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 configurations for our
                                        development testbed based off of
                                        the <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 examples. <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 We&#39;ve been able to get
                                        the test suite to run manually
                                        on <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Mellanox CX5 devices
                                        once. <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 However, we are running
                                        into an issue where, when RCF
                                        starts, <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 the RCF connection times
                                        out very frequently. We aren&#39;t
                                        sure <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 why this is the case. <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 It works sometimes, but
                                        most of the time when we try to
                                        run <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 the test engine, it
                                        encounters this issue. <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 I&#39;ve tried changing the
                                        RCF port by setting <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 &quot;TE_RCF_PORT=3D&lt;some
                                        port number&gt;&quot; and rebooting
                                        the testbed <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 machines. Neither seems
                                        to fix the issue. <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 It also seems like the
                                        timeout takes far longer than 60
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 seconds, even when
                                        running &quot;export
                                        TE_RCFUNIX_TIMEOUT=3D60&quot; <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 before I try to run the
                                        test suite. <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 I assume the unit for
                                        this variable is seconds? <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Thanks, <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Adam <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 On Mon, Aug 21, 2023 at
                                        10:19=E2=80=AFAM Adam Hassick <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 &lt;<a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">aha=
ssick@iol.unh.edu</a>
                                        <a href=3D"mailto:ahassick@iol.unh.=
edu" target=3D"_blank">&lt;mailto:ahassick@iol.unh.edu&gt;</a>&gt;
                                        wrote: <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Andrew, <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Thanks, I&#39;ve cloned
                                        the example repository and will
                                        start <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 setting up a
                                        configuration for our
                                        development testbed <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 today. I&#39;ll let you
                                        know if I run into any
                                        difficulties <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 or have any
                                        questions. <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0- Adam <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On Sun, Aug 20, 2023
                                        at 4:40=E2=80=AFAM Andrew Rybchenko=
 <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &lt;<a href=3D"mailto:andrew.rybchenko@ok=
tetlabs.ru" target=3D"_blank">andrew.rybchenko@oktetlabs.ru</a>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"mailto:andrew.rybchenko@oktetl=
abs.ru" target=3D"_blank">&lt;mailto:andrew.rybchenko@oktetlabs.ru&gt;</a>&=
gt;
                                        wrote: <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Adam, <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;ve publishe=
d <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"https:=
//github.com/ts-factory/ts-rigs-sample" target=3D"_blank">https://github.co=
m/ts-factory/ts-rigs-sample</a>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"https:=
//github.com/ts-factory/ts-rigs-sample" target=3D"_blank">&lt;https://githu=
b.com/ts-factory/ts-rigs-sample&gt;</a>.
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hopefully it
                                        will help to define your test
                                        rigs and <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 successfully run
                                        some tests manually. Feel free
                                        to <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ask any
                                        questions and I&#39;ll answer here
                                        and try to <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 update
                                        documentation. <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Meanwhile I&#39;l=
l
                                        prepare missing bits for steps
                                        (2) and <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (3). <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hopefully
                                        everything is in place for step
                                        (4), but we <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 need to make
                                        steps (2) and (3) first. <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Andrew. <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 8/18/23
                                        21:40, Andrew Rybchenko wrote: <br>
                                        <blockquote type=3D"cite">=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0
                                          Hi Adam, <br>
                                          <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &gt; I&#39;ve
                                          conferred with the rest of the
                                          team, and we <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 think it would
                                          be best to move forward with
                                          mainly <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 option B. <br>
                                          <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 OK, I&#39;ll
                                          provide the sample on Monday
                                          for you. It is <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 almost ready
                                          right now, but I need to
                                          double-check <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 it before
                                          publishing. <br>
                                          <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Regards, <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Andrew. <br>
                                          <br>
                                          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 8/17/23
                                          20:03, Adam Hassick wrote: <br>
                                          <blockquote type=3D"cite">=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            Hi Andrew, <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;m adding
                                            the CI mailing list to this
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                                            conversation. Others in the
                                            community might find <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 this
                                            conversation valuable. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We do want
                                            to run testing on a regular
                                            basis. The <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Jenkins
                                            integration will be very
                                            useful for us, as <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 most of our
                                            CI is orchestrated by
                                            Jenkins. <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;ve
                                            conferred with the rest of
                                            the team, and we <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 think it
                                            would be best to move
                                            forward with mainly <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 option B. <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 If you would
                                            like to know anything about
                                            our <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 testbeds
                                            that would help you with
                                            creating an <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 example
                                            ts-rigs repo, I&#39;d be happy
                                            to answer any <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 questions
                                            you have. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We have
                                            multiple test rigs (we call
                                            these <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &quot;DUT-test=
er
                                            pairs&quot;) that we run our
                                            existing <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hardware
                                            testing on, with differing
                                            network <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hardware and
                                            CPU architecture. I figured
                                            this might <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 be an
                                            important detail. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Thanks, <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Adam <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On Thu, Aug
                                            17, 2023 at 11:44=E2=80=AFAM An=
drew
                                            Rybchenko <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &lt;<a href=3D=
"mailto:andrew.rybchenko@oktetlabs.ru" target=3D"_blank">andrew.rybchenko@o=
ktetlabs.ru</a>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"mai=
lto:andrew.rybchenko@oktetlabs.ru" target=3D"_blank">&lt;mailto:andrew.rybc=
henko@oktetlabs.ru&gt;</a>&gt;
                                            wrote: <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            Greatings Adam, <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 I&#39;m
                                            happy to hear that you&#39;re
                                            trying to bring <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 it up. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 As I
                                            understand the final goal is
                                            to run it on <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 regular
                                            basis. So, we need to make
                                            it properly <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 from the
                                            very beginning. <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Bring up
                                            of all features consists of
                                            4 steps: <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 1.
                                            Create site-specific
                                            repository (we call it <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 ts-rigs)
                                            which contains information
                                            about test <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 rigs and
                                            other site-specific
                                            information like <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 where to
                                            send mails, where to store
                                            logs etc. <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 It is
                                            required for manual
                                            execution as well, <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 since
                                            test rigs description is
                                            essential. I&#39;ll <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 return
                                            to the topic below. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 2. Setup
                                            logs storage for automated
                                            runs. <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            Basically it is a disk space
                                            plus apache2 web <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 server
                                            with few CGI scripts which
                                            help a lot to <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 save
                                            disk space. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 3. Setup
                                            Bublik web application which
                                            provides <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 web
                                            interface to view testing
                                            results. Same as <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 <a href=3D"https://ts-factory.io/bublik" target=3D"_blank">https:=
//ts-factory.io/bublik</a>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 <a href=3D"https://ts-factory.io/bublik" target=3D"_blank">&lt;ht=
tps://ts-factory.io/bublik&gt;</a>
                                            <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 4. Setup
                                            Jenkins to run tests on
                                            regularly, <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 save
                                            logs in log storage (2) and
                                            import it to <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 bublik
                                            (3). <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Last few
                                            month we spent on our
                                            homework to make <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 it
                                            simpler to bring up
                                            automated execution <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 using
                                            Jenkins - <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 <a href=3D"https://github.com/ts-factory/te-jenkins" target=3D"_b=
lank">https://github.com/ts-factory/te-jenkins</a>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 <a href=3D"https://github.com/ts-factory/te-jenkins" target=3D"_b=
lank">&lt;https://github.com/ts-factory/te-jenkins&gt;</a>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            Corresponding bits in
                                            dpdk-ethdev-ts will be <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            available tomorrow. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Let&#39;s
                                            return to the step (1). <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            Unfortunately there is no
                                            publicly available <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 example
                                            of the ts-rigs repository
                                            since <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            sensitive site-specific
                                            information is located <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 there.
                                            But I&#39;m ready to help you t=
o
                                            create it <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 for UNH.
                                            I see two options here: <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 (A) I&#39;ll
                                            ask questions and based on
                                            your <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 answers
                                            will create the first draft
                                            with my <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            comments. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 (B) I&#39;ll
                                            make a template/example
                                            ts-rigs repo, <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 publish
                                            it and you&#39;ll create UNH
                                            ts-rigs based <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 on it. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Of
                                            course, I&#39;ll help to debug
                                            and finally bring <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 it up in
                                            any case. <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 (A) is a
                                            bit simpler for me and you,
                                            but (B) is <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 a bit
                                            more generic and will help
                                            other <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0
                                            potential users to bring it
                                            up. <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 We can
                                            combine (A)+(B). I.e. start
                                            from (A). <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 What do
                                            you think? <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Thanks,
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Andrew.
                                            <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 On
                                            8/17/23 15:18, Konstantin
                                            Ushakov wrote: <br>
                                            <blockquote type=3D"cite">=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                                              Greetings Adam, <br>
                                              <br>
                                              <br>
                                              =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 Thanks
                                              for contacting us. I copy
                                              Andrew who <br>
                                              =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 would
                                              be happy to help <br>
                                              <br>
                                              =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                              Thanks, <br>
                                              =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                              Konstantin <br>
                                              <br>
                                              <blockquote type=3D"cite">=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                                                On 16 Aug 2023, at
                                                21:50, Adam Hassick <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">&lt;=
ahassick@iol.unh.edu&gt;</a>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">&lt;=
mailto:ahassick@iol.unh.edu&gt;</a>
                                                wrote: <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 =EF=BB=BF <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                Greetings Konstantin, <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 I am
                                                in the process of
                                                setting up the DPDK <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 Poll
                                                Mode Driver test suite
                                                as an addition to <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 our
                                                testing coverage for
                                                DPDK at the UNH lab. <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 I
                                                have some questions
                                                about how to set the <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 test
                                                suite arguments. <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 I
                                                have been able to
                                                configure the Test
                                                Engine <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 to
                                                connect to the hosts in
                                                the testbed. The <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 RCF,
                                                Configurator, and Tester
                                                all begin to <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 run,
                                                however the prelude of
                                                the test suite <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                fails to run. <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 <a href=3D"https://ts-factory.io/doc/dpdk-ethdev-ts/index.html=
#test-parameters" target=3D"_blank">https://ts-factory.io/doc/dpdk-ethdev-t=
s/index.html#test-parameters</a>
                                                <a href=3D"https://ts-facto=
ry.io/doc/dpdk-ethdev-ts/index.html#test-parameters" target=3D"_blank">&lt;=
https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters&gt;</a>
                                                <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 The
                                                documentation mentions
                                                that there are <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                several test parameters
                                                for the test suite, <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 like
                                                for the IUT test link
                                                MAC, etc. These <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 seem
                                                like they would need to
                                                be set somewhere <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 to
                                                run many of the tests. <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 I
                                                see in the Test Engine
                                                documentation, there <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 are
                                                instructions on how to
                                                create new <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                parameters for test
                                                suites in the Tester <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                configuration, but there
                                                is nothing in the <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 user
                                                guide or in the Tester
                                                guide for how to <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 set
                                                the arguments for the
                                                parameters when <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                running the test suite
                                                that I can find. I&#39;m <b=
r>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 not
                                                sure if I need to write
                                                my own Tester <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                config, or if I should
                                                be setting these in <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 some
                                                other way. <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 How
                                                should these values be
                                                set? <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 I&#39;m
                                                also not sure what
                                                environment <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                variables/arguments are
                                                strictly necessary or <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                which are optional. <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                Regards, <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 Adam
                                                <br>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 --
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                *Adam Hassick* <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0
                                                Senior Developer <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 UNH
                                                InterOperability Lab <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">ahas=
sick@iol.unh.edu</a> <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">&lt;=
mailto:ahassick@iol.unh.edu&gt;</a>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 <a href=3D"http://iol.unh.edu" target=3D"_blank">iol.unh.edu</=
a>
                                                <a href=3D"https://www.iol.=
unh.edu/" target=3D"_blank">&lt;https://www.iol.unh.edu/&gt;</a>
                                                <br>
                                                =C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 +1
                                                (603) 475-8248 <br>
                                              </blockquote>
                                            </blockquote>
                                            <br>
                                            <br>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 --
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Adam
                                            Hassick* <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Senior
                                            Developer <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 UNH
                                            InterOperability Lab <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"mai=
lto:ahassick@iol.unh.edu" target=3D"_blank">ahassick@iol.unh.edu</a>
                                            <a href=3D"mailto:ahassick@iol.=
unh.edu" target=3D"_blank">&lt;mailto:ahassick@iol.unh.edu&gt;</a>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"htt=
p://iol.unh.edu" target=3D"_blank">iol.unh.edu</a>
                                            <a href=3D"https://www.iol.unh.=
edu/" target=3D"_blank">&lt;https://www.iol.unh.edu/&gt;</a>
                                            <br>
                                            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 +1 (603)
                                            475-8248 <br>
                                          </blockquote>
                                          <br>
                                        </blockquote>
                                        <br>
                                        <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Adam
                                        Hassick* <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Senior Developer <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 UNH InterOperability
                                        Lab <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"mailto:ahassick@iol.unh.edu" t=
arget=3D"_blank">ahassick@iol.unh.edu</a>
                                        <a href=3D"mailto:ahassick@iol.unh.=
edu" target=3D"_blank">&lt;mailto:ahassick@iol.unh.edu&gt;</a>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"http://iol.unh.edu" target=3D"=
_blank">iol.unh.edu</a>
                                        <a href=3D"https://www.iol.unh.edu/=
" target=3D"_blank">&lt;https://www.iol.unh.edu/&gt;</a>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 +1 (603) 475-8248 <br>
                                        <br>
                                        <br>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Adam
                                        Hassick* <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 Senior Developer <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 UNH InterOperability Lab
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">ahassic=
k@iol.unh.edu</a>
                                        <a href=3D"mailto:ahassick@iol.unh.=
edu" target=3D"_blank">&lt;mailto:ahassick@iol.unh.edu&gt;</a>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 <a href=3D"http://iol.unh.edu" target=3D"_blank">iol.unh.edu</a>
                                        <a href=3D"https://www.iol.unh.edu/=
" target=3D"_blank">&lt;https://www.iol.unh.edu/&gt;</a>
                                        <br>
                                        =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 +1 (603) 475-8248 <br>
                                      </blockquote>
                                      <br>
                                      <br>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=
=A0 *Adam Hassick* <br>
                                      =C2=A0=C2=A0=C2=A0 Senior Developer <=
br>
                                      =C2=A0=C2=A0=C2=A0 UNH InterOperabili=
ty Lab <br>
                                      =C2=A0=C2=A0=C2=A0 <a href=3D"mailto:=
ahassick@iol.unh.edu" target=3D"_blank">ahassick@iol.unh.edu</a>
                                      <a href=3D"mailto:ahassick@iol.unh.ed=
u" target=3D"_blank">&lt;mailto:ahassick@iol.unh.edu&gt;</a>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 <a href=3D"http://=
iol.unh.edu" target=3D"_blank">iol.unh.edu</a>
                                      <a href=3D"https://www.iol.unh.edu/" =
target=3D"_blank">&lt;https://www.iol.unh.edu/&gt;</a>
                                      <br>
                                      =C2=A0=C2=A0=C2=A0 +1 (603) 475-8248 =
<br>
                                    </blockquote>
                                    <br>
                                    <br>
                                    <br>
                                    --=C2=A0<br>
                                    *Adam Hassick* <br>
                                    Senior Developer <br>
                                    UNH InterOperability Lab <br>
                                    <a href=3D"mailto:ahassick@iol.unh.edu"=
 target=3D"_blank">ahassick@iol.unh.edu</a>
                                    <a href=3D"mailto:ahassick@iol.unh.edu"=
 target=3D"_blank">&lt;mailto:ahassick@iol.unh.edu&gt;</a>
                                    <br>
                                    <a href=3D"http://iol.unh.edu" target=
=3D"_blank">iol.unh.edu</a>
                                    <a href=3D"https://www.iol.unh.edu/" ta=
rget=3D"_blank">&lt;https://www.iol.unh.edu/&gt;</a>
                                    <br>
                                    +1 (603) 475-8248 <br>
                                  </blockquote>
                                  <br>
                                </blockquote>
                                <br>
                              </div>
                            </blockquote>
                          </div>
                          <br clear=3D"all">
                          <br>
                          <span class=3D"gmail_signature_prefix">-- </span>=
<br>
                          <div dir=3D"ltr" class=3D"gmail_signature">
                            <div dir=3D"ltr">
                              <div>
                                <div><b><span style=3D"background-color:rgb=
(255,255,255)"><span style=3D"color:rgb(102,102,102)">Adam
                                        Hassick</span></span></b><br>
                                </div>
                                <span style=3D"color:rgb(102,102,102)"></sp=
an></div>
                              <div><span style=3D"color:rgb(102,102,102)">S=
enior
                                  Developer</span></div>
                              <div><span style=3D"color:rgb(102,102,102)"><=
span style=3D"color:rgb(11,83,148)"><span style=3D"background-color:rgb(255=
,255,255)">UNH
                                      InterOperability Lab</span></span></s=
pan><span style=3D"color:rgb(102,102,102)"></span></div>
                              <div><span style=3D"color:rgb(102,102,102)"><=
a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">ahassick@iol.unh.e=
du</a><br>
                                </span></div>
                              <div><span style=3D"color:rgb(102,102,102)"><=
a href=3D"https://www.iol.unh.edu/" target=3D"_blank">iol.unh.edu</a><br>
                                </span></div>
                              +1 (603) 475-8248<br>
                            </div>
                          </div>
                        </blockquote>
                        <br>
                      </blockquote>
                      <br>
                    </div>
                  </blockquote>
                </div>
                <br clear=3D"all">
                <br>
                <span class=3D"gmail_signature_prefix">-- </span><br>
                <div dir=3D"ltr" class=3D"gmail_signature">
                  <div dir=3D"ltr">
                    <div>
                      <div><b><span style=3D"background-color:rgb(255,255,2=
55)"><span style=3D"color:rgb(102,102,102)">Adam
                              Hassick</span></span></b><br>
                      </div>
                      <span style=3D"color:rgb(102,102,102)"></span></div>
                    <div><span style=3D"color:rgb(102,102,102)">Senior
                        Developer</span></div>
                    <div><span style=3D"color:rgb(102,102,102)"><span style=
=3D"color:rgb(11,83,148)"><span style=3D"background-color:rgb(255,255,255)"=
>UNH
                            InterOperability Lab</span></span></span><span =
style=3D"color:rgb(102,102,102)"></span></div>
                    <div><span style=3D"color:rgb(102,102,102)"><a href=3D"=
mailto:ahassick@iol.unh.edu" target=3D"_blank">ahassick@iol.unh.edu</a><br>
                      </span></div>
                    <div><span style=3D"color:rgb(102,102,102)"><a href=3D"=
https://www.iol.unh.edu/" target=3D"_blank">iol.unh.edu</a><br>
                      </span></div>
                    +1 (603) 475-8248<br>
                  </div>
                </div>
              </blockquote>
              <br>
            </blockquote>
            <br>
          </div>
        </blockquote>
      </div>
      <br clear=3D"all">
      <br>
      <span class=3D"gmail_signature_prefix">-- </span><br>
      <div dir=3D"ltr" class=3D"gmail_signature">
        <div dir=3D"ltr">
          <div>
            <div><b><span style=3D"background-color:rgb(255,255,255)"><span=
 style=3D"color:rgb(102,102,102)">Adam Hassick</span></span></b><br>
            </div>
            <span style=3D"color:rgb(102,102,102)"></span></div>
          <div><span style=3D"color:rgb(102,102,102)">Senior Developer</spa=
n></div>
          <div><span style=3D"color:rgb(102,102,102)"><span style=3D"color:=
rgb(11,83,148)"><span style=3D"background-color:rgb(255,255,255)">UNH
                  InterOperability Lab</span></span></span><span style=3D"c=
olor:rgb(102,102,102)"></span></div>
          <div><span style=3D"color:rgb(102,102,102)"><a href=3D"mailto:aha=
ssick@iol.unh.edu" target=3D"_blank">ahassick@iol.unh.edu</a><br>
            </span></div>
          <div><span style=3D"color:rgb(102,102,102)"><a href=3D"https://ww=
w.iol.unh.edu/" target=3D"_blank">iol.unh.edu</a><br>
            </span></div>
          +1 (603) 475-8248<br>
        </div>
      </div>
    </blockquote>
    <br>
  </div>

</blockquote></div></div></div>

--0000000000009dd73806043d2dac--