From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ci-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
	by inbox.dpdk.org (Postfix) with ESMTP id A80E243101;
	Fri, 25 Aug 2023 16:05:31 +0200 (CEST)
Received: from mails.dpdk.org (localhost [127.0.0.1])
	by mails.dpdk.org (Postfix) with ESMTP id 8302640695;
	Fri, 25 Aug 2023 16:05:31 +0200 (CEST)
Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com
 [209.85.128.173])
 by mails.dpdk.org (Postfix) with ESMTP id CD8A5400D5
 for <ci@dpdk.org>; Fri, 25 Aug 2023 16:05:29 +0200 (CEST)
Received: by mail-yw1-f173.google.com with SMTP id
 00721157ae682-58d40c2debeso11475297b3.2
 for <ci@dpdk.org>; Fri, 25 Aug 2023 07:05:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=iol.unh.edu; s=unh-iol; t=1692972329; x=1693577129;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:from:to:cc:subject:date:message-id:reply-to;
 bh=ANEmiUHybmqlLi9hLJhcMUwgO4wvqKMIZW1ADMo30QI=;
 b=ZcGHFzXmPF0BzxdZcj7+3FtJwGgV0hqtZiHTDo1r7RWDFdGUIntApgUEGnopyx76UM
 rmRCduUbTMdGTkr2o+ecBAFR/FNlDe6UIc0ZlkmJIQyGoUbgT6nQShbpaQhRgJfl2Jdw
 grj/on8C0gYIUotRVKheaRTAVP9rQgdbV/yMo=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20221208; t=1692972329; x=1693577129;
 h=cc:to:subject:message-id:date:from:in-reply-to:references
 :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
 :reply-to;
 bh=ANEmiUHybmqlLi9hLJhcMUwgO4wvqKMIZW1ADMo30QI=;
 b=a5oPhoaNL4Ec3bkslORL49lg5vnRdbbVu54BmqyMLt63kBKx7Nzh7ybuWr6bfmA93H
 bKe3Pco8XMbXRJOA6DstPQCbkUEWYrqjsMZItuThTj+3l6xV2hInnEDTkJiHI7XbdcNQ
 isGvLGUxHLbWGRq9Zv6rI0GFMJIqO+KbeGxWAoD2hZrqOnsPkjwZs+Uh9EdIc7dc1/H0
 An++nXxY/xEkEQwnCu/2EsRSfvtHFAyIKbtr2f6zgT637RdvibqLq9uhyMN/IGiN0cpK
 gRr5NXFvqFhyJA2v8+9pCt+dJ0DdW/65cs8SRrzxC/gaKE7afC0CqTphk1yNos91CWMN
 dsMw==
X-Gm-Message-State: AOJu0YzqPCd8VDWANgixrsN0khAuJQXPMSrQGW1IwIKVjDywnxnCtz86
 MToBvkrX+6DWupcVb1MlCEEZSsXtQhUTm7MjAsjR2SktcKib7QrzTEzbQA==
X-Google-Smtp-Source: AGHT+IESo90MdQbfwL0gXEEwOC0kzdjCioPcShFX15mmO1efez4WgGA0zJlsPkPkNOS1rYy9XxbDL75poJRefAFl5Cw=
X-Received: by 2002:a0d:d712:0:b0:570:28a9:fe40 with SMTP id
 z18-20020a0dd712000000b0057028a9fe40mr18308533ywd.5.1692972328830; Fri, 25
 Aug 2023 07:05:28 -0700 (PDT)
MIME-Version: 1.0
References: <CAC-YWqiQfH4Rx-Et1jGHhGK9i47d0AArKy-B2P77iYbbM+Lpig@mail.gmail.com>
 <C3B08390-DA6D-4BDC-BBD7-98561F92FE33@oktetlabs.ru>
 <35340484-1d7e-7e5f-cad4-c965ba541397@oktetlabs.ru>
 <CAC-YWqhJtLYycknQNLaRL4vT4SE-MmF5gq+wfzs0AZ66ptvmBw@mail.gmail.com>
 <9ce9d7fd-4051-6d51-26bb-7e96e98c677e@oktetlabs.ru>
 <781ca146-955f-85af-5727-66015ae1d326@oktetlabs.ru>
 <CAC-YWqg9LaOXGo9DT0oz0Co6oSqb2QVZTeWrQ2i=Em=R0X-AYQ@mail.gmail.com>
 <CAC-YWqjbaiVOWqb1-1kmp2Ci+VLACVkUygyMb83HfoNHur9NhQ@mail.gmail.com>
 <7734826a-840d-d0d9-e7a5-91951223398c@oktetlabs.ru>
 <CAC-YWqgvgKRgffwWY3mqWbC8o-LJ_o0BkRuuBxDSqg8Pnj0h1Q@mail.gmail.com>
 <9d920676-485d-3b4d-ca20-2b5ea3a5b606@oktetlabs.ru>
 <CAC-YWqhth9q5QDVhtfDoXkfQGwg4b3w2Q0d5EzhYZMFEhhzE6Q@mail.gmail.com>
 <873c7972-3e5a-9e82-9449-4d12b2c96032@oktetlabs.ru>
 <cc758ad7-2f40-7c6c-ffab-c574011ba770@oktetlabs.ru>
In-Reply-To: <cc758ad7-2f40-7c6c-ffab-c574011ba770@oktetlabs.ru>
From: Adam Hassick <ahassick@iol.unh.edu>
Date: Fri, 25 Aug 2023 10:06:00 -0400
Message-ID: <CAC-YWqj28X-YhLzM-rChuKMdaoTrDS6fN68YxzS+rb1N-i5AkQ@mail.gmail.com>
Subject: Re: Setting up DPDK PMD Test Suite
To: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Cc: Patrick Robb <probb@iol.unh.edu>,
 Konstantin Ushakov <Konstantin.Ushakov@oktetlabs.ru>, ci@dpdk.org
Content-Type: multipart/alternative; boundary="0000000000004660700603bfd6d9"
X-BeenThere: ci@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK CI discussions <ci.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/ci>,
 <mailto:ci-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/ci/>
List-Post: <mailto:ci@dpdk.org>
List-Help: <mailto:ci-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/ci>,
 <mailto:ci-request@dpdk.org?subject=subscribe>
Errors-To: ci-bounces@dpdk.org

--0000000000004660700603bfd6d9
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Andrew,

Two of our systems (the Test Engine runner and the DUT host) are running
Ubuntu 20.04 LTS, however this morning I noticed that the tester system
(the one having issues) is running Ubuntu 22.04 LTS.
This could be the source of the problem. I encountered a dependency issue
trying to run the Test Engine on 22.04 LTS, so I downgraded the system.
Since the tester is also the host having connection issues, I will try
downgrading that system to 20.04, and see if that changes anything.

I did try passing in the "--vg-rcf" argument to the run.sh script of the
test suite after installing valgrind, but there was no additional output
that I saw.

I will try pulling in the changes you've pushed up, and will see if that
fixes anything.

Thanks,
Adam

On Fri, Aug 25, 2023 at 9:57=E2=80=AFAM Andrew Rybchenko <
andrew.rybchenko@oktetlabs.ru> wrote:

> Hello Adam,
>
> On 8/24/23 23:54, Andrew Rybchenko wrote:
>
> I'd like to try to repeat the problem locally. Which Linux distro is
> running on test engine and agents?
>
> In fact I know one problem with Debian 12 and Fedora 38 and we have
> patch in review to fix it, however, the behaviour is different in
> this case, so it is unlike the same problem.
>
>
> I've just published a new tag which fixes known test engine side problems
> on Debian 12 and Fedora 38.
>
>
> One more idea is to install valgrind on the test engine host and
> run with option --vg-rcf to check if something weird is happening.
>
> What I don't understand right now is why I see just one failed attempt
> to connect in your log.txt and then Logger shutdown after 9 minutes.
>
> Andrew.
>
> On 8/24/23 23:29, Adam Hassick wrote:
>
>  > Is there any firewall in the network or on test hosts which could bloc=
k
> incoming TCP connection to the port 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host where you
> run test engine?
>
> Our test engine host and the testbed are on the same subnet. The
> connection does work sometimes.
>
>  > If behaviour the same on the next try and you see that test agent is
> kept running, could you check using
>  >
>  > # netstat -tnlp
>  >
>  > that Test Agent is listening on the port and try to establish TCP
> connection from test agent using
>  >
>  > $ telnet iol-dts-tester.dpdklab.iol.unh.edu
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>  >
>  > and check if TCP connection could be established.
>
> I was able to replicate the same behavior again, where it hangs while RCF
> is trying to start.
> Running this command, I see this in the output:
>
> tcp        0      0 0.0.0.0:23571 <http://0.0.0.0:23571>
> <http://0.0.0.0:23571>           0.0.0.0:*               LISTEN
>  18599/ta
>
> So it seems like it is listening on the correct port.
> Additionally, I was able to connect to the Tester machine from our Test
> Engine host using telnet. It printed the PID of the process once the
> connection was opened.
>
> I tried running the "ta" application manually on the command line, and it
> didn't print anything at all.
> Maybe the issue is something on the Test Engine side.
>
> On Thu, Aug 24, 2023 at 2:35=E2=80=AFPM Andrew Rybchenko <
> andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>     Hi Adam,
>
>      > On the tester host (which appears to be the Peer agent), there
>     are four processes that I see running, which look like the test
>     agent processes.
>
>     Before the next try I'd recommend to kill these processes.
>
>     Is there any firewall in the network or on test hosts which could
>     block incoming TCP connection to the port 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> from the host
>     where you run test engine?
>
>     If behaviour the same on the next try and you see that test agent is
>     kept running, could you check using
>
>     # netstat -tnlp
>
>     that Test Agent is listening on the port and try to establish TCP
>     connection from test agent using
>
>     $ telnet iol-dts-tester.dpdklab.iol.unh.edu
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571> 23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
>     and check if TCP connection could be established.
>
>     Another idea is to login Tester under root as testing does, get
>     start TA command from the log and try it by hands without -n and
>     remove extra escaping.
>
>     # sudo PATH=3D${PATH}:/tmp/linux_x86_root_76872_1692885663_1
>
> LD_LIBRARY_PATH=3D${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_r=
oot_76872_1692885663_1
> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
> host=3Diol-dts-tester.dpdklab.iol.unh.edu:
> port=3D23571:user=3Droot:key=3D/opt/tsf/keys/id_ed25519:ssh_port=3D22:cop=
y_timeout=3D15:kill_timeout=3D15:sudo=3D:shell=3D
>
>     Hopefully in this case test agent directory remains in the /tmp and
>     you don't need to copy it as testing does.
>     May be output could shed some light on what's going on.
>
>     Andrew.
>
>     On 8/24/23 17:30, Adam Hassick wrote:
>
>     Hi Andrew,
>
>     This is the output that I see in the terminal when this failure
>     occurs, after the test agent binaries build and the test engine
>     starts:
>
>     Platform default build - pass
>     Simple RCF consistency check succeeded
>     --->>> Starting Logger...done
>     --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
>     out iol-dts-tester.dpdklab.iol.unh.edu:23571
>     <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
> <http://iol-dts-tester.dpdklab.iol.unh.edu:23571>
>
>     Then, it hangs here until I kill the "te_rcf" and "te_tee"
>     processes. I let it hang for around 9 minutes.
>
>     On the tester host (which appears to be the Peer agent), there are
>     four processes that I see running, which look like the test agent
>     processes.
>
>     ta.Peer is an empty file. I've attached the log.txt from this run.
>
>      - Adam
>
>     On Thu, Aug 24, 2023 at 4:22=E2=80=AFAM Andrew Rybchenko
>     <andrew.rybchenko@oktetlabs.ru
>     <mailto:andrew.rybchenko@oktetlabs.ru> <andrew.rybchenko@oktetlabs.ru=
>>
> wrote:
>
>         Hi Adam,
>
>         Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
>         that it goes to 'copy_timeout' in ts-conf/rcf.conf.
>         Description in in doc/sphinx/pages/group_te_engine_rcf.rst
>         says that copy_timeout is in seconds and implementation in
>         lib/rcfunix/rcfunix.c passes the value to select() tv_sec.
>         Theoretically select() could be interrupted by signal, but I
>         think it is unlikely here.
>
>         I'm not sure that I understand what do you mean by RCF
>         connection timeout. Does it happen on TE startup when RCF
>         starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
>         does it happen when tests are in progress, e.g. in the middle
>         of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
>         likely either host with test agent dies or test agent itself
>         crashes. It would be easier for me if classify it if you share
>         text log (log.txt, full or just corresponding fragment with
>         some context). Also content of ta.DPDK or ta.Peer file
>         depending on which agent has problems could shed some light.
>         Corresponding files contain stdout/stderr of test agents.
>
>         Andrew.
>
>         On 8/23/23 17:45, Adam Hassick wrote:
>
>         Hi Andrew,
>
>         I've set up a test rig repository here, and have created
>         configurations for our development testbed based off of the
>         examples.
>         We've been able to get the test suite to run manually on
>         Mellanox CX5 devices once.
>         However, we are running into an issue where, when RCF starts,
>         the RCF connection times out very frequently. We aren't sure
>         why this is the case.
>         It works sometimes, but most of the time when we try to run
>         the test engine, it encounters this issue.
>         I've tried changing the RCF port by setting
>         "TE_RCF_PORT=3D<some port number>" and rebooting the testbed
>         machines. Neither seems to fix the issue.
>
>         It also seems like the timeout takes far longer than 60
>         seconds, even when running "export TE_RCFUNIX_TIMEOUT=3D60"
>         before I try to run the test suite.
>         I assume the unit for this variable is seconds?
>
>         Thanks,
>         Adam
>
>         On Mon, Aug 21, 2023 at 10:19=E2=80=AFAM Adam Hassick
>         <ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>> wrote:
>
>             Hi Andrew,
>
>             Thanks, I've cloned the example repository and will start
>             setting up a configuration for our development testbed
>             today. I'll let you know if I run into any difficulties
>             or have any questions.
>
>              - Adam
>
>             On Sun, Aug 20, 2023 at 4:40=E2=80=AFAM Andrew Rybchenko
>             <andrew.rybchenko@oktetlabs.ru
>             <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>                 Hi Adam,
>
>                 I've published
>                 https://github.com/ts-factory/ts-rigs-sample
>                 <https://github.com/ts-factory/ts-rigs-sample>
> <https://github.com/ts-factory/ts-rigs-sample>.
>                 Hopefully it will help to define your test rigs and
>                 successfully run some tests manually. Feel free to
>                 ask any questions and I'll answer here and try to
>                 update documentation.
>
>                 Meanwhile I'll prepare missing bits for steps (2) and
>                 (3).
>                 Hopefully everything is in place for step (4), but we
>                 need to make steps (2) and (3) first.
>
>                 Andrew.
>
>                 On 8/18/23 21:40, Andrew Rybchenko wrote:
>
>                 Hi Adam,
>
>                 > I've conferred with the rest of the team, and we
>                 think it would be best to move forward with mainly
>                 option B.
>
>                 OK, I'll provide the sample on Monday for you. It is
>                 almost ready right now, but I need to double-check
>                 it before publishing.
>
>                 Regards,
>                 Andrew.
>
>                 On 8/17/23 20:03, Adam Hassick wrote:
>
>                 Hi Andrew,
>
>                 I'm adding the CI mailing list to this
>                 conversation. Others in the community might find
>                 this conversation valuable.
>
>                 We do want to run testing on a regular basis. The
>                 Jenkins integration will be very useful for us, as
>                 most of our CI is orchestrated by Jenkins.
>                 I've conferred with the rest of the team, and we
>                 think it would be best to move forward with mainly
>                 option B.
>                 If you would like to know anything about our
>                 testbeds that would help you with creating an
>                 example ts-rigs repo, I'd be happy to answer any
>                 questions you have.
>
>                 We have multiple test rigs (we call these
>                 "DUT-tester pairs") that we run our existing
>                 hardware testing on, with differing network
>                 hardware and CPU architecture. I figured this might
>                 be an important detail.
>
>                 Thanks,
>                 Adam
>
>                 On Thu, Aug 17, 2023 at 11:44=E2=80=AFAM Andrew Rybchenko
>                 <andrew.rybchenko@oktetlabs.ru
>                 <mailto:andrew.rybchenko@oktetlabs.ru>
> <andrew.rybchenko@oktetlabs.ru>> wrote:
>
>                     Greatings Adam,
>
>                     I'm happy to hear that you're trying to bring
>                     it up.
>
>                     As I understand the final goal is to run it on
>                     regular basis. So, we need to make it properly
>                     from the very beginning.
>                     Bring up of all features consists of 4 steps:
>
>                     1. Create site-specific repository (we call it
>                     ts-rigs) which contains information about test
>                     rigs and other site-specific information like
>                     where to send mails, where to store logs etc.
>                     It is required for manual execution as well,
>                     since test rigs description is essential. I'll
>                     return to the topic below.
>
>                     2. Setup logs storage for automated runs.
>                     Basically it is a disk space plus apache2 web
>                     server with few CGI scripts which help a lot to
>                     save disk space.
>
>                     3. Setup Bublik web application which provides
>                     web interface to view testing results. Same as
>                     https://ts-factory.io/bublik
>                     <https://ts-factory.io/bublik>
> <https://ts-factory.io/bublik>
>
>                     4. Setup Jenkins to run tests on regularly,
>                     save logs in log storage (2) and import it to
>                     bublik (3).
>
>                     Last few month we spent on our homework to make
>                     it simpler to bring up automated execution
>                     using Jenkins -
>                     https://github.com/ts-factory/te-jenkins
>                     <https://github.com/ts-factory/te-jenkins>
> <https://github.com/ts-factory/te-jenkins>
>                     Corresponding bits in dpdk-ethdev-ts will be
>                     available tomorrow.
>
>                     Let's return to the step (1).
>
>                     Unfortunately there is no publicly available
>                     example of the ts-rigs repository since
>                     sensitive site-specific information is located
>                     there. But I'm ready to help you to create it
>                     for UNH. I see two options here:
>
>                     (A) I'll ask questions and based on your
>                     answers will create the first draft with my
>                     comments.
>
>                     (B) I'll make a template/example ts-rigs repo,
>                     publish it and you'll create UNH ts-rigs based
>                     on it.
>
>                     Of course, I'll help to debug and finally bring
>                     it up in any case.
>
>                     (A) is a bit simpler for me and you, but (B) is
>                     a bit more generic and will help other
>                     potential users to bring it up.
>                     We can combine (A)+(B). I.e. start from (A).
>                     What do you think?
>
>                     Thanks,
>                     Andrew.
>
>                     On 8/17/23 15:18, Konstantin Ushakov wrote:
>
>                     Greetings Adam,
>
>
>                     Thanks for contacting us. I copy Andrew who
>                     would be happy to help
>
>                     Thanks,
>                     Konstantin
>
>                     On 16 Aug 2023, at 21:50, Adam Hassick
>                     <ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
> wrote:
>
>                     =EF=BB=BF
>                     Greetings Konstantin,
>
>                     I am in the process of setting up the DPDK
>                     Poll Mode Driver test suite as an addition to
>                     our testing coverage for DPDK at the UNH lab.
>
>                     I have some questions about how to set the
>                     test suite arguments.
>
>                     I have been able to configure the Test Engine
>                     to connect to the hosts in the testbed. The
>                     RCF, Configurator, and Tester all begin to
>                     run, however the prelude of the test suite
>                     fails to run.
>
>
> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters
> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
> <https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>
>
>                     The documentation mentions that there are
>                     several test parameters for the test suite,
>                     like for the IUT test link MAC, etc. These
>                     seem like they would need to be set somewhere
>                     to run many of the tests.
>
>                     I see in the Test Engine documentation, there
>                     are instructions on how to create new
>                     parameters for test suites in the Tester
>                     configuration, but there is nothing in the
>                     user guide or in the Tester guide for how to
>                     set the arguments for the parameters when
>                     running the test suite that I can find. I'm
>                     not sure if I need to write my own Tester
>                     config, or if I should be setting these in
>                     some other way.
>
>                     How should these values be set?
>
>                     I'm also not sure what environment
>                     variables/arguments are strictly necessary or
>                     which are optional.
>
>                     Regards,
>                     Adam
>
>                     --                     *Adam Hassick*
>                     Senior Developer
>                     UNH InterOperability Lab
>                     ahassick@iol.unh.edu
>                     <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
>                     iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>                     +1 (603) 475-8248
>
>
>
>
>                 --                 *Adam Hassick*
>                 Senior Developer
>                 UNH InterOperability Lab
>                 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>                 iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>                 +1 (603) 475-8248
>
>
>
>
>
>             --             *Adam Hassick*
>             Senior Developer
>             UNH InterOperability Lab
>             ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>             iol.unh.edu <https://www.iol.unh.edu/>
> <https://www.iol.unh.edu/>
>             +1 (603) 475-8248
>
>
>
>         --         *Adam Hassick*
>         Senior Developer
>         UNH InterOperability Lab
>         ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>         iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>         +1 (603) 475-8248
>
>
>
>
>     --     *Adam Hassick*
>     Senior Developer
>     UNH InterOperability Lab
>     ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
> <ahassick@iol.unh.edu>
>     iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
>     +1 (603) 475-8248
>
>
>
>
> --
> *Adam Hassick*
> Senior Developer
> UNH InterOperability Lab
> ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu> <ahassick@iol.unh.edu>
> iol.unh.edu <https://www.iol.unh.edu/> <https://www.iol.unh.edu/>
> +1 (603) 475-8248
>
>
>
>

--=20
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu
iol.unh.edu <https://www.iol.unh.edu/>
+1 (603) 475-8248

--0000000000004660700603bfd6d9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi Andrew,<br><br></div>Two of our systems (the =
Test Engine=20
runner and the DUT host) are running Ubuntu 20.04 LTS, however this morning=
 I noticed that the tester system (the one having issues) is running Ubuntu
 22.04 LTS.<br></div><div>This could be the source of the problem. I encoun=
tered a dependency issue trying to run the Test Engine on 22.04 LTS, so I d=
owngraded the system. Since the tester is also the host having connection i=
ssues, I will try downgrading that system to 20.04, and see if that changes=
 anything.<br></div><div><br></div><div>I did try passing in the &quot;--vg=
-rcf&quot; argument to the run.sh script of the test suite after installing=
 valgrind, but there was no additional output that I saw.<br><br></div><div=
>I will try pulling in the changes you&#39;ve pushed up, and will see if th=
at fixes anything.<br><br></div><div>Thanks,<br></div><div>Adam<br></div></=
div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On=
 Fri, Aug 25, 2023 at 9:57=E2=80=AFAM Andrew Rybchenko &lt;<a href=3D"mailt=
o:andrew.rybchenko@oktetlabs.ru">andrew.rybchenko@oktetlabs.ru</a>&gt; wrot=
e:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0=
.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
 =20
   =20
 =20
  <div>
    <div>Hello Adam, <br>
      <br>
      On 8/24/23 23:54, Andrew Rybchenko wrote:<br>
    </div>
    <blockquote type=3D"cite">I&#39;d
      like to try to repeat the problem locally. Which Linux distro is
      running on test engine and agents?
      <br>
      <br>
      In fact I know one problem with Debian 12 and Fedora 38 and we
      have
      <br>
      patch in review to fix it, however, the behaviour is different in
      <br>
      this case, so it is unlike the same problem.
      <br>
    </blockquote>
    <br>
    I&#39;ve just published a new tag which fixes known test engine side
    problems on Debian 12 and Fedora 38.<br>
    <br>
    <blockquote type=3D"cite">
      <br>
      One more idea is to install valgrind on the test engine host and
      <br>
      run with option --vg-rcf to check if something weird is happening.
      <br>
      <br>
      What I don&#39;t understand right now is why I see just one failed
      attempt
      <br>
      to connect in your log.txt and then Logger shutdown after 9
      minutes.
      <br>
      <br>
      Andrew.
      <br>
      <br>
      On 8/24/23 23:29, Adam Hassick wrote:
      <br>
      <blockquote type=3D"cite">=C2=A0&gt; Is there any firewall in the net=
work
        or on test hosts which could block incoming TCP connection to
        the port 23571
        <a href=3D"http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target=
=3D"_blank">&lt;http://iol-dts-tester.dpdklab.iol.unh.edu:23571&gt;</a> fro=
m the
        host where you run test engine?
        <br>
        <br>
        Our test engine host and the testbed are on the same subnet. The
        connection does work sometimes.
        <br>
        <br>
        =C2=A0&gt; If behaviour the same on the next try and you see that
        test agent is kept running, could you check using
        <br>
        =C2=A0&gt;
        <br>
        =C2=A0&gt; # netstat -tnlp
        <br>
        =C2=A0&gt;
        <br>
        =C2=A0&gt; that Test Agent is listening on the port and try to
        establish TCP connection from test agent using
        <br>
        =C2=A0&gt;
        <br>
        =C2=A0&gt; $ telnet <a href=3D"http://iol-dts-tester.dpdklab.iol.un=
h.edu" target=3D"_blank">iol-dts-tester.dpdklab.iol.unh.edu</a>
        <a href=3D"http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target=
=3D"_blank">&lt;http://iol-dts-tester.dpdklab.iol.unh.edu:23571&gt;</a> 235=
71
        <a href=3D"http://iol-dts-tester.dpdklab.iol.unh.edu:23571" target=
=3D"_blank">&lt;http://iol-dts-tester.dpdklab.iol.unh.edu:23571&gt;</a>
        <br>
        =C2=A0&gt;
        <br>
        =C2=A0&gt; and check if TCP connection could be established.
        <br>
        <br>
        I was able to replicate the same behavior again, where it hangs
        while RCF is trying to start.
        <br>
        Running this command, I see this in the output:
        <br>
        <br>
        tcp =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A00 <a href=3D"h=
ttp://0.0.0.0:23571" target=3D"_blank">0.0.0.0:23571</a>
        <a href=3D"http://0.0.0.0:23571" target=3D"_blank">&lt;http://0.0.0=
.0:23571&gt;</a>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 0.0.0.0:* =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
        LISTEN =C2=A0 =C2=A0 =C2=A018599/ta
        <br>
        <br>
        So it seems like it is listening on the correct port.
        <br>
        Additionally, I was able to connect to the Tester machine from
        our Test Engine host using telnet. It printed the PID of the
        process once the connection was opened.
        <br>
        <br>
        I tried running the &quot;ta&quot; application manually on the comm=
and
        line, and it didn&#39;t print anything at all.
        <br>
        Maybe the issue is something on the Test Engine side.
        <br>
        <br>
        On Thu, Aug 24, 2023 at 2:35=E2=80=AFPM Andrew Rybchenko
        &lt;<a href=3D"mailto:andrew.rybchenko@oktetlabs.ru" target=3D"_bla=
nk">andrew.rybchenko@oktetlabs.ru</a>
        <a href=3D"mailto:andrew.rybchenko@oktetlabs.ru" target=3D"_blank">=
&lt;mailto:andrew.rybchenko@oktetlabs.ru&gt;</a>&gt; wrote:
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 Hi Adam,
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0=C2=A0 &gt; On the tester host (which appears to =
be the Peer
        agent), there
        <br>
        =C2=A0=C2=A0=C2=A0 are four processes that I see running, which loo=
k like the
        test
        <br>
        =C2=A0=C2=A0=C2=A0 agent processes.
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 Before the next try I&#39;d recommend to kill th=
ese processes.
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 Is there any firewall in the network or on test =
hosts which
        could
        <br>
        =C2=A0=C2=A0=C2=A0 block incoming TCP connection to the port 23571
        <br>
        =C2=A0=C2=A0=C2=A0 <a href=3D"http://iol-dts-tester.dpdklab.iol.unh=
.edu:23571" target=3D"_blank">&lt;http://iol-dts-tester.dpdklab.iol.unh.edu=
:23571&gt;</a> from
        the host
        <br>
        =C2=A0=C2=A0=C2=A0 where you run test engine?
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 If behaviour the same on the next try and you se=
e that test
        agent is
        <br>
        =C2=A0=C2=A0=C2=A0 kept running, could you check using
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 # netstat -tnlp
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 that Test Agent is listening on the port and try=
 to
        establish TCP
        <br>
        =C2=A0=C2=A0=C2=A0 connection from test agent using
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 $ telnet <a href=3D"http://iol-dts-tester.dpdkla=
b.iol.unh.edu" target=3D"_blank">iol-dts-tester.dpdklab.iol.unh.edu</a>
        <br>
        =C2=A0=C2=A0=C2=A0 <a href=3D"http://iol-dts-tester.dpdklab.iol.unh=
.edu:23571" target=3D"_blank">&lt;http://iol-dts-tester.dpdklab.iol.unh.edu=
:23571&gt;</a>
        23571
        <br>
        =C2=A0=C2=A0=C2=A0 <a href=3D"http://iol-dts-tester.dpdklab.iol.unh=
.edu:23571" target=3D"_blank">&lt;http://iol-dts-tester.dpdklab.iol.unh.edu=
:23571&gt;</a>
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 and check if TCP connection could be established=
.
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 Another idea is to login Tester under root as te=
sting does,
        get
        <br>
        =C2=A0=C2=A0=C2=A0 start TA command from the log and try it by hand=
s without -n
        and
        <br>
        =C2=A0=C2=A0=C2=A0 remove extra escaping.
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 # sudo PATH=3D${PATH}:/tmp/linux_x86_root_76872_=
1692885663_1
        <br>
        =C2=A0=C2=A0=C2=A0
LD_LIBRARY_PATH=3D${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_roo=
t_76872_1692885663_1
        /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571
host=3Diol-dts-tester.dpdklab.iol.unh.edu:port=3D23571:user=3Droot:key=3D/o=
pt/tsf/keys/id_ed25519:ssh_port=3D22:copy_timeout=3D15:kill_timeout=3D15:su=
do=3D:shell=3D<br>
        <br>
        =C2=A0=C2=A0=C2=A0 Hopefully in this case test agent directory rema=
ins in the
        /tmp and
        <br>
        =C2=A0=C2=A0=C2=A0 you don&#39;t need to copy it as testing does.
        <br>
        =C2=A0=C2=A0=C2=A0 May be output could shed some light on what&#39;=
s going on.
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 Andrew.
        <br>
        <br>
        =C2=A0=C2=A0=C2=A0 On 8/24/23 17:30, Adam Hassick wrote:
        <br>
        <blockquote type=3D"cite">=C2=A0=C2=A0=C2=A0 Hi Andrew,
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0 This is the output that I see in the terminal =
when this
          failure
          <br>
          =C2=A0=C2=A0=C2=A0 occurs, after the test agent binaries build an=
d the test
          engine
          <br>
          =C2=A0=C2=A0=C2=A0 starts:
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0 Platform default build - pass
          <br>
          =C2=A0=C2=A0=C2=A0 Simple RCF consistency check succeeded
          <br>
          =C2=A0=C2=A0=C2=A0 ---&gt;&gt;&gt; Starting Logger...done
          <br>
          =C2=A0=C2=A0=C2=A0 ---&gt;&gt;&gt; Starting RCF...rcf_net_engine_=
connect():
          Connection timed
          <br>
          =C2=A0=C2=A0=C2=A0 out <a href=3D"http://iol-dts-tester.dpdklab.i=
ol.unh.edu:23571" target=3D"_blank">iol-dts-tester.dpdklab.iol.unh.edu:2357=
1</a>
          <br>
          =C2=A0=C2=A0=C2=A0 <a href=3D"http://iol-dts-tester.dpdklab.iol.u=
nh.edu:23571" target=3D"_blank">&lt;http://iol-dts-tester.dpdklab.iol.unh.e=
du:23571&gt;</a>
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0 Then, it hangs here until I kill the &quot;te_=
rcf&quot; and &quot;te_tee&quot;
          <br>
          =C2=A0=C2=A0=C2=A0 processes. I let it hang for around 9 minutes.
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0 On the tester host (which appears to be the Pe=
er agent),
          there are
          <br>
          =C2=A0=C2=A0=C2=A0 four processes that I see running, which look =
like the
          test agent
          <br>
          =C2=A0=C2=A0=C2=A0 processes.
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0 ta.Peer is an empty file. I&#39;ve attached th=
e log.txt from
          this run.
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0 =C2=A0- Adam
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0 On Thu, Aug 24, 2023 at 4:22=E2=80=AFAM Andrew=
 Rybchenko
          <br>
          =C2=A0=C2=A0=C2=A0 &lt;<a href=3D"mailto:andrew.rybchenko@oktetla=
bs.ru" target=3D"_blank">andrew.rybchenko@oktetlabs.ru</a>
          <br>
          =C2=A0=C2=A0=C2=A0 <a href=3D"mailto:andrew.rybchenko@oktetlabs.r=
u" target=3D"_blank">&lt;mailto:andrew.rybchenko@oktetlabs.ru&gt;</a>&gt; w=
rote:
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Adam,
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Yes, TE_RCFUNIX_TIMEOU=
T is in seconds. I&#39;ve
          double-checked
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 that it goes to &#39;c=
opy_timeout&#39; in ts-conf/rcf.conf.
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Description in in
          doc/sphinx/pages/group_te_engine_rcf.rst
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 says that copy_timeout=
 is in seconds and
          implementation in
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 lib/rcfunix/rcfunix.c =
passes the value to select()
          tv_sec.
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Theoretically select()=
 could be interrupted by signal,
          but I
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 think it is unlikely h=
ere.
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;m not sure that =
I understand what do you mean by RCF
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 connection timeout. Do=
es it happen on TE startup when
          RCF
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 starts test agents. If=
 so, TE_RCFUNIX_TIMEOUT could
          help. Or
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 does it happen when te=
sts are in progress, e.g. in the
          middle
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 of a test. If so, TE_R=
CFUNIX_TIMEOUT is unrelated and
          most
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 likely either host wit=
h test agent dies or test agent
          itself
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 crashes. It would be e=
asier for me if classify it if
          you share
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 text log (log.txt, ful=
l or just corresponding fragment
          with
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 some context). Also co=
ntent of ta.DPDK or ta.Peer file
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 depending on which age=
nt has problems could shed some
          light.
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Corresponding files co=
ntain stdout/stderr of test
          agents.
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Andrew.
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 8/23/23 17:45, Adam=
 Hassick wrote:
          <br>
          <blockquote type=3D"cite">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Hi Andrew,
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;ve set up a te=
st rig repository here, and have
            created
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 configurations for o=
ur development testbed based off
            of the
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 examples.
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We&#39;ve been able =
to get the test suite to run
            manually on
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Mellanox CX5 devices=
 once.
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 However, we are runn=
ing into an issue where, when
            RCF starts,
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 the RCF connection t=
imes out very frequently. We
            aren&#39;t sure
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 why this is the case=
.
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 It works sometimes, =
but most of the time when we try
            to run
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 the test engine, it =
encounters this issue.
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;ve tried chang=
ing the RCF port by setting
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &quot;TE_RCF_PORT=3D=
&lt;some port number&gt;&quot; and rebooting
            the testbed
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 machines. Neither se=
ems to fix the issue.
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 It also seems like t=
he timeout takes far longer than
            60
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 seconds, even when r=
unning &quot;export
            TE_RCFUNIX_TIMEOUT=3D60&quot;
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 before I try to run =
the test suite.
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I assume the unit fo=
r this variable is seconds?
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Thanks,
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Adam
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On Mon, Aug 21, 2023=
 at 10:19=E2=80=AFAM Adam Hassick
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &lt;<a href=3D"mailt=
o:ahassick@iol.unh.edu" target=3D"_blank">ahassick@iol.unh.edu</a>
            <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">&lt;m=
ailto:ahassick@iol.unh.edu&gt;</a>&gt; wrote:
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Hi Andrew,
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Thanks, I&#39;ve cloned the example repository and
            will start
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 setting up a configuration for our development
            testbed
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 today. I&#39;ll let you know if I run into any
            difficulties
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 or have any questions.
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 =C2=A0- Adam
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 On Sun, Aug 20, 2023 at 4:40=E2=80=AFAM Andrew Rybchenko
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 &lt;<a href=3D"mailto:andrew.rybchenko@oktetlabs.ru" target=3D"_blank">=
andrew.rybchenko@oktetlabs.ru</a>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 <a href=3D"mailto:andrew.rybchenko@oktetlabs.ru" target=3D"_blank">&lt;=
mailto:andrew.rybchenko@oktetlabs.ru&gt;</a>&gt;
            wrote:
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Adam,
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;ve published
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"https://github.com/ts-factory/ts-rig=
s-sample" target=3D"_blank">https://github.com/ts-factory/ts-rigs-sample</a=
>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0
            <a href=3D"https://github.com/ts-factory/ts-rigs-sample" target=
=3D"_blank">&lt;https://github.com/ts-factory/ts-rigs-sample&gt;</a>.
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hopefully it will help to define your test
            rigs and
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 successfully run some tests manually. Feel
            free to
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 ask any questions and I&#39;ll answer here and
            try to
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 update documentation.
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 Meanwhile I&#39;ll prepare missing bits for
            steps (2) and
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 (3).
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hopefully everything is in place for step
            (4), but we
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 need to make steps (2) and (3) first.
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 Andrew.
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 8/18/23 21:40, Andrew Rybchenko wrote:
            <br>
            <blockquote type=3D"cite">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Adam,
              <br>
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &gt; I&#39;ve conferred with the rest of the
              team, and we
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 think it would be best to move forward
              with mainly
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 option B.
              <br>
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 OK, I&#39;ll provide the sample on Monday fo=
r
              you. It is
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 almost ready right now, but I need to
              double-check
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 it before publishing.
              <br>
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Regards,
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Andrew.
              <br>
              <br>
              =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 8/17/23 20:03, Adam Hassick wrote:
              <br>
              <blockquote type=3D"cite">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Andrew,
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;m adding the CI mailing list to thi=
s
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 conversation. Others in the community
                might find
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 this conversation valuable.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We do want to run testing on a regular
                basis. The
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Jenkins integration will be very useful
                for us, as
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 most of our CI is orchestrated by
                Jenkins.
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;ve conferred with the rest of the
                team, and we
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 think it would be best to move forward
                with mainly
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 option B.
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 If you would like to know anything about
                our
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 testbeds that would help you with
                creating an
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 example ts-rigs repo, I&#39;d be happy to
                answer any
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 questions you have.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We have multiple test rigs (we call
                these
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &quot;DUT-tester pairs&quot;) that we run=
 our
                existing
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hardware testing on, with differing
                network
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hardware and CPU architecture. I figured
                this might
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 be an important detail.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Thanks,
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Adam
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On Thu, Aug 17, 2023 at 11:44=E2=80=AFAM =
Andrew
                Rybchenko
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 &lt;<a href=3D"mailto:andrew.rybchenko@ok=
tetlabs.ru" target=3D"_blank">andrew.rybchenko@oktetlabs.ru</a>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                <a href=3D"mailto:andrew.rybchenko@oktetlabs.ru" target=3D"=
_blank">&lt;mailto:andrew.rybchenko@oktetlabs.ru&gt;</a>&gt; wrote:
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Greatings Adam,
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;m happy to =
hear that you&#39;re trying
                to bring
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 it up.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 As I understand t=
he final goal is to
                run it on
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 regular basis. So=
, we need to make
                it properly
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 from the very beg=
inning.
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Bring up of all f=
eatures consists of
                4 steps:
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1. Create site-sp=
ecific repository
                (we call it
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ts-rigs) which co=
ntains information
                about test
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rigs and other si=
te-specific
                information like
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 where to send mai=
ls, where to store
                logs etc.
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 It is required fo=
r manual execution
                as well,
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 since test rigs d=
escription is
                essential. I&#39;ll
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return to the top=
ic below.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2. Setup logs sto=
rage for automated
                runs.
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Basically it is a=
 disk space plus
                apache2 web
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 server with few C=
GI scripts which
                help a lot to
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 save disk space.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 3. Setup Bublik w=
eb application
                which provides
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 web interface to =
view testing
                results. Same as
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"https:=
//ts-factory.io/bublik" target=3D"_blank">https://ts-factory.io/bublik</a>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"https:=
//ts-factory.io/bublik" target=3D"_blank">&lt;https://ts-factory.io/bublik&=
gt;</a>
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4. Setup Jenkins =
to run tests on
                regularly,
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 save logs in log =
storage (2) and
                import it to
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bublik (3).
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Last few month we=
 spent on our
                homework to make
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 it simpler to bri=
ng up automated
                execution
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using Jenkins -
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                <a href=3D"https://github.com/ts-factory/te-jenkins" target=
=3D"_blank">https://github.com/ts-factory/te-jenkins</a>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                <a href=3D"https://github.com/ts-factory/te-jenkins" target=
=3D"_blank">&lt;https://github.com/ts-factory/te-jenkins&gt;</a>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Corresponding bit=
s in dpdk-ethdev-ts
                will be
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 available tomorro=
w.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Let&#39;s return =
to the step (1).
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Unfortunately the=
re is no publicly
                available
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 example of the ts=
-rigs repository
                since
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sensitive site-sp=
ecific information
                is located
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 there. But I&#39;=
m ready to help you to
                create it
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for UNH. I see tw=
o options here:
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (A) I&#39;ll ask =
questions and based on
                your
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 answers will crea=
te the first draft
                with my
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 comments.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (B) I&#39;ll make=
 a template/example
                ts-rigs repo,
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 publish it and yo=
u&#39;ll create UNH
                ts-rigs based
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 on it.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Of course, I&#39;=
ll help to debug and
                finally bring
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 it up in any case=
.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (A) is a bit simp=
ler for me and you,
                but (B) is
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 a bit more generi=
c and will help
                other
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 potential users t=
o bring it up.
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We can combine (A=
)+(B). I.e. start
                from (A).
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 What do you think=
?
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Thanks,
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Andrew.
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 8/17/23 15:18,=
 Konstantin Ushakov
                wrote:
                <br>
                <blockquote type=3D"cite">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 Greetings
                  Adam,
                  <br>
                  <br>
                  <br>
                  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Thanks for contac=
ting us. I copy
                  Andrew who
                  <br>
                  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 would be happy to=
 help
                  <br>
                  <br>
                  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Thanks,
                  <br>
                  =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Konstantin
                  <br>
                  <br>
                  <blockquote type=3D"cite">=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 On 16 Aug
                    2023, at 21:50, Adam Hassick
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"mai=
lto:ahassick@iol.unh.edu" target=3D"_blank">&lt;ahassick@iol.unh.edu&gt;</a=
>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                    <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blan=
k">&lt;mailto:ahassick@iol.unh.edu&gt;</a> wrote:
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =EF=BB=BF
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Greetings Kons=
tantin,
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I am in the pr=
ocess of setting
                    up the DPDK
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Poll Mode Driv=
er test suite as
                    an addition to
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 our testing co=
verage for DPDK at
                    the UNH lab.
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I have some qu=
estions about how
                    to set the
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 test suite arg=
uments.
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I have been ab=
le to configure
                    the Test Engine
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 to connect to =
the hosts in the
                    testbed. The
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 RCF, Configura=
tor, and Tester
                    all begin to
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 run, however t=
he prelude of the
                    test suite
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 fails to run.
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                    <a href=3D"https://ts-factory.io/doc/dpdk-ethdev-ts/ind=
ex.html#test-parameters" target=3D"_blank">https://ts-factory.io/doc/dpdk-e=
thdev-ts/index.html#test-parameters</a>
<a href=3D"https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-paramet=
ers" target=3D"_blank">&lt;https://ts-factory.io/doc/dpdk-ethdev-ts/index.h=
tml#test-parameters&gt;</a>
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 The documentat=
ion mentions that
                    there are
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 several test p=
arameters for the
                    test suite,
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 like for the I=
UT test link MAC,
                    etc. These
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 seem like they=
 would need to be
                    set somewhere
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 to run many of=
 the tests.
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I see in the T=
est Engine
                    documentation, there
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 are instructio=
ns on how to
                    create new
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 parameters for=
 test suites in
                    the Tester
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 configuration,=
 but there is
                    nothing in the
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 user guide or =
in the Tester
                    guide for how to
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 set the argume=
nts for the
                    parameters when
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 running the te=
st suite that I
                    can find. I&#39;m
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 not sure if I =
need to write my
                    own Tester
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 config, or if =
I should be
                    setting these in
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 some other way=
.
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 How should the=
se values be set?
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&#39;m also n=
ot sure what
                    environment
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 variables/argu=
ments are strictly
                    necessary or
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 which are opti=
onal.
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Regards,
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Adam
                    <br>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Adam
                    Hassick*
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Senior Develop=
er
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 UNH InterOpera=
bility Lab
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"mai=
lto:ahassick@iol.unh.edu" target=3D"_blank">ahassick@iol.unh.edu</a>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
                    <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blan=
k">&lt;mailto:ahassick@iol.unh.edu&gt;</a>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"htt=
p://iol.unh.edu" target=3D"_blank">iol.unh.edu</a>
                    <a href=3D"https://www.iol.unh.edu/" target=3D"_blank">=
&lt;https://www.iol.unh.edu/&gt;</a>
                    <br>
                    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 +1 (603) 475-8=
248
                    <br>
                  </blockquote>
                </blockquote>
                <br>
                <br>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Adam Hassick*
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Senior Developer
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 UNH InterOperability Lab
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"mailto:ahassick@iol.unh.edu" t=
arget=3D"_blank">ahassick@iol.unh.edu</a>
                <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">&=
lt;mailto:ahassick@iol.unh.edu&gt;</a>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"http://iol.unh.edu" target=3D"=
_blank">iol.unh.edu</a>
                <a href=3D"https://www.iol.unh.edu/" target=3D"_blank">&lt;=
https://www.iol.unh.edu/&gt;</a>
                <br>
                =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 +1 (603) 475-8248
                <br>
              </blockquote>
              <br>
            </blockquote>
            <br>
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 -- =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *=
Adam Hassick*
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 Senior Developer
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 UNH InterOperability Lab
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">ahassick@iol.=
unh.edu</a>
            <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">&lt;m=
ailto:ahassick@iol.unh.edu&gt;</a>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 <a href=3D"http://iol.unh.edu" target=3D"_blank">iol.unh.edu</a> <a hre=
f=3D"https://www.iol.unh.edu/" target=3D"_blank">&lt;https://www.iol.unh.ed=
u/&gt;</a>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 +1 (603) 475-8248
            <br>
            <br>
            <br>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Adam Hassick*
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Senior Developer
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 UNH InterOperability=
 Lab
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"mailto:ah=
assick@iol.unh.edu" target=3D"_blank">ahassick@iol.unh.edu</a>
            <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">&lt;m=
ailto:ahassick@iol.unh.edu&gt;</a>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <a href=3D"http://io=
l.unh.edu" target=3D"_blank">iol.unh.edu</a> <a href=3D"https://www.iol.unh=
.edu/" target=3D"_blank">&lt;https://www.iol.unh.edu/&gt;</a>
            <br>
            =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 +1 (603) 475-8248
            <br>
          </blockquote>
          <br>
          <br>
          <br>
          =C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0 *Adam Hassick*
          <br>
          =C2=A0=C2=A0=C2=A0 Senior Developer
          <br>
          =C2=A0=C2=A0=C2=A0 UNH InterOperability Lab
          <br>
          =C2=A0=C2=A0=C2=A0 <a href=3D"mailto:ahassick@iol.unh.edu" target=
=3D"_blank">ahassick@iol.unh.edu</a> <a href=3D"mailto:ahassick@iol.unh.edu=
" target=3D"_blank">&lt;mailto:ahassick@iol.unh.edu&gt;</a>
          <br>
          =C2=A0=C2=A0=C2=A0 <a href=3D"http://iol.unh.edu" target=3D"_blan=
k">iol.unh.edu</a> <a href=3D"https://www.iol.unh.edu/" target=3D"_blank">&=
lt;https://www.iol.unh.edu/&gt;</a>
          <br>
          =C2=A0=C2=A0=C2=A0 +1 (603) 475-8248
          <br>
        </blockquote>
        <br>
        <br>
        <br>
        --=C2=A0<br>
        *Adam Hassick*
        <br>
        Senior Developer
        <br>
        UNH InterOperability Lab
        <br>
        <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">ahassick@=
iol.unh.edu</a> <a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank">&=
lt;mailto:ahassick@iol.unh.edu&gt;</a>
        <br>
        <a href=3D"http://iol.unh.edu" target=3D"_blank">iol.unh.edu</a> <a=
 href=3D"https://www.iol.unh.edu/" target=3D"_blank">&lt;https://www.iol.un=
h.edu/&gt;</a>
        <br>
        +1 (603) 475-8248
        <br>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </div>

</blockquote></div><br clear=3D"all"><br><span class=3D"gmail_signature_pre=
fix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"l=
tr"><div><div><b><span style=3D"background-color:rgb(255,255,255)"><span st=
yle=3D"color:rgb(102,102,102)">Adam Hassick</span></span></b><br></div><spa=
n style=3D"color:rgb(102,102,102)"></span></div><div><span style=3D"color:r=
gb(102,102,102)">Senior Developer</span></div><div><span style=3D"color:rgb=
(102,102,102)"><span style=3D"color:rgb(11,83,148)"><span style=3D"backgrou=
nd-color:rgb(255,255,255)">UNH InterOperability Lab</span></span></span><sp=
an style=3D"color:rgb(102,102,102)"></span></div><div><span style=3D"color:=
rgb(102,102,102)"><a href=3D"mailto:ahassick@iol.unh.edu" target=3D"_blank"=
>ahassick@iol.unh.edu</a><br></span></div><div><span style=3D"color:rgb(102=
,102,102)"><a href=3D"https://www.iol.unh.edu/" target=3D"_blank">iol.unh.e=
du</a><br></span></div>+1 (603) 475-8248<br></div></div>

--0000000000004660700603bfd6d9--