From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2A13141F5F; Tue, 29 Aug 2023 16:01:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1A28D40293; Tue, 29 Aug 2023 16:01:52 +0200 (CEST) Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) by mails.dpdk.org (Postfix) with ESMTP id E9CBD40279 for ; Tue, 29 Aug 2023 16:01:50 +0200 (CEST) Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-5920efd91c7so50023447b3.2 for ; Tue, 29 Aug 2023 07:01:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1693317710; x=1693922510; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=IZGR3VMAYcyn+KQNBegVFWvBJ13F/RjZ36ce7XSRF6I=; b=A7gxghkVnEw7s753yAZi4cbZ6H5l1fSfOjXfEn06OPrbR0lk9+IQacrEbob7xXwPMC NpffuQUc7IZXrYj8jewHpRVgnYeF4UCNpANzT5SdARvVZRwoM0vFF+hIRWZYaJHZZoBh /npCR6j4t1i5jaT1ALHshCvj0SII1TS0NROKw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693317710; x=1693922510; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IZGR3VMAYcyn+KQNBegVFWvBJ13F/RjZ36ce7XSRF6I=; b=jXXAkdnDVtu8ppZ8ZCNrVXpEW2+tR++Ej4dpnghOZ9EXz6lV4AcEUN8XONl/K3KvN/ tYH3t/njxUSG3/CNvG6Ab9tCFL+sJmbMFw3FHjxlvGGAeLa3nLlrI5vGbPThYDhswFdC ZWxD8veGizoD6xCbJ7Iiyk+XtpDPjnLYb9Me3cEg0bDGkJBoPkIcdYl7SzCTpf5Njhuc jQyLS8bb1vBqNrjUNkgxTODnAb8w55PjegG3Np7kfh4NZyQ98OfBAUtIetQdnuzOTy+q 9k2+qFUMJhvs7syi6Zk3hd0GEGGXlpPwc8d7loUgHjGCoXnBxyQwZACUVLksbJhgpdgg GZfQ== X-Gm-Message-State: AOJu0YyuY2BfvMUfXQZwJZ4Yh8arAmb0dgsdDVa8WyaFXips4FnOPuJ6 UGsOvYwN/InENsV0ZzkOiDr1Wzh6iOGITF9cbPYe+Q== X-Google-Smtp-Source: AGHT+IEokkDcDxNKqSo/zZUZ1Ni5arg4XAJLlE2AKn7nrvIfHJrVKN+nmNKX7oJq6Q7n9dm1stkGd3YkmUaNAX73K18= X-Received: by 2002:a05:690c:887:b0:595:352f:2296 with SMTP id cd7-20020a05690c088700b00595352f2296mr5864814ywb.25.1693317709823; Tue, 29 Aug 2023 07:01:49 -0700 (PDT) MIME-Version: 1.0 References: <9ce9d7fd-4051-6d51-26bb-7e96e98c677e@oktetlabs.ru> <781ca146-955f-85af-5727-66015ae1d326@oktetlabs.ru> <7734826a-840d-d0d9-e7a5-91951223398c@oktetlabs.ru> <9d920676-485d-3b4d-ca20-2b5ea3a5b606@oktetlabs.ru> <873c7972-3e5a-9e82-9449-4d12b2c96032@oktetlabs.ru> <6c9eea95-7d53-c82c-bced-823e4e9db62a@oktetlabs.ru> <74cec43e-43d5-eb4c-caa2-8ebada2680c1@oktetlabs.ru> <6f77b754-4b12-4e64-e25d-3037af475410@oktetlabs.ru> In-Reply-To: <6f77b754-4b12-4e64-e25d-3037af475410@oktetlabs.ru> From: Adam Hassick Date: Tue, 29 Aug 2023 10:02:21 -0400 Message-ID: Subject: Re: Setting up DPDK PMD Test Suite To: Andrew Rybchenko Cc: Patrick Robb , Konstantin Ushakov , ci@dpdk.org Content-Type: multipart/alternative; boundary="000000000000961db70604104034" X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --000000000000961db70604104034 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Andrew, That fix seems to have resolved the issue, thanks for the quick turnaround time on that patch. Now that we have the RCF timeout issue resolved, there are a few other questions and issues that we have about the tests themselves. 1. The test suite fails to build with a couple warnings. Below is the stderr log from compilation: FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o > cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib > -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include > -fdiagnostics-color=3Dalways -pipe -D_FILE_OFFSET_BITS=3D64 -Wall -Winval= id-pch > -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ 'lib/76b5a35@ > @ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_= pmd_ts.c.o.d' > -o 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c > ../../lib/dpdk_pmd_ts.c > ../../lib/dpdk_pmd_ts.c: In function > =E2=80=98test_create_traffic_generator_params=E2=80=99: > ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no > format arguments [-Werror=3Dformat-security] > 5577 | rc =3D te_kvpair_add(result, buf, mode); > | ^~ > cc1: all warnings being treated as errors > ninja: build stopped: subcommand failed. > ninja: Entering directory `.' > FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o > cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib > -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include > -fdiagnostics-color=3Dalways -pipe -D_FILE_OFFSET_BITS=3D64 -Wall -Winval= id-pch > -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ 'lib/76b5a35@ > @ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_= pmd_ts.c.o.d' > -o 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c > ../../lib/dpdk_pmd_ts.c > ../../lib/dpdk_pmd_ts.c: In function > =E2=80=98test_create_traffic_generator_params=E2=80=99: > ../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string literal and no > format arguments [-Werror=3Dformat-security] > 5577 | rc =3D te_kvpair_add(result, buf, mode); > | ^~ > cc1: all warnings being treated as errors > This error wasn't occurring last week, which was the last time I ran the tests. The TE host and the DUT have GCC v9.4.0 installed, and the tester has GCC v11.4.0 installed, if this information is helpful. 2. On the Mellanox CX5s, there are over 6,000 tests run, which collectively take around 9 hours. Is it possible, and would it make sense, to lower the test coverage and have the test suite run faster? For some context, we run immediate testing on incoming patches for DPDK main and development branches, as well as periodic test runs on the main, stable, and LTS branches. For us to consider including this test suite as part of our immediate testing on patches, we would have to reduce the test coverage to the most important tests. This is primarily to reduce the testing time to, for example, less than 30 minutes. Testing on patches can't take too long because the lab can receive numerous patches each day, which each require individual testing runs. At what frequency we run these tests, and on what, still needs to be discussed with the DPDK community, but it would be nice to know if the test suite had a "dial" to control the testing coverage. 3. We see a lot of test failures on our Mellanox CX5 NICs. Around 2,300 of ~6,600 tests passed. Is there anything we can do to diagnose these test failures? Thanks, Adam On Tue, Aug 29, 2023 at 8:07=E2=80=AFAM Andrew Rybchenko < andrew.rybchenko@oktetlabs.ru> wrote: > Hi Adam, > > I've pushed the fix in main branch and a new tag v1.18.1. It should solve > the problem with IPv6 address from DNS. > > Andrew. > > On 8/29/23 00:05, Andrew Rybchenko wrote: > > Hi Adam, > > > Does the test engine prefer to use IPv6 over IPv4 for initiating the RC= F > connection to the test bed hosts? And if so, is there a way to force it t= o > use IPv4? > > Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your case, > I guess it is the root cause of the problem. > Of course, it is TE problem since I see really weird code in > lib/comm_net_engine/comm_net_engine.c line 135. > > I've pushed fix to the branch user/arybchik/fix_ipv4_only in > ts-factory/test-environment repository. Please, try. > > It is late night fix with minimal testing and no review. I'll pass it > through review process tomorrow and > hopefully it will be released in one-two days. > > Andrew. > > On 8/28/23 18:02, Adam Hassick wrote: > > Hi Andrew, > > We have yet to notice a distinct pattern with the failures. Sometimes, th= e > RCF will start and connect without issue a few times in a row before > failing to connect again. Once the issue begins to occur, neither rebooti= ng > all of the hosts (test engine VM, tester, IUT) or deleting all of the bui= ld > directories (suites, agents, inst) and rebooting the hosts afterward > resolves the issue. When it begins working again seems very arbitrary to = us. > > I do usually try to terminate the test engine with Ctrl+C, but when it > hangs while trying to start RCF, that does not work. > > Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF > connection to the test bed hosts? And if so, is there a way to force it t= o > use IPv4? > > - Adam > > On Fri, Aug 25, 2023 at 1:35=E2=80=AFPM Andrew Rybchenko < > andrew.rybchenko@oktetlabs.ru> wrote: > >> > I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04. >> >> Done. It works fine for me without any issues. >> >> Have you noticed any pattern when it works or does not work? >> May be it is a problem of not clean state after termination? >> Does it work fine the first time after DUTs reboot? >> How do you terminate testing? It should be done using Ctrl+C in terminal >> where you execute run.sh command. >> In this case it should shutdown gracefully and close all test agents an= d >> engine applications. >> >> (I'm trying to understand why you've seen many test agent processes. It >> should not happen.) >> >> Andrew. >> >> On 8/25/23 17:41, Andrew Rybchenko wrote: >> >> On 8/25/23 17:06, Adam Hassick wrote: >> >> Hi Andrew, >> >> Two of our systems (the Test Engine runner and the DUT host) are running >> Ubuntu 20.04 LTS, however this morning I noticed that the tester system >> (the one having issues) is running Ubuntu 22.04 LTS. >> This could be the source of the problem. I encountered a dependency issu= e >> trying to run the Test Engine on 22.04 LTS, so I downgraded the system. >> Since the tester is also the host having connection issues, I will try >> downgrading that system to 20.04, and see if that changes anything. >> >> >> Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu >> 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night. >> Right now Debian 11 is used for test engine in nightly regressions. >> >> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04. >> >> I did try passing in the "--vg-rcf" argument to the run.sh script of the >> test suite after installing valgrind, but there was no additional output >> that I saw. >> >> >> Sorry, I should valgrind output should be in valgrind.te_rcf (direction >> where you run test engine). >> >> >> I will try pulling in the changes you've pushed up, and will see if that >> fixes anything. >> >> Thanks, >> Adam >> >> On Fri, Aug 25, 2023 at 9:57=E2=80=AFAM Andrew Rybchenko < >> andrew.rybchenko@oktetlabs.ru> wrote: >> >>> Hello Adam, >>> >>> On 8/24/23 23:54, Andrew Rybchenko wrote: >>> >>> I'd like to try to repeat the problem locally. Which Linux distro is >>> running on test engine and agents? >>> >>> In fact I know one problem with Debian 12 and Fedora 38 and we have >>> patch in review to fix it, however, the behaviour is different in >>> this case, so it is unlike the same problem. >>> >>> >>> I've just published a new tag which fixes known test engine side >>> problems on Debian 12 and Fedora 38. >>> >>> >>> One more idea is to install valgrind on the test engine host and >>> run with option --vg-rcf to check if something weird is happening. >>> >>> What I don't understand right now is why I see just one failed attempt >>> to connect in your log.txt and then Logger shutdown after 9 minutes. >>> >>> Andrew. >>> >>> On 8/24/23 23:29, Adam Hassick wrote: >>> >>> > Is there any firewall in the network or on test hosts which could >>> block incoming TCP connection to the port 23571 >>> >>> from the host where >>> you run test engine? >>> >>> Our test engine host and the testbed are on the same subnet. The >>> connection does work sometimes. >>> >>> > If behaviour the same on the next try and you see that test agent is >>> kept running, could you check using >>> > >>> > # netstat -tnlp >>> > >>> > that Test Agent is listening on the port and try to establish TCP >>> connection from test agent using >>> > >>> > $ telnet iol-dts-tester.dpdklab.iol.unh.edu >>> >>> 23571 >>> >>> >>> > >>> > and check if TCP connection could be established. >>> >>> I was able to replicate the same behavior again, where it hangs while >>> RCF is trying to start. >>> Running this command, I see this in the output: >>> >>> tcp 0 0 0.0.0.0:23571 >>> 0.0.0.0:* LISTEN >>> 18599/ta >>> >>> So it seems like it is listening on the correct port. >>> Additionally, I was able to connect to the Tester machine from our Test >>> Engine host using telnet. It printed the PID of the process once the >>> connection was opened. >>> >>> I tried running the "ta" application manually on the command line, and >>> it didn't print anything at all. >>> Maybe the issue is something on the Test Engine side. >>> >>> On Thu, Aug 24, 2023 at 2:35=E2=80=AFPM Andrew Rybchenko < >>> andrew.rybchenko@oktetlabs.ru >>> > wrote: >>> >>> Hi Adam, >>> >>> > On the tester host (which appears to be the Peer agent), there >>> are four processes that I see running, which look like the test >>> agent processes. >>> >>> Before the next try I'd recommend to kill these processes. >>> >>> Is there any firewall in the network or on test hosts which could >>> block incoming TCP connection to the port 23571 >>> >>> from the host >>> where you run test engine? >>> >>> If behaviour the same on the next try and you see that test agent i= s >>> kept running, could you check using >>> >>> # netstat -tnlp >>> >>> that Test Agent is listening on the port and try to establish TCP >>> connection from test agent using >>> >>> $ telnet iol-dts-tester.dpdklab.iol.unh.edu >>> >>> 23571 >>> >>> >>> >>> and check if TCP connection could be established. >>> >>> Another idea is to login Tester under root as testing does, get >>> start TA command from the log and try it by hands without -n and >>> remove extra escaping. >>> >>> # sudo PATH=3D${PATH}:/tmp/linux_x86_root_76872_1692885663_1 >>> >>> LD_LIBRARY_PATH=3D${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86= _root_76872_1692885663_1 >>> /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 >>> host=3Diol-dts-tester.dpdklab.iol.unh.edu: >>> port=3D23571:user=3Droot:key=3D/opt/tsf/keys/id_ed25519:ssh_port=3D22:c= opy_timeout=3D15:kill_timeout=3D15:sudo=3D:shell=3D >>> >>> Hopefully in this case test agent directory remains in the /tmp and >>> you don't need to copy it as testing does. >>> May be output could shed some light on what's going on. >>> >>> Andrew. >>> >>> On 8/24/23 17:30, Adam Hassick wrote: >>> >>> Hi Andrew, >>> >>> This is the output that I see in the terminal when this failure >>> occurs, after the test agent binaries build and the test engine >>> starts: >>> >>> Platform default build - pass >>> Simple RCF consistency check succeeded >>> --->>> Starting Logger...done >>> --->>> Starting RCF...rcf_net_engine_connect(): Connection timed >>> out iol-dts-tester.dpdklab.iol.unh.edu:23571 >>> >>> >>> >>> Then, it hangs here until I kill the "te_rcf" and "te_tee" >>> processes. I let it hang for around 9 minutes. >>> >>> On the tester host (which appears to be the Peer agent), there are >>> four processes that I see running, which look like the test agent >>> processes. >>> >>> ta.Peer is an empty file. I've attached the log.txt from this run. >>> >>> - Adam >>> >>> On Thu, Aug 24, 2023 at 4:22=E2=80=AFAM Andrew Rybchenko >>> >> >>> > wrote: >>> >>> Hi Adam, >>> >>> Yes, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked >>> that it goes to 'copy_timeout' in ts-conf/rcf.conf. >>> Description in in doc/sphinx/pages/group_te_engine_rcf.rst >>> says that copy_timeout is in seconds and implementation in >>> lib/rcfunix/rcfunix.c passes the value to select() tv_sec. >>> Theoretically select() could be interrupted by signal, but I >>> think it is unlikely here. >>> >>> I'm not sure that I understand what do you mean by RCF >>> connection timeout. Does it happen on TE startup when RCF >>> starts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or >>> does it happen when tests are in progress, e.g. in the middle >>> of a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most >>> likely either host with test agent dies or test agent itself >>> crashes. It would be easier for me if classify it if you share >>> text log (log.txt, full or just corresponding fragment with >>> some context). Also content of ta.DPDK or ta.Peer file >>> depending on which agent has problems could shed some light. >>> Corresponding files contain stdout/stderr of test agents. >>> >>> Andrew. >>> >>> On 8/23/23 17:45, Adam Hassick wrote: >>> >>> Hi Andrew, >>> >>> I've set up a test rig repository here, and have created >>> configurations for our development testbed based off of the >>> examples. >>> We've been able to get the test suite to run manually on >>> Mellanox CX5 devices once. >>> However, we are running into an issue where, when RCF starts, >>> the RCF connection times out very frequently. We aren't sure >>> why this is the case. >>> It works sometimes, but most of the time when we try to run >>> the test engine, it encounters this issue. >>> I've tried changing the RCF port by setting >>> "TE_RCF_PORT=3D" and rebooting the testbed >>> machines. Neither seems to fix the issue. >>> >>> It also seems like the timeout takes far longer than 60 >>> seconds, even when running "export TE_RCFUNIX_TIMEOUT=3D60" >>> before I try to run the test suite. >>> I assume the unit for this variable is seconds? >>> >>> Thanks, >>> Adam >>> >>> On Mon, Aug 21, 2023 at 10:19=E2=80=AFAM Adam Hassick >>> >>> > wrote: >>> >>> Hi Andrew, >>> >>> Thanks, I've cloned the example repository and will start >>> setting up a configuration for our development testbed >>> today. I'll let you know if I run into any difficulties >>> or have any questions. >>> >>> - Adam >>> >>> On Sun, Aug 20, 2023 at 4:40=E2=80=AFAM Andrew Rybchenko >>> >> >>> > wrote: >>> >>> Hi Adam, >>> >>> I've published >>> https://github.com/ts-factory/ts-rigs-sample >>> >>> . >>> Hopefully it will help to define your test rigs and >>> successfully run some tests manually. Feel free to >>> ask any questions and I'll answer here and try to >>> update documentation. >>> >>> Meanwhile I'll prepare missing bits for steps (2) and >>> (3). >>> Hopefully everything is in place for step (4), but we >>> need to make steps (2) and (3) first. >>> >>> Andrew. >>> >>> On 8/18/23 21:40, Andrew Rybchenko wrote: >>> >>> Hi Adam, >>> >>> > I've conferred with the rest of the team, and we >>> think it would be best to move forward with mainly >>> option B. >>> >>> OK, I'll provide the sample on Monday for you. It is >>> almost ready right now, but I need to double-check >>> it before publishing. >>> >>> Regards, >>> Andrew. >>> >>> On 8/17/23 20:03, Adam Hassick wrote: >>> >>> Hi Andrew, >>> >>> I'm adding the CI mailing list to this >>> conversation. Others in the community might find >>> this conversation valuable. >>> >>> We do want to run testing on a regular basis. The >>> Jenkins integration will be very useful for us, as >>> most of our CI is orchestrated by Jenkins. >>> I've conferred with the rest of the team, and we >>> think it would be best to move forward with mainly >>> option B. >>> If you would like to know anything about our >>> testbeds that would help you with creating an >>> example ts-rigs repo, I'd be happy to answer any >>> questions you have. >>> >>> We have multiple test rigs (we call these >>> "DUT-tester pairs") that we run our existing >>> hardware testing on, with differing network >>> hardware and CPU architecture. I figured this might >>> be an important detail. >>> >>> Thanks, >>> Adam >>> >>> On Thu, Aug 17, 2023 at 11:44=E2=80=AFAM Andrew Rybchen= ko >>> >> >>> > wrote: >>> >>> Greatings Adam, >>> >>> I'm happy to hear that you're trying to bring >>> it up. >>> >>> As I understand the final goal is to run it on >>> regular basis. So, we need to make it properly >>> from the very beginning. >>> Bring up of all features consists of 4 steps: >>> >>> 1. Create site-specific repository (we call it >>> ts-rigs) which contains information about test >>> rigs and other site-specific information like >>> where to send mails, where to store logs etc. >>> It is required for manual execution as well, >>> since test rigs description is essential. I'll >>> return to the topic below. >>> >>> 2. Setup logs storage for automated runs. >>> Basically it is a disk space plus apache2 web >>> server with few CGI scripts which help a lot to >>> save disk space. >>> >>> 3. Setup Bublik web application which provides >>> web interface to view testing results. Same as >>> https://ts-factory.io/bublik >>> >>> >>> >>> 4. Setup Jenkins to run tests on regularly, >>> save logs in log storage (2) and import it to >>> bublik (3). >>> >>> Last few month we spent on our homework to make >>> it simpler to bring up automated execution >>> using Jenkins - >>> https://github.com/ts-factory/te-jenkins >>> >>> >>> Corresponding bits in dpdk-ethdev-ts will be >>> available tomorrow. >>> >>> Let's return to the step (1). >>> >>> Unfortunately there is no publicly available >>> example of the ts-rigs repository since >>> sensitive site-specific information is located >>> there. But I'm ready to help you to create it >>> for UNH. I see two options here: >>> >>> (A) I'll ask questions and based on your >>> answers will create the first draft with my >>> comments. >>> >>> (B) I'll make a template/example ts-rigs repo, >>> publish it and you'll create UNH ts-rigs based >>> on it. >>> >>> Of course, I'll help to debug and finally bring >>> it up in any case. >>> >>> (A) is a bit simpler for me and you, but (B) is >>> a bit more generic and will help other >>> potential users to bring it up. >>> We can combine (A)+(B). I.e. start from (A). >>> What do you think? >>> >>> Thanks, >>> Andrew. >>> >>> On 8/17/23 15:18, Konstantin Ushakov wrote: >>> >>> Greetings Adam, >>> >>> >>> Thanks for contacting us. I copy Andrew who >>> would be happy to help >>> >>> Thanks, >>> Konstantin >>> >>> On 16 Aug 2023, at 21:50, Adam Hassick >>> >>> >>> wrote: >>> >>> =EF=BB=BF >>> Greetings Konstantin, >>> >>> I am in the process of setting up the DPDK >>> Poll Mode Driver test suite as an addition to >>> our testing coverage for DPDK at the UNH lab. >>> >>> I have some questions about how to set the >>> test suite arguments. >>> >>> I have been able to configure the Test Engine >>> to connect to the hosts in the testbed. The >>> RCF, Configurator, and Tester all begin to >>> run, however the prelude of the test suite >>> fails to run. >>> >>> >>> https://ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters >>> >>> >>> >>> The documentation mentions that there are >>> several test parameters for the test suite, >>> like for the IUT test link MAC, etc. These >>> seem like they would need to be set somewhere >>> to run many of the tests. >>> >>> I see in the Test Engine documentation, there >>> are instructions on how to create new >>> parameters for test suites in the Tester >>> configuration, but there is nothing in the >>> user guide or in the Tester guide for how to >>> set the arguments for the parameters when >>> running the test suite that I can find. I'm >>> not sure if I need to write my own Tester >>> config, or if I should be setting these in >>> some other way. >>> >>> How should these values be set? >>> >>> I'm also not sure what environment >>> variables/arguments are strictly necessary or >>> which are optional. >>> >>> Regards, >>> Adam >>> >>> -- *Adam Hassick* >>> Senior Developer >>> UNH InterOperability Lab >>> ahassick@iol.unh.edu >>> >>> iol.unh.edu >>> >>> +1 (603) 475-8248 >>> >>> >>> >>> >>> -- *Adam Hassick* >>> Senior Developer >>> UNH InterOperability Lab >>> ahassick@iol.unh.edu >>> >>> iol.unh.edu >>> >>> +1 (603) 475-8248 >>> >>> >>> >>> >>> >>> -- *Adam Hassick* >>> Senior Developer >>> UNH InterOperability Lab >>> ahassick@iol.unh.edu >>> >>> iol.unh.edu >>> >>> +1 (603) 475-8248 >>> >>> >>> >>> -- *Adam Hassick* >>> Senior Developer >>> UNH InterOperability Lab >>> ahassick@iol.unh.edu >>> >>> iol.unh.edu >>> >>> +1 (603) 475-8248 >>> >>> >>> >>> >>> -- *Adam Hassick* >>> Senior Developer >>> UNH InterOperability Lab >>> ahassick@iol.unh.edu >>> >>> iol.unh.edu >>> +1 (603) 475-8248 >>> >>> >>> >>> >>> -- >>> *Adam Hassick* >>> Senior Developer >>> UNH InterOperability Lab >>> ahassick@iol.unh.edu >>> >>> iol.unh.edu >>> +1 (603) 475-8248 >>> >>> >>> >>> >> >> -- >> *Adam Hassick* >> Senior Developer >> UNH InterOperability Lab >> ahassick@iol.unh.edu >> iol.unh.edu >> +1 (603) 475-8248 >> >> >> >> > > -- > *Adam Hassick* > Senior Developer > UNH InterOperability Lab > ahassick@iol.unh.edu > iol.unh.edu > +1 (603) 475-8248 > > > > --=20 *Adam Hassick* Senior Developer UNH InterOperability Lab ahassick@iol.unh.edu iol.unh.edu +1 (603) 475-8248 --000000000000961db70604104034 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Andrew,

That fix seems to h= ave resolved the issue, thanks for the quick turnaround time on that patch.=
Now that we have the RCF timeout issue resolved, there are a= few other questions and issues that we have about the tests themselves.
1. The test suite fails to build with a couple warnings.<= br>

Below is the stderr log from compilation:
<= /div>
FAILED: lib/76b5= a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
cc -Ilib/76b5a35@@ts_dpdk_pmd@sta -= Ilib -I../../lib -I/opt/tsf/dpdk-ethdev-ts/ts/inst/default/include -fdiagno= stics-color=3Dalways -pipe -D_FILE_OFFSET_BITS=3D64 -Wall -Winvalid-pch -We= rror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -fPIC -MD -MQ 'lib/76b5a35@@ts= _dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF 'lib/76b5a35@@ts_dpdk_pmd@sta/dp= dk_pmd_ts.c.o.d' -o 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o&#= 39; -c ../../lib/dpdk_pmd_ts.c
../../lib/dpdk_pmd_ts.c: In function =E2= =80=98test_create_traffic_generator_params=E2=80=99:
../../lib/dpdk_pmd_= ts.c:5577:5: error: format not a string literal and no format arguments [-W= error=3Dformat-security]
5577 | =C2=A0 =C2=A0 rc =3D te_kvpair_add(resul= t, buf, mode);
| =C2=A0 =C2=A0 ^~
cc1: all warnings being treated as = errors
ninja: build stopped: subcommand failed.
ninja: Entering direc= tory `.'
FAILED: lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o
cc = -Ilib/76b5a35@@ts_dpdk_pmd@sta -Ilib -I../../lib -I/opt/tsf/dpdk-ethdev-ts/= ts/inst/default/include -fdiagnostics-color=3Dalways -pipe -D_FILE_OFFSET_B= ITS=3D64 -Wall -Winvalid-pch -Werror -g -D_GNU_SOURCE -O0 -ggdb -Wall -W -f= PIC -MD -MQ 'lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -MF '= ;lib/76b5a35@@ts_dpdk_pmd@sta/dpdk_pmd_ts.c.o.d' -o 'lib/76b5a35@@t= s_dpdk_pmd@sta/dpdk_pmd_ts.c.o' -c ../../lib/dpdk_pmd_ts.c
../../lib= /dpdk_pmd_ts.c: In function =E2=80=98test_create_traffic_generator_params= =E2=80=99:
../../lib/dpdk_pmd_ts.c:5577:5: error: format not a string li= teral and no format arguments [-Werror=3Dformat-security]
5577 | =C2=A0 = =C2=A0 rc =3D te_kvpair_add(result, buf, mode);
| =C2=A0 =C2=A0 ^~
cc= 1: all warnings being treated as errors

=
This error wasn't occurring last week, which was the last ti= me I ran the tests.
The TE host and the DUT have GCC v9.4.0 i= nstalled, and the tester has GCC v11.4.0 installed, if this information is = helpful.

2. On the Mellanox CX5s, there are ov= er 6,000 tests run, which collectively take around 9 hours. Is it possible,= and would it make sense, to lower the test coverage and have the test suit= e run faster?

For some context, we run immediate testing = on incoming patches for DPDK main and development branches, as well as peri= odic test runs on the main, stable, and LTS branches.
For us = to consider including this test suite as part of our immediate testing on p= atches, we would have to reduce the test coverage to the most important tes= ts.
This is primarily to reduce the testing time to, for example, less t= han 30 minutes. Testing on patches can't take too long because the lab = can receive numerous patches each day, which each require individual testin= g runs.

At what frequency we run these tests, and on what= , still needs to be discussed with the DPDK community, but it would be nice= to know if the test suite had a "dial" to control the testing co= verage.

3. We see a lot of test failures on ou= r Mellanox CX5 NICs. Around 2,300 of ~6,600 tests passed. Is there anything= we can do to diagnose these test failures?

Th= anks,
Adam


On Tue, Aug 29= , 2023 at 8:07=E2=80=AFAM Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> wrote:
=
=20 =20 =20
Hi Adam,

I've pushed the fix in main branch and a new tag v1.18.1. It should solve the problem with IPv6 address from DNS.

Andrew.

On 8/29/23 00:05, Andrew Rybchenko wrote:
=20
Hi Adam,

> Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF connection to the test bed hosts? And if so, is there a way to force it to use IPv4?

Brilliant idea. If DNS returns both IPv4 and IPv6 addresses in your case, I guess it is the root cause of the problem.
Of course, it is TE problem since I see really weird code in lib/comm_net_engine/comm_net_engine.c line 135.

I've pushed fix to the branch user/arybchik/fix_ipv4_only in ts-factory/test-environment repository. Please, try.

It is late night fix with minimal testing and no review. I'll pass it through review process tomorrow and
hopefully it will be released in one-two days.

Andrew.

On 8/28/23 18:02, Adam Hassick wrote:
=20
Hi Andrew,

We have yet to notice a distinct pattern with the failures. Sometimes, the RCF will start and connect without issue a few times in a row before failing to connect again. Once the issue begins to occur, neither rebooting all of the hosts (test engine VM, tester, IUT) or deleting all of the build directories (suites, agents, inst) and rebooting the hosts afterward resolves the issue. When it begins working again seems very arbitrary to us.

I do usually try to terminate the test engine with Ctrl+C, but when it hangs while trying to start RCF, that does not work.

Does the test engine prefer to use IPv6 over IPv4 for initiating the RCF connection to the test bed hosts? And if so, is there a way to force it to use IPv4?

=C2=A0- Adam

On Fri, Aug 25, 2023 at 1:35=E2=80=AFPM Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru&= gt; wrote:
> I'll double-check test engine on Ubuntu 20.04 and Ubuntu 22.04.

Done. It works fine for me without any issues.

Have you noticed any pattern when it works or does not work?
May be it is a problem of not clean state after termination?
Does it work fine the first time after DUTs reboot?
How do you terminate testing? It should be done using Ctrl+C in terminal where you execute run.sh command.
=C2=A0In this case it should shutdown gracefully and close all test agents and engine applications.

(I'm trying to understand why you've seen many test agent processes. It should not happen.)

Andrew.

On 8/25/23 17:41, Andrew Rybchenko wrote:
On 8/25/23 17:06, Adam Hassick wrote:
Hi Andrew,

Two of our systems (the Test Engine runner and the DUT host) are running Ubuntu 20.04 LTS, however this morning I noticed that the tester system (the one having issues) is running Ubuntu 22.04 LTS.
This could be the source of the problem. I encountered a dependency issue trying to run the Test Engine on 22.04 LTS, so I downgraded the system. Since the tester is also the host having connection issues, I will try downgrading that system to 20.04, and see if that changes anything.

Unlikely, but who knows. We run tests (DUTs) on Ubuntu 20.04, Ubuntu 22.04, Ubuntu 22.10, Ubuntu 23.04, Debian 11 and Fedora 38 every night.
Right now Debian 11 is used for test engine in nightly regressions.

I'll double-check test engine on Ubuntu 20.04 and Ubunt= u 22.04.

I did try passing in the "--vg-rcf" argu= ment to the run.sh script of the test suite after installing valgrind, but there was no additional output that I saw.

Sorry, I should valgrind output should be in valgrind.te_rcf (direction where you run test engine).


I will try pulling in the changes you've pushe= d up, and will see if that fixes anything.

Thanks,
Adam

On Fri, Aug 25, 2023 at 9:57=E2=80=AFAM Andrew Rybchenko <andrew.rybchenk= o@oktetlabs.ru> wrote:
Hello Adam,

On 8/24/23 23:54, Andrew Rybchenko wrote:
I'd like to try to repeat the problem locally. Which Linux distro is running on test engine and agents?

In fact I know one problem with Debian 12 and Fedora 38 and we have
patch in review to fix it, however, the behaviour is different in
this case, so it is unlike the same problem.

I've just published a new tag which fixes known test engine side problems on Debian 12 and Fedora 38.


One more idea is to install valgrind on the test engine host and
run with option --vg-rcf to check if something weird is happening.

What I don't understand right now is why I se= e just one failed attempt
to connect in your log.txt and then Logger shutdown after 9 minutes.

Andrew.

On 8/24/23 23:29, Adam Hassick wrote:
=C2=A0> Is there any firewall in the network or on test hosts which could block incoming TCP connection to the port 23571 <http://iol-dts-tester.dpd= klab.iol.unh.edu:23571> from the host where you run test engine?

Our test engine host and the testbed are on the same subnet. The connection does work sometimes.

=C2=A0> If behaviour the same on the next tr= y and you see that test agent is kept running, could you check using
=C2=A0>
=C2=A0> # netstat -tnlp
=C2=A0>
=C2=A0> that Test Agent is listening on the port and try to establish TCP connection from test agent using
=C2=A0>
=C2=A0> $ telnet iol-dts-tester.dpdklab.iol.unh= .edu <http://iol-dts-tester.dpdklab.iol.unh.ed= u:23571> 23571 <http://iol-dts-tester.dpdklab.iol.= unh.edu:23571>
=C2=A0>
=C2=A0> and check if TCP connection could be established.

I was able to replicate the same behavior again, where it hangs while RCF is trying to start.
Running this command, I see this in the output:

tcp =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 = =C2=A00 0.0.0.0:23571 <http://0.0.0.0:23571>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 0.0.0.0:* =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 LISTEN =C2=A0 =C2=A0 =C2=A018599/ta

So it seems like it is listening on the correct port.
Additionally, I was able to connect to the Tester machine from our Test Engine host using telnet. It printed the PID of the process once the connection was opened.

I tried running the "ta" application manually on the command line, and it didn't print anything at all.
Maybe the issue is something on the Test Engine side.

On Thu, Aug 24, 2023 at 2:35=E2=80=AFPM Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:

=C2=A0=C2=A0=C2=A0 Hi Adam,

=C2=A0=C2=A0=C2=A0=C2=A0 > On the tester hos= t (which appears to be the Peer agent), there
=C2=A0=C2=A0=C2=A0 are four processes that I se= e running, which look like the test
=C2=A0=C2=A0=C2=A0 agent processes.

=C2=A0=C2=A0=C2=A0 Before the next try I'd = recommend to kill these processes.

=C2=A0=C2=A0=C2=A0 Is there any firewall in the= network or on test hosts which could
=C2=A0=C2=A0=C2=A0 block incoming TCP connectio= n to the port 23571
=C2=A0=C2=A0=C2=A0 <http://iol-dts-tester= .dpdklab.iol.unh.edu:23571> from the host
=C2=A0=C2=A0=C2=A0 where you run test engine? <= br>
=C2=A0=C2=A0=C2=A0 If behaviour the same on the= next try and you see that test agent is
=C2=A0=C2=A0=C2=A0 kept running, could you chec= k using

=C2=A0=C2=A0=C2=A0 # netstat -tnlp

=C2=A0=C2=A0=C2=A0 that Test Agent is listening= on the port and try to establish TCP
=C2=A0=C2=A0=C2=A0 connection from test agent u= sing

=C2=A0=C2=A0=C2=A0 $ telnet iol-dts-tester.dpdklab= .iol.unh.edu
=C2=A0=C2=A0=C2=A0 <http://iol-dts-tester= .dpdklab.iol.unh.edu:23571> 23571
=C2=A0=C2=A0=C2=A0 <http://iol-dts-tester= .dpdklab.iol.unh.edu:23571>

=C2=A0=C2=A0=C2=A0 and check if TCP connection = could be established.

=C2=A0=C2=A0=C2=A0 Another idea is to login Tes= ter under root as testing does, get
=C2=A0=C2=A0=C2=A0 start TA command from the lo= g and try it by hands without -n and
=C2=A0=C2=A0=C2=A0 remove extra escaping.

=C2=A0=C2=A0=C2=A0 # sudo PATH=3D${PATH}:/tmp/linux_x86_root_76872_169288= 5663_1
=C2=A0=C2=A0=C2=A0 LD_LIBRARY_PATH=3D${LD_LIBRARY_PATH}${LD_LIBRARY_PATH:+:}/tmp/linux_x86_roo= t_76872_1692885663_1 /tmp/linux_x86_root_76872_1692885663_1/ta Peer 23571 host=3Diol-dts-tester.dpdklab.iol.unh.edu:port=3D23571:user=3Droot:key=3D/o= pt/tsf/keys/id_ed25519:ssh_port=3D22:copy_timeout=3D15:kill_timeout=3D15:su= do=3D:shell=3D

=C2=A0=C2=A0=C2=A0 Hopefully in this case test = agent directory remains in the /tmp and
=C2=A0=C2=A0=C2=A0 you don't need to copy i= t as testing does.
=C2=A0=C2=A0=C2=A0 May be output could shed som= e light on what's going on.

=C2=A0=C2=A0=C2=A0 Andrew.

=C2=A0=C2=A0=C2=A0 On 8/24/23 17:30, Adam Hassi= ck wrote:
=C2=A0=C2=A0=C2=A0 Hi= Andrew,

=C2=A0=C2=A0=C2=A0 This is the output that I = see in the terminal when this failure
=C2=A0=C2=A0=C2=A0 occurs, after the test age= nt binaries build and the test engine
=C2=A0=C2=A0=C2=A0 starts:

=C2=A0=C2=A0=C2=A0 Platform default build - p= ass
=C2=A0=C2=A0=C2=A0 Simple RCF consistency che= ck succeeded
=C2=A0=C2=A0=C2=A0 --->>> Starting L= ogger...done
=C2=A0=C2=A0=C2=A0 --->>> Starting RCF...rcf_net_engine_connect(): Connection timed
=C2=A0=C2=A0=C2=A0 out iol-dts-tester.dpdk= lab.iol.unh.edu:23571
=C2=A0=C2=A0=C2=A0 <http://iol-dts-test= er.dpdklab.iol.unh.edu:23571>

=C2=A0=C2=A0=C2=A0 Then, it hangs here until = I kill the "te_rcf" and "te_tee" =C2=A0=C2=A0=C2=A0 processes. I let it hang f= or around 9 minutes.

=C2=A0=C2=A0=C2=A0 On the tester host (which = appears to be the Peer agent), there are
=C2=A0=C2=A0=C2=A0 four processes that I see = running, which look like the test agent
=C2=A0=C2=A0=C2=A0 processes.

=C2=A0=C2=A0=C2=A0 ta.Peer is an empty file. = I've attached the log.txt from this run.

=C2=A0=C2=A0=C2=A0 =C2=A0- Adam

=C2=A0=C2=A0=C2=A0 On Thu, Aug 24, 2023 at 4:= 22=E2=80=AFAM Andrew Rybchenko
=C2=A0=C2=A0=C2=A0 <andrew.rybchenko@oktetlabs.ru<= /a>
=C2=A0=C2=A0=C2=A0 <mailto:andrew.rybchenko@oktetl= abs.ru>> wrote:

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi= Adam,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Ye= s, TE_RCFUNIX_TIMEOUT is in seconds. I've double-checked
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 th= at it goes to 'copy_timeout' in ts-conf/rcf.conf.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 De= scription in in doc/sphinx/pages/group_te_engine_rcf.rst
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sa= ys that copy_timeout is in seconds and implementation in
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 li= b/rcfunix/rcfunix.c passes the value to select() tv_sec.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Th= eoretically select() could be interrupted by signal, but I
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 th= ink it is unlikely here.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I&= #39;m not sure that I understand what do you mean by RCF
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 co= nnection timeout. Does it happen on TE startup when RCF
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 st= arts test agents. If so, TE_RCFUNIX_TIMEOUT could help. Or
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 do= es it happen when tests are in progress, e.g. in the middle
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 of= a test. If so, TE_RCFUNIX_TIMEOUT is unrelated and most
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 li= kely either host with test agent dies or test agent itself
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 cr= ashes. It would be easier for me if classify it if you share
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 te= xt log (log.txt, full or just corresponding fragment with
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 so= me context). Also content of ta.DPDK or ta.Peer file
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 de= pending on which agent has problems could shed some light.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Co= rresponding files contain stdout/stderr of test agents.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 An= drew.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On= 8/23/23 17:45, Adam Hassick wrote:
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 Hi Andrew,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = I've set up a test rig repository here, and have created
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = configurations for our development testbed based off of the
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = examples.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = We've been able to get the test suite to run manually on
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Mellanox CX5 devices once.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = However, we are running into an issue where, when RCF starts,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = the RCF connection times out very frequently. We aren't sure
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = why this is the case.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = It works sometimes, but most of the time when we try to run
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = the test engine, it encounters this issue.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = I've tried changing the RCF port by setting
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = "TE_RCF_PORT=3D<some port number>" and rebooting the testbed =
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = machines. Neither seems to fix the issue.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = It also seems like the timeout takes far longer than 60
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = seconds, even when running "export TE_RCFUNIX_TIMEOUT=3D60" =
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = before I try to run the test suite.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = I assume the unit for this variable is seconds?

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Thanks,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Adam

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = On Mon, Aug 21, 2023 at 10:19=E2=80=AFAM Adam Hassick
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = <ahassick@iol.= unh.edu <mailto:ahassick@iol.unh.edu>> wrote:

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 Hi Andrew,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 Thanks, I've cloned the example repository and will start
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 setting up a configuration for our development testbed
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 today. I'll let you know if I run into any difficulties
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 or have any questions.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0- Adam

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 On Sun, Aug 20, 2023 at 4:40=E2=80=AFAM Andrew Rybchenko
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 <andrew.rybchenko@oktetlabs.ru
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 <mailto:andrew.rybchenko@oktetlabs.ru>> wrote:

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Adam,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I've published
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 https://github.com/ts-fact= ory/ts-rigs-sample
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <https://github.com/ts-= factory/ts-rigs-sample>.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hopefully it will help to define your test rigs and
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 successfully run some tests manually. Feel free to
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ask any questions and I'll answer here and try to
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 update documentation.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Meanwhile I'll prepare missing bits for steps (2) and
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (3).
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hopefully everything is in place for step (4), but we
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 need to make steps (2) and (3) first.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Andrew.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 8/18/23 21:40, Andrew Rybchenko wrote:
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Adam,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > I've conferred with the rest of the team, and we
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 think it would be best to move forward with mainly
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 option B.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 OK, I'll provide th= e sample on Monday for you. It is
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 almost ready right now, but I need to double-check
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 it before publishing.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Regards,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Andrew.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 8/17/23 20:03, Adam Hassick wrote:
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Hi Andrew,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I'm adding the CI mailing list to this
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 conversation. Others in the community might find
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 this conversation valuable.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We do want to run testing on a regular basis. The
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Jenkins integration will be very useful for us, as
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 most of our CI is orchestrated by Jenkins.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 I've conferred with the rest of the team, and we
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 think it would be best to move forward with mainly
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 option B.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 If you would like to know anything about our
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 testbeds that would help you with creating an
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 example ts-rigs repo, I'd be happy to answer any =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 questions you have.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 We have multiple test rigs (we call these
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 "DUT-tester pairs&= quot;) that we run our existing
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hardware testing on, with differing network
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hardware and CPU architecture. I figured this might
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 be an important detail.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Thanks,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Adam

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On Thu, Aug 17, 2023 at 11:44=E2=80=AFAM Andrew Rybchenko =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <andrew.rybchenko@oktetlabs.= ru
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <mailto:andrew.rybchenko@okt= etlabs.ru>> wrote:

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Greatings Adam,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= I'm happy to hear that you're trying to bring =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= it up.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= As I understand the final goal is to run it on
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= regular basis. So, we need to make it properly
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= from the very beginning.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Bring up of all features consists of 4 steps:

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 1. Create site-specific repository (we call it
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= ts-rigs) which contains information about test
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= rigs and other site-specific information like
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= where to send mails, where to store logs etc.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= It is required for manual execution as well,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= since test rigs description is essential. I'll
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= return to the topic below.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 2. Setup logs storage for automated runs.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Basically it is a disk space plus apache2 web
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= server with few CGI scripts which help a lot to
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= save disk space.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 3. Setup Bublik web application which provides
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= web interface to view testing results. Same as
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= https://ts-fact= ory.io/bublik
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= <https://ts-= factory.io/bublik>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 4. Setup Jenkins to run tests on regularly,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= save logs in log storage (2) and import it to
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= bublik (3).

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Last few month we spent on our homework to make
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= it simpler to bring up automated execution
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= using Jenkins -
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= htt= ps://github.com/ts-factory/te-jenkins
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= <= ;https://github.com/ts-factory/te-jenkins>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Corresponding bits in dpdk-ethdev-ts will be
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= available tomorrow.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Let's return to the step (1).

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Unfortunately there is no publicly available
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= example of the ts-rigs repository since
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= sensitive site-specific information is located
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= there. But I'm ready to help you to create it
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= for UNH. I see two options here:

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= (A) I'll ask questions and based on your
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= answers will create the first draft with my
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= comments.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= (B) I'll make a template/example ts-rigs repo,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= publish it and you'll create UNH ts-rigs based =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= on it.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Of course, I'll help to debug and finally bring
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= it up in any case.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= (A) is a bit simpler for me and you, but (B) is
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= a bit more generic and will help other
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= potential users to bring it up.
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= We can combine (A)+(B). I.e. start from (A).
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= What do you think?

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Thanks,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= Andrew.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= On 8/17/23 15:18, Konstantin Ushakov wrote:
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Greetings Adam,


=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Thanks for contacting us. I copy Andrew who
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 would be happy to help

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Thanks,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Konstantin

=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 On 16 Aug 2023, at 21:50, Adam Hassick
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 <ahassi= ck@iol.unh.edu>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 <mailto= :ahassick@iol.unh.edu> wrote:

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 =EF=BB=BF
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 Greetings Konstantin,

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 I am in the process of setting up the DPDK
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 Poll Mode Driver test suite as an addition to
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 our testing coverage for DPDK at the UNH lab.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 I have some questions about how to set the
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 test suite arguments.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 I have been able to configure the Test Engine
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 to connect to the hosts in the testbed. The
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 RCF, Configurator, and Tester all begin to
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 run, however the prelude of the test suite
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 fails to run.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 https://ts-factory.io/doc/dpdk-ethdev-ts/inde= x.html#test-parameters <https://= ts-factory.io/doc/dpdk-ethdev-ts/index.html#test-parameters>

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 The documentation mentions that there are
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 several test parameters for the test suite,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 like for the IUT test link MAC, etc. These
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 seem like they would need to be set somewhere
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 to run many of the tests.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 I see in the Test Engine documentation, there
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 are instructions on how to create new
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 parameters for test suites in the Tester
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 configuration, but there is nothing in the
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 user guide or in the Tester guide for how to
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 set the arguments for the parameters when
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 running the test suite that I can find. I'm
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 not sure if I need to write my own Tester
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 config, or if I should be setting these in
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 some other way.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 How should these values be set?

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 I'm also not sure what environment
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 variables/arguments are strictly necessary or
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 which are optional.

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 Regards,
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 Adam

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 -- =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 *Adam Hassick*
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 Senior Developer
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 UNH InterOperability Lab
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 ahassick@i= ol.unh.edu
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 <mailto= :ahassick@iol.unh.edu>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 iol.unh.edu <https://www.iol.unh.edu/>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 +1 (603) 475-8248



=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Adam Hassick*
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Senior Developer
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 UNH InterOperability Lab
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 iol.unh.edu <https://www.iol.unh.edu/>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 +1 (603) 475-8248




=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 *Adam Hassick*
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 Senior Developer
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 UNH InterOperability Lab
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 i= ol.unh.edu <https://www.iol.unh.edu/>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 +1 (603) 475-8248



=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = -- =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Adam Hassick*
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Senior Developer
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = UNH InterOperability Lab
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ahassick@iol.unh.= edu <mailto:ahassick@iol.unh.edu>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = iol.unh.edu <https://www.iol.unh.edu/>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = +1 (603) 475-8248



=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0 *Ada= m Hassick*
=C2=A0=C2=A0=C2=A0 Senior Developer
=C2=A0=C2=A0=C2=A0 UNH InterOperability Lab <= br> =C2=A0=C2=A0=C2=A0 ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
=C2=A0=C2=A0=C2=A0 iol.unh.edu <https://www.iol.unh.edu/>
=C2=A0=C2=A0=C2=A0 +1 (603) 475-8248



--=C2=A0
*Adam Hassick*
Senior Developer
UNH InterOperability Lab
ahassick@iol.unh.edu <mailto:ahassick@iol.unh.edu>
iol.unh.edu &l= t;https://www.iol.unh.edu/>
+1 (603) 475-8248




--
Adam Hassick
Senior Developer
UNH InterOperability Lab
+1 (603) 475-8248




--
Adam Hassick
Senior Developer
UNH InterOperability Lab
+1 (603) 475-8248




--
Adam Hassick
Senior Developer
UNH InterOperability Lab
+1 (603) 475-8248
--000000000000961db70604104034--