Hi Brandon, Thanks for this. I have been stretched thin lately, and this will surely help. Thanks Ajit On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo wrote: > Hi Ajit, > > I'm going to re-enable the Broadcom 25G NIC in production performance > and functional testing. > > If you would like to do testing/debugging on the machine, such as on > the 100G NIC, then please disable these two pipelines on our Jenkins: > > https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/ > > https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/ > > You will need to sign in using the same credentials as your VPN. > > This will prevent any overlapping of DTS/TREX runs which is known to > cause errors. > Please let me know if you run into any issues. > > Thanks, > Brandon > > On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde > wrote: > > > > Not yet Brandon. > > But I believe I might have to look at what trex is doing to make > progress. > > I will let you know if I need any help. > > > > Thanks > > Ajit > > > > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo wrote: > >> > >> Hi Ajit, > >> > >> Did you find anything of interest on the machine? > >> I could help out if needed. > >> > >> Thanks, > >> Brandon > >> > >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde > >> wrote: > >> > > >> > Hi Brandon, > >> > I do see some issues while running trex on this setup. > >> > I will have to dig further. I will let you know once I find something > or may be a fix. > >> > > >> > Thanks > >> > Ajit > >> > > >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo wrote: > >> >> > >> >> Hi Ajit, > >> >> > >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs > >> >> testpmd (through an ssh session from io --> rhea) to catch > information > >> >> as a DUT. > >> >> > >> >> The only commands that I run (on io) are: > >> >> cd /opt/dts > >> >> export DTS_CFG_FOLDER='conf_100g' > >> >> ./dts -s > >> >> > >> >> The rest is managed by DTS itself. > >> >> The issue occurs when DTS is trying to send packets to rhea from io > on > >> >> the new 100G NIC. > >> >> It seems to happen after it tries to measure throughput using TREX's > API. > >> >> > >> >> Thanks, > >> >> Brandon > >> >> > >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde > >> >> wrote: > >> >> > > >> >> > Hi Brandon, > >> >> > I was trying to see what exactly is happening on the setup and > what is causing the problem. > >> >> > But I will need your help to use the proper code and commands. > >> >> > > >> >> > I believe you run the testpmd command on io. And the trex is > executed on rhea? > >> >> > Can you point me to the location of the code and the steps you are > following while running the test? > >> >> > > >> >> > Thanks > >> >> > Ajit > >> >> > > >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde < > ajit.khaparde@broadcom.com> wrote: > >> >> >> > >> >> >> Hi Brandon, > >> >> >> I haven't. I tried to login as well. But I had some issues doing > it from the office. > >> >> >> I just have to remind myself to try it again once I get home > before I connect to the company VPN. > >> >> >> Thanks for checking in. I will try to update you as soon as I can. > >> >> >> > >> >> >> Thanks > >> >> >> Ajit > >> >> >> > >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo > wrote: > >> >> >>> > >> >> >>> Hi Ajit, > >> >> >>> > >> >> >>> I'm just checking in; have you heard of any updates on this > issue? > >> >> >>> > >> >> >>> Thanks, > >> >> >>> Brandon > >> >> >>> > >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo > wrote: > >> >> >>>> > >> >> >>>> Hi Ajit, > >> >> >>>> > >> >> >>>> Yes, I believe the issue is coming from the trex/tester system > with the 100G NIC. > >> >> >>>> I'm not sure what causes this issue; if I run trex using the > command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", > which is the same command used in DTS, it seems to launch without failing. > >> >> >>>> > >> >> >>>> If you want to replicate it, here are the steps that I ran: > >> >> >>>> > >> >> >>>> (on io) cd /opt/dts > >> >> >>>> export DTS_CFG_FOLDER='conf_100g' > >> >> >>>> > >> >> >>>> conf_100g has the new configuration files to use the new PCI id > and pktgen config file > >> >> >>>> > >> >> >>>> ./dts -s > >> >> >>>> > >> >> >>>> > >> >> >>>> Thanks for your help, > >> >> >>>> Brandon > >> >> >>>> > >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde < > ajit.khaparde@broadcom.com> wrote: > >> >> >>>>> > >> >> >>>>> Hi Brandon, > >> >> >>>>> No, I haven't seen or heard this before. > >> >> >>>>> But I will try to have someone run it again. > >> >> >>>>> > >> >> >>>>> Just to make sure - > >> >> >>>>> You are running trex on the 100G NIC and the problem is > encountered on that setup? > >> >> >>>>> Or is it the system that is running testpmd where you are > running into the issue? > >> >> >>>>> > >> >> >>>>> Thanks > >> >> >>>>> Ajit > >> >> >>>>> > >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo > wrote: > >> >> >>>>>> > >> >> >>>>>> Hi Ajit, > >> >> >>>>>> > >> >> >>>>>> I'm running into a problem with trying to run > nic_single_core_perf on the new NIC. > >> >> >>>>>> The current configuration uses trex version v2.82. > >> >> >>>>>> However, I'm running into an error when it tries to actually > do a test case in the nic_single_core_perf. > >> >> >>>>>> > >> >> >>>>>> The output looks like this when it reaches a test case: > >> >> >>>>>> > >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: > framesize: 64, rxd/txd: 512 > >> >> >>>>>>> dut.rhea: > ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w > 0000:81:00.1 --file-prefix=dpdk_11307_20200804160513 -- -i > --portmask=0x3 --txd=512 --rxd=512 > >> >> >>>>>>> dut.rhea: start > >> >> >>>>>>> TestNicSingleCorePerf: Test Case > test_perf_nic_single_core Result ERROR: Traceback (most recent call last): > >> >> >>>>>>> File "/opt/dts/framework/test_case.py", line 316, in > _execute_test_case > >> >> >>>>>>> case_obj() > >> >> >>>>>>> File "tests/TestSuite_nic_single_core_perf.py", line 198, > in test_perf_nic_single_core > >> >> >>>>>>> self.perf_test(self.nb_ports) > >> >> >>>>>>> File "tests/TestSuite_nic_single_core_perf.py", line 259, > in perf_test > >> >> >>>>>>> _, packets_received = > self.tester.pktgen.measure_throughput(stream_ids=streams, > options=traffic_opt) > >> >> >>>>>>> File "/opt/dts/framework/pktgen_base.py", line 245, in > measure_throughput > >> >> >>>>>>> self._prepare_transmission(stream_ids=stream_ids) > >> >> >>>>>>> File "/opt/dts/framework/pktgen_trex.py", line 779, in > _prepare_transmission > >> >> >>>>>>> self._conn.reset(ports=self._ports) > >> >> >>>>>>> File > "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", > line 51, in wrap2 > >> >> >>>>>>> ret = f(*args, **kwargs) > >> >> >>>>>>> File > "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", > line 339, in reset > >> >> >>>>>>> self.clear_stats(ports) > >> >> >>>>>>> File > "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", > line 51, in wrap2 > >> >> >>>>>>> ret = f(*args, **kwargs) > >> >> >>>>>>> File > "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", > line 1467, in clear_stats > >> >> >>>>>>> self._clear_stats_common(ports, clear_global, > clear_xstats) > >> >> >>>>>>> File > "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", > line 2840, in _clear_stats_common > >> >> >>>>>>> raise TRexError(rc) > >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to > get server response from tcp://127.0.0.1:4501 > >> >> >>>>>> > >> >> >>>>>> > >> >> >>>>>> I have found one similar case on the github repository for > trex, but the solution was vendor-specific: > https://github.com/cisco-system-traffic-generator/trex-core/issues/147. > >> >> >>>>>> Have you ran into this issue before? > >> >> >>>>>> > >> >> >>>>>> Thanks, > >> >> >>>>>> Brandon > >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> -- > >> >> >>>> > >> >> >>>> Brandon Lo > >> >> >>>> > >> >> >>>> UNH InterOperability Laboratory > >> >> >>>> > >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824 > >> >> >>>> > >> >> >>>> blo@iol.unh.edu > >> >> >>>> > >> >> >>>> www.iol.unh.edu > >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> -- > >> >> >>> > >> >> >>> Brandon Lo > >> >> >>> > >> >> >>> UNH InterOperability Laboratory > >> >> >>> > >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824 > >> >> >>> > >> >> >>> blo@iol.unh.edu > >> >> >>> > >> >> >>> www.iol.unh.edu > >> >> > >> >> > >> >> > >> >> -- > >> >> > >> >> Brandon Lo > >> >> > >> >> UNH InterOperability Laboratory > >> >> > >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 > >> >> > >> >> blo@iol.unh.edu > >> >> > >> >> www.iol.unh.edu > >> > >> > >> > >> -- > >> > >> Brandon Lo > >> > >> UNH InterOperability Laboratory > >> > >> 21 Madbury Rd, Suite 100, Durham, NH 03824 > >> > >> blo@iol.unh.edu > >> > >> www.iol.unh.edu > > > > -- > > Brandon Lo > > UNH InterOperability Laboratory > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > blo@iol.unh.edu > > www.iol.unh.edu >