Re: [dpdk-ci] [dpdklab] Re: New 100G Broadcom NIC Ubuntu

[Brandon Lo <blo@iol.unh.edu>]

Hi Ajit,

Hope you are doing well. Have you had a chance to investigate what
could cause this issue?
At UNH, we are facing a similar issue with some of the new Arm systems.
Maybe they could be related and help us figure out how to fix the
Broadcom machine as well.
I will let you know if anything pops up that could be interesting.

Thanks,
Brandon

On Thu, Oct 22, 2020 at 3:27 PM 'Ajit Khaparde' via dpdklab
<dpdklab@iol.unh.edu> wrote:
>
> Hi Brandon,
> Yes. I haven't been able to spend time on this because of the release.
> But I will try to get to it soon.
> Thanks for checking.
>
> Thanks
> Ajit
>
> On Thu, Oct 22, 2020 at 12:04 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > Just checking in to make sure this issue is not dropped.
> > I have tested the 100G NIC running TREX/DTS again, it still seems that
> > the 100G NIC is not able to use TREX as the console is outputting
> > "Connection lost - Subscriber timeout: no data from TRex server."
> > However, I know that some developers are busy for DPDK so I understand
> > if you are a bit pressed on time currently.
> >
> > Thanks,
> > Brandon
> >
> > On Tue, Sep 1, 2020 at 2:12 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> > >
> > > Hi Brandon,
> > > Thanks for this.
> > > I have been stretched thin lately, and this will surely help.
> > >
> > > Thanks
> > > Ajit
> > >
> > > On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo <blo@iol.unh.edu> wrote:
> > >>
> > >> Hi Ajit,
> > >>
> > >> I'm going to re-enable the Broadcom 25G NIC in production performance
> > >> and functional testing.
> > >>
> > >> If you would like to do testing/debugging on the machine, such as on
> > >> the 100G NIC, then please disable these two pipelines on our Jenkins:
> > >>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/
> > >>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/
> > >>
> > >> You will need to sign in using the same credentials as your VPN.
> > >>
> > >> This will prevent any overlapping of DTS/TREX runs which is known to
> > >> cause errors.
> > >> Please let me know if you run into any issues.
> > >>
> > >> Thanks,
> > >> Brandon
> > >>
> > >> On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde
> > >> <ajit.khaparde@broadcom.com> wrote:
> > >> >
> > >> > Not yet Brandon.
> > >> > But I believe I might have to look at what trex is doing to make progress.
> > >> > I will let you know if I need any help.
> > >> >
> > >> > Thanks
> > >> > Ajit
> > >> >
> > >> > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >>
> > >> >> Hi Ajit,
> > >> >>
> > >> >> Did you find anything of interest on the machine?
> > >> >> I could help out if needed.
> > >> >>
> > >> >> Thanks,
> > >> >> Brandon
> > >> >>
> > >> >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
> > >> >> <ajit.khaparde@broadcom.com> wrote:
> > >> >> >
> > >> >> > Hi Brandon,
> > >> >> > I do see some issues while running trex on this setup.
> > >> >> > I will have to dig further. I will let you know once I find something or may be a fix.
> > >> >> >
> > >> >> > Thanks
> > >> >> > Ajit
> > >> >> >
> > >> >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >> >>
> > >> >> >> Hi Ajit,
> > >> >> >>
> > >> >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
> > >> >> >> testpmd (through an ssh session from io --> rhea) to catch information
> > >> >> >> as a DUT.
> > >> >> >>
> > >> >> >> The only commands that I run (on io) are:
> > >> >> >> cd /opt/dts
> > >> >> >> export DTS_CFG_FOLDER='conf_100g'
> > >> >> >> ./dts -s
> > >> >> >>
> > >> >> >> The rest is managed by DTS itself.
> > >> >> >> The issue occurs when DTS is trying to send packets to rhea from io on
> > >> >> >> the new 100G NIC.
> > >> >> >> It seems to happen after it tries to measure throughput using TREX's API.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> Brandon
> > >> >> >>
> > >> >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
> > >> >> >> <ajit.khaparde@broadcom.com> wrote:
> > >> >> >> >
> > >> >> >> > Hi Brandon,
> > >> >> >> > I was trying to see what exactly is happening on the setup and what is causing the problem.
> > >> >> >> > But I will need your help to use the proper code and commands.
> > >> >> >> >
> > >> >> >> > I believe you run the testpmd command on io. And the trex is executed on rhea?
> > >> >> >> > Can you point me to the location of the code and the steps you are following while running the test?
> > >> >> >> >
> > >> >> >> > Thanks
> > >> >> >> > Ajit
> > >> >> >> >
> > >> >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> > >> >> >> >>
> > >> >> >> >> Hi Brandon,
> > >> >> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office.
> > >> >> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN.
> > >> >> >> >> Thanks for checking in. I will try to update you as soon as I can.
> > >> >> >> >>
> > >> >> >> >> Thanks
> > >> >> >> >> Ajit
> > >> >> >> >>
> > >> >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >> >> >>>
> > >> >> >> >>> Hi Ajit,
> > >> >> >> >>>
> > >> >> >> >>> I'm just checking in; have you heard of any updates on this issue?
> > >> >> >> >>>
> > >> >> >> >>> Thanks,
> > >> >> >> >>> Brandon
> > >> >> >> >>>
> > >> >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >> >> >>>>
> > >> >> >> >>>> Hi Ajit,
> > >> >> >> >>>>
> > >> >> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.
> > >> >> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.
> > >> >> >> >>>>
> > >> >> >> >>>> If you want to replicate it, here are the steps that I ran:
> > >> >> >> >>>>
> > >> >> >> >>>> (on io) cd /opt/dts
> > >> >> >> >>>> export DTS_CFG_FOLDER='conf_100g'
> > >> >> >> >>>>
> > >> >> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file
> > >> >> >> >>>>
> > >> >> >> >>>> ./dts -s
> > >> >> >> >>>>
> > >> >> >> >>>>
> > >> >> >> >>>> Thanks for your help,
> > >> >> >> >>>> Brandon
> > >> >> >> >>>>
> > >> >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> > >> >> >> >>>>>
> > >> >> >> >>>>> Hi Brandon,
> > >> >> >> >>>>> No, I haven't seen or heard this before.
> > >> >> >> >>>>> But I will try to have someone run it again.
> > >> >> >> >>>>>
> > >> >> >> >>>>> Just to make sure -
> > >> >> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?
> > >> >> >> >>>>> Or is it the system that is running testpmd where you are running into the issue?
> > >> >> >> >>>>>
> > >> >> >> >>>>> Thanks
> > >> >> >> >>>>> Ajit
> > >> >> >> >>>>>
> > >> >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> Hi Ajit,
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.
> > >> >> >> >>>>>> The current configuration uses trex version v2.82.
> > >> >> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> The output looks like this when it reaches a test case:
> > >> >> >> >>>>>>
> > >> >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
> > >> >> >> >>>>>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512 --rxd=512
> > >> >> >> >>>>>>>                       dut.rhea: start
> > >> >> >> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> > >> >> >> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
> > >> >> >> >>>>>>>     case_obj()
> > >> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core
> > >> >> >> >>>>>>>     self.perf_test(self.nb_ports)
> > >> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
> > >> >> >> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
> > >> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> > >> >> >> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> > >> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission
> > >> >> >> >>>>>>>     self._conn.reset(ports=self._ports)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> > >> >> >> >>>>>>>     ret = f(*args, **kwargs)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
> > >> >> >> >>>>>>>     self.clear_stats(ports)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> > >> >> >> >>>>>>>     ret = f(*args, **kwargs)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats
> > >> >> >> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common
> > >> >> >> >>>>>>>     raise TRexError(rc)
> > >> >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501
> > >> >> >> >>>>>>
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
> > >> >> >> >>>>>> Have you ran into this issue before?
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> Thanks,
> > >> >> >> >>>>>> Brandon
> > >> >> >> >>>>
> > >> >> >> >>>>
> > >> >> >> >>>>
> > >> >> >> >>>> --
> > >> >> >> >>>>
> > >> >> >> >>>> Brandon Lo
> > >> >> >> >>>>
> > >> >> >> >>>> UNH InterOperability Laboratory
> > >> >> >> >>>>
> > >> >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >> >> >> >>>>
> > >> >> >> >>>> blo@iol.unh.edu
> > >> >> >> >>>>
> > >> >> >> >>>> www.iol.unh.edu
> > >> >> >> >>>
> > >> >> >> >>>
> > >> >> >> >>>
> > >> >> >> >>> --
> > >> >> >> >>>
> > >> >> >> >>> Brandon Lo
> > >> >> >> >>>
> > >> >> >> >>> UNH InterOperability Laboratory
> > >> >> >> >>>
> > >> >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >> >> >> >>>
> > >> >> >> >>> blo@iol.unh.edu
> > >> >> >> >>>
> > >> >> >> >>> www.iol.unh.edu
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> --
> > >> >> >>
> > >> >> >> Brandon Lo
> > >> >> >>
> > >> >> >> UNH InterOperability Laboratory
> > >> >> >>
> > >> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >> >> >>
> > >> >> >> blo@iol.unh.edu
> > >> >> >>
> > >> >> >> www.iol.unh.edu
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >>
> > >> >> Brandon Lo
> > >> >>
> > >> >> UNH InterOperability Laboratory
> > >> >>
> > >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >> >>
> > >> >> blo@iol.unh.edu
> > >> >>
> > >> >> www.iol.unh.edu
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Brandon Lo
> > >>
> > >> UNH InterOperability Laboratory
> > >>
> > >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >>
> > >> blo@iol.unh.edu
> > >>
> > >> www.iol.unh.edu
> >
> >
> >
> > --
> >
> > Brandon Lo
> >
> > UNH InterOperability Laboratory
> >
> > 21 Madbury Rd, Suite 100, Durham, NH 03824
> >
> > blo@iol.unh.edu
> >
> > www.iol.unh.edu



-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu