From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 87110A0524 for ; Thu, 3 Dec 2020 17:49:07 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 60D13C970; Thu, 3 Dec 2020 17:49:06 +0100 (CET) Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) by dpdk.org (Postfix) with ESMTP id 83768C954 for ; Thu, 3 Dec 2020 17:49:03 +0100 (CET) Received: by mail-lf1-f52.google.com with SMTP id d20so3661410lfe.11 for ; Thu, 03 Dec 2020 08:49:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qu88GxGf5AY6oB2M5Yt2lC7n/ZO6G+dosFP021Ievmk=; b=iaze1t7/BDTThYDzWGTOwlgakE8o1a0Qm5tYfye338FqL2OQy1uHWqy7fDs2pZy1kv HWYXo55fuTDzqT9MECMxAwC2Ix7yG5tBJ7ZHpDsFQWVltnmIxVx3/z658V/Z2fGtonMG ZehLunMqJcnYzwqtpb1l/sZsnDxKPk2FQPf+8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qu88GxGf5AY6oB2M5Yt2lC7n/ZO6G+dosFP021Ievmk=; b=MohNvHvtez1waZUVRUsUxOoxpIjjKSSVH4bmnBi2YyRi/u/24hLh/ePq1F7ScthhES g6FR+2/E/KSizIPaEOoBAzzfFLSELKbl3XbBS8POPN8gNZD8DKq/Fkc+RrZe3FADxYTh hyHEIy7fvLdq1CMR0WHKFKxWfYFC6LyQrbAFQyxeg5h//6o+sT4Qn4znNLC0cI8go+oj PLepxeadbQ6BGAx11/7DSFas/MZCnO+GRPgVybDY5/YzIEhs3sZGnhRULNKb/BENJ4pB orPNfkbtLmO806vTZIIECV+Z8vn4cl+nwE5KVC1P14f1gPcfujGVjJLrp0Y0BHajpEZl Nv3A== X-Gm-Message-State: AOAM533TTFKSmXQ8bvMgZ6Wa5q6rjwlfcuStDQ1lQTifl+j+iXEj/hlW fmtuE7UFBfIN90FiRbjqZBu/DD0Y+z0NRIhVQGWg/Q== X-Google-Smtp-Source: ABdhPJxeTdc21txOhmp4taC+E94sZuR0dKwQY/ueCbTGiFONwfJ0GBKo1HhdNonEDtAdgVnMQm1dkQNMqg96FpSysEw= X-Received: by 2002:a19:794:: with SMTP id 142mr1664192lfh.232.1607014142753; Thu, 03 Dec 2020 08:49:02 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Brandon Lo Date: Thu, 3 Dec 2020 11:48:26 -0500 Message-ID: To: Ajit Khaparde Cc: ci@dpdk.org, dpdklab Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-ci] [dpdklab] Re: New 100G Broadcom NIC Ubuntu X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" Hi Ajit, Hope you are doing well. Have you had a chance to investigate what could cause this issue? At UNH, we are facing a similar issue with some of the new Arm systems. Maybe they could be related and help us figure out how to fix the Broadcom machine as well. I will let you know if anything pops up that could be interesting. Thanks, Brandon On Thu, Oct 22, 2020 at 3:27 PM 'Ajit Khaparde' via dpdklab wrote: > > Hi Brandon, > Yes. I haven't been able to spend time on this because of the release. > But I will try to get to it soon. > Thanks for checking. > > Thanks > Ajit > > On Thu, Oct 22, 2020 at 12:04 PM Brandon Lo wrote: > > > > Hi Ajit, > > > > Just checking in to make sure this issue is not dropped. > > I have tested the 100G NIC running TREX/DTS again, it still seems that > > the 100G NIC is not able to use TREX as the console is outputting > > "Connection lost - Subscriber timeout: no data from TRex server." > > However, I know that some developers are busy for DPDK so I understand > > if you are a bit pressed on time currently. > > > > Thanks, > > Brandon > > > > On Tue, Sep 1, 2020 at 2:12 PM Ajit Khaparde wrote: > > > > > > Hi Brandon, > > > Thanks for this. > > > I have been stretched thin lately, and this will surely help. > > > > > > Thanks > > > Ajit > > > > > > On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo wrote: > > >> > > >> Hi Ajit, > > >> > > >> I'm going to re-enable the Broadcom 25G NIC in production performance > > >> and functional testing. > > >> > > >> If you would like to do testing/debugging on the machine, such as on > > >> the 100G NIC, then please disable these two pipelines on our Jenkins: > > >> https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/ > > >> https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/ > > >> > > >> You will need to sign in using the same credentials as your VPN. > > >> > > >> This will prevent any overlapping of DTS/TREX runs which is known to > > >> cause errors. > > >> Please let me know if you run into any issues. > > >> > > >> Thanks, > > >> Brandon > > >> > > >> On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde > > >> wrote: > > >> > > > >> > Not yet Brandon. > > >> > But I believe I might have to look at what trex is doing to make progress. > > >> > I will let you know if I need any help. > > >> > > > >> > Thanks > > >> > Ajit > > >> > > > >> > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo wrote: > > >> >> > > >> >> Hi Ajit, > > >> >> > > >> >> Did you find anything of interest on the machine? > > >> >> I could help out if needed. > > >> >> > > >> >> Thanks, > > >> >> Brandon > > >> >> > > >> >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde > > >> >> wrote: > > >> >> > > > >> >> > Hi Brandon, > > >> >> > I do see some issues while running trex on this setup. > > >> >> > I will have to dig further. I will let you know once I find something or may be a fix. > > >> >> > > > >> >> > Thanks > > >> >> > Ajit > > >> >> > > > >> >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo wrote: > > >> >> >> > > >> >> >> Hi Ajit, > > >> >> >> > > >> >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs > > >> >> >> testpmd (through an ssh session from io --> rhea) to catch information > > >> >> >> as a DUT. > > >> >> >> > > >> >> >> The only commands that I run (on io) are: > > >> >> >> cd /opt/dts > > >> >> >> export DTS_CFG_FOLDER='conf_100g' > > >> >> >> ./dts -s > > >> >> >> > > >> >> >> The rest is managed by DTS itself. > > >> >> >> The issue occurs when DTS is trying to send packets to rhea from io on > > >> >> >> the new 100G NIC. > > >> >> >> It seems to happen after it tries to measure throughput using TREX's API. > > >> >> >> > > >> >> >> Thanks, > > >> >> >> Brandon > > >> >> >> > > >> >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde > > >> >> >> wrote: > > >> >> >> > > > >> >> >> > Hi Brandon, > > >> >> >> > I was trying to see what exactly is happening on the setup and what is causing the problem. > > >> >> >> > But I will need your help to use the proper code and commands. > > >> >> >> > > > >> >> >> > I believe you run the testpmd command on io. And the trex is executed on rhea? > > >> >> >> > Can you point me to the location of the code and the steps you are following while running the test? > > >> >> >> > > > >> >> >> > Thanks > > >> >> >> > Ajit > > >> >> >> > > > >> >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde wrote: > > >> >> >> >> > > >> >> >> >> Hi Brandon, > > >> >> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office. > > >> >> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN. > > >> >> >> >> Thanks for checking in. I will try to update you as soon as I can. > > >> >> >> >> > > >> >> >> >> Thanks > > >> >> >> >> Ajit > > >> >> >> >> > > >> >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo wrote: > > >> >> >> >>> > > >> >> >> >>> Hi Ajit, > > >> >> >> >>> > > >> >> >> >>> I'm just checking in; have you heard of any updates on this issue? > > >> >> >> >>> > > >> >> >> >>> Thanks, > > >> >> >> >>> Brandon > > >> >> >> >>> > > >> >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo wrote: > > >> >> >> >>>> > > >> >> >> >>>> Hi Ajit, > > >> >> >> >>>> > > >> >> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC. > > >> >> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing. > > >> >> >> >>>> > > >> >> >> >>>> If you want to replicate it, here are the steps that I ran: > > >> >> >> >>>> > > >> >> >> >>>> (on io) cd /opt/dts > > >> >> >> >>>> export DTS_CFG_FOLDER='conf_100g' > > >> >> >> >>>> > > >> >> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file > > >> >> >> >>>> > > >> >> >> >>>> ./dts -s > > >> >> >> >>>> > > >> >> >> >>>> > > >> >> >> >>>> Thanks for your help, > > >> >> >> >>>> Brandon > > >> >> >> >>>> > > >> >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde wrote: > > >> >> >> >>>>> > > >> >> >> >>>>> Hi Brandon, > > >> >> >> >>>>> No, I haven't seen or heard this before. > > >> >> >> >>>>> But I will try to have someone run it again. > > >> >> >> >>>>> > > >> >> >> >>>>> Just to make sure - > > >> >> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup? > > >> >> >> >>>>> Or is it the system that is running testpmd where you are running into the issue? > > >> >> >> >>>>> > > >> >> >> >>>>> Thanks > > >> >> >> >>>>> Ajit > > >> >> >> >>>>> > > >> >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo wrote: > > >> >> >> >>>>>> > > >> >> >> >>>>>> Hi Ajit, > > >> >> >> >>>>>> > > >> >> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC. > > >> >> >> >>>>>> The current configuration uses trex version v2.82. > > >> >> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf. > > >> >> >> >>>>>> > > >> >> >> >>>>>> The output looks like this when it reaches a test case: > > >> >> >> >>>>>> > > >> >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512 > > >> >> >> >>>>>>> dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1 --file-prefix=dpdk_11307_20200804160513 -- -i --portmask=0x3 --txd=512 --rxd=512 > > >> >> >> >>>>>>> dut.rhea: start > > >> >> >> >>>>>>> TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last): > > >> >> >> >>>>>>> File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case > > >> >> >> >>>>>>> case_obj() > > >> >> >> >>>>>>> File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core > > >> >> >> >>>>>>> self.perf_test(self.nb_ports) > > >> >> >> >>>>>>> File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test > > >> >> >> >>>>>>> _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt) > > >> >> >> >>>>>>> File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput > > >> >> >> >>>>>>> self._prepare_transmission(stream_ids=stream_ids) > > >> >> >> >>>>>>> File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission > > >> >> >> >>>>>>> self._conn.reset(ports=self._ports) > > >> >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2 > > >> >> >> >>>>>>> ret = f(*args, **kwargs) > > >> >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset > > >> >> >> >>>>>>> self.clear_stats(ports) > > >> >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2 > > >> >> >> >>>>>>> ret = f(*args, **kwargs) > > >> >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats > > >> >> >> >>>>>>> self._clear_stats_common(ports, clear_global, clear_xstats) > > >> >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common > > >> >> >> >>>>>>> raise TRexError(rc) > > >> >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501 > > >> >> >> >>>>>> > > >> >> >> >>>>>> > > >> >> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147. > > >> >> >> >>>>>> Have you ran into this issue before? > > >> >> >> >>>>>> > > >> >> >> >>>>>> Thanks, > > >> >> >> >>>>>> Brandon > > >> >> >> >>>> > > >> >> >> >>>> > > >> >> >> >>>> > > >> >> >> >>>> -- > > >> >> >> >>>> > > >> >> >> >>>> Brandon Lo > > >> >> >> >>>> > > >> >> >> >>>> UNH InterOperability Laboratory > > >> >> >> >>>> > > >> >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824 > > >> >> >> >>>> > > >> >> >> >>>> blo@iol.unh.edu > > >> >> >> >>>> > > >> >> >> >>>> www.iol.unh.edu > > >> >> >> >>> > > >> >> >> >>> > > >> >> >> >>> > > >> >> >> >>> -- > > >> >> >> >>> > > >> >> >> >>> Brandon Lo > > >> >> >> >>> > > >> >> >> >>> UNH InterOperability Laboratory > > >> >> >> >>> > > >> >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824 > > >> >> >> >>> > > >> >> >> >>> blo@iol.unh.edu > > >> >> >> >>> > > >> >> >> >>> www.iol.unh.edu > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> -- > > >> >> >> > > >> >> >> Brandon Lo > > >> >> >> > > >> >> >> UNH InterOperability Laboratory > > >> >> >> > > >> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 > > >> >> >> > > >> >> >> blo@iol.unh.edu > > >> >> >> > > >> >> >> www.iol.unh.edu > > >> >> > > >> >> > > >> >> > > >> >> -- > > >> >> > > >> >> Brandon Lo > > >> >> > > >> >> UNH InterOperability Laboratory > > >> >> > > >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 > > >> >> > > >> >> blo@iol.unh.edu > > >> >> > > >> >> www.iol.unh.edu > > >> > > >> > > >> > > >> -- > > >> > > >> Brandon Lo > > >> > > >> UNH InterOperability Laboratory > > >> > > >> 21 Madbury Rd, Suite 100, Durham, NH 03824 > > >> > > >> blo@iol.unh.edu > > >> > > >> www.iol.unh.edu > > > > > > > > -- > > > > Brandon Lo > > > > UNH InterOperability Laboratory > > > > 21 Madbury Rd, Suite 100, Durham, NH 03824 > > > > blo@iol.unh.edu > > > > www.iol.unh.edu -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu