From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BED68A04AC for ; Tue, 1 Sep 2020 20:09:58 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 93ADA1C0B1; Tue, 1 Sep 2020 20:09:58 +0200 (CEST) Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) by dpdk.org (Postfix) with ESMTP id E17D21C0AF for ; Tue, 1 Sep 2020 20:09:57 +0200 (CEST) Received: by mail-io1-f45.google.com with SMTP id b6so1817350iof.6 for ; Tue, 01 Sep 2020 11:09:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=CHRNG8K9rQbast0/0qF/wS1eZokx/vWC8rOYvUroar0=; b=bHKIsUoMzyE339vSXlaEdYkT1fvb7q1xNyJAI+ZUq46gZRL0rr40S8pBncK2elKuiQ SffCfGT6etm8t6tG+ODlTVN0wkVpaOGmA2WKukXdSNxLPbgyMbzJmQzpXX3VtF5+F3w7 ae0TeeOApjmKFpQy/SzQ73aWMRmopUo/lAxXM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CHRNG8K9rQbast0/0qF/wS1eZokx/vWC8rOYvUroar0=; b=rlBiC7lfzUOhfg6+QWo9/TSpwNjqIaqjdzBsaY8P0XCR9mh7aHu00loNg2Uf2cZjmw wwZfGy5lF6f2QGeQo5lwmbwVi2zJ+gmA0PrRTvGjVX4EnahW1bRHoQtsqv7oLKAh/G9H 4MLV5rqKWxdlGjIrRFXPMHDZ9to0MC8D4H3eeL5NpvWTPYInuVaybWHlv2e2FSnhSX87 mXxu1xh698ODyNdfoYiuBpdK0oFd/aOwVf8YFREjGwVit6z6dvkVS7z3lMbYqTWH+BcI DY2RLd6u1+UMydidI/HIRIwlpnneFOTk1iQf5n6wqBRRVc15pVjRt2vN6lTGDd7Du/FO Yz8A== X-Gm-Message-State: AOAM530O0o+YwHJ+F4dYYU8QJdCLXA+HZEsygrzEvx1HoJnJQU1V/KuK kLOZxQUDNVDN2MM7RTbXbenGNrCG2hWmsQdllNUfDw== X-Google-Smtp-Source: ABdhPJxtow2x6x7LR6TIBQZ0jDfE8w26MoK+Hrs2+YO36cKN989/rdd8LSr3xV7ZaNt2GnNUoSVpsAq5Sam6zREE45A= X-Received: by 2002:a5d:888b:: with SMTP id d11mr253019ioo.188.1598983796946; Tue, 01 Sep 2020 11:09:56 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Brandon Lo Date: Tue, 1 Sep 2020 14:09:21 -0400 Message-ID: To: Ajit Khaparde Cc: ci@dpdk.org, dpdklab Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org Sender: "ci" Hi Ajit, I'm going to re-enable the Broadcom 25G NIC in production performance and functional testing. If you would like to do testing/debugging on the machine, such as on the 100G NIC, then please disable these two pipelines on our Jenkins: https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/ https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/ You will need to sign in using the same credentials as your VPN. This will prevent any overlapping of DTS/TREX runs which is known to cause errors. Please let me know if you run into any issues. Thanks, Brandon On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde wrote: > > Not yet Brandon. > But I believe I might have to look at what trex is doing to make progress. > I will let you know if I need any help. > > Thanks > Ajit > > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo wrote: >> >> Hi Ajit, >> >> Did you find anything of interest on the machine? >> I could help out if needed. >> >> Thanks, >> Brandon >> >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde >> wrote: >> > >> > Hi Brandon, >> > I do see some issues while running trex on this setup. >> > I will have to dig further. I will let you know once I find something or may be a fix. >> > >> > Thanks >> > Ajit >> > >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo wrote: >> >> >> >> Hi Ajit, >> >> >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs >> >> testpmd (through an ssh session from io --> rhea) to catch information >> >> as a DUT. >> >> >> >> The only commands that I run (on io) are: >> >> cd /opt/dts >> >> export DTS_CFG_FOLDER='conf_100g' >> >> ./dts -s >> >> >> >> The rest is managed by DTS itself. >> >> The issue occurs when DTS is trying to send packets to rhea from io on >> >> the new 100G NIC. >> >> It seems to happen after it tries to measure throughput using TREX's API. >> >> >> >> Thanks, >> >> Brandon >> >> >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde >> >> wrote: >> >> > >> >> > Hi Brandon, >> >> > I was trying to see what exactly is happening on the setup and what is causing the problem. >> >> > But I will need your help to use the proper code and commands. >> >> > >> >> > I believe you run the testpmd command on io. And the trex is executed on rhea? >> >> > Can you point me to the location of the code and the steps you are following while running the test? >> >> > >> >> > Thanks >> >> > Ajit >> >> > >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde wrote: >> >> >> >> >> >> Hi Brandon, >> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office. >> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN. >> >> >> Thanks for checking in. I will try to update you as soon as I can. >> >> >> >> >> >> Thanks >> >> >> Ajit >> >> >> >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo wrote: >> >> >>> >> >> >>> Hi Ajit, >> >> >>> >> >> >>> I'm just checking in; have you heard of any updates on this issue? >> >> >>> >> >> >>> Thanks, >> >> >>> Brandon >> >> >>> >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo wrote: >> >> >>>> >> >> >>>> Hi Ajit, >> >> >>>> >> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC. >> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing. >> >> >>>> >> >> >>>> If you want to replicate it, here are the steps that I ran: >> >> >>>> >> >> >>>> (on io) cd /opt/dts >> >> >>>> export DTS_CFG_FOLDER='conf_100g' >> >> >>>> >> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file >> >> >>>> >> >> >>>> ./dts -s >> >> >>>> >> >> >>>> >> >> >>>> Thanks for your help, >> >> >>>> Brandon >> >> >>>> >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde wrote: >> >> >>>>> >> >> >>>>> Hi Brandon, >> >> >>>>> No, I haven't seen or heard this before. >> >> >>>>> But I will try to have someone run it again. >> >> >>>>> >> >> >>>>> Just to make sure - >> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup? >> >> >>>>> Or is it the system that is running testpmd where you are running into the issue? >> >> >>>>> >> >> >>>>> Thanks >> >> >>>>> Ajit >> >> >>>>> >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo wrote: >> >> >>>>>> >> >> >>>>>> Hi Ajit, >> >> >>>>>> >> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC. >> >> >>>>>> The current configuration uses trex version v2.82. >> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf. >> >> >>>>>> >> >> >>>>>> The output looks like this when it reaches a test case: >> >> >>>>>> >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512 >> >> >>>>>>> dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1 --file-prefix=dpdk_11307_20200804160513 -- -i --portmask=0x3 --txd=512 --rxd=512 >> >> >>>>>>> dut.rhea: start >> >> >>>>>>> TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last): >> >> >>>>>>> File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case >> >> >>>>>>> case_obj() >> >> >>>>>>> File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core >> >> >>>>>>> self.perf_test(self.nb_ports) >> >> >>>>>>> File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test >> >> >>>>>>> _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt) >> >> >>>>>>> File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput >> >> >>>>>>> self._prepare_transmission(stream_ids=stream_ids) >> >> >>>>>>> File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission >> >> >>>>>>> self._conn.reset(ports=self._ports) >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2 >> >> >>>>>>> ret = f(*args, **kwargs) >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset >> >> >>>>>>> self.clear_stats(ports) >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2 >> >> >>>>>>> ret = f(*args, **kwargs) >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats >> >> >>>>>>> self._clear_stats_common(ports, clear_global, clear_xstats) >> >> >>>>>>> File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common >> >> >>>>>>> raise TRexError(rc) >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501 >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147. >> >> >>>>>> Have you ran into this issue before? >> >> >>>>>> >> >> >>>>>> Thanks, >> >> >>>>>> Brandon >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> -- >> >> >>>> >> >> >>>> Brandon Lo >> >> >>>> >> >> >>>> UNH InterOperability Laboratory >> >> >>>> >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> >>>> >> >> >>>> blo@iol.unh.edu >> >> >>>> >> >> >>>> www.iol.unh.edu >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> >> >> >>> Brandon Lo >> >> >>> >> >> >>> UNH InterOperability Laboratory >> >> >>> >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> >>> >> >> >>> blo@iol.unh.edu >> >> >>> >> >> >>> www.iol.unh.edu >> >> >> >> >> >> >> >> -- >> >> >> >> Brandon Lo >> >> >> >> UNH InterOperability Laboratory >> >> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> >> >> blo@iol.unh.edu >> >> >> >> www.iol.unh.edu >> >> >> >> -- >> >> Brandon Lo >> >> UNH InterOperability Laboratory >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824 >> >> blo@iol.unh.edu >> >> www.iol.unh.edu -- Brandon Lo UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 blo@iol.unh.edu www.iol.unh.edu