DPDK CI discussions
 help / color / mirror / Atom feed
* [dpdk-ci] New 100G Broadcom NIC Ubuntu
@ 2020-07-21 20:22 Brandon Lo
  2020-07-21 21:27 ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-07-21 20:22 UTC (permalink / raw)
  To: Ajit Khaparde, dpdklab, ci

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

Hi Ajit,

We've received the Broadcom 100G NIC and installed it onto the machine.
However, the drivers don't seem to support Ubuntu, information from
https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g
.

I've also updated the kernel to the latest version. The card appears with
the correct name in lspci and lshw, but it has no available MAC address, so
I can't use it in setting up the nic single core performance testing.

Do you know what could be causing this issue?

Thanks,
Brandon

-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 2303 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-07-21 20:22 [dpdk-ci] New 100G Broadcom NIC Ubuntu Brandon Lo
@ 2020-07-21 21:27 ` Ajit Khaparde
  2020-07-28 16:47   ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-07-21 21:27 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

Hi Brandon,
Maybe I might have to update it to a newer firmware.
Let me check internally and get back.

Thanks
Ajit


On Tue, Jul 21, 2020 at 1:23 PM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> We've received the Broadcom 100G NIC and installed it onto the machine.
> However, the drivers don't seem to support Ubuntu, information from
> https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g
> .
>
> I've also updated the kernel to the latest version. The card appears with
> the correct name in lspci and lshw, but it has no available MAC address, so
> I can't use it in setting up the nic single core performance testing.
>
> Do you know what could be causing this issue?
>
> Thanks,
> Brandon
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
> <https://www.google.com/maps/search/21+Madbury+Rd,+Suite+100,+Durham,+NH+03824?entry=gmail&source=g>
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>

[-- Attachment #2: Type: text/html, Size: 3463 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-07-21 21:27 ` Ajit Khaparde
@ 2020-07-28 16:47   ` Brandon Lo
  2020-07-28 17:22     ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-07-28 16:47 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 1420 bytes --]

Hi Ajit,

Have you gotten a response yet?
I've tried the firmware from the official website but it's supported on
Fedora/Redhat instead of Ubuntu.

Thanks,
Brandon

On Tue, Jul 21, 2020 at 5:28 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi Brandon,
> Maybe I might have to update it to a newer firmware.
> Let me check internally and get back.
>
> Thanks
> Ajit
>
>
> On Tue, Jul 21, 2020 at 1:23 PM Brandon Lo <blo@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> We've received the Broadcom 100G NIC and installed it onto the machine.
>> However, the drivers don't seem to support Ubuntu, information from
>> https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g
>> .
>>
>> I've also updated the kernel to the latest version. The card appears with
>> the correct name in lspci and lshw, but it has no available MAC address, so
>> I can't use it in setting up the nic single core performance testing.
>>
>> Do you know what could be causing this issue?
>>
>> Thanks,
>> Brandon
>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> <https://www.google.com/maps/search/21+Madbury+Rd,+Suite+100,+Durham,+NH+03824?entry=gmail&source=g>
>>
>> blo@iol.unh.edu
>>
>> www.iol.unh.edu
>>
>

-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 5545 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-07-28 16:47   ` Brandon Lo
@ 2020-07-28 17:22     ` Ajit Khaparde
  2020-07-28 17:32       ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-07-28 17:22 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

Brandon,
I have sent them a reminder. I will keep you posted.
But I tried to login into the system to peek around and the VPN seems to
continuously negotiate/authenticate. For some reason the connection
is getting reset.
Any idea on what might be causing this?

Thanks
Ajit

On Tue, Jul 28, 2020 at 9:48 AM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> Have you gotten a response yet?
> I've tried the firmware from the official website but it's supported on
> Fedora/Redhat instead of Ubuntu.
>
> Thanks,
> Brandon
>
> On Tue, Jul 21, 2020 at 5:28 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Hi Brandon,
>> Maybe I might have to update it to a newer firmware.
>> Let me check internally and get back.
>>
>> Thanks
>> Ajit
>>
>>
>> On Tue, Jul 21, 2020 at 1:23 PM Brandon Lo <blo@iol.unh.edu> wrote:
>>
>>> Hi Ajit,
>>>
>>> We've received the Broadcom 100G NIC and installed it onto the machine.
>>> However, the drivers don't seem to support Ubuntu, information from
>>> https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g
>>> .
>>>
>>> I've also updated the kernel to the latest version. The card appears
>>> with the correct name in lspci and lshw, but it has no available MAC
>>> address, so I can't use it in setting up the nic single core performance
>>> testing.
>>>
>>> Do you know what could be causing this issue?
>>>
>>> Thanks,
>>> Brandon
>>>
>>>
>>> --
>>>
>>> Brandon Lo
>>>
>>> UNH InterOperability Laboratory
>>>
>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>> <https://www.google.com/maps/search/21+Madbury+Rd,+Suite+100,+Durham,+NH+03824?entry=gmail&source=g>
>>>
>>> blo@iol.unh.edu
>>>
>>> www.iol.unh.edu
>>>
>>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>

[-- Attachment #2: Type: text/html, Size: 6797 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-07-28 17:22     ` Ajit Khaparde
@ 2020-07-28 17:32       ` Brandon Lo
  2020-07-28 19:19         ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-07-28 17:32 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 2341 bytes --]

Hi Ajit,

That issue occurs sometimes on my end as well. There's no clear solution
other than to try again later.
I will look into the issue and try to make a permanent fix.

Thanks,
Brandon

On Tue, Jul 28, 2020 at 1:22 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Brandon,
> I have sent them a reminder. I will keep you posted.
> But I tried to login into the system to peek around and the VPN seems to
> continuously negotiate/authenticate. For some reason the connection
> is getting reset.
> Any idea on what might be causing this?
>
> Thanks
> Ajit
>
> On Tue, Jul 28, 2020 at 9:48 AM Brandon Lo <blo@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> Have you gotten a response yet?
>> I've tried the firmware from the official website but it's supported on
>> Fedora/Redhat instead of Ubuntu.
>>
>> Thanks,
>> Brandon
>>
>> On Tue, Jul 21, 2020 at 5:28 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>>> Hi Brandon,
>>> Maybe I might have to update it to a newer firmware.
>>> Let me check internally and get back.
>>>
>>> Thanks
>>> Ajit
>>>
>>>
>>> On Tue, Jul 21, 2020 at 1:23 PM Brandon Lo <blo@iol.unh.edu> wrote:
>>>
>>>> Hi Ajit,
>>>>
>>>> We've received the Broadcom 100G NIC and installed it onto the machine.
>>>> However, the drivers don't seem to support Ubuntu, information from
>>>> https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g
>>>> .
>>>>
>>>> I've also updated the kernel to the latest version. The card appears
>>>> with the correct name in lspci and lshw, but it has no available MAC
>>>> address, so I can't use it in setting up the nic single core performance
>>>> testing.
>>>>
>>>> Do you know what could be causing this issue?
>>>>
>>>> Thanks,
>>>> Brandon
>>>>
>>>>
>>>> --
>>>>
>>>> Brandon Lo
>>>>
>>>> UNH InterOperability Laboratory
>>>>
>>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>>> <https://www.google.com/maps/search/21+Madbury+Rd,+Suite+100,+Durham,+NH+03824?entry=gmail&source=g>
>>>>
>>>> blo@iol.unh.edu
>>>>
>>>> www.iol.unh.edu
>>>>
>>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>
>> blo@iol.unh.edu
>>
>> www.iol.unh.edu
>>
>

-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 8918 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-07-28 17:32       ` Brandon Lo
@ 2020-07-28 19:19         ` Ajit Khaparde
  2020-07-28 19:41           ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-07-28 19:19 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 2935 bytes --]

Hi Brandon,
I copied over a more recent code for the NIC.
And it loads fine on the two systems.
The code is currently in /home/ajitkhaparde/bnxt_en-1.10.1-216.1.142.0
Till I find a release version of the driver for Ubuntu, we could use this?
We can copy the bnxt_en.ko over to
/lib/modules/4.15.0-112-generic/kernel/drivers/net/ethernet/broa
dcom/bnxt/bnxt_en.ko

Please let me know.

Thanks
Ajit

On Tue, Jul 28, 2020 at 10:33 AM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> That issue occurs sometimes on my end as well. There's no clear solution
> other than to try again later.
> I will look into the issue and try to make a permanent fix.
>
> Thanks,
> Brandon
>
> On Tue, Jul 28, 2020 at 1:22 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Brandon,
>> I have sent them a reminder. I will keep you posted.
>> But I tried to login into the system to peek around and the VPN seems to
>> continuously negotiate/authenticate. For some reason the connection
>> is getting reset.
>> Any idea on what might be causing this?
>>
>> Thanks
>> Ajit
>>
>> On Tue, Jul 28, 2020 at 9:48 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>
>>> Hi Ajit,
>>>
>>> Have you gotten a response yet?
>>> I've tried the firmware from the official website but it's supported on
>>> Fedora/Redhat instead of Ubuntu.
>>>
>>> Thanks,
>>> Brandon
>>>
>>> On Tue, Jul 21, 2020 at 5:28 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>>
>>>> Hi Brandon,
>>>> Maybe I might have to update it to a newer firmware.
>>>> Let me check internally and get back.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>>
>>>> On Tue, Jul 21, 2020 at 1:23 PM Brandon Lo <blo@iol.unh.edu> wrote:
>>>>
>>>>> Hi Ajit,
>>>>>
>>>>> We've received the Broadcom 100G NIC and installed it onto the
>>>>> machine. However, the drivers don't seem to support Ubuntu, information
>>>>> from
>>>>> https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g
>>>>> .
>>>>>
>>>>> I've also updated the kernel to the latest version. The card appears
>>>>> with the correct name in lspci and lshw, but it has no available MAC
>>>>> address, so I can't use it in setting up the nic single core performance
>>>>> testing.
>>>>>
>>>>> Do you know what could be causing this issue?
>>>>>
>>>>> Thanks,
>>>>> Brandon
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Brandon Lo
>>>>>
>>>>> UNH InterOperability Laboratory
>>>>>
>>>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>>>> <https://www.google.com/maps/search/21+Madbury+Rd,+Suite+100,+Durham,+NH+03824?entry=gmail&source=g>
>>>>>
>>>>> blo@iol.unh.edu
>>>>>
>>>>> www.iol.unh.edu
>>>>>
>>>>
>>>
>>> --
>>>
>>> Brandon Lo
>>>
>>> UNH InterOperability Laboratory
>>>
>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>>
>>> blo@iol.unh.edu
>>>
>>> www.iol.unh.edu
>>>
>>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>

[-- Attachment #2: Type: text/html, Size: 10360 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-07-28 19:19         ` Ajit Khaparde
@ 2020-07-28 19:41           ` Brandon Lo
  2020-08-04 16:16             ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-07-28 19:41 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 3443 bytes --]

Hi Ajit,

Yes, I believe we can use this current version as the driver.
I'll let you know if I run into any issues with this.

Thank you for your help,
Brandon

On Tue, Jul 28, 2020 at 3:19 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi Brandon,
> I copied over a more recent code for the NIC.
> And it loads fine on the two systems.
> The code is currently in /home/ajitkhaparde/bnxt_en-1.10.1-216.1.142.0
> Till I find a release version of the driver for Ubuntu, we could use this?
> We can copy the bnxt_en.ko over to
> /lib/modules/4.15.0-112-generic/kernel/drivers/net/ethernet/broa
> dcom/bnxt/bnxt_en.ko
>
> Please let me know.
>
> Thanks
> Ajit
>
> On Tue, Jul 28, 2020 at 10:33 AM Brandon Lo <blo@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> That issue occurs sometimes on my end as well. There's no clear solution
>> other than to try again later.
>> I will look into the issue and try to make a permanent fix.
>>
>> Thanks,
>> Brandon
>>
>> On Tue, Jul 28, 2020 at 1:22 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>>> Brandon,
>>> I have sent them a reminder. I will keep you posted.
>>> But I tried to login into the system to peek around and the VPN seems to
>>> continuously negotiate/authenticate. For some reason the connection
>>> is getting reset.
>>> Any idea on what might be causing this?
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Tue, Jul 28, 2020 at 9:48 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>>
>>>> Hi Ajit,
>>>>
>>>> Have you gotten a response yet?
>>>> I've tried the firmware from the official website but it's supported on
>>>> Fedora/Redhat instead of Ubuntu.
>>>>
>>>> Thanks,
>>>> Brandon
>>>>
>>>> On Tue, Jul 21, 2020 at 5:28 PM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>>
>>>>> Hi Brandon,
>>>>> Maybe I might have to update it to a newer firmware.
>>>>> Let me check internally and get back.
>>>>>
>>>>> Thanks
>>>>> Ajit
>>>>>
>>>>>
>>>>> On Tue, Jul 21, 2020 at 1:23 PM Brandon Lo <blo@iol.unh.edu> wrote:
>>>>>
>>>>>> Hi Ajit,
>>>>>>
>>>>>> We've received the Broadcom 100G NIC and installed it onto the
>>>>>> machine. However, the drivers don't seem to support Ubuntu, information
>>>>>> from
>>>>>> https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp/p2100g
>>>>>> .
>>>>>>
>>>>>> I've also updated the kernel to the latest version. The card appears
>>>>>> with the correct name in lspci and lshw, but it has no available MAC
>>>>>> address, so I can't use it in setting up the nic single core performance
>>>>>> testing.
>>>>>>
>>>>>> Do you know what could be causing this issue?
>>>>>>
>>>>>> Thanks,
>>>>>> Brandon
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Brandon Lo
>>>>>>
>>>>>> UNH InterOperability Laboratory
>>>>>>
>>>>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>>>>> <https://www.google.com/maps/search/21+Madbury+Rd,+Suite+100,+Durham,+NH+03824?entry=gmail&source=g>
>>>>>>
>>>>>> blo@iol.unh.edu
>>>>>>
>>>>>> www.iol.unh.edu
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Brandon Lo
>>>>
>>>> UNH InterOperability Laboratory
>>>>
>>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>>>
>>>> blo@iol.unh.edu
>>>>
>>>> www.iol.unh.edu
>>>>
>>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>
>> blo@iol.unh.edu
>>
>> www.iol.unh.edu
>>
>

-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 12396 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-07-28 19:41           ` Brandon Lo
@ 2020-08-04 16:16             ` Brandon Lo
  2020-08-04 16:37               ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-08-04 16:16 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 2568 bytes --]

Hi Ajit,

I'm running into a problem with trying to run nic_single_core_perf on the
new NIC.
The current configuration uses trex version v2.82.
However, I'm running into an error when it tries to actually do a test case
in the nic_single_core_perf.

The output looks like this when it reaches a test case:

TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd:
> 512
>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd
> -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1
>  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512
> --rxd=512
>                       dut.rhea: start
>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result
> ERROR: Traceback (most recent call last):
>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
>     case_obj()
>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in
> test_perf_nic_single_core
>     self.perf_test(self.nb_ports)
>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>     self._prepare_transmission(stream_ids=stream_ids)
>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
> _prepare_transmission
>     self._conn.reset(ports=self._ports)
>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
>     ret = f(*args, **kwargs)
>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
>     self.clear_stats(ports)
>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
>     ret = f(*args, **kwargs)
>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1467, in clear_stats
>     self._clear_stats_common(ports, clear_global, clear_xstats)
>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2840, in _clear_stats_common
>     raise TRexError(rc)
> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
> response from tcp://127.0.0.1:4501


I have found one similar case on the github repository for trex, but the
solution was vendor-specific:
https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
Have you ran into this issue before?

Thanks,
Brandon

[-- Attachment #2: Type: text/html, Size: 3598 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-04 16:16             ` Brandon Lo
@ 2020-08-04 16:37               ` Ajit Khaparde
  2020-08-04 17:42                 ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-08-04 16:37 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 3017 bytes --]

Hi Brandon,
No, I haven't seen or heard this before.
But I will try to have someone run it again.

Just to make sure -
You are running trex on the 100G NIC and the problem is encountered on that
setup?
Or is it the system that is running testpmd where you are running into the
issue?

Thanks
Ajit

On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> I'm running into a problem with trying to run nic_single_core_perf on the
> new NIC.
> The current configuration uses trex version v2.82.
> However, I'm running into an error when it tries to actually do a test
> case in the nic_single_core_perf.
>
> The output looks like this when it reaches a test case:
>
> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd:
>> 512
>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd
>> -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1
>>  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512
>> --rxd=512
>>                       dut.rhea: start
>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>> Result ERROR: Traceback (most recent call last):
>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
>>     case_obj()
>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in
>> test_perf_nic_single_core
>>     self.perf_test(self.nb_ports)
>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>>     _, packets_received =
>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>> options=traffic_opt)
>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>> measure_throughput
>>     self._prepare_transmission(stream_ids=stream_ids)
>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
>> _prepare_transmission
>>     self._conn.reset(ports=self._ports)
>>   File
>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>>     ret = f(*args, **kwargs)
>>   File
>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 339, in reset
>>     self.clear_stats(ports)
>>   File
>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>>     ret = f(*args, **kwargs)
>>   File
>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 1467, in clear_stats
>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>   File
>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>> line 2840, in _clear_stats_common
>>     raise TRexError(rc)
>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
>> response from tcp://127.0.0.1:4501
>
>
> I have found one similar case on the github repository for trex, but the
> solution was vendor-specific:
> https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
> Have you ran into this issue before?
>
> Thanks,
> Brandon
>

[-- Attachment #2: Type: text/html, Size: 4990 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-04 16:37               ` Ajit Khaparde
@ 2020-08-04 17:42                 ` Brandon Lo
  2020-08-11 17:45                   ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-08-04 17:42 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 3901 bytes --]

Hi Ajit,

Yes, I believe the issue is coming from the trex/tester system with the
100G NIC.
I'm not sure what causes this issue; if I run trex using the command "cd
/opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the
same command used in DTS, it seems to launch without failing.

If you want to replicate it, here are the steps that I ran:

   1. (on io) cd /opt/dts
   2. export DTS_CFG_FOLDER='conf_100g'
      - conf_100g has the new configuration files to use the new PCI id and
      pktgen config file
   3. ./dts -s


Thanks for your help,
Brandon

On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi Brandon,
> No, I haven't seen or heard this before.
> But I will try to have someone run it again.
>
> Just to make sure -
> You are running trex on the 100G NIC and the problem is encountered on
> that setup?
> Or is it the system that is running testpmd where you are running into the
> issue?
>
> Thanks
> Ajit
>
> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> I'm running into a problem with trying to run nic_single_core_perf on the
>> new NIC.
>> The current configuration uses trex version v2.82.
>> However, I'm running into an error when it tries to actually do a test
>> case in the nic_single_core_perf.
>>
>> The output looks like this when it reaches a test case:
>>
>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>>> rxd/txd: 512
>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd
>>> -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1
>>>  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512
>>> --rxd=512
>>>                       dut.rhea: start
>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>>> Result ERROR: Traceback (most recent call last):
>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
>>>     case_obj()
>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in
>>> test_perf_nic_single_core
>>>     self.perf_test(self.nb_ports)
>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>>>     _, packets_received =
>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>> options=traffic_opt)
>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>> measure_throughput
>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
>>> _prepare_transmission
>>>     self._conn.reset(ports=self._ports)
>>>   File
>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>>     ret = f(*args, **kwargs)
>>>   File
>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 339, in reset
>>>     self.clear_stats(ports)
>>>   File
>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>>     ret = f(*args, **kwargs)
>>>   File
>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 1467, in clear_stats
>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>   File
>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>> line 2840, in _clear_stats_common
>>>     raise TRexError(rc)
>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
>>> response from tcp://127.0.0.1:4501
>>
>>
>> I have found one similar case on the github repository for trex, but the
>> solution was vendor-specific:
>> https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
>> Have you ran into this issue before?
>>
>> Thanks,
>> Brandon
>>
>

-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 7559 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-04 17:42                 ` Brandon Lo
@ 2020-08-11 17:45                   ` Brandon Lo
  2020-08-11 17:47                     ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-08-11 17:45 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 4338 bytes --]

Hi Ajit,

I'm just checking in; have you heard of any updates on this issue?

Thanks,
Brandon

On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> Yes, I believe the issue is coming from the trex/tester system with the
> 100G NIC.
> I'm not sure what causes this issue; if I run trex using the command "cd
> /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the
> same command used in DTS, it seems to launch without failing.
>
> If you want to replicate it, here are the steps that I ran:
>
>    1. (on io) cd /opt/dts
>    2. export DTS_CFG_FOLDER='conf_100g'
>       - conf_100g has the new configuration files to use the new PCI id
>       and pktgen config file
>    3. ./dts -s
>
>
> Thanks for your help,
> Brandon
>
> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Hi Brandon,
>> No, I haven't seen or heard this before.
>> But I will try to have someone run it again.
>>
>> Just to make sure -
>> You are running trex on the 100G NIC and the problem is encountered on
>> that setup?
>> Or is it the system that is running testpmd where you are running into
>> the issue?
>>
>> Thanks
>> Ajit
>>
>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>
>>> Hi Ajit,
>>>
>>> I'm running into a problem with trying to run nic_single_core_perf on
>>> the new NIC.
>>> The current configuration uses trex version v2.82.
>>> However, I'm running into an error when it tries to actually do a test
>>> case in the nic_single_core_perf.
>>>
>>> The output looks like this when it reaches a test case:
>>>
>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>>>> rxd/txd: 512
>>>>                       dut.rhea:
>>>> ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w
>>>> 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i
>>>>  --portmask=0x3 --txd=512 --rxd=512
>>>>                       dut.rhea: start
>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>>>> Result ERROR: Traceback (most recent call last):
>>>>   File "/opt/dts/framework/test_case.py", line 316, in
>>>> _execute_test_case
>>>>     case_obj()
>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in
>>>> test_perf_nic_single_core
>>>>     self.perf_test(self.nb_ports)
>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>>>>     _, packets_received =
>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>> options=traffic_opt)
>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>>> measure_throughput
>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
>>>> _prepare_transmission
>>>>     self._conn.reset(ports=self._ports)
>>>>   File
>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>> line 51, in wrap2
>>>>     ret = f(*args, **kwargs)
>>>>   File
>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>> line 339, in reset
>>>>     self.clear_stats(ports)
>>>>   File
>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>> line 51, in wrap2
>>>>     ret = f(*args, **kwargs)
>>>>   File
>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>> line 1467, in clear_stats
>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>   File
>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>> line 2840, in _clear_stats_common
>>>>     raise TRexError(rc)
>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
>>>> response from tcp://127.0.0.1:4501
>>>
>>>
>>> I have found one similar case on the github repository for trex, but the
>>> solution was vendor-specific:
>>> https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
>>> Have you ran into this issue before?
>>>
>>> Thanks,
>>> Brandon
>>>
>>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>


-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

[-- Attachment #2: Type: text/html, Size: 9531 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-11 17:45                   ` Brandon Lo
@ 2020-08-11 17:47                     ` Ajit Khaparde
  2020-08-20  4:29                       ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-08-11 17:47 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 4843 bytes --]

Hi Brandon,
I haven't. I tried to login as well. But I had some issues doing it from
the office.
I just have to remind myself to try it again once I get home before I
connect to the company VPN.
Thanks for checking in. I will try to update you as soon as I can.

Thanks
Ajit

On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> I'm just checking in; have you heard of any updates on this issue?
>
> Thanks,
> Brandon
>
> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> Yes, I believe the issue is coming from the trex/tester system with the
>> 100G NIC.
>> I'm not sure what causes this issue; if I run trex using the command "cd
>> /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the
>> same command used in DTS, it seems to launch without failing.
>>
>> If you want to replicate it, here are the steps that I ran:
>>
>>    1. (on io) cd /opt/dts
>>    2. export DTS_CFG_FOLDER='conf_100g'
>>       - conf_100g has the new configuration files to use the new PCI id
>>       and pktgen config file
>>    3. ./dts -s
>>
>>
>> Thanks for your help,
>> Brandon
>>
>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>>> Hi Brandon,
>>> No, I haven't seen or heard this before.
>>> But I will try to have someone run it again.
>>>
>>> Just to make sure -
>>> You are running trex on the 100G NIC and the problem is encountered on
>>> that setup?
>>> Or is it the system that is running testpmd where you are running into
>>> the issue?
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>>
>>>> Hi Ajit,
>>>>
>>>> I'm running into a problem with trying to run nic_single_core_perf on
>>>> the new NIC.
>>>> The current configuration uses trex version v2.82.
>>>> However, I'm running into an error when it tries to actually do a test
>>>> case in the nic_single_core_perf.
>>>>
>>>> The output looks like this when it reaches a test case:
>>>>
>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>>>>> rxd/txd: 512
>>>>>                       dut.rhea:
>>>>> ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w
>>>>> 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i
>>>>>  --portmask=0x3 --txd=512 --rxd=512
>>>>>                       dut.rhea: start
>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>>>>> Result ERROR: Traceback (most recent call last):
>>>>>   File "/opt/dts/framework/test_case.py", line 316, in
>>>>> _execute_test_case
>>>>>     case_obj()
>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in
>>>>> test_perf_nic_single_core
>>>>>     self.perf_test(self.nb_ports)
>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in
>>>>> perf_test
>>>>>     _, packets_received =
>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>> options=traffic_opt)
>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>>>> measure_throughput
>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
>>>>> _prepare_transmission
>>>>>     self._conn.reset(ports=self._ports)
>>>>>   File
>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>> line 51, in wrap2
>>>>>     ret = f(*args, **kwargs)
>>>>>   File
>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>> line 339, in reset
>>>>>     self.clear_stats(ports)
>>>>>   File
>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>> line 51, in wrap2
>>>>>     ret = f(*args, **kwargs)
>>>>>   File
>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>> line 1467, in clear_stats
>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>>   File
>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>> line 2840, in _clear_stats_common
>>>>>     raise TRexError(rc)
>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get
>>>>> server response from tcp://127.0.0.1:4501
>>>>
>>>>
>>>> I have found one similar case on the github repository for trex, but
>>>> the solution was vendor-specific:
>>>> https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
>>>> Have you ran into this issue before?
>>>>
>>>> Thanks,
>>>> Brandon
>>>>
>>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>
>> blo@iol.unh.edu
>>
>> www.iol.unh.edu
>>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>

[-- Attachment #2: Type: text/html, Size: 10644 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-11 17:47                     ` Ajit Khaparde
@ 2020-08-20  4:29                       ` Ajit Khaparde
  2020-08-20 15:39                         ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-08-20  4:29 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 5463 bytes --]

Hi Brandon,
I was trying to see what exactly is happening on the setup and what is
causing the problem.
But I will need your help to use the proper code and commands.

I believe you run the testpmd command on io. And the trex is executed on
rhea?
Can you point me to the location of the code and the steps you are
following while running the test?

Thanks
Ajit

On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi Brandon,
> I haven't. I tried to login as well. But I had some issues doing it from
> the office.
> I just have to remind myself to try it again once I get home before I
> connect to the company VPN.
> Thanks for checking in. I will try to update you as soon as I can.
>
> Thanks
> Ajit
>
> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> I'm just checking in; have you heard of any updates on this issue?
>>
>> Thanks,
>> Brandon
>>
>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
>>
>>> Hi Ajit,
>>>
>>> Yes, I believe the issue is coming from the trex/tester system with the
>>> 100G NIC.
>>> I'm not sure what causes this issue; if I run trex using the command "cd
>>> /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the
>>> same command used in DTS, it seems to launch without failing.
>>>
>>> If you want to replicate it, here are the steps that I ran:
>>>
>>>    1. (on io) cd /opt/dts
>>>    2. export DTS_CFG_FOLDER='conf_100g'
>>>       - conf_100g has the new configuration files to use the new PCI id
>>>       and pktgen config file
>>>    3. ./dts -s
>>>
>>>
>>> Thanks for your help,
>>> Brandon
>>>
>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>>
>>>> Hi Brandon,
>>>> No, I haven't seen or heard this before.
>>>> But I will try to have someone run it again.
>>>>
>>>> Just to make sure -
>>>> You are running trex on the 100G NIC and the problem is encountered on
>>>> that setup?
>>>> Or is it the system that is running testpmd where you are running into
>>>> the issue?
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>>>
>>>>> Hi Ajit,
>>>>>
>>>>> I'm running into a problem with trying to run nic_single_core_perf on
>>>>> the new NIC.
>>>>> The current configuration uses trex version v2.82.
>>>>> However, I'm running into an error when it tries to actually do a test
>>>>> case in the nic_single_core_perf.
>>>>>
>>>>> The output looks like this when it reaches a test case:
>>>>>
>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>>>>>> rxd/txd: 512
>>>>>>                       dut.rhea:
>>>>>> ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w
>>>>>> 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i
>>>>>>  --portmask=0x3 --txd=512 --rxd=512
>>>>>>                       dut.rhea: start
>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>>>>>> Result ERROR: Traceback (most recent call last):
>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in
>>>>>> _execute_test_case
>>>>>>     case_obj()
>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in
>>>>>> test_perf_nic_single_core
>>>>>>     self.perf_test(self.nb_ports)
>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in
>>>>>> perf_test
>>>>>>     _, packets_received =
>>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>>> options=traffic_opt)
>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>>>>> measure_throughput
>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
>>>>>> _prepare_transmission
>>>>>>     self._conn.reset(ports=self._ports)
>>>>>>   File
>>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>> line 51, in wrap2
>>>>>>     ret = f(*args, **kwargs)
>>>>>>   File
>>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>> line 339, in reset
>>>>>>     self.clear_stats(ports)
>>>>>>   File
>>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>> line 51, in wrap2
>>>>>>     ret = f(*args, **kwargs)
>>>>>>   File
>>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>> line 1467, in clear_stats
>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>>>   File
>>>>>> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>>> line 2840, in _clear_stats_common
>>>>>>     raise TRexError(rc)
>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get
>>>>>> server response from tcp://127.0.0.1:4501
>>>>>
>>>>>
>>>>> I have found one similar case on the github repository for trex, but
>>>>> the solution was vendor-specific:
>>>>> https://github.com/cisco-system-traffic-generator/trex-core/issues/147
>>>>> .
>>>>> Have you ran into this issue before?
>>>>>
>>>>> Thanks,
>>>>> Brandon
>>>>>
>>>>
>>>
>>> --
>>>
>>> Brandon Lo
>>>
>>> UNH InterOperability Laboratory
>>>
>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>>
>>> blo@iol.unh.edu
>>>
>>> www.iol.unh.edu
>>>
>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>
>> blo@iol.unh.edu
>>
>> www.iol.unh.edu
>>
>

[-- Attachment #2: Type: text/html, Size: 12032 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-20  4:29                       ` Ajit Khaparde
@ 2020-08-20 15:39                         ` Brandon Lo
  2020-08-20 21:07                           ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-08-20 15:39 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

Hi Ajit,

I believe DTS/TREX is run on the io machine as a tester. Rhea runs
testpmd (through an ssh session from io --> rhea) to catch information
as a DUT.

The only commands that I run (on io) are:
cd /opt/dts
export DTS_CFG_FOLDER='conf_100g'
./dts -s

The rest is managed by DTS itself.
The issue occurs when DTS is trying to send packets to rhea from io on
the new 100G NIC.
It seems to happen after it tries to measure throughput using TREX's API.

Thanks,
Brandon

On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
<ajit.khaparde@broadcom.com> wrote:
>
> Hi Brandon,
> I was trying to see what exactly is happening on the setup and what is causing the problem.
> But I will need your help to use the proper code and commands.
>
> I believe you run the testpmd command on io. And the trex is executed on rhea?
> Can you point me to the location of the code and the steps you are following while running the test?
>
> Thanks
> Ajit
>
> On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>
>> Hi Brandon,
>> I haven't. I tried to login as well. But I had some issues doing it from the office.
>> I just have to remind myself to try it again once I get home before I connect to the company VPN.
>> Thanks for checking in. I will try to update you as soon as I can.
>>
>> Thanks
>> Ajit
>>
>> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>>
>>> Hi Ajit,
>>>
>>> I'm just checking in; have you heard of any updates on this issue?
>>>
>>> Thanks,
>>> Brandon
>>>
>>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
>>>>
>>>> Hi Ajit,
>>>>
>>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.
>>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.
>>>>
>>>> If you want to replicate it, here are the steps that I ran:
>>>>
>>>> (on io) cd /opt/dts
>>>> export DTS_CFG_FOLDER='conf_100g'
>>>>
>>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file
>>>>
>>>> ./dts -s
>>>>
>>>>
>>>> Thanks for your help,
>>>> Brandon
>>>>
>>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>>
>>>>> Hi Brandon,
>>>>> No, I haven't seen or heard this before.
>>>>> But I will try to have someone run it again.
>>>>>
>>>>> Just to make sure -
>>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?
>>>>> Or is it the system that is running testpmd where you are running into the issue?
>>>>>
>>>>> Thanks
>>>>> Ajit
>>>>>
>>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>>>>>
>>>>>> Hi Ajit,
>>>>>>
>>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.
>>>>>> The current configuration uses trex version v2.82.
>>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.
>>>>>>
>>>>>> The output looks like this when it reaches a test case:
>>>>>>
>>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>>>>>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512 --rxd=512
>>>>>>>                       dut.rhea: start
>>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
>>>>>>>     case_obj()
>>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core
>>>>>>>     self.perf_test(self.nb_ports)
>>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission
>>>>>>>     self._conn.reset(ports=self._ports)
>>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>>>>>     ret = f(*args, **kwargs)
>>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>>>>>>>     self.clear_stats(ports)
>>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>>>>>     ret = f(*args, **kwargs)
>>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats
>>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common
>>>>>>>     raise TRexError(rc)
>>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501
>>>>>>
>>>>>>
>>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
>>>>>> Have you ran into this issue before?
>>>>>>
>>>>>> Thanks,
>>>>>> Brandon
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Brandon Lo
>>>>
>>>> UNH InterOperability Laboratory
>>>>
>>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>>>
>>>> blo@iol.unh.edu
>>>>
>>>> www.iol.unh.edu
>>>
>>>
>>>
>>> --
>>>
>>> Brandon Lo
>>>
>>> UNH InterOperability Laboratory
>>>
>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>>
>>> blo@iol.unh.edu
>>>
>>> www.iol.unh.edu



-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-20 15:39                         ` Brandon Lo
@ 2020-08-20 21:07                           ` Ajit Khaparde
  2020-08-25 20:13                             ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-08-20 21:07 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 6679 bytes --]

Hi Brandon,
I do see some issues while running trex on this setup.
I will have to dig further. I will let you know once I find something or
may be a fix.

Thanks
Ajit

On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
> testpmd (through an ssh session from io --> rhea) to catch information
> as a DUT.
>
> The only commands that I run (on io) are:
> cd /opt/dts
> export DTS_CFG_FOLDER='conf_100g'
> ./dts -s
>
> The rest is managed by DTS itself.
> The issue occurs when DTS is trying to send packets to rhea from io on
> the new 100G NIC.
> It seems to happen after it tries to measure throughput using TREX's API.
>
> Thanks,
> Brandon
>
> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
> <ajit.khaparde@broadcom.com> wrote:
> >
> > Hi Brandon,
> > I was trying to see what exactly is happening on the setup and what is
> causing the problem.
> > But I will need your help to use the proper code and commands.
> >
> > I believe you run the testpmd command on io. And the trex is executed on
> rhea?
> > Can you point me to the location of the code and the steps you are
> following while running the test?
> >
> > Thanks
> > Ajit
> >
> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>
> >> Hi Brandon,
> >> I haven't. I tried to login as well. But I had some issues doing it
> from the office.
> >> I just have to remind myself to try it again once I get home before I
> connect to the company VPN.
> >> Thanks for checking in. I will try to update you as soon as I can.
> >>
> >> Thanks
> >> Ajit
> >>
> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >>>
> >>> Hi Ajit,
> >>>
> >>> I'm just checking in; have you heard of any updates on this issue?
> >>>
> >>> Thanks,
> >>> Brandon
> >>>
> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >>>>
> >>>> Hi Ajit,
> >>>>
> >>>> Yes, I believe the issue is coming from the trex/tester system with
> the 100G NIC.
> >>>> I'm not sure what causes this issue; if I run trex using the command
> "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is
> the same command used in DTS, it seems to launch without failing.
> >>>>
> >>>> If you want to replicate it, here are the steps that I ran:
> >>>>
> >>>> (on io) cd /opt/dts
> >>>> export DTS_CFG_FOLDER='conf_100g'
> >>>>
> >>>> conf_100g has the new configuration files to use the new PCI id and
> pktgen config file
> >>>>
> >>>> ./dts -s
> >>>>
> >>>>
> >>>> Thanks for your help,
> >>>> Brandon
> >>>>
> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>>>
> >>>>> Hi Brandon,
> >>>>> No, I haven't seen or heard this before.
> >>>>> But I will try to have someone run it again.
> >>>>>
> >>>>> Just to make sure -
> >>>>> You are running trex on the 100G NIC and the problem is encountered
> on that setup?
> >>>>> Or is it the system that is running testpmd where you are running
> into the issue?
> >>>>>
> >>>>> Thanks
> >>>>> Ajit
> >>>>>
> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >>>>>>
> >>>>>> Hi Ajit,
> >>>>>>
> >>>>>> I'm running into a problem with trying to run nic_single_core_perf
> on the new NIC.
> >>>>>> The current configuration uses trex version v2.82.
> >>>>>> However, I'm running into an error when it tries to actually do a
> test case in the nic_single_core_perf.
> >>>>>>
> >>>>>> The output looks like this when it reaches a test case:
> >>>>>>
> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
> rxd/txd: 512
> >>>>>>>                       dut.rhea:
> ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w
> 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i
> --portmask=0x3 --txd=512 --rxd=512
> >>>>>>>                       dut.rhea: start
> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in
> _execute_test_case
> >>>>>>>     case_obj()
> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in
> test_perf_nic_single_core
> >>>>>>>     self.perf_test(self.nb_ports)
> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in
> perf_test
> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
> measure_throughput
> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
> _prepare_transmission
> >>>>>>>     self._conn.reset(ports=self._ports)
> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>>>>>     ret = f(*args, **kwargs)
> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >>>>>>>     self.clear_stats(ports)
> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>>>>>     ret = f(*args, **kwargs)
> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1467, in clear_stats
> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2840, in _clear_stats_common
> >>>>>>>     raise TRexError(rc)
> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get
> server response from tcp://127.0.0.1:4501
> >>>>>>
> >>>>>>
> >>>>>> I have found one similar case on the github repository for trex,
> but the solution was vendor-specific:
> https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
> >>>>>> Have you ran into this issue before?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Brandon
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Brandon Lo
> >>>>
> >>>> UNH InterOperability Laboratory
> >>>>
> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >>>>
> >>>> blo@iol.unh.edu
> >>>>
> >>>> www.iol.unh.edu
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Brandon Lo
> >>>
> >>> UNH InterOperability Laboratory
> >>>
> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >>>
> >>> blo@iol.unh.edu
> >>>
> >>> www.iol.unh.edu
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>

[-- Attachment #2: Type: text/html, Size: 10716 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-20 21:07                           ` Ajit Khaparde
@ 2020-08-25 20:13                             ` Brandon Lo
  2020-08-26 19:57                               ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-08-25 20:13 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

Hi Ajit,

Did you find anything of interest on the machine?
I could help out if needed.

Thanks,
Brandon

On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
<ajit.khaparde@broadcom.com> wrote:
>
> Hi Brandon,
> I do see some issues while running trex on this setup.
> I will have to dig further. I will let you know once I find something or may be a fix.
>
> Thanks
> Ajit
>
> On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>
>> Hi Ajit,
>>
>> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
>> testpmd (through an ssh session from io --> rhea) to catch information
>> as a DUT.
>>
>> The only commands that I run (on io) are:
>> cd /opt/dts
>> export DTS_CFG_FOLDER='conf_100g'
>> ./dts -s
>>
>> The rest is managed by DTS itself.
>> The issue occurs when DTS is trying to send packets to rhea from io on
>> the new 100G NIC.
>> It seems to happen after it tries to measure throughput using TREX's API.
>>
>> Thanks,
>> Brandon
>>
>> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
>> <ajit.khaparde@broadcom.com> wrote:
>> >
>> > Hi Brandon,
>> > I was trying to see what exactly is happening on the setup and what is causing the problem.
>> > But I will need your help to use the proper code and commands.
>> >
>> > I believe you run the testpmd command on io. And the trex is executed on rhea?
>> > Can you point me to the location of the code and the steps you are following while running the test?
>> >
>> > Thanks
>> > Ajit
>> >
>> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>
>> >> Hi Brandon,
>> >> I haven't. I tried to login as well. But I had some issues doing it from the office.
>> >> I just have to remind myself to try it again once I get home before I connect to the company VPN.
>> >> Thanks for checking in. I will try to update you as soon as I can.
>> >>
>> >> Thanks
>> >> Ajit
>> >>
>> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
>> >>>
>> >>> Hi Ajit,
>> >>>
>> >>> I'm just checking in; have you heard of any updates on this issue?
>> >>>
>> >>> Thanks,
>> >>> Brandon
>> >>>
>> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
>> >>>>
>> >>>> Hi Ajit,
>> >>>>
>> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.
>> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.
>> >>>>
>> >>>> If you want to replicate it, here are the steps that I ran:
>> >>>>
>> >>>> (on io) cd /opt/dts
>> >>>> export DTS_CFG_FOLDER='conf_100g'
>> >>>>
>> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file
>> >>>>
>> >>>> ./dts -s
>> >>>>
>> >>>>
>> >>>> Thanks for your help,
>> >>>> Brandon
>> >>>>
>> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>>>
>> >>>>> Hi Brandon,
>> >>>>> No, I haven't seen or heard this before.
>> >>>>> But I will try to have someone run it again.
>> >>>>>
>> >>>>> Just to make sure -
>> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?
>> >>>>> Or is it the system that is running testpmd where you are running into the issue?
>> >>>>>
>> >>>>> Thanks
>> >>>>> Ajit
>> >>>>>
>> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
>> >>>>>>
>> >>>>>> Hi Ajit,
>> >>>>>>
>> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.
>> >>>>>> The current configuration uses trex version v2.82.
>> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.
>> >>>>>>
>> >>>>>> The output looks like this when it reaches a test case:
>> >>>>>>
>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> >>>>>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512 --rxd=512
>> >>>>>>>                       dut.rhea: start
>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
>> >>>>>>>     case_obj()
>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core
>> >>>>>>>     self.perf_test(self.nb_ports)
>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission
>> >>>>>>>     self._conn.reset(ports=self._ports)
>> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> >>>>>>>     self.clear_stats(ports)
>> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats
>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common
>> >>>>>>>     raise TRexError(rc)
>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501
>> >>>>>>
>> >>>>>>
>> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
>> >>>>>> Have you ran into this issue before?
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Brandon
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>>
>> >>>> Brandon Lo
>> >>>>
>> >>>> UNH InterOperability Laboratory
>> >>>>
>> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >>>>
>> >>>> blo@iol.unh.edu
>> >>>>
>> >>>> www.iol.unh.edu
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>>
>> >>> Brandon Lo
>> >>>
>> >>> UNH InterOperability Laboratory
>> >>>
>> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >>>
>> >>> blo@iol.unh.edu
>> >>>
>> >>> www.iol.unh.edu
>>
>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>
>> blo@iol.unh.edu
>>
>> www.iol.unh.edu



-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-25 20:13                             ` Brandon Lo
@ 2020-08-26 19:57                               ` Ajit Khaparde
  2020-09-01 18:09                                 ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-08-26 19:57 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 7796 bytes --]

Not yet Brandon.
But I believe I might have to look at what trex is doing to make progress.
I will let you know if I need any help.

Thanks
Ajit

On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> Did you find anything of interest on the machine?
> I could help out if needed.
>
> Thanks,
> Brandon
>
> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
> <ajit.khaparde@broadcom.com> wrote:
> >
> > Hi Brandon,
> > I do see some issues while running trex on this setup.
> > I will have to dig further. I will let you know once I find something or
> may be a fix.
> >
> > Thanks
> > Ajit
> >
> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >>
> >> Hi Ajit,
> >>
> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
> >> testpmd (through an ssh session from io --> rhea) to catch information
> >> as a DUT.
> >>
> >> The only commands that I run (on io) are:
> >> cd /opt/dts
> >> export DTS_CFG_FOLDER='conf_100g'
> >> ./dts -s
> >>
> >> The rest is managed by DTS itself.
> >> The issue occurs when DTS is trying to send packets to rhea from io on
> >> the new 100G NIC.
> >> It seems to happen after it tries to measure throughput using TREX's
> API.
> >>
> >> Thanks,
> >> Brandon
> >>
> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
> >> <ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > Hi Brandon,
> >> > I was trying to see what exactly is happening on the setup and what
> is causing the problem.
> >> > But I will need your help to use the proper code and commands.
> >> >
> >> > I believe you run the testpmd command on io. And the trex is executed
> on rhea?
> >> > Can you point me to the location of the code and the steps you are
> following while running the test?
> >> >
> >> > Thanks
> >> > Ajit
> >> >
> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >>
> >> >> Hi Brandon,
> >> >> I haven't. I tried to login as well. But I had some issues doing it
> from the office.
> >> >> I just have to remind myself to try it again once I get home before
> I connect to the company VPN.
> >> >> Thanks for checking in. I will try to update you as soon as I can.
> >> >>
> >> >> Thanks
> >> >> Ajit
> >> >>
> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >> >>>
> >> >>> Hi Ajit,
> >> >>>
> >> >>> I'm just checking in; have you heard of any updates on this issue?
> >> >>>
> >> >>> Thanks,
> >> >>> Brandon
> >> >>>
> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >> >>>>
> >> >>>> Hi Ajit,
> >> >>>>
> >> >>>> Yes, I believe the issue is coming from the trex/tester system
> with the 100G NIC.
> >> >>>> I'm not sure what causes this issue; if I run trex using the
> command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7",
> which is the same command used in DTS, it seems to launch without failing.
> >> >>>>
> >> >>>> If you want to replicate it, here are the steps that I ran:
> >> >>>>
> >> >>>> (on io) cd /opt/dts
> >> >>>> export DTS_CFG_FOLDER='conf_100g'
> >> >>>>
> >> >>>> conf_100g has the new configuration files to use the new PCI id
> and pktgen config file
> >> >>>>
> >> >>>> ./dts -s
> >> >>>>
> >> >>>>
> >> >>>> Thanks for your help,
> >> >>>> Brandon
> >> >>>>
> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >>>>>
> >> >>>>> Hi Brandon,
> >> >>>>> No, I haven't seen or heard this before.
> >> >>>>> But I will try to have someone run it again.
> >> >>>>>
> >> >>>>> Just to make sure -
> >> >>>>> You are running trex on the 100G NIC and the problem is
> encountered on that setup?
> >> >>>>> Or is it the system that is running testpmd where you are running
> into the issue?
> >> >>>>>
> >> >>>>> Thanks
> >> >>>>> Ajit
> >> >>>>>
> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu>
> wrote:
> >> >>>>>>
> >> >>>>>> Hi Ajit,
> >> >>>>>>
> >> >>>>>> I'm running into a problem with trying to run
> nic_single_core_perf on the new NIC.
> >> >>>>>> The current configuration uses trex version v2.82.
> >> >>>>>> However, I'm running into an error when it tries to actually do
> a test case in the nic_single_core_perf.
> >> >>>>>>
> >> >>>>>> The output looks like this when it reaches a test case:
> >> >>>>>>
> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize:
> 64, rxd/txd: 512
> >> >>>>>>>                       dut.rhea:
> ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w
> 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i
> --portmask=0x3 --txd=512 --rxd=512
> >> >>>>>>>                       dut.rhea: start
> >> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in
> _execute_test_case
> >> >>>>>>>     case_obj()
> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in
> test_perf_nic_single_core
> >> >>>>>>>     self.perf_test(self.nb_ports)
> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in
> perf_test
> >> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
> measure_throughput
> >> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
> _prepare_transmission
> >> >>>>>>>     self._conn.reset(ports=self._ports)
> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> >>>>>>>     ret = f(*args, **kwargs)
> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >> >>>>>>>     self.clear_stats(ports)
> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> >>>>>>>     ret = f(*args, **kwargs)
> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1467, in clear_stats
> >> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2840, in _clear_stats_common
> >> >>>>>>>     raise TRexError(rc)
> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to
> get server response from tcp://127.0.0.1:4501
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> I have found one similar case on the github repository for trex,
> but the solution was vendor-specific:
> https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
> >> >>>>>> Have you ran into this issue before?
> >> >>>>>>
> >> >>>>>> Thanks,
> >> >>>>>> Brandon
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>>
> >> >>>> Brandon Lo
> >> >>>>
> >> >>>> UNH InterOperability Laboratory
> >> >>>>
> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >>>>
> >> >>>> blo@iol.unh.edu
> >> >>>>
> >> >>>> www.iol.unh.edu
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>>
> >> >>> Brandon Lo
> >> >>>
> >> >>> UNH InterOperability Laboratory
> >> >>>
> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >>>
> >> >>> blo@iol.unh.edu
> >> >>>
> >> >>> www.iol.unh.edu
> >>
> >>
> >>
> >> --
> >>
> >> Brandon Lo
> >>
> >> UNH InterOperability Laboratory
> >>
> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >>
> >> blo@iol.unh.edu
> >>
> >> www.iol.unh.edu
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>

[-- Attachment #2: Type: text/html, Size: 13220 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-08-26 19:57                               ` Ajit Khaparde
@ 2020-09-01 18:09                                 ` Brandon Lo
  2020-09-01 18:11                                   ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-09-01 18:09 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

Hi Ajit,

I'm going to re-enable the Broadcom 25G NIC in production performance
and functional testing.

If you would like to do testing/debugging on the machine, such as on
the 100G NIC, then please disable these two pipelines on our Jenkins:
    https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/
    https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/

You will need to sign in using the same credentials as your VPN.

This will prevent any overlapping of DTS/TREX runs which is known to
cause errors.
Please let me know if you run into any issues.

Thanks,
Brandon

On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde
<ajit.khaparde@broadcom.com> wrote:
>
> Not yet Brandon.
> But I believe I might have to look at what trex is doing to make progress.
> I will let you know if I need any help.
>
> Thanks
> Ajit
>
> On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <blo@iol.unh.edu> wrote:
>>
>> Hi Ajit,
>>
>> Did you find anything of interest on the machine?
>> I could help out if needed.
>>
>> Thanks,
>> Brandon
>>
>> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
>> <ajit.khaparde@broadcom.com> wrote:
>> >
>> > Hi Brandon,
>> > I do see some issues while running trex on this setup.
>> > I will have to dig further. I will let you know once I find something or may be a fix.
>> >
>> > Thanks
>> > Ajit
>> >
>> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:
>> >>
>> >> Hi Ajit,
>> >>
>> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
>> >> testpmd (through an ssh session from io --> rhea) to catch information
>> >> as a DUT.
>> >>
>> >> The only commands that I run (on io) are:
>> >> cd /opt/dts
>> >> export DTS_CFG_FOLDER='conf_100g'
>> >> ./dts -s
>> >>
>> >> The rest is managed by DTS itself.
>> >> The issue occurs when DTS is trying to send packets to rhea from io on
>> >> the new 100G NIC.
>> >> It seems to happen after it tries to measure throughput using TREX's API.
>> >>
>> >> Thanks,
>> >> Brandon
>> >>
>> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
>> >> <ajit.khaparde@broadcom.com> wrote:
>> >> >
>> >> > Hi Brandon,
>> >> > I was trying to see what exactly is happening on the setup and what is causing the problem.
>> >> > But I will need your help to use the proper code and commands.
>> >> >
>> >> > I believe you run the testpmd command on io. And the trex is executed on rhea?
>> >> > Can you point me to the location of the code and the steps you are following while running the test?
>> >> >
>> >> > Thanks
>> >> > Ajit
>> >> >
>> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> >>
>> >> >> Hi Brandon,
>> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office.
>> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN.
>> >> >> Thanks for checking in. I will try to update you as soon as I can.
>> >> >>
>> >> >> Thanks
>> >> >> Ajit
>> >> >>
>> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
>> >> >>>
>> >> >>> Hi Ajit,
>> >> >>>
>> >> >>> I'm just checking in; have you heard of any updates on this issue?
>> >> >>>
>> >> >>> Thanks,
>> >> >>> Brandon
>> >> >>>
>> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
>> >> >>>>
>> >> >>>> Hi Ajit,
>> >> >>>>
>> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.
>> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.
>> >> >>>>
>> >> >>>> If you want to replicate it, here are the steps that I ran:
>> >> >>>>
>> >> >>>> (on io) cd /opt/dts
>> >> >>>> export DTS_CFG_FOLDER='conf_100g'
>> >> >>>>
>> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file
>> >> >>>>
>> >> >>>> ./dts -s
>> >> >>>>
>> >> >>>>
>> >> >>>> Thanks for your help,
>> >> >>>> Brandon
>> >> >>>>
>> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> >>>>>
>> >> >>>>> Hi Brandon,
>> >> >>>>> No, I haven't seen or heard this before.
>> >> >>>>> But I will try to have someone run it again.
>> >> >>>>>
>> >> >>>>> Just to make sure -
>> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?
>> >> >>>>> Or is it the system that is running testpmd where you are running into the issue?
>> >> >>>>>
>> >> >>>>> Thanks
>> >> >>>>> Ajit
>> >> >>>>>
>> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
>> >> >>>>>>
>> >> >>>>>> Hi Ajit,
>> >> >>>>>>
>> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.
>> >> >>>>>> The current configuration uses trex version v2.82.
>> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.
>> >> >>>>>>
>> >> >>>>>> The output looks like this when it reaches a test case:
>> >> >>>>>>
>> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> >> >>>>>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512 --rxd=512
>> >> >>>>>>>                       dut.rhea: start
>> >> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
>> >> >>>>>>>     case_obj()
>> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core
>> >> >>>>>>>     self.perf_test(self.nb_ports)
>> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>> >> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> >> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> >> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission
>> >> >>>>>>>     self._conn.reset(ports=self._ports)
>> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >> >>>>>>>     ret = f(*args, **kwargs)
>> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> >> >>>>>>>     self.clear_stats(ports)
>> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >> >>>>>>>     ret = f(*args, **kwargs)
>> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats
>> >> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common
>> >> >>>>>>>     raise TRexError(rc)
>> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501
>> >> >>>>>>
>> >> >>>>>>
>> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
>> >> >>>>>> Have you ran into this issue before?
>> >> >>>>>>
>> >> >>>>>> Thanks,
>> >> >>>>>> Brandon
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>>
>> >> >>>> Brandon Lo
>> >> >>>>
>> >> >>>> UNH InterOperability Laboratory
>> >> >>>>
>> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >> >>>>
>> >> >>>> blo@iol.unh.edu
>> >> >>>>
>> >> >>>> www.iol.unh.edu
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>>
>> >> >>> Brandon Lo
>> >> >>>
>> >> >>> UNH InterOperability Laboratory
>> >> >>>
>> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >> >>>
>> >> >>> blo@iol.unh.edu
>> >> >>>
>> >> >>> www.iol.unh.edu
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Brandon Lo
>> >>
>> >> UNH InterOperability Laboratory
>> >>
>> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >>
>> >> blo@iol.unh.edu
>> >>
>> >> www.iol.unh.edu
>>
>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>
>> blo@iol.unh.edu
>>
>> www.iol.unh.edu



-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-09-01 18:09                                 ` Brandon Lo
@ 2020-09-01 18:11                                   ` Ajit Khaparde
  2020-10-22 19:03                                     ` Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-09-01 18:11 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 9517 bytes --]

Hi Brandon,
Thanks for this.
I have been stretched thin lately, and this will surely help.

Thanks
Ajit

On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo <blo@iol.unh.edu> wrote:

> Hi Ajit,
>
> I'm going to re-enable the Broadcom 25G NIC in production performance
> and functional testing.
>
> If you would like to do testing/debugging on the machine, such as on
> the 100G NIC, then please disable these two pipelines on our Jenkins:
>
> https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/
>
> https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/
>
> You will need to sign in using the same credentials as your VPN.
>
> This will prevent any overlapping of DTS/TREX runs which is known to
> cause errors.
> Please let me know if you run into any issues.
>
> Thanks,
> Brandon
>
> On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde
> <ajit.khaparde@broadcom.com> wrote:
> >
> > Not yet Brandon.
> > But I believe I might have to look at what trex is doing to make
> progress.
> > I will let you know if I need any help.
> >
> > Thanks
> > Ajit
> >
> > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >>
> >> Hi Ajit,
> >>
> >> Did you find anything of interest on the machine?
> >> I could help out if needed.
> >>
> >> Thanks,
> >> Brandon
> >>
> >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
> >> <ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > Hi Brandon,
> >> > I do see some issues while running trex on this setup.
> >> > I will have to dig further. I will let you know once I find something
> or may be a fix.
> >> >
> >> > Thanks
> >> > Ajit
> >> >
> >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >> >>
> >> >> Hi Ajit,
> >> >>
> >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
> >> >> testpmd (through an ssh session from io --> rhea) to catch
> information
> >> >> as a DUT.
> >> >>
> >> >> The only commands that I run (on io) are:
> >> >> cd /opt/dts
> >> >> export DTS_CFG_FOLDER='conf_100g'
> >> >> ./dts -s
> >> >>
> >> >> The rest is managed by DTS itself.
> >> >> The issue occurs when DTS is trying to send packets to rhea from io
> on
> >> >> the new 100G NIC.
> >> >> It seems to happen after it tries to measure throughput using TREX's
> API.
> >> >>
> >> >> Thanks,
> >> >> Brandon
> >> >>
> >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
> >> >> <ajit.khaparde@broadcom.com> wrote:
> >> >> >
> >> >> > Hi Brandon,
> >> >> > I was trying to see what exactly is happening on the setup and
> what is causing the problem.
> >> >> > But I will need your help to use the proper code and commands.
> >> >> >
> >> >> > I believe you run the testpmd command on io. And the trex is
> executed on rhea?
> >> >> > Can you point me to the location of the code and the steps you are
> following while running the test?
> >> >> >
> >> >> > Thanks
> >> >> > Ajit
> >> >> >
> >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> >>
> >> >> >> Hi Brandon,
> >> >> >> I haven't. I tried to login as well. But I had some issues doing
> it from the office.
> >> >> >> I just have to remind myself to try it again once I get home
> before I connect to the company VPN.
> >> >> >> Thanks for checking in. I will try to update you as soon as I can.
> >> >> >>
> >> >> >> Thanks
> >> >> >> Ajit
> >> >> >>
> >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu>
> wrote:
> >> >> >>>
> >> >> >>> Hi Ajit,
> >> >> >>>
> >> >> >>> I'm just checking in; have you heard of any updates on this
> issue?
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>> Brandon
> >> >> >>>
> >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu>
> wrote:
> >> >> >>>>
> >> >> >>>> Hi Ajit,
> >> >> >>>>
> >> >> >>>> Yes, I believe the issue is coming from the trex/tester system
> with the 100G NIC.
> >> >> >>>> I'm not sure what causes this issue; if I run trex using the
> command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7",
> which is the same command used in DTS, it seems to launch without failing.
> >> >> >>>>
> >> >> >>>> If you want to replicate it, here are the steps that I ran:
> >> >> >>>>
> >> >> >>>> (on io) cd /opt/dts
> >> >> >>>> export DTS_CFG_FOLDER='conf_100g'
> >> >> >>>>
> >> >> >>>> conf_100g has the new configuration files to use the new PCI id
> and pktgen config file
> >> >> >>>>
> >> >> >>>> ./dts -s
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> Thanks for your help,
> >> >> >>>> Brandon
> >> >> >>>>
> >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> >>>>>
> >> >> >>>>> Hi Brandon,
> >> >> >>>>> No, I haven't seen or heard this before.
> >> >> >>>>> But I will try to have someone run it again.
> >> >> >>>>>
> >> >> >>>>> Just to make sure -
> >> >> >>>>> You are running trex on the 100G NIC and the problem is
> encountered on that setup?
> >> >> >>>>> Or is it the system that is running testpmd where you are
> running into the issue?
> >> >> >>>>>
> >> >> >>>>> Thanks
> >> >> >>>>> Ajit
> >> >> >>>>>
> >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu>
> wrote:
> >> >> >>>>>>
> >> >> >>>>>> Hi Ajit,
> >> >> >>>>>>
> >> >> >>>>>> I'm running into a problem with trying to run
> nic_single_core_perf on the new NIC.
> >> >> >>>>>> The current configuration uses trex version v2.82.
> >> >> >>>>>> However, I'm running into an error when it tries to actually
> do a test case in the nic_single_core_perf.
> >> >> >>>>>>
> >> >> >>>>>> The output looks like this when it reaches a test case:
> >> >> >>>>>>
> >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
> framesize: 64, rxd/txd: 512
> >> >> >>>>>>>                       dut.rhea:
> ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w
> 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i
> --portmask=0x3 --txd=512 --rxd=512
> >> >> >>>>>>>                       dut.rhea: start
> >> >> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >> >> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in
> _execute_test_case
> >> >> >>>>>>>     case_obj()
> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198,
> in test_perf_nic_single_core
> >> >> >>>>>>>     self.perf_test(self.nb_ports)
> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259,
> in perf_test
> >> >> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
> measure_throughput
> >> >> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in
> _prepare_transmission
> >> >> >>>>>>>     self._conn.reset(ports=self._ports)
> >> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> >> >>>>>>>     ret = f(*args, **kwargs)
> >> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >> >> >>>>>>>     self.clear_stats(ports)
> >> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> >> >>>>>>>     ret = f(*args, **kwargs)
> >> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1467, in clear_stats
> >> >> >>>>>>>     self._clear_stats_common(ports, clear_global,
> clear_xstats)
> >> >> >>>>>>>   File
> "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2840, in _clear_stats_common
> >> >> >>>>>>>     raise TRexError(rc)
> >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to
> get server response from tcp://127.0.0.1:4501
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>> I have found one similar case on the github repository for
> trex, but the solution was vendor-specific:
> https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
> >> >> >>>>>> Have you ran into this issue before?
> >> >> >>>>>>
> >> >> >>>>>> Thanks,
> >> >> >>>>>> Brandon
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> --
> >> >> >>>>
> >> >> >>>> Brandon Lo
> >> >> >>>>
> >> >> >>>> UNH InterOperability Laboratory
> >> >> >>>>
> >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >> >>>>
> >> >> >>>> blo@iol.unh.edu
> >> >> >>>>
> >> >> >>>> www.iol.unh.edu
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> --
> >> >> >>>
> >> >> >>> Brandon Lo
> >> >> >>>
> >> >> >>> UNH InterOperability Laboratory
> >> >> >>>
> >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >> >>>
> >> >> >>> blo@iol.unh.edu
> >> >> >>>
> >> >> >>> www.iol.unh.edu
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Brandon Lo
> >> >>
> >> >> UNH InterOperability Laboratory
> >> >>
> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >>
> >> >> blo@iol.unh.edu
> >> >>
> >> >> www.iol.unh.edu
> >>
> >>
> >>
> >> --
> >>
> >> Brandon Lo
> >>
> >> UNH InterOperability Laboratory
> >>
> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >>
> >> blo@iol.unh.edu
> >>
> >> www.iol.unh.edu
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu
>

[-- Attachment #2: Type: text/html, Size: 16827 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-09-01 18:11                                   ` Ajit Khaparde
@ 2020-10-22 19:03                                     ` Brandon Lo
  2020-10-22 19:27                                       ` Ajit Khaparde
  0 siblings, 1 reply; 22+ messages in thread
From: Brandon Lo @ 2020-10-22 19:03 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

Hi Ajit,

Just checking in to make sure this issue is not dropped.
I have tested the 100G NIC running TREX/DTS again, it still seems that
the 100G NIC is not able to use TREX as the console is outputting
"Connection lost - Subscriber timeout: no data from TRex server."
However, I know that some developers are busy for DPDK so I understand
if you are a bit pressed on time currently.

Thanks,
Brandon

On Tue, Sep 1, 2020 at 2:12 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
> Hi Brandon,
> Thanks for this.
> I have been stretched thin lately, and this will surely help.
>
> Thanks
> Ajit
>
> On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo <blo@iol.unh.edu> wrote:
>>
>> Hi Ajit,
>>
>> I'm going to re-enable the Broadcom 25G NIC in production performance
>> and functional testing.
>>
>> If you would like to do testing/debugging on the machine, such as on
>> the 100G NIC, then please disable these two pipelines on our Jenkins:
>>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/
>>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/
>>
>> You will need to sign in using the same credentials as your VPN.
>>
>> This will prevent any overlapping of DTS/TREX runs which is known to
>> cause errors.
>> Please let me know if you run into any issues.
>>
>> Thanks,
>> Brandon
>>
>> On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde
>> <ajit.khaparde@broadcom.com> wrote:
>> >
>> > Not yet Brandon.
>> > But I believe I might have to look at what trex is doing to make progress.
>> > I will let you know if I need any help.
>> >
>> > Thanks
>> > Ajit
>> >
>> > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <blo@iol.unh.edu> wrote:
>> >>
>> >> Hi Ajit,
>> >>
>> >> Did you find anything of interest on the machine?
>> >> I could help out if needed.
>> >>
>> >> Thanks,
>> >> Brandon
>> >>
>> >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
>> >> <ajit.khaparde@broadcom.com> wrote:
>> >> >
>> >> > Hi Brandon,
>> >> > I do see some issues while running trex on this setup.
>> >> > I will have to dig further. I will let you know once I find something or may be a fix.
>> >> >
>> >> > Thanks
>> >> > Ajit
>> >> >
>> >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:
>> >> >>
>> >> >> Hi Ajit,
>> >> >>
>> >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
>> >> >> testpmd (through an ssh session from io --> rhea) to catch information
>> >> >> as a DUT.
>> >> >>
>> >> >> The only commands that I run (on io) are:
>> >> >> cd /opt/dts
>> >> >> export DTS_CFG_FOLDER='conf_100g'
>> >> >> ./dts -s
>> >> >>
>> >> >> The rest is managed by DTS itself.
>> >> >> The issue occurs when DTS is trying to send packets to rhea from io on
>> >> >> the new 100G NIC.
>> >> >> It seems to happen after it tries to measure throughput using TREX's API.
>> >> >>
>> >> >> Thanks,
>> >> >> Brandon
>> >> >>
>> >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
>> >> >> <ajit.khaparde@broadcom.com> wrote:
>> >> >> >
>> >> >> > Hi Brandon,
>> >> >> > I was trying to see what exactly is happening on the setup and what is causing the problem.
>> >> >> > But I will need your help to use the proper code and commands.
>> >> >> >
>> >> >> > I believe you run the testpmd command on io. And the trex is executed on rhea?
>> >> >> > Can you point me to the location of the code and the steps you are following while running the test?
>> >> >> >
>> >> >> > Thanks
>> >> >> > Ajit
>> >> >> >
>> >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> >> >>
>> >> >> >> Hi Brandon,
>> >> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office.
>> >> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN.
>> >> >> >> Thanks for checking in. I will try to update you as soon as I can.
>> >> >> >>
>> >> >> >> Thanks
>> >> >> >> Ajit
>> >> >> >>
>> >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
>> >> >> >>>
>> >> >> >>> Hi Ajit,
>> >> >> >>>
>> >> >> >>> I'm just checking in; have you heard of any updates on this issue?
>> >> >> >>>
>> >> >> >>> Thanks,
>> >> >> >>> Brandon
>> >> >> >>>
>> >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
>> >> >> >>>>
>> >> >> >>>> Hi Ajit,
>> >> >> >>>>
>> >> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.
>> >> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.
>> >> >> >>>>
>> >> >> >>>> If you want to replicate it, here are the steps that I ran:
>> >> >> >>>>
>> >> >> >>>> (on io) cd /opt/dts
>> >> >> >>>> export DTS_CFG_FOLDER='conf_100g'
>> >> >> >>>>
>> >> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file
>> >> >> >>>>
>> >> >> >>>> ./dts -s
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> Thanks for your help,
>> >> >> >>>> Brandon
>> >> >> >>>>
>> >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> >> >>>>>
>> >> >> >>>>> Hi Brandon,
>> >> >> >>>>> No, I haven't seen or heard this before.
>> >> >> >>>>> But I will try to have someone run it again.
>> >> >> >>>>>
>> >> >> >>>>> Just to make sure -
>> >> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?
>> >> >> >>>>> Or is it the system that is running testpmd where you are running into the issue?
>> >> >> >>>>>
>> >> >> >>>>> Thanks
>> >> >> >>>>> Ajit
>> >> >> >>>>>
>> >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
>> >> >> >>>>>>
>> >> >> >>>>>> Hi Ajit,
>> >> >> >>>>>>
>> >> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.
>> >> >> >>>>>> The current configuration uses trex version v2.82.
>> >> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.
>> >> >> >>>>>>
>> >> >> >>>>>> The output looks like this when it reaches a test case:
>> >> >> >>>>>>
>> >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> >> >> >>>>>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512 --rxd=512
>> >> >> >>>>>>>                       dut.rhea: start
>> >> >> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >> >> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
>> >> >> >>>>>>>     case_obj()
>> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core
>> >> >> >>>>>>>     self.perf_test(self.nb_ports)
>> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
>> >> >> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> >> >> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission
>> >> >> >>>>>>>     self._conn.reset(ports=self._ports)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >> >> >>>>>>>     ret = f(*args, **kwargs)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> >> >> >>>>>>>     self.clear_stats(ports)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >> >> >>>>>>>     ret = f(*args, **kwargs)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats
>> >> >> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common
>> >> >> >>>>>>>     raise TRexError(rc)
>> >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501
>> >> >> >>>>>>
>> >> >> >>>>>>
>> >> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
>> >> >> >>>>>> Have you ran into this issue before?
>> >> >> >>>>>>
>> >> >> >>>>>> Thanks,
>> >> >> >>>>>> Brandon
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>>> --
>> >> >> >>>>
>> >> >> >>>> Brandon Lo
>> >> >> >>>>
>> >> >> >>>> UNH InterOperability Laboratory
>> >> >> >>>>
>> >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >> >> >>>>
>> >> >> >>>> blo@iol.unh.edu
>> >> >> >>>>
>> >> >> >>>> www.iol.unh.edu
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>>
>> >> >> >>> Brandon Lo
>> >> >> >>>
>> >> >> >>> UNH InterOperability Laboratory
>> >> >> >>>
>> >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >> >> >>>
>> >> >> >>> blo@iol.unh.edu
>> >> >> >>>
>> >> >> >>> www.iol.unh.edu
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >>
>> >> >> Brandon Lo
>> >> >>
>> >> >> UNH InterOperability Laboratory
>> >> >>
>> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >> >>
>> >> >> blo@iol.unh.edu
>> >> >>
>> >> >> www.iol.unh.edu
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Brandon Lo
>> >>
>> >> UNH InterOperability Laboratory
>> >>
>> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
>> >>
>> >> blo@iol.unh.edu
>> >>
>> >> www.iol.unh.edu
>>
>>
>>
>> --
>>
>> Brandon Lo
>>
>> UNH InterOperability Laboratory
>>
>> 21 Madbury Rd, Suite 100, Durham, NH 03824
>>
>> blo@iol.unh.edu
>>
>> www.iol.unh.edu



-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] New 100G Broadcom NIC Ubuntu
  2020-10-22 19:03                                     ` Brandon Lo
@ 2020-10-22 19:27                                       ` Ajit Khaparde
  2020-12-03 16:48                                         ` [dpdk-ci] [dpdklab] " Brandon Lo
  0 siblings, 1 reply; 22+ messages in thread
From: Ajit Khaparde @ 2020-10-22 19:27 UTC (permalink / raw)
  To: Brandon Lo; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 11076 bytes --]

Hi Brandon,
Yes. I haven't been able to spend time on this because of the release.
But I will try to get to it soon.
Thanks for checking.

Thanks
Ajit

On Thu, Oct 22, 2020 at 12:04 PM Brandon Lo <blo@iol.unh.edu> wrote:
>
> Hi Ajit,
>
> Just checking in to make sure this issue is not dropped.
> I have tested the 100G NIC running TREX/DTS again, it still seems that
> the 100G NIC is not able to use TREX as the console is outputting
> "Connection lost - Subscriber timeout: no data from TRex server."
> However, I know that some developers are busy for DPDK so I understand
> if you are a bit pressed on time currently.
>
> Thanks,
> Brandon
>
> On Tue, Sep 1, 2020 at 2:12 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >
> > Hi Brandon,
> > Thanks for this.
> > I have been stretched thin lately, and this will surely help.
> >
> > Thanks
> > Ajit
> >
> > On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >>
> >> Hi Ajit,
> >>
> >> I'm going to re-enable the Broadcom 25G NIC in production performance
> >> and functional testing.
> >>
> >> If you would like to do testing/debugging on the machine, such as on
> >> the 100G NIC, then please disable these two pipelines on our Jenkins:
> >>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/
> >>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/
> >>
> >> You will need to sign in using the same credentials as your VPN.
> >>
> >> This will prevent any overlapping of DTS/TREX runs which is known to
> >> cause errors.
> >> Please let me know if you run into any issues.
> >>
> >> Thanks,
> >> Brandon
> >>
> >> On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde
> >> <ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > Not yet Brandon.
> >> > But I believe I might have to look at what trex is doing to make progress.
> >> > I will let you know if I need any help.
> >> >
> >> > Thanks
> >> > Ajit
> >> >
> >> > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >> >>
> >> >> Hi Ajit,
> >> >>
> >> >> Did you find anything of interest on the machine?
> >> >> I could help out if needed.
> >> >>
> >> >> Thanks,
> >> >> Brandon
> >> >>
> >> >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
> >> >> <ajit.khaparde@broadcom.com> wrote:
> >> >> >
> >> >> > Hi Brandon,
> >> >> > I do see some issues while running trex on this setup.
> >> >> > I will have to dig further. I will let you know once I find something or may be a fix.
> >> >> >
> >> >> > Thanks
> >> >> > Ajit
> >> >> >
> >> >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >> >> >>
> >> >> >> Hi Ajit,
> >> >> >>
> >> >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
> >> >> >> testpmd (through an ssh session from io --> rhea) to catch information
> >> >> >> as a DUT.
> >> >> >>
> >> >> >> The only commands that I run (on io) are:
> >> >> >> cd /opt/dts
> >> >> >> export DTS_CFG_FOLDER='conf_100g'
> >> >> >> ./dts -s
> >> >> >>
> >> >> >> The rest is managed by DTS itself.
> >> >> >> The issue occurs when DTS is trying to send packets to rhea from io on
> >> >> >> the new 100G NIC.
> >> >> >> It seems to happen after it tries to measure throughput using TREX's API.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Brandon
> >> >> >>
> >> >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
> >> >> >> <ajit.khaparde@broadcom.com> wrote:
> >> >> >> >
> >> >> >> > Hi Brandon,
> >> >> >> > I was trying to see what exactly is happening on the setup and what is causing the problem.
> >> >> >> > But I will need your help to use the proper code and commands.
> >> >> >> >
> >> >> >> > I believe you run the testpmd command on io. And the trex is executed on rhea?
> >> >> >> > Can you point me to the location of the code and the steps you are following while running the test?
> >> >> >> >
> >> >> >> > Thanks
> >> >> >> > Ajit
> >> >> >> >
> >> >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >> >> >>
> >> >> >> >> Hi Brandon,
> >> >> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office.
> >> >> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN.
> >> >> >> >> Thanks for checking in. I will try to update you as soon as I can.
> >> >> >> >>
> >> >> >> >> Thanks
> >> >> >> >> Ajit
> >> >> >> >>
> >> >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >> >> >> >>>
> >> >> >> >>> Hi Ajit,
> >> >> >> >>>
> >> >> >> >>> I'm just checking in; have you heard of any updates on this issue?
> >> >> >> >>>
> >> >> >> >>> Thanks,
> >> >> >> >>> Brandon
> >> >> >> >>>
> >> >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >> >> >> >>>>
> >> >> >> >>>> Hi Ajit,
> >> >> >> >>>>
> >> >> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.
> >> >> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.
> >> >> >> >>>>
> >> >> >> >>>> If you want to replicate it, here are the steps that I ran:
> >> >> >> >>>>
> >> >> >> >>>> (on io) cd /opt/dts
> >> >> >> >>>> export DTS_CFG_FOLDER='conf_100g'
> >> >> >> >>>>
> >> >> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file
> >> >> >> >>>>
> >> >> >> >>>> ./dts -s
> >> >> >> >>>>
> >> >> >> >>>>
> >> >> >> >>>> Thanks for your help,
> >> >> >> >>>> Brandon
> >> >> >> >>>>
> >> >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >> >> >>>>>
> >> >> >> >>>>> Hi Brandon,
> >> >> >> >>>>> No, I haven't seen or heard this before.
> >> >> >> >>>>> But I will try to have someone run it again.
> >> >> >> >>>>>
> >> >> >> >>>>> Just to make sure -
> >> >> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?
> >> >> >> >>>>> Or is it the system that is running testpmd where you are running into the issue?
> >> >> >> >>>>>
> >> >> >> >>>>> Thanks
> >> >> >> >>>>> Ajit
> >> >> >> >>>>>
> >> >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
> >> >> >> >>>>>>
> >> >> >> >>>>>> Hi Ajit,
> >> >> >> >>>>>>
> >> >> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.
> >> >> >> >>>>>> The current configuration uses trex version v2.82.
> >> >> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.
> >> >> >> >>>>>>
> >> >> >> >>>>>> The output looks like this when it reaches a test case:
> >> >> >> >>>>>>
> >> >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
> >> >> >> >>>>>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512 --rxd=512
> >> >> >> >>>>>>>                       dut.rhea: start
> >> >> >> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >> >> >> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
> >> >> >> >>>>>>>     case_obj()
> >> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core
> >> >> >> >>>>>>>     self.perf_test(self.nb_ports)
> >> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
> >> >> >> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
> >> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >> >> >> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission
> >> >> >> >>>>>>>     self._conn.reset(ports=self._ports)
> >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >> >> >> >>>>>>>     ret = f(*args, **kwargs)
> >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
> >> >> >> >>>>>>>     self.clear_stats(ports)
> >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >> >> >> >>>>>>>     ret = f(*args, **kwargs)
> >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats
> >> >> >> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common
> >> >> >> >>>>>>>     raise TRexError(rc)
> >> >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501
> >> >> >> >>>>>>
> >> >> >> >>>>>>
> >> >> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
> >> >> >> >>>>>> Have you ran into this issue before?
> >> >> >> >>>>>>
> >> >> >> >>>>>> Thanks,
> >> >> >> >>>>>> Brandon
> >> >> >> >>>>
> >> >> >> >>>>
> >> >> >> >>>>
> >> >> >> >>>> --
> >> >> >> >>>>
> >> >> >> >>>> Brandon Lo
> >> >> >> >>>>
> >> >> >> >>>> UNH InterOperability Laboratory
> >> >> >> >>>>
> >> >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >> >> >>>>
> >> >> >> >>>> blo@iol.unh.edu
> >> >> >> >>>>
> >> >> >> >>>> www.iol.unh.edu
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>>
> >> >> >> >>> Brandon Lo
> >> >> >> >>>
> >> >> >> >>> UNH InterOperability Laboratory
> >> >> >> >>>
> >> >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >> >> >>>
> >> >> >> >>> blo@iol.unh.edu
> >> >> >> >>>
> >> >> >> >>> www.iol.unh.edu
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >>
> >> >> >> Brandon Lo
> >> >> >>
> >> >> >> UNH InterOperability Laboratory
> >> >> >>
> >> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >> >>
> >> >> >> blo@iol.unh.edu
> >> >> >>
> >> >> >> www.iol.unh.edu
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >> Brandon Lo
> >> >>
> >> >> UNH InterOperability Laboratory
> >> >>
> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >> >>
> >> >> blo@iol.unh.edu
> >> >>
> >> >> www.iol.unh.edu
> >>
> >>
> >>
> >> --
> >>
> >> Brandon Lo
> >>
> >> UNH InterOperability Laboratory
> >>
> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> >>
> >> blo@iol.unh.edu
> >>
> >> www.iol.unh.edu
>
>
>
> --
>
> Brandon Lo
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> blo@iol.unh.edu
>
> www.iol.unh.edu

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 22+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: New 100G Broadcom NIC Ubuntu
  2020-10-22 19:27                                       ` Ajit Khaparde
@ 2020-12-03 16:48                                         ` Brandon Lo
  0 siblings, 0 replies; 22+ messages in thread
From: Brandon Lo @ 2020-12-03 16:48 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

Hi Ajit,

Hope you are doing well. Have you had a chance to investigate what
could cause this issue?
At UNH, we are facing a similar issue with some of the new Arm systems.
Maybe they could be related and help us figure out how to fix the
Broadcom machine as well.
I will let you know if anything pops up that could be interesting.

Thanks,
Brandon

On Thu, Oct 22, 2020 at 3:27 PM 'Ajit Khaparde' via dpdklab
<dpdklab@iol.unh.edu> wrote:
>
> Hi Brandon,
> Yes. I haven't been able to spend time on this because of the release.
> But I will try to get to it soon.
> Thanks for checking.
>
> Thanks
> Ajit
>
> On Thu, Oct 22, 2020 at 12:04 PM Brandon Lo <blo@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > Just checking in to make sure this issue is not dropped.
> > I have tested the 100G NIC running TREX/DTS again, it still seems that
> > the 100G NIC is not able to use TREX as the console is outputting
> > "Connection lost - Subscriber timeout: no data from TRex server."
> > However, I know that some developers are busy for DPDK so I understand
> > if you are a bit pressed on time currently.
> >
> > Thanks,
> > Brandon
> >
> > On Tue, Sep 1, 2020 at 2:12 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> > >
> > > Hi Brandon,
> > > Thanks for this.
> > > I have been stretched thin lately, and this will surely help.
> > >
> > > Thanks
> > > Ajit
> > >
> > > On Tue, Sep 1, 2020 at 11:09 AM Brandon Lo <blo@iol.unh.edu> wrote:
> > >>
> > >> Hi Ajit,
> > >>
> > >> I'm going to re-enable the Broadcom 25G NIC in production performance
> > >> and functional testing.
> > >>
> > >> If you would like to do testing/debugging on the machine, such as on
> > >> the 100G NIC, then please disable these two pipelines on our Jenkins:
> > >>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Functional-Test-Pipeline/
> > >>     https://dpdklab.iol.unh.edu/jenkins/job/Broadcom-25G-Performance-Test-Pipeline/
> > >>
> > >> You will need to sign in using the same credentials as your VPN.
> > >>
> > >> This will prevent any overlapping of DTS/TREX runs which is known to
> > >> cause errors.
> > >> Please let me know if you run into any issues.
> > >>
> > >> Thanks,
> > >> Brandon
> > >>
> > >> On Wed, Aug 26, 2020 at 3:58 PM Ajit Khaparde
> > >> <ajit.khaparde@broadcom.com> wrote:
> > >> >
> > >> > Not yet Brandon.
> > >> > But I believe I might have to look at what trex is doing to make progress.
> > >> > I will let you know if I need any help.
> > >> >
> > >> > Thanks
> > >> > Ajit
> > >> >
> > >> > On Tue, Aug 25, 2020 at 1:13 PM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >>
> > >> >> Hi Ajit,
> > >> >>
> > >> >> Did you find anything of interest on the machine?
> > >> >> I could help out if needed.
> > >> >>
> > >> >> Thanks,
> > >> >> Brandon
> > >> >>
> > >> >> On Thu, Aug 20, 2020 at 5:07 PM Ajit Khaparde
> > >> >> <ajit.khaparde@broadcom.com> wrote:
> > >> >> >
> > >> >> > Hi Brandon,
> > >> >> > I do see some issues while running trex on this setup.
> > >> >> > I will have to dig further. I will let you know once I find something or may be a fix.
> > >> >> >
> > >> >> > Thanks
> > >> >> > Ajit
> > >> >> >
> > >> >> > On Thu, Aug 20, 2020 at 8:39 AM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >> >>
> > >> >> >> Hi Ajit,
> > >> >> >>
> > >> >> >> I believe DTS/TREX is run on the io machine as a tester. Rhea runs
> > >> >> >> testpmd (through an ssh session from io --> rhea) to catch information
> > >> >> >> as a DUT.
> > >> >> >>
> > >> >> >> The only commands that I run (on io) are:
> > >> >> >> cd /opt/dts
> > >> >> >> export DTS_CFG_FOLDER='conf_100g'
> > >> >> >> ./dts -s
> > >> >> >>
> > >> >> >> The rest is managed by DTS itself.
> > >> >> >> The issue occurs when DTS is trying to send packets to rhea from io on
> > >> >> >> the new 100G NIC.
> > >> >> >> It seems to happen after it tries to measure throughput using TREX's API.
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> Brandon
> > >> >> >>
> > >> >> >> On Thu, Aug 20, 2020 at 12:29 AM Ajit Khaparde
> > >> >> >> <ajit.khaparde@broadcom.com> wrote:
> > >> >> >> >
> > >> >> >> > Hi Brandon,
> > >> >> >> > I was trying to see what exactly is happening on the setup and what is causing the problem.
> > >> >> >> > But I will need your help to use the proper code and commands.
> > >> >> >> >
> > >> >> >> > I believe you run the testpmd command on io. And the trex is executed on rhea?
> > >> >> >> > Can you point me to the location of the code and the steps you are following while running the test?
> > >> >> >> >
> > >> >> >> > Thanks
> > >> >> >> > Ajit
> > >> >> >> >
> > >> >> >> > On Tue, Aug 11, 2020 at 10:47 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> > >> >> >> >>
> > >> >> >> >> Hi Brandon,
> > >> >> >> >> I haven't. I tried to login as well. But I had some issues doing it from the office.
> > >> >> >> >> I just have to remind myself to try it again once I get home before I connect to the company VPN.
> > >> >> >> >> Thanks for checking in. I will try to update you as soon as I can.
> > >> >> >> >>
> > >> >> >> >> Thanks
> > >> >> >> >> Ajit
> > >> >> >> >>
> > >> >> >> >> On Tue, Aug 11, 2020 at 10:46 AM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >> >> >>>
> > >> >> >> >>> Hi Ajit,
> > >> >> >> >>>
> > >> >> >> >>> I'm just checking in; have you heard of any updates on this issue?
> > >> >> >> >>>
> > >> >> >> >>> Thanks,
> > >> >> >> >>> Brandon
> > >> >> >> >>>
> > >> >> >> >>> On Tue, Aug 4, 2020 at 1:42 PM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >> >> >>>>
> > >> >> >> >>>> Hi Ajit,
> > >> >> >> >>>>
> > >> >> >> >>>> Yes, I believe the issue is coming from the trex/tester system with the 100G NIC.
> > >> >> >> >>>> I'm not sure what causes this issue; if I run trex using the command "cd /opt/v2.82;./t-rex-64 -i --cfg /etc/trex_cfg_100g.yaml -c 7", which is the same command used in DTS, it seems to launch without failing.
> > >> >> >> >>>>
> > >> >> >> >>>> If you want to replicate it, here are the steps that I ran:
> > >> >> >> >>>>
> > >> >> >> >>>> (on io) cd /opt/dts
> > >> >> >> >>>> export DTS_CFG_FOLDER='conf_100g'
> > >> >> >> >>>>
> > >> >> >> >>>> conf_100g has the new configuration files to use the new PCI id and pktgen config file
> > >> >> >> >>>>
> > >> >> >> >>>> ./dts -s
> > >> >> >> >>>>
> > >> >> >> >>>>
> > >> >> >> >>>> Thanks for your help,
> > >> >> >> >>>> Brandon
> > >> >> >> >>>>
> > >> >> >> >>>> On Tue, Aug 4, 2020 at 12:38 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> > >> >> >> >>>>>
> > >> >> >> >>>>> Hi Brandon,
> > >> >> >> >>>>> No, I haven't seen or heard this before.
> > >> >> >> >>>>> But I will try to have someone run it again.
> > >> >> >> >>>>>
> > >> >> >> >>>>> Just to make sure -
> > >> >> >> >>>>> You are running trex on the 100G NIC and the problem is encountered on that setup?
> > >> >> >> >>>>> Or is it the system that is running testpmd where you are running into the issue?
> > >> >> >> >>>>>
> > >> >> >> >>>>> Thanks
> > >> >> >> >>>>> Ajit
> > >> >> >> >>>>>
> > >> >> >> >>>>> On Tue, Aug 4, 2020 at 9:16 AM Brandon Lo <blo@iol.unh.edu> wrote:
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> Hi Ajit,
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> I'm running into a problem with trying to run nic_single_core_perf on the new NIC.
> > >> >> >> >>>>>> The current configuration uses trex version v2.82.
> > >> >> >> >>>>>> However, I'm running into an error when it tries to actually do a test case in the nic_single_core_perf.
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> The output looks like this when it reaches a test case:
> > >> >> >> >>>>>>
> > >> >> >> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
> > >> >> >> >>>>>>>                       dut.rhea: ./x86_64-native-linuxapp-gcc/app/testpmd -l 16,17 -n 4 -w 0000:81:00.0 -w 0000:81:00.1  --file-prefix=dpdk_11307_20200804160513    -- -i  --portmask=0x3 --txd=512 --rxd=512
> > >> >> >> >>>>>>>                       dut.rhea: start
> > >> >> >> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> > >> >> >> >>>>>>>   File "/opt/dts/framework/test_case.py", line 316, in _execute_test_case
> > >> >> >> >>>>>>>     case_obj()
> > >> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 198, in test_perf_nic_single_core
> > >> >> >> >>>>>>>     self.perf_test(self.nb_ports)
> > >> >> >> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 259, in perf_test
> > >> >> >> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
> > >> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> > >> >> >> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> > >> >> >> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 779, in _prepare_transmission
> > >> >> >> >>>>>>>     self._conn.reset(ports=self._ports)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> > >> >> >> >>>>>>>     ret = f(*args, **kwargs)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
> > >> >> >> >>>>>>>     self.clear_stats(ports)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> > >> >> >> >>>>>>>     ret = f(*args, **kwargs)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1467, in clear_stats
> > >> >> >> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> > >> >> >> >>>>>>>   File "/opt/v2.82/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2840, in _clear_stats_common
> > >> >> >> >>>>>>>     raise TRexError(rc)
> > >> >> >> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://127.0.0.1:4501
> > >> >> >> >>>>>>
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> I have found one similar case on the github repository for trex, but the solution was vendor-specific: https://github.com/cisco-system-traffic-generator/trex-core/issues/147.
> > >> >> >> >>>>>> Have you ran into this issue before?
> > >> >> >> >>>>>>
> > >> >> >> >>>>>> Thanks,
> > >> >> >> >>>>>> Brandon
> > >> >> >> >>>>
> > >> >> >> >>>>
> > >> >> >> >>>>
> > >> >> >> >>>> --
> > >> >> >> >>>>
> > >> >> >> >>>> Brandon Lo
> > >> >> >> >>>>
> > >> >> >> >>>> UNH InterOperability Laboratory
> > >> >> >> >>>>
> > >> >> >> >>>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >> >> >> >>>>
> > >> >> >> >>>> blo@iol.unh.edu
> > >> >> >> >>>>
> > >> >> >> >>>> www.iol.unh.edu
> > >> >> >> >>>
> > >> >> >> >>>
> > >> >> >> >>>
> > >> >> >> >>> --
> > >> >> >> >>>
> > >> >> >> >>> Brandon Lo
> > >> >> >> >>>
> > >> >> >> >>> UNH InterOperability Laboratory
> > >> >> >> >>>
> > >> >> >> >>> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >> >> >> >>>
> > >> >> >> >>> blo@iol.unh.edu
> > >> >> >> >>>
> > >> >> >> >>> www.iol.unh.edu
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> --
> > >> >> >>
> > >> >> >> Brandon Lo
> > >> >> >>
> > >> >> >> UNH InterOperability Laboratory
> > >> >> >>
> > >> >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >> >> >>
> > >> >> >> blo@iol.unh.edu
> > >> >> >>
> > >> >> >> www.iol.unh.edu
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >>
> > >> >> Brandon Lo
> > >> >>
> > >> >> UNH InterOperability Laboratory
> > >> >>
> > >> >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >> >>
> > >> >> blo@iol.unh.edu
> > >> >>
> > >> >> www.iol.unh.edu
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Brandon Lo
> > >>
> > >> UNH InterOperability Laboratory
> > >>
> > >> 21 Madbury Rd, Suite 100, Durham, NH 03824
> > >>
> > >> blo@iol.unh.edu
> > >>
> > >> www.iol.unh.edu
> >
> >
> >
> > --
> >
> > Brandon Lo
> >
> > UNH InterOperability Laboratory
> >
> > 21 Madbury Rd, Suite 100, Durham, NH 03824
> >
> > blo@iol.unh.edu
> >
> > www.iol.unh.edu



-- 

Brandon Lo

UNH InterOperability Laboratory

21 Madbury Rd, Suite 100, Durham, NH 03824

blo@iol.unh.edu

www.iol.unh.edu

^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2020-12-03 16:49 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-21 20:22 [dpdk-ci] New 100G Broadcom NIC Ubuntu Brandon Lo
2020-07-21 21:27 ` Ajit Khaparde
2020-07-28 16:47   ` Brandon Lo
2020-07-28 17:22     ` Ajit Khaparde
2020-07-28 17:32       ` Brandon Lo
2020-07-28 19:19         ` Ajit Khaparde
2020-07-28 19:41           ` Brandon Lo
2020-08-04 16:16             ` Brandon Lo
2020-08-04 16:37               ` Ajit Khaparde
2020-08-04 17:42                 ` Brandon Lo
2020-08-11 17:45                   ` Brandon Lo
2020-08-11 17:47                     ` Ajit Khaparde
2020-08-20  4:29                       ` Ajit Khaparde
2020-08-20 15:39                         ` Brandon Lo
2020-08-20 21:07                           ` Ajit Khaparde
2020-08-25 20:13                             ` Brandon Lo
2020-08-26 19:57                               ` Ajit Khaparde
2020-09-01 18:09                                 ` Brandon Lo
2020-09-01 18:11                                   ` Ajit Khaparde
2020-10-22 19:03                                     ` Brandon Lo
2020-10-22 19:27                                       ` Ajit Khaparde
2020-12-03 16:48                                         ` [dpdk-ci] [dpdklab] " Brandon Lo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).