DPDK CI discussions
 help / color / mirror / Atom feed
* [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
@ 2021-01-13 20:07 David Liu
  2021-01-13 22:34 ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-01-13 20:07 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 4686 bytes --]

Hi Ajit,

We have a 25G Broadcom NIC that is installed on an Arm machine.
And I am running into a problem when I try to run nic_single_core_perf on
the NIC.
Currently, we are using: Trex v2.86 and dts from
http://git.dpdk.org/tools/dts/
All the NICs are up and running when testing.

The problem is an error when I try to run the test case inside
the nic_single_core_perf:

TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd:
> 512
>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>                dut.172.18.0.41: start
>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result
> ERROR: Traceback (most recent call last):
>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>     case_obj()
>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
> test_perf_nic_single_core
>     self.perf_test(self.nb_ports)
>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>     self._prepare_transmission(stream_ids=stream_ids)
>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
> _prepare_transmission
>     self._conn.reset(ports=self._ports)
>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
>     ret = f(*args, **kwargs)
>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
>     self.clear_stats(ports)
>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
>     ret = f(*args, **kwargs)
>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
>     self._clear_stats_common(ports, clear_global, clear_xstats)
>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
>     raise TRexError(rc)
> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
> response from tcp://172.18.0.40:4501


I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
/etc/trex_cfg.yaml and  connect with ./trex-console

Then start sending traffic with command

> trex>start -f stl/imix.py


But this will cause an error to turn off Trex.

trex>
> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown -
> cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than
> 1.00135 seconds - timeout is 1 seconds
> *** traceback follows ***
> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)
> [0x7feea3ac0980]
> 3       0x561be1a05a2a rte_delay_us_block + 106
> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned
> short, CVirtualIFPerSideStats*) + 3220
> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*,
> CFlowGenListPerThread*, bool&) + 92
> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char,
> CGenNode*, CFlowGenListPerThread*, bool) + 184
> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23,
> false>(double, double, CFlowGenListPerThread*, double&) + 881
> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
> 11      0x561be1883ae9 TrexDpCore::start() + 89
> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> 15      0x561be1a1dfaa eal_thread_loop + 426
> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)
> [0x7feea3ab56db]
> 17      0x7feea2a8571f clone + 63
>
> *** addr2line information follows ***
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> ??:0
> '


I believe if everything is working, this should not turn off trex but
please correct me if I am wrong.
I wonder if you have any suggestion of how I can approach this issue?

Best,
David

[-- Attachment #2: Type: text/html, Size: 14428 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-13 20:07 [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware David Liu
@ 2021-01-13 22:34 ` Ajit Khaparde
  2021-01-14  0:19   ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-01-13 22:34 UTC (permalink / raw)
  To: David Liu; +Cc: ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 5086 bytes --]

Hi David,
I will take a look.
Do you see similar issues on x86? I am asking because
even I would start with that to create a baseline and then
attempt on ARM.

Thanks
Ajit

On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit,
>
> We have a 25G Broadcom NIC that is installed on an Arm machine.
> And I am running into a problem when I try to run nic_single_core_perf on
> the NIC.
> Currently, we are using: Trex v2.86 and dts from
> http://git.dpdk.org/tools/dts/
> All the NICs are up and running when testing.
>
> The problem is an error when I try to run the test case inside
> the nic_single_core_perf:
>
> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd:
>> 512
>>                dut.172.18.0.41:
>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>                dut.172.18.0.41: start
>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>> Result ERROR: Traceback (most recent call last):
>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>     case_obj()
>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
>> test_perf_nic_single_core
>>     self.perf_test(self.nb_ports)
>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>     _, packets_received =
>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>> options=traffic_opt)
>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>> measure_throughput
>>     self._prepare_transmission(stream_ids=stream_ids)
>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>> _prepare_transmission
>>     self._conn.reset(ports=self._ports)
>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>>     ret = f(*args, **kwargs)
>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 339, in reset
>>     self.clear_stats(ports)
>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>>     ret = f(*args, **kwargs)
>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 1492, in clear_stats
>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>> line 2876, in _clear_stats_common
>>     raise TRexError(rc)
>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
>> response from tcp://172.18.0.40:4501
>
>
> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
> /etc/trex_cfg.yaml and  connect with ./trex-console
>
> Then start sending traffic with command
>
>> trex>start -f stl/imix.py
>
>
> But this will cause an error to turn off Trex.
>
> trex>
>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown -
>> cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than
>> 1.00135 seconds - timeout is 1 seconds
>> *** traceback follows ***
>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)
>> [0x7feea3ac0980]
>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned
>> short, CVirtualIFPerSideStats*) + 3220
>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> 6       0x561be171e3d9
>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*,
>> CFlowGenListPerThread*, bool&) + 92
>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned
>> char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23,
>> false>(double, double, CFlowGenListPerThread*, double&) + 881
>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> 12      0x561be1714113
>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)
>> [0x7feea3ab56db]
>> 17      0x7feea2a8571f clone + 63
>>
>> *** addr2line information follows ***
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> ??:0
>> '
>
>
> I believe if everything is working, this should not turn off trex but
> please correct me if I am wrong.
> I wonder if you have any suggestion of how I can approach this issue?
>
> Best,
> David
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 15677 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-13 22:34 ` Ajit Khaparde
@ 2021-01-14  0:19   ` David Liu
  2021-01-14  6:11     ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-01-14  0:19 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 5423 bytes --]

Hi Ajit,

Thank you for helping out.

We have it working on the x86, but not the Arm.


Thanks,
David

On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi David,
> I will take a look.
> Do you see similar issues on x86? I am asking because
> even I would start with that to create a baseline and then
> attempt on ARM.
>
> Thanks
> Ajit
>
> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>> And I am running into a problem when I try to run nic_single_core_perf on
>> the NIC.
>> Currently, we are using: Trex v2.86 and dts from
>> http://git.dpdk.org/tools/dts/
>> All the NICs are up and running when testing.
>>
>> The problem is an error when I try to run the test case inside
>> the nic_single_core_perf:
>>
>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>>> rxd/txd: 512
>>>                dut.172.18.0.41:
>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>                dut.172.18.0.41: start
>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>>> Result ERROR: Traceback (most recent call last):
>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>>     case_obj()
>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
>>> test_perf_nic_single_core
>>>     self.perf_test(self.nb_ports)
>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>     _, packets_received =
>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>> options=traffic_opt)
>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>> measure_throughput
>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>>> _prepare_transmission
>>>     self._conn.reset(ports=self._ports)
>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>>     ret = f(*args, **kwargs)
>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 339, in reset
>>>     self.clear_stats(ports)
>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>>     ret = f(*args, **kwargs)
>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 1492, in clear_stats
>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>> line 2876, in _clear_stats_common
>>>     raise TRexError(rc)
>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
>>> response from tcp://172.18.0.40:4501
>>
>>
>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
>> /etc/trex_cfg.yaml and  connect with ./trex-console
>>
>> Then start sending traffic with command
>>
>>> trex>start -f stl/imix.py
>>
>>
>> But this will cause an error to turn off Trex.
>>
>> trex>
>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown -
>>> cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than
>>> 1.00135 seconds - timeout is 1 seconds
>>> *** traceback follows ***
>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)
>>> [0x7feea3ac0980]
>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned
>>> short, CVirtualIFPerSideStats*) + 3220
>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>> 6       0x561be171e3d9
>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*,
>>> CFlowGenListPerThread*, bool&) + 92
>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned
>>> char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23,
>>> false>(double, double, CFlowGenListPerThread*, double&) + 881
>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>> 12      0x561be1714113
>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)
>>> [0x7feea3ab56db]
>>> 17      0x7feea2a8571f clone + 63
>>>
>>> *** addr2line information follows ***
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> ??:0
>>> '
>>
>>
>> I believe if everything is working, this should not turn off trex but
>> please correct me if I am wrong.
>> I wonder if you have any suggestion of how I can approach this issue?
>>
>> Best,
>> David
>>
>>
>>

[-- Attachment #2: Type: text/html, Size: 16261 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-14  0:19   ` David Liu
@ 2021-01-14  6:11     ` Ajit Khaparde
  2021-01-14 19:12       ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-01-14  6:11 UTC (permalink / raw)
  To: David Liu; +Cc: ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 5879 bytes --]

Does the same NIC work on x86?
Also do you see link before you start the trex? I am trying to see if the
port failed to successfully come up
or if there is a problem with the link. Can you check and tell?

Thanks
Ajit

On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit,
>
> Thank you for helping out.
>
> We have it working on the x86, but not the Arm.
>
>
> Thanks,
> David
>
> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Hi David,
>> I will take a look.
>> Do you see similar issues on x86? I am asking because
>> even I would start with that to create a baseline and then
>> attempt on ARM.
>>
>> Thanks
>> Ajit
>>
>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>>
>>> Hi Ajit,
>>>
>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>> And I am running into a problem when I try to run nic_single_core_perf
>>> on the NIC.
>>> Currently, we are using: Trex v2.86 and dts from
>>> http://git.dpdk.org/tools/dts/
>>> All the NICs are up and running when testing.
>>>
>>> The problem is an error when I try to run the test case inside
>>> the nic_single_core_perf:
>>>
>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>>>> rxd/txd: 512
>>>>                dut.172.18.0.41:
>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>                dut.172.18.0.41: start
>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>>>> Result ERROR: Traceback (most recent call last):
>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>>> _execute_test_case
>>>>     case_obj()
>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
>>>> test_perf_nic_single_core
>>>>     self.perf_test(self.nb_ports)
>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>>     _, packets_received =
>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>> options=traffic_opt)
>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>>> measure_throughput
>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>>>> _prepare_transmission
>>>>     self._conn.reset(ports=self._ports)
>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>> line 51, in wrap2
>>>>     ret = f(*args, **kwargs)
>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>> line 339, in reset
>>>>     self.clear_stats(ports)
>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>> line 51, in wrap2
>>>>     ret = f(*args, **kwargs)
>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>> line 1492, in clear_stats
>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>> line 2876, in _clear_stats_common
>>>>     raise TRexError(rc)
>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
>>>> response from tcp://172.18.0.40:4501
>>>
>>>
>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
>>> /etc/trex_cfg.yaml and  connect with ./trex-console
>>>
>>> Then start sending traffic with command
>>>
>>>> trex>start -f stl/imix.py
>>>
>>>
>>> But this will cause an error to turn off Trex.
>>>
>>> trex>
>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown -
>>>> cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than
>>>> 1.00135 seconds - timeout is 1 seconds
>>>> *** traceback follows ***
>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)
>>>> [0x7feea3ac0980]
>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned
>>>> short, CVirtualIFPerSideStats*) + 3220
>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>> 6       0x561be171e3d9
>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*,
>>>> CFlowGenListPerThread*, bool&) + 92
>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned
>>>> char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23,
>>>> false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>> 12      0x561be1714113
>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)
>>>> [0x7feea3ab56db]
>>>> 17      0x7feea2a8571f clone + 63
>>>>
>>>> *** addr2line information follows ***
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> ??:0
>>>> '
>>>
>>>
>>> I believe if everything is working, this should not turn off trex but
>>> please correct me if I am wrong.
>>> I wonder if you have any suggestion of how I can approach this issue?
>>>
>>> Best,
>>> David
>>>
>>>
>>>

[-- Attachment #1.2: Type: text/html, Size: 17266 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-14  6:11     ` Ajit Khaparde
@ 2021-01-14 19:12       ` David Liu
  2021-01-19 21:40         ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-01-14 19:12 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 6321 bytes --]

Yes, so this is the same NIC work on x86.
By seeing the link, I am assuming you mean the ports that are using for
Trex. If so they are UP before I start the Trex.

Thank,
David


On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Does the same NIC work on x86?
> Also do you see link before you start the trex? I am trying to see if the
> port failed to successfully come up
> or if there is a problem with the link. Can you check and tell?
>
> Thanks
> Ajit
>
> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> Thank you for helping out.
>>
>> We have it working on the x86, but not the Arm.
>>
>>
>> Thanks,
>> David
>>
>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>>> Hi David,
>>> I will take a look.
>>> Do you see similar issues on x86? I am asking because
>>> even I would start with that to create a baseline and then
>>> attempt on ARM.
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>>>
>>>> Hi Ajit,
>>>>
>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>> And I am running into a problem when I try to run nic_single_core_perf
>>>> on the NIC.
>>>> Currently, we are using: Trex v2.86 and dts from
>>>> http://git.dpdk.org/tools/dts/
>>>> All the NICs are up and running when testing.
>>>>
>>>> The problem is an error when I try to run the test case inside
>>>> the nic_single_core_perf:
>>>>
>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>>>>> rxd/txd: 512
>>>>>                dut.172.18.0.41:
>>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>>  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>>                dut.172.18.0.41: start
>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>>>>> Result ERROR: Traceback (most recent call last):
>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>>>> _execute_test_case
>>>>>     case_obj()
>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
>>>>> test_perf_nic_single_core
>>>>>     self.perf_test(self.nb_ports)
>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in
>>>>> perf_test
>>>>>     _, packets_received =
>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>> options=traffic_opt)
>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>>>> measure_throughput
>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>>>>> _prepare_transmission
>>>>>     self._conn.reset(ports=self._ports)
>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>> line 51, in wrap2
>>>>>     ret = f(*args, **kwargs)
>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>> line 339, in reset
>>>>>     self.clear_stats(ports)
>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>> line 51, in wrap2
>>>>>     ret = f(*args, **kwargs)
>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>> line 1492, in clear_stats
>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>> line 2876, in _clear_stats_common
>>>>>     raise TRexError(rc)
>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get
>>>>> server response from tcp://172.18.0.40:4501
>>>>
>>>>
>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
>>>> /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>
>>>> Then start sending traffic with command
>>>>
>>>>> trex>start -f stl/imix.py
>>>>
>>>>
>>>> But this will cause an error to turn off Trex.
>>>>
>>>> trex>
>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown -
>>>>> cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than
>>>>> 1.00135 seconds - timeout is 1 seconds
>>>>> *** traceback follows ***
>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)
>>>>> [0x7feea3ac0980]
>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned
>>>>> short, CVirtualIFPerSideStats*) + 3220
>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>>> 6       0x561be171e3d9
>>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*,
>>>>> CFlowGenListPerThread*, bool&) + 92
>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned
>>>>> char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23,
>>>>> false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>> 12      0x561be1714113
>>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)
>>>>> [0x7feea3ab56db]
>>>>> 17      0x7feea2a8571f clone + 63
>>>>>
>>>>> *** addr2line information follows ***
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> ??:0
>>>>> '
>>>>
>>>>
>>>> I believe if everything is working, this should not turn off trex but
>>>> please correct me if I am wrong.
>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>>
>>>> Best,
>>>> David
>>>>
>>>>
>>>>

[-- Attachment #2: Type: text/html, Size: 17906 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-14 19:12       ` David Liu
@ 2021-01-19 21:40         ` David Liu
  2021-01-19 22:19           ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-01-19 21:40 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 6816 bytes --]

Hi Ajit,

Thank you for helping out.

How are you doing with this issue?

Please let me know if you need anything information from us.

Just some extra information that both Trex and dts are on the same
intel tester machine.

Best,
David

On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:

> Yes, so this is the same NIC work on x86.
> By seeing the link, I am assuming you mean the ports that are using for
> Trex. If so they are UP before I start the Trex.
>
> Thank,
> David
>
>
> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Does the same NIC work on x86?
>> Also do you see link before you start the trex? I am trying to see if the
>> port failed to successfully come up
>> or if there is a problem with the link. Can you check and tell?
>>
>> Thanks
>> Ajit
>>
>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>>
>>> Hi Ajit,
>>>
>>> Thank you for helping out.
>>>
>>> We have it working on the x86, but not the Arm.
>>>
>>>
>>> Thanks,
>>> David
>>>
>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>>
>>>> Hi David,
>>>> I will take a look.
>>>> Do you see similar issues on x86? I am asking because
>>>> even I would start with that to create a baseline and then
>>>> attempt on ARM.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>
>>>>> Hi Ajit,
>>>>>
>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>>> And I am running into a problem when I try to run nic_single_core_perf
>>>>> on the NIC.
>>>>> Currently, we are using: Trex v2.86 and dts from
>>>>> http://git.dpdk.org/tools/dts/
>>>>> All the NICs are up and running when testing.
>>>>>
>>>>> The problem is an error when I try to run the test case inside
>>>>> the nic_single_core_perf:
>>>>>
>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>>>>>> rxd/txd: 512
>>>>>>                dut.172.18.0.41:
>>>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>>>  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>>>                dut.172.18.0.41: start
>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core
>>>>>> Result ERROR: Traceback (most recent call last):
>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>>>>> _execute_test_case
>>>>>>     case_obj()
>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
>>>>>> test_perf_nic_single_core
>>>>>>     self.perf_test(self.nb_ports)
>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in
>>>>>> perf_test
>>>>>>     _, packets_received =
>>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>>> options=traffic_opt)
>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>>>>> measure_throughput
>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>>>>>> _prepare_transmission
>>>>>>     self._conn.reset(ports=self._ports)
>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>> line 51, in wrap2
>>>>>>     ret = f(*args, **kwargs)
>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>> line 339, in reset
>>>>>>     self.clear_stats(ports)
>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>> line 51, in wrap2
>>>>>>     ret = f(*args, **kwargs)
>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>> line 1492, in clear_stats
>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>>> line 2876, in _clear_stats_common
>>>>>>     raise TRexError(rc)
>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get
>>>>>> server response from tcp://172.18.0.40:4501
>>>>>
>>>>>
>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
>>>>> /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>>
>>>>> Then start sending traffic with command
>>>>>
>>>>>> trex>start -f stl/imix.py
>>>>>
>>>>>
>>>>> But this will cause an error to turn off Trex.
>>>>>
>>>>> trex>
>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown -
>>>>>> cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than
>>>>>> 1.00135 seconds - timeout is 1 seconds
>>>>>> *** traceback follows ***
>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>>>> 2       0x7feea3ac0980
>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned
>>>>>> short, CVirtualIFPerSideStats*) + 3220
>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>>>> 6       0x561be171e3d9
>>>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*,
>>>>>> CFlowGenListPerThread*, bool&) + 92
>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned
>>>>>> char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23,
>>>>>> false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>>> 12      0x561be1714113
>>>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)
>>>>>> [0x7feea3ab56db]
>>>>>> 17      0x7feea2a8571f clone + 63
>>>>>>
>>>>>> *** addr2line information follows ***
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> ??:0
>>>>>> '
>>>>>
>>>>>
>>>>> I believe if everything is working, this should not turn off trex but
>>>>> please correct me if I am wrong.
>>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>>>
>>>>> Best,
>>>>> David
>>>>>
>>>>>
>>>>>

[-- Attachment #2: Type: text/html, Size: 18638 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-19 21:40         ` David Liu
@ 2021-01-19 22:19           ` Ajit Khaparde
  2021-01-26 21:54             ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-01-19 22:19 UTC (permalink / raw)
  To: David Liu; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 7085 bytes --]

Hi David,
Thanks for pinging me. Do you think I could login into the setup and
see what is happening?
I do have the credentials for two systems in your lab -
rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
If you can provide me details of the systems you are trying to
configure, I could take a look.

Thanks
Ajit

On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
> Thank you for helping out.
>
> How are you doing with this issue?
>
> Please let me know if you need anything information from us.
>
> Just some extra information that both Trex and dts are on the same intel tester machine.
>
> Best,
> David
>
> On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>>
>> Yes, so this is the same NIC work on x86.
>> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>>
>> Thank,
>> David
>>
>>
>> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>
>>> Does the same NIC work on x86?
>>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>>> or if there is a problem with the link. Can you check and tell?
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>
>>>> Hi Ajit,
>>>>
>>>> Thank you for helping out.
>>>>
>>>> We have it working on the x86, but not the Arm.
>>>>
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>>
>>>>> Hi David,
>>>>> I will take a look.
>>>>> Do you see similar issues on x86? I am asking because
>>>>> even I would start with that to create a baseline and then
>>>>> attempt on ARM.
>>>>>
>>>>> Thanks
>>>>> Ajit
>>>>>
>>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>>>
>>>>>> Hi Ajit,
>>>>>>
>>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>>>>>> All the NICs are up and running when testing.
>>>>>>
>>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>>>>>>
>>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>>>>                dut.172.18.0.41: start
>>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>>>>>>     case_obj()
>>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>>>>>>>     self.perf_test(self.nb_ports)
>>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>>>>>>     self._conn.reset(ports=self._ports)
>>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>>>>>     ret = f(*args, **kwargs)
>>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>>>>>>>     self.clear_stats(ports)
>>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>>>>>     ret = f(*args, **kwargs)
>>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>>>>>>>     raise TRexError(rc)
>>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>>>>>>
>>>>>>
>>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>>>
>>>>>> Then start sending traffic with command
>>>>>>>
>>>>>>> trex>start -f stl/imix.py
>>>>>>
>>>>>>
>>>>>> But this will cause an error to turn off Trex.
>>>>>>
>>>>>>> trex>
>>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>>>>>>> *** traceback follows ***
>>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>>>>> 17      0x7feea2a8571f clone + 63
>>>>>>>
>>>>>>> *** addr2line information follows ***
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> ??:0
>>>>>>> '
>>>>>>
>>>>>>
>>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>>>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>>>>
>>>>>> Best,
>>>>>> David
>>>>>>
>>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-19 22:19           ` Ajit Khaparde
@ 2021-01-26 21:54             ` David Liu
  2021-01-26 21:59               ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-01-26 21:54 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 7919 bytes --]

Hi Ajit,
Sorry for the late reply.

You can ssh into the machine using the same VPN Username and Password.
Can you try and see if that works?

Informatio:
arm-dell.dpdklab.iol.unh.edu
- dts - located in /opt/dts
- Trex - located in /opt/trex/v2.87

arm-gigabyte.dpdklab.iol.unh.edu
- dpdk - located in /root/dpdk


Thanks,
David

On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi David,
> Thanks for pinging me. Do you think I could login into the setup and
> see what is happening?
> I do have the credentials for two systems in your lab -
> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> If you can provide me details of the systems you are trying to
> configure, I could take a look.
>
> Thanks
> Ajit
>
> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > Thank you for helping out.
> >
> > How are you doing with this issue?
> >
> > Please let me know if you need anything information from us.
> >
> > Just some extra information that both Trex and dts are on the same intel
> tester machine.
> >
> > Best,
> > David
> >
> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
> >>
> >> Yes, so this is the same NIC work on x86.
> >> By seeing the link, I am assuming you mean the ports that are using for
> Trex. If so they are UP before I start the Trex.
> >>
> >> Thank,
> >> David
> >>
> >>
> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>
> >>> Does the same NIC work on x86?
> >>> Also do you see link before you start the trex? I am trying to see if
> the port failed to successfully come up
> >>> or if there is a problem with the link. Can you check and tell?
> >>>
> >>> Thanks
> >>> Ajit
> >>>
> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>
> >>>> Hi Ajit,
> >>>>
> >>>> Thank you for helping out.
> >>>>
> >>>> We have it working on the x86, but not the Arm.
> >>>>
> >>>>
> >>>> Thanks,
> >>>> David
> >>>>
> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>>>
> >>>>> Hi David,
> >>>>> I will take a look.
> >>>>> Do you see similar issues on x86? I am asking because
> >>>>> even I would start with that to create a baseline and then
> >>>>> attempt on ARM.
> >>>>>
> >>>>> Thanks
> >>>>> Ajit
> >>>>>
> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>>>
> >>>>>> Hi Ajit,
> >>>>>>
> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
> >>>>>> And I am running into a problem when I try to run
> nic_single_core_perf on the NIC.
> >>>>>> Currently, we are using: Trex v2.86 and dts from
> http://git.dpdk.org/tools/dts/
> >>>>>> All the NICs are up and running when testing.
> >>>>>>
> >>>>>> The problem is an error when I try to run the test case inside the
> nic_single_core_perf:
> >>>>>>
> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
> rxd/txd: 512
> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >>>>>>>                dut.172.18.0.41: start
> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
> _execute_test_case
> >>>>>>>     case_obj()
> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
> test_perf_nic_single_core
> >>>>>>>     self.perf_test(self.nb_ports)
> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in
> perf_test
> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
> measure_throughput
> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
> _prepare_transmission
> >>>>>>>     self._conn.reset(ports=self._ports)
> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>>>>>     ret = f(*args, **kwargs)
> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >>>>>>>     self.clear_stats(ports)
> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>>>>>     ret = f(*args, **kwargs)
> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >>>>>>>     raise TRexError(rc)
> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get
> server response from tcp://172.18.0.40:4501
> >>>>>>
> >>>>>>
> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
> /etc/trex_cfg.yaml and  connect with ./trex-console
> >>>>>>
> >>>>>> Then start sending traffic with command
> >>>>>>>
> >>>>>>> trex>start -f stl/imix.py
> >>>>>>
> >>>>>>
> >>>>>> But this will cause an error to turn off Trex.
> >>>>>>
> >>>>>>> trex>
> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been
> shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for
> more than 1.00135 seconds - timeout is 1 seconds
> >>>>>>> *** traceback follows ***
> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*,
> unsigned short, CVirtualIFPerSideStats*) + 3220
> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*,
> CFlowGenListPerThread*, bool&) + 92
> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23,
> false>(double, double, CFlowGenListPerThread*, double&) + 881
> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) +
> 487
> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >>>>>>> 17      0x7feea2a8571f clone + 63
> >>>>>>>
> >>>>>>> *** addr2line information follows ***
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> ??:0
> >>>>>>> '
> >>>>>>
> >>>>>>
> >>>>>> I believe if everything is working, this should not turn off trex
> but please correct me if I am wrong.
> >>>>>> I wonder if you have any suggestion of how I can approach this
> issue?
> >>>>>>
> >>>>>> Best,
> >>>>>> David
> >>>>>>
> >>>>>>
>

[-- Attachment #2: Type: text/html, Size: 12324 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-26 21:54             ` David Liu
@ 2021-01-26 21:59               ` David Liu
  2021-02-02 16:58                 ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-01-26 21:59 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 8362 bytes --]

Also, I had changed the configuration when I run Broadcom nic.
In the dts folder, you can start it with ./dts -s

On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit,
> Sorry for the late reply.
>
> You can ssh into the machine using the same VPN Username and Password.
> Can you try and see if that works?
>
> Informatio:
> arm-dell.dpdklab.iol.unh.edu
> - dts - located in /opt/dts
> - Trex - located in /opt/trex/v2.87
>
> arm-gigabyte.dpdklab.iol.unh.edu
> - dpdk - located in /root/dpdk
>
>
> Thanks,
> David
>
> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Hi David,
>> Thanks for pinging me. Do you think I could login into the setup and
>> see what is happening?
>> I do have the credentials for two systems in your lab -
>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>> If you can provide me details of the systems you are trying to
>> configure, I could take a look.
>>
>> Thanks
>> Ajit
>>
>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> > Thank you for helping out.
>> >
>> > How are you doing with this issue?
>> >
>> > Please let me know if you need anything information from us.
>> >
>> > Just some extra information that both Trex and dts are on the same
>> intel tester machine.
>> >
>> > Best,
>> > David
>> >
>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>
>> >> Yes, so this is the same NIC work on x86.
>> >> By seeing the link, I am assuming you mean the ports that are using
>> for Trex. If so they are UP before I start the Trex.
>> >>
>> >> Thank,
>> >> David
>> >>
>> >>
>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>
>> >>> Does the same NIC work on x86?
>> >>> Also do you see link before you start the trex? I am trying to see if
>> the port failed to successfully come up
>> >>> or if there is a problem with the link. Can you check and tell?
>> >>>
>> >>> Thanks
>> >>> Ajit
>> >>>
>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>>
>> >>>> Hi Ajit,
>> >>>>
>> >>>> Thank you for helping out.
>> >>>>
>> >>>> We have it working on the x86, but not the Arm.
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>> David
>> >>>>
>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>>>
>> >>>>> Hi David,
>> >>>>> I will take a look.
>> >>>>> Do you see similar issues on x86? I am asking because
>> >>>>> even I would start with that to create a baseline and then
>> >>>>> attempt on ARM.
>> >>>>>
>> >>>>> Thanks
>> >>>>> Ajit
>> >>>>>
>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >>>>>>
>> >>>>>> Hi Ajit,
>> >>>>>>
>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>> >>>>>> And I am running into a problem when I try to run
>> nic_single_core_perf on the NIC.
>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>> http://git.dpdk.org/tools/dts/
>> >>>>>> All the NICs are up and running when testing.
>> >>>>>>
>> >>>>>> The problem is an error when I try to run the test case inside the
>> nic_single_core_perf:
>> >>>>>>
>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64,
>> rxd/txd: 512
>> >>>>>>>                dut.172.18.0.41:
>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> >>>>>>>                dut.172.18.0.41: start
>> >>>>>>>          TestNicSingleCorePerf: Test Case
>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>> _execute_test_case
>> >>>>>>>     case_obj()
>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
>> test_perf_nic_single_core
>> >>>>>>>     self.perf_test(self.nb_ports)
>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in
>> perf_test
>> >>>>>>>     _, packets_received =
>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>> options=traffic_opt)
>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>> measure_throughput
>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>> _prepare_transmission
>> >>>>>>>     self._conn.reset(ports=self._ports)
>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 339, in reset
>> >>>>>>>     self.clear_stats(ports)
>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 1492, in clear_stats
>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>> line 2876, in _clear_stats_common
>> >>>>>>>     raise TRexError(rc)
>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get
>> server response from tcp://172.18.0.40:4501
>> >>>>>>
>> >>>>>>
>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
>> /etc/trex_cfg.yaml and  connect with ./trex-console
>> >>>>>>
>> >>>>>> Then start sending traffic with command
>> >>>>>>>
>> >>>>>>> trex>start -f stl/imix.py
>> >>>>>>
>> >>>>>>
>> >>>>>> But this will cause an error to turn off Trex.
>> >>>>>>
>> >>>>>>> trex>
>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been
>> shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for
>> more than 1.00135 seconds - timeout is 1 seconds
>> >>>>>>> *** traceback follows ***
>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> >>>>>>> 2       0x7feea3ac0980
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*,
>> unsigned short, CVirtualIFPerSideStats*) + 3220
>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> >>>>>>> 6       0x561be171e3d9
>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> >>>>>>> 7       0x561be171f7ec
>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>> + 92
>> >>>>>>> 8       0x561be171fc98
>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>> CFlowGenListPerThread*, bool) + 184
>> >>>>>>> 9       0x561be16cb5f1 int
>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>> CFlowGenListPerThread*, double&) + 881
>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() +
>> 226
>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> >>>>>>> 12      0x561be1714113
>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) +
>> 487
>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> >>>>>>> 16      0x7feea3ab56db
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> >>>>>>>
>> >>>>>>> *** addr2line information follows ***
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> ??:0
>> >>>>>>> '
>> >>>>>>
>> >>>>>>
>> >>>>>> I believe if everything is working, this should not turn off trex
>> but please correct me if I am wrong.
>> >>>>>> I wonder if you have any suggestion of how I can approach this
>> issue?
>> >>>>>>
>> >>>>>> Best,
>> >>>>>> David
>> >>>>>>
>> >>>>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 12795 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-01-26 21:59               ` David Liu
@ 2021-02-02 16:58                 ` David Liu
  2021-02-02 18:57                   ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-02-02 16:58 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 9383 bytes --]

Hi Ajit,

Ping again in case if you miss the last email.

Did you have a chance to give it a try on login to the machines?

Just some notes, we are currently running compile and some
performance/functional testing.
Which it may be hard for you to run dts, so please let me know if you want
me to stop these tests for you to debug.

Step I run:
CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the
dts configuration to Broadcom
./dts -s                                                           #To
start the dts

Informatio:
arm-dell.dpdklab.iol.unh.edu
- dts - located in /opt/dts
- Trex - located in /opt/trex/v2.87

arm-gigabyte.dpdklab.iol.unh.edu
- dpdk - located in /root/dpdk

Best,
David

On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:

> Also, I had changed the configuration when I run Broadcom nic.
> In the dts folder, you can start it with ./dts -s
>
> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>
>> Hi Ajit,
>> Sorry for the late reply.
>>
>> You can ssh into the machine using the same VPN Username and Password.
>> Can you try and see if that works?
>>
>> Informatio:
>> arm-dell.dpdklab.iol.unh.edu
>> - dts - located in /opt/dts
>> - Trex - located in /opt/trex/v2.87
>>
>> arm-gigabyte.dpdklab.iol.unh.edu
>> - dpdk - located in /root/dpdk
>>
>>
>> Thanks,
>> David
>>
>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>>> Hi David,
>>> Thanks for pinging me. Do you think I could login into the setup and
>>> see what is happening?
>>> I do have the credentials for two systems in your lab -
>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>> If you can provide me details of the systems you are trying to
>>> configure, I could take a look.
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>>> >
>>> > Hi Ajit,
>>> >
>>> > Thank you for helping out.
>>> >
>>> > How are you doing with this issue?
>>> >
>>> > Please let me know if you need anything information from us.
>>> >
>>> > Just some extra information that both Trex and dts are on the same
>>> intel tester machine.
>>> >
>>> > Best,
>>> > David
>>> >
>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>>> >>
>>> >> Yes, so this is the same NIC work on x86.
>>> >> By seeing the link, I am assuming you mean the ports that are using
>>> for Trex. If so they are UP before I start the Trex.
>>> >>
>>> >> Thank,
>>> >> David
>>> >>
>>> >>
>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>
>>> >>> Does the same NIC work on x86?
>>> >>> Also do you see link before you start the trex? I am trying to see
>>> if the port failed to successfully come up
>>> >>> or if there is a problem with the link. Can you check and tell?
>>> >>>
>>> >>> Thanks
>>> >>> Ajit
>>> >>>
>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>>> >>>>
>>> >>>> Hi Ajit,
>>> >>>>
>>> >>>> Thank you for helping out.
>>> >>>>
>>> >>>> We have it working on the x86, but not the Arm.
>>> >>>>
>>> >>>>
>>> >>>> Thanks,
>>> >>>> David
>>> >>>>
>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>>>
>>> >>>>> Hi David,
>>> >>>>> I will take a look.
>>> >>>>> Do you see similar issues on x86? I am asking because
>>> >>>>> even I would start with that to create a baseline and then
>>> >>>>> attempt on ARM.
>>> >>>>>
>>> >>>>> Thanks
>>> >>>>> Ajit
>>> >>>>>
>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >>>>>>
>>> >>>>>> Hi Ajit,
>>> >>>>>>
>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>> >>>>>> And I am running into a problem when I try to run
>>> nic_single_core_perf on the NIC.
>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>> http://git.dpdk.org/tools/dts/
>>> >>>>>> All the NICs are up and running when testing.
>>> >>>>>>
>>> >>>>>> The problem is an error when I try to run the test case inside
>>> the nic_single_core_perf:
>>> >>>>>>
>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize:
>>> 64, rxd/txd: 512
>>> >>>>>>>                dut.172.18.0.41:
>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>> >>>>>>>                dut.172.18.0.41: start
>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>> _execute_test_case
>>> >>>>>>>     case_obj()
>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in
>>> test_perf_nic_single_core
>>> >>>>>>>     self.perf_test(self.nb_ports)
>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in
>>> perf_test
>>> >>>>>>>     _, packets_received =
>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>> options=traffic_opt)
>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>> measure_throughput
>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>>> _prepare_transmission
>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 339, in reset
>>> >>>>>>>     self.clear_stats(ports)
>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 1492, in clear_stats
>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>> line 2876, in _clear_stats_common
>>> >>>>>>>     raise TRexError(rc)
>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get
>>> server response from tcp://172.18.0.40:4501
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg
>>> /etc/trex_cfg.yaml and  connect with ./trex-console
>>> >>>>>>
>>> >>>>>> Then start sending traffic with command
>>> >>>>>>>
>>> >>>>>>> trex>start -f stl/imix.py
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> But this will cause an error to turn off Trex.
>>> >>>>>>
>>> >>>>>>> trex>
>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been
>>> shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for
>>> more than 1.00135 seconds - timeout is 1 seconds
>>> >>>>>>> *** traceback follows ***
>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>> >>>>>>> 2       0x7feea3ac0980
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*,
>>> unsigned short, CVirtualIFPerSideStats*) + 3220
>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>> >>>>>>> 6       0x561be171e3d9
>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>> >>>>>>> 7       0x561be171f7ec
>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>> + 92
>>> >>>>>>> 8       0x561be171fc98
>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>> CFlowGenListPerThread*, bool) + 184
>>> >>>>>>> 9       0x561be16cb5f1 int
>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>> CFlowGenListPerThread*, double&) + 881
>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() +
>>> 226
>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>> >>>>>>> 12      0x561be1714113
>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) +
>>> 487
>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>> >>>>>>> 16      0x7feea3ab56db
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>> >>>>>>>
>>> >>>>>>> *** addr2line information follows ***
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> ??:0
>>> >>>>>>> '
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> I believe if everything is working, this should not turn off trex
>>> but please correct me if I am wrong.
>>> >>>>>> I wonder if you have any suggestion of how I can approach this
>>> issue?
>>> >>>>>>
>>> >>>>>> Best,
>>> >>>>>> David
>>> >>>>>>
>>> >>>>>>
>>>
>>

[-- Attachment #2: Type: text/html, Size: 14334 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-02 16:58                 ` David Liu
@ 2021-02-02 18:57                   ` Ajit Khaparde
  2021-02-08  4:03                     ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-02-02 18:57 UTC (permalink / raw)
  To: David Liu; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 9568 bytes --]

Hi David,
I actually did miss it. I will take a look.
Thanks for the refresh mail.

Thanks
Ajit

On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
> Ping again in case if you miss the last email.
>
> Did you have a chance to give it a try on login to the machines?
>
> Just some notes, we are currently running compile and some performance/functional testing.
> Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>
> Step I run:
> CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
> ./dts -s                                                           #To start the dts
>
> Informatio:
> arm-dell.dpdklab.iol.unh.edu
> - dts - located in /opt/dts
> - Trex - located in /opt/trex/v2.87
>
> arm-gigabyte.dpdklab.iol.unh.edu
> - dpdk - located in /root/dpdk
>
> Best,
> David
>
> On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>>
>> Also, I had changed the configuration when I run Broadcom nic.
>> In the dts folder, you can start it with ./dts -s
>>
>> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>>>
>>> Hi Ajit,
>>> Sorry for the late reply.
>>>
>>> You can ssh into the machine using the same VPN Username and Password.
>>> Can you try and see if that works?
>>>
>>> Informatio:
>>> arm-dell.dpdklab.iol.unh.edu
>>> - dts - located in /opt/dts
>>> - Trex - located in /opt/trex/v2.87
>>>
>>> arm-gigabyte.dpdklab.iol.unh.edu
>>> - dpdk - located in /root/dpdk
>>>
>>>
>>> Thanks,
>>> David
>>>
>>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>
>>>> Hi David,
>>>> Thanks for pinging me. Do you think I could login into the setup and
>>>> see what is happening?
>>>> I do have the credentials for two systems in your lab -
>>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>> If you can provide me details of the systems you are trying to
>>>> configure, I could take a look.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >
>>>> > Hi Ajit,
>>>> >
>>>> > Thank you for helping out.
>>>> >
>>>> > How are you doing with this issue?
>>>> >
>>>> > Please let me know if you need anything information from us.
>>>> >
>>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>>>> >
>>>> > Best,
>>>> > David
>>>> >
>>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >>
>>>> >> Yes, so this is the same NIC work on x86.
>>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>>>> >>
>>>> >> Thank,
>>>> >> David
>>>> >>
>>>> >>
>>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >>>
>>>> >>> Does the same NIC work on x86?
>>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>>>> >>> or if there is a problem with the link. Can you check and tell?
>>>> >>>
>>>> >>> Thanks
>>>> >>> Ajit
>>>> >>>
>>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >>>>
>>>> >>>> Hi Ajit,
>>>> >>>>
>>>> >>>> Thank you for helping out.
>>>> >>>>
>>>> >>>> We have it working on the x86, but not the Arm.
>>>> >>>>
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> David
>>>> >>>>
>>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >>>>>
>>>> >>>>> Hi David,
>>>> >>>>> I will take a look.
>>>> >>>>> Do you see similar issues on x86? I am asking because
>>>> >>>>> even I would start with that to create a baseline and then
>>>> >>>>> attempt on ARM.
>>>> >>>>>
>>>> >>>>> Thanks
>>>> >>>>> Ajit
>>>> >>>>>
>>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >>>>>>
>>>> >>>>>> Hi Ajit,
>>>> >>>>>>
>>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>>>> >>>>>> All the NICs are up and running when testing.
>>>> >>>>>>
>>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>>>> >>>>>>
>>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>> >>>>>>>                dut.172.18.0.41: start
>>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>>> >>>>>>>     case_obj()
>>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>>>> >>>>>>>     self.clear_stats(ports)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>>>> >>>>>>>     raise TRexError(rc)
>>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>> >>>>>>
>>>> >>>>>> Then start sending traffic with command
>>>> >>>>>>>
>>>> >>>>>>> trex>start -f stl/imix.py
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> But this will cause an error to turn off Trex.
>>>> >>>>>>
>>>> >>>>>>> trex>
>>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>>>> >>>>>>> *** traceback follows ***
>>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>> >>>>>>>
>>>> >>>>>>> *** addr2line information follows ***
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> '
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>> >>>>>>
>>>> >>>>>> Best,
>>>> >>>>>> David
>>>> >>>>>>
>>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-02 18:57                   ` Ajit Khaparde
@ 2021-02-08  4:03                     ` Ajit Khaparde
  2021-02-09 14:35                       ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-02-08  4:03 UTC (permalink / raw)
  To: David Liu; +Cc: ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 10427 bytes --]

David,
I had some trouble with my system over the past week.
So I could not login into the setup. I am hoping that my system will be
restored early this week
I will login into the setup after that.

Thanks
Ajit

On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi David,
> I actually did miss it. I will take a look.
> Thanks for the refresh mail.
>
> Thanks
> Ajit
>
> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > Ping again in case if you miss the last email.
> >
> > Did you have a chance to give it a try on login to the machines?
> >
> > Just some notes, we are currently running compile and some
> performance/functional testing.
> > Which it may be hard for you to run dts, so please let me know if you
> want me to stop these tests for you to debug.
> >
> > Step I run:
> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
> configuration to Broadcom
> > ./dts -s                                                           #To
> start the dts
> >
> > Informatio:
> > arm-dell.dpdklab.iol.unh.edu
> > - dts - located in /opt/dts
> > - Trex - located in /opt/trex/v2.87
> >
> > arm-gigabyte.dpdklab.iol.unh.edu
> > - dpdk - located in /root/dpdk
> >
> > Best,
> > David
> >
> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
> >>
> >> Also, I had changed the configuration when I run Broadcom nic.
> >> In the dts folder, you can start it with ./dts -s
> >>
> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>
> >>> Hi Ajit,
> >>> Sorry for the late reply.
> >>>
> >>> You can ssh into the machine using the same VPN Username and Password.
> >>> Can you try and see if that works?
> >>>
> >>> Informatio:
> >>> arm-dell.dpdklab.iol.unh.edu
> >>> - dts - located in /opt/dts
> >>> - Trex - located in /opt/trex/v2.87
> >>>
> >>> arm-gigabyte.dpdklab.iol.unh.edu
> >>> - dpdk - located in /root/dpdk
> >>>
> >>>
> >>> Thanks,
> >>> David
> >>>
> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>>
> >>>> Hi David,
> >>>> Thanks for pinging me. Do you think I could login into the setup and
> >>>> see what is happening?
> >>>> I do have the credentials for two systems in your lab -
> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> >>>> If you can provide me details of the systems you are trying to
> >>>> configure, I could take a look.
> >>>>
> >>>> Thanks
> >>>> Ajit
> >>>>
> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >
> >>>> > Hi Ajit,
> >>>> >
> >>>> > Thank you for helping out.
> >>>> >
> >>>> > How are you doing with this issue?
> >>>> >
> >>>> > Please let me know if you need anything information from us.
> >>>> >
> >>>> > Just some extra information that both Trex and dts are on the same
> intel tester machine.
> >>>> >
> >>>> > Best,
> >>>> > David
> >>>> >
> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >>
> >>>> >> Yes, so this is the same NIC work on x86.
> >>>> >> By seeing the link, I am assuming you mean the ports that are
> using for Trex. If so they are UP before I start the Trex.
> >>>> >>
> >>>> >> Thank,
> >>>> >> David
> >>>> >>
> >>>> >>
> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >>>
> >>>> >>> Does the same NIC work on x86?
> >>>> >>> Also do you see link before you start the trex? I am trying to
> see if the port failed to successfully come up
> >>>> >>> or if there is a problem with the link. Can you check and tell?
> >>>> >>>
> >>>> >>> Thanks
> >>>> >>> Ajit
> >>>> >>>
> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >>>> >>>>
> >>>> >>>> Hi Ajit,
> >>>> >>>>
> >>>> >>>> Thank you for helping out.
> >>>> >>>>
> >>>> >>>> We have it working on the x86, but not the Arm.
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> Thanks,
> >>>> >>>> David
> >>>> >>>>
> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >>>>>
> >>>> >>>>> Hi David,
> >>>> >>>>> I will take a look.
> >>>> >>>>> Do you see similar issues on x86? I am asking because
> >>>> >>>>> even I would start with that to create a baseline and then
> >>>> >>>>> attempt on ARM.
> >>>> >>>>>
> >>>> >>>>> Thanks
> >>>> >>>>> Ajit
> >>>> >>>>>
> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >>>> >>>>>>
> >>>> >>>>>> Hi Ajit,
> >>>> >>>>>>
> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
> >>>> >>>>>> And I am running into a problem when I try to run
> nic_single_core_perf on the NIC.
> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
> http://git.dpdk.org/tools/dts/
> >>>> >>>>>> All the NICs are up and running when testing.
> >>>> >>>>>>
> >>>> >>>>>> The problem is an error when I try to run the test case inside
> the nic_single_core_perf:
> >>>> >>>>>>
> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize:
> 64, rxd/txd: 512
> >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >>>> >>>>>>>                dut.172.18.0.41: start
> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
> _execute_test_case
> >>>> >>>>>>>     case_obj()
> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196,
> in test_perf_nic_single_core
> >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270,
> in perf_test
> >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
> measure_throughput
> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
> _prepare_transmission
> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >>>> >>>>>>>     self.clear_stats(ports)
> >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
> clear_xstats)
> >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >>>> >>>>>>>     raise TRexError(rc)
> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to
> get server response from tcp://172.18.0.40:4501
> >>>> >>>>>>
> >>>> >>>>>>
> >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i
> --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >>>> >>>>>>
> >>>> >>>>>> Then start sending traffic with command
> >>>> >>>>>>>
> >>>> >>>>>>> trex>start -f stl/imix.py
> >>>> >>>>>>
> >>>> >>>>>>
> >>>> >>>>>> But this will cause an error to turn off Trex.
> >>>> >>>>>>
> >>>> >>>>>>> trex>
> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been
> shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for
> more than 1.00135 seconds - timeout is 1 seconds
> >>>> >>>>>>> *** traceback follows ***
> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*,
> unsigned short, CVirtualIFPerSideStats*) + 3220
> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler()
> + 226
> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned
> char) + 487
> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >>>> >>>>>>>
> >>>> >>>>>>> *** addr2line information follows ***
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> '
> >>>> >>>>>>
> >>>> >>>>>>
> >>>> >>>>>> I believe if everything is working, this should not turn off
> trex but please correct me if I am wrong.
> >>>> >>>>>> I wonder if you have any suggestion of how I can approach this
> issue?
> >>>> >>>>>>
> >>>> >>>>>> Best,
> >>>> >>>>>> David
> >>>> >>>>>>
> >>>> >>>>>>
>

[-- Attachment #1.2: Type: text/html, Size: 18030 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-08  4:03                     ` Ajit Khaparde
@ 2021-02-09 14:35                       ` David Liu
  2021-02-12  2:38                         ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-02-09 14:35 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 10917 bytes --]

Hi Ajit,

Thanks for the update. Let me know if you need anything from me.


Best,
David

On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> David,
> I had some trouble with my system over the past week.
> So I could not login into the setup. I am hoping that my system will be
> restored early this week
> I will login into the setup after that.
>
> Thanks
> Ajit
>
> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Hi David,
>> I actually did miss it. I will take a look.
>> Thanks for the refresh mail.
>>
>> Thanks
>> Ajit
>>
>> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> > Ping again in case if you miss the last email.
>> >
>> > Did you have a chance to give it a try on login to the machines?
>> >
>> > Just some notes, we are currently running compile and some
>> performance/functional testing.
>> > Which it may be hard for you to run dts, so please let me know if you
>> want me to stop these tests for you to debug.
>> >
>> > Step I run:
>> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
>> configuration to Broadcom
>> > ./dts -s                                                           #To
>> start the dts
>> >
>> > Informatio:
>> > arm-dell.dpdklab.iol.unh.edu
>> > - dts - located in /opt/dts
>> > - Trex - located in /opt/trex/v2.87
>> >
>> > arm-gigabyte.dpdklab.iol.unh.edu
>> > - dpdk - located in /root/dpdk
>> >
>> > Best,
>> > David
>> >
>> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>
>> >> Also, I had changed the configuration when I run Broadcom nic.
>> >> In the dts folder, you can start it with ./dts -s
>> >>
>> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>
>> >>> Hi Ajit,
>> >>> Sorry for the late reply.
>> >>>
>> >>> You can ssh into the machine using the same VPN Username and Password.
>> >>> Can you try and see if that works?
>> >>>
>> >>> Informatio:
>> >>> arm-dell.dpdklab.iol.unh.edu
>> >>> - dts - located in /opt/dts
>> >>> - Trex - located in /opt/trex/v2.87
>> >>>
>> >>> arm-gigabyte.dpdklab.iol.unh.edu
>> >>> - dpdk - located in /root/dpdk
>> >>>
>> >>>
>> >>> Thanks,
>> >>> David
>> >>>
>> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>>
>> >>>> Hi David,
>> >>>> Thanks for pinging me. Do you think I could login into the setup and
>> >>>> see what is happening?
>> >>>> I do have the credentials for two systems in your lab -
>> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>> >>>> If you can provide me details of the systems you are trying to
>> >>>> configure, I could take a look.
>> >>>>
>> >>>> Thanks
>> >>>> Ajit
>> >>>>
>> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >
>> >>>> > Hi Ajit,
>> >>>> >
>> >>>> > Thank you for helping out.
>> >>>> >
>> >>>> > How are you doing with this issue?
>> >>>> >
>> >>>> > Please let me know if you need anything information from us.
>> >>>> >
>> >>>> > Just some extra information that both Trex and dts are on the same
>> intel tester machine.
>> >>>> >
>> >>>> > Best,
>> >>>> > David
>> >>>> >
>> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >>>> >>
>> >>>> >> Yes, so this is the same NIC work on x86.
>> >>>> >> By seeing the link, I am assuming you mean the ports that are
>> using for Trex. If so they are UP before I start the Trex.
>> >>>> >>
>> >>>> >> Thank,
>> >>>> >> David
>> >>>> >>
>> >>>> >>
>> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >>>
>> >>>> >>> Does the same NIC work on x86?
>> >>>> >>> Also do you see link before you start the trex? I am trying to
>> see if the port failed to successfully come up
>> >>>> >>> or if there is a problem with the link. Can you check and tell?
>> >>>> >>>
>> >>>> >>> Thanks
>> >>>> >>> Ajit
>> >>>> >>>
>> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >>>> >>>>
>> >>>> >>>> Hi Ajit,
>> >>>> >>>>
>> >>>> >>>> Thank you for helping out.
>> >>>> >>>>
>> >>>> >>>> We have it working on the x86, but not the Arm.
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>> Thanks,
>> >>>> >>>> David
>> >>>> >>>>
>> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >>>>>
>> >>>> >>>>> Hi David,
>> >>>> >>>>> I will take a look.
>> >>>> >>>>> Do you see similar issues on x86? I am asking because
>> >>>> >>>>> even I would start with that to create a baseline and then
>> >>>> >>>>> attempt on ARM.
>> >>>> >>>>>
>> >>>> >>>>> Thanks
>> >>>> >>>>> Ajit
>> >>>> >>>>>
>> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >>>> >>>>>>
>> >>>> >>>>>> Hi Ajit,
>> >>>> >>>>>>
>> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>> machine.
>> >>>> >>>>>> And I am running into a problem when I try to run
>> nic_single_core_perf on the NIC.
>> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>> http://git.dpdk.org/tools/dts/
>> >>>> >>>>>> All the NICs are up and running when testing.
>> >>>> >>>>>>
>> >>>> >>>>>> The problem is an error when I try to run the test case
>> inside the nic_single_core_perf:
>> >>>> >>>>>>
>> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>> framesize: 64, rxd/txd: 512
>> >>>> >>>>>>>                dut.172.18.0.41:
>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> >>>> >>>>>>>                dut.172.18.0.41: start
>> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>> _execute_test_case
>> >>>> >>>>>>>     case_obj()
>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196,
>> in test_perf_nic_single_core
>> >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270,
>> in perf_test
>> >>>> >>>>>>>     _, packets_received =
>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>> options=traffic_opt)
>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>> measure_throughput
>> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>> _prepare_transmission
>> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>> >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 339, in reset
>> >>>> >>>>>>>     self.clear_stats(ports)
>> >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 1492, in clear_stats
>> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>> clear_xstats)
>> >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>> line 2876, in _clear_stats_common
>> >>>> >>>>>>>     raise TRexError(rc)
>> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to
>> get server response from tcp://172.18.0.40:4501
>> >>>> >>>>>>
>> >>>> >>>>>>
>> >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i
>> --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>> >>>> >>>>>>
>> >>>> >>>>>> Then start sending traffic with command
>> >>>> >>>>>>>
>> >>>> >>>>>>> trex>start -f stl/imix.py
>> >>>> >>>>>>
>> >>>> >>>>>>
>> >>>> >>>>>> But this will cause an error to turn off Trex.
>> >>>> >>>>>>
>> >>>> >>>>>>> trex>
>> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been
>> shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for
>> more than 1.00135 seconds - timeout is 1 seconds
>> >>>> >>>>>>> *** traceback follows ***
>> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>> [0x561be173cf5a]
>> >>>> >>>>>>> 2       0x7feea3ac0980
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*,
>> unsigned short, CVirtualIFPerSideStats*) + 3220
>> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> >>>> >>>>>>> 6       0x561be171e3d9
>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> >>>> >>>>>>> 7       0x561be171f7ec
>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>> + 92
>> >>>> >>>>>>> 8       0x561be171fc98
>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>> CFlowGenListPerThread*, bool) + 184
>> >>>> >>>>>>> 9       0x561be16cb5f1 int
>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>> CFlowGenListPerThread*, double&) + 881
>> >>>> >>>>>>> 10      0x561be1905212
>> TrexStatelessDpCore::start_scheduler() + 226
>> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> >>>> >>>>>>> 12      0x561be1714113
>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned
>> char) + 487
>> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>> [0x561be16d11ad]
>> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> >>>> >>>>>>> 16      0x7feea3ab56db
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> >>>> >>>>>>>
>> >>>> >>>>>>> *** addr2line information follows ***
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> ??:0
>> >>>> >>>>>>> '
>> >>>> >>>>>>
>> >>>> >>>>>>
>> >>>> >>>>>> I believe if everything is working, this should not turn off
>> trex but please correct me if I am wrong.
>> >>>> >>>>>> I wonder if you have any suggestion of how I can approach
>> this issue?
>> >>>> >>>>>>
>> >>>> >>>>>> Best,
>> >>>> >>>>>> David
>> >>>> >>>>>>
>> >>>> >>>>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 18578 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-09 14:35                       ` David Liu
@ 2021-02-12  2:38                         ` Ajit Khaparde
       [not found]                           ` <CAHx6DYDLOBOMMo_vfB0YAhLcQMbDV3-H=RY6Oa+Tp7tWdFxo8A@mail.gmail.com>
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-02-12  2:38 UTC (permalink / raw)
  To: David Liu; +Cc: ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 11644 bytes --]

David,
I can use my credentials to login to rhea.
But the same credentials are not allowing me to login onto the servers you
provided.

Thanks
Ajit

ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
Connection closed by 172.18.0.41 port 22

ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
Connection closed by 172.18.0.40 port 22

On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit,
>
> Thanks for the update. Let me know if you need anything from me.
>
>
> Best,
> David
>
> On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> David,
>> I had some trouble with my system over the past week.
>> So I could not login into the setup. I am hoping that my system will be
>> restored early this week
>> I will login into the setup after that.
>>
>> Thanks
>> Ajit
>>
>> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>>> Hi David,
>>> I actually did miss it. I will take a look.
>>> Thanks for the refresh mail.
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >
>>> > Hi Ajit,
>>> >
>>> > Ping again in case if you miss the last email.
>>> >
>>> > Did you have a chance to give it a try on login to the machines?
>>> >
>>> > Just some notes, we are currently running compile and some
>>> performance/functional testing.
>>> > Which it may be hard for you to run dts, so please let me know if you
>>> want me to stop these tests for you to debug.
>>> >
>>> > Step I run:
>>> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
>>> configuration to Broadcom
>>> > ./dts -s                                                           #To
>>> start the dts
>>> >
>>> > Informatio:
>>> > arm-dell.dpdklab.iol.unh.edu
>>> > - dts - located in /opt/dts
>>> > - Trex - located in /opt/trex/v2.87
>>> >
>>> > arm-gigabyte.dpdklab.iol.unh.edu
>>> > - dpdk - located in /root/dpdk
>>> >
>>> > Best,
>>> > David
>>> >
>>> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>>> >>
>>> >> Also, I had changed the configuration when I run Broadcom nic.
>>> >> In the dts folder, you can start it with ./dts -s
>>> >>
>>> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>>> >>>
>>> >>> Hi Ajit,
>>> >>> Sorry for the late reply.
>>> >>>
>>> >>> You can ssh into the machine using the same VPN Username and
>>> Password.
>>> >>> Can you try and see if that works?
>>> >>>
>>> >>> Informatio:
>>> >>> arm-dell.dpdklab.iol.unh.edu
>>> >>> - dts - located in /opt/dts
>>> >>> - Trex - located in /opt/trex/v2.87
>>> >>>
>>> >>> arm-gigabyte.dpdklab.iol.unh.edu
>>> >>> - dpdk - located in /root/dpdk
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>> David
>>> >>>
>>> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>>
>>> >>>> Hi David,
>>> >>>> Thanks for pinging me. Do you think I could login into the setup and
>>> >>>> see what is happening?
>>> >>>> I do have the credentials for two systems in your lab -
>>> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>> >>>> If you can provide me details of the systems you are trying to
>>> >>>> configure, I could take a look.
>>> >>>>
>>> >>>> Thanks
>>> >>>> Ajit
>>> >>>>
>>> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>>> >>>> >
>>> >>>> > Hi Ajit,
>>> >>>> >
>>> >>>> > Thank you for helping out.
>>> >>>> >
>>> >>>> > How are you doing with this issue?
>>> >>>> >
>>> >>>> > Please let me know if you need anything information from us.
>>> >>>> >
>>> >>>> > Just some extra information that both Trex and dts are on the
>>> same intel tester machine.
>>> >>>> >
>>> >>>> > Best,
>>> >>>> > David
>>> >>>> >
>>> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >>>> >>
>>> >>>> >> Yes, so this is the same NIC work on x86.
>>> >>>> >> By seeing the link, I am assuming you mean the ports that are
>>> using for Trex. If so they are UP before I start the Trex.
>>> >>>> >>
>>> >>>> >> Thank,
>>> >>>> >> David
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >>>
>>> >>>> >>> Does the same NIC work on x86?
>>> >>>> >>> Also do you see link before you start the trex? I am trying to
>>> see if the port failed to successfully come up
>>> >>>> >>> or if there is a problem with the link. Can you check and tell?
>>> >>>> >>>
>>> >>>> >>> Thanks
>>> >>>> >>> Ajit
>>> >>>> >>>
>>> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >>>> >>>>
>>> >>>> >>>> Hi Ajit,
>>> >>>> >>>>
>>> >>>> >>>> Thank you for helping out.
>>> >>>> >>>>
>>> >>>> >>>> We have it working on the x86, but not the Arm.
>>> >>>> >>>>
>>> >>>> >>>>
>>> >>>> >>>> Thanks,
>>> >>>> >>>> David
>>> >>>> >>>>
>>> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >>>>>
>>> >>>> >>>>> Hi David,
>>> >>>> >>>>> I will take a look.
>>> >>>> >>>>> Do you see similar issues on x86? I am asking because
>>> >>>> >>>>> even I would start with that to create a baseline and then
>>> >>>> >>>>> attempt on ARM.
>>> >>>> >>>>>
>>> >>>> >>>>> Thanks
>>> >>>> >>>>> Ajit
>>> >>>> >>>>>
>>> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >>>> >>>>>>
>>> >>>> >>>>>> Hi Ajit,
>>> >>>> >>>>>>
>>> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>>> machine.
>>> >>>> >>>>>> And I am running into a problem when I try to run
>>> nic_single_core_perf on the NIC.
>>> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>> http://git.dpdk.org/tools/dts/
>>> >>>> >>>>>> All the NICs are up and running when testing.
>>> >>>> >>>>>>
>>> >>>> >>>>>> The problem is an error when I try to run the test case
>>> inside the nic_single_core_perf:
>>> >>>> >>>>>>
>>> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>> framesize: 64, rxd/txd: 512
>>> >>>> >>>>>>>                dut.172.18.0.41:
>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>> >>>> >>>>>>>                dut.172.18.0.41: start
>>> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>> _execute_test_case
>>> >>>> >>>>>>>     case_obj()
>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196,
>>> in test_perf_nic_single_core
>>> >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270,
>>> in perf_test
>>> >>>> >>>>>>>     _, packets_received =
>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>> options=traffic_opt)
>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>> measure_throughput
>>> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>>> _prepare_transmission
>>> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>> >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 339, in reset
>>> >>>> >>>>>>>     self.clear_stats(ports)
>>> >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 1492, in clear_stats
>>> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>> clear_xstats)
>>> >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>> line 2876, in _clear_stats_common
>>> >>>> >>>>>>>     raise TRexError(rc)
>>> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed
>>> to get server response from tcp://172.18.0.40:4501
>>> >>>> >>>>>>
>>> >>>> >>>>>>
>>> >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i
>>> --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>> >>>> >>>>>>
>>> >>>> >>>>>> Then start sending traffic with command
>>> >>>> >>>>>>>
>>> >>>> >>>>>>> trex>start -f stl/imix.py
>>> >>>> >>>>>>
>>> >>>> >>>>>>
>>> >>>> >>>>>> But this will cause an error to turn off Trex.
>>> >>>> >>>>>>
>>> >>>> >>>>>>> trex>
>>> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been
>>> shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for
>>> more than 1.00135 seconds - timeout is 1 seconds
>>> >>>> >>>>>>> *** traceback follows ***
>>> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>> [0x561be173cf5a]
>>> >>>> >>>>>>> 2       0x7feea3ac0980
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>> >>>> >>>>>>> 4       0x561be16ce874
>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>> CVirtualIFPerSideStats*) + 3220
>>> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>> >>>> >>>>>>> 6       0x561be171e3d9
>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>> >>>> >>>>>>> 7       0x561be171f7ec
>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>> + 92
>>> >>>> >>>>>>> 8       0x561be171fc98
>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>> CFlowGenListPerThread*, bool) + 184
>>> >>>> >>>>>>> 9       0x561be16cb5f1 int
>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>> CFlowGenListPerThread*, double&) + 881
>>> >>>> >>>>>>> 10      0x561be1905212
>>> TrexStatelessDpCore::start_scheduler() + 226
>>> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>> >>>> >>>>>>> 12      0x561be1714113
>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>> >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned
>>> char) + 487
>>> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>> [0x561be16d11ad]
>>> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>> >>>> >>>>>>> 16      0x7feea3ab56db
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>> >>>> >>>>>>>
>>> >>>> >>>>>>> *** addr2line information follows ***
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> ??:0
>>> >>>> >>>>>>> '
>>> >>>> >>>>>>
>>> >>>> >>>>>>
>>> >>>> >>>>>> I believe if everything is working, this should not turn off
>>> trex but please correct me if I am wrong.
>>> >>>> >>>>>> I wonder if you have any suggestion of how I can approach
>>> this issue?
>>> >>>> >>>>>>
>>> >>>> >>>>>> Best,
>>> >>>> >>>>>> David
>>> >>>> >>>>>>
>>> >>>> >>>>>>
>>>
>>

[-- Attachment #1.2: Type: text/html, Size: 20104 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
       [not found]                           ` <CAHx6DYDLOBOMMo_vfB0YAhLcQMbDV3-H=RY6Oa+Tp7tWdFxo8A@mail.gmail.com>
@ 2021-02-12 22:48                             ` Ajit Khaparde
  2021-02-18 18:00                               ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-02-12 22:48 UTC (permalink / raw)
  To: Owen Hilyard, ci, dpdklab, David Liu


[-- Attachment #1.1: Type: text/plain, Size: 13151 bytes --]

Thanks Owen.

When I run the dts command line suggested, I get the following error.

Thanks
Ajit

Traceback (most recent call last):
  File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
    self.password, original_prompt='[$#>]',
password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
.+:)')
  File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440,
in login
    raise ExceptionPxssh('password refused')
pexpect.pxssh.ExceptionPxssh: password refused
                                         ^^^^^^^^^^^^^^



On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:

> Hello Ajit,
>
> There was a misconfiguration in our identity management system. The issue
> should now be resolved.
>
> Owen Hilyard
>
> On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
> dpdklab@iol.unh.edu> wrote:
>
>> David,
>> I can use my credentials to login to rhea.
>> But the same credentials are not allowing me to login onto the servers
>> you provided.
>>
>> Thanks
>> Ajit
>>
>> ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>> Connection closed by 172.18.0.41 port 22
>>
>> ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>> Connection closed by 172.18.0.40 port 22
>>
>> On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>
>>> Hi Ajit,
>>>
>>> Thanks for the update. Let me know if you need anything from me.
>>>
>>>
>>> Best,
>>> David
>>>
>>> On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>>
>>>> David,
>>>> I had some trouble with my system over the past week.
>>>> So I could not login into the setup. I am hoping that my system will be
>>>> restored early this week
>>>> I will login into the setup after that.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>>
>>>>> Hi David,
>>>>> I actually did miss it. I will take a look.
>>>>> Thanks for the refresh mail.
>>>>>
>>>>> Thanks
>>>>> Ajit
>>>>>
>>>>> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >
>>>>> > Hi Ajit,
>>>>> >
>>>>> > Ping again in case if you miss the last email.
>>>>> >
>>>>> > Did you have a chance to give it a try on login to the machines?
>>>>> >
>>>>> > Just some notes, we are currently running compile and some
>>>>> performance/functional testing.
>>>>> > Which it may be hard for you to run dts, so please let me know if
>>>>> you want me to stop these tests for you to debug.
>>>>> >
>>>>> > Step I run:
>>>>> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
>>>>> configuration to Broadcom
>>>>> > ./dts -s
>>>>>  #To start the dts
>>>>> >
>>>>> > Informatio:
>>>>> > arm-dell.dpdklab.iol.unh.edu
>>>>> > - dts - located in /opt/dts
>>>>> > - Trex - located in /opt/trex/v2.87
>>>>> >
>>>>> > arm-gigabyte.dpdklab.iol.unh.edu
>>>>> > - dpdk - located in /root/dpdk
>>>>> >
>>>>> > Best,
>>>>> > David
>>>>> >
>>>>> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >>
>>>>> >> Also, I had changed the configuration when I run Broadcom nic.
>>>>> >> In the dts folder, you can start it with ./dts -s
>>>>> >>
>>>>> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >>>
>>>>> >>> Hi Ajit,
>>>>> >>> Sorry for the late reply.
>>>>> >>>
>>>>> >>> You can ssh into the machine using the same VPN Username and
>>>>> Password.
>>>>> >>> Can you try and see if that works?
>>>>> >>>
>>>>> >>> Informatio:
>>>>> >>> arm-dell.dpdklab.iol.unh.edu
>>>>> >>> - dts - located in /opt/dts
>>>>> >>> - Trex - located in /opt/trex/v2.87
>>>>> >>>
>>>>> >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>>> >>> - dpdk - located in /root/dpdk
>>>>> >>>
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> David
>>>>> >>>
>>>>> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>> >>>>
>>>>> >>>> Hi David,
>>>>> >>>> Thanks for pinging me. Do you think I could login into the setup
>>>>> and
>>>>> >>>> see what is happening?
>>>>> >>>> I do have the credentials for two systems in your lab -
>>>>> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>>> >>>> If you can provide me details of the systems you are trying to
>>>>> >>>> configure, I could take a look.
>>>>> >>>>
>>>>> >>>> Thanks
>>>>> >>>> Ajit
>>>>> >>>>
>>>>> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>>>>> wrote:
>>>>> >>>> >
>>>>> >>>> > Hi Ajit,
>>>>> >>>> >
>>>>> >>>> > Thank you for helping out.
>>>>> >>>> >
>>>>> >>>> > How are you doing with this issue?
>>>>> >>>> >
>>>>> >>>> > Please let me know if you need anything information from us.
>>>>> >>>> >
>>>>> >>>> > Just some extra information that both Trex and dts are on the
>>>>> same intel tester machine.
>>>>> >>>> >
>>>>> >>>> > Best,
>>>>> >>>> > David
>>>>> >>>> >
>>>>> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>>>>> wrote:
>>>>> >>>> >>
>>>>> >>>> >> Yes, so this is the same NIC work on x86.
>>>>> >>>> >> By seeing the link, I am assuming you mean the ports that are
>>>>> using for Trex. If so they are UP before I start the Trex.
>>>>> >>>> >>
>>>>> >>>> >> Thank,
>>>>> >>>> >> David
>>>>> >>>> >>
>>>>> >>>> >>
>>>>> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>> >>>> >>>
>>>>> >>>> >>> Does the same NIC work on x86?
>>>>> >>>> >>> Also do you see link before you start the trex? I am trying
>>>>> to see if the port failed to successfully come up
>>>>> >>>> >>> or if there is a problem with the link. Can you check and
>>>>> tell?
>>>>> >>>> >>>
>>>>> >>>> >>> Thanks
>>>>> >>>> >>> Ajit
>>>>> >>>> >>>
>>>>> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
>>>>> wrote:
>>>>> >>>> >>>>
>>>>> >>>> >>>> Hi Ajit,
>>>>> >>>> >>>>
>>>>> >>>> >>>> Thank you for helping out.
>>>>> >>>> >>>>
>>>>> >>>> >>>> We have it working on the x86, but not the Arm.
>>>>> >>>> >>>>
>>>>> >>>> >>>>
>>>>> >>>> >>>> Thanks,
>>>>> >>>> >>>> David
>>>>> >>>> >>>>
>>>>> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>> >>>> >>>>>
>>>>> >>>> >>>>> Hi David,
>>>>> >>>> >>>>> I will take a look.
>>>>> >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>>> >>>> >>>>> even I would start with that to create a baseline and then
>>>>> >>>> >>>>> attempt on ARM.
>>>>> >>>> >>>>>
>>>>> >>>> >>>>> Thanks
>>>>> >>>> >>>>> Ajit
>>>>> >>>> >>>>>
>>>>> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>>>>> dliu@iol.unh.edu> wrote:
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> Hi Ajit,
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>>>>> machine.
>>>>> >>>> >>>>>> And I am running into a problem when I try to run
>>>>> nic_single_core_perf on the NIC.
>>>>> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>>>> http://git.dpdk.org/tools/dts/
>>>>> >>>> >>>>>> All the NICs are up and running when testing.
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> The problem is an error when I try to run the test case
>>>>> inside the nic_single_core_perf:
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>>>> framesize: 64, rxd/txd: 512
>>>>> >>>> >>>>>>>                dut.172.18.0.41:
>>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>> >>>> >>>>>>>                dut.172.18.0.41: start
>>>>> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>>>> _execute_test_case
>>>>> >>>> >>>>>>>     case_obj()
>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>>>> 196, in test_perf_nic_single_core
>>>>> >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>>>> 270, in perf_test
>>>>> >>>> >>>>>>>     _, packets_received =
>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>> options=traffic_opt)
>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>>>> measure_throughput
>>>>> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>>>>> _prepare_transmission
>>>>> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>>> >>>> >>>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>> line 51, in wrap2
>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>> >>>> >>>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>> line 339, in reset
>>>>> >>>> >>>>>>>     self.clear_stats(ports)
>>>>> >>>> >>>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>> line 51, in wrap2
>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>> >>>> >>>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>> line 1492, in clear_stats
>>>>> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>>>> clear_xstats)
>>>>> >>>> >>>>>>>   File
>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>> line 2876, in _clear_stats_common
>>>>> >>>> >>>>>>>     raise TRexError(rc)
>>>>> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed
>>>>> to get server response from tcp://172.18.0.40:4501
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i
>>>>> --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> Then start sending traffic with command
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>> trex>start -f stl/imix.py
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> But this will cause an error to turn off Trex.
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>> trex>
>>>>> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
>>>>> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
>>>>> for more than 1.00135 seconds - timeout is 1 seconds
>>>>> >>>> >>>>>>> *** traceback follows ***
>>>>> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>>>> [0x561be173cf5a]
>>>>> >>>> >>>>>>> 2       0x7feea3ac0980
>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>> >>>> >>>>>>> 4       0x561be16ce874
>>>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>>>> CVirtualIFPerSideStats*) + 3220
>>>>> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>>> >>>> >>>>>>> 6       0x561be171e3d9
>>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>> >>>> >>>>>>> 7       0x561be171f7ec
>>>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>>>> + 92
>>>>> >>>> >>>>>>> 8       0x561be171fc98
>>>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>>>> CFlowGenListPerThread*, bool) + 184
>>>>> >>>> >>>>>>> 9       0x561be16cb5f1 int
>>>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>>>> CFlowGenListPerThread*, double&) + 881
>>>>> >>>> >>>>>>> 10      0x561be1905212
>>>>> TrexStatelessDpCore::start_scheduler() + 226
>>>>> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>> >>>> >>>>>>> 12      0x561be1714113
>>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>> >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned
>>>>> char) + 487
>>>>> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>>>> [0x561be16d11ad]
>>>>> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>> >>>> >>>>>>> 16      0x7feea3ab56db
>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>>> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>>> >>>> >>>>>>>
>>>>> >>>> >>>>>>> *** addr2line information follows ***
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> ??:0
>>>>> >>>> >>>>>>> '
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> I believe if everything is working, this should not turn
>>>>> off trex but please correct me if I am wrong.
>>>>> >>>> >>>>>> I wonder if you have any suggestion of how I can approach
>>>>> this issue?
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>> Best,
>>>>> >>>> >>>>>> David
>>>>> >>>> >>>>>>
>>>>> >>>> >>>>>>
>>>>>
>>>>

[-- Attachment #1.2: Type: text/html, Size: 22546 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-12 22:48                             ` [dpdk-ci] [dpdklab] " Ajit Khaparde
@ 2021-02-18 18:00                               ` David Liu
  2021-02-18 18:03                                 ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-02-18 18:00 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 13977 bytes --]

Hi Ajit,

This is likely caused by when dts try to ssh into another machine as a
normal user which will prompt you for the password.

One solution would be "sudo su" as root to run the command line since you
should have root access.

Another solution would be to let me add your public ssh key into both
machines.

Let me know if this helps.

Best,
David

On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Thanks Owen.
>
> When I run the dts command line suggested, I get the following error.
>
> Thanks
> Ajit
>
> Traceback (most recent call last):
>   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
>   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line
> 440, in login
>     raise ExceptionPxssh('password refused')
> pexpect.pxssh.ExceptionPxssh: password refused
>                                          ^^^^^^^^^^^^^^
>
>
>
> On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
> wrote:
>
>> Hello Ajit,
>>
>> There was a misconfiguration in our identity management system. The issue
>> should now be resolved.
>>
>> Owen Hilyard
>>
>> On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
>> dpdklab@iol.unh.edu> wrote:
>>
>>> David,
>>> I can use my credentials to login to rhea.
>>> But the same credentials are not allowing me to login onto the servers
>>> you provided.
>>>
>>> Thanks
>>> Ajit
>>>
>>> ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>> Connection closed by 172.18.0.41 port 22
>>>
>>> ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>> Connection closed by 172.18.0.40 port 22
>>>
>>> On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>>
>>>> Hi Ajit,
>>>>
>>>> Thanks for the update. Let me know if you need anything from me.
>>>>
>>>>
>>>> Best,
>>>> David
>>>>
>>>> On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>>
>>>>> David,
>>>>> I had some trouble with my system over the past week.
>>>>> So I could not login into the setup. I am hoping that my system will
>>>>> be restored early this week
>>>>> I will login into the setup after that.
>>>>>
>>>>> Thanks
>>>>> Ajit
>>>>>
>>>>> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>
>>>>>> Hi David,
>>>>>> I actually did miss it. I will take a look.
>>>>>> Thanks for the refresh mail.
>>>>>>
>>>>>> Thanks
>>>>>> Ajit
>>>>>>
>>>>>> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>>> >
>>>>>> > Hi Ajit,
>>>>>> >
>>>>>> > Ping again in case if you miss the last email.
>>>>>> >
>>>>>> > Did you have a chance to give it a try on login to the machines?
>>>>>> >
>>>>>> > Just some notes, we are currently running compile and some
>>>>>> performance/functional testing.
>>>>>> > Which it may be hard for you to run dts, so please let me know if
>>>>>> you want me to stop these tests for you to debug.
>>>>>> >
>>>>>> > Step I run:
>>>>>> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
>>>>>> configuration to Broadcom
>>>>>> > ./dts -s
>>>>>>  #To start the dts
>>>>>> >
>>>>>> > Informatio:
>>>>>> > arm-dell.dpdklab.iol.unh.edu
>>>>>> > - dts - located in /opt/dts
>>>>>> > - Trex - located in /opt/trex/v2.87
>>>>>> >
>>>>>> > arm-gigabyte.dpdklab.iol.unh.edu
>>>>>> > - dpdk - located in /root/dpdk
>>>>>> >
>>>>>> > Best,
>>>>>> > David
>>>>>> >
>>>>>> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>>> >>
>>>>>> >> Also, I had changed the configuration when I run Broadcom nic.
>>>>>> >> In the dts folder, you can start it with ./dts -s
>>>>>> >>
>>>>>> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Hi Ajit,
>>>>>> >>> Sorry for the late reply.
>>>>>> >>>
>>>>>> >>> You can ssh into the machine using the same VPN Username and
>>>>>> Password.
>>>>>> >>> Can you try and see if that works?
>>>>>> >>>
>>>>>> >>> Informatio:
>>>>>> >>> arm-dell.dpdklab.iol.unh.edu
>>>>>> >>> - dts - located in /opt/dts
>>>>>> >>> - Trex - located in /opt/trex/v2.87
>>>>>> >>>
>>>>>> >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>>>> >>> - dpdk - located in /root/dpdk
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> Thanks,
>>>>>> >>> David
>>>>>> >>>
>>>>>> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>> >>>>
>>>>>> >>>> Hi David,
>>>>>> >>>> Thanks for pinging me. Do you think I could login into the setup
>>>>>> and
>>>>>> >>>> see what is happening?
>>>>>> >>>> I do have the credentials for two systems in your lab -
>>>>>> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>>>> >>>> If you can provide me details of the systems you are trying to
>>>>>> >>>> configure, I could take a look.
>>>>>> >>>>
>>>>>> >>>> Thanks
>>>>>> >>>> Ajit
>>>>>> >>>>
>>>>>> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>>>>>> wrote:
>>>>>> >>>> >
>>>>>> >>>> > Hi Ajit,
>>>>>> >>>> >
>>>>>> >>>> > Thank you for helping out.
>>>>>> >>>> >
>>>>>> >>>> > How are you doing with this issue?
>>>>>> >>>> >
>>>>>> >>>> > Please let me know if you need anything information from us.
>>>>>> >>>> >
>>>>>> >>>> > Just some extra information that both Trex and dts are on the
>>>>>> same intel tester machine.
>>>>>> >>>> >
>>>>>> >>>> > Best,
>>>>>> >>>> > David
>>>>>> >>>> >
>>>>>> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>>>>>> wrote:
>>>>>> >>>> >>
>>>>>> >>>> >> Yes, so this is the same NIC work on x86.
>>>>>> >>>> >> By seeing the link, I am assuming you mean the ports that are
>>>>>> using for Trex. If so they are UP before I start the Trex.
>>>>>> >>>> >>
>>>>>> >>>> >> Thank,
>>>>>> >>>> >> David
>>>>>> >>>> >>
>>>>>> >>>> >>
>>>>>> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>> >>>> >>>
>>>>>> >>>> >>> Does the same NIC work on x86?
>>>>>> >>>> >>> Also do you see link before you start the trex? I am trying
>>>>>> to see if the port failed to successfully come up
>>>>>> >>>> >>> or if there is a problem with the link. Can you check and
>>>>>> tell?
>>>>>> >>>> >>>
>>>>>> >>>> >>> Thanks
>>>>>> >>>> >>> Ajit
>>>>>> >>>> >>>
>>>>>> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
>>>>>> wrote:
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> Hi Ajit,
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> Thank you for helping out.
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> We have it working on the x86, but not the Arm.
>>>>>> >>>> >>>>
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> Thanks,
>>>>>> >>>> >>>> David
>>>>>> >>>> >>>>
>>>>>> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>> Hi David,
>>>>>> >>>> >>>>> I will take a look.
>>>>>> >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>>>> >>>> >>>>> even I would start with that to create a baseline and then
>>>>>> >>>> >>>>> attempt on ARM.
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>> Thanks
>>>>>> >>>> >>>>> Ajit
>>>>>> >>>> >>>>>
>>>>>> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>>>>>> dliu@iol.unh.edu> wrote:
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> Hi Ajit,
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>>>>>> machine.
>>>>>> >>>> >>>>>> And I am running into a problem when I try to run
>>>>>> nic_single_core_perf on the NIC.
>>>>>> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>>>>> http://git.dpdk.org/tools/dts/
>>>>>> >>>> >>>>>> All the NICs are up and running when testing.
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> The problem is an error when I try to run the test case
>>>>>> inside the nic_single_core_perf:
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>>>>> framesize: 64, rxd/txd: 512
>>>>>> >>>> >>>>>>>                dut.172.18.0.41:
>>>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>>> >>>> >>>>>>>                dut.172.18.0.41: start
>>>>>> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>>>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>>>>> _execute_test_case
>>>>>> >>>> >>>>>>>     case_obj()
>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>>>>> 196, in test_perf_nic_single_core
>>>>>> >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>>>>> 270, in perf_test
>>>>>> >>>> >>>>>>>     _, packets_received =
>>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>>> options=traffic_opt)
>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
>>>>>> measure_throughput
>>>>>> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
>>>>>> _prepare_transmission
>>>>>> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>>>> >>>> >>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>> line 51, in wrap2
>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>> >>>> >>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>> line 339, in reset
>>>>>> >>>> >>>>>>>     self.clear_stats(ports)
>>>>>> >>>> >>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>> line 51, in wrap2
>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>> >>>> >>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>> line 1492, in clear_stats
>>>>>> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>>>>> clear_xstats)
>>>>>> >>>> >>>>>>>   File
>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>>> line 2876, in _clear_stats_common
>>>>>> >>>> >>>>>>>     raise TRexError(rc)
>>>>>> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
>>>>>> Failed to get server response from tcp://172.18.0.40:4501
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64
>>>>>> -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> Then start sending traffic with command
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>> trex>start -f stl/imix.py
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> But this will cause an error to turn off Trex.
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>> trex>
>>>>>> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
>>>>>> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
>>>>>> for more than 1.00135 seconds - timeout is 1 seconds
>>>>>> >>>> >>>>>>> *** traceback follows ***
>>>>>> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>>>>> [0x561be173cf5a]
>>>>>> >>>> >>>>>>> 2       0x7feea3ac0980
>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>>> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>>> >>>> >>>>>>> 4       0x561be16ce874
>>>>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>>>>> CVirtualIFPerSideStats*) + 3220
>>>>>> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>>>> >>>> >>>>>>> 6       0x561be171e3d9
>>>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>>> >>>> >>>>>>> 7       0x561be171f7ec
>>>>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>>>>> + 92
>>>>>> >>>> >>>>>>> 8       0x561be171fc98
>>>>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>>>>> CFlowGenListPerThread*, bool) + 184
>>>>>> >>>> >>>>>>> 9       0x561be16cb5f1 int
>>>>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>>>>> CFlowGenListPerThread*, double&) + 881
>>>>>> >>>> >>>>>>> 10      0x561be1905212
>>>>>> TrexStatelessDpCore::start_scheduler() + 226
>>>>>> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>>> >>>> >>>>>>> 12      0x561be1714113
>>>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>>> >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned
>>>>>> char) + 487
>>>>>> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>>>>> [0x561be16d11ad]
>>>>>> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>>> >>>> >>>>>>> 16      0x7feea3ab56db
>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>>>> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>>>> >>>> >>>>>>>
>>>>>> >>>> >>>>>>> *** addr2line information follows ***
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> ??:0
>>>>>> >>>> >>>>>>> '
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> I believe if everything is working, this should not turn
>>>>>> off trex but please correct me if I am wrong.
>>>>>> >>>> >>>>>> I wonder if you have any suggestion of how I can approach
>>>>>> this issue?
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>> Best,
>>>>>> >>>> >>>>>> David
>>>>>> >>>> >>>>>>
>>>>>> >>>> >>>>>>
>>>>>>
>>>>>

[-- Attachment #2: Type: text/html, Size: 25018 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-18 18:00                               ` David Liu
@ 2021-02-18 18:03                                 ` Ajit Khaparde
  2021-02-23  4:10                                   ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-02-18 18:03 UTC (permalink / raw)
  To: David Liu; +Cc: Owen Hilyard, ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 14585 bytes --]

On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit,
>
> This is likely caused by when dts try to ssh into another machine as a
> normal user which will prompt you for the password.
>
> One solution would be "sudo su" as root to run the command line since you
> should have root access.
>
David,
Let me try this method first.
If it does not work, we could try with the public ssh keys.
I will let you know.

Thanks
Ajit

>
> Another solution would be to let me add your public ssh key into both
> machines.
>
> Let me know if this helps.
>
> Best,
> David
>
> On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Thanks Owen.
>>
>> When I run the dts command line suggested, I get the following error.
>>
>> Thanks
>> Ajit
>>
>> Traceback (most recent call last):
>>   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>>     self.password, original_prompt='[$#>]',
>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>> .+:)')
>>   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line
>> 440, in login
>>     raise ExceptionPxssh('password refused')
>> pexpect.pxssh.ExceptionPxssh: password refused
>>                                          ^^^^^^^^^^^^^^
>>
>>
>>
>> On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
>> wrote:
>>
>>> Hello Ajit,
>>>
>>> There was a misconfiguration in our identity management system. The
>>> issue should now be resolved.
>>>
>>> Owen Hilyard
>>>
>>> On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
>>> dpdklab@iol.unh.edu> wrote:
>>>
>>>> David,
>>>> I can use my credentials to login to rhea.
>>>> But the same credentials are not allowing me to login onto the servers
>>>> you provided.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>>> Connection closed by 172.18.0.41 port 22
>>>>
>>>> ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>>> Connection closed by 172.18.0.40 port 22
>>>>
>>>> On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>
>>>>> Hi Ajit,
>>>>>
>>>>> Thanks for the update. Let me know if you need anything from me.
>>>>>
>>>>>
>>>>> Best,
>>>>> David
>>>>>
>>>>> On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>
>>>>>> David,
>>>>>> I had some trouble with my system over the past week.
>>>>>> So I could not login into the setup. I am hoping that my system will
>>>>>> be restored early this week
>>>>>> I will login into the setup after that.
>>>>>>
>>>>>> Thanks
>>>>>> Ajit
>>>>>>
>>>>>> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>
>>>>>>> Hi David,
>>>>>>> I actually did miss it. I will take a look.
>>>>>>> Thanks for the refresh mail.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Ajit
>>>>>>>
>>>>>>> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>>>> >
>>>>>>> > Hi Ajit,
>>>>>>> >
>>>>>>> > Ping again in case if you miss the last email.
>>>>>>> >
>>>>>>> > Did you have a chance to give it a try on login to the machines?
>>>>>>> >
>>>>>>> > Just some notes, we are currently running compile and some
>>>>>>> performance/functional testing.
>>>>>>> > Which it may be hard for you to run dts, so please let me know if
>>>>>>> you want me to stop these tests for you to debug.
>>>>>>> >
>>>>>>> > Step I run:
>>>>>>> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
>>>>>>> configuration to Broadcom
>>>>>>> > ./dts -s
>>>>>>>  #To start the dts
>>>>>>> >
>>>>>>> > Informatio:
>>>>>>> > arm-dell.dpdklab.iol.unh.edu
>>>>>>> > - dts - located in /opt/dts
>>>>>>> > - Trex - located in /opt/trex/v2.87
>>>>>>> >
>>>>>>> > arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>> > - dpdk - located in /root/dpdk
>>>>>>> >
>>>>>>> > Best,
>>>>>>> > David
>>>>>>> >
>>>>>>> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Also, I had changed the configuration when I run Broadcom nic.
>>>>>>> >> In the dts folder, you can start it with ./dts -s
>>>>>>> >>
>>>>>>> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> Hi Ajit,
>>>>>>> >>> Sorry for the late reply.
>>>>>>> >>>
>>>>>>> >>> You can ssh into the machine using the same VPN Username and
>>>>>>> Password.
>>>>>>> >>> Can you try and see if that works?
>>>>>>> >>>
>>>>>>> >>> Informatio:
>>>>>>> >>> arm-dell.dpdklab.iol.unh.edu
>>>>>>> >>> - dts - located in /opt/dts
>>>>>>> >>> - Trex - located in /opt/trex/v2.87
>>>>>>> >>>
>>>>>>> >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>> >>> - dpdk - located in /root/dpdk
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> Thanks,
>>>>>>> >>> David
>>>>>>> >>>
>>>>>>> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>> >>>>
>>>>>>> >>>> Hi David,
>>>>>>> >>>> Thanks for pinging me. Do you think I could login into the
>>>>>>> setup and
>>>>>>> >>>> see what is happening?
>>>>>>> >>>> I do have the credentials for two systems in your lab -
>>>>>>> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>>>>> >>>> If you can provide me details of the systems you are trying to
>>>>>>> >>>> configure, I could take a look.
>>>>>>> >>>>
>>>>>>> >>>> Thanks
>>>>>>> >>>> Ajit
>>>>>>> >>>>
>>>>>>> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>>>>>>> wrote:
>>>>>>> >>>> >
>>>>>>> >>>> > Hi Ajit,
>>>>>>> >>>> >
>>>>>>> >>>> > Thank you for helping out.
>>>>>>> >>>> >
>>>>>>> >>>> > How are you doing with this issue?
>>>>>>> >>>> >
>>>>>>> >>>> > Please let me know if you need anything information from us.
>>>>>>> >>>> >
>>>>>>> >>>> > Just some extra information that both Trex and dts are on the
>>>>>>> same intel tester machine.
>>>>>>> >>>> >
>>>>>>> >>>> > Best,
>>>>>>> >>>> > David
>>>>>>> >>>> >
>>>>>>> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>>>>>>> wrote:
>>>>>>> >>>> >>
>>>>>>> >>>> >> Yes, so this is the same NIC work on x86.
>>>>>>> >>>> >> By seeing the link, I am assuming you mean the ports that
>>>>>>> are using for Trex. If so they are UP before I start the Trex.
>>>>>>> >>>> >>
>>>>>>> >>>> >> Thank,
>>>>>>> >>>> >> David
>>>>>>> >>>> >>
>>>>>>> >>>> >>
>>>>>>> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>> >>>> >>>
>>>>>>> >>>> >>> Does the same NIC work on x86?
>>>>>>> >>>> >>> Also do you see link before you start the trex? I am trying
>>>>>>> to see if the port failed to successfully come up
>>>>>>> >>>> >>> or if there is a problem with the link. Can you check and
>>>>>>> tell?
>>>>>>> >>>> >>>
>>>>>>> >>>> >>> Thanks
>>>>>>> >>>> >>> Ajit
>>>>>>> >>>> >>>
>>>>>>> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
>>>>>>> wrote:
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> Hi Ajit,
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> Thank you for helping out.
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> We have it working on the x86, but not the Arm.
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> Thanks,
>>>>>>> >>>> >>>> David
>>>>>>> >>>> >>>>
>>>>>>> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>> Hi David,
>>>>>>> >>>> >>>>> I will take a look.
>>>>>>> >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>>>>> >>>> >>>>> even I would start with that to create a baseline and then
>>>>>>> >>>> >>>>> attempt on ARM.
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>> Thanks
>>>>>>> >>>> >>>>> Ajit
>>>>>>> >>>> >>>>>
>>>>>>> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>>>>>>> dliu@iol.unh.edu> wrote:
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> Hi Ajit,
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>>>>>>> machine.
>>>>>>> >>>> >>>>>> And I am running into a problem when I try to run
>>>>>>> nic_single_core_perf on the NIC.
>>>>>>> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>>>>>> http://git.dpdk.org/tools/dts/
>>>>>>> >>>> >>>>>> All the NICs are up and running when testing.
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> The problem is an error when I try to run the test case
>>>>>>> inside the nic_single_core_perf:
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>>>>>> framesize: 64, rxd/txd: 512
>>>>>>> >>>> >>>>>>>                dut.172.18.0.41:
>>>>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>>>> >>>> >>>>>>>                dut.172.18.0.41: start
>>>>>>> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>>>>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>>>>>> _execute_test_case
>>>>>>> >>>> >>>>>>>     case_obj()
>>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>>>>>> 196, in test_perf_nic_single_core
>>>>>>> >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>>>>>> 270, in perf_test
>>>>>>> >>>> >>>>>>>     _, packets_received =
>>>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>>>> options=traffic_opt)
>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245,
>>>>>>> in measure_throughput
>>>>>>> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778,
>>>>>>> in _prepare_transmission
>>>>>>> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>>>>> >>>> >>>>>>>   File
>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>>> line 51, in wrap2
>>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>>> >>>> >>>>>>>   File
>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>>> line 339, in reset
>>>>>>> >>>> >>>>>>>     self.clear_stats(ports)
>>>>>>> >>>> >>>>>>>   File
>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>>> line 51, in wrap2
>>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>>> >>>> >>>>>>>   File
>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>>> line 1492, in clear_stats
>>>>>>> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>>>>>> clear_xstats)
>>>>>>> >>>> >>>>>>>   File
>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>>>> line 2876, in _clear_stats_common
>>>>>>> >>>> >>>>>>>     raise TRexError(rc)
>>>>>>> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
>>>>>>> Failed to get server response from tcp://172.18.0.40:4501
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64
>>>>>>> -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> Then start sending traffic with command
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>> trex>start -f stl/imix.py
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> But this will cause an error to turn off Trex.
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>> trex>
>>>>>>> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
>>>>>>> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
>>>>>>> for more than 1.00135 seconds - timeout is 1 seconds
>>>>>>> >>>> >>>>>>> *** traceback follows ***
>>>>>>> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>>>>>> [0x561be173cf5a]
>>>>>>> >>>> >>>>>>> 2       0x7feea3ac0980
>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>>>> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>>>> >>>> >>>>>>> 4       0x561be16ce874
>>>>>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>>>>>> CVirtualIFPerSideStats*) + 3220
>>>>>>> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>>>>> >>>> >>>>>>> 6       0x561be171e3d9
>>>>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>>>> >>>> >>>>>>> 7       0x561be171f7ec
>>>>>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>>>>>> + 92
>>>>>>> >>>> >>>>>>> 8       0x561be171fc98
>>>>>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>>>>>> CFlowGenListPerThread*, bool) + 184
>>>>>>> >>>> >>>>>>> 9       0x561be16cb5f1 int
>>>>>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>>>>>> CFlowGenListPerThread*, double&) + 881
>>>>>>> >>>> >>>>>>> 10      0x561be1905212
>>>>>>> TrexStatelessDpCore::start_scheduler() + 226
>>>>>>> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>>>> >>>> >>>>>>> 12      0x561be1714113
>>>>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>>>> >>>> >>>>>>> 13      0x561be16af8dd
>>>>>>> CGlobalTRex::run_in_core(unsigned char) + 487
>>>>>>> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>>>>>> [0x561be16d11ad]
>>>>>>> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>>>> >>>> >>>>>>> 16      0x7feea3ab56db
>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>>>>> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>>>>> >>>> >>>>>>>
>>>>>>> >>>> >>>>>>> *** addr2line information follows ***
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> ??:0
>>>>>>> >>>> >>>>>>> '
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> I believe if everything is working, this should not turn
>>>>>>> off trex but please correct me if I am wrong.
>>>>>>> >>>> >>>>>> I wonder if you have any suggestion of how I can
>>>>>>> approach this issue?
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>> Best,
>>>>>>> >>>> >>>>>> David
>>>>>>> >>>> >>>>>>
>>>>>>> >>>> >>>>>>
>>>>>>>
>>>>>>

[-- Attachment #1.2: Type: text/html, Size: 25735 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4175 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-18 18:03                                 ` Ajit Khaparde
@ 2021-02-23  4:10                                   ` Ajit Khaparde
  2021-02-25 17:55                                     ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-02-23  4:10 UTC (permalink / raw)
  To: David Liu; +Cc: Owen Hilyard, ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 15561 bytes --]

David,
I tried the option to use sudo. And that didn't work.

Thanks
Ajit

ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident
will be reported.

----
sudo su
[sudo] password for ajitkhaparde:
ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be
reported.
ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
[sudo] password for ajitkhaparde:
ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be
reported.

On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

>
>
> On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> This is likely caused by when dts try to ssh into another machine as a
>> normal user which will prompt you for the password.
>>
>> One solution would be "sudo su" as root to run the command line since you
>> should have root access.
>>
> David,
> Let me try this method first.
> If it does not work, we could try with the public ssh keys.
> I will let you know.
>
> Thanks
> Ajit
>
>>
>> Another solution would be to let me add your public ssh key into both
>> machines.
>>
>> Let me know if this helps.
>>
>> Best,
>> David
>>
>> On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>>> Thanks Owen.
>>>
>>> When I run the dts command line suggested, I get the following error.
>>>
>>> Thanks
>>> Ajit
>>>
>>> Traceback (most recent call last):
>>>   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>>>     self.password, original_prompt='[$#>]',
>>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>>> .+:)')
>>>   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line
>>> 440, in login
>>>     raise ExceptionPxssh('password refused')
>>> pexpect.pxssh.ExceptionPxssh: password refused
>>>                                          ^^^^^^^^^^^^^^
>>>
>>>
>>>
>>> On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
>>> wrote:
>>>
>>>> Hello Ajit,
>>>>
>>>> There was a misconfiguration in our identity management system. The
>>>> issue should now be resolved.
>>>>
>>>> Owen Hilyard
>>>>
>>>> On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
>>>> dpdklab@iol.unh.edu> wrote:
>>>>
>>>>> David,
>>>>> I can use my credentials to login to rhea.
>>>>> But the same credentials are not allowing me to login onto the servers
>>>>> you provided.
>>>>>
>>>>> Thanks
>>>>> Ajit
>>>>>
>>>>> ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>>>> Connection closed by 172.18.0.41 port 22
>>>>>
>>>>> ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>>>> Connection closed by 172.18.0.40 port 22
>>>>>
>>>>> On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>>
>>>>>> Hi Ajit,
>>>>>>
>>>>>> Thanks for the update. Let me know if you need anything from me.
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> David
>>>>>>
>>>>>> On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>
>>>>>>> David,
>>>>>>> I had some trouble with my system over the past week.
>>>>>>> So I could not login into the setup. I am hoping that my system will
>>>>>>> be restored early this week
>>>>>>> I will login into the setup after that.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Ajit
>>>>>>>
>>>>>>> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>
>>>>>>>> Hi David,
>>>>>>>> I actually did miss it. I will take a look.
>>>>>>>> Thanks for the refresh mail.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Ajit
>>>>>>>>
>>>>>>>> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>>>>> >
>>>>>>>> > Hi Ajit,
>>>>>>>> >
>>>>>>>> > Ping again in case if you miss the last email.
>>>>>>>> >
>>>>>>>> > Did you have a chance to give it a try on login to the machines?
>>>>>>>> >
>>>>>>>> > Just some notes, we are currently running compile and some
>>>>>>>> performance/functional testing.
>>>>>>>> > Which it may be hard for you to run dts, so please let me know if
>>>>>>>> you want me to stop these tests for you to debug.
>>>>>>>> >
>>>>>>>> > Step I run:
>>>>>>>> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the
>>>>>>>> dts configuration to Broadcom
>>>>>>>> > ./dts -s
>>>>>>>>  #To start the dts
>>>>>>>> >
>>>>>>>> > Informatio:
>>>>>>>> > arm-dell.dpdklab.iol.unh.edu
>>>>>>>> > - dts - located in /opt/dts
>>>>>>>> > - Trex - located in /opt/trex/v2.87
>>>>>>>> >
>>>>>>>> > arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>>> > - dpdk - located in /root/dpdk
>>>>>>>> >
>>>>>>>> > Best,
>>>>>>>> > David
>>>>>>>> >
>>>>>>>> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> Also, I had changed the configuration when I run Broadcom nic.
>>>>>>>> >> In the dts folder, you can start it with ./dts -s
>>>>>>>> >>
>>>>>>>> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> Hi Ajit,
>>>>>>>> >>> Sorry for the late reply.
>>>>>>>> >>>
>>>>>>>> >>> You can ssh into the machine using the same VPN Username and
>>>>>>>> Password.
>>>>>>>> >>> Can you try and see if that works?
>>>>>>>> >>>
>>>>>>>> >>> Informatio:
>>>>>>>> >>> arm-dell.dpdklab.iol.unh.edu
>>>>>>>> >>> - dts - located in /opt/dts
>>>>>>>> >>> - Trex - located in /opt/trex/v2.87
>>>>>>>> >>>
>>>>>>>> >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>>> >>> - dpdk - located in /root/dpdk
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> Thanks,
>>>>>>>> >>> David
>>>>>>>> >>>
>>>>>>>> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Hi David,
>>>>>>>> >>>> Thanks for pinging me. Do you think I could login into the
>>>>>>>> setup and
>>>>>>>> >>>> see what is happening?
>>>>>>>> >>>> I do have the credentials for two systems in your lab -
>>>>>>>> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>>>>>> >>>> If you can provide me details of the systems you are trying to
>>>>>>>> >>>> configure, I could take a look.
>>>>>>>> >>>>
>>>>>>>> >>>> Thanks
>>>>>>>> >>>> Ajit
>>>>>>>> >>>>
>>>>>>>> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>>>>>>>> wrote:
>>>>>>>> >>>> >
>>>>>>>> >>>> > Hi Ajit,
>>>>>>>> >>>> >
>>>>>>>> >>>> > Thank you for helping out.
>>>>>>>> >>>> >
>>>>>>>> >>>> > How are you doing with this issue?
>>>>>>>> >>>> >
>>>>>>>> >>>> > Please let me know if you need anything information from us.
>>>>>>>> >>>> >
>>>>>>>> >>>> > Just some extra information that both Trex and dts are on
>>>>>>>> the same intel tester machine.
>>>>>>>> >>>> >
>>>>>>>> >>>> > Best,
>>>>>>>> >>>> > David
>>>>>>>> >>>> >
>>>>>>>> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>>>>>>>> wrote:
>>>>>>>> >>>> >>
>>>>>>>> >>>> >> Yes, so this is the same NIC work on x86.
>>>>>>>> >>>> >> By seeing the link, I am assuming you mean the ports that
>>>>>>>> are using for Trex. If so they are UP before I start the Trex.
>>>>>>>> >>>> >>
>>>>>>>> >>>> >> Thank,
>>>>>>>> >>>> >> David
>>>>>>>> >>>> >>
>>>>>>>> >>>> >>
>>>>>>>> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>> >>>> >>>
>>>>>>>> >>>> >>> Does the same NIC work on x86?
>>>>>>>> >>>> >>> Also do you see link before you start the trex? I am
>>>>>>>> trying to see if the port failed to successfully come up
>>>>>>>> >>>> >>> or if there is a problem with the link. Can you check and
>>>>>>>> tell?
>>>>>>>> >>>> >>>
>>>>>>>> >>>> >>> Thanks
>>>>>>>> >>>> >>> Ajit
>>>>>>>> >>>> >>>
>>>>>>>> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
>>>>>>>> dliu@iol.unh.edu> wrote:
>>>>>>>> >>>> >>>>
>>>>>>>> >>>> >>>> Hi Ajit,
>>>>>>>> >>>> >>>>
>>>>>>>> >>>> >>>> Thank you for helping out.
>>>>>>>> >>>> >>>>
>>>>>>>> >>>> >>>> We have it working on the x86, but not the Arm.
>>>>>>>> >>>> >>>>
>>>>>>>> >>>> >>>>
>>>>>>>> >>>> >>>> Thanks,
>>>>>>>> >>>> >>>> David
>>>>>>>> >>>> >>>>
>>>>>>>> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>> >>>> >>>>>
>>>>>>>> >>>> >>>>> Hi David,
>>>>>>>> >>>> >>>>> I will take a look.
>>>>>>>> >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>>>>>> >>>> >>>>> even I would start with that to create a baseline and
>>>>>>>> then
>>>>>>>> >>>> >>>>> attempt on ARM.
>>>>>>>> >>>> >>>>>
>>>>>>>> >>>> >>>>> Thanks
>>>>>>>> >>>> >>>>> Ajit
>>>>>>>> >>>> >>>>>
>>>>>>>> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>>>>>>>> dliu@iol.unh.edu> wrote:
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>> Hi Ajit,
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>>>>>>>> machine.
>>>>>>>> >>>> >>>>>> And I am running into a problem when I try to run
>>>>>>>> nic_single_core_perf on the NIC.
>>>>>>>> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>>>>>>> http://git.dpdk.org/tools/dts/
>>>>>>>> >>>> >>>>>> All the NICs are up and running when testing.
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>> The problem is an error when I try to run the test case
>>>>>>>> inside the nic_single_core_perf:
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>>>>>>> framesize: 64, rxd/txd: 512
>>>>>>>> >>>> >>>>>>>                dut.172.18.0.41:
>>>>>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>>>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>>>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>>>>> >>>> >>>>>>>                dut.172.18.0.41: start
>>>>>>>> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>>>>>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>>>>>>> _execute_test_case
>>>>>>>> >>>> >>>>>>>     case_obj()
>>>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>>>>>>> 196, in test_perf_nic_single_core
>>>>>>>> >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>>>>>>> 270, in perf_test
>>>>>>>> >>>> >>>>>>>     _, packets_received =
>>>>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>>>>> options=traffic_opt)
>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245,
>>>>>>>> in measure_throughput
>>>>>>>> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778,
>>>>>>>> in _prepare_transmission
>>>>>>>> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>>>>>> >>>> >>>>>>>   File
>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>>>> line 51, in wrap2
>>>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>>>> >>>> >>>>>>>   File
>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>>>> line 339, in reset
>>>>>>>> >>>> >>>>>>>     self.clear_stats(ports)
>>>>>>>> >>>> >>>>>>>   File
>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>>>> line 51, in wrap2
>>>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>>>> >>>> >>>>>>>   File
>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>>>> line 1492, in clear_stats
>>>>>>>> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>>>>>>> clear_xstats)
>>>>>>>> >>>> >>>>>>>   File
>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>>>>> line 2876, in _clear_stats_common
>>>>>>>> >>>> >>>>>>>     raise TRexError(rc)
>>>>>>>> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
>>>>>>>> Failed to get server response from tcp://172.18.0.40:4501
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64
>>>>>>>> -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>> Then start sending traffic with command
>>>>>>>> >>>> >>>>>>>
>>>>>>>> >>>> >>>>>>> trex>start -f stl/imix.py
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>> But this will cause an error to turn off Trex.
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>>> trex>
>>>>>>>> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
>>>>>>>> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
>>>>>>>> for more than 1.00135 seconds - timeout is 1 seconds
>>>>>>>> >>>> >>>>>>> *** traceback follows ***
>>>>>>>> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>>>>>>> [0x561be173cf5a]
>>>>>>>> >>>> >>>>>>> 2       0x7feea3ac0980
>>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>>>>> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>>>>> >>>> >>>>>>> 4       0x561be16ce874
>>>>>>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>>>>>>> CVirtualIFPerSideStats*) + 3220
>>>>>>>> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() +
>>>>>>>> 31
>>>>>>>> >>>> >>>>>>> 6       0x561be171e3d9
>>>>>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>>>>> >>>> >>>>>>> 7       0x561be171f7ec
>>>>>>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>>>>>>> + 92
>>>>>>>> >>>> >>>>>>> 8       0x561be171fc98
>>>>>>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>>>>>>> CFlowGenListPerThread*, bool) + 184
>>>>>>>> >>>> >>>>>>> 9       0x561be16cb5f1 int
>>>>>>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>>>>>>> CFlowGenListPerThread*, double&) + 881
>>>>>>>> >>>> >>>>>>> 10      0x561be1905212
>>>>>>>> TrexStatelessDpCore::start_scheduler() + 226
>>>>>>>> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>>>>> >>>> >>>>>>> 12      0x561be1714113
>>>>>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>>>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>>>>> >>>> >>>>>>> 13      0x561be16af8dd
>>>>>>>> CGlobalTRex::run_in_core(unsigned char) + 487
>>>>>>>> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>>>>>>> [0x561be16d11ad]
>>>>>>>> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>>>>> >>>> >>>>>>> 16      0x7feea3ab56db
>>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>>>>>> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>>>>>> >>>> >>>>>>>
>>>>>>>> >>>> >>>>>>> *** addr2line information follows ***
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>> >>>> >>>>>>> '
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>> I believe if everything is working, this should not
>>>>>>>> turn off trex but please correct me if I am wrong.
>>>>>>>> >>>> >>>>>> I wonder if you have any suggestion of how I can
>>>>>>>> approach this issue?
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>> Best,
>>>>>>>> >>>> >>>>>> David
>>>>>>>> >>>> >>>>>>
>>>>>>>> >>>> >>>>>>
>>>>>>>>
>>>>>>>

[-- Attachment #1.2: Type: text/html, Size: 27348 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-23  4:10                                   ` Ajit Khaparde
@ 2021-02-25 17:55                                     ` David Liu
  2021-02-25 18:46                                       ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-02-25 17:55 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 16222 bytes --]

Hi Ajit,

I changed the permission to allow you to run sudo.
This should work now.
Let me know if you still have issues.

Best,
David

On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> David,
> I tried the option to use sudo. And that didn't work.
>
> Thanks
> Ajit
>
> ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident
> will be reported.
>
> ----
> sudo su
> [sudo] password for ajitkhaparde:
> ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will
> be reported.
> ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> [sudo] password for ajitkhaparde:
> ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will
> be reported.
>
> On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>>
>>
>> On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>>
>>> Hi Ajit,
>>>
>>> This is likely caused by when dts try to ssh into another machine as a
>>> normal user which will prompt you for the password.
>>>
>>> One solution would be "sudo su" as root to run the command line since
>>> you should have root access.
>>>
>> David,
>> Let me try this method first.
>> If it does not work, we could try with the public ssh keys.
>> I will let you know.
>>
>> Thanks
>> Ajit
>>
>>>
>>> Another solution would be to let me add your public ssh key into both
>>> machines.
>>>
>>> Let me know if this helps.
>>>
>>> Best,
>>> David
>>>
>>> On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>>
>>>> Thanks Owen.
>>>>
>>>> When I run the dts command line suggested, I get the following error.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> Traceback (most recent call last):
>>>>   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>>>>     self.password, original_prompt='[$#>]',
>>>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>>>> .+:)')
>>>>   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line
>>>> 440, in login
>>>>     raise ExceptionPxssh('password refused')
>>>> pexpect.pxssh.ExceptionPxssh: password refused
>>>>                                          ^^^^^^^^^^^^^^
>>>>
>>>>
>>>>
>>>> On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
>>>> wrote:
>>>>
>>>>> Hello Ajit,
>>>>>
>>>>> There was a misconfiguration in our identity management system. The
>>>>> issue should now be resolved.
>>>>>
>>>>> Owen Hilyard
>>>>>
>>>>> On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
>>>>> dpdklab@iol.unh.edu> wrote:
>>>>>
>>>>>> David,
>>>>>> I can use my credentials to login to rhea.
>>>>>> But the same credentials are not allowing me to login onto the
>>>>>> servers you provided.
>>>>>>
>>>>>> Thanks
>>>>>> Ajit
>>>>>>
>>>>>> ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>>>>> Connection closed by 172.18.0.41 port 22
>>>>>>
>>>>>> ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>>>>> Connection closed by 172.18.0.40 port 22
>>>>>>
>>>>>> On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>>>
>>>>>>> Hi Ajit,
>>>>>>>
>>>>>>> Thanks for the update. Let me know if you need anything from me.
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> David
>>>>>>>
>>>>>>> On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>
>>>>>>>> David,
>>>>>>>> I had some trouble with my system over the past week.
>>>>>>>> So I could not login into the setup. I am hoping that my system
>>>>>>>> will be restored early this week
>>>>>>>> I will login into the setup after that.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Ajit
>>>>>>>>
>>>>>>>> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>
>>>>>>>>> Hi David,
>>>>>>>>> I actually did miss it. I will take a look.
>>>>>>>>> Thanks for the refresh mail.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Ajit
>>>>>>>>>
>>>>>>>>> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>>>>>> >
>>>>>>>>> > Hi Ajit,
>>>>>>>>> >
>>>>>>>>> > Ping again in case if you miss the last email.
>>>>>>>>> >
>>>>>>>>> > Did you have a chance to give it a try on login to the machines?
>>>>>>>>> >
>>>>>>>>> > Just some notes, we are currently running compile and some
>>>>>>>>> performance/functional testing.
>>>>>>>>> > Which it may be hard for you to run dts, so please let me know
>>>>>>>>> if you want me to stop these tests for you to debug.
>>>>>>>>> >
>>>>>>>>> > Step I run:
>>>>>>>>> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the
>>>>>>>>> dts configuration to Broadcom
>>>>>>>>> > ./dts -s
>>>>>>>>>    #To start the dts
>>>>>>>>> >
>>>>>>>>> > Informatio:
>>>>>>>>> > arm-dell.dpdklab.iol.unh.edu
>>>>>>>>> > - dts - located in /opt/dts
>>>>>>>>> > - Trex - located in /opt/trex/v2.87
>>>>>>>>> >
>>>>>>>>> > arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>>>> > - dpdk - located in /root/dpdk
>>>>>>>>> >
>>>>>>>>> > Best,
>>>>>>>>> > David
>>>>>>>>> >
>>>>>>>>> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu>
>>>>>>>>> wrote:
>>>>>>>>> >>
>>>>>>>>> >> Also, I had changed the configuration when I run Broadcom nic.
>>>>>>>>> >> In the dts folder, you can start it with ./dts -s
>>>>>>>>> >>
>>>>>>>>> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
>>>>>>>>> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> Hi Ajit,
>>>>>>>>> >>> Sorry for the late reply.
>>>>>>>>> >>>
>>>>>>>>> >>> You can ssh into the machine using the same VPN Username and
>>>>>>>>> Password.
>>>>>>>>> >>> Can you try and see if that works?
>>>>>>>>> >>>
>>>>>>>>> >>> Informatio:
>>>>>>>>> >>> arm-dell.dpdklab.iol.unh.edu
>>>>>>>>> >>> - dts - located in /opt/dts
>>>>>>>>> >>> - Trex - located in /opt/trex/v2.87
>>>>>>>>> >>>
>>>>>>>>> >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>>>> >>> - dpdk - located in /root/dpdk
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> Thanks,
>>>>>>>>> >>> David
>>>>>>>>> >>>
>>>>>>>>> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> Hi David,
>>>>>>>>> >>>> Thanks for pinging me. Do you think I could login into the
>>>>>>>>> setup and
>>>>>>>>> >>>> see what is happening?
>>>>>>>>> >>>> I do have the credentials for two systems in your lab -
>>>>>>>>> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>>>>>>> >>>> If you can provide me details of the systems you are trying to
>>>>>>>>> >>>> configure, I could take a look.
>>>>>>>>> >>>>
>>>>>>>>> >>>> Thanks
>>>>>>>>> >>>> Ajit
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>>>>>>>>> wrote:
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > Hi Ajit,
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > Thank you for helping out.
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > How are you doing with this issue?
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > Please let me know if you need anything information from us.
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > Just some extra information that both Trex and dts are on
>>>>>>>>> the same intel tester machine.
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > Best,
>>>>>>>>> >>>> > David
>>>>>>>>> >>>> >
>>>>>>>>> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>>>>>>>>> wrote:
>>>>>>>>> >>>> >>
>>>>>>>>> >>>> >> Yes, so this is the same NIC work on x86.
>>>>>>>>> >>>> >> By seeing the link, I am assuming you mean the ports that
>>>>>>>>> are using for Trex. If so they are UP before I start the Trex.
>>>>>>>>> >>>> >>
>>>>>>>>> >>>> >> Thank,
>>>>>>>>> >>>> >> David
>>>>>>>>> >>>> >>
>>>>>>>>> >>>> >>
>>>>>>>>> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>> >>>> >>>
>>>>>>>>> >>>> >>> Does the same NIC work on x86?
>>>>>>>>> >>>> >>> Also do you see link before you start the trex? I am
>>>>>>>>> trying to see if the port failed to successfully come up
>>>>>>>>> >>>> >>> or if there is a problem with the link. Can you check and
>>>>>>>>> tell?
>>>>>>>>> >>>> >>>
>>>>>>>>> >>>> >>> Thanks
>>>>>>>>> >>>> >>> Ajit
>>>>>>>>> >>>> >>>
>>>>>>>>> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
>>>>>>>>> dliu@iol.unh.edu> wrote:
>>>>>>>>> >>>> >>>>
>>>>>>>>> >>>> >>>> Hi Ajit,
>>>>>>>>> >>>> >>>>
>>>>>>>>> >>>> >>>> Thank you for helping out.
>>>>>>>>> >>>> >>>>
>>>>>>>>> >>>> >>>> We have it working on the x86, but not the Arm.
>>>>>>>>> >>>> >>>>
>>>>>>>>> >>>> >>>>
>>>>>>>>> >>>> >>>> Thanks,
>>>>>>>>> >>>> >>>> David
>>>>>>>>> >>>> >>>>
>>>>>>>>> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>> >>>> >>>>>
>>>>>>>>> >>>> >>>>> Hi David,
>>>>>>>>> >>>> >>>>> I will take a look.
>>>>>>>>> >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>>>>>>> >>>> >>>>> even I would start with that to create a baseline and
>>>>>>>>> then
>>>>>>>>> >>>> >>>>> attempt on ARM.
>>>>>>>>> >>>> >>>>>
>>>>>>>>> >>>> >>>>> Thanks
>>>>>>>>> >>>> >>>>> Ajit
>>>>>>>>> >>>> >>>>>
>>>>>>>>> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>>>>>>>>> dliu@iol.unh.edu> wrote:
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>> Hi Ajit,
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>>>>>>>>> machine.
>>>>>>>>> >>>> >>>>>> And I am running into a problem when I try to run
>>>>>>>>> nic_single_core_perf on the NIC.
>>>>>>>>> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>>>>>>>> http://git.dpdk.org/tools/dts/
>>>>>>>>> >>>> >>>>>> All the NICs are up and running when testing.
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>> The problem is an error when I try to run the test
>>>>>>>>> case inside the nic_single_core_perf:
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>>>>>>>> framesize: 64, rxd/txd: 512
>>>>>>>>> >>>> >>>>>>>                dut.172.18.0.41:
>>>>>>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>>>>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>>>>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>>>>>> >>>> >>>>>>>                dut.172.18.0.41: start
>>>>>>>>> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>>>>>>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319,
>>>>>>>>> in _execute_test_case
>>>>>>>>> >>>> >>>>>>>     case_obj()
>>>>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py",
>>>>>>>>> line 196, in test_perf_nic_single_core
>>>>>>>>> >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py",
>>>>>>>>> line 270, in perf_test
>>>>>>>>> >>>> >>>>>>>     _, packets_received =
>>>>>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>>>>>> options=traffic_opt)
>>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245,
>>>>>>>>> in measure_throughput
>>>>>>>>> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778,
>>>>>>>>> in _prepare_transmission
>>>>>>>>> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>>>>> line 51, in wrap2
>>>>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>>>>> line 339, in reset
>>>>>>>>> >>>> >>>>>>>     self.clear_stats(ports)
>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>>>>> line 51, in wrap2
>>>>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>>>>> line 1492, in clear_stats
>>>>>>>>> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>>>>>>>> clear_xstats)
>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>>>>>> line 2876, in _clear_stats_common
>>>>>>>>> >>>> >>>>>>>     raise TRexError(rc)
>>>>>>>>> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
>>>>>>>>> Failed to get server response from tcp://172.18.0.40:4501
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>> I also try to run TRex in stateless mode
>>>>>>>>> with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>> Then start sending traffic with command
>>>>>>>>> >>>> >>>>>>>
>>>>>>>>> >>>> >>>>>>> trex>start -f stl/imix.py
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>> But this will cause an error to turn off Trex.
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>>> trex>
>>>>>>>>> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
>>>>>>>>> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
>>>>>>>>> for more than 1.00135 seconds - timeout is 1 seconds
>>>>>>>>> >>>> >>>>>>> *** traceback follows ***
>>>>>>>>> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>>>>>>>> [0x561be173cf5a]
>>>>>>>>> >>>> >>>>>>> 2       0x7feea3ac0980
>>>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>>>>>> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>>>>>> >>>> >>>>>>> 4       0x561be16ce874
>>>>>>>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>>>>>>>> CVirtualIFPerSideStats*) + 3220
>>>>>>>>> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() +
>>>>>>>>> 31
>>>>>>>>> >>>> >>>>>>> 6       0x561be171e3d9
>>>>>>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>>>>>> >>>> >>>>>>> 7       0x561be171f7ec
>>>>>>>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>>>>>>>> + 92
>>>>>>>>> >>>> >>>>>>> 8       0x561be171fc98
>>>>>>>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>>>>>>>> CFlowGenListPerThread*, bool) + 184
>>>>>>>>> >>>> >>>>>>> 9       0x561be16cb5f1 int
>>>>>>>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>>>>>>>> CFlowGenListPerThread*, double&) + 881
>>>>>>>>> >>>> >>>>>>> 10      0x561be1905212
>>>>>>>>> TrexStatelessDpCore::start_scheduler() + 226
>>>>>>>>> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>>>>>> >>>> >>>>>>> 12      0x561be1714113
>>>>>>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>>>>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>>>>>> >>>> >>>>>>> 13      0x561be16af8dd
>>>>>>>>> CGlobalTRex::run_in_core(unsigned char) + 487
>>>>>>>>> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>>>>>>>> [0x561be16d11ad]
>>>>>>>>> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>>>>>> >>>> >>>>>>> 16      0x7feea3ab56db
>>>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>>>>>>> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>>>>>>> >>>> >>>>>>>
>>>>>>>>> >>>> >>>>>>> *** addr2line information follows ***
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>> >>>> >>>>>>> '
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>> I believe if everything is working, this should not
>>>>>>>>> turn off trex but please correct me if I am wrong.
>>>>>>>>> >>>> >>>>>> I wonder if you have any suggestion of how I can
>>>>>>>>> approach this issue?
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>> Best,
>>>>>>>>> >>>> >>>>>> David
>>>>>>>>> >>>> >>>>>>
>>>>>>>>> >>>> >>>>>>
>>>>>>>>>
>>>>>>>>

[-- Attachment #2: Type: text/html, Size: 27949 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-25 17:55                                     ` David Liu
@ 2021-02-25 18:46                                       ` David Liu
  2021-03-01 15:00                                         ` Juraj Linkeš
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-02-25 18:46 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 16975 bytes --]

Hi Ajit,

Sorry for this issue.
If possible I would like to set up a time where we can do this in a call to
help get this solved faster.
Please let me know what would be a good time for you.

Best,
David


On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit,
>
> I changed the permission to allow you to run sudo.
> This should work now.
> Let me know if you still have issues.
>
> Best,
> David
>
> On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> David,
>> I tried the option to use sudo. And that didn't work.
>>
>> Thanks
>> Ajit
>>
>> ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident
>> will be reported.
>>
>> ----
>> sudo su
>> [sudo] password for ajitkhaparde:
>> ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will
>> be reported.
>> ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> [sudo] password for ajitkhaparde:
>> ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will
>> be reported.
>>
>> On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>>
>>>
>>>
>>> On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>>>
>>>> Hi Ajit,
>>>>
>>>> This is likely caused by when dts try to ssh into another machine as a
>>>> normal user which will prompt you for the password.
>>>>
>>>> One solution would be "sudo su" as root to run the command line since
>>>> you should have root access.
>>>>
>>> David,
>>> Let me try this method first.
>>> If it does not work, we could try with the public ssh keys.
>>> I will let you know.
>>>
>>> Thanks
>>> Ajit
>>>
>>>>
>>>> Another solution would be to let me add your public ssh key into both
>>>> machines.
>>>>
>>>> Let me know if this helps.
>>>>
>>>> Best,
>>>> David
>>>>
>>>> On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>>
>>>>> Thanks Owen.
>>>>>
>>>>> When I run the dts command line suggested, I get the following error.
>>>>>
>>>>> Thanks
>>>>> Ajit
>>>>>
>>>>> Traceback (most recent call last):
>>>>>   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>>>>>     self.password, original_prompt='[$#>]',
>>>>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>>>>> .+:)')
>>>>>   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line
>>>>> 440, in login
>>>>>     raise ExceptionPxssh('password refused')
>>>>> pexpect.pxssh.ExceptionPxssh: password refused
>>>>>                                          ^^^^^^^^^^^^^^
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
>>>>> wrote:
>>>>>
>>>>>> Hello Ajit,
>>>>>>
>>>>>> There was a misconfiguration in our identity management system. The
>>>>>> issue should now be resolved.
>>>>>>
>>>>>> Owen Hilyard
>>>>>>
>>>>>> On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
>>>>>> dpdklab@iol.unh.edu> wrote:
>>>>>>
>>>>>>> David,
>>>>>>> I can use my credentials to login to rhea.
>>>>>>> But the same credentials are not allowing me to login onto the
>>>>>>> servers you provided.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Ajit
>>>>>>>
>>>>>>> ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>> Connection closed by 172.18.0.41 port 22
>>>>>>>
>>>>>>> ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>>>>>> Connection closed by 172.18.0.40 port 22
>>>>>>>
>>>>>>> On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>>>>
>>>>>>>> Hi Ajit,
>>>>>>>>
>>>>>>>> Thanks for the update. Let me know if you need anything from me.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> David
>>>>>>>>
>>>>>>>> On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>
>>>>>>>>> David,
>>>>>>>>> I had some trouble with my system over the past week.
>>>>>>>>> So I could not login into the setup. I am hoping that my system
>>>>>>>>> will be restored early this week
>>>>>>>>> I will login into the setup after that.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Ajit
>>>>>>>>>
>>>>>>>>> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi David,
>>>>>>>>>> I actually did miss it. I will take a look.
>>>>>>>>>> Thanks for the refresh mail.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Ajit
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu>
>>>>>>>>>> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Hi Ajit,
>>>>>>>>>> >
>>>>>>>>>> > Ping again in case if you miss the last email.
>>>>>>>>>> >
>>>>>>>>>> > Did you have a chance to give it a try on login to the machines?
>>>>>>>>>> >
>>>>>>>>>> > Just some notes, we are currently running compile and some
>>>>>>>>>> performance/functional testing.
>>>>>>>>>> > Which it may be hard for you to run dts, so please let me know
>>>>>>>>>> if you want me to stop these tests for you to debug.
>>>>>>>>>> >
>>>>>>>>>> > Step I run:
>>>>>>>>>> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the
>>>>>>>>>> dts configuration to Broadcom
>>>>>>>>>> > ./dts -s
>>>>>>>>>>    #To start the dts
>>>>>>>>>> >
>>>>>>>>>> > Informatio:
>>>>>>>>>> > arm-dell.dpdklab.iol.unh.edu
>>>>>>>>>> > - dts - located in /opt/dts
>>>>>>>>>> > - Trex - located in /opt/trex/v2.87
>>>>>>>>>> >
>>>>>>>>>> > arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>>>>> > - dpdk - located in /root/dpdk
>>>>>>>>>> >
>>>>>>>>>> > Best,
>>>>>>>>>> > David
>>>>>>>>>> >
>>>>>>>>>> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu>
>>>>>>>>>> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> Also, I had changed the configuration when I run Broadcom nic.
>>>>>>>>>> >> In the dts folder, you can start it with ./dts -s
>>>>>>>>>> >>
>>>>>>>>>> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
>>>>>>>>>> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>> Hi Ajit,
>>>>>>>>>> >>> Sorry for the late reply.
>>>>>>>>>> >>>
>>>>>>>>>> >>> You can ssh into the machine using the same VPN Username and
>>>>>>>>>> Password.
>>>>>>>>>> >>> Can you try and see if that works?
>>>>>>>>>> >>>
>>>>>>>>>> >>> Informatio:
>>>>>>>>>> >>> arm-dell.dpdklab.iol.unh.edu
>>>>>>>>>> >>> - dts - located in /opt/dts
>>>>>>>>>> >>> - Trex - located in /opt/trex/v2.87
>>>>>>>>>> >>>
>>>>>>>>>> >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>>>>>>>> >>> - dpdk - located in /root/dpdk
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>> Thanks,
>>>>>>>>>> >>> David
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Hi David,
>>>>>>>>>> >>>> Thanks for pinging me. Do you think I could login into the
>>>>>>>>>> setup and
>>>>>>>>>> >>>> see what is happening?
>>>>>>>>>> >>>> I do have the credentials for two systems in your lab -
>>>>>>>>>> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>>>>>>>> >>>> If you can provide me details of the systems you are trying
>>>>>>>>>> to
>>>>>>>>>> >>>> configure, I could take a look.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Thanks
>>>>>>>>>> >>>> Ajit
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>>>>>>>>>> wrote:
>>>>>>>>>> >>>> >
>>>>>>>>>> >>>> > Hi Ajit,
>>>>>>>>>> >>>> >
>>>>>>>>>> >>>> > Thank you for helping out.
>>>>>>>>>> >>>> >
>>>>>>>>>> >>>> > How are you doing with this issue?
>>>>>>>>>> >>>> >
>>>>>>>>>> >>>> > Please let me know if you need anything information from
>>>>>>>>>> us.
>>>>>>>>>> >>>> >
>>>>>>>>>> >>>> > Just some extra information that both Trex and dts are on
>>>>>>>>>> the same intel tester machine.
>>>>>>>>>> >>>> >
>>>>>>>>>> >>>> > Best,
>>>>>>>>>> >>>> > David
>>>>>>>>>> >>>> >
>>>>>>>>>> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
>>>>>>>>>> dliu@iol.unh.edu> wrote:
>>>>>>>>>> >>>> >>
>>>>>>>>>> >>>> >> Yes, so this is the same NIC work on x86.
>>>>>>>>>> >>>> >> By seeing the link, I am assuming you mean the ports that
>>>>>>>>>> are using for Trex. If so they are UP before I start the Trex.
>>>>>>>>>> >>>> >>
>>>>>>>>>> >>>> >> Thank,
>>>>>>>>>> >>>> >> David
>>>>>>>>>> >>>> >>
>>>>>>>>>> >>>> >>
>>>>>>>>>> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>>> >>>> >>>
>>>>>>>>>> >>>> >>> Does the same NIC work on x86?
>>>>>>>>>> >>>> >>> Also do you see link before you start the trex? I am
>>>>>>>>>> trying to see if the port failed to successfully come up
>>>>>>>>>> >>>> >>> or if there is a problem with the link. Can you check
>>>>>>>>>> and tell?
>>>>>>>>>> >>>> >>>
>>>>>>>>>> >>>> >>> Thanks
>>>>>>>>>> >>>> >>> Ajit
>>>>>>>>>> >>>> >>>
>>>>>>>>>> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
>>>>>>>>>> dliu@iol.unh.edu> wrote:
>>>>>>>>>> >>>> >>>>
>>>>>>>>>> >>>> >>>> Hi Ajit,
>>>>>>>>>> >>>> >>>>
>>>>>>>>>> >>>> >>>> Thank you for helping out.
>>>>>>>>>> >>>> >>>>
>>>>>>>>>> >>>> >>>> We have it working on the x86, but not the Arm.
>>>>>>>>>> >>>> >>>>
>>>>>>>>>> >>>> >>>>
>>>>>>>>>> >>>> >>>> Thanks,
>>>>>>>>>> >>>> >>>> David
>>>>>>>>>> >>>> >>>>
>>>>>>>>>> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>>>>>>>>> ajit.khaparde@broadcom.com> wrote:
>>>>>>>>>> >>>> >>>>>
>>>>>>>>>> >>>> >>>>> Hi David,
>>>>>>>>>> >>>> >>>>> I will take a look.
>>>>>>>>>> >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>>>>>>>> >>>> >>>>> even I would start with that to create a baseline and
>>>>>>>>>> then
>>>>>>>>>> >>>> >>>>> attempt on ARM.
>>>>>>>>>> >>>> >>>>>
>>>>>>>>>> >>>> >>>>> Thanks
>>>>>>>>>> >>>> >>>>> Ajit
>>>>>>>>>> >>>> >>>>>
>>>>>>>>>> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>>>>>>>>>> dliu@iol.unh.edu> wrote:
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>> Hi Ajit,
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an
>>>>>>>>>> Arm machine.
>>>>>>>>>> >>>> >>>>>> And I am running into a problem when I try to run
>>>>>>>>>> nic_single_core_perf on the NIC.
>>>>>>>>>> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>>>>>>>>> http://git.dpdk.org/tools/dts/
>>>>>>>>>> >>>> >>>>>> All the NICs are up and running when testing.
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>> The problem is an error when I try to run the test
>>>>>>>>>> case inside the nic_single_core_perf:
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>>>>>>>>> framesize: 64, rxd/txd: 512
>>>>>>>>>> >>>> >>>>>>>                dut.172.18.0.41:
>>>>>>>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>>>>>>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>>>>>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>>>>>>> >>>> >>>>>>>                dut.172.18.0.41: start
>>>>>>>>>> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>>>>>>>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319,
>>>>>>>>>> in _execute_test_case
>>>>>>>>>> >>>> >>>>>>>     case_obj()
>>>>>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py",
>>>>>>>>>> line 196, in test_perf_nic_single_core
>>>>>>>>>> >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>>>>>>>> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py",
>>>>>>>>>> line 270, in perf_test
>>>>>>>>>> >>>> >>>>>>>     _, packets_received =
>>>>>>>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>>>>>>>> options=traffic_opt)
>>>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line
>>>>>>>>>> 245, in measure_throughput
>>>>>>>>>> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>>>>>>> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line
>>>>>>>>>> 778, in _prepare_transmission
>>>>>>>>>> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>>>>>> line 51, in wrap2
>>>>>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>>>>>> line 339, in reset
>>>>>>>>>> >>>> >>>>>>>     self.clear_stats(ports)
>>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>>>>>>>> line 51, in wrap2
>>>>>>>>>> >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>>>>>>>> line 1492, in clear_stats
>>>>>>>>>> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>>>>>>>>> clear_xstats)
>>>>>>>>>> >>>> >>>>>>>   File
>>>>>>>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>>>>>>>> line 2876, in _clear_stats_common
>>>>>>>>>> >>>> >>>>>>>     raise TRexError(rc)
>>>>>>>>>> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
>>>>>>>>>> Failed to get server response from tcp://172.18.0.40:4501
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>> I also try to run TRex in stateless mode
>>>>>>>>>> with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>> Then start sending traffic with command
>>>>>>>>>> >>>> >>>>>>>
>>>>>>>>>> >>>> >>>>>>> trex>start -f stl/imix.py
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>> But this will cause an error to turn off Trex.
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>>> trex>
>>>>>>>>>> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server
>>>>>>>>>> has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not
>>>>>>>>>> responded for more than 1.00135 seconds - timeout is 1 seconds
>>>>>>>>>> >>>> >>>>>>> *** traceback follows ***
>>>>>>>>>> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>>>>>>>>> [0x561be173cf5a]
>>>>>>>>>> >>>> >>>>>>> 2       0x7feea3ac0980
>>>>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>>>>>>> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>>>>>>> >>>> >>>>>>> 4       0x561be16ce874
>>>>>>>>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>>>>>>>>> CVirtualIFPerSideStats*) + 3220
>>>>>>>>>> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue()
>>>>>>>>>> + 31
>>>>>>>>>> >>>> >>>>>>> 6       0x561be171e3d9
>>>>>>>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>>>>>>> >>>> >>>>>>> 7       0x561be171f7ec
>>>>>>>>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>>>>>>>>> + 92
>>>>>>>>>> >>>> >>>>>>> 8       0x561be171fc98
>>>>>>>>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>>>>>>>>> CFlowGenListPerThread*, bool) + 184
>>>>>>>>>> >>>> >>>>>>> 9       0x561be16cb5f1 int
>>>>>>>>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>>>>>>>>> CFlowGenListPerThread*, double&) + 881
>>>>>>>>>> >>>> >>>>>>> 10      0x561be1905212
>>>>>>>>>> TrexStatelessDpCore::start_scheduler() + 226
>>>>>>>>>> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>>>>>>> >>>> >>>>>>> 12      0x561be1714113
>>>>>>>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>>>>>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>>>>>>> >>>> >>>>>>> 13      0x561be16af8dd
>>>>>>>>>> CGlobalTRex::run_in_core(unsigned char) + 487
>>>>>>>>>> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>>>>>>>>> [0x561be16d11ad]
>>>>>>>>>> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>>>>>>> >>>> >>>>>>> 16      0x7feea3ab56db
>>>>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>>>>>>>> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>>>>>>>> >>>> >>>>>>>
>>>>>>>>>> >>>> >>>>>>> *** addr2line information follows ***
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> ??:0
>>>>>>>>>> >>>> >>>>>>> '
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>> I believe if everything is working, this should not
>>>>>>>>>> turn off trex but please correct me if I am wrong.
>>>>>>>>>> >>>> >>>>>> I wonder if you have any suggestion of how I can
>>>>>>>>>> approach this issue?
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>> Best,
>>>>>>>>>> >>>> >>>>>> David
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>> >>>> >>>>>>
>>>>>>>>>>
>>>>>>>>>

[-- Attachment #2: Type: text/html, Size: 28610 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-02-25 18:46                                       ` David Liu
@ 2021-03-01 15:00                                         ` Juraj Linkeš
  2021-03-01 15:12                                           ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: Juraj Linkeš @ 2021-03-01 15:00 UTC (permalink / raw)
  To: David Liu, Ajit Khaparde; +Cc: Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 14708 bytes --]

Hi Ajit, David,

If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.

Thanks,
Juraj

From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
Sent: Thursday, February 25, 2021 7:47 PM
To: Ajit Khaparde <ajit.khaparde@broadcom.com>
Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware

Hi Ajit,

Sorry for this issue.
If possible I would like to set up a time where we can do this in a call to help get this solved faster.
Please let me know what would be a good time for you.

Best,
David


On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
Hi Ajit,

I changed the permission to allow you to run sudo.
This should work now.
Let me know if you still have issues.

Best,
David

On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
David,
I tried the option to use sudo. And that didn't work.

Thanks
Ajit

ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.

----
sudo su
[sudo] password for ajitkhaparde:
ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
[sudo] password for ajitkhaparde:
ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.

On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:


On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
Hi Ajit,

This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.

One solution would be "sudo su" as root to run the command line since you should have root access.
David,
Let me try this method first.
If it does not work, we could try with the public ssh keys.
I will let you know.

Thanks
Ajit

Another solution would be to let me add your public ssh key into both machines.

Let me know if this helps.

Best,
David

On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
Thanks Owen.

When I run the dts command line suggested, I get the following error.

Thanks
Ajit

Traceback (most recent call last):
  File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
    self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
  File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
    raise ExceptionPxssh('password refused')
pexpect.pxssh.ExceptionPxssh: password refused
                                         ^^^^^^^^^^^^^^



On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>> wrote:
Hello Ajit,

There was a misconfiguration in our identity management system. The issue should now be resolved.

Owen Hilyard

On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>> wrote:
David,
I can use my credentials to login to rhea.
But the same credentials are not allowing me to login onto the servers you provided.

Thanks
Ajit

ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
Connection closed by 172.18.0.41 port 22

ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
Connection closed by 172.18.0.40 port 22

On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
Hi Ajit,

Thanks for the update. Let me know if you need anything from me.


Best,
David

On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
David,
I had some trouble with my system over the past week.
So I could not login into the setup. I am hoping that my system will be restored early this week
I will login into the setup after that.

Thanks
Ajit

On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
Hi David,
I actually did miss it. I will take a look.
Thanks for the refresh mail.

Thanks
Ajit

On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>
> Hi Ajit,
>
> Ping again in case if you miss the last email.
>
> Did you have a chance to give it a try on login to the machines?
>
> Just some notes, we are currently running compile and some performance/functional testing.
> Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>
> Step I run:
> CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
> ./dts -s                                                           #To start the dts
>
> Informatio:
> arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
> - dts - located in /opt/dts
> - Trex - located in /opt/trex/v2.87
>
> arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
> - dpdk - located in /root/dpdk
>
> Best,
> David
>
> On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>
>> Also, I had changed the configuration when I run Broadcom nic.
>> In the dts folder, you can start it with ./dts -s
>>
>> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>
>>> Hi Ajit,
>>> Sorry for the late reply.
>>>
>>> You can ssh into the machine using the same VPN Username and Password.
>>> Can you try and see if that works?
>>>
>>> Informatio:
>>> arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>>> - dts - located in /opt/dts
>>> - Trex - located in /opt/trex/v2.87
>>>
>>> arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>>> - dpdk - located in /root/dpdk
>>>
>>>
>>> Thanks,
>>> David
>>>
>>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>>
>>>> Hi David,
>>>> Thanks for pinging me. Do you think I could login into the setup and
>>>> see what is happening?
>>>> I do have the credentials for two systems in your lab -
>>>> rhea.dpdklab.iol.unh.edu<http://rhea.dpdklab.iol.unh.edu> and io.dpdklab.iol.unh.edu<http://io.dpdklab.iol.unh.edu>
>>>> If you can provide me details of the systems you are trying to
>>>> configure, I could take a look.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >
>>>> > Hi Ajit,
>>>> >
>>>> > Thank you for helping out.
>>>> >
>>>> > How are you doing with this issue?
>>>> >
>>>> > Please let me know if you need anything information from us.
>>>> >
>>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>>>> >
>>>> > Best,
>>>> > David
>>>> >
>>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >>
>>>> >> Yes, so this is the same NIC work on x86.
>>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>>>> >>
>>>> >> Thank,
>>>> >> David
>>>> >>
>>>> >>
>>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >>>
>>>> >>> Does the same NIC work on x86?
>>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>>>> >>> or if there is a problem with the link. Can you check and tell?
>>>> >>>
>>>> >>> Thanks
>>>> >>> Ajit
>>>> >>>
>>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >>>>
>>>> >>>> Hi Ajit,
>>>> >>>>
>>>> >>>> Thank you for helping out.
>>>> >>>>
>>>> >>>> We have it working on the x86, but not the Arm.
>>>> >>>>
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> David
>>>> >>>>
>>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >>>>>
>>>> >>>>> Hi David,
>>>> >>>>> I will take a look.
>>>> >>>>> Do you see similar issues on x86? I am asking because
>>>> >>>>> even I would start with that to create a baseline and then
>>>> >>>>> attempt on ARM.
>>>> >>>>>
>>>> >>>>> Thanks
>>>> >>>>> Ajit
>>>> >>>>>
>>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >>>>>>
>>>> >>>>>> Hi Ajit,
>>>> >>>>>>
>>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>>>> >>>>>> All the NICs are up and running when testing.
>>>> >>>>>>
>>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>>>> >>>>>>
>>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>> >>>>>>>                dut.172.18.0.41: start
>>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>>> >>>>>>>     case_obj()
>>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>>>> >>>>>>>     self.clear_stats(ports)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>>>> >>>>>>>     raise TRexError(rc)
>>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501<http://172.18.0.40:4501>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>> >>>>>>
>>>> >>>>>> Then start sending traffic with command
>>>> >>>>>>>
>>>> >>>>>>> trex>start -f stl/imix.py
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> But this will cause an error to turn off Trex.
>>>> >>>>>>
>>>> >>>>>>> trex>
>>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>>>> >>>>>>> *** traceback follows ***
>>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>> >>>>>>>
>>>> >>>>>>> *** addr2line information follows ***
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> ??:0
>>>> >>>>>>> '
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>> >>>>>>
>>>> >>>>>> Best,
>>>> >>>>>> David
>>>> >>>>>>
>>>> >>>>>>

[-- Attachment #2: Type: text/html, Size: 37401 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-01 15:00                                         ` Juraj Linkeš
@ 2021-03-01 15:12                                           ` Ajit Khaparde
  2021-03-02 14:28                                             ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-03-01 15:12 UTC (permalink / raw)
  To: Juraj Linkeš; +Cc: David Liu, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 15142 bytes --]

David,
I was busy with a customer issue and could not get to my emails.
I am seeing this now. Let me close out any issues needing urgent attention.
Let's sync on a suitable time after that.

Juraj,
Sure. We will add you.

Thanks
Ajit

On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> Hi Ajit, David,
>
>
>
> If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>
>
>
> Thanks,
>
> Juraj
>
>
>
> From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> Sent: Thursday, February 25, 2021 7:47 PM
> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>
>
>
> Hi Ajit,
>
>
>
> Sorry for this issue.
>
> If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>
> Please let me know what would be a good time for you.
>
>
>
> Best,
>
> David
>
>
>
>
>
> On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
>
>
> I changed the permission to allow you to run sudo.
>
> This should work now.
>
> Let me know if you still have issues.
>
>
>
> Best,
>
> David
>
>
>
> On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
> David,
>
> I tried the option to use sudo. And that didn't work.
>
>
>
> Thanks
>
> Ajit
>
>
>
> ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>
>
>
> ----
>
> sudo su
> [sudo] password for ajitkhaparde:
> ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
> ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> [sudo] password for ajitkhaparde:
> ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>
>
>
> On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
>
>
>
>
> On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
>
>
> This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>
>
>
> One solution would be "sudo su" as root to run the command line since you should have root access.
>
> David,
>
> Let me try this method first.
>
> If it does not work, we could try with the public ssh keys.
>
> I will let you know.
>
>
>
> Thanks
>
> Ajit
>
>
>
> Another solution would be to let me add your public ssh key into both machines.
>
>
>
> Let me know if this helps.
>
>
>
> Best,
>
> David
>
>
>
> On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
> Thanks Owen.
>
>
>
> When I run the dts command line suggested, I get the following error.
>
>
>
> Thanks
>
> Ajit
>
>
>
> Traceback (most recent call last):
>   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>     raise ExceptionPxssh('password refused')
> pexpect.pxssh.ExceptionPxssh: password refused
>
>                                          ^^^^^^^^^^^^^^
>
>
>
>
>
>
>
> On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
>
> Hello Ajit,
>
>
>
> There was a misconfiguration in our identity management system. The issue should now be resolved.
>
>
>
> Owen Hilyard
>
>
>
> On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
>
> David,
>
> I can use my credentials to login to rhea.
>
> But the same credentials are not allowing me to login onto the servers you provided.
>
>
>
> Thanks
>
> Ajit
>
>
>
> ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> Connection closed by 172.18.0.41 port 22
>
>
>
> ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> Connection closed by 172.18.0.40 port 22
>
>
>
> On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
>
>
> Thanks for the update. Let me know if you need anything from me.
>
>
>
>
>
> Best,
>
> David
>
>
>
> On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
> David,
>
> I had some trouble with my system over the past week.
>
> So I could not login into the setup. I am hoping that my system will be restored early this week
>
> I will login into the setup after that.
>
>
>
> Thanks
>
> Ajit
>
>
>
> On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
> Hi David,
> I actually did miss it. I will take a look.
> Thanks for the refresh mail.
>
> Thanks
> Ajit
>
> On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > Ping again in case if you miss the last email.
> >
> > Did you have a chance to give it a try on login to the machines?
> >
> > Just some notes, we are currently running compile and some performance/functional testing.
> > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
> >
> > Step I run:
> > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
> > ./dts -s                                                           #To start the dts
> >
> > Informatio:
> > arm-dell.dpdklab.iol.unh.edu
> > - dts - located in /opt/dts
> > - Trex - located in /opt/trex/v2.87
> >
> > arm-gigabyte.dpdklab.iol.unh.edu
> > - dpdk - located in /root/dpdk
> >
> > Best,
> > David
> >
> > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
> >>
> >> Also, I had changed the configuration when I run Broadcom nic.
> >> In the dts folder, you can start it with ./dts -s
> >>
> >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>
> >>> Hi Ajit,
> >>> Sorry for the late reply.
> >>>
> >>> You can ssh into the machine using the same VPN Username and Password.
> >>> Can you try and see if that works?
> >>>
> >>> Informatio:
> >>> arm-dell.dpdklab.iol.unh.edu
> >>> - dts - located in /opt/dts
> >>> - Trex - located in /opt/trex/v2.87
> >>>
> >>> arm-gigabyte.dpdklab.iol.unh.edu
> >>> - dpdk - located in /root/dpdk
> >>>
> >>>
> >>> Thanks,
> >>> David
> >>>
> >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>
> >>>> Hi David,
> >>>> Thanks for pinging me. Do you think I could login into the setup and
> >>>> see what is happening?
> >>>> I do have the credentials for two systems in your lab -
> >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> >>>> If you can provide me details of the systems you are trying to
> >>>> configure, I could take a look.
> >>>>
> >>>> Thanks
> >>>> Ajit
> >>>>
> >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >
> >>>> > Hi Ajit,
> >>>> >
> >>>> > Thank you for helping out.
> >>>> >
> >>>> > How are you doing with this issue?
> >>>> >
> >>>> > Please let me know if you need anything information from us.
> >>>> >
> >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
> >>>> >
> >>>> > Best,
> >>>> > David
> >>>> >
> >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >>
> >>>> >> Yes, so this is the same NIC work on x86.
> >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
> >>>> >>
> >>>> >> Thank,
> >>>> >> David
> >>>> >>
> >>>> >>
> >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >>>
> >>>> >>> Does the same NIC work on x86?
> >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
> >>>> >>> or if there is a problem with the link. Can you check and tell?
> >>>> >>>
> >>>> >>> Thanks
> >>>> >>> Ajit
> >>>> >>>
> >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >>>>
> >>>> >>>> Hi Ajit,
> >>>> >>>>
> >>>> >>>> Thank you for helping out.
> >>>> >>>>
> >>>> >>>> We have it working on the x86, but not the Arm.
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> Thanks,
> >>>> >>>> David
> >>>> >>>>
> >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >>>>>
> >>>> >>>>> Hi David,
> >>>> >>>>> I will take a look.
> >>>> >>>>> Do you see similar issues on x86? I am asking because
> >>>> >>>>> even I would start with that to create a baseline and then
> >>>> >>>>> attempt on ARM.
> >>>> >>>>>
> >>>> >>>>> Thanks
> >>>> >>>>> Ajit
> >>>> >>>>>
> >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >>>>>>
> >>>> >>>>>> Hi Ajit,
> >>>> >>>>>>
> >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
> >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
> >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
> >>>> >>>>>> All the NICs are up and running when testing.
> >>>> >>>>>>
> >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
> >>>> >>>>>>
> >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
> >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >>>> >>>>>>>                dut.172.18.0.41: start
> >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
> >>>> >>>>>>>     case_obj()
> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
> >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
> >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
> >>>> >>>>>>>     self.clear_stats(ports)
> >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
> >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
> >>>> >>>>>>>     raise TRexError(rc)
> >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
> >>>> >>>>>>
> >>>> >>>>>>
> >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >>>> >>>>>>
> >>>> >>>>>> Then start sending traffic with command
> >>>> >>>>>>>
> >>>> >>>>>>> trex>start -f stl/imix.py
> >>>> >>>>>>
> >>>> >>>>>>
> >>>> >>>>>> But this will cause an error to turn off Trex.
> >>>> >>>>>>
> >>>> >>>>>>> trex>
> >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
> >>>> >>>>>>> *** traceback follows ***
> >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
> >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
> >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
> >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
> >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
> >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
> >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >>>> >>>>>>>
> >>>> >>>>>>> *** addr2line information follows ***
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> ??:0
> >>>> >>>>>>> '
> >>>> >>>>>>
> >>>> >>>>>>
> >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
> >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
> >>>> >>>>>>
> >>>> >>>>>> Best,
> >>>> >>>>>> David
> >>>> >>>>>>
> >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-01 15:12                                           ` Ajit Khaparde
@ 2021-03-02 14:28                                             ` David Liu
  2021-03-02 15:27                                               ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-03-02 14:28 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 16789 bytes --]

Hi Ajit and Juraj,

I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to
adjust the time or let me know if there is a better time that works for you.

For me, I can do:
Tuesday 10:00 AM - 1:00 PM.
Thursday 10:00 AM - 1:00 PM.
Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM

Best,
David

On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> David,
> I was busy with a customer issue and could not get to my emails.
> I am seeing this now. Let me close out any issues needing urgent attention.
> Let's sync on a suitable time after that.
>
> Juraj,
> Sure. We will add you.
>
> Thanks
> Ajit
>
> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
> wrote:
> >
> > Hi Ajit, David,
> >
> >
> >
> > If you're setting up a call, could you please send me an invite? I'd
> like to get some insight into the issues on Arm's behalf.
> >
> >
> >
> > Thanks,
> >
> > Juraj
> >
> >
> >
> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> > Sent: Thursday, February 25, 2021 7:47 PM
> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <
> dpdklab@iol.unh.edu>
> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm
> hardware
> >
> >
> >
> > Hi Ajit,
> >
> >
> >
> > Sorry for this issue.
> >
> > If possible I would like to set up a time where we can do this in a call
> to help get this solved faster.
> >
> > Please let me know what would be a good time for you.
> >
> >
> >
> > Best,
> >
> > David
> >
> >
> >
> >
> >
> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> >
> >
> > I changed the permission to allow you to run sudo.
> >
> > This should work now.
> >
> > Let me know if you still have issues.
> >
> >
> >
> > Best,
> >
> > David
> >
> >
> >
> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >
> > David,
> >
> > I tried the option to use sudo. And that didn't work.
> >
> >
> >
> > Thanks
> >
> > Ajit
> >
> >
> >
> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident
> will be reported.
> >
> >
> >
> > ----
> >
> > sudo su
> > [sudo] password for ajitkhaparde:
> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will
> be reported.
> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> > [sudo] password for ajitkhaparde:
> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will
> be reported.
> >
> >
> >
> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >
> >
> >
> >
> >
> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> >
> >
> > This is likely caused by when dts try to ssh into another machine as a
> normal user which will prompt you for the password.
> >
> >
> >
> > One solution would be "sudo su" as root to run the command line since
> you should have root access.
> >
> > David,
> >
> > Let me try this method first.
> >
> > If it does not work, we could try with the public ssh keys.
> >
> > I will let you know.
> >
> >
> >
> > Thanks
> >
> > Ajit
> >
> >
> >
> > Another solution would be to let me add your public ssh key into both
> machines.
> >
> >
> >
> > Let me know if this helps.
> >
> >
> >
> > Best,
> >
> > David
> >
> >
> >
> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >
> > Thanks Owen.
> >
> >
> >
> > When I run the dts command line suggested, I get the following error.
> >
> >
> >
> > Thanks
> >
> > Ajit
> >
> >
> >
> > Traceback (most recent call last):
> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line
> 440, in login
> >     raise ExceptionPxssh('password refused')
> > pexpect.pxssh.ExceptionPxssh: password refused
> >
> >                                          ^^^^^^^^^^^^^^
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
> wrote:
> >
> > Hello Ajit,
> >
> >
> >
> > There was a misconfiguration in our identity management system. The
> issue should now be resolved.
> >
> >
> >
> > Owen Hilyard
> >
> >
> >
> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
> dpdklab@iol.unh.edu> wrote:
> >
> > David,
> >
> > I can use my credentials to login to rhea.
> >
> > But the same credentials are not allowing me to login onto the servers
> you provided.
> >
> >
> >
> > Thanks
> >
> > Ajit
> >
> >
> >
> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> > Connection closed by 172.18.0.41 port 22
> >
> >
> >
> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> > Connection closed by 172.18.0.40 port 22
> >
> >
> >
> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> >
> >
> > Thanks for the update. Let me know if you need anything from me.
> >
> >
> >
> >
> >
> > Best,
> >
> > David
> >
> >
> >
> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >
> > David,
> >
> > I had some trouble with my system over the past week.
> >
> > So I could not login into the setup. I am hoping that my system will be
> restored early this week
> >
> > I will login into the setup after that.
> >
> >
> >
> > Thanks
> >
> > Ajit
> >
> >
> >
> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >
> > Hi David,
> > I actually did miss it. I will take a look.
> > Thanks for the refresh mail.
> >
> > Thanks
> > Ajit
> >
> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
> > >
> > > Hi Ajit,
> > >
> > > Ping again in case if you miss the last email.
> > >
> > > Did you have a chance to give it a try on login to the machines?
> > >
> > > Just some notes, we are currently running compile and some
> performance/functional testing.
> > > Which it may be hard for you to run dts, so please let me know if you
> want me to stop these tests for you to debug.
> > >
> > > Step I run:
> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
> configuration to Broadcom
> > > ./dts -s                                                           #To
> start the dts
> > >
> > > Informatio:
> > > arm-dell.dpdklab.iol.unh.edu
> > > - dts - located in /opt/dts
> > > - Trex - located in /opt/trex/v2.87
> > >
> > > arm-gigabyte.dpdklab.iol.unh.edu
> > > - dpdk - located in /root/dpdk
> > >
> > > Best,
> > > David
> > >
> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
> > >>
> > >> Also, I had changed the configuration when I run Broadcom nic.
> > >> In the dts folder, you can start it with ./dts -s
> > >>
> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
> > >>>
> > >>> Hi Ajit,
> > >>> Sorry for the late reply.
> > >>>
> > >>> You can ssh into the machine using the same VPN Username and
> Password.
> > >>> Can you try and see if that works?
> > >>>
> > >>> Informatio:
> > >>> arm-dell.dpdklab.iol.unh.edu
> > >>> - dts - located in /opt/dts
> > >>> - Trex - located in /opt/trex/v2.87
> > >>>
> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> > >>> - dpdk - located in /root/dpdk
> > >>>
> > >>>
> > >>> Thanks,
> > >>> David
> > >>>
> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>
> > >>>> Hi David,
> > >>>> Thanks for pinging me. Do you think I could login into the setup and
> > >>>> see what is happening?
> > >>>> I do have the credentials for two systems in your lab -
> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> > >>>> If you can provide me details of the systems you are trying to
> > >>>> configure, I could take a look.
> > >>>>
> > >>>> Thanks
> > >>>> Ajit
> > >>>>
> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
> > >>>> >
> > >>>> > Hi Ajit,
> > >>>> >
> > >>>> > Thank you for helping out.
> > >>>> >
> > >>>> > How are you doing with this issue?
> > >>>> >
> > >>>> > Please let me know if you need anything information from us.
> > >>>> >
> > >>>> > Just some extra information that both Trex and dts are on the
> same intel tester machine.
> > >>>> >
> > >>>> > Best,
> > >>>> > David
> > >>>> >
> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
> wrote:
> > >>>> >>
> > >>>> >> Yes, so this is the same NIC work on x86.
> > >>>> >> By seeing the link, I am assuming you mean the ports that are
> using for Trex. If so they are UP before I start the Trex.
> > >>>> >>
> > >>>> >> Thank,
> > >>>> >> David
> > >>>> >>
> > >>>> >>
> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>> >>>
> > >>>> >>> Does the same NIC work on x86?
> > >>>> >>> Also do you see link before you start the trex? I am trying to
> see if the port failed to successfully come up
> > >>>> >>> or if there is a problem with the link. Can you check and tell?
> > >>>> >>>
> > >>>> >>> Thanks
> > >>>> >>> Ajit
> > >>>> >>>
> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
> wrote:
> > >>>> >>>>
> > >>>> >>>> Hi Ajit,
> > >>>> >>>>
> > >>>> >>>> Thank you for helping out.
> > >>>> >>>>
> > >>>> >>>> We have it working on the x86, but not the Arm.
> > >>>> >>>>
> > >>>> >>>>
> > >>>> >>>> Thanks,
> > >>>> >>>> David
> > >>>> >>>>
> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>> >>>>>
> > >>>> >>>>> Hi David,
> > >>>> >>>>> I will take a look.
> > >>>> >>>>> Do you see similar issues on x86? I am asking because
> > >>>> >>>>> even I would start with that to create a baseline and then
> > >>>> >>>>> attempt on ARM.
> > >>>> >>>>>
> > >>>> >>>>> Thanks
> > >>>> >>>>> Ajit
> > >>>> >>>>>
> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu>
> wrote:
> > >>>> >>>>>>
> > >>>> >>>>>> Hi Ajit,
> > >>>> >>>>>>
> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
> machine.
> > >>>> >>>>>> And I am running into a problem when I try to run
> nic_single_core_perf on the NIC.
> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
> http://git.dpdk.org/tools/dts/
> > >>>> >>>>>> All the NICs are up and running when testing.
> > >>>> >>>>>>
> > >>>> >>>>>> The problem is an error when I try to run the test case
> inside the nic_single_core_perf:
> > >>>> >>>>>>
> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
> framesize: 64, rxd/txd: 512
> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> > >>>> >>>>>>>                dut.172.18.0.41: start
> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
> _execute_test_case
> > >>>> >>>>>>>     case_obj()
> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196,
> in test_perf_nic_single_core
> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270,
> in perf_test
> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
> measure_throughput
> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
> _prepare_transmission
> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> > >>>> >>>>>>>     self.clear_stats(ports)
> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
> clear_xstats)
> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> > >>>> >>>>>>>     raise TRexError(rc)
> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed
> to get server response from tcp://172.18.0.40:4501
> > >>>> >>>>>>
> > >>>> >>>>>>
> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i
> --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> > >>>> >>>>>>
> > >>>> >>>>>> Then start sending traffic with command
> > >>>> >>>>>>>
> > >>>> >>>>>>> trex>start -f stl/imix.py
> > >>>> >>>>>>
> > >>>> >>>>>>
> > >>>> >>>>>> But this will cause an error to turn off Trex.
> > >>>> >>>>>>
> > >>>> >>>>>>> trex>
> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been
> shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for
> more than 1.00135 seconds - timeout is 1 seconds
> > >>>> >>>>>>> *** traceback follows ***
> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
> [0x561be173cf5a]
> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned
> char) + 487
> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
> [0x561be16d11ad]
> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> > >>>> >>>>>>>
> > >>>> >>>>>>> *** addr2line information follows ***
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> ??:0
> > >>>> >>>>>>> '
> > >>>> >>>>>>
> > >>>> >>>>>>
> > >>>> >>>>>> I believe if everything is working, this should not turn off
> trex but please correct me if I am wrong.
> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach
> this issue?
> > >>>> >>>>>>
> > >>>> >>>>>> Best,
> > >>>> >>>>>> David
> > >>>> >>>>>>
> > >>>> >>>>>>
>

[-- Attachment #2: Type: text/html, Size: 26813 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-02 14:28                                             ` David Liu
@ 2021-03-02 15:27                                               ` Ajit Khaparde
  2021-03-02 18:05                                                 ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-03-02 15:27 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 17224 bytes --]

Hi David,
My Thursdays are crazy. Can we do Friday instead?
Juraj, can you make Friday 10AM (Eastern time) call?

Thanks
Ajit

On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit and Juraj,
>
> I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>
> For me, I can do:
> Tuesday 10:00 AM - 1:00 PM.
> Thursday 10:00 AM - 1:00 PM.
> Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>
> Best,
> David
>
> On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>
>> David,
>> I was busy with a customer issue and could not get to my emails.
>> I am seeing this now. Let me close out any issues needing urgent attention.
>> Let's sync on a suitable time after that.
>>
>> Juraj,
>> Sure. We will add you.
>>
>> Thanks
>> Ajit
>>
>> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>> >
>> > Hi Ajit, David,
>> >
>> >
>> >
>> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Juraj
>> >
>> >
>> >
>> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>> > Sent: Thursday, February 25, 2021 7:47 PM
>> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>> >
>> >
>> >
>> > Hi Ajit,
>> >
>> >
>> >
>> > Sorry for this issue.
>> >
>> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>> >
>> > Please let me know what would be a good time for you.
>> >
>> >
>> >
>> > Best,
>> >
>> > David
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> >
>> >
>> > I changed the permission to allow you to run sudo.
>> >
>> > This should work now.
>> >
>> > Let me know if you still have issues.
>> >
>> >
>> >
>> > Best,
>> >
>> > David
>> >
>> >
>> >
>> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >
>> > David,
>> >
>> > I tried the option to use sudo. And that didn't work.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>> >
>> >
>> >
>> > ----
>> >
>> > sudo su
>> > [sudo] password for ajitkhaparde:
>> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> > [sudo] password for ajitkhaparde:
>> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> >
>> >
>> >
>> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> >
>> >
>> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>> >
>> >
>> >
>> > One solution would be "sudo su" as root to run the command line since you should have root access.
>> >
>> > David,
>> >
>> > Let me try this method first.
>> >
>> > If it does not work, we could try with the public ssh keys.
>> >
>> > I will let you know.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > Another solution would be to let me add your public ssh key into both machines.
>> >
>> >
>> >
>> > Let me know if this helps.
>> >
>> >
>> >
>> > Best,
>> >
>> > David
>> >
>> >
>> >
>> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >
>> > Thanks Owen.
>> >
>> >
>> >
>> > When I run the dts command line suggested, I get the following error.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > Traceback (most recent call last):
>> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>> >     raise ExceptionPxssh('password refused')
>> > pexpect.pxssh.ExceptionPxssh: password refused
>> >
>> >                                          ^^^^^^^^^^^^^^
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
>> >
>> > Hello Ajit,
>> >
>> >
>> >
>> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>> >
>> >
>> >
>> > Owen Hilyard
>> >
>> >
>> >
>> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
>> >
>> > David,
>> >
>> > I can use my credentials to login to rhea.
>> >
>> > But the same credentials are not allowing me to login onto the servers you provided.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>> > Connection closed by 172.18.0.41 port 22
>> >
>> >
>> >
>> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>> > Connection closed by 172.18.0.40 port 22
>> >
>> >
>> >
>> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> >
>> >
>> > Thanks for the update. Let me know if you need anything from me.
>> >
>> >
>> >
>> >
>> >
>> > Best,
>> >
>> > David
>> >
>> >
>> >
>> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >
>> > David,
>> >
>> > I had some trouble with my system over the past week.
>> >
>> > So I could not login into the setup. I am hoping that my system will be restored early this week
>> >
>> > I will login into the setup after that.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >
>> > Hi David,
>> > I actually did miss it. I will take a look.
>> > Thanks for the refresh mail.
>> >
>> > Thanks
>> > Ajit
>> >
>> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>> > >
>> > > Hi Ajit,
>> > >
>> > > Ping again in case if you miss the last email.
>> > >
>> > > Did you have a chance to give it a try on login to the machines?
>> > >
>> > > Just some notes, we are currently running compile and some performance/functional testing.
>> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>> > >
>> > > Step I run:
>> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>> > > ./dts -s                                                           #To start the dts
>> > >
>> > > Informatio:
>> > > arm-dell.dpdklab.iol.unh.edu
>> > > - dts - located in /opt/dts
>> > > - Trex - located in /opt/trex/v2.87
>> > >
>> > > arm-gigabyte.dpdklab.iol.unh.edu
>> > > - dpdk - located in /root/dpdk
>> > >
>> > > Best,
>> > > David
>> > >
>> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>
>> > >> Also, I had changed the configuration when I run Broadcom nic.
>> > >> In the dts folder, you can start it with ./dts -s
>> > >>
>> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>
>> > >>> Hi Ajit,
>> > >>> Sorry for the late reply.
>> > >>>
>> > >>> You can ssh into the machine using the same VPN Username and Password.
>> > >>> Can you try and see if that works?
>> > >>>
>> > >>> Informatio:
>> > >>> arm-dell.dpdklab.iol.unh.edu
>> > >>> - dts - located in /opt/dts
>> > >>> - Trex - located in /opt/trex/v2.87
>> > >>>
>> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>> > >>> - dpdk - located in /root/dpdk
>> > >>>
>> > >>>
>> > >>> Thanks,
>> > >>> David
>> > >>>
>> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>
>> > >>>> Hi David,
>> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>> > >>>> see what is happening?
>> > >>>> I do have the credentials for two systems in your lab -
>> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>> > >>>> If you can provide me details of the systems you are trying to
>> > >>>> configure, I could take a look.
>> > >>>>
>> > >>>> Thanks
>> > >>>> Ajit
>> > >>>>
>> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>> >
>> > >>>> > Hi Ajit,
>> > >>>> >
>> > >>>> > Thank you for helping out.
>> > >>>> >
>> > >>>> > How are you doing with this issue?
>> > >>>> >
>> > >>>> > Please let me know if you need anything information from us.
>> > >>>> >
>> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>> > >>>> >
>> > >>>> > Best,
>> > >>>> > David
>> > >>>> >
>> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>> >>
>> > >>>> >> Yes, so this is the same NIC work on x86.
>> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>> > >>>> >>
>> > >>>> >> Thank,
>> > >>>> >> David
>> > >>>> >>
>> > >>>> >>
>> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>> >>>
>> > >>>> >>> Does the same NIC work on x86?
>> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>> > >>>> >>>
>> > >>>> >>> Thanks
>> > >>>> >>> Ajit
>> > >>>> >>>
>> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>> >>>>
>> > >>>> >>>> Hi Ajit,
>> > >>>> >>>>
>> > >>>> >>>> Thank you for helping out.
>> > >>>> >>>>
>> > >>>> >>>> We have it working on the x86, but not the Arm.
>> > >>>> >>>>
>> > >>>> >>>>
>> > >>>> >>>> Thanks,
>> > >>>> >>>> David
>> > >>>> >>>>
>> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>> >>>>>
>> > >>>> >>>>> Hi David,
>> > >>>> >>>>> I will take a look.
>> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>> > >>>> >>>>> even I would start with that to create a baseline and then
>> > >>>> >>>>> attempt on ARM.
>> > >>>> >>>>>
>> > >>>> >>>>> Thanks
>> > >>>> >>>>> Ajit
>> > >>>> >>>>>
>> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>> >>>>>>
>> > >>>> >>>>>> Hi Ajit,
>> > >>>> >>>>>>
>> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>> > >>>> >>>>>> All the NICs are up and running when testing.
>> > >>>> >>>>>>
>> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>> > >>>> >>>>>>
>> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> > >>>> >>>>>>>                dut.172.18.0.41: start
>> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>> > >>>> >>>>>>>     case_obj()
>> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> > >>>> >>>>>>>     self.clear_stats(ports)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>> > >>>> >>>>>>>     raise TRexError(rc)
>> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>> > >>>> >>>>>>
>> > >>>> >>>>>>
>> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>> > >>>> >>>>>>
>> > >>>> >>>>>> Then start sending traffic with command
>> > >>>> >>>>>>>
>> > >>>> >>>>>>> trex>start -f stl/imix.py
>> > >>>> >>>>>>
>> > >>>> >>>>>>
>> > >>>> >>>>>> But this will cause an error to turn off Trex.
>> > >>>> >>>>>>
>> > >>>> >>>>>>> trex>
>> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>> > >>>> >>>>>>> *** traceback follows ***
>> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> > >>>> >>>>>>>
>> > >>>> >>>>>>> *** addr2line information follows ***
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> '
>> > >>>> >>>>>>
>> > >>>> >>>>>>
>> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>> > >>>> >>>>>>
>> > >>>> >>>>>> Best,
>> > >>>> >>>>>> David
>> > >>>> >>>>>>
>> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-02 15:27                                               ` Ajit Khaparde
@ 2021-03-02 18:05                                                 ` David Liu
  2021-03-03  9:05                                                   ` Juraj Linkeš
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-03-02 18:05 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 18686 bytes --]

Hi, Ajit
I changed it to Friday at 10 AM for now.
We can change again if that doesn't work for Juraj
Best,
David


On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi David,
> My Thursdays are crazy. Can we do Friday instead?
> Juraj, can you make Friday 10AM (Eastern time) call?
>
> Thanks
> Ajit
>
> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit and Juraj,
> >
> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to
> adjust the time or let me know if there is a better time that works for you.
> >
> > For me, I can do:
> > Tuesday 10:00 AM - 1:00 PM.
> > Thursday 10:00 AM - 1:00 PM.
> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >
> > Best,
> > David
> >
> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>
> >> David,
> >> I was busy with a customer issue and could not get to my emails.
> >> I am seeing this now. Let me close out any issues needing urgent
> attention.
> >> Let's sync on a suitable time after that.
> >>
> >> Juraj,
> >> Sure. We will add you.
> >>
> >> Thanks
> >> Ajit
> >>
> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
> wrote:
> >> >
> >> > Hi Ajit, David,
> >> >
> >> >
> >> >
> >> > If you're setting up a call, could you please send me an invite? I'd
> like to get some insight into the issues on Arm's behalf.
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Juraj
> >> >
> >> >
> >> >
> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <
> dpdklab@iol.unh.edu>
> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in
> Arm hardware
> >> >
> >> >
> >> >
> >> > Hi Ajit,
> >> >
> >> >
> >> >
> >> > Sorry for this issue.
> >> >
> >> > If possible I would like to set up a time where we can do this in a
> call to help get this solved faster.
> >> >
> >> > Please let me know what would be a good time for you.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > David
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >
> >> > Hi Ajit,
> >> >
> >> >
> >> >
> >> > I changed the permission to allow you to run sudo.
> >> >
> >> > This should work now.
> >> >
> >> > Let me know if you still have issues.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > David
> >> >
> >> >
> >> >
> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > David,
> >> >
> >> > I tried the option to use sudo. And that didn't work.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This
> incident will be reported.
> >> >
> >> >
> >> >
> >> > ----
> >> >
> >> > sudo su
> >> > [sudo] password for ajitkhaparde:
> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident
> will be reported.
> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >> > [sudo] password for ajitkhaparde:
> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident
> will be reported.
> >> >
> >> >
> >> >
> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >
> >> > Hi Ajit,
> >> >
> >> >
> >> >
> >> > This is likely caused by when dts try to ssh into another machine as
> a normal user which will prompt you for the password.
> >> >
> >> >
> >> >
> >> > One solution would be "sudo su" as root to run the command line since
> you should have root access.
> >> >
> >> > David,
> >> >
> >> > Let me try this method first.
> >> >
> >> > If it does not work, we could try with the public ssh keys.
> >> >
> >> > I will let you know.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > Another solution would be to let me add your public ssh key into both
> machines.
> >> >
> >> >
> >> >
> >> > Let me know if this helps.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > David
> >> >
> >> >
> >> >
> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > Thanks Owen.
> >> >
> >> >
> >> >
> >> > When I run the dts command line suggested, I get the following error.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > Traceback (most recent call last):
> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
> >> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py",
> line 440, in login
> >> >     raise ExceptionPxssh('password refused')
> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >> >
> >> >                                          ^^^^^^^^^^^^^^
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
> wrote:
> >> >
> >> > Hello Ajit,
> >> >
> >> >
> >> >
> >> > There was a misconfiguration in our identity management system. The
> issue should now be resolved.
> >> >
> >> >
> >> >
> >> > Owen Hilyard
> >> >
> >> >
> >> >
> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
> dpdklab@iol.unh.edu> wrote:
> >> >
> >> > David,
> >> >
> >> > I can use my credentials to login to rhea.
> >> >
> >> > But the same credentials are not allowing me to login onto the
> servers you provided.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> >> > Connection closed by 172.18.0.41 port 22
> >> >
> >> >
> >> >
> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >> > Connection closed by 172.18.0.40 port 22
> >> >
> >> >
> >> >
> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >
> >> > Hi Ajit,
> >> >
> >> >
> >> >
> >> > Thanks for the update. Let me know if you need anything from me.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > David
> >> >
> >> >
> >> >
> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > David,
> >> >
> >> > I had some trouble with my system over the past week.
> >> >
> >> > So I could not login into the setup. I am hoping that my system will
> be restored early this week
> >> >
> >> > I will login into the setup after that.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > Hi David,
> >> > I actually did miss it. I will take a look.
> >> > Thanks for the refresh mail.
> >> >
> >> > Thanks
> >> > Ajit
> >> >
> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
> >> > >
> >> > > Hi Ajit,
> >> > >
> >> > > Ping again in case if you miss the last email.
> >> > >
> >> > > Did you have a chance to give it a try on login to the machines?
> >> > >
> >> > > Just some notes, we are currently running compile and some
> performance/functional testing.
> >> > > Which it may be hard for you to run dts, so please let me know if
> you want me to stop these tests for you to debug.
> >> > >
> >> > > Step I run:
> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
> configuration to Broadcom
> >> > > ./dts -s
>  #To start the dts
> >> > >
> >> > > Informatio:
> >> > > arm-dell.dpdklab.iol.unh.edu
> >> > > - dts - located in /opt/dts
> >> > > - Trex - located in /opt/trex/v2.87
> >> > >
> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >> > > - dpdk - located in /root/dpdk
> >> > >
> >> > > Best,
> >> > > David
> >> > >
> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
> >> > >>
> >> > >> Also, I had changed the configuration when I run Broadcom nic.
> >> > >> In the dts folder, you can start it with ./dts -s
> >> > >>
> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>
> >> > >>> Hi Ajit,
> >> > >>> Sorry for the late reply.
> >> > >>>
> >> > >>> You can ssh into the machine using the same VPN Username and
> Password.
> >> > >>> Can you try and see if that works?
> >> > >>>
> >> > >>> Informatio:
> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >> > >>> - dts - located in /opt/dts
> >> > >>> - Trex - located in /opt/trex/v2.87
> >> > >>>
> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >> > >>> - dpdk - located in /root/dpdk
> >> > >>>
> >> > >>>
> >> > >>> Thanks,
> >> > >>> David
> >> > >>>
> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>
> >> > >>>> Hi David,
> >> > >>>> Thanks for pinging me. Do you think I could login into the setup
> and
> >> > >>>> see what is happening?
> >> > >>>> I do have the credentials for two systems in your lab -
> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> >> > >>>> If you can provide me details of the systems you are trying to
> >> > >>>> configure, I could take a look.
> >> > >>>>
> >> > >>>> Thanks
> >> > >>>> Ajit
> >> > >>>>
> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>> >
> >> > >>>> > Hi Ajit,
> >> > >>>> >
> >> > >>>> > Thank you for helping out.
> >> > >>>> >
> >> > >>>> > How are you doing with this issue?
> >> > >>>> >
> >> > >>>> > Please let me know if you need anything information from us.
> >> > >>>> >
> >> > >>>> > Just some extra information that both Trex and dts are on the
> same intel tester machine.
> >> > >>>> >
> >> > >>>> > Best,
> >> > >>>> > David
> >> > >>>> >
> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>> >>
> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are
> using for Trex. If so they are UP before I start the Trex.
> >> > >>>> >>
> >> > >>>> >> Thank,
> >> > >>>> >> David
> >> > >>>> >>
> >> > >>>> >>
> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>> >>>
> >> > >>>> >>> Does the same NIC work on x86?
> >> > >>>> >>> Also do you see link before you start the trex? I am trying
> to see if the port failed to successfully come up
> >> > >>>> >>> or if there is a problem with the link. Can you check and
> tell?
> >> > >>>> >>>
> >> > >>>> >>> Thanks
> >> > >>>> >>> Ajit
> >> > >>>> >>>
> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>> >>>>
> >> > >>>> >>>> Hi Ajit,
> >> > >>>> >>>>
> >> > >>>> >>>> Thank you for helping out.
> >> > >>>> >>>>
> >> > >>>> >>>> We have it working on the x86, but not the Arm.
> >> > >>>> >>>>
> >> > >>>> >>>>
> >> > >>>> >>>> Thanks,
> >> > >>>> >>>> David
> >> > >>>> >>>>
> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>> >>>>>
> >> > >>>> >>>>> Hi David,
> >> > >>>> >>>>> I will take a look.
> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
> >> > >>>> >>>>> even I would start with that to create a baseline and then
> >> > >>>> >>>>> attempt on ARM.
> >> > >>>> >>>>>
> >> > >>>> >>>>> Thanks
> >> > >>>> >>>>> Ajit
> >> > >>>> >>>>>
> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> Hi Ajit,
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
> machine.
> >> > >>>> >>>>>> And I am running into a problem when I try to run
> nic_single_core_perf on the NIC.
> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
> http://git.dpdk.org/tools/dts/
> >> > >>>> >>>>>> All the NICs are up and running when testing.
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> The problem is an error when I try to run the test case
> inside the nic_single_core_perf:
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
> framesize: 64, rxd/txd: 512
> >> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
> _execute_test_case
> >> > >>>> >>>>>>>     case_obj()
> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
> 196, in test_perf_nic_single_core
> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
> 270, in perf_test
> >> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
> measure_throughput
> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
> _prepare_transmission
> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
> clear_xstats)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >> > >>>> >>>>>>>     raise TRexError(rc)
> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
> Failed to get server response from tcp://172.18.0.40:4501
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64
> -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> Then start sending traffic with command
> >> > >>>> >>>>>>>
> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>> trex>
> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
> for more than 1.00135 seconds - timeout is 1 seconds
> >> > >>>> >>>>>>> *** traceback follows ***
> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
> [0x561be173cf5a]
> >> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> >> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> >> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> >> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> >> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> >> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned
> char) + 487
> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
> [0x561be16d11ad]
> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >> > >>>> >>>>>>>
> >> > >>>> >>>>>>> *** addr2line information follows ***
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> '
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> I believe if everything is working, this should not turn
> off trex but please correct me if I am wrong.
> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach
> this issue?
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> Best,
> >> > >>>> >>>>>> David
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>
>

[-- Attachment #2: Type: text/html, Size: 31822 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-02 18:05                                                 ` David Liu
@ 2021-03-03  9:05                                                   ` Juraj Linkeš
  2021-03-05 21:29                                                     ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Juraj Linkeš @ 2021-03-03  9:05 UTC (permalink / raw)
  To: David Liu, Ajit Khaparde; +Cc: Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 19015 bytes --]

Hi David, Ajit,

Friday 10am is 4pm for me a that's fine.

Juraj

From: David Liu <dliu@iol.unh.edu>
Sent: Tuesday, March 2, 2021 7:06 PM
To: Ajit Khaparde <ajit.khaparde@broadcom.com>
Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware

Hi, Ajit
I changed it to Friday at 10 AM for now.
We can change again if that doesn't work for Juraj
Best,
David


On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
Hi David,
My Thursdays are crazy. Can we do Friday instead?
Juraj, can you make Friday 10AM (Eastern time) call?

Thanks
Ajit

On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>
> Hi Ajit and Juraj,
>
> I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>
> For me, I can do:
> Tuesday 10:00 AM - 1:00 PM.
> Thursday 10:00 AM - 1:00 PM.
> Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>
> Best,
> David
>
> On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>
>> David,
>> I was busy with a customer issue and could not get to my emails.
>> I am seeing this now. Let me close out any issues needing urgent attention.
>> Let's sync on a suitable time after that.
>>
>> Juraj,
>> Sure. We will add you.
>>
>> Thanks
>> Ajit
>>
>> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>> wrote:
>> >
>> > Hi Ajit, David,
>> >
>> >
>> >
>> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Juraj
>> >
>> >
>> >
>> > From: ci <ci-bounces@dpdk.org<mailto:ci-bounces@dpdk.org>> On Behalf Of David Liu
>> > Sent: Thursday, February 25, 2021 7:47 PM
>> > To: Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>>
>> > Cc: Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>>; ci@dpdk.org<mailto:ci@dpdk.org>; dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>>
>> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>> >
>> >
>> >
>> > Hi Ajit,
>> >
>> >
>> >
>> > Sorry for this issue.
>> >
>> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>> >
>> > Please let me know what would be a good time for you.
>> >
>> >
>> >
>> > Best,
>> >
>> > David
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> >
>> > Hi Ajit,
>> >
>> >
>> >
>> > I changed the permission to allow you to run sudo.
>> >
>> > This should work now.
>> >
>> > Let me know if you still have issues.
>> >
>> >
>> >
>> > Best,
>> >
>> > David
>> >
>> >
>> >
>> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>> >
>> > David,
>> >
>> > I tried the option to use sudo. And that didn't work.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>> >
>> >
>> >
>> > ----
>> >
>> > sudo su
>> > [sudo] password for ajitkhaparde:
>> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> > [sudo] password for ajitkhaparde:
>> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> >
>> >
>> >
>> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> >
>> > Hi Ajit,
>> >
>> >
>> >
>> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>> >
>> >
>> >
>> > One solution would be "sudo su" as root to run the command line since you should have root access.
>> >
>> > David,
>> >
>> > Let me try this method first.
>> >
>> > If it does not work, we could try with the public ssh keys.
>> >
>> > I will let you know.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > Another solution would be to let me add your public ssh key into both machines.
>> >
>> >
>> >
>> > Let me know if this helps.
>> >
>> >
>> >
>> > Best,
>> >
>> > David
>> >
>> >
>> >
>> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>> >
>> > Thanks Owen.
>> >
>> >
>> >
>> > When I run the dts command line suggested, I get the following error.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > Traceback (most recent call last):
>> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>> >     raise ExceptionPxssh('password refused')
>> > pexpect.pxssh.ExceptionPxssh: password refused
>> >
>> >                                          ^^^^^^^^^^^^^^
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>> wrote:
>> >
>> > Hello Ajit,
>> >
>> >
>> >
>> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>> >
>> >
>> >
>> > Owen Hilyard
>> >
>> >
>> >
>> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>> wrote:
>> >
>> > David,
>> >
>> > I can use my credentials to login to rhea.
>> >
>> > But the same credentials are not allowing me to login onto the servers you provided.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>> > Connection closed by 172.18.0.41 port 22
>> >
>> >
>> >
>> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>> > Connection closed by 172.18.0.40 port 22
>> >
>> >
>> >
>> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> >
>> > Hi Ajit,
>> >
>> >
>> >
>> > Thanks for the update. Let me know if you need anything from me.
>> >
>> >
>> >
>> >
>> >
>> > Best,
>> >
>> > David
>> >
>> >
>> >
>> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>> >
>> > David,
>> >
>> > I had some trouble with my system over the past week.
>> >
>> > So I could not login into the setup. I am hoping that my system will be restored early this week
>> >
>> > I will login into the setup after that.
>> >
>> >
>> >
>> > Thanks
>> >
>> > Ajit
>> >
>> >
>> >
>> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>> >
>> > Hi David,
>> > I actually did miss it. I will take a look.
>> > Thanks for the refresh mail.
>> >
>> > Thanks
>> > Ajit
>> >
>> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> > >
>> > > Hi Ajit,
>> > >
>> > > Ping again in case if you miss the last email.
>> > >
>> > > Did you have a chance to give it a try on login to the machines?
>> > >
>> > > Just some notes, we are currently running compile and some performance/functional testing.
>> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>> > >
>> > > Step I run:
>> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>> > > ./dts -s                                                           #To start the dts
>> > >
>> > > Informatio:
>> > > arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>> > > - dts - located in /opt/dts
>> > > - Trex - located in /opt/trex/v2.87
>> > >
>> > > arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>> > > - dpdk - located in /root/dpdk
>> > >
>> > > Best,
>> > > David
>> > >
>> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> > >>
>> > >> Also, I had changed the configuration when I run Broadcom nic.
>> > >> In the dts folder, you can start it with ./dts -s
>> > >>
>> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> > >>>
>> > >>> Hi Ajit,
>> > >>> Sorry for the late reply.
>> > >>>
>> > >>> You can ssh into the machine using the same VPN Username and Password.
>> > >>> Can you try and see if that works?
>> > >>>
>> > >>> Informatio:
>> > >>> arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>> > >>> - dts - located in /opt/dts
>> > >>> - Trex - located in /opt/trex/v2.87
>> > >>>
>> > >>> arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>> > >>> - dpdk - located in /root/dpdk
>> > >>>
>> > >>>
>> > >>> Thanks,
>> > >>> David
>> > >>>
>> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>> > >>>>
>> > >>>> Hi David,
>> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>> > >>>> see what is happening?
>> > >>>> I do have the credentials for two systems in your lab -
>> > >>>> rhea.dpdklab.iol.unh.edu<http://rhea.dpdklab.iol.unh.edu> and io.dpdklab.iol.unh.edu<http://io.dpdklab.iol.unh.edu>
>> > >>>> If you can provide me details of the systems you are trying to
>> > >>>> configure, I could take a look.
>> > >>>>
>> > >>>> Thanks
>> > >>>> Ajit
>> > >>>>
>> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> > >>>> >
>> > >>>> > Hi Ajit,
>> > >>>> >
>> > >>>> > Thank you for helping out.
>> > >>>> >
>> > >>>> > How are you doing with this issue?
>> > >>>> >
>> > >>>> > Please let me know if you need anything information from us.
>> > >>>> >
>> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>> > >>>> >
>> > >>>> > Best,
>> > >>>> > David
>> > >>>> >
>> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> > >>>> >>
>> > >>>> >> Yes, so this is the same NIC work on x86.
>> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>> > >>>> >>
>> > >>>> >> Thank,
>> > >>>> >> David
>> > >>>> >>
>> > >>>> >>
>> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>> > >>>> >>>
>> > >>>> >>> Does the same NIC work on x86?
>> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>> > >>>> >>>
>> > >>>> >>> Thanks
>> > >>>> >>> Ajit
>> > >>>> >>>
>> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> > >>>> >>>>
>> > >>>> >>>> Hi Ajit,
>> > >>>> >>>>
>> > >>>> >>>> Thank you for helping out.
>> > >>>> >>>>
>> > >>>> >>>> We have it working on the x86, but not the Arm.
>> > >>>> >>>>
>> > >>>> >>>>
>> > >>>> >>>> Thanks,
>> > >>>> >>>> David
>> > >>>> >>>>
>> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>> > >>>> >>>>>
>> > >>>> >>>>> Hi David,
>> > >>>> >>>>> I will take a look.
>> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>> > >>>> >>>>> even I would start with that to create a baseline and then
>> > >>>> >>>>> attempt on ARM.
>> > >>>> >>>>>
>> > >>>> >>>>> Thanks
>> > >>>> >>>>> Ajit
>> > >>>> >>>>>
>> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>> > >>>> >>>>>>
>> > >>>> >>>>>> Hi Ajit,
>> > >>>> >>>>>>
>> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>> > >>>> >>>>>> All the NICs are up and running when testing.
>> > >>>> >>>>>>
>> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>> > >>>> >>>>>>
>> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> > >>>> >>>>>>>                dut.172.18.0.41: start
>> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>> > >>>> >>>>>>>     case_obj()
>> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> > >>>> >>>>>>>     self.clear_stats(ports)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>> > >>>> >>>>>>>     raise TRexError(rc)
>> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501<http://172.18.0.40:4501>
>> > >>>> >>>>>>
>> > >>>> >>>>>>
>> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>> > >>>> >>>>>>
>> > >>>> >>>>>> Then start sending traffic with command
>> > >>>> >>>>>>>
>> > >>>> >>>>>>> trex>start -f stl/imix.py
>> > >>>> >>>>>>
>> > >>>> >>>>>>
>> > >>>> >>>>>> But this will cause an error to turn off Trex.
>> > >>>> >>>>>>
>> > >>>> >>>>>>> trex>
>> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>> > >>>> >>>>>>> *** traceback follows ***
>> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> > >>>> >>>>>>>
>> > >>>> >>>>>>> *** addr2line information follows ***
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> ??:0
>> > >>>> >>>>>>> '
>> > >>>> >>>>>>
>> > >>>> >>>>>>
>> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>> > >>>> >>>>>>
>> > >>>> >>>>>> Best,
>> > >>>> >>>>>> David
>> > >>>> >>>>>>
>> > >>>> >>>>>>

[-- Attachment #2: Type: text/html, Size: 36578 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-03  9:05                                                   ` Juraj Linkeš
@ 2021-03-05 21:29                                                     ` David Liu
  2021-03-05 22:06                                                       ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-03-05 21:29 UTC (permalink / raw)
  To: Juraj Linkeš; +Cc: Ajit Khaparde, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 19598 bytes --]

Hi Ajit, Juraj,

For simplicity, we will keep the functional and performance testing disable
during the weekend.

In this way, you will have no testing interfere with you during
troubleshooting.

Then we will rerun all the jobs that are skipped.

Please let me know if you have any questions.

Best,
David

On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:

> Hi David, Ajit,
>
>
>
> Friday 10am is 4pm for me a that's fine.
>
>
>
> Juraj
>
>
>
> *From:* David Liu <dliu@iol.unh.edu>
> *Sent:* Tuesday, March 2, 2021 7:06 PM
> *To:* Ajit Khaparde <ajit.khaparde@broadcom.com>
> *Cc:* Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <
> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> *Subject:* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm
> hardware
>
>
>
> Hi, Ajit
> I changed it to Friday at 10 AM for now.
>
> We can change again if that doesn't work for Juraj
>
> Best,
>
> David
>
>
>
>
>
> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
> Hi David,
> My Thursdays are crazy. Can we do Friday instead?
> Juraj, can you make Friday 10AM (Eastern time) call?
>
> Thanks
> Ajit
>
> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit and Juraj,
> >
> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to
> adjust the time or let me know if there is a better time that works for you.
> >
> > For me, I can do:
> > Tuesday 10:00 AM - 1:00 PM.
> > Thursday 10:00 AM - 1:00 PM.
> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >
> > Best,
> > David
> >
> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>
> >> David,
> >> I was busy with a customer issue and could not get to my emails.
> >> I am seeing this now. Let me close out any issues needing urgent
> attention.
> >> Let's sync on a suitable time after that.
> >>
> >> Juraj,
> >> Sure. We will add you.
> >>
> >> Thanks
> >> Ajit
> >>
> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
> wrote:
> >> >
> >> > Hi Ajit, David,
> >> >
> >> >
> >> >
> >> > If you're setting up a call, could you please send me an invite? I'd
> like to get some insight into the issues on Arm's behalf.
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Juraj
> >> >
> >> >
> >> >
> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <
> dpdklab@iol.unh.edu>
> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in
> Arm hardware
> >> >
> >> >
> >> >
> >> > Hi Ajit,
> >> >
> >> >
> >> >
> >> > Sorry for this issue.
> >> >
> >> > If possible I would like to set up a time where we can do this in a
> call to help get this solved faster.
> >> >
> >> > Please let me know what would be a good time for you.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > David
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >
> >> > Hi Ajit,
> >> >
> >> >
> >> >
> >> > I changed the permission to allow you to run sudo.
> >> >
> >> > This should work now.
> >> >
> >> > Let me know if you still have issues.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > David
> >> >
> >> >
> >> >
> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > David,
> >> >
> >> > I tried the option to use sudo. And that didn't work.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This
> incident will be reported.
> >> >
> >> >
> >> >
> >> > ----
> >> >
> >> > sudo su
> >> > [sudo] password for ajitkhaparde:
> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident
> will be reported.
> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >> > [sudo] password for ajitkhaparde:
> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident
> will be reported.
> >> >
> >> >
> >> >
> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >
> >> > Hi Ajit,
> >> >
> >> >
> >> >
> >> > This is likely caused by when dts try to ssh into another machine as
> a normal user which will prompt you for the password.
> >> >
> >> >
> >> >
> >> > One solution would be "sudo su" as root to run the command line since
> you should have root access.
> >> >
> >> > David,
> >> >
> >> > Let me try this method first.
> >> >
> >> > If it does not work, we could try with the public ssh keys.
> >> >
> >> > I will let you know.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > Another solution would be to let me add your public ssh key into both
> machines.
> >> >
> >> >
> >> >
> >> > Let me know if this helps.
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > David
> >> >
> >> >
> >> >
> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > Thanks Owen.
> >> >
> >> >
> >> >
> >> > When I run the dts command line suggested, I get the following error.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > Traceback (most recent call last):
> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
> >> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py",
> line 440, in login
> >> >     raise ExceptionPxssh('password refused')
> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >> >
> >> >                                          ^^^^^^^^^^^^^^
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
> wrote:
> >> >
> >> > Hello Ajit,
> >> >
> >> >
> >> >
> >> > There was a misconfiguration in our identity management system. The
> issue should now be resolved.
> >> >
> >> >
> >> >
> >> > Owen Hilyard
> >> >
> >> >
> >> >
> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
> dpdklab@iol.unh.edu> wrote:
> >> >
> >> > David,
> >> >
> >> > I can use my credentials to login to rhea.
> >> >
> >> > But the same credentials are not allowing me to login onto the
> servers you provided.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> >> > Connection closed by 172.18.0.41 port 22
> >> >
> >> >
> >> >
> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >> > Connection closed by 172.18.0.40 port 22
> >> >
> >> >
> >> >
> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >
> >> > Hi Ajit,
> >> >
> >> >
> >> >
> >> > Thanks for the update. Let me know if you need anything from me.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > David
> >> >
> >> >
> >> >
> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > David,
> >> >
> >> > I had some trouble with my system over the past week.
> >> >
> >> > So I could not login into the setup. I am hoping that my system will
> be restored early this week
> >> >
> >> > I will login into the setup after that.
> >> >
> >> >
> >> >
> >> > Thanks
> >> >
> >> > Ajit
> >> >
> >> >
> >> >
> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > Hi David,
> >> > I actually did miss it. I will take a look.
> >> > Thanks for the refresh mail.
> >> >
> >> > Thanks
> >> > Ajit
> >> >
> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
> >> > >
> >> > > Hi Ajit,
> >> > >
> >> > > Ping again in case if you miss the last email.
> >> > >
> >> > > Did you have a chance to give it a try on login to the machines?
> >> > >
> >> > > Just some notes, we are currently running compile and some
> performance/functional testing.
> >> > > Which it may be hard for you to run dts, so please let me know if
> you want me to stop these tests for you to debug.
> >> > >
> >> > > Step I run:
> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
> configuration to Broadcom
> >> > > ./dts -s
>  #To start the dts
> >> > >
> >> > > Informatio:
> >> > > arm-dell.dpdklab.iol.unh.edu
> >> > > - dts - located in /opt/dts
> >> > > - Trex - located in /opt/trex/v2.87
> >> > >
> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >> > > - dpdk - located in /root/dpdk
> >> > >
> >> > > Best,
> >> > > David
> >> > >
> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
> >> > >>
> >> > >> Also, I had changed the configuration when I run Broadcom nic.
> >> > >> In the dts folder, you can start it with ./dts -s
> >> > >>
> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>
> >> > >>> Hi Ajit,
> >> > >>> Sorry for the late reply.
> >> > >>>
> >> > >>> You can ssh into the machine using the same VPN Username and
> Password.
> >> > >>> Can you try and see if that works?
> >> > >>>
> >> > >>> Informatio:
> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >> > >>> - dts - located in /opt/dts
> >> > >>> - Trex - located in /opt/trex/v2.87
> >> > >>>
> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >> > >>> - dpdk - located in /root/dpdk
> >> > >>>
> >> > >>>
> >> > >>> Thanks,
> >> > >>> David
> >> > >>>
> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>
> >> > >>>> Hi David,
> >> > >>>> Thanks for pinging me. Do you think I could login into the setup
> and
> >> > >>>> see what is happening?
> >> > >>>> I do have the credentials for two systems in your lab -
> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> >> > >>>> If you can provide me details of the systems you are trying to
> >> > >>>> configure, I could take a look.
> >> > >>>>
> >> > >>>> Thanks
> >> > >>>> Ajit
> >> > >>>>
> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>> >
> >> > >>>> > Hi Ajit,
> >> > >>>> >
> >> > >>>> > Thank you for helping out.
> >> > >>>> >
> >> > >>>> > How are you doing with this issue?
> >> > >>>> >
> >> > >>>> > Please let me know if you need anything information from us.
> >> > >>>> >
> >> > >>>> > Just some extra information that both Trex and dts are on the
> same intel tester machine.
> >> > >>>> >
> >> > >>>> > Best,
> >> > >>>> > David
> >> > >>>> >
> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>> >>
> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are
> using for Trex. If so they are UP before I start the Trex.
> >> > >>>> >>
> >> > >>>> >> Thank,
> >> > >>>> >> David
> >> > >>>> >>
> >> > >>>> >>
> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>> >>>
> >> > >>>> >>> Does the same NIC work on x86?
> >> > >>>> >>> Also do you see link before you start the trex? I am trying
> to see if the port failed to successfully come up
> >> > >>>> >>> or if there is a problem with the link. Can you check and
> tell?
> >> > >>>> >>>
> >> > >>>> >>> Thanks
> >> > >>>> >>> Ajit
> >> > >>>> >>>
> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>> >>>>
> >> > >>>> >>>> Hi Ajit,
> >> > >>>> >>>>
> >> > >>>> >>>> Thank you for helping out.
> >> > >>>> >>>>
> >> > >>>> >>>> We have it working on the x86, but not the Arm.
> >> > >>>> >>>>
> >> > >>>> >>>>
> >> > >>>> >>>> Thanks,
> >> > >>>> >>>> David
> >> > >>>> >>>>
> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>> >>>>>
> >> > >>>> >>>>> Hi David,
> >> > >>>> >>>>> I will take a look.
> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
> >> > >>>> >>>>> even I would start with that to create a baseline and then
> >> > >>>> >>>>> attempt on ARM.
> >> > >>>> >>>>>
> >> > >>>> >>>>> Thanks
> >> > >>>> >>>>> Ajit
> >> > >>>> >>>>>
> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> Hi Ajit,
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
> machine.
> >> > >>>> >>>>>> And I am running into a problem when I try to run
> nic_single_core_perf on the NIC.
> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
> http://git.dpdk.org/tools/dts/
> >> > >>>> >>>>>> All the NICs are up and running when testing.
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> The problem is an error when I try to run the test case
> inside the nic_single_core_perf:
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
> framesize: 64, rxd/txd: 512
> >> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
> _execute_test_case
> >> > >>>> >>>>>>>     case_obj()
> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
> 196, in test_perf_nic_single_core
> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
> 270, in perf_test
> >> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in
> measure_throughput
> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in
> _prepare_transmission
> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
> clear_xstats)
> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >> > >>>> >>>>>>>     raise TRexError(rc)
> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
> Failed to get server response from tcp://172.18.0.40:4501
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64
> -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> Then start sending traffic with command
> >> > >>>> >>>>>>>
> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>> trex>
> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
> for more than 1.00135 seconds - timeout is 1 seconds
> >> > >>>> >>>>>>> *** traceback follows ***
> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
> [0x561be173cf5a]
> >> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> >> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> >> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> >> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> >> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> >> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned
> char) + 487
> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
> [0x561be16d11ad]
> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >> > >>>> >>>>>>>
> >> > >>>> >>>>>>> *** addr2line information follows ***
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> ??:0
> >> > >>>> >>>>>>> '
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> I believe if everything is working, this should not turn
> off trex but please correct me if I am wrong.
> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach
> this issue?
> >> > >>>> >>>>>>
> >> > >>>> >>>>>> Best,
> >> > >>>> >>>>>> David
> >> > >>>> >>>>>>
> >> > >>>> >>>>>>
>
>

[-- Attachment #2: Type: text/html, Size: 34980 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-05 21:29                                                     ` David Liu
@ 2021-03-05 22:06                                                       ` Ajit Khaparde
  2021-03-08  1:07                                                         ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-03-05 22:06 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 20475 bytes --]

Hi David,
That's a good idea. That way you don't have to watch out for my email
requesting to stop the tests.

Thanks
Ajit

On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit, Juraj,
>
> For simplicity, we will keep the functional and performance testing
> disable during the weekend.
>
> In this way, you will have no testing interfere with you during
> troubleshooting.
>
> Then we will rerun all the jobs that are skipped.
>
> Please let me know if you have any questions.
>
> Best,
> David
>
> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
> wrote:
>
>> Hi David, Ajit,
>>
>>
>>
>> Friday 10am is 4pm for me a that's fine.
>>
>>
>>
>> Juraj
>>
>>
>>
>> *From:* David Liu <dliu@iol.unh.edu>
>> *Sent:* Tuesday, March 2, 2021 7:06 PM
>> *To:* Ajit Khaparde <ajit.khaparde@broadcom.com>
>> *Cc:* Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <
>> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> *Subject:* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm
>> hardware
>>
>>
>>
>> Hi, Ajit
>> I changed it to Friday at 10 AM for now.
>>
>> We can change again if that doesn't work for Juraj
>>
>> Best,
>>
>> David
>>
>>
>>
>>
>>
>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>> Hi David,
>> My Thursdays are crazy. Can we do Friday instead?
>> Juraj, can you make Friday 10AM (Eastern time) call?
>>
>> Thanks
>> Ajit
>>
>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit and Juraj,
>> >
>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to
>> adjust the time or let me know if there is a better time that works for you.
>> >
>> > For me, I can do:
>> > Tuesday 10:00 AM - 1:00 PM.
>> > Thursday 10:00 AM - 1:00 PM.
>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>> >
>> > Best,
>> > David
>> >
>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>
>> >> David,
>> >> I was busy with a customer issue and could not get to my emails.
>> >> I am seeing this now. Let me close out any issues needing urgent
>> attention.
>> >> Let's sync on a suitable time after that.
>> >>
>> >> Juraj,
>> >> Sure. We will add you.
>> >>
>> >> Thanks
>> >> Ajit
>> >>
>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <
>> juraj.linkes@pantheon.tech> wrote:
>> >> >
>> >> > Hi Ajit, David,
>> >> >
>> >> >
>> >> >
>> >> > If you're setting up a call, could you please send me an invite? I'd
>> like to get some insight into the issues on Arm's behalf.
>> >> >
>> >> >
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Juraj
>> >> >
>> >> >
>> >> >
>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <
>> dpdklab@iol.unh.edu>
>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in
>> Arm hardware
>> >> >
>> >> >
>> >> >
>> >> > Hi Ajit,
>> >> >
>> >> >
>> >> >
>> >> > Sorry for this issue.
>> >> >
>> >> > If possible I would like to set up a time where we can do this in a
>> call to help get this solved faster.
>> >> >
>> >> > Please let me know what would be a good time for you.
>> >> >
>> >> >
>> >> >
>> >> > Best,
>> >> >
>> >> > David
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> >
>> >> > Hi Ajit,
>> >> >
>> >> >
>> >> >
>> >> > I changed the permission to allow you to run sudo.
>> >> >
>> >> > This should work now.
>> >> >
>> >> > Let me know if you still have issues.
>> >> >
>> >> >
>> >> >
>> >> > Best,
>> >> >
>> >> > David
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >
>> >> > David,
>> >> >
>> >> > I tried the option to use sudo. And that didn't work.
>> >> >
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Ajit
>> >> >
>> >> >
>> >> >
>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This
>> incident will be reported.
>> >> >
>> >> >
>> >> >
>> >> > ----
>> >> >
>> >> > sudo su
>> >> > [sudo] password for ajitkhaparde:
>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident
>> will be reported.
>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> >> > [sudo] password for ajitkhaparde:
>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident
>> will be reported.
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> >
>> >> > Hi Ajit,
>> >> >
>> >> >
>> >> >
>> >> > This is likely caused by when dts try to ssh into another machine as
>> a normal user which will prompt you for the password.
>> >> >
>> >> >
>> >> >
>> >> > One solution would be "sudo su" as root to run the command line
>> since you should have root access.
>> >> >
>> >> > David,
>> >> >
>> >> > Let me try this method first.
>> >> >
>> >> > If it does not work, we could try with the public ssh keys.
>> >> >
>> >> > I will let you know.
>> >> >
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Ajit
>> >> >
>> >> >
>> >> >
>> >> > Another solution would be to let me add your public ssh key into
>> both machines.
>> >> >
>> >> >
>> >> >
>> >> > Let me know if this helps.
>> >> >
>> >> >
>> >> >
>> >> > Best,
>> >> >
>> >> > David
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >
>> >> > Thanks Owen.
>> >> >
>> >> >
>> >> >
>> >> > When I run the dts command line suggested, I get the following error.
>> >> >
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Ajit
>> >> >
>> >> >
>> >> >
>> >> > Traceback (most recent call last):
>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>> >> >     self.password, original_prompt='[$#>]',
>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>> .+:)')
>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py",
>> line 440, in login
>> >> >     raise ExceptionPxssh('password refused')
>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>> >> >
>> >> >                                          ^^^^^^^^^^^^^^
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
>> wrote:
>> >> >
>> >> > Hello Ajit,
>> >> >
>> >> >
>> >> >
>> >> > There was a misconfiguration in our identity management system. The
>> issue should now be resolved.
>> >> >
>> >> >
>> >> >
>> >> > Owen Hilyard
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
>> dpdklab@iol.unh.edu> wrote:
>> >> >
>> >> > David,
>> >> >
>> >> > I can use my credentials to login to rhea.
>> >> >
>> >> > But the same credentials are not allowing me to login onto the
>> servers you provided.
>> >> >
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Ajit
>> >> >
>> >> >
>> >> >
>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>> >> > Connection closed by 172.18.0.41 port 22
>> >> >
>> >> >
>> >> >
>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>> >> > Connection closed by 172.18.0.40 port 22
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> >
>> >> > Hi Ajit,
>> >> >
>> >> >
>> >> >
>> >> > Thanks for the update. Let me know if you need anything from me.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Best,
>> >> >
>> >> > David
>> >> >
>> >> >
>> >> >
>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >
>> >> > David,
>> >> >
>> >> > I had some trouble with my system over the past week.
>> >> >
>> >> > So I could not login into the setup. I am hoping that my system will
>> be restored early this week
>> >> >
>> >> > I will login into the setup after that.
>> >> >
>> >> >
>> >> >
>> >> > Thanks
>> >> >
>> >> > Ajit
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >
>> >> > Hi David,
>> >> > I actually did miss it. I will take a look.
>> >> > Thanks for the refresh mail.
>> >> >
>> >> > Thanks
>> >> > Ajit
>> >> >
>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >
>> >> > > Hi Ajit,
>> >> > >
>> >> > > Ping again in case if you miss the last email.
>> >> > >
>> >> > > Did you have a chance to give it a try on login to the machines?
>> >> > >
>> >> > > Just some notes, we are currently running compile and some
>> performance/functional testing.
>> >> > > Which it may be hard for you to run dts, so please let me know if
>> you want me to stop these tests for you to debug.
>> >> > >
>> >> > > Step I run:
>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts
>> configuration to Broadcom
>> >> > > ./dts -s
>>  #To start the dts
>> >> > >
>> >> > > Informatio:
>> >> > > arm-dell.dpdklab.iol.unh.edu
>> >> > > - dts - located in /opt/dts
>> >> > > - Trex - located in /opt/trex/v2.87
>> >> > >
>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>> >> > > - dpdk - located in /root/dpdk
>> >> > >
>> >> > > Best,
>> >> > > David
>> >> > >
>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >> > >>
>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>> >> > >> In the dts folder, you can start it with ./dts -s
>> >> > >>
>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >> > >>>
>> >> > >>> Hi Ajit,
>> >> > >>> Sorry for the late reply.
>> >> > >>>
>> >> > >>> You can ssh into the machine using the same VPN Username and
>> Password.
>> >> > >>> Can you try and see if that works?
>> >> > >>>
>> >> > >>> Informatio:
>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>> >> > >>> - dts - located in /opt/dts
>> >> > >>> - Trex - located in /opt/trex/v2.87
>> >> > >>>
>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>> >> > >>> - dpdk - located in /root/dpdk
>> >> > >>>
>> >> > >>>
>> >> > >>> Thanks,
>> >> > >>> David
>> >> > >>>
>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>
>> >> > >>>> Hi David,
>> >> > >>>> Thanks for pinging me. Do you think I could login into the
>> setup and
>> >> > >>>> see what is happening?
>> >> > >>>> I do have the credentials for two systems in your lab -
>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>> >> > >>>> If you can provide me details of the systems you are trying to
>> >> > >>>> configure, I could take a look.
>> >> > >>>>
>> >> > >>>> Thanks
>> >> > >>>> Ajit
>> >> > >>>>
>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >> > >>>> >
>> >> > >>>> > Hi Ajit,
>> >> > >>>> >
>> >> > >>>> > Thank you for helping out.
>> >> > >>>> >
>> >> > >>>> > How are you doing with this issue?
>> >> > >>>> >
>> >> > >>>> > Please let me know if you need anything information from us.
>> >> > >>>> >
>> >> > >>>> > Just some extra information that both Trex and dts are on the
>> same intel tester machine.
>> >> > >>>> >
>> >> > >>>> > Best,
>> >> > >>>> > David
>> >> > >>>> >
>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >> > >>>> >>
>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that
>> are using for Trex. If so they are UP before I start the Trex.
>> >> > >>>> >>
>> >> > >>>> >> Thank,
>> >> > >>>> >> David
>> >> > >>>> >>
>> >> > >>>> >>
>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> > >>>> >>>
>> >> > >>>> >>> Does the same NIC work on x86?
>> >> > >>>> >>> Also do you see link before you start the trex? I am trying
>> to see if the port failed to successfully come up
>> >> > >>>> >>> or if there is a problem with the link. Can you check and
>> tell?
>> >> > >>>> >>>
>> >> > >>>> >>> Thanks
>> >> > >>>> >>> Ajit
>> >> > >>>> >>>
>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >> > >>>> >>>>
>> >> > >>>> >>>> Hi Ajit,
>> >> > >>>> >>>>
>> >> > >>>> >>>> Thank you for helping out.
>> >> > >>>> >>>>
>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>> >> > >>>> >>>>
>> >> > >>>> >>>>
>> >> > >>>> >>>> Thanks,
>> >> > >>>> >>>> David
>> >> > >>>> >>>>
>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> > >>>> >>>>>
>> >> > >>>> >>>>> Hi David,
>> >> > >>>> >>>>> I will take a look.
>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>> >> > >>>> >>>>> even I would start with that to create a baseline and then
>> >> > >>>> >>>>> attempt on ARM.
>> >> > >>>> >>>>>
>> >> > >>>> >>>>> Thanks
>> >> > >>>> >>>>> Ajit
>> >> > >>>> >>>>>
>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>> Hi Ajit,
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>> machine.
>> >> > >>>> >>>>>> And I am running into a problem when I try to run
>> nic_single_core_perf on the NIC.
>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>> http://git.dpdk.org/tools/dts/
>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>> The problem is an error when I try to run the test case
>> inside the nic_single_core_perf:
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>> framesize: 64, rxd/txd: 512
>> >> > >>>> >>>>>>>                dut.172.18.0.41:
>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>> _execute_test_case
>> >> > >>>> >>>>>>>     case_obj()
>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>> 196, in test_perf_nic_single_core
>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>> 270, in perf_test
>> >> > >>>> >>>>>>>     _, packets_received =
>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>> options=traffic_opt)
>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245,
>> in measure_throughput
>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778,
>> in _prepare_transmission
>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 339, in reset
>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 1492, in clear_stats
>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>> clear_xstats)
>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>> line 2876, in _clear_stats_common
>> >> > >>>> >>>>>>>     raise TRexError(rc)
>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
>> Failed to get server response from tcp://172.18.0.40:4501
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64
>> -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>> Then start sending traffic with command
>> >> > >>>> >>>>>>>
>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>>> trex>
>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
>> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
>> for more than 1.00135 seconds - timeout is 1 seconds
>> >> > >>>> >>>>>>> *** traceback follows ***
>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>> [0x561be173cf5a]
>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> >> > >>>> >>>>>>> 4       0x561be16ce874
>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>> CVirtualIFPerSideStats*) + 3220
>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> >> > >>>> >>>>>>> 6       0x561be171e3d9
>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> >> > >>>> >>>>>>> 7       0x561be171f7ec
>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>> + 92
>> >> > >>>> >>>>>>> 8       0x561be171fc98
>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>> CFlowGenListPerThread*, bool) + 184
>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>> CFlowGenListPerThread*, double&) + 881
>> >> > >>>> >>>>>>> 10      0x561be1905212
>> TrexStatelessDpCore::start_scheduler() + 226
>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> >> > >>>> >>>>>>> 12      0x561be1714113
>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> >> > >>>> >>>>>>> 13      0x561be16af8dd
>> CGlobalTRex::run_in_core(unsigned char) + 487
>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>> [0x561be16d11ad]
>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> >> > >>>> >>>>>>>
>> >> > >>>> >>>>>>> *** addr2line information follows ***
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> ??:0
>> >> > >>>> >>>>>>> '
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>> I believe if everything is working, this should not turn
>> off trex but please correct me if I am wrong.
>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can
>> approach this issue?
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>> Best,
>> >> > >>>> >>>>>> David
>> >> > >>>> >>>>>>
>> >> > >>>> >>>>>>
>>
>>

[-- Attachment #1.2: Type: text/html, Size: 35764 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-05 22:06                                                       ` Ajit Khaparde
@ 2021-03-08  1:07                                                         ` Ajit Khaparde
  2021-03-09 14:23                                                           ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-03-08  1:07 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 21611 bytes --]

David,
I did spend some time on this. For now I can't tell what is wrong.
But I want to update the firmware on the card to take another shot at it.
Since this is an ARM based server, I will have to find a way to update the
firmware.
IIRC our tools are mostly x86 based. So I am looking around.
I will update you soon.

Thanks
Ajit

On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi David,
> That's a good idea. That way you don't have to watch out for my email
> requesting to stop the tests.
>
> Thanks
> Ajit
>
> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
>
>> Hi Ajit, Juraj,
>>
>> For simplicity, we will keep the functional and performance testing
>> disable during the weekend.
>>
>> In this way, you will have no testing interfere with you during
>> troubleshooting.
>>
>> Then we will rerun all the jobs that are skipped.
>>
>> Please let me know if you have any questions.
>>
>> Best,
>> David
>>
>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
>> wrote:
>>
>>> Hi David, Ajit,
>>>
>>>
>>>
>>> Friday 10am is 4pm for me a that's fine.
>>>
>>>
>>>
>>> Juraj
>>>
>>>
>>>
>>> *From:* David Liu <dliu@iol.unh.edu>
>>> *Sent:* Tuesday, March 2, 2021 7:06 PM
>>> *To:* Ajit Khaparde <ajit.khaparde@broadcom.com>
>>> *Cc:* Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <
>>> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>>> *Subject:* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in
>>> Arm hardware
>>>
>>>
>>>
>>> Hi, Ajit
>>> I changed it to Friday at 10 AM for now.
>>>
>>> We can change again if that doesn't work for Juraj
>>>
>>> Best,
>>>
>>> David
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>>
>>> Hi David,
>>> My Thursdays are crazy. Can we do Friday instead?
>>> Juraj, can you make Friday 10AM (Eastern time) call?
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >
>>> > Hi Ajit and Juraj,
>>> >
>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to
>>> adjust the time or let me know if there is a better time that works for you.
>>> >
>>> > For me, I can do:
>>> > Tuesday 10:00 AM - 1:00 PM.
>>> > Thursday 10:00 AM - 1:00 PM.
>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>>> >
>>> > Best,
>>> > David
>>> >
>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>
>>> >> David,
>>> >> I was busy with a customer issue and could not get to my emails.
>>> >> I am seeing this now. Let me close out any issues needing urgent
>>> attention.
>>> >> Let's sync on a suitable time after that.
>>> >>
>>> >> Juraj,
>>> >> Sure. We will add you.
>>> >>
>>> >> Thanks
>>> >> Ajit
>>> >>
>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <
>>> juraj.linkes@pantheon.tech> wrote:
>>> >> >
>>> >> > Hi Ajit, David,
>>> >> >
>>> >> >
>>> >> >
>>> >> > If you're setting up a call, could you please send me an invite?
>>> I'd like to get some insight into the issues on Arm's behalf.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks,
>>> >> >
>>> >> > Juraj
>>> >> >
>>> >> >
>>> >> >
>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <
>>> dpdklab@iol.unh.edu>
>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in
>>> Arm hardware
>>> >> >
>>> >> >
>>> >> >
>>> >> > Hi Ajit,
>>> >> >
>>> >> >
>>> >> >
>>> >> > Sorry for this issue.
>>> >> >
>>> >> > If possible I would like to set up a time where we can do this in a
>>> call to help get this solved faster.
>>> >> >
>>> >> > Please let me know what would be a good time for you.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Best,
>>> >> >
>>> >> > David
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> >
>>> >> > Hi Ajit,
>>> >> >
>>> >> >
>>> >> >
>>> >> > I changed the permission to allow you to run sudo.
>>> >> >
>>> >> > This should work now.
>>> >> >
>>> >> > Let me know if you still have issues.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Best,
>>> >> >
>>> >> > David
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >
>>> >> > David,
>>> >> >
>>> >> > I tried the option to use sudo. And that didn't work.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks
>>> >> >
>>> >> > Ajit
>>> >> >
>>> >> >
>>> >> >
>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This
>>> incident will be reported.
>>> >> >
>>> >> >
>>> >> >
>>> >> > ----
>>> >> >
>>> >> > sudo su
>>> >> > [sudo] password for ajitkhaparde:
>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident
>>> will be reported.
>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>>> >> > [sudo] password for ajitkhaparde:
>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident
>>> will be reported.
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> >
>>> >> > Hi Ajit,
>>> >> >
>>> >> >
>>> >> >
>>> >> > This is likely caused by when dts try to ssh into another machine
>>> as a normal user which will prompt you for the password.
>>> >> >
>>> >> >
>>> >> >
>>> >> > One solution would be "sudo su" as root to run the command line
>>> since you should have root access.
>>> >> >
>>> >> > David,
>>> >> >
>>> >> > Let me try this method first.
>>> >> >
>>> >> > If it does not work, we could try with the public ssh keys.
>>> >> >
>>> >> > I will let you know.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks
>>> >> >
>>> >> > Ajit
>>> >> >
>>> >> >
>>> >> >
>>> >> > Another solution would be to let me add your public ssh key into
>>> both machines.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Let me know if this helps.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Best,
>>> >> >
>>> >> > David
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >
>>> >> > Thanks Owen.
>>> >> >
>>> >> >
>>> >> >
>>> >> > When I run the dts command line suggested, I get the following
>>> error.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks
>>> >> >
>>> >> > Ajit
>>> >> >
>>> >> >
>>> >> >
>>> >> > Traceback (most recent call last):
>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in
>>> _connect_host
>>> >> >     self.password, original_prompt='[$#>]',
>>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>>> .+:)')
>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py",
>>> line 440, in login
>>> >> >     raise ExceptionPxssh('password refused')
>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>>> >> >
>>> >> >                                          ^^^^^^^^^^^^^^
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu>
>>> wrote:
>>> >> >
>>> >> > Hello Ajit,
>>> >> >
>>> >> >
>>> >> >
>>> >> > There was a misconfiguration in our identity management system. The
>>> issue should now be resolved.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Owen Hilyard
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
>>> dpdklab@iol.unh.edu> wrote:
>>> >> >
>>> >> > David,
>>> >> >
>>> >> > I can use my credentials to login to rhea.
>>> >> >
>>> >> > But the same credentials are not allowing me to login onto the
>>> servers you provided.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks
>>> >> >
>>> >> > Ajit
>>> >> >
>>> >> >
>>> >> >
>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>> >> > Connection closed by 172.18.0.41 port 22
>>> >> >
>>> >> >
>>> >> >
>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>> >> > Connection closed by 172.18.0.40 port 22
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >> >
>>> >> > Hi Ajit,
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks for the update. Let me know if you need anything from me.
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> > Best,
>>> >> >
>>> >> > David
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >
>>> >> > David,
>>> >> >
>>> >> > I had some trouble with my system over the past week.
>>> >> >
>>> >> > So I could not login into the setup. I am hoping that my system
>>> will be restored early this week
>>> >> >
>>> >> > I will login into the setup after that.
>>> >> >
>>> >> >
>>> >> >
>>> >> > Thanks
>>> >> >
>>> >> > Ajit
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >
>>> >> > Hi David,
>>> >> > I actually did miss it. I will take a look.
>>> >> > Thanks for the refresh mail.
>>> >> >
>>> >> > Thanks
>>> >> > Ajit
>>> >> >
>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >> > >
>>> >> > > Hi Ajit,
>>> >> > >
>>> >> > > Ping again in case if you miss the last email.
>>> >> > >
>>> >> > > Did you have a chance to give it a try on login to the machines?
>>> >> > >
>>> >> > > Just some notes, we are currently running compile and some
>>> performance/functional testing.
>>> >> > > Which it may be hard for you to run dts, so please let me know if
>>> you want me to stop these tests for you to debug.
>>> >> > >
>>> >> > > Step I run:
>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the
>>> dts configuration to Broadcom
>>> >> > > ./dts -s
>>>  #To start the dts
>>> >> > >
>>> >> > > Informatio:
>>> >> > > arm-dell.dpdklab.iol.unh.edu
>>> >> > > - dts - located in /opt/dts
>>> >> > > - Trex - located in /opt/trex/v2.87
>>> >> > >
>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>>> >> > > - dpdk - located in /root/dpdk
>>> >> > >
>>> >> > > Best,
>>> >> > > David
>>> >> > >
>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> > >>
>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>>> >> > >> In the dts folder, you can start it with ./dts -s
>>> >> > >>
>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> > >>>
>>> >> > >>> Hi Ajit,
>>> >> > >>> Sorry for the late reply.
>>> >> > >>>
>>> >> > >>> You can ssh into the machine using the same VPN Username and
>>> Password.
>>> >> > >>> Can you try and see if that works?
>>> >> > >>>
>>> >> > >>> Informatio:
>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>>> >> > >>> - dts - located in /opt/dts
>>> >> > >>> - Trex - located in /opt/trex/v2.87
>>> >> > >>>
>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>>> >> > >>> - dpdk - located in /root/dpdk
>>> >> > >>>
>>> >> > >>>
>>> >> > >>> Thanks,
>>> >> > >>> David
>>> >> > >>>
>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> > >>>>
>>> >> > >>>> Hi David,
>>> >> > >>>> Thanks for pinging me. Do you think I could login into the
>>> setup and
>>> >> > >>>> see what is happening?
>>> >> > >>>> I do have the credentials for two systems in your lab -
>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>> >> > >>>> If you can provide me details of the systems you are trying to
>>> >> > >>>> configure, I could take a look.
>>> >> > >>>>
>>> >> > >>>> Thanks
>>> >> > >>>> Ajit
>>> >> > >>>>
>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> > >>>> >
>>> >> > >>>> > Hi Ajit,
>>> >> > >>>> >
>>> >> > >>>> > Thank you for helping out.
>>> >> > >>>> >
>>> >> > >>>> > How are you doing with this issue?
>>> >> > >>>> >
>>> >> > >>>> > Please let me know if you need anything information from us.
>>> >> > >>>> >
>>> >> > >>>> > Just some extra information that both Trex and dts are on
>>> the same intel tester machine.
>>> >> > >>>> >
>>> >> > >>>> > Best,
>>> >> > >>>> > David
>>> >> > >>>> >
>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> > >>>> >>
>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that
>>> are using for Trex. If so they are UP before I start the Trex.
>>> >> > >>>> >>
>>> >> > >>>> >> Thank,
>>> >> > >>>> >> David
>>> >> > >>>> >>
>>> >> > >>>> >>
>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> > >>>> >>>
>>> >> > >>>> >>> Does the same NIC work on x86?
>>> >> > >>>> >>> Also do you see link before you start the trex? I am
>>> trying to see if the port failed to successfully come up
>>> >> > >>>> >>> or if there is a problem with the link. Can you check and
>>> tell?
>>> >> > >>>> >>>
>>> >> > >>>> >>> Thanks
>>> >> > >>>> >>> Ajit
>>> >> > >>>> >>>
>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> > >>>> >>>>
>>> >> > >>>> >>>> Hi Ajit,
>>> >> > >>>> >>>>
>>> >> > >>>> >>>> Thank you for helping out.
>>> >> > >>>> >>>>
>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>>> >> > >>>> >>>>
>>> >> > >>>> >>>>
>>> >> > >>>> >>>> Thanks,
>>> >> > >>>> >>>> David
>>> >> > >>>> >>>>
>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> > >>>> >>>>>
>>> >> > >>>> >>>>> Hi David,
>>> >> > >>>> >>>>> I will take a look.
>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>>> >> > >>>> >>>>> even I would start with that to create a baseline and
>>> then
>>> >> > >>>> >>>>> attempt on ARM.
>>> >> > >>>> >>>>>
>>> >> > >>>> >>>>> Thanks
>>> >> > >>>> >>>>> Ajit
>>> >> > >>>> >>>>>
>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>> Hi Ajit,
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>>> machine.
>>> >> > >>>> >>>>>> And I am running into a problem when I try to run
>>> nic_single_core_perf on the NIC.
>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>> http://git.dpdk.org/tools/dts/
>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case
>>> inside the nic_single_core_perf:
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>> framesize: 64, rxd/txd: 512
>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in
>>> _execute_test_case
>>> >> > >>>> >>>>>>>     case_obj()
>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>> 196, in test_perf_nic_single_core
>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line
>>> 270, in perf_test
>>> >> > >>>> >>>>>>>     _, packets_received =
>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>> options=traffic_opt)
>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245,
>>> in measure_throughput
>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778,
>>> in _prepare_transmission
>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 339, in reset
>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 1492, in clear_stats
>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>> clear_xstats)
>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>> line 2876, in _clear_stats_common
>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
>>> Failed to get server response from tcp://172.18.0.40:4501
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64
>>> -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>> Then start sending traffic with command
>>> >> > >>>> >>>>>>>
>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>>> trex>
>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
>>> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
>>> for more than 1.00135 seconds - timeout is 1 seconds
>>> >> > >>>> >>>>>>> *** traceback follows ***
>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>> [0x561be173cf5a]
>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>> >> > >>>> >>>>>>> 4       0x561be16ce874
>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>> CVirtualIFPerSideStats*) + 3220
>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() +
>>> 31
>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>> + 92
>>> >> > >>>> >>>>>>> 8       0x561be171fc98
>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>> CFlowGenListPerThread*, bool) + 184
>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>> CFlowGenListPerThread*, double&) + 881
>>> >> > >>>> >>>>>>> 10      0x561be1905212
>>> TrexStatelessDpCore::start_scheduler() + 226
>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>> >> > >>>> >>>>>>> 12      0x561be1714113
>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
>>> CGlobalTRex::run_in_core(unsigned char) + 487
>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>> [0x561be16d11ad]
>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>> >> > >>>> >>>>>>>
>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> ??:0
>>> >> > >>>> >>>>>>> '
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>> I believe if everything is working, this should not
>>> turn off trex but please correct me if I am wrong.
>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can
>>> approach this issue?
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>> Best,
>>> >> > >>>> >>>>>> David
>>> >> > >>>> >>>>>>
>>> >> > >>>> >>>>>>
>>>
>>>

[-- Attachment #1.2: Type: text/html, Size: 37119 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-08  1:07                                                         ` Ajit Khaparde
@ 2021-03-09 14:23                                                           ` David Liu
  2021-03-09 16:56                                                             ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-03-09 14:23 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 22545 bytes --]

Hi Ajit,

Thank for the update.

May I know if I can enable testing or would you like more time to look into
this?

Best,
David

On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> David,
> I did spend some time on this. For now I can't tell what is wrong.
> But I want to update the firmware on the card to take another shot at it.
> Since this is an ARM based server, I will have to find a way to update the
> firmware.
> IIRC our tools are mostly x86 based. So I am looking around.
> I will update you soon.
>
> Thanks
> Ajit
>
> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Hi David,
>> That's a good idea. That way you don't have to watch out for my email
>> requesting to stop the tests.
>>
>> Thanks
>> Ajit
>>
>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
>>
>>> Hi Ajit, Juraj,
>>>
>>> For simplicity, we will keep the functional and performance testing
>>> disable during the weekend.
>>>
>>> In this way, you will have no testing interfere with you during
>>> troubleshooting.
>>>
>>> Then we will rerun all the jobs that are skipped.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Best,
>>> David
>>>
>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
>>> wrote:
>>>
>>>> Hi David, Ajit,
>>>>
>>>>
>>>>
>>>> Friday 10am is 4pm for me a that's fine.
>>>>
>>>>
>>>>
>>>> Juraj
>>>>
>>>>
>>>>
>>>> *From:* David Liu <dliu@iol.unh.edu>
>>>> *Sent:* Tuesday, March 2, 2021 7:06 PM
>>>> *To:* Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>> *Cc:* Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <
>>>> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>>>> *Subject:* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in
>>>> Arm hardware
>>>>
>>>>
>>>>
>>>> Hi, Ajit
>>>> I changed it to Friday at 10 AM for now.
>>>>
>>>> We can change again if that doesn't work for Juraj
>>>>
>>>> Best,
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>>
>>>> Hi David,
>>>> My Thursdays are crazy. Can we do Friday instead?
>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >
>>>> > Hi Ajit and Juraj,
>>>> >
>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free
>>>> to adjust the time or let me know if there is a better time that works for
>>>> you.
>>>> >
>>>> > For me, I can do:
>>>> > Tuesday 10:00 AM - 1:00 PM.
>>>> > Thursday 10:00 AM - 1:00 PM.
>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>>>> >
>>>> > Best,
>>>> > David
>>>> >
>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >>
>>>> >> David,
>>>> >> I was busy with a customer issue and could not get to my emails.
>>>> >> I am seeing this now. Let me close out any issues needing urgent
>>>> attention.
>>>> >> Let's sync on a suitable time after that.
>>>> >>
>>>> >> Juraj,
>>>> >> Sure. We will add you.
>>>> >>
>>>> >> Thanks
>>>> >> Ajit
>>>> >>
>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <
>>>> juraj.linkes@pantheon.tech> wrote:
>>>> >> >
>>>> >> > Hi Ajit, David,
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > If you're setting up a call, could you please send me an invite?
>>>> I'd like to get some insight into the issues on Arm's behalf.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Thanks,
>>>> >> >
>>>> >> > Juraj
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <
>>>> dpdklab@iol.unh.edu>
>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in
>>>> Arm hardware
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Hi Ajit,
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Sorry for this issue.
>>>> >> >
>>>> >> > If possible I would like to set up a time where we can do this in
>>>> a call to help get this solved faster.
>>>> >> >
>>>> >> > Please let me know what would be a good time for you.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Best,
>>>> >> >
>>>> >> > David
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu>
>>>> wrote:
>>>> >> >
>>>> >> > Hi Ajit,
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > I changed the permission to allow you to run sudo.
>>>> >> >
>>>> >> > This should work now.
>>>> >> >
>>>> >> > Let me know if you still have issues.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Best,
>>>> >> >
>>>> >> > David
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >> >
>>>> >> > David,
>>>> >> >
>>>> >> > I tried the option to use sudo. And that didn't work.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Thanks
>>>> >> >
>>>> >> > Ajit
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This
>>>> incident will be reported.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > ----
>>>> >> >
>>>> >> > sudo su
>>>> >> > [sudo] password for ajitkhaparde:
>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This
>>>> incident will be reported.
>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>>>> >> > [sudo] password for ajitkhaparde:
>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This
>>>> incident will be reported.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu>
>>>> wrote:
>>>> >> >
>>>> >> > Hi Ajit,
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > This is likely caused by when dts try to ssh into another machine
>>>> as a normal user which will prompt you for the password.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > One solution would be "sudo su" as root to run the command line
>>>> since you should have root access.
>>>> >> >
>>>> >> > David,
>>>> >> >
>>>> >> > Let me try this method first.
>>>> >> >
>>>> >> > If it does not work, we could try with the public ssh keys.
>>>> >> >
>>>> >> > I will let you know.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Thanks
>>>> >> >
>>>> >> > Ajit
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Another solution would be to let me add your public ssh key into
>>>> both machines.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Let me know if this helps.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Best,
>>>> >> >
>>>> >> > David
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >> >
>>>> >> > Thanks Owen.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > When I run the dts command line suggested, I get the following
>>>> error.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Thanks
>>>> >> >
>>>> >> > Ajit
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Traceback (most recent call last):
>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in
>>>> _connect_host
>>>> >> >     self.password, original_prompt='[$#>]',
>>>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>>>> .+:)')
>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py",
>>>> line 440, in login
>>>> >> >     raise ExceptionPxssh('password refused')
>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>>>> >> >
>>>> >> >                                          ^^^^^^^^^^^^^^
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
>>>> ohilyard@iol.unh.edu> wrote:
>>>> >> >
>>>> >> > Hello Ajit,
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > There was a misconfiguration in our identity management system.
>>>> The issue should now be resolved.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Owen Hilyard
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
>>>> dpdklab@iol.unh.edu> wrote:
>>>> >> >
>>>> >> > David,
>>>> >> >
>>>> >> > I can use my credentials to login to rhea.
>>>> >> >
>>>> >> > But the same credentials are not allowing me to login onto the
>>>> servers you provided.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Thanks
>>>> >> >
>>>> >> > Ajit
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>>> >> > Connection closed by 172.18.0.41 port 22
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>>> >> > Connection closed by 172.18.0.40 port 22
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >
>>>> >> > Hi Ajit,
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Thanks for the update. Let me know if you need anything from me.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Best,
>>>> >> >
>>>> >> > David
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >> >
>>>> >> > David,
>>>> >> >
>>>> >> > I had some trouble with my system over the past week.
>>>> >> >
>>>> >> > So I could not login into the setup. I am hoping that my system
>>>> will be restored early this week
>>>> >> >
>>>> >> > I will login into the setup after that.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > Thanks
>>>> >> >
>>>> >> > Ajit
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >> >
>>>> >> > Hi David,
>>>> >> > I actually did miss it. I will take a look.
>>>> >> > Thanks for the refresh mail.
>>>> >> >
>>>> >> > Thanks
>>>> >> > Ajit
>>>> >> >
>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> > >
>>>> >> > > Hi Ajit,
>>>> >> > >
>>>> >> > > Ping again in case if you miss the last email.
>>>> >> > >
>>>> >> > > Did you have a chance to give it a try on login to the machines?
>>>> >> > >
>>>> >> > > Just some notes, we are currently running compile and some
>>>> performance/functional testing.
>>>> >> > > Which it may be hard for you to run dts, so please let me know
>>>> if you want me to stop these tests for you to debug.
>>>> >> > >
>>>> >> > > Step I run:
>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the
>>>> dts configuration to Broadcom
>>>> >> > > ./dts -s
>>>>    #To start the dts
>>>> >> > >
>>>> >> > > Informatio:
>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>>>> >> > > - dts - located in /opt/dts
>>>> >> > > - Trex - located in /opt/trex/v2.87
>>>> >> > >
>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>>>> >> > > - dpdk - located in /root/dpdk
>>>> >> > >
>>>> >> > > Best,
>>>> >> > > David
>>>> >> > >
>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu>
>>>> wrote:
>>>> >> > >>
>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>>>> >> > >> In the dts folder, you can start it with ./dts -s
>>>> >> > >>
>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu>
>>>> wrote:
>>>> >> > >>>
>>>> >> > >>> Hi Ajit,
>>>> >> > >>> Sorry for the late reply.
>>>> >> > >>>
>>>> >> > >>> You can ssh into the machine using the same VPN Username and
>>>> Password.
>>>> >> > >>> Can you try and see if that works?
>>>> >> > >>>
>>>> >> > >>> Informatio:
>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>>>> >> > >>> - dts - located in /opt/dts
>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>>>> >> > >>>
>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>> >> > >>> - dpdk - located in /root/dpdk
>>>> >> > >>>
>>>> >> > >>>
>>>> >> > >>> Thanks,
>>>> >> > >>> David
>>>> >> > >>>
>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >> > >>>>
>>>> >> > >>>> Hi David,
>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the
>>>> setup and
>>>> >> > >>>> see what is happening?
>>>> >> > >>>> I do have the credentials for two systems in your lab -
>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>> >> > >>>> If you can provide me details of the systems you are trying to
>>>> >> > >>>> configure, I could take a look.
>>>> >> > >>>>
>>>> >> > >>>> Thanks
>>>> >> > >>>> Ajit
>>>> >> > >>>>
>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu>
>>>> wrote:
>>>> >> > >>>> >
>>>> >> > >>>> > Hi Ajit,
>>>> >> > >>>> >
>>>> >> > >>>> > Thank you for helping out.
>>>> >> > >>>> >
>>>> >> > >>>> > How are you doing with this issue?
>>>> >> > >>>> >
>>>> >> > >>>> > Please let me know if you need anything information from us.
>>>> >> > >>>> >
>>>> >> > >>>> > Just some extra information that both Trex and dts are on
>>>> the same intel tester machine.
>>>> >> > >>>> >
>>>> >> > >>>> > Best,
>>>> >> > >>>> > David
>>>> >> > >>>> >
>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu>
>>>> wrote:
>>>> >> > >>>> >>
>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that
>>>> are using for Trex. If so they are UP before I start the Trex.
>>>> >> > >>>> >>
>>>> >> > >>>> >> Thank,
>>>> >> > >>>> >> David
>>>> >> > >>>> >>
>>>> >> > >>>> >>
>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >> > >>>> >>>
>>>> >> > >>>> >>> Does the same NIC work on x86?
>>>> >> > >>>> >>> Also do you see link before you start the trex? I am
>>>> trying to see if the port failed to successfully come up
>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and
>>>> tell?
>>>> >> > >>>> >>>
>>>> >> > >>>> >>> Thanks
>>>> >> > >>>> >>> Ajit
>>>> >> > >>>> >>>
>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
>>>> dliu@iol.unh.edu> wrote:
>>>> >> > >>>> >>>>
>>>> >> > >>>> >>>> Hi Ajit,
>>>> >> > >>>> >>>>
>>>> >> > >>>> >>>> Thank you for helping out.
>>>> >> > >>>> >>>>
>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>>>> >> > >>>> >>>>
>>>> >> > >>>> >>>>
>>>> >> > >>>> >>>> Thanks,
>>>> >> > >>>> >>>> David
>>>> >> > >>>> >>>>
>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
>>>> ajit.khaparde@broadcom.com> wrote:
>>>> >> > >>>> >>>>>
>>>> >> > >>>> >>>>> Hi David,
>>>> >> > >>>> >>>>> I will take a look.
>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>> >> > >>>> >>>>> even I would start with that to create a baseline and
>>>> then
>>>> >> > >>>> >>>>> attempt on ARM.
>>>> >> > >>>> >>>>>
>>>> >> > >>>> >>>>> Thanks
>>>> >> > >>>> >>>>> Ajit
>>>> >> > >>>> >>>>>
>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>>>> dliu@iol.unh.edu> wrote:
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>> Hi Ajit,
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm
>>>> machine.
>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run
>>>> nic_single_core_perf on the NIC.
>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
>>>> http://git.dpdk.org/tools/dts/
>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test
>>>> case inside the nic_single_core_perf:
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters:
>>>> framesize: 64, rxd/txd: 512
>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
>>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>>>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319,
>>>> in _execute_test_case
>>>> >> > >>>> >>>>>>>     case_obj()
>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py",
>>>> line 196, in test_perf_nic_single_core
>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py",
>>>> line 270, in perf_test
>>>> >> > >>>> >>>>>>>     _, packets_received =
>>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>>> options=traffic_opt)
>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245,
>>>> in measure_throughput
>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778,
>>>> in _prepare_transmission
>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>> >> > >>>> >>>>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>> line 51, in wrap2
>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >> > >>>> >>>>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>> line 339, in reset
>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>>>> >> > >>>> >>>>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>>> line 51, in wrap2
>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >> > >>>> >>>>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>>> line 1492, in clear_stats
>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global,
>>>> clear_xstats)
>>>> >> > >>>> >>>>>>>   File
>>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>>> line 2876, in _clear_stats_common
>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] -
>>>> Failed to get server response from tcp://172.18.0.40:4501
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode
>>>> with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>> Then start sending traffic with command
>>>> >> > >>>> >>>>>>>
>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>>> trex>
>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has
>>>> been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded
>>>> for more than 1.00135 seconds - timeout is 1 seconds
>>>> >> > >>>> >>>>>>> *** traceback follows ***
>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
>>>> [0x561be173cf5a]
>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
>>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>>> CVirtualIFPerSideStats*) + 3220
>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() +
>>>> 31
>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
>>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
>>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>>> + 92
>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
>>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>>> CFlowGenListPerThread*, bool) + 184
>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
>>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>>> CFlowGenListPerThread*, double&) + 881
>>>> >> > >>>> >>>>>>> 10      0x561be1905212
>>>> TrexStatelessDpCore::start_scheduler() + 226
>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>> >> > >>>> >>>>>>> 12      0x561be1714113
>>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
>>>> CGlobalTRex::run_in_core(unsigned char) + 487
>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
>>>> [0x561be16d11ad]
>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>> >> > >>>> >>>>>>>
>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> ??:0
>>>> >> > >>>> >>>>>>> '
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>> I believe if everything is working, this should not
>>>> turn off trex but please correct me if I am wrong.
>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can
>>>> approach this issue?
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>> Best,
>>>> >> > >>>> >>>>>> David
>>>> >> > >>>> >>>>>>
>>>> >> > >>>> >>>>>>
>>>>
>>>>

[-- Attachment #2: Type: text/html, Size: 37731 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-09 14:23                                                           ` David Liu
@ 2021-03-09 16:56                                                             ` Ajit Khaparde
  2021-03-09 19:00                                                               ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-03-09 16:56 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 22719 bytes --]

Hi David,
I am about to login into the setup and try something.
I will let you know once I am done and then you can take control.

Thanks
Ajit

On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
> Thank for the update.
>
> May I know if I can enable testing or would you like more time to look into this?
>
> Best,
> David
>
> On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>
>> David,
>> I did spend some time on this. For now I can't tell what is wrong.
>> But I want to update the firmware on the card to take another shot at it.
>> Since this is an ARM based server, I will have to find a way to update the firmware.
>> IIRC our tools are mostly x86 based. So I am looking around.
>> I will update you soon.
>>
>> Thanks
>> Ajit
>>
>> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>
>>> Hi David,
>>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>
>>>> Hi Ajit, Juraj,
>>>>
>>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
>>>>
>>>> In this way, you will have no testing interfere with you during troubleshooting.
>>>>
>>>> Then we will rerun all the jobs that are skipped.
>>>>
>>>> Please let me know if you have any questions.
>>>>
>>>> Best,
>>>> David
>>>>
>>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>>>>>
>>>>> Hi David, Ajit,
>>>>>
>>>>>
>>>>>
>>>>> Friday 10am is 4pm for me a that's fine.
>>>>>
>>>>>
>>>>>
>>>>> Juraj
>>>>>
>>>>>
>>>>>
>>>>> From: David Liu <dliu@iol.unh.edu>
>>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>>>>
>>>>>
>>>>>
>>>>> Hi, Ajit
>>>>> I changed it to Friday at 10 AM for now.
>>>>>
>>>>> We can change again if that doesn't work for Juraj
>>>>>
>>>>> Best,
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>>
>>>>> Hi David,
>>>>> My Thursdays are crazy. Can we do Friday instead?
>>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>>>>>
>>>>> Thanks
>>>>> Ajit
>>>>>
>>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >
>>>>> > Hi Ajit and Juraj,
>>>>> >
>>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>>>>> >
>>>>> > For me, I can do:
>>>>> > Tuesday 10:00 AM - 1:00 PM.
>>>>> > Thursday 10:00 AM - 1:00 PM.
>>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>>>>> >
>>>>> > Best,
>>>>> > David
>>>>> >
>>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >>
>>>>> >> David,
>>>>> >> I was busy with a customer issue and could not get to my emails.
>>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
>>>>> >> Let's sync on a suitable time after that.
>>>>> >>
>>>>> >> Juraj,
>>>>> >> Sure. We will add you.
>>>>> >>
>>>>> >> Thanks
>>>>> >> Ajit
>>>>> >>
>>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>>>>> >> >
>>>>> >> > Hi Ajit, David,
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> >
>>>>> >> > Juraj
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Hi Ajit,
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Sorry for this issue.
>>>>> >> >
>>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>>>>> >> >
>>>>> >> > Please let me know what would be a good time for you.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Best,
>>>>> >> >
>>>>> >> > David
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> >
>>>>> >> > Hi Ajit,
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > I changed the permission to allow you to run sudo.
>>>>> >> >
>>>>> >> > This should work now.
>>>>> >> >
>>>>> >> > Let me know if you still have issues.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Best,
>>>>> >> >
>>>>> >> > David
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >> >
>>>>> >> > David,
>>>>> >> >
>>>>> >> > I tried the option to use sudo. And that didn't work.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks
>>>>> >> >
>>>>> >> > Ajit
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > ----
>>>>> >> >
>>>>> >> > sudo su
>>>>> >> > [sudo] password for ajitkhaparde:
>>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>>>>> >> > [sudo] password for ajitkhaparde:
>>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> >
>>>>> >> > Hi Ajit,
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
>>>>> >> >
>>>>> >> > David,
>>>>> >> >
>>>>> >> > Let me try this method first.
>>>>> >> >
>>>>> >> > If it does not work, we could try with the public ssh keys.
>>>>> >> >
>>>>> >> > I will let you know.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks
>>>>> >> >
>>>>> >> > Ajit
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Another solution would be to let me add your public ssh key into both machines.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Let me know if this helps.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Best,
>>>>> >> >
>>>>> >> > David
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >> >
>>>>> >> > Thanks Owen.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > When I run the dts command line suggested, I get the following error.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks
>>>>> >> >
>>>>> >> > Ajit
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Traceback (most recent call last):
>>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>>>>> >> >     raise ExceptionPxssh('password refused')
>>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>>>>> >> >
>>>>> >> >                                          ^^^^^^^^^^^^^^
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
>>>>> >> >
>>>>> >> > Hello Ajit,
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Owen Hilyard
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
>>>>> >> >
>>>>> >> > David,
>>>>> >> >
>>>>> >> > I can use my credentials to login to rhea.
>>>>> >> >
>>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks
>>>>> >> >
>>>>> >> > Ajit
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>>>> >> > Connection closed by 172.18.0.41 port 22
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>>>> >> > Connection closed by 172.18.0.40 port 22
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> >
>>>>> >> > Hi Ajit,
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks for the update. Let me know if you need anything from me.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Best,
>>>>> >> >
>>>>> >> > David
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >> >
>>>>> >> > David,
>>>>> >> >
>>>>> >> > I had some trouble with my system over the past week.
>>>>> >> >
>>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
>>>>> >> >
>>>>> >> > I will login into the setup after that.
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > Thanks
>>>>> >> >
>>>>> >> > Ajit
>>>>> >> >
>>>>> >> >
>>>>> >> >
>>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >> >
>>>>> >> > Hi David,
>>>>> >> > I actually did miss it. I will take a look.
>>>>> >> > Thanks for the refresh mail.
>>>>> >> >
>>>>> >> > Thanks
>>>>> >> > Ajit
>>>>> >> >
>>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> > >
>>>>> >> > > Hi Ajit,
>>>>> >> > >
>>>>> >> > > Ping again in case if you miss the last email.
>>>>> >> > >
>>>>> >> > > Did you have a chance to give it a try on login to the machines?
>>>>> >> > >
>>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
>>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>>>>> >> > >
>>>>> >> > > Step I run:
>>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>>>>> >> > > ./dts -s                                                           #To start the dts
>>>>> >> > >
>>>>> >> > > Informatio:
>>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>>>>> >> > > - dts - located in /opt/dts
>>>>> >> > > - Trex - located in /opt/trex/v2.87
>>>>> >> > >
>>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>>>>> >> > > - dpdk - located in /root/dpdk
>>>>> >> > >
>>>>> >> > > Best,
>>>>> >> > > David
>>>>> >> > >
>>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> > >>
>>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>>>>> >> > >> In the dts folder, you can start it with ./dts -s
>>>>> >> > >>
>>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> > >>>
>>>>> >> > >>> Hi Ajit,
>>>>> >> > >>> Sorry for the late reply.
>>>>> >> > >>>
>>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
>>>>> >> > >>> Can you try and see if that works?
>>>>> >> > >>>
>>>>> >> > >>> Informatio:
>>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>>>>> >> > >>> - dts - located in /opt/dts
>>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>>>>> >> > >>>
>>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>>> >> > >>> - dpdk - located in /root/dpdk
>>>>> >> > >>>
>>>>> >> > >>>
>>>>> >> > >>> Thanks,
>>>>> >> > >>> David
>>>>> >> > >>>
>>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >> > >>>>
>>>>> >> > >>>> Hi David,
>>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>>>>> >> > >>>> see what is happening?
>>>>> >> > >>>> I do have the credentials for two systems in your lab -
>>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>>> >> > >>>> If you can provide me details of the systems you are trying to
>>>>> >> > >>>> configure, I could take a look.
>>>>> >> > >>>>
>>>>> >> > >>>> Thanks
>>>>> >> > >>>> Ajit
>>>>> >> > >>>>
>>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> > >>>> >
>>>>> >> > >>>> > Hi Ajit,
>>>>> >> > >>>> >
>>>>> >> > >>>> > Thank you for helping out.
>>>>> >> > >>>> >
>>>>> >> > >>>> > How are you doing with this issue?
>>>>> >> > >>>> >
>>>>> >> > >>>> > Please let me know if you need anything information from us.
>>>>> >> > >>>> >
>>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>>>>> >> > >>>> >
>>>>> >> > >>>> > Best,
>>>>> >> > >>>> > David
>>>>> >> > >>>> >
>>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> > >>>> >>
>>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>>>>> >> > >>>> >>
>>>>> >> > >>>> >> Thank,
>>>>> >> > >>>> >> David
>>>>> >> > >>>> >>
>>>>> >> > >>>> >>
>>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >> > >>>> >>>
>>>>> >> > >>>> >>> Does the same NIC work on x86?
>>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>>>>> >> > >>>> >>>
>>>>> >> > >>>> >>> Thanks
>>>>> >> > >>>> >>> Ajit
>>>>> >> > >>>> >>>
>>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> > >>>> >>>>
>>>>> >> > >>>> >>>> Hi Ajit,
>>>>> >> > >>>> >>>>
>>>>> >> > >>>> >>>> Thank you for helping out.
>>>>> >> > >>>> >>>>
>>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>>>>> >> > >>>> >>>>
>>>>> >> > >>>> >>>>
>>>>> >> > >>>> >>>> Thanks,
>>>>> >> > >>>> >>>> David
>>>>> >> > >>>> >>>>
>>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>> >> > >>>> >>>>>
>>>>> >> > >>>> >>>>> Hi David,
>>>>> >> > >>>> >>>>> I will take a look.
>>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
>>>>> >> > >>>> >>>>> attempt on ARM.
>>>>> >> > >>>> >>>>>
>>>>> >> > >>>> >>>>> Thanks
>>>>> >> > >>>> >>>>> Ajit
>>>>> >> > >>>> >>>>>
>>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>> Hi Ajit,
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>>>> >> > >>>> >>>>>>>     case_obj()
>>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>> Then start sending traffic with command
>>>>> >> > >>>> >>>>>>>
>>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>>> trex>
>>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>>>>> >> > >>>> >>>>>>> *** traceback follows ***
>>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>>> >> > >>>> >>>>>>>
>>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> ??:0
>>>>> >> > >>>> >>>>>>> '
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>> Best,
>>>>> >> > >>>> >>>>>> David
>>>>> >> > >>>> >>>>>>
>>>>> >> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-09 16:56                                                             ` Ajit Khaparde
@ 2021-03-09 19:00                                                               ` Ajit Khaparde
  2021-03-11 14:01                                                                 ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-03-09 19:00 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 24155 bytes --]

David,
The tool has issues when I load the kernel module.
I am looking for another alternative.
Please go ahead and take back the setup.

Thanks
Ajit

On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
> Hi David,
> I am about to login into the setup and try something.
> I will let you know once I am done and then you can take control.
>
> Thanks
> Ajit
>
> On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > Thank for the update.
> >
> > May I know if I can enable testing or would you like more time to look into this?
> >
> > Best,
> > David
> >
> > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>
> >> David,
> >> I did spend some time on this. For now I can't tell what is wrong.
> >> But I want to update the firmware on the card to take another shot at it.
> >> Since this is an ARM based server, I will have to find a way to update the firmware.
> >> IIRC our tools are mostly x86 based. So I am looking around.
> >> I will update you soon.
> >>
> >> Thanks
> >> Ajit
> >>
> >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>
> >>> Hi David,
> >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
> >>>
> >>> Thanks
> >>> Ajit
> >>>
> >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>
> >>>> Hi Ajit, Juraj,
> >>>>
> >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
> >>>>
> >>>> In this way, you will have no testing interfere with you during troubleshooting.
> >>>>
> >>>> Then we will rerun all the jobs that are skipped.
> >>>>
> >>>> Please let me know if you have any questions.
> >>>>
> >>>> Best,
> >>>> David
> >>>>
> >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >>>>>
> >>>>> Hi David, Ajit,
> >>>>>
> >>>>>
> >>>>>
> >>>>> Friday 10am is 4pm for me a that's fine.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Juraj
> >>>>>
> >>>>>
> >>>>>
> >>>>> From: David Liu <dliu@iol.unh.edu>
> >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hi, Ajit
> >>>>> I changed it to Friday at 10 AM for now.
> >>>>>
> >>>>> We can change again if that doesn't work for Juraj
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> David
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>>
> >>>>> Hi David,
> >>>>> My Thursdays are crazy. Can we do Friday instead?
> >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> >>>>>
> >>>>> Thanks
> >>>>> Ajit
> >>>>>
> >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >
> >>>>> > Hi Ajit and Juraj,
> >>>>> >
> >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
> >>>>> >
> >>>>> > For me, I can do:
> >>>>> > Tuesday 10:00 AM - 1:00 PM.
> >>>>> > Thursday 10:00 AM - 1:00 PM.
> >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >>>>> >
> >>>>> > Best,
> >>>>> > David
> >>>>> >
> >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >>
> >>>>> >> David,
> >>>>> >> I was busy with a customer issue and could not get to my emails.
> >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
> >>>>> >> Let's sync on a suitable time after that.
> >>>>> >>
> >>>>> >> Juraj,
> >>>>> >> Sure. We will add you.
> >>>>> >>
> >>>>> >> Thanks
> >>>>> >> Ajit
> >>>>> >>
> >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >>>>> >> >
> >>>>> >> > Hi Ajit, David,
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Thanks,
> >>>>> >> >
> >>>>> >> > Juraj
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Hi Ajit,
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Sorry for this issue.
> >>>>> >> >
> >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
> >>>>> >> >
> >>>>> >> > Please let me know what would be a good time for you.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Best,
> >>>>> >> >
> >>>>> >> > David
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> >
> >>>>> >> > Hi Ajit,
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > I changed the permission to allow you to run sudo.
> >>>>> >> >
> >>>>> >> > This should work now.
> >>>>> >> >
> >>>>> >> > Let me know if you still have issues.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Best,
> >>>>> >> >
> >>>>> >> > David
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >> >
> >>>>> >> > David,
> >>>>> >> >
> >>>>> >> > I tried the option to use sudo. And that didn't work.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Thanks
> >>>>> >> >
> >>>>> >> > Ajit
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > ----
> >>>>> >> >
> >>>>> >> > sudo su
> >>>>> >> > [sudo] password for ajitkhaparde:
> >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
> >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >>>>> >> > [sudo] password for ajitkhaparde:
> >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> >
> >>>>> >> > Hi Ajit,
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
> >>>>> >> >
> >>>>> >> > David,
> >>>>> >> >
> >>>>> >> > Let me try this method first.
> >>>>> >> >
> >>>>> >> > If it does not work, we could try with the public ssh keys.
> >>>>> >> >
> >>>>> >> > I will let you know.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Thanks
> >>>>> >> >
> >>>>> >> > Ajit
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Let me know if this helps.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Best,
> >>>>> >> >
> >>>>> >> > David
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >> >
> >>>>> >> > Thanks Owen.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > When I run the dts command line suggested, I get the following error.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Thanks
> >>>>> >> >
> >>>>> >> > Ajit
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Traceback (most recent call last):
> >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
> >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
> >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
> >>>>> >> >     raise ExceptionPxssh('password refused')
> >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >>>>> >> >
> >>>>> >> >                                          ^^^^^^^^^^^^^^
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
> >>>>> >> >
> >>>>> >> > Hello Ajit,
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Owen Hilyard
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
> >>>>> >> >
> >>>>> >> > David,
> >>>>> >> >
> >>>>> >> > I can use my credentials to login to rhea.
> >>>>> >> >
> >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Thanks
> >>>>> >> >
> >>>>> >> > Ajit
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> >>>>> >> > Connection closed by 172.18.0.41 port 22
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >>>>> >> > Connection closed by 172.18.0.40 port 22
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> >
> >>>>> >> > Hi Ajit,
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Thanks for the update. Let me know if you need anything from me.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Best,
> >>>>> >> >
> >>>>> >> > David
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >> >
> >>>>> >> > David,
> >>>>> >> >
> >>>>> >> > I had some trouble with my system over the past week.
> >>>>> >> >
> >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
> >>>>> >> >
> >>>>> >> > I will login into the setup after that.
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > Thanks
> >>>>> >> >
> >>>>> >> > Ajit
> >>>>> >> >
> >>>>> >> >
> >>>>> >> >
> >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >> >
> >>>>> >> > Hi David,
> >>>>> >> > I actually did miss it. I will take a look.
> >>>>> >> > Thanks for the refresh mail.
> >>>>> >> >
> >>>>> >> > Thanks
> >>>>> >> > Ajit
> >>>>> >> >
> >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> > >
> >>>>> >> > > Hi Ajit,
> >>>>> >> > >
> >>>>> >> > > Ping again in case if you miss the last email.
> >>>>> >> > >
> >>>>> >> > > Did you have a chance to give it a try on login to the machines?
> >>>>> >> > >
> >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
> >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
> >>>>> >> > >
> >>>>> >> > > Step I run:
> >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
> >>>>> >> > > ./dts -s                                                           #To start the dts
> >>>>> >> > >
> >>>>> >> > > Informatio:
> >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> >>>>> >> > > - dts - located in /opt/dts
> >>>>> >> > > - Trex - located in /opt/trex/v2.87
> >>>>> >> > >
> >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >>>>> >> > > - dpdk - located in /root/dpdk
> >>>>> >> > >
> >>>>> >> > > Best,
> >>>>> >> > > David
> >>>>> >> > >
> >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> > >>
> >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
> >>>>> >> > >> In the dts folder, you can start it with ./dts -s
> >>>>> >> > >>
> >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> > >>>
> >>>>> >> > >>> Hi Ajit,
> >>>>> >> > >>> Sorry for the late reply.
> >>>>> >> > >>>
> >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
> >>>>> >> > >>> Can you try and see if that works?
> >>>>> >> > >>>
> >>>>> >> > >>> Informatio:
> >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >>>>> >> > >>> - dts - located in /opt/dts
> >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> >>>>> >> > >>>
> >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >>>>> >> > >>> - dpdk - located in /root/dpdk
> >>>>> >> > >>>
> >>>>> >> > >>>
> >>>>> >> > >>> Thanks,
> >>>>> >> > >>> David
> >>>>> >> > >>>
> >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >> > >>>>
> >>>>> >> > >>>> Hi David,
> >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
> >>>>> >> > >>>> see what is happening?
> >>>>> >> > >>>> I do have the credentials for two systems in your lab -
> >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> >>>>> >> > >>>> If you can provide me details of the systems you are trying to
> >>>>> >> > >>>> configure, I could take a look.
> >>>>> >> > >>>>
> >>>>> >> > >>>> Thanks
> >>>>> >> > >>>> Ajit
> >>>>> >> > >>>>
> >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> > >>>> >
> >>>>> >> > >>>> > Hi Ajit,
> >>>>> >> > >>>> >
> >>>>> >> > >>>> > Thank you for helping out.
> >>>>> >> > >>>> >
> >>>>> >> > >>>> > How are you doing with this issue?
> >>>>> >> > >>>> >
> >>>>> >> > >>>> > Please let me know if you need anything information from us.
> >>>>> >> > >>>> >
> >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
> >>>>> >> > >>>> >
> >>>>> >> > >>>> > Best,
> >>>>> >> > >>>> > David
> >>>>> >> > >>>> >
> >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> > >>>> >>
> >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
> >>>>> >> > >>>> >>
> >>>>> >> > >>>> >> Thank,
> >>>>> >> > >>>> >> David
> >>>>> >> > >>>> >>
> >>>>> >> > >>>> >>
> >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >> > >>>> >>>
> >>>>> >> > >>>> >>> Does the same NIC work on x86?
> >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
> >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
> >>>>> >> > >>>> >>>
> >>>>> >> > >>>> >>> Thanks
> >>>>> >> > >>>> >>> Ajit
> >>>>> >> > >>>> >>>
> >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> > >>>> >>>>
> >>>>> >> > >>>> >>>> Hi Ajit,
> >>>>> >> > >>>> >>>>
> >>>>> >> > >>>> >>>> Thank you for helping out.
> >>>>> >> > >>>> >>>>
> >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
> >>>>> >> > >>>> >>>>
> >>>>> >> > >>>> >>>>
> >>>>> >> > >>>> >>>> Thanks,
> >>>>> >> > >>>> >>>> David
> >>>>> >> > >>>> >>>>
> >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>> >> > >>>> >>>>>
> >>>>> >> > >>>> >>>>> Hi David,
> >>>>> >> > >>>> >>>>> I will take a look.
> >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
> >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
> >>>>> >> > >>>> >>>>> attempt on ARM.
> >>>>> >> > >>>> >>>>>
> >>>>> >> > >>>> >>>>> Thanks
> >>>>> >> > >>>> >>>>> Ajit
> >>>>> >> > >>>> >>>>>
> >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>> Hi Ajit,
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
> >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
> >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
> >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
> >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
> >>>>> >> > >>>> >>>>>>>     case_obj()
> >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
> >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
> >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
> >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
> >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
> >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
> >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> >>>>> >> > >>>> >>>>>>>
> >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>>> trex>
> >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
> >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
> >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
> >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
> >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
> >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
> >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
> >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >>>>> >> > >>>> >>>>>>>
> >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> ??:0
> >>>>> >> > >>>> >>>>>>> '
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
> >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>> Best,
> >>>>> >> > >>>> >>>>>> David
> >>>>> >> > >>>> >>>>>>
> >>>>> >> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-09 19:00                                                               ` Ajit Khaparde
@ 2021-03-11 14:01                                                                 ` David Liu
  2021-03-11 17:05                                                                   ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-03-11 14:01 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 25910 bytes --]

Hi Ajit,

Thank you.

Please let me know when you need to take over the machine again.

Best,
David

On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> David,
> The tool has issues when I load the kernel module.
> I am looking for another alternative.
> Please go ahead and take back the setup.
>
> Thanks
> Ajit
>
> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
> >
> > Hi David,
> > I am about to login into the setup and try something.
> > I will let you know once I am done and then you can take control.
> >
> > Thanks
> > Ajit
> >
> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
> > >
> > > Hi Ajit,
> > >
> > > Thank for the update.
> > >
> > > May I know if I can enable testing or would you like more time to look
> into this?
> > >
> > > Best,
> > > David
> > >
> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>
> > >> David,
> > >> I did spend some time on this. For now I can't tell what is wrong.
> > >> But I want to update the firmware on the card to take another shot at
> it.
> > >> Since this is an ARM based server, I will have to find a way to
> update the firmware.
> > >> IIRC our tools are mostly x86 based. So I am looking around.
> > >> I will update you soon.
> > >>
> > >> Thanks
> > >> Ajit
> > >>
> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>
> > >>> Hi David,
> > >>> That's a good idea. That way you don't have to watch out for my
> email requesting to stop the tests.
> > >>>
> > >>> Thanks
> > >>> Ajit
> > >>>
> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
> > >>>>
> > >>>> Hi Ajit, Juraj,
> > >>>>
> > >>>> For simplicity, we will keep the functional and performance testing
> disable during the weekend.
> > >>>>
> > >>>> In this way, you will have no testing interfere with you during
> troubleshooting.
> > >>>>
> > >>>> Then we will rerun all the jobs that are skipped.
> > >>>>
> > >>>> Please let me know if you have any questions.
> > >>>>
> > >>>> Best,
> > >>>> David
> > >>>>
> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> > >>>>>
> > >>>>> Hi David, Ajit,
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Friday 10am is 4pm for me a that's fine.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Juraj
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> From: David Liu <dliu@iol.unh.edu>
> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <
> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in
> Arm hardware
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Hi, Ajit
> > >>>>> I changed it to Friday at 10 AM for now.
> > >>>>>
> > >>>>> We can change again if that doesn't work for Juraj
> > >>>>>
> > >>>>> Best,
> > >>>>>
> > >>>>> David
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>>
> > >>>>> Hi David,
> > >>>>> My Thursdays are crazy. Can we do Friday instead?
> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> > >>>>>
> > >>>>> Thanks
> > >>>>> Ajit
> > >>>>>
> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
> > >>>>> >
> > >>>>> > Hi Ajit and Juraj,
> > >>>>> >
> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel
> free to adjust the time or let me know if there is a better time that works
> for you.
> > >>>>> >
> > >>>>> > For me, I can do:
> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
> > >>>>> > Thursday 10:00 AM - 1:00 PM.
> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> > >>>>> >
> > >>>>> > Best,
> > >>>>> > David
> > >>>>> >
> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >>
> > >>>>> >> David,
> > >>>>> >> I was busy with a customer issue and could not get to my emails.
> > >>>>> >> I am seeing this now. Let me close out any issues needing
> urgent attention.
> > >>>>> >> Let's sync on a suitable time after that.
> > >>>>> >>
> > >>>>> >> Juraj,
> > >>>>> >> Sure. We will add you.
> > >>>>> >>
> > >>>>> >> Thanks
> > >>>>> >> Ajit
> > >>>>> >>
> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> > >>>>> >> >
> > >>>>> >> > Hi Ajit, David,
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > If you're setting up a call, could you please send me an
> invite? I'd like to get some insight into the issues on Arm's behalf.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Thanks,
> > >>>>> >> >
> > >>>>> >> > Juraj
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org;
> dpdklab <dpdklab@iol.unh.edu>
> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom
> NIC in Arm hardware
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Hi Ajit,
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Sorry for this issue.
> > >>>>> >> >
> > >>>>> >> > If possible I would like to set up a time where we can do
> this in a call to help get this solved faster.
> > >>>>> >> >
> > >>>>> >> > Please let me know what would be a good time for you.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Best,
> > >>>>> >> >
> > >>>>> >> > David
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu>
> wrote:
> > >>>>> >> >
> > >>>>> >> > Hi Ajit,
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > I changed the permission to allow you to run sudo.
> > >>>>> >> >
> > >>>>> >> > This should work now.
> > >>>>> >> >
> > >>>>> >> > Let me know if you still have issues.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Best,
> > >>>>> >> >
> > >>>>> >> > David
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >> >
> > >>>>> >> > David,
> > >>>>> >> >
> > >>>>> >> > I tried the option to use sudo. And that didn't work.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Thanks
> > >>>>> >> >
> > >>>>> >> > Ajit
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.
> This incident will be reported.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > ----
> > >>>>> >> >
> > >>>>> >> > sudo su
> > >>>>> >> > [sudo] password for ajitkhaparde:
> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This
> incident will be reported.
> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> > >>>>> >> > [sudo] password for ajitkhaparde:
> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This
> incident will be reported.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu>
> wrote:
> > >>>>> >> >
> > >>>>> >> > Hi Ajit,
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > This is likely caused by when dts try to ssh into another
> machine as a normal user which will prompt you for the password.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > One solution would be "sudo su" as root to run the command
> line since you should have root access.
> > >>>>> >> >
> > >>>>> >> > David,
> > >>>>> >> >
> > >>>>> >> > Let me try this method first.
> > >>>>> >> >
> > >>>>> >> > If it does not work, we could try with the public ssh keys.
> > >>>>> >> >
> > >>>>> >> > I will let you know.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Thanks
> > >>>>> >> >
> > >>>>> >> > Ajit
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Another solution would be to let me add your public ssh key
> into both machines.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Let me know if this helps.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Best,
> > >>>>> >> >
> > >>>>> >> > David
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >> >
> > >>>>> >> > Thanks Owen.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > When I run the dts command line suggested, I get the
> following error.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Thanks
> > >>>>> >> >
> > >>>>> >> > Ajit
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Traceback (most recent call last):
> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in
> _connect_host
> > >>>>> >> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> > >>>>> >> >   File
> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
> login
> > >>>>> >> >     raise ExceptionPxssh('password refused')
> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> > >>>>> >> >
> > >>>>> >> >                                          ^^^^^^^^^^^^^^
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
> ohilyard@iol.unh.edu> wrote:
> > >>>>> >> >
> > >>>>> >> > Hello Ajit,
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > There was a misconfiguration in our identity management
> system. The issue should now be resolved.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Owen Hilyard
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <
> dpdklab@iol.unh.edu> wrote:
> > >>>>> >> >
> > >>>>> >> > David,
> > >>>>> >> >
> > >>>>> >> > I can use my credentials to login to rhea.
> > >>>>> >> >
> > >>>>> >> > But the same credentials are not allowing me to login onto
> the servers you provided.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Thanks
> > >>>>> >> >
> > >>>>> >> > Ajit
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> > >>>>> >> > Connection closed by 172.18.0.41 port 22
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> > >>>>> >> > Connection closed by 172.18.0.40 port 22
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu>
> wrote:
> > >>>>> >> >
> > >>>>> >> > Hi Ajit,
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Thanks for the update. Let me know if you need anything from
> me.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Best,
> > >>>>> >> >
> > >>>>> >> > David
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >> >
> > >>>>> >> > David,
> > >>>>> >> >
> > >>>>> >> > I had some trouble with my system over the past week.
> > >>>>> >> >
> > >>>>> >> > So I could not login into the setup. I am hoping that my
> system will be restored early this week
> > >>>>> >> >
> > >>>>> >> > I will login into the setup after that.
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > Thanks
> > >>>>> >> >
> > >>>>> >> > Ajit
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> >
> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >> >
> > >>>>> >> > Hi David,
> > >>>>> >> > I actually did miss it. I will take a look.
> > >>>>> >> > Thanks for the refresh mail.
> > >>>>> >> >
> > >>>>> >> > Thanks
> > >>>>> >> > Ajit
> > >>>>> >> >
> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu>
> wrote:
> > >>>>> >> > >
> > >>>>> >> > > Hi Ajit,
> > >>>>> >> > >
> > >>>>> >> > > Ping again in case if you miss the last email.
> > >>>>> >> > >
> > >>>>> >> > > Did you have a chance to give it a try on login to the
> machines?
> > >>>>> >> > >
> > >>>>> >> > > Just some notes, we are currently running compile and some
> performance/functional testing.
> > >>>>> >> > > Which it may be hard for you to run dts, so please let me
> know if you want me to stop these tests for you to debug.
> > >>>>> >> > >
> > >>>>> >> > > Step I run:
> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch
> the dts configuration to Broadcom
> > >>>>> >> > > ./dts -s
>        #To start the dts
> > >>>>> >> > >
> > >>>>> >> > > Informatio:
> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> > >>>>> >> > > - dts - located in /opt/dts
> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
> > >>>>> >> > >
> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> > >>>>> >> > > - dpdk - located in /root/dpdk
> > >>>>> >> > >
> > >>>>> >> > > Best,
> > >>>>> >> > > David
> > >>>>> >> > >
> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu>
> wrote:
> > >>>>> >> > >>
> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom
> nic.
> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
> > >>>>> >> > >>
> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
> dliu@iol.unh.edu> wrote:
> > >>>>> >> > >>>
> > >>>>> >> > >>> Hi Ajit,
> > >>>>> >> > >>> Sorry for the late reply.
> > >>>>> >> > >>>
> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username
> and Password.
> > >>>>> >> > >>> Can you try and see if that works?
> > >>>>> >> > >>>
> > >>>>> >> > >>> Informatio:
> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> > >>>>> >> > >>> - dts - located in /opt/dts
> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> > >>>>> >> > >>>
> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> > >>>>> >> > >>> - dpdk - located in /root/dpdk
> > >>>>> >> > >>>
> > >>>>> >> > >>>
> > >>>>> >> > >>> Thanks,
> > >>>>> >> > >>> David
> > >>>>> >> > >>>
> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >> > >>>>
> > >>>>> >> > >>>> Hi David,
> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into
> the setup and
> > >>>>> >> > >>>> see what is happening?
> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> > >>>>> >> > >>>> If you can provide me details of the systems you are
> trying to
> > >>>>> >> > >>>> configure, I could take a look.
> > >>>>> >> > >>>>
> > >>>>> >> > >>>> Thanks
> > >>>>> >> > >>>> Ajit
> > >>>>> >> > >>>>
> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
> dliu@iol.unh.edu> wrote:
> > >>>>> >> > >>>> >
> > >>>>> >> > >>>> > Hi Ajit,
> > >>>>> >> > >>>> >
> > >>>>> >> > >>>> > Thank you for helping out.
> > >>>>> >> > >>>> >
> > >>>>> >> > >>>> > How are you doing with this issue?
> > >>>>> >> > >>>> >
> > >>>>> >> > >>>> > Please let me know if you need anything information
> from us.
> > >>>>> >> > >>>> >
> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are
> on the same intel tester machine.
> > >>>>> >> > >>>> >
> > >>>>> >> > >>>> > Best,
> > >>>>> >> > >>>> > David
> > >>>>> >> > >>>> >
> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
> dliu@iol.unh.edu> wrote:
> > >>>>> >> > >>>> >>
> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports
> that are using for Trex. If so they are UP before I start the Trex.
> > >>>>> >> > >>>> >>
> > >>>>> >> > >>>> >> Thank,
> > >>>>> >> > >>>> >> David
> > >>>>> >> > >>>> >>
> > >>>>> >> > >>>> >>
> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >> > >>>> >>>
> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am
> trying to see if the port failed to successfully come up
> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you
> check and tell?
> > >>>>> >> > >>>> >>>
> > >>>>> >> > >>>> >>> Thanks
> > >>>>> >> > >>>> >>> Ajit
> > >>>>> >> > >>>> >>>
> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
> dliu@iol.unh.edu> wrote:
> > >>>>> >> > >>>> >>>>
> > >>>>> >> > >>>> >>>> Hi Ajit,
> > >>>>> >> > >>>> >>>>
> > >>>>> >> > >>>> >>>> Thank you for helping out.
> > >>>>> >> > >>>> >>>>
> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
> > >>>>> >> > >>>> >>>>
> > >>>>> >> > >>>> >>>>
> > >>>>> >> > >>>> >>>> Thanks,
> > >>>>> >> > >>>> >>>> David
> > >>>>> >> > >>>> >>>>
> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> > >>>>> >> > >>>> >>>>>
> > >>>>> >> > >>>> >>>>> Hi David,
> > >>>>> >> > >>>> >>>>> I will take a look.
> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking
> because
> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline
> and then
> > >>>>> >> > >>>> >>>>> attempt on ARM.
> > >>>>> >> > >>>> >>>>>
> > >>>>> >> > >>>> >>>>> Thanks
> > >>>>> >> > >>>> >>>>> Ajit
> > >>>>> >> > >>>> >>>>>
> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
> dliu@iol.unh.edu> wrote:
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>> Hi Ajit,
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on
> an Arm machine.
> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run
> nic_single_core_perf on the NIC.
> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from
> http://git.dpdk.org/tools/dts/
> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the
> test case inside the nic_single_core_perf:
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at
> parameters: framesize: 64, rxd/txd: 512
> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line
> 319, in _execute_test_case
> > >>>>> >> > >>>> >>>>>>>     case_obj()
> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 196, in
> test_perf_nic_single_core
> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line
> 245, in measure_throughput
> > >>>>> >> > >>>> >>>>>>>
>  self._prepare_transmission(stream_ids=stream_ids)
> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line
> 778, in _prepare_transmission
> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
> clear_global, clear_xstats)
> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC]
> - Failed to get server response from tcp://172.18.0.40:4501
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode
> with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> > >>>>> >> > >>>> >>>>>>>
> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>>> trex>
> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  -
> Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not
> responded for more than 1.00135 seconds - timeout is 1 seconds
> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
> [0x561be173cf5a]
> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
> CCoreEthIF::flush_tx_queue() + 31
> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
> CGlobalTRex::run_in_core(unsigned char) + 487
> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
> [0x561be16d11ad]
> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> > >>>>> >> > >>>> >>>>>>>
> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> ??:0
> > >>>>> >> > >>>> >>>>>>> '
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should
> not turn off trex but please correct me if I am wrong.
> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can
> approach this issue?
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>> Best,
> > >>>>> >> > >>>> >>>>>> David
> > >>>>> >> > >>>> >>>>>>
> > >>>>> >> > >>>> >>>>>>
>

[-- Attachment #2: Type: text/html, Size: 50198 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-11 14:01                                                                 ` David Liu
@ 2021-03-11 17:05                                                                   ` Ajit Khaparde
  2021-03-16 15:31                                                                     ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-03-11 17:05 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 26458 bytes --]

David,
While testing with TREX, we have one identified one fix.
We will try to submit it to the mailing list tomorrow.
Once it gets merged, I will get back to you for the setup.

Thanks
Ajit

On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
> Thank you.
>
> Please let me know when you need to take over the machine again.
>
> Best,
> David
>
> On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>
>> David,
>> The tool has issues when I load the kernel module.
>> I am looking for another alternative.
>> Please go ahead and take back the setup.
>>
>> Thanks
>> Ajit
>>
>> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >
>> > Hi David,
>> > I am about to login into the setup and try something.
>> > I will let you know once I am done and then you can take control.
>> >
>> > Thanks
>> > Ajit
>> >
>> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
>> > >
>> > > Hi Ajit,
>> > >
>> > > Thank for the update.
>> > >
>> > > May I know if I can enable testing or would you like more time to look into this?
>> > >
>> > > Best,
>> > > David
>> > >
>> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>
>> > >> David,
>> > >> I did spend some time on this. For now I can't tell what is wrong.
>> > >> But I want to update the firmware on the card to take another shot at it.
>> > >> Since this is an ARM based server, I will have to find a way to update the firmware.
>> > >> IIRC our tools are mostly x86 based. So I am looking around.
>> > >> I will update you soon.
>> > >>
>> > >> Thanks
>> > >> Ajit
>> > >>
>> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>
>> > >>> Hi David,
>> > >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
>> > >>>
>> > >>> Thanks
>> > >>> Ajit
>> > >>>
>> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>
>> > >>>> Hi Ajit, Juraj,
>> > >>>>
>> > >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
>> > >>>>
>> > >>>> In this way, you will have no testing interfere with you during troubleshooting.
>> > >>>>
>> > >>>> Then we will rerun all the jobs that are skipped.
>> > >>>>
>> > >>>> Please let me know if you have any questions.
>> > >>>>
>> > >>>> Best,
>> > >>>> David
>> > >>>>
>> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>> > >>>>>
>> > >>>>> Hi David, Ajit,
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> Friday 10am is 4pm for me a that's fine.
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> Juraj
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> From: David Liu <dliu@iol.unh.edu>
>> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> Hi, Ajit
>> > >>>>> I changed it to Friday at 10 AM for now.
>> > >>>>>
>> > >>>>> We can change again if that doesn't work for Juraj
>> > >>>>>
>> > >>>>> Best,
>> > >>>>>
>> > >>>>> David
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>>
>> > >>>>> Hi David,
>> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>> > >>>>>
>> > >>>>> Thanks
>> > >>>>> Ajit
>> > >>>>>
>> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >
>> > >>>>> > Hi Ajit and Juraj,
>> > >>>>> >
>> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>> > >>>>> >
>> > >>>>> > For me, I can do:
>> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>> > >>>>> >
>> > >>>>> > Best,
>> > >>>>> > David
>> > >>>>> >
>> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >>
>> > >>>>> >> David,
>> > >>>>> >> I was busy with a customer issue and could not get to my emails.
>> > >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
>> > >>>>> >> Let's sync on a suitable time after that.
>> > >>>>> >>
>> > >>>>> >> Juraj,
>> > >>>>> >> Sure. We will add you.
>> > >>>>> >>
>> > >>>>> >> Thanks
>> > >>>>> >> Ajit
>> > >>>>> >>
>> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>> > >>>>> >> >
>> > >>>>> >> > Hi Ajit, David,
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Thanks,
>> > >>>>> >> >
>> > >>>>> >> > Juraj
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Hi Ajit,
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Sorry for this issue.
>> > >>>>> >> >
>> > >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>> > >>>>> >> >
>> > >>>>> >> > Please let me know what would be a good time for you.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Best,
>> > >>>>> >> >
>> > >>>>> >> > David
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> >
>> > >>>>> >> > Hi Ajit,
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > I changed the permission to allow you to run sudo.
>> > >>>>> >> >
>> > >>>>> >> > This should work now.
>> > >>>>> >> >
>> > >>>>> >> > Let me know if you still have issues.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Best,
>> > >>>>> >> >
>> > >>>>> >> > David
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >> >
>> > >>>>> >> > David,
>> > >>>>> >> >
>> > >>>>> >> > I tried the option to use sudo. And that didn't work.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Thanks
>> > >>>>> >> >
>> > >>>>> >> > Ajit
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > ----
>> > >>>>> >> >
>> > >>>>> >> > sudo su
>> > >>>>> >> > [sudo] password for ajitkhaparde:
>> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> > >>>>> >> > [sudo] password for ajitkhaparde:
>> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> >
>> > >>>>> >> > Hi Ajit,
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
>> > >>>>> >> >
>> > >>>>> >> > David,
>> > >>>>> >> >
>> > >>>>> >> > Let me try this method first.
>> > >>>>> >> >
>> > >>>>> >> > If it does not work, we could try with the public ssh keys.
>> > >>>>> >> >
>> > >>>>> >> > I will let you know.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Thanks
>> > >>>>> >> >
>> > >>>>> >> > Ajit
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Let me know if this helps.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Best,
>> > >>>>> >> >
>> > >>>>> >> > David
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >> >
>> > >>>>> >> > Thanks Owen.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > When I run the dts command line suggested, I get the following error.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Thanks
>> > >>>>> >> >
>> > >>>>> >> > Ajit
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Traceback (most recent call last):
>> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>> > >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>> > >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>> > >>>>> >> >     raise ExceptionPxssh('password refused')
>> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>> > >>>>> >> >
>> > >>>>> >> >                                          ^^^^^^^^^^^^^^
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
>> > >>>>> >> >
>> > >>>>> >> > Hello Ajit,
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Owen Hilyard
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
>> > >>>>> >> >
>> > >>>>> >> > David,
>> > >>>>> >> >
>> > >>>>> >> > I can use my credentials to login to rhea.
>> > >>>>> >> >
>> > >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Thanks
>> > >>>>> >> >
>> > >>>>> >> > Ajit
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> >
>> > >>>>> >> > Hi Ajit,
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Thanks for the update. Let me know if you need anything from me.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Best,
>> > >>>>> >> >
>> > >>>>> >> > David
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >> >
>> > >>>>> >> > David,
>> > >>>>> >> >
>> > >>>>> >> > I had some trouble with my system over the past week.
>> > >>>>> >> >
>> > >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
>> > >>>>> >> >
>> > >>>>> >> > I will login into the setup after that.
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > Thanks
>> > >>>>> >> >
>> > >>>>> >> > Ajit
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> >
>> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >> >
>> > >>>>> >> > Hi David,
>> > >>>>> >> > I actually did miss it. I will take a look.
>> > >>>>> >> > Thanks for the refresh mail.
>> > >>>>> >> >
>> > >>>>> >> > Thanks
>> > >>>>> >> > Ajit
>> > >>>>> >> >
>> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> > >
>> > >>>>> >> > > Hi Ajit,
>> > >>>>> >> > >
>> > >>>>> >> > > Ping again in case if you miss the last email.
>> > >>>>> >> > >
>> > >>>>> >> > > Did you have a chance to give it a try on login to the machines?
>> > >>>>> >> > >
>> > >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
>> > >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>> > >>>>> >> > >
>> > >>>>> >> > > Step I run:
>> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>> > >>>>> >> > > ./dts -s                                                           #To start the dts
>> > >>>>> >> > >
>> > >>>>> >> > > Informatio:
>> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>> > >>>>> >> > > - dts - located in /opt/dts
>> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>> > >>>>> >> > >
>> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>> > >>>>> >> > > - dpdk - located in /root/dpdk
>> > >>>>> >> > >
>> > >>>>> >> > > Best,
>> > >>>>> >> > > David
>> > >>>>> >> > >
>> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> > >>
>> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
>> > >>>>> >> > >>
>> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> > >>>
>> > >>>>> >> > >>> Hi Ajit,
>> > >>>>> >> > >>> Sorry for the late reply.
>> > >>>>> >> > >>>
>> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
>> > >>>>> >> > >>> Can you try and see if that works?
>> > >>>>> >> > >>>
>> > >>>>> >> > >>> Informatio:
>> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>> > >>>>> >> > >>> - dts - located in /opt/dts
>> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>> > >>>>> >> > >>>
>> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>> > >>>>> >> > >>>
>> > >>>>> >> > >>>
>> > >>>>> >> > >>> Thanks,
>> > >>>>> >> > >>> David
>> > >>>>> >> > >>>
>> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >> > >>>>
>> > >>>>> >> > >>>> Hi David,
>> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>> > >>>>> >> > >>>> see what is happening?
>> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
>> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>> > >>>>> >> > >>>> If you can provide me details of the systems you are trying to
>> > >>>>> >> > >>>> configure, I could take a look.
>> > >>>>> >> > >>>>
>> > >>>>> >> > >>>> Thanks
>> > >>>>> >> > >>>> Ajit
>> > >>>>> >> > >>>>
>> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> > >>>> >
>> > >>>>> >> > >>>> > Hi Ajit,
>> > >>>>> >> > >>>> >
>> > >>>>> >> > >>>> > Thank you for helping out.
>> > >>>>> >> > >>>> >
>> > >>>>> >> > >>>> > How are you doing with this issue?
>> > >>>>> >> > >>>> >
>> > >>>>> >> > >>>> > Please let me know if you need anything information from us.
>> > >>>>> >> > >>>> >
>> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>> > >>>>> >> > >>>> >
>> > >>>>> >> > >>>> > Best,
>> > >>>>> >> > >>>> > David
>> > >>>>> >> > >>>> >
>> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> > >>>> >>
>> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>> > >>>>> >> > >>>> >>
>> > >>>>> >> > >>>> >> Thank,
>> > >>>>> >> > >>>> >> David
>> > >>>>> >> > >>>> >>
>> > >>>>> >> > >>>> >>
>> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >> > >>>> >>>
>> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>> > >>>>> >> > >>>> >>>
>> > >>>>> >> > >>>> >>> Thanks
>> > >>>>> >> > >>>> >>> Ajit
>> > >>>>> >> > >>>> >>>
>> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> > >>>> >>>>
>> > >>>>> >> > >>>> >>>> Hi Ajit,
>> > >>>>> >> > >>>> >>>>
>> > >>>>> >> > >>>> >>>> Thank you for helping out.
>> > >>>>> >> > >>>> >>>>
>> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>> > >>>>> >> > >>>> >>>>
>> > >>>>> >> > >>>> >>>>
>> > >>>>> >> > >>>> >>>> Thanks,
>> > >>>>> >> > >>>> >>>> David
>> > >>>>> >> > >>>> >>>>
>> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> > >>>>> >> > >>>> >>>>>
>> > >>>>> >> > >>>> >>>>> Hi David,
>> > >>>>> >> > >>>> >>>>> I will take a look.
>> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
>> > >>>>> >> > >>>> >>>>> attempt on ARM.
>> > >>>>> >> > >>>> >>>>>
>> > >>>>> >> > >>>> >>>>> Thanks
>> > >>>>> >> > >>>> >>>>> Ajit
>> > >>>>> >> > >>>> >>>>>
>> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>> > >>>>> >> > >>>> >>>>>>>     case_obj()
>> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>> > >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> > >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
>> > >>>>> >> > >>>> >>>>>>>
>> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>>> trex>
>> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> > >>>>> >> > >>>> >>>>>>>
>> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> ??:0
>> > >>>>> >> > >>>> >>>>>>> '
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>> Best,
>> > >>>>> >> > >>>> >>>>>> David
>> > >>>>> >> > >>>> >>>>>>
>> > >>>>> >> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-11 17:05                                                                   ` Ajit Khaparde
@ 2021-03-16 15:31                                                                     ` David Liu
  2021-03-16 15:43                                                                       ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-03-16 15:31 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 28356 bytes --]

Hi Ajit,

Here to check in with you to see if the patch gets submitted.

And for some confirmation, is this fix possibly be a solution to the
current issue?

Best,
David

On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> David,
> While testing with TREX, we have one identified one fix.
> We will try to submit it to the mailing list tomorrow.
> Once it gets merged, I will get back to you for the setup.
>
> Thanks
> Ajit
>
> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > Thank you.
> >
> > Please let me know when you need to take over the machine again.
> >
> > Best,
> > David
> >
> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
> >>
> >> David,
> >> The tool has issues when I load the kernel module.
> >> I am looking for another alternative.
> >> Please go ahead and take back the setup.
> >>
> >> Thanks
> >> Ajit
> >>
> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >
> >> > Hi David,
> >> > I am about to login into the setup and try something.
> >> > I will let you know once I am done and then you can take control.
> >> >
> >> > Thanks
> >> > Ajit
> >> >
> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
> >> > >
> >> > > Hi Ajit,
> >> > >
> >> > > Thank for the update.
> >> > >
> >> > > May I know if I can enable testing or would you like more time to
> look into this?
> >> > >
> >> > > Best,
> >> > > David
> >> > >
> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>
> >> > >> David,
> >> > >> I did spend some time on this. For now I can't tell what is wrong.
> >> > >> But I want to update the firmware on the card to take another shot
> at it.
> >> > >> Since this is an ARM based server, I will have to find a way to
> update the firmware.
> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
> >> > >> I will update you soon.
> >> > >>
> >> > >> Thanks
> >> > >> Ajit
> >> > >>
> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>
> >> > >>> Hi David,
> >> > >>> That's a good idea. That way you don't have to watch out for my
> email requesting to stop the tests.
> >> > >>>
> >> > >>> Thanks
> >> > >>> Ajit
> >> > >>>
> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>>
> >> > >>>> Hi Ajit, Juraj,
> >> > >>>>
> >> > >>>> For simplicity, we will keep the functional and performance
> testing disable during the weekend.
> >> > >>>>
> >> > >>>> In this way, you will have no testing interfere with you during
> troubleshooting.
> >> > >>>>
> >> > >>>> Then we will rerun all the jobs that are skipped.
> >> > >>>>
> >> > >>>> Please let me know if you have any questions.
> >> > >>>>
> >> > >>>> Best,
> >> > >>>> David
> >> > >>>>
> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> >> > >>>>>
> >> > >>>>> Hi David, Ajit,
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> Friday 10am is 4pm for me a that's fine.
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> Juraj
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <
> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC
> in Arm hardware
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> Hi, Ajit
> >> > >>>>> I changed it to Friday at 10 AM for now.
> >> > >>>>>
> >> > >>>>> We can change again if that doesn't work for Juraj
> >> > >>>>>
> >> > >>>>> Best,
> >> > >>>>>
> >> > >>>>> David
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>>
> >> > >>>>> Hi David,
> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> >> > >>>>>
> >> > >>>>> Thanks
> >> > >>>>> Ajit
> >> > >>>>>
> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>>> >
> >> > >>>>> > Hi Ajit and Juraj,
> >> > >>>>> >
> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But
> feel free to adjust the time or let me know if there is a better time that
> works for you.
> >> > >>>>> >
> >> > >>>>> > For me, I can do:
> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >> > >>>>> >
> >> > >>>>> > Best,
> >> > >>>>> > David
> >> > >>>>> >
> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >>
> >> > >>>>> >> David,
> >> > >>>>> >> I was busy with a customer issue and could not get to my
> emails.
> >> > >>>>> >> I am seeing this now. Let me close out any issues needing
> urgent attention.
> >> > >>>>> >> Let's sync on a suitable time after that.
> >> > >>>>> >>
> >> > >>>>> >> Juraj,
> >> > >>>>> >> Sure. We will add you.
> >> > >>>>> >>
> >> > >>>>> >> Thanks
> >> > >>>>> >> Ajit
> >> > >>>>> >>
> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > Hi Ajit, David,
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > If you're setting up a call, could you please send me an
> invite? I'd like to get some insight into the issues on Arm's behalf.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks,
> >> > >>>>> >> >
> >> > >>>>> >> > Juraj
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org;
> dpdklab <dpdklab@iol.unh.edu>
> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
> Broadcom NIC in Arm hardware
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Hi Ajit,
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Sorry for this issue.
> >> > >>>>> >> >
> >> > >>>>> >> > If possible I would like to set up a time where we can do
> this in a call to help get this solved faster.
> >> > >>>>> >> >
> >> > >>>>> >> > Please let me know what would be a good time for you.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Best,
> >> > >>>>> >> >
> >> > >>>>> >> > David
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > Hi Ajit,
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > I changed the permission to allow you to run sudo.
> >> > >>>>> >> >
> >> > >>>>> >> > This should work now.
> >> > >>>>> >> >
> >> > >>>>> >> > Let me know if you still have issues.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Best,
> >> > >>>>> >> >
> >> > >>>>> >> > David
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > David,
> >> > >>>>> >> >
> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks
> >> > >>>>> >> >
> >> > >>>>> >> > Ajit
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.
> This incident will be reported.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > ----
> >> > >>>>> >> >
> >> > >>>>> >> > sudo su
> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This
> incident will be reported.
> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This
> incident will be reported.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > Hi Ajit,
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > This is likely caused by when dts try to ssh into another
> machine as a normal user which will prompt you for the password.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > One solution would be "sudo su" as root to run the command
> line since you should have root access.
> >> > >>>>> >> >
> >> > >>>>> >> > David,
> >> > >>>>> >> >
> >> > >>>>> >> > Let me try this method first.
> >> > >>>>> >> >
> >> > >>>>> >> > If it does not work, we could try with the public ssh keys.
> >> > >>>>> >> >
> >> > >>>>> >> > I will let you know.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks
> >> > >>>>> >> >
> >> > >>>>> >> > Ajit
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Another solution would be to let me add your public ssh
> key into both machines.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Let me know if this helps.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Best,
> >> > >>>>> >> >
> >> > >>>>> >> > David
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks Owen.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > When I run the dts command line suggested, I get the
> following error.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks
> >> > >>>>> >> >
> >> > >>>>> >> > Ajit
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Traceback (most recent call last):
> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in
> _connect_host
> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> >> > >>>>> >> >   File
> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
> login
> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >> > >>>>> >> >
> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
> ohilyard@iol.unh.edu> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > Hello Ajit,
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > There was a misconfiguration in our identity management
> system. The issue should now be resolved.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Owen Hilyard
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via
> dpdklab <dpdklab@iol.unh.edu> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > David,
> >> > >>>>> >> >
> >> > >>>>> >> > I can use my credentials to login to rhea.
> >> > >>>>> >> >
> >> > >>>>> >> > But the same credentials are not allowing me to login onto
> the servers you provided.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks
> >> > >>>>> >> >
> >> > >>>>> >> > Ajit
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > Hi Ajit,
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks for the update. Let me know if you need anything
> from me.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Best,
> >> > >>>>> >> >
> >> > >>>>> >> > David
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > David,
> >> > >>>>> >> >
> >> > >>>>> >> > I had some trouble with my system over the past week.
> >> > >>>>> >> >
> >> > >>>>> >> > So I could not login into the setup. I am hoping that my
> system will be restored early this week
> >> > >>>>> >> >
> >> > >>>>> >> > I will login into the setup after that.
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks
> >> > >>>>> >> >
> >> > >>>>> >> > Ajit
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> >
> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >> >
> >> > >>>>> >> > Hi David,
> >> > >>>>> >> > I actually did miss it. I will take a look.
> >> > >>>>> >> > Thanks for the refresh mail.
> >> > >>>>> >> >
> >> > >>>>> >> > Thanks
> >> > >>>>> >> > Ajit
> >> > >>>>> >> >
> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >> > >>>>> >> > >
> >> > >>>>> >> > > Hi Ajit,
> >> > >>>>> >> > >
> >> > >>>>> >> > > Ping again in case if you miss the last email.
> >> > >>>>> >> > >
> >> > >>>>> >> > > Did you have a chance to give it a try on login to the
> machines?
> >> > >>>>> >> > >
> >> > >>>>> >> > > Just some notes, we are currently running compile and
> some performance/functional testing.
> >> > >>>>> >> > > Which it may be hard for you to run dts, so please let
> me know if you want me to stop these tests for you to debug.
> >> > >>>>> >> > >
> >> > >>>>> >> > > Step I run:
> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to
> switch the dts configuration to Broadcom
> >> > >>>>> >> > > ./dts -s
>            #To start the dts
> >> > >>>>> >> > >
> >> > >>>>> >> > > Informatio:
> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> >> > >>>>> >> > > - dts - located in /opt/dts
> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
> >> > >>>>> >> > >
> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >> > >>>>> >> > > - dpdk - located in /root/dpdk
> >> > >>>>> >> > >
> >> > >>>>> >> > > Best,
> >> > >>>>> >> > > David
> >> > >>>>> >> > >
> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>>> >> > >>
> >> > >>>>> >> > >> Also, I had changed the configuration when I run
> Broadcom nic.
> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
> >> > >>>>> >> > >>
> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>>> >> > >>>
> >> > >>>>> >> > >>> Hi Ajit,
> >> > >>>>> >> > >>> Sorry for the late reply.
> >> > >>>>> >> > >>>
> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN
> Username and Password.
> >> > >>>>> >> > >>> Can you try and see if that works?
> >> > >>>>> >> > >>>
> >> > >>>>> >> > >>> Informatio:
> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >> > >>>>> >> > >>> - dts - located in /opt/dts
> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> >> > >>>>> >> > >>>
> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
> >> > >>>>> >> > >>>
> >> > >>>>> >> > >>>
> >> > >>>>> >> > >>> Thanks,
> >> > >>>>> >> > >>> David
> >> > >>>>> >> > >>>
> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >> > >>>>
> >> > >>>>> >> > >>>> Hi David,
> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login
> into the setup and
> >> > >>>>> >> > >>>> see what is happening?
> >> > >>>>> >> > >>>> I do have the credentials for two systems in your lab
> -
> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> >> > >>>>> >> > >>>> If you can provide me details of the systems you are
> trying to
> >> > >>>>> >> > >>>> configure, I could take a look.
> >> > >>>>> >> > >>>>
> >> > >>>>> >> > >>>> Thanks
> >> > >>>>> >> > >>>> Ajit
> >> > >>>>> >> > >>>>
> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>>> >> > >>>> >
> >> > >>>>> >> > >>>> > Hi Ajit,
> >> > >>>>> >> > >>>> >
> >> > >>>>> >> > >>>> > Thank you for helping out.
> >> > >>>>> >> > >>>> >
> >> > >>>>> >> > >>>> > How are you doing with this issue?
> >> > >>>>> >> > >>>> >
> >> > >>>>> >> > >>>> > Please let me know if you need anything information
> from us.
> >> > >>>>> >> > >>>> >
> >> > >>>>> >> > >>>> > Just some extra information that both Trex and dts
> are on the same intel tester machine.
> >> > >>>>> >> > >>>> >
> >> > >>>>> >> > >>>> > Best,
> >> > >>>>> >> > >>>> > David
> >> > >>>>> >> > >>>> >
> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>>> >> > >>>> >>
> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the
> ports that are using for Trex. If so they are UP before I start the Trex.
> >> > >>>>> >> > >>>> >>
> >> > >>>>> >> > >>>> >> Thank,
> >> > >>>>> >> > >>>> >> David
> >> > >>>>> >> > >>>> >>
> >> > >>>>> >> > >>>> >>
> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >> > >>>> >>>
> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
> >> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I
> am trying to see if the port failed to successfully come up
> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you
> check and tell?
> >> > >>>>> >> > >>>> >>>
> >> > >>>>> >> > >>>> >>> Thanks
> >> > >>>>> >> > >>>> >>> Ajit
> >> > >>>>> >> > >>>> >>>
> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>>> >> > >>>> >>>>
> >> > >>>>> >> > >>>> >>>> Hi Ajit,
> >> > >>>>> >> > >>>> >>>>
> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
> >> > >>>>> >> > >>>> >>>>
> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
> >> > >>>>> >> > >>>> >>>>
> >> > >>>>> >> > >>>> >>>>
> >> > >>>>> >> > >>>> >>>> Thanks,
> >> > >>>>> >> > >>>> >>>> David
> >> > >>>>> >> > >>>> >>>>
> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> > >>>>> >> > >>>> >>>>>
> >> > >>>>> >> > >>>> >>>>> Hi David,
> >> > >>>>> >> > >>>> >>>>> I will take a look.
> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking
> because
> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a
> baseline and then
> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
> >> > >>>>> >> > >>>> >>>>>
> >> > >>>>> >> > >>>> >>>>> Thanks
> >> > >>>>> >> > >>>> >>>>> Ajit
> >> > >>>>> >> > >>>> >>>>>
> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed
> on an Arm machine.
> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to
> run nic_single_core_perf on the NIC.
> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts
> from http://git.dpdk.org/tools/dts/
> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the
> test case inside the nic_single_core_perf:
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at
> parameters: framesize: 64, rxd/txd: 512
> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py",
> line 319, in _execute_test_case
> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 196, in
> test_perf_nic_single_core
> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py",
> line 245, in measure_throughput
> >> > >>>>> >> > >>>> >>>>>>>
>  self._prepare_transmission(stream_ids=stream_ids)
> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py",
> line 778, in _prepare_transmission
> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
> clear_global, clear_xstats)
> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: ***
> [RPC] - Failed to get server response from tcp://172.18.0.40:4501
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode
> with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> >> > >>>>> >> > >>>> >>>>>>>
> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>>> trex>
> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  -
> Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not
> responded for more than 1.00135 seconds - timeout is 1 seconds
> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a)
> [0x561be173cf5a]
> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block +
> 106
> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
> CCoreEthIF::flush_tx_queue() + 31
> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() +
> 89
> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
> CGlobalTRex::run_in_core(unsigned char) + 487
> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad)
> [0x561be16d11ad]
> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >> > >>>>> >> > >>>> >>>>>>>
> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> > >>>>> >> > >>>> >>>>>>> '
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this
> should not turn off trex but please correct me if I am wrong.
> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I
> can approach this issue?
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>> Best,
> >> > >>>>> >> > >>>> >>>>>> David
> >> > >>>>> >> > >>>> >>>>>>
> >> > >>>>> >> > >>>> >>>>>>
>

[-- Attachment #2: Type: text/html, Size: 56429 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-16 15:31                                                                     ` David Liu
@ 2021-03-16 15:43                                                                       ` Ajit Khaparde
  2021-04-01 17:09                                                                         ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-03-16 15:43 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 28808 bytes --]

Hi David,
The patch was in dpdk-next-net.
Let me check if it was applied in the main tree.

Thanks
Ajit

On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
> Here to check in with you to see if the patch gets submitted.
>
> And for some confirmation, is this fix possibly be a solution to the current issue?
>
> Best,
> David
>
> On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>
>> David,
>> While testing with TREX, we have one identified one fix.
>> We will try to submit it to the mailing list tomorrow.
>> Once it gets merged, I will get back to you for the setup.
>>
>> Thanks
>> Ajit
>>
>> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> > Thank you.
>> >
>> > Please let me know when you need to take over the machine again.
>> >
>> > Best,
>> > David
>> >
>> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>
>> >> David,
>> >> The tool has issues when I load the kernel module.
>> >> I am looking for another alternative.
>> >> Please go ahead and take back the setup.
>> >>
>> >> Thanks
>> >> Ajit
>> >>
>> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> >
>> >> > Hi David,
>> >> > I am about to login into the setup and try something.
>> >> > I will let you know once I am done and then you can take control.
>> >> >
>> >> > Thanks
>> >> > Ajit
>> >> >
>> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >
>> >> > > Hi Ajit,
>> >> > >
>> >> > > Thank for the update.
>> >> > >
>> >> > > May I know if I can enable testing or would you like more time to look into this?
>> >> > >
>> >> > > Best,
>> >> > > David
>> >> > >
>> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>
>> >> > >> David,
>> >> > >> I did spend some time on this. For now I can't tell what is wrong.
>> >> > >> But I want to update the firmware on the card to take another shot at it.
>> >> > >> Since this is an ARM based server, I will have to find a way to update the firmware.
>> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
>> >> > >> I will update you soon.
>> >> > >>
>> >> > >> Thanks
>> >> > >> Ajit
>> >> > >>
>> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>
>> >> > >>> Hi David,
>> >> > >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
>> >> > >>>
>> >> > >>> Thanks
>> >> > >>> Ajit
>> >> > >>>
>> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>
>> >> > >>>> Hi Ajit, Juraj,
>> >> > >>>>
>> >> > >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
>> >> > >>>>
>> >> > >>>> In this way, you will have no testing interfere with you during troubleshooting.
>> >> > >>>>
>> >> > >>>> Then we will rerun all the jobs that are skipped.
>> >> > >>>>
>> >> > >>>> Please let me know if you have any questions.
>> >> > >>>>
>> >> > >>>> Best,
>> >> > >>>> David
>> >> > >>>>
>> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>> >> > >>>>>
>> >> > >>>>> Hi David, Ajit,
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>> Juraj
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
>> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>> Hi, Ajit
>> >> > >>>>> I changed it to Friday at 10 AM for now.
>> >> > >>>>>
>> >> > >>>>> We can change again if that doesn't work for Juraj
>> >> > >>>>>
>> >> > >>>>> Best,
>> >> > >>>>>
>> >> > >>>>> David
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>>
>> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>>
>> >> > >>>>> Hi David,
>> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>> >> > >>>>>
>> >> > >>>>> Thanks
>> >> > >>>>> Ajit
>> >> > >>>>>
>> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >
>> >> > >>>>> > Hi Ajit and Juraj,
>> >> > >>>>> >
>> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>> >> > >>>>> >
>> >> > >>>>> > For me, I can do:
>> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>> >> > >>>>> >
>> >> > >>>>> > Best,
>> >> > >>>>> > David
>> >> > >>>>> >
>> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >>
>> >> > >>>>> >> David,
>> >> > >>>>> >> I was busy with a customer issue and could not get to my emails.
>> >> > >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
>> >> > >>>>> >> Let's sync on a suitable time after that.
>> >> > >>>>> >>
>> >> > >>>>> >> Juraj,
>> >> > >>>>> >> Sure. We will add you.
>> >> > >>>>> >>
>> >> > >>>>> >> Thanks
>> >> > >>>>> >> Ajit
>> >> > >>>>> >>
>> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > Hi Ajit, David,
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks,
>> >> > >>>>> >> >
>> >> > >>>>> >> > Juraj
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Hi Ajit,
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Sorry for this issue.
>> >> > >>>>> >> >
>> >> > >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>> >> > >>>>> >> >
>> >> > >>>>> >> > Please let me know what would be a good time for you.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Best,
>> >> > >>>>> >> >
>> >> > >>>>> >> > David
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > Hi Ajit,
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > I changed the permission to allow you to run sudo.
>> >> > >>>>> >> >
>> >> > >>>>> >> > This should work now.
>> >> > >>>>> >> >
>> >> > >>>>> >> > Let me know if you still have issues.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Best,
>> >> > >>>>> >> >
>> >> > >>>>> >> > David
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > David,
>> >> > >>>>> >> >
>> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks
>> >> > >>>>> >> >
>> >> > >>>>> >> > Ajit
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > ----
>> >> > >>>>> >> >
>> >> > >>>>> >> > sudo su
>> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > Hi Ajit,
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
>> >> > >>>>> >> >
>> >> > >>>>> >> > David,
>> >> > >>>>> >> >
>> >> > >>>>> >> > Let me try this method first.
>> >> > >>>>> >> >
>> >> > >>>>> >> > If it does not work, we could try with the public ssh keys.
>> >> > >>>>> >> >
>> >> > >>>>> >> > I will let you know.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks
>> >> > >>>>> >> >
>> >> > >>>>> >> > Ajit
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Let me know if this helps.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Best,
>> >> > >>>>> >> >
>> >> > >>>>> >> > David
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks Owen.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > When I run the dts command line suggested, I get the following error.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks
>> >> > >>>>> >> >
>> >> > >>>>> >> > Ajit
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Traceback (most recent call last):
>> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>> >> > >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>> >> > >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>> >> > >>>>> >> >
>> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > Hello Ajit,
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Owen Hilyard
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > David,
>> >> > >>>>> >> >
>> >> > >>>>> >> > I can use my credentials to login to rhea.
>> >> > >>>>> >> >
>> >> > >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks
>> >> > >>>>> >> >
>> >> > >>>>> >> > Ajit
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > Hi Ajit,
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks for the update. Let me know if you need anything from me.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Best,
>> >> > >>>>> >> >
>> >> > >>>>> >> > David
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > David,
>> >> > >>>>> >> >
>> >> > >>>>> >> > I had some trouble with my system over the past week.
>> >> > >>>>> >> >
>> >> > >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
>> >> > >>>>> >> >
>> >> > >>>>> >> > I will login into the setup after that.
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks
>> >> > >>>>> >> >
>> >> > >>>>> >> > Ajit
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >> >
>> >> > >>>>> >> > Hi David,
>> >> > >>>>> >> > I actually did miss it. I will take a look.
>> >> > >>>>> >> > Thanks for the refresh mail.
>> >> > >>>>> >> >
>> >> > >>>>> >> > Thanks
>> >> > >>>>> >> > Ajit
>> >> > >>>>> >> >
>> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > Hi Ajit,
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > Ping again in case if you miss the last email.
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > Did you have a chance to give it a try on login to the machines?
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
>> >> > >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > Step I run:
>> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>> >> > >>>>> >> > > ./dts -s                                                           #To start the dts
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > Informatio:
>> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>> >> > >>>>> >> > > - dts - located in /opt/dts
>> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > Best,
>> >> > >>>>> >> > > David
>> >> > >>>>> >> > >
>> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> > >>
>> >> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
>> >> > >>>>> >> > >>
>> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> > >>>
>> >> > >>>>> >> > >>> Hi Ajit,
>> >> > >>>>> >> > >>> Sorry for the late reply.
>> >> > >>>>> >> > >>>
>> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
>> >> > >>>>> >> > >>> Can you try and see if that works?
>> >> > >>>>> >> > >>>
>> >> > >>>>> >> > >>> Informatio:
>> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>> >> > >>>>> >> > >>> - dts - located in /opt/dts
>> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>> >> > >>>>> >> > >>>
>> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>> >> > >>>>> >> > >>>
>> >> > >>>>> >> > >>>
>> >> > >>>>> >> > >>> Thanks,
>> >> > >>>>> >> > >>> David
>> >> > >>>>> >> > >>>
>> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >> > >>>>
>> >> > >>>>> >> > >>>> Hi David,
>> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>> >> > >>>>> >> > >>>> see what is happening?
>> >> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
>> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>> >> > >>>>> >> > >>>> If you can provide me details of the systems you are trying to
>> >> > >>>>> >> > >>>> configure, I could take a look.
>> >> > >>>>> >> > >>>>
>> >> > >>>>> >> > >>>> Thanks
>> >> > >>>>> >> > >>>> Ajit
>> >> > >>>>> >> > >>>>
>> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> > >>>> >
>> >> > >>>>> >> > >>>> > Hi Ajit,
>> >> > >>>>> >> > >>>> >
>> >> > >>>>> >> > >>>> > Thank you for helping out.
>> >> > >>>>> >> > >>>> >
>> >> > >>>>> >> > >>>> > How are you doing with this issue?
>> >> > >>>>> >> > >>>> >
>> >> > >>>>> >> > >>>> > Please let me know if you need anything information from us.
>> >> > >>>>> >> > >>>> >
>> >> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>> >> > >>>>> >> > >>>> >
>> >> > >>>>> >> > >>>> > Best,
>> >> > >>>>> >> > >>>> > David
>> >> > >>>>> >> > >>>> >
>> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> > >>>> >>
>> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>> >> > >>>>> >> > >>>> >>
>> >> > >>>>> >> > >>>> >> Thank,
>> >> > >>>>> >> > >>>> >> David
>> >> > >>>>> >> > >>>> >>
>> >> > >>>>> >> > >>>> >>
>> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >> > >>>> >>>
>> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>> >> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>> >> > >>>>> >> > >>>> >>>
>> >> > >>>>> >> > >>>> >>> Thanks
>> >> > >>>>> >> > >>>> >>> Ajit
>> >> > >>>>> >> > >>>> >>>
>> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> > >>>> >>>>
>> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>> >> > >>>>> >> > >>>> >>>>
>> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>> >> > >>>>> >> > >>>> >>>>
>> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>> >> > >>>>> >> > >>>> >>>>
>> >> > >>>>> >> > >>>> >>>>
>> >> > >>>>> >> > >>>> >>>> Thanks,
>> >> > >>>>> >> > >>>> >>>> David
>> >> > >>>>> >> > >>>> >>>>
>> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> > >>>>> >> > >>>> >>>>>
>> >> > >>>>> >> > >>>> >>>>> Hi David,
>> >> > >>>>> >> > >>>> >>>>> I will take a look.
>> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
>> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>> >> > >>>>> >> > >>>> >>>>>
>> >> > >>>>> >> > >>>> >>>>> Thanks
>> >> > >>>>> >> > >>>> >>>>> Ajit
>> >> > >>>>> >> > >>>> >>>>>
>> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> >> > >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
>> >> > >>>>> >> > >>>> >>>>>>>
>> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>>> trex>
>> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> >> > >>>>> >> > >>>> >>>>>>>
>> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> > >>>>> >> > >>>> >>>>>>> '
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>> Best,
>> >> > >>>>> >> > >>>> >>>>>> David
>> >> > >>>>> >> > >>>> >>>>>>
>> >> > >>>>> >> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-03-16 15:43                                                                       ` Ajit Khaparde
@ 2021-04-01 17:09                                                                         ` David Liu
  2021-04-05 14:25                                                                           ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-04-01 17:09 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 30780 bytes --]

Hi Ajit,

I think the patch should be in the main tree by now.

Can you check if that is true?

Just checking, is this patch related to the Trex mapping issue?

Best,
David

On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi David,
> The patch was in dpdk-next-net.
> Let me check if it was applied in the main tree.
>
> Thanks
> Ajit
>
> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > Here to check in with you to see if the patch gets submitted.
> >
> > And for some confirmation, is this fix possibly be a solution to the
> current issue?
> >
> > Best,
> > David
> >
> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>
> >> David,
> >> While testing with TREX, we have one identified one fix.
> >> We will try to submit it to the mailing list tomorrow.
> >> Once it gets merged, I will get back to you for the setup.
> >>
> >> Thanks
> >> Ajit
> >>
> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >
> >> > Hi Ajit,
> >> >
> >> > Thank you.
> >> >
> >> > Please let me know when you need to take over the machine again.
> >> >
> >> > Best,
> >> > David
> >> >
> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >>
> >> >> David,
> >> >> The tool has issues when I load the kernel module.
> >> >> I am looking for another alternative.
> >> >> Please go ahead and take back the setup.
> >> >>
> >> >> Thanks
> >> >> Ajit
> >> >>
> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> >
> >> >> > Hi David,
> >> >> > I am about to login into the setup and try something.
> >> >> > I will let you know once I am done and then you can take control.
> >> >> >
> >> >> > Thanks
> >> >> > Ajit
> >> >> >
> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >> > >
> >> >> > > Hi Ajit,
> >> >> > >
> >> >> > > Thank for the update.
> >> >> > >
> >> >> > > May I know if I can enable testing or would you like more time
> to look into this?
> >> >> > >
> >> >> > > Best,
> >> >> > > David
> >> >> > >
> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>
> >> >> > >> David,
> >> >> > >> I did spend some time on this. For now I can't tell what is
> wrong.
> >> >> > >> But I want to update the firmware on the card to take another
> shot at it.
> >> >> > >> Since this is an ARM based server, I will have to find a way to
> update the firmware.
> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
> >> >> > >> I will update you soon.
> >> >> > >>
> >> >> > >> Thanks
> >> >> > >> Ajit
> >> >> > >>
> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>
> >> >> > >>> Hi David,
> >> >> > >>> That's a good idea. That way you don't have to watch out for
> my email requesting to stop the tests.
> >> >> > >>>
> >> >> > >>> Thanks
> >> >> > >>> Ajit
> >> >> > >>>
> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu>
> wrote:
> >> >> > >>>>
> >> >> > >>>> Hi Ajit, Juraj,
> >> >> > >>>>
> >> >> > >>>> For simplicity, we will keep the functional and performance
> testing disable during the weekend.
> >> >> > >>>>
> >> >> > >>>> In this way, you will have no testing interfere with you
> during troubleshooting.
> >> >> > >>>>
> >> >> > >>>> Then we will rerun all the jobs that are skipped.
> >> >> > >>>>
> >> >> > >>>> Please let me know if you have any questions.
> >> >> > >>>>
> >> >> > >>>> Best,
> >> >> > >>>> David
> >> >> > >>>>
> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> >> >> > >>>>>
> >> >> > >>>>> Hi David, Ajit,
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>> Juraj
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard
> <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom
> NIC in Arm hardware
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>> Hi, Ajit
> >> >> > >>>>> I changed it to Friday at 10 AM for now.
> >> >> > >>>>>
> >> >> > >>>>> We can change again if that doesn't work for Juraj
> >> >> > >>>>>
> >> >> > >>>>> Best,
> >> >> > >>>>>
> >> >> > >>>>> David
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>>
> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>>
> >> >> > >>>>> Hi David,
> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> >> >> > >>>>>
> >> >> > >>>>> Thanks
> >> >> > >>>>> Ajit
> >> >> > >>>>>
> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >> >> > >>>>> >
> >> >> > >>>>> > Hi Ajit and Juraj,
> >> >> > >>>>> >
> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But
> feel free to adjust the time or let me know if there is a better time that
> works for you.
> >> >> > >>>>> >
> >> >> > >>>>> > For me, I can do:
> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >> >> > >>>>> >
> >> >> > >>>>> > Best,
> >> >> > >>>>> > David
> >> >> > >>>>> >
> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >>
> >> >> > >>>>> >> David,
> >> >> > >>>>> >> I was busy with a customer issue and could not get to my
> emails.
> >> >> > >>>>> >> I am seeing this now. Let me close out any issues needing
> urgent attention.
> >> >> > >>>>> >> Let's sync on a suitable time after that.
> >> >> > >>>>> >>
> >> >> > >>>>> >> Juraj,
> >> >> > >>>>> >> Sure. We will add you.
> >> >> > >>>>> >>
> >> >> > >>>>> >> Thanks
> >> >> > >>>>> >> Ajit
> >> >> > >>>>> >>
> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Hi Ajit, David,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > If you're setting up a call, could you please send me
> an invite? I'd like to get some insight into the issues on Arm's behalf.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Juraj
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org;
> dpdklab <dpdklab@iol.unh.edu>
> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
> Broadcom NIC in Arm hardware
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Hi Ajit,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Sorry for this issue.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > If possible I would like to set up a time where we can
> do this in a call to help get this solved faster.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Please let me know what would be a good time for you.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Best,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > David
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Hi Ajit,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > This should work now.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Let me know if you still have issues.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Best,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > David
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > David,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Ajit
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-gigabyte.  This incident will be reported.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > ----
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > sudo su
> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.
> This incident will be reported.
> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.
> This incident will be reported.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Hi Ajit,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into
> another machine as a normal user which will prompt you for the password.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the
> command line since you should have root access.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > David,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Let me try this method first.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > If it does not work, we could try with the public ssh
> keys.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > I will let you know.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Ajit
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Another solution would be to let me add your public ssh
> key into both machines.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Let me know if this helps.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Best,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > David
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks Owen.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > When I run the dts command line suggested, I get the
> following error.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Ajit
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Traceback (most recent call last):
> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in
> _connect_host
> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> >> >> > >>>>> >> >   File
> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
> login
> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
> ohilyard@iol.unh.edu> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Hello Ajit,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > There was a misconfiguration in our identity management
> system. The issue should now be resolved.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Owen Hilyard
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via
> dpdklab <dpdklab@iol.unh.edu> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > David,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > I can use my credentials to login to rhea.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > But the same credentials are not allowing me to login
> onto the servers you provided.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Ajit
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Hi Ajit,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks for the update. Let me know if you need anything
> from me.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Best,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > David
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > David,
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > I had some trouble with my system over the past week.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that
> my system will be restored early this week
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > I will login into the setup after that.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Ajit
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Hi David,
> >> >> > >>>>> >> > I actually did miss it. I will take a look.
> >> >> > >>>>> >> > Thanks for the refresh mail.
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > Thanks
> >> >> > >>>>> >> > Ajit
> >> >> > >>>>> >> >
> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > Hi Ajit,
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to
> the machines?
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > Just some notes, we are currently running compile and
> some performance/functional testing.
> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please
> let me know if you want me to stop these tests for you to debug.
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > Step I run:
> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to
> switch the dts configuration to Broadcom
> >> >> > >>>>> >> > > ./dts -s
>              #To start the dts
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > Informatio:
> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> >> >> > >>>>> >> > > - dts - located in /opt/dts
> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > Best,
> >> >> > >>>>> >> > > David
> >> >> > >>>>> >> > >
> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> > >>
> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run
> Broadcom nic.
> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
> >> >> > >>>>> >> > >>
> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> > >>>
> >> >> > >>>>> >> > >>> Hi Ajit,
> >> >> > >>>>> >> > >>> Sorry for the late reply.
> >> >> > >>>>> >> > >>>
> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN
> Username and Password.
> >> >> > >>>>> >> > >>> Can you try and see if that works?
> >> >> > >>>>> >> > >>>
> >> >> > >>>>> >> > >>> Informatio:
> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> >> >> > >>>>> >> > >>>
> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
> >> >> > >>>>> >> > >>>
> >> >> > >>>>> >> > >>>
> >> >> > >>>>> >> > >>> Thanks,
> >> >> > >>>>> >> > >>> David
> >> >> > >>>>> >> > >>>
> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >> > >>>>
> >> >> > >>>>> >> > >>>> Hi David,
> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login
> into the setup and
> >> >> > >>>>> >> > >>>> see what is happening?
> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in your
> lab -
> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and
> io.dpdklab.iol.unh.edu
> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you
> are trying to
> >> >> > >>>>> >> > >>>> configure, I could take a look.
> >> >> > >>>>> >> > >>>>
> >> >> > >>>>> >> > >>>> Thanks
> >> >> > >>>>> >> > >>>> Ajit
> >> >> > >>>>> >> > >>>>
> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> > >>>> >
> >> >> > >>>>> >> > >>>> > Hi Ajit,
> >> >> > >>>>> >> > >>>> >
> >> >> > >>>>> >> > >>>> > Thank you for helping out.
> >> >> > >>>>> >> > >>>> >
> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
> >> >> > >>>>> >> > >>>> >
> >> >> > >>>>> >> > >>>> > Please let me know if you need anything
> information from us.
> >> >> > >>>>> >> > >>>> >
> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and
> dts are on the same intel tester machine.
> >> >> > >>>>> >> > >>>> >
> >> >> > >>>>> >> > >>>> > Best,
> >> >> > >>>>> >> > >>>> > David
> >> >> > >>>>> >> > >>>> >
> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> > >>>> >>
> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the
> ports that are using for Trex. If so they are UP before I start the Trex.
> >> >> > >>>>> >> > >>>> >>
> >> >> > >>>>> >> > >>>> >> Thank,
> >> >> > >>>>> >> > >>>> >> David
> >> >> > >>>>> >> > >>>> >>
> >> >> > >>>>> >> > >>>> >>
> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >> > >>>> >>>
> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the
> trex? I am trying to see if the port failed to successfully come up
> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can
> you check and tell?
> >> >> > >>>>> >> > >>>> >>>
> >> >> > >>>>> >> > >>>> >>> Thanks
> >> >> > >>>>> >> > >>>> >>> Ajit
> >> >> > >>>>> >> > >>>> >>>
> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> > >>>> >>>>
> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
> >> >> > >>>>> >> > >>>> >>>>
> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
> >> >> > >>>>> >> > >>>> >>>>
> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the
> Arm.
> >> >> > >>>>> >> > >>>> >>>>
> >> >> > >>>>> >> > >>>> >>>>
> >> >> > >>>>> >> > >>>> >>>> Thanks,
> >> >> > >>>>> >> > >>>> >>>> David
> >> >> > >>>>> >> > >>>> >>>>
> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde
> <ajit.khaparde@broadcom.com> wrote:
> >> >> > >>>>> >> > >>>> >>>>>
> >> >> > >>>>> >> > >>>> >>>>> Hi David,
> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am
> asking because
> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a
> baseline and then
> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
> >> >> > >>>>> >> > >>>> >>>>>
> >> >> > >>>>> >> > >>>> >>>>> Thanks
> >> >> > >>>>> >> > >>>> >>>>> Ajit
> >> >> > >>>>> >> > >>>> >>>>>
> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is
> installed on an Arm machine.
> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try
> to run nic_single_core_perf on the NIC.
> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts
> from http://git.dpdk.org/tools/dts/
> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when
> testing.
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run
> the test case inside the nic_single_core_perf:
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at
> parameters: framesize: 64, rxd/txd: 512
> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py",
> line 319, in _execute_test_case
> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 196, in
> test_perf_nic_single_core
> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >> >> > >>>>> >> > >>>> >>>>>>>
>  self._prepare_transmission(stream_ids=stream_ids)
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
> clear_global, clear_xstats)
> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: ***
> [RPC] - Failed to get server response from tcp://172.18.0.40:4501
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode
> with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> >> >> > >>>>> >> > >>>> >>>>>>>
> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off
> Trex.
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>>> trex>
> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  -
> Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not
> responded for more than 1.00135 seconds - timeout is 1 seconds
> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a
> ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block
> + 106
> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
> CCoreEthIF::flush_tx_queue() + 31
> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start()
> + 89
> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
> CGlobalTRex::run_in_core(unsigned char) + 487
> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad
> ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop +
> 426
> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >> >> > >>>>> >> > >>>> >>>>>>>
> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >> > >>>>> >> > >>>> >>>>>>> '
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this
> should not turn off trex but please correct me if I am wrong.
> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how
> I can approach this issue?
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>> Best,
> >> >> > >>>>> >> > >>>> >>>>>> David
> >> >> > >>>>> >> > >>>> >>>>>>
> >> >> > >>>>> >> > >>>> >>>>>>
>

[-- Attachment #2: Type: text/html, Size: 62839 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-01 17:09                                                                         ` David Liu
@ 2021-04-05 14:25                                                                           ` Ajit Khaparde
  2021-04-06 14:29                                                                             ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-04-05 14:25 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 31795 bytes --]

Hi David,
Yes. The changes are in the tree.
Let's try to rerun the test and see what happens.

Thanks
Ajit


On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit,
>
> I think the patch should be in the main tree by now.
>
> Can you check if that is true?
>
> Just checking, is this patch related to the Trex mapping issue?
>
> Best,
> David
>
> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Hi David,
>> The patch was in dpdk-next-net.
>> Let me check if it was applied in the main tree.
>>
>> Thanks
>> Ajit
>>
>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> > Here to check in with you to see if the patch gets submitted.
>> >
>> > And for some confirmation, is this fix possibly be a solution to the
>> current issue?
>> >
>> > Best,
>> > David
>> >
>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>
>> >> David,
>> >> While testing with TREX, we have one identified one fix.
>> >> We will try to submit it to the mailing list tomorrow.
>> >> Once it gets merged, I will get back to you for the setup.
>> >>
>> >> Thanks
>> >> Ajit
>> >>
>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
>> >> >
>> >> > Hi Ajit,
>> >> >
>> >> > Thank you.
>> >> >
>> >> > Please let me know when you need to take over the machine again.
>> >> >
>> >> > Best,
>> >> > David
>> >> >
>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >>
>> >> >> David,
>> >> >> The tool has issues when I load the kernel module.
>> >> >> I am looking for another alternative.
>> >> >> Please go ahead and take back the setup.
>> >> >>
>> >> >> Thanks
>> >> >> Ajit
>> >> >>
>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> >
>> >> >> > Hi David,
>> >> >> > I am about to login into the setup and try something.
>> >> >> > I will let you know once I am done and then you can take control.
>> >> >> >
>> >> >> > Thanks
>> >> >> > Ajit
>> >> >> >
>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >> >> > >
>> >> >> > > Hi Ajit,
>> >> >> > >
>> >> >> > > Thank for the update.
>> >> >> > >
>> >> >> > > May I know if I can enable testing or would you like more time
>> to look into this?
>> >> >> > >
>> >> >> > > Best,
>> >> >> > > David
>> >> >> > >
>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>
>> >> >> > >> David,
>> >> >> > >> I did spend some time on this. For now I can't tell what is
>> wrong.
>> >> >> > >> But I want to update the firmware on the card to take another
>> shot at it.
>> >> >> > >> Since this is an ARM based server, I will have to find a way
>> to update the firmware.
>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
>> >> >> > >> I will update you soon.
>> >> >> > >>
>> >> >> > >> Thanks
>> >> >> > >> Ajit
>> >> >> > >>
>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>
>> >> >> > >>> Hi David,
>> >> >> > >>> That's a good idea. That way you don't have to watch out for
>> my email requesting to stop the tests.
>> >> >> > >>>
>> >> >> > >>> Thanks
>> >> >> > >>> Ajit
>> >> >> > >>>
>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >> >> > >>>>
>> >> >> > >>>> Hi Ajit, Juraj,
>> >> >> > >>>>
>> >> >> > >>>> For simplicity, we will keep the functional and performance
>> testing disable during the weekend.
>> >> >> > >>>>
>> >> >> > >>>> In this way, you will have no testing interfere with you
>> during troubleshooting.
>> >> >> > >>>>
>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
>> >> >> > >>>>
>> >> >> > >>>> Please let me know if you have any questions.
>> >> >> > >>>>
>> >> >> > >>>> Best,
>> >> >> > >>>> David
>> >> >> > >>>>
>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš
>> <juraj.linkes@pantheon.tech> wrote:
>> >> >> > >>>>>
>> >> >> > >>>>> Hi David, Ajit,
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>> Juraj
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen
>> Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu
>> >
>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom
>> NIC in Arm hardware
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>> Hi, Ajit
>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
>> >> >> > >>>>>
>> >> >> > >>>>> We can change again if that doesn't work for Juraj
>> >> >> > >>>>>
>> >> >> > >>>>> Best,
>> >> >> > >>>>>
>> >> >> > >>>>> David
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>>
>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>>
>> >> >> > >>>>> Hi David,
>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>> >> >> > >>>>>
>> >> >> > >>>>> Thanks
>> >> >> > >>>>> Ajit
>> >> >> > >>>>>
>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >> >> > >>>>> >
>> >> >> > >>>>> > Hi Ajit and Juraj,
>> >> >> > >>>>> >
>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But
>> feel free to adjust the time or let me know if there is a better time that
>> works for you.
>> >> >> > >>>>> >
>> >> >> > >>>>> > For me, I can do:
>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>> >> >> > >>>>> >
>> >> >> > >>>>> > Best,
>> >> >> > >>>>> > David
>> >> >> > >>>>> >
>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >>
>> >> >> > >>>>> >> David,
>> >> >> > >>>>> >> I was busy with a customer issue and could not get to my
>> emails.
>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues
>> needing urgent attention.
>> >> >> > >>>>> >> Let's sync on a suitable time after that.
>> >> >> > >>>>> >>
>> >> >> > >>>>> >> Juraj,
>> >> >> > >>>>> >> Sure. We will add you.
>> >> >> > >>>>> >>
>> >> >> > >>>>> >> Thanks
>> >> >> > >>>>> >> Ajit
>> >> >> > >>>>> >>
>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš
>> <juraj.linkes@pantheon.tech> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Hi Ajit, David,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > If you're setting up a call, could you please send me
>> an invite? I'd like to get some insight into the issues on Arm's behalf.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Juraj
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org;
>> dpdklab <dpdklab@iol.unh.edu>
>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
>> Broadcom NIC in Arm hardware
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Hi Ajit,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Sorry for this issue.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > If possible I would like to set up a time where we can
>> do this in a call to help get this solved faster.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Please let me know what would be a good time for you.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Best,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > David
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Hi Ajit,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > This should work now.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Let me know if you still have issues.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Best,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > David
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > David,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Ajit
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
>> arm-gigabyte.  This incident will be reported.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > ----
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > sudo su
>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.
>> This incident will be reported.
>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.
>> This incident will be reported.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Hi Ajit,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into
>> another machine as a normal user which will prompt you for the password.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the
>> command line since you should have root access.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > David,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Let me try this method first.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > If it does not work, we could try with the public ssh
>> keys.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > I will let you know.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Ajit
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Another solution would be to let me add your public
>> ssh key into both machines.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Let me know if this helps.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Best,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > David
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks Owen.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > When I run the dts command line suggested, I get the
>> following error.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Ajit
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Traceback (most recent call last):
>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60,
>> in _connect_host
>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>> .+:)')
>> >> >> > >>>>> >> >   File
>> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
>> login
>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
>> ohilyard@iol.unh.edu> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Hello Ajit,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > There was a misconfiguration in our identity
>> management system. The issue should now be resolved.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Owen Hilyard
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via
>> dpdklab <dpdklab@iol.unh.edu> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > David,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > But the same credentials are not allowing me to login
>> onto the servers you provided.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Ajit
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Hi Ajit,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need
>> anything from me.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Best,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > David
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > David,
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > I had some trouble with my system over the past week.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that
>> my system will be restored early this week
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > I will login into the setup after that.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Ajit
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Hi David,
>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
>> >> >> > >>>>> >> > Thanks for the refresh mail.
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > Thanks
>> >> >> > >>>>> >> > Ajit
>> >> >> > >>>>> >> >
>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > Hi Ajit,
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to
>> the machines?
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > Just some notes, we are currently running compile
>> and some performance/functional testing.
>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please
>> let me know if you want me to stop these tests for you to debug.
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > Step I run:
>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to
>> switch the dts configuration to Broadcom
>> >> >> > >>>>> >> > > ./dts -s
>>                #To start the dts
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > Informatio:
>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>> >> >> > >>>>> >> > > - dts - located in /opt/dts
>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > Best,
>> >> >> > >>>>> >> > > David
>> >> >> > >>>>> >> > >
>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> > >>
>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run
>> Broadcom nic.
>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
>> >> >> > >>>>> >> > >>
>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> > >>>
>> >> >> > >>>>> >> > >>> Hi Ajit,
>> >> >> > >>>>> >> > >>> Sorry for the late reply.
>> >> >> > >>>>> >> > >>>
>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN
>> Username and Password.
>> >> >> > >>>>> >> > >>> Can you try and see if that works?
>> >> >> > >>>>> >> > >>>
>> >> >> > >>>>> >> > >>> Informatio:
>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>> >> >> > >>>>> >> > >>>
>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>> >> >> > >>>>> >> > >>>
>> >> >> > >>>>> >> > >>>
>> >> >> > >>>>> >> > >>> Thanks,
>> >> >> > >>>>> >> > >>> David
>> >> >> > >>>>> >> > >>>
>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >> > >>>>
>> >> >> > >>>>> >> > >>>> Hi David,
>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login
>> into the setup and
>> >> >> > >>>>> >> > >>>> see what is happening?
>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in your
>> lab -
>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and
>> io.dpdklab.iol.unh.edu
>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you
>> are trying to
>> >> >> > >>>>> >> > >>>> configure, I could take a look.
>> >> >> > >>>>> >> > >>>>
>> >> >> > >>>>> >> > >>>> Thanks
>> >> >> > >>>>> >> > >>>> Ajit
>> >> >> > >>>>> >> > >>>>
>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> > >>>> >
>> >> >> > >>>>> >> > >>>> > Hi Ajit,
>> >> >> > >>>>> >> > >>>> >
>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
>> >> >> > >>>>> >> > >>>> >
>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
>> >> >> > >>>>> >> > >>>> >
>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything
>> information from us.
>> >> >> > >>>>> >> > >>>> >
>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and
>> dts are on the same intel tester machine.
>> >> >> > >>>>> >> > >>>> >
>> >> >> > >>>>> >> > >>>> > Best,
>> >> >> > >>>>> >> > >>>> > David
>> >> >> > >>>>> >> > >>>> >
>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> > >>>> >>
>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the
>> ports that are using for Trex. If so they are UP before I start the Trex.
>> >> >> > >>>>> >> > >>>> >>
>> >> >> > >>>>> >> > >>>> >> Thank,
>> >> >> > >>>>> >> > >>>> >> David
>> >> >> > >>>>> >> > >>>> >>
>> >> >> > >>>>> >> > >>>> >>
>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >> > >>>> >>>
>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the
>> trex? I am trying to see if the port failed to successfully come up
>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can
>> you check and tell?
>> >> >> > >>>>> >> > >>>> >>>
>> >> >> > >>>>> >> > >>>> >>> Thanks
>> >> >> > >>>>> >> > >>>> >>> Ajit
>> >> >> > >>>>> >> > >>>> >>>
>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> > >>>> >>>>
>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>> >> >> > >>>>> >> > >>>> >>>>
>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>> >> >> > >>>>> >> > >>>> >>>>
>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the
>> Arm.
>> >> >> > >>>>> >> > >>>> >>>>
>> >> >> > >>>>> >> > >>>> >>>>
>> >> >> > >>>>> >> > >>>> >>>> Thanks,
>> >> >> > >>>>> >> > >>>> >>>> David
>> >> >> > >>>>> >> > >>>> >>>>
>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit
>> Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >> >> > >>>>> >> > >>>> >>>>>
>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am
>> asking because
>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a
>> baseline and then
>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>> >> >> > >>>>> >> > >>>> >>>>>
>> >> >> > >>>>> >> > >>>> >>>>> Thanks
>> >> >> > >>>>> >> > >>>> >>>>> Ajit
>> >> >> > >>>>> >> > >>>> >>>>>
>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is
>> installed on an Arm machine.
>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try
>> to run nic_single_core_perf on the NIC.
>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and
>> dts from http://git.dpdk.org/tools/dts/
>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when
>> testing.
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run
>> the test case inside the nic_single_core_perf:
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at
>> parameters: framesize: 64, rxd/txd: 512
>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case
>> test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py",
>> line 319, in _execute_test_case
>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "tests/TestSuite_nic_single_core_perf.py", line 196, in
>> test_perf_nic_single_core
>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>> options=traffic_opt)
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> >> >> > >>>>> >> > >>>> >>>>>>>
>>  self._prepare_transmission(stream_ids=stream_ids)
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 339, in reset
>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 1492, in clear_stats
>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
>> clear_global, clear_xstats)
>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>> line 2876, in _clear_stats_common
>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError:
>> *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode
>> with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
>> >> >> > >>>>> >> > >>>> >>>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off
>> Trex.
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]
>> - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has
>> not responded for more than 1.00135 seconds - timeout is 1 seconds
>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a
>> ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block
>> + 106
>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>> CVirtualIFPerSideStats*) + 3220
>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
>> CCoreEthIF::flush_tx_queue() + 31
>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>> + 92
>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>> CFlowGenListPerThread*, bool) + 184
>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>> CFlowGenListPerThread*, double&) + 881
>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
>> TrexStatelessDpCore::start_scheduler() + 226
>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9
>> TrexDpCore::start() + 89
>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
>> CGlobalTRex::run_in_core(unsigned char) + 487
>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad
>> ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop +
>> 426
>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> >> >> > >>>>> >> > >>>> >>>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >> >> > >>>>> >> > >>>> >>>>>>> '
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this
>> should not turn off trex but please correct me if I am wrong.
>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how
>> I can approach this issue?
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>> Best,
>> >> >> > >>>>> >> > >>>> >>>>>> David
>> >> >> > >>>>> >> > >>>> >>>>>>
>> >> >> > >>>>> >> > >>>> >>>>>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 63814 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-05 14:25                                                                           ` Ajit Khaparde
@ 2021-04-06 14:29                                                                             ` David Liu
  2021-04-07 20:16                                                                               ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-04-06 14:29 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 34070 bytes --]

Hi Ajit,

I had rerun the test with the recent DTS and Trex (v2.88)

With this, I am getting the mapping error we see during the debugging
session.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                        pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg
/opt/trex/v2.88/trex_cfg_bnxt.yaml
Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg
/opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy
server......... Scapy server is started
Trying to bind to vfio-pci ...
/usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0 0000:5f:00.1
The ports are bound/configured.
Starting  TRex v2.88 please wait  ...
ERROR in DPDK map
Could not find requested interface 5f:00.0
Killing Scapy server... Scapy server is killed]
                           dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64 -i
--cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
                        tester: pkill -f _t-rex-64
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Best,
David



On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Hi David,
> Yes. The changes are in the tree.
> Let's try to rerun the test and see what happens.
>
> Thanks
> Ajit
>
>
> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> I think the patch should be in the main tree by now.
>>
>> Can you check if that is true?
>>
>> Just checking, is this patch related to the Trex mapping issue?
>>
>> Best,
>> David
>>
>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>>
>>> Hi David,
>>> The patch was in dpdk-next-net.
>>> Let me check if it was applied in the main tree.
>>>
>>> Thanks
>>> Ajit
>>>
>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >
>>> > Hi Ajit,
>>> >
>>> > Here to check in with you to see if the patch gets submitted.
>>> >
>>> > And for some confirmation, is this fix possibly be a solution to the
>>> current issue?
>>> >
>>> > Best,
>>> > David
>>> >
>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>
>>> >> David,
>>> >> While testing with TREX, we have one identified one fix.
>>> >> We will try to submit it to the mailing list tomorrow.
>>> >> Once it gets merged, I will get back to you for the setup.
>>> >>
>>> >> Thanks
>>> >> Ajit
>>> >>
>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >> >
>>> >> > Hi Ajit,
>>> >> >
>>> >> > Thank you.
>>> >> >
>>> >> > Please let me know when you need to take over the machine again.
>>> >> >
>>> >> > Best,
>>> >> > David
>>> >> >
>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >>
>>> >> >> David,
>>> >> >> The tool has issues when I load the kernel module.
>>> >> >> I am looking for another alternative.
>>> >> >> Please go ahead and take back the setup.
>>> >> >>
>>> >> >> Thanks
>>> >> >> Ajit
>>> >> >>
>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> >
>>> >> >> > Hi David,
>>> >> >> > I am about to login into the setup and try something.
>>> >> >> > I will let you know once I am done and then you can take control.
>>> >> >> >
>>> >> >> > Thanks
>>> >> >> > Ajit
>>> >> >> >
>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> >> > >
>>> >> >> > > Hi Ajit,
>>> >> >> > >
>>> >> >> > > Thank for the update.
>>> >> >> > >
>>> >> >> > > May I know if I can enable testing or would you like more time
>>> to look into this?
>>> >> >> > >
>>> >> >> > > Best,
>>> >> >> > > David
>>> >> >> > >
>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>
>>> >> >> > >> David,
>>> >> >> > >> I did spend some time on this. For now I can't tell what is
>>> wrong.
>>> >> >> > >> But I want to update the firmware on the card to take another
>>> shot at it.
>>> >> >> > >> Since this is an ARM based server, I will have to find a way
>>> to update the firmware.
>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
>>> >> >> > >> I will update you soon.
>>> >> >> > >>
>>> >> >> > >> Thanks
>>> >> >> > >> Ajit
>>> >> >> > >>
>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>
>>> >> >> > >>> Hi David,
>>> >> >> > >>> That's a good idea. That way you don't have to watch out for
>>> my email requesting to stop the tests.
>>> >> >> > >>>
>>> >> >> > >>> Thanks
>>> >> >> > >>> Ajit
>>> >> >> > >>>
>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> >> > >>>>
>>> >> >> > >>>> Hi Ajit, Juraj,
>>> >> >> > >>>>
>>> >> >> > >>>> For simplicity, we will keep the functional and performance
>>> testing disable during the weekend.
>>> >> >> > >>>>
>>> >> >> > >>>> In this way, you will have no testing interfere with you
>>> during troubleshooting.
>>> >> >> > >>>>
>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
>>> >> >> > >>>>
>>> >> >> > >>>> Please let me know if you have any questions.
>>> >> >> > >>>>
>>> >> >> > >>>> Best,
>>> >> >> > >>>> David
>>> >> >> > >>>>
>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš
>>> <juraj.linkes@pantheon.tech> wrote:
>>> >> >> > >>>>>
>>> >> >> > >>>>> Hi David, Ajit,
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>> Juraj
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen
>>> Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <
>>> dpdklab@iol.unh.edu>
>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
>>> Broadcom NIC in Arm hardware
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>> Hi, Ajit
>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
>>> >> >> > >>>>>
>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
>>> >> >> > >>>>>
>>> >> >> > >>>>> Best,
>>> >> >> > >>>>>
>>> >> >> > >>>>> David
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>>
>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>>
>>> >> >> > >>>>> Hi David,
>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>>> >> >> > >>>>>
>>> >> >> > >>>>> Thanks
>>> >> >> > >>>>> Ajit
>>> >> >> > >>>>>
>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >> >> > >>>>> >
>>> >> >> > >>>>> > Hi Ajit and Juraj,
>>> >> >> > >>>>> >
>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM.
>>> But feel free to adjust the time or let me know if there is a better time
>>> that works for you.
>>> >> >> > >>>>> >
>>> >> >> > >>>>> > For me, I can do:
>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>>> >> >> > >>>>> >
>>> >> >> > >>>>> > Best,
>>> >> >> > >>>>> > David
>>> >> >> > >>>>> >
>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >>
>>> >> >> > >>>>> >> David,
>>> >> >> > >>>>> >> I was busy with a customer issue and could not get to
>>> my emails.
>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues
>>> needing urgent attention.
>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
>>> >> >> > >>>>> >>
>>> >> >> > >>>>> >> Juraj,
>>> >> >> > >>>>> >> Sure. We will add you.
>>> >> >> > >>>>> >>
>>> >> >> > >>>>> >> Thanks
>>> >> >> > >>>>> >> Ajit
>>> >> >> > >>>>> >>
>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš
>>> <juraj.linkes@pantheon.tech> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Hi Ajit, David,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > If you're setting up a call, could you please send me
>>> an invite? I'd like to get some insight into the issues on Arm's behalf.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Juraj
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org;
>>> dpdklab <dpdklab@iol.unh.edu>
>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
>>> Broadcom NIC in Arm hardware
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Hi Ajit,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Sorry for this issue.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > If possible I would like to set up a time where we
>>> can do this in a call to help get this solved faster.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Please let me know what would be a good time for you.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Best,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > David
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Hi Ajit,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > This should work now.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Let me know if you still have issues.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Best,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > David
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > David,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Ajit
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
>>> arm-gigabyte.  This incident will be reported.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > ----
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > sudo su
>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.
>>> This incident will be reported.
>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.
>>> This incident will be reported.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Hi Ajit,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into
>>> another machine as a normal user which will prompt you for the password.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the
>>> command line since you should have root access.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > David,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Let me try this method first.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > If it does not work, we could try with the public ssh
>>> keys.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > I will let you know.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Ajit
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Another solution would be to let me add your public
>>> ssh key into both machines.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Let me know if this helps.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Best,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > David
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks Owen.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get the
>>> following error.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Ajit
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Traceback (most recent call last):
>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60,
>>> in _connect_host
>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
>>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>>> .+:)')
>>> >> >> > >>>>> >> >   File
>>> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
>>> login
>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> ^^^^^^^^^^^^^^
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
>>> ohilyard@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Hello Ajit,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > There was a misconfiguration in our identity
>>> management system. The issue should now be resolved.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Owen Hilyard
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via
>>> dpdklab <dpdklab@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > David,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > But the same credentials are not allowing me to login
>>> onto the servers you provided.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Ajit
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Hi Ajit,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need
>>> anything from me.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Best,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > David
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > David,
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > I had some trouble with my system over the past week.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that
>>> my system will be restored early this week
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > I will login into the setup after that.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Ajit
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Hi David,
>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
>>> >> >> > >>>>> >> > Thanks for the refresh mail.
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > Thanks
>>> >> >> > >>>>> >> > Ajit
>>> >> >> > >>>>> >> >
>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > Hi Ajit,
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to
>>> the machines?
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > Just some notes, we are currently running compile
>>> and some performance/functional testing.
>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please
>>> let me know if you want me to stop these tests for you to debug.
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > Step I run:
>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to
>>> switch the dts configuration to Broadcom
>>> >> >> > >>>>> >> > > ./dts -s
>>>                #To start the dts
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > Informatio:
>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > Best,
>>> >> >> > >>>>> >> > > David
>>> >> >> > >>>>> >> > >
>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> > >>
>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run
>>> Broadcom nic.
>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
>>> >> >> > >>>>> >> > >>
>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> > >>>
>>> >> >> > >>>>> >> > >>> Hi Ajit,
>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
>>> >> >> > >>>>> >> > >>>
>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN
>>> Username and Password.
>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
>>> >> >> > >>>>> >> > >>>
>>> >> >> > >>>>> >> > >>> Informatio:
>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>>> >> >> > >>>>> >> > >>>
>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>>> >> >> > >>>>> >> > >>>
>>> >> >> > >>>>> >> > >>>
>>> >> >> > >>>>> >> > >>> Thanks,
>>> >> >> > >>>>> >> > >>> David
>>> >> >> > >>>>> >> > >>>
>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >> > >>>>
>>> >> >> > >>>>> >> > >>>> Hi David,
>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could
>>> login into the setup and
>>> >> >> > >>>>> >> > >>>> see what is happening?
>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in
>>> your lab -
>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and
>>> io.dpdklab.iol.unh.edu
>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you
>>> are trying to
>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
>>> >> >> > >>>>> >> > >>>>
>>> >> >> > >>>>> >> > >>>> Thanks
>>> >> >> > >>>>> >> > >>>> Ajit
>>> >> >> > >>>>> >> > >>>>
>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> > >>>> >
>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
>>> >> >> > >>>>> >> > >>>> >
>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
>>> >> >> > >>>>> >> > >>>> >
>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
>>> >> >> > >>>>> >> > >>>> >
>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything
>>> information from us.
>>> >> >> > >>>>> >> > >>>> >
>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and
>>> dts are on the same intel tester machine.
>>> >> >> > >>>>> >> > >>>> >
>>> >> >> > >>>>> >> > >>>> > Best,
>>> >> >> > >>>>> >> > >>>> > David
>>> >> >> > >>>>> >> > >>>> >
>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> > >>>> >>
>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean
>>> the ports that are using for Trex. If so they are UP before I start the
>>> Trex.
>>> >> >> > >>>>> >> > >>>> >>
>>> >> >> > >>>>> >> > >>>> >> Thank,
>>> >> >> > >>>>> >> > >>>> >> David
>>> >> >> > >>>>> >> > >>>> >>
>>> >> >> > >>>>> >> > >>>> >>
>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde
>>> <ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >> > >>>> >>>
>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the
>>> trex? I am trying to see if the port failed to successfully come up
>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can
>>> you check and tell?
>>> >> >> > >>>>> >> > >>>> >>>
>>> >> >> > >>>>> >> > >>>> >>> Thanks
>>> >> >> > >>>>> >> > >>>> >>> Ajit
>>> >> >> > >>>>> >> > >>>> >>>
>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> > >>>> >>>>
>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>>> >> >> > >>>>> >> > >>>> >>>>
>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>>> >> >> > >>>>> >> > >>>> >>>>
>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the
>>> Arm.
>>> >> >> > >>>>> >> > >>>> >>>>
>>> >> >> > >>>>> >> > >>>> >>>>
>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
>>> >> >> > >>>>> >> > >>>> >>>> David
>>> >> >> > >>>>> >> > >>>> >>>>
>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit
>>> Khaparde <ajit.khaparde@broadcom.com> wrote:
>>> >> >> > >>>>> >> > >>>> >>>>>
>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am
>>> asking because
>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a
>>> baseline and then
>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>>> >> >> > >>>>> >> > >>>> >>>>>
>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
>>> >> >> > >>>>> >> > >>>> >>>>>
>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu
>>> <dliu@iol.unh.edu> wrote:
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is
>>> installed on an Arm machine.
>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I
>>> try to run nic_single_core_perf on the NIC.
>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and
>>> dts from http://git.dpdk.org/tools/dts/
>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when
>>> testing.
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run
>>> the test case inside the nic_single_core_perf:
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at
>>> parameters: framesize: 64, rxd/txd: 512
>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test
>>> Case test_perf_nic_single_core Result ERROR: Traceback (most recent call
>>> last):
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "tests/TestSuite_nic_single_core_perf.py", line 196, in
>>> test_perf_nic_single_core
>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>> options=traffic_opt)
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>  self._prepare_transmission(stream_ids=stream_ids)
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 339, in reset
>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 1492, in clear_stats
>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
>>> clear_global, clear_xstats)
>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>> line 2876, in _clear_stats_common
>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>>> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError:
>>> *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode
>>> with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off
>>> Trex.
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]
>>> - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has
>>> not responded for more than 1.00135 seconds - timeout is 1 seconds
>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a
>>> ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a
>>> rte_delay_us_block + 106
>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>> CVirtualIFPerSideStats*) + 3220
>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
>>> CCoreEthIF::flush_tx_queue() + 31
>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>> + 92
>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>> CFlowGenListPerThread*, bool) + 184
>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>> CFlowGenListPerThread*, double&) + 881
>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
>>> TrexStatelessDpCore::start_scheduler() + 226
>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9
>>> TrexDpCore::start() + 89
>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
>>> CGlobalTRex::run_in_core(unsigned char) + 487
>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad
>>> ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop +
>>> 426
>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >> >> > >>>>> >> > >>>> >>>>>>> '
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this
>>> should not turn off trex but please correct me if I am wrong.
>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of
>>> how I can approach this issue?
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
>>> >> >> > >>>>> >> > >>>> >>>>>> David
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>
>>

[-- Attachment #2: Type: text/html, Size: 65846 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-06 14:29                                                                             ` David Liu
@ 2021-04-07 20:16                                                                               ` Ajit Khaparde
  2021-04-09 22:09                                                                                 ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-04-07 20:16 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 34367 bytes --]

Thanks David.
While I don't have the ARM server, I am trying to use the new version
of TREX and see if I run into any issues.
I will get back to you once I complete the test.

Thanks
Ajit


On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
> I had rerun the test with the recent DTS and Trex (v2.88)
>
> With this, I am getting the mapping error we see during the debugging session.
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
> Trying to bind to vfio-pci ...
> /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0 0000:5f:00.1
> The ports are bound/configured.
> Starting  TRex v2.88 please wait  ...
> ERROR in DPDK map
> Could not find requested interface 5f:00.0
> Killing Scapy server... Scapy server is killed]
>                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>                         tester: pkill -f _t-rex-64
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Best,
> David
>
>
>
> On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>
>> Hi David,
>> Yes. The changes are in the tree.
>> Let's try to rerun the test and see what happens.
>>
>> Thanks
>> Ajit
>>
>>
>> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
>>>
>>> Hi Ajit,
>>>
>>> I think the patch should be in the main tree by now.
>>>
>>> Can you check if that is true?
>>>
>>> Just checking, is this patch related to the Trex mapping issue?
>>>
>>> Best,
>>> David
>>>
>>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>>
>>>> Hi David,
>>>> The patch was in dpdk-next-net.
>>>> Let me check if it was applied in the main tree.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >
>>>> > Hi Ajit,
>>>> >
>>>> > Here to check in with you to see if the patch gets submitted.
>>>> >
>>>> > And for some confirmation, is this fix possibly be a solution to the current issue?
>>>> >
>>>> > Best,
>>>> > David
>>>> >
>>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >>
>>>> >> David,
>>>> >> While testing with TREX, we have one identified one fix.
>>>> >> We will try to submit it to the mailing list tomorrow.
>>>> >> Once it gets merged, I will get back to you for the setup.
>>>> >>
>>>> >> Thanks
>>>> >> Ajit
>>>> >>
>>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >
>>>> >> > Hi Ajit,
>>>> >> >
>>>> >> > Thank you.
>>>> >> >
>>>> >> > Please let me know when you need to take over the machine again.
>>>> >> >
>>>> >> > Best,
>>>> >> > David
>>>> >> >
>>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >>
>>>> >> >> David,
>>>> >> >> The tool has issues when I load the kernel module.
>>>> >> >> I am looking for another alternative.
>>>> >> >> Please go ahead and take back the setup.
>>>> >> >>
>>>> >> >> Thanks
>>>> >> >> Ajit
>>>> >> >>
>>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> >
>>>> >> >> > Hi David,
>>>> >> >> > I am about to login into the setup and try something.
>>>> >> >> > I will let you know once I am done and then you can take control.
>>>> >> >> >
>>>> >> >> > Thanks
>>>> >> >> > Ajit
>>>> >> >> >
>>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >
>>>> >> >> > > Hi Ajit,
>>>> >> >> > >
>>>> >> >> > > Thank for the update.
>>>> >> >> > >
>>>> >> >> > > May I know if I can enable testing or would you like more time to look into this?
>>>> >> >> > >
>>>> >> >> > > Best,
>>>> >> >> > > David
>>>> >> >> > >
>>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>
>>>> >> >> > >> David,
>>>> >> >> > >> I did spend some time on this. For now I can't tell what is wrong.
>>>> >> >> > >> But I want to update the firmware on the card to take another shot at it.
>>>> >> >> > >> Since this is an ARM based server, I will have to find a way to update the firmware.
>>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
>>>> >> >> > >> I will update you soon.
>>>> >> >> > >>
>>>> >> >> > >> Thanks
>>>> >> >> > >> Ajit
>>>> >> >> > >>
>>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>
>>>> >> >> > >>> Hi David,
>>>> >> >> > >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
>>>> >> >> > >>>
>>>> >> >> > >>> Thanks
>>>> >> >> > >>> Ajit
>>>> >> >> > >>>
>>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>
>>>> >> >> > >>>> Hi Ajit, Juraj,
>>>> >> >> > >>>>
>>>> >> >> > >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
>>>> >> >> > >>>>
>>>> >> >> > >>>> In this way, you will have no testing interfere with you during troubleshooting.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Please let me know if you have any questions.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Best,
>>>> >> >> > >>>> David
>>>> >> >> > >>>>
>>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi David, Ajit,
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Juraj
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
>>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi, Ajit
>>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Best,
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> David
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi David,
>>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Thanks
>>>> >> >> > >>>>> Ajit
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > Hi Ajit and Juraj,
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > For me, I can do:
>>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > Best,
>>>> >> >> > >>>>> > David
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> David,
>>>> >> >> > >>>>> >> I was busy with a customer issue and could not get to my emails.
>>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
>>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> Juraj,
>>>> >> >> > >>>>> >> Sure. We will add you.
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> Thanks
>>>> >> >> > >>>>> >> Ajit
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit, David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Juraj
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Sorry for this issue.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Please let me know what would be a good time for you.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > This should work now.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me know if you still have issues.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ----
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > sudo su
>>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me try this method first.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If it does not work, we could try with the public ssh keys.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I will let you know.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me know if this helps.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks Owen.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get the following error.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Traceback (most recent call last):
>>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>>>> >> >> > >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hello Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Owen Hilyard
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need anything from me.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I had some trouble with my system over the past week.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I will login into the setup after that.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi David,
>>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
>>>> >> >> > >>>>> >> > Thanks for the refresh mail.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Hi Ajit,
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to the machines?
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
>>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Step I run:
>>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>>>> >> >> > >>>>> >> > > ./dts -s                                                           #To start the dts
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Informatio:
>>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
>>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Best,
>>>> >> >> > >>>>> >> > > David
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> > >>
>>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
>>>> >> >> > >>>>> >> > >>
>>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
>>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Informatio:
>>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
>>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Thanks,
>>>> >> >> > >>>>> >> > >>> David
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> Hi David,
>>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>>>> >> >> > >>>>> >> > >>>> see what is happening?
>>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
>>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you are trying to
>>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> Thanks
>>>> >> >> > >>>>> >> > >>>> Ajit
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything information from us.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Best,
>>>> >> >> > >>>>> >> > >>>> > David
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> Thank,
>>>> >> >> > >>>>> >> > >>>> >> David
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> Thanks
>>>> >> >> > >>>>> >> > >>>> >>> Ajit
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
>>>> >> >> > >>>>> >> > >>>> >>>> David
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
>>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
>>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
>>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
>>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
>>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> '
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
>>>> >> >> > >>>>> >> > >>>> >>>>>> David
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-07 20:16                                                                               ` Ajit Khaparde
@ 2021-04-09 22:09                                                                                 ` David Liu
  2021-04-09 23:34                                                                                   ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-04-09 22:09 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 36478 bytes --]

Hi Ajit,

We are working on an issue we have in the Arm server. When we are done, I
will let you know so you can take get into the server.
Best,
David

On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Thanks David.
> While I don't have the ARM server, I am trying to use the new version
> of TREX and see if I run into any issues.
> I will get back to you once I complete the test.
>
> Thanks
> Ajit
>
>
> On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > I had rerun the test with the recent DTS and Trex (v2.88)
> >
> > With this, I am getting the mapping error we see during the debugging
> session.
> >
> >
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg
> /opt/trex/v2.88/trex_cfg_bnxt.yaml
> > Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg
> /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy
> server......... Scapy server is started
> > Trying to bind to vfio-pci ...
> > /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0
> 0000:5f:00.1
> > The ports are bound/configured.
> > Starting  TRex v2.88 please wait  ...
> > ERROR in DPDK map
> > Could not find requested interface 5f:00.0
> > Killing Scapy server... Scapy server is killed]
> >                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64
> -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> >                         tester: pkill -f _t-rex-64
> >
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > Best,
> > David
> >
> >
> >
> > On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>
> >> Hi David,
> >> Yes. The changes are in the tree.
> >> Let's try to rerun the test and see what happens.
> >>
> >> Thanks
> >> Ajit
> >>
> >>
> >> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>
> >>> Hi Ajit,
> >>>
> >>> I think the patch should be in the main tree by now.
> >>>
> >>> Can you check if that is true?
> >>>
> >>> Just checking, is this patch related to the Trex mapping issue?
> >>>
> >>> Best,
> >>> David
> >>>
> >>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>>
> >>>> Hi David,
> >>>> The patch was in dpdk-next-net.
> >>>> Let me check if it was applied in the main tree.
> >>>>
> >>>> Thanks
> >>>> Ajit
> >>>>
> >>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >
> >>>> > Hi Ajit,
> >>>> >
> >>>> > Here to check in with you to see if the patch gets submitted.
> >>>> >
> >>>> > And for some confirmation, is this fix possibly be a solution to
> the current issue?
> >>>> >
> >>>> > Best,
> >>>> > David
> >>>> >
> >>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >>
> >>>> >> David,
> >>>> >> While testing with TREX, we have one identified one fix.
> >>>> >> We will try to submit it to the mailing list tomorrow.
> >>>> >> Once it gets merged, I will get back to you for the setup.
> >>>> >>
> >>>> >> Thanks
> >>>> >> Ajit
> >>>> >>
> >>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >>>> >> >
> >>>> >> > Hi Ajit,
> >>>> >> >
> >>>> >> > Thank you.
> >>>> >> >
> >>>> >> > Please let me know when you need to take over the machine again.
> >>>> >> >
> >>>> >> > Best,
> >>>> >> > David
> >>>> >> >
> >>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >>
> >>>> >> >> David,
> >>>> >> >> The tool has issues when I load the kernel module.
> >>>> >> >> I am looking for another alternative.
> >>>> >> >> Please go ahead and take back the setup.
> >>>> >> >>
> >>>> >> >> Thanks
> >>>> >> >> Ajit
> >>>> >> >>
> >>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> >
> >>>> >> >> > Hi David,
> >>>> >> >> > I am about to login into the setup and try something.
> >>>> >> >> > I will let you know once I am done and then you can take
> control.
> >>>> >> >> >
> >>>> >> >> > Thanks
> >>>> >> >> > Ajit
> >>>> >> >> >
> >>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >>>> >> >> > >
> >>>> >> >> > > Hi Ajit,
> >>>> >> >> > >
> >>>> >> >> > > Thank for the update.
> >>>> >> >> > >
> >>>> >> >> > > May I know if I can enable testing or would you like more
> time to look into this?
> >>>> >> >> > >
> >>>> >> >> > > Best,
> >>>> >> >> > > David
> >>>> >> >> > >
> >>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>
> >>>> >> >> > >> David,
> >>>> >> >> > >> I did spend some time on this. For now I can't tell what
> is wrong.
> >>>> >> >> > >> But I want to update the firmware on the card to take
> another shot at it.
> >>>> >> >> > >> Since this is an ARM based server, I will have to find a
> way to update the firmware.
> >>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking
> around.
> >>>> >> >> > >> I will update you soon.
> >>>> >> >> > >>
> >>>> >> >> > >> Thanks
> >>>> >> >> > >> Ajit
> >>>> >> >> > >>
> >>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>
> >>>> >> >> > >>> Hi David,
> >>>> >> >> > >>> That's a good idea. That way you don't have to watch out
> for my email requesting to stop the tests.
> >>>> >> >> > >>>
> >>>> >> >> > >>> Thanks
> >>>> >> >> > >>> Ajit
> >>>> >> >> > >>>
> >>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Hi Ajit, Juraj,
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> For simplicity, we will keep the functional and
> performance testing disable during the weekend.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> In this way, you will have no testing interfere with you
> during troubleshooting.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Please let me know if you have any questions.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Best,
> >>>> >> >> > >>>> David
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi David, Ajit,
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Juraj
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
> >>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> >>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen
> Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
> Broadcom NIC in Arm hardware
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi, Ajit
> >>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Best,
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> David
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi David,
> >>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
> >>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Thanks
> >>>> >> >> > >>>>> Ajit
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > Hi Ajit and Juraj,
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM.
> But feel free to adjust the time or let me know if there is a better time
> that works for you.
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > For me, I can do:
> >>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
> >>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
> >>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > Best,
> >>>> >> >> > >>>>> > David
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> David,
> >>>> >> >> > >>>>> >> I was busy with a customer issue and could not get
> to my emails.
> >>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues
> needing urgent attention.
> >>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> Juraj,
> >>>> >> >> > >>>>> >> Sure. We will add you.
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> Thanks
> >>>> >> >> > >>>>> >> Ajit
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš
> <juraj.linkes@pantheon.tech> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit, David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If you're setting up a call, could you please send
> me an invite? I'd like to get some insight into the issues on Arm's behalf.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Juraj
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David
> Liu
> >>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>;
> ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
> Broadcom NIC in Arm hardware
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Sorry for this issue.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If possible I would like to set up a time where we
> can do this in a call to help get this solved faster.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Please let me know what would be a good time for
> you.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > This should work now.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me know if you still have issues.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't
> work.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-gigabyte.  This incident will be reported.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ----
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > sudo su
> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-dell.  This incident will be reported.
> >>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-dell.  This incident will be reported.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into
> another machine as a normal user which will prompt you for the password.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the
> command line since you should have root access.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me try this method first.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If it does not work, we could try with the public
> ssh keys.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I will let you know.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Another solution would be to let me add your
> public ssh key into both machines.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me know if this helps.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks Owen.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get
> the following error.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Traceback (most recent call last):
> >>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line
> 60, in _connect_host
> >>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> >>>> >> >> > >>>>> >> >   File
> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
> login
> >>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
> >>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> ^^^^^^^^^^^^^^
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
> ohilyard@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hello Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > There was a misconfiguration in our identity
> management system. The issue should now be resolved.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Owen Hilyard
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde'
> via dpdklab <dpdklab@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > But the same credentials are not allowing me to
> login onto the servers you provided.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde
> arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need
> anything from me.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I had some trouble with my system over the past
> week.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping
> that my system will be restored early this week
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I will login into the setup after that.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi David,
> >>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
> >>>> >> >> > >>>>> >> > Thanks for the refresh mail.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Hi Ajit,
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login
> to the machines?
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Just some notes, we are currently running
> compile and some performance/functional testing.
> >>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so
> please let me know if you want me to stop these tests for you to debug.
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Step I run:
> >>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is
> to switch the dts configuration to Broadcom
> >>>> >> >> > >>>>> >> > > ./dts -s
>                    #To start the dts
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Informatio:
> >>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
> >>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Best,
> >>>> >> >> > >>>>> >> > > David
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>
> >>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I
> run Broadcom nic.
> >>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts
> -s
> >>>> >> >> > >>>>> >> > >>
> >>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same
> VPN Username and Password.
> >>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Informatio:
> >>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
> >>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Thanks,
> >>>> >> >> > >>>>> >> > >>> David
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> Hi David,
> >>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could
> login into the setup and
> >>>> >> >> > >>>>> >> > >>>> see what is happening?
> >>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in
> your lab -
> >>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and
> io.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems
> you are trying to
> >>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> Thanks
> >>>> >> >> > >>>>> >> > >>>> Ajit
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything
> information from us.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex
> and dts are on the same intel tester machine.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Best,
> >>>> >> >> > >>>>> >> > >>>> > David
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean
> the ports that are using for Trex. If so they are UP before I start the
> Trex.
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> Thank,
> >>>> >> >> > >>>>> >> > >>>> >> David
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit
> Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
> >>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the
> trex? I am trying to see if the port failed to successfully come up
> >>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link.
> Can you check and tell?
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> Thanks
> >>>> >> >> > >>>>> >> > >>>> >>> Ajit
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu
> <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not
> the Arm.
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
> >>>> >> >> > >>>>> >> > >>>> >>>> David
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit
> Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
> >>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
> >>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am
> asking because
> >>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create
> a baseline and then
> >>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
> >>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David
> Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is
> installed on an Arm machine.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I
> try to run nic_single_core_perf on the NIC.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86
> and dts from http://git.dpdk.org/tools/dts/
> >>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when
> testing.
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to
> run the test case inside the nic_single_core_perf:
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running
> at parameters: framesize: 64, rxd/txd: 512
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test
> Case test_perf_nic_single_core Result ERROR: Traceback (most recent call
> last):
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 196, in
> test_perf_nic_single_core
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>  self._prepare_transmission(stream_ids=stream_ids)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>  self._conn.reset(ports=self._ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
> clear_global, clear_xstats)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
> response from tcp://172.18.0.40:4501
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless
> mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with
> ./trex-console
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn
> off Trex.
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 -
> [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task
> 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout
> is 1 seconds
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a
> ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a
> rte_delay_us_block + 106
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
> CCoreEthIF::flush_tx_queue() + 31
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9
> TrexDpCore::start() + 89
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
> CGlobalTRex::run_in_core(unsigned char) + 487
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad
> ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa
> eal_thread_loop + 426
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> '
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working,
> this should not turn off trex but please correct me if I am wrong.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of
> how I can approach this issue?
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
> >>>> >> >> > >>>>> >> > >>>> >>>>>> David
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>

[-- Attachment #2: Type: text/html, Size: 76935 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-09 22:09                                                                                 ` David Liu
@ 2021-04-09 23:34                                                                                   ` Ajit Khaparde
  2021-04-15 17:07                                                                                     ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-04-09 23:34 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab


[-- Attachment #1.1: Type: text/plain, Size: 37676 bytes --]

Thanks David.
Next time when I get on the system I will try the 2.87 version of trex.
Apparently my team is having trouble trying with 2.88 version as well.
Let me know when the setup is ready.

Thanks
Ajit

On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu> wrote:

> Hi Ajit,
>
> We are working on an issue we have in the Arm server. When we are done, I
> will let you know so you can take get into the server.
> Best,
> David
>
> On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
>> Thanks David.
>> While I don't have the ARM server, I am trying to use the new version
>> of TREX and see if I run into any issues.
>> I will get back to you once I complete the test.
>>
>> Thanks
>> Ajit
>>
>>
>> On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> > I had rerun the test with the recent DTS and Trex (v2.88)
>> >
>> > With this, I am getting the mapping error we see during the debugging
>> session.
>> >
>> >
>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> >                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg
>> /opt/trex/v2.88/trex_cfg_bnxt.yaml
>> > Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg
>> /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy
>> server......... Scapy server is started
>> > Trying to bind to vfio-pci ...
>> > /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0
>> 0000:5f:00.1
>> > The ports are bound/configured.
>> > Starting  TRex v2.88 please wait  ...
>> > ERROR in DPDK map
>> > Could not find requested interface 5f:00.0
>> > Killing Scapy server... Scapy server is killed]
>> >                            dts: TIMEOUT on cd
>> /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>> >                         tester: pkill -f _t-rex-64
>> >
>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> >
>> > Best,
>> > David
>> >
>> >
>> >
>> > On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>
>> >> Hi David,
>> >> Yes. The changes are in the tree.
>> >> Let's try to rerun the test and see what happens.
>> >>
>> >> Thanks
>> >> Ajit
>> >>
>> >>
>> >> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>
>> >>> Hi Ajit,
>> >>>
>> >>> I think the patch should be in the main tree by now.
>> >>>
>> >>> Can you check if that is true?
>> >>>
>> >>> Just checking, is this patch related to the Trex mapping issue?
>> >>>
>> >>> Best,
>> >>> David
>> >>>
>> >>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>>
>> >>>> Hi David,
>> >>>> The patch was in dpdk-next-net.
>> >>>> Let me check if it was applied in the main tree.
>> >>>>
>> >>>> Thanks
>> >>>> Ajit
>> >>>>
>> >>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >
>> >>>> > Hi Ajit,
>> >>>> >
>> >>>> > Here to check in with you to see if the patch gets submitted.
>> >>>> >
>> >>>> > And for some confirmation, is this fix possibly be a solution to
>> the current issue?
>> >>>> >
>> >>>> > Best,
>> >>>> > David
>> >>>> >
>> >>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >>
>> >>>> >> David,
>> >>>> >> While testing with TREX, we have one identified one fix.
>> >>>> >> We will try to submit it to the mailing list tomorrow.
>> >>>> >> Once it gets merged, I will get back to you for the setup.
>> >>>> >>
>> >>>> >> Thanks
>> >>>> >> Ajit
>> >>>> >>
>> >>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >>>> >> >
>> >>>> >> > Hi Ajit,
>> >>>> >> >
>> >>>> >> > Thank you.
>> >>>> >> >
>> >>>> >> > Please let me know when you need to take over the machine again.
>> >>>> >> >
>> >>>> >> > Best,
>> >>>> >> > David
>> >>>> >> >
>> >>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >>
>> >>>> >> >> David,
>> >>>> >> >> The tool has issues when I load the kernel module.
>> >>>> >> >> I am looking for another alternative.
>> >>>> >> >> Please go ahead and take back the setup.
>> >>>> >> >>
>> >>>> >> >> Thanks
>> >>>> >> >> Ajit
>> >>>> >> >>
>> >>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> >
>> >>>> >> >> > Hi David,
>> >>>> >> >> > I am about to login into the setup and try something.
>> >>>> >> >> > I will let you know once I am done and then you can take
>> control.
>> >>>> >> >> >
>> >>>> >> >> > Thanks
>> >>>> >> >> > Ajit
>> >>>> >> >> >
>> >>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu>
>> wrote:
>> >>>> >> >> > >
>> >>>> >> >> > > Hi Ajit,
>> >>>> >> >> > >
>> >>>> >> >> > > Thank for the update.
>> >>>> >> >> > >
>> >>>> >> >> > > May I know if I can enable testing or would you like more
>> time to look into this?
>> >>>> >> >> > >
>> >>>> >> >> > > Best,
>> >>>> >> >> > > David
>> >>>> >> >> > >
>> >>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>
>> >>>> >> >> > >> David,
>> >>>> >> >> > >> I did spend some time on this. For now I can't tell what
>> is wrong.
>> >>>> >> >> > >> But I want to update the firmware on the card to take
>> another shot at it.
>> >>>> >> >> > >> Since this is an ARM based server, I will have to find a
>> way to update the firmware.
>> >>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking
>> around.
>> >>>> >> >> > >> I will update you soon.
>> >>>> >> >> > >>
>> >>>> >> >> > >> Thanks
>> >>>> >> >> > >> Ajit
>> >>>> >> >> > >>
>> >>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>
>> >>>> >> >> > >>> Hi David,
>> >>>> >> >> > >>> That's a good idea. That way you don't have to watch out
>> for my email requesting to stop the tests.
>> >>>> >> >> > >>>
>> >>>> >> >> > >>> Thanks
>> >>>> >> >> > >>> Ajit
>> >>>> >> >> > >>>
>> >>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> Hi Ajit, Juraj,
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> For simplicity, we will keep the functional and
>> performance testing disable during the weekend.
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> In this way, you will have no testing interfere with
>> you during troubleshooting.
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> Please let me know if you have any questions.
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> Best,
>> >>>> >> >> > >>>> David
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš
>> <juraj.linkes@pantheon.tech> wrote:
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Hi David, Ajit,
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Juraj
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
>> >>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>> >>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen
>> Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu
>> >
>> >>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
>> Broadcom NIC in Arm hardware
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Hi, Ajit
>> >>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Best,
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> David
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Hi David,
>> >>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>> >>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Thanks
>> >>>> >> >> > >>>>> Ajit
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > Hi Ajit and Juraj,
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00
>> AM. But feel free to adjust the time or let me know if there is a better
>> time that works for you.
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > For me, I can do:
>> >>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>> >>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>> >>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > Best,
>> >>>> >> >> > >>>>> > David
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >>
>> >>>> >> >> > >>>>> >> David,
>> >>>> >> >> > >>>>> >> I was busy with a customer issue and could not get
>> to my emails.
>> >>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues
>> needing urgent attention.
>> >>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
>> >>>> >> >> > >>>>> >>
>> >>>> >> >> > >>>>> >> Juraj,
>> >>>> >> >> > >>>>> >> Sure. We will add you.
>> >>>> >> >> > >>>>> >>
>> >>>> >> >> > >>>>> >> Thanks
>> >>>> >> >> > >>>>> >> Ajit
>> >>>> >> >> > >>>>> >>
>> >>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš
>> <juraj.linkes@pantheon.tech> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit, David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > If you're setting up a call, could you please
>> send me an invite? I'd like to get some insight into the issues on Arm's
>> behalf.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Juraj
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of
>> David Liu
>> >>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>> >>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>;
>> ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> >>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB]
>> 25G Broadcom NIC in Arm hardware
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Sorry for this issue.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > If possible I would like to set up a time where
>> we can do this in a call to help get this solved faster.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Please let me know what would be a good time for
>> you.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Best,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > This should work now.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Let me know if you still have issues.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Best,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't
>> work.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
>> arm-gigabyte.  This incident will be reported.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > ----
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > sudo su
>> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
>> arm-dell.  This incident will be reported.
>> >>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
>> arm-dell.  This incident will be reported.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into
>> another machine as a normal user which will prompt you for the password.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run
>> the command line since you should have root access.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Let me try this method first.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > If it does not work, we could try with the public
>> ssh keys.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I will let you know.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Another solution would be to let me add your
>> public ssh key into both machines.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Let me know if this helps.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Best,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks Owen.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get
>> the following error.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Traceback (most recent call last):
>> >>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line
>> 60, in _connect_host
>> >>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>> .+:)')
>> >>>> >> >> > >>>>> >> >   File
>> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
>> login
>> >>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>> >>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> ^^^^^^^^^^^^^^
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
>> ohilyard@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hello Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > There was a misconfiguration in our identity
>> management system. The issue should now be resolved.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Owen Hilyard
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde'
>> via dpdklab <dpdklab@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > But the same credentials are not allowing me to
>> login onto the servers you provided.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde
>> arm-gigabyte.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need
>> anything from me.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Best,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I had some trouble with my system over the past
>> week.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping
>> that my system will be restored early this week
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I will login into the setup after that.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>> ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi David,
>> >>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
>> >>>> >> >> > >>>>> >> > Thanks for the refresh mail.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Hi Ajit,
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login
>> to the machines?
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Just some notes, we are currently running
>> compile and some performance/functional testing.
>> >>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so
>> please let me know if you want me to stop these tests for you to debug.
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Step I run:
>> >>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this
>> is to switch the dts configuration to Broadcom
>> >>>> >> >> > >>>>> >> > > ./dts -s
>>                    #To start the dts
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Informatio:
>> >>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
>> >>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Best,
>> >>>> >> >> > >>>>> >> > > David
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>
>> >>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I
>> run Broadcom nic.
>> >>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts
>> -s
>> >>>> >> >> > >>>>> >> > >>
>> >>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> Hi Ajit,
>> >>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same
>> VPN Username and Password.
>> >>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> Informatio:
>> >>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
>> >>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> Thanks,
>> >>>> >> >> > >>>>> >> > >>> David
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde
>> <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> > >>>>
>> >>>> >> >> > >>>>> >> > >>>> Hi David,
>> >>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could
>> login into the setup and
>> >>>> >> >> > >>>>> >> > >>>> see what is happening?
>> >>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in
>> your lab -
>> >>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and
>> io.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems
>> you are trying to
>> >>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
>> >>>> >> >> > >>>>> >> > >>>>
>> >>>> >> >> > >>>>> >> > >>>> Thanks
>> >>>> >> >> > >>>>> >> > >>>> Ajit
>> >>>> >> >> > >>>>> >> > >>>>
>> >>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything
>> information from us.
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex
>> and dts are on the same intel tester machine.
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Best,
>> >>>> >> >> > >>>>> >> > >>>> > David
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
>> dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>
>> >>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>> >>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you
>> mean the ports that are using for Trex. If so they are UP before I start
>> the Trex.
>> >>>> >> >> > >>>>> >> > >>>> >>
>> >>>> >> >> > >>>>> >> > >>>> >> Thank,
>> >>>> >> >> > >>>>> >> > >>>> >> David
>> >>>> >> >> > >>>>> >> > >>>> >>
>> >>>> >> >> > >>>>> >> > >>>> >>
>> >>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit
>> Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>>
>> >>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>> >>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start
>> the trex? I am trying to see if the port failed to successfully come up
>> >>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link.
>> Can you check and tell?
>> >>>> >> >> > >>>>> >> > >>>> >>>
>> >>>> >> >> > >>>>> >> > >>>> >>> Thanks
>> >>>> >> >> > >>>>> >> > >>>> >>> Ajit
>> >>>> >> >> > >>>>> >> > >>>> >>>
>> >>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David
>> Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not
>> the Arm.
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
>> >>>> >> >> > >>>>> >> > >>>> >>>> David
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit
>> Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
>> >>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
>> >>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am
>> asking because
>> >>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create
>> a baseline and then
>> >>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
>> >>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David
>> Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is
>> installed on an Arm machine.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when
>> I try to run nic_single_core_perf on the NIC.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86
>> and dts from http://git.dpdk.org/tools/dts/
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when
>> testing.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to
>> run the test case inside the nic_single_core_perf:
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running
>> at parameters: framesize: 64, rxd/txd: 512
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test
>> Case test_perf_nic_single_core Result ERROR: Traceback (most recent call
>> last):
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "tests/TestSuite_nic_single_core_perf.py", line 196, in
>> test_perf_nic_single_core
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>> options=traffic_opt)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>  self._prepare_transmission(stream_ids=stream_ids)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>  self._conn.reset(ports=self._ports)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 339, in reset
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>> line 51, in wrap2
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>> line 1492, in clear_stats
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
>> clear_global, clear_xstats)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>> line 2876, in _clear_stats_common
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
>> response from tcp://172.18.0.40:4501
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless
>> mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with
>> ./trex-console
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with
>> command
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn
>> off Trex.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 -
>> [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task
>> 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout
>> is 1 seconds
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a
>> ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a
>> rte_delay_us_block + 106
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>> CVirtualIFPerSideStats*) + 3220
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
>> CCoreEthIF::flush_tx_queue() + 31
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>> + 92
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>> CFlowGenListPerThread*, bool) + 184
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>> CFlowGenListPerThread*, double&) + 881
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
>> TrexStatelessDpCore::start_scheduler() + 226
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9
>> TrexDpCore::start() + 89
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
>> CGlobalTRex::run_in_core(unsigned char) + 487
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad
>> ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa
>> eal_thread_loop + 426
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> '
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working,
>> this should not turn off trex but please correct me if I am wrong.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion
>> of how I can approach this issue?
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> David
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 77998 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-09 23:34                                                                                   ` Ajit Khaparde
@ 2021-04-15 17:07                                                                                     ` David Liu
  2021-04-21  7:39                                                                                       ` Juraj Linkeš
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-04-15 17:07 UTC (permalink / raw)
  To: Ajit Khaparde; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 39015 bytes --]

Hi Ajit,

You are free to hop on to the system now.
But do keep in mind that the dut server is running compile testing and the
other is free.
So let me know if you need that to be turned off any time.

For v2.88, I do have an issue where I have to replace system scapy with the
Trex scapy.

Best,
David

On Fri, Apr 9, 2021 at 7:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
wrote:

> Thanks David.
> Next time when I get on the system I will try the 2.87 version of trex.
> Apparently my team is having trouble trying with 2.88 version as well.
> Let me know when the setup is ready.
>
> Thanks
> Ajit
>
> On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu> wrote:
>
>> Hi Ajit,
>>
>> We are working on an issue we have in the Arm server. When we are done, I
>> will let you know so you can take get into the server.
>> Best,
>> David
>>
>> On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
>> wrote:
>>
>>> Thanks David.
>>> While I don't have the ARM server, I am trying to use the new version
>>> of TREX and see if I run into any issues.
>>> I will get back to you once I complete the test.
>>>
>>> Thanks
>>> Ajit
>>>
>>>
>>> On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >
>>> > Hi Ajit,
>>> >
>>> > I had rerun the test with the recent DTS and Trex (v2.88)
>>> >
>>> > With this, I am getting the mapping error we see during the debugging
>>> session.
>>> >
>>> >
>>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> >                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg
>>> /opt/trex/v2.88/trex_cfg_bnxt.yaml
>>> > Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg
>>> /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy
>>> server......... Scapy server is started
>>> > Trying to bind to vfio-pci ...
>>> > /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0
>>> 0000:5f:00.1
>>> > The ports are bound/configured.
>>> > Starting  TRex v2.88 please wait  ...
>>> > ERROR in DPDK map
>>> > Could not find requested interface 5f:00.0
>>> > Killing Scapy server... Scapy server is killed]
>>> >                            dts: TIMEOUT on cd
>>> /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>>> >                         tester: pkill -f _t-rex-64
>>> >
>>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>> >
>>> > Best,
>>> > David
>>> >
>>> >
>>> >
>>> > On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>
>>> >> Hi David,
>>> >> Yes. The changes are in the tree.
>>> >> Let's try to rerun the test and see what happens.
>>> >>
>>> >> Thanks
>>> >> Ajit
>>> >>
>>> >>
>>> >> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >>>
>>> >>> Hi Ajit,
>>> >>>
>>> >>> I think the patch should be in the main tree by now.
>>> >>>
>>> >>> Can you check if that is true?
>>> >>>
>>> >>> Just checking, is this patch related to the Trex mapping issue?
>>> >>>
>>> >>> Best,
>>> >>> David
>>> >>>
>>> >>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>>
>>> >>>> Hi David,
>>> >>>> The patch was in dpdk-next-net.
>>> >>>> Let me check if it was applied in the main tree.
>>> >>>>
>>> >>>> Thanks
>>> >>>> Ajit
>>> >>>>
>>> >>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
>>> >>>> >
>>> >>>> > Hi Ajit,
>>> >>>> >
>>> >>>> > Here to check in with you to see if the patch gets submitted.
>>> >>>> >
>>> >>>> > And for some confirmation, is this fix possibly be a solution to
>>> the current issue?
>>> >>>> >
>>> >>>> > Best,
>>> >>>> > David
>>> >>>> >
>>> >>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >>
>>> >>>> >> David,
>>> >>>> >> While testing with TREX, we have one identified one fix.
>>> >>>> >> We will try to submit it to the mailing list tomorrow.
>>> >>>> >> Once it gets merged, I will get back to you for the setup.
>>> >>>> >>
>>> >>>> >> Thanks
>>> >>>> >> Ajit
>>> >>>> >>
>>> >>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >>>> >> >
>>> >>>> >> > Hi Ajit,
>>> >>>> >> >
>>> >>>> >> > Thank you.
>>> >>>> >> >
>>> >>>> >> > Please let me know when you need to take over the machine
>>> again.
>>> >>>> >> >
>>> >>>> >> > Best,
>>> >>>> >> > David
>>> >>>> >> >
>>> >>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >>
>>> >>>> >> >> David,
>>> >>>> >> >> The tool has issues when I load the kernel module.
>>> >>>> >> >> I am looking for another alternative.
>>> >>>> >> >> Please go ahead and take back the setup.
>>> >>>> >> >>
>>> >>>> >> >> Thanks
>>> >>>> >> >> Ajit
>>> >>>> >> >>
>>> >>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> >
>>> >>>> >> >> > Hi David,
>>> >>>> >> >> > I am about to login into the setup and try something.
>>> >>>> >> >> > I will let you know once I am done and then you can take
>>> control.
>>> >>>> >> >> >
>>> >>>> >> >> > Thanks
>>> >>>> >> >> > Ajit
>>> >>>> >> >> >
>>> >>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu>
>>> wrote:
>>> >>>> >> >> > >
>>> >>>> >> >> > > Hi Ajit,
>>> >>>> >> >> > >
>>> >>>> >> >> > > Thank for the update.
>>> >>>> >> >> > >
>>> >>>> >> >> > > May I know if I can enable testing or would you like more
>>> time to look into this?
>>> >>>> >> >> > >
>>> >>>> >> >> > > Best,
>>> >>>> >> >> > > David
>>> >>>> >> >> > >
>>> >>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>
>>> >>>> >> >> > >> David,
>>> >>>> >> >> > >> I did spend some time on this. For now I can't tell what
>>> is wrong.
>>> >>>> >> >> > >> But I want to update the firmware on the card to take
>>> another shot at it.
>>> >>>> >> >> > >> Since this is an ARM based server, I will have to find a
>>> way to update the firmware.
>>> >>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking
>>> around.
>>> >>>> >> >> > >> I will update you soon.
>>> >>>> >> >> > >>
>>> >>>> >> >> > >> Thanks
>>> >>>> >> >> > >> Ajit
>>> >>>> >> >> > >>
>>> >>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>
>>> >>>> >> >> > >>> Hi David,
>>> >>>> >> >> > >>> That's a good idea. That way you don't have to watch
>>> out for my email requesting to stop the tests.
>>> >>>> >> >> > >>>
>>> >>>> >> >> > >>> Thanks
>>> >>>> >> >> > >>> Ajit
>>> >>>> >> >> > >>>
>>> >>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>
>>> >>>> >> >> > >>>> Hi Ajit, Juraj,
>>> >>>> >> >> > >>>>
>>> >>>> >> >> > >>>> For simplicity, we will keep the functional and
>>> performance testing disable during the weekend.
>>> >>>> >> >> > >>>>
>>> >>>> >> >> > >>>> In this way, you will have no testing interfere with
>>> you during troubleshooting.
>>> >>>> >> >> > >>>>
>>> >>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
>>> >>>> >> >> > >>>>
>>> >>>> >> >> > >>>> Please let me know if you have any questions.
>>> >>>> >> >> > >>>>
>>> >>>> >> >> > >>>> Best,
>>> >>>> >> >> > >>>> David
>>> >>>> >> >> > >>>>
>>> >>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš
>>> <juraj.linkes@pantheon.tech> wrote:
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> Hi David, Ajit,
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> Juraj
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
>>> >>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>>> >>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>> >>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen
>>> Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <
>>> dpdklab@iol.unh.edu>
>>> >>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
>>> Broadcom NIC in Arm hardware
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> Hi, Ajit
>>> >>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> Best,
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> David
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> Hi David,
>>> >>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>>> >>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> Thanks
>>> >>>> >> >> > >>>>> Ajit
>>> >>>> >> >> > >>>>>
>>> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >
>>> >>>> >> >> > >>>>> > Hi Ajit and Juraj,
>>> >>>> >> >> > >>>>> >
>>> >>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00
>>> AM. But feel free to adjust the time or let me know if there is a better
>>> time that works for you.
>>> >>>> >> >> > >>>>> >
>>> >>>> >> >> > >>>>> > For me, I can do:
>>> >>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>>> >>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>>> >>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>>> >>>> >> >> > >>>>> >
>>> >>>> >> >> > >>>>> > Best,
>>> >>>> >> >> > >>>>> > David
>>> >>>> >> >> > >>>>> >
>>> >>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >>
>>> >>>> >> >> > >>>>> >> David,
>>> >>>> >> >> > >>>>> >> I was busy with a customer issue and could not get
>>> to my emails.
>>> >>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues
>>> needing urgent attention.
>>> >>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
>>> >>>> >> >> > >>>>> >>
>>> >>>> >> >> > >>>>> >> Juraj,
>>> >>>> >> >> > >>>>> >> Sure. We will add you.
>>> >>>> >> >> > >>>>> >>
>>> >>>> >> >> > >>>>> >> Thanks
>>> >>>> >> >> > >>>>> >> Ajit
>>> >>>> >> >> > >>>>> >>
>>> >>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš
>>> <juraj.linkes@pantheon.tech> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Hi Ajit, David,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > If you're setting up a call, could you please
>>> send me an invite? I'd like to get some insight into the issues on Arm's
>>> behalf.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Juraj
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of
>>> David Liu
>>> >>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>>> >>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>> >>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>;
>>> ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>>> >>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB]
>>> 25G Broadcom NIC in Arm hardware
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Hi Ajit,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Sorry for this issue.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > If possible I would like to set up a time where
>>> we can do this in a call to help get this solved faster.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Please let me know what would be a good time for
>>> you.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Best,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > David
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Hi Ajit,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > I changed the permission to allow you to run
>>> sudo.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > This should work now.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Let me know if you still have issues.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Best,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > David
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > David,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't
>>> work.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Ajit
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
>>> arm-gigabyte.  This incident will be reported.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > ----
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > sudo su
>>> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
>>> arm-dell.  This incident will be reported.
>>> >>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>>> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
>>> arm-dell.  This incident will be reported.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Hi Ajit,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh
>>> into another machine as a normal user which will prompt you for the
>>> password.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run
>>> the command line since you should have root access.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > David,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Let me try this method first.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > If it does not work, we could try with the
>>> public ssh keys.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > I will let you know.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Ajit
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Another solution would be to let me add your
>>> public ssh key into both machines.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Let me know if this helps.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Best,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > David
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks Owen.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get
>>> the following error.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Ajit
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Traceback (most recent call last):
>>> >>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line
>>> 60, in _connect_host
>>> >>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
>>> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
>>> .+:)')
>>> >>>> >> >> > >>>>> >> >   File
>>> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
>>> login
>>> >>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>>> >>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> ^^^^^^^^^^^^^^
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
>>> ohilyard@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Hello Ajit,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > There was a misconfiguration in our identity
>>> management system. The issue should now be resolved.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Owen Hilyard
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde'
>>> via dpdklab <dpdklab@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > David,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > But the same credentials are not allowing me to
>>> login onto the servers you provided.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Ajit
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde
>>> arm-gigabyte.dpdklab.iol.unh.edu
>>> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>>> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Hi Ajit,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need
>>> anything from me.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Best,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > David
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > David,
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > I had some trouble with my system over the past
>>> week.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping
>>> that my system will be restored early this week
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > I will login into the setup after that.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Ajit
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
>>> ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Hi David,
>>> >>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
>>> >>>> >> >> > >>>>> >> > Thanks for the refresh mail.
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > Thanks
>>> >>>> >> >> > >>>>> >> > Ajit
>>> >>>> >> >> > >>>>> >> >
>>> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > Hi Ajit,
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on
>>> login to the machines?
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > Just some notes, we are currently running
>>> compile and some performance/functional testing.
>>> >>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so
>>> please let me know if you want me to stop these tests for you to debug.
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > Step I run:
>>> >>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this
>>> is to switch the dts configuration to Broadcom
>>> >>>> >> >> > >>>>> >> > > ./dts -s
>>>                      #To start the dts
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > Informatio:
>>> >>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>>> >>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
>>> >>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>>> >>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > Best,
>>> >>>> >> >> > >>>>> >> > > David
>>> >>>> >> >> > >>>>> >> > >
>>> >>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> > >>
>>> >>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I
>>> run Broadcom nic.
>>> >>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with
>>> ./dts -s
>>> >>>> >> >> > >>>>> >> > >>
>>> >>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> > >>>
>>> >>>> >> >> > >>>>> >> > >>> Hi Ajit,
>>> >>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
>>> >>>> >> >> > >>>>> >> > >>>
>>> >>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same
>>> VPN Username and Password.
>>> >>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
>>> >>>> >> >> > >>>>> >> > >>>
>>> >>>> >> >> > >>>>> >> > >>> Informatio:
>>> >>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>>> >>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
>>> >>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>>> >>>> >> >> > >>>>> >> > >>>
>>> >>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>>> >>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>>> >>>> >> >> > >>>>> >> > >>>
>>> >>>> >> >> > >>>>> >> > >>>
>>> >>>> >> >> > >>>>> >> > >>> Thanks,
>>> >>>> >> >> > >>>>> >> > >>> David
>>> >>>> >> >> > >>>>> >> > >>>
>>> >>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit
>>> Khaparde <ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >> > >>>>
>>> >>>> >> >> > >>>>> >> > >>>> Hi David,
>>> >>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could
>>> login into the setup and
>>> >>>> >> >> > >>>>> >> > >>>> see what is happening?
>>> >>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems
>>> in your lab -
>>> >>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and
>>> io.dpdklab.iol.unh.edu
>>> >>>> >> >> > >>>>> >> > >>>> If you can provide me details of the
>>> systems you are trying to
>>> >>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
>>> >>>> >> >> > >>>>> >> > >>>>
>>> >>>> >> >> > >>>>> >> > >>>> Thanks
>>> >>>> >> >> > >>>>> >> > >>>> Ajit
>>> >>>> >> >> > >>>>> >> > >>>>
>>> >>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
>>> dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> > >>>> >
>>> >>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
>>> >>>> >> >> > >>>>> >> > >>>> >
>>> >>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
>>> >>>> >> >> > >>>>> >> > >>>> >
>>> >>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
>>> >>>> >> >> > >>>>> >> > >>>> >
>>> >>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything
>>> information from us.
>>> >>>> >> >> > >>>>> >> > >>>> >
>>> >>>> >> >> > >>>>> >> > >>>> > Just some extra information that both
>>> Trex and dts are on the same intel tester machine.
>>> >>>> >> >> > >>>>> >> > >>>> >
>>> >>>> >> >> > >>>>> >> > >>>> > Best,
>>> >>>> >> >> > >>>>> >> > >>>> > David
>>> >>>> >> >> > >>>>> >> > >>>> >
>>> >>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu
>>> <dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> > >>>> >>
>>> >>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>>> >>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you
>>> mean the ports that are using for Trex. If so they are UP before I start
>>> the Trex.
>>> >>>> >> >> > >>>>> >> > >>>> >>
>>> >>>> >> >> > >>>>> >> > >>>> >> Thank,
>>> >>>> >> >> > >>>>> >> > >>>> >> David
>>> >>>> >> >> > >>>>> >> > >>>> >>
>>> >>>> >> >> > >>>>> >> > >>>> >>
>>> >>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit
>>> Khaparde <ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >> > >>>> >>>
>>> >>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>>> >>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start
>>> the trex? I am trying to see if the port failed to successfully come up
>>> >>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link.
>>> Can you check and tell?
>>> >>>> >> >> > >>>>> >> > >>>> >>>
>>> >>>> >> >> > >>>>> >> > >>>> >>> Thanks
>>> >>>> >> >> > >>>>> >> > >>>> >>> Ajit
>>> >>>> >> >> > >>>>> >> > >>>> >>>
>>> >>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David
>>> Liu <dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> > >>>> >>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>>> >>>> >> >> > >>>>> >> > >>>> >>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not
>>> the Arm.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
>>> >>>> >> >> > >>>>> >> > >>>> >>>> David
>>> >>>> >> >> > >>>>> >> > >>>> >>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit
>>> Khaparde <ajit.khaparde@broadcom.com> wrote:
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
>>> >>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I
>>> am asking because
>>> >>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to
>>> create a baseline and then
>>> >>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
>>> >>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM
>>> David Liu <dliu@iol.unh.edu> wrote:
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is
>>> installed on an Arm machine.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when
>>> I try to run nic_single_core_perf on the NIC.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86
>>> and dts from http://git.dpdk.org/tools/dts/
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when
>>> testing.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try
>>> to run the test case inside the nic_single_core_perf:
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running
>>> at parameters: framesize: 64, rxd/txd: 512
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
>>> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
>>> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
>>> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
>>> start
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf:
>>> Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent
>>> call last):
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "tests/TestSuite_nic_single_core_perf.py", line 196, in
>>> test_perf_nic_single_core
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
>>> self.tester.pktgen.measure_throughput(stream_ids=streams,
>>> options=traffic_opt)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>  self._prepare_transmission(stream_ids=stream_ids)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>  self._conn.reset(ports=self._ports)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 339, in reset
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
>>> line 51, in wrap2
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
>>> line 1492, in clear_stats
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
>>> clear_global, clear_xstats)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
>>> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
>>> line 2876, in _clear_stats_common
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
>>> response from tcp://172.18.0.40:4501
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless
>>> mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with
>>> ./trex-console
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with
>>> command
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn
>>> off Trex.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 -
>>> [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task
>>> 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout
>>> is 1 seconds
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a
>>> ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a
>>> rte_delay_us_block + 106
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
>>> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
>>> CVirtualIFPerSideStats*) + 3220
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
>>> CCoreEthIF::flush_tx_queue() + 31
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
>>> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
>>> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
>>> + 92
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
>>> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
>>> CFlowGenListPerThread*, bool) + 184
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
>>> CNodeGenerator::flush_file_realtime<23, false>(double, double,
>>> CFlowGenListPerThread*, double&) + 881
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
>>> TrexStatelessDpCore::start_scheduler() + 226
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9
>>> TrexDpCore::start() + 89
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
>>> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
>>> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
>>> CGlobalTRex::run_in_core(unsigned char) + 487
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad
>>> ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa
>>> eal_thread_loop + 426
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows
>>> ***
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> '
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working,
>>> this should not turn off trex but please correct me if I am wrong.
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion
>>> of how I can approach this issue?
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>> David
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>
>>

[-- Attachment #2: Type: text/html, Size: 78797 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-15 17:07                                                                                     ` David Liu
@ 2021-04-21  7:39                                                                                       ` Juraj Linkeš
  2021-04-21 14:20                                                                                         ` Ajit Khaparde
  2021-04-22  3:55                                                                                         ` David Liu
  0 siblings, 2 replies; 50+ messages in thread
From: Juraj Linkeš @ 2021-04-21  7:39 UTC (permalink / raw)
  To: David Liu, Ajit Khaparde; +Cc: Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 37778 bytes --]

Hi Ajit, David,

What is the current status of this issue (what the current problem is, what we need to investigate, any blockers)? Do we have a bugzilla ticket where we track this? If not, we could just update https://bugs.dpdk.org/show_bug.cgi?id=408.

Thanks,
Juraj

From: David Liu <dliu@iol.unh.edu>
Sent: Thursday, April 15, 2021 7:08 PM
To: Ajit Khaparde <ajit.khaparde@broadcom.com>
Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware

Hi Ajit,

You are free to hop on to the system now.
But do keep in mind that the dut server is running compile testing and the other is free.
So let me know if you need that to be turned off any time.

For v2.88, I do have an issue where I have to replace system scapy with the Trex scapy.

Best,
David

On Fri, Apr 9, 2021 at 7:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
Thanks David.
Next time when I get on the system I will try the 2.87 version of trex.
Apparently my team is having trouble trying with 2.88 version as well.
Let me know when the setup is ready.

Thanks
Ajit

On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
Hi Ajit,

We are working on an issue we have in the Arm server. When we are done, I will let you know so you can take get into the server.
Best,
David

On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
Thanks David.
While I don't have the ARM server, I am trying to use the new version
of TREX and see if I run into any issues.
I will get back to you once I complete the test.

Thanks
Ajit


On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>
> Hi Ajit,
>
> I had rerun the test with the recent DTS and Trex (v2.88)
>
> With this, I am getting the mapping error we see during the debugging session.
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
> Trying to bind to vfio-pci ...
> /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0 0000:5f:00.1
> The ports are bound/configured.
> Starting  TRex v2.88 please wait  ...
> ERROR in DPDK map
> Could not find requested interface 5f:00.0
> Killing Scapy server... Scapy server is killed]
>                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>                         tester: pkill -f _t-rex-64
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Best,
> David
>
>
>
> On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>
>> Hi David,
>> Yes. The changes are in the tree.
>> Let's try to rerun the test and see what happens.
>>
>> Thanks
>> Ajit
>>
>>
>> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>
>>> Hi Ajit,
>>>
>>> I think the patch should be in the main tree by now.
>>>
>>> Can you check if that is true?
>>>
>>> Just checking, is this patch related to the Trex mapping issue?
>>>
>>> Best,
>>> David
>>>
>>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>>
>>>> Hi David,
>>>> The patch was in dpdk-next-net.
>>>> Let me check if it was applied in the main tree.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >
>>>> > Hi Ajit,
>>>> >
>>>> > Here to check in with you to see if the patch gets submitted.
>>>> >
>>>> > And for some confirmation, is this fix possibly be a solution to the current issue?
>>>> >
>>>> > Best,
>>>> > David
>>>> >
>>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >>
>>>> >> David,
>>>> >> While testing with TREX, we have one identified one fix.
>>>> >> We will try to submit it to the mailing list tomorrow.
>>>> >> Once it gets merged, I will get back to you for the setup.
>>>> >>
>>>> >> Thanks
>>>> >> Ajit
>>>> >>
>>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >
>>>> >> > Hi Ajit,
>>>> >> >
>>>> >> > Thank you.
>>>> >> >
>>>> >> > Please let me know when you need to take over the machine again.
>>>> >> >
>>>> >> > Best,
>>>> >> > David
>>>> >> >
>>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >>
>>>> >> >> David,
>>>> >> >> The tool has issues when I load the kernel module.
>>>> >> >> I am looking for another alternative.
>>>> >> >> Please go ahead and take back the setup.
>>>> >> >>
>>>> >> >> Thanks
>>>> >> >> Ajit
>>>> >> >>
>>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> >
>>>> >> >> > Hi David,
>>>> >> >> > I am about to login into the setup and try something.
>>>> >> >> > I will let you know once I am done and then you can take control.
>>>> >> >> >
>>>> >> >> > Thanks
>>>> >> >> > Ajit
>>>> >> >> >
>>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >
>>>> >> >> > > Hi Ajit,
>>>> >> >> > >
>>>> >> >> > > Thank for the update.
>>>> >> >> > >
>>>> >> >> > > May I know if I can enable testing or would you like more time to look into this?
>>>> >> >> > >
>>>> >> >> > > Best,
>>>> >> >> > > David
>>>> >> >> > >
>>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>
>>>> >> >> > >> David,
>>>> >> >> > >> I did spend some time on this. For now I can't tell what is wrong.
>>>> >> >> > >> But I want to update the firmware on the card to take another shot at it.
>>>> >> >> > >> Since this is an ARM based server, I will have to find a way to update the firmware.
>>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
>>>> >> >> > >> I will update you soon.
>>>> >> >> > >>
>>>> >> >> > >> Thanks
>>>> >> >> > >> Ajit
>>>> >> >> > >>
>>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>
>>>> >> >> > >>> Hi David,
>>>> >> >> > >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
>>>> >> >> > >>>
>>>> >> >> > >>> Thanks
>>>> >> >> > >>> Ajit
>>>> >> >> > >>>
>>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>
>>>> >> >> > >>>> Hi Ajit, Juraj,
>>>> >> >> > >>>>
>>>> >> >> > >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
>>>> >> >> > >>>>
>>>> >> >> > >>>> In this way, you will have no testing interfere with you during troubleshooting.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Please let me know if you have any questions.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Best,
>>>> >> >> > >>>> David
>>>> >> >> > >>>>
>>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>> wrote:
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi David, Ajit,
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Juraj
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>>
>>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>>
>>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>>; Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>>; ci@dpdk.org<mailto:ci@dpdk.org>; dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>>
>>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi, Ajit
>>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Best,
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> David
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi David,
>>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Thanks
>>>> >> >> > >>>>> Ajit
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > Hi Ajit and Juraj,
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > For me, I can do:
>>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > Best,
>>>> >> >> > >>>>> > David
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> David,
>>>> >> >> > >>>>> >> I was busy with a customer issue and could not get to my emails.
>>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
>>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> Juraj,
>>>> >> >> > >>>>> >> Sure. We will add you.
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> Thanks
>>>> >> >> > >>>>> >> Ajit
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit, David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Juraj
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org<mailto:ci-bounces@dpdk.org>> On Behalf Of David Liu
>>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>>
>>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>>; ci@dpdk.org<mailto:ci@dpdk.org>; dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>>
>>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Sorry for this issue.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Please let me know what would be a good time for you.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > This should work now.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me know if you still have issues.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ----
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > sudo su
>>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me try this method first.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If it does not work, we could try with the public ssh keys.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I will let you know.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me know if this helps.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks Owen.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get the following error.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Traceback (most recent call last):
>>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>>>> >> >> > >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hello Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Owen Hilyard
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need anything from me.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I had some trouble with my system over the past week.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I will login into the setup after that.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi David,
>>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
>>>> >> >> > >>>>> >> > Thanks for the refresh mail.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Hi Ajit,
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to the machines?
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
>>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Step I run:
>>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>>>> >> >> > >>>>> >> > > ./dts -s                                                           #To start the dts
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Informatio:
>>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
>>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Best,
>>>> >> >> > >>>>> >> > > David
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>
>>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
>>>> >> >> > >>>>> >> > >>
>>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
>>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Informatio:
>>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
>>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Thanks,
>>>> >> >> > >>>>> >> > >>> David
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> Hi David,
>>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>>>> >> >> > >>>>> >> > >>>> see what is happening?
>>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
>>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu<http://rhea.dpdklab.iol.unh.edu> and io.dpdklab.iol.unh.edu<http://io.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you are trying to
>>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> Thanks
>>>> >> >> > >>>>> >> > >>>> Ajit
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything information from us.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Best,
>>>> >> >> > >>>>> >> > >>>> > David
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> Thank,
>>>> >> >> > >>>>> >> > >>>> >> David
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> Thanks
>>>> >> >> > >>>>> >> > >>>> >>> Ajit
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
>>>> >> >> > >>>>> >> > >>>> >>>> David
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
>>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
>>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
>>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
>>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501<http://172.18.0.40:4501>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
>>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> '
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
>>>> >> >> > >>>>> >> > >>>> >>>>>> David
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>

[-- Attachment #2: Type: text/html, Size: 85061 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-21  7:39                                                                                       ` Juraj Linkeš
@ 2021-04-21 14:20                                                                                         ` Ajit Khaparde
  2021-04-22  3:55                                                                                         ` David Liu
  1 sibling, 0 replies; 50+ messages in thread
From: Ajit Khaparde @ 2021-04-21 14:20 UTC (permalink / raw)
  To: Juraj Linkeš; +Cc: David Liu, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 37823 bytes --]

I had some personal work to take care of for the last two days and
even today I have something to handle. But tomorrow I should be able
to get on the system.

I will send an update once I spend some time on the setup.

Thanks
Ajit

On Wed, Apr 21, 2021 at 12:39 AM Juraj Linkeš
<juraj.linkes@pantheon.tech> wrote:
>
> Hi Ajit, David,
>
>
>
> What is the current status of this issue (what the current problem is, what we need to investigate, any blockers)? Do we have a bugzilla ticket where we track this? If not, we could just update https://bugs.dpdk.org/show_bug.cgi?id=408.
>
>
>
> Thanks,
>
> Juraj
>
>
>
> From: David Liu <dliu@iol.unh.edu>
> Sent: Thursday, April 15, 2021 7:08 PM
> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>
>
>
> Hi Ajit,
>
>
>
> You are free to hop on to the system now.
> But do keep in mind that the dut server is running compile testing and the other is free.
>
> So let me know if you need that to be turned off any time.
>
>
>
> For v2.88, I do have an issue where I have to replace system scapy with the Trex scapy.
>
>
>
> Best,
>
> David
>
>
>
> On Fri, Apr 9, 2021 at 7:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
> Thanks David.
>
> Next time when I get on the system I will try the 2.87 version of trex.
>
> Apparently my team is having trouble trying with 2.88 version as well.
>
> Let me know when the setup is ready.
>
>
>
> Thanks
>
> Ajit
>
>
>
> On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
>
>
> We are working on an issue we have in the Arm server. When we are done, I will let you know so you can take get into the server.
>
> Best,
>
> David
>
>
>
> On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>
> Thanks David.
> While I don't have the ARM server, I am trying to use the new version
> of TREX and see if I run into any issues.
> I will get back to you once I complete the test.
>
> Thanks
> Ajit
>
>
> On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > I had rerun the test with the recent DTS and Trex (v2.88)
> >
> > With this, I am getting the mapping error we see during the debugging session.
> >
> > -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> > Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
> > Trying to bind to vfio-pci ...
> > /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0 0000:5f:00.1
> > The ports are bound/configured.
> > Starting  TRex v2.88 please wait  ...
> > ERROR in DPDK map
> > Could not find requested interface 5f:00.0
> > Killing Scapy server... Scapy server is killed]
> >                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> >                         tester: pkill -f _t-rex-64
> > -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > Best,
> > David
> >
> >
> >
> > On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>
> >> Hi David,
> >> Yes. The changes are in the tree.
> >> Let's try to rerun the test and see what happens.
> >>
> >> Thanks
> >> Ajit
> >>
> >>
> >> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>
> >>> Hi Ajit,
> >>>
> >>> I think the patch should be in the main tree by now.
> >>>
> >>> Can you check if that is true?
> >>>
> >>> Just checking, is this patch related to the Trex mapping issue?
> >>>
> >>> Best,
> >>> David
> >>>
> >>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>>
> >>>> Hi David,
> >>>> The patch was in dpdk-next-net.
> >>>> Let me check if it was applied in the main tree.
> >>>>
> >>>> Thanks
> >>>> Ajit
> >>>>
> >>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >
> >>>> > Hi Ajit,
> >>>> >
> >>>> > Here to check in with you to see if the patch gets submitted.
> >>>> >
> >>>> > And for some confirmation, is this fix possibly be a solution to the current issue?
> >>>> >
> >>>> > Best,
> >>>> > David
> >>>> >
> >>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >>
> >>>> >> David,
> >>>> >> While testing with TREX, we have one identified one fix.
> >>>> >> We will try to submit it to the mailing list tomorrow.
> >>>> >> Once it gets merged, I will get back to you for the setup.
> >>>> >>
> >>>> >> Thanks
> >>>> >> Ajit
> >>>> >>
> >>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >
> >>>> >> > Hi Ajit,
> >>>> >> >
> >>>> >> > Thank you.
> >>>> >> >
> >>>> >> > Please let me know when you need to take over the machine again.
> >>>> >> >
> >>>> >> > Best,
> >>>> >> > David
> >>>> >> >
> >>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >>
> >>>> >> >> David,
> >>>> >> >> The tool has issues when I load the kernel module.
> >>>> >> >> I am looking for another alternative.
> >>>> >> >> Please go ahead and take back the setup.
> >>>> >> >>
> >>>> >> >> Thanks
> >>>> >> >> Ajit
> >>>> >> >>
> >>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> >
> >>>> >> >> > Hi David,
> >>>> >> >> > I am about to login into the setup and try something.
> >>>> >> >> > I will let you know once I am done and then you can take control.
> >>>> >> >> >
> >>>> >> >> > Thanks
> >>>> >> >> > Ajit
> >>>> >> >> >
> >>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >
> >>>> >> >> > > Hi Ajit,
> >>>> >> >> > >
> >>>> >> >> > > Thank for the update.
> >>>> >> >> > >
> >>>> >> >> > > May I know if I can enable testing or would you like more time to look into this?
> >>>> >> >> > >
> >>>> >> >> > > Best,
> >>>> >> >> > > David
> >>>> >> >> > >
> >>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>
> >>>> >> >> > >> David,
> >>>> >> >> > >> I did spend some time on this. For now I can't tell what is wrong.
> >>>> >> >> > >> But I want to update the firmware on the card to take another shot at it.
> >>>> >> >> > >> Since this is an ARM based server, I will have to find a way to update the firmware.
> >>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
> >>>> >> >> > >> I will update you soon.
> >>>> >> >> > >>
> >>>> >> >> > >> Thanks
> >>>> >> >> > >> Ajit
> >>>> >> >> > >>
> >>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>
> >>>> >> >> > >>> Hi David,
> >>>> >> >> > >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
> >>>> >> >> > >>>
> >>>> >> >> > >>> Thanks
> >>>> >> >> > >>> Ajit
> >>>> >> >> > >>>
> >>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Hi Ajit, Juraj,
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> In this way, you will have no testing interfere with you during troubleshooting.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Please let me know if you have any questions.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Best,
> >>>> >> >> > >>>> David
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi David, Ajit,
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Juraj
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
> >>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> >>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi, Ajit
> >>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Best,
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> David
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi David,
> >>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
> >>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Thanks
> >>>> >> >> > >>>>> Ajit
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > Hi Ajit and Juraj,
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > For me, I can do:
> >>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
> >>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
> >>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > Best,
> >>>> >> >> > >>>>> > David
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> David,
> >>>> >> >> > >>>>> >> I was busy with a customer issue and could not get to my emails.
> >>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
> >>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> Juraj,
> >>>> >> >> > >>>>> >> Sure. We will add you.
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> Thanks
> >>>> >> >> > >>>>> >> Ajit
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit, David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Juraj
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> >>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Sorry for this issue.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Please let me know what would be a good time for you.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > This should work now.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me know if you still have issues.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ----
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > sudo su
> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
> >>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me try this method first.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If it does not work, we could try with the public ssh keys.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I will let you know.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me know if this helps.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks Owen.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get the following error.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Traceback (most recent call last):
> >>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
> >>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
> >>>> >> >> > >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
> >>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
> >>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hello Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Owen Hilyard
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need anything from me.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I had some trouble with my system over the past week.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I will login into the setup after that.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi David,
> >>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
> >>>> >> >> > >>>>> >> > Thanks for the refresh mail.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Hi Ajit,
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to the machines?
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
> >>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Step I run:
> >>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
> >>>> >> >> > >>>>> >> > > ./dts -s                                                           #To start the dts
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Informatio:
> >>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
> >>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Best,
> >>>> >> >> > >>>>> >> > > David
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>
> >>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
> >>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
> >>>> >> >> > >>>>> >> > >>
> >>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
> >>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Informatio:
> >>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
> >>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Thanks,
> >>>> >> >> > >>>>> >> > >>> David
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> Hi David,
> >>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
> >>>> >> >> > >>>>> >> > >>>> see what is happening?
> >>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
> >>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you are trying to
> >>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> Thanks
> >>>> >> >> > >>>>> >> > >>>> Ajit
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything information from us.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Best,
> >>>> >> >> > >>>>> >> > >>>> > David
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> Thank,
> >>>> >> >> > >>>>> >> > >>>> >> David
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
> >>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
> >>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> Thanks
> >>>> >> >> > >>>>> >> > >>>> >>> Ajit
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
> >>>> >> >> > >>>>> >> > >>>> >>>> David
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
> >>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
> >>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
> >>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
> >>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
> >>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
> >>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> '
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
> >>>> >> >> > >>>>> >> > >>>> >>>>>> David
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-21  7:39                                                                                       ` Juraj Linkeš
  2021-04-21 14:20                                                                                         ` Ajit Khaparde
@ 2021-04-22  3:55                                                                                         ` David Liu
  2021-04-22  6:50                                                                                           ` Juraj Linkeš
  1 sibling, 1 reply; 50+ messages in thread
From: David Liu @ 2021-04-22  3:55 UTC (permalink / raw)
  To: Juraj Linkeš; +Cc: Ajit Khaparde, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 39239 bytes --]

Hi Juraj,

The current issue is when running DTS with the newest release of Tex v2.89.
It will report an error:

===========================================================================================================================
pktgen: ssh root@172.18.0.40
                        pktgen: cd /opt/trex/v2.89;./t-rex-64 -i --cfg
/opt/trex/v2.88/trex_cfg_bnxt.yaml
Exception happened in [cd /opt/trex/v2.89;./t-rex-64 -i --cfg
/opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy
server......... Scapy server is started
The ports are bound/configured.

Starting  TRex v2.89 please wait  ...

ERROR in DPDK map
Could not find requested interface 5f:00.0
Killing Scapy server... Scapy server is killed]
                           dts: TIMEOUT on cd /opt/trex/v2.89;./t-rex-64 -i
--cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
                        tester: pkill -f _t-rex-64
===========================================================================================================================

We do not have a Bugzilla ticket for this issue. So I will update the 480
ticket to track this issue.

Thank,
David

On Wed, Apr 21, 2021 at 3:39 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:

> Hi Ajit, David,
>
>
>
> What is the current status of this issue (what the current problem is,
> what we need to investigate, any blockers)? Do we have a bugzilla ticket
> where we track this? If not, we could just update
> https://bugs.dpdk.org/show_bug.cgi?id=408.
>
>
>
> Thanks,
>
> Juraj
>
>
>
> *From:* David Liu <dliu@iol.unh.edu>
> *Sent:* Thursday, April 15, 2021 7:08 PM
> *To:* Ajit Khaparde <ajit.khaparde@broadcom.com>
> *Cc:* Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <
> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> *Subject:* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm
> hardware
>
>
>
> Hi Ajit,
>
>
>
> You are free to hop on to the system now.
> But do keep in mind that the dut server is running compile testing and the
> other is free.
>
> So let me know if you need that to be turned off any time.
>
>
>
> For v2.88, I do have an issue where I have to replace system scapy with
> the Trex scapy.
>
>
>
> Best,
>
> David
>
>
>
> On Fri, Apr 9, 2021 at 7:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
> Thanks David.
>
> Next time when I get on the system I will try the 2.87 version of trex.
>
> Apparently my team is having trouble trying with 2.88 version as well.
>
> Let me know when the setup is ready.
>
>
>
> Thanks
>
> Ajit
>
>
>
> On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
>
>
> We are working on an issue we have in the Arm server. When we are done, I
> will let you know so you can take get into the server.
>
> Best,
>
> David
>
>
>
> On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
> Thanks David.
> While I don't have the ARM server, I am trying to use the new version
> of TREX and see if I run into any issues.
> I will get back to you once I complete the test.
>
> Thanks
> Ajit
>
>
> On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > I had rerun the test with the recent DTS and Trex (v2.88)
> >
> > With this, I am getting the mapping error we see during the debugging
> session.
> >
> >
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg
> /opt/trex/v2.88/trex_cfg_bnxt.yaml
> > Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg
> /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy
> server......... Scapy server is started
> > Trying to bind to vfio-pci ...
> > /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0
> 0000:5f:00.1
> > The ports are bound/configured.
> > Starting  TRex v2.88 please wait  ...
> > ERROR in DPDK map
> > Could not find requested interface 5f:00.0
> > Killing Scapy server... Scapy server is killed]
> >                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64
> -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> >                         tester: pkill -f _t-rex-64
> >
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > Best,
> > David
> >
> >
> >
> > On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>
> >> Hi David,
> >> Yes. The changes are in the tree.
> >> Let's try to rerun the test and see what happens.
> >>
> >> Thanks
> >> Ajit
> >>
> >>
> >> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>
> >>> Hi Ajit,
> >>>
> >>> I think the patch should be in the main tree by now.
> >>>
> >>> Can you check if that is true?
> >>>
> >>> Just checking, is this patch related to the Trex mapping issue?
> >>>
> >>> Best,
> >>> David
> >>>
> >>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>>
> >>>> Hi David,
> >>>> The patch was in dpdk-next-net.
> >>>> Let me check if it was applied in the main tree.
> >>>>
> >>>> Thanks
> >>>> Ajit
> >>>>
> >>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >
> >>>> > Hi Ajit,
> >>>> >
> >>>> > Here to check in with you to see if the patch gets submitted.
> >>>> >
> >>>> > And for some confirmation, is this fix possibly be a solution to
> the current issue?
> >>>> >
> >>>> > Best,
> >>>> > David
> >>>> >
> >>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >>
> >>>> >> David,
> >>>> >> While testing with TREX, we have one identified one fix.
> >>>> >> We will try to submit it to the mailing list tomorrow.
> >>>> >> Once it gets merged, I will get back to you for the setup.
> >>>> >>
> >>>> >> Thanks
> >>>> >> Ajit
> >>>> >>
> >>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >>>> >> >
> >>>> >> > Hi Ajit,
> >>>> >> >
> >>>> >> > Thank you.
> >>>> >> >
> >>>> >> > Please let me know when you need to take over the machine again.
> >>>> >> >
> >>>> >> > Best,
> >>>> >> > David
> >>>> >> >
> >>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >>
> >>>> >> >> David,
> >>>> >> >> The tool has issues when I load the kernel module.
> >>>> >> >> I am looking for another alternative.
> >>>> >> >> Please go ahead and take back the setup.
> >>>> >> >>
> >>>> >> >> Thanks
> >>>> >> >> Ajit
> >>>> >> >>
> >>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> >
> >>>> >> >> > Hi David,
> >>>> >> >> > I am about to login into the setup and try something.
> >>>> >> >> > I will let you know once I am done and then you can take
> control.
> >>>> >> >> >
> >>>> >> >> > Thanks
> >>>> >> >> > Ajit
> >>>> >> >> >
> >>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >>>> >> >> > >
> >>>> >> >> > > Hi Ajit,
> >>>> >> >> > >
> >>>> >> >> > > Thank for the update.
> >>>> >> >> > >
> >>>> >> >> > > May I know if I can enable testing or would you like more
> time to look into this?
> >>>> >> >> > >
> >>>> >> >> > > Best,
> >>>> >> >> > > David
> >>>> >> >> > >
> >>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>
> >>>> >> >> > >> David,
> >>>> >> >> > >> I did spend some time on this. For now I can't tell what
> is wrong.
> >>>> >> >> > >> But I want to update the firmware on the card to take
> another shot at it.
> >>>> >> >> > >> Since this is an ARM based server, I will have to find a
> way to update the firmware.
> >>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking
> around.
> >>>> >> >> > >> I will update you soon.
> >>>> >> >> > >>
> >>>> >> >> > >> Thanks
> >>>> >> >> > >> Ajit
> >>>> >> >> > >>
> >>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>
> >>>> >> >> > >>> Hi David,
> >>>> >> >> > >>> That's a good idea. That way you don't have to watch out
> for my email requesting to stop the tests.
> >>>> >> >> > >>>
> >>>> >> >> > >>> Thanks
> >>>> >> >> > >>> Ajit
> >>>> >> >> > >>>
> >>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Hi Ajit, Juraj,
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> For simplicity, we will keep the functional and
> performance testing disable during the weekend.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> In this way, you will have no testing interfere with you
> during troubleshooting.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Please let me know if you have any questions.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Best,
> >>>> >> >> > >>>> David
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <
> juraj.linkes@pantheon.tech> wrote:
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi David, Ajit,
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Juraj
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
> >>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> >>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen
> Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
> Broadcom NIC in Arm hardware
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi, Ajit
> >>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Best,
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> David
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi David,
> >>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
> >>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Thanks
> >>>> >> >> > >>>>> Ajit
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > Hi Ajit and Juraj,
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM.
> But feel free to adjust the time or let me know if there is a better time
> that works for you.
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > For me, I can do:
> >>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
> >>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
> >>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > Best,
> >>>> >> >> > >>>>> > David
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> David,
> >>>> >> >> > >>>>> >> I was busy with a customer issue and could not get
> to my emails.
> >>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues
> needing urgent attention.
> >>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> Juraj,
> >>>> >> >> > >>>>> >> Sure. We will add you.
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> Thanks
> >>>> >> >> > >>>>> >> Ajit
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <
> juraj.linkes@pantheon.tech> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit, David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If you're setting up a call, could you please send
> me an invite? I'd like to get some insight into the issues on Arm's behalf.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Juraj
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David
> Liu
> >>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>;
> ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
> Broadcom NIC in Arm hardware
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Sorry for this issue.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If possible I would like to set up a time where we
> can do this in a call to help get this solved faster.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Please let me know what would be a good time for
> you.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > This should work now.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me know if you still have issues.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't
> work.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-gigabyte.  This incident will be reported.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ----
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > sudo su
> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-dell.  This incident will be reported.
> >>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-dell.  This incident will be reported.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into
> another machine as a normal user which will prompt you for the password.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the
> command line since you should have root access.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me try this method first.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If it does not work, we could try with the public
> ssh keys.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I will let you know.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Another solution would be to let me add your
> public ssh key into both machines.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me know if this helps.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks Owen.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get
> the following error.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Traceback (most recent call last):
> >>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line
> 60, in _connect_host
> >>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> >>>> >> >> > >>>>> >> >   File
> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
> login
> >>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
> >>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> ^^^^^^^^^^^^^^
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
> ohilyard@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hello Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > There was a misconfiguration in our identity
> management system. The issue should now be resolved.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Owen Hilyard
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde'
> via dpdklab <dpdklab@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > But the same credentials are not allowing me to
> login onto the servers you provided.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde
> arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need
> anything from me.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I had some trouble with my system over the past
> week.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping
> that my system will be restored early this week
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I will login into the setup after that.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi David,
> >>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
> >>>> >> >> > >>>>> >> > Thanks for the refresh mail.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Hi Ajit,
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login
> to the machines?
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Just some notes, we are currently running
> compile and some performance/functional testing.
> >>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so
> please let me know if you want me to stop these tests for you to debug.
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Step I run:
> >>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is
> to switch the dts configuration to Broadcom
> >>>> >> >> > >>>>> >> > > ./dts -s
>                    #To start the dts
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Informatio:
> >>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
> >>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Best,
> >>>> >> >> > >>>>> >> > > David
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>
> >>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I
> run Broadcom nic.
> >>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts
> -s
> >>>> >> >> > >>>>> >> > >>
> >>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same
> VPN Username and Password.
> >>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Informatio:
> >>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
> >>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Thanks,
> >>>> >> >> > >>>>> >> > >>> David
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> Hi David,
> >>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could
> login into the setup and
> >>>> >> >> > >>>>> >> > >>>> see what is happening?
> >>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in
> your lab -
> >>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and
> io.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems
> you are trying to
> >>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> Thanks
> >>>> >> >> > >>>>> >> > >>>> Ajit
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything
> information from us.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex
> and dts are on the same intel tester machine.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Best,
> >>>> >> >> > >>>>> >> > >>>> > David
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean
> the ports that are using for Trex. If so they are UP before I start the
> Trex.
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> Thank,
> >>>> >> >> > >>>>> >> > >>>> >> David
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit
> Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
> >>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the
> trex? I am trying to see if the port failed to successfully come up
> >>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link.
> Can you check and tell?
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> Thanks
> >>>> >> >> > >>>>> >> > >>>> >>> Ajit
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu
> <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not
> the Arm.
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
> >>>> >> >> > >>>>> >> > >>>> >>>> David
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit
> Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
> >>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
> >>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am
> asking because
> >>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create
> a baseline and then
> >>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
> >>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David
> Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is
> installed on an Arm machine.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I
> try to run nic_single_core_perf on the NIC.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86
> and dts from http://git.dpdk.org/tools/dts/
> >>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when
> testing.
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to
> run the test case inside the nic_single_core_perf:
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running
> at parameters: framesize: 64, rxd/txd: 512
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test
> Case test_perf_nic_single_core Result ERROR: Traceback (most recent call
> last):
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 196, in
> test_perf_nic_single_core
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>  self._prepare_transmission(stream_ids=stream_ids)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>  self._conn.reset(ports=self._ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
> clear_global, clear_xstats)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
> response from tcp://172.18.0.40:4501
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless
> mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with
> ./trex-console
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn
> off Trex.
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 -
> [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task
> 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout
> is 1 seconds
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a
> ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a
> rte_delay_us_block + 106
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
> CCoreEthIF::flush_tx_queue() + 31
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9
> TrexDpCore::start() + 89
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
> CGlobalTRex::run_in_core(unsigned char) + 487
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad
> ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa
> eal_thread_loop + 426
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> '
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working,
> this should not turn off trex but please correct me if I am wrong.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of
> how I can approach this issue?
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
> >>>> >> >> > >>>>> >> > >>>> >>>>>> David
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>
>

[-- Attachment #2: Type: text/html, Size: 84485 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-22  3:55                                                                                         ` David Liu
@ 2021-04-22  6:50                                                                                           ` Juraj Linkeš
  2021-04-27 14:09                                                                                             ` David Liu
  0 siblings, 1 reply; 50+ messages in thread
From: Juraj Linkeš @ 2021-04-22  6:50 UTC (permalink / raw)
  To: David Liu; +Cc: Ajit Khaparde, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 39598 bytes --]

Thanks, David.

From: David Liu <dliu@iol.unh.edu>
Sent: Thursday, April 22, 2021 5:55 AM
To: Juraj Linkeš <juraj.linkes@pantheon.tech>
Cc: Ajit Khaparde <ajit.khaparde@broadcom.com>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware

Hi Juraj,

The current issue is when running DTS with the newest release of Tex v2.89. It will report an error:

===========================================================================================================================
pktgen: ssh root@172.18.0.40<mailto:root@172.18.0.40>
                        pktgen: cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
Exception happened in [cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
The ports are bound/configured.
Starting  TRex v2.89 please wait  ...
ERROR in DPDK map
Could not find requested interface 5f:00.0
Killing Scapy server... Scapy server is killed]
                           dts: TIMEOUT on cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
                        tester: pkill -f _t-rex-64
===========================================================================================================================

We do not have a Bugzilla ticket for this issue. So I will update the 480 ticket to track this issue.

Thank,
David

On Wed, Apr 21, 2021 at 3:39 AM Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>> wrote:
Hi Ajit, David,

What is the current status of this issue (what the current problem is, what we need to investigate, any blockers)? Do we have a bugzilla ticket where we track this? If not, we could just update https://bugs.dpdk.org/show_bug.cgi?id=408.

Thanks,
Juraj

From: David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>>
Sent: Thursday, April 15, 2021 7:08 PM
To: Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>>
Cc: Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>>; Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>>; ci@dpdk.org<mailto:ci@dpdk.org>; dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>>
Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware

Hi Ajit,

You are free to hop on to the system now.
But do keep in mind that the dut server is running compile testing and the other is free.
So let me know if you need that to be turned off any time.

For v2.88, I do have an issue where I have to replace system scapy with the Trex scapy.

Best,
David

On Fri, Apr 9, 2021 at 7:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
Thanks David.
Next time when I get on the system I will try the 2.87 version of trex.
Apparently my team is having trouble trying with 2.88 version as well.
Let me know when the setup is ready.

Thanks
Ajit

On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
Hi Ajit,

We are working on an issue we have in the Arm server. When we are done, I will let you know so you can take get into the server.
Best,
David

On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
Thanks David.
While I don't have the ARM server, I am trying to use the new version
of TREX and see if I run into any issues.
I will get back to you once I complete the test.

Thanks
Ajit


On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>
> Hi Ajit,
>
> I had rerun the test with the recent DTS and Trex (v2.88)
>
> With this, I am getting the mapping error we see during the debugging session.
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
> Trying to bind to vfio-pci ...
> /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0 0000:5f:00.1
> The ports are bound/configured.
> Starting  TRex v2.88 please wait  ...
> ERROR in DPDK map
> Could not find requested interface 5f:00.0
> Killing Scapy server... Scapy server is killed]
>                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>                         tester: pkill -f _t-rex-64
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Best,
> David
>
>
>
> On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>
>> Hi David,
>> Yes. The changes are in the tree.
>> Let's try to rerun the test and see what happens.
>>
>> Thanks
>> Ajit
>>
>>
>> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>
>>> Hi Ajit,
>>>
>>> I think the patch should be in the main tree by now.
>>>
>>> Can you check if that is true?
>>>
>>> Just checking, is this patch related to the Trex mapping issue?
>>>
>>> Best,
>>> David
>>>
>>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>>
>>>> Hi David,
>>>> The patch was in dpdk-next-net.
>>>> Let me check if it was applied in the main tree.
>>>>
>>>> Thanks
>>>> Ajit
>>>>
>>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >
>>>> > Hi Ajit,
>>>> >
>>>> > Here to check in with you to see if the patch gets submitted.
>>>> >
>>>> > And for some confirmation, is this fix possibly be a solution to the current issue?
>>>> >
>>>> > Best,
>>>> > David
>>>> >
>>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >>
>>>> >> David,
>>>> >> While testing with TREX, we have one identified one fix.
>>>> >> We will try to submit it to the mailing list tomorrow.
>>>> >> Once it gets merged, I will get back to you for the setup.
>>>> >>
>>>> >> Thanks
>>>> >> Ajit
>>>> >>
>>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >
>>>> >> > Hi Ajit,
>>>> >> >
>>>> >> > Thank you.
>>>> >> >
>>>> >> > Please let me know when you need to take over the machine again.
>>>> >> >
>>>> >> > Best,
>>>> >> > David
>>>> >> >
>>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >>
>>>> >> >> David,
>>>> >> >> The tool has issues when I load the kernel module.
>>>> >> >> I am looking for another alternative.
>>>> >> >> Please go ahead and take back the setup.
>>>> >> >>
>>>> >> >> Thanks
>>>> >> >> Ajit
>>>> >> >>
>>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> >
>>>> >> >> > Hi David,
>>>> >> >> > I am about to login into the setup and try something.
>>>> >> >> > I will let you know once I am done and then you can take control.
>>>> >> >> >
>>>> >> >> > Thanks
>>>> >> >> > Ajit
>>>> >> >> >
>>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >
>>>> >> >> > > Hi Ajit,
>>>> >> >> > >
>>>> >> >> > > Thank for the update.
>>>> >> >> > >
>>>> >> >> > > May I know if I can enable testing or would you like more time to look into this?
>>>> >> >> > >
>>>> >> >> > > Best,
>>>> >> >> > > David
>>>> >> >> > >
>>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>
>>>> >> >> > >> David,
>>>> >> >> > >> I did spend some time on this. For now I can't tell what is wrong.
>>>> >> >> > >> But I want to update the firmware on the card to take another shot at it.
>>>> >> >> > >> Since this is an ARM based server, I will have to find a way to update the firmware.
>>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
>>>> >> >> > >> I will update you soon.
>>>> >> >> > >>
>>>> >> >> > >> Thanks
>>>> >> >> > >> Ajit
>>>> >> >> > >>
>>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>
>>>> >> >> > >>> Hi David,
>>>> >> >> > >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
>>>> >> >> > >>>
>>>> >> >> > >>> Thanks
>>>> >> >> > >>> Ajit
>>>> >> >> > >>>
>>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>
>>>> >> >> > >>>> Hi Ajit, Juraj,
>>>> >> >> > >>>>
>>>> >> >> > >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
>>>> >> >> > >>>>
>>>> >> >> > >>>> In this way, you will have no testing interfere with you during troubleshooting.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Please let me know if you have any questions.
>>>> >> >> > >>>>
>>>> >> >> > >>>> Best,
>>>> >> >> > >>>> David
>>>> >> >> > >>>>
>>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>> wrote:
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi David, Ajit,
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Juraj
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>>
>>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>>
>>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>>; Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>>; ci@dpdk.org<mailto:ci@dpdk.org>; dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>>
>>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi, Ajit
>>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Best,
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> David
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Hi David,
>>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> Thanks
>>>> >> >> > >>>>> Ajit
>>>> >> >> > >>>>>
>>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > Hi Ajit and Juraj,
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > For me, I can do:
>>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > Best,
>>>> >> >> > >>>>> > David
>>>> >> >> > >>>>> >
>>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> David,
>>>> >> >> > >>>>> >> I was busy with a customer issue and could not get to my emails.
>>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
>>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> Juraj,
>>>> >> >> > >>>>> >> Sure. We will add you.
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> Thanks
>>>> >> >> > >>>>> >> Ajit
>>>> >> >> > >>>>> >>
>>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech<mailto:juraj.linkes@pantheon.tech>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit, David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Juraj
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org<mailto:ci-bounces@dpdk.org>> On Behalf Of David Liu
>>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>>
>>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>>; ci@dpdk.org<mailto:ci@dpdk.org>; dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>>
>>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Sorry for this issue.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Please let me know what would be a good time for you.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > This should work now.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me know if you still have issues.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ----
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > sudo su
>>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me try this method first.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > If it does not work, we could try with the public ssh keys.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I will let you know.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Let me know if this helps.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks Owen.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get the following error.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Traceback (most recent call last):
>>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>>>> >> >> > >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu<mailto:ohilyard@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hello Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Owen Hilyard
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu<mailto:dpdklab@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi Ajit,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need anything from me.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Best,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > David,
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I had some trouble with my system over the past week.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > I will login into the setup after that.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Hi David,
>>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
>>>> >> >> > >>>>> >> > Thanks for the refresh mail.
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > Thanks
>>>> >> >> > >>>>> >> > Ajit
>>>> >> >> > >>>>> >> >
>>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Hi Ajit,
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to the machines?
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
>>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Step I run:
>>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>>>> >> >> > >>>>> >> > > ./dts -s                                                           #To start the dts
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Informatio:
>>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
>>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > Best,
>>>> >> >> > >>>>> >> > > David
>>>> >> >> > >>>>> >> > >
>>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>
>>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
>>>> >> >> > >>>>> >> > >>
>>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
>>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Informatio:
>>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu<http://arm-dell.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
>>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu<http://arm-gigabyte.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> Thanks,
>>>> >> >> > >>>>> >> > >>> David
>>>> >> >> > >>>>> >> > >>>
>>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> Hi David,
>>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>>>> >> >> > >>>>> >> > >>>> see what is happening?
>>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
>>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu<http://rhea.dpdklab.iol.unh.edu> and io.dpdklab.iol.unh.edu<http://io.dpdklab.iol.unh.edu>
>>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you are trying to
>>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> Thanks
>>>> >> >> > >>>>> >> > >>>> Ajit
>>>> >> >> > >>>>> >> > >>>>
>>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything information from us.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > Best,
>>>> >> >> > >>>>> >> > >>>> > David
>>>> >> >> > >>>>> >> > >>>> >
>>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> Thank,
>>>> >> >> > >>>>> >> > >>>> >> David
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >>
>>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> Thanks
>>>> >> >> > >>>>> >> > >>>> >>> Ajit
>>>> >> >> > >>>>> >> > >>>> >>>
>>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
>>>> >> >> > >>>>> >> > >>>> >>>> David
>>>> >> >> > >>>>> >> > >>>> >>>>
>>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
>>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
>>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
>>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
>>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
>>>> >> >> > >>>>> >> > >>>> >>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu<mailto:dliu@iol.unh.edu>> wrote:
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501<http://172.18.0.40:4501>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
>>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>>>> >> >> > >>>>> >> > >>>> >>>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>>>> >> >> > >>>>> >> > >>>> >>>>>>> '
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
>>>> >> >> > >>>>> >> > >>>> >>>>>> David
>>>> >> >> > >>>>> >> > >>>> >>>>>>
>>>> >> >> > >>>>> >> > >>>> >>>>>>

[-- Attachment #2: Type: text/html, Size: 91794 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-22  6:50                                                                                           ` Juraj Linkeš
@ 2021-04-27 14:09                                                                                             ` David Liu
  2021-04-28  1:19                                                                                               ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: David Liu @ 2021-04-27 14:09 UTC (permalink / raw)
  To: Juraj Linkeš; +Cc: Ajit Khaparde, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 39965 bytes --]

Hi Ajit

Just a quick check in, have you get a chance to jump onto the arm setup?

Let me know if there anything we can do to help.

Best.
David

On Thu, Apr 22, 2021 at 2:50 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:

> Thanks, David.
>
>
>
> *From:* David Liu <dliu@iol.unh.edu>
> *Sent:* Thursday, April 22, 2021 5:55 AM
> *To:* Juraj Linkeš <juraj.linkes@pantheon.tech>
> *Cc:* Ajit Khaparde <ajit.khaparde@broadcom.com>; Owen Hilyard <
> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> *Subject:* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm
> hardware
>
>
>
> Hi Juraj,
>
>
>
> The current issue is when running DTS with the newest release of Tex
> v2.89. It will report an error:
>
>
> ===========================================================================================================================
> pktgen: ssh root@172.18.0.40
>                         pktgen: cd /opt/trex/v2.89;./t-rex-64 -i --cfg
> /opt/trex/v2.88/trex_cfg_bnxt.yaml
> Exception happened in [cd /opt/trex/v2.89;./t-rex-64 -i --cfg
> /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy
> server......... Scapy server is started
> The ports are bound/configured.
>
> Starting  TRex v2.89 please wait  ...
>
> ERROR in DPDK map
> Could not find requested interface 5f:00.0
> Killing Scapy server... Scapy server is killed]
>                            dts: TIMEOUT on cd /opt/trex/v2.89;./t-rex-64
> -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>                         tester: pkill -f _t-rex-64
>
> ===========================================================================================================================
>
>
> We do not have a Bugzilla ticket for this issue. So I will update the 480
> ticket to track this issue.
>
>
>
> Thank,
>
> David
>
>
>
> On Wed, Apr 21, 2021 at 3:39 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
> wrote:
>
> Hi Ajit, David,
>
>
>
> What is the current status of this issue (what the current problem is,
> what we need to investigate, any blockers)? Do we have a bugzilla ticket
> where we track this? If not, we could just update
> https://bugs.dpdk.org/show_bug.cgi?id=408.
>
>
>
> Thanks,
>
> Juraj
>
>
>
> *From:* David Liu <dliu@iol.unh.edu>
> *Sent:* Thursday, April 15, 2021 7:08 PM
> *To:* Ajit Khaparde <ajit.khaparde@broadcom.com>
> *Cc:* Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <
> ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> *Subject:* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm
> hardware
>
>
>
> Hi Ajit,
>
>
>
> You are free to hop on to the system now.
> But do keep in mind that the dut server is running compile testing and the
> other is free.
>
> So let me know if you need that to be turned off any time.
>
>
>
> For v2.88, I do have an issue where I have to replace system scapy with
> the Trex scapy.
>
>
>
> Best,
>
> David
>
>
>
> On Fri, Apr 9, 2021 at 7:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
> Thanks David.
>
> Next time when I get on the system I will try the 2.87 version of trex.
>
> Apparently my team is having trouble trying with 2.88 version as well.
>
> Let me know when the setup is ready.
>
>
>
> Thanks
>
> Ajit
>
>
>
> On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit,
>
>
>
> We are working on an issue we have in the Arm server. When we are done, I
> will let you know so you can take get into the server.
>
> Best,
>
> David
>
>
>
> On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com>
> wrote:
>
> Thanks David.
> While I don't have the ARM server, I am trying to use the new version
> of TREX and see if I run into any issues.
> I will get back to you once I complete the test.
>
> Thanks
> Ajit
>
>
> On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit,
> >
> > I had rerun the test with the recent DTS and Trex (v2.88)
> >
> > With this, I am getting the mapping error we see during the debugging
> session.
> >
> >
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg
> /opt/trex/v2.88/trex_cfg_bnxt.yaml
> > Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg
> /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy
> server......... Scapy server is started
> > Trying to bind to vfio-pci ...
> > /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0
> 0000:5f:00.1
> > The ports are bound/configured.
> > Starting  TRex v2.88 please wait  ...
> > ERROR in DPDK map
> > Could not find requested interface 5f:00.0
> > Killing Scapy server... Scapy server is killed]
> >                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64
> -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> >                         tester: pkill -f _t-rex-64
> >
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >
> > Best,
> > David
> >
> >
> >
> > On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>
> >> Hi David,
> >> Yes. The changes are in the tree.
> >> Let's try to rerun the test and see what happens.
> >>
> >> Thanks
> >> Ajit
> >>
> >>
> >> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>
> >>> Hi Ajit,
> >>>
> >>> I think the patch should be in the main tree by now.
> >>>
> >>> Can you check if that is true?
> >>>
> >>> Just checking, is this patch related to the Trex mapping issue?
> >>>
> >>> Best,
> >>> David
> >>>
> >>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>>
> >>>> Hi David,
> >>>> The patch was in dpdk-next-net.
> >>>> Let me check if it was applied in the main tree.
> >>>>
> >>>> Thanks
> >>>> Ajit
> >>>>
> >>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
> >>>> >
> >>>> > Hi Ajit,
> >>>> >
> >>>> > Here to check in with you to see if the patch gets submitted.
> >>>> >
> >>>> > And for some confirmation, is this fix possibly be a solution to
> the current issue?
> >>>> >
> >>>> > Best,
> >>>> > David
> >>>> >
> >>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >>
> >>>> >> David,
> >>>> >> While testing with TREX, we have one identified one fix.
> >>>> >> We will try to submit it to the mailing list tomorrow.
> >>>> >> Once it gets merged, I will get back to you for the setup.
> >>>> >>
> >>>> >> Thanks
> >>>> >> Ajit
> >>>> >>
> >>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >>>> >> >
> >>>> >> > Hi Ajit,
> >>>> >> >
> >>>> >> > Thank you.
> >>>> >> >
> >>>> >> > Please let me know when you need to take over the machine again.
> >>>> >> >
> >>>> >> > Best,
> >>>> >> > David
> >>>> >> >
> >>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >>
> >>>> >> >> David,
> >>>> >> >> The tool has issues when I load the kernel module.
> >>>> >> >> I am looking for another alternative.
> >>>> >> >> Please go ahead and take back the setup.
> >>>> >> >>
> >>>> >> >> Thanks
> >>>> >> >> Ajit
> >>>> >> >>
> >>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> >
> >>>> >> >> > Hi David,
> >>>> >> >> > I am about to login into the setup and try something.
> >>>> >> >> > I will let you know once I am done and then you can take
> control.
> >>>> >> >> >
> >>>> >> >> > Thanks
> >>>> >> >> > Ajit
> >>>> >> >> >
> >>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu>
> wrote:
> >>>> >> >> > >
> >>>> >> >> > > Hi Ajit,
> >>>> >> >> > >
> >>>> >> >> > > Thank for the update.
> >>>> >> >> > >
> >>>> >> >> > > May I know if I can enable testing or would you like more
> time to look into this?
> >>>> >> >> > >
> >>>> >> >> > > Best,
> >>>> >> >> > > David
> >>>> >> >> > >
> >>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>
> >>>> >> >> > >> David,
> >>>> >> >> > >> I did spend some time on this. For now I can't tell what
> is wrong.
> >>>> >> >> > >> But I want to update the firmware on the card to take
> another shot at it.
> >>>> >> >> > >> Since this is an ARM based server, I will have to find a
> way to update the firmware.
> >>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking
> around.
> >>>> >> >> > >> I will update you soon.
> >>>> >> >> > >>
> >>>> >> >> > >> Thanks
> >>>> >> >> > >> Ajit
> >>>> >> >> > >>
> >>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>
> >>>> >> >> > >>> Hi David,
> >>>> >> >> > >>> That's a good idea. That way you don't have to watch out
> for my email requesting to stop the tests.
> >>>> >> >> > >>>
> >>>> >> >> > >>> Thanks
> >>>> >> >> > >>> Ajit
> >>>> >> >> > >>>
> >>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Hi Ajit, Juraj,
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> For simplicity, we will keep the functional and
> performance testing disable during the weekend.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> In this way, you will have no testing interfere with you
> during troubleshooting.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Please let me know if you have any questions.
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> Best,
> >>>> >> >> > >>>> David
> >>>> >> >> > >>>>
> >>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <
> juraj.linkes@pantheon.tech> wrote:
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi David, Ajit,
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Juraj
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
> >>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> >>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen
> Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
> Broadcom NIC in Arm hardware
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi, Ajit
> >>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Best,
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> David
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Hi David,
> >>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
> >>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> Thanks
> >>>> >> >> > >>>>> Ajit
> >>>> >> >> > >>>>>
> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > Hi Ajit and Juraj,
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM.
> But feel free to adjust the time or let me know if there is a better time
> that works for you.
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > For me, I can do:
> >>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
> >>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
> >>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > Best,
> >>>> >> >> > >>>>> > David
> >>>> >> >> > >>>>> >
> >>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> David,
> >>>> >> >> > >>>>> >> I was busy with a customer issue and could not get
> to my emails.
> >>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues
> needing urgent attention.
> >>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> Juraj,
> >>>> >> >> > >>>>> >> Sure. We will add you.
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> Thanks
> >>>> >> >> > >>>>> >> Ajit
> >>>> >> >> > >>>>> >>
> >>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <
> juraj.linkes@pantheon.tech> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit, David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If you're setting up a call, could you please send
> me an invite? I'd like to get some insight into the issues on Arm's behalf.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Juraj
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David
> Liu
> >>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>;
> ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G
> Broadcom NIC in Arm hardware
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Sorry for this issue.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If possible I would like to set up a time where we
> can do this in a call to help get this solved faster.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Please let me know what would be a good time for
> you.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > This should work now.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me know if you still have issues.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't
> work.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-gigabyte.  This incident will be reported.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ----
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > sudo su
> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-dell.  This incident will be reported.
> >>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on
> arm-dell.  This incident will be reported.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into
> another machine as a normal user which will prompt you for the password.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the
> command line since you should have root access.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me try this method first.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > If it does not work, we could try with the public
> ssh keys.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I will let you know.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Another solution would be to let me add your
> public ssh key into both machines.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Let me know if this helps.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks Owen.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get
> the following error.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Traceback (most recent call last):
> >>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line
> 60, in _connect_host
> >>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]',
> password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for
> .+:)')
> >>>> >> >> > >>>>> >> >   File
> "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in
> login
> >>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
> >>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> ^^^^^^^^^^^^^^
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <
> ohilyard@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hello Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > There was a misconfiguration in our identity
> management system. The issue should now be resolved.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Owen Hilyard
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde'
> via dpdklab <dpdklab@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > But the same credentials are not allowing me to
> login onto the servers you provided.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde
> arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi Ajit,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need
> anything from me.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Best,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > David,
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I had some trouble with my system over the past
> week.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping
> that my system will be restored early this week
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > I will login into the setup after that.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Hi David,
> >>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
> >>>> >> >> > >>>>> >> > Thanks for the refresh mail.
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > Thanks
> >>>> >> >> > >>>>> >> > Ajit
> >>>> >> >> > >>>>> >> >
> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Hi Ajit,
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login
> to the machines?
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Just some notes, we are currently running
> compile and some performance/functional testing.
> >>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so
> please let me know if you want me to stop these tests for you to debug.
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Step I run:
> >>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is
> to switch the dts configuration to Broadcom
> >>>> >> >> > >>>>> >> > > ./dts -s
>                    #To start the dts
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Informatio:
> >>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
> >>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > Best,
> >>>> >> >> > >>>>> >> > > David
> >>>> >> >> > >>>>> >> > >
> >>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>
> >>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I
> run Broadcom nic.
> >>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts
> -s
> >>>> >> >> > >>>>> >> > >>
> >>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same
> VPN Username and Password.
> >>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Informatio:
> >>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
> >>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> Thanks,
> >>>> >> >> > >>>>> >> > >>> David
> >>>> >> >> > >>>>> >> > >>>
> >>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <
> ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> Hi David,
> >>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could
> login into the setup and
> >>>> >> >> > >>>>> >> > >>>> see what is happening?
> >>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in
> your lab -
> >>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and
> io.dpdklab.iol.unh.edu
> >>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems
> you are trying to
> >>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> Thanks
> >>>> >> >> > >>>>> >> > >>>> Ajit
> >>>> >> >> > >>>>> >> > >>>>
> >>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything
> information from us.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex
> and dts are on the same intel tester machine.
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > Best,
> >>>> >> >> > >>>>> >> > >>>> > David
> >>>> >> >> > >>>>> >> > >>>> >
> >>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <
> dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean
> the ports that are using for Trex. If so they are UP before I start the
> Trex.
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> Thank,
> >>>> >> >> > >>>>> >> > >>>> >> David
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >>
> >>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit
> Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
> >>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the
> trex? I am trying to see if the port failed to successfully come up
> >>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link.
> Can you check and tell?
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> Thanks
> >>>> >> >> > >>>>> >> > >>>> >>> Ajit
> >>>> >> >> > >>>>> >> > >>>> >>>
> >>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu
> <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not
> the Arm.
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
> >>>> >> >> > >>>>> >> > >>>> >>>> David
> >>>> >> >> > >>>>> >> > >>>> >>>>
> >>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit
> Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
> >>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
> >>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am
> asking because
> >>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create
> a baseline and then
> >>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
> >>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David
> Liu <dliu@iol.unh.edu> wrote:
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is
> installed on an Arm machine.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I
> try to run nic_single_core_perf on the NIC.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86
> and dts from http://git.dpdk.org/tools/dts/
> >>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when
> testing.
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to
> run the test case inside the nic_single_core_perf:
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running
> at parameters: framesize: 64, rxd/txd: 512
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41:
> x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4
> --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1
> -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test
> Case test_perf_nic_single_core Result ERROR: Traceback (most recent call
> last):
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 196, in
> test_perf_nic_single_core
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received =
> self.tester.pktgen.measure_throughput(stream_ids=streams,
> options=traffic_opt)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>  self._prepare_transmission(stream_ids=stream_ids)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>  self._conn.reset(ports=self._ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 339, in reset
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py",
> line 51, in wrap2
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py",
> line 1492, in clear_stats
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports,
> clear_global, clear_xstats)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File
> "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py",
> line 2876, in _clear_stats_common
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server
> response from tcp://172.18.0.40:4501
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless
> mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with
> ./trex-console
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn
> off Trex.
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 -
> [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task
> 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout
> is 1 seconds
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a
> ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a
> rte_delay_us_block + 106
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874
> CCoreEthIF::send_burst(CCorePerPort*, unsigned short,
> CVirtualIFPerSideStats*) + 3220
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57
> CCoreEthIF::flush_tx_queue() + 31
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9
> CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec
> CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&)
> + 92
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98
> CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*,
> CFlowGenListPerThread*, bool) + 184
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int
> CNodeGenerator::flush_file_realtime<23, false>(double, double,
> CFlowGenListPerThread*, double&) + 881
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212
> TrexStatelessDpCore::start_scheduler() + 226
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9
> TrexDpCore::start() + 89
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113
> CFlowGenListPerThread::start(std::__cxx11::basic_string<char,
> std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd
> CGlobalTRex::run_in_core(unsigned char) + 487
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad
> ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa
> eal_thread_loop + 426
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >>>> >> >> > >>>>> >> > >>>> >>>>>>> '
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working,
> this should not turn off trex but please correct me if I am wrong.
> >>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of
> how I can approach this issue?
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
> >>>> >> >> > >>>>> >> > >>>> >>>>>> David
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>
>

[-- Attachment #2: Type: text/html, Size: 87087 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-27 14:09                                                                                             ` David Liu
@ 2021-04-28  1:19                                                                                               ` Ajit Khaparde
  2021-04-28  6:48                                                                                                 ` Ajit Khaparde
  0 siblings, 1 reply; 50+ messages in thread
From: Ajit Khaparde @ 2021-04-28  1:19 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 40654 bytes --]

David,
I have been trying different experiments on the setup.
No luck so far. But I am not done with the experiments yet - which
include trying newer trex as well.
If the systems are used for any CI/CD and I may be interfering with
that, please let me know if you can get them out of the grid.

Thanks
Ajit

On Tue, Apr 27, 2021 at 7:10 AM David Liu <dliu@iol.unh.edu> wrote:
>
> Hi Ajit
>
> Just a quick check in, have you get a chance to jump onto the arm setup?
>
> Let me know if there anything we can do to help.
>
> Best.
> David
>
> On Thu, Apr 22, 2021 at 2:50 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>>
>> Thanks, David.
>>
>>
>>
>> From: David Liu <dliu@iol.unh.edu>
>> Sent: Thursday, April 22, 2021 5:55 AM
>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
>> Cc: Ajit Khaparde <ajit.khaparde@broadcom.com>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>
>>
>>
>> Hi Juraj,
>>
>>
>>
>> The current issue is when running DTS with the newest release of Tex v2.89. It will report an error:
>>
>> ===========================================================================================================================
>> pktgen: ssh root@172.18.0.40
>>                         pktgen: cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>> Exception happened in [cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
>> The ports are bound/configured.
>> Starting  TRex v2.89 please wait  ...
>> ERROR in DPDK map
>> Could not find requested interface 5f:00.0
>> Killing Scapy server... Scapy server is killed]
>>                            dts: TIMEOUT on cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>>                         tester: pkill -f _t-rex-64
>> ===========================================================================================================================
>>
>>
>> We do not have a Bugzilla ticket for this issue. So I will update the 480 ticket to track this issue.
>>
>>
>>
>> Thank,
>>
>> David
>>
>>
>>
>> On Wed, Apr 21, 2021 at 3:39 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>>
>> Hi Ajit, David,
>>
>>
>>
>> What is the current status of this issue (what the current problem is, what we need to investigate, any blockers)? Do we have a bugzilla ticket where we track this? If not, we could just update https://bugs.dpdk.org/show_bug.cgi?id=408.
>>
>>
>>
>> Thanks,
>>
>> Juraj
>>
>>
>>
>> From: David Liu <dliu@iol.unh.edu>
>> Sent: Thursday, April 15, 2021 7:08 PM
>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>>
>>
>>
>> Hi Ajit,
>>
>>
>>
>> You are free to hop on to the system now.
>> But do keep in mind that the dut server is running compile testing and the other is free.
>>
>> So let me know if you need that to be turned off any time.
>>
>>
>>
>> For v2.88, I do have an issue where I have to replace system scapy with the Trex scapy.
>>
>>
>>
>> Best,
>>
>> David
>>
>>
>>
>> On Fri, Apr 9, 2021 at 7:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>
>> Thanks David.
>>
>> Next time when I get on the system I will try the 2.87 version of trex.
>>
>> Apparently my team is having trouble trying with 2.88 version as well.
>>
>> Let me know when the setup is ready.
>>
>>
>>
>> Thanks
>>
>> Ajit
>>
>>
>>
>> On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu> wrote:
>>
>> Hi Ajit,
>>
>>
>>
>> We are working on an issue we have in the Arm server. When we are done, I will let you know so you can take get into the server.
>>
>> Best,
>>
>> David
>>
>>
>>
>> On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>>
>> Thanks David.
>> While I don't have the ARM server, I am trying to use the new version
>> of TREX and see if I run into any issues.
>> I will get back to you once I complete the test.
>>
>> Thanks
>> Ajit
>>
>>
>> On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
>> >
>> > Hi Ajit,
>> >
>> > I had rerun the test with the recent DTS and Trex (v2.88)
>> >
>> > With this, I am getting the mapping error we see during the debugging session.
>> >
>> > -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> >                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>> > Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
>> > Trying to bind to vfio-pci ...
>> > /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0 0000:5f:00.1
>> > The ports are bound/configured.
>> > Starting  TRex v2.88 please wait  ...
>> > ERROR in DPDK map
>> > Could not find requested interface 5f:00.0
>> > Killing Scapy server... Scapy server is killed]
>> >                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
>> >                         tester: pkill -f _t-rex-64
>> > -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> >
>> > Best,
>> > David
>> >
>> >
>> >
>> > On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>
>> >> Hi David,
>> >> Yes. The changes are in the tree.
>> >> Let's try to rerun the test and see what happens.
>> >>
>> >> Thanks
>> >> Ajit
>> >>
>> >>
>> >> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>
>> >>> Hi Ajit,
>> >>>
>> >>> I think the patch should be in the main tree by now.
>> >>>
>> >>> Can you check if that is true?
>> >>>
>> >>> Just checking, is this patch related to the Trex mapping issue?
>> >>>
>> >>> Best,
>> >>> David
>> >>>
>> >>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>>
>> >>>> Hi David,
>> >>>> The patch was in dpdk-next-net.
>> >>>> Let me check if it was applied in the main tree.
>> >>>>
>> >>>> Thanks
>> >>>> Ajit
>> >>>>
>> >>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >
>> >>>> > Hi Ajit,
>> >>>> >
>> >>>> > Here to check in with you to see if the patch gets submitted.
>> >>>> >
>> >>>> > And for some confirmation, is this fix possibly be a solution to the current issue?
>> >>>> >
>> >>>> > Best,
>> >>>> > David
>> >>>> >
>> >>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >>
>> >>>> >> David,
>> >>>> >> While testing with TREX, we have one identified one fix.
>> >>>> >> We will try to submit it to the mailing list tomorrow.
>> >>>> >> Once it gets merged, I will get back to you for the setup.
>> >>>> >>
>> >>>> >> Thanks
>> >>>> >> Ajit
>> >>>> >>
>> >>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >
>> >>>> >> > Hi Ajit,
>> >>>> >> >
>> >>>> >> > Thank you.
>> >>>> >> >
>> >>>> >> > Please let me know when you need to take over the machine again.
>> >>>> >> >
>> >>>> >> > Best,
>> >>>> >> > David
>> >>>> >> >
>> >>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >>
>> >>>> >> >> David,
>> >>>> >> >> The tool has issues when I load the kernel module.
>> >>>> >> >> I am looking for another alternative.
>> >>>> >> >> Please go ahead and take back the setup.
>> >>>> >> >>
>> >>>> >> >> Thanks
>> >>>> >> >> Ajit
>> >>>> >> >>
>> >>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> >
>> >>>> >> >> > Hi David,
>> >>>> >> >> > I am about to login into the setup and try something.
>> >>>> >> >> > I will let you know once I am done and then you can take control.
>> >>>> >> >> >
>> >>>> >> >> > Thanks
>> >>>> >> >> > Ajit
>> >>>> >> >> >
>> >>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >
>> >>>> >> >> > > Hi Ajit,
>> >>>> >> >> > >
>> >>>> >> >> > > Thank for the update.
>> >>>> >> >> > >
>> >>>> >> >> > > May I know if I can enable testing or would you like more time to look into this?
>> >>>> >> >> > >
>> >>>> >> >> > > Best,
>> >>>> >> >> > > David
>> >>>> >> >> > >
>> >>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>
>> >>>> >> >> > >> David,
>> >>>> >> >> > >> I did spend some time on this. For now I can't tell what is wrong.
>> >>>> >> >> > >> But I want to update the firmware on the card to take another shot at it.
>> >>>> >> >> > >> Since this is an ARM based server, I will have to find a way to update the firmware.
>> >>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
>> >>>> >> >> > >> I will update you soon.
>> >>>> >> >> > >>
>> >>>> >> >> > >> Thanks
>> >>>> >> >> > >> Ajit
>> >>>> >> >> > >>
>> >>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>
>> >>>> >> >> > >>> Hi David,
>> >>>> >> >> > >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
>> >>>> >> >> > >>>
>> >>>> >> >> > >>> Thanks
>> >>>> >> >> > >>> Ajit
>> >>>> >> >> > >>>
>> >>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> Hi Ajit, Juraj,
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> In this way, you will have no testing interfere with you during troubleshooting.
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> Please let me know if you have any questions.
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> Best,
>> >>>> >> >> > >>>> David
>> >>>> >> >> > >>>>
>> >>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Hi David, Ajit,
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Juraj
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
>> >>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
>> >>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> >>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Hi, Ajit
>> >>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Best,
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> David
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Hi David,
>> >>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
>> >>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> Thanks
>> >>>> >> >> > >>>>> Ajit
>> >>>> >> >> > >>>>>
>> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > Hi Ajit and Juraj,
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > For me, I can do:
>> >>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
>> >>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
>> >>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > Best,
>> >>>> >> >> > >>>>> > David
>> >>>> >> >> > >>>>> >
>> >>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >>
>> >>>> >> >> > >>>>> >> David,
>> >>>> >> >> > >>>>> >> I was busy with a customer issue and could not get to my emails.
>> >>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
>> >>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
>> >>>> >> >> > >>>>> >>
>> >>>> >> >> > >>>>> >> Juraj,
>> >>>> >> >> > >>>>> >> Sure. We will add you.
>> >>>> >> >> > >>>>> >>
>> >>>> >> >> > >>>>> >> Thanks
>> >>>> >> >> > >>>>> >> Ajit
>> >>>> >> >> > >>>>> >>
>> >>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit, David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Juraj
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
>> >>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
>> >>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> >>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
>> >>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Sorry for this issue.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Please let me know what would be a good time for you.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Best,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > This should work now.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Let me know if you still have issues.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Best,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > ----
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > sudo su
>> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> >>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
>> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
>> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Let me try this method first.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > If it does not work, we could try with the public ssh keys.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I will let you know.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Let me know if this helps.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Best,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks Owen.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get the following error.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Traceback (most recent call last):
>> >>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
>> >>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
>> >>>> >> >> > >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
>> >>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
>> >>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hello Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Owen Hilyard
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi Ajit,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need anything from me.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Best,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > David,
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I had some trouble with my system over the past week.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > I will login into the setup after that.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Hi David,
>> >>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
>> >>>> >> >> > >>>>> >> > Thanks for the refresh mail.
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > Thanks
>> >>>> >> >> > >>>>> >> > Ajit
>> >>>> >> >> > >>>>> >> >
>> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Hi Ajit,
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to the machines?
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
>> >>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Step I run:
>> >>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
>> >>>> >> >> > >>>>> >> > > ./dts -s                                                           #To start the dts
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Informatio:
>> >>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
>> >>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > Best,
>> >>>> >> >> > >>>>> >> > > David
>> >>>> >> >> > >>>>> >> > >
>> >>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>
>> >>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
>> >>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
>> >>>> >> >> > >>>>> >> > >>
>> >>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> Hi Ajit,
>> >>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
>> >>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> Informatio:
>> >>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
>> >>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> Thanks,
>> >>>> >> >> > >>>>> >> > >>> David
>> >>>> >> >> > >>>>> >> > >>>
>> >>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> > >>>>
>> >>>> >> >> > >>>>> >> > >>>> Hi David,
>> >>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
>> >>>> >> >> > >>>>> >> > >>>> see what is happening?
>> >>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
>> >>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
>> >>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you are trying to
>> >>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
>> >>>> >> >> > >>>>> >> > >>>>
>> >>>> >> >> > >>>>> >> > >>>> Thanks
>> >>>> >> >> > >>>>> >> > >>>> Ajit
>> >>>> >> >> > >>>>> >> > >>>>
>> >>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything information from us.
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > Best,
>> >>>> >> >> > >>>>> >> > >>>> > David
>> >>>> >> >> > >>>>> >> > >>>> >
>> >>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>
>> >>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
>> >>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
>> >>>> >> >> > >>>>> >> > >>>> >>
>> >>>> >> >> > >>>>> >> > >>>> >> Thank,
>> >>>> >> >> > >>>>> >> > >>>> >> David
>> >>>> >> >> > >>>>> >> > >>>> >>
>> >>>> >> >> > >>>>> >> > >>>> >>
>> >>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>>
>> >>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
>> >>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
>> >>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
>> >>>> >> >> > >>>>> >> > >>>> >>>
>> >>>> >> >> > >>>>> >> > >>>> >>> Thanks
>> >>>> >> >> > >>>>> >> > >>>> >>> Ajit
>> >>>> >> >> > >>>>> >> > >>>> >>>
>> >>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
>> >>>> >> >> > >>>>> >> > >>>> >>>> David
>> >>>> >> >> > >>>>> >> > >>>> >>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
>> >>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
>> >>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
>> >>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
>> >>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
>> >>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
>> >>>> >> >> > >>>>> >> > >>>> >>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>> '
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
>> >>>> >> >> > >>>>> >> > >>>> >>>>>> David
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>
>> >>>> >> >> > >>>>> >> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

* Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
  2021-04-28  1:19                                                                                               ` Ajit Khaparde
@ 2021-04-28  6:48                                                                                                 ` Ajit Khaparde
  0 siblings, 0 replies; 50+ messages in thread
From: Ajit Khaparde @ 2021-04-28  6:48 UTC (permalink / raw)
  To: David Liu; +Cc: Juraj Linkeš, Owen Hilyard, ci, dpdklab

[-- Attachment #1: Type: text/plain, Size: 42743 bytes --]

I found the issue with the v2.88/v2.89 versions of TREX.
I will submit the patch to fix trex tomorrow.

Now we should be able to use the new version of TREX and see if the
results will change.
I will try to do that tomorrow. Let me know if the system can be made available.

Thanks
Ajit

On Tue, Apr 27, 2021 at 6:19 PM Ajit Khaparde
<ajit.khaparde@broadcom.com> wrote:
>
> David,
> I have been trying different experiments on the setup.
> No luck so far. But I am not done with the experiments yet - which
> include trying newer trex as well.
> If the systems are used for any CI/CD and I may be interfering with
> that, please let me know if you can get them out of the grid.
>
> Thanks
> Ajit
>
> On Tue, Apr 27, 2021 at 7:10 AM David Liu <dliu@iol.unh.edu> wrote:
> >
> > Hi Ajit
> >
> > Just a quick check in, have you get a chance to jump onto the arm setup?
> >
> > Let me know if there anything we can do to help.
> >
> > Best.
> > David
> >
> > On Thu, Apr 22, 2021 at 2:50 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >>
> >> Thanks, David.
> >>
> >>
> >>
> >> From: David Liu <dliu@iol.unh.edu>
> >> Sent: Thursday, April 22, 2021 5:55 AM
> >> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> >> Cc: Ajit Khaparde <ajit.khaparde@broadcom.com>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
> >>
> >>
> >>
> >> Hi Juraj,
> >>
> >>
> >>
> >> The current issue is when running DTS with the newest release of Tex v2.89. It will report an error:
> >>
> >> ===========================================================================================================================
> >> pktgen: ssh root@172.18.0.40
> >>                         pktgen: cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> >> Exception happened in [cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
> >> The ports are bound/configured.
> >> Starting  TRex v2.89 please wait  ...
> >> ERROR in DPDK map
> >> Could not find requested interface 5f:00.0
> >> Killing Scapy server... Scapy server is killed]
> >>                            dts: TIMEOUT on cd /opt/trex/v2.89;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> >>                         tester: pkill -f _t-rex-64
> >> ===========================================================================================================================
> >>
> >>
> >> We do not have a Bugzilla ticket for this issue. So I will update the 480 ticket to track this issue.
> >>
> >>
> >>
> >> Thank,
> >>
> >> David
> >>
> >>
> >>
> >> On Wed, Apr 21, 2021 at 3:39 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >>
> >> Hi Ajit, David,
> >>
> >>
> >>
> >> What is the current status of this issue (what the current problem is, what we need to investigate, any blockers)? Do we have a bugzilla ticket where we track this? If not, we could just update https://bugs.dpdk.org/show_bug.cgi?id=408.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Juraj
> >>
> >>
> >>
> >> From: David Liu <dliu@iol.unh.edu>
> >> Sent: Thursday, April 15, 2021 7:08 PM
> >> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
> >>
> >>
> >>
> >> Hi Ajit,
> >>
> >>
> >>
> >> You are free to hop on to the system now.
> >> But do keep in mind that the dut server is running compile testing and the other is free.
> >>
> >> So let me know if you need that to be turned off any time.
> >>
> >>
> >>
> >> For v2.88, I do have an issue where I have to replace system scapy with the Trex scapy.
> >>
> >>
> >>
> >> Best,
> >>
> >> David
> >>
> >>
> >>
> >> On Fri, Apr 9, 2021 at 7:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>
> >> Thanks David.
> >>
> >> Next time when I get on the system I will try the 2.87 version of trex.
> >>
> >> Apparently my team is having trouble trying with 2.88 version as well.
> >>
> >> Let me know when the setup is ready.
> >>
> >>
> >>
> >> Thanks
> >>
> >> Ajit
> >>
> >>
> >>
> >> On Fri, Apr 9, 2021 at 3:10 PM David Liu <dliu@iol.unh.edu> wrote:
> >>
> >> Hi Ajit,
> >>
> >>
> >>
> >> We are working on an issue we have in the Arm server. When we are done, I will let you know so you can take get into the server.
> >>
> >> Best,
> >>
> >> David
> >>
> >>
> >>
> >> On Wed, Apr 7, 2021 at 4:16 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >>
> >> Thanks David.
> >> While I don't have the ARM server, I am trying to use the new version
> >> of TREX and see if I run into any issues.
> >> I will get back to you once I complete the test.
> >>
> >> Thanks
> >> Ajit
> >>
> >>
> >> On Tue, Apr 6, 2021 at 7:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >
> >> > Hi Ajit,
> >> >
> >> > I had rerun the test with the recent DTS and Trex (v2.88)
> >> >
> >> > With this, I am getting the mapping error we see during the debugging session.
> >> >
> >> > -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >> >                         pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> >> > Exception happened in [cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml] and output is [Starting Scapy server......... Scapy server is started
> >> > Trying to bind to vfio-pci ...
> >> > /usr/bin/python3 dpdk_nic_bind.py --bind=vfio-pci 0000:5f:00.0 0000:5f:00.1
> >> > The ports are bound/configured.
> >> > Starting  TRex v2.88 please wait  ...
> >> > ERROR in DPDK map
> >> > Could not find requested interface 5f:00.0
> >> > Killing Scapy server... Scapy server is killed]
> >> >                            dts: TIMEOUT on cd /opt/trex/v2.88;./t-rex-64 -i --cfg /opt/trex/v2.88/trex_cfg_bnxt.yaml
> >> >                         tester: pkill -f _t-rex-64
> >> > -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >> >
> >> > Best,
> >> > David
> >> >
> >> >
> >> >
> >> > On Mon, Apr 5, 2021 at 10:26 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>
> >> >> Hi David,
> >> >> Yes. The changes are in the tree.
> >> >> Let's try to rerun the test and see what happens.
> >> >>
> >> >> Thanks
> >> >> Ajit
> >> >>
> >> >>
> >> >> On Thu, Apr 1, 2021 at 10:09 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>
> >> >>> Hi Ajit,
> >> >>>
> >> >>> I think the patch should be in the main tree by now.
> >> >>>
> >> >>> Can you check if that is true?
> >> >>>
> >> >>> Just checking, is this patch related to the Trex mapping issue?
> >> >>>
> >> >>> Best,
> >> >>> David
> >> >>>
> >> >>> On Tue, Mar 16, 2021 at 11:43 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>>
> >> >>>> Hi David,
> >> >>>> The patch was in dpdk-next-net.
> >> >>>> Let me check if it was applied in the main tree.
> >> >>>>
> >> >>>> Thanks
> >> >>>> Ajit
> >> >>>>
> >> >>>> On Tue, Mar 16, 2021 at 8:32 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >
> >> >>>> > Hi Ajit,
> >> >>>> >
> >> >>>> > Here to check in with you to see if the patch gets submitted.
> >> >>>> >
> >> >>>> > And for some confirmation, is this fix possibly be a solution to the current issue?
> >> >>>> >
> >> >>>> > Best,
> >> >>>> > David
> >> >>>> >
> >> >>>> > On Thu, Mar 11, 2021 at 12:05 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >>
> >> >>>> >> David,
> >> >>>> >> While testing with TREX, we have one identified one fix.
> >> >>>> >> We will try to submit it to the mailing list tomorrow.
> >> >>>> >> Once it gets merged, I will get back to you for the setup.
> >> >>>> >>
> >> >>>> >> Thanks
> >> >>>> >> Ajit
> >> >>>> >>
> >> >>>> >> On Thu, Mar 11, 2021 at 6:02 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >
> >> >>>> >> > Hi Ajit,
> >> >>>> >> >
> >> >>>> >> > Thank you.
> >> >>>> >> >
> >> >>>> >> > Please let me know when you need to take over the machine again.
> >> >>>> >> >
> >> >>>> >> > Best,
> >> >>>> >> > David
> >> >>>> >> >
> >> >>>> >> > On Tue, Mar 9, 2021 at 2:01 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >>
> >> >>>> >> >> David,
> >> >>>> >> >> The tool has issues when I load the kernel module.
> >> >>>> >> >> I am looking for another alternative.
> >> >>>> >> >> Please go ahead and take back the setup.
> >> >>>> >> >>
> >> >>>> >> >> Thanks
> >> >>>> >> >> Ajit
> >> >>>> >> >>
> >> >>>> >> >> On Tue, Mar 9, 2021 at 8:56 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> >
> >> >>>> >> >> > Hi David,
> >> >>>> >> >> > I am about to login into the setup and try something.
> >> >>>> >> >> > I will let you know once I am done and then you can take control.
> >> >>>> >> >> >
> >> >>>> >> >> > Thanks
> >> >>>> >> >> > Ajit
> >> >>>> >> >> >
> >> >>>> >> >> > On Tue, Mar 9, 2021 at 6:24 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >
> >> >>>> >> >> > > Hi Ajit,
> >> >>>> >> >> > >
> >> >>>> >> >> > > Thank for the update.
> >> >>>> >> >> > >
> >> >>>> >> >> > > May I know if I can enable testing or would you like more time to look into this?
> >> >>>> >> >> > >
> >> >>>> >> >> > > Best,
> >> >>>> >> >> > > David
> >> >>>> >> >> > >
> >> >>>> >> >> > > On Sun, Mar 7, 2021 at 8:07 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>
> >> >>>> >> >> > >> David,
> >> >>>> >> >> > >> I did spend some time on this. For now I can't tell what is wrong.
> >> >>>> >> >> > >> But I want to update the firmware on the card to take another shot at it.
> >> >>>> >> >> > >> Since this is an ARM based server, I will have to find a way to update the firmware.
> >> >>>> >> >> > >> IIRC our tools are mostly x86 based. So I am looking around.
> >> >>>> >> >> > >> I will update you soon.
> >> >>>> >> >> > >>
> >> >>>> >> >> > >> Thanks
> >> >>>> >> >> > >> Ajit
> >> >>>> >> >> > >>
> >> >>>> >> >> > >> On Fri, Mar 5, 2021 at 2:06 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>
> >> >>>> >> >> > >>> Hi David,
> >> >>>> >> >> > >>> That's a good idea. That way you don't have to watch out for my email requesting to stop the tests.
> >> >>>> >> >> > >>>
> >> >>>> >> >> > >>> Thanks
> >> >>>> >> >> > >>> Ajit
> >> >>>> >> >> > >>>
> >> >>>> >> >> > >>> On Fri, Mar 5, 2021 at 1:29 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>
> >> >>>> >> >> > >>>> Hi Ajit, Juraj,
> >> >>>> >> >> > >>>>
> >> >>>> >> >> > >>>> For simplicity, we will keep the functional and performance testing disable during the weekend.
> >> >>>> >> >> > >>>>
> >> >>>> >> >> > >>>> In this way, you will have no testing interfere with you during troubleshooting.
> >> >>>> >> >> > >>>>
> >> >>>> >> >> > >>>> Then we will rerun all the jobs that are skipped.
> >> >>>> >> >> > >>>>
> >> >>>> >> >> > >>>> Please let me know if you have any questions.
> >> >>>> >> >> > >>>>
> >> >>>> >> >> > >>>> Best,
> >> >>>> >> >> > >>>> David
> >> >>>> >> >> > >>>>
> >> >>>> >> >> > >>>> On Wed, Mar 3, 2021 at 4:05 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> Hi David, Ajit,
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> Friday 10am is 4pm for me a that's fine.
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> Juraj
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> From: David Liu <dliu@iol.unh.edu>
> >> >>>> >> >> > >>>>> Sent: Tuesday, March 2, 2021 7:06 PM
> >> >>>> >> >> > >>>>> To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> >>>> >> >> > >>>>> Cc: Juraj Linkeš <juraj.linkes@pantheon.tech>; Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >> >>>> >> >> > >>>>> Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> Hi, Ajit
> >> >>>> >> >> > >>>>> I changed it to Friday at 10 AM for now.
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> We can change again if that doesn't work for Juraj
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> Best,
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> David
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 10:28 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> Hi David,
> >> >>>> >> >> > >>>>> My Thursdays are crazy. Can we do Friday instead?
> >> >>>> >> >> > >>>>> Juraj, can you make Friday 10AM (Eastern time) call?
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> Thanks
> >> >>>> >> >> > >>>>> Ajit
> >> >>>> >> >> > >>>>>
> >> >>>> >> >> > >>>>> On Tue, Mar 2, 2021 at 6:29 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >
> >> >>>> >> >> > >>>>> > Hi Ajit and Juraj,
> >> >>>> >> >> > >>>>> >
> >> >>>> >> >> > >>>>> > I send out an invite for Thursday, Mar 4 at 10:00 AM. But feel free to adjust the time or let me know if there is a better time that works for you.
> >> >>>> >> >> > >>>>> >
> >> >>>> >> >> > >>>>> > For me, I can do:
> >> >>>> >> >> > >>>>> > Tuesday 10:00 AM - 1:00 PM.
> >> >>>> >> >> > >>>>> > Thursday 10:00 AM - 1:00 PM.
> >> >>>> >> >> > >>>>> > Friday 10:00 AM - 12:00 PM and 2:00 PM - 5:00 PM
> >> >>>> >> >> > >>>>> >
> >> >>>> >> >> > >>>>> > Best,
> >> >>>> >> >> > >>>>> > David
> >> >>>> >> >> > >>>>> >
> >> >>>> >> >> > >>>>> > On Mon, Mar 1, 2021 at 10:12 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >>
> >> >>>> >> >> > >>>>> >> David,
> >> >>>> >> >> > >>>>> >> I was busy with a customer issue and could not get to my emails.
> >> >>>> >> >> > >>>>> >> I am seeing this now. Let me close out any issues needing urgent attention.
> >> >>>> >> >> > >>>>> >> Let's sync on a suitable time after that.
> >> >>>> >> >> > >>>>> >>
> >> >>>> >> >> > >>>>> >> Juraj,
> >> >>>> >> >> > >>>>> >> Sure. We will add you.
> >> >>>> >> >> > >>>>> >>
> >> >>>> >> >> > >>>>> >> Thanks
> >> >>>> >> >> > >>>>> >> Ajit
> >> >>>> >> >> > >>>>> >>
> >> >>>> >> >> > >>>>> >> On Mon, Mar 1, 2021 at 7:00 AM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Hi Ajit, David,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > If you're setting up a call, could you please send me an invite? I'd like to get some insight into the issues on Arm's behalf.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Juraj
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > From: ci <ci-bounces@dpdk.org> On Behalf Of David Liu
> >> >>>> >> >> > >>>>> >> > Sent: Thursday, February 25, 2021 7:47 PM
> >> >>>> >> >> > >>>>> >> > To: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> >>>> >> >> > >>>>> >> > Cc: Owen Hilyard <ohilyard@iol.unh.edu>; ci@dpdk.org; dpdklab <dpdklab@iol.unh.edu>
> >> >>>> >> >> > >>>>> >> > Subject: Re: [dpdk-ci] [dpdklab] Re: [DPDKLAB] 25G Broadcom NIC in Arm hardware
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Hi Ajit,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Sorry for this issue.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > If possible I would like to set up a time where we can do this in a call to help get this solved faster.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Please let me know what would be a good time for you.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Best,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > David
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Thu, Feb 25, 2021 at 12:55 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Hi Ajit,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > I changed the permission to allow you to run sudo.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > This should work now.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Let me know if you still have issues.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Best,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > David
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Mon, Feb 22, 2021 at 11:11 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > David,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > I tried the option to use sudo. And that didn't work.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Ajit
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-gigabyte.  This incident will be reported.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > ----
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > sudo su
> >> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
> >> >>>> >> >> > >>>>> >> > ajitkhaparde@arm-dell:/opt/dts$ sudo ./dts -s
> >> >>>> >> >> > >>>>> >> > [sudo] password for ajitkhaparde:
> >> >>>> >> >> > >>>>> >> > ajitkhaparde is not allowed to run sudo on arm-dell.  This incident will be reported.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:03 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Thu, Feb 18, 2021 at 10:01 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Hi Ajit,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > This is likely caused by when dts try to ssh into another machine as a normal user which will prompt you for the password.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > One solution would be "sudo su" as root to run the command line since you should have root access.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > David,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Let me try this method first.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > If it does not work, we could try with the public ssh keys.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > I will let you know.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Ajit
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Another solution would be to let me add your public ssh key into both machines.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Let me know if this helps.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Best,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > David
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 5:48 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks Owen.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > When I run the dts command line suggested, I get the following error.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Ajit
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Traceback (most recent call last):
> >> >>>> >> >> > >>>>> >> >   File "/opt/dts/framework/ssh_pexpect.py", line 60, in _connect_host
> >> >>>> >> >> > >>>>> >> >     self.password, original_prompt='[$#>]', password_regex=r'(?i)(?:password:)|(?:passphrase for key)|(?i)(password for .+:)')
> >> >>>> >> >> > >>>>> >> >   File "/usr/local/lib/python3.6/dist-packages/pexpect/pxssh.py", line 440, in login
> >> >>>> >> >> > >>>>> >> >     raise ExceptionPxssh('password refused')
> >> >>>> >> >> > >>>>> >> > pexpect.pxssh.ExceptionPxssh: password refused
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >                                          ^^^^^^^^^^^^^^
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Fri, Feb 12, 2021 at 12:09 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Hello Ajit,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > There was a misconfiguration in our identity management system. The issue should now be resolved.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Owen Hilyard
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Thu, Feb 11, 2021 at 9:38 PM 'Ajit Khaparde' via dpdklab <dpdklab@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > David,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > I can use my credentials to login to rhea.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > But the same credentials are not allowing me to login onto the servers you provided.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Ajit
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-gigabyte.dpdklab.iol.unh.edu
> >> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.41 port 22
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > ssh -l ajitkhaparde arm-dell.dpdklab.iol.unh.edu
> >> >>>> >> >> > >>>>> >> > Connection closed by 172.18.0.40 port 22
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Tue, Feb 9, 2021 at 6:36 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Hi Ajit,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks for the update. Let me know if you need anything from me.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Best,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > David
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Sun, Feb 7, 2021 at 11:04 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > David,
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > I had some trouble with my system over the past week.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > So I could not login into the setup. I am hoping that my system will be restored early this week
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > I will login into the setup after that.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Ajit
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 10:57 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Hi David,
> >> >>>> >> >> > >>>>> >> > I actually did miss it. I will take a look.
> >> >>>> >> >> > >>>>> >> > Thanks for the refresh mail.
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > Thanks
> >> >>>> >> >> > >>>>> >> > Ajit
> >> >>>> >> >> > >>>>> >> >
> >> >>>> >> >> > >>>>> >> > On Tue, Feb 2, 2021 at 8:58 AM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > Hi Ajit,
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > Ping again in case if you miss the last email.
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > Did you have a chance to give it a try on login to the machines?
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > Just some notes, we are currently running compile and some performance/functional testing.
> >> >>>> >> >> > >>>>> >> > > Which it may be hard for you to run dts, so please let me know if you want me to stop these tests for you to debug.
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > Step I run:
> >> >>>> >> >> > >>>>> >> > > CONF="conf_dpdk" ./switch.sh Broadcom   #this is to switch the dts configuration to Broadcom
> >> >>>> >> >> > >>>>> >> > > ./dts -s                                                           #To start the dts
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > Informatio:
> >> >>>> >> >> > >>>>> >> > > arm-dell.dpdklab.iol.unh.edu
> >> >>>> >> >> > >>>>> >> > > - dts - located in /opt/dts
> >> >>>> >> >> > >>>>> >> > > - Trex - located in /opt/trex/v2.87
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > arm-gigabyte.dpdklab.iol.unh.edu
> >> >>>> >> >> > >>>>> >> > > - dpdk - located in /root/dpdk
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > Best,
> >> >>>> >> >> > >>>>> >> > > David
> >> >>>> >> >> > >>>>> >> > >
> >> >>>> >> >> > >>>>> >> > > On Tue, Jan 26, 2021 at 4:59 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> > >>
> >> >>>> >> >> > >>>>> >> > >> Also, I had changed the configuration when I run Broadcom nic.
> >> >>>> >> >> > >>>>> >> > >> In the dts folder, you can start it with ./dts -s
> >> >>>> >> >> > >>>>> >> > >>
> >> >>>> >> >> > >>>>> >> > >> On Tue, Jan 26, 2021 at 4:54 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> > >>>
> >> >>>> >> >> > >>>>> >> > >>> Hi Ajit,
> >> >>>> >> >> > >>>>> >> > >>> Sorry for the late reply.
> >> >>>> >> >> > >>>>> >> > >>>
> >> >>>> >> >> > >>>>> >> > >>> You can ssh into the machine using the same VPN Username and Password.
> >> >>>> >> >> > >>>>> >> > >>> Can you try and see if that works?
> >> >>>> >> >> > >>>>> >> > >>>
> >> >>>> >> >> > >>>>> >> > >>> Informatio:
> >> >>>> >> >> > >>>>> >> > >>> arm-dell.dpdklab.iol.unh.edu
> >> >>>> >> >> > >>>>> >> > >>> - dts - located in /opt/dts
> >> >>>> >> >> > >>>>> >> > >>> - Trex - located in /opt/trex/v2.87
> >> >>>> >> >> > >>>>> >> > >>>
> >> >>>> >> >> > >>>>> >> > >>> arm-gigabyte.dpdklab.iol.unh.edu
> >> >>>> >> >> > >>>>> >> > >>> - dpdk - located in /root/dpdk
> >> >>>> >> >> > >>>>> >> > >>>
> >> >>>> >> >> > >>>>> >> > >>>
> >> >>>> >> >> > >>>>> >> > >>> Thanks,
> >> >>>> >> >> > >>>>> >> > >>> David
> >> >>>> >> >> > >>>>> >> > >>>
> >> >>>> >> >> > >>>>> >> > >>> On Tue, Jan 19, 2021 at 5:20 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >> > >>>>
> >> >>>> >> >> > >>>>> >> > >>>> Hi David,
> >> >>>> >> >> > >>>>> >> > >>>> Thanks for pinging me. Do you think I could login into the setup and
> >> >>>> >> >> > >>>>> >> > >>>> see what is happening?
> >> >>>> >> >> > >>>>> >> > >>>> I do have the credentials for two systems in your lab -
> >> >>>> >> >> > >>>>> >> > >>>> rhea.dpdklab.iol.unh.edu and io.dpdklab.iol.unh.edu
> >> >>>> >> >> > >>>>> >> > >>>> If you can provide me details of the systems you are trying to
> >> >>>> >> >> > >>>>> >> > >>>> configure, I could take a look.
> >> >>>> >> >> > >>>>> >> > >>>>
> >> >>>> >> >> > >>>>> >> > >>>> Thanks
> >> >>>> >> >> > >>>>> >> > >>>> Ajit
> >> >>>> >> >> > >>>>> >> > >>>>
> >> >>>> >> >> > >>>>> >> > >>>> On Tue, Jan 19, 2021 at 1:41 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> > >>>> >
> >> >>>> >> >> > >>>>> >> > >>>> > Hi Ajit,
> >> >>>> >> >> > >>>>> >> > >>>> >
> >> >>>> >> >> > >>>>> >> > >>>> > Thank you for helping out.
> >> >>>> >> >> > >>>>> >> > >>>> >
> >> >>>> >> >> > >>>>> >> > >>>> > How are you doing with this issue?
> >> >>>> >> >> > >>>>> >> > >>>> >
> >> >>>> >> >> > >>>>> >> > >>>> > Please let me know if you need anything information from us.
> >> >>>> >> >> > >>>>> >> > >>>> >
> >> >>>> >> >> > >>>>> >> > >>>> > Just some extra information that both Trex and dts are on the same intel tester machine.
> >> >>>> >> >> > >>>>> >> > >>>> >
> >> >>>> >> >> > >>>>> >> > >>>> > Best,
> >> >>>> >> >> > >>>>> >> > >>>> > David
> >> >>>> >> >> > >>>>> >> > >>>> >
> >> >>>> >> >> > >>>>> >> > >>>> > On Thu, Jan 14, 2021 at 2:12 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> > >>>> >>
> >> >>>> >> >> > >>>>> >> > >>>> >> Yes, so this is the same NIC work on x86.
> >> >>>> >> >> > >>>>> >> > >>>> >> By seeing the link, I am assuming you mean the ports that are using for Trex. If so they are UP before I start the Trex.
> >> >>>> >> >> > >>>>> >> > >>>> >>
> >> >>>> >> >> > >>>>> >> > >>>> >> Thank,
> >> >>>> >> >> > >>>>> >> > >>>> >> David
> >> >>>> >> >> > >>>>> >> > >>>> >>
> >> >>>> >> >> > >>>>> >> > >>>> >>
> >> >>>> >> >> > >>>>> >> > >>>> >> On Thu, Jan 14, 2021 at 1:11 AM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >> > >>>> >>>
> >> >>>> >> >> > >>>>> >> > >>>> >>> Does the same NIC work on x86?
> >> >>>> >> >> > >>>>> >> > >>>> >>> Also do you see link before you start the trex? I am trying to see if the port failed to successfully come up
> >> >>>> >> >> > >>>>> >> > >>>> >>> or if there is a problem with the link. Can you check and tell?
> >> >>>> >> >> > >>>>> >> > >>>> >>>
> >> >>>> >> >> > >>>>> >> > >>>> >>> Thanks
> >> >>>> >> >> > >>>>> >> > >>>> >>> Ajit
> >> >>>> >> >> > >>>>> >> > >>>> >>>
> >> >>>> >> >> > >>>>> >> > >>>> >>> On Wed, Jan 13, 2021 at 4:20 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> > >>>> >>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>> Hi Ajit,
> >> >>>> >> >> > >>>>> >> > >>>> >>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>> Thank you for helping out.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>> We have it working on the x86, but not the Arm.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>> Thanks,
> >> >>>> >> >> > >>>>> >> > >>>> >>>> David
> >> >>>> >> >> > >>>>> >> > >>>> >>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>> On Wed, Jan 13, 2021 at 5:34 PM Ajit Khaparde <ajit.khaparde@broadcom.com> wrote:
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>> Hi David,
> >> >>>> >> >> > >>>>> >> > >>>> >>>>> I will take a look.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>> Do you see similar issues on x86? I am asking because
> >> >>>> >> >> > >>>>> >> > >>>> >>>>> even I would start with that to create a baseline and then
> >> >>>> >> >> > >>>>> >> > >>>> >>>>> attempt on ARM.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>> Thanks
> >> >>>> >> >> > >>>>> >> > >>>> >>>>> Ajit
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>> On Wed, Jan 13, 2021 at 12:08 PM David Liu <dliu@iol.unh.edu> wrote:
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> Hi Ajit,
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> We have a 25G Broadcom NIC that is installed on an Arm machine.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> And I am running into a problem when I try to run nic_single_core_perf on the NIC.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> Currently, we are using: Trex v2.86 and dts from http://git.dpdk.org/tools/dts/
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> All the NICs are up and running when testing.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> The problem is an error when I try to run the test case inside the nic_single_core_perf:
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> TestNicSingleCorePerf: Test running at parameters: framesize: 64, rxd/txd: 512
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 32,33 -n 4  --file-prefix=dpdk_15311_20210113190237   -a 0000:93:00.0 -a 0000:93:00.1 -- -i --portmask=0x3 --txd=512 --rxd=512 --nb-cores=1
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>                dut.172.18.0.41: start
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>          TestNicSingleCorePerf: Test Case test_perf_nic_single_core Result ERROR: Traceback (most recent call last):
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/test_case.py", line 319, in _execute_test_case
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     case_obj()
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 196, in test_perf_nic_single_core
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.perf_test(self.nb_ports)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "tests/TestSuite_nic_single_core_perf.py", line 270, in perf_test
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_base.py", line 245, in measure_throughput
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._prepare_transmission(stream_ids=stream_ids)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/dts/framework/pktgen_trex.py", line 778, in _prepare_transmission
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._conn.reset(ports=self._ports)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 339, in reset
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self.clear_stats(ports)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_api_annotators.py", line 51, in wrap2
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     ret = f(*args, **kwargs)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/stl/trex_stl_client.py", line 1492, in clear_stats
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     self._clear_stats_common(ports, clear_global, clear_xstats)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>   File "/opt/trex/v2.86/automation/trex_control_plane/interactive/trex/common/trex_client.py", line 2876, in _clear_stats_common
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>     raise TRexError(rc)
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex.common.trex_exceptions.TRexError: *** [RPC] - Failed to get server response from tcp://172.18.0.40:4501
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> I also try to run TRex in stateless mode with./t-rex-64 -i --cfg /etc/trex_cfg.yaml and  connect with ./trex-console
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> Then start sending traffic with command
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>start -f stl/imix.py
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> But this will cause an error to turn off Trex.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> trex>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2021-01-13 19:56:21 - [server][warning]  - Server has been shutdown - cause: 'WATCHDOG: task 'Trex DP core 1' has not responded for more than 1.00135 seconds - timeout is 1 seconds
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** traceback follows ***
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 1       0x561be173cf5a ./_t-rex-64(+0x19af5a) [0x561be173cf5a]
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 2       0x7feea3ac0980 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7feea3ac0980]
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 3       0x561be1a05a2a rte_delay_us_block + 106
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 4       0x561be16ce874 CCoreEthIF::send_burst(CCorePerPort*, unsigned short, CVirtualIFPerSideStats*) + 3220
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 5       0x561be16abf57 CCoreEthIF::flush_tx_queue() + 31
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 6       0x561be171e3d9 CNodeGenerator::handle_maintenance(CFlowGenListPerThread*) + 265
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 7       0x561be171f7ec CNodeGenerator::handle_flow_sync(CGenNode*, CFlowGenListPerThread*, bool&) + 92
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 8       0x561be171fc98 CNodeGenerator::handle_slow_messages(unsigned char, CGenNode*, CFlowGenListPerThread*, bool) + 184
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 9       0x561be16cb5f1 int CNodeGenerator::flush_file_realtime<23, false>(double, double, CFlowGenListPerThread*, double&) + 881
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 10      0x561be1905212 TrexStatelessDpCore::start_scheduler() + 226
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 11      0x561be1883ae9 TrexDpCore::start() + 89
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 12      0x561be1714113 CFlowGenListPerThread::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, CPreviewMode&) + 115
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 13      0x561be16af8dd CGlobalTRex::run_in_core(unsigned char) + 487
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 14      0x561be16d11ad ./_t-rex-64(+0x12f1ad) [0x561be16d11ad]
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 15      0x561be1a1dfaa eal_thread_loop + 426
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 16      0x7feea3ab56db /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7feea3ab56db]
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> 17      0x7feea2a8571f clone + 63
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> *** addr2line information follows ***
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> ??:0
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>> '
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> I believe if everything is working, this should not turn off trex but please correct me if I am wrong.
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> I wonder if you have any suggestion of how I can approach this issue?
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> Best,
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>> David
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>
> >> >>>> >> >> > >>>>> >> > >>>> >>>>>>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 50+ messages in thread

end of thread, other threads:[~2021-04-28  6:48 UTC | newest]

Thread overview: 50+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-13 20:07 [dpdk-ci] [DPDKLAB] 25G Broadcom NIC in Arm hardware David Liu
2021-01-13 22:34 ` Ajit Khaparde
2021-01-14  0:19   ` David Liu
2021-01-14  6:11     ` Ajit Khaparde
2021-01-14 19:12       ` David Liu
2021-01-19 21:40         ` David Liu
2021-01-19 22:19           ` Ajit Khaparde
2021-01-26 21:54             ` David Liu
2021-01-26 21:59               ` David Liu
2021-02-02 16:58                 ` David Liu
2021-02-02 18:57                   ` Ajit Khaparde
2021-02-08  4:03                     ` Ajit Khaparde
2021-02-09 14:35                       ` David Liu
2021-02-12  2:38                         ` Ajit Khaparde
     [not found]                           ` <CAHx6DYDLOBOMMo_vfB0YAhLcQMbDV3-H=RY6Oa+Tp7tWdFxo8A@mail.gmail.com>
2021-02-12 22:48                             ` [dpdk-ci] [dpdklab] " Ajit Khaparde
2021-02-18 18:00                               ` David Liu
2021-02-18 18:03                                 ` Ajit Khaparde
2021-02-23  4:10                                   ` Ajit Khaparde
2021-02-25 17:55                                     ` David Liu
2021-02-25 18:46                                       ` David Liu
2021-03-01 15:00                                         ` Juraj Linkeš
2021-03-01 15:12                                           ` Ajit Khaparde
2021-03-02 14:28                                             ` David Liu
2021-03-02 15:27                                               ` Ajit Khaparde
2021-03-02 18:05                                                 ` David Liu
2021-03-03  9:05                                                   ` Juraj Linkeš
2021-03-05 21:29                                                     ` David Liu
2021-03-05 22:06                                                       ` Ajit Khaparde
2021-03-08  1:07                                                         ` Ajit Khaparde
2021-03-09 14:23                                                           ` David Liu
2021-03-09 16:56                                                             ` Ajit Khaparde
2021-03-09 19:00                                                               ` Ajit Khaparde
2021-03-11 14:01                                                                 ` David Liu
2021-03-11 17:05                                                                   ` Ajit Khaparde
2021-03-16 15:31                                                                     ` David Liu
2021-03-16 15:43                                                                       ` Ajit Khaparde
2021-04-01 17:09                                                                         ` David Liu
2021-04-05 14:25                                                                           ` Ajit Khaparde
2021-04-06 14:29                                                                             ` David Liu
2021-04-07 20:16                                                                               ` Ajit Khaparde
2021-04-09 22:09                                                                                 ` David Liu
2021-04-09 23:34                                                                                   ` Ajit Khaparde
2021-04-15 17:07                                                                                     ` David Liu
2021-04-21  7:39                                                                                       ` Juraj Linkeš
2021-04-21 14:20                                                                                         ` Ajit Khaparde
2021-04-22  3:55                                                                                         ` David Liu
2021-04-22  6:50                                                                                           ` Juraj Linkeš
2021-04-27 14:09                                                                                             ` David Liu
2021-04-28  1:19                                                                                               ` Ajit Khaparde
2021-04-28  6:48                                                                                                 ` Ajit Khaparde

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).