https://bugs.dpdk.org/show_bug.cgi?id=670 Bug ID: 670 Summary: Exception occurs with running nic_single_core_perf with latest DTS version Product: DTS Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: Normal Component: tests Assignee: dts@dpdk.org Reporter: alialnu@nvidia.com Target Milestone: --- Created attachment 154 --> https://bugs.dpdk.org/attachment.cgi?id=154&action=edit DTS log with scapy 2.4.4 When running the test 'nic_single_core_perf' on latest DTS version (082321a2 test_plans/vm2vm_virtio_net_perf_test_plan.rst), the following exception occurs: """ File "/path/to/venv/lib/python3.6/site-packages/scapy/packet.py", line 1600, in getfield_and_val raise AttributeError(attr) AttributeError: _offset """ Followed by the following exception: """ exception.TimeoutException: TIMEOUT on rm -rf x86_64-native-linux-gcc """ See full log in attached file. The previous exceptions happen when using the scapy version defined in requirements.txt (scapy 2.4.4). Forcing DTS to use TRex's scapy (2.3.1) causes another exception: """ $ PYTHONPATH=/opt/trex-core-2.59/external_libs/scapy-2.3.1/python3 ./dts module 'scapy.contrib.gtp' has no attribute 'GTPPDUSessionContainer' No module named 'scapy.contrib.lldp' No module named 'scapy.contrib.pfcp' No module named 'scapy.contrib.nsh' Traceback (most recent call last): File "./main.py", line 50, in <module> import dts File "/path/to/dts/framework/dts.py", line 47, in <module> from tester import Tester File "/path/to/dts/framework/tester.py", line 52, in <module> from packet import SCAPY_IMP_CMD File "/path/to/dts/framework/packet.py", line 129, in <module> class scapy(object): File "/path/to/dts/framework/packet.py", line 144, in scapy 'sctp': SCTP(), NameError: name 'SCTP' is not defined """ Configuration files: """ $ cat conf/crbs.cfg [HOSTNAME] dut_ip=HOSTNAME dut_user=root dut_passwd= os=linux dut_arch= tester_ip=127.0.0.1 tester_passwd= ixia_group= pktgen_group=TREX channels=4 bypass_core0=True dut_cores= snapshot_load_side=tester $ cat conf/nic_single_core_perf.cfg [suite] update_expected = True test_parameters = {'1C/1T': {64: [256], 512: [256]}} rx_desc_16byte = 'y' test_duration = 60 accepted_tolerance = 1 expected_throughput = {'ConnectX5_MT4121': {'1C/1T': {64: {256: 0.0}, 512: {256: 0.0}}}} $ cat conf/pktgen.cfg [TREX] trex_root_path=/opt/trex-core-2.59 trex_lib_path=/opt/trex-core-2.59/automation/trex_control_plane/interactive config_file=/etc/trex_cfg.yaml server=127.0.0.1 pcap_file=/opt/trex-core-2.59/stl/sample.pcap core_num=14 start_trex=yes $ cat conf/ports.cfg [HOSTNAME] ports = pci=0000:82:00.0,peer=TREX:0; pci=0000:82:00.1,peer=TREX:1; $ cat execution.cfg [Execution1] crbs=HOSTNAME drivername=mlx5_core build_type=meson test_suites= nic_single_core_perf, targets= x86_64-native-linux-gcc parameters=nic_type=cfg:perf=true """ OS: Ubuntu 18.04.2 Python: 3.6.9 -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 --- Comment #1 from Ali Alnubani (alialnu@nvidia.com) --- The last known good version is 54931c7f5706511486328c0b73e53a6b7d216f06. -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 --- Comment #2 from Ali Alnubani (alialnu@nvidia.com) --- The exceptions don't reproduce when updating TRex to v2.87 and running with: """ export PYTHONPATH=/opt/trex-core-2.87/external_libs/scapy-2.4.3 ./dts """ However, I can't update TRex to this version in production due to a regression unrelated to DTS that I'm facing with it. -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 Ali Alnubani (alialnu@nvidia.com) changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Exception occurs with |Exception occurs when |running |running |nic_single_core_perf with |nic_single_core_perf with |latest DTS version |latest DTS version -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 dliu (dliu@iol.unh.edu) changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dliu@iol.unh.edu --- Comment #3 from dliu (dliu@iol.unh.edu) --- Hi Ali, I was testing our Mellanox machine with Trex V2.88 and recent dts But because we have an older driver version, it was not able to start Trex. pktgen: ssh root@127.0.0.1 pktgen: cd /opt/v2.88;./t-rex-64 -i --cfg /etc/trex_cfg_cx4lx25g.yaml -c 7 Exception happened in [cd /opt/v2.88;./t-rex-64 -i --cfg /etc/trex_cfg_cx4lx25g.yaml -c 7] and output is [ Warning: Mellanox NICs were tested only with RedHat/CentOS 7.9 Correct usage with other Linux distributions is not guaranteed. Installed OFED version is 'b'MLNX_OFED_LINUX-5.0-2.1.8.0 (OFED-5.0-2.1.8):'', should be at least '5.2' and up. ERROR encountered while configuring TRex system] dts: TIMEOUT on cd /opt/v2.88;./t-rex-64 -i --cfg /etc/trex_cfg_cx4lx25g.yaml -c 7 tester: pkill -f _t-rex-64 Traceback (most recent call last): File "/home/jenkins/dts_per_test/framework/dts.py", line 342, in dts_crbs_init -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 --- Comment #4 from dliu (dliu@iol.unh.edu) --- (In reply to Ali Alnubani from comment #2) > The exceptions don't reproduce when updating TRex to v2.87 and running with: > """ > export PYTHONPATH=/opt/trex-core-2.87/external_libs/scapy-2.4.3 > ./dts > """ > > However, I can't update TRex to this version in production due to a > regression unrelated to DTS that I'm facing with it. For I understand of DTS, it will use the scapy from the Trex rather than the system scapy. That's why you need to repoint the scapy folder. Since trex-core-2.59 have scapy version of 2.3.1. DTS could be using something in scapy 2.4.3(Trex v2.88) that does not exist in scapy 2.3.1(Trex 2.59). -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 Lijuan Tu (lijuan.tu@intel.com) changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lijuan.tu@intel.com --- Comment #5 from Lijuan Tu (lijuan.tu@intel.com) --- (In reply to dliu from comment #4) > (In reply to Ali Alnubani from comment #2) > > The exceptions don't reproduce when updating TRex to v2.87 and running > with: > > """ > > export PYTHONPATH=/opt/trex-core-2.87/external_libs/scapy-2.4.3 > > ./dts > > """ > > > > However, I can't update TRex to this version in production due to a > > regression unrelated to DTS that I'm facing with it. > > For I understand of DTS, it will use the scapy from the Trex rather than the > system scapy. That's why you need to repoint the scapy folder. > > Since trex-core-2.59 have scapy version of 2.3.1. DTS could be using > something in scapy 2.4.3(Trex v2.88) that does not exist in scapy 2.3.1(Trex > 2.59). agree with David, scapy 2.3.1 doesn't have SCTP, but DTS take it as a basic protocol. I have a workaround you may try: 1, download sctp from https://github.com/secdev/scapy/blob/master/scapy/layers/sctp.py to dts/dep/scapy_modules/ 2, apply the following patch: diff --git a/framework/packet.py b/framework/packet.py index 6d9bf24..92bbd2c 100644 --- a/framework/packet.py +++ b/framework/packet.py @@ -58,6 +58,7 @@ scapy_modules_required = {'gtp': ['GTP_U_Header', 'GTPPDUSessionContainer'], 'nsh': ['NSH'], 'igmp': ['IGMP'], 'mpls': ['MPLS'], + 'sctp': ['SCTP'], } local_modules = [m[:-3] for m in os.listdir(DEP_FOLDER + '/scapy_modules') if (m.endswith('.py') and not m.startswith('__'))] Now DTS will import SCTP from local module: dts/dep/scapy_modules/sctp.py -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 --- Comment #6 from Ali Alnubani (alialnu@nvidia.com) --- Thanks all for helping, and sorry for the delay, I have 2 issues in trex that are blocking me from upgrading it on production: - https://github.com/cisco-system-traffic-generator/trex-core/issues/632 - https://github.com/cisco-system-traffic-generator/trex-core/issues/678 The first was recently resolved in v2.89, but we are still doing more testing to verify. Hope I can update soon. -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 --- Comment #7 from Ali Alnubani (alialnu@nvidia.com) --- Sorry didn't have a chance to join the last meeting and update about this. The issue https://github.com/cisco-system-traffic-generator/trex-core/issues/678 isn't resolved yet. We are still debugging it internally and I'll update when it's resolved. -- You are receiving this mail because: You are the assignee for the bug.
https://bugs.dpdk.org/show_bug.cgi?id=670 Lincoln Lavoie (lylavoie@iol.unh.edu) changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lylavoie@iol.unh.edu --- Comment #8 from Lincoln Lavoie (lylavoie@iol.unh.edu) --- TRex exception for both 512B and 1024B frames -- You are receiving this mail because: You are the assignee for the bug.