[dts] [Bug 670] Exception occurs with running nic_single_core_perf with latest DTS version

[dts] [Bug 670] Exception occurs with running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

            Bug ID: 670
           Summary: Exception occurs with running nic_single_core_perf
                    with latest DTS version
           Product: DTS
           Version: unspecified
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: tests
          Assignee: dts@dpdk.org
          Reporter: alialnu@nvidia.com
  Target Milestone: ---

Created attachment 154
  --> https://bugs.dpdk.org/attachment.cgi?id=154&action=edit
DTS log with scapy 2.4.4

When running the test 'nic_single_core_perf' on latest DTS version (082321a2
test_plans/vm2vm_virtio_net_perf_test_plan.rst), the following exception
occurs:

"""
  File "/path/to/venv/lib/python3.6/site-packages/scapy/packet.py", line 1600,
in getfield_and_val
    raise AttributeError(attr)
AttributeError: _offset
"""

Followed by the following exception:

"""
exception.TimeoutException: TIMEOUT on rm -rf x86_64-native-linux-gcc
"""

See full log in attached file.

The previous exceptions happen when using the scapy version defined in
requirements.txt (scapy 2.4.4).

Forcing DTS to use TRex's scapy (2.3.1) causes another exception:

"""
$ PYTHONPATH=/opt/trex-core-2.59/external_libs/scapy-2.3.1/python3 ./dts

module 'scapy.contrib.gtp' has no attribute 'GTPPDUSessionContainer'
No module named 'scapy.contrib.lldp'
No module named 'scapy.contrib.pfcp'
No module named 'scapy.contrib.nsh'
Traceback (most recent call last):
  File "./main.py", line 50, in <module>
    import dts
  File "/path/to/dts/framework/dts.py", line 47, in <module>
    from tester import Tester
  File "/path/to/dts/framework/tester.py", line 52, in <module>
    from packet import SCAPY_IMP_CMD
  File "/path/to/dts/framework/packet.py", line 129, in <module>
    class scapy(object):
  File "/path/to/dts/framework/packet.py", line 144, in scapy
    'sctp': SCTP(),
NameError: name 'SCTP' is not defined
"""

Configuration files:

"""
$ cat conf/crbs.cfg
[HOSTNAME]
dut_ip=HOSTNAME
dut_user=root
dut_passwd=
os=linux
dut_arch=
tester_ip=127.0.0.1
tester_passwd=
ixia_group=
pktgen_group=TREX
channels=4
bypass_core0=True
dut_cores=
snapshot_load_side=tester

$ cat conf/nic_single_core_perf.cfg
[suite]
update_expected = True
test_parameters = {'1C/1T': {64: [256], 512: [256]}}
rx_desc_16byte = 'y'
test_duration = 60
accepted_tolerance = 1
expected_throughput = {'ConnectX5_MT4121': {'1C/1T': {64: {256: 0.0}, 512:
{256: 0.0}}}}

$ cat conf/pktgen.cfg
[TREX]
trex_root_path=/opt/trex-core-2.59
trex_lib_path=/opt/trex-core-2.59/automation/trex_control_plane/interactive
config_file=/etc/trex_cfg.yaml
server=127.0.0.1
pcap_file=/opt/trex-core-2.59/stl/sample.pcap
core_num=14
start_trex=yes

$ cat conf/ports.cfg
[HOSTNAME]
ports =
    pci=0000:82:00.0,peer=TREX:0;
    pci=0000:82:00.1,peer=TREX:1;

$ cat execution.cfg
[Execution1]
crbs=HOSTNAME
drivername=mlx5_core
build_type=meson
test_suites=
    nic_single_core_perf,
targets=
    x86_64-native-linux-gcc
parameters=nic_type=cfg:perf=true
"""

OS: Ubuntu 18.04.2
Python: 3.6.9

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs with running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

--- Comment #1 from Ali Alnubani (alialnu@nvidia.com) ---
The last known good version is 54931c7f5706511486328c0b73e53a6b7d216f06.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs with running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

--- Comment #2 from Ali Alnubani (alialnu@nvidia.com) ---
The exceptions don't reproduce when updating TRex to v2.87 and running with:
"""
export PYTHONPATH=/opt/trex-core-2.87/external_libs/scapy-2.4.3
./dts
"""

However, I can't update TRex to this version in production due to a regression
unrelated to DTS that I'm facing with it.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs when running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

Ali Alnubani (alialnu@nvidia.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Exception occurs with       |Exception occurs when
                   |running                     |running
                   |nic_single_core_perf with   |nic_single_core_perf with
                   |latest DTS version          |latest DTS version

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs when running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

dliu (dliu@iol.unh.edu) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dliu@iol.unh.edu

--- Comment #3 from dliu (dliu@iol.unh.edu) ---
Hi Ali,

I was testing our Mellanox machine with Trex V2.88 and recent dts

But because we have an older driver version, it was not able to start Trex.

                        pktgen: ssh root@127.0.0.1
                        pktgen: cd /opt/v2.88;./t-rex-64 -i --cfg
/etc/trex_cfg_cx4lx25g.yaml -c 7
Exception happened in [cd /opt/v2.88;./t-rex-64 -i --cfg
/etc/trex_cfg_cx4lx25g.yaml -c 7] and output is [ Warning: Mellanox NICs were
tested only with RedHat/CentOS 7.9
Correct usage with other Linux distributions is not guaranteed.
Installed OFED version is 'b'MLNX_OFED_LINUX-5.0-2.1.8.0 (OFED-5.0-2.1.8):'',
should be at least '5.2' and up.
ERROR encountered while configuring TRex system]
                           dts: TIMEOUT on cd /opt/v2.88;./t-rex-64 -i --cfg
/etc/trex_cfg_cx4lx25g.yaml -c 7
                        tester: pkill -f _t-rex-64
Traceback (most recent call last):
  File "/home/jenkins/dts_per_test/framework/dts.py", line 342, in
dts_crbs_init

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs when running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

--- Comment #4 from dliu (dliu@iol.unh.edu) ---
(In reply to Ali Alnubani from comment #2)
> The exceptions don't reproduce when updating TRex to v2.87 and running with:
> """
> export PYTHONPATH=/opt/trex-core-2.87/external_libs/scapy-2.4.3
> ./dts
> """
> 
> However, I can't update TRex to this version in production due to a
> regression unrelated to DTS that I'm facing with it.

For I understand of DTS, it will use the scapy from the Trex rather than the
system scapy. That's why you need to repoint the scapy folder.

Since trex-core-2.59 have scapy version of 2.3.1. DTS could be using something
in scapy 2.4.3(Trex v2.88) that does not exist in scapy 2.3.1(Trex 2.59).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs when running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

Lijuan Tu (lijuan.tu@intel.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lijuan.tu@intel.com

--- Comment #5 from Lijuan Tu (lijuan.tu@intel.com) ---
(In reply to dliu from comment #4)
> (In reply to Ali Alnubani from comment #2)
> > The exceptions don't reproduce when updating TRex to v2.87 and running
> with:
> > """
> > export PYTHONPATH=/opt/trex-core-2.87/external_libs/scapy-2.4.3
> > ./dts
> > """
> > 
> > However, I can't update TRex to this version in production due to a
> > regression unrelated to DTS that I'm facing with it.
> 
> For I understand of DTS, it will use the scapy from the Trex rather than the
> system scapy. That's why you need to repoint the scapy folder.
> 
> Since trex-core-2.59 have scapy version of 2.3.1. DTS could be using
> something in scapy 2.4.3(Trex v2.88) that does not exist in scapy 2.3.1(Trex
> 2.59).

agree with David, scapy 2.3.1 doesn't have SCTP, but DTS take it as a basic
protocol. I have a workaround you may try:

1, download sctp from
https://github.com/secdev/scapy/blob/master/scapy/layers/sctp.py to
dts/dep/scapy_modules/
2, apply the following patch:
diff --git a/framework/packet.py b/framework/packet.py
index 6d9bf24..92bbd2c 100644
--- a/framework/packet.py
+++ b/framework/packet.py
@@ -58,6 +58,7 @@ scapy_modules_required = {'gtp': ['GTP_U_Header',
'GTPPDUSessionContainer'],
                           'nsh': ['NSH'],
                           'igmp': ['IGMP'],
                           'mpls': ['MPLS'],
+                          'sctp': ['SCTP'],
                           }
 local_modules = [m[:-3] for m in os.listdir(DEP_FOLDER + '/scapy_modules') if
(m.endswith('.py') and not m.startswith('__'))]

Now DTS will import SCTP from local module: dts/dep/scapy_modules/sctp.py

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs when running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

--- Comment #6 from Ali Alnubani (alialnu@nvidia.com) ---
Thanks all for helping, and sorry for the delay,

I have 2 issues in trex that are blocking me from upgrading it on production:
- https://github.com/cisco-system-traffic-generator/trex-core/issues/632
- https://github.com/cisco-system-traffic-generator/trex-core/issues/678

The first was recently resolved in v2.89, but we are still doing more testing
to verify. Hope I can update soon.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs when running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

--- Comment #7 from Ali Alnubani (alialnu@nvidia.com) ---
Sorry didn't have a chance to join the last meeting and update about this.

The issue
https://github.com/cisco-system-traffic-generator/trex-core/issues/678 isn't
resolved yet. We are still debugging it internally and I'll update when it's
resolved.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[dts] [Bug 670] Exception occurs when running nic_single_core_perf with latest DTS version

[bugzilla]

https://bugs.dpdk.org/show_bug.cgi?id=670

Lincoln Lavoie (lylavoie@iol.unh.edu) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |lylavoie@iol.unh.edu

--- Comment #8 from Lincoln Lavoie (lylavoie@iol.unh.edu) ---
TRex exception for both 512B and 1024B frames

-- 
You are receiving this mail because:
You are the assignee for the bug.