Issue seen in DPDK behavior for virtio driver

Issue seen in DPDK behavior for virtio driver

[Chinmaya Agarwal]

[-- Attachment #1: Type: text/plain, Size: 4265 bytes --]

Hi,

We are running VPP v22.02 and DPDK v21.11.0 on Centos 8 VM. We are facing an issue where if we configure an SRv6 policy on VPP with sid list having 4 SIDs we don't see packet coming out of the VPP interface and if we configure 5 SIDs, again we don't see the packets coming out of the interface plus VPP crashes after some time. We analyzed the crash dump, and it seems that the crash is happening in DPDK library in the virtio driver code leg. The interface we are using has virtio driver associated with it. Below is the DPDK core dump:-

Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault.
0x00007fe67f1cf256 in virtio_update_packet_stats () from /usr/lib/vpp_plugins//dpdk_plugin.so
(gdb) bt
#0  0x00007fe67f1cf256 in virtio_update_packet_stats () from /usr/lib/vpp_plugins//dpdk_plugin.so
#1  0x00007fe67f1d6646 in virtio_xmit_pkts () from /usr/lib/vpp_plugins//dpdk_plugin.so
#2  0x00007fe67f42d60e in rte_eth_tx_burst (nb_pkts=<optimized out>, tx_pkts=0x7fe6862afc00, queue_id=<optimized out>,
    port_id=<optimized out>) at /opt/vpp/external/x86_64/include/rte_ethdev.h:5680
#3  tx_burst_vector_internal (n_left=1, mb=0x7fe6862afc00, xd=<optimized out>, vm=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/plugins/dpdk/device/device.c:175
#4  dpdk_device_class_tx_fn_hsw (vm=<optimized out>, node=<optimized out>, f=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/plugins/dpdk/device/device.c:435
#5  0x00007fe6c66bd802 in dispatch_node (last_time_stamp=<optimized out>, frame=<optimized out>,
    dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INTERNAL, node=0x7fe685e93c00, vm=0x7fe68547a680)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:975
#6  dispatch_pending_node (vm=vm@entry=0x7fe68547a680, pending_frame_index=pending_frame_index@entry=10,
    last_time_stamp=<optimized out>) at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1134
#7  0x00007fe6c66c1ebf in vlib_main_or_worker_loop (is_main=1, vm=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1600
#8  vlib_main_loop (vm=<optimized out>) at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1728
#9  vlib_main (vm=<optimized out>, vm@entry=0x7fe68547a680, input=input@entry=0x7fe675df5fa0)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:2017
#10 0x00007fe6c670cc86 in thread0 (arg=140628055271040)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/unix/main.c:671
#11 0x00007fe6c5c29388 in clib_calljmp () at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vppinfra/longjmp.S:123
#12 0x00007ffcdf319c80 in ?? ()
#13 0x00007fe6c670e210 in vlib_unix_main (argc=<optimized out>, argv=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/unix/main.c:751
#14 0xffffffffffffffff in ?? ()
#15 0x0001a53c00005137 in ?? ()

We repeated the above test with interface having ixgbe driver and we don't see this issue and we can see packet coming out of the interface with correct sid list.
What could be the possible reason for this issue? Can we modify any parameter at dpdk code level as part of debugging this issue?

Also, we tried subscribing to dpdk dev mailing list (using the same email id), but our subscription request is still pending. This issue is kind of a blocker for us that's why sending it on mail.

Thanks and Regards,
Chinmaya Agarwal.
DISCLAIMER: This electronic message and all of its contents, contains information which is privileged, confidential or otherwise protected from disclosure. The information contained in this electronic mail transmission is intended for use only by the individual or entity to which it is addressed. If you are not the intended recipient or may have received this electronic mail transmission in error, please notify the sender immediately and delete / destroy all copies of this electronic mail transmission without disclosing, copying, distributing, forwarding, printing or retaining any part of it. Hughes Systique accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus.

[-- Attachment #2: Type: text/html, Size: 5961 bytes --]

RE: Issue seen in DPDK behavior for virtio driver

[Xia, Chenbo]

Hi,

Please check the cause of segfaults, null pointer, or? Then you can have some clues

Thanks,
Chenbo

From: Chinmaya Agarwal <Chinmaya.Agarwal@hsc.com> 
Sent: Monday, July 25, 2022 4:25 PM
To: dev@dpdk.org
Subject: Issue seen in DPDK behavior for virtio driver

Hi,

We are running VPP v22.02 and DPDK v21.11.0 on Centos 8 VM. We are facing an issue where if we configure an SRv6 policy on VPP with sid list having 4 SIDs we don't see packet coming out of the VPP interface and if we configure 5 SIDs, again we don't see the packets coming out of the interface plus VPP crashes after some time. We analyzed the crash dump, and it seems that the crash is happening in DPDK library in the virtio driver code leg. The interface we are using has virtio driver associated with it. Below is the DPDK core dump:- 

Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault.
0x00007fe67f1cf256 in virtio_update_packet_stats () from /usr/lib/vpp_plugins//dpdk_plugin.so
(gdb) bt
#0  0x00007fe67f1cf256 in virtio_update_packet_stats () from /usr/lib/vpp_plugins//dpdk_plugin.so
#1  0x00007fe67f1d6646 in virtio_xmit_pkts () from /usr/lib/vpp_plugins//dpdk_plugin.so
#2  0x00007fe67f42d60e in rte_eth_tx_burst (nb_pkts=<optimized out>, tx_pkts=0x7fe6862afc00, queue_id=<optimized out>,
    port_id=<optimized out>) at /opt/vpp/external/x86_64/include/rte_ethdev.h:5680
#3  tx_burst_vector_internal (n_left=1, mb=0x7fe6862afc00, xd=<optimized out>, vm=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/plugins/dpdk/device/device.c:175
#4  dpdk_device_class_tx_fn_hsw (vm=<optimized out>, node=<optimized out>, f=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/plugins/dpdk/device/device.c:435
#5  0x00007fe6c66bd802 in dispatch_node (last_time_stamp=<optimized out>, frame=<optimized out>,
    dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INTERNAL, node=0x7fe685e93c00, vm=0x7fe68547a680)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:975
#6  dispatch_pending_node (vm=vm@entry=0x7fe68547a680, pending_frame_index=pending_frame_index@entry=10,
    last_time_stamp=<optimized out>) at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1134
#7  0x00007fe6c66c1ebf in vlib_main_or_worker_loop (is_main=1, vm=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1600
#8  vlib_main_loop (vm=<optimized out>) at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1728
#9  vlib_main (vm=<optimized out>, vm@entry=0x7fe68547a680, input=input@entry=0x7fe675df5fa0)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:2017
#10 0x00007fe6c670cc86 in thread0 (arg=140628055271040)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/unix/main.c:671
#11 0x00007fe6c5c29388 in clib_calljmp () at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vppinfra/longjmp.S:123
#12 0x00007ffcdf319c80 in ?? ()
#13 0x00007fe6c670e210 in vlib_unix_main (argc=<optimized out>, argv=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/unix/main.c:751
#14 0xffffffffffffffff in ?? ()
#15 0x0001a53c00005137 in ?? ()

We repeated the above test with interface having ixgbe driver and we don't see this issue and we can see packet coming out of the interface with correct sid list. 
What could be the possible reason for this issue? Can we modify any parameter at dpdk code level as part of debugging this issue?

Also, we tried subscribing to dpdk dev mailing list (using the same email id), but our subscription request is still pending. This issue is kind of a blocker for us that's why sending it on mail.

Thanks and Regards,
Chinmaya Agarwal.
DISCLAIMER: This electronic message and all of its contents, contains information which is privileged, confidential or otherwise protected from disclosure. The information contained in this electronic mail transmission is intended for use only by the individual or entity to which it is addressed. If you are not the intended recipient or may have received this electronic mail transmission in error, please notify the sender immediately and delete / destroy all copies of this electronic mail transmission without disclosing, copying, distributing, forwarding, printing or retaining any part of it. Hughes Systique accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus. 

Re: Issue seen in DPDK behavior for virtio driver

[Chinmaya Agarwal]

[-- Attachment #1: Type: text/plain, Size: 4618 bytes --]

Hi,

Can we have some pointers on how to debug this issue?
Also, is there a way we can increase logging for DPDK plugin.

Thanks and Regards,
Chinmaya Agarwal.
________________________________
From: Chinmaya Agarwal
Sent: Monday, July 25, 2022 1:55 PM
To: dev@dpdk.org <dev@dpdk.org>
Subject: Issue seen in DPDK behavior for virtio driver

Hi,

We are running VPP v22.02 and DPDK v21.11.0 on Centos 8 VM. We are facing an issue where if we configure an SRv6 policy on VPP with sid list having 4 SIDs we don't see packet coming out of the VPP interface and if we configure 5 SIDs, again we don't see the packets coming out of the interface plus VPP crashes after some time. We analyzed the crash dump, and it seems that the crash is happening in DPDK library in the virtio driver code leg. The interface we are using has virtio driver associated with it. Below is the DPDK core dump:-

Thread 1 "vpp_main" received signal SIGSEGV, Segmentation fault.
0x00007fe67f1cf256 in virtio_update_packet_stats () from /usr/lib/vpp_plugins//dpdk_plugin.so
(gdb) bt
#0  0x00007fe67f1cf256 in virtio_update_packet_stats () from /usr/lib/vpp_plugins//dpdk_plugin.so
#1  0x00007fe67f1d6646 in virtio_xmit_pkts () from /usr/lib/vpp_plugins//dpdk_plugin.so
#2  0x00007fe67f42d60e in rte_eth_tx_burst (nb_pkts=<optimized out>, tx_pkts=0x7fe6862afc00, queue_id=<optimized out>,
    port_id=<optimized out>) at /opt/vpp/external/x86_64/include/rte_ethdev.h:5680
#3  tx_burst_vector_internal (n_left=1, mb=0x7fe6862afc00, xd=<optimized out>, vm=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/plugins/dpdk/device/device.c:175
#4  dpdk_device_class_tx_fn_hsw (vm=<optimized out>, node=<optimized out>, f=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/plugins/dpdk/device/device.c:435
#5  0x00007fe6c66bd802 in dispatch_node (last_time_stamp=<optimized out>, frame=<optimized out>,
    dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INTERNAL, node=0x7fe685e93c00, vm=0x7fe68547a680)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:975
#6  dispatch_pending_node (vm=vm@entry=0x7fe68547a680, pending_frame_index=pending_frame_index@entry=10,
    last_time_stamp=<optimized out>) at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1134
#7  0x00007fe6c66c1ebf in vlib_main_or_worker_loop (is_main=1, vm=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1600
#8  vlib_main_loop (vm=<optimized out>) at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:1728
#9  vlib_main (vm=<optimized out>, vm@entry=0x7fe68547a680, input=input@entry=0x7fe675df5fa0)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/main.c:2017
#10 0x00007fe6c670cc86 in thread0 (arg=140628055271040)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/unix/main.c:671
#11 0x00007fe6c5c29388 in clib_calljmp () at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vppinfra/longjmp.S:123
#12 0x00007ffcdf319c80 in ?? ()
#13 0x00007fe6c670e210 in vlib_unix_main (argc=<optimized out>, argv=<optimized out>)
    at /usr/src/debug/vpp-22.02.0-35~ge3c583654.x86_64/src/vlib/unix/main.c:751
#14 0xffffffffffffffff in ?? ()
#15 0x0001a53c00005137 in ?? ()

We repeated the above test with interface having ixgbe driver and we don't see this issue and we can see packet coming out of the interface with correct sid list.
What could be the possible reason for this issue? Can we modify any parameter at dpdk code level as part of debugging this issue?

Also, we tried subscribing to dpdk dev mailing list (using the same email id), but our subscription request is still pending. This issue is kind of a blocker for us that's why sending it on mail.

Thanks and Regards,
Chinmaya Agarwal.
DISCLAIMER: This electronic message and all of its contents, contains information which is privileged, confidential or otherwise protected from disclosure. The information contained in this electronic mail transmission is intended for use only by the individual or entity to which it is addressed. If you are not the intended recipient or may have received this electronic mail transmission in error, please notify the sender immediately and delete / destroy all copies of this electronic mail transmission without disclosing, copying, distributing, forwarding, printing or retaining any part of it. Hughes Systique accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus.

[-- Attachment #2: Type: text/html, Size: 7578 bytes --]