[dpdk-users] Crash in virtio pmd - dpdk 18.11

[dpdk-users] Crash in virtio pmd - dpdk 18.11

[Rao, Sanjay]

I upgraded to dodk 18.11 and still see the crash at the exact same place in the driver code. Greatly appreciate if somebody familiar with the code help me resolve this.

Thread 15 "lcore-slave-3" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffda1fc700 (LWP 5881)]
0x00000000005b4889 in virtqueue_enqueue_xmit (txvq=0x17ffc3e00, cookie=0x17530be80, needed=1, use_indirect=0, can_push=1, in_order=0)
    at /home/srao/github/athena-master/source/dpdk/drivers/net/virtio/virtio_rxtx.c:476
476         /home/srao/github/athena-master/source/dpdk/drivers/net/virtio/virtio_rxtx.c: No such file or directory.
(gdb) bt
#0  0x00000000005b4889 in virtqueue_enqueue_xmit (txvq=0x17ffc3e00, cookie=0x17530be80, needed=1, use_indirect=0, can_push=1, in_order=0)
    at /home/srao/github/athena-master/source/dpdk/drivers/net/virtio/virtio_rxtx.c:476
#1  0x00000000005b6e2d in virtio_xmit_pkts (tx_queue=0x17ffc3e00, tx_pkts=0x7fd0fcc0 <lcore_conf+18304>, nb_pkts=25)
    at /home/srao/github/athena-master/source/dpdk/drivers/net/virtio/virtio_rxtx.c:1423
#2  0x00000000005fae28 in rte_eth_tx_burst (nb_pkts=<optimized out>, tx_pkts=0x7fd0fcc0 <lcore_conf+18304>, queue_id=0, port_id=1)
    at /home/srao/github/athena-master/output/RHEL-6/gcc492/mode1/dpdk/include/rte_ethdev.h:4158
#3  send_burst (port=1 '\001', n=25, qconf=0x7fd0f980 <lcore_conf+17472>) at source/fanout/fastfanout/ffmain.c:559
#4  worker_loop (lcore_id=<optimized out>) at source/fanout/fastfanout/ffmain.c:2922
#5  0x00000000004c3255 in eal_thread_loop ()
#6  0x00007ffff7782aa1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff6b26bdd in clone () from /lib64/libc.so.6
(gdb) p head_idx
$1 = 32768
(gdb) p vq->vq_descx[idx]
$2 = {cookie = 0x0, ndescs = 0}
(gdb) p dxp->cookie
$3 = (void *) 0x0
(gdb) p dxp
$4 = (struct vq_desc_extra *) 0x180043fa0
(gdb) p *dxp
$5 = {cookie = 0x0, ndescs = 0}
(gdb) p cookie
$6 = (struct rte_mbuf *) 0x17530be80


From: Rao Sanjay <srao@verisign.com>
Date: Monday, May 6, 2019 at 5:09 PM
To: "users@dpdk.org" <users@dpdk.org>
Subject: Crash in virtio pmd - dpdk 17.11.4

My apologies if this is already reported. I am seeing a crash in virtqueue_enqueue_xmit() function at line#291 in drivers/net/virtio_rxtx.c. This is in DPDK version 17.11.4 and happens under a sustained heavy traffic.

static inline void
virtqueue_enqueue_xmit(struct virtnet_tx *txvq, struct rte_mbuf *cookie,
                       uint16_t needed, int use_indirect, int can_push)
{
        struct virtio_tx_region *txr = txvq->virtio_net_hdr_mz->addr;
        struct vq_desc_extra *dxp;
        struct virtqueue *vq = txvq->vq;
        struct vring_desc *start_dp;
        uint16_t seg_num = cookie->nb_segs;
        uint16_t head_idx, idx;
        uint16_t head_size = vq->hw->vtnet_hdr_size;
        struct virtio_net_hdr *hdr;
        int offload;

       offload = tx_offload_enabled(vq->hw);
        head_idx = vq->vq_desc_head_idx;
        idx = head_idx;
        dxp = &vq->vq_descx[idx];     <-- idx is VQ_RING_DESC_CHAIN_END

        ---------
}

Can somebody familiar with this code let me know if there is any work around for this.

Thanks
Sanjay

Re: [dpdk-users] Crash in virtio pmd - dpdk 18.11

[Tiwei Bie]

Hi Sanjay,

On Mon, May 20, 2019 at 04:30:33AM +0000, Rao, Sanjay wrote:
> I upgraded to dodk 18.11 and still see the crash at the exact same place in the driver code. Greatly appreciate if somebody familiar with the code help me resolve this.

Could you provide some steps about how to reproduce this crash?

Thanks,
Tiwei

> 
> Thread 15 "lcore-slave-3" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7fffda1fc700 (LWP 5881)]
> 0x00000000005b4889 in virtqueue_enqueue_xmit (txvq=0x17ffc3e00, cookie=0x17530be80, needed=1, use_indirect=0, can_push=1, in_order=0)
>     at /home/srao/github/athena-master/source/dpdk/drivers/net/virtio/virtio_rxtx.c:476
> 476         /home/srao/github/athena-master/source/dpdk/drivers/net/virtio/virtio_rxtx.c: No such file or directory.
> (gdb) bt
> #0  0x00000000005b4889 in virtqueue_enqueue_xmit (txvq=0x17ffc3e00, cookie=0x17530be80, needed=1, use_indirect=0, can_push=1, in_order=0)
>     at /home/srao/github/athena-master/source/dpdk/drivers/net/virtio/virtio_rxtx.c:476
> #1  0x00000000005b6e2d in virtio_xmit_pkts (tx_queue=0x17ffc3e00, tx_pkts=0x7fd0fcc0 <lcore_conf+18304>, nb_pkts=25)
>     at /home/srao/github/athena-master/source/dpdk/drivers/net/virtio/virtio_rxtx.c:1423
> #2  0x00000000005fae28 in rte_eth_tx_burst (nb_pkts=<optimized out>, tx_pkts=0x7fd0fcc0 <lcore_conf+18304>, queue_id=0, port_id=1)
>     at /home/srao/github/athena-master/output/RHEL-6/gcc492/mode1/dpdk/include/rte_ethdev.h:4158
> #3  send_burst (port=1 '\001', n=25, qconf=0x7fd0f980 <lcore_conf+17472>) at source/fanout/fastfanout/ffmain.c:559
> #4  worker_loop (lcore_id=<optimized out>) at source/fanout/fastfanout/ffmain.c:2922
> #5  0x00000000004c3255 in eal_thread_loop ()
> #6  0x00007ffff7782aa1 in start_thread () from /lib64/libpthread.so.0
> #7  0x00007ffff6b26bdd in clone () from /lib64/libc.so.6
> (gdb) p head_idx
> $1 = 32768
> (gdb) p vq->vq_descx[idx]
> $2 = {cookie = 0x0, ndescs = 0}
> (gdb) p dxp->cookie
> $3 = (void *) 0x0
> (gdb) p dxp
> $4 = (struct vq_desc_extra *) 0x180043fa0
> (gdb) p *dxp
> $5 = {cookie = 0x0, ndescs = 0}
> (gdb) p cookie
> $6 = (struct rte_mbuf *) 0x17530be80
> 
> 
> From: Rao Sanjay <srao@verisign.com>
> Date: Monday, May 6, 2019 at 5:09 PM
> To: "users@dpdk.org" <users@dpdk.org>
> Subject: Crash in virtio pmd - dpdk 17.11.4
> 
> My apologies if this is already reported. I am seeing a crash in virtqueue_enqueue_xmit() function at line#291 in drivers/net/virtio_rxtx.c. This is in DPDK version 17.11.4 and happens under a sustained heavy traffic.
> 
> static inline void
> virtqueue_enqueue_xmit(struct virtnet_tx *txvq, struct rte_mbuf *cookie,
>                        uint16_t needed, int use_indirect, int can_push)
> {
>         struct virtio_tx_region *txr = txvq->virtio_net_hdr_mz->addr;
>         struct vq_desc_extra *dxp;
>         struct virtqueue *vq = txvq->vq;
>         struct vring_desc *start_dp;
>         uint16_t seg_num = cookie->nb_segs;
>         uint16_t head_idx, idx;
>         uint16_t head_size = vq->hw->vtnet_hdr_size;
>         struct virtio_net_hdr *hdr;
>         int offload;
> 
>        offload = tx_offload_enabled(vq->hw);
>         head_idx = vq->vq_desc_head_idx;
>         idx = head_idx;
>         dxp = &vq->vq_descx[idx];     <-- idx is VQ_RING_DESC_CHAIN_END
> 
>         ---------
> }
> 
> Can somebody familiar with this code let me know if there is any work around for this.
> 
> Thanks
> Sanjay
> 
>