DPDK patches and discussions
 help / color / mirror / Atom feed
From: bugzilla@dpdk.org
To: dev@dpdk.org
Subject: [dpdk-dev] [Bug 382] rte_eth: rx/tx callbacks invoked without lock protection
Date: Thu, 09 Jan 2020 07:52:34 +0000	[thread overview]
Message-ID: <bug-382-3@http.bugs.dpdk.org/> (raw)

https://bugs.dpdk.org/show_bug.cgi?id=382

            Bug ID: 382
           Summary: rte_eth: rx/tx callbacks invoked without lock
                    protection
           Product: DPDK
           Version: 18.11
          Hardware: All
                OS: All
            Status: UNCONFIRMED
          Severity: normal
          Priority: Normal
         Component: ethdev
          Assignee: dev@dpdk.org
          Reporter: zhongdahulinfan@163.com
  Target Milestone: ---

Hi, all  
    I launch my DPDK app, and then use dpdk-pdump to capture wire packets. When
I stop dpdk-pdump with ctrl+c, my DPDK app crash. Here is the coredump
backtrace:  

```
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/dpvs -- -l 0,1,2,3,4,5,6,7,8 -w 0000:04:00.0
-w 0000:04:00.1 --legacy'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00005555558d24e3 in rte_eth_rx_burst (port_id=1, queue_id=2,
rx_pkts=0x55555721ed10 <lcore_conf+2386256>, nb_pkts=32)
    at
/builds/lbc/kigw/kigw_scripts/dpdk-stable-18.11.2/build/include/rte_ethdev.h:3888
3888   
/builds/lbc/kigw/kigw_scripts/dpdk-stable-18.11.2/build/include/rte_ethdev.h:
No such file or directory.
[Current thread is 1 (Thread 0x7ffff3d36700 (LWP 29228))]
(gdb) bt
#0  0x00005555558d24e3 in rte_eth_rx_burst (port_id=1, queue_id=2,
rx_pkts=0x55555721ed10 <lcore_conf+2386256>, nb_pkts=32)
    at
/builds/lbc/kigw/kigw_scripts/dpdk-stable-18.11.2/build/include/rte_ethdev.h:3888
#1  0x00005555558d8743 in netif_rx_burst (pid=1, qconf=0x55555721ed00
<lcore_conf+2386240>) at /builds/lbc/kigw/src/netif.c:1618
#2  0x00005555558dc331 in lcore_job_recv_fwd (arg=0x0) at
/builds/lbc/kigw/src/netif.c:2424
#3  0x00005555558e16de in do_lcore_job (job=0x555555f6f980 <netif_jobs>) at
/builds/lbc/kigw/src/netif.c:4265
#4  0x00005555558e184d in netif_loop (dummy=0x0) at
/builds/lbc/kigw/src/netif.c:4307
#5  0x00005555556cfb9d in eal_thread_loop ()
#6  0x00007ffff67744a4 in start_thread (arg=0x7ffff3d36700) at
pthread_create.c:456
#7  0x00007ffff62abd0f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
```

I followed the source code in rte_ethdev.h where the segmentation fault
occurred in line 3888:  

```
static inline uint16_t
rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id,
                 struct rte_mbuf **rx_pkts, const uint16_t nb_pkts)
{
        struct rte_eth_dev *dev = &rte_eth_devices[port_id];
        uint16_t nb_rx;

#ifdef RTE_LIBRTE_ETHDEV_DEBUG
        RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, 0);
        RTE_FUNC_PTR_OR_ERR_RET(*dev->rx_pkt_burst, 0);

        if (queue_id >= dev->data->nb_rx_queues) {
                RTE_ETHDEV_LOG(ERR, "Invalid RX queue_id=%u\n", queue_id);
                return 0;
        }
#endif
        nb_rx = (*dev->rx_pkt_burst)(dev->data->rx_queues[queue_id],
                                     rx_pkts, nb_pkts);

#ifdef RTE_ETHDEV_RXTX_CALLBACKS
        if (unlikely(dev->post_rx_burst_cbs[queue_id] != NULL)) {
                struct rte_eth_rxtx_callback *cb =
                                dev->post_rx_burst_cbs[queue_id];

                do {
                        nb_rx = cb->fn.rx(port_id, queue_id, rx_pkts, nb_rx,
                                                nb_pkts, cb->param);
                        cb = cb->next;
                } while (cb != NULL);
        }
#endif

        return nb_rx;
}
```

Found that callback list doesn't protected by any lock. These may have
concurrent issues when another process like dpdk-pdump unregister tx/rx
callbacks before it exits. The registration/unregistration of tx/rx callbacks
hold a spinlock. I wonder why there is no lock in invocation of callback list.  
```
int
rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id,
                const struct rte_eth_rxtx_callback *user_cb)
{
#ifndef RTE_ETHDEV_RXTX_CALLBACKS
        return -ENOTSUP;
#endif
        /* Check input parameters. */
        RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL);
        if (user_cb == NULL ||
                        queue_id >=
rte_eth_devices[port_id].data->nb_rx_queues)
                return -EINVAL;

        struct rte_eth_dev *dev = &rte_eth_devices[port_id];
        struct rte_eth_rxtx_callback *cb;
        struct rte_eth_rxtx_callback **prev_cb;
        int ret = -EINVAL;

        rte_spinlock_lock(&rte_eth_rx_cb_lock);
        prev_cb = &dev->post_rx_burst_cbs[queue_id];
        for (; *prev_cb != NULL; prev_cb = &cb->next) {
                cb = *prev_cb;
                if (cb == user_cb) {
                        /* Remove the user cb from the callback list. */
                        *prev_cb = cb->next;
                        ret = 0;
                        break;
                }
        }
        rte_spinlock_unlock(&rte_eth_rx_cb_lock);

        return ret;
} 
```

Please check and conform if this is a bug.

Thanks,
Linfan Hu

-- 
You are receiving this mail because:
You are the assignee for the bug.

             reply	other threads:[~2020-01-09  7:52 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-09  7:52 bugzilla [this message]
2020-01-09 16:10 ` Stephen Hemminger

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=bug-382-3@http.bugs.dpdk.org/ \
    --to=bugzilla@dpdk.org \
    --cc=dev@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).