* [RFC PATCH 0/5] vhost lock annotations
@ 2022-03-28 12:17 David Marchand
2022-03-28 12:17 ` [RFC PATCH 1/5] vhost: fix missing virtqueue lock protection David Marchand
` (9 more replies)
0 siblings, 10 replies; 110+ messages in thread
From: David Marchand @ 2022-03-28 12:17 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia
vhost internals involves multiple locks to protect data access by
multiple threads.
This series is a try at using clang thread safety checks [1] to catch
issues during compilation: spinlock and rwlock are wrapped into vhost types
and annotations are put all over the code.
Patch 1 is a fix from Maxime that I had to take in the series so
that the CI won't fail.
Patch 2 annotates existing manipulations of the access_lock vq lock.
It does the minimal stuff, more annotations are used/introduced later
in the series.
Patch 4 is an example of extending the work by annotating the async field.
This patch raised the issues fixed in patch 3.
Patch 5 further extends this by annotating rwlocks used in IOTLB.
This raised two suspicious call sites (in vdpa and vhost_crypto code,
see added FIXME). Because of a limitation in the check, the IOTLB lock is
changed to be always taken. I don't expect a big impact, but this needs a
confirmation.
This is still a work in progress.
Those annotations are quite heavy to maintain because the full path of
code must be annotated, but I think it would be worth using.
1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
--
David Marchand
David Marchand (4):
vhost: annotate virtqueue access lock
vhost: fix async access
vhost: annotate async locking requirement
vhost: annotate IOTLB locks
Maxime Coquelin (1):
vhost: fix missing virtqueue lock protection
lib/vhost/iotlb.c | 36 ++++-----
lib/vhost/iotlb.h | 24 ------
lib/vhost/meson.build | 3 +
lib/vhost/vdpa.c | 1 +
lib/vhost/vhost.c | 85 ++++++++++----------
lib/vhost/vhost.h | 169 +++++++++++++++++++++++++++++++++++----
lib/vhost/vhost_crypto.c | 7 ++
lib/vhost/vhost_user.c | 32 +++++---
lib/vhost/virtio_net.c | 96 ++++++++++++++++------
9 files changed, 320 insertions(+), 133 deletions(-)
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH 1/5] vhost: fix missing virtqueue lock protection
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
@ 2022-03-28 12:17 ` David Marchand
2022-03-28 12:17 ` [RFC PATCH 2/5] vhost: annotate virtqueue access lock David Marchand
` (8 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-28 12:17 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, stable, Yuanhan Liu, Stefan Hajnoczi
From: Maxime Coquelin <maxime.coquelin@redhat.com>
This patch ensures virtqueue metadata are not being
modified while rte_vhost_vring_call() is executed.
Fixes: 6c299bb7322f ("vhost: introduce vring call API")
Cc: stable@dpdk.org
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
---
This is the same as: https://patchwork.dpdk.org/project/dpdk/patch/20220324124638.32672-2-maxime.coquelin@redhat.com/
---
lib/vhost/vhost.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index bc88148347..2f96a28dac 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1291,11 +1291,15 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)
if (!vq)
return -1;
+ rte_spinlock_lock(&vq->access_lock);
+
if (vq_is_packed(dev))
vhost_vring_call_packed(dev, vq);
else
vhost_vring_call_split(dev, vq);
+ rte_spinlock_unlock(&vq->access_lock);
+
return 0;
}
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH 2/5] vhost: annotate virtqueue access lock
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
2022-03-28 12:17 ` [RFC PATCH 1/5] vhost: fix missing virtqueue lock protection David Marchand
@ 2022-03-28 12:17 ` David Marchand
2022-03-28 12:17 ` [RFC PATCH 3/5] vhost: fix async access David Marchand
` (7 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-28 12:17 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia
Introduce a wrapper around spinlocks and annotate the access_lock lock
protecting vq.
One small change is required in the code: vhost_poll_enqueue_completed
was getting a queue_id to get hold of the vq, while its callers already
knew of the vq. vq is now directly passed.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/meson.build | 3 ++
lib/vhost/vhost.c | 30 +++++++++---------
lib/vhost/vhost.h | 70 ++++++++++++++++++++++++++++++++++++++++--
lib/vhost/vhost_user.c | 14 +++++----
lib/vhost/virtio_net.c | 33 ++++++++++++--------
5 files changed, 115 insertions(+), 35 deletions(-)
diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
index bc7272053b..6e0bf9e7e8 100644
--- a/lib/vhost/meson.build
+++ b/lib/vhost/meson.build
@@ -17,6 +17,9 @@ elif (toolchain == 'icc' and cc.version().version_compare('>=16.0.0'))
endif
dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY', cc.has_header('linux/userfaultfd.h'))
cflags += '-fno-strict-aliasing'
+if cc.has_argument('-Wthread-safety')
+ cflags += '-Wthread-safety'
+endif
sources = files(
'fd_man.c',
'iotlb.c',
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 2f96a28dac..863cd9131e 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -622,7 +622,7 @@ alloc_vring_queue(struct virtio_net *dev, uint32_t vring_idx)
dev->virtqueue[i] = vq;
init_vring_queue(dev, i);
- rte_spinlock_init(&vq->access_lock);
+ vhost_spinlock_init(&vq->access_lock);
vq->avail_wrap_counter = 1;
vq->used_wrap_counter = 1;
vq->signalled_used_valid = false;
@@ -1291,14 +1291,14 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)
if (!vq)
return -1;
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
if (vq_is_packed(dev))
vhost_vring_call_packed(dev, vq);
else
vhost_vring_call_split(dev, vq);
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return 0;
}
@@ -1321,7 +1321,7 @@ rte_vhost_avail_entries(int vid, uint16_t queue_id)
if (!vq)
return 0;
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
if (unlikely(!vq->enabled || vq->avail == NULL))
goto out;
@@ -1329,7 +1329,7 @@ rte_vhost_avail_entries(int vid, uint16_t queue_id)
ret = *(volatile uint16_t *)&vq->avail->idx - vq->last_used_idx;
out:
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return ret;
}
@@ -1413,12 +1413,12 @@ rte_vhost_enable_guest_notification(int vid, uint16_t queue_id, int enable)
if (!vq)
return -1;
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
vq->notif_enable = enable;
ret = vhost_enable_guest_notification(dev, vq, enable);
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return ret;
}
@@ -1475,7 +1475,7 @@ rte_vhost_rx_queue_count(int vid, uint16_t qid)
if (vq == NULL)
return 0;
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
if (unlikely(!vq->enabled || vq->avail == NULL))
goto out;
@@ -1483,7 +1483,7 @@ rte_vhost_rx_queue_count(int vid, uint16_t qid)
ret = *((volatile uint16_t *)&vq->avail->idx) - vq->last_avail_idx;
out:
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return ret;
}
@@ -1708,9 +1708,9 @@ rte_vhost_async_channel_register(int vid, uint16_t queue_id)
if (unlikely(vq == NULL || !dev->async_copy))
return -1;
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
ret = async_channel_register(vid, queue_id);
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return ret;
}
@@ -1758,7 +1758,7 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
if (!vq->async)
return ret;
- if (!rte_spinlock_trylock(&vq->access_lock)) {
+ if (vhost_spinlock_trylock(&vq->access_lock) == 0) {
VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel, virtqueue busy.\n",
dev->ifname);
return -1;
@@ -1774,7 +1774,7 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
vhost_free_async_mem(vq);
out:
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return ret;
}
@@ -1894,7 +1894,7 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
if (!vq->async)
return ret;
- if (!rte_spinlock_trylock(&vq->access_lock)) {
+ if (vhost_spinlock_trylock(&vq->access_lock) == 0) {
VHOST_LOG_CONFIG(DEBUG,
"(%s) failed to check in-flight packets. virtqueue busy.\n",
dev->ifname);
@@ -1902,7 +1902,7 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
}
ret = vq->async->pkts_inflight_n;
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return ret;
}
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index a9edc271aa..619f073fb2 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -85,6 +85,71 @@
for (iter = val; iter < num; iter++)
#endif
+#ifndef __has_feature
+#define __vhost_has_feature(x) 0
+#else
+#define __vhost_has_feature __has_feature
+#endif
+
+#if __vhost_has_feature(c_thread_safety_attributes)
+#define VHOST_NO_THREAD_SAFETY_ANALYSIS \
+ __attribute__((no_thread_safety_analysis))
+#define VHOST_LOCKABLE \
+ __attribute__((lockable))
+
+#define VHOST_SPINLOCK_REQUIRES(...) \
+ __attribute__((exclusive_locks_required(__VA_ARGS__)))
+#define VHOST_SPINLOCK_ACQUIRES(...) \
+ __attribute__((exclusive_lock_function(__VA_ARGS__)))
+#define VHOST_SPINLOCK_TRYLOCK(ret, ...) \
+ __attribute__((exclusive_trylock_function(ret, __VA_ARGS__)))
+#define VHOST_SPINLOCK_RELEASES(...) \
+ __attribute__((unlock_function(__VA_ARGS__)))
+
+#else
+#define VHOST_NO_THREAD_SAFETY_ANALYSIS
+#define VHOST_LOCKABLE
+
+#define VHOST_SPINLOCK_REQUIRES(...)
+#define VHOST_SPINLOCK_ACQUIRES(...)
+#define VHOST_SPINLOCK_TRYLOCK(...)
+#define VHOST_SPINLOCK_RELEASES(...)
+#endif
+
+typedef struct VHOST_LOCKABLE {
+ rte_spinlock_t l;
+} vhost_spinlock_t;
+
+static __rte_always_inline void
+vhost_spinlock_init(vhost_spinlock_t *l)
+{
+ rte_spinlock_init(&l->l);
+}
+
+static __rte_always_inline void
+vhost_spinlock_lock(vhost_spinlock_t *l)
+ VHOST_SPINLOCK_ACQUIRES(l)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
+{
+ rte_spinlock_lock(&l->l);
+}
+
+static __rte_always_inline int
+vhost_spinlock_trylock(vhost_spinlock_t *l)
+ VHOST_SPINLOCK_TRYLOCK(1, l)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
+{
+ return rte_spinlock_trylock(&l->l);
+}
+
+static __rte_always_inline void
+vhost_spinlock_unlock(vhost_spinlock_t *l)
+ VHOST_SPINLOCK_RELEASES(l)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
+{
+ rte_spinlock_unlock(&l->l);
+}
+
/**
* Structure contains buffer address, length and descriptor index
* from vring to do scatter RX.
@@ -255,8 +320,7 @@ struct vhost_virtqueue {
bool access_ok;
bool ready;
- rte_spinlock_t access_lock;
-
+ vhost_spinlock_t access_lock;
union {
struct vring_used_elem *shadow_used_split;
@@ -834,6 +898,7 @@ vhost_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old)
static __rte_always_inline void
vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
/* Flush used->idx update before we read avail->flags. */
rte_atomic_thread_fence(__ATOMIC_SEQ_CST);
@@ -872,6 +937,7 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
static __rte_always_inline void
vhost_vring_call_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint16_t old, new, off, off_wrap;
bool signalled_used_valid, kick = false;
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 1d390677fa..caf4cf14b5 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2571,9 +2571,9 @@ vhost_user_iotlb_msg(struct virtio_net **pdev,
len, imsg->perm);
if (is_vring_iotlb(dev, vq, imsg)) {
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
*pdev = dev = translate_ring_addresses(dev, i);
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
}
}
break;
@@ -2588,9 +2588,9 @@ vhost_user_iotlb_msg(struct virtio_net **pdev,
imsg->size);
if (is_vring_iotlb(dev, vq, imsg)) {
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
vring_invalidate(dev, vq);
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
}
}
break;
@@ -2909,6 +2909,7 @@ vhost_user_check_and_alloc_queue_pair(struct virtio_net *dev,
static void
vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
{
unsigned int i = 0;
unsigned int vq_num = 0;
@@ -2917,7 +2918,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
struct vhost_virtqueue *vq = dev->virtqueue[i];
if (vq) {
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
vq_num++;
}
i++;
@@ -2926,6 +2927,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
static void
vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
{
unsigned int i = 0;
unsigned int vq_num = 0;
@@ -2934,7 +2936,7 @@ vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
struct vhost_virtqueue *vq = dev->virtqueue[i];
if (vq) {
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
vq_num++;
}
i++;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 5f432b0d77..25bdb1479d 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -1246,6 +1246,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1441,6 +1442,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -1488,7 +1490,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
vq = dev->virtqueue[queue_id];
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
if (unlikely(!vq->enabled))
goto out_access_unlock;
@@ -1514,7 +1516,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return nb_tx;
}
@@ -1955,11 +1957,11 @@ write_back_completed_descs_packed(struct vhost_virtqueue *vq,
}
static __rte_always_inline uint16_t
-vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id,
+vhost_poll_enqueue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
- struct vhost_virtqueue *vq = dev->virtqueue[queue_id];
struct vhost_async *async = vq->async;
struct async_inflight_info *pkts_info = async->pkts_info;
uint16_t nr_cpl_pkts = 0;
@@ -2050,7 +2052,7 @@ rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
vq = dev->virtqueue[queue_id];
- if (!rte_spinlock_trylock(&vq->access_lock)) {
+ if (vhost_spinlock_trylock(&vq->access_lock) == 0) {
VHOST_LOG_DATA(DEBUG, "(%s) %s: virtqueue %u is busy.\n", dev->ifname, __func__,
queue_id);
return 0;
@@ -2062,10 +2064,10 @@ rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
goto out;
}
- n_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count, dma_id, vchan_id);
+ n_pkts_cpl = vhost_poll_enqueue_completed(dev, vq, pkts, count, dma_id, vchan_id);
out:
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return n_pkts_cpl;
}
@@ -2074,6 +2076,7 @@ uint16_t
rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
{
struct virtio_net *dev = get_device(vid);
struct vhost_virtqueue *vq;
@@ -2104,7 +2107,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
return 0;
}
- n_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count, dma_id, vchan_id);
+ n_pkts_cpl = vhost_poll_enqueue_completed(dev, vq, pkts, count, dma_id, vchan_id);
return n_pkts_cpl;
}
@@ -2132,7 +2135,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,
vq = dev->virtqueue[queue_id];
- rte_spinlock_lock(&vq->access_lock);
+ vhost_spinlock_lock(&vq->access_lock);
if (unlikely(!vq->enabled || !vq->async))
goto out_access_unlock;
@@ -2160,7 +2163,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,
vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
return nb_tx;
}
@@ -2679,6 +2682,7 @@ static uint16_t
virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint16_t i;
uint16_t free_entries;
@@ -2774,6 +2778,7 @@ static uint16_t
virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2783,6 +2788,7 @@ static uint16_t
virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -2982,6 +2988,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count,
bool legacy_ol_flags)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -3025,6 +3032,7 @@ static uint16_t
virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3034,6 +3042,7 @@ static uint16_t
virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3065,7 +3074,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
vq = dev->virtqueue[queue_id];
- if (unlikely(rte_spinlock_trylock(&vq->access_lock) == 0))
+ if (unlikely(vhost_spinlock_trylock(&vq->access_lock) == 0))
return 0;
if (unlikely(!vq->enabled)) {
@@ -3134,7 +3143,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
- rte_spinlock_unlock(&vq->access_lock);
+ vhost_spinlock_unlock(&vq->access_lock);
if (unlikely(rarp_mbuf != NULL))
count += 1;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH 3/5] vhost: fix async access
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
2022-03-28 12:17 ` [RFC PATCH 1/5] vhost: fix missing virtqueue lock protection David Marchand
2022-03-28 12:17 ` [RFC PATCH 2/5] vhost: annotate virtqueue access lock David Marchand
@ 2022-03-28 12:17 ` David Marchand
2022-03-28 12:17 ` [RFC PATCH 4/5] vhost: annotate async locking requirement David Marchand
` (6 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-28 12:17 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, stable, Patrick Fu, Jiayu Hu
vq->async is protected with vq->access_lock.
Fixes: eb666d24085f ("vhost: fix async unregister deadlock")
Fixes: 0c0935c5f794 ("vhost: allow to check in-flight packets for async vhost")
Cc: stable@dpdk.org
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
This has been reported by Jiayu too:
https://patchwork.dpdk.org/project/dpdk/patch/20220328020754.1155063-1-jiayu.hu@intel.com/
---
lib/vhost/vhost.c | 25 ++++++++++---------------
1 file changed, 10 insertions(+), 15 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 863cd9131e..9e6ca506ff 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1753,27 +1753,23 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- ret = 0;
-
- if (!vq->async)
- return ret;
-
if (vhost_spinlock_trylock(&vq->access_lock) == 0) {
VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel, virtqueue busy.\n",
dev->ifname);
- return -1;
+ return ret;
}
- if (vq->async->pkts_inflight_n) {
+ if (!vq->async) {
+ ret = 0;
+ } else if (vq->async->pkts_inflight_n) {
VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel.\n", dev->ifname);
VHOST_LOG_CONFIG(ERR, "(%s) inflight packets must be completed before unregistration.\n",
dev->ifname);
- ret = -1;
- goto out;
+ } else {
+ vhost_free_async_mem(vq);
+ ret = 0;
}
- vhost_free_async_mem(vq);
-out:
vhost_spinlock_unlock(&vq->access_lock);
return ret;
@@ -1891,9 +1887,6 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- if (!vq->async)
- return ret;
-
if (vhost_spinlock_trylock(&vq->access_lock) == 0) {
VHOST_LOG_CONFIG(DEBUG,
"(%s) failed to check in-flight packets. virtqueue busy.\n",
@@ -1901,7 +1894,9 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
return ret;
}
- ret = vq->async->pkts_inflight_n;
+ if (vq->async)
+ ret = vq->async->pkts_inflight_n;
+
vhost_spinlock_unlock(&vq->access_lock);
return ret;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH 4/5] vhost: annotate async locking requirement
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
` (2 preceding siblings ...)
2022-03-28 12:17 ` [RFC PATCH 3/5] vhost: fix async access David Marchand
@ 2022-03-28 12:17 ` David Marchand
2022-03-28 12:17 ` [RFC PATCH 5/5] vhost: annotate IOTLB locks David Marchand
` (5 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-28 12:17 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia
Instrument that the async field of a virtqueue must be protected by a
lock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.c | 14 +++++++++-----
lib/vhost/vhost.h | 5 ++++-
lib/vhost/vhost_user.c | 2 ++
lib/vhost/virtio_net.c | 14 ++++++++++++++
4 files changed, 29 insertions(+), 6 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 9e6ca506ff..ea28323367 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -334,6 +334,7 @@ cleanup_device(struct virtio_net *dev, int destroy)
static void
vhost_free_async_mem(struct vhost_virtqueue *vq)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
if (!vq->async)
return;
@@ -352,6 +353,7 @@ vhost_free_async_mem(struct vhost_virtqueue *vq)
void
free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
{
if (vq_is_packed(dev))
rte_free(vq->shadow_used_packed);
@@ -1622,10 +1624,10 @@ rte_vhost_extern_callback_register(int vid,
}
static __rte_always_inline int
-async_channel_register(int vid, uint16_t queue_id)
+async_channel_register(struct virtio_net *dev, struct vhost_virtqueue *vq,
+ uint16_t queue_id)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
- struct virtio_net *dev = get_device(vid);
- struct vhost_virtqueue *vq = dev->virtqueue[queue_id];
struct vhost_async *async;
int node = vq->numa_node;
@@ -1709,7 +1711,7 @@ rte_vhost_async_channel_register(int vid, uint16_t queue_id)
return -1;
vhost_spinlock_lock(&vq->access_lock);
- ret = async_channel_register(vid, queue_id);
+ ret = async_channel_register(dev, vq, queue_id);
vhost_spinlock_unlock(&vq->access_lock);
return ret;
@@ -1717,6 +1719,7 @@ rte_vhost_async_channel_register(int vid, uint16_t queue_id)
int
rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
{
struct vhost_virtqueue *vq;
struct virtio_net *dev = get_device(vid);
@@ -1732,7 +1735,7 @@ rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
if (unlikely(vq == NULL || !dev->async_copy))
return -1;
- return async_channel_register(vid, queue_id);
+ return async_channel_register(dev, vq, queue_id);
}
int
@@ -1777,6 +1780,7 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
int
rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
{
struct vhost_virtqueue *vq;
struct virtio_net *dev = get_device(vid);
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 619f073fb2..4b301ec152 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -96,6 +96,8 @@
__attribute__((no_thread_safety_analysis))
#define VHOST_LOCKABLE \
__attribute__((lockable))
+#define VHOST_GUARDED \
+ __attribute__((guarded_var))
#define VHOST_SPINLOCK_REQUIRES(...) \
__attribute__((exclusive_locks_required(__VA_ARGS__)))
@@ -109,6 +111,7 @@
#else
#define VHOST_NO_THREAD_SAFETY_ANALYSIS
#define VHOST_LOCKABLE
+#define VHOST_GUARDED
#define VHOST_SPINLOCK_REQUIRES(...)
#define VHOST_SPINLOCK_ACQUIRES(...)
@@ -363,7 +366,7 @@ struct vhost_virtqueue {
struct rte_vhost_resubmit_info *resubmit_inflight;
uint64_t global_counter;
- struct vhost_async *async;
+ struct vhost_async *async VHOST_GUARDED;
int notif_enable;
#define VIRTIO_UNINITIALIZED_NOTIF (-1)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index caf4cf14b5..35f1e23995 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2199,6 +2199,8 @@ static int
vhost_user_set_vring_enable(struct virtio_net **pdev,
struct vhu_msg_context *ctx,
int main_fd __rte_unused)
+ /* vq->access_lock is taken in vhost_user_lock_all_queue_pairs() */
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
{
struct virtio_net *dev = *pdev;
bool enable = !!ctx->msg.payload.state.num;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 25bdb1479d..9bdba992dd 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -51,6 +51,7 @@ static __rte_always_inline int64_t
vhost_async_dma_transfer_one(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t flag_idx,
struct vhost_iov_iter *pkt)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
uint16_t ring_mask = dma_info->ring_mask;
@@ -99,6 +100,7 @@ static __rte_always_inline uint16_t
vhost_async_dma_transfer(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t head_idx,
struct vhost_iov_iter *pkts, uint16_t nr_pkts)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
int64_t ret, nr_copies = 0;
@@ -1000,6 +1002,7 @@ static __rte_always_inline int
async_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1057,6 +1060,7 @@ static __rte_always_inline int
mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1186,6 +1190,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkt,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1417,6 +1422,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1541,6 +1547,7 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
static __rte_always_inline uint16_t
async_get_first_inflight_pkt_idx(struct vhost_virtqueue *vq)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct vhost_async *async = vq->async;
@@ -1587,6 +1594,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1691,6 +1699,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs,
uint16_t *nr_buffers)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1748,6 +1757,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1766,6 +1776,7 @@ virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
uint32_t nr_err, uint32_t *pkt_idx)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint16_t descs_err = 0;
uint16_t buffers_err = 0;
@@ -1793,6 +1804,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
uint32_t pkt_idx = 0;
uint32_t remained = count;
@@ -1863,6 +1875,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
static __rte_always_inline void
write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t nr_left = n_descs;
@@ -1895,6 +1908,7 @@ write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
static __rte_always_inline void
write_back_completed_descs_packed(struct vhost_virtqueue *vq,
uint16_t n_buffers)
+ VHOST_SPINLOCK_REQUIRES(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t from = async->last_buffer_idx_packed;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH 5/5] vhost: annotate IOTLB locks
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
` (3 preceding siblings ...)
2022-03-28 12:17 ` [RFC PATCH 4/5] vhost: annotate async locking requirement David Marchand
@ 2022-03-28 12:17 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
` (4 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-28 12:17 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia
The IOTLB code uses r/w locks.
Introduce a wrapper around this type of locks and annotate iotlb_lock
and iotlb_pending_lock.
clang does not support conditionally held locks, so always take iotlb
locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
vdpa and vhost_crypto code are annotated though they end up not taking
a IOTLB lock and have been marked with a FIXME.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/iotlb.c | 36 +++++++--------
lib/vhost/iotlb.h | 24 ----------
lib/vhost/vdpa.c | 1 +
lib/vhost/vhost.c | 16 +++----
lib/vhost/vhost.h | 94 +++++++++++++++++++++++++++++++++++-----
lib/vhost/vhost_crypto.c | 7 +++
lib/vhost/vhost_user.c | 16 +++++--
lib/vhost/virtio_net.c | 49 ++++++++++++++++-----
8 files changed, 164 insertions(+), 79 deletions(-)
diff --git a/lib/vhost/iotlb.c b/lib/vhost/iotlb.c
index 5a5ba8b82a..e5367d3af9 100644
--- a/lib/vhost/iotlb.c
+++ b/lib/vhost/iotlb.c
@@ -30,14 +30,14 @@ vhost_user_iotlb_pending_remove_all(struct vhost_virtqueue *vq)
{
struct vhost_iotlb_entry *node, *temp_node;
- rte_rwlock_write_lock(&vq->iotlb_pending_lock);
+ vhost_rwlock_write_lock(&vq->iotlb_pending_lock);
RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_pending_list, next, temp_node) {
TAILQ_REMOVE(&vq->iotlb_pending_list, node, next);
rte_mempool_put(vq->iotlb_pool, node);
}
- rte_rwlock_write_unlock(&vq->iotlb_pending_lock);
+ vhost_rwlock_write_unlock(&vq->iotlb_pending_lock);
}
bool
@@ -47,7 +47,7 @@ vhost_user_iotlb_pending_miss(struct vhost_virtqueue *vq, uint64_t iova,
struct vhost_iotlb_entry *node;
bool found = false;
- rte_rwlock_read_lock(&vq->iotlb_pending_lock);
+ vhost_rwlock_read_lock(&vq->iotlb_pending_lock);
TAILQ_FOREACH(node, &vq->iotlb_pending_list, next) {
if ((node->iova == iova) && (node->perm == perm)) {
@@ -56,7 +56,7 @@ vhost_user_iotlb_pending_miss(struct vhost_virtqueue *vq, uint64_t iova,
}
}
- rte_rwlock_read_unlock(&vq->iotlb_pending_lock);
+ vhost_rwlock_read_unlock(&vq->iotlb_pending_lock);
return found;
}
@@ -89,11 +89,11 @@ vhost_user_iotlb_pending_insert(struct virtio_net *dev, struct vhost_virtqueue *
node->iova = iova;
node->perm = perm;
- rte_rwlock_write_lock(&vq->iotlb_pending_lock);
+ vhost_rwlock_write_lock(&vq->iotlb_pending_lock);
TAILQ_INSERT_TAIL(&vq->iotlb_pending_list, node, next);
- rte_rwlock_write_unlock(&vq->iotlb_pending_lock);
+ vhost_rwlock_write_unlock(&vq->iotlb_pending_lock);
}
void
@@ -102,7 +102,7 @@ vhost_user_iotlb_pending_remove(struct vhost_virtqueue *vq,
{
struct vhost_iotlb_entry *node, *temp_node;
- rte_rwlock_write_lock(&vq->iotlb_pending_lock);
+ vhost_rwlock_write_lock(&vq->iotlb_pending_lock);
RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_pending_list, next,
temp_node) {
@@ -116,7 +116,7 @@ vhost_user_iotlb_pending_remove(struct vhost_virtqueue *vq,
rte_mempool_put(vq->iotlb_pool, node);
}
- rte_rwlock_write_unlock(&vq->iotlb_pending_lock);
+ vhost_rwlock_write_unlock(&vq->iotlb_pending_lock);
}
static void
@@ -124,7 +124,7 @@ vhost_user_iotlb_cache_remove_all(struct vhost_virtqueue *vq)
{
struct vhost_iotlb_entry *node, *temp_node;
- rte_rwlock_write_lock(&vq->iotlb_lock);
+ vhost_rwlock_write_lock(&vq->iotlb_lock);
RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) {
TAILQ_REMOVE(&vq->iotlb_list, node, next);
@@ -133,7 +133,7 @@ vhost_user_iotlb_cache_remove_all(struct vhost_virtqueue *vq)
vq->iotlb_cache_nr = 0;
- rte_rwlock_write_unlock(&vq->iotlb_lock);
+ vhost_rwlock_write_unlock(&vq->iotlb_lock);
}
static void
@@ -142,7 +142,7 @@ vhost_user_iotlb_cache_random_evict(struct vhost_virtqueue *vq)
struct vhost_iotlb_entry *node, *temp_node;
int entry_idx;
- rte_rwlock_write_lock(&vq->iotlb_lock);
+ vhost_rwlock_write_lock(&vq->iotlb_lock);
entry_idx = rte_rand() % vq->iotlb_cache_nr;
@@ -156,7 +156,7 @@ vhost_user_iotlb_cache_random_evict(struct vhost_virtqueue *vq)
entry_idx--;
}
- rte_rwlock_write_unlock(&vq->iotlb_lock);
+ vhost_rwlock_write_unlock(&vq->iotlb_lock);
}
void
@@ -190,7 +190,7 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq
new_node->size = size;
new_node->perm = perm;
- rte_rwlock_write_lock(&vq->iotlb_lock);
+ vhost_rwlock_write_lock(&vq->iotlb_lock);
TAILQ_FOREACH(node, &vq->iotlb_list, next) {
/*
@@ -213,7 +213,7 @@ vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq
unlock:
vhost_user_iotlb_pending_remove(vq, iova, size, perm);
- rte_rwlock_write_unlock(&vq->iotlb_lock);
+ vhost_rwlock_write_unlock(&vq->iotlb_lock);
}
@@ -226,7 +226,7 @@ vhost_user_iotlb_cache_remove(struct vhost_virtqueue *vq,
if (unlikely(!size))
return;
- rte_rwlock_write_lock(&vq->iotlb_lock);
+ vhost_rwlock_write_lock(&vq->iotlb_lock);
RTE_TAILQ_FOREACH_SAFE(node, &vq->iotlb_list, next, temp_node) {
/* Sorted list */
@@ -240,7 +240,7 @@ vhost_user_iotlb_cache_remove(struct vhost_virtqueue *vq,
}
}
- rte_rwlock_write_unlock(&vq->iotlb_lock);
+ vhost_rwlock_write_unlock(&vq->iotlb_lock);
}
uint64_t
@@ -312,8 +312,8 @@ vhost_user_iotlb_init(struct virtio_net *dev, int vq_index)
socket = 0;
#endif
- rte_rwlock_init(&vq->iotlb_lock);
- rte_rwlock_init(&vq->iotlb_pending_lock);
+ vhost_rwlock_init(&vq->iotlb_lock);
+ vhost_rwlock_init(&vq->iotlb_pending_lock);
TAILQ_INIT(&vq->iotlb_list);
TAILQ_INIT(&vq->iotlb_pending_list);
diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h
index 8d0ff7473b..96ec4d608f 100644
--- a/lib/vhost/iotlb.h
+++ b/lib/vhost/iotlb.h
@@ -9,30 +9,6 @@
#include "vhost.h"
-static __rte_always_inline void
-vhost_user_iotlb_rd_lock(struct vhost_virtqueue *vq)
-{
- rte_rwlock_read_lock(&vq->iotlb_lock);
-}
-
-static __rte_always_inline void
-vhost_user_iotlb_rd_unlock(struct vhost_virtqueue *vq)
-{
- rte_rwlock_read_unlock(&vq->iotlb_lock);
-}
-
-static __rte_always_inline void
-vhost_user_iotlb_wr_lock(struct vhost_virtqueue *vq)
-{
- rte_rwlock_write_lock(&vq->iotlb_lock);
-}
-
-static __rte_always_inline void
-vhost_user_iotlb_wr_unlock(struct vhost_virtqueue *vq)
-{
- rte_rwlock_write_unlock(&vq->iotlb_lock);
-}
-
void vhost_user_iotlb_cache_insert(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t uaddr,
uint64_t size, uint8_t perm);
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index 8fa2153023..406f13288b 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -130,6 +130,7 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int
rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS /* FIXME: requires iotlb_lock? */
{
struct virtio_net *dev = get_device(vid);
uint16_t idx, idx_m, desc_id;
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index ea28323367..466a24dabe 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -50,7 +50,7 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
* which could cause a deadlock with QEMU if an IOTLB update
* is being handled. We can safely unlock here to avoid it.
*/
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_rwlock_read_unlock(&vq->iotlb_lock);
vhost_user_iotlb_pending_insert(dev, vq, iova, perm);
if (vhost_user_iotlb_miss(dev, iova, perm)) {
@@ -59,7 +59,7 @@ __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
vhost_user_iotlb_pending_remove(vq, iova, 1, perm);
}
- vhost_user_iotlb_rd_lock(vq);
+ vhost_rwlock_read_lock(&vq->iotlb_lock);
}
return 0;
@@ -383,6 +383,7 @@ free_device(struct virtio_net *dev)
static __rte_always_inline int
log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
if (likely(!(vq->ring_addrs.flags & (1 << VHOST_VRING_F_LOG))))
return 0;
@@ -434,6 +435,7 @@ translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
/* Caller should have iotlb_lock read-locked */
static int
vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -473,6 +475,7 @@ vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
/* Caller should have iotlb_lock read-locked */
static int
vring_translate_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -506,7 +509,6 @@ vring_translate_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
int
vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
{
-
if (!(dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)))
return -1;
@@ -527,19 +529,13 @@ vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
}
void
-vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+vring_invalidate(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq)
{
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_lock(vq);
-
vq->access_ok = false;
vq->desc = NULL;
vq->avail = NULL;
vq->used = NULL;
vq->log_guest_addr = 0;
-
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_unlock(vq);
}
static void
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 4b301ec152..08ace9994a 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -108,6 +108,19 @@
#define VHOST_SPINLOCK_RELEASES(...) \
__attribute__((unlock_function(__VA_ARGS__)))
+#define VHOST_RDLOCK_REQUIRES(...) \
+ __attribute__((shared_locks_required(__VA_ARGS__)))
+#define VHOST_RDLOCK_ACQUIRES(...) \
+ __attribute__((shared_lock_function(__VA_ARGS__)))
+#define VHOST_RDLOCK_RELEASES(...) \
+ __attribute__((unlock_function(__VA_ARGS__)))
+#define VHOST_WRLOCK_REQUIRES(...) \
+ __attribute__((exclusive_locks_required(__VA_ARGS__)))
+#define VHOST_WRLOCK_ACQUIRES(...) \
+ __attribute__((exclusive_lock_function(__VA_ARGS__)))
+#define VHOST_WRLOCK_RELEASES(...) \
+ __attribute__((unlock_function(__VA_ARGS__)))
+
#else
#define VHOST_NO_THREAD_SAFETY_ANALYSIS
#define VHOST_LOCKABLE
@@ -117,6 +130,13 @@
#define VHOST_SPINLOCK_ACQUIRES(...)
#define VHOST_SPINLOCK_TRYLOCK(...)
#define VHOST_SPINLOCK_RELEASES(...)
+
+#define VHOST_RDLOCK_ACQUIRES(...)
+#define VHOST_RDLOCK_REQUIRES(...)
+#define VHOST_RDLOCK_RELEASES(...)
+#define VHOST_WRLOCK_REQUIRES(...)
+#define VHOST_WRLOCK_ACQUIRES(...)
+#define VHOST_WRLOCK_RELEASES(...)
#endif
typedef struct VHOST_LOCKABLE {
@@ -153,6 +173,48 @@ vhost_spinlock_unlock(vhost_spinlock_t *l)
rte_spinlock_unlock(&l->l);
}
+typedef struct VHOST_LOCKABLE {
+ rte_rwlock_t l;
+} vhost_rwlock_t;
+
+static __rte_always_inline void
+vhost_rwlock_init(vhost_rwlock_t *l)
+{
+ rte_rwlock_init(&l->l);
+}
+
+static __rte_always_inline void
+vhost_rwlock_read_lock(vhost_rwlock_t *l)
+ VHOST_RDLOCK_ACQUIRES(l)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
+{
+ rte_rwlock_read_lock(&l->l);
+}
+
+static __rte_always_inline void
+vhost_rwlock_read_unlock(vhost_rwlock_t *l)
+ VHOST_RDLOCK_RELEASES(l)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
+{
+ rte_rwlock_read_unlock(&l->l);
+}
+
+static __rte_always_inline void
+vhost_rwlock_write_lock(vhost_rwlock_t *l)
+ VHOST_WRLOCK_ACQUIRES(l)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
+{
+ rte_rwlock_write_lock(&l->l);
+}
+
+static __rte_always_inline void
+vhost_rwlock_write_unlock(vhost_rwlock_t *l)
+ VHOST_WRLOCK_RELEASES(l)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS
+{
+ rte_rwlock_write_unlock(&l->l);
+}
+
/**
* Structure contains buffer address, length and descriptor index
* from vring to do scatter RX.
@@ -346,8 +408,8 @@ struct vhost_virtqueue {
uint64_t log_guest_addr;
struct log_cache_entry *log_cache;
- rte_rwlock_t iotlb_lock;
- rte_rwlock_t iotlb_pending_lock;
+ vhost_rwlock_t iotlb_lock;
+ vhost_rwlock_t iotlb_pending_lock;
struct rte_mempool *iotlb_pool;
TAILQ_HEAD(, vhost_iotlb_entry) iotlb_list;
TAILQ_HEAD(, vhost_iotlb_entry) iotlb_pending_list;
@@ -592,12 +654,15 @@ void __vhost_log_cache_write(struct virtio_net *dev,
uint64_t addr, uint64_t len);
void __vhost_log_cache_write_iova(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock);
void __vhost_log_cache_sync(struct virtio_net *dev,
struct vhost_virtqueue *vq);
+
void __vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len);
void __vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock);
static __rte_always_inline void
vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len)
@@ -647,6 +712,7 @@ vhost_log_used_vring(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -660,6 +726,7 @@ vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -863,18 +930,23 @@ struct rte_vhost_device_ops const *vhost_driver_callback_get(const char *path);
void vhost_backend_cleanup(struct virtio_net *dev);
uint64_t __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t *len, uint8_t perm);
-void *vhost_alloc_copy_ind_table(struct virtio_net *dev,
- struct vhost_virtqueue *vq,
- uint64_t desc_addr, uint64_t desc_len);
-int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq);
+ uint64_t iova, uint64_t *len, uint8_t perm)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock);
+void *vhost_alloc_copy_ind_table(struct virtio_net *dev, struct vhost_virtqueue *vq,
+ uint64_t desc_addr, uint64_t desc_len)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock);
+int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock);
uint64_t translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t log_addr);
-void vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq);
+ uint64_t log_addr)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock);
+void vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_WRLOCK_REQUIRES(vq->iotlb_lock);
static __rte_always_inline uint64_t
vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *len, uint8_t perm)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
if (!(dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)))
return rte_vhost_va_from_guest_pa(dev->mem, iova, len);
diff --git a/lib/vhost/vhost_crypto.c b/lib/vhost/vhost_crypto.c
index b1c0eb6a0f..2d0a1250fb 100644
--- a/lib/vhost/vhost_crypto.c
+++ b/lib/vhost/vhost_crypto.c
@@ -506,6 +506,7 @@ static __rte_always_inline struct virtio_crypto_inhdr *
reach_inhdr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ VHOST_RDLOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct virtio_crypto_inhdr *inhdr;
struct vhost_crypto_desc *last = head + (max_n_descs - 1);
@@ -552,6 +553,7 @@ static __rte_always_inline void *
get_data_ptr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *cur_desc,
uint8_t perm)
+ VHOST_RDLOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
void *data;
uint64_t dlen = cur_desc->len;
@@ -570,6 +572,7 @@ copy_data(void *dst_data, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
struct vhost_crypto_desc **cur_desc,
uint32_t size, uint32_t max_n_descs)
+ VHOST_RDLOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = *cur_desc;
uint64_t remain, addr, dlen, len;
@@ -718,6 +721,7 @@ prepare_write_back_data(struct vhost_crypto_data_req *vc_req,
uint32_t offset,
uint64_t write_back_len,
uint32_t max_n_descs)
+ VHOST_RDLOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_writeback_data *wb_data, *head;
struct vhost_crypto_desc *desc = *cur_desc;
@@ -838,6 +842,7 @@ prepare_sym_cipher_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_cipher_data_req *cipher,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ VHOST_RDLOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head;
struct vhost_crypto_writeback_data *ewb = NULL;
@@ -990,6 +995,7 @@ prepare_sym_chain_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_alg_chain_data_req *chain,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ VHOST_RDLOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head, *digest_desc;
struct vhost_crypto_writeback_data *ewb = NULL, *ewb2 = NULL;
@@ -1172,6 +1178,7 @@ vhost_crypto_process_one_req(struct vhost_crypto *vcrypto,
struct vhost_virtqueue *vq, struct rte_crypto_op *op,
struct vring_desc *head, struct vhost_crypto_desc *descs,
uint16_t desc_idx)
+ VHOST_NO_THREAD_SAFETY_ANALYSIS /* FIXME: requires iotlb_lock? */
{
struct vhost_crypto_data_req *vc_req = rte_mbuf_to_priv(op->sym->m_src);
struct rte_cryptodev_sym_session *session;
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 35f1e23995..0482c00801 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -754,10 +754,10 @@ ring_addr_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) {
uint64_t vva;
- vhost_user_iotlb_rd_lock(vq);
+ vhost_rwlock_read_lock(&vq->iotlb_lock);
vva = vhost_iova_to_vva(dev, vq, ra,
size, VHOST_ACCESS_RW);
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_rwlock_read_unlock(&vq->iotlb_lock);
return vva;
}
@@ -770,9 +770,9 @@ log_addr_to_gpa(struct virtio_net *dev, struct vhost_virtqueue *vq)
{
uint64_t log_gpa;
- vhost_user_iotlb_rd_lock(vq);
+ vhost_rwlock_read_lock(&vq->iotlb_lock);
log_gpa = translate_log_addr(dev, vq, vq->ring_addrs.log_guest_addr);
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_rwlock_read_unlock(&vq->iotlb_lock);
return log_gpa;
}
@@ -927,7 +927,9 @@ vhost_user_set_vring_addr(struct virtio_net **pdev,
*/
memcpy(&vq->ring_addrs, addr, sizeof(*addr));
+ vhost_rwlock_write_lock(&vq->iotlb_lock);
vring_invalidate(dev, vq);
+ vhost_rwlock_write_unlock(&vq->iotlb_lock);
if ((vq->enabled && (dev->features &
(1ULL << VHOST_USER_F_PROTOCOL_FEATURES))) ||
@@ -1437,7 +1439,9 @@ vhost_user_set_mem_table(struct virtio_net **pdev,
* need to be translated again as virtual addresses have
* changed.
*/
+ vhost_rwlock_write_lock(&vq->iotlb_lock);
vring_invalidate(dev, vq);
+ vhost_rwlock_write_unlock(&vq->iotlb_lock);
dev = translate_ring_addresses(dev, i);
if (!dev) {
@@ -2186,7 +2190,9 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
vhost_user_iotlb_flush_all(vq);
+ vhost_rwlock_write_lock(&vq->iotlb_lock);
vring_invalidate(dev, vq);
+ vhost_rwlock_write_unlock(&vq->iotlb_lock);
return RTE_VHOST_MSG_RESULT_REPLY;
}
@@ -2591,7 +2597,9 @@ vhost_user_iotlb_msg(struct virtio_net **pdev,
if (is_vring_iotlb(dev, vq, imsg)) {
vhost_spinlock_lock(&vq->access_lock);
+ vhost_rwlock_write_lock(&vq->iotlb_lock);
vring_invalidate(dev, vq);
+ vhost_rwlock_write_unlock(&vq->iotlb_lock);
vhost_spinlock_unlock(&vq->access_lock);
}
}
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 9bdba992dd..ec342e772d 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -180,6 +180,7 @@ vhost_async_dma_check_completed(struct virtio_net *dev, int16_t dma_id, uint16_t
static inline void
do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
struct batch_copy_elem *elem = vq->batch_copy_elems;
uint16_t count = vq->batch_copy_nb_elems;
@@ -526,6 +527,7 @@ vhost_shadow_enqueue_single_packed(struct virtio_net *dev,
uint16_t *id,
uint16_t *count,
uint16_t num_buffers)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
vhost_shadow_enqueue_packed(vq, len, id, count, num_buffers);
@@ -607,6 +609,7 @@ static __rte_always_inline int
map_one_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint64_t desc_iova, uint64_t desc_len, uint8_t perm)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t vec_id = *vec_idx;
@@ -644,6 +647,7 @@ fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t avail_idx, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint16_t *desc_chain_head,
uint32_t *desc_chain_len, uint8_t perm)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
uint16_t vec_id = *vec_idx;
@@ -727,6 +731,7 @@ reserve_avail_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t size, struct buf_vector *buf_vec,
uint16_t *num_buffers, uint16_t avail_head,
uint16_t *nr_vec)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t cur_idx;
uint16_t vec_idx = 0;
@@ -777,6 +782,7 @@ fill_vec_buf_packed_indirect(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct vring_packed_desc *desc, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint32_t *len, uint8_t perm)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t i;
uint32_t nr_descs;
@@ -835,6 +841,7 @@ fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t avail_idx, uint16_t *desc_count,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint16_t *buf_id, uint32_t *len, uint8_t perm)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -900,6 +907,7 @@ static __rte_noinline void
copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec,
struct virtio_net_hdr_mrg_rxbuf *hdr)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint64_t len;
uint64_t remain = dev->vhost_hlen;
@@ -1036,6 +1044,7 @@ static __rte_always_inline void
sync_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_addr, uint64_t buf_iova, uint32_t cpy_len)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
@@ -1061,6 +1070,7 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1191,6 +1201,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1252,6 +1263,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1309,6 +1321,7 @@ virtio_dev_rx_sync_batch_check(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -1360,6 +1373,7 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
@@ -1401,6 +1415,7 @@ static __rte_always_inline int
virtio_dev_rx_sync_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint64_t desc_addrs[PACKED_BATCH_SIZE];
uint64_t lens[PACKED_BATCH_SIZE];
@@ -1423,6 +1438,7 @@ virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1449,6 +1465,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -1501,8 +1518,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
if (unlikely(!vq->enabled))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_rwlock_read_lock(&vq->iotlb_lock);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -1518,8 +1534,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
nb_tx = virtio_dev_rx_split(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_rwlock_read_unlock(&vq->iotlb_lock);
out_access_unlock:
vhost_spinlock_unlock(&vq->access_lock);
@@ -1595,6 +1610,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1700,6 +1716,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
uint16_t *nr_descs,
uint16_t *nr_buffers)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1758,6 +1775,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1805,6 +1823,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint32_t remained = count;
@@ -2154,8 +2173,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,
if (unlikely(!vq->enabled || !vq->async))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_rwlock_read_lock(&vq->iotlb_lock);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -2173,8 +2191,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,
pkts, count, dma_id, vchan_id);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_rwlock_read_unlock(&vq->iotlb_lock);
out_access_unlock:
vhost_spinlock_unlock(&vq->access_lock);
@@ -2697,6 +2714,7 @@ virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t i;
uint16_t free_entries;
@@ -2793,6 +2811,7 @@ virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2803,6 +2822,7 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -2814,6 +2834,7 @@ vhost_reserve_avail_batch_packed(struct virtio_net *dev,
uint16_t avail_idx,
uintptr_t *desc_addrs,
uint16_t *ids)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
bool wrap = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -2883,6 +2904,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts,
bool legacy_ol_flags)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t avail_idx = vq->last_avail_idx;
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
@@ -2929,6 +2951,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *buf_id,
uint16_t *desc_count,
bool legacy_ol_flags)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -2972,6 +2995,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mempool *mbuf_pool,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3003,6 +3027,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
uint32_t count,
bool legacy_ol_flags)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -3047,6 +3072,7 @@ virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3057,6 +3083,7 @@ virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
VHOST_SPINLOCK_REQUIRES(vq->access_lock)
+ VHOST_RDLOCK_REQUIRES(vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3096,8 +3123,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_rwlock_read_lock(&vq->iotlb_lock);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -3153,8 +3179,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
}
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_rwlock_read_unlock(&vq->iotlb_lock);
out_access_unlock:
vhost_spinlock_unlock(&vq->access_lock);
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 0/9] vhost lock annotations
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
` (4 preceding siblings ...)
2022-03-28 12:17 ` [RFC PATCH 5/5] vhost: annotate IOTLB locks David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 1/9] vhost: fix missing virtqueue lock protection David Marchand
` (9 more replies)
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
` (3 subsequent siblings)
9 siblings, 10 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vhost internals involves multiple locks to protect data access by
multiple threads.
This series is a try at using clang thread safety checks [1] to catch
issues during compilation: EAL spinlock and rwlock are annotated and
vhost code is instrumented so that clang can statically check
correctness.
This is still a work in progress (some documentation and a release note
update are missing).
Those annotations are quite heavy to maintain because the full path of
code must be annotated, but I think it is worth using.
1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
--
David Marchand
Changes since RFC v1:
- Cc'd people who have pending patches for vhost,
- moved annotations to EAL and removed wrappers in vhost,
- as a result of moving to EAL, this series will be tested against
the main repo, so patch 1 has been kept as part of the series
even if already applied to next-virtio,
- refined/split patches and annotated all spinlocks in vhost,
David Marchand (8):
eal: annotate spinlock and rwlock
vhost: annotate virtqueue access lock
vhost: fix async access
vhost: annotate async acesses
vhost: annotate need reply handling
vhost: annotate VDPA device list accesses
vhost: annotate IOTLB locks
vhost: enable lock check
Maxime Coquelin (1):
vhost: fix missing virtqueue lock protection
drivers/meson.build | 5 ++
lib/eal/arm/include/rte_rwlock.h | 4 ++
lib/eal/arm/include/rte_spinlock.h | 6 ++
lib/eal/include/generic/rte_rwlock.h | 21 ++++---
lib/eal/include/generic/rte_spinlock.h | 19 ++++--
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 67 +++++++++++++++++++++
lib/eal/ppc/include/rte_rwlock.h | 4 ++
lib/eal/ppc/include/rte_spinlock.h | 9 +++
lib/eal/x86/include/rte_rwlock.h | 4 ++
lib/eal/x86/include/rte_spinlock.h | 9 +++
lib/meson.build | 5 ++
lib/vhost/iotlb.h | 8 +++
lib/vhost/meson.build | 2 +
lib/vhost/vdpa.c | 18 +++---
lib/vhost/vhost.c | 54 +++++++++--------
lib/vhost/vhost.h | 26 ++++++---
lib/vhost/vhost_crypto.c | 7 +++
lib/vhost/vhost_user.c | 6 ++
lib/vhost/virtio_net.c | 80 ++++++++++++++++++++------
20 files changed, 287 insertions(+), 68 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 1/9] vhost: fix missing virtqueue lock protection
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock David Marchand
` (8 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding,
stable, Yuanhan Liu, Stefan Hajnoczi
From: Maxime Coquelin <maxime.coquelin@redhat.com>
This patch ensures virtqueue metadata are not being
modified while rte_vhost_vring_call() is executed.
Fixes: 6c299bb7322f ("vhost: introduce vring call API")
Cc: stable@dpdk.org
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index bc88148347..2f96a28dac 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1291,11 +1291,15 @@ rte_vhost_vring_call(int vid, uint16_t vring_idx)
if (!vq)
return -1;
+ rte_spinlock_lock(&vq->access_lock);
+
if (vq_is_packed(dev))
vhost_vring_call_packed(dev, vq);
else
vhost_vring_call_split(dev, vq);
+ rte_spinlock_unlock(&vq->access_lock);
+
return 0;
}
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 1/9] vhost: fix missing virtqueue lock protection David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-31 9:22 ` David Marchand
2022-04-04 6:21 ` Stephen Hemminger
2022-03-30 13:49 ` [RFC PATCH v2 3/9] vhost: annotate virtqueue access lock David Marchand
` (7 subsequent siblings)
9 siblings, 2 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding,
Ruifeng Wang, Jan Viktorin, David Christensen, Bruce Richardson,
Konstantin Ananyev
clang offers some thread safety checks, statically verifying that locks
are taken and released in the code.
To use those checks, the full code leading to taking or releasing locks
must be annotated with some attributes.
Wrap those attributes into our own set of macros.
Only rwlock and the "normal" spinlock are instrumented for now.
A component may enable this check by setting annotate_locks = true
in its meson.build.
Note: those checks might be of interest out of DPDK, but it
requires that the including application locks are annotated.
On the other hand, applications out there might have been using
those same checks.
To be on the safe side, keep this instrumentation under a
RTE_ANNOTATE_LOCKS internal build flag.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
drivers/meson.build | 5 ++
lib/eal/arm/include/rte_rwlock.h | 4 ++
lib/eal/arm/include/rte_spinlock.h | 6 +++
lib/eal/include/generic/rte_rwlock.h | 21 +++++---
lib/eal/include/generic/rte_spinlock.h | 19 +++++---
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 67 ++++++++++++++++++++++++++
lib/eal/ppc/include/rte_rwlock.h | 4 ++
lib/eal/ppc/include/rte_spinlock.h | 9 ++++
lib/eal/x86/include/rte_rwlock.h | 4 ++
lib/eal/x86/include/rte_spinlock.h | 9 ++++
lib/meson.build | 5 ++
12 files changed, 141 insertions(+), 13 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
diff --git a/drivers/meson.build b/drivers/meson.build
index 1d8123b00c..c81c8c0eff 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -87,6 +87,7 @@ foreach subpath:subdirs
build = true # set to false to disable, e.g. missing deps
reason = '<unknown reason>' # set if build == false to explain
name = drv
+ annotate_locks = false
sources = []
headers = []
objs = []
@@ -152,6 +153,10 @@ foreach subpath:subdirs
enabled_drivers += name
lib_name = '_'.join(['rte', class, name])
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=' + '.'.join([log_prefix, name])
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
dpdk_conf.set(lib_name.to_upper(), 1)
dpdk_extra_ldflags += pkgconfig_extra_libs
diff --git a/lib/eal/arm/include/rte_rwlock.h b/lib/eal/arm/include/rte_rwlock.h
index 18bb37b036..bea398b6f1 100644
--- a/lib/eal/arm/include/rte_rwlock.h
+++ b/lib/eal/arm/include/rte_rwlock.h
@@ -13,24 +13,28 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_read_lock(rwl);
}
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_read_unlock(rwl);
}
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_write_lock(rwl);
}
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_write_unlock(rwl);
}
diff --git a/lib/eal/arm/include/rte_spinlock.h b/lib/eal/arm/include/rte_spinlock.h
index a973763c23..d47b13070b 100644
--- a/lib/eal/arm/include/rte_spinlock.h
+++ b/lib/eal/arm/include/rte_spinlock.h
@@ -23,36 +23,42 @@ static inline int rte_tm_supported(void)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_spinlock_lock(sl); /* fall-back */
}
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
return rte_spinlock_trylock(sl);
}
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_spinlock_unlock(sl);
}
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_spinlock_recursive_lock(slr); /* fall-back */
}
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_spinlock_recursive_unlock(slr);
}
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
return rte_spinlock_recursive_trylock(slr);
}
diff --git a/lib/eal/include/generic/rte_rwlock.h b/lib/eal/include/generic/rte_rwlock.h
index da9bc3e9c0..dabbac0131 100644
--- a/lib/eal/include/generic/rte_rwlock.h
+++ b/lib/eal/include/generic/rte_rwlock.h
@@ -23,6 +23,7 @@ extern "C" {
#include <rte_common.h>
#include <rte_atomic.h>
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
@@ -30,7 +31,7 @@ extern "C" {
*
* cnt is -1 when write lock is held, and > 0 when read locks are held.
*/
-typedef struct {
+typedef struct RTE_ANNOTATED_LOCK {
volatile int32_t cnt; /**< -1 when W lock held, > 0 when R locks held. */
} rte_rwlock_t;
@@ -58,7 +59,8 @@ rte_rwlock_init(rte_rwlock_t *rwl)
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_read_lock(rte_rwlock_t *rwl)
+rte_rwlock_read_lock(rte_rwlock_t *rwl) RTE_SHARED_LOCK_ACQUIRES(rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int32_t x;
int success = 0;
@@ -90,7 +92,8 @@ rte_rwlock_read_lock(rte_rwlock_t *rwl)
*/
__rte_experimental
static inline int
-rte_rwlock_read_trylock(rte_rwlock_t *rwl)
+rte_rwlock_read_trylock(rte_rwlock_t *rwl) RTE_SHARED_LOCK_TRYLOCK(1, rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int32_t x;
int success = 0;
@@ -114,7 +117,8 @@ rte_rwlock_read_trylock(rte_rwlock_t *rwl)
* A pointer to the rwlock structure.
*/
static inline void
-rte_rwlock_read_unlock(rte_rwlock_t *rwl)
+rte_rwlock_read_unlock(rte_rwlock_t *rwl) RTE_LOCK_RELEASES(rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
__atomic_fetch_sub(&rwl->cnt, 1, __ATOMIC_RELEASE);
}
@@ -134,7 +138,8 @@ rte_rwlock_read_unlock(rte_rwlock_t *rwl)
*/
__rte_experimental
static inline int
-rte_rwlock_write_trylock(rte_rwlock_t *rwl)
+rte_rwlock_write_trylock(rte_rwlock_t *rwl) RTE_EXC_LOCK_TRYLOCK(1, rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int32_t x;
@@ -153,7 +158,8 @@ rte_rwlock_write_trylock(rte_rwlock_t *rwl)
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_lock(rte_rwlock_t *rwl)
+rte_rwlock_write_lock(rte_rwlock_t *rwl) RTE_EXC_LOCK_ACQUIRES(rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int32_t x;
int success = 0;
@@ -177,7 +183,8 @@ rte_rwlock_write_lock(rte_rwlock_t *rwl)
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_unlock(rte_rwlock_t *rwl)
+rte_rwlock_write_unlock(rte_rwlock_t *rwl) RTE_LOCK_RELEASES(rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
__atomic_store_n(&rwl->cnt, 0, __ATOMIC_RELEASE);
}
diff --git a/lib/eal/include/generic/rte_spinlock.h b/lib/eal/include/generic/rte_spinlock.h
index 40fe49d5ad..c54421f5d3 100644
--- a/lib/eal/include/generic/rte_spinlock.h
+++ b/lib/eal/include/generic/rte_spinlock.h
@@ -22,12 +22,13 @@
#ifdef RTE_FORCE_INTRINSICS
#include <rte_common.h>
#endif
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
* The rte_spinlock_t type.
*/
-typedef struct {
+typedef struct RTE_ANNOTATED_LOCK {
volatile int locked; /**< lock status 0 = unlocked, 1 = locked */
} rte_spinlock_t;
@@ -55,11 +56,12 @@ rte_spinlock_init(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock(rte_spinlock_t *sl);
+rte_spinlock_lock(rte_spinlock_t *sl) RTE_EXC_LOCK_ACQUIRES(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int exp = 0;
@@ -79,11 +81,12 @@ rte_spinlock_lock(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl);
+rte_spinlock_unlock(rte_spinlock_t *sl) RTE_LOCK_RELEASES(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl)
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
__atomic_store_n(&sl->locked, 0, __ATOMIC_RELEASE);
}
@@ -98,11 +101,12 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
* 1 if the lock is successfully taken; 0 otherwise.
*/
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl);
+rte_spinlock_trylock(rte_spinlock_t *sl) RTE_EXC_LOCK_TRYLOCK(1, sl);
#ifdef RTE_FORCE_INTRINSICS
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl)
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int exp = 0;
return __atomic_compare_exchange_n(&sl->locked, &exp, 1,
@@ -211,6 +215,7 @@ static inline void rte_spinlock_recursive_init(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int id = rte_gettid();
@@ -227,6 +232,7 @@ static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (--(slr->count) == 0) {
slr->user = -1;
@@ -244,6 +250,7 @@ static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
* 1 if the lock is successfully taken; 0 otherwise.
*/
static inline int rte_spinlock_recursive_trylock(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int id = rte_gettid();
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index 9700494816..a20b8d7e23 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -27,6 +27,7 @@ headers += files(
'rte_keepalive.h',
'rte_launch.h',
'rte_lcore.h',
+ 'rte_lock_annotations.h',
'rte_log.h',
'rte_malloc.h',
'rte_memory.h',
diff --git a/lib/eal/include/rte_lock_annotations.h b/lib/eal/include/rte_lock_annotations.h
new file mode 100644
index 0000000000..143087a56c
--- /dev/null
+++ b/lib/eal/include/rte_lock_annotations.h
@@ -0,0 +1,67 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Red Hat, Inc.
+ */
+
+#ifndef RTE_LOCK_ANNOTATIONS_H
+#define RTE_LOCK_ANNOTATIONS_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifdef RTE_ANNOTATE_LOCKS
+
+#define RTE_ANNOTATED_LOCK \
+ __attribute__((lockable))
+
+#define RTE_GUARDED_BY(...) \
+ __attribute__((guarded_by(__VA_ARGS__)))
+#define RTE_GUARDED_VAR \
+ __attribute__((guarded_var))
+
+#define RTE_EXC_LOCK_REQUIRES(...) \
+ __attribute__((exclusive_locks_required(__VA_ARGS__)))
+#define RTE_EXC_LOCK_ACQUIRES(...) \
+ __attribute__((exclusive_lock_function(__VA_ARGS__)))
+#define RTE_EXC_LOCK_TRYLOCK(ret, ...) \
+ __attribute__((exclusive_trylock_function(ret, __VA_ARGS__)))
+
+#define RTE_SHARED_LOCK_REQUIRES(...) \
+ __attribute__((shared_locks_required(__VA_ARGS__)))
+#define RTE_SHARED_LOCK_ACQUIRES(...) \
+ __attribute__((shared_lock_function(__VA_ARGS__)))
+#define RTE_SHARED_LOCK_TRYLOCK(ret, ...) \
+ __attribute__((shared_trylock_function(ret, __VA_ARGS__)))
+
+#define RTE_LOCK_RELEASES(...) \
+ __attribute__((unlock_function(__VA_ARGS__)))
+
+#define RTE_NO_ANNOTATED_LOCK_CHECK \
+ __attribute__((no_thread_safety_analysis))
+
+#else /* ! RTE_ANNOTATE_LOCKS */
+
+#define RTE_ANNOTATED_LOCK
+
+#define RTE_GUARDED_BY(...)
+#define RTE_GUARDED_VAR
+
+#define RTE_EXC_LOCK_REQUIRES(...)
+#define RTE_EXC_LOCK_ACQUIRES(...)
+#define RTE_EXC_LOCK_TRYLOCK(...)
+
+#define RTE_SHARED_LOCK_REQUIRES(...)
+#define RTE_SHARED_LOCK_ACQUIRES(...)
+#define RTE_SHARED_LOCK_TRYLOCK(...)
+
+#define RTE_LOCK_RELEASES(...)
+
+#define RTE_NO_ANNOTATED_LOCK_CHECK
+
+#endif /* RTE_ANNOTATE_LOCKS */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_LOCK_ANNOTATIONS_H */
diff --git a/lib/eal/ppc/include/rte_rwlock.h b/lib/eal/ppc/include/rte_rwlock.h
index 9fadc04076..9260a9f0a2 100644
--- a/lib/eal/ppc/include/rte_rwlock.h
+++ b/lib/eal/ppc/include/rte_rwlock.h
@@ -11,24 +11,28 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_read_lock(rwl);
}
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_read_unlock(rwl);
}
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_write_lock(rwl);
}
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_write_unlock(rwl);
}
diff --git a/lib/eal/ppc/include/rte_spinlock.h b/lib/eal/ppc/include/rte_spinlock.h
index 149ec245c7..1d64535ddd 100644
--- a/lib/eal/ppc/include/rte_spinlock.h
+++ b/lib/eal/ppc/include/rte_spinlock.h
@@ -20,6 +20,7 @@ extern "C" {
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
while (__sync_lock_test_and_set(&sl->locked, 1))
while (sl->locked)
@@ -28,12 +29,14 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
__sync_lock_release(&sl->locked);
}
static inline int
rte_spinlock_trylock(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
return __sync_lock_test_and_set(&sl->locked, 1) == 0;
}
@@ -47,36 +50,42 @@ static inline int rte_tm_supported(void)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_spinlock_lock(sl); /* fall-back */
}
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
return rte_spinlock_trylock(sl);
}
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_spinlock_unlock(sl);
}
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_spinlock_recursive_lock(slr); /* fall-back */
}
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_spinlock_recursive_unlock(slr);
}
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
return rte_spinlock_recursive_trylock(slr);
}
diff --git a/lib/eal/x86/include/rte_rwlock.h b/lib/eal/x86/include/rte_rwlock.h
index eec4c7123c..7857192d3e 100644
--- a/lib/eal/x86/include/rte_rwlock.h
+++ b/lib/eal/x86/include/rte_rwlock.h
@@ -14,6 +14,7 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -22,6 +23,7 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (unlikely(rwl->cnt))
rte_rwlock_read_unlock(rwl);
@@ -31,6 +33,7 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -39,6 +42,7 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (unlikely(rwl->cnt))
rte_rwlock_write_unlock(rwl);
diff --git a/lib/eal/x86/include/rte_spinlock.h b/lib/eal/x86/include/rte_spinlock.h
index e2e2b2643c..a0be70570b 100644
--- a/lib/eal/x86/include/rte_spinlock.h
+++ b/lib/eal/x86/include/rte_spinlock.h
@@ -23,6 +23,7 @@ extern "C" {
#ifndef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int lock_val = 1;
asm volatile (
@@ -43,6 +44,7 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock (rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int unlock_val = 0;
asm volatile (
@@ -54,6 +56,7 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock (rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int lockval = 1;
@@ -121,6 +124,7 @@ rte_try_tm(volatile int *lock)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (likely(rte_try_tm(&sl->locked)))
return;
@@ -130,6 +134,7 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (likely(rte_try_tm(&sl->locked)))
return 1;
@@ -139,6 +144,7 @@ rte_spinlock_trylock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (unlikely(sl->locked))
rte_spinlock_unlock(sl);
@@ -148,6 +154,7 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (likely(rte_try_tm(&slr->sl.locked)))
return;
@@ -157,6 +164,7 @@ rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (unlikely(slr->sl.locked))
rte_spinlock_recursive_unlock(slr);
@@ -166,6 +174,7 @@ rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (likely(rte_try_tm(&slr->sl.locked)))
return 1;
diff --git a/lib/meson.build b/lib/meson.build
index 24adbe44c9..909133ea64 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -112,6 +112,7 @@ foreach l:libraries
reason = '<unknown reason>' # set if build == false to explain why
name = l
use_function_versioning = false
+ annotate_locks = false
sources = []
headers = []
indirect_headers = [] # public headers not directly included by apps
@@ -184,6 +185,10 @@ foreach l:libraries
cflags += '-DRTE_USE_FUNCTION_VERSIONING'
endif
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=lib.' + l
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
# first build static lib
static_lib = static_library(libname,
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 3/9] vhost: annotate virtqueue access lock
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 1/9] vhost: fix missing virtqueue lock protection David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-04-07 1:40 ` Hu, Jiayu
2022-03-30 13:49 ` [RFC PATCH v2 4/9] vhost: fix async access David Marchand
` (6 subsequent siblings)
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
This change simply annotates existing paths of the code leading to
manipulations of the vq->access_lock.
One small change is required: vhost_poll_enqueue_completed was getting
a queue_id to get hold of the vq, while its callers already knew of
the vq. For the annotation sake, vq is now directly passed.
vhost_user_lock_all_queue_pairs and vhost_user_unlock_all_queue_pairs
are skipped since vq->access_lock are conditionally held.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.h | 2 ++
lib/vhost/vhost_user.c | 2 ++
lib/vhost/virtio_net.c | 16 ++++++++++++----
3 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index a9edc271aa..158460b7d7 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -834,6 +834,7 @@ vhost_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old)
static __rte_always_inline void
vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
/* Flush used->idx update before we read avail->flags. */
rte_atomic_thread_fence(__ATOMIC_SEQ_CST);
@@ -872,6 +873,7 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
static __rte_always_inline void
vhost_vring_call_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint16_t old, new, off, off_wrap;
bool signalled_used_valid, kick = false;
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 1d390677fa..87eaa2ab4a 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2909,6 +2909,7 @@ vhost_user_check_and_alloc_queue_pair(struct virtio_net *dev,
static void
vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
unsigned int i = 0;
unsigned int vq_num = 0;
@@ -2926,6 +2927,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
static void
vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
unsigned int i = 0;
unsigned int vq_num = 0;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 5f432b0d77..514ee00993 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -1246,6 +1246,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1441,6 +1442,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -1955,11 +1957,11 @@ write_back_completed_descs_packed(struct vhost_virtqueue *vq,
}
static __rte_always_inline uint16_t
-vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id,
+vhost_poll_enqueue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
- struct vhost_virtqueue *vq = dev->virtqueue[queue_id];
struct vhost_async *async = vq->async;
struct async_inflight_info *pkts_info = async->pkts_info;
uint16_t nr_cpl_pkts = 0;
@@ -2062,7 +2064,7 @@ rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
goto out;
}
- n_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count, dma_id, vchan_id);
+ n_pkts_cpl = vhost_poll_enqueue_completed(dev, vq, pkts, count, dma_id, vchan_id);
out:
rte_spinlock_unlock(&vq->access_lock);
@@ -2104,7 +2106,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
return 0;
}
- n_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count, dma_id, vchan_id);
+ n_pkts_cpl = vhost_poll_enqueue_completed(dev, vq, pkts, count, dma_id, vchan_id);
return n_pkts_cpl;
}
@@ -2679,6 +2681,7 @@ static uint16_t
virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint16_t i;
uint16_t free_entries;
@@ -2774,6 +2777,7 @@ static uint16_t
virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2783,6 +2787,7 @@ static uint16_t
virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -2982,6 +2987,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count,
bool legacy_ol_flags)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -3025,6 +3031,7 @@ static uint16_t
virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3034,6 +3041,7 @@ static uint16_t
virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 4/9] vhost: fix async access
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
` (2 preceding siblings ...)
2022-03-30 13:49 ` [RFC PATCH v2 3/9] vhost: annotate virtqueue access lock David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-31 8:00 ` Maxime Coquelin
2022-04-04 6:57 ` Pai G, Sunil
2022-03-30 13:49 ` [RFC PATCH v2 5/9] vhost: annotate async acesses David Marchand
` (5 subsequent siblings)
9 siblings, 2 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding,
stable, Patrick Fu
vq->async accesses must be protected with vq->access_lock.
Fixes: eb666d24085f ("vhost: fix async unregister deadlock")
Fixes: 0c0935c5f794 ("vhost: allow to check in-flight packets for async vhost")
Cc: stable@dpdk.org
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.c | 25 ++++++++++---------------
1 file changed, 10 insertions(+), 15 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 2f96a28dac..a93e41f314 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1753,27 +1753,23 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- ret = 0;
-
- if (!vq->async)
- return ret;
-
if (!rte_spinlock_trylock(&vq->access_lock)) {
VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel, virtqueue busy.\n",
dev->ifname);
- return -1;
+ return ret;
}
- if (vq->async->pkts_inflight_n) {
+ if (!vq->async) {
+ ret = 0;
+ } else if (vq->async->pkts_inflight_n) {
VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel.\n", dev->ifname);
VHOST_LOG_CONFIG(ERR, "(%s) inflight packets must be completed before unregistration.\n",
dev->ifname);
- ret = -1;
- goto out;
+ } else {
+ vhost_free_async_mem(vq);
+ ret = 0;
}
- vhost_free_async_mem(vq);
-out:
rte_spinlock_unlock(&vq->access_lock);
return ret;
@@ -1891,9 +1887,6 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- if (!vq->async)
- return ret;
-
if (!rte_spinlock_trylock(&vq->access_lock)) {
VHOST_LOG_CONFIG(DEBUG,
"(%s) failed to check in-flight packets. virtqueue busy.\n",
@@ -1901,7 +1894,9 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
return ret;
}
- ret = vq->async->pkts_inflight_n;
+ if (vq->async)
+ ret = vq->async->pkts_inflight_n;
+
rte_spinlock_unlock(&vq->access_lock);
return ret;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 5/9] vhost: annotate async acesses
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
` (3 preceding siblings ...)
2022-03-30 13:49 ` [RFC PATCH v2 4/9] vhost: fix async access David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 6/9] vhost: annotate need reply handling David Marchand
` (4 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vq->async is initialised and must be accessed under vq->access_lock.
Top level "_thread_unsafe" functions could be checked at runtime (clang
provides a lock aware assert()-like check), but they are simply skipped
because those functions are not called in-tree, and as a result,
their annotations would not be tested.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.c | 14 +++++++++-----
lib/vhost/vhost.h | 2 +-
lib/vhost/vhost_user.c | 2 ++
lib/vhost/virtio_net.c | 15 +++++++++++++++
4 files changed, 27 insertions(+), 6 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index a93e41f314..637aa65ffb 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -334,6 +334,7 @@ cleanup_device(struct virtio_net *dev, int destroy)
static void
vhost_free_async_mem(struct vhost_virtqueue *vq)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
if (!vq->async)
return;
@@ -352,6 +353,7 @@ vhost_free_async_mem(struct vhost_virtqueue *vq)
void
free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
if (vq_is_packed(dev))
rte_free(vq->shadow_used_packed);
@@ -1622,10 +1624,10 @@ rte_vhost_extern_callback_register(int vid,
}
static __rte_always_inline int
-async_channel_register(int vid, uint16_t queue_id)
+async_channel_register(struct virtio_net *dev, struct vhost_virtqueue *vq,
+ uint16_t queue_id)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
- struct virtio_net *dev = get_device(vid);
- struct vhost_virtqueue *vq = dev->virtqueue[queue_id];
struct vhost_async *async;
int node = vq->numa_node;
@@ -1709,7 +1711,7 @@ rte_vhost_async_channel_register(int vid, uint16_t queue_id)
return -1;
rte_spinlock_lock(&vq->access_lock);
- ret = async_channel_register(vid, queue_id);
+ ret = async_channel_register(dev, vq, queue_id);
rte_spinlock_unlock(&vq->access_lock);
return ret;
@@ -1717,6 +1719,7 @@ rte_vhost_async_channel_register(int vid, uint16_t queue_id)
int
rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
struct vhost_virtqueue *vq;
struct virtio_net *dev = get_device(vid);
@@ -1732,7 +1735,7 @@ rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
if (unlikely(vq == NULL || !dev->async_copy))
return -1;
- return async_channel_register(vid, queue_id);
+ return async_channel_register(dev, vq, queue_id);
}
int
@@ -1777,6 +1780,7 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
int
rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
struct vhost_virtqueue *vq;
struct virtio_net *dev = get_device(vid);
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 158460b7d7..7c44fba0a1 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -299,7 +299,7 @@ struct vhost_virtqueue {
struct rte_vhost_resubmit_info *resubmit_inflight;
uint64_t global_counter;
- struct vhost_async *async;
+ struct vhost_async *async RTE_GUARDED_VAR;
int notif_enable;
#define VIRTIO_UNINITIALIZED_NOTIF (-1)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 87eaa2ab4a..0cfa2d6f82 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2199,6 +2199,8 @@ static int
vhost_user_set_vring_enable(struct virtio_net **pdev,
struct vhu_msg_context *ctx,
int main_fd __rte_unused)
+ /* vq->access_lock is taken in vhost_user_lock_all_queue_pairs() */
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
struct virtio_net *dev = *pdev;
bool enable = !!ctx->msg.payload.state.num;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 514ee00993..c0d9d3db3e 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -51,6 +51,7 @@ static __rte_always_inline int64_t
vhost_async_dma_transfer_one(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t flag_idx,
struct vhost_iov_iter *pkt)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
uint16_t ring_mask = dma_info->ring_mask;
@@ -99,6 +100,7 @@ static __rte_always_inline uint16_t
vhost_async_dma_transfer(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t head_idx,
struct vhost_iov_iter *pkts, uint16_t nr_pkts)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
int64_t ret, nr_copies = 0;
@@ -1000,6 +1002,7 @@ static __rte_always_inline int
async_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1057,6 +1060,7 @@ static __rte_always_inline int
mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1186,6 +1190,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkt,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1417,6 +1422,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1541,6 +1547,7 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
static __rte_always_inline uint16_t
async_get_first_inflight_pkt_idx(struct vhost_virtqueue *vq)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct vhost_async *async = vq->async;
@@ -1587,6 +1594,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1691,6 +1699,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs,
uint16_t *nr_buffers)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1748,6 +1757,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1766,6 +1776,7 @@ virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
uint32_t nr_err, uint32_t *pkt_idx)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint16_t descs_err = 0;
uint16_t buffers_err = 0;
@@ -1793,6 +1804,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
uint32_t pkt_idx = 0;
uint32_t remained = count;
@@ -1863,6 +1875,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
static __rte_always_inline void
write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t nr_left = n_descs;
@@ -1895,6 +1908,7 @@ write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
static __rte_always_inline void
write_back_completed_descs_packed(struct vhost_virtqueue *vq,
uint16_t n_buffers)
+ RTE_EXC_LOCK_REQUIRES(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t from = async->last_buffer_idx_packed;
@@ -2076,6 +2090,7 @@ uint16_t
rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
struct virtio_net *dev = get_device(vid);
struct vhost_virtqueue *vq;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 6/9] vhost: annotate need reply handling
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
` (4 preceding siblings ...)
2022-03-30 13:49 ` [RFC PATCH v2 5/9] vhost: annotate async acesses David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 7/9] vhost: annotate VDPA device list accesses David Marchand
` (3 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
When a reply from the slave is required (VHOST_USER_NEED_REPLY flag),
a spinlock is taken before sending the message.
This spinlock is released if an error occurs when sending the message, and
once a reply is received.
A problem is that this lock is taken under a branch and annotating
conditionally held locks is not supported.
The code seems currently correct and, while we may rework the code,
it is easier to simply skip checks on slave_req_lock for those helpers.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost_user.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 0cfa2d6f82..273cce5e41 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2854,6 +2854,7 @@ send_vhost_reply(struct virtio_net *dev, int sockfd, struct vhu_msg_context *ctx
static int
send_vhost_slave_message(struct virtio_net *dev,
struct vhu_msg_context *ctx)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
int ret;
@@ -3165,6 +3166,7 @@ vhost_user_msg_handler(int vid, int fd)
static int process_slave_message_reply(struct virtio_net *dev,
const struct vhu_msg_context *ctx)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
struct vhu_msg_context msg_reply;
int ret;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 7/9] vhost: annotate VDPA device list accesses
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
` (5 preceding siblings ...)
2022-03-30 13:49 ` [RFC PATCH v2 6/9] vhost: annotate need reply handling David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 8/9] vhost: annotate IOTLB locks David Marchand
` (2 subsequent siblings)
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vdpa_device_list access must be protected with vdpa_device_list_lock
spinlock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vdpa.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index 8fa2153023..e2caa2bf28 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -22,21 +22,24 @@
/** Double linked list of vDPA devices. */
TAILQ_HEAD(vdpa_device_list, rte_vdpa_device);
-static struct vdpa_device_list vdpa_device_list =
- TAILQ_HEAD_INITIALIZER(vdpa_device_list);
+static struct vdpa_device_list vdpa_device_list__ =
+ TAILQ_HEAD_INITIALIZER(vdpa_device_list__);
static rte_spinlock_t vdpa_device_list_lock = RTE_SPINLOCK_INITIALIZER;
+static struct vdpa_device_list * const vdpa_device_list
+ RTE_GUARDED_BY(&vdpa_device_list_lock) = &vdpa_device_list__;
/* Unsafe, needs to be called with vdpa_device_list_lock held */
static struct rte_vdpa_device *
__vdpa_find_device_by_name(const char *name)
+ RTE_EXC_LOCK_REQUIRES(&vdpa_device_list_lock)
{
struct rte_vdpa_device *dev, *ret = NULL;
if (name == NULL)
return NULL;
- TAILQ_FOREACH(dev, &vdpa_device_list, next) {
+ TAILQ_FOREACH(dev, vdpa_device_list, next) {
if (!strncmp(dev->device->name, name, RTE_DEV_NAME_MAX_LEN)) {
ret = dev;
break;
@@ -100,7 +103,7 @@ rte_vdpa_register_device(struct rte_device *rte_dev,
dev->device = rte_dev;
dev->ops = ops;
- TAILQ_INSERT_TAIL(&vdpa_device_list, dev, next);
+ TAILQ_INSERT_TAIL(vdpa_device_list, dev, next);
out_unlock:
rte_spinlock_unlock(&vdpa_device_list_lock);
@@ -114,11 +117,11 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int ret = -1;
rte_spinlock_lock(&vdpa_device_list_lock);
- RTE_TAILQ_FOREACH_SAFE(cur_dev, &vdpa_device_list, next, tmp_dev) {
+ RTE_TAILQ_FOREACH_SAFE(cur_dev, vdpa_device_list, next, tmp_dev) {
if (dev != cur_dev)
continue;
- TAILQ_REMOVE(&vdpa_device_list, dev, next);
+ TAILQ_REMOVE(vdpa_device_list, dev, next);
rte_free(dev);
ret = 0;
break;
@@ -316,7 +319,7 @@ vdpa_find_device(const struct rte_vdpa_device *start, rte_vdpa_cmp_t cmp,
rte_spinlock_lock(&vdpa_device_list_lock);
if (start == NULL)
- dev = TAILQ_FIRST(&vdpa_device_list);
+ dev = TAILQ_FIRST(vdpa_device_list);
else
dev = TAILQ_NEXT(start, next);
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 8/9] vhost: annotate IOTLB locks
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
` (6 preceding siblings ...)
2022-03-30 13:49 ` [RFC PATCH v2 7/9] vhost: annotate VDPA device list accesses David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 9/9] vhost: enable lock check David Marchand
2022-03-30 14:03 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
This change simply annotates existing paths of the code leading to
manipulations of the IOTLB r/w locks.
clang does not support conditionally held locks, so always take iotlb
locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
vdpa and vhost_crypto code are annotated though they end up not taking
a IOTLB lock and have been marked with a FIXME.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/iotlb.h | 8 +++++++
lib/vhost/vdpa.c | 1 +
lib/vhost/vhost.c | 11 +++++----
lib/vhost/vhost.h | 22 +++++++++++++-----
lib/vhost/vhost_crypto.c | 7 ++++++
lib/vhost/virtio_net.c | 49 ++++++++++++++++++++++++++++++----------
6 files changed, 75 insertions(+), 23 deletions(-)
diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h
index 8d0ff7473b..af5ec2da55 100644
--- a/lib/vhost/iotlb.h
+++ b/lib/vhost/iotlb.h
@@ -11,24 +11,32 @@
static __rte_always_inline void
vhost_user_iotlb_rd_lock(struct vhost_virtqueue *vq)
+ RTE_SHARED_LOCK_ACQUIRES(vq->iotlb_lock)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_read_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_rd_unlock(struct vhost_virtqueue *vq)
+ RTE_LOCK_RELEASES(vq->iotlb_lock)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_read_unlock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_lock(struct vhost_virtqueue *vq)
+ RTE_EXC_LOCK_ACQUIRES(vq->iotlb_lock)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_write_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_unlock(struct vhost_virtqueue *vq)
+ RTE_LOCK_RELEASES(vq->iotlb_lock)
+ RTE_NO_ANNOTATED_LOCK_CHECK
{
rte_rwlock_write_unlock(&vq->iotlb_lock);
}
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index e2caa2bf28..9304c0083d 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -133,6 +133,7 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int
rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
+ RTE_NO_ANNOTATED_LOCK_CHECK /* FIXME: requires iotlb_lock? */
{
struct virtio_net *dev = get_device(vid);
uint16_t idx, idx_m, desc_id;
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 637aa65ffb..f87d9735d2 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -383,6 +383,7 @@ free_device(struct virtio_net *dev)
static __rte_always_inline int
log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
if (likely(!(vq->ring_addrs.flags & (1 << VHOST_VRING_F_LOG))))
return 0;
@@ -434,6 +435,7 @@ translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
/* Caller should have iotlb_lock read-locked */
static int
vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -473,6 +475,7 @@ vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
/* Caller should have iotlb_lock read-locked */
static int
vring_translate_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -527,10 +530,9 @@ vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
}
void
-vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+vring_invalidate(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq)
{
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_lock(vq);
+ vhost_user_iotlb_wr_lock(vq);
vq->access_ok = false;
vq->desc = NULL;
@@ -538,8 +540,7 @@ vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
vq->used = NULL;
vq->log_guest_addr = 0;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_unlock(vq);
+ vhost_user_iotlb_wr_unlock(vq);
}
static void
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 7c44fba0a1..f26171b601 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -525,12 +525,15 @@ void __vhost_log_cache_write(struct virtio_net *dev,
uint64_t addr, uint64_t len);
void __vhost_log_cache_write_iova(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock);
void __vhost_log_cache_sync(struct virtio_net *dev,
struct vhost_virtqueue *vq);
+
void __vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len);
void __vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock);
static __rte_always_inline void
vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len)
@@ -580,6 +583,7 @@ vhost_log_used_vring(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -593,6 +597,7 @@ vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -796,18 +801,23 @@ struct rte_vhost_device_ops const *vhost_driver_callback_get(const char *path);
void vhost_backend_cleanup(struct virtio_net *dev);
uint64_t __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t *len, uint8_t perm);
+ uint64_t iova, uint64_t *len, uint8_t perm)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock);
void *vhost_alloc_copy_ind_table(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t desc_addr, uint64_t desc_len);
-int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq);
+ uint64_t desc_addr, uint64_t desc_len)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock);
+int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock);
uint64_t translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t log_addr);
+ uint64_t log_addr)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock);
void vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq);
static __rte_always_inline uint64_t
vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *len, uint8_t perm)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
if (!(dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)))
return rte_vhost_va_from_guest_pa(dev->mem, iova, len);
diff --git a/lib/vhost/vhost_crypto.c b/lib/vhost/vhost_crypto.c
index b1c0eb6a0f..fccd28a71a 100644
--- a/lib/vhost/vhost_crypto.c
+++ b/lib/vhost/vhost_crypto.c
@@ -506,6 +506,7 @@ static __rte_always_inline struct virtio_crypto_inhdr *
reach_inhdr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ RTE_SHARED_LOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct virtio_crypto_inhdr *inhdr;
struct vhost_crypto_desc *last = head + (max_n_descs - 1);
@@ -552,6 +553,7 @@ static __rte_always_inline void *
get_data_ptr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *cur_desc,
uint8_t perm)
+ RTE_SHARED_LOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
void *data;
uint64_t dlen = cur_desc->len;
@@ -570,6 +572,7 @@ copy_data(void *dst_data, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
struct vhost_crypto_desc **cur_desc,
uint32_t size, uint32_t max_n_descs)
+ RTE_SHARED_LOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = *cur_desc;
uint64_t remain, addr, dlen, len;
@@ -718,6 +721,7 @@ prepare_write_back_data(struct vhost_crypto_data_req *vc_req,
uint32_t offset,
uint64_t write_back_len,
uint32_t max_n_descs)
+ RTE_SHARED_LOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_writeback_data *wb_data, *head;
struct vhost_crypto_desc *desc = *cur_desc;
@@ -838,6 +842,7 @@ prepare_sym_cipher_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_cipher_data_req *cipher,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ RTE_SHARED_LOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head;
struct vhost_crypto_writeback_data *ewb = NULL;
@@ -990,6 +995,7 @@ prepare_sym_chain_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_alg_chain_data_req *chain,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ RTE_SHARED_LOCK_REQUIRES(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head, *digest_desc;
struct vhost_crypto_writeback_data *ewb = NULL, *ewb2 = NULL;
@@ -1172,6 +1178,7 @@ vhost_crypto_process_one_req(struct vhost_crypto *vcrypto,
struct vhost_virtqueue *vq, struct rte_crypto_op *op,
struct vring_desc *head, struct vhost_crypto_desc *descs,
uint16_t desc_idx)
+ RTE_NO_ANNOTATED_LOCK_CHECK /* FIXME: requires iotlb_lock? */
{
struct vhost_crypto_data_req *vc_req = rte_mbuf_to_priv(op->sym->m_src);
struct rte_cryptodev_sym_session *session;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index c0d9d3db3e..867492a338 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -180,6 +180,7 @@ vhost_async_dma_check_completed(struct virtio_net *dev, int16_t dma_id, uint16_t
static inline void
do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
struct batch_copy_elem *elem = vq->batch_copy_elems;
uint16_t count = vq->batch_copy_nb_elems;
@@ -526,6 +527,7 @@ vhost_shadow_enqueue_single_packed(struct virtio_net *dev,
uint16_t *id,
uint16_t *count,
uint16_t num_buffers)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
vhost_shadow_enqueue_packed(vq, len, id, count, num_buffers);
@@ -607,6 +609,7 @@ static __rte_always_inline int
map_one_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint64_t desc_iova, uint64_t desc_len, uint8_t perm)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t vec_id = *vec_idx;
@@ -644,6 +647,7 @@ fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t avail_idx, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint16_t *desc_chain_head,
uint32_t *desc_chain_len, uint8_t perm)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
uint16_t vec_id = *vec_idx;
@@ -727,6 +731,7 @@ reserve_avail_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t size, struct buf_vector *buf_vec,
uint16_t *num_buffers, uint16_t avail_head,
uint16_t *nr_vec)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t cur_idx;
uint16_t vec_idx = 0;
@@ -777,6 +782,7 @@ fill_vec_buf_packed_indirect(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct vring_packed_desc *desc, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint32_t *len, uint8_t perm)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t i;
uint32_t nr_descs;
@@ -835,6 +841,7 @@ fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t avail_idx, uint16_t *desc_count,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint16_t *buf_id, uint32_t *len, uint8_t perm)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -900,6 +907,7 @@ static __rte_noinline void
copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec,
struct virtio_net_hdr_mrg_rxbuf *hdr)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint64_t len;
uint64_t remain = dev->vhost_hlen;
@@ -1036,6 +1044,7 @@ static __rte_always_inline void
sync_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_addr, uint64_t buf_iova, uint32_t cpy_len)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
@@ -1061,6 +1070,7 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1191,6 +1201,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1252,6 +1263,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1309,6 +1321,7 @@ virtio_dev_rx_sync_batch_check(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -1360,6 +1373,7 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
@@ -1401,6 +1415,7 @@ static __rte_always_inline int
virtio_dev_rx_sync_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint64_t desc_addrs[PACKED_BATCH_SIZE];
uint64_t lens[PACKED_BATCH_SIZE];
@@ -1423,6 +1438,7 @@ virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1449,6 +1465,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -1501,8 +1518,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
if (unlikely(!vq->enabled))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -1518,8 +1534,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
nb_tx = virtio_dev_rx_split(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -1595,6 +1610,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1700,6 +1716,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
uint16_t *nr_descs,
uint16_t *nr_buffers)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1758,6 +1775,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1805,6 +1823,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint32_t remained = count;
@@ -2154,8 +2173,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,
if (unlikely(!vq->enabled || !vq->async))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -2173,8 +2191,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,
pkts, count, dma_id, vchan_id);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -2697,6 +2714,7 @@ virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t i;
uint16_t free_entries;
@@ -2793,6 +2811,7 @@ virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2803,6 +2822,7 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -2814,6 +2834,7 @@ vhost_reserve_avail_batch_packed(struct virtio_net *dev,
uint16_t avail_idx,
uintptr_t *desc_addrs,
uint16_t *ids)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
bool wrap = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -2883,6 +2904,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts,
bool legacy_ol_flags)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t avail_idx = vq->last_avail_idx;
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
@@ -2929,6 +2951,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *buf_id,
uint16_t *desc_count,
bool legacy_ol_flags)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -2972,6 +2995,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mempool *mbuf_pool,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3003,6 +3027,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
uint32_t count,
bool legacy_ol_flags)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -3047,6 +3072,7 @@ virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3057,6 +3083,7 @@ virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
RTE_EXC_LOCK_REQUIRES(vq->access_lock)
+ RTE_SHARED_LOCK_REQUIRES(vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3096,8 +3123,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -3153,8 +3179,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
}
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v2 9/9] vhost: enable lock check
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
` (7 preceding siblings ...)
2022-03-30 13:49 ` [RFC PATCH v2 8/9] vhost: annotate IOTLB locks David Marchand
@ 2022-03-30 13:49 ` David Marchand
2022-03-30 14:03 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
9 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-30 13:49 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
Now that all locks in this library are annotated, we can enable the
check.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/meson.build | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
index bc7272053b..197a51d936 100644
--- a/lib/vhost/meson.build
+++ b/lib/vhost/meson.build
@@ -17,6 +17,8 @@ elif (toolchain == 'icc' and cc.version().version_compare('>=16.0.0'))
endif
dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY', cc.has_header('linux/userfaultfd.h'))
cflags += '-fno-strict-aliasing'
+
+annotate_locks = true
sources = files(
'fd_man.c',
'iotlb.c',
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v2 0/9] vhost lock annotations
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
` (8 preceding siblings ...)
2022-03-30 13:49 ` [RFC PATCH v2 9/9] vhost: enable lock check David Marchand
@ 2022-03-30 14:03 ` David Marchand
2022-03-30 14:37 ` Ali Alnubani
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-03-30 14:03 UTC (permalink / raw)
To: dev, Ali Alnubani
Cc: Maxime Coquelin, Xia, Chenbo, Jiayu Hu, Wang, YuanX, Xuan Ding
On Wed, Mar 30, 2022 at 3:50 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> vhost internals involves multiple locks to protect data access by
> multiple threads.
>
> This series is a try at using clang thread safety checks [1] to catch
> issues during compilation: EAL spinlock and rwlock are annotated and
> vhost code is instrumented so that clang can statically check
> correctness.
>
> This is still a work in progress (some documentation and a release note
> update are missing).
>
> Those annotations are quite heavy to maintain because the full path of
> code must be annotated, but I think it is worth using.
>
>
> 1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
It looks like mimecast shot the first patch (which I sent in place of
Maxime, because this series should go through the main repo).
Looking at the mail source, I see:
X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation
Protection Definition;Similar Internal Domain=false;Similar Monitored
External Domain=false;Custom External Domain=false;Mimecast External
Domain=false;Newly Observed Domain=false;Internal User
Name=false;Custom Display Name List=false;Reply-to Address
Mismatch=false;Targeted Threat Dictionary=false;Mimecast Threat
Dictionary=false;Custom Threat Dictionary=false
I don't know how to understand this...
But as a result, the series is missing this patch in patchwork.
Patch 1 is still available from RFC v1:
https://patchwork.dpdk.org/project/dpdk/patch/20220328121758.26632-2-david.marchand@redhat.com/
or via next-virtio.
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [RFC PATCH v2 0/9] vhost lock annotations
2022-03-30 14:03 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
@ 2022-03-30 14:37 ` Ali Alnubani
2022-04-05 7:11 ` David Marchand
0 siblings, 1 reply; 110+ messages in thread
From: Ali Alnubani @ 2022-03-30 14:37 UTC (permalink / raw)
To: David Marchand, dev
Cc: Maxime Coquelin, Xia, Chenbo, Jiayu Hu, Wang, YuanX, Xuan Ding
[..]
> It looks like mimecast shot the first patch (which I sent in place of
> Maxime, because this series should go through the main repo).
>
> Looking at the mail source, I see:
>
> X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation
> Protection Definition;Similar Internal Domain=false;Similar Monitored
> External Domain=false;Custom External Domain=false;Mimecast External
> Domain=false;Newly Observed Domain=false;Internal User
> Name=false;Custom Display Name List=false;Reply-to Address
> Mismatch=false;Targeted Threat Dictionary=false;Mimecast Threat
> Dictionary=false;Custom Threat Dictionary=false
>
> I don't know how to understand this...
> But as a result, the series is missing this patch in patchwork.
I believe it was ignored by Patchwork because of its content-type (application/octet-stream), which indicates that the message contains binary data instead of text:
> Content-Transfer-Encoding: 8bit
> Content-Type: application/octet-stream; x-default=true
Regards,
Ali
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v2 4/9] vhost: fix async access
2022-03-30 13:49 ` [RFC PATCH v2 4/9] vhost: fix async access David Marchand
@ 2022-03-31 8:00 ` Maxime Coquelin
2022-03-31 10:23 ` Hu, Jiayu
2022-04-04 6:57 ` Pai G, Sunil
1 sibling, 1 reply; 110+ messages in thread
From: Maxime Coquelin @ 2022-03-31 8:00 UTC (permalink / raw)
To: David Marchand, dev
Cc: chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding, stable, Patrick Fu
Hi Jiayu,
On 3/30/22 15:49, David Marchand wrote:
> vq->async accesses must be protected with vq->access_lock.
>
> Fixes: eb666d24085f ("vhost: fix async unregister deadlock")
> Fixes: 0c0935c5f794 ("vhost: allow to check in-flight packets for async vhost")
> Cc: stable@dpdk.org
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost.c | 25 ++++++++++---------------
> 1 file changed, 10 insertions(+), 15 deletions(-)
Could you please test and review below patch?
We may want to apply it early, before the annotation series is applied.
Thanks!
Maxime
> diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
> index 2f96a28dac..a93e41f314 100644
> --- a/lib/vhost/vhost.c
> +++ b/lib/vhost/vhost.c
> @@ -1753,27 +1753,23 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
> if (vq == NULL)
> return ret;
>
> - ret = 0;
> -
> - if (!vq->async)
> - return ret;
> -
> if (!rte_spinlock_trylock(&vq->access_lock)) {
> VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel, virtqueue busy.\n",
> dev->ifname);
> - return -1;
> + return ret;
> }
>
> - if (vq->async->pkts_inflight_n) {
> + if (!vq->async) {
> + ret = 0;
> + } else if (vq->async->pkts_inflight_n) {
> VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel.\n", dev->ifname);
> VHOST_LOG_CONFIG(ERR, "(%s) inflight packets must be completed before unregistration.\n",
> dev->ifname);
> - ret = -1;
> - goto out;
> + } else {
> + vhost_free_async_mem(vq);
> + ret = 0;
> }
>
> - vhost_free_async_mem(vq);
> -out:
> rte_spinlock_unlock(&vq->access_lock);
>
> return ret;
> @@ -1891,9 +1887,6 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
> if (vq == NULL)
> return ret;
>
> - if (!vq->async)
> - return ret;
> -
> if (!rte_spinlock_trylock(&vq->access_lock)) {
> VHOST_LOG_CONFIG(DEBUG,
> "(%s) failed to check in-flight packets. virtqueue busy.\n",
> @@ -1901,7 +1894,9 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
> return ret;
> }
>
> - ret = vq->async->pkts_inflight_n;
> + if (vq->async)
> + ret = vq->async->pkts_inflight_n;
> +
> rte_spinlock_unlock(&vq->access_lock);
>
> return ret;
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock
2022-03-30 13:49 ` [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock David Marchand
@ 2022-03-31 9:22 ` David Marchand
2022-04-04 6:21 ` Stephen Hemminger
1 sibling, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-03-31 9:22 UTC (permalink / raw)
To: dev
Cc: Maxime Coquelin, Xia, Chenbo, Jiayu Hu, Wang, YuanX, Xuan Ding,
Ruifeng Wang, Jan Viktorin, David Christensen, Bruce Richardson,
Konstantin Ananyev
On Wed, Mar 30, 2022 at 3:50 PM David Marchand
<david.marchand@redhat.com> wrote:
> diff --git a/lib/eal/include/generic/rte_rwlock.h b/lib/eal/include/generic/rte_rwlock.h
> index da9bc3e9c0..dabbac0131 100644
> --- a/lib/eal/include/generic/rte_rwlock.h
> +++ b/lib/eal/include/generic/rte_rwlock.h
[snip]
> @@ -90,7 +92,8 @@ rte_rwlock_read_lock(rte_rwlock_t *rwl)
> */
> __rte_experimental
> static inline int
> -rte_rwlock_read_trylock(rte_rwlock_t *rwl)
> +rte_rwlock_read_trylock(rte_rwlock_t *rwl) RTE_SHARED_LOCK_TRYLOCK(1, rwl)
Should be 0.
> + RTE_NO_ANNOTATED_LOCK_CHECK
> {
> int32_t x;
> int success = 0;
> @@ -114,7 +117,8 @@ rte_rwlock_read_trylock(rte_rwlock_t *rwl)
> * A pointer to the rwlock structure.
> */
> static inline void
> -rte_rwlock_read_unlock(rte_rwlock_t *rwl)
> +rte_rwlock_read_unlock(rte_rwlock_t *rwl) RTE_LOCK_RELEASES(rwl)
> + RTE_NO_ANNOTATED_LOCK_CHECK
> {
> __atomic_fetch_sub(&rwl->cnt, 1, __ATOMIC_RELEASE);
> }
> @@ -134,7 +138,8 @@ rte_rwlock_read_unlock(rte_rwlock_t *rwl)
> */
> __rte_experimental
> static inline int
> -rte_rwlock_write_trylock(rte_rwlock_t *rwl)
> +rte_rwlock_write_trylock(rte_rwlock_t *rwl) RTE_EXC_LOCK_TRYLOCK(1, rwl)
Should be 0.
> + RTE_NO_ANNOTATED_LOCK_CHECK
> {
> int32_t x;
>
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [RFC PATCH v2 4/9] vhost: fix async access
2022-03-31 8:00 ` Maxime Coquelin
@ 2022-03-31 10:23 ` Hu, Jiayu
0 siblings, 0 replies; 110+ messages in thread
From: Hu, Jiayu @ 2022-03-31 10:23 UTC (permalink / raw)
To: Maxime Coquelin, David Marchand, dev
Cc: Xia, Chenbo, Wang, YuanX, Ding, Xuan, stable, Patrick Fu
> -----Original Message-----
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> Sent: Thursday, March 31, 2022 4:00 PM
> To: David Marchand <david.marchand@redhat.com>; dev@dpdk.org
> Cc: Xia, Chenbo <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>;
> Wang, YuanX <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> stable@dpdk.org; Patrick Fu <patrick.fu@intel.com>
> Subject: Re: [RFC PATCH v2 4/9] vhost: fix async access
>
> Hi Jiayu,
>
> On 3/30/22 15:49, David Marchand wrote:
> > vq->async accesses must be protected with vq->access_lock.
> >
> > Fixes: eb666d24085f ("vhost: fix async unregister deadlock")
> > Fixes: 0c0935c5f794 ("vhost: allow to check in-flight packets for
> > async vhost")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > lib/vhost/vhost.c | 25 ++++++++++---------------
> > 1 file changed, 10 insertions(+), 15 deletions(-)
>
> Could you please test and review below patch?
> We may want to apply it early, before the annotation series is applied.
Sure, I will review them.
Thanks,
Jiayu
>
> Thanks!
> Maxime
>
> > diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c index
> > 2f96a28dac..a93e41f314 100644
> > --- a/lib/vhost/vhost.c
> > +++ b/lib/vhost/vhost.c
> > @@ -1753,27 +1753,23 @@ rte_vhost_async_channel_unregister(int vid,
> uint16_t queue_id)
> > if (vq == NULL)
> > return ret;
> >
> > - ret = 0;
> > -
> > - if (!vq->async)
> > - return ret;
> > -
> > if (!rte_spinlock_trylock(&vq->access_lock)) {
> > VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async
> channel, virtqueue busy.\n",
> > dev->ifname);
> > - return -1;
> > + return ret;
> > }
> >
> > - if (vq->async->pkts_inflight_n) {
> > + if (!vq->async) {
> > + ret = 0;
> > + } else if (vq->async->pkts_inflight_n) {
> > VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async
> channel.\n", dev->ifname);
> > VHOST_LOG_CONFIG(ERR, "(%s) inflight packets must be
> completed before unregistration.\n",
> > dev->ifname);
> > - ret = -1;
> > - goto out;
> > + } else {
> > + vhost_free_async_mem(vq);
> > + ret = 0;
> > }
> >
> > - vhost_free_async_mem(vq);
> > -out:
> > rte_spinlock_unlock(&vq->access_lock);
> >
> > return ret;
> > @@ -1891,9 +1887,6 @@ rte_vhost_async_get_inflight(int vid, uint16_t
> queue_id)
> > if (vq == NULL)
> > return ret;
> >
> > - if (!vq->async)
> > - return ret;
> > -
> > if (!rte_spinlock_trylock(&vq->access_lock)) {
> > VHOST_LOG_CONFIG(DEBUG,
> > "(%s) failed to check in-flight packets. virtqueue
> busy.\n", @@
> > -1901,7 +1894,9 @@ rte_vhost_async_get_inflight(int vid, uint16_t
> queue_id)
> > return ret;
> > }
> >
> > - ret = vq->async->pkts_inflight_n;
> > + if (vq->async)
> > + ret = vq->async->pkts_inflight_n;
> > +
> > rte_spinlock_unlock(&vq->access_lock);
> >
> > return ret;
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock
2022-03-30 13:49 ` [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock David Marchand
2022-03-31 9:22 ` David Marchand
@ 2022-04-04 6:21 ` Stephen Hemminger
2022-04-07 8:20 ` David Marchand
1 sibling, 1 reply; 110+ messages in thread
From: Stephen Hemminger @ 2022-04-04 6:21 UTC (permalink / raw)
To: David Marchand
Cc: dev, maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, Ruifeng Wang, Jan Viktorin, David Christensen,
Bruce Richardson, Konstantin Ananyev
On Wed, 30 Mar 2022 15:49:49 +0200
David Marchand <david.marchand@redhat.com> wrote:
> +#ifdef RTE_ANNOTATE_LOCKS
> +
> +#define RTE_ANNOTATED_LOCK \
> + __attribute__((lockable))
> +
> +#define RTE_GUARDED_BY(...) \
> + __attribute__((guarded_by(__VA_ARGS__)))
> +#define RTE_GUARDED_VAR \
> + __attribute__((guarded_var))
Could we use attributes that match the existing syntax and lower case.
That is what was done for hot/cold and format attributes.
#define __rte_lockable
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [RFC PATCH v2 4/9] vhost: fix async access
2022-03-30 13:49 ` [RFC PATCH v2 4/9] vhost: fix async access David Marchand
2022-03-31 8:00 ` Maxime Coquelin
@ 2022-04-04 6:57 ` Pai G, Sunil
1 sibling, 0 replies; 110+ messages in thread
From: Pai G, Sunil @ 2022-04-04 6:57 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, Xia, Chenbo, Hu, Jiayu, Wang, YuanX, Ding, Xuan,
stable, Patrick Fu
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, March 30, 2022 7:20 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; Xia, Chenbo <chenbo.xia@intel.com>; Hu,
> Jiayu <jiayu.hu@intel.com>; Wang, YuanX <yuanx.wang@intel.com>; Ding, Xuan
> <xuan.ding@intel.com>; stable@dpdk.org; Patrick Fu <patrick.fu@intel.com>
> Subject: [RFC PATCH v2 4/9] vhost: fix async access
>
> vq->async accesses must be protected with vq->access_lock.
>
> Fixes: eb666d24085f ("vhost: fix async unregister deadlock")
> Fixes: 0c0935c5f794 ("vhost: allow to check in-flight packets for async
> vhost")
> Cc: stable@dpdk.org
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost.c | 25 ++++++++++---------------
> 1 file changed, 10 insertions(+), 15 deletions(-)
Thanks for the fix, David.
Tested the changes for rte_vhost_async_get_inflight, works as expected.
Although I couldn't test, the changes for rte_vhost_async_channel_unregister looks good to me .
Based on that,
Acked-by: Sunil Pai G <sunil.pai.g@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v2 0/9] vhost lock annotations
2022-03-30 14:37 ` Ali Alnubani
@ 2022-04-05 7:11 ` David Marchand
0 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-04-05 7:11 UTC (permalink / raw)
To: Ali Alnubani
Cc: dev, Maxime Coquelin, Xia, Chenbo, Jiayu Hu, Wang, YuanX,
Xuan Ding, Kevin Traynor
On Wed, Mar 30, 2022 at 4:37 PM Ali Alnubani <alialnu@nvidia.com> wrote:
>
> [..]
> > It looks like mimecast shot the first patch (which I sent in place of
> > Maxime, because this series should go through the main repo).
> >
> > Looking at the mail source, I see:
> >
> > X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation
> > Protection Definition;Similar Internal Domain=false;Similar Monitored
> > External Domain=false;Custom External Domain=false;Mimecast External
> > Domain=false;Newly Observed Domain=false;Internal User
> > Name=false;Custom Display Name List=false;Reply-to Address
> > Mismatch=false;Targeted Threat Dictionary=false;Mimecast Threat
> > Dictionary=false;Custom Threat Dictionary=false
> >
> > I don't know how to understand this...
> > But as a result, the series is missing this patch in patchwork.
>
> I believe it was ignored by Patchwork because of its content-type (application/octet-stream), which indicates that the message contains binary data instead of text:
> > Content-Transfer-Encoding: 8bit
> > Content-Type: application/octet-stream; x-default=true
Probably the consequence of some Mimecast mangling.
I noticed similar issues in my inbox for some Luca mails on stable@
and some mails from @Intel people on dev@ and even on netdev@.
In all cases where From: contains a name != sender, my inbox got the
same "content stripped and attached" symptom.
On the other hand, those mails end up fine in public mail archives.
Looking at headers of publicly archived mails and comparing with what
I got, there is an additional trace of Mimecast between dpdk.org
server and my inbox.
I opened a IT ticket internally.
I hope it will get fixed quickly.
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [RFC PATCH v2 3/9] vhost: annotate virtqueue access lock
2022-03-30 13:49 ` [RFC PATCH v2 3/9] vhost: annotate virtqueue access lock David Marchand
@ 2022-04-07 1:40 ` Hu, Jiayu
2022-04-07 7:03 ` David Marchand
0 siblings, 1 reply; 110+ messages in thread
From: Hu, Jiayu @ 2022-04-07 1:40 UTC (permalink / raw)
To: David Marchand, dev; +Cc: maxime.coquelin, Xia, Chenbo, Wang, YuanX, Ding, Xuan
Hi David,
Please see replies inline.
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, March 30, 2022 9:50 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; Xia, Chenbo <chenbo.xia@intel.com>; Hu,
> Jiayu <jiayu.hu@intel.com>; Wang, YuanX <yuanx.wang@intel.com>; Ding,
> Xuan <xuan.ding@intel.com>
> Subject: [RFC PATCH v2 3/9] vhost: annotate virtqueue access lock
>
> This change simply annotates existing paths of the code leading to
> manipulations of the vq->access_lock.
>
> One small change is required: vhost_poll_enqueue_completed was getting a
> queue_id to get hold of the vq, while its callers already knew of the vq. For
> the annotation sake, vq is now directly passed.
>
> vhost_user_lock_all_queue_pairs and vhost_user_unlock_all_queue_pairs
> are skipped since vq->access_lock are conditionally held.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost.h | 2 ++
> lib/vhost/vhost_user.c | 2 ++
> lib/vhost/virtio_net.c | 16 ++++++++++++----
> 3 files changed, 16 insertions(+), 4 deletions(-)
>
> diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index
> a9edc271aa..158460b7d7 100644
> --- a/lib/vhost/vhost.h
> +++ b/lib/vhost/vhost.h
> @@ -834,6 +834,7 @@ vhost_need_event(uint16_t event_idx, uint16_t
> new_idx, uint16_t old)
>
> static __rte_always_inline void
> vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
vhost_vring_call_split() is called in rte_vhost_vring_call() too, but it doesn't
acquire vq->access_lock before calling vhost_vring_call_split().
> {
> /* Flush used->idx update before we read avail->flags. */
> rte_atomic_thread_fence(__ATOMIC_SEQ_CST);
> @@ -872,6 +873,7 @@ vhost_vring_call_split(struct virtio_net *dev, struct
> vhost_virtqueue *vq)
>
> static __rte_always_inline void
> vhost_vring_call_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
Ditto.
> {
> uint16_t old, new, off, off_wrap;
> bool signalled_used_valid, kick = false; diff --git
> a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c index
> 1d390677fa..87eaa2ab4a 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -2909,6 +2909,7 @@ vhost_user_check_and_alloc_queue_pair(struct
> virtio_net *dev,
>
> static void
> vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
> + RTE_NO_ANNOTATED_LOCK_CHECK
> {
> unsigned int i = 0;
> unsigned int vq_num = 0;
> @@ -2926,6 +2927,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net
> *dev)
>
> static void
> vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
> + RTE_NO_ANNOTATED_LOCK_CHECK
> {
> unsigned int i = 0;
> unsigned int vq_num = 0;
> diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c index
> 5f432b0d77..514ee00993 100644
> --- a/lib/vhost/virtio_net.c
> +++ b/lib/vhost/virtio_net.c
> @@ -1246,6 +1246,7 @@ vhost_enqueue_single_packed(struct virtio_net
> *dev, static __rte_noinline uint32_t virtio_dev_rx_split(struct virtio_net
> *dev, struct vhost_virtqueue *vq,
> struct rte_mbuf **pkts, uint32_t count)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
> {
> uint32_t pkt_idx = 0;
> uint16_t num_buffers;
> @@ -1441,6 +1442,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
> struct vhost_virtqueue *__rte_restrict vq,
> struct rte_mbuf **__rte_restrict pkts,
> uint32_t count)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
> {
> uint32_t pkt_idx = 0;
>
> @@ -1955,11 +1957,11 @@ write_back_completed_descs_packed(struct
> vhost_virtqueue *vq, }
>
> static __rte_always_inline uint16_t
> -vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id,
> +vhost_poll_enqueue_completed(struct virtio_net *dev, struct
> +vhost_virtqueue *vq,
> struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
> uint16_t vchan_id)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
rte_vhost_clear_queue_thread_unsafe() doesn't acquire vq->access_lock.
Will it cause a compiler warning?
Thanks,
Jiayu
> {
> - struct vhost_virtqueue *vq = dev->virtqueue[queue_id];
> struct vhost_async *async = vq->async;
> struct async_inflight_info *pkts_info = async->pkts_info;
> uint16_t nr_cpl_pkts = 0;
> @@ -2062,7 +2064,7 @@ rte_vhost_poll_enqueue_completed(int vid,
> uint16_t queue_id,
> goto out;
> }
>
> - n_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts,
> count, dma_id, vchan_id);
> + n_pkts_cpl = vhost_poll_enqueue_completed(dev, vq, pkts, count,
> +dma_id, vchan_id);
>
> out:
> rte_spinlock_unlock(&vq->access_lock);
> @@ -2104,7 +2106,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid,
> uint16_t queue_id,
> return 0;
> }
>
> - n_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts,
> count, dma_id, vchan_id);
> + n_pkts_cpl = vhost_poll_enqueue_completed(dev, vq, pkts, count,
> +dma_id, vchan_id);
>
> return n_pkts_cpl;
> }
> @@ -2679,6 +2681,7 @@ static uint16_t
> virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
> struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t
> count,
> bool legacy_ol_flags)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
> {
> uint16_t i;
> uint16_t free_entries;
> @@ -2774,6 +2777,7 @@ static uint16_t
> virtio_dev_tx_split_legacy(struct virtio_net *dev,
> struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
> struct rte_mbuf **pkts, uint16_t count)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
> {
> return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true); }
> @@ -2783,6 +2787,7 @@ static uint16_t
> virtio_dev_tx_split_compliant(struct virtio_net *dev,
> struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
> struct rte_mbuf **pkts, uint16_t count)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
> {
> return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false); }
> @@ -2982,6 +2987,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
> struct rte_mbuf **__rte_restrict pkts,
> uint32_t count,
> bool legacy_ol_flags)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
> {
> uint32_t pkt_idx = 0;
>
> @@ -3025,6 +3031,7 @@ static uint16_t
> virtio_dev_tx_packed_legacy(struct virtio_net *dev,
> struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool
> *mbuf_pool,
> struct rte_mbuf **__rte_restrict pkts, uint32_t count)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
> {
> return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true); }
> @@ -3034,6 +3041,7 @@ static uint16_t
> virtio_dev_tx_packed_compliant(struct virtio_net *dev,
> struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool
> *mbuf_pool,
> struct rte_mbuf **__rte_restrict pkts, uint32_t count)
> + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
> {
> return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false); }
> --
> 2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v2 3/9] vhost: annotate virtqueue access lock
2022-04-07 1:40 ` Hu, Jiayu
@ 2022-04-07 7:03 ` David Marchand
0 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-04-07 7:03 UTC (permalink / raw)
To: Hu, Jiayu; +Cc: dev, maxime.coquelin, Xia, Chenbo, Wang, YuanX, Ding, Xuan
On Thu, Apr 7, 2022 at 3:40 AM Hu, Jiayu <jiayu.hu@intel.com> wrote:
> > diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h index
> > a9edc271aa..158460b7d7 100644
> > --- a/lib/vhost/vhost.h
> > +++ b/lib/vhost/vhost.h
> > @@ -834,6 +834,7 @@ vhost_need_event(uint16_t event_idx, uint16_t
> > new_idx, uint16_t old)
> >
> > static __rte_always_inline void
> > vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
> > + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
>
> vhost_vring_call_split() is called in rte_vhost_vring_call() too, but it doesn't
> acquire vq->access_lock before calling vhost_vring_call_split().
I have some issues with sending patches from other people (Mimecast
seems to think I try to impersonate them and strip the content of the
mail?).
You'll notice the series in patchwork starts at patch 2.
https://patchwork.dpdk.org/project/dpdk/list/?series=22292&state=*
My intent was to have Maxime fix (already in next-virtio:
https://git.dpdk.org/next/dpdk-next-virtio/commit/?id=53d8fffcf8e3c89c9785f8ce50db892f2cdfd7c7)
in this series.
[snip]
> > @@ -1955,11 +1957,11 @@ write_back_completed_descs_packed(struct
> > vhost_virtqueue *vq, }
> >
> > static __rte_always_inline uint16_t
> > -vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id,
> > +vhost_poll_enqueue_completed(struct virtio_net *dev, struct
> > +vhost_virtqueue *vq,
> > struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
> > uint16_t vchan_id)
> > + RTE_EXC_LOCK_REQUIRES(vq->access_lock)
>
> rte_vhost_clear_queue_thread_unsafe() doesn't acquire vq->access_lock.
> Will it cause a compiler warning?
Mm, probably a rebase/split error on my side when doing rfc v2.
On the other hand I don't think we can enable the check at this point
of the series in any case (there would be other warnings, at least for
rwlocks).
I'll double check before sending next revision, thanks for pointing out.
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock
2022-04-04 6:21 ` Stephen Hemminger
@ 2022-04-07 8:20 ` David Marchand
0 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-04-07 8:20 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, Maxime Coquelin, Xia, Chenbo, Jiayu Hu, Wang, YuanX,
Xuan Ding, Ruifeng Wang, Jan Viktorin, David Christensen,
Bruce Richardson, Konstantin Ananyev
On Mon, Apr 4, 2022 at 8:21 AM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Wed, 30 Mar 2022 15:49:49 +0200
> David Marchand <david.marchand@redhat.com> wrote:
>
> > +#ifdef RTE_ANNOTATE_LOCKS
> > +
> > +#define RTE_ANNOTATED_LOCK \
> > + __attribute__((lockable))
> > +
> > +#define RTE_GUARDED_BY(...) \
> > + __attribute__((guarded_by(__VA_ARGS__)))
> > +#define RTE_GUARDED_VAR \
> > + __attribute__((guarded_var))
>
> Could we use attributes that match the existing syntax and lower case.
> That is what was done for hot/cold and format attributes.
Yes, I reconsidered and I'll do that.
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 0/8] vhost lock annotations
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
` (5 preceding siblings ...)
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 1/8] eal: annotate spinlock and rwlock David Marchand
` (7 more replies)
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
` (2 subsequent siblings)
9 siblings, 8 replies; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vhost internals involves multiple locks to protect data access by
multiple threads.
This series is a try at using clang thread safety checks [1] to catch
issues during compilation: EAL spinlock and rwlock are annotated and
vhost code is instrumented so that clang can statically check
correctness.
This is still a work in progress (some documentation and a release note
update are missing).
Those annotations are quite heavy to maintain because the full path of
code must be annotated, but I think it is worth using.
1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
Depends-on: patch-108840 ("vhost: fix missing virtqueue lock protection")
--
David Marchand
Changes since RFC v2:
- fixed trylock annotations for rwlock,
- annotated _tm flavors of spinlock and rwlock,
- removed Maxime vhost fix from series (since Mimecast does not like
me sending Maxime patch...), added a dependency on original fix
as a hint for reviewers,
- renamed attributes,
Changes since RFC v1:
- Cc'd people who have pending patches for vhost,
- moved annotations to EAL and removed wrappers in vhost,
- as a result of moving to EAL, this series will be tested against
the main repo, so patch 1 has been kept as part of the series
even if already applied to next-virtio,
- refined/split patches and annotated all spinlocks in vhost,
David Marchand (8):
eal: annotate spinlock and rwlock
vhost: annotate virtqueue access lock
vhost: fix async access
vhost: annotate async accesses
vhost: annotate need reply handling
vhost: annotate vDPA device list accesses
vhost: annotate IOTLB locks
vhost: enable lock check
drivers/meson.build | 5 ++
lib/eal/arm/include/rte_rwlock.h | 4 ++
lib/eal/arm/include/rte_spinlock.h | 6 ++
lib/eal/include/generic/rte_rwlock.h | 27 +++++++--
lib/eal/include/generic/rte_spinlock.h | 40 +++++++++----
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 67 +++++++++++++++++++++
lib/eal/ppc/include/rte_rwlock.h | 4 ++
lib/eal/ppc/include/rte_spinlock.h | 9 +++
lib/eal/x86/include/rte_rwlock.h | 4 ++
lib/eal/x86/include/rte_spinlock.h | 9 +++
lib/meson.build | 5 ++
lib/vhost/iotlb.h | 8 +++
lib/vhost/meson.build | 2 +
lib/vhost/vdpa.c | 18 +++---
lib/vhost/vhost.c | 50 ++++++++--------
lib/vhost/vhost.h | 26 ++++++---
lib/vhost/vhost_crypto.c | 7 +++
lib/vhost/vhost_user.c | 6 ++
lib/vhost/virtio_net.c | 80 ++++++++++++++++++++------
20 files changed, 306 insertions(+), 72 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 1/8] eal: annotate spinlock and rwlock
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-21 13:48 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 2/8] vhost: annotate virtqueue access lock David Marchand
` (6 subsequent siblings)
7 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, Ruifeng Wang, Jan Viktorin, David Christensen,
Bruce Richardson, Konstantin Ananyev
clang offers some thread safety checks, statically verifying that locks
are taken and released in the code.
To use those checks, the full code leading to taking or releasing locks
must be annotated with some attributes.
Wrap those attributes into our own set of macros.
rwlock and the "normal" spinlock are instrumented.
A component may enable this check by setting annotate_locks = true
in its meson.build.
Note: those checks might be of interest out of DPDK, but it
requires that the including application locks are annotated.
On the other hand, applications out there might have been using
those same checks.
To be on the safe side, keep this instrumentation under a
RTE_ANNOTATE_LOCKS internal build flag.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since RFC v2:
- fixed rwlock trylock,
- instrumented _tm spinlocks,
- aligned attribute names to clang,
---
drivers/meson.build | 5 ++
lib/eal/arm/include/rte_rwlock.h | 4 ++
lib/eal/arm/include/rte_spinlock.h | 6 +++
lib/eal/include/generic/rte_rwlock.h | 27 +++++++++--
lib/eal/include/generic/rte_spinlock.h | 40 ++++++++++-----
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 67 ++++++++++++++++++++++++++
lib/eal/ppc/include/rte_rwlock.h | 4 ++
lib/eal/ppc/include/rte_spinlock.h | 9 ++++
lib/eal/x86/include/rte_rwlock.h | 4 ++
lib/eal/x86/include/rte_spinlock.h | 9 ++++
lib/meson.build | 5 ++
12 files changed, 164 insertions(+), 17 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
diff --git a/drivers/meson.build b/drivers/meson.build
index 1d8123b00c..c81c8c0eff 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -87,6 +87,7 @@ foreach subpath:subdirs
build = true # set to false to disable, e.g. missing deps
reason = '<unknown reason>' # set if build == false to explain
name = drv
+ annotate_locks = false
sources = []
headers = []
objs = []
@@ -152,6 +153,10 @@ foreach subpath:subdirs
enabled_drivers += name
lib_name = '_'.join(['rte', class, name])
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=' + '.'.join([log_prefix, name])
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
dpdk_conf.set(lib_name.to_upper(), 1)
dpdk_extra_ldflags += pkgconfig_extra_libs
diff --git a/lib/eal/arm/include/rte_rwlock.h b/lib/eal/arm/include/rte_rwlock.h
index 18bb37b036..575f171670 100644
--- a/lib/eal/arm/include/rte_rwlock.h
+++ b/lib/eal/arm/include/rte_rwlock.h
@@ -13,24 +13,28 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_read_lock(rwl);
}
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_read_unlock(rwl);
}
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_write_lock(rwl);
}
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_write_unlock(rwl);
}
diff --git a/lib/eal/arm/include/rte_spinlock.h b/lib/eal/arm/include/rte_spinlock.h
index a973763c23..2970de5c3c 100644
--- a/lib/eal/arm/include/rte_spinlock.h
+++ b/lib/eal/arm/include/rte_spinlock.h
@@ -23,36 +23,42 @@ static inline int rte_tm_supported(void)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
rte_spinlock_lock(sl); /* fall-back */
}
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
return rte_spinlock_trylock(sl);
}
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
rte_spinlock_unlock(sl);
}
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
rte_spinlock_recursive_lock(slr); /* fall-back */
}
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
rte_spinlock_recursive_unlock(slr);
}
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
return rte_spinlock_recursive_trylock(slr);
}
diff --git a/lib/eal/include/generic/rte_rwlock.h b/lib/eal/include/generic/rte_rwlock.h
index da9bc3e9c0..145851f9c0 100644
--- a/lib/eal/include/generic/rte_rwlock.h
+++ b/lib/eal/include/generic/rte_rwlock.h
@@ -23,6 +23,7 @@ extern "C" {
#include <rte_common.h>
#include <rte_atomic.h>
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
@@ -30,7 +31,7 @@ extern "C" {
*
* cnt is -1 when write lock is held, and > 0 when read locks are held.
*/
-typedef struct {
+typedef struct __rte_lockable {
volatile int32_t cnt; /**< -1 when W lock held, > 0 when R locks held. */
} rte_rwlock_t;
@@ -59,6 +60,8 @@ rte_rwlock_init(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_read_lock(rte_rwlock_t *rwl)
+ __rte_shared_lock_function(rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
int success = 0;
@@ -91,6 +94,8 @@ rte_rwlock_read_lock(rte_rwlock_t *rwl)
__rte_experimental
static inline int
rte_rwlock_read_trylock(rte_rwlock_t *rwl)
+ __rte_shared_trylock_function(0, rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
int success = 0;
@@ -115,6 +120,8 @@ rte_rwlock_read_trylock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_read_unlock(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl)
+ __rte_no_thread_safety_analysis
{
__atomic_fetch_sub(&rwl->cnt, 1, __ATOMIC_RELEASE);
}
@@ -135,6 +142,8 @@ rte_rwlock_read_unlock(rte_rwlock_t *rwl)
__rte_experimental
static inline int
rte_rwlock_write_trylock(rte_rwlock_t *rwl)
+ __rte_exclusive_trylock_function(0, rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -154,6 +163,8 @@ rte_rwlock_write_trylock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_write_lock(rte_rwlock_t *rwl)
+ __rte_exclusive_lock_function(rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
int success = 0;
@@ -178,6 +189,8 @@ rte_rwlock_write_lock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_write_unlock(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl)
+ __rte_no_thread_safety_analysis
{
__atomic_store_n(&rwl->cnt, 0, __ATOMIC_RELEASE);
}
@@ -196,7 +209,8 @@ rte_rwlock_write_unlock(rte_rwlock_t *rwl)
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_read_lock_tm(rte_rwlock_t *rwl);
+rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_shared_lock_function(rwl);
/**
* Commit hardware memory transaction or release the read lock if the lock is used as a fall-back
@@ -205,7 +219,8 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl);
* A pointer to the rwlock structure.
*/
static inline void
-rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl);
+rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl);
/**
* Try to execute critical section in a hardware memory transaction, if it
@@ -221,7 +236,8 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl);
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_lock_tm(rte_rwlock_t *rwl);
+rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_exclusive_lock_function(rwl);
/**
* Commit hardware memory transaction or release the write lock if the lock is used as a fall-back
@@ -230,7 +246,8 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl);
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl);
+rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl);
#ifdef __cplusplus
}
diff --git a/lib/eal/include/generic/rte_spinlock.h b/lib/eal/include/generic/rte_spinlock.h
index 40fe49d5ad..684bfac96c 100644
--- a/lib/eal/include/generic/rte_spinlock.h
+++ b/lib/eal/include/generic/rte_spinlock.h
@@ -22,12 +22,13 @@
#ifdef RTE_FORCE_INTRINSICS
#include <rte_common.h>
#endif
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
* The rte_spinlock_t type.
*/
-typedef struct {
+typedef struct __rte_lockable {
volatile int locked; /**< lock status 0 = unlocked, 1 = locked */
} rte_spinlock_t;
@@ -55,11 +56,13 @@ rte_spinlock_init(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock(rte_spinlock_t *sl);
+rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_exclusive_lock_function(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int exp = 0;
@@ -79,11 +82,13 @@ rte_spinlock_lock(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl);
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_unlock_function(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl)
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
__atomic_store_n(&sl->locked, 0, __ATOMIC_RELEASE);
}
@@ -98,11 +103,13 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
* 1 if the lock is successfully taken; 0 otherwise.
*/
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl);
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_exclusive_trylock_function(1, sl);
#ifdef RTE_FORCE_INTRINSICS
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl)
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int exp = 0;
return __atomic_compare_exchange_n(&sl->locked, &exp, 1,
@@ -146,7 +153,8 @@ static inline int rte_tm_supported(void);
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock_tm(rte_spinlock_t *sl);
+rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_exclusive_lock_function(sl);
/**
* Commit hardware memory transaction or release the spinlock if
@@ -156,7 +164,8 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl);
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock_tm(rte_spinlock_t *sl);
+rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_unlock_function(sl);
/**
* Try to execute critical section in a hardware memory transaction,
@@ -175,7 +184,8 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl);
* or lock is successfully taken; 0 otherwise.
*/
static inline int
-rte_spinlock_trylock_tm(rte_spinlock_t *sl);
+rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_exclusive_trylock_function(1, sl);
/**
* The rte_spinlock_recursive_t type.
@@ -211,6 +221,7 @@ static inline void rte_spinlock_recursive_init(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
int id = rte_gettid();
@@ -227,6 +238,7 @@ static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (--(slr->count) == 0) {
slr->user = -1;
@@ -244,6 +256,7 @@ static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
* 1 if the lock is successfully taken; 0 otherwise.
*/
static inline int rte_spinlock_recursive_trylock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
int id = rte_gettid();
@@ -271,7 +284,8 @@ static inline int rte_spinlock_recursive_trylock(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_lock_tm(
- rte_spinlock_recursive_t *slr);
+ rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis;
/**
* Commit hardware memory transaction or release the recursive spinlock
@@ -281,7 +295,8 @@ static inline void rte_spinlock_recursive_lock_tm(
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_unlock_tm(
- rte_spinlock_recursive_t *slr);
+ rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis;
/**
* Try to execute critical section in a hardware memory transaction,
@@ -300,6 +315,7 @@ static inline void rte_spinlock_recursive_unlock_tm(
* or lock is successfully taken; 0 otherwise.
*/
static inline int rte_spinlock_recursive_trylock_tm(
- rte_spinlock_recursive_t *slr);
+ rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis;
#endif /* _RTE_SPINLOCK_H_ */
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index 9700494816..a20b8d7e23 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -27,6 +27,7 @@ headers += files(
'rte_keepalive.h',
'rte_launch.h',
'rte_lcore.h',
+ 'rte_lock_annotations.h',
'rte_log.h',
'rte_malloc.h',
'rte_memory.h',
diff --git a/lib/eal/include/rte_lock_annotations.h b/lib/eal/include/rte_lock_annotations.h
new file mode 100644
index 0000000000..bcaf4193f7
--- /dev/null
+++ b/lib/eal/include/rte_lock_annotations.h
@@ -0,0 +1,67 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Red Hat, Inc.
+ */
+
+#ifndef RTE_LOCK_ANNOTATIONS_H
+#define RTE_LOCK_ANNOTATIONS_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifdef RTE_ANNOTATE_LOCKS
+
+#define __rte_lockable \
+ __attribute__((lockable))
+
+#define __rte_guarded_by(...) \
+ __attribute__((guarded_by(__VA_ARGS__)))
+#define __rte_guarded_var \
+ __attribute__((guarded_var))
+
+#define __rte_exclusive_locks_required(...) \
+ __attribute__((exclusive_locks_required(__VA_ARGS__)))
+#define __rte_exclusive_lock_function(...) \
+ __attribute__((exclusive_lock_function(__VA_ARGS__)))
+#define __rte_exclusive_trylock_function(ret, ...) \
+ __attribute__((exclusive_trylock_function(ret, __VA_ARGS__)))
+
+#define __rte_shared_locks_required(...) \
+ __attribute__((shared_locks_required(__VA_ARGS__)))
+#define __rte_shared_lock_function(...) \
+ __attribute__((shared_lock_function(__VA_ARGS__)))
+#define __rte_shared_trylock_function(ret, ...) \
+ __attribute__((shared_trylock_function(ret, __VA_ARGS__)))
+
+#define __rte_unlock_function(...) \
+ __attribute__((unlock_function(__VA_ARGS__)))
+
+#define __rte_no_thread_safety_analysis \
+ __attribute__((no_thread_safety_analysis))
+
+#else /* ! RTE_ANNOTATE_LOCKS */
+
+#define __rte_lockable
+
+#define __rte_guarded_by(...)
+#define __rte_guarded_var
+
+#define __rte_exclusive_locks_required(...)
+#define __rte_exclusive_lock_function(...)
+#define __rte_exclusive_trylock_function(...)
+
+#define __rte_shared_locks_required(...)
+#define __rte_shared_lock_function(...)
+#define __rte_shared_trylock_function(...)
+
+#define __rte_unlock_function(...)
+
+#define __rte_no_thread_safety_analysis
+
+#endif /* RTE_ANNOTATE_LOCKS */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_LOCK_ANNOTATIONS_H */
diff --git a/lib/eal/ppc/include/rte_rwlock.h b/lib/eal/ppc/include/rte_rwlock.h
index 9fadc04076..d7a5b61b8e 100644
--- a/lib/eal/ppc/include/rte_rwlock.h
+++ b/lib/eal/ppc/include/rte_rwlock.h
@@ -11,24 +11,28 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_read_lock(rwl);
}
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_read_unlock(rwl);
}
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_write_lock(rwl);
}
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_write_unlock(rwl);
}
diff --git a/lib/eal/ppc/include/rte_spinlock.h b/lib/eal/ppc/include/rte_spinlock.h
index 149ec245c7..979abea16d 100644
--- a/lib/eal/ppc/include/rte_spinlock.h
+++ b/lib/eal/ppc/include/rte_spinlock.h
@@ -20,6 +20,7 @@ extern "C" {
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
while (__sync_lock_test_and_set(&sl->locked, 1))
while (sl->locked)
@@ -28,12 +29,14 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
__sync_lock_release(&sl->locked);
}
static inline int
rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
return __sync_lock_test_and_set(&sl->locked, 1) == 0;
}
@@ -47,36 +50,42 @@ static inline int rte_tm_supported(void)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
rte_spinlock_lock(sl); /* fall-back */
}
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
return rte_spinlock_trylock(sl);
}
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
rte_spinlock_unlock(sl);
}
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
rte_spinlock_recursive_lock(slr); /* fall-back */
}
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
rte_spinlock_recursive_unlock(slr);
}
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
return rte_spinlock_recursive_trylock(slr);
}
diff --git a/lib/eal/x86/include/rte_rwlock.h b/lib/eal/x86/include/rte_rwlock.h
index eec4c7123c..1796b69265 100644
--- a/lib/eal/x86/include/rte_rwlock.h
+++ b/lib/eal/x86/include/rte_rwlock.h
@@ -14,6 +14,7 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -22,6 +23,7 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(rwl->cnt))
rte_rwlock_read_unlock(rwl);
@@ -31,6 +33,7 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -39,6 +42,7 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(rwl->cnt))
rte_rwlock_write_unlock(rwl);
diff --git a/lib/eal/x86/include/rte_spinlock.h b/lib/eal/x86/include/rte_spinlock.h
index e2e2b2643c..0b20ddfd73 100644
--- a/lib/eal/x86/include/rte_spinlock.h
+++ b/lib/eal/x86/include/rte_spinlock.h
@@ -23,6 +23,7 @@ extern "C" {
#ifndef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int lock_val = 1;
asm volatile (
@@ -43,6 +44,7 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock (rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int unlock_val = 0;
asm volatile (
@@ -54,6 +56,7 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock (rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int lockval = 1;
@@ -121,6 +124,7 @@ rte_try_tm(volatile int *lock)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&sl->locked)))
return;
@@ -130,6 +134,7 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&sl->locked)))
return 1;
@@ -139,6 +144,7 @@ rte_spinlock_trylock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(sl->locked))
rte_spinlock_unlock(sl);
@@ -148,6 +154,7 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&slr->sl.locked)))
return;
@@ -157,6 +164,7 @@ rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (unlikely(slr->sl.locked))
rte_spinlock_recursive_unlock(slr);
@@ -166,6 +174,7 @@ rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&slr->sl.locked)))
return 1;
diff --git a/lib/meson.build b/lib/meson.build
index 24adbe44c9..909133ea64 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -112,6 +112,7 @@ foreach l:libraries
reason = '<unknown reason>' # set if build == false to explain why
name = l
use_function_versioning = false
+ annotate_locks = false
sources = []
headers = []
indirect_headers = [] # public headers not directly included by apps
@@ -184,6 +185,10 @@ foreach l:libraries
cflags += '-DRTE_USE_FUNCTION_VERSIONING'
endif
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=lib.' + l
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
# first build static lib
static_lib = static_library(libname,
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 2/8] vhost: annotate virtqueue access lock
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 1/8] eal: annotate spinlock and rwlock David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-21 15:25 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 3/8] vhost: fix async access David Marchand
` (5 subsequent siblings)
7 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
This change simply annotates existing paths of the code leading to
manipulations of the vq->access_lock.
One small change is required: vhost_poll_enqueue_completed was getting
a queue_id to get hold of the vq, while its callers already knew of
the vq. For the annotation sake, vq is now directly passed.
vhost_user_lock_all_queue_pairs and vhost_user_unlock_all_queue_pairs
are skipped since vq->access_lock are conditionally held.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.h | 2 ++
lib/vhost/vhost_user.c | 2 ++
lib/vhost/virtio_net.c | 16 ++++++++++++----
3 files changed, 16 insertions(+), 4 deletions(-)
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index a9edc271aa..4a0aa35306 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -834,6 +834,7 @@ vhost_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old)
static __rte_always_inline void
vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(vq->access_lock)
{
/* Flush used->idx update before we read avail->flags. */
rte_atomic_thread_fence(__ATOMIC_SEQ_CST);
@@ -872,6 +873,7 @@ vhost_vring_call_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
static __rte_always_inline void
vhost_vring_call_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint16_t old, new, off, off_wrap;
bool signalled_used_valid, kick = false;
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 1d390677fa..dfa24fec09 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2909,6 +2909,7 @@ vhost_user_check_and_alloc_queue_pair(struct virtio_net *dev,
static void
vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
+ __rte_no_thread_safety_analysis
{
unsigned int i = 0;
unsigned int vq_num = 0;
@@ -2926,6 +2927,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
static void
vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
+ __rte_no_thread_safety_analysis
{
unsigned int i = 0;
unsigned int vq_num = 0;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 5f432b0d77..a8b91d4b20 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -1246,6 +1246,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1441,6 +1442,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -1955,11 +1957,11 @@ write_back_completed_descs_packed(struct vhost_virtqueue *vq,
}
static __rte_always_inline uint16_t
-vhost_poll_enqueue_completed(struct virtio_net *dev, uint16_t queue_id,
+vhost_poll_enqueue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id)
+ __rte_exclusive_locks_required(vq->access_lock)
{
- struct vhost_virtqueue *vq = dev->virtqueue[queue_id];
struct vhost_async *async = vq->async;
struct async_inflight_info *pkts_info = async->pkts_info;
uint16_t nr_cpl_pkts = 0;
@@ -2062,7 +2064,7 @@ rte_vhost_poll_enqueue_completed(int vid, uint16_t queue_id,
goto out;
}
- n_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count, dma_id, vchan_id);
+ n_pkts_cpl = vhost_poll_enqueue_completed(dev, vq, pkts, count, dma_id, vchan_id);
out:
rte_spinlock_unlock(&vq->access_lock);
@@ -2104,7 +2106,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
return 0;
}
- n_pkts_cpl = vhost_poll_enqueue_completed(dev, queue_id, pkts, count, dma_id, vchan_id);
+ n_pkts_cpl = vhost_poll_enqueue_completed(dev, vq, pkts, count, dma_id, vchan_id);
return n_pkts_cpl;
}
@@ -2679,6 +2681,7 @@ static uint16_t
virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint16_t i;
uint16_t free_entries;
@@ -2774,6 +2777,7 @@ static uint16_t
virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2783,6 +2787,7 @@ static uint16_t
virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -2982,6 +2987,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -3025,6 +3031,7 @@ static uint16_t
virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ __rte_exclusive_locks_required(vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3034,6 +3041,7 @@ static uint16_t
virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ __rte_exclusive_locks_required(vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 3/8] vhost: fix async access
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 1/8] eal: annotate spinlock and rwlock David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 2/8] vhost: annotate virtqueue access lock David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-21 19:21 ` Maxime Coquelin
2022-05-17 13:24 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 4/8] vhost: annotate async accesses David Marchand
` (4 subsequent siblings)
7 siblings, 2 replies; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, stable, Sunil Pai G, Patrick Fu
vq->async accesses must be protected with vq->access_lock.
Fixes: eb666d24085f ("vhost: fix async unregister deadlock")
Fixes: 0c0935c5f794 ("vhost: allow to check in-flight packets for async vhost")
Cc: stable@dpdk.org
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Sunil Pai G <sunil.pai.g@intel.com>
---
lib/vhost/vhost.c | 25 ++++++++++---------------
1 file changed, 10 insertions(+), 15 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 2f96a28dac..a93e41f314 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1753,27 +1753,23 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- ret = 0;
-
- if (!vq->async)
- return ret;
-
if (!rte_spinlock_trylock(&vq->access_lock)) {
VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel, virtqueue busy.\n",
dev->ifname);
- return -1;
+ return ret;
}
- if (vq->async->pkts_inflight_n) {
+ if (!vq->async) {
+ ret = 0;
+ } else if (vq->async->pkts_inflight_n) {
VHOST_LOG_CONFIG(ERR, "(%s) failed to unregister async channel.\n", dev->ifname);
VHOST_LOG_CONFIG(ERR, "(%s) inflight packets must be completed before unregistration.\n",
dev->ifname);
- ret = -1;
- goto out;
+ } else {
+ vhost_free_async_mem(vq);
+ ret = 0;
}
- vhost_free_async_mem(vq);
-out:
rte_spinlock_unlock(&vq->access_lock);
return ret;
@@ -1891,9 +1887,6 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- if (!vq->async)
- return ret;
-
if (!rte_spinlock_trylock(&vq->access_lock)) {
VHOST_LOG_CONFIG(DEBUG,
"(%s) failed to check in-flight packets. virtqueue busy.\n",
@@ -1901,7 +1894,9 @@ rte_vhost_async_get_inflight(int vid, uint16_t queue_id)
return ret;
}
- ret = vq->async->pkts_inflight_n;
+ if (vq->async)
+ ret = vq->async->pkts_inflight_n;
+
rte_spinlock_unlock(&vq->access_lock);
return ret;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 4/8] vhost: annotate async accesses
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
` (2 preceding siblings ...)
2022-04-11 11:00 ` [RFC PATCH v3 3/8] vhost: fix async access David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-22 7:20 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 5/8] vhost: annotate need reply handling David Marchand
` (3 subsequent siblings)
7 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vq->async is initialised and must be accessed under vq->access_lock.
Top level "_thread_unsafe" functions could be checked at runtime (clang
provides a lock aware assert()-like check), but they are simply skipped
because those functions are not called in-tree, and as a result,
their annotations would not be tested.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.c | 14 +++++++++-----
lib/vhost/vhost.h | 2 +-
lib/vhost/vhost_user.c | 2 ++
lib/vhost/virtio_net.c | 15 +++++++++++++++
4 files changed, 27 insertions(+), 6 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index a93e41f314..79236df000 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -334,6 +334,7 @@ cleanup_device(struct virtio_net *dev, int destroy)
static void
vhost_free_async_mem(struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(vq->access_lock)
{
if (!vq->async)
return;
@@ -352,6 +353,7 @@ vhost_free_async_mem(struct vhost_virtqueue *vq)
void
free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_no_thread_safety_analysis
{
if (vq_is_packed(dev))
rte_free(vq->shadow_used_packed);
@@ -1622,10 +1624,10 @@ rte_vhost_extern_callback_register(int vid,
}
static __rte_always_inline int
-async_channel_register(int vid, uint16_t queue_id)
+async_channel_register(struct virtio_net *dev, struct vhost_virtqueue *vq,
+ uint16_t queue_id)
+ __rte_exclusive_locks_required(vq->access_lock)
{
- struct virtio_net *dev = get_device(vid);
- struct vhost_virtqueue *vq = dev->virtqueue[queue_id];
struct vhost_async *async;
int node = vq->numa_node;
@@ -1709,7 +1711,7 @@ rte_vhost_async_channel_register(int vid, uint16_t queue_id)
return -1;
rte_spinlock_lock(&vq->access_lock);
- ret = async_channel_register(vid, queue_id);
+ ret = async_channel_register(dev, vq, queue_id);
rte_spinlock_unlock(&vq->access_lock);
return ret;
@@ -1717,6 +1719,7 @@ rte_vhost_async_channel_register(int vid, uint16_t queue_id)
int
rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
+ __rte_no_thread_safety_analysis
{
struct vhost_virtqueue *vq;
struct virtio_net *dev = get_device(vid);
@@ -1732,7 +1735,7 @@ rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
if (unlikely(vq == NULL || !dev->async_copy))
return -1;
- return async_channel_register(vid, queue_id);
+ return async_channel_register(dev, vq, queue_id);
}
int
@@ -1777,6 +1780,7 @@ rte_vhost_async_channel_unregister(int vid, uint16_t queue_id)
int
rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)
+ __rte_no_thread_safety_analysis
{
struct vhost_virtqueue *vq;
struct virtio_net *dev = get_device(vid);
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 4a0aa35306..c2065e5324 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -299,7 +299,7 @@ struct vhost_virtqueue {
struct rte_vhost_resubmit_info *resubmit_inflight;
uint64_t global_counter;
- struct vhost_async *async;
+ struct vhost_async *async __rte_guarded_var;
int notif_enable;
#define VIRTIO_UNINITIALIZED_NOTIF (-1)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index dfa24fec09..ee276a28f1 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2199,6 +2199,8 @@ static int
vhost_user_set_vring_enable(struct virtio_net **pdev,
struct vhu_msg_context *ctx,
int main_fd __rte_unused)
+ /* vq->access_lock is taken in vhost_user_lock_all_queue_pairs() */
+ __rte_no_thread_safety_analysis
{
struct virtio_net *dev = *pdev;
bool enable = !!ctx->msg.payload.state.num;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index a8b91d4b20..0c40d5a7a5 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -51,6 +51,7 @@ static __rte_always_inline int64_t
vhost_async_dma_transfer_one(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t flag_idx,
struct vhost_iov_iter *pkt)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
uint16_t ring_mask = dma_info->ring_mask;
@@ -99,6 +100,7 @@ static __rte_always_inline uint16_t
vhost_async_dma_transfer(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t head_idx,
struct vhost_iov_iter *pkts, uint16_t nr_pkts)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
int64_t ret, nr_copies = 0;
@@ -1000,6 +1002,7 @@ static __rte_always_inline int
async_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1057,6 +1060,7 @@ static __rte_always_inline int
mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1186,6 +1190,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkt,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1417,6 +1422,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1541,6 +1547,7 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
static __rte_always_inline uint16_t
async_get_first_inflight_pkt_idx(struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct vhost_async *async = vq->async;
@@ -1587,6 +1594,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1691,6 +1699,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs,
uint16_t *nr_buffers)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1748,6 +1757,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1766,6 +1776,7 @@ virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
uint32_t nr_err, uint32_t *pkt_idx)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint16_t descs_err = 0;
uint16_t buffers_err = 0;
@@ -1793,6 +1804,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(vq->access_lock)
{
uint32_t pkt_idx = 0;
uint32_t remained = count;
@@ -1863,6 +1875,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
static __rte_always_inline void
write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t nr_left = n_descs;
@@ -1895,6 +1908,7 @@ write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
static __rte_always_inline void
write_back_completed_descs_packed(struct vhost_virtqueue *vq,
uint16_t n_buffers)
+ __rte_exclusive_locks_required(vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t from = async->last_buffer_idx_packed;
@@ -2076,6 +2090,7 @@ uint16_t
rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id)
+ __rte_no_thread_safety_analysis
{
struct virtio_net *dev = get_device(vid);
struct vhost_virtqueue *vq;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 5/8] vhost: annotate need reply handling
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
` (3 preceding siblings ...)
2022-04-11 11:00 ` [RFC PATCH v3 4/8] vhost: annotate async accesses David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-22 7:25 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 6/8] vhost: annotate vDPA device list accesses David Marchand
` (2 subsequent siblings)
7 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
When a reply from the slave is required (VHOST_USER_NEED_REPLY flag),
a spinlock is taken before sending the message.
This spinlock is released if an error occurs when sending the message, and
once a reply is received.
A problem is that this lock is taken under a branch and annotating
conditionally held locks is not supported.
The code seems currently correct and, while we may rework the code,
it is easier to simply skip checks on slave_req_lock for those helpers.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost_user.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index ee276a28f1..d101d5072f 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2854,6 +2854,7 @@ send_vhost_reply(struct virtio_net *dev, int sockfd, struct vhu_msg_context *ctx
static int
send_vhost_slave_message(struct virtio_net *dev,
struct vhu_msg_context *ctx)
+ __rte_no_thread_safety_analysis
{
int ret;
@@ -3165,6 +3166,7 @@ vhost_user_msg_handler(int vid, int fd)
static int process_slave_message_reply(struct virtio_net *dev,
const struct vhu_msg_context *ctx)
+ __rte_no_thread_safety_analysis
{
struct vhu_msg_context msg_reply;
int ret;
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 6/8] vhost: annotate vDPA device list accesses
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
` (4 preceding siblings ...)
2022-04-11 11:00 ` [RFC PATCH v3 5/8] vhost: annotate need reply handling David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-22 7:26 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 7/8] vhost: annotate IOTLB locks David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 8/8] vhost: enable lock check David Marchand
7 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vdpa_device_list access must be protected with vdpa_device_list_lock
spinlock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vdpa.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index 8fa2153023..a3f9f8f072 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -22,21 +22,24 @@
/** Double linked list of vDPA devices. */
TAILQ_HEAD(vdpa_device_list, rte_vdpa_device);
-static struct vdpa_device_list vdpa_device_list =
- TAILQ_HEAD_INITIALIZER(vdpa_device_list);
+static struct vdpa_device_list vdpa_device_list__ =
+ TAILQ_HEAD_INITIALIZER(vdpa_device_list__);
static rte_spinlock_t vdpa_device_list_lock = RTE_SPINLOCK_INITIALIZER;
+static struct vdpa_device_list * const vdpa_device_list
+ __rte_guarded_by(&vdpa_device_list_lock) = &vdpa_device_list__;
/* Unsafe, needs to be called with vdpa_device_list_lock held */
static struct rte_vdpa_device *
__vdpa_find_device_by_name(const char *name)
+ __rte_exclusive_locks_required(&vdpa_device_list_lock)
{
struct rte_vdpa_device *dev, *ret = NULL;
if (name == NULL)
return NULL;
- TAILQ_FOREACH(dev, &vdpa_device_list, next) {
+ TAILQ_FOREACH(dev, vdpa_device_list, next) {
if (!strncmp(dev->device->name, name, RTE_DEV_NAME_MAX_LEN)) {
ret = dev;
break;
@@ -100,7 +103,7 @@ rte_vdpa_register_device(struct rte_device *rte_dev,
dev->device = rte_dev;
dev->ops = ops;
- TAILQ_INSERT_TAIL(&vdpa_device_list, dev, next);
+ TAILQ_INSERT_TAIL(vdpa_device_list, dev, next);
out_unlock:
rte_spinlock_unlock(&vdpa_device_list_lock);
@@ -114,11 +117,11 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int ret = -1;
rte_spinlock_lock(&vdpa_device_list_lock);
- RTE_TAILQ_FOREACH_SAFE(cur_dev, &vdpa_device_list, next, tmp_dev) {
+ RTE_TAILQ_FOREACH_SAFE(cur_dev, vdpa_device_list, next, tmp_dev) {
if (dev != cur_dev)
continue;
- TAILQ_REMOVE(&vdpa_device_list, dev, next);
+ TAILQ_REMOVE(vdpa_device_list, dev, next);
rte_free(dev);
ret = 0;
break;
@@ -316,7 +319,7 @@ vdpa_find_device(const struct rte_vdpa_device *start, rte_vdpa_cmp_t cmp,
rte_spinlock_lock(&vdpa_device_list_lock);
if (start == NULL)
- dev = TAILQ_FIRST(&vdpa_device_list);
+ dev = TAILQ_FIRST(vdpa_device_list);
else
dev = TAILQ_NEXT(start, next);
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 7/8] vhost: annotate IOTLB locks
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
` (5 preceding siblings ...)
2022-04-11 11:00 ` [RFC PATCH v3 6/8] vhost: annotate vDPA device list accesses David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-22 7:46 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 8/8] vhost: enable lock check David Marchand
7 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
This change simply annotates existing paths of the code leading to
manipulations of the IOTLB r/w locks.
clang does not support conditionally held locks, so always take iotlb
locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
vdpa and vhost_crypto code are annotated though they end up not taking
a IOTLB lock and have been marked with a FIXME.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/iotlb.h | 8 +++++++
lib/vhost/vdpa.c | 1 +
lib/vhost/vhost.c | 11 +++++----
lib/vhost/vhost.h | 22 +++++++++++++-----
lib/vhost/vhost_crypto.c | 7 ++++++
lib/vhost/virtio_net.c | 49 ++++++++++++++++++++++++++++++----------
6 files changed, 75 insertions(+), 23 deletions(-)
diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h
index 8d0ff7473b..8b97d308d1 100644
--- a/lib/vhost/iotlb.h
+++ b/lib/vhost/iotlb.h
@@ -11,24 +11,32 @@
static __rte_always_inline void
vhost_user_iotlb_rd_lock(struct vhost_virtqueue *vq)
+ __rte_shared_lock_function(vq->iotlb_lock)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_read_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_rd_unlock(struct vhost_virtqueue *vq)
+ __rte_unlock_function(vq->iotlb_lock)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_read_unlock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_lock(struct vhost_virtqueue *vq)
+ __rte_exclusive_lock_function(vq->iotlb_lock)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_write_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_unlock(struct vhost_virtqueue *vq)
+ __rte_unlock_function(vq->iotlb_lock)
+ __rte_no_thread_safety_analysis
{
rte_rwlock_write_unlock(&vq->iotlb_lock);
}
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index a3f9f8f072..22b8f48b39 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -133,6 +133,7 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int
rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
+ __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct virtio_net *dev = get_device(vid);
uint16_t idx, idx_m, desc_id;
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 79236df000..f96e8e0376 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -383,6 +383,7 @@ free_device(struct virtio_net *dev)
static __rte_always_inline int
log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
if (likely(!(vq->ring_addrs.flags & (1 << VHOST_VRING_F_LOG))))
return 0;
@@ -434,6 +435,7 @@ translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
/* Caller should have iotlb_lock read-locked */
static int
vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -473,6 +475,7 @@ vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
/* Caller should have iotlb_lock read-locked */
static int
vring_translate_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -527,10 +530,9 @@ vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
}
void
-vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+vring_invalidate(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq)
{
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_lock(vq);
+ vhost_user_iotlb_wr_lock(vq);
vq->access_ok = false;
vq->desc = NULL;
@@ -538,8 +540,7 @@ vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
vq->used = NULL;
vq->log_guest_addr = 0;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_unlock(vq);
+ vhost_user_iotlb_wr_unlock(vq);
}
static void
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index c2065e5324..b5724c6d2a 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -525,12 +525,15 @@ void __vhost_log_cache_write(struct virtio_net *dev,
uint64_t addr, uint64_t len);
void __vhost_log_cache_write_iova(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(vq->iotlb_lock);
void __vhost_log_cache_sync(struct virtio_net *dev,
struct vhost_virtqueue *vq);
+
void __vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len);
void __vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(vq->iotlb_lock);
static __rte_always_inline void
vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len)
@@ -580,6 +583,7 @@ vhost_log_used_vring(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -593,6 +597,7 @@ vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -796,18 +801,23 @@ struct rte_vhost_device_ops const *vhost_driver_callback_get(const char *path);
void vhost_backend_cleanup(struct virtio_net *dev);
uint64_t __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t *len, uint8_t perm);
+ uint64_t iova, uint64_t *len, uint8_t perm)
+ __rte_shared_locks_required(vq->iotlb_lock);
void *vhost_alloc_copy_ind_table(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t desc_addr, uint64_t desc_len);
-int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq);
+ uint64_t desc_addr, uint64_t desc_len)
+ __rte_shared_locks_required(vq->iotlb_lock);
+int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(vq->iotlb_lock);
uint64_t translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t log_addr);
+ uint64_t log_addr)
+ __rte_shared_locks_required(vq->iotlb_lock);
void vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq);
static __rte_always_inline uint64_t
vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *len, uint8_t perm)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
if (!(dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)))
return rte_vhost_va_from_guest_pa(dev->mem, iova, len);
diff --git a/lib/vhost/vhost_crypto.c b/lib/vhost/vhost_crypto.c
index b1c0eb6a0f..7f8d11ad9f 100644
--- a/lib/vhost/vhost_crypto.c
+++ b/lib/vhost/vhost_crypto.c
@@ -506,6 +506,7 @@ static __rte_always_inline struct virtio_crypto_inhdr *
reach_inhdr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(vc_req->vq->iotlb_lock)
{
struct virtio_crypto_inhdr *inhdr;
struct vhost_crypto_desc *last = head + (max_n_descs - 1);
@@ -552,6 +553,7 @@ static __rte_always_inline void *
get_data_ptr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *cur_desc,
uint8_t perm)
+ __rte_shared_locks_required(vc_req->vq->iotlb_lock)
{
void *data;
uint64_t dlen = cur_desc->len;
@@ -570,6 +572,7 @@ copy_data(void *dst_data, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
struct vhost_crypto_desc **cur_desc,
uint32_t size, uint32_t max_n_descs)
+ __rte_shared_locks_required(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = *cur_desc;
uint64_t remain, addr, dlen, len;
@@ -718,6 +721,7 @@ prepare_write_back_data(struct vhost_crypto_data_req *vc_req,
uint32_t offset,
uint64_t write_back_len,
uint32_t max_n_descs)
+ __rte_shared_locks_required(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_writeback_data *wb_data, *head;
struct vhost_crypto_desc *desc = *cur_desc;
@@ -838,6 +842,7 @@ prepare_sym_cipher_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_cipher_data_req *cipher,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head;
struct vhost_crypto_writeback_data *ewb = NULL;
@@ -990,6 +995,7 @@ prepare_sym_chain_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_alg_chain_data_req *chain,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head, *digest_desc;
struct vhost_crypto_writeback_data *ewb = NULL, *ewb2 = NULL;
@@ -1172,6 +1178,7 @@ vhost_crypto_process_one_req(struct vhost_crypto *vcrypto,
struct vhost_virtqueue *vq, struct rte_crypto_op *op,
struct vring_desc *head, struct vhost_crypto_desc *descs,
uint16_t desc_idx)
+ __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct vhost_crypto_data_req *vc_req = rte_mbuf_to_priv(op->sym->m_src);
struct rte_cryptodev_sym_session *session;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 0c40d5a7a5..8a3e3aa297 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -180,6 +180,7 @@ vhost_async_dma_check_completed(struct virtio_net *dev, int16_t dma_id, uint16_t
static inline void
do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
struct batch_copy_elem *elem = vq->batch_copy_elems;
uint16_t count = vq->batch_copy_nb_elems;
@@ -526,6 +527,7 @@ vhost_shadow_enqueue_single_packed(struct virtio_net *dev,
uint16_t *id,
uint16_t *count,
uint16_t num_buffers)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
vhost_shadow_enqueue_packed(vq, len, id, count, num_buffers);
@@ -607,6 +609,7 @@ static __rte_always_inline int
map_one_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint64_t desc_iova, uint64_t desc_len, uint8_t perm)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t vec_id = *vec_idx;
@@ -644,6 +647,7 @@ fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t avail_idx, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint16_t *desc_chain_head,
uint32_t *desc_chain_len, uint8_t perm)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
uint16_t vec_id = *vec_idx;
@@ -727,6 +731,7 @@ reserve_avail_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t size, struct buf_vector *buf_vec,
uint16_t *num_buffers, uint16_t avail_head,
uint16_t *nr_vec)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t cur_idx;
uint16_t vec_idx = 0;
@@ -777,6 +782,7 @@ fill_vec_buf_packed_indirect(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct vring_packed_desc *desc, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint32_t *len, uint8_t perm)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t i;
uint32_t nr_descs;
@@ -835,6 +841,7 @@ fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t avail_idx, uint16_t *desc_count,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint16_t *buf_id, uint32_t *len, uint8_t perm)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -900,6 +907,7 @@ static __rte_noinline void
copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec,
struct virtio_net_hdr_mrg_rxbuf *hdr)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint64_t len;
uint64_t remain = dev->vhost_hlen;
@@ -1036,6 +1044,7 @@ static __rte_always_inline void
sync_mbuf_to_desc_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_addr, uint64_t buf_iova, uint32_t cpy_len)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
@@ -1061,6 +1070,7 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1191,6 +1201,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1252,6 +1263,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1309,6 +1321,7 @@ virtio_dev_rx_sync_batch_check(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -1360,6 +1373,7 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
@@ -1401,6 +1415,7 @@ static __rte_always_inline int
virtio_dev_rx_sync_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint64_t desc_addrs[PACKED_BATCH_SIZE];
uint64_t lens[PACKED_BATCH_SIZE];
@@ -1423,6 +1438,7 @@ virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1449,6 +1465,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -1501,8 +1518,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
if (unlikely(!vq->enabled))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -1518,8 +1534,7 @@ virtio_dev_rx(struct virtio_net *dev, uint16_t queue_id,
nb_tx = virtio_dev_rx_split(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -1595,6 +1610,7 @@ virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1700,6 +1716,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
uint16_t *nr_descs,
uint16_t *nr_buffers)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1758,6 +1775,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1805,6 +1823,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
uint16_t queue_id, struct rte_mbuf **pkts, uint32_t count,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint32_t remained = count;
@@ -2154,8 +2173,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,
if (unlikely(!vq->enabled || !vq->async))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -2173,8 +2191,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, uint16_t queue_id,
pkts, count, dma_id, vchan_id);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -2697,6 +2714,7 @@ virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t i;
uint16_t free_entries;
@@ -2793,6 +2811,7 @@ virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2803,6 +2822,7 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -2814,6 +2834,7 @@ vhost_reserve_avail_batch_packed(struct virtio_net *dev,
uint16_t avail_idx,
uintptr_t *desc_addrs,
uint16_t *ids)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
bool wrap = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -2883,6 +2904,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts,
bool legacy_ol_flags)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t avail_idx = vq->last_avail_idx;
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
@@ -2929,6 +2951,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *buf_id,
uint16_t *desc_count,
bool legacy_ol_flags)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -2972,6 +2995,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mempool *mbuf_pool,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3003,6 +3027,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
uint32_t count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -3047,6 +3072,7 @@ virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3057,6 +3083,7 @@ virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
__rte_exclusive_locks_required(vq->access_lock)
+ __rte_shared_locks_required(vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3096,8 +3123,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -3153,8 +3179,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
}
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [RFC PATCH v3 8/8] vhost: enable lock check
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
` (6 preceding siblings ...)
2022-04-11 11:00 ` [RFC PATCH v3 7/8] vhost: annotate IOTLB locks David Marchand
@ 2022-04-11 11:00 ` David Marchand
2022-04-22 7:47 ` Maxime Coquelin
7 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2022-04-11 11:00 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
Now that all locks in this library are annotated, we can enable the
check.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/meson.build | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
index bc7272053b..197a51d936 100644
--- a/lib/vhost/meson.build
+++ b/lib/vhost/meson.build
@@ -17,6 +17,8 @@ elif (toolchain == 'icc' and cc.version().version_compare('>=16.0.0'))
endif
dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY', cc.has_header('linux/userfaultfd.h'))
cflags += '-fno-strict-aliasing'
+
+annotate_locks = true
sources = files(
'fd_man.c',
'iotlb.c',
--
2.23.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 1/8] eal: annotate spinlock and rwlock
2022-04-11 11:00 ` [RFC PATCH v3 1/8] eal: annotate spinlock and rwlock David Marchand
@ 2022-04-21 13:48 ` Maxime Coquelin
2022-04-28 12:16 ` David Marchand
0 siblings, 1 reply; 110+ messages in thread
From: Maxime Coquelin @ 2022-04-21 13:48 UTC (permalink / raw)
To: David Marchand, dev
Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding,
Ruifeng Wang, Jan Viktorin, David Christensen, Bruce Richardson,
Konstantin Ananyev
Hi David,
On 4/11/22 13:00, David Marchand wrote:
> clang offers some thread safety checks, statically verifying that locks
> are taken and released in the code.
> To use those checks, the full code leading to taking or releasing locks
> must be annotated with some attributes.
>
> Wrap those attributes into our own set of macros.
>
> rwlock and the "normal" spinlock are instrumented.
>
> A component may enable this check by setting annotate_locks = true
> in its meson.build.
>
> Note: those checks might be of interest out of DPDK, but it
> requires that the including application locks are annotated.
> On the other hand, applications out there might have been using
> those same checks.
> To be on the safe side, keep this instrumentation under a
> RTE_ANNOTATE_LOCKS internal build flag.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changes since RFC v2:
> - fixed rwlock trylock,
> - instrumented _tm spinlocks,
> - aligned attribute names to clang,
>
> ---
> drivers/meson.build | 5 ++
> lib/eal/arm/include/rte_rwlock.h | 4 ++
> lib/eal/arm/include/rte_spinlock.h | 6 +++
> lib/eal/include/generic/rte_rwlock.h | 27 +++++++++--
> lib/eal/include/generic/rte_spinlock.h | 40 ++++++++++-----
> lib/eal/include/meson.build | 1 +
> lib/eal/include/rte_lock_annotations.h | 67 ++++++++++++++++++++++++++
> lib/eal/ppc/include/rte_rwlock.h | 4 ++
> lib/eal/ppc/include/rte_spinlock.h | 9 ++++
> lib/eal/x86/include/rte_rwlock.h | 4 ++
> lib/eal/x86/include/rte_spinlock.h | 9 ++++
> lib/meson.build | 5 ++
> 12 files changed, 164 insertions(+), 17 deletions(-)
> create mode 100644 lib/eal/include/rte_lock_annotations.h
>
Thanks for working on this. I think this lock annotation feature is very
useful, and will help to catch bugs when libs and drivers will adopt it.
Acked-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Regards,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 2/8] vhost: annotate virtqueue access lock
2022-04-11 11:00 ` [RFC PATCH v3 2/8] vhost: annotate virtqueue access lock David Marchand
@ 2022-04-21 15:25 ` Maxime Coquelin
2022-04-22 9:49 ` David Marchand
0 siblings, 1 reply; 110+ messages in thread
From: Maxime Coquelin @ 2022-04-21 15:25 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 4/11/22 13:00, David Marchand wrote:
> This change simply annotates existing paths of the code leading to
> manipulations of the vq->access_lock.
>
> One small change is required: vhost_poll_enqueue_completed was getting
> a queue_id to get hold of the vq, while its callers already knew of
> the vq. For the annotation sake, vq is now directly passed.
It is anyway more consistent with the rest of the code to pass directly
the vq in internal API when queue ID is not needed.
> vhost_user_lock_all_queue_pairs and vhost_user_unlock_all_queue_pairs
> are skipped since vq->access_lock are conditionally held.
As discussed off-list, I wonder whether it could be possible to rework
the conditional lock holding using the static array and some macros so
that we could statically specify for each request if the lock is
required.
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost.h | 2 ++
> lib/vhost/vhost_user.c | 2 ++
> lib/vhost/virtio_net.c | 16 ++++++++++++----
> 3 files changed, 16 insertions(+), 4 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 3/8] vhost: fix async access
2022-04-11 11:00 ` [RFC PATCH v3 3/8] vhost: fix async access David Marchand
@ 2022-04-21 19:21 ` Maxime Coquelin
2022-05-17 13:24 ` Maxime Coquelin
1 sibling, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2022-04-21 19:21 UTC (permalink / raw)
To: David Marchand, dev
Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding, stable,
Sunil Pai G, Patrick Fu
On 4/11/22 13:00, David Marchand wrote:
> vq->async accesses must be protected with vq->access_lock.
>
> Fixes: eb666d24085f ("vhost: fix async unregister deadlock")
> Fixes: 0c0935c5f794 ("vhost: allow to check in-flight packets for async vhost")
> Cc: stable@dpdk.org
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Sunil Pai G <sunil.pai.g@intel.com>
> ---
> lib/vhost/vhost.c | 25 ++++++++++---------------
> 1 file changed, 10 insertions(+), 15 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 4/8] vhost: annotate async accesses
2022-04-11 11:00 ` [RFC PATCH v3 4/8] vhost: annotate async accesses David Marchand
@ 2022-04-22 7:20 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2022-04-22 7:20 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 4/11/22 13:00, David Marchand wrote:
> vq->async is initialised and must be accessed under vq->access_lock.
>
> Top level "_thread_unsafe" functions could be checked at runtime (clang
> provides a lock aware assert()-like check), but they are simply skipped
> because those functions are not called in-tree, and as a result,
> their annotations would not be tested.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost.c | 14 +++++++++-----
> lib/vhost/vhost.h | 2 +-
> lib/vhost/vhost_user.c | 2 ++
> lib/vhost/virtio_net.c | 15 +++++++++++++++
> 4 files changed, 27 insertions(+), 6 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 5/8] vhost: annotate need reply handling
2022-04-11 11:00 ` [RFC PATCH v3 5/8] vhost: annotate need reply handling David Marchand
@ 2022-04-22 7:25 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2022-04-22 7:25 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 4/11/22 13:00, David Marchand wrote:
> When a reply from the slave is required (VHOST_USER_NEED_REPLY flag),
> a spinlock is taken before sending the message.
> This spinlock is released if an error occurs when sending the message, and
> once a reply is received.
>
> A problem is that this lock is taken under a branch and annotating
> conditionally held locks is not supported.
> The code seems currently correct and, while we may rework the code,
> it is easier to simply skip checks on slave_req_lock for those helpers.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost_user.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index ee276a28f1..d101d5072f 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -2854,6 +2854,7 @@ send_vhost_reply(struct virtio_net *dev, int sockfd, struct vhu_msg_context *ctx
> static int
> send_vhost_slave_message(struct virtio_net *dev,
> struct vhu_msg_context *ctx)
> + __rte_no_thread_safety_analysis
> {
> int ret;
>
> @@ -3165,6 +3166,7 @@ vhost_user_msg_handler(int vid, int fd)
>
> static int process_slave_message_reply(struct virtio_net *dev,
> const struct vhu_msg_context *ctx)
> + __rte_no_thread_safety_analysis
> {
> struct vhu_msg_context msg_reply;
> int ret;
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 6/8] vhost: annotate vDPA device list accesses
2022-04-11 11:00 ` [RFC PATCH v3 6/8] vhost: annotate vDPA device list accesses David Marchand
@ 2022-04-22 7:26 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2022-04-22 7:26 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 4/11/22 13:00, David Marchand wrote:
> vdpa_device_list access must be protected with vdpa_device_list_lock
> spinlock.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vdpa.c | 17 ++++++++++-------
> 1 file changed, 10 insertions(+), 7 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 7/8] vhost: annotate IOTLB locks
2022-04-11 11:00 ` [RFC PATCH v3 7/8] vhost: annotate IOTLB locks David Marchand
@ 2022-04-22 7:46 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2022-04-22 7:46 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 4/11/22 13:00, David Marchand wrote:
> This change simply annotates existing paths of the code leading to
> manipulations of the IOTLB r/w locks.
>
> clang does not support conditionally held locks, so always take iotlb
> locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
>
> vdpa and vhost_crypto code are annotated though they end up not taking
> a IOTLB lock and have been marked with a FIXME.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/iotlb.h | 8 +++++++
> lib/vhost/vdpa.c | 1 +
> lib/vhost/vhost.c | 11 +++++----
> lib/vhost/vhost.h | 22 +++++++++++++-----
> lib/vhost/vhost_crypto.c | 7 ++++++
> lib/vhost/virtio_net.c | 49 ++++++++++++++++++++++++++++++----------
> 6 files changed, 75 insertions(+), 23 deletions(-)
>
I agree with the change. I don't expect performance impact of taking the
lock unconditionally, because there won't be cache line sharing since it
is per-vq lock and the locking cost will be offset by removing the
feature check.
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 8/8] vhost: enable lock check
2022-04-11 11:00 ` [RFC PATCH v3 8/8] vhost: enable lock check David Marchand
@ 2022-04-22 7:47 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2022-04-22 7:47 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 4/11/22 13:00, David Marchand wrote:
> Now that all locks in this library are annotated, we can enable the
> check.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/meson.build | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
> index bc7272053b..197a51d936 100644
> --- a/lib/vhost/meson.build
> +++ b/lib/vhost/meson.build
> @@ -17,6 +17,8 @@ elif (toolchain == 'icc' and cc.version().version_compare('>=16.0.0'))
> endif
> dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY', cc.has_header('linux/userfaultfd.h'))
> cflags += '-fno-strict-aliasing'
> +
> +annotate_locks = true
> sources = files(
> 'fd_man.c',
> 'iotlb.c',
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 2/8] vhost: annotate virtqueue access lock
2022-04-21 15:25 ` Maxime Coquelin
@ 2022-04-22 9:49 ` David Marchand
0 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-04-22 9:49 UTC (permalink / raw)
To: Maxime Coquelin
Cc: dev, Stephen Hemminger, Xia, Chenbo, Jiayu Hu, Wang, YuanX, Xuan Ding
On Thu, Apr 21, 2022 at 5:25 PM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
> On 4/11/22 13:00, David Marchand wrote:
> > This change simply annotates existing paths of the code leading to
> > manipulations of the vq->access_lock.
> >
> > One small change is required: vhost_poll_enqueue_completed was getting
> > a queue_id to get hold of the vq, while its callers already knew of
> > the vq. For the annotation sake, vq is now directly passed.
>
> It is anyway more consistent with the rest of the code to pass directly
> the vq in internal API when queue ID is not needed.
>
> > vhost_user_lock_all_queue_pairs and vhost_user_unlock_all_queue_pairs
> > are skipped since vq->access_lock are conditionally held.
>
> As discussed off-list, I wonder whether it could be possible to rework
> the conditional lock holding using the static array and some macros so
> that we could statically specify for each request if the lock is
> required.
We did discuss some ideas off-list, but in the end, since we have
multiple locks being dynamically taken in
vhost_user_lock_all_queue_pairs, I see no way to statically annotate
the code.
We could rework the code to have message handlers in a consolidated
static array, but that would not help with annotations.
I had some patches going in that direction (related to some fd fixes I
sent before), but it needs more work.
I'll see if I can send this later in the release or it will go to next release.
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 1/8] eal: annotate spinlock and rwlock
2022-04-21 13:48 ` Maxime Coquelin
@ 2022-04-28 12:16 ` David Marchand
0 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2022-04-28 12:16 UTC (permalink / raw)
To: Maxime Coquelin
Cc: dev, Stephen Hemminger, Xia, Chenbo, Jiayu Hu, Wang, YuanX,
Xuan Ding, Ruifeng Wang, Jan Viktorin, David Christensen,
Bruce Richardson, Konstantin Ananyev
On Thu, Apr 21, 2022 at 3:48 PM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
> Hi David,
>
> On 4/11/22 13:00, David Marchand wrote:
> > clang offers some thread safety checks, statically verifying that locks
> > are taken and released in the code.
> > To use those checks, the full code leading to taking or releasing locks
> > must be annotated with some attributes.
> >
> > Wrap those attributes into our own set of macros.
> >
> > rwlock and the "normal" spinlock are instrumented.
> >
> > A component may enable this check by setting annotate_locks = true
> > in its meson.build.
> >
> > Note: those checks might be of interest out of DPDK, but it
> > requires that the including application locks are annotated.
> > On the other hand, applications out there might have been using
> > those same checks.
> > To be on the safe side, keep this instrumentation under a
> > RTE_ANNOTATE_LOCKS internal build flag.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> > Changes since RFC v2:
> > - fixed rwlock trylock,
> > - instrumented _tm spinlocks,
> > - aligned attribute names to clang,
> >
> > ---
> > drivers/meson.build | 5 ++
> > lib/eal/arm/include/rte_rwlock.h | 4 ++
> > lib/eal/arm/include/rte_spinlock.h | 6 +++
> > lib/eal/include/generic/rte_rwlock.h | 27 +++++++++--
> > lib/eal/include/generic/rte_spinlock.h | 40 ++++++++++-----
> > lib/eal/include/meson.build | 1 +
> > lib/eal/include/rte_lock_annotations.h | 67 ++++++++++++++++++++++++++
> > lib/eal/ppc/include/rte_rwlock.h | 4 ++
> > lib/eal/ppc/include/rte_spinlock.h | 9 ++++
> > lib/eal/x86/include/rte_rwlock.h | 4 ++
> > lib/eal/x86/include/rte_spinlock.h | 9 ++++
> > lib/meson.build | 5 ++
> > 12 files changed, 164 insertions(+), 17 deletions(-)
> > create mode 100644 lib/eal/include/rte_lock_annotations.h
> >
>
>
> Thanks for working on this. I think this lock annotation feature is very
> useful, and will help to catch bugs when libs and drivers will adopt it.
>
Thanks for the review.
We need some documentation about those annotations.
I'll add this and post a non RFC series.
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [RFC PATCH v3 3/8] vhost: fix async access
2022-04-11 11:00 ` [RFC PATCH v3 3/8] vhost: fix async access David Marchand
2022-04-21 19:21 ` Maxime Coquelin
@ 2022-05-17 13:24 ` Maxime Coquelin
1 sibling, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2022-05-17 13:24 UTC (permalink / raw)
To: David Marchand, dev
Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding, stable,
Sunil Pai G, Patrick Fu
On 4/11/22 13:00, David Marchand wrote:
> vq->async accesses must be protected with vq->access_lock.
>
> Fixes: eb666d24085f ("vhost: fix async unregister deadlock")
> Fixes: 0c0935c5f794 ("vhost: allow to check in-flight packets for async vhost")
> Cc: stable@dpdk.org
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Sunil Pai G <sunil.pai.g@intel.com>
> ---
> lib/vhost/vhost.c | 25 ++++++++++---------------
> 1 file changed, 10 insertions(+), 15 deletions(-)
>
Applied to dpdk-next-virtio/main.
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 0/9] vhost lock annotations
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
` (6 preceding siblings ...)
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-19 18:46 ` [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
` (9 more replies)
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
9 siblings, 10 replies; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vhost internals involves multiple locks to protect data access by
multiple threads.
This series uses clang thread safety checks [1] to catch issues during
compilation: EAL spinlock, seqlock and rwlock are annotated and vhost
code is instrumented so that clang can statically check correctness.
Those annotations are quite heavy to maintain because the full path of
code must be annotated (as can be seen in the vhost datapath code),
but I think it is worth using.
This has been tested against the whole tree and some fixes are already
flying on the mailing list (see [2] for a list).
If this first series is merged, I will prepare a followup series for EAL
and other libraries.
1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
2: https://patchwork.dpdk.org/bundle/dmarchand/lock_fixes/?state=*&archive=both
--
David Marchand
Changes since RFC v3:
- sorry Maxime, it has been too long since RFC v3 and the code evolved,
so I dropped all your review tags,
- rebased,
- added documentation,
- dropped/fixed annotations in arch-specific and EAL headers,
- rewrote need reply handling so that we don't have to waive the check
on the associated functions,
- separated IOTLB lock unconditional acquire from the annotation patch,
- rewrote runtime checks for "unsafe" functions using a panicking assert
helper,
Changes since RFC v2:
- fixed trylock annotations for rwlock,
- annotated _tm flavors of spinlock and rwlock,
- removed Maxime vhost fix from series (since Mimecast does not like
me sending Maxime patch...), added a dependency on original fix
as a hint for reviewers,
- renamed attributes,
Changes since RFC v1:
- Cc'd people who have pending patches for vhost,
- moved annotations to EAL and removed wrappers in vhost,
- as a result of moving to EAL, this series will be tested against
the main repo, so patch 1 has been kept as part of the series
even if already applied to next-virtio,
- refined/split patches and annotated all spinlocks in vhost,
David Marchand (9):
eal: annotate spinlock, rwlock and seqlock
vhost: simplify need reply handling
vhost: terminate when access lock is not taken
vhost: annotate virtqueue access lock
vhost: annotate async accesses
vhost: always take IOTLB lock
vhost: annotate IOTLB lock
vhost: annotate vDPA device list accesses
vhost: enable lock check
.../prog_guide/env_abstraction_layer.rst | 24 ++++
doc/guides/rel_notes/release_23_03.rst | 5 +
drivers/meson.build | 5 +
lib/eal/include/generic/rte_rwlock.h | 27 +++-
lib/eal/include/generic/rte_spinlock.h | 31 +++--
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 73 ++++++++++
lib/eal/include/rte_seqlock.h | 6 +
lib/eal/ppc/include/rte_spinlock.h | 3 +
lib/eal/x86/include/rte_rwlock.h | 4 +
lib/eal/x86/include/rte_spinlock.h | 9 ++
lib/meson.build | 5 +
lib/vhost/iotlb.h | 4 +
lib/vhost/meson.build | 2 +
lib/vhost/vdpa.c | 20 +--
lib/vhost/vhost.c | 38 ++---
lib/vhost/vhost.h | 34 ++++-
lib/vhost/vhost_crypto.c | 8 ++
lib/vhost/vhost_user.c | 131 ++++++++----------
lib/vhost/virtio_net.c | 109 +++++++++++----
20 files changed, 389 insertions(+), 150 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-19 19:42 ` Stephen Hemminger
` (2 more replies)
2023-01-19 18:46 ` [PATCH v4 2/9] vhost: simplify need reply handling David Marchand
` (8 subsequent siblings)
9 siblings, 3 replies; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
clang offers some thread safety checks, statically verifying that locks
are taken and released in the code.
To use those checks, the full code leading to taking or releasing locks
must be annotated with some attributes.
Wrap those attributes into our own set of macros.
rwlock, seqlock and the "normal" spinlock are instrumented.
Those checks might be of interest out of DPDK, but it requires that the
including application locks are annotated.
On the other hand, applications out there might have been using
those same checks.
To be on the safe side, keep this instrumentation under a
RTE_ANNOTATE_LOCKS internal build flag.
A component may en/disable this check by setting
annotate_locks = true/false in its meson.build.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since RFC v3:
- rebased,
- added some documentation,
- added assert attribute,
- instrumented seqlock,
- cleaned annotations in arch-specific headers,
Changes since RFC v2:
- fixed rwlock trylock,
- instrumented _tm spinlocks,
- aligned attribute names to clang,
---
.../prog_guide/env_abstraction_layer.rst | 24 ++++++
doc/guides/rel_notes/release_23_03.rst | 5 ++
drivers/meson.build | 5 ++
lib/eal/include/generic/rte_rwlock.h | 27 +++++--
lib/eal/include/generic/rte_spinlock.h | 31 +++++---
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 73 +++++++++++++++++++
lib/eal/include/rte_seqlock.h | 6 ++
lib/eal/ppc/include/rte_spinlock.h | 3 +
lib/eal/x86/include/rte_rwlock.h | 4 +
lib/eal/x86/include/rte_spinlock.h | 9 +++
lib/meson.build | 5 ++
12 files changed, 179 insertions(+), 14 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 35fbebe1be..6c1e8ba985 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -529,6 +529,30 @@ Misc Functions
Locks and atomic operations are per-architecture (i686 and x86_64).
+Lock annotations
+~~~~~~~~~~~~~~~~
+
+R/W locks, seq locks and spinlocks have been instrumented to help developers in
+catching issues in DPDK.
+
+This instrumentation relies on
+`clang Thread Safety checks <https://clang.llvm.org/docs/ThreadSafetyAnalysis.html>`_.
+All attributes are prefixed with __rte and are fully described in the clang
+documentation.
+
+Some general comments:
+
+- it is important that lock requirements are expressed at the function
+ declaration level in headers so that other code units can be inspected,
+- when some global lock is necessary to some user-exposed API, it is preferred
+ to expose it via an internal helper rather than expose the global variable,
+- there are a list of known limitations with clang instrumentation, but before
+ waiving checks with ``__rte_no_thread_safety_analysis`` in your code, please
+ discuss it on the mailing list,
+
+A DPDK library/driver can enabled/disable the checks by setting
+``annotate_locks`` accordingly in its ``meson.build`` file.
+
IOVA Mode Detection
~~~~~~~~~~~~~~~~~~~
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index c15f6fbb9f..5425e59c65 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -55,6 +55,11 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Introduced lock annotations.**
+
+ Added lock annotations attributes to that clang can statically analyze lock
+ correctness.
+
* **Updated Intel QuickAssist Technology (QAT) crypto driver.**
* Added support for SHA3 224/256/384/512 plain hash in QAT GEN 3.
diff --git a/drivers/meson.build b/drivers/meson.build
index c6d619200f..bddc4a6cc4 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -91,6 +91,7 @@ foreach subpath:subdirs
build = true # set to false to disable, e.g. missing deps
reason = '<unknown reason>' # set if build == false to explain
name = drv
+ annotate_locks = false
sources = []
headers = []
driver_sdk_headers = [] # public headers included by drivers
@@ -167,6 +168,10 @@ foreach subpath:subdirs
enabled_drivers += name
lib_name = '_'.join(['rte', class, name])
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=' + '.'.join([log_prefix, name])
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
dpdk_conf.set(lib_name.to_upper(), 1)
dpdk_extra_ldflags += pkgconfig_extra_libs
diff --git a/lib/eal/include/generic/rte_rwlock.h b/lib/eal/include/generic/rte_rwlock.h
index 233d4262be..d45c22c189 100644
--- a/lib/eal/include/generic/rte_rwlock.h
+++ b/lib/eal/include/generic/rte_rwlock.h
@@ -30,6 +30,7 @@ extern "C" {
#include <rte_branch_prediction.h>
#include <rte_common.h>
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
@@ -55,7 +56,7 @@ extern "C" {
/* Writer is waiting or has lock */
#define RTE_RWLOCK_READ 0x4 /* Reader increment */
-typedef struct {
+typedef struct __rte_lockable {
int32_t cnt;
} rte_rwlock_t;
@@ -84,6 +85,8 @@ rte_rwlock_init(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_read_lock(rte_rwlock_t *rwl)
+ __rte_shared_lock_function(rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -119,6 +122,8 @@ rte_rwlock_read_lock(rte_rwlock_t *rwl)
*/
static inline int
rte_rwlock_read_trylock(rte_rwlock_t *rwl)
+ __rte_shared_trylock_function(0, rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -150,6 +155,8 @@ rte_rwlock_read_trylock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_read_unlock(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl)
+ __rte_no_thread_safety_analysis
{
__atomic_fetch_sub(&rwl->cnt, RTE_RWLOCK_READ, __ATOMIC_RELEASE);
}
@@ -166,6 +173,8 @@ rte_rwlock_read_unlock(rte_rwlock_t *rwl)
*/
static inline int
rte_rwlock_write_trylock(rte_rwlock_t *rwl)
+ __rte_exclusive_trylock_function(0, rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -186,6 +195,8 @@ rte_rwlock_write_trylock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_write_lock(rte_rwlock_t *rwl)
+ __rte_exclusive_lock_function(rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -219,6 +230,8 @@ rte_rwlock_write_lock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_write_unlock(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl)
+ __rte_no_thread_safety_analysis
{
__atomic_fetch_sub(&rwl->cnt, RTE_RWLOCK_WRITE, __ATOMIC_RELEASE);
}
@@ -237,7 +250,8 @@ rte_rwlock_write_unlock(rte_rwlock_t *rwl)
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_read_lock_tm(rte_rwlock_t *rwl);
+rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_shared_lock_function(rwl);
/**
* Commit hardware memory transaction or release the read lock if the lock is used as a fall-back
@@ -246,7 +260,8 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl);
* A pointer to the rwlock structure.
*/
static inline void
-rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl);
+rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl);
/**
* Try to execute critical section in a hardware memory transaction, if it
@@ -262,7 +277,8 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl);
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_lock_tm(rte_rwlock_t *rwl);
+rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_exclusive_lock_function(rwl);
/**
* Commit hardware memory transaction or release the write lock if the lock is used as a fall-back
@@ -271,7 +287,8 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl);
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl);
+rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl);
#ifdef __cplusplus
}
diff --git a/lib/eal/include/generic/rte_spinlock.h b/lib/eal/include/generic/rte_spinlock.h
index 73ed4bfbdc..8ca47bbfaa 100644
--- a/lib/eal/include/generic/rte_spinlock.h
+++ b/lib/eal/include/generic/rte_spinlock.h
@@ -22,12 +22,13 @@
#ifdef RTE_FORCE_INTRINSICS
#include <rte_common.h>
#endif
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
* The rte_spinlock_t type.
*/
-typedef struct {
+typedef struct __rte_lockable {
volatile int locked; /**< lock status 0 = unlocked, 1 = locked */
} rte_spinlock_t;
@@ -55,11 +56,13 @@ rte_spinlock_init(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock(rte_spinlock_t *sl);
+rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_exclusive_lock_function(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int exp = 0;
@@ -79,11 +82,13 @@ rte_spinlock_lock(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl);
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_unlock_function(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl)
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
__atomic_store_n(&sl->locked, 0, __ATOMIC_RELEASE);
}
@@ -99,11 +104,13 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
*/
__rte_warn_unused_result
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl);
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_exclusive_trylock_function(1, sl);
#ifdef RTE_FORCE_INTRINSICS
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl)
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int exp = 0;
return __atomic_compare_exchange_n(&sl->locked, &exp, 1,
@@ -147,7 +154,8 @@ static inline int rte_tm_supported(void);
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock_tm(rte_spinlock_t *sl);
+rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_exclusive_lock_function(sl);
/**
* Commit hardware memory transaction or release the spinlock if
@@ -157,7 +165,8 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl);
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock_tm(rte_spinlock_t *sl);
+rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_unlock_function(sl);
/**
* Try to execute critical section in a hardware memory transaction,
@@ -177,7 +186,8 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl);
*/
__rte_warn_unused_result
static inline int
-rte_spinlock_trylock_tm(rte_spinlock_t *sl);
+rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_exclusive_trylock_function(1, sl);
/**
* The rte_spinlock_recursive_t type.
@@ -213,6 +223,7 @@ static inline void rte_spinlock_recursive_init(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
int id = rte_gettid();
@@ -229,6 +240,7 @@ static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (--(slr->count) == 0) {
slr->user = -1;
@@ -247,6 +259,7 @@ static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
*/
__rte_warn_unused_result
static inline int rte_spinlock_recursive_trylock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
int id = rte_gettid();
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index cfcd40aaed..b0db9b3b3a 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -27,6 +27,7 @@ headers += files(
'rte_keepalive.h',
'rte_launch.h',
'rte_lcore.h',
+ 'rte_lock_annotations.h',
'rte_log.h',
'rte_malloc.h',
'rte_mcslock.h',
diff --git a/lib/eal/include/rte_lock_annotations.h b/lib/eal/include/rte_lock_annotations.h
new file mode 100644
index 0000000000..9fc50082d6
--- /dev/null
+++ b/lib/eal/include/rte_lock_annotations.h
@@ -0,0 +1,73 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Red Hat, Inc.
+ */
+
+#ifndef RTE_LOCK_ANNOTATIONS_H
+#define RTE_LOCK_ANNOTATIONS_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifdef RTE_ANNOTATE_LOCKS
+
+#define __rte_lockable \
+ __attribute__((lockable))
+
+#define __rte_guarded_by(...) \
+ __attribute__((guarded_by(__VA_ARGS__)))
+#define __rte_guarded_var \
+ __attribute__((guarded_var))
+
+#define __rte_exclusive_locks_required(...) \
+ __attribute__((exclusive_locks_required(__VA_ARGS__)))
+#define __rte_exclusive_lock_function(...) \
+ __attribute__((exclusive_lock_function(__VA_ARGS__)))
+#define __rte_exclusive_trylock_function(ret, ...) \
+ __attribute__((exclusive_trylock_function(ret, __VA_ARGS__)))
+#define __rte_assert_exclusive_lock(...) \
+ __attribute__((assert_exclusive_lock(__VA_ARGS__)))
+
+#define __rte_shared_locks_required(...) \
+ __attribute__((shared_locks_required(__VA_ARGS__)))
+#define __rte_shared_lock_function(...) \
+ __attribute__((shared_lock_function(__VA_ARGS__)))
+#define __rte_shared_trylock_function(ret, ...) \
+ __attribute__((shared_trylock_function(ret, __VA_ARGS__)))
+#define __rte_assert_shared_lock(...) \
+ __attribute__((assert_shared_lock(__VA_ARGS__)))
+
+#define __rte_unlock_function(...) \
+ __attribute__((unlock_function(__VA_ARGS__)))
+
+#define __rte_no_thread_safety_analysis \
+ __attribute__((no_thread_safety_analysis))
+
+#else /* ! RTE_ANNOTATE_LOCKS */
+
+#define __rte_lockable
+
+#define __rte_guarded_by(...)
+#define __rte_guarded_var
+
+#define __rte_exclusive_locks_required(...)
+#define __rte_exclusive_lock_function(...)
+#define __rte_exclusive_trylock_function(...)
+#define __rte_assert_exclusive_lock(...)
+
+#define __rte_shared_locks_required(...)
+#define __rte_shared_lock_function(...)
+#define __rte_shared_trylock_function(...)
+#define __rte_assert_shared_lock(...)
+
+#define __rte_unlock_function(...)
+
+#define __rte_no_thread_safety_analysis
+
+#endif /* RTE_ANNOTATE_LOCKS */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_LOCK_ANNOTATIONS_H */
diff --git a/lib/eal/include/rte_seqlock.h b/lib/eal/include/rte_seqlock.h
index 1663af62e8..f34fa2005d 100644
--- a/lib/eal/include/rte_seqlock.h
+++ b/lib/eal/include/rte_seqlock.h
@@ -215,6 +215,9 @@ rte_seqlock_read_retry(const rte_seqlock_t *seqlock, uint32_t begin_sn)
__rte_experimental
static inline void
rte_seqlock_write_lock(rte_seqlock_t *seqlock)
+#ifndef __DOXYGEN__
+ __rte_exclusive_lock_function(&seqlock->lock)
+#endif
{
/* To synchronize with other writers. */
rte_spinlock_lock(&seqlock->lock);
@@ -240,6 +243,9 @@ rte_seqlock_write_lock(rte_seqlock_t *seqlock)
__rte_experimental
static inline void
rte_seqlock_write_unlock(rte_seqlock_t *seqlock)
+#ifndef __DOXYGEN__
+ __rte_unlock_function(&seqlock->lock)
+#endif
{
rte_seqcount_write_end(&seqlock->count);
diff --git a/lib/eal/ppc/include/rte_spinlock.h b/lib/eal/ppc/include/rte_spinlock.h
index 149ec245c7..3a4c905b22 100644
--- a/lib/eal/ppc/include/rte_spinlock.h
+++ b/lib/eal/ppc/include/rte_spinlock.h
@@ -20,6 +20,7 @@ extern "C" {
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
while (__sync_lock_test_and_set(&sl->locked, 1))
while (sl->locked)
@@ -28,12 +29,14 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
__sync_lock_release(&sl->locked);
}
static inline int
rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
return __sync_lock_test_and_set(&sl->locked, 1) == 0;
}
diff --git a/lib/eal/x86/include/rte_rwlock.h b/lib/eal/x86/include/rte_rwlock.h
index eec4c7123c..1796b69265 100644
--- a/lib/eal/x86/include/rte_rwlock.h
+++ b/lib/eal/x86/include/rte_rwlock.h
@@ -14,6 +14,7 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -22,6 +23,7 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(rwl->cnt))
rte_rwlock_read_unlock(rwl);
@@ -31,6 +33,7 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -39,6 +42,7 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(rwl->cnt))
rte_rwlock_write_unlock(rwl);
diff --git a/lib/eal/x86/include/rte_spinlock.h b/lib/eal/x86/include/rte_spinlock.h
index e2e2b2643c..0b20ddfd73 100644
--- a/lib/eal/x86/include/rte_spinlock.h
+++ b/lib/eal/x86/include/rte_spinlock.h
@@ -23,6 +23,7 @@ extern "C" {
#ifndef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int lock_val = 1;
asm volatile (
@@ -43,6 +44,7 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock (rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int unlock_val = 0;
asm volatile (
@@ -54,6 +56,7 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock (rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int lockval = 1;
@@ -121,6 +124,7 @@ rte_try_tm(volatile int *lock)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&sl->locked)))
return;
@@ -130,6 +134,7 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&sl->locked)))
return 1;
@@ -139,6 +144,7 @@ rte_spinlock_trylock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(sl->locked))
rte_spinlock_unlock(sl);
@@ -148,6 +154,7 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&slr->sl.locked)))
return;
@@ -157,6 +164,7 @@ rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (unlikely(slr->sl.locked))
rte_spinlock_recursive_unlock(slr);
@@ -166,6 +174,7 @@ rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&slr->sl.locked)))
return 1;
diff --git a/lib/meson.build b/lib/meson.build
index a90fee31b7..450c061d2b 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -120,6 +120,7 @@ foreach l:libraries
reason = '<unknown reason>' # set if build == false to explain why
name = l
use_function_versioning = false
+ annotate_locks = false
sources = []
headers = []
indirect_headers = [] # public headers not directly included by apps
@@ -202,6 +203,10 @@ foreach l:libraries
cflags += '-DRTE_USE_FUNCTION_VERSIONING'
endif
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=lib.' + l
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
# first build static lib
static_lib = static_library(libname,
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 2/9] vhost: simplify need reply handling
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
2023-01-19 18:46 ` [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-31 16:41 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 3/9] vhost: terminate when access lock is not taken David Marchand
` (7 subsequent siblings)
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
Dedicate send_vhost_slave_message() helper to the case when no reply is
needed.
Add a send_vhost_slave_message_process_reply() helper for the opposite.
This new helper merges both send_vhost_slave_message() and the code
previously in process_slave_message_reply().
The slave_req_lock lock is then only handled in this helper which will
make lock checks trivial.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost_user.c | 119 ++++++++++++++++++-----------------------
1 file changed, 51 insertions(+), 68 deletions(-)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 9902ae9944..3f6c5df900 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2868,18 +2868,46 @@ send_vhost_reply(struct virtio_net *dev, int sockfd, struct vhu_msg_context *ctx
}
static int
-send_vhost_slave_message(struct virtio_net *dev,
- struct vhu_msg_context *ctx)
+send_vhost_slave_message(struct virtio_net *dev, struct vhu_msg_context *ctx)
+{
+ return send_vhost_message(dev, dev->slave_req_fd, ctx);
+}
+
+static int
+send_vhost_slave_message_process_reply(struct virtio_net *dev, struct vhu_msg_context *ctx)
{
+ struct vhu_msg_context msg_reply;
int ret;
- if (ctx->msg.flags & VHOST_USER_NEED_REPLY)
- rte_spinlock_lock(&dev->slave_req_lock);
+ rte_spinlock_lock(&dev->slave_req_lock);
+ ret = send_vhost_slave_message(dev, ctx);
+ if (ret < 0) {
+ VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
+ goto out;
+ }
- ret = send_vhost_message(dev, dev->slave_req_fd, ctx);
- if (ret < 0 && (ctx->msg.flags & VHOST_USER_NEED_REPLY))
- rte_spinlock_unlock(&dev->slave_req_lock);
+ ret = read_vhost_message(dev, dev->slave_req_fd, &msg_reply);
+ if (ret <= 0) {
+ if (ret < 0)
+ VHOST_LOG_CONFIG(dev->ifname, ERR,
+ "vhost read slave message reply failed\n");
+ else
+ VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer closed\n");
+ ret = -1;
+ goto out;
+ }
+
+ if (msg_reply.msg.request.slave != ctx->msg.request.slave) {
+ VHOST_LOG_CONFIG(dev->ifname, ERR,
+ "received unexpected msg type (%u), expected %u\n",
+ msg_reply.msg.request.slave, ctx->msg.request.slave);
+ ret = -1;
+ goto out;
+ }
+ ret = msg_reply.msg.payload.u64 ? -1 : 0;
+out:
+ rte_spinlock_unlock(&dev->slave_req_lock);
return ret;
}
@@ -3203,42 +3231,6 @@ vhost_user_msg_handler(int vid, int fd)
return ret;
}
-static int process_slave_message_reply(struct virtio_net *dev,
- const struct vhu_msg_context *ctx)
-{
- struct vhu_msg_context msg_reply;
- int ret;
-
- if ((ctx->msg.flags & VHOST_USER_NEED_REPLY) == 0)
- return 0;
-
- ret = read_vhost_message(dev, dev->slave_req_fd, &msg_reply);
- if (ret <= 0) {
- if (ret < 0)
- VHOST_LOG_CONFIG(dev->ifname, ERR,
- "vhost read slave message reply failed\n");
- else
- VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer closed\n");
- ret = -1;
- goto out;
- }
-
- ret = 0;
- if (msg_reply.msg.request.slave != ctx->msg.request.slave) {
- VHOST_LOG_CONFIG(dev->ifname, ERR,
- "received unexpected msg type (%u), expected %u\n",
- msg_reply.msg.request.slave, ctx->msg.request.slave);
- ret = -1;
- goto out;
- }
-
- ret = msg_reply.msg.payload.u64 ? -1 : 0;
-
-out:
- rte_spinlock_unlock(&dev->slave_req_lock);
- return ret;
-}
-
int
vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm)
{
@@ -3267,10 +3259,9 @@ vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm)
return 0;
}
-static int
-vhost_user_slave_config_change(struct virtio_net *dev, bool need_reply)
+int
+rte_vhost_slave_config_change(int vid, bool need_reply)
{
- int ret;
struct vhu_msg_context ctx = {
.msg = {
.request.slave = VHOST_USER_SLAVE_CONFIG_CHANGE_MSG,
@@ -3278,29 +3269,23 @@ vhost_user_slave_config_change(struct virtio_net *dev, bool need_reply)
.size = 0,
}
};
-
- if (need_reply)
- ctx.msg.flags |= VHOST_USER_NEED_REPLY;
-
- ret = send_vhost_slave_message(dev, &ctx);
- if (ret < 0) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
- return ret;
- }
-
- return process_slave_message_reply(dev, &ctx);
-}
-
-int
-rte_vhost_slave_config_change(int vid, bool need_reply)
-{
struct virtio_net *dev;
+ int ret;
dev = get_device(vid);
if (!dev)
return -ENODEV;
- return vhost_user_slave_config_change(dev, need_reply);
+ if (!need_reply) {
+ ret = send_vhost_slave_message(dev, &ctx);
+ } else {
+ ctx.msg.flags |= VHOST_USER_NEED_REPLY;
+ ret = send_vhost_slave_message_process_reply(dev, &ctx);
+ }
+
+ if (ret < 0)
+ VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
+ return ret;
}
static int vhost_user_slave_set_vring_host_notifier(struct virtio_net *dev,
@@ -3329,13 +3314,11 @@ static int vhost_user_slave_set_vring_host_notifier(struct virtio_net *dev,
ctx.fd_num = 1;
}
- ret = send_vhost_slave_message(dev, &ctx);
- if (ret < 0) {
+ ret = send_vhost_slave_message_process_reply(dev, &ctx);
+ if (ret < 0)
VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to set host notifier (%d)\n", ret);
- return ret;
- }
- return process_slave_message_reply(dev, &ctx);
+ return ret;
}
int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable)
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 3/9] vhost: terminate when access lock is not taken
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
2023-01-19 18:46 ` [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
2023-01-19 18:46 ` [PATCH v4 2/9] vhost: simplify need reply handling David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-31 16:47 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 4/9] vhost: annotate virtqueue access lock David Marchand
` (6 subsequent siblings)
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
Be a bit more strict when a programmatic error is detected wrt to the
access_lock not being taken.
Mark the new helper with __rte_assert_exclusive_lock so that clang
understands where locks are expected to be taken.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.c | 18 +++---------------
lib/vhost/vhost.h | 10 ++++++++++
lib/vhost/virtio_net.c | 6 +-----
3 files changed, 14 insertions(+), 20 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 19c7b92c32..8cd727ca2f 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1781,11 +1781,7 @@ rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
if (unlikely(vq == NULL || !dev->async_copy))
return -1;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
return async_channel_register(dev, vq);
}
@@ -1847,11 +1843,7 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)
if (vq == NULL)
return -1;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (!vq->async)
return 0;
@@ -1994,11 +1986,7 @@ rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (!vq->async)
return ret;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index ef211ed519..f6b2930efd 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -512,6 +512,16 @@ struct virtio_net {
struct rte_vhost_user_extern_ops extern_ops;
} __rte_cache_aligned;
+static inline void
+vq_assert_lock__(struct virtio_net *dev, struct vhost_virtqueue *vq, const char *func)
+ __rte_assert_exclusive_lock(&vq->access_lock)
+{
+ if (unlikely(!rte_spinlock_is_locked(&vq->access_lock)))
+ rte_panic("VHOST_CONFIG: (%s) %s() called without access lock taken.\n",
+ dev->ifname, func);
+}
+#define vq_assert_lock(dev, vq) vq_assert_lock__(dev, vq, __func__)
+
static __rte_always_inline bool
vq_is_packed(struct virtio_net *dev)
{
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 9abf752f30..2a75cda7b6 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -2185,11 +2185,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
vq = dev->virtqueue[queue_id];
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_DATA(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (unlikely(!vq->async)) {
VHOST_LOG_DATA(dev->ifname, ERR,
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 4/9] vhost: annotate virtqueue access lock
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
` (2 preceding siblings ...)
2023-01-19 18:46 ` [PATCH v4 3/9] vhost: terminate when access lock is not taken David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-31 16:50 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 5/9] vhost: annotate async accesses David Marchand
` (5 subsequent siblings)
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vhost_user_lock/unlock_all_queue_pairs must be waived since clang
annotations can't express taking a runtime number of locks.
vhost_queue_stats_update() requirement can be expressed with a required
tag.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since RFC v3:
- removed annotations needed for vhost async which went to the next
patch,
---
lib/vhost/vhost_user.c | 2 ++
lib/vhost/virtio_net.c | 4 +---
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 3f6c5df900..c57f092975 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2955,6 +2955,7 @@ vhost_user_check_and_alloc_queue_pair(struct virtio_net *dev,
static void
vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
+ __rte_no_thread_safety_analysis
{
unsigned int i = 0;
unsigned int vq_num = 0;
@@ -2972,6 +2973,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
static void
vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
+ __rte_no_thread_safety_analysis
{
unsigned int i = 0;
unsigned int vq_num = 0;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 2a75cda7b6..f05e379316 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -52,12 +52,10 @@ is_valid_virt_queue_idx(uint32_t idx, int is_tx, uint32_t nr_vring)
return (is_tx ^ (idx & 1)) == 0 && idx < nr_vring;
}
-/*
- * This function must be called with virtqueue's access_lock taken.
- */
static inline void
vhost_queue_stats_update(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct virtqueue_stats *stats = &vq->stats;
int i;
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 5/9] vhost: annotate async accesses
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
` (3 preceding siblings ...)
2023-01-19 18:46 ` [PATCH v4 4/9] vhost: annotate virtqueue access lock David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-31 16:54 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 6/9] vhost: always take IOTLB lock David Marchand
` (4 subsequent siblings)
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
vq->async is initialised and must be accessed under vq->access_lock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since RFC v3:
- rebased,
- fixed annotations vq->access_lock -> &vq->access_lock,
- reworked free_vq,
---
lib/vhost/vhost.c | 4 ++++
lib/vhost/vhost.h | 2 +-
lib/vhost/vhost_user.c | 10 +++++++---
lib/vhost/virtio_net.c | 35 +++++++++++++++++++++++++++++++++++
4 files changed, 47 insertions(+), 4 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 8cd727ca2f..8bccdd8584 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -369,6 +369,7 @@ cleanup_device(struct virtio_net *dev, int destroy)
static void
vhost_free_async_mem(struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
if (!vq->async)
return;
@@ -393,7 +394,9 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq)
else
rte_free(vq->shadow_used_split);
+ rte_spinlock_lock(&vq->access_lock);
vhost_free_async_mem(vq);
+ rte_spinlock_unlock(&vq->access_lock);
rte_free(vq->batch_copy_elems);
vhost_user_iotlb_destroy(vq);
rte_free(vq->log_cache);
@@ -1669,6 +1672,7 @@ rte_vhost_extern_callback_register(int vid,
static __rte_always_inline int
async_channel_register(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async;
int node = vq->numa_node;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index f6b2930efd..239ed02bd4 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -325,7 +325,7 @@ struct vhost_virtqueue {
struct rte_vhost_resubmit_info *resubmit_inflight;
uint64_t global_counter;
- struct vhost_async *async;
+ struct vhost_async *async __rte_guarded_var;
int notif_enable;
#define VIRTIO_UNINITIALIZED_NOTIF (-1)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index c57f092975..75c7fc851e 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2159,6 +2159,7 @@ vhost_user_set_vring_enable(struct virtio_net **pdev,
int main_fd __rte_unused)
{
struct virtio_net *dev = *pdev;
+ struct vhost_virtqueue *vq;
bool enable = !!ctx->msg.payload.state.num;
int index = (int)ctx->msg.payload.state.index;
@@ -2166,15 +2167,18 @@ vhost_user_set_vring_enable(struct virtio_net **pdev,
"set queue enable: %d to qp idx: %d\n",
enable, index);
- if (enable && dev->virtqueue[index]->async) {
- if (dev->virtqueue[index]->async->pkts_inflight_n) {
+ vq = dev->virtqueue[index];
+ /* vhost_user_lock_all_queue_pairs locked all qps */
+ vq_assert_lock(dev, vq);
+ if (enable && vq->async) {
+ if (vq->async->pkts_inflight_n) {
VHOST_LOG_CONFIG(dev->ifname, ERR,
"failed to enable vring. Inflight packets must be completed first\n");
return RTE_VHOST_MSG_RESULT_ERR;
}
}
- dev->virtqueue[index]->enabled = enable;
+ vq->enabled = enable;
return RTE_VHOST_MSG_RESULT_OK;
}
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index f05e379316..eedaf0fbf3 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -102,6 +102,7 @@ static __rte_always_inline int64_t
vhost_async_dma_transfer_one(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t flag_idx,
struct vhost_iov_iter *pkt)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
uint16_t ring_mask = dma_info->ring_mask;
@@ -151,6 +152,7 @@ static __rte_always_inline uint16_t
vhost_async_dma_transfer(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t head_idx,
struct vhost_iov_iter *pkts, uint16_t nr_pkts)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
int64_t ret, nr_copies = 0;
@@ -1063,6 +1065,7 @@ static __rte_always_inline int
async_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1140,6 +1143,7 @@ static __rte_always_inline int
mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1268,6 +1272,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkt,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1328,6 +1333,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1497,6 +1503,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1521,6 +1528,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -1620,6 +1628,7 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
static __rte_always_inline uint16_t
async_get_first_inflight_pkt_idx(struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
@@ -1665,6 +1674,7 @@ store_dma_desc_info_packed(struct vring_used_elem_packed *s_ring,
static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1771,6 +1781,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs,
uint16_t *nr_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1828,6 +1839,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1847,6 +1859,7 @@ virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
uint32_t nr_err, uint32_t *pkt_idx)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t descs_err = 0;
uint16_t buffers_err = 0;
@@ -1873,6 +1886,7 @@ dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t n_xfer;
@@ -1942,6 +1956,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
static __rte_always_inline void
write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t nr_left = n_descs;
@@ -1974,6 +1989,7 @@ write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
static __rte_always_inline void
write_back_completed_descs_packed(struct vhost_virtqueue *vq,
uint16_t n_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t from = async->last_buffer_idx_packed;
@@ -2038,6 +2054,7 @@ write_back_completed_descs_packed(struct vhost_virtqueue *vq,
static __rte_always_inline uint16_t
vhost_poll_enqueue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
struct async_inflight_info *pkts_info = async->pkts_info;
@@ -2642,6 +2659,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t nr_vec,
struct rte_mbuf *m, struct rte_mempool *mbuf_pool,
bool legacy_ol_flags, uint16_t slot_idx, bool is_async)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t buf_avail, buf_offset, buf_len;
uint64_t buf_addr, buf_iova;
@@ -2847,6 +2865,7 @@ static uint16_t
virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t i;
uint16_t avail_entries;
@@ -2950,6 +2969,7 @@ static uint16_t
virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2959,6 +2979,7 @@ static uint16_t
virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3085,6 +3106,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *buf_id,
uint16_t *desc_count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -3133,6 +3155,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mempool *mbuf_pool,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3163,6 +3186,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -3206,6 +3230,7 @@ static uint16_t
virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3215,6 +3240,7 @@ static uint16_t
virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3332,6 +3358,7 @@ static __rte_always_inline uint16_t
async_poll_dequeue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t start_idx, from, i;
uint16_t nr_cpl_pkts = 0;
@@ -3378,6 +3405,7 @@ static __rte_always_inline uint16_t
virtio_dev_tx_async_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
static bool allocerr_warned;
bool dropped = false;
@@ -3524,6 +3552,7 @@ virtio_dev_tx_async_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3535,6 +3564,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
@@ -3543,6 +3573,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
static __rte_always_inline void
vhost_async_shadow_dequeue_single_packed(struct vhost_virtqueue *vq,
uint16_t buf_id, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t idx = async->buffer_idx_packed;
@@ -3564,6 +3595,7 @@ virtio_dev_tx_async_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkts,
uint16_t slot_idx,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
int err;
uint16_t buf_id, desc_count = 0;
@@ -3614,6 +3646,7 @@ static __rte_always_inline uint16_t
virtio_dev_tx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t pkt_idx;
uint16_t slot_idx = 0;
@@ -3707,6 +3740,7 @@ static uint16_t
virtio_dev_tx_async_packed_legacy(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3717,6 +3751,7 @@ static uint16_t
virtio_dev_tx_async_packed_compliant(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 6/9] vhost: always take IOTLB lock
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
` (4 preceding siblings ...)
2023-01-19 18:46 ` [PATCH v4 5/9] vhost: annotate async accesses David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-31 16:59 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 7/9] vhost: annotate " David Marchand
` (3 subsequent siblings)
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
clang does not support conditionally held locks when statically analysing
taken locks with thread safety checks.
Always take iotlb locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/vhost.c | 8 +++-----
lib/vhost/virtio_net.c | 24 ++++++++----------------
2 files changed, 11 insertions(+), 21 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 8bccdd8584..1e0c30791e 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -563,10 +563,9 @@ vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
}
void
-vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+vring_invalidate(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq)
{
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_lock(vq);
+ vhost_user_iotlb_wr_lock(vq);
vq->access_ok = false;
vq->desc = NULL;
@@ -574,8 +573,7 @@ vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
vq->used = NULL;
vq->log_guest_addr = 0;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_unlock(vq);
+ vhost_user_iotlb_wr_unlock(vq);
}
static void
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index eedaf0fbf3..ed92a855f8 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -1572,8 +1572,7 @@ virtio_dev_rx(struct virtio_net *dev, struct vhost_virtqueue *vq,
if (unlikely(!vq->enabled))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -1591,8 +1590,7 @@ virtio_dev_rx(struct virtio_net *dev, struct vhost_virtqueue *vq,
vhost_queue_stats_update(dev, vq, pkts, nb_tx);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -2312,8 +2310,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, struct vhost_virtqueue *vq,
if (unlikely(!vq->enabled || !vq->async))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -2333,8 +2330,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, struct vhost_virtqueue *vq,
vq->stats.inflight_submitted += nb_tx;
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -3282,8 +3278,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -3342,8 +3337,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
vhost_queue_stats_update(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -3815,8 +3809,7 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(vq->access_ok == 0))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -3880,8 +3873,7 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
vhost_queue_stats_update(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 7/9] vhost: annotate IOTLB lock
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
` (5 preceding siblings ...)
2023-01-19 18:46 ` [PATCH v4 6/9] vhost: always take IOTLB lock David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-31 17:05 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 8/9] vhost: annotate vDPA device list accesses David Marchand
` (2 subsequent siblings)
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
The starting point for this is __vhost_iova_to_vva() which requires the
lock to be taken. Annotate all the code leading to a call to it.
vdpa and vhost_crypto code are annotated but they end up not taking
a IOTLB lock and have been marked with a FIXME at the top level.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since RFC v3:
- rebased,
- moved unconditionnal lock acquire in separate patch,
- fixed annotations,
---
lib/vhost/iotlb.h | 4 ++++
lib/vhost/vdpa.c | 1 +
lib/vhost/vhost.c | 8 +++-----
lib/vhost/vhost.h | 22 ++++++++++++++++------
lib/vhost/vhost_crypto.c | 8 ++++++++
lib/vhost/virtio_net.c | 40 ++++++++++++++++++++++++++++++++++++++++
6 files changed, 72 insertions(+), 11 deletions(-)
diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h
index e27ebebcf5..be079a8aa7 100644
--- a/lib/vhost/iotlb.h
+++ b/lib/vhost/iotlb.h
@@ -11,24 +11,28 @@
static __rte_always_inline void
vhost_user_iotlb_rd_lock(struct vhost_virtqueue *vq)
+ __rte_shared_lock_function(&vq->iotlb_lock)
{
rte_rwlock_read_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_rd_unlock(struct vhost_virtqueue *vq)
+ __rte_unlock_function(&vq->iotlb_lock)
{
rte_rwlock_read_unlock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_lock(struct vhost_virtqueue *vq)
+ __rte_exclusive_lock_function(&vq->iotlb_lock)
{
rte_rwlock_write_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_unlock(struct vhost_virtqueue *vq)
+ __rte_unlock_function(&vq->iotlb_lock)
{
rte_rwlock_write_unlock(&vq->iotlb_lock);
}
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index 577cb00a43..a430a66970 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -146,6 +146,7 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int
rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
+ __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct virtio_net *dev = get_device(vid);
uint16_t idx, idx_m, desc_id;
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 1e0c30791e..c36dc06a0a 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -52,7 +52,6 @@ static const struct vhost_vq_stats_name_off vhost_vq_stat_strings[] = {
#define VHOST_NB_VQ_STATS RTE_DIM(vhost_vq_stat_strings)
-/* Called with iotlb_lock read-locked */
uint64_t
__vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *size, uint8_t perm)
@@ -419,6 +418,7 @@ free_device(struct virtio_net *dev)
static __rte_always_inline int
log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(vq->ring_addrs.flags & (1 << VHOST_VRING_F_LOG))))
return 0;
@@ -435,8 +435,6 @@ log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
* Converts vring log address to GPA
* If IOMMU is enabled, the log address is IOVA
* If IOMMU not enabled, the log address is already GPA
- *
- * Caller should have iotlb_lock read-locked
*/
uint64_t
translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
@@ -467,9 +465,9 @@ translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
return log_addr;
}
-/* Caller should have iotlb_lock read-locked */
static int
vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -506,9 +504,9 @@ vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
return 0;
}
-/* Caller should have iotlb_lock read-locked */
static int
vring_translate_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t req_size, size;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 239ed02bd4..46aacd79c0 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -562,12 +562,15 @@ void __vhost_log_cache_write(struct virtio_net *dev,
uint64_t addr, uint64_t len);
void __vhost_log_cache_write_iova(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void __vhost_log_cache_sync(struct virtio_net *dev,
struct vhost_virtqueue *vq);
+
void __vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len);
void __vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
static __rte_always_inline void
vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len)
@@ -617,6 +620,7 @@ vhost_log_used_vring(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -630,6 +634,7 @@ vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -833,18 +838,23 @@ struct rte_vhost_device_ops const *vhost_driver_callback_get(const char *path);
void vhost_backend_cleanup(struct virtio_net *dev);
uint64_t __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t *len, uint8_t perm);
+ uint64_t iova, uint64_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void *vhost_alloc_copy_ind_table(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t desc_addr, uint64_t desc_len);
-int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq);
+ uint64_t desc_addr, uint64_t desc_len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
+int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock);
uint64_t translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t log_addr);
+ uint64_t log_addr)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq);
static __rte_always_inline uint64_t
vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (!(dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)))
return rte_vhost_va_from_guest_pa(dev->mem, iova, len);
diff --git a/lib/vhost/vhost_crypto.c b/lib/vhost/vhost_crypto.c
index b448b6685d..f02bf865c3 100644
--- a/lib/vhost/vhost_crypto.c
+++ b/lib/vhost/vhost_crypto.c
@@ -490,6 +490,7 @@ static __rte_always_inline struct virtio_crypto_inhdr *
reach_inhdr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct virtio_crypto_inhdr *inhdr;
struct vhost_crypto_desc *last = head + (max_n_descs - 1);
@@ -536,6 +537,7 @@ static __rte_always_inline void *
get_data_ptr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *cur_desc,
uint8_t perm)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
void *data;
uint64_t dlen = cur_desc->len;
@@ -552,6 +554,7 @@ get_data_ptr(struct vhost_crypto_data_req *vc_req,
static __rte_always_inline uint32_t
copy_data_from_desc(void *dst, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *desc, uint32_t size)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
uint64_t remain;
uint64_t addr;
@@ -582,6 +585,7 @@ static __rte_always_inline int
copy_data(void *data, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head, struct vhost_crypto_desc **cur_desc,
uint32_t size, uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = *cur_desc;
uint32_t left = size;
@@ -665,6 +669,7 @@ prepare_write_back_data(struct vhost_crypto_data_req *vc_req,
uint32_t offset,
uint64_t write_back_len,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_writeback_data *wb_data, *head;
struct vhost_crypto_desc *desc = *cur_desc;
@@ -785,6 +790,7 @@ prepare_sym_cipher_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_cipher_data_req *cipher,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head;
struct vhost_crypto_writeback_data *ewb = NULL;
@@ -938,6 +944,7 @@ prepare_sym_chain_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_alg_chain_data_req *chain,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head, *digest_desc;
struct vhost_crypto_writeback_data *ewb = NULL, *ewb2 = NULL;
@@ -1123,6 +1130,7 @@ vhost_crypto_process_one_req(struct vhost_crypto *vcrypto,
struct vhost_virtqueue *vq, struct rte_crypto_op *op,
struct vring_desc *head, struct vhost_crypto_desc *descs,
uint16_t desc_idx)
+ __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct vhost_crypto_data_req *vc_req = rte_mbuf_to_priv(op->sym->m_src);
struct rte_cryptodev_sym_session *session;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index ed92a855f8..9853adee1a 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -233,6 +233,7 @@ vhost_async_dma_check_completed(struct virtio_net *dev, int16_t dma_id, uint16_t
static inline void
do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct batch_copy_elem *elem = vq->batch_copy_elems;
uint16_t count = vq->batch_copy_nb_elems;
@@ -579,6 +580,7 @@ vhost_shadow_enqueue_single_packed(struct virtio_net *dev,
uint16_t *id,
uint16_t *count,
uint16_t num_buffers)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
vhost_shadow_enqueue_packed(vq, len, id, count, num_buffers);
@@ -670,6 +672,7 @@ static __rte_always_inline int
map_one_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint64_t desc_iova, uint64_t desc_len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t vec_id = *vec_idx;
@@ -707,6 +710,7 @@ fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t avail_idx, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint16_t *desc_chain_head,
uint32_t *desc_chain_len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
uint16_t vec_id = *vec_idx;
@@ -790,6 +794,7 @@ reserve_avail_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t size, struct buf_vector *buf_vec,
uint16_t *num_buffers, uint16_t avail_head,
uint16_t *nr_vec)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t cur_idx;
uint16_t vec_idx = 0;
@@ -840,6 +845,7 @@ fill_vec_buf_packed_indirect(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct vring_packed_desc *desc, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint32_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t i;
uint32_t nr_descs;
@@ -898,6 +904,7 @@ fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t avail_idx, uint16_t *desc_count,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint16_t *buf_id, uint32_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -963,6 +970,7 @@ static __rte_noinline void
copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec,
struct virtio_net_hdr_mrg_rxbuf *hdr)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t len;
uint64_t remain = dev->vhost_hlen;
@@ -1066,6 +1074,7 @@ async_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1106,6 +1115,7 @@ static __rte_always_inline void
sync_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_addr, uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
@@ -1144,6 +1154,7 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1273,6 +1284,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1334,6 +1346,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1390,6 +1403,7 @@ virtio_dev_rx_sync_batch_check(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -1441,6 +1455,7 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
@@ -1482,6 +1497,7 @@ static __rte_always_inline int
virtio_dev_rx_sync_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t desc_addrs[PACKED_BATCH_SIZE];
uint64_t lens[PACKED_BATCH_SIZE];
@@ -1504,6 +1520,7 @@ virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1529,6 +1546,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -1673,6 +1691,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1780,6 +1799,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
uint16_t *nr_descs,
uint16_t *nr_buffers)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1838,6 +1858,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1885,6 +1906,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t n_xfer;
@@ -2656,6 +2678,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct rte_mempool *mbuf_pool,
bool legacy_ol_flags, uint16_t slot_idx, bool is_async)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t buf_avail, buf_offset, buf_len;
uint64_t buf_addr, buf_iova;
@@ -2862,6 +2885,7 @@ virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t i;
uint16_t avail_entries;
@@ -2966,6 +2990,7 @@ virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2976,6 +3001,7 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -2987,6 +3013,7 @@ vhost_reserve_avail_batch_packed(struct virtio_net *dev,
uint16_t avail_idx,
uintptr_t *desc_addrs,
uint16_t *ids)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -3056,6 +3083,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts,
bool legacy_ol_flags)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t avail_idx = vq->last_avail_idx;
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
@@ -3103,6 +3131,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *desc_count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -3152,6 +3181,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3183,6 +3213,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
uint32_t count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -3227,6 +3258,7 @@ virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3237,6 +3269,7 @@ virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3400,6 +3433,7 @@ virtio_dev_tx_async_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
static bool allocerr_warned;
bool dropped = false;
@@ -3547,6 +3581,7 @@ virtio_dev_tx_async_split_legacy(struct virtio_net *dev,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3559,6 +3594,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
@@ -3590,6 +3626,7 @@ virtio_dev_tx_async_single_packed(struct virtio_net *dev,
uint16_t slot_idx,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
int err;
uint16_t buf_id, desc_count = 0;
@@ -3641,6 +3678,7 @@ virtio_dev_tx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t pkt_idx;
uint16_t slot_idx = 0;
@@ -3735,6 +3773,7 @@ virtio_dev_tx_async_packed_legacy(struct virtio_net *dev, struct vhost_virtqueue
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3746,6 +3785,7 @@ virtio_dev_tx_async_packed_compliant(struct virtio_net *dev, struct vhost_virtqu
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 8/9] vhost: annotate vDPA device list accesses
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
` (6 preceding siblings ...)
2023-01-19 18:46 ` [PATCH v4 7/9] vhost: annotate " David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-31 17:08 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 9/9] vhost: enable lock check David Marchand
2023-01-19 19:20 ` [PATCH v4 0/9] vhost lock annotations Morten Brørup
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
Access to vdpa_device_list must be protected with vdpa_device_list_lock
spinlock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Changes since RFC v3:
- rebased,
---
lib/vhost/vdpa.c | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index a430a66970..6284ea2ed1 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -23,21 +23,22 @@
/** Double linked list of vDPA devices. */
TAILQ_HEAD(vdpa_device_list, rte_vdpa_device);
-static struct vdpa_device_list vdpa_device_list =
- TAILQ_HEAD_INITIALIZER(vdpa_device_list);
+static struct vdpa_device_list vdpa_device_list__ =
+ TAILQ_HEAD_INITIALIZER(vdpa_device_list__);
static rte_spinlock_t vdpa_device_list_lock = RTE_SPINLOCK_INITIALIZER;
+static struct vdpa_device_list * const vdpa_device_list
+ __rte_guarded_by(&vdpa_device_list_lock) = &vdpa_device_list__;
-
-/* Unsafe, needs to be called with vdpa_device_list_lock held */
static struct rte_vdpa_device *
__vdpa_find_device_by_name(const char *name)
+ __rte_exclusive_locks_required(&vdpa_device_list_lock)
{
struct rte_vdpa_device *dev, *ret = NULL;
if (name == NULL)
return NULL;
- TAILQ_FOREACH(dev, &vdpa_device_list, next) {
+ TAILQ_FOREACH(dev, vdpa_device_list, next) {
if (!strncmp(dev->device->name, name, RTE_DEV_NAME_MAX_LEN)) {
ret = dev;
break;
@@ -116,7 +117,7 @@ rte_vdpa_register_device(struct rte_device *rte_dev,
dev->type = RTE_VHOST_VDPA_DEVICE_TYPE_NET;
}
- TAILQ_INSERT_TAIL(&vdpa_device_list, dev, next);
+ TAILQ_INSERT_TAIL(vdpa_device_list, dev, next);
out_unlock:
rte_spinlock_unlock(&vdpa_device_list_lock);
@@ -130,11 +131,11 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int ret = -1;
rte_spinlock_lock(&vdpa_device_list_lock);
- RTE_TAILQ_FOREACH_SAFE(cur_dev, &vdpa_device_list, next, tmp_dev) {
+ RTE_TAILQ_FOREACH_SAFE(cur_dev, vdpa_device_list, next, tmp_dev) {
if (dev != cur_dev)
continue;
- TAILQ_REMOVE(&vdpa_device_list, dev, next);
+ TAILQ_REMOVE(vdpa_device_list, dev, next);
rte_free(dev);
ret = 0;
break;
@@ -336,7 +337,7 @@ vdpa_find_device(const struct rte_vdpa_device *start, rte_vdpa_cmp_t cmp,
rte_spinlock_lock(&vdpa_device_list_lock);
if (start == NULL)
- dev = TAILQ_FIRST(&vdpa_device_list);
+ dev = TAILQ_FIRST(vdpa_device_list);
else
dev = TAILQ_NEXT(start, next);
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v4 9/9] vhost: enable lock check
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
` (7 preceding siblings ...)
2023-01-19 18:46 ` [PATCH v4 8/9] vhost: annotate vDPA device list accesses David Marchand
@ 2023-01-19 18:46 ` David Marchand
2023-01-31 17:14 ` Maxime Coquelin
2023-01-19 19:20 ` [PATCH v4 0/9] vhost lock annotations Morten Brørup
9 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 18:46 UTC (permalink / raw)
To: dev; +Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
Now that all locks in this library are annotated, we can enable the
check.
Signed-off-by: David Marchand <david.marchand@redhat.com>
---
lib/vhost/meson.build | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
index bc7272053b..197a51d936 100644
--- a/lib/vhost/meson.build
+++ b/lib/vhost/meson.build
@@ -17,6 +17,8 @@ elif (toolchain == 'icc' and cc.version().version_compare('>=16.0.0'))
endif
dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY', cc.has_header('linux/userfaultfd.h'))
cflags += '-fno-strict-aliasing'
+
+annotate_locks = true
sources = files(
'fd_man.c',
'iotlb.c',
--
2.39.0
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v4 0/9] vhost lock annotations
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
` (8 preceding siblings ...)
2023-01-19 18:46 ` [PATCH v4 9/9] vhost: enable lock check David Marchand
@ 2023-01-19 19:20 ` Morten Brørup
9 siblings, 0 replies; 110+ messages in thread
From: Morten Brørup @ 2023-01-19 19:20 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
> From: David Marchand [mailto:david.marchand@redhat.com]
> Sent: Thursday, 19 January 2023 19.46
>
> vhost internals involves multiple locks to protect data access by
> multiple threads.
>
> This series uses clang thread safety checks [1] to catch issues during
> compilation: EAL spinlock, seqlock and rwlock are annotated and vhost
> code is instrumented so that clang can statically check correctness.
>
> Those annotations are quite heavy to maintain because the full path of
> code must be annotated (as can be seen in the vhost datapath code),
> but I think it is worth using.
>
> This has been tested against the whole tree and some fixes are already
> flying on the mailing list (see [2] for a list).
>
> If this first series is merged, I will prepare a followup series for
> EAL
> and other libraries.
>
>
> 1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
> 2:
> https://patchwork.dpdk.org/bundle/dmarchand/lock_fixes/?state=*&archive
> =both
>
> --
Lock annotations seems like a good idea. I probably didn't notice the previous patch versions because the email subject camouflaged them as "vhost something...", so I skipped them.
Series-acked-by: Morten Brørup <mb@smartsharesystems.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 18:46 ` [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
@ 2023-01-19 19:42 ` Stephen Hemminger
2023-01-19 20:39 ` Tyler Retzlaff
2023-01-19 20:55 ` David Marchand
2023-01-19 19:43 ` Stephen Hemminger
2023-01-31 16:18 ` Maxime Coquelin
2 siblings, 2 replies; 110+ messages in thread
From: Stephen Hemminger @ 2023-01-19 19:42 UTC (permalink / raw)
To: David Marchand
Cc: dev, maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
On Thu, 19 Jan 2023 19:46:12 +0100
David Marchand <david.marchand@redhat.com> wrote:
> +#ifndef __DOXYGEN__
> + __rte_exclusive_lock_function(&seqlock->lock)
> +#endif
> {
Would be cleaner any required ifdefs was in rte_lock_annotations
rather than sprinkling the code
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 18:46 ` [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
2023-01-19 19:42 ` Stephen Hemminger
@ 2023-01-19 19:43 ` Stephen Hemminger
2023-01-31 16:18 ` Maxime Coquelin
2 siblings, 0 replies; 110+ messages in thread
From: Stephen Hemminger @ 2023-01-19 19:43 UTC (permalink / raw)
To: David Marchand
Cc: dev, maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
On Thu, 19 Jan 2023 19:46:12 +0100
David Marchand <david.marchand@redhat.com> wrote:
> clang offers some thread safety checks, statically verifying that locks
> are taken and released in the code.
> To use those checks, the full code leading to taking or releasing locks
> must be annotated with some attributes.
>
> Wrap those attributes into our own set of macros.
>
> rwlock, seqlock and the "normal" spinlock are instrumented.
>
> Those checks might be of interest out of DPDK, but it requires that the
> including application locks are annotated.
> On the other hand, applications out there might have been using
> those same checks.
> To be on the safe side, keep this instrumentation under a
> RTE_ANNOTATE_LOCKS internal build flag.
>
> A component may en/disable this check by setting
> annotate_locks = true/false in its meson.build.
Could this be made to work with sparse as well?
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 19:42 ` Stephen Hemminger
@ 2023-01-19 20:39 ` Tyler Retzlaff
2023-01-19 21:16 ` David Marchand
2023-01-19 20:55 ` David Marchand
1 sibling, 1 reply; 110+ messages in thread
From: Tyler Retzlaff @ 2023-01-19 20:39 UTC (permalink / raw)
To: Stephen Hemminger
Cc: David Marchand, dev, maxime.coquelin, chenbo.xia, jiayu.hu,
yuanx.wang, xuan.ding, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
On Thu, Jan 19, 2023 at 11:42:02AM -0800, Stephen Hemminger wrote:
> On Thu, 19 Jan 2023 19:46:12 +0100
> David Marchand <david.marchand@redhat.com> wrote:
>
> > +#ifndef __DOXYGEN__
> > + __rte_exclusive_lock_function(&seqlock->lock)
> > +#endif
> > {
>
> Would be cleaner any required ifdefs was in rte_lock_annotations
> rather than sprinkling the code
we briefly touched on abstracting annotations in another thread. it
would be favorable if annotations were stashed behind macros that could
be expanded for more than just clang/internal/under doxygen to make
available opportunities to use other annotation dialects that may be
compatible.
no change requested, just thoughts for discussion.
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 19:42 ` Stephen Hemminger
2023-01-19 20:39 ` Tyler Retzlaff
@ 2023-01-19 20:55 ` David Marchand
1 sibling, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-01-19 20:55 UTC (permalink / raw)
To: Stephen Hemminger
Cc: dev, maxime.coquelin, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
On Thu, Jan 19, 2023 at 8:42 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Thu, 19 Jan 2023 19:46:12 +0100
> David Marchand <david.marchand@redhat.com> wrote:
>
> > +#ifndef __DOXYGEN__
> > + __rte_exclusive_lock_function(&seqlock->lock)
> > +#endif
> > {
>
> Would be cleaner any required ifdefs was in rte_lock_annotations
> rather than sprinkling the code
>
IIRC doxygen was getting confused about parens around seqlock->lock
for this place only.
Other places seem fine, so it seemed more like a doxygen bug and I
waived the only location where it was needed.
I kind of forgot to investigate again.
Here is the trace:
$ ninja-build -C build-gcc doc
ninja: Entering directory `build-gcc'
[1/2] Generating doc/api/doxygen with a custom command
FAILED: doc/api/html
/usr/bin/python3 ../doc/api/generate_doxygen.py doc/api/html
/usr/bin/doxygen doc/api/doxy-api.conf
/home/dmarchan/git/pub/dpdk.org/main/lib/eal/include/rte_seqlock.h:218:
error: Found ')' without opening '(' for trailing return type ' ->
lock)...' (warning treated as error, aborting now)
Traceback (most recent call last):
File "/home/dmarchan/git/pub/dpdk.org/main/build-gcc/../doc/api/generate_doxygen.py",
line 13, in <module>
subprocess.run(doxygen_command, check=True, stdout=out)
File "/usr/lib64/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/doxygen',
'doc/api/doxy-api.conf']' returned non-zero exit status 1.
ninja: build stopped: subcommand failed.
$ doxygen --version
1.9.5
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 20:39 ` Tyler Retzlaff
@ 2023-01-19 21:16 ` David Marchand
2023-01-19 21:50 ` Tyler Retzlaff
0 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-01-19 21:16 UTC (permalink / raw)
To: Tyler Retzlaff
Cc: Stephen Hemminger, dev, maxime.coquelin, chenbo.xia, jiayu.hu,
yuanx.wang, xuan.ding, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
On Thu, Jan 19, 2023 at 9:39 PM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
>
> On Thu, Jan 19, 2023 at 11:42:02AM -0800, Stephen Hemminger wrote:
> > On Thu, 19 Jan 2023 19:46:12 +0100
> > David Marchand <david.marchand@redhat.com> wrote:
> >
> > > +#ifndef __DOXYGEN__
> > > + __rte_exclusive_lock_function(&seqlock->lock)
> > > +#endif
> > > {
> >
> > Would be cleaner any required ifdefs was in rte_lock_annotations
> > rather than sprinkling the code
>
> we briefly touched on abstracting annotations in another thread. it
> would be favorable if annotations were stashed behind macros that could
> be expanded for more than just clang/internal/under doxygen to make
> available opportunities to use other annotation dialects that may be
> compatible.
I am open to abstractions.
Do you have pointers for an equivalent functionnality in other
compilers/tooling?
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 21:16 ` David Marchand
@ 2023-01-19 21:50 ` Tyler Retzlaff
2023-01-26 12:18 ` David Marchand
0 siblings, 1 reply; 110+ messages in thread
From: Tyler Retzlaff @ 2023-01-19 21:50 UTC (permalink / raw)
To: David Marchand
Cc: Stephen Hemminger, dev, maxime.coquelin, chenbo.xia, jiayu.hu,
yuanx.wang, xuan.ding, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
On Thu, Jan 19, 2023 at 10:16:35PM +0100, David Marchand wrote:
> On Thu, Jan 19, 2023 at 9:39 PM Tyler Retzlaff
> <roretzla@linux.microsoft.com> wrote:
> >
> > On Thu, Jan 19, 2023 at 11:42:02AM -0800, Stephen Hemminger wrote:
> > > On Thu, 19 Jan 2023 19:46:12 +0100
> > > David Marchand <david.marchand@redhat.com> wrote:
> > >
> > > > +#ifndef __DOXYGEN__
> > > > + __rte_exclusive_lock_function(&seqlock->lock)
> > > > +#endif
> > > > {
> > >
> > > Would be cleaner any required ifdefs was in rte_lock_annotations
> > > rather than sprinkling the code
> >
> > we briefly touched on abstracting annotations in another thread. it
> > would be favorable if annotations were stashed behind macros that could
> > be expanded for more than just clang/internal/under doxygen to make
> > available opportunities to use other annotation dialects that may be
> > compatible.
>
> I am open to abstractions.
> Do you have pointers for an equivalent functionnality in other
> compilers/tooling?
aye, reference documentation for SALv2 is here.
https://learn.microsoft.com/en-us/cpp/code-quality/using-sal-annotations-to-reduce-c-cpp-code-defects?view=msvc-170
locking annotations are here.
https://learn.microsoft.com/en-us/cpp/code-quality/annotating-locking-behavior?view=msvc-170
but just to reiterate i'm not pushing any particular implementation, or
saying that SAL will ever be used. i think pragmatically all that would
be nice for now is not creating a direct dependency on any tool that
allow space for others in the future. if the burden to do that is too
much though let's just do what we can to get the benefits.
>
>
> --
> David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 21:50 ` Tyler Retzlaff
@ 2023-01-26 12:18 ` David Marchand
0 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-01-26 12:18 UTC (permalink / raw)
To: Tyler Retzlaff
Cc: Stephen Hemminger, dev, maxime.coquelin, chenbo.xia, jiayu.hu,
yuanx.wang, xuan.ding, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
On Thu, Jan 19, 2023 at 10:50 PM Tyler Retzlaff
<roretzla@linux.microsoft.com> wrote:
> > > we briefly touched on abstracting annotations in another thread. it
> > > would be favorable if annotations were stashed behind macros that could
> > > be expanded for more than just clang/internal/under doxygen to make
> > > available opportunities to use other annotation dialects that may be
> > > compatible.
> >
> > I am open to abstractions.
> > Do you have pointers for an equivalent functionnality in other
> > compilers/tooling?
>
> aye, reference documentation for SALv2 is here.
> https://learn.microsoft.com/en-us/cpp/code-quality/using-sal-annotations-to-reduce-c-cpp-code-defects?view=msvc-170
>
> locking annotations are here.
> https://learn.microsoft.com/en-us/cpp/code-quality/annotating-locking-behavior?view=msvc-170
I had a brief look at it and it seems close to what clang proposes.
I think there would be no issue with my current proposal since SAL
uses function attributes-like syntax for locking.
Do you see anything blocking?
Thanks Tyler.
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock
2023-01-19 18:46 ` [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
2023-01-19 19:42 ` Stephen Hemminger
2023-01-19 19:43 ` Stephen Hemminger
@ 2023-01-31 16:18 ` Maxime Coquelin
2 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 16:18 UTC (permalink / raw)
To: David Marchand, dev
Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding,
Anatoly Burakov, Mattias Rönnblom, David Christensen,
Bruce Richardson, Konstantin Ananyev
On 1/19/23 19:46, David Marchand wrote:
> clang offers some thread safety checks, statically verifying that locks
> are taken and released in the code.
> To use those checks, the full code leading to taking or releasing locks
> must be annotated with some attributes.
>
> Wrap those attributes into our own set of macros.
>
> rwlock, seqlock and the "normal" spinlock are instrumented.
>
> Those checks might be of interest out of DPDK, but it requires that the
> including application locks are annotated.
> On the other hand, applications out there might have been using
> those same checks.
> To be on the safe side, keep this instrumentation under a
> RTE_ANNOTATE_LOCKS internal build flag.
>
> A component may en/disable this check by setting
> annotate_locks = true/false in its meson.build.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changes since RFC v3:
> - rebased,
> - added some documentation,
> - added assert attribute,
> - instrumented seqlock,
> - cleaned annotations in arch-specific headers,
>
> Changes since RFC v2:
> - fixed rwlock trylock,
> - instrumented _tm spinlocks,
> - aligned attribute names to clang,
>
> ---
> .../prog_guide/env_abstraction_layer.rst | 24 ++++++
> doc/guides/rel_notes/release_23_03.rst | 5 ++
> drivers/meson.build | 5 ++
> lib/eal/include/generic/rte_rwlock.h | 27 +++++--
> lib/eal/include/generic/rte_spinlock.h | 31 +++++---
> lib/eal/include/meson.build | 1 +
> lib/eal/include/rte_lock_annotations.h | 73 +++++++++++++++++++
> lib/eal/include/rte_seqlock.h | 6 ++
> lib/eal/ppc/include/rte_spinlock.h | 3 +
> lib/eal/x86/include/rte_rwlock.h | 4 +
> lib/eal/x86/include/rte_spinlock.h | 9 +++
> lib/meson.build | 5 ++
> 12 files changed, 179 insertions(+), 14 deletions(-)
> create mode 100644 lib/eal/include/rte_lock_annotations.h
>
> diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
> index 35fbebe1be..6c1e8ba985 100644
> --- a/doc/guides/prog_guide/env_abstraction_layer.rst
> +++ b/doc/guides/prog_guide/env_abstraction_layer.rst
> @@ -529,6 +529,30 @@ Misc Functions
>
> Locks and atomic operations are per-architecture (i686 and x86_64).
>
> +Lock annotations
> +~~~~~~~~~~~~~~~~
> +
> +R/W locks, seq locks and spinlocks have been instrumented to help developers in
> +catching issues in DPDK.
> +
> +This instrumentation relies on
> +`clang Thread Safety checks <https://clang.llvm.org/docs/ThreadSafetyAnalysis.html>`_.
> +All attributes are prefixed with __rte and are fully described in the clang
> +documentation.
> +
> +Some general comments:
> +
> +- it is important that lock requirements are expressed at the function
> + declaration level in headers so that other code units can be inspected,
> +- when some global lock is necessary to some user-exposed API, it is preferred
> + to expose it via an internal helper rather than expose the global variable,
> +- there are a list of known limitations with clang instrumentation, but before
> + waiving checks with ``__rte_no_thread_safety_analysis`` in your code, please
> + discuss it on the mailing list,
> +
> +A DPDK library/driver can enabled/disable the checks by setting
s/enabled/enable/
> +``annotate_locks`` accordingly in its ``meson.build`` file.
> +
> IOVA Mode Detection
> ~~~~~~~~~~~~~~~~~~~
>
> diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
> index c15f6fbb9f..5425e59c65 100644
> --- a/doc/guides/rel_notes/release_23_03.rst
> +++ b/doc/guides/rel_notes/release_23_03.rst
> @@ -55,6 +55,11 @@ New Features
> Also, make sure to start the actual text at the margin.
> =======================================================
>
> +* **Introduced lock annotations.**
> +
> + Added lock annotations attributes to that clang can statically analyze lock
s/to/so/
> + correctness.
> +
With above typos fixed:
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 2/9] vhost: simplify need reply handling
2023-01-19 18:46 ` [PATCH v4 2/9] vhost: simplify need reply handling David Marchand
@ 2023-01-31 16:41 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 16:41 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 1/19/23 19:46, David Marchand wrote:
> Dedicate send_vhost_slave_message() helper to the case when no reply is
> needed.
>
> Add a send_vhost_slave_message_process_reply() helper for the opposite.
> This new helper merges both send_vhost_slave_message() and the code
> previously in process_slave_message_reply().
> The slave_req_lock lock is then only handled in this helper which will
> make lock checks trivial.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost_user.c | 119 ++++++++++++++++++-----------------------
> 1 file changed, 51 insertions(+), 68 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 3/9] vhost: terminate when access lock is not taken
2023-01-19 18:46 ` [PATCH v4 3/9] vhost: terminate when access lock is not taken David Marchand
@ 2023-01-31 16:47 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 16:47 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 1/19/23 19:46, David Marchand wrote:
> Be a bit more strict when a programmatic error is detected wrt to the
with regards to*
> access_lock not being taken.
> Mark the new helper with __rte_assert_exclusive_lock so that clang
> understands where locks are expected to be taken.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost.c | 18 +++---------------
> lib/vhost/vhost.h | 10 ++++++++++
> lib/vhost/virtio_net.c | 6 +-----
> 3 files changed, 14 insertions(+), 20 deletions(-)
>
Nice!
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 4/9] vhost: annotate virtqueue access lock
2023-01-19 18:46 ` [PATCH v4 4/9] vhost: annotate virtqueue access lock David Marchand
@ 2023-01-31 16:50 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 16:50 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 1/19/23 19:46, David Marchand wrote:
> vhost_user_lock/unlock_all_queue_pairs must be waived since clang
> annotations can't express taking a runtime number of locks.
>
> vhost_queue_stats_update() requirement can be expressed with a required
> tag.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changes since RFC v3:
> - removed annotations needed for vhost async which went to the next
> patch,
>
> ---
> lib/vhost/vhost_user.c | 2 ++
> lib/vhost/virtio_net.c | 4 +---
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 5/9] vhost: annotate async accesses
2023-01-19 18:46 ` [PATCH v4 5/9] vhost: annotate async accesses David Marchand
@ 2023-01-31 16:54 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 16:54 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 1/19/23 19:46, David Marchand wrote:
> vq->async is initialised and must be accessed under vq->access_lock.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changes since RFC v3:
> - rebased,
> - fixed annotations vq->access_lock -> &vq->access_lock,
> - reworked free_vq,
>
> ---
> lib/vhost/vhost.c | 4 ++++
> lib/vhost/vhost.h | 2 +-
> lib/vhost/vhost_user.c | 10 +++++++---
> lib/vhost/virtio_net.c | 35 +++++++++++++++++++++++++++++++++++
> 4 files changed, 47 insertions(+), 4 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 6/9] vhost: always take IOTLB lock
2023-01-19 18:46 ` [PATCH v4 6/9] vhost: always take IOTLB lock David Marchand
@ 2023-01-31 16:59 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 16:59 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 1/19/23 19:46, David Marchand wrote:
> clang does not support conditionally held locks when statically analysing
> taken locks with thread safety checks.
> Always take iotlb locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/vhost.c | 8 +++-----
> lib/vhost/virtio_net.c | 24 ++++++++----------------
> 2 files changed, 11 insertions(+), 21 deletions(-)
>
Since there will never be contention when VIRTIO_F_IOMMU_PLATFORM hasn't
been negotiated, there should not be noticeable overhead by taking it
unconditionally, especially since it is removing checking the feature
bit.
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 7/9] vhost: annotate IOTLB lock
2023-01-19 18:46 ` [PATCH v4 7/9] vhost: annotate " David Marchand
@ 2023-01-31 17:05 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 17:05 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 1/19/23 19:46, David Marchand wrote:
> The starting point for this is __vhost_iova_to_vva() which requires the
> lock to be taken. Annotate all the code leading to a call to it.
>
> vdpa and vhost_crypto code are annotated but they end up not taking
> a IOTLB lock and have been marked with a FIXME at the top level.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changes since RFC v3:
> - rebased,
> - moved unconditionnal lock acquire in separate patch,
> - fixed annotations,
>
> ---
> lib/vhost/iotlb.h | 4 ++++
> lib/vhost/vdpa.c | 1 +
> lib/vhost/vhost.c | 8 +++-----
> lib/vhost/vhost.h | 22 ++++++++++++++++------
> lib/vhost/vhost_crypto.c | 8 ++++++++
> lib/vhost/virtio_net.c | 40 ++++++++++++++++++++++++++++++++++++++++
> 6 files changed, 72 insertions(+), 11 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 8/9] vhost: annotate vDPA device list accesses
2023-01-19 18:46 ` [PATCH v4 8/9] vhost: annotate vDPA device list accesses David Marchand
@ 2023-01-31 17:08 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 17:08 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 1/19/23 19:46, David Marchand wrote:
> Access to vdpa_device_list must be protected with vdpa_device_list_lock
> spinlock.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Changes since RFC v3:
> - rebased,
>
> ---
> lib/vhost/vdpa.c | 19 ++++++++++---------
> 1 file changed, 10 insertions(+), 9 deletions(-)
>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v4 9/9] vhost: enable lock check
2023-01-19 18:46 ` [PATCH v4 9/9] vhost: enable lock check David Marchand
@ 2023-01-31 17:14 ` Maxime Coquelin
0 siblings, 0 replies; 110+ messages in thread
From: Maxime Coquelin @ 2023-01-31 17:14 UTC (permalink / raw)
To: David Marchand, dev; +Cc: stephen, chenbo.xia, jiayu.hu, yuanx.wang, xuan.ding
On 1/19/23 19:46, David Marchand wrote:
> Now that all locks in this library are annotated, we can enable the
> check.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> lib/vhost/meson.build | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
> index bc7272053b..197a51d936 100644
> --- a/lib/vhost/meson.build
> +++ b/lib/vhost/meson.build
> @@ -17,6 +17,8 @@ elif (toolchain == 'icc' and cc.version().version_compare('>=16.0.0'))
> endif
> dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY', cc.has_header('linux/userfaultfd.h'))
> cflags += '-fno-strict-aliasing'
> +
> +annotate_locks = true
> sources = files(
> 'fd_man.c',
> 'iotlb.c',
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Thanks,
Maxime
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 0/9] Lock annotations
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
` (7 preceding siblings ...)
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
` (8 more replies)
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
9 siblings, 9 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
vhost internals involves multiple locks to protect data access by
multiple threads.
This series uses clang thread safety checks [1] to catch issues during
compilation: EAL spinlock, seqlock and rwlock are annotated and vhost
code is instrumented so that clang can statically check correctness.
Those annotations are quite heavy to maintain because the full path of
code must be annotated (as can be seen in the vhost datapath code),
but I think it is worth using.
This has been tested against the whole tree and some fixes are already
flying on the mailing list (see [2] for a list).
If this first series is merged, I will prepare a followup series for EAL
and other libraries.
1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
2: https://patchwork.dpdk.org/bundle/dmarchand/lock_fixes/?state=*&archive=both
--
David Marchand
Changes since v4:
- masked annotations from Doxygen as it seems confused with some
constructs,
- fixed typos,
Changes since RFC v3:
- sorry Maxime, it has been too long since RFC v3 and the code evolved,
so I dropped all your review tags,
- rebased,
- added documentation,
- dropped/fixed annotations in arch-specific and EAL headers,
- rewrote need reply handling so that we don't have to waive the check
on the associated functions,
- separated IOTLB lock unconditional acquire from the annotation patch,
- rewrote runtime checks for "unsafe" functions using a panicking assert
helper,
Changes since RFC v2:
- fixed trylock annotations for rwlock,
- annotated _tm flavors of spinlock and rwlock,
- removed Maxime vhost fix from series (since Mimecast does not like
me sending Maxime patch...), added a dependency on original fix
as a hint for reviewers,
- renamed attributes,
Changes since RFC v1:
- Cc'd people who have pending patches for vhost,
- moved annotations to EAL and removed wrappers in vhost,
- as a result of moving to EAL, this series will be tested against
the main repo, so patch 1 has been kept as part of the series
even if already applied to next-virtio,
- refined/split patches and annotated all spinlocks in vhost,
David Marchand (9):
eal: annotate spinlock, rwlock and seqlock
vhost: simplify need reply handling
vhost: terminate when access lock is not taken
vhost: annotate virtqueue access lock
vhost: annotate async accesses
vhost: always take IOTLB lock
vhost: annotate IOTLB lock
vhost: annotate vDPA device list accesses
vhost: enable lock check
doc/api/doxy-api.conf.in | 11 ++
.../prog_guide/env_abstraction_layer.rst | 24 ++++
doc/guides/rel_notes/release_23_03.rst | 5 +
drivers/meson.build | 5 +
lib/eal/include/generic/rte_rwlock.h | 27 +++-
lib/eal/include/generic/rte_spinlock.h | 31 +++--
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 73 ++++++++++
lib/eal/include/rte_seqlock.h | 2 +
lib/eal/ppc/include/rte_spinlock.h | 3 +
lib/eal/x86/include/rte_rwlock.h | 4 +
lib/eal/x86/include/rte_spinlock.h | 9 ++
lib/meson.build | 5 +
lib/vhost/iotlb.h | 4 +
lib/vhost/meson.build | 2 +
lib/vhost/vdpa.c | 20 +--
lib/vhost/vhost.c | 38 ++---
lib/vhost/vhost.h | 34 ++++-
lib/vhost/vhost_crypto.c | 8 ++
lib/vhost/vhost_user.c | 131 ++++++++----------
lib/vhost/virtio_net.c | 109 +++++++++++----
21 files changed, 396 insertions(+), 150 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 12:32 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 2/9] vhost: simplify need reply handling David Marchand
` (7 subsequent siblings)
8 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
clang offers some thread safety checks, statically verifying that locks
are taken and released in the code.
To use those checks, the full code leading to taking or releasing locks
must be annotated with some attributes.
Wrap those attributes into our own set of macros.
rwlock, seqlock and the "normal" spinlock are instrumented.
Those checks might be of interest out of DPDK, but it requires that the
including application locks are annotated.
On the other hand, applications out there might have been using
those same checks.
To be on the safe side, keep this instrumentation under a
RTE_ANNOTATE_LOCKS internal build flag.
A component may en/disable this check by setting
annotate_locks = true/false in its meson.build.
Note:
Doxygen preprocessor does not understand trailing function attributes
(this can be observed with the rte_seqlock.h header).
One would think that expanding the annotation macros to a noop in
rte_lock_annotations.h would be enough (since RTE_ANNOTATE_LOCKS is not
set during doxygen processing)). Unfortunately, the use of
EXPAND_ONLY_PREDEF defeats this.
Removing EXPAND_ONLY_PREDEF entirely is not an option as it would expand
all other DPDK macros.
The chosen solution is to expand the annotation macros explicitly to a
noop in PREDEFINED.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since v4:
- hid annotations from Doxygen,
- fixed typos,
Changes since RFC v3:
- rebased,
- added some documentation,
- added assert attribute,
- instrumented seqlock,
- cleaned annotations in arch-specific headers,
Changes since RFC v2:
- fixed rwlock trylock,
- instrumented _tm spinlocks,
- aligned attribute names to clang,
---
doc/api/doxy-api.conf.in | 11 +++
.../prog_guide/env_abstraction_layer.rst | 24 ++++++
doc/guides/rel_notes/release_23_03.rst | 5 ++
drivers/meson.build | 5 ++
lib/eal/include/generic/rte_rwlock.h | 27 +++++--
lib/eal/include/generic/rte_spinlock.h | 31 +++++---
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 73 +++++++++++++++++++
lib/eal/include/rte_seqlock.h | 2 +
lib/eal/ppc/include/rte_spinlock.h | 3 +
lib/eal/x86/include/rte_rwlock.h | 4 +
lib/eal/x86/include/rte_spinlock.h | 9 +++
lib/meson.build | 5 ++
13 files changed, 186 insertions(+), 14 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index f0886c3bd1..e859426099 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -84,6 +84,17 @@ FILE_PATTERNS = rte_*.h \
PREDEFINED = __DOXYGEN__ \
RTE_HAS_CPUSET \
VFIO_PRESENT \
+ __rte_lockable= \
+ __rte_guarded_by(x)= \
+ __rte_exclusive_locks_required(x)= \
+ __rte_exclusive_lock_function(x)= \
+ __rte_exclusive_trylock_function(x)= \
+ __rte_assert_exclusive_lock(x)= \
+ __rte_shared_locks_required(x)= \
+ __rte_shared_lock_function(x)= \
+ __rte_shared_trylock_function(x)= \
+ __rte_assert_shared_lock(x)= \
+ __rte_unlock_function(x)= \
__attribute__(x)=
OPTIMIZE_OUTPUT_FOR_C = YES
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 35fbebe1be..3f33621e05 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -529,6 +529,30 @@ Misc Functions
Locks and atomic operations are per-architecture (i686 and x86_64).
+Lock annotations
+~~~~~~~~~~~~~~~~
+
+R/W locks, seq locks and spinlocks have been instrumented to help developers in
+catching issues in DPDK.
+
+This instrumentation relies on
+`clang Thread Safety checks <https://clang.llvm.org/docs/ThreadSafetyAnalysis.html>`_.
+All attributes are prefixed with __rte and are fully described in the clang
+documentation.
+
+Some general comments:
+
+- it is important that lock requirements are expressed at the function
+ declaration level in headers so that other code units can be inspected,
+- when some global lock is necessary to some user-exposed API, it is preferred
+ to expose it via an internal helper rather than expose the global variable,
+- there are a list of known limitations with clang instrumentation, but before
+ waiving checks with ``__rte_no_thread_safety_analysis`` in your code, please
+ discuss it on the mailing list,
+
+A DPDK library/driver can enable/disable the checks by setting
+``annotate_locks`` accordingly in its ``meson.build`` file.
+
IOVA Mode Detection
~~~~~~~~~~~~~~~~~~~
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index c15f6fbb9f..b12aabaaa8 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -55,6 +55,11 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Introduced lock annotations.**
+
+ Added lock annotations attributes so that clang can statically analyze lock
+ correctness.
+
* **Updated Intel QuickAssist Technology (QAT) crypto driver.**
* Added support for SHA3 224/256/384/512 plain hash in QAT GEN 3.
diff --git a/drivers/meson.build b/drivers/meson.build
index c6d619200f..bddc4a6cc4 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -91,6 +91,7 @@ foreach subpath:subdirs
build = true # set to false to disable, e.g. missing deps
reason = '<unknown reason>' # set if build == false to explain
name = drv
+ annotate_locks = false
sources = []
headers = []
driver_sdk_headers = [] # public headers included by drivers
@@ -167,6 +168,10 @@ foreach subpath:subdirs
enabled_drivers += name
lib_name = '_'.join(['rte', class, name])
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=' + '.'.join([log_prefix, name])
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
dpdk_conf.set(lib_name.to_upper(), 1)
dpdk_extra_ldflags += pkgconfig_extra_libs
diff --git a/lib/eal/include/generic/rte_rwlock.h b/lib/eal/include/generic/rte_rwlock.h
index 233d4262be..d45c22c189 100644
--- a/lib/eal/include/generic/rte_rwlock.h
+++ b/lib/eal/include/generic/rte_rwlock.h
@@ -30,6 +30,7 @@ extern "C" {
#include <rte_branch_prediction.h>
#include <rte_common.h>
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
@@ -55,7 +56,7 @@ extern "C" {
/* Writer is waiting or has lock */
#define RTE_RWLOCK_READ 0x4 /* Reader increment */
-typedef struct {
+typedef struct __rte_lockable {
int32_t cnt;
} rte_rwlock_t;
@@ -84,6 +85,8 @@ rte_rwlock_init(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_read_lock(rte_rwlock_t *rwl)
+ __rte_shared_lock_function(rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -119,6 +122,8 @@ rte_rwlock_read_lock(rte_rwlock_t *rwl)
*/
static inline int
rte_rwlock_read_trylock(rte_rwlock_t *rwl)
+ __rte_shared_trylock_function(0, rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -150,6 +155,8 @@ rte_rwlock_read_trylock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_read_unlock(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl)
+ __rte_no_thread_safety_analysis
{
__atomic_fetch_sub(&rwl->cnt, RTE_RWLOCK_READ, __ATOMIC_RELEASE);
}
@@ -166,6 +173,8 @@ rte_rwlock_read_unlock(rte_rwlock_t *rwl)
*/
static inline int
rte_rwlock_write_trylock(rte_rwlock_t *rwl)
+ __rte_exclusive_trylock_function(0, rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -186,6 +195,8 @@ rte_rwlock_write_trylock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_write_lock(rte_rwlock_t *rwl)
+ __rte_exclusive_lock_function(rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -219,6 +230,8 @@ rte_rwlock_write_lock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_write_unlock(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl)
+ __rte_no_thread_safety_analysis
{
__atomic_fetch_sub(&rwl->cnt, RTE_RWLOCK_WRITE, __ATOMIC_RELEASE);
}
@@ -237,7 +250,8 @@ rte_rwlock_write_unlock(rte_rwlock_t *rwl)
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_read_lock_tm(rte_rwlock_t *rwl);
+rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_shared_lock_function(rwl);
/**
* Commit hardware memory transaction or release the read lock if the lock is used as a fall-back
@@ -246,7 +260,8 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl);
* A pointer to the rwlock structure.
*/
static inline void
-rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl);
+rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl);
/**
* Try to execute critical section in a hardware memory transaction, if it
@@ -262,7 +277,8 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl);
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_lock_tm(rte_rwlock_t *rwl);
+rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_exclusive_lock_function(rwl);
/**
* Commit hardware memory transaction or release the write lock if the lock is used as a fall-back
@@ -271,7 +287,8 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl);
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl);
+rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl);
#ifdef __cplusplus
}
diff --git a/lib/eal/include/generic/rte_spinlock.h b/lib/eal/include/generic/rte_spinlock.h
index 73ed4bfbdc..8ca47bbfaa 100644
--- a/lib/eal/include/generic/rte_spinlock.h
+++ b/lib/eal/include/generic/rte_spinlock.h
@@ -22,12 +22,13 @@
#ifdef RTE_FORCE_INTRINSICS
#include <rte_common.h>
#endif
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
* The rte_spinlock_t type.
*/
-typedef struct {
+typedef struct __rte_lockable {
volatile int locked; /**< lock status 0 = unlocked, 1 = locked */
} rte_spinlock_t;
@@ -55,11 +56,13 @@ rte_spinlock_init(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock(rte_spinlock_t *sl);
+rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_exclusive_lock_function(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int exp = 0;
@@ -79,11 +82,13 @@ rte_spinlock_lock(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl);
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_unlock_function(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl)
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
__atomic_store_n(&sl->locked, 0, __ATOMIC_RELEASE);
}
@@ -99,11 +104,13 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
*/
__rte_warn_unused_result
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl);
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_exclusive_trylock_function(1, sl);
#ifdef RTE_FORCE_INTRINSICS
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl)
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int exp = 0;
return __atomic_compare_exchange_n(&sl->locked, &exp, 1,
@@ -147,7 +154,8 @@ static inline int rte_tm_supported(void);
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock_tm(rte_spinlock_t *sl);
+rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_exclusive_lock_function(sl);
/**
* Commit hardware memory transaction or release the spinlock if
@@ -157,7 +165,8 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl);
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock_tm(rte_spinlock_t *sl);
+rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_unlock_function(sl);
/**
* Try to execute critical section in a hardware memory transaction,
@@ -177,7 +186,8 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl);
*/
__rte_warn_unused_result
static inline int
-rte_spinlock_trylock_tm(rte_spinlock_t *sl);
+rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_exclusive_trylock_function(1, sl);
/**
* The rte_spinlock_recursive_t type.
@@ -213,6 +223,7 @@ static inline void rte_spinlock_recursive_init(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
int id = rte_gettid();
@@ -229,6 +240,7 @@ static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (--(slr->count) == 0) {
slr->user = -1;
@@ -247,6 +259,7 @@ static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
*/
__rte_warn_unused_result
static inline int rte_spinlock_recursive_trylock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
int id = rte_gettid();
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index cfcd40aaed..b0db9b3b3a 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -27,6 +27,7 @@ headers += files(
'rte_keepalive.h',
'rte_launch.h',
'rte_lcore.h',
+ 'rte_lock_annotations.h',
'rte_log.h',
'rte_malloc.h',
'rte_mcslock.h',
diff --git a/lib/eal/include/rte_lock_annotations.h b/lib/eal/include/rte_lock_annotations.h
new file mode 100644
index 0000000000..9fc50082d6
--- /dev/null
+++ b/lib/eal/include/rte_lock_annotations.h
@@ -0,0 +1,73 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Red Hat, Inc.
+ */
+
+#ifndef RTE_LOCK_ANNOTATIONS_H
+#define RTE_LOCK_ANNOTATIONS_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifdef RTE_ANNOTATE_LOCKS
+
+#define __rte_lockable \
+ __attribute__((lockable))
+
+#define __rte_guarded_by(...) \
+ __attribute__((guarded_by(__VA_ARGS__)))
+#define __rte_guarded_var \
+ __attribute__((guarded_var))
+
+#define __rte_exclusive_locks_required(...) \
+ __attribute__((exclusive_locks_required(__VA_ARGS__)))
+#define __rte_exclusive_lock_function(...) \
+ __attribute__((exclusive_lock_function(__VA_ARGS__)))
+#define __rte_exclusive_trylock_function(ret, ...) \
+ __attribute__((exclusive_trylock_function(ret, __VA_ARGS__)))
+#define __rte_assert_exclusive_lock(...) \
+ __attribute__((assert_exclusive_lock(__VA_ARGS__)))
+
+#define __rte_shared_locks_required(...) \
+ __attribute__((shared_locks_required(__VA_ARGS__)))
+#define __rte_shared_lock_function(...) \
+ __attribute__((shared_lock_function(__VA_ARGS__)))
+#define __rte_shared_trylock_function(ret, ...) \
+ __attribute__((shared_trylock_function(ret, __VA_ARGS__)))
+#define __rte_assert_shared_lock(...) \
+ __attribute__((assert_shared_lock(__VA_ARGS__)))
+
+#define __rte_unlock_function(...) \
+ __attribute__((unlock_function(__VA_ARGS__)))
+
+#define __rte_no_thread_safety_analysis \
+ __attribute__((no_thread_safety_analysis))
+
+#else /* ! RTE_ANNOTATE_LOCKS */
+
+#define __rte_lockable
+
+#define __rte_guarded_by(...)
+#define __rte_guarded_var
+
+#define __rte_exclusive_locks_required(...)
+#define __rte_exclusive_lock_function(...)
+#define __rte_exclusive_trylock_function(...)
+#define __rte_assert_exclusive_lock(...)
+
+#define __rte_shared_locks_required(...)
+#define __rte_shared_lock_function(...)
+#define __rte_shared_trylock_function(...)
+#define __rte_assert_shared_lock(...)
+
+#define __rte_unlock_function(...)
+
+#define __rte_no_thread_safety_analysis
+
+#endif /* RTE_ANNOTATE_LOCKS */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_LOCK_ANNOTATIONS_H */
diff --git a/lib/eal/include/rte_seqlock.h b/lib/eal/include/rte_seqlock.h
index 1663af62e8..fcbb9c5866 100644
--- a/lib/eal/include/rte_seqlock.h
+++ b/lib/eal/include/rte_seqlock.h
@@ -215,6 +215,7 @@ rte_seqlock_read_retry(const rte_seqlock_t *seqlock, uint32_t begin_sn)
__rte_experimental
static inline void
rte_seqlock_write_lock(rte_seqlock_t *seqlock)
+ __rte_exclusive_lock_function(&seqlock->lock)
{
/* To synchronize with other writers. */
rte_spinlock_lock(&seqlock->lock);
@@ -240,6 +241,7 @@ rte_seqlock_write_lock(rte_seqlock_t *seqlock)
__rte_experimental
static inline void
rte_seqlock_write_unlock(rte_seqlock_t *seqlock)
+ __rte_unlock_function(&seqlock->lock)
{
rte_seqcount_write_end(&seqlock->count);
diff --git a/lib/eal/ppc/include/rte_spinlock.h b/lib/eal/ppc/include/rte_spinlock.h
index 149ec245c7..3a4c905b22 100644
--- a/lib/eal/ppc/include/rte_spinlock.h
+++ b/lib/eal/ppc/include/rte_spinlock.h
@@ -20,6 +20,7 @@ extern "C" {
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
while (__sync_lock_test_and_set(&sl->locked, 1))
while (sl->locked)
@@ -28,12 +29,14 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
__sync_lock_release(&sl->locked);
}
static inline int
rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
return __sync_lock_test_and_set(&sl->locked, 1) == 0;
}
diff --git a/lib/eal/x86/include/rte_rwlock.h b/lib/eal/x86/include/rte_rwlock.h
index eec4c7123c..1796b69265 100644
--- a/lib/eal/x86/include/rte_rwlock.h
+++ b/lib/eal/x86/include/rte_rwlock.h
@@ -14,6 +14,7 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -22,6 +23,7 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(rwl->cnt))
rte_rwlock_read_unlock(rwl);
@@ -31,6 +33,7 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -39,6 +42,7 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(rwl->cnt))
rte_rwlock_write_unlock(rwl);
diff --git a/lib/eal/x86/include/rte_spinlock.h b/lib/eal/x86/include/rte_spinlock.h
index e2e2b2643c..0b20ddfd73 100644
--- a/lib/eal/x86/include/rte_spinlock.h
+++ b/lib/eal/x86/include/rte_spinlock.h
@@ -23,6 +23,7 @@ extern "C" {
#ifndef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int lock_val = 1;
asm volatile (
@@ -43,6 +44,7 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock (rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int unlock_val = 0;
asm volatile (
@@ -54,6 +56,7 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock (rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int lockval = 1;
@@ -121,6 +124,7 @@ rte_try_tm(volatile int *lock)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&sl->locked)))
return;
@@ -130,6 +134,7 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&sl->locked)))
return 1;
@@ -139,6 +144,7 @@ rte_spinlock_trylock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(sl->locked))
rte_spinlock_unlock(sl);
@@ -148,6 +154,7 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&slr->sl.locked)))
return;
@@ -157,6 +164,7 @@ rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (unlikely(slr->sl.locked))
rte_spinlock_recursive_unlock(slr);
@@ -166,6 +174,7 @@ rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&slr->sl.locked)))
return 1;
diff --git a/lib/meson.build b/lib/meson.build
index a90fee31b7..450c061d2b 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -120,6 +120,7 @@ foreach l:libraries
reason = '<unknown reason>' # set if build == false to explain why
name = l
use_function_versioning = false
+ annotate_locks = false
sources = []
headers = []
indirect_headers = [] # public headers not directly included by apps
@@ -202,6 +203,10 @@ foreach l:libraries
cflags += '-DRTE_USE_FUNCTION_VERSIONING'
endif
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=lib.' + l
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
# first build static lib
static_lib = static_library(libname,
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 2/9] vhost: simplify need reply handling
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
2023-02-01 11:14 ` [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 3/9] vhost: terminate when access lock is not taken David Marchand
` (6 subsequent siblings)
8 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
Dedicate send_vhost_slave_message() helper to the case when no reply is
needed.
Add a send_vhost_slave_message_process_reply() helper for the opposite.
This new helper merges both send_vhost_slave_message() and the code
previously in process_slave_message_reply().
The slave_req_lock lock is then only handled in this helper which will
make lock checks trivial.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost_user.c | 119 ++++++++++++++++++-----------------------
1 file changed, 51 insertions(+), 68 deletions(-)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 9902ae9944..3f6c5df900 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2868,18 +2868,46 @@ send_vhost_reply(struct virtio_net *dev, int sockfd, struct vhu_msg_context *ctx
}
static int
-send_vhost_slave_message(struct virtio_net *dev,
- struct vhu_msg_context *ctx)
+send_vhost_slave_message(struct virtio_net *dev, struct vhu_msg_context *ctx)
+{
+ return send_vhost_message(dev, dev->slave_req_fd, ctx);
+}
+
+static int
+send_vhost_slave_message_process_reply(struct virtio_net *dev, struct vhu_msg_context *ctx)
{
+ struct vhu_msg_context msg_reply;
int ret;
- if (ctx->msg.flags & VHOST_USER_NEED_REPLY)
- rte_spinlock_lock(&dev->slave_req_lock);
+ rte_spinlock_lock(&dev->slave_req_lock);
+ ret = send_vhost_slave_message(dev, ctx);
+ if (ret < 0) {
+ VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
+ goto out;
+ }
- ret = send_vhost_message(dev, dev->slave_req_fd, ctx);
- if (ret < 0 && (ctx->msg.flags & VHOST_USER_NEED_REPLY))
- rte_spinlock_unlock(&dev->slave_req_lock);
+ ret = read_vhost_message(dev, dev->slave_req_fd, &msg_reply);
+ if (ret <= 0) {
+ if (ret < 0)
+ VHOST_LOG_CONFIG(dev->ifname, ERR,
+ "vhost read slave message reply failed\n");
+ else
+ VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer closed\n");
+ ret = -1;
+ goto out;
+ }
+
+ if (msg_reply.msg.request.slave != ctx->msg.request.slave) {
+ VHOST_LOG_CONFIG(dev->ifname, ERR,
+ "received unexpected msg type (%u), expected %u\n",
+ msg_reply.msg.request.slave, ctx->msg.request.slave);
+ ret = -1;
+ goto out;
+ }
+ ret = msg_reply.msg.payload.u64 ? -1 : 0;
+out:
+ rte_spinlock_unlock(&dev->slave_req_lock);
return ret;
}
@@ -3203,42 +3231,6 @@ vhost_user_msg_handler(int vid, int fd)
return ret;
}
-static int process_slave_message_reply(struct virtio_net *dev,
- const struct vhu_msg_context *ctx)
-{
- struct vhu_msg_context msg_reply;
- int ret;
-
- if ((ctx->msg.flags & VHOST_USER_NEED_REPLY) == 0)
- return 0;
-
- ret = read_vhost_message(dev, dev->slave_req_fd, &msg_reply);
- if (ret <= 0) {
- if (ret < 0)
- VHOST_LOG_CONFIG(dev->ifname, ERR,
- "vhost read slave message reply failed\n");
- else
- VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer closed\n");
- ret = -1;
- goto out;
- }
-
- ret = 0;
- if (msg_reply.msg.request.slave != ctx->msg.request.slave) {
- VHOST_LOG_CONFIG(dev->ifname, ERR,
- "received unexpected msg type (%u), expected %u\n",
- msg_reply.msg.request.slave, ctx->msg.request.slave);
- ret = -1;
- goto out;
- }
-
- ret = msg_reply.msg.payload.u64 ? -1 : 0;
-
-out:
- rte_spinlock_unlock(&dev->slave_req_lock);
- return ret;
-}
-
int
vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm)
{
@@ -3267,10 +3259,9 @@ vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm)
return 0;
}
-static int
-vhost_user_slave_config_change(struct virtio_net *dev, bool need_reply)
+int
+rte_vhost_slave_config_change(int vid, bool need_reply)
{
- int ret;
struct vhu_msg_context ctx = {
.msg = {
.request.slave = VHOST_USER_SLAVE_CONFIG_CHANGE_MSG,
@@ -3278,29 +3269,23 @@ vhost_user_slave_config_change(struct virtio_net *dev, bool need_reply)
.size = 0,
}
};
-
- if (need_reply)
- ctx.msg.flags |= VHOST_USER_NEED_REPLY;
-
- ret = send_vhost_slave_message(dev, &ctx);
- if (ret < 0) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
- return ret;
- }
-
- return process_slave_message_reply(dev, &ctx);
-}
-
-int
-rte_vhost_slave_config_change(int vid, bool need_reply)
-{
struct virtio_net *dev;
+ int ret;
dev = get_device(vid);
if (!dev)
return -ENODEV;
- return vhost_user_slave_config_change(dev, need_reply);
+ if (!need_reply) {
+ ret = send_vhost_slave_message(dev, &ctx);
+ } else {
+ ctx.msg.flags |= VHOST_USER_NEED_REPLY;
+ ret = send_vhost_slave_message_process_reply(dev, &ctx);
+ }
+
+ if (ret < 0)
+ VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
+ return ret;
}
static int vhost_user_slave_set_vring_host_notifier(struct virtio_net *dev,
@@ -3329,13 +3314,11 @@ static int vhost_user_slave_set_vring_host_notifier(struct virtio_net *dev,
ctx.fd_num = 1;
}
- ret = send_vhost_slave_message(dev, &ctx);
- if (ret < 0) {
+ ret = send_vhost_slave_message_process_reply(dev, &ctx);
+ if (ret < 0)
VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to set host notifier (%d)\n", ret);
- return ret;
- }
- return process_slave_message_reply(dev, &ctx);
+ return ret;
}
int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable)
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 3/9] vhost: terminate when access lock is not taken
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
2023-02-01 11:14 ` [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
2023-02-01 11:14 ` [PATCH v5 2/9] vhost: simplify need reply handling David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 4/9] vhost: annotate virtqueue access lock David Marchand
` (5 subsequent siblings)
8 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
Be a bit more strict when a programmatic error is detected with regards to
the access_lock not being taken.
Mark the new helper with __rte_assert_exclusive_lock so that clang
understands where locks are expected to be taken.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost.c | 18 +++---------------
lib/vhost/vhost.h | 10 ++++++++++
lib/vhost/virtio_net.c | 6 +-----
3 files changed, 14 insertions(+), 20 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 19c7b92c32..8cd727ca2f 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1781,11 +1781,7 @@ rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
if (unlikely(vq == NULL || !dev->async_copy))
return -1;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
return async_channel_register(dev, vq);
}
@@ -1847,11 +1843,7 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)
if (vq == NULL)
return -1;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (!vq->async)
return 0;
@@ -1994,11 +1986,7 @@ rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (!vq->async)
return ret;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index ef211ed519..f6b2930efd 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -512,6 +512,16 @@ struct virtio_net {
struct rte_vhost_user_extern_ops extern_ops;
} __rte_cache_aligned;
+static inline void
+vq_assert_lock__(struct virtio_net *dev, struct vhost_virtqueue *vq, const char *func)
+ __rte_assert_exclusive_lock(&vq->access_lock)
+{
+ if (unlikely(!rte_spinlock_is_locked(&vq->access_lock)))
+ rte_panic("VHOST_CONFIG: (%s) %s() called without access lock taken.\n",
+ dev->ifname, func);
+}
+#define vq_assert_lock(dev, vq) vq_assert_lock__(dev, vq, __func__)
+
static __rte_always_inline bool
vq_is_packed(struct virtio_net *dev)
{
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 9abf752f30..2a75cda7b6 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -2185,11 +2185,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
vq = dev->virtqueue[queue_id];
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_DATA(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (unlikely(!vq->async)) {
VHOST_LOG_DATA(dev->ifname, ERR,
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 4/9] vhost: annotate virtqueue access lock
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
` (2 preceding siblings ...)
2023-02-01 11:14 ` [PATCH v5 3/9] vhost: terminate when access lock is not taken David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 5/9] vhost: annotate async accesses David Marchand
` (4 subsequent siblings)
8 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
vhost_user_lock/unlock_all_queue_pairs must be waived since clang
annotations can't express taking a runtime number of locks.
vhost_queue_stats_update() requirement can be expressed with a required
tag.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since RFC v3:
- removed annotations needed for vhost async which went to the next
patch,
---
lib/vhost/vhost_user.c | 2 ++
lib/vhost/virtio_net.c | 4 +---
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 3f6c5df900..c57f092975 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2955,6 +2955,7 @@ vhost_user_check_and_alloc_queue_pair(struct virtio_net *dev,
static void
vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
+ __rte_no_thread_safety_analysis
{
unsigned int i = 0;
unsigned int vq_num = 0;
@@ -2972,6 +2973,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
static void
vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
+ __rte_no_thread_safety_analysis
{
unsigned int i = 0;
unsigned int vq_num = 0;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 2a75cda7b6..f05e379316 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -52,12 +52,10 @@ is_valid_virt_queue_idx(uint32_t idx, int is_tx, uint32_t nr_vring)
return (is_tx ^ (idx & 1)) == 0 && idx < nr_vring;
}
-/*
- * This function must be called with virtqueue's access_lock taken.
- */
static inline void
vhost_queue_stats_update(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct virtqueue_stats *stats = &vq->stats;
int i;
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 5/9] vhost: annotate async accesses
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
` (3 preceding siblings ...)
2023-02-01 11:14 ` [PATCH v5 4/9] vhost: annotate virtqueue access lock David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 6/9] vhost: always take IOTLB lock David Marchand
` (3 subsequent siblings)
8 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
vq->async is initialised and must be accessed under vq->access_lock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since RFC v3:
- rebased,
- fixed annotations vq->access_lock -> &vq->access_lock,
- reworked free_vq,
---
lib/vhost/vhost.c | 4 ++++
lib/vhost/vhost.h | 2 +-
lib/vhost/vhost_user.c | 10 +++++++---
lib/vhost/virtio_net.c | 35 +++++++++++++++++++++++++++++++++++
4 files changed, 47 insertions(+), 4 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 8cd727ca2f..8bccdd8584 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -369,6 +369,7 @@ cleanup_device(struct virtio_net *dev, int destroy)
static void
vhost_free_async_mem(struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
if (!vq->async)
return;
@@ -393,7 +394,9 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq)
else
rte_free(vq->shadow_used_split);
+ rte_spinlock_lock(&vq->access_lock);
vhost_free_async_mem(vq);
+ rte_spinlock_unlock(&vq->access_lock);
rte_free(vq->batch_copy_elems);
vhost_user_iotlb_destroy(vq);
rte_free(vq->log_cache);
@@ -1669,6 +1672,7 @@ rte_vhost_extern_callback_register(int vid,
static __rte_always_inline int
async_channel_register(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async;
int node = vq->numa_node;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index f6b2930efd..239ed02bd4 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -325,7 +325,7 @@ struct vhost_virtqueue {
struct rte_vhost_resubmit_info *resubmit_inflight;
uint64_t global_counter;
- struct vhost_async *async;
+ struct vhost_async *async __rte_guarded_var;
int notif_enable;
#define VIRTIO_UNINITIALIZED_NOTIF (-1)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index c57f092975..75c7fc851e 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2159,6 +2159,7 @@ vhost_user_set_vring_enable(struct virtio_net **pdev,
int main_fd __rte_unused)
{
struct virtio_net *dev = *pdev;
+ struct vhost_virtqueue *vq;
bool enable = !!ctx->msg.payload.state.num;
int index = (int)ctx->msg.payload.state.index;
@@ -2166,15 +2167,18 @@ vhost_user_set_vring_enable(struct virtio_net **pdev,
"set queue enable: %d to qp idx: %d\n",
enable, index);
- if (enable && dev->virtqueue[index]->async) {
- if (dev->virtqueue[index]->async->pkts_inflight_n) {
+ vq = dev->virtqueue[index];
+ /* vhost_user_lock_all_queue_pairs locked all qps */
+ vq_assert_lock(dev, vq);
+ if (enable && vq->async) {
+ if (vq->async->pkts_inflight_n) {
VHOST_LOG_CONFIG(dev->ifname, ERR,
"failed to enable vring. Inflight packets must be completed first\n");
return RTE_VHOST_MSG_RESULT_ERR;
}
}
- dev->virtqueue[index]->enabled = enable;
+ vq->enabled = enable;
return RTE_VHOST_MSG_RESULT_OK;
}
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index f05e379316..eedaf0fbf3 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -102,6 +102,7 @@ static __rte_always_inline int64_t
vhost_async_dma_transfer_one(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t flag_idx,
struct vhost_iov_iter *pkt)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
uint16_t ring_mask = dma_info->ring_mask;
@@ -151,6 +152,7 @@ static __rte_always_inline uint16_t
vhost_async_dma_transfer(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t head_idx,
struct vhost_iov_iter *pkts, uint16_t nr_pkts)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
int64_t ret, nr_copies = 0;
@@ -1063,6 +1065,7 @@ static __rte_always_inline int
async_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1140,6 +1143,7 @@ static __rte_always_inline int
mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1268,6 +1272,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkt,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1328,6 +1333,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1497,6 +1503,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1521,6 +1528,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -1620,6 +1628,7 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
static __rte_always_inline uint16_t
async_get_first_inflight_pkt_idx(struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
@@ -1665,6 +1674,7 @@ store_dma_desc_info_packed(struct vring_used_elem_packed *s_ring,
static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1771,6 +1781,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs,
uint16_t *nr_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1828,6 +1839,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1847,6 +1859,7 @@ virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
uint32_t nr_err, uint32_t *pkt_idx)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t descs_err = 0;
uint16_t buffers_err = 0;
@@ -1873,6 +1886,7 @@ dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t n_xfer;
@@ -1942,6 +1956,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
static __rte_always_inline void
write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t nr_left = n_descs;
@@ -1974,6 +1989,7 @@ write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
static __rte_always_inline void
write_back_completed_descs_packed(struct vhost_virtqueue *vq,
uint16_t n_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t from = async->last_buffer_idx_packed;
@@ -2038,6 +2054,7 @@ write_back_completed_descs_packed(struct vhost_virtqueue *vq,
static __rte_always_inline uint16_t
vhost_poll_enqueue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
struct async_inflight_info *pkts_info = async->pkts_info;
@@ -2642,6 +2659,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t nr_vec,
struct rte_mbuf *m, struct rte_mempool *mbuf_pool,
bool legacy_ol_flags, uint16_t slot_idx, bool is_async)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t buf_avail, buf_offset, buf_len;
uint64_t buf_addr, buf_iova;
@@ -2847,6 +2865,7 @@ static uint16_t
virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t i;
uint16_t avail_entries;
@@ -2950,6 +2969,7 @@ static uint16_t
virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2959,6 +2979,7 @@ static uint16_t
virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3085,6 +3106,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *buf_id,
uint16_t *desc_count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -3133,6 +3155,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mempool *mbuf_pool,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3163,6 +3186,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -3206,6 +3230,7 @@ static uint16_t
virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3215,6 +3240,7 @@ static uint16_t
virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3332,6 +3358,7 @@ static __rte_always_inline uint16_t
async_poll_dequeue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t start_idx, from, i;
uint16_t nr_cpl_pkts = 0;
@@ -3378,6 +3405,7 @@ static __rte_always_inline uint16_t
virtio_dev_tx_async_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
static bool allocerr_warned;
bool dropped = false;
@@ -3524,6 +3552,7 @@ virtio_dev_tx_async_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3535,6 +3564,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
@@ -3543,6 +3573,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
static __rte_always_inline void
vhost_async_shadow_dequeue_single_packed(struct vhost_virtqueue *vq,
uint16_t buf_id, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t idx = async->buffer_idx_packed;
@@ -3564,6 +3595,7 @@ virtio_dev_tx_async_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkts,
uint16_t slot_idx,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
int err;
uint16_t buf_id, desc_count = 0;
@@ -3614,6 +3646,7 @@ static __rte_always_inline uint16_t
virtio_dev_tx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t pkt_idx;
uint16_t slot_idx = 0;
@@ -3707,6 +3740,7 @@ static uint16_t
virtio_dev_tx_async_packed_legacy(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3717,6 +3751,7 @@ static uint16_t
virtio_dev_tx_async_packed_compliant(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 6/9] vhost: always take IOTLB lock
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
` (4 preceding siblings ...)
2023-02-01 11:14 ` [PATCH v5 5/9] vhost: annotate async accesses David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 7/9] vhost: annotate " David Marchand
` (2 subsequent siblings)
8 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
clang does not support conditionally held locks when statically analysing
taken locks with thread safety checks.
Always take iotlb locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost.c | 8 +++-----
lib/vhost/virtio_net.c | 24 ++++++++----------------
2 files changed, 11 insertions(+), 21 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 8bccdd8584..1e0c30791e 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -563,10 +563,9 @@ vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
}
void
-vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+vring_invalidate(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq)
{
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_lock(vq);
+ vhost_user_iotlb_wr_lock(vq);
vq->access_ok = false;
vq->desc = NULL;
@@ -574,8 +573,7 @@ vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
vq->used = NULL;
vq->log_guest_addr = 0;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_unlock(vq);
+ vhost_user_iotlb_wr_unlock(vq);
}
static void
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index eedaf0fbf3..ed92a855f8 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -1572,8 +1572,7 @@ virtio_dev_rx(struct virtio_net *dev, struct vhost_virtqueue *vq,
if (unlikely(!vq->enabled))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -1591,8 +1590,7 @@ virtio_dev_rx(struct virtio_net *dev, struct vhost_virtqueue *vq,
vhost_queue_stats_update(dev, vq, pkts, nb_tx);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -2312,8 +2310,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, struct vhost_virtqueue *vq,
if (unlikely(!vq->enabled || !vq->async))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -2333,8 +2330,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, struct vhost_virtqueue *vq,
vq->stats.inflight_submitted += nb_tx;
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -3282,8 +3278,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -3342,8 +3337,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
vhost_queue_stats_update(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -3815,8 +3809,7 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(vq->access_ok == 0))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -3880,8 +3873,7 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
vhost_queue_stats_update(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 7/9] vhost: annotate IOTLB lock
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
` (5 preceding siblings ...)
2023-02-01 11:14 ` [PATCH v5 6/9] vhost: always take IOTLB lock David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 8/9] vhost: annotate vDPA device list accesses David Marchand
2023-02-01 11:14 ` [PATCH v5 9/9] vhost: enable lock check David Marchand
8 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
The starting point for this is __vhost_iova_to_vva() which requires the
lock to be taken. Annotate all the code leading to a call to it.
vdpa and vhost_crypto code are annotated but they end up not taking
a IOTLB lock and have been marked with a FIXME at the top level.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since RFC v3:
- rebased,
- moved unconditionnal lock acquire in separate patch,
- fixed annotations,
---
lib/vhost/iotlb.h | 4 ++++
lib/vhost/vdpa.c | 1 +
lib/vhost/vhost.c | 8 +++-----
lib/vhost/vhost.h | 22 ++++++++++++++++------
lib/vhost/vhost_crypto.c | 8 ++++++++
lib/vhost/virtio_net.c | 40 ++++++++++++++++++++++++++++++++++++++++
6 files changed, 72 insertions(+), 11 deletions(-)
diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h
index e27ebebcf5..be079a8aa7 100644
--- a/lib/vhost/iotlb.h
+++ b/lib/vhost/iotlb.h
@@ -11,24 +11,28 @@
static __rte_always_inline void
vhost_user_iotlb_rd_lock(struct vhost_virtqueue *vq)
+ __rte_shared_lock_function(&vq->iotlb_lock)
{
rte_rwlock_read_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_rd_unlock(struct vhost_virtqueue *vq)
+ __rte_unlock_function(&vq->iotlb_lock)
{
rte_rwlock_read_unlock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_lock(struct vhost_virtqueue *vq)
+ __rte_exclusive_lock_function(&vq->iotlb_lock)
{
rte_rwlock_write_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_unlock(struct vhost_virtqueue *vq)
+ __rte_unlock_function(&vq->iotlb_lock)
{
rte_rwlock_write_unlock(&vq->iotlb_lock);
}
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index 577cb00a43..a430a66970 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -146,6 +146,7 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int
rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
+ __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct virtio_net *dev = get_device(vid);
uint16_t idx, idx_m, desc_id;
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 1e0c30791e..c36dc06a0a 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -52,7 +52,6 @@ static const struct vhost_vq_stats_name_off vhost_vq_stat_strings[] = {
#define VHOST_NB_VQ_STATS RTE_DIM(vhost_vq_stat_strings)
-/* Called with iotlb_lock read-locked */
uint64_t
__vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *size, uint8_t perm)
@@ -419,6 +418,7 @@ free_device(struct virtio_net *dev)
static __rte_always_inline int
log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(vq->ring_addrs.flags & (1 << VHOST_VRING_F_LOG))))
return 0;
@@ -435,8 +435,6 @@ log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
* Converts vring log address to GPA
* If IOMMU is enabled, the log address is IOVA
* If IOMMU not enabled, the log address is already GPA
- *
- * Caller should have iotlb_lock read-locked
*/
uint64_t
translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
@@ -467,9 +465,9 @@ translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
return log_addr;
}
-/* Caller should have iotlb_lock read-locked */
static int
vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -506,9 +504,9 @@ vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
return 0;
}
-/* Caller should have iotlb_lock read-locked */
static int
vring_translate_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t req_size, size;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 239ed02bd4..46aacd79c0 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -562,12 +562,15 @@ void __vhost_log_cache_write(struct virtio_net *dev,
uint64_t addr, uint64_t len);
void __vhost_log_cache_write_iova(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void __vhost_log_cache_sync(struct virtio_net *dev,
struct vhost_virtqueue *vq);
+
void __vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len);
void __vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
static __rte_always_inline void
vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len)
@@ -617,6 +620,7 @@ vhost_log_used_vring(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -630,6 +634,7 @@ vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -833,18 +838,23 @@ struct rte_vhost_device_ops const *vhost_driver_callback_get(const char *path);
void vhost_backend_cleanup(struct virtio_net *dev);
uint64_t __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t *len, uint8_t perm);
+ uint64_t iova, uint64_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void *vhost_alloc_copy_ind_table(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t desc_addr, uint64_t desc_len);
-int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq);
+ uint64_t desc_addr, uint64_t desc_len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
+int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock);
uint64_t translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t log_addr);
+ uint64_t log_addr)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq);
static __rte_always_inline uint64_t
vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (!(dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)))
return rte_vhost_va_from_guest_pa(dev->mem, iova, len);
diff --git a/lib/vhost/vhost_crypto.c b/lib/vhost/vhost_crypto.c
index b448b6685d..f02bf865c3 100644
--- a/lib/vhost/vhost_crypto.c
+++ b/lib/vhost/vhost_crypto.c
@@ -490,6 +490,7 @@ static __rte_always_inline struct virtio_crypto_inhdr *
reach_inhdr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct virtio_crypto_inhdr *inhdr;
struct vhost_crypto_desc *last = head + (max_n_descs - 1);
@@ -536,6 +537,7 @@ static __rte_always_inline void *
get_data_ptr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *cur_desc,
uint8_t perm)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
void *data;
uint64_t dlen = cur_desc->len;
@@ -552,6 +554,7 @@ get_data_ptr(struct vhost_crypto_data_req *vc_req,
static __rte_always_inline uint32_t
copy_data_from_desc(void *dst, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *desc, uint32_t size)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
uint64_t remain;
uint64_t addr;
@@ -582,6 +585,7 @@ static __rte_always_inline int
copy_data(void *data, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head, struct vhost_crypto_desc **cur_desc,
uint32_t size, uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = *cur_desc;
uint32_t left = size;
@@ -665,6 +669,7 @@ prepare_write_back_data(struct vhost_crypto_data_req *vc_req,
uint32_t offset,
uint64_t write_back_len,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_writeback_data *wb_data, *head;
struct vhost_crypto_desc *desc = *cur_desc;
@@ -785,6 +790,7 @@ prepare_sym_cipher_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_cipher_data_req *cipher,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head;
struct vhost_crypto_writeback_data *ewb = NULL;
@@ -938,6 +944,7 @@ prepare_sym_chain_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_alg_chain_data_req *chain,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head, *digest_desc;
struct vhost_crypto_writeback_data *ewb = NULL, *ewb2 = NULL;
@@ -1123,6 +1130,7 @@ vhost_crypto_process_one_req(struct vhost_crypto *vcrypto,
struct vhost_virtqueue *vq, struct rte_crypto_op *op,
struct vring_desc *head, struct vhost_crypto_desc *descs,
uint16_t desc_idx)
+ __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct vhost_crypto_data_req *vc_req = rte_mbuf_to_priv(op->sym->m_src);
struct rte_cryptodev_sym_session *session;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index ed92a855f8..9853adee1a 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -233,6 +233,7 @@ vhost_async_dma_check_completed(struct virtio_net *dev, int16_t dma_id, uint16_t
static inline void
do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct batch_copy_elem *elem = vq->batch_copy_elems;
uint16_t count = vq->batch_copy_nb_elems;
@@ -579,6 +580,7 @@ vhost_shadow_enqueue_single_packed(struct virtio_net *dev,
uint16_t *id,
uint16_t *count,
uint16_t num_buffers)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
vhost_shadow_enqueue_packed(vq, len, id, count, num_buffers);
@@ -670,6 +672,7 @@ static __rte_always_inline int
map_one_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint64_t desc_iova, uint64_t desc_len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t vec_id = *vec_idx;
@@ -707,6 +710,7 @@ fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t avail_idx, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint16_t *desc_chain_head,
uint32_t *desc_chain_len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
uint16_t vec_id = *vec_idx;
@@ -790,6 +794,7 @@ reserve_avail_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t size, struct buf_vector *buf_vec,
uint16_t *num_buffers, uint16_t avail_head,
uint16_t *nr_vec)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t cur_idx;
uint16_t vec_idx = 0;
@@ -840,6 +845,7 @@ fill_vec_buf_packed_indirect(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct vring_packed_desc *desc, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint32_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t i;
uint32_t nr_descs;
@@ -898,6 +904,7 @@ fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t avail_idx, uint16_t *desc_count,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint16_t *buf_id, uint32_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -963,6 +970,7 @@ static __rte_noinline void
copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec,
struct virtio_net_hdr_mrg_rxbuf *hdr)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t len;
uint64_t remain = dev->vhost_hlen;
@@ -1066,6 +1074,7 @@ async_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1106,6 +1115,7 @@ static __rte_always_inline void
sync_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_addr, uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
@@ -1144,6 +1154,7 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1273,6 +1284,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1334,6 +1346,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1390,6 +1403,7 @@ virtio_dev_rx_sync_batch_check(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -1441,6 +1455,7 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
@@ -1482,6 +1497,7 @@ static __rte_always_inline int
virtio_dev_rx_sync_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t desc_addrs[PACKED_BATCH_SIZE];
uint64_t lens[PACKED_BATCH_SIZE];
@@ -1504,6 +1520,7 @@ virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1529,6 +1546,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -1673,6 +1691,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1780,6 +1799,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
uint16_t *nr_descs,
uint16_t *nr_buffers)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1838,6 +1858,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1885,6 +1906,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t n_xfer;
@@ -2656,6 +2678,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct rte_mempool *mbuf_pool,
bool legacy_ol_flags, uint16_t slot_idx, bool is_async)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t buf_avail, buf_offset, buf_len;
uint64_t buf_addr, buf_iova;
@@ -2862,6 +2885,7 @@ virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t i;
uint16_t avail_entries;
@@ -2966,6 +2990,7 @@ virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -2976,6 +3001,7 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -2987,6 +3013,7 @@ vhost_reserve_avail_batch_packed(struct virtio_net *dev,
uint16_t avail_idx,
uintptr_t *desc_addrs,
uint16_t *ids)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -3056,6 +3083,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts,
bool legacy_ol_flags)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t avail_idx = vq->last_avail_idx;
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
@@ -3103,6 +3131,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *desc_count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -3152,6 +3181,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3183,6 +3213,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
uint32_t count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -3227,6 +3258,7 @@ virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3237,6 +3269,7 @@ virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3400,6 +3433,7 @@ virtio_dev_tx_async_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
static bool allocerr_warned;
bool dropped = false;
@@ -3547,6 +3581,7 @@ virtio_dev_tx_async_split_legacy(struct virtio_net *dev,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3559,6 +3594,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
@@ -3590,6 +3626,7 @@ virtio_dev_tx_async_single_packed(struct virtio_net *dev,
uint16_t slot_idx,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
int err;
uint16_t buf_id, desc_count = 0;
@@ -3641,6 +3678,7 @@ virtio_dev_tx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t pkt_idx;
uint16_t slot_idx = 0;
@@ -3735,6 +3773,7 @@ virtio_dev_tx_async_packed_legacy(struct virtio_net *dev, struct vhost_virtqueue
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3746,6 +3785,7 @@ virtio_dev_tx_async_packed_compliant(struct virtio_net *dev, struct vhost_virtqu
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 8/9] vhost: annotate vDPA device list accesses
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
` (6 preceding siblings ...)
2023-02-01 11:14 ` [PATCH v5 7/9] vhost: annotate " David Marchand
@ 2023-02-01 11:14 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 9/9] vhost: enable lock check David Marchand
8 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
Access to vdpa_device_list must be protected with vdpa_device_list_lock
spinlock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since RFC v3:
- rebased,
---
lib/vhost/vdpa.c | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index a430a66970..6284ea2ed1 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -23,21 +23,22 @@
/** Double linked list of vDPA devices. */
TAILQ_HEAD(vdpa_device_list, rte_vdpa_device);
-static struct vdpa_device_list vdpa_device_list =
- TAILQ_HEAD_INITIALIZER(vdpa_device_list);
+static struct vdpa_device_list vdpa_device_list__ =
+ TAILQ_HEAD_INITIALIZER(vdpa_device_list__);
static rte_spinlock_t vdpa_device_list_lock = RTE_SPINLOCK_INITIALIZER;
+static struct vdpa_device_list * const vdpa_device_list
+ __rte_guarded_by(&vdpa_device_list_lock) = &vdpa_device_list__;
-
-/* Unsafe, needs to be called with vdpa_device_list_lock held */
static struct rte_vdpa_device *
__vdpa_find_device_by_name(const char *name)
+ __rte_exclusive_locks_required(&vdpa_device_list_lock)
{
struct rte_vdpa_device *dev, *ret = NULL;
if (name == NULL)
return NULL;
- TAILQ_FOREACH(dev, &vdpa_device_list, next) {
+ TAILQ_FOREACH(dev, vdpa_device_list, next) {
if (!strncmp(dev->device->name, name, RTE_DEV_NAME_MAX_LEN)) {
ret = dev;
break;
@@ -116,7 +117,7 @@ rte_vdpa_register_device(struct rte_device *rte_dev,
dev->type = RTE_VHOST_VDPA_DEVICE_TYPE_NET;
}
- TAILQ_INSERT_TAIL(&vdpa_device_list, dev, next);
+ TAILQ_INSERT_TAIL(vdpa_device_list, dev, next);
out_unlock:
rte_spinlock_unlock(&vdpa_device_list_lock);
@@ -130,11 +131,11 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int ret = -1;
rte_spinlock_lock(&vdpa_device_list_lock);
- RTE_TAILQ_FOREACH_SAFE(cur_dev, &vdpa_device_list, next, tmp_dev) {
+ RTE_TAILQ_FOREACH_SAFE(cur_dev, vdpa_device_list, next, tmp_dev) {
if (dev != cur_dev)
continue;
- TAILQ_REMOVE(&vdpa_device_list, dev, next);
+ TAILQ_REMOVE(vdpa_device_list, dev, next);
rte_free(dev);
ret = 0;
break;
@@ -336,7 +337,7 @@ vdpa_find_device(const struct rte_vdpa_device *start, rte_vdpa_cmp_t cmp,
rte_spinlock_lock(&vdpa_device_list_lock);
if (start == NULL)
- dev = TAILQ_FIRST(&vdpa_device_list);
+ dev = TAILQ_FIRST(vdpa_device_list);
else
dev = TAILQ_NEXT(start, next);
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v5 9/9] vhost: enable lock check
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
` (7 preceding siblings ...)
2023-02-01 11:14 ` [PATCH v5 8/9] vhost: annotate vDPA device list accesses David Marchand
@ 2023-02-01 11:14 ` David Marchand
8 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-01 11:14 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
Now that all locks in this library are annotated, we can enable the
check.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/meson.build | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
index bc7272053b..197a51d936 100644
--- a/lib/vhost/meson.build
+++ b/lib/vhost/meson.build
@@ -17,6 +17,8 @@ elif (toolchain == 'icc' and cc.version().version_compare('>=16.0.0'))
endif
dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY', cc.has_header('linux/userfaultfd.h'))
cflags += '-fno-strict-aliasing'
+
+annotate_locks = true
sources = files(
'fd_man.c',
'iotlb.c',
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock
2023-02-01 11:14 ` [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
@ 2023-02-01 12:32 ` David Marchand
2023-02-06 1:01 ` Tu, Lijuan
0 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-01 12:32 UTC (permalink / raw)
To: Chen, Zhaoyan, Tu, Lijuan, sys_stv
Cc: dev, maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
Hello Intel CI team,
On Wed, Feb 1, 2023 at 12:16 PM David Marchand
<david.marchand@redhat.com> wrote:
[snip]
> Note:
> Doxygen preprocessor does not understand trailing function attributes
> (this can be observed with the rte_seqlock.h header).
> One would think that expanding the annotation macros to a noop in
> rte_lock_annotations.h would be enough (since RTE_ANNOTATE_LOCKS is not
> set during doxygen processing)). Unfortunately, the use of
> EXPAND_ONLY_PREDEF defeats this.
>
> Removing EXPAND_ONLY_PREDEF entirely is not an option as it would expand
> all other DPDK macros.
>
> The chosen solution is to expand the annotation macros explicitly to a
> noop in PREDEFINED.
>
[snip]
> ---
> Changes since v4:
> - hid annotations from Doxygen,
> - fixed typos,
[snip]
> diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
> index f0886c3bd1..e859426099 100644
> --- a/doc/api/doxy-api.conf.in
> +++ b/doc/api/doxy-api.conf.in
> @@ -84,6 +84,17 @@ FILE_PATTERNS = rte_*.h \
> PREDEFINED = __DOXYGEN__ \
> RTE_HAS_CPUSET \
> VFIO_PRESENT \
> + __rte_lockable= \
> + __rte_guarded_by(x)= \
> + __rte_exclusive_locks_required(x)= \
> + __rte_exclusive_lock_function(x)= \
> + __rte_exclusive_trylock_function(x)= \
> + __rte_assert_exclusive_lock(x)= \
> + __rte_shared_locks_required(x)= \
> + __rte_shared_lock_function(x)= \
> + __rte_shared_trylock_function(x)= \
> + __rte_assert_shared_lock(x)= \
> + __rte_unlock_function(x)= \
> __attribute__(x)=
>
> OPTIMIZE_OUTPUT_FOR_C = YES
I received this report:
http://mails.dpdk.org/archives/test-report/2023-February/345705.html
FAILED: doc/api/html
/usr/bin/python3 ../doc/api/generate_doxygen.py doc/api/html
/usr/bin/doxygen doc/api/doxy-api.conf
/root/UB2204-64_K5.15.0_GCC11.3.0/x86_64-native-linuxapp-doc/26733/dpdk/lib/eal/include/rte_seqlock.h:218:
error: Found ')' without opening '(' for trailing return type ' ->
lock)...' (warning treated as error, aborting now)
Traceback (most recent call last):
File "/root/UB2204-64_K5.15.0_GCC11.3.0/x86_64-native-linuxapp-doc/26733/dpdk/x86_64-native-linuxapp-doc/../doc/api/generate_doxygen.py",
line 13, in <module>
subprocess.run(doxygen_command, check=True, stdout=out)
File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/doxygen',
'doc/api/doxy-api.conf']' returned non-zero exit status 1.
This error is what the patch 1 in this series (with the
doc/api/doxy-api.conf.in update) tries to avoid.
I tested my series in a fresh Ubuntu 22.04 container and I can't
reproduce this error.
Please share how this test was run in your lab so I can replicate.
Thanks.
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock
2023-02-01 12:32 ` David Marchand
@ 2023-02-06 1:01 ` Tu, Lijuan
2023-02-06 8:12 ` David Marchand
0 siblings, 1 reply; 110+ messages in thread
From: Tu, Lijuan @ 2023-02-06 1:01 UTC (permalink / raw)
To: David Marchand, Chen, Zhaoyan, sys_stv
Cc: dev, Coquelin, Maxime, stephen, Xia, Chenbo, Hu, Jiayu, Wang,
YuanX, Ding, Xuan, mb, Burakov, Anatoly, mattias.ronnblom,
David Christensen, Richardson, Bruce, Konstantin Ananyev
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday, February 1, 2023 8:33 PM
> To: Chen, Zhaoyan <zhaoyan.chen@intel.com>; Tu, Lijuan
> <lijuan.tu@intel.com>; sys_stv <sys_stv@intel.com>
> Cc: dev@dpdk.org; Coquelin, Maxime <maxime.coquelin@redhat.com>;
> stephen@networkplumber.org; Xia, Chenbo <chenbo.xia@intel.com>; Hu, Jiayu
> <jiayu.hu@intel.com>; Wang, YuanX <yuanx.wang@intel.com>; Ding, Xuan
> <xuan.ding@intel.com>; mb@smartsharesystems.com; Burakov, Anatoly
> <anatoly.burakov@intel.com>; mattias.ronnblom
> <mattias.ronnblom@ericsson.com>; David Christensen
> <drc@linux.vnet.ibm.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>
> Subject: Re: [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock
>
> Hello Intel CI team,
>
> On Wed, Feb 1, 2023 at 12:16 PM David Marchand
> <david.marchand@redhat.com> wrote:
>
> [snip]
>
> > Note:
> > Doxygen preprocessor does not understand trailing function attributes
> > (this can be observed with the rte_seqlock.h header).
> > One would think that expanding the annotation macros to a noop in
> > rte_lock_annotations.h would be enough (since RTE_ANNOTATE_LOCKS is
> > not set during doxygen processing)). Unfortunately, the use of
> > EXPAND_ONLY_PREDEF defeats this.
> >
> > Removing EXPAND_ONLY_PREDEF entirely is not an option as it would
> > expand all other DPDK macros.
> >
> > The chosen solution is to expand the annotation macros explicitly to a
> > noop in PREDEFINED.
> >
>
> [snip]
>
> > ---
> > Changes since v4:
> > - hid annotations from Doxygen,
> > - fixed typos,
>
> [snip]
>
> > diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in index
> > f0886c3bd1..e859426099 100644
> > --- a/doc/api/doxy-api.conf.in
> > +++ b/doc/api/doxy-api.conf.in
> > @@ -84,6 +84,17 @@ FILE_PATTERNS = rte_*.h \
> > PREDEFINED = __DOXYGEN__ \
> > RTE_HAS_CPUSET \
> > VFIO_PRESENT \
> > + __rte_lockable= \
> > + __rte_guarded_by(x)= \
> > + __rte_exclusive_locks_required(x)= \
> > + __rte_exclusive_lock_function(x)= \
> > + __rte_exclusive_trylock_function(x)= \
> > + __rte_assert_exclusive_lock(x)= \
> > + __rte_shared_locks_required(x)= \
> > + __rte_shared_lock_function(x)= \
> > + __rte_shared_trylock_function(x)= \
> > + __rte_assert_shared_lock(x)= \
> > + __rte_unlock_function(x)= \
> > __attribute__(x)=
> >
> > OPTIMIZE_OUTPUT_FOR_C = YES
>
> I received this report:
> http://mails.dpdk.org/archives/test-report/2023-February/345705.html
>
> FAILED: doc/api/html
> /usr/bin/python3 ../doc/api/generate_doxygen.py doc/api/html
> /usr/bin/doxygen doc/api/doxy-api.conf
> /root/UB2204-64_K5.15.0_GCC11.3.0/x86_64-native-linuxapp-
> doc/26733/dpdk/lib/eal/include/rte_seqlock.h:218:
> error: Found ')' without opening '(' for trailing return type ' -> lock)...' (warning
> treated as error, aborting now) Traceback (most recent call last):
> File "/root/UB2204-64_K5.15.0_GCC11.3.0/x86_64-native-linuxapp-
> doc/26733/dpdk/x86_64-native-linuxapp-doc/../doc/api/generate_doxygen.py",
> line 13, in <module>
> subprocess.run(doxygen_command, check=True, stdout=out)
> File "/usr/lib/python3.10/subprocess.py", line 524, in run
> raise CalledProcessError(retcode, process.args,
> subprocess.CalledProcessError: Command '['/usr/bin/doxygen', 'doc/api/doxy-
> api.conf']' returned non-zero exit status 1.
>
> This error is what the patch 1 in this series (with the doc/api/doxy-api.conf.in
> update) tries to avoid.
>
> I tested my series in a fresh Ubuntu 22.04 container and I can't reproduce this
> error.
> Please share how this test was run in your lab so I can replicate.
Hi David,
Sorry, bothered you. There is no error with this series.
The error is caused by the CI doesn't apply any changes with doc/*.
Community decided to exclude doc/*, as doc/* change frequently,
especially the release notes, cause a lot of conflict with main tree.
Here is command we used:
git pw series apply 26733 --exclude=doc/**
We will raise a discussion for this in community CI meeting.
My thought is to narrow down exclude scope to doc/guides or
doc/guides/rel_notes/*
thanks,
Lijuan
>
>
> Thanks.
>
> --
> David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock
2023-02-06 1:01 ` Tu, Lijuan
@ 2023-02-06 8:12 ` David Marchand
0 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-06 8:12 UTC (permalink / raw)
To: Tu, Lijuan
Cc: Chen, Zhaoyan, sys_stv, dev, Coquelin, Maxime, stephen, Xia,
Chenbo, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb, Burakov, Anatoly,
mattias.ronnblom, David Christensen, Richardson, Bruce,
Konstantin Ananyev, Thomas Monjalon
Hello,
On Mon, Feb 6, 2023 at 2:02 AM Tu, Lijuan <lijuan.tu@intel.com> wrote:
> > I received this report:
> > http://mails.dpdk.org/archives/test-report/2023-February/345705.html
> >
> > FAILED: doc/api/html
> > /usr/bin/python3 ../doc/api/generate_doxygen.py doc/api/html
> > /usr/bin/doxygen doc/api/doxy-api.conf
> > /root/UB2204-64_K5.15.0_GCC11.3.0/x86_64-native-linuxapp-
> > doc/26733/dpdk/lib/eal/include/rte_seqlock.h:218:
> > error: Found ')' without opening '(' for trailing return type ' -> lock)...' (warning
> > treated as error, aborting now) Traceback (most recent call last):
> > File "/root/UB2204-64_K5.15.0_GCC11.3.0/x86_64-native-linuxapp-
> > doc/26733/dpdk/x86_64-native-linuxapp-doc/../doc/api/generate_doxygen.py",
> > line 13, in <module>
> > subprocess.run(doxygen_command, check=True, stdout=out)
> > File "/usr/lib/python3.10/subprocess.py", line 524, in run
> > raise CalledProcessError(retcode, process.args,
> > subprocess.CalledProcessError: Command '['/usr/bin/doxygen', 'doc/api/doxy-
> > api.conf']' returned non-zero exit status 1.
> >
> > This error is what the patch 1 in this series (with the doc/api/doxy-api.conf.in
> > update) tries to avoid.
> >
> > I tested my series in a fresh Ubuntu 22.04 container and I can't reproduce this
> > error.
> > Please share how this test was run in your lab so I can replicate.
>
> Sorry, bothered you. There is no error with this series.
>
> The error is caused by the CI doesn't apply any changes with doc/*.
> Community decided to exclude doc/*, as doc/* change frequently,
> especially the release notes, cause a lot of conflict with main tree.
>
> Here is command we used:
> git pw series apply 26733 --exclude=doc/**
>
> We will raise a discussion for this in community CI meeting.
> My thought is to narrow down exclude scope to doc/guides or
> doc/guides/rel_notes/*
Ok, thanks for the analysis.
- I was not aware of such behavior (I probably missed it in CI minutes).
Is this something that only Intel CI does?
At least for GHA, I am sure it is not the case.
- What is the point of calling the documentation generation target if
doc/ updates are ignored?
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 0/9] Lock annotations
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
` (8 preceding siblings ...)
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-07 10:45 ` [PATCH v6 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
` (10 more replies)
9 siblings, 11 replies; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
vhost internals involves multiple locks to protect data access by
multiple threads.
This series uses clang thread safety checks [1] to catch issues during
compilation: EAL spinlock, seqlock and rwlock are annotated and vhost
code is instrumented so that clang can statically check correctness.
Those annotations are quite heavy to maintain because the full path of
code must be annotated (as can be seen in the vhost datapath code),
but I think it is worth using.
This has been tested against the whole tree and some fixes are already
flying on the mailing list (see [2] for a list).
If this first series is merged, I will prepare a followup series for EAL
and other libraries.
1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
2: https://patchwork.dpdk.org/bundle/dmarchand/lock_fixes/?state=*&archive=both
--
David Marchand
Changes since v5:
- rebased after lib/vhost updates (patches 5 and 7),
Changes since v4:
- masked annotations from Doxygen as it seems confused with some
constructs,
- fixed typos,
Changes since RFC v3:
- sorry Maxime, it has been too long since RFC v3 and the code evolved,
so I dropped all your review tags,
- rebased,
- added documentation,
- dropped/fixed annotations in arch-specific and EAL headers,
- rewrote need reply handling so that we don't have to waive the check
on the associated functions,
- separated IOTLB lock unconditional acquire from the annotation patch,
- rewrote runtime checks for "unsafe" functions using a panicking assert
helper,
Changes since RFC v2:
- fixed trylock annotations for rwlock,
- annotated _tm flavors of spinlock and rwlock,
- removed Maxime vhost fix from series (since Mimecast does not like
me sending Maxime patch...), added a dependency on original fix
as a hint for reviewers,
- renamed attributes,
Changes since RFC v1:
- Cc'd people who have pending patches for vhost,
- moved annotations to EAL and removed wrappers in vhost,
- as a result of moving to EAL, this series will be tested against
the main repo, so patch 1 has been kept as part of the series
even if already applied to next-virtio,
- refined/split patches and annotated all spinlocks in vhost,
David Marchand (9):
eal: annotate spinlock, rwlock and seqlock
vhost: simplify need reply handling
vhost: terminate when access lock is not taken
vhost: annotate virtqueue access lock
vhost: annotate async accesses
vhost: always take IOTLB lock
vhost: annotate IOTLB lock
vhost: annotate vDPA device list accesses
vhost: enable lock check
doc/api/doxy-api.conf.in | 11 ++
.../prog_guide/env_abstraction_layer.rst | 24 ++++
doc/guides/rel_notes/release_23_03.rst | 5 +
drivers/meson.build | 5 +
lib/eal/include/generic/rte_rwlock.h | 27 +++-
lib/eal/include/generic/rte_spinlock.h | 31 +++--
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 73 ++++++++++
lib/eal/include/rte_seqlock.h | 2 +
lib/eal/ppc/include/rte_spinlock.h | 3 +
lib/eal/x86/include/rte_rwlock.h | 4 +
lib/eal/x86/include/rte_spinlock.h | 9 ++
lib/meson.build | 5 +
lib/vhost/iotlb.h | 4 +
lib/vhost/meson.build | 2 +
lib/vhost/vdpa.c | 20 +--
lib/vhost/vhost.c | 38 ++---
lib/vhost/vhost.h | 34 ++++-
lib/vhost/vhost_crypto.c | 8 ++
lib/vhost/vhost_user.c | 131 ++++++++----------
lib/vhost/virtio_net.c | 118 ++++++++++++----
21 files changed, 405 insertions(+), 150 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 1/9] eal: annotate spinlock, rwlock and seqlock
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:00 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 2/9] vhost: simplify need reply handling David Marchand
` (9 subsequent siblings)
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb, Anatoly Burakov, Mattias Rönnblom,
David Christensen, Bruce Richardson, Konstantin Ananyev
clang offers some thread safety checks, statically verifying that locks
are taken and released in the code.
To use those checks, the full code leading to taking or releasing locks
must be annotated with some attributes.
Wrap those attributes into our own set of macros.
rwlock, seqlock and the "normal" spinlock are instrumented.
Those checks might be of interest out of DPDK, but it requires that the
including application locks are annotated.
On the other hand, applications out there might have been using
those same checks.
To be on the safe side, keep this instrumentation under a
RTE_ANNOTATE_LOCKS internal build flag.
A component may en/disable this check by setting
annotate_locks = true/false in its meson.build.
Note:
Doxygen preprocessor does not understand trailing function attributes
(this can be observed with the rte_seqlock.h header).
One would think that expanding the annotation macros to a noop in
rte_lock_annotations.h would be enough (since RTE_ANNOTATE_LOCKS is not
set during doxygen processing)). Unfortunately, the use of
EXPAND_ONLY_PREDEF defeats this.
Removing EXPAND_ONLY_PREDEF entirely is not an option as it would expand
all other DPDK macros.
The chosen solution is to expand the annotation macros explicitly to a
noop in PREDEFINED.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since v4:
- hid annotations from Doxygen,
- fixed typos,
Changes since RFC v3:
- rebased,
- added some documentation,
- added assert attribute,
- instrumented seqlock,
- cleaned annotations in arch-specific headers,
Changes since RFC v2:
- fixed rwlock trylock,
- instrumented _tm spinlocks,
- aligned attribute names to clang,
---
doc/api/doxy-api.conf.in | 11 +++
.../prog_guide/env_abstraction_layer.rst | 24 ++++++
doc/guides/rel_notes/release_23_03.rst | 5 ++
drivers/meson.build | 5 ++
lib/eal/include/generic/rte_rwlock.h | 27 +++++--
lib/eal/include/generic/rte_spinlock.h | 31 +++++---
lib/eal/include/meson.build | 1 +
lib/eal/include/rte_lock_annotations.h | 73 +++++++++++++++++++
lib/eal/include/rte_seqlock.h | 2 +
lib/eal/ppc/include/rte_spinlock.h | 3 +
lib/eal/x86/include/rte_rwlock.h | 4 +
lib/eal/x86/include/rte_spinlock.h | 9 +++
lib/meson.build | 5 ++
13 files changed, 186 insertions(+), 14 deletions(-)
create mode 100644 lib/eal/include/rte_lock_annotations.h
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index f0886c3bd1..e859426099 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -84,6 +84,17 @@ FILE_PATTERNS = rte_*.h \
PREDEFINED = __DOXYGEN__ \
RTE_HAS_CPUSET \
VFIO_PRESENT \
+ __rte_lockable= \
+ __rte_guarded_by(x)= \
+ __rte_exclusive_locks_required(x)= \
+ __rte_exclusive_lock_function(x)= \
+ __rte_exclusive_trylock_function(x)= \
+ __rte_assert_exclusive_lock(x)= \
+ __rte_shared_locks_required(x)= \
+ __rte_shared_lock_function(x)= \
+ __rte_shared_trylock_function(x)= \
+ __rte_assert_shared_lock(x)= \
+ __rte_unlock_function(x)= \
__attribute__(x)=
OPTIMIZE_OUTPUT_FOR_C = YES
diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst
index 35fbebe1be..3f33621e05 100644
--- a/doc/guides/prog_guide/env_abstraction_layer.rst
+++ b/doc/guides/prog_guide/env_abstraction_layer.rst
@@ -529,6 +529,30 @@ Misc Functions
Locks and atomic operations are per-architecture (i686 and x86_64).
+Lock annotations
+~~~~~~~~~~~~~~~~
+
+R/W locks, seq locks and spinlocks have been instrumented to help developers in
+catching issues in DPDK.
+
+This instrumentation relies on
+`clang Thread Safety checks <https://clang.llvm.org/docs/ThreadSafetyAnalysis.html>`_.
+All attributes are prefixed with __rte and are fully described in the clang
+documentation.
+
+Some general comments:
+
+- it is important that lock requirements are expressed at the function
+ declaration level in headers so that other code units can be inspected,
+- when some global lock is necessary to some user-exposed API, it is preferred
+ to expose it via an internal helper rather than expose the global variable,
+- there are a list of known limitations with clang instrumentation, but before
+ waiving checks with ``__rte_no_thread_safety_analysis`` in your code, please
+ discuss it on the mailing list,
+
+A DPDK library/driver can enable/disable the checks by setting
+``annotate_locks`` accordingly in its ``meson.build`` file.
+
IOVA Mode Detection
~~~~~~~~~~~~~~~~~~~
diff --git a/doc/guides/rel_notes/release_23_03.rst b/doc/guides/rel_notes/release_23_03.rst
index 1fa101c420..bf90feba68 100644
--- a/doc/guides/rel_notes/release_23_03.rst
+++ b/doc/guides/rel_notes/release_23_03.rst
@@ -55,6 +55,11 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+* **Introduced lock annotations.**
+
+ Added lock annotations attributes so that clang can statically analyze lock
+ correctness.
+
* **Updated AMD axgbe driver.**
* Added multi-process support.
diff --git a/drivers/meson.build b/drivers/meson.build
index c6d619200f..bddc4a6cc4 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -91,6 +91,7 @@ foreach subpath:subdirs
build = true # set to false to disable, e.g. missing deps
reason = '<unknown reason>' # set if build == false to explain
name = drv
+ annotate_locks = false
sources = []
headers = []
driver_sdk_headers = [] # public headers included by drivers
@@ -167,6 +168,10 @@ foreach subpath:subdirs
enabled_drivers += name
lib_name = '_'.join(['rte', class, name])
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=' + '.'.join([log_prefix, name])
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
dpdk_conf.set(lib_name.to_upper(), 1)
dpdk_extra_ldflags += pkgconfig_extra_libs
diff --git a/lib/eal/include/generic/rte_rwlock.h b/lib/eal/include/generic/rte_rwlock.h
index 233d4262be..d45c22c189 100644
--- a/lib/eal/include/generic/rte_rwlock.h
+++ b/lib/eal/include/generic/rte_rwlock.h
@@ -30,6 +30,7 @@ extern "C" {
#include <rte_branch_prediction.h>
#include <rte_common.h>
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
@@ -55,7 +56,7 @@ extern "C" {
/* Writer is waiting or has lock */
#define RTE_RWLOCK_READ 0x4 /* Reader increment */
-typedef struct {
+typedef struct __rte_lockable {
int32_t cnt;
} rte_rwlock_t;
@@ -84,6 +85,8 @@ rte_rwlock_init(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_read_lock(rte_rwlock_t *rwl)
+ __rte_shared_lock_function(rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -119,6 +122,8 @@ rte_rwlock_read_lock(rte_rwlock_t *rwl)
*/
static inline int
rte_rwlock_read_trylock(rte_rwlock_t *rwl)
+ __rte_shared_trylock_function(0, rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -150,6 +155,8 @@ rte_rwlock_read_trylock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_read_unlock(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl)
+ __rte_no_thread_safety_analysis
{
__atomic_fetch_sub(&rwl->cnt, RTE_RWLOCK_READ, __ATOMIC_RELEASE);
}
@@ -166,6 +173,8 @@ rte_rwlock_read_unlock(rte_rwlock_t *rwl)
*/
static inline int
rte_rwlock_write_trylock(rte_rwlock_t *rwl)
+ __rte_exclusive_trylock_function(0, rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -186,6 +195,8 @@ rte_rwlock_write_trylock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_write_lock(rte_rwlock_t *rwl)
+ __rte_exclusive_lock_function(rwl)
+ __rte_no_thread_safety_analysis
{
int32_t x;
@@ -219,6 +230,8 @@ rte_rwlock_write_lock(rte_rwlock_t *rwl)
*/
static inline void
rte_rwlock_write_unlock(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl)
+ __rte_no_thread_safety_analysis
{
__atomic_fetch_sub(&rwl->cnt, RTE_RWLOCK_WRITE, __ATOMIC_RELEASE);
}
@@ -237,7 +250,8 @@ rte_rwlock_write_unlock(rte_rwlock_t *rwl)
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_read_lock_tm(rte_rwlock_t *rwl);
+rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_shared_lock_function(rwl);
/**
* Commit hardware memory transaction or release the read lock if the lock is used as a fall-back
@@ -246,7 +260,8 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl);
* A pointer to the rwlock structure.
*/
static inline void
-rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl);
+rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl);
/**
* Try to execute critical section in a hardware memory transaction, if it
@@ -262,7 +277,8 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl);
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_lock_tm(rte_rwlock_t *rwl);
+rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_exclusive_lock_function(rwl);
/**
* Commit hardware memory transaction or release the write lock if the lock is used as a fall-back
@@ -271,7 +287,8 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl);
* A pointer to a rwlock structure.
*/
static inline void
-rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl);
+rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_unlock_function(rwl);
#ifdef __cplusplus
}
diff --git a/lib/eal/include/generic/rte_spinlock.h b/lib/eal/include/generic/rte_spinlock.h
index 73ed4bfbdc..8ca47bbfaa 100644
--- a/lib/eal/include/generic/rte_spinlock.h
+++ b/lib/eal/include/generic/rte_spinlock.h
@@ -22,12 +22,13 @@
#ifdef RTE_FORCE_INTRINSICS
#include <rte_common.h>
#endif
+#include <rte_lock_annotations.h>
#include <rte_pause.h>
/**
* The rte_spinlock_t type.
*/
-typedef struct {
+typedef struct __rte_lockable {
volatile int locked; /**< lock status 0 = unlocked, 1 = locked */
} rte_spinlock_t;
@@ -55,11 +56,13 @@ rte_spinlock_init(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock(rte_spinlock_t *sl);
+rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_exclusive_lock_function(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int exp = 0;
@@ -79,11 +82,13 @@ rte_spinlock_lock(rte_spinlock_t *sl)
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl);
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_unlock_function(sl);
#ifdef RTE_FORCE_INTRINSICS
static inline void
-rte_spinlock_unlock (rte_spinlock_t *sl)
+rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
__atomic_store_n(&sl->locked, 0, __ATOMIC_RELEASE);
}
@@ -99,11 +104,13 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
*/
__rte_warn_unused_result
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl);
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_exclusive_trylock_function(1, sl);
#ifdef RTE_FORCE_INTRINSICS
static inline int
-rte_spinlock_trylock (rte_spinlock_t *sl)
+rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int exp = 0;
return __atomic_compare_exchange_n(&sl->locked, &exp, 1,
@@ -147,7 +154,8 @@ static inline int rte_tm_supported(void);
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_lock_tm(rte_spinlock_t *sl);
+rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_exclusive_lock_function(sl);
/**
* Commit hardware memory transaction or release the spinlock if
@@ -157,7 +165,8 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl);
* A pointer to the spinlock.
*/
static inline void
-rte_spinlock_unlock_tm(rte_spinlock_t *sl);
+rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_unlock_function(sl);
/**
* Try to execute critical section in a hardware memory transaction,
@@ -177,7 +186,8 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl);
*/
__rte_warn_unused_result
static inline int
-rte_spinlock_trylock_tm(rte_spinlock_t *sl);
+rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_exclusive_trylock_function(1, sl);
/**
* The rte_spinlock_recursive_t type.
@@ -213,6 +223,7 @@ static inline void rte_spinlock_recursive_init(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
int id = rte_gettid();
@@ -229,6 +240,7 @@ static inline void rte_spinlock_recursive_lock(rte_spinlock_recursive_t *slr)
* A pointer to the recursive spinlock.
*/
static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (--(slr->count) == 0) {
slr->user = -1;
@@ -247,6 +259,7 @@ static inline void rte_spinlock_recursive_unlock(rte_spinlock_recursive_t *slr)
*/
__rte_warn_unused_result
static inline int rte_spinlock_recursive_trylock(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
int id = rte_gettid();
diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build
index cfcd40aaed..b0db9b3b3a 100644
--- a/lib/eal/include/meson.build
+++ b/lib/eal/include/meson.build
@@ -27,6 +27,7 @@ headers += files(
'rte_keepalive.h',
'rte_launch.h',
'rte_lcore.h',
+ 'rte_lock_annotations.h',
'rte_log.h',
'rte_malloc.h',
'rte_mcslock.h',
diff --git a/lib/eal/include/rte_lock_annotations.h b/lib/eal/include/rte_lock_annotations.h
new file mode 100644
index 0000000000..9fc50082d6
--- /dev/null
+++ b/lib/eal/include/rte_lock_annotations.h
@@ -0,0 +1,73 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Red Hat, Inc.
+ */
+
+#ifndef RTE_LOCK_ANNOTATIONS_H
+#define RTE_LOCK_ANNOTATIONS_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#ifdef RTE_ANNOTATE_LOCKS
+
+#define __rte_lockable \
+ __attribute__((lockable))
+
+#define __rte_guarded_by(...) \
+ __attribute__((guarded_by(__VA_ARGS__)))
+#define __rte_guarded_var \
+ __attribute__((guarded_var))
+
+#define __rte_exclusive_locks_required(...) \
+ __attribute__((exclusive_locks_required(__VA_ARGS__)))
+#define __rte_exclusive_lock_function(...) \
+ __attribute__((exclusive_lock_function(__VA_ARGS__)))
+#define __rte_exclusive_trylock_function(ret, ...) \
+ __attribute__((exclusive_trylock_function(ret, __VA_ARGS__)))
+#define __rte_assert_exclusive_lock(...) \
+ __attribute__((assert_exclusive_lock(__VA_ARGS__)))
+
+#define __rte_shared_locks_required(...) \
+ __attribute__((shared_locks_required(__VA_ARGS__)))
+#define __rte_shared_lock_function(...) \
+ __attribute__((shared_lock_function(__VA_ARGS__)))
+#define __rte_shared_trylock_function(ret, ...) \
+ __attribute__((shared_trylock_function(ret, __VA_ARGS__)))
+#define __rte_assert_shared_lock(...) \
+ __attribute__((assert_shared_lock(__VA_ARGS__)))
+
+#define __rte_unlock_function(...) \
+ __attribute__((unlock_function(__VA_ARGS__)))
+
+#define __rte_no_thread_safety_analysis \
+ __attribute__((no_thread_safety_analysis))
+
+#else /* ! RTE_ANNOTATE_LOCKS */
+
+#define __rte_lockable
+
+#define __rte_guarded_by(...)
+#define __rte_guarded_var
+
+#define __rte_exclusive_locks_required(...)
+#define __rte_exclusive_lock_function(...)
+#define __rte_exclusive_trylock_function(...)
+#define __rte_assert_exclusive_lock(...)
+
+#define __rte_shared_locks_required(...)
+#define __rte_shared_lock_function(...)
+#define __rte_shared_trylock_function(...)
+#define __rte_assert_shared_lock(...)
+
+#define __rte_unlock_function(...)
+
+#define __rte_no_thread_safety_analysis
+
+#endif /* RTE_ANNOTATE_LOCKS */
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RTE_LOCK_ANNOTATIONS_H */
diff --git a/lib/eal/include/rte_seqlock.h b/lib/eal/include/rte_seqlock.h
index 1663af62e8..fcbb9c5866 100644
--- a/lib/eal/include/rte_seqlock.h
+++ b/lib/eal/include/rte_seqlock.h
@@ -215,6 +215,7 @@ rte_seqlock_read_retry(const rte_seqlock_t *seqlock, uint32_t begin_sn)
__rte_experimental
static inline void
rte_seqlock_write_lock(rte_seqlock_t *seqlock)
+ __rte_exclusive_lock_function(&seqlock->lock)
{
/* To synchronize with other writers. */
rte_spinlock_lock(&seqlock->lock);
@@ -240,6 +241,7 @@ rte_seqlock_write_lock(rte_seqlock_t *seqlock)
__rte_experimental
static inline void
rte_seqlock_write_unlock(rte_seqlock_t *seqlock)
+ __rte_unlock_function(&seqlock->lock)
{
rte_seqcount_write_end(&seqlock->count);
diff --git a/lib/eal/ppc/include/rte_spinlock.h b/lib/eal/ppc/include/rte_spinlock.h
index 149ec245c7..3a4c905b22 100644
--- a/lib/eal/ppc/include/rte_spinlock.h
+++ b/lib/eal/ppc/include/rte_spinlock.h
@@ -20,6 +20,7 @@ extern "C" {
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
while (__sync_lock_test_and_set(&sl->locked, 1))
while (sl->locked)
@@ -28,12 +29,14 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
__sync_lock_release(&sl->locked);
}
static inline int
rte_spinlock_trylock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
return __sync_lock_test_and_set(&sl->locked, 1) == 0;
}
diff --git a/lib/eal/x86/include/rte_rwlock.h b/lib/eal/x86/include/rte_rwlock.h
index eec4c7123c..1796b69265 100644
--- a/lib/eal/x86/include/rte_rwlock.h
+++ b/lib/eal/x86/include/rte_rwlock.h
@@ -14,6 +14,7 @@ extern "C" {
static inline void
rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -22,6 +23,7 @@ rte_rwlock_read_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(rwl->cnt))
rte_rwlock_read_unlock(rwl);
@@ -31,6 +33,7 @@ rte_rwlock_read_unlock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&rwl->cnt)))
return;
@@ -39,6 +42,7 @@ rte_rwlock_write_lock_tm(rte_rwlock_t *rwl)
static inline void
rte_rwlock_write_unlock_tm(rte_rwlock_t *rwl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(rwl->cnt))
rte_rwlock_write_unlock(rwl);
diff --git a/lib/eal/x86/include/rte_spinlock.h b/lib/eal/x86/include/rte_spinlock.h
index e2e2b2643c..0b20ddfd73 100644
--- a/lib/eal/x86/include/rte_spinlock.h
+++ b/lib/eal/x86/include/rte_spinlock.h
@@ -23,6 +23,7 @@ extern "C" {
#ifndef RTE_FORCE_INTRINSICS
static inline void
rte_spinlock_lock(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int lock_val = 1;
asm volatile (
@@ -43,6 +44,7 @@ rte_spinlock_lock(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock (rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int unlock_val = 0;
asm volatile (
@@ -54,6 +56,7 @@ rte_spinlock_unlock (rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock (rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
int lockval = 1;
@@ -121,6 +124,7 @@ rte_try_tm(volatile int *lock)
static inline void
rte_spinlock_lock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&sl->locked)))
return;
@@ -130,6 +134,7 @@ rte_spinlock_lock_tm(rte_spinlock_t *sl)
static inline int
rte_spinlock_trylock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&sl->locked)))
return 1;
@@ -139,6 +144,7 @@ rte_spinlock_trylock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_unlock_tm(rte_spinlock_t *sl)
+ __rte_no_thread_safety_analysis
{
if (unlikely(sl->locked))
rte_spinlock_unlock(sl);
@@ -148,6 +154,7 @@ rte_spinlock_unlock_tm(rte_spinlock_t *sl)
static inline void
rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&slr->sl.locked)))
return;
@@ -157,6 +164,7 @@ rte_spinlock_recursive_lock_tm(rte_spinlock_recursive_t *slr)
static inline void
rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (unlikely(slr->sl.locked))
rte_spinlock_recursive_unlock(slr);
@@ -166,6 +174,7 @@ rte_spinlock_recursive_unlock_tm(rte_spinlock_recursive_t *slr)
static inline int
rte_spinlock_recursive_trylock_tm(rte_spinlock_recursive_t *slr)
+ __rte_no_thread_safety_analysis
{
if (likely(rte_try_tm(&slr->sl.locked)))
return 1;
diff --git a/lib/meson.build b/lib/meson.build
index a90fee31b7..450c061d2b 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -120,6 +120,7 @@ foreach l:libraries
reason = '<unknown reason>' # set if build == false to explain why
name = l
use_function_versioning = false
+ annotate_locks = false
sources = []
headers = []
indirect_headers = [] # public headers not directly included by apps
@@ -202,6 +203,10 @@ foreach l:libraries
cflags += '-DRTE_USE_FUNCTION_VERSIONING'
endif
cflags += '-DRTE_LOG_DEFAULT_LOGTYPE=lib.' + l
+ if annotate_locks and cc.has_argument('-Wthread-safety')
+ cflags += '-DRTE_ANNOTATE_LOCKS'
+ cflags += '-Wthread-safety'
+ endif
# first build static lib
static_lib = static_library(libname,
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 2/9] vhost: simplify need reply handling
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
2023-02-07 10:45 ` [PATCH v6 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:00 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 3/9] vhost: terminate when access lock is not taken David Marchand
` (8 subsequent siblings)
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
Dedicate send_vhost_slave_message() helper to the case when no reply is
needed.
Add a send_vhost_slave_message_process_reply() helper for the opposite.
This new helper merges both send_vhost_slave_message() and the code
previously in process_slave_message_reply().
The slave_req_lock lock is then only handled in this helper which will
make lock checks trivial.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost_user.c | 119 ++++++++++++++++++-----------------------
1 file changed, 51 insertions(+), 68 deletions(-)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 8f33d5f4d9..60ec1bf5f6 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2878,18 +2878,46 @@ send_vhost_reply(struct virtio_net *dev, int sockfd, struct vhu_msg_context *ctx
}
static int
-send_vhost_slave_message(struct virtio_net *dev,
- struct vhu_msg_context *ctx)
+send_vhost_slave_message(struct virtio_net *dev, struct vhu_msg_context *ctx)
+{
+ return send_vhost_message(dev, dev->slave_req_fd, ctx);
+}
+
+static int
+send_vhost_slave_message_process_reply(struct virtio_net *dev, struct vhu_msg_context *ctx)
{
+ struct vhu_msg_context msg_reply;
int ret;
- if (ctx->msg.flags & VHOST_USER_NEED_REPLY)
- rte_spinlock_lock(&dev->slave_req_lock);
+ rte_spinlock_lock(&dev->slave_req_lock);
+ ret = send_vhost_slave_message(dev, ctx);
+ if (ret < 0) {
+ VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
+ goto out;
+ }
- ret = send_vhost_message(dev, dev->slave_req_fd, ctx);
- if (ret < 0 && (ctx->msg.flags & VHOST_USER_NEED_REPLY))
- rte_spinlock_unlock(&dev->slave_req_lock);
+ ret = read_vhost_message(dev, dev->slave_req_fd, &msg_reply);
+ if (ret <= 0) {
+ if (ret < 0)
+ VHOST_LOG_CONFIG(dev->ifname, ERR,
+ "vhost read slave message reply failed\n");
+ else
+ VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer closed\n");
+ ret = -1;
+ goto out;
+ }
+
+ if (msg_reply.msg.request.slave != ctx->msg.request.slave) {
+ VHOST_LOG_CONFIG(dev->ifname, ERR,
+ "received unexpected msg type (%u), expected %u\n",
+ msg_reply.msg.request.slave, ctx->msg.request.slave);
+ ret = -1;
+ goto out;
+ }
+ ret = msg_reply.msg.payload.u64 ? -1 : 0;
+out:
+ rte_spinlock_unlock(&dev->slave_req_lock);
return ret;
}
@@ -3213,42 +3241,6 @@ vhost_user_msg_handler(int vid, int fd)
return ret;
}
-static int process_slave_message_reply(struct virtio_net *dev,
- const struct vhu_msg_context *ctx)
-{
- struct vhu_msg_context msg_reply;
- int ret;
-
- if ((ctx->msg.flags & VHOST_USER_NEED_REPLY) == 0)
- return 0;
-
- ret = read_vhost_message(dev, dev->slave_req_fd, &msg_reply);
- if (ret <= 0) {
- if (ret < 0)
- VHOST_LOG_CONFIG(dev->ifname, ERR,
- "vhost read slave message reply failed\n");
- else
- VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer closed\n");
- ret = -1;
- goto out;
- }
-
- ret = 0;
- if (msg_reply.msg.request.slave != ctx->msg.request.slave) {
- VHOST_LOG_CONFIG(dev->ifname, ERR,
- "received unexpected msg type (%u), expected %u\n",
- msg_reply.msg.request.slave, ctx->msg.request.slave);
- ret = -1;
- goto out;
- }
-
- ret = msg_reply.msg.payload.u64 ? -1 : 0;
-
-out:
- rte_spinlock_unlock(&dev->slave_req_lock);
- return ret;
-}
-
int
vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm)
{
@@ -3277,10 +3269,9 @@ vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm)
return 0;
}
-static int
-vhost_user_slave_config_change(struct virtio_net *dev, bool need_reply)
+int
+rte_vhost_slave_config_change(int vid, bool need_reply)
{
- int ret;
struct vhu_msg_context ctx = {
.msg = {
.request.slave = VHOST_USER_SLAVE_CONFIG_CHANGE_MSG,
@@ -3288,29 +3279,23 @@ vhost_user_slave_config_change(struct virtio_net *dev, bool need_reply)
.size = 0,
}
};
-
- if (need_reply)
- ctx.msg.flags |= VHOST_USER_NEED_REPLY;
-
- ret = send_vhost_slave_message(dev, &ctx);
- if (ret < 0) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
- return ret;
- }
-
- return process_slave_message_reply(dev, &ctx);
-}
-
-int
-rte_vhost_slave_config_change(int vid, bool need_reply)
-{
struct virtio_net *dev;
+ int ret;
dev = get_device(vid);
if (!dev)
return -ENODEV;
- return vhost_user_slave_config_change(dev, need_reply);
+ if (!need_reply) {
+ ret = send_vhost_slave_message(dev, &ctx);
+ } else {
+ ctx.msg.flags |= VHOST_USER_NEED_REPLY;
+ ret = send_vhost_slave_message_process_reply(dev, &ctx);
+ }
+
+ if (ret < 0)
+ VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config change (%d)\n", ret);
+ return ret;
}
static int vhost_user_slave_set_vring_host_notifier(struct virtio_net *dev,
@@ -3339,13 +3324,11 @@ static int vhost_user_slave_set_vring_host_notifier(struct virtio_net *dev,
ctx.fd_num = 1;
}
- ret = send_vhost_slave_message(dev, &ctx);
- if (ret < 0) {
+ ret = send_vhost_slave_message_process_reply(dev, &ctx);
+ if (ret < 0)
VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to set host notifier (%d)\n", ret);
- return ret;
- }
- return process_slave_message_reply(dev, &ctx);
+ return ret;
}
int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable)
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 3/9] vhost: terminate when access lock is not taken
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
2023-02-07 10:45 ` [PATCH v6 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
2023-02-07 10:45 ` [PATCH v6 2/9] vhost: simplify need reply handling David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:01 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 4/9] vhost: annotate virtqueue access lock David Marchand
` (7 subsequent siblings)
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
Be a bit more strict when a programmatic error is detected with regards to
the access_lock not being taken.
Mark the new helper with __rte_assert_exclusive_lock so that clang
understands where locks are expected to be taken.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost.c | 18 +++---------------
lib/vhost/vhost.h | 10 ++++++++++
lib/vhost/virtio_net.c | 6 +-----
3 files changed, 14 insertions(+), 20 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 19c7b92c32..8cd727ca2f 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -1781,11 +1781,7 @@ rte_vhost_async_channel_register_thread_unsafe(int vid, uint16_t queue_id)
if (unlikely(vq == NULL || !dev->async_copy))
return -1;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
return async_channel_register(dev, vq);
}
@@ -1847,11 +1843,7 @@ rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t queue_id)
if (vq == NULL)
return -1;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (!vq->async)
return 0;
@@ -1994,11 +1986,7 @@ rte_vhost_async_get_inflight_thread_unsafe(int vid, uint16_t queue_id)
if (vq == NULL)
return ret;
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (!vq->async)
return ret;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 1f913803f6..82fe9b5fda 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -513,6 +513,16 @@ struct virtio_net {
struct rte_vhost_user_extern_ops extern_ops;
} __rte_cache_aligned;
+static inline void
+vq_assert_lock__(struct virtio_net *dev, struct vhost_virtqueue *vq, const char *func)
+ __rte_assert_exclusive_lock(&vq->access_lock)
+{
+ if (unlikely(!rte_spinlock_is_locked(&vq->access_lock)))
+ rte_panic("VHOST_CONFIG: (%s) %s() called without access lock taken.\n",
+ dev->ifname, func);
+}
+#define vq_assert_lock(dev, vq) vq_assert_lock__(dev, vq, __func__)
+
static __rte_always_inline bool
vq_is_packed(struct virtio_net *dev)
{
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index aac7aa9d01..cc9675ebe5 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -2367,11 +2367,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid, uint16_t queue_id,
vq = dev->virtqueue[queue_id];
- if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
- VHOST_LOG_DATA(dev->ifname, ERR, "%s() called without access lock taken.\n",
- __func__);
- return -1;
- }
+ vq_assert_lock(dev, vq);
if (unlikely(!vq->async)) {
VHOST_LOG_DATA(dev->ifname, ERR,
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 4/9] vhost: annotate virtqueue access lock
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
` (2 preceding siblings ...)
2023-02-07 10:45 ` [PATCH v6 3/9] vhost: terminate when access lock is not taken David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:01 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 5/9] vhost: annotate async accesses David Marchand
` (6 subsequent siblings)
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
vhost_user_lock/unlock_all_queue_pairs must be waived since clang
annotations can't express taking a runtime number of locks.
vhost_queue_stats_update() requirement can be expressed with a required
tag.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since RFC v3:
- removed annotations needed for vhost async which went to the next
patch,
---
lib/vhost/vhost_user.c | 2 ++
lib/vhost/virtio_net.c | 4 +---
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 60ec1bf5f6..70d221b9f6 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2965,6 +2965,7 @@ vhost_user_check_and_alloc_queue_pair(struct virtio_net *dev,
static void
vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
+ __rte_no_thread_safety_analysis
{
unsigned int i = 0;
unsigned int vq_num = 0;
@@ -2982,6 +2983,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
static void
vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
+ __rte_no_thread_safety_analysis
{
unsigned int i = 0;
unsigned int vq_num = 0;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index cc9675ebe5..f2ab6dba15 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -52,12 +52,10 @@ is_valid_virt_queue_idx(uint32_t idx, int is_tx, uint32_t nr_vring)
return (is_tx ^ (idx & 1)) == 0 && idx < nr_vring;
}
-/*
- * This function must be called with virtqueue's access_lock taken.
- */
static inline void
vhost_queue_stats_update(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct virtqueue_stats *stats = &vq->stats;
int i;
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 5/9] vhost: annotate async accesses
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
` (3 preceding siblings ...)
2023-02-07 10:45 ` [PATCH v6 4/9] vhost: annotate virtqueue access lock David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:01 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 6/9] vhost: always take IOTLB lock David Marchand
` (5 subsequent siblings)
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
vq->async is initialised and must be accessed under vq->access_lock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since v5:
- rebased after packed support was added to async code,
Changes since RFC v3:
- rebased,
- fixed annotations vq->access_lock -> &vq->access_lock,
- reworked free_vq,
---
lib/vhost/vhost.c | 4 ++++
lib/vhost/vhost.h | 2 +-
lib/vhost/vhost_user.c | 10 +++++++---
lib/vhost/virtio_net.c | 41 +++++++++++++++++++++++++++++++++++++++++
4 files changed, 53 insertions(+), 4 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 8cd727ca2f..8bccdd8584 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -369,6 +369,7 @@ cleanup_device(struct virtio_net *dev, int destroy)
static void
vhost_free_async_mem(struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
if (!vq->async)
return;
@@ -393,7 +394,9 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue *vq)
else
rte_free(vq->shadow_used_split);
+ rte_spinlock_lock(&vq->access_lock);
vhost_free_async_mem(vq);
+ rte_spinlock_unlock(&vq->access_lock);
rte_free(vq->batch_copy_elems);
vhost_user_iotlb_destroy(vq);
rte_free(vq->log_cache);
@@ -1669,6 +1672,7 @@ rte_vhost_extern_callback_register(int vid,
static __rte_always_inline int
async_channel_register(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async;
int node = vq->numa_node;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index 82fe9b5fda..c05313cf37 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -326,7 +326,7 @@ struct vhost_virtqueue {
struct rte_vhost_resubmit_info *resubmit_inflight;
uint64_t global_counter;
- struct vhost_async *async;
+ struct vhost_async *async __rte_guarded_var;
int notif_enable;
#define VIRTIO_UNINITIALIZED_NOTIF (-1)
diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
index 70d221b9f6..8c1d60b76b 100644
--- a/lib/vhost/vhost_user.c
+++ b/lib/vhost/vhost_user.c
@@ -2168,6 +2168,7 @@ vhost_user_set_vring_enable(struct virtio_net **pdev,
int main_fd __rte_unused)
{
struct virtio_net *dev = *pdev;
+ struct vhost_virtqueue *vq;
bool enable = !!ctx->msg.payload.state.num;
int index = (int)ctx->msg.payload.state.index;
@@ -2175,15 +2176,18 @@ vhost_user_set_vring_enable(struct virtio_net **pdev,
"set queue enable: %d to qp idx: %d\n",
enable, index);
- if (enable && dev->virtqueue[index]->async) {
- if (dev->virtqueue[index]->async->pkts_inflight_n) {
+ vq = dev->virtqueue[index];
+ /* vhost_user_lock_all_queue_pairs locked all qps */
+ vq_assert_lock(dev, vq);
+ if (enable && vq->async) {
+ if (vq->async->pkts_inflight_n) {
VHOST_LOG_CONFIG(dev->ifname, ERR,
"failed to enable vring. Inflight packets must be completed first\n");
return RTE_VHOST_MSG_RESULT_ERR;
}
}
- dev->virtqueue[index]->enabled = enable;
+ vq->enabled = enable;
return RTE_VHOST_MSG_RESULT_OK;
}
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index f2ab6dba15..6672caac49 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -102,6 +102,7 @@ static __rte_always_inline int64_t
vhost_async_dma_transfer_one(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t flag_idx,
struct vhost_iov_iter *pkt)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
uint16_t ring_mask = dma_info->ring_mask;
@@ -151,6 +152,7 @@ static __rte_always_inline uint16_t
vhost_async_dma_transfer(struct virtio_net *dev, struct vhost_virtqueue *vq,
int16_t dma_id, uint16_t vchan_id, uint16_t head_idx,
struct vhost_iov_iter *pkts, uint16_t nr_pkts)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct async_dma_vchan_info *dma_info = &dma_copy_track[dma_id].vchans[vchan_id];
int64_t ret, nr_copies = 0;
@@ -434,6 +436,7 @@ static __rte_always_inline void
vhost_async_shadow_enqueue_packed_batch(struct vhost_virtqueue *vq,
uint64_t *lens,
uint16_t *ids)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t i;
struct vhost_async *async = vq->async;
@@ -450,6 +453,7 @@ vhost_async_shadow_enqueue_packed_batch(struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_async_shadow_dequeue_packed_batch(struct vhost_virtqueue *vq, uint16_t *ids)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t i;
struct vhost_async *async = vq->async;
@@ -611,6 +615,7 @@ vhost_async_shadow_enqueue_packed(struct vhost_virtqueue *vq,
uint16_t *id,
uint16_t *count,
uint16_t num_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t i;
struct vhost_async *async = vq->async;
@@ -1118,6 +1123,7 @@ static __rte_always_inline int
async_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1195,6 +1201,7 @@ static __rte_always_inline int
mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1323,6 +1330,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkt,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1383,6 +1391,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1610,6 +1619,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1634,6 +1644,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -1733,6 +1744,7 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
static __rte_always_inline uint16_t
async_get_first_inflight_pkt_idx(struct vhost_virtqueue *vq)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
@@ -1761,6 +1773,7 @@ store_dma_desc_info_split(struct vring_used_elem *s_ring, struct vring_used_elem
static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1867,6 +1880,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs,
uint16_t *nr_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1925,6 +1939,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1947,6 +1962,7 @@ virtio_dev_rx_async_packed_batch_enqueue(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
@@ -2007,6 +2023,7 @@ virtio_dev_rx_async_packed_batch(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint64_t desc_addrs[PACKED_BATCH_SIZE];
uint64_t lens[PACKED_BATCH_SIZE];
@@ -2022,6 +2039,7 @@ virtio_dev_rx_async_packed_batch(struct virtio_net *dev,
static __rte_always_inline void
dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
uint32_t nr_err, uint32_t *pkt_idx)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t descs_err = 0;
uint16_t buffers_err = 0;
@@ -2052,6 +2070,7 @@ dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t n_xfer;
@@ -2124,6 +2143,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue
static __rte_always_inline void
write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t nr_left = n_descs;
@@ -2156,6 +2176,7 @@ write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t n_descs)
static __rte_always_inline void
write_back_completed_descs_packed(struct vhost_virtqueue *vq,
uint16_t n_buffers)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t from = async->last_buffer_idx_packed;
@@ -2220,6 +2241,7 @@ write_back_completed_descs_packed(struct vhost_virtqueue *vq,
static __rte_always_inline uint16_t
vhost_poll_enqueue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
struct async_inflight_info *pkts_info = async->pkts_info;
@@ -2824,6 +2846,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t nr_vec,
struct rte_mbuf *m, struct rte_mempool *mbuf_pool,
bool legacy_ol_flags, uint16_t slot_idx, bool is_async)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t buf_avail, buf_offset, buf_len;
uint64_t buf_addr, buf_iova;
@@ -3029,6 +3052,7 @@ static uint16_t
virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t i;
uint16_t avail_entries;
@@ -3132,6 +3156,7 @@ static uint16_t
virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3141,6 +3166,7 @@ static uint16_t
virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3341,6 +3367,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *buf_id,
uint16_t *desc_count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -3389,6 +3416,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mempool *mbuf_pool,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3419,6 +3447,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
@@ -3462,6 +3491,7 @@ static uint16_t
virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3471,6 +3501,7 @@ static uint16_t
virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3588,6 +3619,7 @@ static __rte_always_inline uint16_t
async_poll_dequeue_completed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t start_idx, from, i;
uint16_t nr_cpl_pkts = 0;
@@ -3634,6 +3666,7 @@ static __rte_always_inline uint16_t
virtio_dev_tx_async_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
static bool allocerr_warned;
bool dropped = false;
@@ -3780,6 +3813,7 @@ virtio_dev_tx_async_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3791,6 +3825,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
@@ -3799,6 +3834,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
static __rte_always_inline void
vhost_async_shadow_dequeue_single_packed(struct vhost_virtqueue *vq,
uint16_t buf_id, uint16_t count)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
struct vhost_async *async = vq->async;
uint16_t idx = async->buffer_idx_packed;
@@ -3820,6 +3856,7 @@ virtio_dev_tx_async_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkts,
uint16_t slot_idx,
bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
int err;
uint16_t buf_id, desc_count = 0;
@@ -3871,6 +3908,7 @@ virtio_dev_tx_async_packed_batch(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint16_t slot_idx,
uint16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint16_t avail_idx = vq->last_avail_idx;
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
@@ -3927,6 +3965,7 @@ static __rte_always_inline uint16_t
virtio_dev_tx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
uint32_t pkt_idx = 0;
uint16_t slot_idx = 0;
@@ -4036,6 +4075,7 @@ static uint16_t
virtio_dev_tx_async_packed_legacy(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -4046,6 +4086,7 @@ static uint16_t
virtio_dev_tx_async_packed_compliant(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
+ __rte_exclusive_locks_required(&vq->access_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 6/9] vhost: always take IOTLB lock
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
` (4 preceding siblings ...)
2023-02-07 10:45 ` [PATCH v6 5/9] vhost: annotate async accesses David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:01 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 7/9] vhost: annotate " David Marchand
` (4 subsequent siblings)
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
clang does not support conditionally held locks when statically analysing
taken locks with thread safety checks.
Always take iotlb locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/vhost.c | 8 +++-----
lib/vhost/virtio_net.c | 24 ++++++++----------------
2 files changed, 11 insertions(+), 21 deletions(-)
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 8bccdd8584..1e0c30791e 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -563,10 +563,9 @@ vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
}
void
-vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+vring_invalidate(struct virtio_net *dev __rte_unused, struct vhost_virtqueue *vq)
{
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_lock(vq);
+ vhost_user_iotlb_wr_lock(vq);
vq->access_ok = false;
vq->desc = NULL;
@@ -574,8 +573,7 @@ vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
vq->used = NULL;
vq->log_guest_addr = 0;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_wr_unlock(vq);
+ vhost_user_iotlb_wr_unlock(vq);
}
static void
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 6672caac49..49fc46e127 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -1688,8 +1688,7 @@ virtio_dev_rx(struct virtio_net *dev, struct vhost_virtqueue *vq,
if (unlikely(!vq->enabled))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -1707,8 +1706,7 @@ virtio_dev_rx(struct virtio_net *dev, struct vhost_virtqueue *vq,
vhost_queue_stats_update(dev, vq, pkts, nb_tx);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -2499,8 +2497,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, struct vhost_virtqueue *vq,
if (unlikely(!vq->enabled || !vq->async))
goto out_access_unlock;
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0))
@@ -2520,8 +2517,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev, struct vhost_virtqueue *vq,
vq->stats.inflight_submitted += nb_tx;
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -3543,8 +3539,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(!vq->access_ok))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -3603,8 +3598,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
vhost_queue_stats_update(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
@@ -4150,8 +4144,7 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
goto out_access_unlock;
}
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_lock(vq);
+ vhost_user_iotlb_rd_lock(vq);
if (unlikely(vq->access_ok == 0))
if (unlikely(vring_translate(dev, vq) < 0)) {
@@ -4215,8 +4208,7 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t queue_id,
vhost_queue_stats_update(dev, vq, pkts, count);
out:
- if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
- vhost_user_iotlb_rd_unlock(vq);
+ vhost_user_iotlb_rd_unlock(vq);
out_access_unlock:
rte_spinlock_unlock(&vq->access_lock);
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 7/9] vhost: annotate IOTLB lock
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
` (5 preceding siblings ...)
2023-02-07 10:45 ` [PATCH v6 6/9] vhost: always take IOTLB lock David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:02 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 8/9] vhost: annotate vDPA device list accesses David Marchand
` (3 subsequent siblings)
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
The starting point for this is __vhost_iova_to_vva() which requires the
lock to be taken. Annotate all the code leading to a call to it.
vdpa and vhost_crypto code are annotated but they end up not taking
a IOTLB lock and have been marked with a FIXME at the top level.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since v5:
- rebased after packed support was added to async code,
Changes since RFC v3:
- rebased,
- moved unconditionnal lock acquire in separate patch,
- fixed annotations,
---
lib/vhost/iotlb.h | 4 ++++
lib/vhost/vdpa.c | 1 +
lib/vhost/vhost.c | 8 +++-----
lib/vhost/vhost.h | 22 ++++++++++++++------
lib/vhost/vhost_crypto.c | 8 ++++++++
lib/vhost/virtio_net.c | 43 ++++++++++++++++++++++++++++++++++++++++
6 files changed, 75 insertions(+), 11 deletions(-)
diff --git a/lib/vhost/iotlb.h b/lib/vhost/iotlb.h
index e27ebebcf5..be079a8aa7 100644
--- a/lib/vhost/iotlb.h
+++ b/lib/vhost/iotlb.h
@@ -11,24 +11,28 @@
static __rte_always_inline void
vhost_user_iotlb_rd_lock(struct vhost_virtqueue *vq)
+ __rte_shared_lock_function(&vq->iotlb_lock)
{
rte_rwlock_read_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_rd_unlock(struct vhost_virtqueue *vq)
+ __rte_unlock_function(&vq->iotlb_lock)
{
rte_rwlock_read_unlock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_lock(struct vhost_virtqueue *vq)
+ __rte_exclusive_lock_function(&vq->iotlb_lock)
{
rte_rwlock_write_lock(&vq->iotlb_lock);
}
static __rte_always_inline void
vhost_user_iotlb_wr_unlock(struct vhost_virtqueue *vq)
+ __rte_unlock_function(&vq->iotlb_lock)
{
rte_rwlock_write_unlock(&vq->iotlb_lock);
}
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index 577cb00a43..a430a66970 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -146,6 +146,7 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int
rte_vdpa_relay_vring_used(int vid, uint16_t qid, void *vring_m)
+ __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct virtio_net *dev = get_device(vid);
uint16_t idx, idx_m, desc_id;
diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
index 1e0c30791e..c36dc06a0a 100644
--- a/lib/vhost/vhost.c
+++ b/lib/vhost/vhost.c
@@ -52,7 +52,6 @@ static const struct vhost_vq_stats_name_off vhost_vq_stat_strings[] = {
#define VHOST_NB_VQ_STATS RTE_DIM(vhost_vq_stat_strings)
-/* Called with iotlb_lock read-locked */
uint64_t
__vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *size, uint8_t perm)
@@ -419,6 +418,7 @@ free_device(struct virtio_net *dev)
static __rte_always_inline int
log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(vq->ring_addrs.flags & (1 << VHOST_VRING_F_LOG))))
return 0;
@@ -435,8 +435,6 @@ log_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
* Converts vring log address to GPA
* If IOMMU is enabled, the log address is IOVA
* If IOMMU not enabled, the log address is already GPA
- *
- * Caller should have iotlb_lock read-locked
*/
uint64_t
translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
@@ -467,9 +465,9 @@ translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
return log_addr;
}
-/* Caller should have iotlb_lock read-locked */
static int
vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t req_size, size;
@@ -506,9 +504,9 @@ vring_translate_split(struct virtio_net *dev, struct vhost_virtqueue *vq)
return 0;
}
-/* Caller should have iotlb_lock read-locked */
static int
vring_translate_packed(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t req_size, size;
diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
index c05313cf37..5750f0c005 100644
--- a/lib/vhost/vhost.h
+++ b/lib/vhost/vhost.h
@@ -563,12 +563,15 @@ void __vhost_log_cache_write(struct virtio_net *dev,
uint64_t addr, uint64_t len);
void __vhost_log_cache_write_iova(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void __vhost_log_cache_sync(struct virtio_net *dev,
struct vhost_virtqueue *vq);
+
void __vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len);
void __vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t len);
+ uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
static __rte_always_inline void
vhost_log_write(struct virtio_net *dev, uint64_t addr, uint64_t len)
@@ -618,6 +621,7 @@ vhost_log_used_vring(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -631,6 +635,7 @@ vhost_log_cache_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
static __rte_always_inline void
vhost_log_write_iova(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t len)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (likely(!(dev->features & (1ULL << VHOST_F_LOG_ALL))))
return;
@@ -834,18 +839,23 @@ struct rte_vhost_device_ops const *vhost_driver_callback_get(const char *path);
void vhost_backend_cleanup(struct virtio_net *dev);
uint64_t __vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t iova, uint64_t *len, uint8_t perm);
+ uint64_t iova, uint64_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void *vhost_alloc_copy_ind_table(struct virtio_net *dev,
struct vhost_virtqueue *vq,
- uint64_t desc_addr, uint64_t desc_len);
-int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq);
+ uint64_t desc_addr, uint64_t desc_len)
+ __rte_shared_locks_required(&vq->iotlb_lock);
+int vring_translate(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock);
uint64_t translate_log_addr(struct virtio_net *dev, struct vhost_virtqueue *vq,
- uint64_t log_addr);
+ uint64_t log_addr)
+ __rte_shared_locks_required(&vq->iotlb_lock);
void vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq);
static __rte_always_inline uint64_t
vhost_iova_to_vva(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t iova, uint64_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
if (!(dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)))
return rte_vhost_va_from_guest_pa(dev->mem, iova, len);
diff --git a/lib/vhost/vhost_crypto.c b/lib/vhost/vhost_crypto.c
index b448b6685d..f02bf865c3 100644
--- a/lib/vhost/vhost_crypto.c
+++ b/lib/vhost/vhost_crypto.c
@@ -490,6 +490,7 @@ static __rte_always_inline struct virtio_crypto_inhdr *
reach_inhdr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct virtio_crypto_inhdr *inhdr;
struct vhost_crypto_desc *last = head + (max_n_descs - 1);
@@ -536,6 +537,7 @@ static __rte_always_inline void *
get_data_ptr(struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *cur_desc,
uint8_t perm)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
void *data;
uint64_t dlen = cur_desc->len;
@@ -552,6 +554,7 @@ get_data_ptr(struct vhost_crypto_data_req *vc_req,
static __rte_always_inline uint32_t
copy_data_from_desc(void *dst, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *desc, uint32_t size)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
uint64_t remain;
uint64_t addr;
@@ -582,6 +585,7 @@ static __rte_always_inline int
copy_data(void *data, struct vhost_crypto_data_req *vc_req,
struct vhost_crypto_desc *head, struct vhost_crypto_desc **cur_desc,
uint32_t size, uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = *cur_desc;
uint32_t left = size;
@@ -665,6 +669,7 @@ prepare_write_back_data(struct vhost_crypto_data_req *vc_req,
uint32_t offset,
uint64_t write_back_len,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_writeback_data *wb_data, *head;
struct vhost_crypto_desc *desc = *cur_desc;
@@ -785,6 +790,7 @@ prepare_sym_cipher_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_cipher_data_req *cipher,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head;
struct vhost_crypto_writeback_data *ewb = NULL;
@@ -938,6 +944,7 @@ prepare_sym_chain_op(struct vhost_crypto *vcrypto, struct rte_crypto_op *op,
struct virtio_crypto_alg_chain_data_req *chain,
struct vhost_crypto_desc *head,
uint32_t max_n_descs)
+ __rte_shared_locks_required(&vc_req->vq->iotlb_lock)
{
struct vhost_crypto_desc *desc = head, *digest_desc;
struct vhost_crypto_writeback_data *ewb = NULL, *ewb2 = NULL;
@@ -1123,6 +1130,7 @@ vhost_crypto_process_one_req(struct vhost_crypto *vcrypto,
struct vhost_virtqueue *vq, struct rte_crypto_op *op,
struct vring_desc *head, struct vhost_crypto_desc *descs,
uint16_t desc_idx)
+ __rte_no_thread_safety_analysis /* FIXME: requires iotlb_lock? */
{
struct vhost_crypto_data_req *vc_req = rte_mbuf_to_priv(op->sym->m_src);
struct rte_cryptodev_sym_session *session;
diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
index 49fc46e127..51dc3c96d3 100644
--- a/lib/vhost/virtio_net.c
+++ b/lib/vhost/virtio_net.c
@@ -233,6 +233,7 @@ vhost_async_dma_check_completed(struct virtio_net *dev, int16_t dma_id, uint16_t
static inline void
do_data_copy_enqueue(struct virtio_net *dev, struct vhost_virtqueue *vq)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct batch_copy_elem *elem = vq->batch_copy_elems;
uint16_t count = vq->batch_copy_nb_elems;
@@ -637,6 +638,7 @@ vhost_shadow_enqueue_single_packed(struct virtio_net *dev,
uint16_t *id,
uint16_t *count,
uint16_t num_buffers)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
vhost_shadow_enqueue_packed(vq, len, id, count, num_buffers);
@@ -728,6 +730,7 @@ static __rte_always_inline int
map_one_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint64_t desc_iova, uint64_t desc_len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t vec_id = *vec_idx;
@@ -765,6 +768,7 @@ fill_vec_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint32_t avail_idx, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint16_t *desc_chain_head,
uint32_t *desc_chain_len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t idx = vq->avail->ring[avail_idx & (vq->size - 1)];
uint16_t vec_id = *vec_idx;
@@ -848,6 +852,7 @@ reserve_avail_buf_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint64_t size, struct buf_vector *buf_vec,
uint16_t *num_buffers, uint16_t avail_head,
uint16_t *nr_vec)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t cur_idx;
uint16_t vec_idx = 0;
@@ -898,6 +903,7 @@ fill_vec_buf_packed_indirect(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct vring_packed_desc *desc, uint16_t *vec_idx,
struct buf_vector *buf_vec, uint32_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t i;
uint32_t nr_descs;
@@ -956,6 +962,7 @@ fill_vec_buf_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
uint16_t avail_idx, uint16_t *desc_count,
struct buf_vector *buf_vec, uint16_t *vec_idx,
uint16_t *buf_id, uint32_t *len, uint8_t perm)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -1021,6 +1028,7 @@ static __rte_noinline void
copy_vnet_hdr_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct buf_vector *buf_vec,
struct virtio_net_hdr_mrg_rxbuf *hdr)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t len;
uint64_t remain = dev->vhost_hlen;
@@ -1124,6 +1132,7 @@ async_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct vhost_async *async = vq->async;
uint64_t mapped_len;
@@ -1164,6 +1173,7 @@ static __rte_always_inline void
sync_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, uint32_t mbuf_offset,
uint64_t buf_addr, uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct batch_copy_elem *batch_copy = vq->batch_copy_elems;
@@ -1202,6 +1212,7 @@ mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct buf_vector *buf_vec,
uint16_t nr_vec, uint16_t num_buffers, bool is_async)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t vec_idx = 0;
uint32_t mbuf_offset, mbuf_avail;
@@ -1331,6 +1342,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
struct buf_vector *buf_vec,
uint16_t *nr_descs)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1392,6 +1404,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t num_buffers;
@@ -1448,6 +1461,7 @@ virtio_dev_rx_sync_batch_check(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap_counter = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -1551,6 +1565,7 @@ virtio_dev_rx_batch_packed_copy(struct virtio_net *dev,
struct rte_mbuf **pkts,
uint64_t *desc_addrs,
uint64_t *lens)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
@@ -1598,6 +1613,7 @@ static __rte_always_inline int
virtio_dev_rx_sync_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t desc_addrs[PACKED_BATCH_SIZE];
uint64_t lens[PACKED_BATCH_SIZE];
@@ -1620,6 +1636,7 @@ virtio_dev_rx_single_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf *pkt)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint16_t nr_descs = 0;
@@ -1645,6 +1662,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
struct rte_mbuf **__rte_restrict pkts,
uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -1772,6 +1790,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t pkt_idx = 0;
@@ -1879,6 +1898,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
uint16_t *nr_descs,
uint16_t *nr_buffers)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t nr_vec = 0;
uint16_t avail_idx = vq->last_avail_idx;
@@ -1938,6 +1958,7 @@ static __rte_always_inline int16_t
virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t *nr_buffers)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
@@ -1961,6 +1982,7 @@ virtio_dev_rx_async_packed_batch_enqueue(struct virtio_net *dev,
uint64_t *desc_addrs,
uint64_t *lens)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
@@ -2022,6 +2044,7 @@ virtio_dev_rx_async_packed_batch(struct virtio_net *dev,
struct rte_mbuf **pkts,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint64_t desc_addrs[PACKED_BATCH_SIZE];
uint64_t lens[PACKED_BATCH_SIZE];
@@ -2069,6 +2092,7 @@ static __rte_noinline uint32_t
virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t n_xfer;
@@ -2843,6 +2867,7 @@ desc_to_mbuf(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mbuf *m, struct rte_mempool *mbuf_pool,
bool legacy_ol_flags, uint16_t slot_idx, bool is_async)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t buf_avail, buf_offset, buf_len;
uint64_t buf_addr, buf_iova;
@@ -3049,6 +3074,7 @@ virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t i;
uint16_t avail_entries;
@@ -3153,6 +3179,7 @@ virtio_dev_tx_split_legacy(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3163,6 +3190,7 @@ virtio_dev_tx_split_compliant(struct virtio_net *dev,
struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **pkts, uint16_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3174,6 +3202,7 @@ vhost_reserve_avail_batch_packed(struct virtio_net *dev,
uint16_t avail_idx,
uintptr_t *desc_addrs,
uint16_t *ids)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
bool wrap = vq->avail_wrap_counter;
struct vring_packed_desc *descs = vq->desc_packed;
@@ -3317,6 +3346,7 @@ virtio_dev_tx_batch_packed(struct virtio_net *dev,
struct vhost_virtqueue *vq,
struct rte_mbuf **pkts,
bool legacy_ol_flags)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t avail_idx = vq->last_avail_idx;
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
@@ -3364,6 +3394,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
uint16_t *desc_count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
struct buf_vector buf_vec[BUF_VECTOR_MAX];
uint32_t buf_len;
@@ -3413,6 +3444,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
struct rte_mbuf *pkts,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t buf_id, desc_count = 0;
@@ -3444,6 +3476,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
uint32_t count,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
@@ -3488,6 +3521,7 @@ virtio_dev_tx_packed_legacy(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
}
@@ -3498,6 +3532,7 @@ virtio_dev_tx_packed_compliant(struct virtio_net *dev,
struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool *mbuf_pool,
struct rte_mbuf **__rte_restrict pkts, uint32_t count)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
}
@@ -3661,6 +3696,7 @@ virtio_dev_tx_async_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
static bool allocerr_warned;
bool dropped = false;
@@ -3808,6 +3844,7 @@ virtio_dev_tx_async_split_legacy(struct virtio_net *dev,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -3820,6 +3857,7 @@ virtio_dev_tx_async_split_compliant(struct virtio_net *dev,
struct rte_mbuf **pkts, uint16_t count,
int16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
@@ -3851,6 +3889,7 @@ virtio_dev_tx_async_single_packed(struct virtio_net *dev,
uint16_t slot_idx,
bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
int err;
uint16_t buf_id, desc_count = 0;
@@ -3903,6 +3942,7 @@ virtio_dev_tx_async_packed_batch(struct virtio_net *dev,
struct rte_mbuf **pkts, uint16_t slot_idx,
uint16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint16_t avail_idx = vq->last_avail_idx;
uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
@@ -3960,6 +4000,7 @@ virtio_dev_tx_async_packed(struct virtio_net *dev, struct vhost_virtqueue *vq,
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
uint32_t pkt_idx = 0;
uint16_t slot_idx = 0;
@@ -4070,6 +4111,7 @@ virtio_dev_tx_async_packed_legacy(struct virtio_net *dev, struct vhost_virtqueue
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, true);
@@ -4081,6 +4123,7 @@ virtio_dev_tx_async_packed_compliant(struct virtio_net *dev, struct vhost_virtqu
struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
uint16_t count, uint16_t dma_id, uint16_t vchan_id)
__rte_exclusive_locks_required(&vq->access_lock)
+ __rte_shared_locks_required(&vq->iotlb_lock)
{
return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
pkts, count, dma_id, vchan_id, false);
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 8/9] vhost: annotate vDPA device list accesses
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
` (6 preceding siblings ...)
2023-02-07 10:45 ` [PATCH v6 7/9] vhost: annotate " David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:02 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 9/9] vhost: enable lock check David Marchand
` (2 subsequent siblings)
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
Access to vdpa_device_list must be protected with vdpa_device_list_lock
spinlock.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
Changes since RFC v3:
- rebased,
---
lib/vhost/vdpa.c | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)
diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
index a430a66970..6284ea2ed1 100644
--- a/lib/vhost/vdpa.c
+++ b/lib/vhost/vdpa.c
@@ -23,21 +23,22 @@
/** Double linked list of vDPA devices. */
TAILQ_HEAD(vdpa_device_list, rte_vdpa_device);
-static struct vdpa_device_list vdpa_device_list =
- TAILQ_HEAD_INITIALIZER(vdpa_device_list);
+static struct vdpa_device_list vdpa_device_list__ =
+ TAILQ_HEAD_INITIALIZER(vdpa_device_list__);
static rte_spinlock_t vdpa_device_list_lock = RTE_SPINLOCK_INITIALIZER;
+static struct vdpa_device_list * const vdpa_device_list
+ __rte_guarded_by(&vdpa_device_list_lock) = &vdpa_device_list__;
-
-/* Unsafe, needs to be called with vdpa_device_list_lock held */
static struct rte_vdpa_device *
__vdpa_find_device_by_name(const char *name)
+ __rte_exclusive_locks_required(&vdpa_device_list_lock)
{
struct rte_vdpa_device *dev, *ret = NULL;
if (name == NULL)
return NULL;
- TAILQ_FOREACH(dev, &vdpa_device_list, next) {
+ TAILQ_FOREACH(dev, vdpa_device_list, next) {
if (!strncmp(dev->device->name, name, RTE_DEV_NAME_MAX_LEN)) {
ret = dev;
break;
@@ -116,7 +117,7 @@ rte_vdpa_register_device(struct rte_device *rte_dev,
dev->type = RTE_VHOST_VDPA_DEVICE_TYPE_NET;
}
- TAILQ_INSERT_TAIL(&vdpa_device_list, dev, next);
+ TAILQ_INSERT_TAIL(vdpa_device_list, dev, next);
out_unlock:
rte_spinlock_unlock(&vdpa_device_list_lock);
@@ -130,11 +131,11 @@ rte_vdpa_unregister_device(struct rte_vdpa_device *dev)
int ret = -1;
rte_spinlock_lock(&vdpa_device_list_lock);
- RTE_TAILQ_FOREACH_SAFE(cur_dev, &vdpa_device_list, next, tmp_dev) {
+ RTE_TAILQ_FOREACH_SAFE(cur_dev, vdpa_device_list, next, tmp_dev) {
if (dev != cur_dev)
continue;
- TAILQ_REMOVE(&vdpa_device_list, dev, next);
+ TAILQ_REMOVE(vdpa_device_list, dev, next);
rte_free(dev);
ret = 0;
break;
@@ -336,7 +337,7 @@ vdpa_find_device(const struct rte_vdpa_device *start, rte_vdpa_cmp_t cmp,
rte_spinlock_lock(&vdpa_device_list_lock);
if (start == NULL)
- dev = TAILQ_FIRST(&vdpa_device_list);
+ dev = TAILQ_FIRST(vdpa_device_list);
else
dev = TAILQ_NEXT(start, next);
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* [PATCH v6 9/9] vhost: enable lock check
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
` (7 preceding siblings ...)
2023-02-07 10:45 ` [PATCH v6 8/9] vhost: annotate vDPA device list accesses David Marchand
@ 2023-02-07 10:45 ` David Marchand
2023-02-09 8:05 ` Xia, Chenbo
2023-02-09 7:59 ` [PATCH v6 0/9] Lock annotations Xia, Chenbo
2023-02-09 13:48 ` David Marchand
10 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-07 10:45 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb
Now that all locks in this library are annotated, we can enable the
check.
Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
lib/vhost/meson.build | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
index bc7272053b..197a51d936 100644
--- a/lib/vhost/meson.build
+++ b/lib/vhost/meson.build
@@ -17,6 +17,8 @@ elif (toolchain == 'icc' and cc.version().version_compare('>=16.0.0'))
endif
dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY', cc.has_header('linux/userfaultfd.h'))
cflags += '-fno-strict-aliasing'
+
+annotate_locks = true
sources = files(
'fd_man.c',
'iotlb.c',
--
2.39.1
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 0/9] Lock annotations
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
` (8 preceding siblings ...)
2023-02-07 10:45 ` [PATCH v6 9/9] vhost: enable lock check David Marchand
@ 2023-02-09 7:59 ` Xia, Chenbo
2023-02-09 8:08 ` David Marchand
2023-02-09 13:48 ` David Marchand
10 siblings, 1 reply; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 7:59 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:45 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 0/9] Lock annotations
>
> vhost internals involves multiple locks to protect data access by
> multiple threads.
>
> This series uses clang thread safety checks [1] to catch issues during
> compilation: EAL spinlock, seqlock and rwlock are annotated and vhost
> code is instrumented so that clang can statically check correctness.
>
> Those annotations are quite heavy to maintain because the full path of
> code must be annotated (as can be seen in the vhost datapath code),
> but I think it is worth using.
>
> This has been tested against the whole tree and some fixes are already
> flying on the mailing list (see [2] for a list).
>
> If this first series is merged, I will prepare a followup series for EAL
> and other libraries.
>
>
> 1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
> 2:
> https://patchwork.dpdk.org/bundle/dmarchand/lock_fixes/?state=*&archive=bo
> th
>
> --
> David Marchand
>
> Changes since v5:
> - rebased after lib/vhost updates (patches 5 and 7),
>
> Changes since v4:
> - masked annotations from Doxygen as it seems confused with some
> constructs,
> - fixed typos,
>
> Changes since RFC v3:
> - sorry Maxime, it has been too long since RFC v3 and the code evolved,
> so I dropped all your review tags,
> - rebased,
> - added documentation,
> - dropped/fixed annotations in arch-specific and EAL headers,
> - rewrote need reply handling so that we don't have to waive the check
> on the associated functions,
> - separated IOTLB lock unconditional acquire from the annotation patch,
> - rewrote runtime checks for "unsafe" functions using a panicking assert
> helper,
>
> Changes since RFC v2:
> - fixed trylock annotations for rwlock,
> - annotated _tm flavors of spinlock and rwlock,
> - removed Maxime vhost fix from series (since Mimecast does not like
> me sending Maxime patch...), added a dependency on original fix
> as a hint for reviewers,
> - renamed attributes,
>
> Changes since RFC v1:
> - Cc'd people who have pending patches for vhost,
> - moved annotations to EAL and removed wrappers in vhost,
> - as a result of moving to EAL, this series will be tested against
> the main repo, so patch 1 has been kept as part of the series
> even if already applied to next-virtio,
> - refined/split patches and annotated all spinlocks in vhost,
>
>
> David Marchand (9):
> eal: annotate spinlock, rwlock and seqlock
> vhost: simplify need reply handling
> vhost: terminate when access lock is not taken
> vhost: annotate virtqueue access lock
> vhost: annotate async accesses
> vhost: always take IOTLB lock
> vhost: annotate IOTLB lock
> vhost: annotate vDPA device list accesses
> vhost: enable lock check
>
> doc/api/doxy-api.conf.in | 11 ++
> .../prog_guide/env_abstraction_layer.rst | 24 ++++
> doc/guides/rel_notes/release_23_03.rst | 5 +
> drivers/meson.build | 5 +
> lib/eal/include/generic/rte_rwlock.h | 27 +++-
> lib/eal/include/generic/rte_spinlock.h | 31 +++--
> lib/eal/include/meson.build | 1 +
> lib/eal/include/rte_lock_annotations.h | 73 ++++++++++
> lib/eal/include/rte_seqlock.h | 2 +
> lib/eal/ppc/include/rte_spinlock.h | 3 +
> lib/eal/x86/include/rte_rwlock.h | 4 +
> lib/eal/x86/include/rte_spinlock.h | 9 ++
> lib/meson.build | 5 +
> lib/vhost/iotlb.h | 4 +
> lib/vhost/meson.build | 2 +
> lib/vhost/vdpa.c | 20 +--
> lib/vhost/vhost.c | 38 ++---
> lib/vhost/vhost.h | 34 ++++-
> lib/vhost/vhost_crypto.c | 8 ++
> lib/vhost/vhost_user.c | 131 ++++++++----------
> lib/vhost/virtio_net.c | 118 ++++++++++++----
> 21 files changed, 405 insertions(+), 150 deletions(-)
> create mode 100644 lib/eal/include/rte_lock_annotations.h
>
> --
> 2.39.1
Seems one compilation error reported? Not sure it's related or not.
Thanks,
Chenbo
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 1/9] eal: annotate spinlock, rwlock and seqlock
2023-02-07 10:45 ` [PATCH v6 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
@ 2023-02-09 8:00 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:00 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb,
Burakov, Anatoly, mattias.ronnblom, David Christensen,
Richardson, Bruce, Konstantin Ananyev
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:45 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com; Burakov, Anatoly <anatoly.burakov@intel.com>;
> mattias.ronnblom <mattias.ronnblom@ericsson.com>; David Christensen
> <drc@linux.vnet.ibm.com>; Richardson, Bruce <bruce.richardson@intel.com>;
> Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>
> Subject: [PATCH v6 1/9] eal: annotate spinlock, rwlock and seqlock
>
> clang offers some thread safety checks, statically verifying that locks
> are taken and released in the code.
> To use those checks, the full code leading to taking or releasing locks
> must be annotated with some attributes.
>
> Wrap those attributes into our own set of macros.
>
> rwlock, seqlock and the "normal" spinlock are instrumented.
>
> Those checks might be of interest out of DPDK, but it requires that the
> including application locks are annotated.
> On the other hand, applications out there might have been using
> those same checks.
> To be on the safe side, keep this instrumentation under a
> RTE_ANNOTATE_LOCKS internal build flag.
>
> A component may en/disable this check by setting
> annotate_locks = true/false in its meson.build.
>
> Note:
> Doxygen preprocessor does not understand trailing function attributes
> (this can be observed with the rte_seqlock.h header).
> One would think that expanding the annotation macros to a noop in
> rte_lock_annotations.h would be enough (since RTE_ANNOTATE_LOCKS is not
> set during doxygen processing)). Unfortunately, the use of
> EXPAND_ONLY_PREDEF defeats this.
>
> Removing EXPAND_ONLY_PREDEF entirely is not an option as it would expand
> all other DPDK macros.
>
> The chosen solution is to expand the annotation macros explicitly to a
> noop in PREDEFINED.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> --
> 2.39.1
Acked-by: Chenbo Xia <chenbo.xia@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 2/9] vhost: simplify need reply handling
2023-02-07 10:45 ` [PATCH v6 2/9] vhost: simplify need reply handling David Marchand
@ 2023-02-09 8:00 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:00 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:45 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 2/9] vhost: simplify need reply handling
>
> Dedicate send_vhost_slave_message() helper to the case when no reply is
> needed.
>
> Add a send_vhost_slave_message_process_reply() helper for the opposite.
> This new helper merges both send_vhost_slave_message() and the code
> previously in process_slave_message_reply().
> The slave_req_lock lock is then only handled in this helper which will
> make lock checks trivial.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> lib/vhost/vhost_user.c | 119 ++++++++++++++++++-----------------------
> 1 file changed, 51 insertions(+), 68 deletions(-)
>
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 8f33d5f4d9..60ec1bf5f6 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -2878,18 +2878,46 @@ send_vhost_reply(struct virtio_net *dev, int
> sockfd, struct vhu_msg_context *ctx
> }
>
> static int
> -send_vhost_slave_message(struct virtio_net *dev,
> - struct vhu_msg_context *ctx)
> +send_vhost_slave_message(struct virtio_net *dev, struct vhu_msg_context
> *ctx)
> +{
> + return send_vhost_message(dev, dev->slave_req_fd, ctx);
> +}
> +
> +static int
> +send_vhost_slave_message_process_reply(struct virtio_net *dev, struct
> vhu_msg_context *ctx)
> {
> + struct vhu_msg_context msg_reply;
> int ret;
>
> - if (ctx->msg.flags & VHOST_USER_NEED_REPLY)
> - rte_spinlock_lock(&dev->slave_req_lock);
> + rte_spinlock_lock(&dev->slave_req_lock);
> + ret = send_vhost_slave_message(dev, ctx);
> + if (ret < 0) {
> + VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config
> change (%d)\n", ret);
> + goto out;
> + }
>
> - ret = send_vhost_message(dev, dev->slave_req_fd, ctx);
> - if (ret < 0 && (ctx->msg.flags & VHOST_USER_NEED_REPLY))
> - rte_spinlock_unlock(&dev->slave_req_lock);
> + ret = read_vhost_message(dev, dev->slave_req_fd, &msg_reply);
> + if (ret <= 0) {
> + if (ret < 0)
> + VHOST_LOG_CONFIG(dev->ifname, ERR,
> + "vhost read slave message reply failed\n");
> + else
> + VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer
> closed\n");
> + ret = -1;
> + goto out;
> + }
> +
> + if (msg_reply.msg.request.slave != ctx->msg.request.slave) {
> + VHOST_LOG_CONFIG(dev->ifname, ERR,
> + "received unexpected msg type (%u), expected %u\n",
> + msg_reply.msg.request.slave, ctx->msg.request.slave);
> + ret = -1;
> + goto out;
> + }
>
> + ret = msg_reply.msg.payload.u64 ? -1 : 0;
> +out:
> + rte_spinlock_unlock(&dev->slave_req_lock);
> return ret;
> }
>
> @@ -3213,42 +3241,6 @@ vhost_user_msg_handler(int vid, int fd)
> return ret;
> }
>
> -static int process_slave_message_reply(struct virtio_net *dev,
> - const struct vhu_msg_context *ctx)
> -{
> - struct vhu_msg_context msg_reply;
> - int ret;
> -
> - if ((ctx->msg.flags & VHOST_USER_NEED_REPLY) == 0)
> - return 0;
> -
> - ret = read_vhost_message(dev, dev->slave_req_fd, &msg_reply);
> - if (ret <= 0) {
> - if (ret < 0)
> - VHOST_LOG_CONFIG(dev->ifname, ERR,
> - "vhost read slave message reply failed\n");
> - else
> - VHOST_LOG_CONFIG(dev->ifname, INFO, "vhost peer
> closed\n");
> - ret = -1;
> - goto out;
> - }
> -
> - ret = 0;
> - if (msg_reply.msg.request.slave != ctx->msg.request.slave) {
> - VHOST_LOG_CONFIG(dev->ifname, ERR,
> - "received unexpected msg type (%u), expected %u\n",
> - msg_reply.msg.request.slave, ctx->msg.request.slave);
> - ret = -1;
> - goto out;
> - }
> -
> - ret = msg_reply.msg.payload.u64 ? -1 : 0;
> -
> -out:
> - rte_spinlock_unlock(&dev->slave_req_lock);
> - return ret;
> -}
> -
> int
> vhost_user_iotlb_miss(struct virtio_net *dev, uint64_t iova, uint8_t perm)
> {
> @@ -3277,10 +3269,9 @@ vhost_user_iotlb_miss(struct virtio_net *dev,
> uint64_t iova, uint8_t perm)
> return 0;
> }
>
> -static int
> -vhost_user_slave_config_change(struct virtio_net *dev, bool need_reply)
> +int
> +rte_vhost_slave_config_change(int vid, bool need_reply)
> {
> - int ret;
> struct vhu_msg_context ctx = {
> .msg = {
> .request.slave = VHOST_USER_SLAVE_CONFIG_CHANGE_MSG,
> @@ -3288,29 +3279,23 @@ vhost_user_slave_config_change(struct virtio_net
> *dev, bool need_reply)
> .size = 0,
> }
> };
> -
> - if (need_reply)
> - ctx.msg.flags |= VHOST_USER_NEED_REPLY;
> -
> - ret = send_vhost_slave_message(dev, &ctx);
> - if (ret < 0) {
> - VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config
> change (%d)\n", ret);
> - return ret;
> - }
> -
> - return process_slave_message_reply(dev, &ctx);
> -}
> -
> -int
> -rte_vhost_slave_config_change(int vid, bool need_reply)
> -{
> struct virtio_net *dev;
> + int ret;
>
> dev = get_device(vid);
> if (!dev)
> return -ENODEV;
>
> - return vhost_user_slave_config_change(dev, need_reply);
> + if (!need_reply) {
> + ret = send_vhost_slave_message(dev, &ctx);
> + } else {
> + ctx.msg.flags |= VHOST_USER_NEED_REPLY;
> + ret = send_vhost_slave_message_process_reply(dev, &ctx);
> + }
> +
> + if (ret < 0)
> + VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to send config
> change (%d)\n", ret);
> + return ret;
> }
>
> static int vhost_user_slave_set_vring_host_notifier(struct virtio_net
> *dev,
> @@ -3339,13 +3324,11 @@ static int
> vhost_user_slave_set_vring_host_notifier(struct virtio_net *dev,
> ctx.fd_num = 1;
> }
>
> - ret = send_vhost_slave_message(dev, &ctx);
> - if (ret < 0) {
> + ret = send_vhost_slave_message_process_reply(dev, &ctx);
> + if (ret < 0)
> VHOST_LOG_CONFIG(dev->ifname, ERR, "failed to set host
> notifier (%d)\n", ret);
> - return ret;
> - }
>
> - return process_slave_message_reply(dev, &ctx);
> + return ret;
> }
>
> int rte_vhost_host_notifier_ctrl(int vid, uint16_t qid, bool enable)
> --
> 2.39.1
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 3/9] vhost: terminate when access lock is not taken
2023-02-07 10:45 ` [PATCH v6 3/9] vhost: terminate when access lock is not taken David Marchand
@ 2023-02-09 8:01 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:01 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:45 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 3/9] vhost: terminate when access lock is not taken
>
> Be a bit more strict when a programmatic error is detected with regards to
> the access_lock not being taken.
> Mark the new helper with __rte_assert_exclusive_lock so that clang
> understands where locks are expected to be taken.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> lib/vhost/vhost.c | 18 +++---------------
> lib/vhost/vhost.h | 10 ++++++++++
> lib/vhost/virtio_net.c | 6 +-----
> 3 files changed, 14 insertions(+), 20 deletions(-)
>
> diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
> index 19c7b92c32..8cd727ca2f 100644
> --- a/lib/vhost/vhost.c
> +++ b/lib/vhost/vhost.c
> @@ -1781,11 +1781,7 @@ rte_vhost_async_channel_register_thread_unsafe(int
> vid, uint16_t queue_id)
> if (unlikely(vq == NULL || !dev->async_copy))
> return -1;
>
> - if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
> - VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access
> lock taken.\n",
> - __func__);
> - return -1;
> - }
> + vq_assert_lock(dev, vq);
>
> return async_channel_register(dev, vq);
> }
> @@ -1847,11 +1843,7 @@
> rte_vhost_async_channel_unregister_thread_unsafe(int vid, uint16_t
> queue_id)
> if (vq == NULL)
> return -1;
>
> - if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
> - VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access
> lock taken.\n",
> - __func__);
> - return -1;
> - }
> + vq_assert_lock(dev, vq);
>
> if (!vq->async)
> return 0;
> @@ -1994,11 +1986,7 @@ rte_vhost_async_get_inflight_thread_unsafe(int vid,
> uint16_t queue_id)
> if (vq == NULL)
> return ret;
>
> - if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
> - VHOST_LOG_CONFIG(dev->ifname, ERR, "%s() called without access
> lock taken.\n",
> - __func__);
> - return -1;
> - }
> + vq_assert_lock(dev, vq);
>
> if (!vq->async)
> return ret;
> diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
> index 1f913803f6..82fe9b5fda 100644
> --- a/lib/vhost/vhost.h
> +++ b/lib/vhost/vhost.h
> @@ -513,6 +513,16 @@ struct virtio_net {
> struct rte_vhost_user_extern_ops extern_ops;
> } __rte_cache_aligned;
>
> +static inline void
> +vq_assert_lock__(struct virtio_net *dev, struct vhost_virtqueue *vq,
> const char *func)
> + __rte_assert_exclusive_lock(&vq->access_lock)
> +{
> + if (unlikely(!rte_spinlock_is_locked(&vq->access_lock)))
> + rte_panic("VHOST_CONFIG: (%s) %s() called without access lock
> taken.\n",
> + dev->ifname, func);
> +}
> +#define vq_assert_lock(dev, vq) vq_assert_lock__(dev, vq, __func__)
> +
> static __rte_always_inline bool
> vq_is_packed(struct virtio_net *dev)
> {
> diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
> index aac7aa9d01..cc9675ebe5 100644
> --- a/lib/vhost/virtio_net.c
> +++ b/lib/vhost/virtio_net.c
> @@ -2367,11 +2367,7 @@ rte_vhost_clear_queue_thread_unsafe(int vid,
> uint16_t queue_id,
>
> vq = dev->virtqueue[queue_id];
>
> - if (unlikely(!rte_spinlock_is_locked(&vq->access_lock))) {
> - VHOST_LOG_DATA(dev->ifname, ERR, "%s() called without access
> lock taken.\n",
> - __func__);
> - return -1;
> - }
> + vq_assert_lock(dev, vq);
>
> if (unlikely(!vq->async)) {
> VHOST_LOG_DATA(dev->ifname, ERR,
> --
> 2.39.1
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 4/9] vhost: annotate virtqueue access lock
2023-02-07 10:45 ` [PATCH v6 4/9] vhost: annotate virtqueue access lock David Marchand
@ 2023-02-09 8:01 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:01 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:45 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 4/9] vhost: annotate virtqueue access lock
>
> vhost_user_lock/unlock_all_queue_pairs must be waived since clang
> annotations can't express taking a runtime number of locks.
>
> vhost_queue_stats_update() requirement can be expressed with a required
> tag.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> Changes since RFC v3:
> - removed annotations needed for vhost async which went to the next
> patch,
>
> ---
> lib/vhost/vhost_user.c | 2 ++
> lib/vhost/virtio_net.c | 4 +---
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 60ec1bf5f6..70d221b9f6 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -2965,6 +2965,7 @@ vhost_user_check_and_alloc_queue_pair(struct
> virtio_net *dev,
>
> static void
> vhost_user_lock_all_queue_pairs(struct virtio_net *dev)
> + __rte_no_thread_safety_analysis
> {
> unsigned int i = 0;
> unsigned int vq_num = 0;
> @@ -2982,6 +2983,7 @@ vhost_user_lock_all_queue_pairs(struct virtio_net
> *dev)
>
> static void
> vhost_user_unlock_all_queue_pairs(struct virtio_net *dev)
> + __rte_no_thread_safety_analysis
> {
> unsigned int i = 0;
> unsigned int vq_num = 0;
> diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
> index cc9675ebe5..f2ab6dba15 100644
> --- a/lib/vhost/virtio_net.c
> +++ b/lib/vhost/virtio_net.c
> @@ -52,12 +52,10 @@ is_valid_virt_queue_idx(uint32_t idx, int is_tx,
> uint32_t nr_vring)
> return (is_tx ^ (idx & 1)) == 0 && idx < nr_vring;
> }
>
> -/*
> - * This function must be called with virtqueue's access_lock taken.
> - */
> static inline void
> vhost_queue_stats_update(struct virtio_net *dev, struct vhost_virtqueue
> *vq,
> struct rte_mbuf **pkts, uint16_t count)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct virtqueue_stats *stats = &vq->stats;
> int i;
> --
> 2.39.1
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 5/9] vhost: annotate async accesses
2023-02-07 10:45 ` [PATCH v6 5/9] vhost: annotate async accesses David Marchand
@ 2023-02-09 8:01 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:01 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:45 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 5/9] vhost: annotate async accesses
>
> vq->async is initialised and must be accessed under vq->access_lock.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> Changes since v5:
> - rebased after packed support was added to async code,
>
> Changes since RFC v3:
> - rebased,
> - fixed annotations vq->access_lock -> &vq->access_lock,
> - reworked free_vq,
>
> ---
> lib/vhost/vhost.c | 4 ++++
> lib/vhost/vhost.h | 2 +-
> lib/vhost/vhost_user.c | 10 +++++++---
> lib/vhost/virtio_net.c | 41 +++++++++++++++++++++++++++++++++++++++++
> 4 files changed, 53 insertions(+), 4 deletions(-)
>
> diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
> index 8cd727ca2f..8bccdd8584 100644
> --- a/lib/vhost/vhost.c
> +++ b/lib/vhost/vhost.c
> @@ -369,6 +369,7 @@ cleanup_device(struct virtio_net *dev, int destroy)
>
> static void
> vhost_free_async_mem(struct vhost_virtqueue *vq)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> if (!vq->async)
> return;
> @@ -393,7 +394,9 @@ free_vq(struct virtio_net *dev, struct vhost_virtqueue
> *vq)
> else
> rte_free(vq->shadow_used_split);
>
> + rte_spinlock_lock(&vq->access_lock);
> vhost_free_async_mem(vq);
> + rte_spinlock_unlock(&vq->access_lock);
> rte_free(vq->batch_copy_elems);
> vhost_user_iotlb_destroy(vq);
> rte_free(vq->log_cache);
> @@ -1669,6 +1672,7 @@ rte_vhost_extern_callback_register(int vid,
>
> static __rte_always_inline int
> async_channel_register(struct virtio_net *dev, struct vhost_virtqueue *vq)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct vhost_async *async;
> int node = vq->numa_node;
> diff --git a/lib/vhost/vhost.h b/lib/vhost/vhost.h
> index 82fe9b5fda..c05313cf37 100644
> --- a/lib/vhost/vhost.h
> +++ b/lib/vhost/vhost.h
> @@ -326,7 +326,7 @@ struct vhost_virtqueue {
> struct rte_vhost_resubmit_info *resubmit_inflight;
> uint64_t global_counter;
>
> - struct vhost_async *async;
> + struct vhost_async *async __rte_guarded_var;
>
> int notif_enable;
> #define VIRTIO_UNINITIALIZED_NOTIF (-1)
> diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c
> index 70d221b9f6..8c1d60b76b 100644
> --- a/lib/vhost/vhost_user.c
> +++ b/lib/vhost/vhost_user.c
> @@ -2168,6 +2168,7 @@ vhost_user_set_vring_enable(struct virtio_net **pdev,
> int main_fd __rte_unused)
> {
> struct virtio_net *dev = *pdev;
> + struct vhost_virtqueue *vq;
> bool enable = !!ctx->msg.payload.state.num;
> int index = (int)ctx->msg.payload.state.index;
>
> @@ -2175,15 +2176,18 @@ vhost_user_set_vring_enable(struct virtio_net
> **pdev,
> "set queue enable: %d to qp idx: %d\n",
> enable, index);
>
> - if (enable && dev->virtqueue[index]->async) {
> - if (dev->virtqueue[index]->async->pkts_inflight_n) {
> + vq = dev->virtqueue[index];
> + /* vhost_user_lock_all_queue_pairs locked all qps */
> + vq_assert_lock(dev, vq);
> + if (enable && vq->async) {
> + if (vq->async->pkts_inflight_n) {
> VHOST_LOG_CONFIG(dev->ifname, ERR,
> "failed to enable vring. Inflight packets must be
> completed first\n");
> return RTE_VHOST_MSG_RESULT_ERR;
> }
> }
>
> - dev->virtqueue[index]->enabled = enable;
> + vq->enabled = enable;
>
> return RTE_VHOST_MSG_RESULT_OK;
> }
> diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
> index f2ab6dba15..6672caac49 100644
> --- a/lib/vhost/virtio_net.c
> +++ b/lib/vhost/virtio_net.c
> @@ -102,6 +102,7 @@ static __rte_always_inline int64_t
> vhost_async_dma_transfer_one(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> int16_t dma_id, uint16_t vchan_id, uint16_t flag_idx,
> struct vhost_iov_iter *pkt)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct async_dma_vchan_info *dma_info =
> &dma_copy_track[dma_id].vchans[vchan_id];
> uint16_t ring_mask = dma_info->ring_mask;
> @@ -151,6 +152,7 @@ static __rte_always_inline uint16_t
> vhost_async_dma_transfer(struct virtio_net *dev, struct vhost_virtqueue
> *vq,
> int16_t dma_id, uint16_t vchan_id, uint16_t head_idx,
> struct vhost_iov_iter *pkts, uint16_t nr_pkts)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct async_dma_vchan_info *dma_info =
> &dma_copy_track[dma_id].vchans[vchan_id];
> int64_t ret, nr_copies = 0;
> @@ -434,6 +436,7 @@ static __rte_always_inline void
> vhost_async_shadow_enqueue_packed_batch(struct vhost_virtqueue *vq,
> uint64_t *lens,
> uint16_t *ids)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t i;
> struct vhost_async *async = vq->async;
> @@ -450,6 +453,7 @@ vhost_async_shadow_enqueue_packed_batch(struct
> vhost_virtqueue *vq,
>
> static __rte_always_inline void
> vhost_async_shadow_dequeue_packed_batch(struct vhost_virtqueue *vq,
> uint16_t *ids)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t i;
> struct vhost_async *async = vq->async;
> @@ -611,6 +615,7 @@ vhost_async_shadow_enqueue_packed(struct
> vhost_virtqueue *vq,
> uint16_t *id,
> uint16_t *count,
> uint16_t num_buffers)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t i;
> struct vhost_async *async = vq->async;
> @@ -1118,6 +1123,7 @@ static __rte_always_inline int
> async_fill_seg(struct virtio_net *dev, struct vhost_virtqueue *vq,
> struct rte_mbuf *m, uint32_t mbuf_offset,
> uint64_t buf_iova, uint32_t cpy_len, bool to_desc)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct vhost_async *async = vq->async;
> uint64_t mapped_len;
> @@ -1195,6 +1201,7 @@ static __rte_always_inline int
> mbuf_to_desc(struct virtio_net *dev, struct vhost_virtqueue *vq,
> struct rte_mbuf *m, struct buf_vector *buf_vec,
> uint16_t nr_vec, uint16_t num_buffers, bool is_async)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint32_t vec_idx = 0;
> uint32_t mbuf_offset, mbuf_avail;
> @@ -1323,6 +1330,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
> struct rte_mbuf *pkt,
> struct buf_vector *buf_vec,
> uint16_t *nr_descs)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t nr_vec = 0;
> uint16_t avail_idx = vq->last_avail_idx;
> @@ -1383,6 +1391,7 @@ vhost_enqueue_single_packed(struct virtio_net *dev,
> static __rte_noinline uint32_t
> virtio_dev_rx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
> struct rte_mbuf **pkts, uint32_t count)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint32_t pkt_idx = 0;
> uint16_t num_buffers;
> @@ -1610,6 +1619,7 @@ static __rte_always_inline int16_t
> virtio_dev_rx_single_packed(struct virtio_net *dev,
> struct vhost_virtqueue *vq,
> struct rte_mbuf *pkt)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct buf_vector buf_vec[BUF_VECTOR_MAX];
> uint16_t nr_descs = 0;
> @@ -1634,6 +1644,7 @@ virtio_dev_rx_packed(struct virtio_net *dev,
> struct vhost_virtqueue *__rte_restrict vq,
> struct rte_mbuf **__rte_restrict pkts,
> uint32_t count)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint32_t pkt_idx = 0;
>
> @@ -1733,6 +1744,7 @@ rte_vhost_enqueue_burst(int vid, uint16_t queue_id,
>
> static __rte_always_inline uint16_t
> async_get_first_inflight_pkt_idx(struct vhost_virtqueue *vq)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct vhost_async *async = vq->async;
>
> @@ -1761,6 +1773,7 @@ store_dma_desc_info_split(struct vring_used_elem
> *s_ring, struct vring_used_elem
> static __rte_noinline uint32_t
> virtio_dev_rx_async_submit_split(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t
> vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct buf_vector buf_vec[BUF_VECTOR_MAX];
> uint32_t pkt_idx = 0;
> @@ -1867,6 +1880,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
> struct buf_vector *buf_vec,
> uint16_t *nr_descs,
> uint16_t *nr_buffers)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t nr_vec = 0;
> uint16_t avail_idx = vq->last_avail_idx;
> @@ -1925,6 +1939,7 @@ vhost_enqueue_async_packed(struct virtio_net *dev,
> static __rte_always_inline int16_t
> virtio_dev_rx_async_packed(struct virtio_net *dev, struct vhost_virtqueue
> *vq,
> struct rte_mbuf *pkt, uint16_t *nr_descs, uint16_t
> *nr_buffers)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct buf_vector buf_vec[BUF_VECTOR_MAX];
>
> @@ -1947,6 +1962,7 @@ virtio_dev_rx_async_packed_batch_enqueue(struct
> virtio_net *dev,
> struct rte_mbuf **pkts,
> uint64_t *desc_addrs,
> uint64_t *lens)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
> struct virtio_net_hdr_mrg_rxbuf *hdrs[PACKED_BATCH_SIZE];
> @@ -2007,6 +2023,7 @@ virtio_dev_rx_async_packed_batch(struct virtio_net
> *dev,
> struct vhost_virtqueue *vq,
> struct rte_mbuf **pkts,
> int16_t dma_id, uint16_t vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint64_t desc_addrs[PACKED_BATCH_SIZE];
> uint64_t lens[PACKED_BATCH_SIZE];
> @@ -2022,6 +2039,7 @@ virtio_dev_rx_async_packed_batch(struct virtio_net
> *dev,
> static __rte_always_inline void
> dma_error_handler_packed(struct vhost_virtqueue *vq, uint16_t slot_idx,
> uint32_t nr_err, uint32_t *pkt_idx)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t descs_err = 0;
> uint16_t buffers_err = 0;
> @@ -2052,6 +2070,7 @@ dma_error_handler_packed(struct vhost_virtqueue *vq,
> uint16_t slot_idx,
> static __rte_noinline uint32_t
> virtio_dev_rx_async_submit_packed(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> struct rte_mbuf **pkts, uint32_t count, int16_t dma_id, uint16_t
> vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint32_t pkt_idx = 0;
> uint16_t n_xfer;
> @@ -2124,6 +2143,7 @@ virtio_dev_rx_async_submit_packed(struct virtio_net
> *dev, struct vhost_virtqueue
>
> static __rte_always_inline void
> write_back_completed_descs_split(struct vhost_virtqueue *vq, uint16_t
> n_descs)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct vhost_async *async = vq->async;
> uint16_t nr_left = n_descs;
> @@ -2156,6 +2176,7 @@ write_back_completed_descs_split(struct
> vhost_virtqueue *vq, uint16_t n_descs)
> static __rte_always_inline void
> write_back_completed_descs_packed(struct vhost_virtqueue *vq,
> uint16_t n_buffers)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct vhost_async *async = vq->async;
> uint16_t from = async->last_buffer_idx_packed;
> @@ -2220,6 +2241,7 @@ write_back_completed_descs_packed(struct
> vhost_virtqueue *vq,
> static __rte_always_inline uint16_t
> vhost_poll_enqueue_completed(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> struct rte_mbuf **pkts, uint16_t count, int16_t dma_id, uint16_t
> vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct vhost_async *async = vq->async;
> struct async_inflight_info *pkts_info = async->pkts_info;
> @@ -2824,6 +2846,7 @@ desc_to_mbuf(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> struct buf_vector *buf_vec, uint16_t nr_vec,
> struct rte_mbuf *m, struct rte_mempool *mbuf_pool,
> bool legacy_ol_flags, uint16_t slot_idx, bool is_async)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint32_t buf_avail, buf_offset, buf_len;
> uint64_t buf_addr, buf_iova;
> @@ -3029,6 +3052,7 @@ static uint16_t
> virtio_dev_tx_split(struct virtio_net *dev, struct vhost_virtqueue *vq,
> struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts, uint16_t
> count,
> bool legacy_ol_flags)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t i;
> uint16_t avail_entries;
> @@ -3132,6 +3156,7 @@ static uint16_t
> virtio_dev_tx_split_legacy(struct virtio_net *dev,
> struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
> struct rte_mbuf **pkts, uint16_t count)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, true);
> }
> @@ -3141,6 +3166,7 @@ static uint16_t
> virtio_dev_tx_split_compliant(struct virtio_net *dev,
> struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
> struct rte_mbuf **pkts, uint16_t count)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> return virtio_dev_tx_split(dev, vq, mbuf_pool, pkts, count, false);
> }
> @@ -3341,6 +3367,7 @@ vhost_dequeue_single_packed(struct virtio_net *dev,
> uint16_t *buf_id,
> uint16_t *desc_count,
> bool legacy_ol_flags)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct buf_vector buf_vec[BUF_VECTOR_MAX];
> uint32_t buf_len;
> @@ -3389,6 +3416,7 @@ virtio_dev_tx_single_packed(struct virtio_net *dev,
> struct rte_mempool *mbuf_pool,
> struct rte_mbuf *pkts,
> bool legacy_ol_flags)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
>
> uint16_t buf_id, desc_count = 0;
> @@ -3419,6 +3447,7 @@ virtio_dev_tx_packed(struct virtio_net *dev,
> struct rte_mbuf **__rte_restrict pkts,
> uint32_t count,
> bool legacy_ol_flags)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint32_t pkt_idx = 0;
>
> @@ -3462,6 +3491,7 @@ static uint16_t
> virtio_dev_tx_packed_legacy(struct virtio_net *dev,
> struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool
> *mbuf_pool,
> struct rte_mbuf **__rte_restrict pkts, uint32_t count)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, true);
> }
> @@ -3471,6 +3501,7 @@ static uint16_t
> virtio_dev_tx_packed_compliant(struct virtio_net *dev,
> struct vhost_virtqueue *__rte_restrict vq, struct rte_mempool
> *mbuf_pool,
> struct rte_mbuf **__rte_restrict pkts, uint32_t count)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> return virtio_dev_tx_packed(dev, vq, mbuf_pool, pkts, count, false);
> }
> @@ -3588,6 +3619,7 @@ static __rte_always_inline uint16_t
> async_poll_dequeue_completed(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> struct rte_mbuf **pkts, uint16_t count, int16_t dma_id,
> uint16_t vchan_id, bool legacy_ol_flags)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t start_idx, from, i;
> uint16_t nr_cpl_pkts = 0;
> @@ -3634,6 +3666,7 @@ static __rte_always_inline uint16_t
> virtio_dev_tx_async_split(struct virtio_net *dev, struct vhost_virtqueue
> *vq,
> struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
> uint16_t count,
> int16_t dma_id, uint16_t vchan_id, bool legacy_ol_flags)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> static bool allocerr_warned;
> bool dropped = false;
> @@ -3780,6 +3813,7 @@ virtio_dev_tx_async_split_legacy(struct virtio_net
> *dev,
> struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
> struct rte_mbuf **pkts, uint16_t count,
> int16_t dma_id, uint16_t vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
> pkts, count, dma_id, vchan_id, true);
> @@ -3791,6 +3825,7 @@ virtio_dev_tx_async_split_compliant(struct
> virtio_net *dev,
> struct vhost_virtqueue *vq, struct rte_mempool *mbuf_pool,
> struct rte_mbuf **pkts, uint16_t count,
> int16_t dma_id, uint16_t vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> return virtio_dev_tx_async_split(dev, vq, mbuf_pool,
> pkts, count, dma_id, vchan_id, false);
> @@ -3799,6 +3834,7 @@ virtio_dev_tx_async_split_compliant(struct
> virtio_net *dev,
> static __rte_always_inline void
> vhost_async_shadow_dequeue_single_packed(struct vhost_virtqueue *vq,
> uint16_t buf_id, uint16_t count)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> struct vhost_async *async = vq->async;
> uint16_t idx = async->buffer_idx_packed;
> @@ -3820,6 +3856,7 @@ virtio_dev_tx_async_single_packed(struct virtio_net
> *dev,
> struct rte_mbuf *pkts,
> uint16_t slot_idx,
> bool legacy_ol_flags)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> int err;
> uint16_t buf_id, desc_count = 0;
> @@ -3871,6 +3908,7 @@ virtio_dev_tx_async_packed_batch(struct virtio_net
> *dev,
> struct vhost_virtqueue *vq,
> struct rte_mbuf **pkts, uint16_t slot_idx,
> uint16_t dma_id, uint16_t vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint16_t avail_idx = vq->last_avail_idx;
> uint32_t buf_offset = sizeof(struct virtio_net_hdr_mrg_rxbuf);
> @@ -3927,6 +3965,7 @@ static __rte_always_inline uint16_t
> virtio_dev_tx_async_packed(struct virtio_net *dev, struct vhost_virtqueue
> *vq,
> struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
> uint16_t count, uint16_t dma_id, uint16_t vchan_id, bool
> legacy_ol_flags)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> uint32_t pkt_idx = 0;
> uint16_t slot_idx = 0;
> @@ -4036,6 +4075,7 @@ static uint16_t
> virtio_dev_tx_async_packed_legacy(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
> uint16_t count, uint16_t dma_id, uint16_t vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
> pkts, count, dma_id, vchan_id, true);
> @@ -4046,6 +4086,7 @@ static uint16_t
> virtio_dev_tx_async_packed_compliant(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> struct rte_mempool *mbuf_pool, struct rte_mbuf **pkts,
> uint16_t count, uint16_t dma_id, uint16_t vchan_id)
> + __rte_exclusive_locks_required(&vq->access_lock)
> {
> return virtio_dev_tx_async_packed(dev, vq, mbuf_pool,
> pkts, count, dma_id, vchan_id, false);
> --
> 2.39.1
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 6/9] vhost: always take IOTLB lock
2023-02-07 10:45 ` [PATCH v6 6/9] vhost: always take IOTLB lock David Marchand
@ 2023-02-09 8:01 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:01 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:45 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 6/9] vhost: always take IOTLB lock
>
> clang does not support conditionally held locks when statically analysing
> taken locks with thread safety checks.
> Always take iotlb locks regardless of VIRTIO_F_IOMMU_PLATFORM feature.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> lib/vhost/vhost.c | 8 +++-----
> lib/vhost/virtio_net.c | 24 ++++++++----------------
> 2 files changed, 11 insertions(+), 21 deletions(-)
>
> diff --git a/lib/vhost/vhost.c b/lib/vhost/vhost.c
> index 8bccdd8584..1e0c30791e 100644
> --- a/lib/vhost/vhost.c
> +++ b/lib/vhost/vhost.c
> @@ -563,10 +563,9 @@ vring_translate(struct virtio_net *dev, struct
> vhost_virtqueue *vq)
> }
>
> void
> -vring_invalidate(struct virtio_net *dev, struct vhost_virtqueue *vq)
> +vring_invalidate(struct virtio_net *dev __rte_unused, struct
> vhost_virtqueue *vq)
> {
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_wr_lock(vq);
> + vhost_user_iotlb_wr_lock(vq);
>
> vq->access_ok = false;
> vq->desc = NULL;
> @@ -574,8 +573,7 @@ vring_invalidate(struct virtio_net *dev, struct
> vhost_virtqueue *vq)
> vq->used = NULL;
> vq->log_guest_addr = 0;
>
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_wr_unlock(vq);
> + vhost_user_iotlb_wr_unlock(vq);
> }
>
> static void
> diff --git a/lib/vhost/virtio_net.c b/lib/vhost/virtio_net.c
> index 6672caac49..49fc46e127 100644
> --- a/lib/vhost/virtio_net.c
> +++ b/lib/vhost/virtio_net.c
> @@ -1688,8 +1688,7 @@ virtio_dev_rx(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> if (unlikely(!vq->enabled))
> goto out_access_unlock;
>
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_rd_lock(vq);
> + vhost_user_iotlb_rd_lock(vq);
>
> if (unlikely(!vq->access_ok))
> if (unlikely(vring_translate(dev, vq) < 0))
> @@ -1707,8 +1706,7 @@ virtio_dev_rx(struct virtio_net *dev, struct
> vhost_virtqueue *vq,
> vhost_queue_stats_update(dev, vq, pkts, nb_tx);
>
> out:
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_rd_unlock(vq);
> + vhost_user_iotlb_rd_unlock(vq);
>
> out_access_unlock:
> rte_spinlock_unlock(&vq->access_lock);
> @@ -2499,8 +2497,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev,
> struct vhost_virtqueue *vq,
> if (unlikely(!vq->enabled || !vq->async))
> goto out_access_unlock;
>
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_rd_lock(vq);
> + vhost_user_iotlb_rd_lock(vq);
>
> if (unlikely(!vq->access_ok))
> if (unlikely(vring_translate(dev, vq) < 0))
> @@ -2520,8 +2517,7 @@ virtio_dev_rx_async_submit(struct virtio_net *dev,
> struct vhost_virtqueue *vq,
> vq->stats.inflight_submitted += nb_tx;
>
> out:
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_rd_unlock(vq);
> + vhost_user_iotlb_rd_unlock(vq);
>
> out_access_unlock:
> rte_spinlock_unlock(&vq->access_lock);
> @@ -3543,8 +3539,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
> goto out_access_unlock;
> }
>
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_rd_lock(vq);
> + vhost_user_iotlb_rd_lock(vq);
>
> if (unlikely(!vq->access_ok))
> if (unlikely(vring_translate(dev, vq) < 0)) {
> @@ -3603,8 +3598,7 @@ rte_vhost_dequeue_burst(int vid, uint16_t queue_id,
> vhost_queue_stats_update(dev, vq, pkts, count);
>
> out:
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_rd_unlock(vq);
> + vhost_user_iotlb_rd_unlock(vq);
>
> out_access_unlock:
> rte_spinlock_unlock(&vq->access_lock);
> @@ -4150,8 +4144,7 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t
> queue_id,
> goto out_access_unlock;
> }
>
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_rd_lock(vq);
> + vhost_user_iotlb_rd_lock(vq);
>
> if (unlikely(vq->access_ok == 0))
> if (unlikely(vring_translate(dev, vq) < 0)) {
> @@ -4215,8 +4208,7 @@ rte_vhost_async_try_dequeue_burst(int vid, uint16_t
> queue_id,
> vhost_queue_stats_update(dev, vq, pkts, count);
>
> out:
> - if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM))
> - vhost_user_iotlb_rd_unlock(vq);
> + vhost_user_iotlb_rd_unlock(vq);
>
> out_access_unlock:
> rte_spinlock_unlock(&vq->access_lock);
> --
> 2.39.1
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 7/9] vhost: annotate IOTLB lock
2023-02-07 10:45 ` [PATCH v6 7/9] vhost: annotate " David Marchand
@ 2023-02-09 8:02 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:02 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:46 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 7/9] vhost: annotate IOTLB lock
>
> The starting point for this is __vhost_iova_to_vva() which requires the
> lock to be taken. Annotate all the code leading to a call to it.
>
> vdpa and vhost_crypto code are annotated but they end up not taking
> a IOTLB lock and have been marked with a FIXME at the top level.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> 2.39.1
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 8/9] vhost: annotate vDPA device list accesses
2023-02-07 10:45 ` [PATCH v6 8/9] vhost: annotate vDPA device list accesses David Marchand
@ 2023-02-09 8:02 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:02 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:46 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 8/9] vhost: annotate vDPA device list accesses
>
> Access to vdpa_device_list must be protected with vdpa_device_list_lock
> spinlock.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> Changes since RFC v3:
> - rebased,
>
> ---
> lib/vhost/vdpa.c | 19 ++++++++++---------
> 1 file changed, 10 insertions(+), 9 deletions(-)
>
> diff --git a/lib/vhost/vdpa.c b/lib/vhost/vdpa.c
> index a430a66970..6284ea2ed1 100644
> --- a/lib/vhost/vdpa.c
> +++ b/lib/vhost/vdpa.c
> @@ -23,21 +23,22 @@
> /** Double linked list of vDPA devices. */
> TAILQ_HEAD(vdpa_device_list, rte_vdpa_device);
>
> -static struct vdpa_device_list vdpa_device_list =
> - TAILQ_HEAD_INITIALIZER(vdpa_device_list);
> +static struct vdpa_device_list vdpa_device_list__ =
> + TAILQ_HEAD_INITIALIZER(vdpa_device_list__);
> static rte_spinlock_t vdpa_device_list_lock = RTE_SPINLOCK_INITIALIZER;
> +static struct vdpa_device_list * const vdpa_device_list
> + __rte_guarded_by(&vdpa_device_list_lock) = &vdpa_device_list__;
>
> -
> -/* Unsafe, needs to be called with vdpa_device_list_lock held */
> static struct rte_vdpa_device *
> __vdpa_find_device_by_name(const char *name)
> + __rte_exclusive_locks_required(&vdpa_device_list_lock)
> {
> struct rte_vdpa_device *dev, *ret = NULL;
>
> if (name == NULL)
> return NULL;
>
> - TAILQ_FOREACH(dev, &vdpa_device_list, next) {
> + TAILQ_FOREACH(dev, vdpa_device_list, next) {
> if (!strncmp(dev->device->name, name, RTE_DEV_NAME_MAX_LEN)) {
> ret = dev;
> break;
> @@ -116,7 +117,7 @@ rte_vdpa_register_device(struct rte_device *rte_dev,
> dev->type = RTE_VHOST_VDPA_DEVICE_TYPE_NET;
> }
>
> - TAILQ_INSERT_TAIL(&vdpa_device_list, dev, next);
> + TAILQ_INSERT_TAIL(vdpa_device_list, dev, next);
> out_unlock:
> rte_spinlock_unlock(&vdpa_device_list_lock);
>
> @@ -130,11 +131,11 @@ rte_vdpa_unregister_device(struct rte_vdpa_device
> *dev)
> int ret = -1;
>
> rte_spinlock_lock(&vdpa_device_list_lock);
> - RTE_TAILQ_FOREACH_SAFE(cur_dev, &vdpa_device_list, next, tmp_dev) {
> + RTE_TAILQ_FOREACH_SAFE(cur_dev, vdpa_device_list, next, tmp_dev) {
> if (dev != cur_dev)
> continue;
>
> - TAILQ_REMOVE(&vdpa_device_list, dev, next);
> + TAILQ_REMOVE(vdpa_device_list, dev, next);
> rte_free(dev);
> ret = 0;
> break;
> @@ -336,7 +337,7 @@ vdpa_find_device(const struct rte_vdpa_device *start,
> rte_vdpa_cmp_t cmp,
>
> rte_spinlock_lock(&vdpa_device_list_lock);
> if (start == NULL)
> - dev = TAILQ_FIRST(&vdpa_device_list);
> + dev = TAILQ_FIRST(vdpa_device_list);
> else
> dev = TAILQ_NEXT(start, next);
>
> --
> 2.39.1
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 9/9] vhost: enable lock check
2023-02-07 10:45 ` [PATCH v6 9/9] vhost: enable lock check David Marchand
@ 2023-02-09 8:05 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:05 UTC (permalink / raw)
To: David Marchand, dev
Cc: maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, February 7, 2023 6:46 PM
> To: dev@dpdk.org
> Cc: maxime.coquelin@redhat.com; stephen@networkplumber.org; Xia, Chenbo
> <chenbo.xia@intel.com>; Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX
> <yuanx.wang@intel.com>; Ding, Xuan <xuan.ding@intel.com>;
> mb@smartsharesystems.com
> Subject: [PATCH v6 9/9] vhost: enable lock check
>
> Now that all locks in this library are annotated, we can enable the
> check.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
> lib/vhost/meson.build | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/lib/vhost/meson.build b/lib/vhost/meson.build
> index bc7272053b..197a51d936 100644
> --- a/lib/vhost/meson.build
> +++ b/lib/vhost/meson.build
> @@ -17,6 +17,8 @@ elif (toolchain == 'icc' and
> cc.version().version_compare('>=16.0.0'))
> endif
> dpdk_conf.set('RTE_LIBRTE_VHOST_POSTCOPY',
> cc.has_header('linux/userfaultfd.h'))
> cflags += '-fno-strict-aliasing'
> +
> +annotate_locks = true
> sources = files(
> 'fd_man.c',
> 'iotlb.c',
> --
> 2.39.1
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
After going through the whole series, I agree it does need much effort to
add all these annotations..
But I also believe it will bring more benefits :)
Thanks for the work, David!
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v6 0/9] Lock annotations
2023-02-09 7:59 ` [PATCH v6 0/9] Lock annotations Xia, Chenbo
@ 2023-02-09 8:08 ` David Marchand
2023-02-09 8:24 ` Xia, Chenbo
0 siblings, 1 reply; 110+ messages in thread
From: David Marchand @ 2023-02-09 8:08 UTC (permalink / raw)
To: Xia, Chenbo
Cc: dev, maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
Hello,
On Thu, Feb 9, 2023 at 8:59 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > Subject: [PATCH v6 0/9] Lock annotations
> >
> > vhost internals involves multiple locks to protect data access by
> > multiple threads.
> >
> > This series uses clang thread safety checks [1] to catch issues during
> > compilation: EAL spinlock, seqlock and rwlock are annotated and vhost
> > code is instrumented so that clang can statically check correctness.
> >
> > Those annotations are quite heavy to maintain because the full path of
> > code must be annotated (as can be seen in the vhost datapath code),
> > but I think it is worth using.
> >
> > This has been tested against the whole tree and some fixes are already
> > flying on the mailing list (see [2] for a list).
> >
> > If this first series is merged, I will prepare a followup series for EAL
> > and other libraries.
> >
> >
> > 1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
> > 2:
> > https://patchwork.dpdk.org/bundle/dmarchand/lock_fixes/?state=*&archive=bo
> > th
> >
> > --
> > David Marchand
> >
> > Changes since v5:
> > - rebased after lib/vhost updates (patches 5 and 7),
> >
> > Changes since v4:
> > - masked annotations from Doxygen as it seems confused with some
> > constructs,
> > - fixed typos,
> >
[snip]
> >
> >
> > David Marchand (9):
> > eal: annotate spinlock, rwlock and seqlock
> > vhost: simplify need reply handling
> > vhost: terminate when access lock is not taken
> > vhost: annotate virtqueue access lock
> > vhost: annotate async accesses
> > vhost: always take IOTLB lock
> > vhost: annotate IOTLB lock
> > vhost: annotate vDPA device list accesses
> > vhost: enable lock check
> >
> > doc/api/doxy-api.conf.in | 11 ++
> > .../prog_guide/env_abstraction_layer.rst | 24 ++++
> > doc/guides/rel_notes/release_23_03.rst | 5 +
> > drivers/meson.build | 5 +
> > lib/eal/include/generic/rte_rwlock.h | 27 +++-
> > lib/eal/include/generic/rte_spinlock.h | 31 +++--
> > lib/eal/include/meson.build | 1 +
> > lib/eal/include/rte_lock_annotations.h | 73 ++++++++++
> > lib/eal/include/rte_seqlock.h | 2 +
> > lib/eal/ppc/include/rte_spinlock.h | 3 +
> > lib/eal/x86/include/rte_rwlock.h | 4 +
> > lib/eal/x86/include/rte_spinlock.h | 9 ++
> > lib/meson.build | 5 +
> > lib/vhost/iotlb.h | 4 +
> > lib/vhost/meson.build | 2 +
> > lib/vhost/vdpa.c | 20 +--
> > lib/vhost/vhost.c | 38 ++---
> > lib/vhost/vhost.h | 34 ++++-
> > lib/vhost/vhost_crypto.c | 8 ++
> > lib/vhost/vhost_user.c | 131 ++++++++----------
> > lib/vhost/virtio_net.c | 118 ++++++++++++----
> > 21 files changed, 405 insertions(+), 150 deletions(-)
> > create mode 100644 lib/eal/include/rte_lock_annotations.h
> >
> > --
> > 2.39.1
>
> Seems one compilation error reported? Not sure it's related or not.
We discovered recently that Intel CI filters out doc/ updates in patches (?!).
https://inbox.dpdk.org/dev/20220328121758.26632-1-david.marchand@redhat.com/T/#mb42fa6342204dd01c923339ec0b1587bc0b5ac0a
So yes, it is "related" to the series, but you can ignore Intel CI
report because the reported issue is fixed since the v4 revision.
Btw, thanks for the review Chenbo!
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* RE: [PATCH v6 0/9] Lock annotations
2023-02-09 8:08 ` David Marchand
@ 2023-02-09 8:24 ` Xia, Chenbo
0 siblings, 0 replies; 110+ messages in thread
From: Xia, Chenbo @ 2023-02-09 8:24 UTC (permalink / raw)
To: David Marchand
Cc: dev, maxime.coquelin, stephen, Hu, Jiayu, Wang, YuanX, Ding, Xuan, mb
> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Thursday, February 9, 2023 4:08 PM
> To: Xia, Chenbo <chenbo.xia@intel.com>
> Cc: dev@dpdk.org; maxime.coquelin@redhat.com; stephen@networkplumber.org;
> Hu, Jiayu <jiayu.hu@intel.com>; Wang, YuanX <yuanx.wang@intel.com>; Ding,
> Xuan <xuan.ding@intel.com>; mb@smartsharesystems.com
> Subject: Re: [PATCH v6 0/9] Lock annotations
>
> Hello,
>
> On Thu, Feb 9, 2023 at 8:59 AM Xia, Chenbo <chenbo.xia@intel.com> wrote:
> > > Subject: [PATCH v6 0/9] Lock annotations
> > >
> > > vhost internals involves multiple locks to protect data access by
> > > multiple threads.
> > >
> > > This series uses clang thread safety checks [1] to catch issues during
> > > compilation: EAL spinlock, seqlock and rwlock are annotated and vhost
> > > code is instrumented so that clang can statically check correctness.
> > >
> > > Those annotations are quite heavy to maintain because the full path of
> > > code must be annotated (as can be seen in the vhost datapath code),
> > > but I think it is worth using.
> > >
> > > This has been tested against the whole tree and some fixes are already
> > > flying on the mailing list (see [2] for a list).
> > >
> > > If this first series is merged, I will prepare a followup series for
> EAL
> > > and other libraries.
> > >
> > >
> > > 1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
> > > 2:
> > >
> https://patchwork.dpdk.org/bundle/dmarchand/lock_fixes/?state=*&archive=bo
> > > th
> > >
> > > --
> > > David Marchand
> > >
> > > Changes since v5:
> > > - rebased after lib/vhost updates (patches 5 and 7),
> > >
> > > Changes since v4:
> > > - masked annotations from Doxygen as it seems confused with some
> > > constructs,
> > > - fixed typos,
> > >
>
> [snip]
>
> > >
> > >
> > > David Marchand (9):
> > > eal: annotate spinlock, rwlock and seqlock
> > > vhost: simplify need reply handling
> > > vhost: terminate when access lock is not taken
> > > vhost: annotate virtqueue access lock
> > > vhost: annotate async accesses
> > > vhost: always take IOTLB lock
> > > vhost: annotate IOTLB lock
> > > vhost: annotate vDPA device list accesses
> > > vhost: enable lock check
> > >
> > > doc/api/doxy-api.conf.in | 11 ++
> > > .../prog_guide/env_abstraction_layer.rst | 24 ++++
> > > doc/guides/rel_notes/release_23_03.rst | 5 +
> > > drivers/meson.build | 5 +
> > > lib/eal/include/generic/rte_rwlock.h | 27 +++-
> > > lib/eal/include/generic/rte_spinlock.h | 31 +++--
> > > lib/eal/include/meson.build | 1 +
> > > lib/eal/include/rte_lock_annotations.h | 73 ++++++++++
> > > lib/eal/include/rte_seqlock.h | 2 +
> > > lib/eal/ppc/include/rte_spinlock.h | 3 +
> > > lib/eal/x86/include/rte_rwlock.h | 4 +
> > > lib/eal/x86/include/rte_spinlock.h | 9 ++
> > > lib/meson.build | 5 +
> > > lib/vhost/iotlb.h | 4 +
> > > lib/vhost/meson.build | 2 +
> > > lib/vhost/vdpa.c | 20 +--
> > > lib/vhost/vhost.c | 38 ++---
> > > lib/vhost/vhost.h | 34 ++++-
> > > lib/vhost/vhost_crypto.c | 8 ++
> > > lib/vhost/vhost_user.c | 131 ++++++++---------
> -
> > > lib/vhost/virtio_net.c | 118 ++++++++++++----
> > > 21 files changed, 405 insertions(+), 150 deletions(-)
> > > create mode 100644 lib/eal/include/rte_lock_annotations.h
> > >
> > > --
> > > 2.39.1
> >
> > Seems one compilation error reported? Not sure it's related or not.
>
> We discovered recently that Intel CI filters out doc/ updates in patches
> (?!).
Oh, I remember you reported some CI issue to us but didn't realize this is
the one. Good to know it's resolved :)
Thanks,
Chenbo
> https://inbox.dpdk.org/dev/20220328121758.26632-1-
> david.marchand@redhat.com/T/#mb42fa6342204dd01c923339ec0b1587bc0b5ac0a
>
> So yes, it is "related" to the series, but you can ignore Intel CI
> report because the reported issue is fixed since the v4 revision.
>
>
> Btw, thanks for the review Chenbo!
>
>
> --
> David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
* Re: [PATCH v6 0/9] Lock annotations
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
` (9 preceding siblings ...)
2023-02-09 7:59 ` [PATCH v6 0/9] Lock annotations Xia, Chenbo
@ 2023-02-09 13:48 ` David Marchand
10 siblings, 0 replies; 110+ messages in thread
From: David Marchand @ 2023-02-09 13:48 UTC (permalink / raw)
To: dev
Cc: maxime.coquelin, stephen, chenbo.xia, jiayu.hu, yuanx.wang,
xuan.ding, mb, Tyler Retzlaff, Thomas Monjalon
On Tue, Feb 7, 2023 at 11:45 AM David Marchand
<david.marchand@redhat.com> wrote:
>
> vhost internals involves multiple locks to protect data access by
> multiple threads.
>
> This series uses clang thread safety checks [1] to catch issues during
> compilation: EAL spinlock, seqlock and rwlock are annotated and vhost
> code is instrumented so that clang can statically check correctness.
>
> Those annotations are quite heavy to maintain because the full path of
> code must be annotated (as can be seen in the vhost datapath code),
> but I think it is worth using.
>
> This has been tested against the whole tree and some fixes are already
> flying on the mailing list (see [2] for a list).
>
> If this first series is merged, I will prepare a followup series for EAL
> and other libraries.
>
>
> 1: https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
> 2: https://patchwork.dpdk.org/bundle/dmarchand/lock_fixes/?state=*&archive=both
>
We have been discussing the annotations naming convention in another
thread (I copied involved people).
I saw no objection to the approach of simply prefixing with __rte the
clang attributes.
I'll stick to this de facto convention we have in dpdk.
If any objection arises, I will do the necessary adjustments before -rc2.
Series applied, thanks to all reviewers!
--
David Marchand
^ permalink raw reply [flat|nested] 110+ messages in thread
end of thread, other threads:[~2023-02-09 13:49 UTC | newest]
Thread overview: 110+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-28 12:17 [RFC PATCH 0/5] vhost lock annotations David Marchand
2022-03-28 12:17 ` [RFC PATCH 1/5] vhost: fix missing virtqueue lock protection David Marchand
2022-03-28 12:17 ` [RFC PATCH 2/5] vhost: annotate virtqueue access lock David Marchand
2022-03-28 12:17 ` [RFC PATCH 3/5] vhost: fix async access David Marchand
2022-03-28 12:17 ` [RFC PATCH 4/5] vhost: annotate async locking requirement David Marchand
2022-03-28 12:17 ` [RFC PATCH 5/5] vhost: annotate IOTLB locks David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 1/9] vhost: fix missing virtqueue lock protection David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 2/9] eal: annotate spinlock and rwlock David Marchand
2022-03-31 9:22 ` David Marchand
2022-04-04 6:21 ` Stephen Hemminger
2022-04-07 8:20 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 3/9] vhost: annotate virtqueue access lock David Marchand
2022-04-07 1:40 ` Hu, Jiayu
2022-04-07 7:03 ` David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 4/9] vhost: fix async access David Marchand
2022-03-31 8:00 ` Maxime Coquelin
2022-03-31 10:23 ` Hu, Jiayu
2022-04-04 6:57 ` Pai G, Sunil
2022-03-30 13:49 ` [RFC PATCH v2 5/9] vhost: annotate async acesses David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 6/9] vhost: annotate need reply handling David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 7/9] vhost: annotate VDPA device list accesses David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 8/9] vhost: annotate IOTLB locks David Marchand
2022-03-30 13:49 ` [RFC PATCH v2 9/9] vhost: enable lock check David Marchand
2022-03-30 14:03 ` [RFC PATCH v2 0/9] vhost lock annotations David Marchand
2022-03-30 14:37 ` Ali Alnubani
2022-04-05 7:11 ` David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 0/8] " David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 1/8] eal: annotate spinlock and rwlock David Marchand
2022-04-21 13:48 ` Maxime Coquelin
2022-04-28 12:16 ` David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 2/8] vhost: annotate virtqueue access lock David Marchand
2022-04-21 15:25 ` Maxime Coquelin
2022-04-22 9:49 ` David Marchand
2022-04-11 11:00 ` [RFC PATCH v3 3/8] vhost: fix async access David Marchand
2022-04-21 19:21 ` Maxime Coquelin
2022-05-17 13:24 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 4/8] vhost: annotate async accesses David Marchand
2022-04-22 7:20 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 5/8] vhost: annotate need reply handling David Marchand
2022-04-22 7:25 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 6/8] vhost: annotate vDPA device list accesses David Marchand
2022-04-22 7:26 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 7/8] vhost: annotate IOTLB locks David Marchand
2022-04-22 7:46 ` Maxime Coquelin
2022-04-11 11:00 ` [RFC PATCH v3 8/8] vhost: enable lock check David Marchand
2022-04-22 7:47 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 0/9] vhost lock annotations David Marchand
2023-01-19 18:46 ` [PATCH v4 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
2023-01-19 19:42 ` Stephen Hemminger
2023-01-19 20:39 ` Tyler Retzlaff
2023-01-19 21:16 ` David Marchand
2023-01-19 21:50 ` Tyler Retzlaff
2023-01-26 12:18 ` David Marchand
2023-01-19 20:55 ` David Marchand
2023-01-19 19:43 ` Stephen Hemminger
2023-01-31 16:18 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 2/9] vhost: simplify need reply handling David Marchand
2023-01-31 16:41 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 3/9] vhost: terminate when access lock is not taken David Marchand
2023-01-31 16:47 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 4/9] vhost: annotate virtqueue access lock David Marchand
2023-01-31 16:50 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 5/9] vhost: annotate async accesses David Marchand
2023-01-31 16:54 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 6/9] vhost: always take IOTLB lock David Marchand
2023-01-31 16:59 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 7/9] vhost: annotate " David Marchand
2023-01-31 17:05 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 8/9] vhost: annotate vDPA device list accesses David Marchand
2023-01-31 17:08 ` Maxime Coquelin
2023-01-19 18:46 ` [PATCH v4 9/9] vhost: enable lock check David Marchand
2023-01-31 17:14 ` Maxime Coquelin
2023-01-19 19:20 ` [PATCH v4 0/9] vhost lock annotations Morten Brørup
2023-02-01 11:14 ` [PATCH v5 0/9] Lock annotations David Marchand
2023-02-01 11:14 ` [PATCH v5 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
2023-02-01 12:32 ` David Marchand
2023-02-06 1:01 ` Tu, Lijuan
2023-02-06 8:12 ` David Marchand
2023-02-01 11:14 ` [PATCH v5 2/9] vhost: simplify need reply handling David Marchand
2023-02-01 11:14 ` [PATCH v5 3/9] vhost: terminate when access lock is not taken David Marchand
2023-02-01 11:14 ` [PATCH v5 4/9] vhost: annotate virtqueue access lock David Marchand
2023-02-01 11:14 ` [PATCH v5 5/9] vhost: annotate async accesses David Marchand
2023-02-01 11:14 ` [PATCH v5 6/9] vhost: always take IOTLB lock David Marchand
2023-02-01 11:14 ` [PATCH v5 7/9] vhost: annotate " David Marchand
2023-02-01 11:14 ` [PATCH v5 8/9] vhost: annotate vDPA device list accesses David Marchand
2023-02-01 11:14 ` [PATCH v5 9/9] vhost: enable lock check David Marchand
2023-02-07 10:45 ` [PATCH v6 0/9] Lock annotations David Marchand
2023-02-07 10:45 ` [PATCH v6 1/9] eal: annotate spinlock, rwlock and seqlock David Marchand
2023-02-09 8:00 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 2/9] vhost: simplify need reply handling David Marchand
2023-02-09 8:00 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 3/9] vhost: terminate when access lock is not taken David Marchand
2023-02-09 8:01 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 4/9] vhost: annotate virtqueue access lock David Marchand
2023-02-09 8:01 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 5/9] vhost: annotate async accesses David Marchand
2023-02-09 8:01 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 6/9] vhost: always take IOTLB lock David Marchand
2023-02-09 8:01 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 7/9] vhost: annotate " David Marchand
2023-02-09 8:02 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 8/9] vhost: annotate vDPA device list accesses David Marchand
2023-02-09 8:02 ` Xia, Chenbo
2023-02-07 10:45 ` [PATCH v6 9/9] vhost: enable lock check David Marchand
2023-02-09 8:05 ` Xia, Chenbo
2023-02-09 7:59 ` [PATCH v6 0/9] Lock annotations Xia, Chenbo
2023-02-09 8:08 ` David Marchand
2023-02-09 8:24 ` Xia, Chenbo
2023-02-09 13:48 ` David Marchand
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).