From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 614C2A04DD for ; Wed, 18 Nov 2020 17:37:23 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 518574C90; Wed, 18 Nov 2020 17:37:22 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by dpdk.org (Postfix) with ESMTP id AC62B160 for ; Wed, 18 Nov 2020 17:37:20 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605717439; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aorJzxJkgv0VzCnuyotViRuJZJ+QLNJ3KZCSItWclv4=; b=HBzpnEb15L5zoMnEPh8UR3aWqF/kNXxgL5K7lDEvXI3cmbTLwdt2mRtzRUPmlfepelQb81 ME8ZonkoWPlEJGoJ634QrGipiI2mduVpE3pOGXJJYd2hOjB98Pu8OFggBwuHemCDRVI5gl vJi1lO3Q1mJkAxNKuIGlRnEbdcwRDCE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-257-mbg-vkprNmiOT69YCTnXbQ-1; Wed, 18 Nov 2020 11:37:15 -0500 X-MC-Unique: mbg-vkprNmiOT69YCTnXbQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2FE7A8010FC; Wed, 18 Nov 2020 16:36:47 +0000 (UTC) Received: from rh.redhat.com (ovpn-113-249.ams2.redhat.com [10.36.113.249]) by smtp.corp.redhat.com (Postfix) with ESMTP id 919615C1A3; Wed, 18 Nov 2020 16:36:45 +0000 (UTC) From: Kevin Traynor To: Honnappa Nagarahalli Cc: Ola Liljedahl , Phil Yang , Konstantin Ananyev , dpdk stable Date: Wed, 18 Nov 2020 16:35:03 +0000 Message-Id: <20201118163558.1101823-17-ktraynor@redhat.com> In-Reply-To: <20201118163558.1101823-1-ktraynor@redhat.com> References: <20201118163558.1101823-1-ktraynor@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ktraynor@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII" Subject: [dpdk-stable] patch 'ethdev: fix memory ordering for callback functions' has been queued to LTS release 18.11.11 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" Hi, FYI, your patch has been queued to LTS release 18.11.11 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 11/24/20. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/kevintraynor/dpdk-stable-queue This queued commit can be viewed at: https://github.com/kevintraynor/dpdk-stable-queue/commit/79711c90de1314b9b012da0815db2e8ac57f14ed Thanks. Kevin. --- >From 79711c90de1314b9b012da0815db2e8ac57f14ed Mon Sep 17 00:00:00 2001 From: Honnappa Nagarahalli Date: Tue, 13 Oct 2020 11:25:37 -0500 Subject: [PATCH] ethdev: fix memory ordering for callback functions [ upstream commit 2b69bd1179162f69c0a90d5b09309498d362b3d7 ] Call back functions are registered on the control plane. They are accessed from the data plane. Hence, correct memory orderings should be used to avoid race conditions. Fixes: 4dc294158cac ("ethdev: support optional Rx and Tx callbacks") Fixes: c8231c63ddcb ("ethdev: insert Rx callback as head of list") Signed-off-by: Honnappa Nagarahalli Reviewed-by: Ola Liljedahl Reviewed-by: Phil Yang Acked-by: Konstantin Ananyev --- lib/librte_ethdev/rte_ethdev.c | 28 ++++++++++++++++++----- lib/librte_ethdev/rte_ethdev.h | 42 ++++++++++++++++++++++++++-------- 2 files changed, 55 insertions(+), 15 deletions(-) diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c index 336ad1b83a..3f3f9505b2 100644 --- a/lib/librte_ethdev/rte_ethdev.c +++ b/lib/librte_ethdev/rte_ethdev.c @@ -3825,10 +3825,18 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, if (!tail) { - rte_eth_devices[port_id].post_rx_burst_cbs[queue_id] = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n( + &rte_eth_devices[port_id].post_rx_burst_cbs[queue_id], + cb, __ATOMIC_RELEASE); } else { while (tail->next) tail = tail->next; - tail->next = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n(&tail->next, cb, __ATOMIC_RELEASE); } rte_spinlock_unlock(&rte_eth_rx_cb_lock); @@ -3903,10 +3911,18 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, if (!tail) { - rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id] = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n( + &rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id], + cb, __ATOMIC_RELEASE); } else { while (tail->next) tail = tail->next; - tail->next = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n(&tail->next, cb, __ATOMIC_RELEASE); } rte_spinlock_unlock(&rte_eth_tx_cb_lock); @@ -3939,5 +3955,5 @@ rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, if (cb == user_cb) { /* Remove the user cb from the callback list. */ - *prev_cb = cb->next; + __atomic_store_n(prev_cb, cb->next, __ATOMIC_RELAXED); ret = 0; break; @@ -3973,5 +3989,5 @@ rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id, if (cb == user_cb) { /* Remove the user cb from the callback list. */ - *prev_cb = cb->next; + __atomic_store_n(prev_cb, cb->next, __ATOMIC_RELAXED); ret = 0; break; diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h index 7b4221d7b4..e1f299c993 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h @@ -3216,5 +3216,6 @@ struct rte_eth_rxtx_callback; * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return @@ -3245,5 +3246,6 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return @@ -3273,5 +3275,6 @@ rte_eth_add_first_rx_callback(uint16_t port_id, uint16_t queue_id, * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return @@ -3298,5 +3301,7 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, * * - After a short delay - where the delay is sufficient to allow any - * in-flight callbacks to complete. + * in-flight callbacks to complete. Alternately, the RCU mechanism can be + * used to detect when data plane threads have ceased referencing the + * callback memory. * * @param port_id @@ -3331,5 +3336,7 @@ int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, * * - After a short delay - where the delay is sufficient to allow any - * in-flight callbacks to complete. + * in-flight callbacks to complete. Alternately, the RCU mechanism can be + * used to detect when data plane threads have ceased referencing the + * callback memory. * * @param port_id @@ -3881,8 +3888,16 @@ rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, #ifdef RTE_ETHDEV_RXTX_CALLBACKS - if (unlikely(dev->post_rx_burst_cbs[queue_id] != NULL)) { - struct rte_eth_rxtx_callback *cb = - dev->post_rx_burst_cbs[queue_id]; + struct rte_eth_rxtx_callback *cb; + /* __ATOMIC_RELEASE memory order was used when the + * call back was inserted into the list. + * Since there is a clear dependency between loading + * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is + * not required. + */ + cb = __atomic_load_n(&dev->post_rx_burst_cbs[queue_id], + __ATOMIC_RELAXED); + + if (unlikely(cb != NULL)) { do { nb_rx = cb->fn.rx(port_id, queue_id, rx_pkts, nb_rx, @@ -4145,5 +4160,14 @@ rte_eth_tx_burst(uint16_t port_id, uint16_t queue_id, #ifdef RTE_ETHDEV_RXTX_CALLBACKS - struct rte_eth_rxtx_callback *cb = dev->pre_tx_burst_cbs[queue_id]; + struct rte_eth_rxtx_callback *cb; + + /* __ATOMIC_RELEASE memory order was used when the + * call back was inserted into the list. + * Since there is a clear dependency between loading + * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is + * not required. + */ + cb = __atomic_load_n(&dev->pre_tx_burst_cbs[queue_id], + __ATOMIC_RELAXED); if (unlikely(cb != NULL)) { -- 2.26.2 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2020-11-18 16:33:38.296348094 +0000 +++ 0017-ethdev-fix-memory-ordering-for-callback-functions.patch 2020-11-18 16:33:37.924215059 +0000 @@ -1 +1 @@ -From 2b69bd1179162f69c0a90d5b09309498d362b3d7 Mon Sep 17 00:00:00 2001 +From 79711c90de1314b9b012da0815db2e8ac57f14ed Mon Sep 17 00:00:00 2001 @@ -5,0 +6,2 @@ +[ upstream commit 2b69bd1179162f69c0a90d5b09309498d362b3d7 ] + @@ -12 +13,0 @@ -Cc: stable@dpdk.org @@ -24 +25 @@ -index 2198548663..7eaeccc19d 100644 +index 336ad1b83a..3f3f9505b2 100644 @@ -27 +28 @@ -@@ -4611,10 +4611,18 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3825,10 +3825,18 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, @@ -48 +49 @@ -@@ -4701,10 +4709,18 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3903,10 +3911,18 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, @@ -69 +70 @@ -@@ -4737,5 +4753,5 @@ rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3939,5 +3955,5 @@ rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, @@ -76 +77 @@ -@@ -4771,5 +4787,5 @@ rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3973,5 +3989,5 @@ rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id, @@ -84 +85 @@ -index f07dbc4d26..c37b3a1a81 100644 +index 7b4221d7b4..e1f299c993 100644 @@ -87 +88 @@ -@@ -3912,5 +3912,6 @@ struct rte_eth_rxtx_callback; +@@ -3216,5 +3216,6 @@ struct rte_eth_rxtx_callback; @@ -95 +96 @@ -@@ -3941,5 +3942,6 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3245,5 +3246,6 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, @@ -103 +104 @@ -@@ -3969,5 +3971,6 @@ rte_eth_add_first_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3273,5 +3275,6 @@ rte_eth_add_first_rx_callback(uint16_t port_id, uint16_t queue_id, @@ -111 +112 @@ -@@ -3994,5 +3997,7 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3298,5 +3301,7 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, @@ -120 +121 @@ -@@ -4027,5 +4032,7 @@ int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3331,5 +3336,7 @@ int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, @@ -129 +130 @@ -@@ -4688,8 +4695,16 @@ rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, +@@ -3881,8 +3888,16 @@ rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, @@ -149 +150 @@ -@@ -4954,5 +4969,14 @@ rte_eth_tx_burst(uint16_t port_id, uint16_t queue_id, +@@ -4145,5 +4160,14 @@ rte_eth_tx_burst(uint16_t port_id, uint16_t queue_id,