From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 781C4A04DD for ; Wed, 28 Oct 2020 11:55:52 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 70B11CAB7; Wed, 28 Oct 2020 11:55:51 +0100 (CET) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by dpdk.org (Postfix) with ESMTP id 43BFDCAB7 for ; Wed, 28 Oct 2020 11:55:50 +0100 (CET) Received: by mail-wr1-f44.google.com with SMTP id a9so1359707wrg.12 for ; Wed, 28 Oct 2020 03:55:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZV3vtXG+aUB7DOT+ei00v4K6J7ApbEyKR4lN2JbqLdc=; b=U2un03gIKIpGjTME3pdUb0TV5wV+IIM2/1UZdF2oo3OB4b3Nl4CDfEefRz/3yAwxiI 9po8/P3UNwnNN29Ibl988avj4/Nmlvxtcdptg10kWRdIe2XcHz5/YX62BzaHxaGKyhhB AfeurlBJF/G0OgM1R+Lz0tfvPkTYt9LI928ogciYPDy9/HNXf4PAzU68MmThesIaDbrP iTkWn0o1PTEhmf5M1JPFbbEQRq3lrA16QpIkEyKJIEzkthSBaNxNP4x/YrndV3nSy0xn 0KCw1++CVlT6EiM4kcyo3M0bJV9sF/ZaZizYou35Cczt+4e86OCQqxOp6ryHZpIihfZ8 Uiyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZV3vtXG+aUB7DOT+ei00v4K6J7ApbEyKR4lN2JbqLdc=; b=sU35h7XCfO/z8WPyeWLnViGT+V7zQNs6rHfiIQjJz2Lx12t5t7XFPnHSaDzvX27XCo INJ9BHXb1oiFJpOUFXdNtddED4P8h7RuVWSh7hSRhuVY5ra63ct0zJ9s/BdyBsvk/zt9 ToCm/1OQRDXi617s9FLaSJpCJkI7NhCY9I6pTQiutLB2rmNdM5hn8E77yIJ5nBpPueZc DfmfpOuVnn2UOJDaA6mJisKKa8E7G5Po+e98rLAhQ51Pt6DERFuw0d1jlIwt8i7AZf4t KzmraKFm4e/37I4e2HEraVdgf199Z1Hy7vYZq6qDOYmiaNxG/fVTM8XXTBEfFQ8ws9m5 NT4A== X-Gm-Message-State: AOAM532pVx8HsqdJuuLkk6+BKTQdVMNDNsWdanUSNLXTEzkC6W+kPqAt InIx6vgle0dzMdkMOUJtYpYpz+TE1FW8Dd7u X-Google-Smtp-Source: ABdhPJyJgkkRXeUhdBOBUFqNhXi+LuWqCTHigOkbeEkEfkUyVWTGOxbthTphbPbhvKM4Tl6D6vXLpw== X-Received: by 2002:a5d:4dc7:: with SMTP id f7mr7499223wru.375.1603882548950; Wed, 28 Oct 2020 03:55:48 -0700 (PDT) Received: from localhost ([88.98.246.218]) by smtp.gmail.com with ESMTPSA id l6sm6518151wrt.7.2020.10.28.03.55.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Oct 2020 03:55:48 -0700 (PDT) From: luca.boccassi@gmail.com To: Honnappa Nagarahalli Cc: Ola Liljedahl , Phil Yang , Konstantin Ananyev , dpdk stable Date: Wed, 28 Oct 2020 10:45:40 +0000 Message-Id: <20201028104606.3504127-181-luca.boccassi@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201028104606.3504127-1-luca.boccassi@gmail.com> References: <20201028104606.3504127-1-luca.boccassi@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-stable] patch 'ethdev: fix memory ordering for callback functions' has been queued to stable release 19.11.6 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" Hi, FYI, your patch has been queued to stable release 19.11.6 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 10/30/20. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Thanks. Luca Boccassi --- >From 67ba3dd4691d213d5d8d3988384edfa90bfc747d Mon Sep 17 00:00:00 2001 From: Honnappa Nagarahalli Date: Tue, 13 Oct 2020 11:25:37 -0500 Subject: [PATCH] ethdev: fix memory ordering for callback functions [ upstream commit 2b69bd1179162f69c0a90d5b09309498d362b3d7 ] Call back functions are registered on the control plane. They are accessed from the data plane. Hence, correct memory orderings should be used to avoid race conditions. Fixes: 4dc294158cac ("ethdev: support optional Rx and Tx callbacks") Fixes: c8231c63ddcb ("ethdev: insert Rx callback as head of list") Signed-off-by: Honnappa Nagarahalli Reviewed-by: Ola Liljedahl Reviewed-by: Phil Yang Acked-by: Konstantin Ananyev --- lib/librte_ethdev/rte_ethdev.c | 28 ++++++++++++++++++----- lib/librte_ethdev/rte_ethdev.h | 42 ++++++++++++++++++++++++++-------- 2 files changed, 55 insertions(+), 15 deletions(-) diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c index 984b5caa9d..d74429a9e7 100644 --- a/lib/librte_ethdev/rte_ethdev.c +++ b/lib/librte_ethdev/rte_ethdev.c @@ -4435,12 +4435,20 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, rte_eth_devices[port_id].post_rx_burst_cbs[queue_id]; if (!tail) { - rte_eth_devices[port_id].post_rx_burst_cbs[queue_id] = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n( + &rte_eth_devices[port_id].post_rx_burst_cbs[queue_id], + cb, __ATOMIC_RELEASE); } else { while (tail->next) tail = tail->next; - tail->next = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n(&tail->next, cb, __ATOMIC_RELEASE); } rte_spinlock_unlock(&rte_eth_rx_cb_lock); @@ -4521,12 +4529,20 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id]; if (!tail) { - rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id] = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n( + &rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id], + cb, __ATOMIC_RELEASE); } else { while (tail->next) tail = tail->next; - tail->next = cb; + /* Stores to cb->fn and cb->param should complete before + * cb is visible to data plane. + */ + __atomic_store_n(&tail->next, cb, __ATOMIC_RELEASE); } rte_spinlock_unlock(&rte_eth_tx_cb_lock); @@ -4557,7 +4573,7 @@ rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, cb = *prev_cb; if (cb == user_cb) { /* Remove the user cb from the callback list. */ - *prev_cb = cb->next; + __atomic_store_n(prev_cb, cb->next, __ATOMIC_RELAXED); ret = 0; break; } @@ -4591,7 +4607,7 @@ rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id, cb = *prev_cb; if (cb == user_cb) { /* Remove the user cb from the callback list. */ - *prev_cb = cb->next; + __atomic_store_n(prev_cb, cb->next, __ATOMIC_RELAXED); ret = 0; break; } diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h index d1a593ad11..51f9f2569f 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h @@ -3612,7 +3612,8 @@ struct rte_eth_rxtx_callback; * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return * NULL on error. @@ -3641,7 +3642,8 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return * NULL on error. @@ -3669,7 +3671,8 @@ rte_eth_add_first_rx_callback(uint16_t port_id, uint16_t queue_id, * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the - * callback function on this port and queue. + * callback function on this port and queue. Inter-thread synchronization + * of any user data changes is the responsibility of the user. * * @return * NULL on error. @@ -3694,7 +3697,9 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, * on that queue. * * - After a short delay - where the delay is sufficient to allow any - * in-flight callbacks to complete. + * in-flight callbacks to complete. Alternately, the RCU mechanism can be + * used to detect when data plane threads have ceased referencing the + * callback memory. * * @param port_id * The port identifier of the Ethernet device. @@ -3727,7 +3732,9 @@ int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, * on that queue. * * - After a short delay - where the delay is sufficient to allow any - * in-flight callbacks to complete. + * in-flight callbacks to complete. Alternately, the RCU mechanism can be + * used to detect when data plane threads have ceased referencing the + * callback memory. * * @param port_id * The port identifier of the Ethernet device. @@ -4388,10 +4395,18 @@ rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, rx_pkts, nb_pkts); #ifdef RTE_ETHDEV_RXTX_CALLBACKS - if (unlikely(dev->post_rx_burst_cbs[queue_id] != NULL)) { - struct rte_eth_rxtx_callback *cb = - dev->post_rx_burst_cbs[queue_id]; + struct rte_eth_rxtx_callback *cb; + /* __ATOMIC_RELEASE memory order was used when the + * call back was inserted into the list. + * Since there is a clear dependency between loading + * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is + * not required. + */ + cb = __atomic_load_n(&dev->post_rx_burst_cbs[queue_id], + __ATOMIC_RELAXED); + + if (unlikely(cb != NULL)) { do { nb_rx = cb->fn.rx(port_id, queue_id, rx_pkts, nb_rx, nb_pkts, cb->param); @@ -4652,7 +4667,16 @@ rte_eth_tx_burst(uint16_t port_id, uint16_t queue_id, #endif #ifdef RTE_ETHDEV_RXTX_CALLBACKS - struct rte_eth_rxtx_callback *cb = dev->pre_tx_burst_cbs[queue_id]; + struct rte_eth_rxtx_callback *cb; + + /* __ATOMIC_RELEASE memory order was used when the + * call back was inserted into the list. + * Since there is a clear dependency between loading + * cb and cb->fn/cb->next, __ATOMIC_ACQUIRE memory order is + * not required. + */ + cb = __atomic_load_n(&dev->pre_tx_burst_cbs[queue_id], + __ATOMIC_RELAXED); if (unlikely(cb != NULL)) { do { -- 2.20.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2020-10-28 10:35:17.325166133 +0000 +++ 0181-ethdev-fix-memory-ordering-for-callback-functions.patch 2020-10-28 10:35:11.792834263 +0000 @@ -1,15 +1,16 @@ -From 2b69bd1179162f69c0a90d5b09309498d362b3d7 Mon Sep 17 00:00:00 2001 +From 67ba3dd4691d213d5d8d3988384edfa90bfc747d Mon Sep 17 00:00:00 2001 From: Honnappa Nagarahalli Date: Tue, 13 Oct 2020 11:25:37 -0500 Subject: [PATCH] ethdev: fix memory ordering for callback functions +[ upstream commit 2b69bd1179162f69c0a90d5b09309498d362b3d7 ] + Call back functions are registered on the control plane. They are accessed from the data plane. Hence, correct memory orderings should be used to avoid race conditions. Fixes: 4dc294158cac ("ethdev: support optional Rx and Tx callbacks") Fixes: c8231c63ddcb ("ethdev: insert Rx callback as head of list") -Cc: stable@dpdk.org Signed-off-by: Honnappa Nagarahalli Reviewed-by: Ola Liljedahl @@ -21,10 +22,10 @@ 2 files changed, 55 insertions(+), 15 deletions(-) diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c -index 2198548663..7eaeccc19d 100644 +index 984b5caa9d..d74429a9e7 100644 --- a/lib/librte_ethdev/rte_ethdev.c +++ b/lib/librte_ethdev/rte_ethdev.c -@@ -4610,12 +4610,20 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -4435,12 +4435,20 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, rte_eth_devices[port_id].post_rx_burst_cbs[queue_id]; if (!tail) { @@ -47,7 +48,7 @@ } rte_spinlock_unlock(&rte_eth_rx_cb_lock); -@@ -4700,12 +4708,20 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, +@@ -4521,12 +4529,20 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, rte_eth_devices[port_id].pre_tx_burst_cbs[queue_id]; if (!tail) { @@ -70,7 +71,7 @@ } rte_spinlock_unlock(&rte_eth_tx_cb_lock); -@@ -4736,7 +4752,7 @@ rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -4557,7 +4573,7 @@ rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, cb = *prev_cb; if (cb == user_cb) { /* Remove the user cb from the callback list. */ @@ -79,7 +80,7 @@ ret = 0; break; } -@@ -4770,7 +4786,7 @@ rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id, +@@ -4591,7 +4607,7 @@ rte_eth_remove_tx_callback(uint16_t port_id, uint16_t queue_id, cb = *prev_cb; if (cb == user_cb) { /* Remove the user cb from the callback list. */ @@ -89,10 +90,10 @@ break; } diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h -index f07dbc4d26..c37b3a1a81 100644 +index d1a593ad11..51f9f2569f 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h -@@ -3911,7 +3911,8 @@ struct rte_eth_rxtx_callback; +@@ -3612,7 +3612,8 @@ struct rte_eth_rxtx_callback; * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the @@ -102,7 +103,7 @@ * * @return * NULL on error. -@@ -3940,7 +3941,8 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3641,7 +3642,8 @@ rte_eth_add_rx_callback(uint16_t port_id, uint16_t queue_id, * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the @@ -112,7 +113,7 @@ * * @return * NULL on error. -@@ -3968,7 +3970,8 @@ rte_eth_add_first_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3669,7 +3671,8 @@ rte_eth_add_first_rx_callback(uint16_t port_id, uint16_t queue_id, * The callback function * @param user_param * A generic pointer parameter which will be passed to each invocation of the @@ -122,7 +123,7 @@ * * @return * NULL on error. -@@ -3993,7 +3996,9 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3694,7 +3697,9 @@ rte_eth_add_tx_callback(uint16_t port_id, uint16_t queue_id, * on that queue. * * - After a short delay - where the delay is sufficient to allow any @@ -133,7 +134,7 @@ * * @param port_id * The port identifier of the Ethernet device. -@@ -4026,7 +4031,9 @@ int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, +@@ -3727,7 +3732,9 @@ int rte_eth_remove_rx_callback(uint16_t port_id, uint16_t queue_id, * on that queue. * * - After a short delay - where the delay is sufficient to allow any @@ -144,7 +145,7 @@ * * @param port_id * The port identifier of the Ethernet device. -@@ -4687,10 +4694,18 @@ rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, +@@ -4388,10 +4395,18 @@ rte_eth_rx_burst(uint16_t port_id, uint16_t queue_id, rx_pkts, nb_pkts); #ifdef RTE_ETHDEV_RXTX_CALLBACKS @@ -166,7 +167,7 @@ do { nb_rx = cb->fn.rx(port_id, queue_id, rx_pkts, nb_rx, nb_pkts, cb->param); -@@ -4953,7 +4968,16 @@ rte_eth_tx_burst(uint16_t port_id, uint16_t queue_id, +@@ -4652,7 +4667,16 @@ rte_eth_tx_burst(uint16_t port_id, uint16_t queue_id, #endif #ifdef RTE_ETHDEV_RXTX_CALLBACKS