From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7C7E4A0547 for ; Mon, 16 Aug 2021 22:37:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6EB3541150; Mon, 16 Aug 2021 22:37:53 +0200 (CEST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mails.dpdk.org (Postfix) with ESMTP id 8BD6B41150 for ; Mon, 16 Aug 2021 22:37:52 +0200 (CEST) Received: by mail-pl1-f180.google.com with SMTP id u15so6593341plg.13 for ; Mon, 16 Aug 2021 13:37:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=6SZ199Vj0x9Shl23gQ16qTHITdpOAO4MzA4E5V60opQ=; b=QfE6EQEOuZqFCT+T1OG6MEFkK7+/rKo3Hd635ULvsmA8OzN90TmPd6ltm4Xo3mnzc1 YHkYcl6VKq7DU2JyEJkdDbty0G6CLnlnlsqtOb8lKpHClyfP7V2i096FU5/3SAe9HzVK y+z2VsOQ5Oc9ev9vfNlVDpQis6GrwSjlemxzQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=6SZ199Vj0x9Shl23gQ16qTHITdpOAO4MzA4E5V60opQ=; b=iPOOzBDvsUXmkbhekroFNplbLzhifMj/wpOwmTgh2u5g/qQnqmrh4G8i81fz6HwVFA gvEYSZtrDPA7SXNNZdwipmTxtdg1i5Ss3SwLvwAW/Q72hDiDHdI9smzb+d8UE5xoTWF7 fmjMDe1rZJ7SZbiui7VROFTzKcgALz3yqLL6SZXzZykOzLMsIc/i3GbrI6U2dgn62g2A VCsnH5Rs8+3nnPXSYtqQHxYq/7wfkVrjbn5iwlh+S+p2Mj6dtZehyesVJugd4g9LuJzC fLKh2d/ytZyZhT1YNEm1bURmGIE+AUyfPIbtPrlhQjnlqd2x11y4MqyksZ2hdzPoRzFz wXQg== X-Gm-Message-State: AOAM5315ywCECqZu+aC+fImCGMy+XNfzY3Jany4GhKFI6t/+u49jKPFB fqPqPmGpRgzfRSGQXZ0NOBQ5ByLUjZNEmB1YmWUZnKuxF6sXxAiilS1OfCHWBd9Be9C/kdRtxCe aJbgZlZWREIjuHlcToYpjjAZB4rgReit2NGjpI1uTd0cL0fj0DZe+QqBbWkZtw08iLhmlzX4= X-Google-Smtp-Source: ABdhPJx9hIrsoUZcwvQsbD/WkZxl67g6xnCeFYEEdAxVNTAIJlIrPDA2NmU7mjBep+uma7jGWWV97A== X-Received: by 2002:a17:90a:a389:: with SMTP id x9mr447707pjp.167.1629146271354; Mon, 16 Aug 2021 13:37:51 -0700 (PDT) Received: from lrichardson-VirtualBox.dhcp.broadcom.net ([192.19.231.250]) by smtp.gmail.com with ESMTPSA id d20sm59404pfl.32.2021.08.16.13.37.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Aug 2021 13:37:50 -0700 (PDT) From: Lance Richardson To: stable@dpdk.org Cc: Ajit Khaparde , Ruifeng Wang Date: Mon, 16 Aug 2021 16:37:02 -0400 Message-Id: <20210816203702.7235-4-lance.richardson@broadcom.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210816203702.7235-1-lance.richardson@broadcom.com> References: <20210816203702.7235-1-lance.richardson@broadcom.com> MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="000000000000d1587405c9b32c3b" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [dpdk-stable] [PATCH 19.11 3/3] net/bnxt: fix missing barriers in completion handling X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" --000000000000d1587405c9b32c3b Content-Transfer-Encoding: 8bit [ upstream commit 5ed30db87fa810e210fec27a417a3f0d69f7b425 ] Ensure that Rx/Tx/Async completion entry fields are accessed only after the completion's valid flag has been loaded and verified. This is needed for correct operation on systems that use relaxed memory consistency models. Fixes: 2eb53b134aae ("net/bnxt: add initial Rx code") Fixes: 6eb3cc2294fd ("net/bnxt: add initial Tx code") Cc: stable@dpdk.org Signed-off-by: Lance Richardson Reviewed-by: Ajit Khaparde Reviewed-by: Ruifeng Wang --- drivers/net/bnxt/bnxt_cpr.h | 40 ++++++++++++++++++++++------ drivers/net/bnxt/bnxt_ethdev.c | 14 +++++----- drivers/net/bnxt/bnxt_irq.c | 6 ++--- drivers/net/bnxt/bnxt_rxr.c | 9 ++++--- drivers/net/bnxt/bnxt_rxtx_vec_sse.c | 11 +++++--- drivers/net/bnxt/bnxt_txr.c | 2 +- 6 files changed, 56 insertions(+), 26 deletions(-) diff --git a/drivers/net/bnxt/bnxt_cpr.h b/drivers/net/bnxt/bnxt_cpr.h index c769bde61..26cbf97fb 100644 --- a/drivers/net/bnxt/bnxt_cpr.h +++ b/drivers/net/bnxt/bnxt_cpr.h @@ -8,17 +8,10 @@ #include #include +#include "hsi_struct_def_dpdk.h" struct bnxt_db_info; -#define CMP_VALID(cmp, raw_cons, ring) \ - (!!(rte_le_to_cpu_32(((struct cmpl_base *)(cmp))->info3_v) & \ - CMPL_BASE_V) == !((raw_cons) & ((ring)->ring_size))) - -#define CMPL_VALID(cmp, v) \ - (!!(rte_le_to_cpu_32(((struct cmpl_base *)(cmp))->info3_v) & \ - CMPL_BASE_V) == !(v)) - #define NQ_CMP_VALID(nqcmp, raw_cons, ring) \ (!!((nqcmp)->v & rte_cpu_to_le_32(NQ_CN_V)) == \ !((raw_cons) & ((ring)->ring_size))) @@ -131,4 +124,35 @@ bool bnxt_is_recovery_enabled(struct bnxt *bp); bool bnxt_is_master_func(struct bnxt *bp); void bnxt_stop_rxtx(struct bnxt *bp); + +/** + * Check validity of a completion ring entry. If the entry is valid, include a + * C11 __ATOMIC_ACQUIRE fence to ensure that subsequent loads of fields in the + * completion are not hoisted by the compiler or by the CPU to come before the + * loading of the "valid" field. + * + * Note: the caller must not access any fields in the specified completion + * entry prior to calling this function. + * + * @param cmpl + * Pointer to an entry in the completion ring. + * @param raw_cons + * Raw consumer index of entry in completion ring. + * @param ring_size + * Size of completion ring. + */ +static __rte_always_inline bool +bnxt_cpr_cmp_valid(const void *cmpl, uint32_t raw_cons, uint32_t ring_size) +{ + const struct cmpl_base *c = cmpl; + bool expected, valid; + + expected = !(raw_cons & ring_size); + valid = !!(rte_le_to_cpu_32(c->info3_v) & CMPL_BASE_V); + if (valid == expected) { + rte_smp_rmb(); + return true; + } + return false; +} #endif diff --git a/drivers/net/bnxt/bnxt_ethdev.c b/drivers/net/bnxt/bnxt_ethdev.c index a257850d6..b029077af 100644 --- a/drivers/net/bnxt/bnxt_ethdev.c +++ b/drivers/net/bnxt/bnxt_ethdev.c @@ -2453,7 +2453,7 @@ bnxt_rx_queue_count_op(struct rte_eth_dev *dev, uint16_t rx_queue_id) { struct bnxt *bp = (struct bnxt *)dev->data->dev_private; struct bnxt_cp_ring_info *cpr; - uint32_t desc = 0, raw_cons; + uint32_t desc = 0, raw_cons, cp_ring_size; struct bnxt_rx_queue *rxq; struct rx_pkt_cmpl *rxcmp; int rc; @@ -2465,6 +2465,7 @@ bnxt_rx_queue_count_op(struct rte_eth_dev *dev, uint16_t rx_queue_id) rxq = dev->data->rx_queues[rx_queue_id]; cpr = rxq->cp_ring; raw_cons = cpr->cp_raw_cons; + cp_ring_size = cpr->cp_ring_struct->ring_size; while (1) { uint32_t agg_cnt, cons, cmpl_type; @@ -2472,7 +2473,7 @@ bnxt_rx_queue_count_op(struct rte_eth_dev *dev, uint16_t rx_queue_id) cons = RING_CMP(cpr->cp_ring_struct, raw_cons); rxcmp = (struct rx_pkt_cmpl *)&cpr->cp_desc_ring[cons]; - if (!CMP_VALID(rxcmp, raw_cons, cpr->cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(rxcmp, raw_cons, cp_ring_size)) break; cmpl_type = CMP_TYPE(rxcmp); @@ -2516,7 +2517,7 @@ bnxt_rx_descriptor_status_op(void *rx_queue, uint16_t offset) struct bnxt_cp_ring_info *cpr; struct bnxt_sw_rx_bd *rx_buf; struct bnxt_rx_ring_info *rxr; - uint32_t desc, raw_cons; + uint32_t desc, raw_cons, cp_ring_size; struct bnxt *bp = rxq->bp; struct rx_pkt_cmpl *rxcmp; int rc; @@ -2530,6 +2531,7 @@ bnxt_rx_descriptor_status_op(void *rx_queue, uint16_t offset) rxr = rxq->rx_ring; cpr = rxq->cp_ring; + cp_ring_size = cpr->cp_ring_struct->ring_size; raw_cons = cpr->cp_raw_cons; desc = 0; @@ -2539,7 +2541,7 @@ bnxt_rx_descriptor_status_op(void *rx_queue, uint16_t offset) cons = RING_CMP(cpr->cp_ring_struct, raw_cons); rxcmp = (struct rx_pkt_cmpl *)&cpr->cp_desc_ring[cons]; - if (!CMP_VALID(rxcmp, raw_cons, cpr->cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(rxcmp, raw_cons, cp_ring_size)) break; cmpl_type = CMP_TYPE(rxcmp); @@ -2593,7 +2595,6 @@ bnxt_tx_descriptor_status_op(void *tx_queue, uint16_t offset) struct bnxt_tx_queue *txq = (struct bnxt_tx_queue *)tx_queue; struct bnxt_cp_ring_info *cpr = txq->cp_ring; uint32_t ring_mask, raw_cons, nb_tx_pkts = 0; - struct bnxt_ring *cp_ring_struct; struct cmpl_base *cp_desc_ring; int rc; @@ -2610,7 +2611,6 @@ bnxt_tx_descriptor_status_op(void *tx_queue, uint16_t offset) raw_cons = cpr->cp_raw_cons; cp_desc_ring = cpr->cp_desc_ring; - cp_ring_struct = cpr->cp_ring_struct; ring_mask = cpr->cp_ring_struct->ring_mask; /* Check to see if hw has posted a completion for the descriptor. */ @@ -2621,7 +2621,7 @@ bnxt_tx_descriptor_status_op(void *tx_queue, uint16_t offset) cons = RING_CMPL(ring_mask, raw_cons); txcmp = (struct tx_cmpl *)&cp_desc_ring[cons]; - if (!CMP_VALID(txcmp, raw_cons, cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(txcmp, raw_cons, ring_mask + 1)) break; if (CMP_TYPE(txcmp) == TX_CMPL_TYPE_TX_L2) diff --git a/drivers/net/bnxt/bnxt_irq.c b/drivers/net/bnxt/bnxt_irq.c index 93779f4e0..f442edacc 100644 --- a/drivers/net/bnxt/bnxt_irq.c +++ b/drivers/net/bnxt/bnxt_irq.c @@ -21,10 +21,9 @@ void bnxt_int_handler(void *param) { struct rte_eth_dev *eth_dev = (struct rte_eth_dev *)param; struct bnxt *bp = eth_dev->data->dev_private; + uint32_t cons, raw_cons, cp_ring_size; struct bnxt_cp_ring_info *cpr; struct cmpl_base *cmp; - uint32_t raw_cons; - uint32_t cons; if (bp == NULL) return; @@ -33,6 +32,7 @@ void bnxt_int_handler(void *param) return; raw_cons = cpr->cp_raw_cons; + cp_ring_size = cpr->cp_ring_struct->ring_size; pthread_mutex_lock(&bp->def_cp_lock); while (1) { if (!cpr || !cpr->cp_ring_struct || !cpr->cp_db.doorbell) { @@ -48,7 +48,7 @@ void bnxt_int_handler(void *param) cons = RING_CMP(cpr->cp_ring_struct, raw_cons); cmp = &cpr->cp_desc_ring[cons]; - if (!CMP_VALID(cmp, raw_cons, cpr->cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(cmp, raw_cons, cp_ring_size)) break; bnxt_event_hwrm_resp_handler(bp, cmp); diff --git a/drivers/net/bnxt/bnxt_rxr.c b/drivers/net/bnxt/bnxt_rxr.c index 8abd391cf..a92115aae 100644 --- a/drivers/net/bnxt/bnxt_rxr.c +++ b/drivers/net/bnxt/bnxt_rxr.c @@ -186,7 +186,8 @@ static int bnxt_agg_bufs_valid(struct bnxt_cp_ring_info *cpr, cpr->valid = FLIP_VALID(raw_cp_cons, cpr->cp_ring_struct->ring_mask, cpr->valid); - return CMP_VALID(agg_cmpl, raw_cp_cons, cpr->cp_ring_struct); + return bnxt_cpr_cmp_valid(agg_cmpl, raw_cp_cons, + cpr->cp_ring_struct->ring_size); } /* TPA consume agg buffer out of order, allocate connected data only */ @@ -443,7 +444,8 @@ static int bnxt_rx_pkt(struct rte_mbuf **rx_pkt, cp_cons = RING_CMP(cpr->cp_ring_struct, tmp_raw_cons); rxcmp1 = (struct rx_pkt_cmpl_hi *)&cpr->cp_desc_ring[cp_cons]; - if (!CMP_VALID(rxcmp1, tmp_raw_cons, cpr->cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(rxcmp1, tmp_raw_cons, + cpr->cp_ring_struct->ring_size)) return -EBUSY; cpr->valid = FLIP_VALID(cp_cons, @@ -630,7 +632,8 @@ uint16_t bnxt_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, cons = RING_CMP(cpr->cp_ring_struct, raw_cons); rxcmp = (struct rx_pkt_cmpl *)&cpr->cp_desc_ring[cons]; - if (!CMP_VALID(rxcmp, raw_cons, cpr->cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(rxcmp, raw_cons, + cpr->cp_ring_struct->ring_size)) break; cpr->valid = FLIP_VALID(cons, cpr->cp_ring_struct->ring_mask, diff --git a/drivers/net/bnxt/bnxt_rxtx_vec_sse.c b/drivers/net/bnxt/bnxt_rxtx_vec_sse.c index 369301eb3..9c3ae397e 100644 --- a/drivers/net/bnxt/bnxt_rxtx_vec_sse.c +++ b/drivers/net/bnxt/bnxt_rxtx_vec_sse.c @@ -210,6 +210,7 @@ recv_burst_vec_sse(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) struct bnxt_cp_ring_info *cpr = rxq->cp_ring; struct bnxt_rx_ring_info *rxr = rxq->rx_ring; uint32_t raw_cons = cpr->cp_raw_cons; + uint32_t cp_ring_size; uint32_t cons; int nb_rx_pkts = 0; struct rx_pkt_cmpl *rxcmp; @@ -237,13 +238,15 @@ recv_burst_vec_sse(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) if (!nb_pkts) return 0; + cp_ring_size = cpr->cp_ring_struct->ring_size; + /* Handle RX burst request */ while (1) { cons = RING_CMP(cpr->cp_ring_struct, raw_cons); rxcmp = (struct rx_pkt_cmpl *)&cpr->cp_desc_ring[cons]; - if (!CMP_VALID(rxcmp, raw_cons, cpr->cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(rxcmp, raw_cons, cp_ring_size)) break; if (likely(CMP_TYPE(rxcmp) == RX_PKT_CMPL_TYPE_RX_L2)) { @@ -258,8 +261,8 @@ recv_burst_vec_sse(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) rxcmp1 = (struct rx_pkt_cmpl_hi *) &cpr->cp_desc_ring[cp_cons]; - if (!CMP_VALID(rxcmp1, tmp_raw_cons, - cpr->cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(rxcmp1, tmp_raw_cons, + cp_ring_size)) break; raw_cons = tmp_raw_cons; @@ -387,7 +390,7 @@ bnxt_handle_tx_cp_vec(struct bnxt_tx_queue *txq) cons = RING_CMPL(ring_mask, raw_cons); txcmp = (struct tx_cmpl *)&cp_desc_ring[cons]; - if (!CMP_VALID(txcmp, raw_cons, cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(txcmp, raw_cons, ring_mask + 1)) break; if (likely(CMP_TYPE(txcmp) == TX_CMPL_TYPE_TX_L2)) diff --git a/drivers/net/bnxt/bnxt_txr.c b/drivers/net/bnxt/bnxt_txr.c index 892f8c1a4..0d6d485ab 100644 --- a/drivers/net/bnxt/bnxt_txr.c +++ b/drivers/net/bnxt/bnxt_txr.c @@ -414,7 +414,7 @@ static int bnxt_handle_tx_cp(struct bnxt_tx_queue *txq) cons = RING_CMPL(ring_mask, raw_cons); txcmp = (struct tx_cmpl *)&cpr->cp_desc_ring[cons]; - if (!CMP_VALID(txcmp, raw_cons, cp_ring_struct)) + if (!bnxt_cpr_cmp_valid(txcmp, raw_cons, ring_mask + 1)) break; opaque = rte_le_to_cpu_32(txcmp->opaque); -- 2.25.1 --000000000000d1587405c9b32c3b--