From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CA5644612D; Fri, 24 Jan 2025 17:32:20 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0826B42E10; Fri, 24 Jan 2025 17:30:22 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.9]) by mails.dpdk.org (Postfix) with ESMTP id 04B1342E12 for ; Fri, 24 Jan 2025 17:30:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1737736220; x=1769272220; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/aO9EikQCpYBQHaEDHDKRPGGDtfrnad0RdJPTqbxYt8=; b=iy0vEwrlvxztMgkeuKUMmOPGYEGqcB26TmhewRjl1+XfhAuuNSeiB19I sl2ezHJ5qVt87Q8nfghPbKmuUBl7xv1AEWuO49yD1V2pwZyOfQWM/rH6w EtXXWsCsDJB0GdZiZL15mmqUrK4IczSlBkt6WFtjkoiISMWgyyIaK1Ds5 2z94hexsu/IwJOTPEDoG9YbvZvwhyuJQdt8VoznBvgF6r9pL446zcm9jg noAewfHye52Nq/7sKzaYYIjgPwjVlqMC1IdYG6nCqEqkn1ONJu8oYT6hC GYdSizRAY5hKRrWSf1mGqaXMKV7sxjbYvUb+XjjlmzdC1wKjtq4Qqx1fD w==; X-CSE-ConnectionGUID: ReAT0FRSS9+8xm5Z1aVqSg== X-CSE-MsgGUID: L4li/07LSdqVCgLLGiOHHw== X-IronPort-AV: E=McAfee;i="6700,10204,11325"; a="60740274" X-IronPort-AV: E=Sophos;i="6.13,231,1732608000"; d="scan'208";a="60740274" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2025 08:30:19 -0800 X-CSE-ConnectionGUID: CRRhdIlCRcSp8RgegoubFw== X-CSE-MsgGUID: 7oYKEgJVTveiyH2RBgKj7Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.13,231,1732608000"; d="scan'208";a="108352833" Received: from silpixa00401197coob.ir.intel.com (HELO silpixa00401385.ir.intel.com) ([10.237.214.45]) by fmviesa010.fm.intel.com with ESMTP; 24 Jan 2025 08:30:17 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: david.marchand@redhat.com, anatoly.burakov@intel.com, vladimir.medvedkin@intel.com, ian.stokes@intel.com, praveen.shetty@intel.com, Bruce Richardson Subject: [PATCH v6 21/25] net/iavf: use vector SW ring for all vector paths Date: Fri, 24 Jan 2025 16:29:16 +0000 Message-ID: <20250124162921.1406103-22-bruce.richardson@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250124162921.1406103-1-bruce.richardson@intel.com> References: <20241122125418.2857301-1-bruce.richardson@intel.com> <20250124162921.1406103-1-bruce.richardson@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The AVX-512 code path used a smaller SW ring structure only containing the mbuf pointer, but no other fields. The other fields are only used in the scalar code path, so update all vector driver code paths (AVX2, SSE) to use the smaller, faster structure. Signed-off-by: Bruce Richardson --- drivers/net/intel/iavf/iavf_rxtx.c | 7 ------- drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c | 12 ++++++------ drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 8 -------- drivers/net/intel/iavf/iavf_rxtx_vec_common.h | 6 ------ drivers/net/intel/iavf/iavf_rxtx_vec_sse.c | 14 +++++++------- 5 files changed, 13 insertions(+), 34 deletions(-) diff --git a/drivers/net/intel/iavf/iavf_rxtx.c b/drivers/net/intel/iavf/iavf_rxtx.c index 22639da965..80ea38369b 100644 --- a/drivers/net/intel/iavf/iavf_rxtx.c +++ b/drivers/net/intel/iavf/iavf_rxtx.c @@ -4188,14 +4188,7 @@ iavf_set_tx_function(struct rte_eth_dev *dev) txq = dev->data->tx_queues[i]; if (!txq) continue; -#ifdef CC_AVX512_SUPPORT - if (use_avx512) - iavf_txq_vec_setup_avx512(txq); - else - iavf_txq_vec_setup(txq); -#else iavf_txq_vec_setup(txq); -#endif } if (no_poll_on_link_down) { diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c index fdb98b417a..b847886081 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx2.c @@ -1736,14 +1736,14 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, { struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; volatile struct iavf_tx_desc *txdp; - struct ci_tx_entry *txep; + struct ci_tx_entry_vec *txep; uint16_t n, nb_commit, tx_id; /* bit2 is reserved and must be set to 1 according to Spec */ uint64_t flags = IAVF_TX_DESC_CMD_EOP | IAVF_TX_DESC_CMD_ICRC; uint64_t rs = IAVF_TX_DESC_CMD_RS | flags; if (txq->nb_tx_free < txq->tx_free_thresh) - iavf_tx_free_bufs(txq); + ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, false); nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts); if (unlikely(nb_pkts == 0)) @@ -1752,13 +1752,13 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, tx_id = txq->tx_tail; txdp = &txq->iavf_tx_ring[tx_id]; - txep = &txq->sw_ring[tx_id]; + txep = &txq->sw_ring_vec[tx_id]; txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts); n = (uint16_t)(txq->nb_tx_desc - tx_id); if (nb_commit >= n) { - ci_tx_backlog_entry(txep, tx_pkts, n); + ci_tx_backlog_entry_vec(txep, tx_pkts, n); iavf_vtx(txdp, tx_pkts, n - 1, flags, offload); tx_pkts += (n - 1); @@ -1773,10 +1773,10 @@ iavf_xmit_fixed_burst_vec_avx2(void *tx_queue, struct rte_mbuf **tx_pkts, /* avoid reach the end of ring */ txdp = &txq->iavf_tx_ring[tx_id]; - txep = &txq->sw_ring[tx_id]; + txep = &txq->sw_ring_vec[tx_id]; } - ci_tx_backlog_entry(txep, tx_pkts, nb_commit); + ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit); iavf_vtx(txdp, tx_pkts, nb_commit, flags, offload); diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c index 007759e451..641f3311eb 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c @@ -2357,14 +2357,6 @@ iavf_xmit_pkts_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts, return iavf_xmit_pkts_vec_avx512_cmn(tx_queue, tx_pkts, nb_pkts, false); } -int __rte_cold -iavf_txq_vec_setup_avx512(struct ci_tx_queue *txq) -{ - txq->vector_tx = true; - txq->vector_sw_ring = true; - return 0; -} - uint16_t iavf_xmit_pkts_vec_avx512_offload(void *tx_queue, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h index 8f272044b9..777fb1ce14 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_common.h +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_common.h @@ -24,12 +24,6 @@ iavf_tx_desc_done(struct ci_tx_queue *txq, uint16_t idx) rte_cpu_to_le_64(IAVF_TX_DESC_DTYPE_DESC_DONE); } -static __rte_always_inline int -iavf_tx_free_bufs(struct ci_tx_queue *txq) -{ - return ci_tx_free_bufs(txq, iavf_tx_desc_done); -} - static inline void _iavf_rx_queue_release_mbufs_vec(struct iavf_rx_queue *rxq) { diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c index 3adf2a59e4..9f7db80bfd 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_sse.c @@ -1368,14 +1368,14 @@ iavf_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, { struct ci_tx_queue *txq = (struct ci_tx_queue *)tx_queue; volatile struct iavf_tx_desc *txdp; - struct ci_tx_entry *txep; + struct ci_tx_entry_vec *txep; uint16_t n, nb_commit, tx_id; uint64_t flags = IAVF_TX_DESC_CMD_EOP | 0x04; /* bit 2 must be set */ uint64_t rs = IAVF_TX_DESC_CMD_RS | flags; int i; if (txq->nb_tx_free < txq->tx_free_thresh) - iavf_tx_free_bufs(txq); + ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, false); nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts); if (unlikely(nb_pkts == 0)) @@ -1384,13 +1384,13 @@ iavf_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, tx_id = txq->tx_tail; txdp = &txq->iavf_tx_ring[tx_id]; - txep = &txq->sw_ring[tx_id]; + txep = &txq->sw_ring_vec[tx_id]; txq->nb_tx_free = (uint16_t)(txq->nb_tx_free - nb_pkts); n = (uint16_t)(txq->nb_tx_desc - tx_id); if (nb_commit >= n) { - ci_tx_backlog_entry(txep, tx_pkts, n); + ci_tx_backlog_entry_vec(txep, tx_pkts, n); for (i = 0; i < n - 1; ++i, ++tx_pkts, ++txdp) vtx1(txdp, *tx_pkts, flags); @@ -1404,10 +1404,10 @@ iavf_xmit_fixed_burst_vec(void *tx_queue, struct rte_mbuf **tx_pkts, /* avoid reach the end of ring */ txdp = &txq->iavf_tx_ring[tx_id]; - txep = &txq->sw_ring[tx_id]; + txep = &txq->sw_ring_vec[tx_id]; } - ci_tx_backlog_entry(txep, tx_pkts, nb_commit); + ci_tx_backlog_entry_vec(txep, tx_pkts, nb_commit); iavf_vtx(txdp, tx_pkts, nb_commit, flags); @@ -1462,7 +1462,7 @@ int __rte_cold iavf_txq_vec_setup(struct ci_tx_queue *txq) { txq->vector_tx = true; - txq->vector_sw_ring = false; + txq->vector_sw_ring = txq->vector_tx; return 0; } -- 2.43.0