From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D5EAC4612D; Fri, 24 Jan 2025 17:31:32 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0680A42DA3; Fri, 24 Jan 2025 17:30:06 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.9]) by mails.dpdk.org (Postfix) with ESMTP id 3E31E42D99 for ; Fri, 24 Jan 2025 17:30:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1737736204; x=1769272204; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=m7MncIAXUe1eKh+eWSaXWi4s+/qZs5yH3IAsbxbLTDs=; b=Jcm76QMFWrFL2+BzyaVX9DQxWObFGI9xwyM/eeVaz3RMQvZkmL86cDKu Y1za/0qu5VxBE4ZnMwZ+w8rH1y3hmmQo0H9R1RxVe/4lG5n+yHgB5TNBP DMYtNjMPW/5+iVeBe55qVjO6Z6VNrxo7J6QcfniEmuAIjbXNhQ399+tEz gkXxYYIgbN5gDTiMsR0d9MYrDpxBSxn8lxsI0ZTTlKTTNAEXGxx6mLDJN LlyJMVaQZkCEPvRgSVpXm0+UBa88NMAsI9aJkYzNPW6gj2GsaVQArnoRX rEbHS+9x3U/WPxFbgBcSM/ULPKT1P8Aeue/YUvssSfE3oQDPekbCDvsLJ w==; X-CSE-ConnectionGUID: NTAt+v94Q6mEz8T88Kc4Eg== X-CSE-MsgGUID: quLd5nKXQ/S/6yafXPyfgw== X-IronPort-AV: E=McAfee;i="6700,10204,11325"; a="60740253" X-IronPort-AV: E=Sophos;i="6.13,231,1732608000"; d="scan'208";a="60740253" Received: from fmviesa010.fm.intel.com ([10.60.135.150]) by orvoesa101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Jan 2025 08:30:04 -0800 X-CSE-ConnectionGUID: 0tfmfUaySyWQocccLKSbMg== X-CSE-MsgGUID: fe9FqjHcTv6lQjMaXOUYgw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.13,231,1732608000"; d="scan'208";a="108352591" Received: from silpixa00401197coob.ir.intel.com (HELO silpixa00401385.ir.intel.com) ([10.237.214.45]) by fmviesa010.fm.intel.com with ESMTP; 24 Jan 2025 08:30:02 -0800 From: Bruce Richardson To: dev@dpdk.org Cc: david.marchand@redhat.com, anatoly.burakov@intel.com, vladimir.medvedkin@intel.com, ian.stokes@intel.com, praveen.shetty@intel.com, Bruce Richardson Subject: [PATCH v6 14/25] net/iavf: use common Tx free fn for AVX-512 Date: Fri, 24 Jan 2025 16:29:09 +0000 Message-ID: <20250124162921.1406103-15-bruce.richardson@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250124162921.1406103-1-bruce.richardson@intel.com> References: <20241122125418.2857301-1-bruce.richardson@intel.com> <20250124162921.1406103-1-bruce.richardson@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Switch the iavf driver to use the common Tx free function. This requires one additional parameter to that function, since iavf sometimes uses context descriptors which means that we have double the descriptors per SW ring slot. Signed-off-by: Bruce Richardson --- drivers/net/intel/common/tx.h | 6 +- drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c | 2 +- drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c | 119 +----------------- drivers/net/intel/ice/ice_rxtx_vec_avx512.c | 2 +- 4 files changed, 7 insertions(+), 122 deletions(-) diff --git a/drivers/net/intel/common/tx.h b/drivers/net/intel/common/tx.h index 84ff839672..26aef528fa 100644 --- a/drivers/net/intel/common/tx.h +++ b/drivers/net/intel/common/tx.h @@ -179,7 +179,7 @@ ci_tx_free_bufs(struct ci_tx_queue *txq, ci_desc_done_fn desc_done) } static __rte_always_inline int -ci_tx_free_bufs_vec(struct ci_tx_queue *txq, ci_desc_done_fn desc_done) +ci_tx_free_bufs_vec(struct ci_tx_queue *txq, ci_desc_done_fn desc_done, bool ctx_descs) { int nb_free = 0; struct rte_mbuf *free[IETH_VPMD_TX_MAX_FREE_BUF]; @@ -189,13 +189,13 @@ ci_tx_free_bufs_vec(struct ci_tx_queue *txq, ci_desc_done_fn desc_done) if (!desc_done(txq, txq->tx_next_dd)) return 0; - const uint32_t n = txq->tx_rs_thresh; + const uint32_t n = txq->tx_rs_thresh >> ctx_descs; /* first buffer to free from S/W ring is at index * tx_next_dd - (tx_rs_thresh - 1) */ struct ci_tx_entry_vec *txep = txq->sw_ring_vec; - txep += txq->tx_next_dd - (n - 1); + txep += (txq->tx_next_dd >> ctx_descs) - (n - 1); if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE && (n & 31) == 0) { struct rte_mempool *mp = txep[0].mbuf->pool; diff --git a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c index 9bb2a44231..c555c3491d 100644 --- a/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c +++ b/drivers/net/intel/i40e/i40e_rxtx_vec_avx512.c @@ -829,7 +829,7 @@ i40e_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts, uint64_t rs = I40E_TX_DESC_CMD_RS | I40E_TD_CMD; if (txq->nb_tx_free < txq->tx_free_thresh) - ci_tx_free_bufs_vec(txq, i40e_tx_desc_done); + ci_tx_free_bufs_vec(txq, i40e_tx_desc_done, false); nb_commit = nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts); if (unlikely(nb_pkts == 0)) diff --git a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c index 9cf7171524..8543490c70 100644 --- a/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c +++ b/drivers/net/intel/iavf/iavf_rxtx_vec_avx512.c @@ -1844,121 +1844,6 @@ iavf_recv_scattered_pkts_vec_avx512_flex_rxd_offload(void *rx_queue, true); } -static __rte_always_inline int -iavf_tx_free_bufs_avx512(struct ci_tx_queue *txq) -{ - struct ci_tx_entry_vec *txep; - uint32_t n; - uint32_t i; - int nb_free = 0; - struct rte_mbuf *m, *free[IAVF_VPMD_TX_MAX_FREE_BUF]; - - /* check DD bits on threshold descriptor */ - if ((txq->iavf_tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & - rte_cpu_to_le_64(IAVF_TXD_QW1_DTYPE_MASK)) != - rte_cpu_to_le_64(IAVF_TX_DESC_DTYPE_DESC_DONE)) - return 0; - - n = txq->tx_rs_thresh >> txq->use_ctx; - - /* first buffer to free from S/W ring is at index - * tx_next_dd - (tx_rs_thresh-1) - */ - txep = (void *)txq->sw_ring; - txep += (txq->tx_next_dd >> txq->use_ctx) - (n - 1); - - if (txq->offloads & RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE && (n & 31) == 0) { - struct rte_mempool *mp = txep[0].mbuf->pool; - struct rte_mempool_cache *cache = rte_mempool_default_cache(mp, - rte_lcore_id()); - void **cache_objs; - - if (!cache || cache->len == 0) - goto normal; - - cache_objs = &cache->objs[cache->len]; - - if (n > RTE_MEMPOOL_CACHE_MAX_SIZE) { - rte_mempool_ops_enqueue_bulk(mp, (void *)txep, n); - goto done; - } - - /* The cache follows the following algorithm - * 1. Add the objects to the cache - * 2. Anything greater than the cache min value (if it crosses the - * cache flush threshold) is flushed to the ring. - */ - /* Add elements back into the cache */ - uint32_t copied = 0; - /* n is multiple of 32 */ - while (copied < n) { -#ifdef RTE_ARCH_64 - const __m512i a = _mm512_loadu_si512(&txep[copied]); - const __m512i b = _mm512_loadu_si512(&txep[copied + 8]); - const __m512i c = _mm512_loadu_si512(&txep[copied + 16]); - const __m512i d = _mm512_loadu_si512(&txep[copied + 24]); - - _mm512_storeu_si512(&cache_objs[copied], a); - _mm512_storeu_si512(&cache_objs[copied + 8], b); - _mm512_storeu_si512(&cache_objs[copied + 16], c); - _mm512_storeu_si512(&cache_objs[copied + 24], d); -#else - const __m512i a = _mm512_loadu_si512(&txep[copied]); - const __m512i b = _mm512_loadu_si512(&txep[copied + 16]); - _mm512_storeu_si512(&cache_objs[copied], a); - _mm512_storeu_si512(&cache_objs[copied + 16], b); -#endif - copied += 32; - } - cache->len += n; - - if (cache->len >= cache->flushthresh) { - rte_mempool_ops_enqueue_bulk(mp, - &cache->objs[cache->size], - cache->len - cache->size); - cache->len = cache->size; - } - goto done; - } - -normal: - m = rte_pktmbuf_prefree_seg(txep[0].mbuf); - if (likely(m)) { - free[0] = m; - nb_free = 1; - for (i = 1; i < n; i++) { - m = rte_pktmbuf_prefree_seg(txep[i].mbuf); - if (likely(m)) { - if (likely(m->pool == free[0]->pool)) { - free[nb_free++] = m; - } else { - rte_mempool_put_bulk(free[0]->pool, - (void *)free, - nb_free); - free[0] = m; - nb_free = 1; - } - } - } - rte_mempool_put_bulk(free[0]->pool, (void **)free, nb_free); - } else { - for (i = 1; i < n; i++) { - m = rte_pktmbuf_prefree_seg(txep[i].mbuf); - if (m) - rte_mempool_put(m->pool, m); - } - } - -done: - /* buffers were freed, update counters */ - txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh); - txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh); - if (txq->tx_next_dd >= txq->nb_tx_desc) - txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); - - return txq->tx_rs_thresh; -} - static __rte_always_inline void tx_backlog_entry_avx512(struct ci_tx_entry_vec *txep, struct rte_mbuf **tx_pkts, uint16_t nb_pkts) @@ -2320,7 +2205,7 @@ iavf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts, uint64_t rs = IAVF_TX_DESC_CMD_RS | flags; if (txq->nb_tx_free < txq->tx_free_thresh) - iavf_tx_free_bufs_avx512(txq); + ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, false); nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts); if (unlikely(nb_pkts == 0)) @@ -2388,7 +2273,7 @@ iavf_xmit_fixed_burst_vec_avx512_ctx(void *tx_queue, struct rte_mbuf **tx_pkts, uint64_t rs = IAVF_TX_DESC_CMD_RS | flags; if (txq->nb_tx_free < txq->tx_free_thresh) - iavf_tx_free_bufs_avx512(txq); + ci_tx_free_bufs_vec(txq, iavf_tx_desc_done, true); nb_commit = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts << 1); nb_commit &= 0xFFFE; diff --git a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c index 538be707ef..f6ec593f96 100644 --- a/drivers/net/intel/ice/ice_rxtx_vec_avx512.c +++ b/drivers/net/intel/ice/ice_rxtx_vec_avx512.c @@ -949,7 +949,7 @@ ice_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf **tx_pkts, nb_pkts = RTE_MIN(nb_pkts, txq->tx_rs_thresh); if (txq->nb_tx_free < txq->tx_free_thresh) - ci_tx_free_bufs_vec(txq, ice_tx_desc_done); + ci_tx_free_bufs_vec(txq, ice_tx_desc_done, false); nb_commit = nb_pkts = (uint16_t)RTE_MIN(txq->nb_tx_free, nb_pkts); if (unlikely(nb_pkts == 0)) -- 2.43.0