From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CD847A04F9; Thu, 9 Jan 2020 18:16:48 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EC7F21E4C9; Thu, 9 Jan 2020 18:16:26 +0100 (CET) Received: from mellanox.co.il (mail-il-dmz.mellanox.com [193.47.165.129]) by dpdk.org (Postfix) with ESMTP id 6381C1E4C1 for ; Thu, 9 Jan 2020 18:16:23 +0100 (CET) Received: from Internal Mail-Server by MTLPINE1 (envelope-from viacheslavo@mellanox.com) with ESMTPS (AES256-SHA encrypted); 9 Jan 2020 19:16:21 +0200 Received: from pegasus11.mtr.labs.mlnx (pegasus11.mtr.labs.mlnx [10.210.16.104]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 009HGL5U030306; Thu, 9 Jan 2020 19:16:21 +0200 Received: from pegasus11.mtr.labs.mlnx (localhost [127.0.0.1]) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7) with ESMTP id 009HGLfP010276; Thu, 9 Jan 2020 17:16:21 GMT Received: (from viacheslavo@localhost) by pegasus11.mtr.labs.mlnx (8.14.7/8.14.7/Submit) id 009HGLrb010275; Thu, 9 Jan 2020 17:16:21 GMT X-Authentication-Warning: pegasus11.mtr.labs.mlnx: viacheslavo set sender to viacheslavo@mellanox.com using -f From: Viacheslav Ovsiienko To: dev@dpdk.org Cc: matan@mellanox.com, rasland@mellanox.com, orika@mellanox.com Date: Thu, 9 Jan 2020 17:16:07 +0000 Message-Id: <1578590167-10167-5-git-send-email-viacheslavo@mellanox.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1578590167-10167-1-git-send-email-viacheslavo@mellanox.com> References: <1578500161-20156-1-git-send-email-viacheslavo@mellanox.com> <1578590167-10167-1-git-send-email-viacheslavo@mellanox.com> Subject: [dpdk-dev] [PATCH v3 4/4] net/mlx5: engage free on completion queue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The free on completion queue keeps the indices of elts array, all mbuf stored below this index should be freed on arrival of normal send completion. In debug version it also contains an index of completed transmitting descriptor (WQE) to check queues synchronization. Signed-off-by: Viacheslav Ovsiienko Acked-by: Matan Azrad --- drivers/net/mlx5/mlx5_rxtx.c | 33 +++++++++++++++++---------------- drivers/net/mlx5/mlx5_rxtx.h | 4 +--- drivers/net/mlx5/mlx5_txq.c | 2 -- 3 files changed, 18 insertions(+), 21 deletions(-) diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index b7b40ac..b11c5eb 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -2043,8 +2043,7 @@ enum mlx5_txcmp_code { uint16_t tail; txq->wqe_pi = rte_be_to_cpu_16(last_cqe->wqe_counter); - tail = ((volatile struct mlx5_wqe_cseg *) - (txq->wqes + (txq->wqe_pi & txq->wqe_m)))->misc; + tail = txq->fcqs[(txq->cq_ci - 1) & txq->cqe_m]; if (likely(tail != txq->elts_tail)) { mlx5_tx_free_elts(txq, tail, olx); assert(tail == txq->elts_tail); @@ -2095,6 +2094,7 @@ enum mlx5_txcmp_code { * here, before we might perform SQ reset. */ rte_wmb(); + txq->cq_ci = ci; ret = mlx5_tx_error_cqe_handle (txq, (volatile struct mlx5_err_cqe *)cqe); if (unlikely(ret < 0)) { @@ -2108,17 +2108,18 @@ enum mlx5_txcmp_code { /* * We are going to fetch all entries with * MLX5_CQE_SYNDROME_WR_FLUSH_ERR status. + * The send queue is supposed to be empty. */ ++ci; + txq->cq_pi = ci; + last_cqe = NULL; continue; } /* Normal transmit completion. */ + assert(ci != txq->cq_pi); + assert((txq->fcqs[ci & txq->cqe_m] >> 16) == cqe->wqe_counter); ++ci; last_cqe = cqe; -#ifndef NDEBUG - if (txq->cq_pi) - --txq->cq_pi; -#endif /* * We have to restrict the amount of processed CQEs * in one tx_burst routine call. The CQ may be large @@ -2127,7 +2128,7 @@ enum mlx5_txcmp_code { * multiple iterations may introduce significant * latency. */ - if (--count == 0) + if (likely(--count == 0)) break; } while (true); if (likely(ci != txq->cq_ci)) { @@ -2177,15 +2178,15 @@ enum mlx5_txcmp_code { /* Request unconditional completion on last WQE. */ last->cseg.flags = RTE_BE32(MLX5_COMP_ALWAYS << MLX5_COMP_MODE_OFFSET); - /* Save elts_head in unused "immediate" field of WQE. */ - last->cseg.misc = head; - /* - * A CQE slot must always be available. Count the - * issued CEQ "always" request instead of production - * index due to here can be CQE with errors and - * difference with ci may become inconsistent. - */ - assert(txq->cqe_s > ++txq->cq_pi); + /* Save elts_head in dedicated free on completion queue. */ +#ifdef NDEBUG + txq->fcqs[txq->cq_pi++ & txq->cqe_m] = head; +#else + txq->fcqs[txq->cq_pi++ & txq->cqe_m] = head | + (last->cseg.opcode >> 8) << 16; +#endif + /* A CQE slot must always be available. */ + assert((txq->cq_pi - txq->cq_ci) <= txq->cqe_s); } } diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index 7d1b2fa..e362b4a 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -273,9 +273,7 @@ struct mlx5_txq_data { uint16_t wqe_thres; /* WQE threshold to request completion in CQ. */ /* WQ related fields. */ uint16_t cq_ci; /* Consumer index for completion queue. */ -#ifndef NDEBUG - uint16_t cq_pi; /* Counter of issued CQE "always" requests. */ -#endif + uint16_t cq_pi; /* Production index for completion queue. */ uint16_t cqe_s; /* Number of CQ elements. */ uint16_t cqe_m; /* Mask for CQ indices. */ /* CQ related fields. */ diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index aee0970..c750082 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -717,9 +717,7 @@ struct mlx5_txq_obj * txq_data->cq_db = cq_info.dbrec; txq_data->cqes = (volatile struct mlx5_cqe *)cq_info.buf; txq_data->cq_ci = 0; -#ifndef NDEBUG txq_data->cq_pi = 0; -#endif txq_data->wqe_ci = 0; txq_data->wqe_pi = 0; txq_data->wqe_comp = 0; -- 1.8.3.1