From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BACD3A00C2; Mon, 28 Mar 2022 17:17:37 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5D12B427F9; Mon, 28 Mar 2022 17:17:37 +0200 (CEST) Received: from m12-18.163.com (m12-18.163.com [220.181.12.18]) by mails.dpdk.org (Postfix) with ESMTP id 5963541104 for ; Mon, 28 Mar 2022 17:17:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:Subject:Date:Message-Id:MIME-Version; bh=53R/t 5qdQVzg1UK+veatTnb2nO8P/UuRnY2xYZyobjo=; b=lVgZb8TvREVRVjkYS9uDC 3SvspydTxRw1zJxlyZ0JTeJ0YUQeINw0Q//+hj6SknV0APqrc1qJhfCfEK7QaxXk rz7pZ8P7ARLBouGEJ5IKGNj1SLh2+U+b2MOOnSCXLj4L6jIJOAmmHea/JY8LJtwT qfbDSx7d43qYmlNDrhIe6M= Received: from DESKTOP-ONA2IA7.localdomain (unknown [122.235.128.125]) by smtp14 (Coremail) with SMTP id EsCowABXghlt0UFiC_5cCA--.57435S4; Mon, 28 Mar 2022 23:17:24 +0800 (CST) From: Gaoxiang Liu To: chas3@att.com, humin29@huawei.com Cc: olivier.matz@6wind.com, dev@dpdk.org, liugaoxiang@huawei.com, Gaoxiang Liu Subject: [PATCH v7] net/bonding: another fix to LACP mempool size Date: Mon, 28 Mar 2022 23:16:52 +0800 Message-Id: <20220328151652.221-1-gaoxiangliu0@163.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220325133426.2916-1-gaoxiangliu0@163.com> References: <20220325133426.2916-1-gaoxiangliu0@163.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CM-TRANSID: EsCowABXghlt0UFiC_5cCA--.57435S4 X-Coremail-Antispam: 1Uf129KBjvJXoWxGF13Kry7Xw1DWr47Jw1rtFb_yoW5tr17pr W7Way5tr1UArZI9ws7X3WSkws5ur92vr4UK395Zas8Zr47AF1Ygw1UtrW5urW8GrWktFs0 vF4UuasIgF4UG3DanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0pR92-bUUUUU= X-Originating-IP: [122.235.128.125] X-CM-SenderInfo: xjdr5xxdqjzxjxq6il2tof0z/1tbiMhvROlWByNmICwAAsE X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org The following log message may appear after a slave is idle(or nearly idle) for a few minutes:"PMD: Failed to allocate LACP packet from pool". And bond mode 4 negotiation may fail. Problem:When bond mode 4 has been chosed and delicated queue has not been enable, all mbufs from a slave' private pool(used exclusively for transmitting LACPDUs) have been allocated in interrupt thread, and are still sitting in the device's tx descriptor ring and other cores' mempool caches in fwd thread. Thus the interrupt thread can not alloc LACP packet from pool. Solution: Ensure that each slave'tx (LACPDU) mempool owns more than n-tx-queues * n-tx-descriptor + fwd_core_num * per-core-mmempool-flush-threshold mbufs. Note that the LACP tx machine fuction is the only code that allocates from a slave's private pool. It runs in the context of the interrupt thread, and thus it has no mempool cache of its own. Signed-off-by: Gaoxiang Liu --- v2: * Fixed compile issues. v3: * delete duplicate code. v4; * Fixed some issues. 1. total_tx_desc should use += 2. add detailed logs v5: * Fixed some issues. 1. move CACHE_FLUSHTHRESH_MULTIPLIER to rte_eth_bond-8023ad.c 2. use RTE_MIN v6: * add a comment of CACHE_FLUSHTHRESH_MULTIPLIER macro v7: * Fixed some issues. 1. move CACHE_FLUSHTHRESH_MULTIPLIER to rte_mempool.h --- drivers/net/bonding/rte_eth_bond_8023ad.c | 7 ++++--- lib/mempool/rte_mempool.h | 2 ++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/net/bonding/rte_eth_bond_8023ad.c b/drivers/net/bonding/rte_eth_bond_8023ad.c index ca50583d62..f7f6828126 100644 --- a/drivers/net/bonding/rte_eth_bond_8023ad.c +++ b/drivers/net/bonding/rte_eth_bond_8023ad.c @@ -1050,6 +1050,7 @@ bond_mode_8023ad_activate_slave(struct rte_eth_dev *bond_dev, uint32_t total_tx_desc; struct bond_tx_queue *bd_tx_q; uint16_t q_id; + uint32_t cache_size; /* Given slave mus not be in active list */ RTE_ASSERT(find_slave_by_id(internals->active_slaves, @@ -1100,11 +1101,11 @@ bond_mode_8023ad_activate_slave(struct rte_eth_dev *bond_dev, total_tx_desc += bd_tx_q->nb_tx_desc; } + cache_size = RTE_MIN(RTE_MEMPOOL_CACHE_MAX_SIZE, 32); + total_tx_desc += rte_lcore_count() * cache_size * RTE_MEMPOOL_CACHE_FLUSHTHRESH_MULTIPLIER; snprintf(mem_name, RTE_DIM(mem_name), "slave_port%u_pool", slave_id); port->mbuf_pool = rte_pktmbuf_pool_create(mem_name, total_tx_desc, - RTE_MEMPOOL_CACHE_MAX_SIZE >= 32 ? - 32 : RTE_MEMPOOL_CACHE_MAX_SIZE, - 0, element_size, socket_id); + cache_size, 0, element_size, socket_id); /* Any memory allocation failure in initialization is critical because * resources can't be free, so reinitialization is impossible. */ diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h index 1e7a3c1527..fa15ed710f 100644 --- a/lib/mempool/rte_mempool.h +++ b/lib/mempool/rte_mempool.h @@ -56,6 +56,8 @@ extern "C" { #endif +#define RTE_MEMPOOL_CACHE_FLUSHTHRESH_MULTIPLIER 1.5 + #define RTE_MEMPOOL_HEADER_COOKIE1 0xbadbadbadadd2e55ULL /**< Header cookie. */ #define RTE_MEMPOOL_HEADER_COOKIE2 0xf2eef2eedadd2e55ULL /**< Header cookie. */ #define RTE_MEMPOOL_TRAILER_COOKIE 0xadd2e55badbadbadULL /**< Trailer cookie.*/ -- 2.32.0