From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EFC40A00C3; Fri, 25 Mar 2022 14:14:06 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9611F40687; Fri, 25 Mar 2022 14:14:06 +0100 (CET) Received: from smartserver.smartsharesystems.com (smartserver.smartsharesystems.com [77.243.40.215]) by mails.dpdk.org (Postfix) with ESMTP id A14D040140 for ; Fri, 25 Mar 2022 14:14:05 +0100 (CET) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: [PATCH v5] net/bonding: another fix to LACP mempool size Date: Fri, 25 Mar 2022 14:13:59 +0100 Message-ID: <98CBD80474FA8B44BF855DF32C47DC35D86F73@smartserver.smartshare.dk> In-Reply-To: <20220325130135.2207-1-gaoxiangliu0@163.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PATCH v5] net/bonding: another fix to LACP mempool size Thread-Index: AdhASIifKYRtxbvlTb2eN7PxCbXVIQAAMlyA References: <20220325121023.746-1-gaoxiangliu0@163.com> <20220325130135.2207-1-gaoxiangliu0@163.com> From: =?iso-8859-1?Q?Morten_Br=F8rup?= To: "Gaoxiang Liu" , , Cc: , , , X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org +CC mempool maintainers > From: Gaoxiang Liu [mailto:gaoxiangliu0@163.com] > Sent: Friday, 25 March 2022 14.02 >=20 > The following log message may appear after a slave is idle(or nearly > idle) > for a few minutes:"PMD: Failed to allocate LACP packet from pool". > And bond mode 4 negotiation may fail. >=20 > Problem:When bond mode 4 has been chosed and delicated queue has > not been enable, all mbufs from a slave' private pool(used > exclusively for transmitting LACPDUs) have been allocated in > interrupt thread, and are still sitting in the device's tx > descriptor ring and other cores' mempool caches in fwd thread. > Thus the interrupt thread can not alloc LACP packet from pool. >=20 > Solution: Ensure that each slave'tx (LACPDU) mempool owns more than > n-tx-queues * n-tx-descriptor + fwd_core_num * > per-core-mmempool-flush-threshold mbufs. >=20 > Note that the LACP tx machine fuction is the only code that allocates > from a slave's private pool. It runs in the context of the interrupt > thread, and thus it has no mempool cache of its own. >=20 > Signed-off-by: Gaoxiang Liu >=20 > --- > v2: > * Fixed compile issues. >=20 > v3: > * delete duplicate code. >=20 > v4; > * Fixed some issues. > 1. total_tx_desc should use +=3D > 2. add detailed logs >=20 > v5: > * Fixed some issues. > 1. move CACHE_FLUSHTHRESH_MULTIPLIER to rte_eth_bond-8023ad.c > 2. use RTE_MIN > --- > drivers/net/bonding/rte_eth_bond_8023ad.c | 11 ++++++++--- > 1 file changed, 8 insertions(+), 3 deletions(-) >=20 > diff --git a/drivers/net/bonding/rte_eth_bond_8023ad.c > b/drivers/net/bonding/rte_eth_bond_8023ad.c > index ca50583d62..2c39b0d062 100644 > --- a/drivers/net/bonding/rte_eth_bond_8023ad.c > +++ b/drivers/net/bonding/rte_eth_bond_8023ad.c > @@ -1050,6 +1050,7 @@ bond_mode_8023ad_activate_slave(struct > rte_eth_dev *bond_dev, > uint32_t total_tx_desc; > struct bond_tx_queue *bd_tx_q; > uint16_t q_id; > + uint32_t cache_size; >=20 > /* Given slave mus not be in active list */ > RTE_ASSERT(find_slave_by_id(internals->active_slaves, > @@ -1100,11 +1101,15 @@ bond_mode_8023ad_activate_slave(struct > rte_eth_dev *bond_dev, > total_tx_desc +=3D bd_tx_q->nb_tx_desc; > } >=20 > +/* BONDING_8023AD_CACHE_FLUSHTHRESH_MULTIPLIER is the same as > + * CACHE_FLUSHTHRESH_MULTIPLIER already defined in rte_mempool.c */ > +#define BONDING_8023AD_CACHE_FLUSHTHRESH_MULTIPLIER 1.5 Very important comment. Thank you! May I suggest that a similar comment is added to the rte_mempool.c file, = so if CACHE_FLUSHTHRESH_MULTIPLIER is changed there, we don't forget to = change the copy-pasted code in the rte_eth_bond_8023ad.c file too. It = has previously been discussed changing it from 1.5 to 2 for symmetry = reasons. > + > + cache_size =3D RTE_MIN(RTE_MEMPOOL_CACHE_MAX_SIZE, 32); > + total_tx_desc +=3D rte_lcore_count() * cache_size * > BONDING_8023AD_CACHE_FLUSHTHRESH_MULTIPLIER; > snprintf(mem_name, RTE_DIM(mem_name), "slave_port%u_pool", > slave_id); > port->mbuf_pool =3D rte_pktmbuf_pool_create(mem_name, > total_tx_desc, > - RTE_MEMPOOL_CACHE_MAX_SIZE >=3D 32 ? > - 32 : RTE_MEMPOOL_CACHE_MAX_SIZE, > - 0, element_size, socket_id); > + cache_size, 0, element_size, socket_id); >=20 > /* Any memory allocation failure in initialization is critical > because > * resources can't be free, so reinitialization is impossible. */ > -- > 2.32.0 >=20