From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 82438422D2; Sun, 19 Mar 2023 17:11:19 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1CE93410ED; Sun, 19 Mar 2023 17:11:19 +0100 (CET) Received: from forward500a.mail.yandex.net (forward500a.mail.yandex.net [178.154.239.80]) by mails.dpdk.org (Postfix) with ESMTP id 5FD89410EC for ; Sun, 19 Mar 2023 17:11:17 +0100 (CET) Received: from vla3-fd3176e90be6.qloud-c.yandex.net (vla3-fd3176e90be6.qloud-c.yandex.net [IPv6:2a02:6b8:c15:2584:0:640:fd31:76e9]) by forward500a.mail.yandex.net (Yandex) with ESMTP id A8B075E8EC; Sun, 19 Mar 2023 19:11:16 +0300 (MSK) Received: by vla3-fd3176e90be6.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id DBqNVcxdnGk1-CgPrs3Vs; Sun, 19 Mar 2023 19:11:16 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1679242276; bh=9Gk0m3cZdSb4GLGKVD7FvJJ1yIrEGFK9F1EtU2BzD9c=; h=From:In-Reply-To:Cc:Date:References:To:Subject:Message-ID; b=Jt+lJXX9foEbWl5xbwbUTfv1908Prgb6V2tHpbp0S7rTC9FZy2NJirdbI6Kaix1CF lRu3qamADZNzFIuQeL41pcnd5Tw4tD/TlI+0Wwo7dRfNU+NwbLcDGO9IwQDjMbvM7i BNMuwrOJUj21jlNtFqfTKovdGM3ehOpMza8zkOHM= Authentication-Results: vla3-fd3176e90be6.qloud-c.yandex.net; dkim=pass header.i=@yandex.ru Message-ID: <5ff6e7c0-e38f-cc8d-d08d-c612e5505184@yandex.ru> Date: Sun, 19 Mar 2023 16:11:13 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: =?UTF-8?B?UmU6IOWbnuWkjTogW1BBVENIIHYzIDIvM10gbmV0L2k0MGU6IGVuYWJs?= =?UTF-8?Q?e_direct_rearm_with_separate_API?= To: Feifei Wang , Konstantin Ananyev , Yuying Zhang , Beilei Xing , Ruifeng Wang Cc: "dev@dpdk.org" , nd , Honnappa Nagarahalli References: <20220420081650.2043183-1-feifei.wang2@arm.com> <20230104073043.1120168-1-feifei.wang2@arm.com> <20230104073043.1120168-3-feifei.wang2@arm.com> <445b0731-8e91-31dc-2ce5-90616c03f431@yandex.ru> <67ef44538dfd41af829c47891ced9e8e@huawei.com> Content-Language: en-US From: Konstantin Ananyev In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org >>>>>>> +int >>>>>>> +i40e_tx_fill_sw_ring(void *tx_queue, >>>>>>> + struct rte_eth_rxq_rearm_data *rxq_rearm_data) { >>>>>>> + struct i40e_tx_queue *txq = tx_queue; >>>>>>> + struct i40e_tx_entry *txep; >>>>>>> + void **rxep; >>>>>>> + struct rte_mbuf *m; >>>>>>> + int i, n; >>>>>>> + int nb_rearm = 0; >>>>>>> + >>>>>>> + if (*rxq_rearm_data->rearm_nb < txq->tx_rs_thresh || >>>>>>> + txq->nb_tx_free > txq->tx_free_thresh) >>>>>>> + return 0; >>>>>>> + >>>>>>> + /* check DD bits on threshold descriptor */ >>>>>>> + if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz & >>>>>>> + >> rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) != >>>>>>> + >>>>>> rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) >>>>>>> + return 0; >>>>>>> + >>>>>>> + n = txq->tx_rs_thresh; >>>>>>> + >>>>>>> + /* first buffer to free from S/W ring is at index >>>>>>> + * tx_next_dd - (tx_rs_thresh-1) >>>>>>> + */ >>>>>>> + txep = &txq->sw_ring[txq->tx_next_dd - (n - 1)]; >>>>>>> + rxep = rxq_rearm_data->rx_sw_ring; >>>>>>> + rxep += *rxq_rearm_data->rearm_start; >>>>>>> + >>>>>>> + if (txq->offloads & >> RTE_ETH_TX_OFFLOAD_MBUF_FAST_FREE) { >>>>>>> + /* directly put mbufs from Tx to Rx */ >>>>>>> + for (i = 0; i < n; i++, rxep++, txep++) >>>>>>> + *rxep = txep[0].mbuf; >>>>>>> + } else { >>>>>>> + for (i = 0; i < n; i++, rxep++) { >>>>>>> + m = rte_pktmbuf_prefree_seg(txep[i].mbuf); >>>> >>>> One thing I forgot to ask: >>>> What would happen if this mbuf belongs to different mempool (not one >>>> that we specify at rx_queue_setup())? >>>> Do we need to check it here? >>>> Or would it be upper layer constraint? >>>> Or...? >>>> >>> >>> First, 'different mempool' is valid for no FAST_FREE path in tx_free_buffers. >>> >>> If buffers belong to different mempool, we can have an example here: >>> Buffer 1 from mempool 1, its recycle path is: >>> ---------------------------------------------------------------------- >>> ------------------- 1. queue_setup: rearm from mempool 1 into Rx >>> sw-ring 2. rte_eth_Rx_burst: used by user app (Rx) 3. >>> rte_eth_Tx_burst: mount on Tx sw-ring 4. rte_eth_direct_rearm: free >>> into Rx sw-ring: >>> or >>> tx_free_buffers: free into mempool 1 (no fast_free path) >>> ---------------------------------------------------------------------- >>> ------------------- >>> >>> Buffer 2 from mempool 2, its recycle path is: >>> ---------------------------------------------------------------------- >>> ------------------- 1. queue_setup: rearm from mempool 2 into Rx >>> sw-ring 2. rte_eth_Rx_burst: used by user app (Rx) 3. >>> rte_eth_Tx_burst: mount on Tx sw-ring 4. rte_eth_direct_rearm: free >>> into Rx sw-ring >>> or >>> tx_free_buffers: free into mempool 2 (no fast_free_path) >>> ---------------------------------------------------------------------- >>> ------------------- >>> >>> Thus, buffers from Tx different mempools are the same for Rx. The >>> difference point is that they will be freed into different mempool if the >> thread uses generic free buffers. >>> I think this cannot affect direct-rearm mode, and we do not need to check >> this. >> >> I understand that it should work even with multiple mempools. >> What I am trying to say - user may not want to use mbufs from particular >> mempool for RX (while it is still ok to use it for TX). >> Let say user can have a separate mempool with small data-buffers (less then >> normal MTU) to send some 'special' paclets, or even use this memppol with >> small buffers for zero-copy updating of packet L2/L3 headers, etc. >> Or it could be some 'special' user provided mempool. >> That's why I wonder should we allow only mbufs from mempool that is >> assigned to that RX queue. > > Sorry for my misleading. If I understand correctly this time, you means a special > mempool. Maybe its buffer size is very small and this Tx buffer is generated from control plane. > > However, if we recycle this Tx buffer into Rx buffer ring, there maybe some error due to its > size is so small. > > Thus we can only allow general buffers which is valid for Rx buffer ring. Furthermore, this should be > user's responsibility to ensure the Tx recycling buffers should be valid. If we check this in the data plane, > it will cost a lot of CPU cycles. At last, what we can do is to add constraint in the notes to remind users. As I thought: in theory we can add 'struct rte_mempool *mp' into rte_eth_rxq_rearm_data. And then: if (mbuf->pool == rxq_rearm_data->mp) /* put mbuf into rearm buffer */ else /* free mbuf */ For the 'proper' config (when txq contains mbufs from expected mempool) the overhead will be minimal. In other case it might be higher, but still would work and no need for extra limitations. >> >>> >>>>>>> + if (m != NULL) { >>>>>>> + *rxep = m; >>>>>>> + nb_rearm++; >>>>>>> + } >>>>>>> + } >>>>>>> + n = nb_rearm; >>>>>>> + } >>>>>>> + >>>>>>> + /* update counters for Tx */ >>>>>>> + txq->nb_tx_free = (uint16_t)(txq->nb_tx_free + txq- >>> tx_rs_thresh); >>>>>>> + txq->tx_next_dd = (uint16_t)(txq->tx_next_dd + txq- >>> tx_rs_thresh); >>>>>>> + if (txq->tx_next_dd >= txq->nb_tx_desc) >>>>>>> + txq->tx_next_dd = (uint16_t)(txq->tx_rs_thresh - 1); >>>>>>> + >>>>>>> + return n; >>>>>>> +} >>>>>>> +