Re: 回复: [PATCH v2 0/3] Direct re-arming of buffers on receive side

[Ferruh Yigit <ferruh.yigit@amd.com>]

On 9/29/2022 7:19 AM, Feifei Wang wrote:
>> -----邮件原件-----
>> 发件人: Feifei Wang <feifei.wang2@arm.com>
>> 发送时间: Tuesday, September 27, 2022 10:48 AM
>> 抄送: dev@dpdk.org; nd <nd@arm.com>; Feifei Wang
>> <Feifei.Wang2@arm.com>
>> 主题: [PATCH v2 0/3] Direct re-arming of buffers on receive side
>>
>> Currently, the transmit side frees the buffers into the lcore cache and the
>> receive side allocates buffers from the lcore cache. The transmit side typically
>> frees 32 buffers resulting in 32*8=256B of stores to lcore cache. The receive
>> side allocates 32 buffers and stores them in the receive side software ring,
>> resulting in 32*8=256B of stores and 256B of load from the lcore cache.
>>
>> This patch proposes a mechanism to avoid freeing to/allocating from the
>> lcore cache. i.e. the receive side will free the buffers from transmit side
>> directly into it's software ring. This will avoid the 256B of loads and stores
>> introduced by the lcore cache. It also frees up the cache lines used by the
>> lcore cache.
>>
>> However, this solution poses several constraints:
>>
>> 1)The receive queue needs to know which transmit queue it should take the
>> buffers from. The application logic decides which transmit port to use to send
>> out the packets. In many use cases the NIC might have a single port ([1], [2],
>> [3]), in which case a given transmit queue is always mapped to a single
>> receive queue (1:1 Rx queue: Tx queue). This is easy to configure.
>>
>> If the NIC has 2 ports (there are several references), then we will have
>> 1:2 (RX queue: TX queue) mapping which is still easy to configure.
>> However, if this is generalized to 'N' ports, the configuration can be long.
>> More over the PMD would have to scan a list of transmit queues to pull the
>> buffers from.
>>
>> 2)The other factor that needs to be considered is 'run-to-completion' vs
>> 'pipeline' models. In the run-to-completion model, the receive side and the
>> transmit side are running on the same lcore serially. In the pipeline model.
>> The receive side and transmit side might be running on different lcores in
>> parallel. This requires locking. This is not supported at this point.
>>
>> 3)Tx and Rx buffers must be from the same mempool. And we also must
>> ensure Tx buffer free number is equal to Rx buffer free number:
>> (txq->tx_rs_thresh == RTE_I40E_RXQ_REARM_THRESH) Thus, 'tx_next_dd'
>> can be updated correctly in direct-rearm mode. This is due to tx_next_dd is a
>> variable to compute tx sw-ring free location.
>> Its value will be one more round than the position where next time free
>> starts.
>>
>> Current status in this patch:
>> 1)Two APIs are added for users to enable direct-rearm mode:
>>    In control plane, users can call 'rte_eth_txq_data_get' to get Tx sw_ring
>>    pointer and its txq_info (This avoid Rx load Tx data directly);
>>
>>    In data plane, users can  call 'rte_eth_rx_direct_rearm' to rearm Rx
>>    buffers and free Tx buffers at the same time (Currently it supports 1:1
>>    (rxq:txq) mapping:)
>> -----------------------------------------------------------------------
>>    control plane:
>>    	rte_eth_txq_data_get(*txq_data);
>>    data plane:
>>    	loop {
>>    		rte_eth_rx_direct_rearm(*txq_data){
>>       			for (i = 0; i <= 32; i++) {
>>         				rx.mbuf[i] = tx.mbuf[i];
>>         				initialize descs[i];
>>      			}
>> 		}
>> 		rte_eth_rx_burst;
>> 		rte_eth_tx_burst;
>>    	}
>> -----------------------------------------------------------------------
>> 2)The i40e driver is changed to do the direct re-arm of the receive
>>    side.
>> 3)L3fwd application is modified to enable direct rearm mode. Users can
>>    enable direct-rearm and map queues by input parameters.
>>
>> Testing status:
>> 1.The testing results for L3fwd are as follows:
>> -------------------------------------------------------------------
>> enabled direct rearm
>> -------------------------------------------------------------------
>> Arm:
>> N1SDP(neon path):
>> without fast-free mode		with fast-free mode
>> 	+15.09%				+4.2%
>>
>> Ampere Altra(neon path):
>> without fast-free mode		with fast-free mode
>> 	+10.9%				+14.6%
>> -------------------------------------------------------------------
>>
>> 2.The testing results for VPP-L3fwd are as follows:
>> -------------------------------------------------------------------
>> Arm:
>> N1SDP(neon path):
>> with direct re-arm mode enabled
>> 	+4.5%
>>
>> Ampere Altra(neon path):
>> with direct re-arm mode enabled
>>          +6.5%
>> -------------------------------------------------------------------
>>
>> Reference:
>> [1] https://store.nvidia.com/en-
>> us/networking/store/product/MCX623105AN-
>> CDAT/NVIDIAMCX623105ANCDATConnectX6DxENAdapterCard100GbECrypt
>> oDisabled/
>> [2] https://www.intel.com/content/www/us/en/products/sku/192561/intel-
>> ethernet-network-adapter-e810cqda1/specifications.html
>> [3] https://www.broadcom.com/products/ethernet-connectivity/network-
>> adapters/100gb-nic-ocp/n1100g
>>
>> V2:
>> 1. Use data-plane API to enable direct-rearm (Konstantin, Honnappa) 2. Add
>> 'txq_data_get' API to get txq info for Rx (Konstantin) 3. Use input parameter
>> to enable direct rearm in l3fwd (Konstantin) 4. Add condition detection for
>> direct rearm API (Morten, Andrew Rybchenko)
>>
> PING
> 
> Hi,
> 
> Would you please give some comments for this version?
> Thanks very much.
> 

Hi Feifei,

Because of the restrictions this feature has, although it gives a 
performance improvement on some specific tests I believe it doesn't have 
much practical usage, so I have constraints about the feature.