[dpdk-dev] 答复: [disscussion] mlx4 driver MLX4_PMD_TX_MP_CACHE default value

[dpdk-dev] 答复: [disscussion] mlx4 driver MLX4_PMD_TX_MP_CACHE default value

[chenchanghu]

Hi Adrien,
  Your suggestion is appreciated *very much*.
 1)  We can make sure the 5ms time spent is in  the function send_pending()/send_flush().
 The test code is below remarks:
 	gettimeofday(&start_time, NULL);
	err = txq->if_qp->send_pending
		(txq->qp,
		 addr,
		 length,
		 lkey,
		 send_flags);
	gettimeofday(&end_time, NULL);
	time_process(start_time, end_time, "txq->if_qp->send_pending");		

	/* Ring QP doorbell. */
   	gettimeofday(&start_time, NULL);
  	err = txq->if_qp->send_flush(txq->qp);
        	gettimeofday(&end_time, NULL);
   	time_process(start_time, end_time, "txq->if_qp->send_flush");

2) The test function is the *small packet really executed*  which would not do segmentation. So it maybe not the reason .
3) The config CONFIG_RTE_LIBRTE_MLX4_MAX_INLINE we used is 0. And it also maybe not the reason .
4) *We will Enable CONFIG_RTE_LIBRTE_MLX4_DEBUG, and look for the message 'send_flush() failed with error ', maybe the send_flush function returned error.* It maybe the reason.
5) *We tested in VM, so the cpu is vcpu, and we didn't bind the vcpu threads to specific cpu core in the host machine,  so maybe the instance DPDK memory is spread across NUMA nodes. We will bind  the vcpu threads to one NUMA node and test is again.* It maybe the reason too.
Thanks very much again!

-----邮件原件-----
发件人: Adrien Mazarguil [mailto:adrien.mazarguil@6wind.com] 
发送时间: 2017年7月28日 20:00
收件人: chenchanghu <chenchanghu@huawei.com>
抄送: dev@dpdk.org; nelio.laranjeiro@6wind.com; Zhoujingbin <zhoujingbin@huawei.com>; Zhoulei (G) <stone.zhou@huawei.com>; Deng Kairong <dengkairong@huawei.com>; Chenrujie <chenrujie@huawei.com>; cuiyayun <cuiyayun@huawei.com>; Chengwei (Titus) <titus.chengwei@huawei.com>; Lixuan (Alex) <Awesome.li@huawei.com>; Lilijun (Jerry) <jerry.lilijun@huawei.com>
主题: Re: [disscussion] mlx4 driver MLX4_PMD_TX_MP_CACHE default vaule

Hi Changhu,

On Fri, Jul 28, 2017 at 10:52:45AM +0000, chenchanghu wrote:
> Hi   Adrien,
> Thanks very much! I have got the  question about MLX4_PMD_TX_MP_CACHE value, we will modify this value to suit our applications.
>   However, in the 2 clients or more clients test, we found that the function 'txq->if_qp->send_pending' and 'txq->if_qp->send_flush(txq->qp)'  in 'mlx4_tx_burst'   probabilistic cost almost *5ms* each function . The probability is about 1/50000, which means every 50000 packets sending appeared once.
>   Does this phenomenon is normal? Or do we ignored some configurations that not showed documented?

5 ms for these function calls is strange and certainly not normal. Are you sure this time is spent in send_pending()/send_flush() and not in
mlx4_tx_burst() itself?

Given the MP cache size and number of mempools involved in your setup, cache look-up might be longer than normal, but this alone does not explain it.
Might be something else, such as:

- txq_mp2mr() fails to register a mempool of one of these packets for some
  reason (chunked mempool?) Enable CONFIG_RTE_LIBRTE_MLX4_DEBUG and look
  for "unable to get MP <-> MR association" messages.

- You've enabled TX inline mode using a large value and CPU cycles are
  wasted by the PMD doing memcpy() on large packets. Don't enable inline TX
  (set CONFIG_RTE_LIBRTE_MLX4_MAX_INLINE to 0).

- Sent packets have too many segments (more than MLX4_PMD_SGE_WR_N). This is
  super expensive as the PMD needs to linearize extra segments. You can set
  MLX4_PMD_SGE_WR_N to the next power of two (8), however beware doing so
  will degrade performance.

This might also be caused by external factors that depend on the application or the host system, if for instance DPDK memory is spread across NUMA nodes. Make sure it's not the case.

--
Adrien Mazarguil
6WIND