From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f44.google.com (mail-wm0-f44.google.com [74.125.82.44]) by dpdk.org (Postfix) with ESMTP id C2BBB9988 for ; Fri, 28 Jul 2017 14:00:34 +0200 (CEST) Received: by mail-wm0-f44.google.com with SMTP id m85so20005253wma.0 for ; Fri, 28 Jul 2017 05:00:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=xfVBjpmjC3WsUZaVwAMsG4YTXPL26nmkqa1VXMuWnak=; b=ipAL0Ic0qOsigOh9L8grUr5pBE7ATplxDXn5Xwpxqny0sNwfxxzBwT1qsWTjhANtUG utuw+3UMaMIcBvIaGiQXvpUtFzDdZycL6nBd/Wa2gjSu1btlf+Gytu+NEdDVYEaJqtrc PU/jbG/CKVnCxmtsgwKPQ+Gc98cIJd8MO7a5COEMfwsSnpKbpMk9Tcl9rz3MmLHix8t+ 7jrCznoGPtGXxZ3ghHuYhyup7bFN1aOxhrOFfuzWY7VX/ceNcjvFjMkJ84LPNnXzcc36 IpQ1tPw7KrHNyAI/8HrWQRv81s2LfZ0f6e/b/J7amcb39VC+HKV4yozvZeEoRwsJBIsW LlBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=xfVBjpmjC3WsUZaVwAMsG4YTXPL26nmkqa1VXMuWnak=; b=nCCSHRZ1GLfm3eNr0b0+EOoV2G1ViTcb8ZnPha5ymUYsH9IFHDruA2WCHMSAMy9O9U iFzDsvhGIN2LKefQymBfRQsHkhluYLKelUs5xpdPECUhJ6cQKxqvC4utWSqPbLWJiD3L 2G0xNoH2l0q86XxJ/z1YboD0x3bYcEW2GpGqxr1CRZeoxou2XX9gQERu/S/fVYt/hzPq XMBIUjI8YsLKcZFTsQWfzvtbqgFuAbk7Y+gxnrR5qhDv3xsaYQ6jafokZ1IUFQk+O5x+ NiQ+ndX2cLgaIrZh4k05RmSXpwmzhW8NdjqI8GDTYQXoBK2ChqyeT335F3H1f4trBwYl SZMg== X-Gm-Message-State: AIVw111c9ydvuz43CiRNbPBEmh3Zm5nV9BE9jkcs2HJWc3AHIp1YBRBl 2BtSizKvL1b0d2Eo X-Received: by 10.28.140.67 with SMTP id o64mr5821000wmd.68.1501243234113; Fri, 28 Jul 2017 05:00:34 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 199sm4302244wmu.0.2017.07.28.05.00.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Jul 2017 05:00:33 -0700 (PDT) Date: Fri, 28 Jul 2017 14:00:18 +0200 From: Adrien Mazarguil To: chenchanghu Cc: "dev@dpdk.org" , "nelio.laranjeiro@6wind.com" , Zhoujingbin , "Zhoulei (G)" , Deng Kairong , Chenrujie , cuiyayun , "Chengwei (Titus)" , "Lixuan (Alex)" , "Lilijun (Jerry)" Message-ID: <20170728120018.GJ19852@6wind.com> References: <859E1CB9FBF08C4B839DCF451B09C5032D62C090@dggeml505-mbx.china.huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <859E1CB9FBF08C4B839DCF451B09C5032D62C090@dggeml505-mbx.china.huawei.com> Subject: Re: [dpdk-dev] [disscussion] mlx4 driver MLX4_PMD_TX_MP_CACHE default vaule X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jul 2017 12:00:35 -0000 Hi Changhu, On Fri, Jul 28, 2017 at 10:52:45AM +0000, chenchanghu wrote: > Hi Adrien, > Thanks very much! I have got the question about MLX4_PMD_TX_MP_CACHE value, we will modify this value to suit our applications. > However, in the 2 clients or more clients test, we found that the function 'txq->if_qp->send_pending' and 'txq->if_qp->send_flush(txq->qp)' in 'mlx4_tx_burst' probabilistic cost almost *5ms* each function . The probability is about 1/50000, which means every 50000 packets sending appeared once. > Does this phenomenon is normal? Or do we ignored some configurations that not showed documented? 5 ms for these function calls is strange and certainly not normal. Are you sure this time is spent in send_pending()/send_flush() and not in mlx4_tx_burst() itself? Given the MP cache size and number of mempools involved in your setup, cache look-up might be longer than normal, but this alone does not explain it. Might be something else, such as: - txq_mp2mr() fails to register a mempool of one of these packets for some reason (chunked mempool?) Enable CONFIG_RTE_LIBRTE_MLX4_DEBUG and look for "unable to get MP <-> MR association" messages. - You've enabled TX inline mode using a large value and CPU cycles are wasted by the PMD doing memcpy() on large packets. Don't enable inline TX (set CONFIG_RTE_LIBRTE_MLX4_MAX_INLINE to 0). - Sent packets have too many segments (more than MLX4_PMD_SGE_WR_N). This is super expensive as the PMD needs to linearize extra segments. You can set MLX4_PMD_SGE_WR_N to the next power of two (8), however beware doing so will degrade performance. This might also be caused by external factors that depend on the application or the host system, if for instance DPDK memory is spread across NUMA nodes. Make sure it's not the case. -- Adrien Mazarguil 6WIND