From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by dpdk.org (Postfix) with ESMTP id AC04E9137 for ; Fri, 28 Jul 2017 10:40:58 +0200 (CEST) Received: by mail-wm0-f41.google.com with SMTP id t201so119411989wmt.1 for ; Fri, 28 Jul 2017 01:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=84Br9HoS1lyi9Zxf3pSk04/uKXRWSl4FMZ3ByiFWDgw=; b=Eis8E3zm6fROTZAWp9RFV+2zAOXN9Pf7amFYZwhTxi2rHz50UEezioRE5R8U36kyIx YBuFxeew6gOQq+igigf+Pp20Y11NoTxHdAQzPn2rEdlD76nrSKO6KU+2/fohtwSI854z EdxcpUeV3ot2Vvid1Q43pAmHBxdnQCM3LAHD3vBOGwpZsivSD4vXbd7c/YZtToiL9Kse NCPbpB+AwXq4LVyJFFRgdmLqXV96tQom1d9SCEuvXuE0Ji3pInnf4W1u4h6L3jpwl4Mj hFiJHmDO76LqsASdRonqYaqDmSp7Yijfdj1CYZm0ikFKHkP3m19bqQfNgAiHvl6/zcOU nr0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=84Br9HoS1lyi9Zxf3pSk04/uKXRWSl4FMZ3ByiFWDgw=; b=cWwUWTqaBrXZ6SH1IEsui+RBeq1ot4ZX6LUgR2LWBRx7SMkgd1PiYZ6Jexh3zwUkbS nRKzev9jhhLNc1CEG0PBHwZBfgITBzQT9RZ+9CZsw9bZHhosK5gTiqk88ijomDgY4nbe xiyiXOWKflakW4oZMuvsIDeBQzfxeJA4Ujsd5HltiplFulfctIrFCDXh1V5g/A5qzfxn 6Y90zpLmH5cE7zZ7tJC4MGNV42PoV4hYLwxjQeBa/G/PbdxKVMCg0iBOOP11J7KNKh9d laTqaokXGm1/NbiqeLjy5X92ILoa0yj0SjWDUa0tdyfbpodS/a9RRh8jtxK6rKhuhlj4 NoYQ== X-Gm-Message-State: AIVw110hX2uAV6zGotpN6guhx5Z8DM6dQJggJQEeUhg6QHoTL/oZWjWa RyTLhGjfMYp2T1sq X-Received: by 10.28.18.194 with SMTP id 185mr4650726wms.19.1501231258212; Fri, 28 Jul 2017 01:40:58 -0700 (PDT) Received: from 6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id a3sm25933321wra.17.2017.07.28.01.40.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Jul 2017 01:40:57 -0700 (PDT) Date: Fri, 28 Jul 2017 10:40:48 +0200 From: Adrien Mazarguil To: chenchanghu Cc: "dev@dpdk.org" , "nelio.laranjeiro@6wind.com" , Zhoujingbin , "Zhoulei (G)" , Deng Kairong , Chenrujie , cuiyayun , "Chengwei (Titus)" , "Lixuan (Alex)" Message-ID: <20170728084048.GI19852@6wind.com> References: <859E1CB9FBF08C4B839DCF451B09C5032D62BFA3@dggeml505-mbx.china.huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <859E1CB9FBF08C4B839DCF451B09C5032D62BFA3@dggeml505-mbx.china.huawei.com> Subject: Re: [dpdk-dev] [disscussion] mlx4 driver MLX4_PMD_TX_MP_CACHE default vaule X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jul 2017 08:40:58 -0000 Hi, On Fri, Jul 28, 2017 at 07:58:48AM +0000, chenchanghu wrote: > Hi, > When I used the mlx4 pmd, I meet a problem about MLX4_PMD_TX_MP_CACHE vaule, which is used for Memory pool to Memory region translation. The detail test is descripted below. > 1.Test environmemt infomation: > a. Linux distribution: CentOS > b. dpdk version: dpdk-16.04 > c. Ethernet device : mlx4 VF > d. pmd info: mlx4 poll-mode-driver > > 2.Test diagram: > +----------------------+ +---------------------+ +-----------------------+ > | client1 | | client2 | ..... | clientN | > +----------------+----+ +-------+------------+ +----------+------------+ > | | | > | | | > +----v---------------v------------------------------v------+ > | share memory queue | > +----------------------------+------------------------------+ > | > | > +-------------------------------v--------------------------------+ > | server | > +-------------------------------+--------------------------------+ > | > | > +-------------------------------v--------------------------------+ > | dpdk rte_eth_tx_burst | > +-------------------------------+--------------------------------+ > | > +-------------------------------v---------------------------------+ > | mlx4 pmd driver | > +-----------------------------------------------------------------+ > a. Every client has one memory pool, all clients send message to server queue in the shared memory. > b. Server is only one thread, and mlx4 pmd use one tx queue. > > 3.Test step: > a. We start 30 clients, which means total mempool number reaching 30, every client will send 20 packets/second, every packet length is 10k.However,the server will do large packet segmentation before the packet send to rte_eth_tx_burst. > b. When we use the mlx4 pmd default MLX4_PMD_TX_MP_CACHE value which is 8, we found that the function 'rte_eth_tx_burst' cost about 40ms, which is mostly cost by the function 'ibv_reg_mr'. > c. Then we modify the MLX4_PMD_TX_MP_CACHE vaule to 32, which is configured the vaule 'CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE' in the config/common_base file, we found the function 'rte_eth_tx_burst' running time is less than 5ms. Yep, this is an old limitation that also affects the mlx5 PMD (and recently discussed [1]). > Would the community modify the default CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE value to 32 to adapt the scenario like above description, avoiding the slow operation when use too many mempool number > which is more than the CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE value in one tx queue. > > Please send your reply to chenchanghu@huawei.com, any suggestion is to be greatefully appreciated. > > 4. Patch: > diff --git a/config/common_base b/config/common_base > index a0580d1..af6ba47 100644 > --- a/config/common_base > +++ b/config/common_base > @@ -207,7 +207,7 @@ CONFIG_RTE_LIBRTE_MLX4_PMD=y > CONFIG_RTE_LIBRTE_MLX4_DEBUG=n > CONFIG_RTE_LIBRTE_MLX4_SGE_WR_N=4 > CONFIG_RTE_LIBRTE_MLX4_MAX_INLINE=0 > -CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=8 > +CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE=32 > CONFIG_RTE_LIBRTE_MLX4_SOFT_COUNTERS=1 In the past, using 8 as the default MP cache size was deemed a fair trade-off since most applications only used a small number of mempools. Also, recompiling DPDK after tweaking compilation options to suit specific applications needs was not much of an issue. Today DPDK is more library-like so setting compile-time parameters is not always possible for applications, this is why we're in the process of removing these options altogether as part of a major refactoring. They will become either run-time options (through device parameters) or fully automatic whenever possible. This should be the case for mlx4 in the 17.11 release (it's too late for 17.08). In the meantime, considering CONFIG_RTE_LIBRTE_MLX4_PMD is disabled by default and needs to be enabled manually, users that need a larger MP cache need to also increase CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE like you did. Because updating the default value cannot possibly satisfy all use cases, I think it's better to leave it as is for the time being in order to not affect existing applications. [1] http://dpdk.org/ml/archives/dev/2017-July/071405.html -- Adrien Mazarguil 6WIND