From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f44.google.com (mail-wm0-f44.google.com [74.125.82.44]) by dpdk.org (Postfix) with ESMTP id 8A8C9376C for ; Mon, 17 Jul 2017 22:47:28 +0200 (CEST) Received: by mail-wm0-f44.google.com with SMTP id b134so5795160wma.0 for ; Mon, 17 Jul 2017 13:47:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=BndqGhOQ9ZLlJ+QmE21IGguA/trtgAWf3BA7W0YnCQo=; b=oveKnxoXczZoIv06aHS+wDdNAL+qTSKy+h+SpuZCcfhxvPvjBC22Jo4/lkHWIV/d6M PKsiNLyfPnBdILmnJw84pm/bBVn/1RMJeh3eqoGWBWfLuc6FAaoQM3r2ltvXBgjHAVVQ vYMQ8RGLUc0NshPjYiDjoVQW0l7+NkV/te7XfdkYwFRwkgtx5EExwgeVZH5T0GXIFjZf /iq6SAm6e7KLzwkh89030UtReO81j1hKIl+W/up/wIixXXVR5z9HiB0z1BcQQ/oQ7+f7 5FgP1EOFfQiDEcLnVLN1K17lGPj7bVFb16cyzTeosb0ocuWqtKoEpFeo2aX3VncIYrAK upqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=BndqGhOQ9ZLlJ+QmE21IGguA/trtgAWf3BA7W0YnCQo=; b=prV/dZPV67cxQXR76thjTUsVqzhPQ3a2Tu9sEPJA57PEKvxGTDEYSfeGz8tiktn/0A 16V812zUHHe2F1oIfN/qaUM7cuL1m3tL/FTPnqRhzurGi9sqGrJUxuaFz30oYjdLGQo1 QDRYAWZv0TNT0HCL2YGZEtgue9DQEnBXTdNY3vaVwK25HXRJkZbA/ITQhGPZxHR6BjYq hrcgIanYZrrS99SKbyOtIT+0J9t3XLGLqDIEKfJYSxl3Qixw5gu1cL5KJNjfxHJiYOx0 VBW4IW91Nxd/Z3zkukuMBtTVzyqXu5JWNd38k6X8Fmvl/QiYra6N3lOG72JACT8OeypE LOmg== X-Gm-Message-State: AIVw111wC6txb6SLxXlRlYUjCnrNghrbssPg+mEB4AMP+Q8obi1kw1qq +/MAx1FuUAaaou9AzQwl9CKc X-Received: by 10.28.107.69 with SMTP id g66mr6013074wmc.110.1500324448208; Mon, 17 Jul 2017 13:47:28 -0700 (PDT) Received: from shalom (bzq-164-168-31-206.red.bezeqint.net. [31.168.164.206]) by smtp.gmail.com with ESMTPSA id 21sm11103328wmo.16.2017.07.17.13.47.26 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 17 Jul 2017 13:47:27 -0700 (PDT) Date: Mon, 17 Jul 2017 23:02:24 +0200 From: =?iso-8859-1?Q?N=E9lio?= Laranjeiro To: Sagi Grimberg Cc: dev@dpdk.org, Shahaf Shuler , Yongseok Koh , Roy Shterman , Alexander Solganik Message-ID: <20170717210222.j4dwxiujqdlqhlp2@shalom> References: <75d08202-1882-7660-924c-b6dbb4455b88@grimberg.me> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <75d08202-1882-7660-924c-b6dbb4455b88@grimberg.me> User-Agent: NeoMutt/20170113 (1.7.2) Subject: Re: [dpdk-dev] Question on mlx5 PMD txq memory registration X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Jul 2017 20:47:28 -0000 Hello Sagi, On Mon, Jul 17, 2017 at 04:29:34PM +0300, Sagi Grimberg wrote: > Hi, > > Looking at the code, it looks like mlx5 keeps a MR cache per TX queue > (each MR registers a rte_mempool). > > Once a TX queue is created, mlx5 scans existing mempools and > pre-registers a MR for each mempool it meets (using rte_mempool_walk). > For each MR registration that exceeds the TX queue cache, it removes the > first entry from the cache and deregisters that MR (in txq_mp2mr_reg). > > Now on TX burst, if the driver sees a mbuf from an unknown mempool, it > registers the mempool on the fly and again *deregister* the first MR in > the cache. > > My question is, what guarantees that no inflight send operations posted > on the TX queue when we deregister and remove a MR from the cache? There is none, if you send a burst of 9 packets each one coming from a different mempool the first one will be dropped. > AFAICT, it is the driver responsibility to guarantee to never deregister > a memory region that has inflight send operations posted, otherwise > the send operation *will* complete with a local protection error. Is > that taken care of? Up to now we have assumed that the user knows its application needs and he can increase this cache size to its needs through the configuration item. This way this limit and guarantee becomes true. > Another question, why is the MR cache maintained per TX queue and not > per device? If the application starts N TX queues then a single mempool > will be registered N times instead of just once. Having lots of MR > instances will pollute the device ICMC pretty badly. Am I missing > something? Having this cache per device needs a lock on the device structure while threads are sending packets. Having such locks cost cycles, that is why the cache is per queue. Another point is, having several mempool per device is something common, whereas having several mempool per queues is not, it seems logical to have this cache per queue for those two reasons. I am currently re-working this part of the code to improve it using reference counters instead. The cache will remain for performance purpose. This will fix the issues you are pointing. Are you facing some kind of issue? Maybe you can share it, it can help to improve things. Thanks, -- Nélio Laranjeiro 6WIND