From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f180.google.com (mail-wr0-f180.google.com [209.85.128.180]) by dpdk.org (Postfix) with ESMTP id 12462235 for ; Tue, 4 Jul 2017 08:55:04 +0200 (CEST) Received: by mail-wr0-f180.google.com with SMTP id r103so243998572wrb.0 for ; Mon, 03 Jul 2017 23:55:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=gUxAV9GVSD4AYUCwPekfyzzqS+Ln5wYX1dvtezPJQkE=; b=T2/LSEvDS60AA0M50u0WvMhjYMnL8u4+q/wD6ffJWoR9AxAK2yuudVREIRv0SKOQc9 KjMpPD3f84lqUaBh3d3D2Pe0S6gf7JZUt9e1VRE+MXVb9W6TB1+i0otGLzLKlev4bxyE nwrX/hrFn0QI/98scLBNBP467XX9K91KUQU4AUphKC8sG5blJdwhzzkyqpw6Sz+7WV9T orqaah3GQJRdj+/VJZgajs5mjbWFPiz3j64EPZO9LhOEyMwkWoV4M3paP8ygqSncItJ6 qzKV+jy0vLvAjylQV3j2yemdj8lI+4deE+AsJiXWXhniXMPycmDB7RxJIWOZf6JiNzpd oHBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=gUxAV9GVSD4AYUCwPekfyzzqS+Ln5wYX1dvtezPJQkE=; b=VmKWzuzR6vXb3k/yfTl3cOFIPPuJP349/zO3ra/fYjSNwcHOeUucMNVhgTGT9QhFNT B4bqV6HhEPyyL4SCaHLiOeaR+hQwOzKpF74gzCAxwAMGti9GPLtb4RAlOHMHh1jB5v5d 4/cDEi187jx2r/mzjQMxECy17eKWv0AwNCwlCvrPhBI2SCzTsLBCE7ItVwDHNsdzJjFA wgDnN/iWpJYDnw6UkOLr8LzMJMauEBKZ8U9L7jP85VH7fKvlCuEAwjcUJgA/sJPpCZaf 92lKwk7gth9hKoQdHX/aeKWksyPecaaieYFF8w4hLOM3NoIUj/H6J9J1wD5UVA4CvEPN Kdcw== X-Gm-Message-State: AKS2vOyMQUZTHKGeMejXxcTtUT9kcmhcWabI+e/DqcXkBH/CtY+tWF0K qdrArkk0X71Xv2uz X-Received: by 10.223.172.67 with SMTP id v61mr38912006wrc.112.1499151304619; Mon, 03 Jul 2017 23:55:04 -0700 (PDT) Received: from autoinstall.dev.6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id 142sm5400278wmi.1.2017.07.03.23.55.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Jul 2017 23:55:04 -0700 (PDT) Date: Tue, 4 Jul 2017 08:54:54 +0200 From: =?iso-8859-1?Q?N=E9lio?= Laranjeiro To: Yongseok Koh Cc: Ferruh Yigit , "dev@dpdk.org" , Adrien Mazarguil Message-ID: <20170704065436.GA21379@autoinstall.dev.6wind.com> References: <20170628230403.10142-1-yskoh@mellanox.com> <1342e608a5a7c45b7af17e9228d6ce643e7ae40e.1498850005.git.yskoh@mellanox.com> <20170703140605.GE18305@autoinstall.dev.6wind.com> <4771B0EC-2A1E-4A0C-86C5-231289F3E0F2@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <4771B0EC-2A1E-4A0C-86C5-231289F3E0F2@mellanox.com> User-Agent: Mutt/1.5.23 (2014-03-12) Subject: Re: [dpdk-dev] [PATCH v2 3/5] net/mlx5: use buffer address for LKEY search X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jul 2017 06:55:05 -0000 On Mon, Jul 03, 2017 at 08:54:43PM +0000, Yongseok Koh wrote: > > > On Jul 3, 2017, at 7:06 AM, Nélio Laranjeiro wrote: > > > > On Fri, Jun 30, 2017 at 12:23:31PM -0700, Yongseok Koh wrote: > >> When searching LKEY, if search key is mempool pointer, the 2nd cacheline > >> has to be accessed and it even requires to check whether a buffer is > >> indirect per every search. Instead, using address for search key can reduce > >> cycles taken. And caching the last hit entry is beneficial as well. > >> > >> Signed-off-by: Yongseok Koh > >> --- > >> drivers/net/mlx5/mlx5_mr.c | 17 ++++++++++++++--- > >> drivers/net/mlx5/mlx5_rxtx.c | 39 +++++++++++++++++++++------------------ > >> drivers/net/mlx5/mlx5_rxtx.h | 4 +++- > >> drivers/net/mlx5/mlx5_txq.c | 3 +-- > >> 4 files changed, 39 insertions(+), 24 deletions(-) > >> > >> diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c > >> index 0a3638460..287335179 100644 > >> --- a/drivers/net/mlx5/mlx5_mr.c > >> +++ b/drivers/net/mlx5/mlx5_mr.c > >> @@ -265,18 +266,28 @@ txq_mp2mr_iter(struct rte_mempool *mp, void *arg) > >> struct txq_mp2mr_mbuf_check_data data = { > >> .ret = 0, > >> }; > >> + uintptr_t start; > >> + uintptr_t end; > >> unsigned int i; > >> > >> /* Register mempool only if the first element looks like a mbuf. */ > >> if (rte_mempool_obj_iter(mp, txq_mp2mr_mbuf_check, &data) == 0 || > >> data.ret == -1) > >> return; > >> + if (mlx5_check_mempool(mp, &start, &end) != 0) { > >> + ERROR("mempool %p: not virtually contiguous", > >> + (void *)mp); > >> + return; > >> + } > >> for (i = 0; (i != RTE_DIM(txq_ctrl->txq.mp2mr)); ++i) { > >> - if (unlikely(txq_ctrl->txq.mp2mr[i].mp == NULL)) { > >> + struct ibv_mr *mr = txq_ctrl->txq.mp2mr[i].mr; > >> + > >> + if (unlikely(mr == NULL)) { > >> /* Unknown MP, add a new MR for it. */ > >> break; > >> } > >> - if (txq_ctrl->txq.mp2mr[i].mp == mp) > >> + if (start >= (uintptr_t)mr->addr && > >> + end <= (uintptr_t)mr->addr + mr->length) > >> return; > >> } > >> txq_mp2mr_reg(&txq_ctrl->txq, mp, i); > > > > if (start >= (uintptr_t)mr->addr && > > end <= (uintptr_t)mr->addr + mr->length) > > > > Is this expected to have a memory region bigger than the memory pool > > space? I mean I was expecting to see strict equality in the addresses. > In mlx5_mp2mr(), start/end of a memory region are rounded up to make it > aligned to its hugepage size. > > struct ibv_mr * > mlx5_mp2mr(struct ibv_pd *pd, struct rte_mempool *mp) > { > [...] > /* Round start and end to page boundary if found in memory segments. */ > for (i = 0; (i < RTE_MAX_MEMSEG) && (ms[i].addr != NULL); ++i) { > uintptr_t addr = (uintptr_t)ms[i].addr; > size_t len = ms[i].len; > unsigned int align = ms[i].hugepage_sz; > > if ((start > addr) && (start < addr + len)) > start = RTE_ALIGN_FLOOR(start, align); > if ((end > addr) && (end < addr + len)) > end = RTE_ALIGN_CEIL(end, align); > } > Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND