Re: [dpdk-dev] [PATCH v2 3/5] net/mlx5: use buffer address for LKEY search

["Nélio Laranjeiro" <nelio.laranjeiro@6wind.com>]

On Mon, Jul 03, 2017 at 08:54:43PM +0000, Yongseok Koh wrote:
> 
> > On Jul 3, 2017, at 7:06 AM, Nélio Laranjeiro <nelio.laranjeiro@6wind.com> wrote:
> > 
> > On Fri, Jun 30, 2017 at 12:23:31PM -0700, Yongseok Koh wrote:
> >> When searching LKEY, if search key is mempool pointer, the 2nd cacheline
> >> has to be accessed and it even requires to check whether a buffer is
> >> indirect per every search. Instead, using address for search key can reduce
> >> cycles taken. And caching the last hit entry is beneficial as well.
> >> 
> >> Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
> >> ---
> >> drivers/net/mlx5/mlx5_mr.c   | 17 ++++++++++++++---
> >> drivers/net/mlx5/mlx5_rxtx.c | 39 +++++++++++++++++++++------------------
> >> drivers/net/mlx5/mlx5_rxtx.h |  4 +++-
> >> drivers/net/mlx5/mlx5_txq.c  |  3 +--
> >> 4 files changed, 39 insertions(+), 24 deletions(-)
> >> 
> >> diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c
> >> index 0a3638460..287335179 100644
> >> --- a/drivers/net/mlx5/mlx5_mr.c
> >> +++ b/drivers/net/mlx5/mlx5_mr.c
> >> @@ -265,18 +266,28 @@ txq_mp2mr_iter(struct rte_mempool *mp, void *arg)
> >> 	struct txq_mp2mr_mbuf_check_data data = {
> >> 		.ret = 0,
> >> 	};
> >> +	uintptr_t start;
> >> +	uintptr_t end;
> >> 	unsigned int i;
> >> 
> >> 	/* Register mempool only if the first element looks like a mbuf. */
> >> 	if (rte_mempool_obj_iter(mp, txq_mp2mr_mbuf_check, &data) == 0 ||
> >> 			data.ret == -1)
> >> 		return;
> >> +	if (mlx5_check_mempool(mp, &start, &end) != 0) {
> >> +		ERROR("mempool %p: not virtually contiguous",
> >> +		      (void *)mp);
> >> +		return;
> >> +	}
> >> 	for (i = 0; (i != RTE_DIM(txq_ctrl->txq.mp2mr)); ++i) {
> >> -		if (unlikely(txq_ctrl->txq.mp2mr[i].mp == NULL)) {
> >> +		struct ibv_mr *mr = txq_ctrl->txq.mp2mr[i].mr;
> >> +
> >> +		if (unlikely(mr == NULL)) {
> >> 			/* Unknown MP, add a new MR for it. */
> >> 			break;
> >> 		}
> >> -		if (txq_ctrl->txq.mp2mr[i].mp == mp)
> >> +		if (start >= (uintptr_t)mr->addr &&
> >> +		    end <= (uintptr_t)mr->addr + mr->length)
> >> 			return;
> >> 	}
> >> 	txq_mp2mr_reg(&txq_ctrl->txq, mp, i);
> > 
> > if (start >= (uintptr_t)mr->addr &&
> >     end <= (uintptr_t)mr->addr + mr->length)
> > 
> > Is this expected to have a memory region bigger than the memory pool
> > space?  I mean I was expecting to see strict equality in the addresses.
> In mlx5_mp2mr(), start/end of a memory region are rounded up to make it
> aligned to its hugepage size.
> 
> struct ibv_mr *
> mlx5_mp2mr(struct ibv_pd *pd, struct rte_mempool *mp)
> {
> [...]
>         /* Round start and end to page boundary if found in memory segments. */
>         for (i = 0; (i < RTE_MAX_MEMSEG) && (ms[i].addr != NULL); ++i) {
>                 uintptr_t addr = (uintptr_t)ms[i].addr;
>                 size_t len = ms[i].len;
>                 unsigned int align = ms[i].hugepage_sz;
> 
>                 if ((start > addr) && (start < addr + len))
>                         start = RTE_ALIGN_FLOOR(start, align);
>                 if ((end > addr) && (end < addr + len))
>                         end = RTE_ALIGN_CEIL(end, align);
>         }
> 

Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>

-- 
Nélio Laranjeiro
6WIND