From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <sagigrim@gmail.com>
Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com
 [209.85.128.193]) by dpdk.org (Postfix) with ESMTP id A5624377E
 for <dev@dpdk.org>; Mon, 17 Jul 2017 15:29:46 +0200 (CEST)
Received: by mail-wr0-f193.google.com with SMTP id 40so2679230wrw.0
 for <dev@dpdk.org>; Mon, 17 Jul 2017 06:29:46 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:to:from:subject:message-id:date:user-agent
 :mime-version:content-language:content-transfer-encoding;
 bh=PtE8nOJ3la60K75cep/YW8upnSgqdygmBnmU79tS5Lg=;
 b=nzh0Ztd3q344VpTRW/N/ExZl6C3jcFeaChzyy55MkkZn7USBZZbgl652SEwaPTQ9gw
 97fWVs5yJyZ3WUM2qExSeLPUjXFDyLUjAEGMY5ktgmd7ew6hylAQzj3foXxIvn7wQ70n
 6RuTqf7OXP1lr2/9EcmORyV29RL1T95OpoGFLVHBgivD0pGCnom+DgQaOkf9/UvB7Ml4
 F6DMJqUmDDrPy/If+VSSzCyFf0oXkF1qTGVgndzoURlGJ2bZzw9N7sbTVaZjxLhksx/1
 ILwTSQC+q5s+FvVfak+/vEcDvKz1qwRSsM6vt1WyMV4WTrvRPouVBROff64MwZzhA7VI
 uPUA==
X-Gm-Message-State: AIVw1139JzbaqaGtcWZ8xRCIkwlAEybxLAWoWxekflqjZT7NEYTsOqHW
 i6jrgiQJkvzpKN1SlUc=
X-Received: by 10.223.161.219 with SMTP id v27mr10858698wrv.135.1500298186427; 
 Mon, 17 Jul 2017 06:29:46 -0700 (PDT)
Received: from [192.168.64.116] (bzq-82-81-101-184.red.bezeqint.net.
 [82.81.101.184])
 by smtp.gmail.com with ESMTPSA id z190sm11941519wmz.31.2017.07.17.06.29.40
 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
 Mon, 17 Jul 2017 06:29:45 -0700 (PDT)
To: dev@dpdk.org, Shahaf Shuler <shahafs@mellanox.com>,
 =?UTF-8?Q?N=c3=a9lio_Laranjeiro?= <nelio.laranjeiro@6wind.com>,
 Yongseok Koh <yskoh@mellanox.com>, Roy Shterman <roys@lightbitslabs.com>,
 Alexander Solganik <sashas@lightbitslabs.com>
From: Sagi Grimberg <sagi@grimberg.me>
Message-ID: <75d08202-1882-7660-924c-b6dbb4455b88@grimberg.me>
Date: Mon, 17 Jul 2017 16:29:34 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.2.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Subject: [dpdk-dev] Question on mlx5 PMD txq memory registration
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 17 Jul 2017 13:29:47 -0000

Hi,

Looking at the code, it looks like mlx5 keeps a MR cache per TX queue
(each MR registers a rte_mempool).

Once a TX queue is created, mlx5 scans existing mempools and
pre-registers a MR for each mempool it meets (using rte_mempool_walk).
For each MR registration that exceeds the TX queue cache, it removes the
first entry from the cache and deregisters that MR (in txq_mp2mr_reg).

Now on TX burst, if the driver sees a mbuf from an unknown mempool, it
registers the mempool on the fly and again *deregister* the first MR in
the cache.

My question is, what guarantees that no inflight send operations posted
on the TX queue when we deregister and remove a MR from the cache?

AFAICT, it is the driver responsibility to guarantee to never deregister
a memory region that has inflight send operations posted, otherwise
the send operation *will* complete with a local protection error. Is
that taken care of?

Another question, why is the MR cache maintained per TX queue and not
per device? If the application starts N TX queues then a single mempool
will be registered N times instead of just once. Having lots of MR
instances will pollute the device ICMC pretty badly. Am I missing
something?

Thanks,
Sagi.