[dpdk-dev] Question on mlx5 PMD txq memory registration

[Sagi Grimberg <sagi@grimberg.me>]

Hi,

Looking at the code, it looks like mlx5 keeps a MR cache per TX queue
(each MR registers a rte_mempool).

Once a TX queue is created, mlx5 scans existing mempools and
pre-registers a MR for each mempool it meets (using rte_mempool_walk).
For each MR registration that exceeds the TX queue cache, it removes the
first entry from the cache and deregisters that MR (in txq_mp2mr_reg).

Now on TX burst, if the driver sees a mbuf from an unknown mempool, it
registers the mempool on the fly and again *deregister* the first MR in
the cache.

My question is, what guarantees that no inflight send operations posted
on the TX queue when we deregister and remove a MR from the cache?

AFAICT, it is the driver responsibility to guarantee to never deregister
a memory region that has inflight send operations posted, otherwise
the send operation *will* complete with a local protection error. Is
that taken care of?

Another question, why is the MR cache maintained per TX queue and not
per device? If the application starts N TX queues then a single mempool
will be registered N times instead of just once. Having lots of MR
instances will pollute the device ICMC pretty badly. Am I missing
something?

Thanks,
Sagi.