[dpdk-dev] [RFC] mempool: add non-IO flag

[Dmitry Kozlyuk <dkozlyuk@nvidia.com>]

We propose to add a mempool flag MEMPOOL_F_NON_IO to mark pools of objects that
will not be used with device IO and their memory for DMA.  This will allow
saving IOMMU entries by not mapping the memory used by such pools.

Immediate use case is MLX5 PMD.  The hardware has its internal IOMMU where PMD
registers the memory.  On the data path, PMD translates VA into a key consumed
by the device IOMMU.  It is impractical for the PMD to register all allocated
memory because of increased lookup cost both in HW and SW.  Most often mbuf
memory comes from mempools, so if PMD tracked them, it could almost always have
mbuf memory registered before an mbuf hits the PMD. The new flag would prevent
the PMD for registering memory that will never need it. Tracking the mempools
and dealing with them in MLX5 PMD is the next step after the proposed change.

A possible use case is IOMMU management in EAL.  Mempool could translate the new
flag to a hint to the memory manager, which would use it to skip adding IOMMU
entries in some cases.

It was considered to add MEMPOOL_F_IO with the opposite meaning. It would be
automatically set for pktmbuf pools; user would be able to set it for other
pools.  However, current assumption is that all DPDK memory is DMA-able,
it is controversial to have a flag asserting this fact.