Dmitry,

"On the contrary: rte_pktmbuf_pool_create() takes the amount
of usable memory (dataroom) and adds space for rte_mbuf and the headroom.
Furthermore, the underlying rte_mempool_create() ensures element (mbuf)
alignment, may spread the elements between pages, etc."

Thanks. This is a crucial correction to my erroneous statement.

I'd like to press-on then with one of my questions that, after some additional thought

is answered however implicitly. For the benefit of other programmers who are new

to this work. I'll explain. If wrong, please hammer on it.

The other crucial insight is: so long as memory is allocated on the same NUMA

node as the RXQ/TXQ runs that ultimately uses it, there is only marginal performance

advantage to having per-core caching of mbufs in a mempool as provided by the
private_data_size formal argument in rte_mempool_create() here:

https://doc.dpdk.org/api/rte__mempool_8h.html#a503f2f889043a48ca9995878846db2fd

In fact the API doc should really point out the advantage; perhaps it eliminates some

cache sloshing to get the last few percent of performance. It probably is not a major

factor in latency or bandwidth with or without private_data_size==0.

Memory access from an lcore x (aka H/W thread, vCPU) on NUMA N is fairly unchanged

to any other distinct lcore y != x provided y also runs on N and the memory was allocated

for N. Therefore, lcore affinity to a mempool is pretty much a red herring.

Consider this code which originally I used as indicative of good mempool creation,

but upon further thinking got me confused:

https://github.com/erpc-io/eRPC/blob/master/src/transport_impl/dpdk/dpdk_init.cc#L76

for (size_t i = 0; i < kMaxQueuesPerPort; i++) {

const std::string pname = get_mempool_name(phy_port, i);

rte_mempool *mempool =

rte_pktmbuf_pool_create(pname.c_str(), kNumMbufs, 0 /* cache */,

0 /* priv size */, kMbufSize, numa_node);

This has the appearance of creating one mempool per each RXQ and each TXQ. And in

fact this is what it does. The programmer here ensures the numa_node passed in as the

last argument is the same numa_node the RXQ/TXQ eventually runs. Since each lcore

has its own mempool and because rte_pktmbuf_create never calls into rte_mempool_create()

with a non-zero private_data_size, per lcore caching doesn't arise. (I briefly checked

mbuf/rte_mbuf.c to confirm). Indeed lcore v. mempool affinity is irrelevant provided the RXQ

for a given mempool runs on the same numa_node as specified in the last argument to

rte_pktmbuf_pool_create.

Let's turn then to a larger issue: what happens if different RXQ/TXQs have radically different

needs?

As the code above illustrates, one merely allocates a size appropriate to an individual RXQ/TXQ
by changing the count and size of mbufs ---- which is as simple as it can get. You have 10 queues

each with their own memory needs? OK, then allocate one memory pool for each. None of the other

9 queues will have that mempool pointer. Each queue will use the mempool only that was specified

for it. To beat a dead horse just make sure the numa_node in the allocation and the numa node which

will ultimately run the RXQ/TXQ are the same.

On Sat, Jan 29, 2022 at 8:23 PM Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:

2022-01-29 18:46 (UTC-0500), fwefew 4t4tg:
[...]
> 1. Does cache_size include or exclude data_room_size?
> 2. Does cache_size include or exclude sizeof(struct rtre_mbuf)?
> 3. Does cache size include or exclude RTE_PKTMBUF_HEADROOM?

Cache size is measured in the number of elements, irrelevant of their size.
It is not a memory size, so the questions above are not really meaningful.

> 4. What lcore is the allocated memory pinned to?

Memory is associated with a NUMA node (DPDK calls it "socket"), not an lcore.
Each lcore belongs to one NUMA node, see rte_lcore_to_socket_id().

> The lcore of the caller
> when this method is run? The answer here is important. If it's the lcore of
> the caller when called, this routine should be called in the lcore's entry
> point so it's on the right lcore the memory is intended. Calling it on the
> lcore that happens to be running main, for example, could have a bad side
> effect if it's different from where the memory will be ultimately used.

The NUMA node is controlled by "socket_id" parameter.
Your considerations are correct, often you should create separate mempools
for each NUMA node to avoid this performance issue. (You should also
consider which NUMA node each device belongs to.)

> 5. Which one of the formal arguments represents tail room indicated in
> https://doc.dpdk.org/guides/prog_guide/mbuf_lib.html#figure-mbuf1
[...]
> 5. Unknown. Perhaps if you want private data which corresponds to tail room
> in the diagram above one has to call rte_mempool_create() instead and focus
> on private_data_size.

Incorrect; tail room is simply an unused part at the end of the data room.
Private data is for the entire mempool, not for individual mbufs.

> Mempool creation is like malloc: you request the total number of absolute
> bytes required. The API will not add or remove bytes to the number you
> specify. Therefore the number you give must be inclusive of all needs
> including your payload, any DPDK overheader, headroom, tailroom, and so on.
> DPDK is not adding to the number you give for its own purposes. Clearer?
> Perhaps ... but what needs? Read on ...

On the contrary: rte_pktmbuf_pool_create() takes the amount
of usable memory (dataroom) and adds space for rte_mbuf and the headroom.
Furthermore, the underlying rte_mempool_create() ensures element (mbuf)
alignment, may spread the elements between pages, etc.

[...]
> No. I might not. I might have half my TXQ and RXQs dealing with tiny
> mbufs/packets, and the other half dealing with completely different traffic
> of a completely different size and structure. So I might want memory pool
> allocation to be done on a smaller scale e.g. per RXQ/TXQ/lcore. DPDK
> doesn't seem to permit this.

You can create different mempools for each purpose
and specify the proper mempool to rte_eth_rx_queue_setup().
When creating them, you can and should also take NUMA into account.
Take a look at init_mem() function of examples/l3fwd.