From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5EDD9A034C for ; Sun, 30 Jan 2022 00:46:21 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D89E940E50; Sun, 30 Jan 2022 00:46:20 +0100 (CET) Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by mails.dpdk.org (Postfix) with ESMTP id 09AFB40041 for ; Sun, 30 Jan 2022 00:46:19 +0100 (CET) Received: by mail-wr1-f44.google.com with SMTP id e2so18150769wra.2 for ; Sat, 29 Jan 2022 15:46:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:from:date:message-id:subject:to; bh=kTySaU1cl4PLBTgZo9SLF00nIgk+W5MQV5jH7gdCoTM=; b=maOS6qldyO9CfUIE7Knu/6/rzmutk3SlK1wqNwsVbAwnaGC4BwUDeRMbzkziDHNGlN 5B2MrvJIqax4HWS9MRA+WVelUQ1uo564Vdtd5+ncUeig1XsoXrdsROpkl7Jn8d7WIuc5 Hzk7UVfMDNTU2StxGfgd+EF6fqBvqttTdZNwEy/Z/J2ii1dtNKBX/D73plXkY+Cyxb/x ICFL+uRYFHdQ2Po4jKY4c7aSSAXs7hfnw92u8tVJlsRolm1CEUYUFVvMPbbbm7j87Dcc mGUN4j3G5YMSch4v4kqhYw5LFiI7Bez31qL9vYPWZMrf/3Y9pHWjeD++5x/g42esrS7c d2cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=kTySaU1cl4PLBTgZo9SLF00nIgk+W5MQV5jH7gdCoTM=; b=nQHS40nijREbtR5qDwBvafjDJaflTPKsulxCvsONYEM8Jn5U+mpwYYWqE5/ufMblOI 4LDW1T//kgvGQLqYj9mEKFIHw0UAtMUaCcYxFHot7eIeMQeqUNdpOhox8F5ET8VtLO78 HAeE2lhVyTlcRBK8QxS69bq/vf6HaHq0ZxBuwFG5f0/SWIbfRIo2m5Gd8aG65vV1vUBA PcvrRArrWfAgNJqfnl0w1SWtaQIscUcxRCRcnybAtc4RaJsnSXoeuRoCPkS7Xh8aAZjW 8GA6WCV71PqBLAdjEljZJFq/OTH2NFKvJDocTlMOYCq5Fm8f2u+U9DmK1QVIY0i6CyoG +fvQ== X-Gm-Message-State: AOAM532UEfbZOyX2rJZn+MDgIypFpwYIDKXkQUROcp0RlXpf2ZuZ+fcx Uu9A4An69dRSo53BEK0AnF8bbLRmiUr5p/XhrPjlTS9dm7E= X-Google-Smtp-Source: ABdhPJwxhj6/ugrX1F4Lm1ITWokpEcJHbviFxGR6A7PNTz6L7fbMW17/ezDeaOmCmcV0An/50ZDbChF30wXH465R+Ms= X-Received: by 2002:adf:aade:: with SMTP id i30mr12167533wrc.179.1643499978243; Sat, 29 Jan 2022 15:46:18 -0800 (PST) MIME-Version: 1.0 From: fwefew 4t4tg <7532yahoo@gmail.com> Date: Sat, 29 Jan 2022 18:46:07 -0500 Message-ID: Subject: allocating a mempool w/ rte_pktmbuf_pool_create() To: users@dpdk.org Content-Type: multipart/alternative; boundary="000000000000643f5e05d6c128ea" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000643f5e05d6c128ea Content-Type: text/plain; charset="UTF-8" The API rte_pktmbuf_pool_create() https://doc.dpdk.org/api/rte__mbuf_8h.html#a593921f13307803b94bbb4e0932db962 at first glance seems minimal, complete. It's not. It's really not. The doc's link to `rte_mempool_create()` helps a little but not much. It lacks at least the following important facets. I would appreciate answers from whomever knows how DPDK thinks here: 1. Does cache_size include or exclude data_room_size? 2. Does cache_size include or exclude sizeof(struct rtre_mbuf)? 3. Does cache size include or exclude RTE_PKTMBUF_HEADROOM? 4. What lcore is the allocated memory pinned to? The lcore of the caller when this method is run? The answer here is important. If it's the lcore of the caller when called, this routine should be called in the lcore's entry point so it's on the right lcore the memory is intended. Calling it on the lcore that happens to be running main, for example, could have a bad side effect if it's different from where the memory will be ultimately used. 5. Which one of the formal arguments represents tail room indicated in https://doc.dpdk.org/guides/prog_guide/mbuf_lib.html#figure-mbuf1 My answers best I can tell follow: 1. Excludes 2. Excludes 3. Excludes 4. Caller does not enter into this situation; see below 5. Unknown. Perhaps if you want private data which corresponds to tail room in the diagram above one has to call rte_mempool_create() instead and focus on private_data_size. Discussion: Mempool creation is like malloc: you request the total number of absolute bytes required. The API will not add or remove bytes to the number you specify. Therefore the number you give must be inclusive of all needs including your payload, any DPDK overheader, headroom, tailroom, and so on. DPDK is not adding to the number you give for its own purposes. Clearer? Perhaps ... but what needs? Read on ... Unlike malloc rte_pktmbuf_pool_create() takes *n* the number of objects that memory will hold. Therefore the cache_size mod n should be 0. Indeed, some DPDK code like that of https://github.com/erpc-io/eRPC the author allocates one mempool per RXQ or TXQ where the memory requested and number of objects in that memory pool are appropriate for exactly one RXQ or exactly one TXQ. Clearly then the total amount of memory in a specific mempool divided by the number of objects it's intended to hold should mod to zero. This then begs the question, ok, if DPDK can do this what lcore is the memory pinned or cached for? The caller's lcore? If so it depends on what lcore one allocates the memory which can depend on where/when in the program's history the call is made. Note the API does not take a lcore argument. A *careful reading, however, suggests the eRPC code is misguided*. DPDK does not support creating a mempool for usage on one lcore. The doc reads the formal argument *cache_size* gives the *size of the per-core object cache. See rte_mempool_create() for details.* Also the doc reads: *the optimum size (in terms of memory usage) for a mempool is when n is a power of two minus one: n = (2^q - 1) *also contradicts a mod 0 scenario. Now I'm a little but not totally surprised here. Yes, in some applications TXQs and RXQs are bouncing around uniform packets. So telling DPDK I need X bytes and letting DPDK do the spade work of breaking up the memory for efficient per core access ultimately per RXQ or per TXQ is a real benefit. DPDK will give nice per lcore cache, lockless memory access. Gotta like that. Right? No. I might not. I might have half my TXQ and RXQs dealing with tiny mbufs/packets, and the other half dealing with completely different traffic of a completely different size and structure. So I might want memory pool allocation to be done on a smaller scale e.g. per RXQ/TXQ/lcore. DPDK doesn't seem to permit this. DPDK seems to require me to allocate for the largest possible application mbuf size possible for all cases and, then upon allocating a mbuf from the pool, *use as much or as few bytes of allocation as needed for the particular purpose at hand. *If that's what DPDK wants, fine. But I think it ought to be a hell of lot easy to see that's so. --000000000000643f5e05d6c128ea Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

My answers=C2=A0best I can t= ell follow:

1. Excludes
2. Excludes
3. Excludes<= /div>
4. Caller does not enter into this situation; see below
5. Unknown. Perhaps if you want private data which corresponds to tail roo= m in the diagram above one has to call rte_mempool_create() instead and foc= us on private_data_size.

Discussion:

Mempool creation is like= malloc: you request the total number of absolute bytes required. The API w= ill not add or remove bytes to the number you specify. Therefore the number= you give must be inclusive of all needs including your payload, any DPDK o= verheader, headroom, tailroom, and so on. DPDK is not adding to the number = you give for its own purposes. Clearer? Perhaps ... but what needs? Read on= ...

Unlike malloc=C2=A0rte_pktmbuf_pool_create() = takes n the number of objects that memory will hold. Therefore the c= ache_size mod n should be 0. Indeed, some DPDK code like that of https://github.com/erpc-io/eRPC the a= uthor allocates one mempool per RXQ or TXQ where the memory requested and n= umber of objects in that memory pool are appropriate for exactly one RXQ or= exactly one TXQ. Clearly then the total amount of memory in a specific mem= pool divided by the number of objects it's intended to hold should mod= =C2=A0to zero.=C2=A0

This then begs the question, = ok, if DPDK can do this what lcore is the memory pinned or cached for? The = caller's lcore? If so it depends on what lcore one allocates the memory= which can depend on where/when in the program's history the call is ma= de. Note the API does not take a lcore argument.

A= careful reading, however, suggests the eRPC code is misguided. DPDK= does not support creating a mempool for usage on one lcore. The doc reads = the formal argument cache_size=C2=A0gives the size of the per-cor= e object cache. See rte_mempool_create() for details.=C2=A0Also the doc= reads:=C2=A0the optimum size (in terms of memory usage) for a mempool i= s when n is a power of two minus one: n =3D (2^q - 1) also contradicts = a mod 0 scenario.

Now I'm a little=C2=A0but no= t totally=C2=A0surprised here. Yes, in some applications TXQs and RXQs are = bouncing around uniform packets. So telling DPDK I need X bytes and letting= DPDK do the spade work of breaking up the memory for efficient per core ac= cess ultimately per RXQ or per TXQ is a real benefit. DPDK will give nice p= er lcore cache, lockless memory access. Gotta like that. Right?

No. = I might not. I might have half my TXQ and RXQs dealing with tiny mbufs/pack= ets, and the other half dealing with completely different=C2=A0traffic of a= completely different size and structure. So I might want memory pool alloc= ation to be done on a smaller scale e.g. per RXQ/TXQ/lcore. DPDK doesn'= t seem to permit this. DPDK seems to require me to allocate for the largest= possible application mbuf size possible for all cases and, then upon alloc= ating a mbuf from the pool, use as much or as few bytes of allocation as= needed for the particular purpose at hand. If that's what DPDK wan= ts, fine. But I think it ought to be a hell of lot easy to see that's s= o.
--000000000000643f5e05d6c128ea--