DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Sanford, Robert" <rsanford@akamai.com>
To: "Min Hu (Connor)" <humin29@huawei.com>,
	Robert Sanford <rsanford2@gmail.com>,
	"dev@dpdk.org" <dev@dpdk.org>
Cc: "chas3@att.com" <chas3@att.com>
Subject: Re: [PATCH 3/7] net/bonding: change mbuf pool and ring allocation
Date: Mon, 20 Dec 2021 16:47:00 +0000	[thread overview]
Message-ID: <8C168DC6-124B-42B4-A766-11A6FA898256@akamai.com> (raw)
In-Reply-To: <ce2c4371-e719-5e1e-2e59-d23697d83fab@huawei.com>

Hello Connor,

Please see responses inline.

On 12/17/21, 10:44 PM, "Min Hu (Connor)" <humin29@huawei.com> wrote:

> > When the number of used tx-descs (0..255) + number of mbufs in the
> > cache (0..47) reaches 257, then allocation fails.
> > 
> > If I understand the LACP tx-burst code correctly, it would be
> > worse if nb_tx_queues > 1, because (assuming multiple tx-cores)
> > any queue/lcore could xmit an LACPDU. Thus, up to nb_tx_queues *
> > 47 mbufs could be cached, and not accessible from tx_machine().
> > 
> > You would not see this problem if the app xmits other (non-LACP)
> > mbufs on a regular basis, to expedite the clean-up of tx-descs
> > including LACPDU mbufs (unless nb_tx_queues tx-core caches
> > could hold all LACPDU mbufs).
> > 
> I think, we could not see this problem only because the mempool can
> offer much more mbufs than cache size on no-LACP circumstance.
>
> > If we make mempool's cache size 0, then allocation will not fail.
> How about enlarge the size of mempool, i.e., up to 4096 ? I think
> it can also avoid this bug.
> > 
> > A mempool cache for LACPDUs does not offer much additional speed:
> > during alloc, the intr thread does not have default mempool caches
> Why? as I know, all the core has its own default mempool caches ?

These are private mbuf pools that we use *only* for LACPDUs, *one*
mbuf per second, at most. (When LACP link peer selects long timeouts,
we get/put one mbuf every 30 seconds.)

There is *NO* benefit for the consumer thread (interrupt thread
executing tx_machine()) to have caches on per-slave LACPDU pools.
The interrupt thread is a control thread, i.e., a non-EAL thread.
Its lcore_id is LCORE_ID_ANY, so it has no "default cache" in any
mempool.

There is little or no benefit for active data-plane threads to have
caches on per-slave LACPDU pools, because on each pool, the producer
thread puts back, at most, one mbuf per second. There is not much
contention with the consumer (interrupt thread).

I contend that caches are not necessary for these private LACPDU
mbuf pools, but just waste RAM and CPU-cache. If we still insist
on creating them *with* caches, then we should add at least
(cache-size x 1.5 x nb-tx-queues) mbufs per pool.

 
> > Q: Why reserve one additional slot in the rx and tx rings?
> > 
> > A: rte_ring_create() requires the ring size N, to be a power of 2,
> > but it can only store N-1 items. Thus, if we want to store X items,
> Hi, Robert, could you describe it for me?
> I cannot understand why it
> "only store N -1 items". I check the source code, It writes:
> "The real usable ring size is *count-1* instead of *count* to
> differentiate a free ring from an empty ring."
> But I still can not get what it wrote.

I believe there is a mistake in the ring comments (in 3 places).
It would be better if they replace "free" with "full":
"... to differentiate a *full* ring from an empty ring."


> > we need to ask for (at least) X+1. Original code fails when the real
> > desired size is a power of 2, because in such a case, align32pow2
> > does not round up.
> > 
> > For example, say we want a ring to hold 4:
> > 
> >      rte_ring_create(... rte_align32pow2(4) ...)
> > 
> > rte_align32pow2(4) returns 4, and we end up with a ring that only
> > stores 3 items.
> > 
> >      rte_ring_create(... rte_align32pow2(4+1) ...)
> > 
> > rte_align32pow2(5) returns 8, and we end up with a ring that
> > stores up to 7 items, more than we need, but acceptable.
> To fix the bug, how about just setting the flags "RING_F_EXACT_SZ"

Yes, this is a good idea. I will look for examples or test code that
use this flag.

 --
Regards,
Robert Sanford



  reply	other threads:[~2021-12-20 16:47 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-15 18:19 [PATCH 0/7] net/bonding: fixes and LACP short timeout Robert Sanford
2021-12-15 18:19 ` [PATCH 1/7] net/bonding: fix typos and whitespace Robert Sanford
2021-12-21 19:57   ` [PATCH v2 0/8] net/bonding: fixes and LACP short timeout Robert Sanford
2021-12-21 19:57     ` [PATCH v2 1/8] net/bonding: fix typos and whitespace Robert Sanford
2022-02-04 15:06       ` Ferruh Yigit
2021-12-21 19:57     ` [PATCH v2 2/8] net/bonding: fix bonded dev configuring slave dev Robert Sanford
2021-12-21 19:57     ` [PATCH v2 3/8] net/bonding: change mbuf pool and ring creation Robert Sanford
2021-12-21 19:57     ` [PATCH v2 4/8] net/bonding: support enabling LACP short timeout Robert Sanford
2022-02-04 14:46       ` Ferruh Yigit
2021-12-21 19:57     ` [PATCH v2 5/8] net/bonding: add bond_8023ad and bond_alb to doc Robert Sanford
2022-02-04 14:48       ` Ferruh Yigit
2021-12-21 19:57     ` [PATCH v2 6/8] remove self from timers maintainers Robert Sanford
2022-03-08 23:26       ` Thomas Monjalon
2021-12-21 19:57     ` [PATCH v2 7/8] net/ring: add promisc and all-MC stubs Robert Sanford
2022-02-04 14:36       ` Ferruh Yigit
2022-02-04 14:49         ` Bruce Richardson
2022-02-11 19:57           ` Ferruh Yigit
2021-12-21 19:57     ` [PATCH v2 8/8] net/bonding: add LACP short timeout tests Robert Sanford
2022-02-04 14:49       ` Ferruh Yigit
2021-12-22  3:27     ` [PATCH v2 0/8] net/bonding: fixes and LACP short timeout Min Hu (Connor)
2022-01-11 16:41     ` Kevin Traynor
2022-02-04 15:09     ` Ferruh Yigit
2021-12-15 18:19 ` [PATCH 2/7] net/bonding: fix bonded dev configuring slave dev Robert Sanford
2021-12-15 18:19 ` [PATCH 3/7] net/bonding: change mbuf pool and ring allocation Robert Sanford
2021-12-16  8:59   ` Min Hu (Connor)
2021-12-17 19:49     ` Sanford, Robert
2021-12-18  3:44       ` Min Hu (Connor)
2021-12-20 16:47         ` Sanford, Robert [this message]
2021-12-21  2:01           ` Min Hu (Connor)
2021-12-21 15:31             ` Sanford, Robert
2021-12-22  3:25               ` Min Hu (Connor)
2021-12-15 18:19 ` [PATCH 4/7] net/bonding: support enabling LACP short timeout Robert Sanford
2021-12-15 18:19 ` [PATCH 5/7] net/bonding: add LACP short timeout to tests Robert Sanford
2021-12-15 18:20 ` [PATCH 6/7] net/bonding: add bond_8023ad and bond_alb to doc Robert Sanford
2021-12-15 18:20 ` [PATCH 7/7] Remove self from Timers maintainers Robert Sanford

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8C168DC6-124B-42B4-A766-11A6FA898256@akamai.com \
    --to=rsanford@akamai.com \
    --cc=chas3@att.com \
    --cc=dev@dpdk.org \
    --cc=humin29@huawei.com \
    --cc=rsanford2@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).