[dpdk-dev] DPDK Max Mbuf Allocation

[dpdk-dev] DPDK Max Mbuf Allocation

[Kamaraj P]

Hello All,

Would like to understand or if there are any guidelines to allocate the max
no of mbuf per NIC ?
For example, if i have defined as below:
#define RX_RING_SIZE 1024
#define TX_RING_SIZE 1024

The Maximum RX/TX queues can be defined as 8 per NIC, What would be the max
no of mbuf can be allocated per NIC ?
Please share if there is any guildliness or any limitation to increase the
mbuf ?

Thanks,
Kamaraj

Re: [dpdk-dev] DPDK Max Mbuf Allocation

[Ferruh Yigit]

On 9/13/2021 5:56 AM, Kamaraj P wrote:
> Hello All,
> 
> Would like to understand or if there are any guidelines to allocate the max
> no of mbuf per NIC ?
> For example, if i have defined as below:
> #define RX_RING_SIZE 1024
> #define TX_RING_SIZE 1024
> 
> The Maximum RX/TX queues can be defined as 8 per NIC, What would be the max
> no of mbuf can be allocated per NIC ?
> Please share if there is any guildliness or any limitation to increase the
> mbuf ?
> 

Hi Kamaraj,

Max number of the queues and max number of the descriptors per queue depends to
HW and changes form HW to HW.
This information is shared by the PMDs that application needs to take into
account. For example the descriptor limitations are provided by
'rx_desc_lim'/'tx_desc_lim' etc.

After descriptor number is defined, testpmd uses the mbuf count as following,
which can be taken as sample:

nb_mbuf_per_pool = RTE_TEST_RX_DESC_MAX + RTE_TEST_TX_DESC_MAX + MAX_PKT_BURST +
                   (nb_lcores * mb_mempool_cache);




> Thanks,
> Kamaraj
> 

Re: [dpdk-dev] DPDK Max Mbuf Allocation

[Stephen Hemminger]

On Mon, 13 Sep 2021 16:43:18 +0100
Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> On 9/13/2021 5:56 AM, Kamaraj P wrote:
> > Hello All,
> > 
> > Would like to understand or if there are any guidelines to allocate the max
> > no of mbuf per NIC ?
> > For example, if i have defined as below:
> > #define RX_RING_SIZE 1024
> > #define TX_RING_SIZE 1024
> > 
> > The Maximum RX/TX queues can be defined as 8 per NIC, What would be the max
> > no of mbuf can be allocated per NIC ?
> > Please share if there is any guildliness or any limitation to increase the
> > mbuf ?
> >   
> 
> Hi Kamaraj,
> 
> Max number of the queues and max number of the descriptors per queue depends to
> HW and changes form HW to HW.
> This information is shared by the PMDs that application needs to take into
> account. For example the descriptor limitations are provided by
> 'rx_desc_lim'/'tx_desc_lim' etc.
> 
> After descriptor number is defined, testpmd uses the mbuf count as following,
> which can be taken as sample:
> 
> nb_mbuf_per_pool = RTE_TEST_RX_DESC_MAX + RTE_TEST_TX_DESC_MAX + MAX_PKT_BURST +
>                    (nb_lcores * mb_mempool_cache);
> 

It is a a little more complicated since some devices (like bnxt) allocate
multiple mbuf's per packet. Something like

 nb_mbuf_per_pool = MAX_RX_QUEUES * (RTE_TEST_RX_DESC_MAX * MBUF_PER_RX + MBUF_PER_Q)
                + MAX_TX_QUEUE * RTE_TEST_TX_DESC_MAX * MBUF_PER_TX
                + nb_lcores * MAX_PKT_BURST
                + nb_lcores * mb_mempool_cache
                + nb_lcores * PKTMBUF_POOL_RESERVED;

Ended up with
   MBUF_PER_RX = 3
   MBUF_PER_Q  = 6
and when using jumbo
   MBUF_PER_TX = MAX_MTU / MBUF_DATA_SIZE = 2

Re: [dpdk-dev] DPDK Max Mbuf Allocation

[Lance Richardson]

[-- Attachment #1: Type: text/plain, Size: 2677 bytes --]

On Mon, Sep 13, 2021 at 11:51 AM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 13 Sep 2021 16:43:18 +0100
> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> > On 9/13/2021 5:56 AM, Kamaraj P wrote:
> > > Hello All,
> > >
> > > Would like to understand or if there are any guidelines to allocate the max
> > > no of mbuf per NIC ?
> > > For example, if i have defined as below:
> > > #define RX_RING_SIZE 1024
> > > #define TX_RING_SIZE 1024
> > >
> > > The Maximum RX/TX queues can be defined as 8 per NIC, What would be the max
> > > no of mbuf can be allocated per NIC ?
> > > Please share if there is any guildliness or any limitation to increase the
> > > mbuf ?
> > >
> >
> > Hi Kamaraj,
> >
> > Max number of the queues and max number of the descriptors per queue depends to
> > HW and changes form HW to HW.
> > This information is shared by the PMDs that application needs to take into
> > account. For example the descriptor limitations are provided by
> > 'rx_desc_lim'/'tx_desc_lim' etc.
> >
> > After descriptor number is defined, testpmd uses the mbuf count as following,
> > which can be taken as sample:
> >
> > nb_mbuf_per_pool = RTE_TEST_RX_DESC_MAX + RTE_TEST_TX_DESC_MAX + MAX_PKT_BURST +
> >                    (nb_lcores * mb_mempool_cache);
> >
>
> It is a a little more complicated since some devices (like bnxt) allocate
> multiple mbuf's per packet. Something like

+1, and it's worth noting that this makes it difficult to run many
sample applications
on the bnxt PMD.

>
>  nb_mbuf_per_pool = MAX_RX_QUEUES * (RTE_TEST_RX_DESC_MAX * MBUF_PER_RX + MBUF_PER_Q)
>                 + MAX_TX_QUEUE * RTE_TEST_TX_DESC_MAX * MBUF_PER_TX
>                 + nb_lcores * MAX_PKT_BURST
>                 + nb_lcores * mb_mempool_cache
>                 + nb_lcores * PKTMBUF_POOL_RESERVED;
>
> Ended up with
>    MBUF_PER_RX = 3

For releases up to around 20.11, 3 is the correct value (one mbuf per RX
ring entry, two mbufs in each aggregation ring per RX ring entry). Currently
the value for MBUF_PER_RX would be 5 (four mbufs in each aggregation
ring for each RX ring entry). BTW, a future version will avoid populating
aggregation rings with mbufs when LRO or scattered receive are not
enabled.

>    MBUF_PER_Q  = 6

Hmm, it's not clear where these would be allocated in the bnxt PMD. It
seems to me that MBUF_PER_Q is zero for the bnxt PMD.

> and when using jumbo
>    MBUF_PER_TX = MAX_MTU / MBUF_DATA_SIZE = 2

I don't think this is correct... the bnxt PMD allocates TX descriptor rings
with the requested number of descriptors from tx_queue_setup(), this is
the maximum number of mbufs that can be present in a TX ring.

>
>
>
>
>

Re: [dpdk-dev] DPDK Max Mbuf Allocation

[Kamaraj P]

Hello All,
Thank you for the clarification. Is there any guildliness(readme) that can
be added as part of each release notes.
This would help us to align our DPDK upgrade in our product.

Thanks again for your support


Thanks,
Kamaraj

On Tue, Sep 14, 2021 at 7:35 AM Lance Richardson <
lance.richardson@broadcom.com> wrote:

> On Mon, Sep 13, 2021 at 11:51 AM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Mon, 13 Sep 2021 16:43:18 +0100
> > Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> >
> > > On 9/13/2021 5:56 AM, Kamaraj P wrote:
> > > > Hello All,
> > > >
> > > > Would like to understand or if there are any guidelines to allocate
> the max
> > > > no of mbuf per NIC ?
> > > > For example, if i have defined as below:
> > > > #define RX_RING_SIZE 1024
> > > > #define TX_RING_SIZE 1024
> > > >
> > > > The Maximum RX/TX queues can be defined as 8 per NIC, What would be
> the max
> > > > no of mbuf can be allocated per NIC ?
> > > > Please share if there is any guildliness or any limitation to
> increase the
> > > > mbuf ?
> > > >
> > >
> > > Hi Kamaraj,
> > >
> > > Max number of the queues and max number of the descriptors per queue
> depends to
> > > HW and changes form HW to HW.
> > > This information is shared by the PMDs that application needs to take
> into
> > > account. For example the descriptor limitations are provided by
> > > 'rx_desc_lim'/'tx_desc_lim' etc.
> > >
> > > After descriptor number is defined, testpmd uses the mbuf count as
> following,
> > > which can be taken as sample:
> > >
> > > nb_mbuf_per_pool = RTE_TEST_RX_DESC_MAX + RTE_TEST_TX_DESC_MAX +
> MAX_PKT_BURST +
> > >                    (nb_lcores * mb_mempool_cache);
> > >
> >
> > It is a a little more complicated since some devices (like bnxt) allocate
> > multiple mbuf's per packet. Something like
>
> +1, and it's worth noting that this makes it difficult to run many
> sample applications
> on the bnxt PMD.
>
> >
> >  nb_mbuf_per_pool = MAX_RX_QUEUES * (RTE_TEST_RX_DESC_MAX * MBUF_PER_RX
> + MBUF_PER_Q)
> >                 + MAX_TX_QUEUE * RTE_TEST_TX_DESC_MAX * MBUF_PER_TX
> >                 + nb_lcores * MAX_PKT_BURST
> >                 + nb_lcores * mb_mempool_cache
> >                 + nb_lcores * PKTMBUF_POOL_RESERVED;
> >
> > Ended up with
> >    MBUF_PER_RX = 3
>
> For releases up to around 20.11, 3 is the correct value (one mbuf per RX
> ring entry, two mbufs in each aggregation ring per RX ring entry).
> Currently
> the value for MBUF_PER_RX would be 5 (four mbufs in each aggregation
> ring for each RX ring entry). BTW, a future version will avoid populating
> aggregation rings with mbufs when LRO or scattered receive are not
> enabled.
>
> >    MBUF_PER_Q  = 6
>
> Hmm, it's not clear where these would be allocated in the bnxt PMD. It
> seems to me that MBUF_PER_Q is zero for the bnxt PMD.
>
> > and when using jumbo
> >    MBUF_PER_TX = MAX_MTU / MBUF_DATA_SIZE = 2
>
> I don't think this is correct... the bnxt PMD allocates TX descriptor rings
> with the requested number of descriptors from tx_queue_setup(), this is
> the maximum number of mbufs that can be present in a TX ring.
>
> >
> >
> >
> >
> >
>