DPDK usage discussions
 help / color / mirror / Atom feed
From: "Soni, Shivam" <shivsoni@amazon.com>
To: Stephen Hemminger <stephen@networkplumber.org>
Cc: "dev@dpdk.org" <dev@dpdk.org>, "users@dpdk.org" <users@dpdk.org>
Subject: Re: [dpdk-users] [dpdk-dev] TX unable to enqueue packets to NIC due to no free TX descriptor
Date: Sat, 12 Jan 2019 00:26:51 +0000	[thread overview]
Message-ID: <E5C5DF95-5AF8-4F9F-8FED-6946E788F15F@amazon.com> (raw)
In-Reply-To: <20190111153709.3117c539@hermes.lan>

Hi Stephen,

Thanks for the reply.

Our mbuf pool is big enough. We have 2 RX cores, 2 TX cores and 8 worker cores.
NTxd and NRxd is 1024 each and we have 16 Rx rings (shared between Rx and workers) and 8 Tx rings (between Tx and workers)
Mempool cache size is 256 and burst size is 32.

So overall calculation comes out to be 
((NIC_RX_QUEUE_SIZE * RX_LCORES) + (NIC_TX_QUEUE_SIZE * TX_LCORES) + \
                          (WORKER_RX_RING_SIZE * RX_LCORES * NAT_WORKER_LCORES) + (WORKER_TX_RING_SIZE * NAT_WORKER_LCORES) + \
                          ((MBUF_ARRAY_SIZE + CACHE_SIZE) * (RX_LCORES + TX_LCORES + NAT_WORKER_LCORES)))

With this the  mbuf pool size should be 32128. To round off as power of 2 we have kept mbuf pool size as 32767.

Also the incoming packet rate Is pretty low.

For testing I have doubled the pool size for now. Not sure whether this will solve the issue.

Thanks.

On 1/11/19, 3:38 PM, "Stephen Hemminger" <stephen@networkplumber.org> wrote:

    On Fri, 11 Jan 2019 22:10:39 +0000
    "Soni, Shivam" <shivsoni@amazon.com> wrote:
    
    > Hi All,
    > 
    > We are trying to debug and fix an issue. After the deployment, in few of the hosts we see an issue where TX is unable to enqueue packets to NIC. On rebouncing or restarting our packet processor daemon, issue gets resolved.
    > 
    > We are using IntelDPDK version 17.11.4 and i40e drivers.
    > 
    > On looking into driver’s code, we found that whenever the issue is happening the value for nb_tx_free is ‘0’. And then it tries to free the buffer by calling function ‘i40e_tx_free_bufs’.
    > 
    > This method returns early as the buffer its trying to free says it hasn’t finished transmitting yet. The method returns at this if condition:
    > 
    > /* check DD bits on threshold descriptor */
    > if ((txq->tx_ring[txq->tx_next_dd].cmd_type_offset_bsz &
    >                 rte_cpu_to_le_64(I40E_TXD_QW1_DTYPE_MASK)) !=
    >                 rte_cpu_to_le_64(I40E_TX_DESC_DTYPE_DESC_DONE)) {
    > return 0;
    > }
    > 
    > Hence nb_tx_free remains 0.
    > 
    > Our tx descriptor count is 1024.
    > 
    > How can we fix this issue.  Can someone help us out here please
    
    Use bigger mbuf pool.  For safety the mbuf pool has to be big enough
    for Nports * (NRxd + NTxd) + NCore * (mbuf_pool_cache_size + burst_size)
    
    Each NIC might get full receive ring and full transmit ring
    and each active core might be processing a burst of packets and have
    free buffers sitting in the mbuf pool cache. This doesn't account for additional
    mbuf's created if doing things like reassembly, encryption, re-encapsulation, or compression
    
    Anything smaller and your application is relying on statistical averages
    to never see resource exhaustion; overcommitment
    


  reply	other threads:[~2019-01-12  0:26 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-11 22:10 [dpdk-users] " Soni, Shivam
2019-01-11 23:37 ` [dpdk-users] [dpdk-dev] " Stephen Hemminger
2019-01-12  0:26   ` Soni, Shivam [this message]
2019-01-14 17:54     ` Soni, Shivam
2019-01-16 21:45       ` Soni, Shivam
2019-01-17 10:56         ` Bruce Richardson
2019-04-15  5:35         ` Xiao, Xiaohong (NSB - CN/Shanghai)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=E5C5DF95-5AF8-4F9F-8FED-6946E788F15F@amazon.com \
    --to=shivsoni@amazon.com \
    --cc=dev@dpdk.org \
    --cc=stephen@networkplumber.org \
    --cc=users@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).