From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: Kamaraj P <pkamaraj@gmail.com>
Cc: Bruce Richardson <bruce.richardson@intel.com>,
dev@dpdk.org, mmahmoud@ciso.com
Subject: Re: [dpdk-dev] DPDK hugepage memory fragmentation
Date: Tue, 28 Jul 2020 11:10:11 +0100 [thread overview]
Message-ID: <ce80fe59-04d4-b848-0fd8-c40f68c784ff@intel.com> (raw)
In-Reply-To: <CAG8PAap4avO913UHOE6TVFk7U_+njtbEoFe0K7wkVjHFt2ruQg@mail.gmail.com>
On 27-Jul-20 4:30 PM, Kamaraj P wrote:
> Hi Anatoly,
> Since we do not have the driver support of SRIOv with VFIO, we are using
> IGB_UIO .
I believe it's coming :)
> Basically our application is crashing due to the buffer allocation
> failure. I believe because it didn't get a contiguous memory location
> and fails to allocate the memory.
Again, "crashing due to buffer allocation failure" is not very
descriptive. When allocation fails, EAL will produce an error log, so if
your failures are indeed due to memory allocation failures, an EAL log
will tell you if it's actually the case (and enabling debug level
logging will tell you more).
By default, all memory allocations will *not* be contiguous and
therefore will not fail if the memory is not contiguous. In order for
such an allocation to fail, you actually have to run out of memory.
If there is indeed a place where you are specifically requesting
contiguous memory, it will be signified by a call to memzone reserve API
with a RTE_MEMZONE_IOVA_CONTIG flag (or a call to
rte_eth_dma_zone_reserve(), if your driver makes use of that API). So if
you're not willing to provide any logs to help with debugging, i would
at least suggest you grep your codebase for the above two things, and
put GDB breakpoints right after the calls to either memzone reserve API
or a ethdev DMA zone reserve API.
To summarize: a regular allocation *will not fail* if memory is non
contiguous, so you can disregard those. If you find all places where
you're requesting *contiguous* memory (which should be at most one or
two), you'll be in a better position to determine whether this is what's
causing the failures.
> Is there any API, I can use to dump before our application dies ?
> Please let me know.
Not sure what you mean by that, but you could use
rte_dump_physmem_layout() function to dump your hugepage layout. That
said, i believe a debugger is, in most cases, a better way to diagnose
the issue.
>
> Thanks,
> Kamaraj
>
>
> On Mon, Jul 13, 2020 at 2:57 PM Burakov, Anatoly
> <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:
>
> On 11-Jul-20 8:51 AM, Kamaraj P wrote:
> > Hello Anatoly/Bruce,
> >
> > We are using the 18_11 version of DPDK and we are using igb_uio.
> > The way we observe an issue here is that, after we tried multiple
> > iterations of start/stop of container application(which has DPDK),
> > we were not able to allocate the memory for port during the init.
> > We thought that it could be an issue of not getting continuous
> > allocation hence it fails.
> >
> > Is there an API where I can check if the memory is fragmented
> before we
> > invoke an allocation ?
> > Or do we have any such mechanism to defragment the memory allocation
> > once we exist from the application ?
> > Please advise.
> >
>
> This is unlikely due to fragmentation, because the only way for
> 18.11 to
> be affected my memory fragmentation is 1) if you're using legacy mem
> mode, or 2) you're using IOVA as PA mode and you need huge amounts of
> contiguous memory. (you are using igb_uio, so you would be in IOVA
> as PA
> mode)
>
> NICs very rarely, if ever, allocate more than a 2M-page worth of
> contiguous memory, because their descriptor rings aren't that big, and
> they'll usually get all the IOVA-contiguous space they need even in the
> face of heavily fragmented memory. Similarly, while 18.11 mempools will
> request IOVA-contiguous memory first, they have a fallback to using
> non-contiguous memory and thus too work just fine in the face of high
> memory fragmentation.
>
> The nature of the 18.11 memory subsystem is such that IOVA layout is
> decoupled from VA layout, so fragmentation does not affect DPDK as much
> as it has for previous versions. The first thing i'd suggest is using
> VFIO as opposed to igb_uio, as it's safer to use in a container
> environment, and it's less susceptible to memory fragmentation issues
> because it can remap memory to appear IOVA-contiguous.
>
> Could you please provide detailed logs of the init process? You can add
> '--log-level=eal,8' to the EAL command-line to enable debug logging in
> the EAL.
>
> > Thanks,
> > Kamaraj
> >
> >
> >
> > On Fri, Jul 10, 2020 at 9:14 PM Burakov, Anatoly
> > <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
> <mailto:anatoly.burakov@intel.com
> <mailto:anatoly.burakov@intel.com>>> wrote:
> >
> > On 10-Jul-20 11:28 AM, Bruce Richardson wrote:
> > > On Fri, Jul 10, 2020 at 02:52:16PM +0530, Kamaraj P wrote:
> > >> Hello All,
> > >>
> > >> We are running to run DPDK based application in a
> container mode,
> > >> When we do multiple start/stop of our container
> application, the
> > DPDK
> > >> initialization seems to be failing.
> > >> This is because the hugepage memory fragementated and is not
> > able to find
> > >> the continuous allocation of the memory to initialize the
> buffer
> > in the
> > >> dpdk init.
> > >>
> > >> As part of the cleanup of the container, we do call
> > rte_eal_cleanup() to
> > >> cleanup the memory w.r.t our application. However after
> > iterations we still
> > >> see the memory allocation failure due to the
> fragmentation issue.
> > >>
> > >> We also tried to set the "--huge-unlink" as an argument
> before
> > when we
> > >> called the rte_eal_init() and it did not help.
> > >>
> > >> Could you please suggest if there is an option or any
> existing
> > patches
> > >> available to clean up the memory to avoid fragmentation
> issues
> > in the
> > >> future.
> > >>
> > >> Please advise.
> > >>
> > > What version of DPDK are you using, and what kernel driver
> for NIC
> > > interfacing are you using?
> > > DPDK versions since 18.05 should be more forgiving of
> fragmented
> > memory,
> > > especially if using the vfio-pci kernel driver.
> > >
> >
> > This sounds odd, to be honest.
> >
> > Unless you're allocating huge chunks of IOVA-contiguous memory,
> > fragmentation shouldn't be an issue. How did you determine
> that this
> > was
> > in fact due to fragmentation?
> >
> > > Regards,
> > > /Bruce
> > >
> >
> >
> > --
> > Thanks,
> > Anatoly
> >
>
>
> --
> Thanks,
> Anatoly
>
--
Thanks,
Anatoly
next prev parent reply other threads:[~2020-07-28 10:10 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-10 9:22 Kamaraj P
2020-07-10 10:28 ` Bruce Richardson
2020-07-10 15:44 ` Burakov, Anatoly
2020-07-11 7:51 ` Kamaraj P
2020-07-13 9:27 ` Burakov, Anatoly
2020-07-27 15:30 ` Kamaraj P
2020-07-28 10:10 ` Burakov, Anatoly [this message]
2020-09-16 4:32 ` Kamaraj P
2020-09-16 11:19 ` Burakov, Anatoly
2020-09-16 11:47 ` Burakov, Anatoly
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ce80fe59-04d4-b848-0fd8-c40f68c784ff@intel.com \
--to=anatoly.burakov@intel.com \
--cc=bruce.richardson@intel.com \
--cc=dev@dpdk.org \
--cc=mmahmoud@ciso.com \
--cc=pkamaraj@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).