From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2157FA053D; Mon, 27 Jul 2020 17:30:55 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E14952C6E; Mon, 27 Jul 2020 17:30:53 +0200 (CEST) Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by dpdk.org (Postfix) with ESMTP id 5879F1023 for ; Mon, 27 Jul 2020 17:30:52 +0200 (CEST) Received: by mail-wr1-f52.google.com with SMTP id a15so15316167wrh.10 for ; Mon, 27 Jul 2020 08:30:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Mhktq6+8fsqgwbelSWaCuSQZd1MYa0tf71U2le8GuKA=; b=Ww2Yjy08gbC5BKLbJax9wYPEv315qa1FszvgtFME2OdAy1JuucG3c3eZKDEnHO41jM pz5IpiYMVEGkONl0sQVnwQMhC94L3qn2h1FIfsP1XANIMd16NrjaiFiJcqiHMCdFAQxN sAXQibA3P5inMZoxu1x373pBxdHNVx9WcVndZmdwdNTUJD+mAJQiijeJa2ozC/QqDGhD YVqF4OR+9YO39qPaapeNAL3Liu/aSrtHbs+qqQ78ymO9D13Hx392RpkBn4KHpjPYTk70 ShuPJbxztY2zhr9dtJ4E59y0Rz3JIOImM3VpzBsYwNRbB0az8T8ksde/lZxfwt8Gw4uV E0ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Mhktq6+8fsqgwbelSWaCuSQZd1MYa0tf71U2le8GuKA=; b=Pn+Uk6f9Eyr7tZd1ScAxqm9LX6K9mTr7JSnv8FSjH5OGRFWuIQi0z1MHnL69Ow2Hra Myp/fPuMax1PCq/KmUVlXeEaePtcjVIY+rkdhRxh84s0QvY8cg/w9PHZGXHaOUqWcCVa u1tS6QfX9gAbxKFLRR2cMx11/JnF7lon7ds9JihrvGoECLwJBEwHsTTvg4Mjp2E+msua zg8v/JE7NVrlDdZJRGbcb0xICydQ5SV3QciHsI1RawrHy7KCEt2d2RrBiaAX+E6V9Xrx NOSkLobJ662cgfx173UgpR7P0TARKsLLPWLZ/lYuEpA/ZLyvkR6rKMvmpIQSX7c6cOUp nI6w== X-Gm-Message-State: AOAM530PKYCa6PWTgFtbckwK60v2SQ6YOkpvzmDT6NsA2d8m8IxiD0nK CsrfyoCXq1DAONWl3bTulEtHB+adGrEvb1a7Qrw= X-Google-Smtp-Source: ABdhPJyw29ObTgVZ+DCLbwbHagSOYF6DnjNZ3jEcXF7/B154VPlJ09Yt3hO1IyRzSfCayiBF5ylv42WuB/o3aX1Rc3c= X-Received: by 2002:adf:ed48:: with SMTP id u8mr21963490wro.64.1595863851881; Mon, 27 Jul 2020 08:30:51 -0700 (PDT) MIME-Version: 1.0 References: <20200710102837.GB684@bricha3-MOBL.ger.corp.intel.com> <75824075-9690-814a-1849-1107504ce344@intel.com> In-Reply-To: <75824075-9690-814a-1849-1107504ce344@intel.com> From: Kamaraj P Date: Mon, 27 Jul 2020 21:00:40 +0530 Message-ID: To: "Burakov, Anatoly" Cc: Bruce Richardson , dev@dpdk.org, mmahmoud@ciso.com Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] DPDK hugepage memory fragmentation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Anatoly, Since we do not have the driver support of SRIOv with VFIO, we are using IGB_UIO . Basically our application is crashing due to the buffer allocation failure. I believe because it didn't get a contiguous memory location and fails to allocate the memory. Is there any API, I can use to dump before our application dies ? Please let me know. Thanks, Kamaraj On Mon, Jul 13, 2020 at 2:57 PM Burakov, Anatoly wrote: > On 11-Jul-20 8:51 AM, Kamaraj P wrote: > > Hello Anatoly/Bruce, > > > > We are using the 18_11 version of DPDK and we are using igb_uio. > > The way we observe an issue here is that, after we tried multiple > > iterations of start/stop of container application(which has DPDK), > > we were not able to allocate the memory for port during the init. > > We thought that it could be an issue of not getting continuous > > allocation hence it fails. > > > > Is there an API where I can check if the memory is fragmented before we > > invoke an allocation ? > > Or do we have any such mechanism to defragment the memory allocation > > once we exist from the application ? > > Please advise. > > > > This is unlikely due to fragmentation, because the only way for 18.11 to > be affected my memory fragmentation is 1) if you're using legacy mem > mode, or 2) you're using IOVA as PA mode and you need huge amounts of > contiguous memory. (you are using igb_uio, so you would be in IOVA as PA > mode) > > NICs very rarely, if ever, allocate more than a 2M-page worth of > contiguous memory, because their descriptor rings aren't that big, and > they'll usually get all the IOVA-contiguous space they need even in the > face of heavily fragmented memory. Similarly, while 18.11 mempools will > request IOVA-contiguous memory first, they have a fallback to using > non-contiguous memory and thus too work just fine in the face of high > memory fragmentation. > > The nature of the 18.11 memory subsystem is such that IOVA layout is > decoupled from VA layout, so fragmentation does not affect DPDK as much > as it has for previous versions. The first thing i'd suggest is using > VFIO as opposed to igb_uio, as it's safer to use in a container > environment, and it's less susceptible to memory fragmentation issues > because it can remap memory to appear IOVA-contiguous. > > Could you please provide detailed logs of the init process? You can add > '--log-level=eal,8' to the EAL command-line to enable debug logging in > the EAL. > > > Thanks, > > Kamaraj > > > > > > > > On Fri, Jul 10, 2020 at 9:14 PM Burakov, Anatoly > > > wrote: > > > > On 10-Jul-20 11:28 AM, Bruce Richardson wrote: > > > On Fri, Jul 10, 2020 at 02:52:16PM +0530, Kamaraj P wrote: > > >> Hello All, > > >> > > >> We are running to run DPDK based application in a container mode, > > >> When we do multiple start/stop of our container application, the > > DPDK > > >> initialization seems to be failing. > > >> This is because the hugepage memory fragementated and is not > > able to find > > >> the continuous allocation of the memory to initialize the buffer > > in the > > >> dpdk init. > > >> > > >> As part of the cleanup of the container, we do call > > rte_eal_cleanup() to > > >> cleanup the memory w.r.t our application. However after > > iterations we still > > >> see the memory allocation failure due to the fragmentation issue. > > >> > > >> We also tried to set the "--huge-unlink" as an argument before > > when we > > >> called the rte_eal_init() and it did not help. > > >> > > >> Could you please suggest if there is an option or any existing > > patches > > >> available to clean up the memory to avoid fragmentation issues > > in the > > >> future. > > >> > > >> Please advise. > > >> > > > What version of DPDK are you using, and what kernel driver for NIC > > > interfacing are you using? > > > DPDK versions since 18.05 should be more forgiving of fragmented > > memory, > > > especially if using the vfio-pci kernel driver. > > > > > > > This sounds odd, to be honest. > > > > Unless you're allocating huge chunks of IOVA-contiguous memory, > > fragmentation shouldn't be an issue. How did you determine that this > > was > > in fact due to fragmentation? > > > > > Regards, > > > /Bruce > > > > > > > > > -- > > Thanks, > > Anatoly > > > > > -- > Thanks, > Anatoly >