From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 89DA9A04C7; Wed, 16 Sep 2020 13:19:32 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 98F7E1C25A; Wed, 16 Sep 2020 13:19:25 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id D412A1C1C3 for ; Wed, 16 Sep 2020 13:19:23 +0200 (CEST) IronPort-SDR: C7BMli3QXzEuuafYLRZPot+LHjppAv+7bl3lgYAj87hpAwsB/Mv6smc2vdWSXUho7o+tTD8qPz JCI+xRQi5mBQ== X-IronPort-AV: E=McAfee;i="6000,8403,9745"; a="160372566" X-IronPort-AV: E=Sophos;i="5.76,432,1592895600"; d="scan'208";a="160372566" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2020 04:19:22 -0700 IronPort-SDR: YwmIaYcb21Jw5a2f1qolAriP49yBTK8HASRRP+tzkKWitwzx6olEwEdLpAj2Pt92X4fF/lAmSI ZeHEtoMpM69w== X-IronPort-AV: E=Sophos;i="5.76,432,1592895600"; d="scan'208";a="451815893" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.213.216.44]) ([10.213.216.44]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Sep 2020 04:19:21 -0700 To: Kamaraj P Cc: Bruce Richardson , dev@dpdk.org References: <20200710102837.GB684@bricha3-MOBL.ger.corp.intel.com> <75824075-9690-814a-1849-1107504ce344@intel.com> From: "Burakov, Anatoly" Message-ID: <0b3a544d-4765-b284-1755-8b95eb86725d@intel.com> Date: Wed, 16 Sep 2020 12:19:19 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] DPDK hugepage memory fragmentation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 16-Sep-20 5:32 AM, Kamaraj P wrote: > Hi Anatoly, > > We just dump the memory contents when it fails to allocate the memory. > Hi Kamaraj, Yes, i can see that the memory is fragmented. That's not what i was asking though, because memory fragmentation is *expected* if you're using igb_uio. You're not using VFIO and IOVA as VA addressing, so you're at the mercy of your kernel when it comes to getting IOVA-contiguous address. We in DPDK cannot do anything about it, because we don't control which pages we get and what addresses they get assigned. There's nothing to fix on our side in this situation. Here are the things you can do to avoid allocation failure in your case: 1) Drop the IOVA-contiguous allocation flag [1] 2) Use legacy mode [2] [3] 3) Switch to using VFIO [4] 4) Use bigger page size The first point is crucial! Your allocation *wouldn't have failed* if your code didn't specify for the allocation to require IOVA-contiguousness. The fact that it has failed means that your allocation has requested such memory, so it's on you to ensure that whatever you're allocating, IOVA-contiguousness is *required*. I cannot decide that for you as it is your code, so it is up to you to figure out if whatever you're allocating actually requires such memory, or if you can safely remove this allocation flag from your code. [1] http://doc.dpdk.org/api/rte__memzone_8h.html#a3ccbea77ccab608c6e683817a3eb170f [2] http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#memory-mapping-discovery-and-memory-reservation [3] http://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html#id3 [4] I understand the requirement for PF driver, but i think support for PF in VFIO is coming in 20.11 release -- Thanks, Anatoly