From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1BADCA31F3 for ; Fri, 18 Oct 2019 18:53:11 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 74E441C0B2; Fri, 18 Oct 2019 18:53:09 +0200 (CEST) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 878191C08C for ; Fri, 18 Oct 2019 18:53:07 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 18 Oct 2019 09:53:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.67,312,1566889200"; d="scan'208";a="186882084" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.104]) ([10.237.220.104]) by orsmga007.jf.intel.com with ESMTP; 18 Oct 2019 09:53:04 -0700 To: Rajesh Ravi Cc: Ajit Khaparde , dev@dpdk.org, Jonathan Richardson , Scott Branden , Vikram Mysore Prakash , Srinath Mannam References: <20191015053047.52260-1-ajit.khaparde@broadcom.com> From: "Burakov, Anatoly" Message-ID: <83009bb3-1e0c-a22e-eff8-41a437817cb7@intel.com> Date: Fri, 18 Oct 2019 17:53:03 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH] eal: add option --iso-cmem for external custom memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 18-Oct-19 11:54 AM, Rajesh Ravi wrote: > + Srinath > > Thanks Anatoly for reviewing this. Please find my reply inline  below: > > [Anatoly] First of all, what is "iso-cmem"? It doesn't seem to have any > defined > meaning nor any relation to any existing functionality, and it's not > explained anywhere what is "isolated cmem". > [Rajesh] I 'll correct commit message to include clearer explanation. > _ > _ > _Context & purpose_ > We 're using this to improve SPDK performance. When NVMe completion > queues are allocated from a certain memory range, > out of order competions completions  are enabled with our PCIe and it > improves performance. > > _Usage_ > spdk_env_init()-->(calls rte_eal_init(), and then calls > )spdk_env_dpdk_post_init() [file: spdk/lib/env_dpdk/init.c] > --> spdk_mem_map_init() > [lib/env_dpdk/memory.c]-->map_cmem_virtual_area() [this is our custom > function] > In map_cmem_virtual_area(): we 're mmaping anonymous &  private region > and then creating iova table which makes iova addresses which fall in > custom memory region. > Then we call rte_malloc_heap_memory_add() and then allocate NVMe > completion queues from this heap. > > [Anatoly] More importantly, why is this necessary? Type1 map only bypasses > external segments when adding memory at startup - it doesn't stop you > from calling rte_vfio_dma_map() to map the memory with VFIO when you > create the segment. > [Rajesh] Please correct me if I'm wrong or  missing some thing here. >   A) We wanted to create a heap from which we can allocate dynamically > >   B) I see that type1_map() > [file:dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c] getting called once > rte_malloc_heap_memory_add()() was invoked. >        SPDK uses type1 dma map [spdk/lib/env_dpdk/memory.c] >         vfio_type1_dma_map() is registered for .dma_map_func member > [dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c] >         So type1_map is called. Not sure whether we can change this. > I'm not sure i'm following. You mean you *don't* want this mapping to be performed? -- Thanks, Anatoly