From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D3A11A3201 for ; Mon, 21 Oct 2019 09:52:37 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 453233256; Mon, 21 Oct 2019 09:52:37 +0200 (CEST) Received: from mail-il1-f196.google.com (mail-il1-f196.google.com [209.85.166.196]) by dpdk.org (Postfix) with ESMTP id B86F11D14B for ; Fri, 18 Oct 2019 12:54:48 +0200 (CEST) Received: by mail-il1-f196.google.com with SMTP id j9so5136192ilr.2 for ; Fri, 18 Oct 2019 03:54:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=UBr5Eg2Tz09vc7XOO0pWLbMbHGObMfVdO0plEDcDPAU=; b=Ka12T1HtgUmCwL2et0fYyGbnEpvqf7JFEfwCz4VMNDYDhXdVniddaZ3rqLDPde+WR6 s4JYSvA0a/tlejw03LlfT2ssPpSEPe2oWPFmzZ8aHXhFodIq545BaOgzsguM+mO9KXcd hcaGMuwwX/RSEia53MZDH9pwZan9JjhR+WsfA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=UBr5Eg2Tz09vc7XOO0pWLbMbHGObMfVdO0plEDcDPAU=; b=rbClXQOnjNrGEZKylmxWMQ07455Tzga2W3sOWjQYp6HsufkFAEMeP92zAyJ4AwC/fs +j9wG/iwcB1iH95M9jHKSdOS7VqR5KnT0AU5T7nu5p7wbndsyfWrUEbHzLfuZVS8VBVb YVtV5lWXkv7DlDnyp94DKeIeQvQg1XhWw4nlQU99c08VKgBUes/5H0sQZj833BAHMkxv Nizn2pBLNwK+qrbzkg0Q2eOWiderEvYreOnxRp6I1FGgkhdGUDcueZWc+kL1gMYUgo68 T3v6snKXM7ilr/p/8snYED9+mxt6sD+bVyxmdl4QwVUBazrUl5leXiW39f7jYFBdrjxv 0FNg== X-Gm-Message-State: APjAAAV+nyntMn5IgrrE50InIdERnT2Ycg/F+ToIBo0w91OFRs/cbSdd +wVSsqyItIAeJGEGBKK3tvLWMa7mLUZuqDbcqZL0gg== X-Google-Smtp-Source: APXvYqyYhOCsIF7SM6KJn8+Ukt6c13hsRFMShcqvl66PIJu0KrZUvobw3K5w9rT6NyNHV7uviaKSK/p+jnwca7EXVcM= X-Received: by 2002:a92:a1dd:: with SMTP id b90mr9602019ill.207.1571396087921; Fri, 18 Oct 2019 03:54:47 -0700 (PDT) MIME-Version: 1.0 References: <20191015053047.52260-1-ajit.khaparde@broadcom.com> In-Reply-To: From: Rajesh Ravi Date: Fri, 18 Oct 2019 16:24:11 +0530 Message-ID: To: "Burakov, Anatoly" Cc: Ajit Khaparde , dev@dpdk.org, Jonathan Richardson , Scott Branden , Vikram Mysore Prakash , Srinath Mannam X-Mailman-Approved-At: Mon, 21 Oct 2019 09:52:36 +0200 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] eal: add option --iso-cmem for external custom memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" + Srinath Thanks Anatoly for reviewing this. Please find my reply inline below: [Anatoly] First of all, what is "iso-cmem"? It doesn't seem to have any defined meaning nor any relation to any existing functionality, and it's not explained anywhere what is "isolated cmem". [Rajesh] I 'll correct commit message to include clearer explanation. *Context & purpose* We 're using this to improve SPDK performance. When NVMe completion queues are allocated from a certain memory range, out of order competions completions are enabled with our PCIe and it improves performance. *Usage* spdk_env_init()-->(calls rte_eal_init(), and then calls )spdk_env_dpdk_post_init() [file: spdk/lib/env_dpdk/init.c] --> spdk_mem_map_init() [lib/env_dpdk/memory.c]-->map_cmem_virtual_area() [this is our custom function] In map_cmem_virtual_area(): we 're mmaping anonymous & private region and then creating iova table which makes iova addresses which fall in custom memory region. Then we call rte_malloc_heap_memory_add() and then allocate NVMe completion queues from this heap. [Anatoly] More importantly, why is this necessary? Type1 map only bypasses external segments when adding memory at startup - it doesn't stop you from calling rte_vfio_dma_map() to map the memory with VFIO when you create the segment. [Rajesh] Please correct me if I'm wrong or missing some thing here. A) We wanted to create a heap from which we can allocate dynamically B) I see that type1_map() [file:dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c] getting called once rte_malloc_heap_memory_add()() was invoked. SPDK uses type1 dma map [spdk/lib/env_dpdk/memory.c] vfio_type1_dma_map() is registered for .dma_map_func member [dpdk/lib/librte_eal/linuxapp/eal/eal_vfio.c] So type1_map is called. Not sure whether we can change this. Regards, Rajesh On Thu, Oct 17, 2019 at 9:15 PM Burakov, Anatoly wrote: > On 15-Oct-19 6:30 AM, Ajit Khaparde wrote: > > From: Rajesh Ravi > > > > Support external custom memory added to heap to be used with vfio with > > --iso-cmem option.Type1 memory mapping was by passed for external memory. > > But an exception is added to allow external custom memory to be used with > > vfio. > > > > Signed-off-by: Rajesh Ravi > > Reviewed-by: Jonathan Richardson > > Reviewed-by: Scott Branden > > Reviewed-by: Vikram Mysore Prakash > > Reviewed-by: Ajit Khaparde > > --- > > Hi, > > First of all, what is "iso-cmem"? It doesn't seem to have any defined > meaning nor any relation to any existing functionality, and it's not > explained anywhere what is "isolated cmem". > > More importantly, why is this necessary? Type1 map only bypasses > external segments when adding memory at startup - it doesn't stop you > from calling rte_vfio_dma_map() to map the memory with VFIO when you > create the segment. > > -- > Thanks, > Anatoly > -- Regards, Rajesh