From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1EAC5A04AD; Wed, 6 Nov 2019 19:17:30 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7B96C1D427; Wed, 6 Nov 2019 19:17:29 +0100 (CET) Received: from mail-il1-f196.google.com (mail-il1-f196.google.com [209.85.166.196]) by dpdk.org (Postfix) with ESMTP id 814691BE97 for ; Tue, 5 Nov 2019 15:11:28 +0100 (CET) Received: by mail-il1-f196.google.com with SMTP id o18so6143119ils.12 for ; Tue, 05 Nov 2019 06:11:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=eH6zrxaN3n9/vkUfiSTVGKXALXDdweZ672Cni89mNAs=; b=YV4Rdci6jYRObtBZxUm9zaiELq/C9jsWTH6fx/Y+BAnq5jJsTahsIo4YquHJUFVfZD FglY+nj1H0P+VGWZL0yt0etggZiTP6vnCVwWLF7f4Dj5v593ydBmMQwN2PB4eizAdqWG ZMYm7cO92lwHxkOfPsZi68HdocipeFrTQZWK0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=eH6zrxaN3n9/vkUfiSTVGKXALXDdweZ672Cni89mNAs=; b=SULousmhuzm/XXgG2XrsCoT+h5O2whxsWPpHgrxoffgRh1gQK55Uy7o8OLBMLS9Y6c jvttOniURxAqwbOEVIqfHEKWJTyK/xUo11Ww3YU1f4nUEA9Vuvxozkf4fasggSt1NmH4 OAC2aUi6L6iHRKvPR+JtFvHA/5lHDMcrWpeKFu2QznYR38IcV6B9sb03NXTaT+NJccjm JAiDQdFrTTu+UzldZZ0YBYx1coRtC27o4J4d8ztZ2BpJhgm7HCpkTMNfjB58ZU9w1/03 9IRMIhhkRAbR33M4ruyPwBGUap37Hq77JshHhzyM+HDFV/Ox7HXVGk66Ub5VeOYVusXT JgjQ== X-Gm-Message-State: APjAAAVHOinhAzNXgNaY0L1NHt2XKNad4Ou2IIuksYDkh4H/PDuMVF+P OyubFdmtq5o+QvEMewQzVNu4X0t9VEcerOTzhWNhSQ== X-Google-Smtp-Source: APXvYqyhfTZYh6d+s1dzU1sCzcP2dRuYt1CnSMFx62q19aUq88LPWqZ8GrE+Suk64WvUJzhvdi2a6lxA2Pn+rQnmwMo= X-Received: by 2002:a92:b686:: with SMTP id m6mr4173136ill.35.1572963087612; Tue, 05 Nov 2019 06:11:27 -0800 (PST) MIME-Version: 1.0 References: <20191015053047.52260-1-ajit.khaparde@broadcom.com> <83009bb3-1e0c-a22e-eff8-41a437817cb7@intel.com> <64edebee-3686-beca-2b30-c6ec1f26c162@intel.com> <31bff5b7-169c-e158-3a87-6448272c571c@intel.com> <32f3cd1a-08bb-e4bc-c22c-53453b936dd3@intel.com> <3a954a3f-1fc5-47f0-64f2-a7c8e3fbf964@intel.com> In-Reply-To: <3a954a3f-1fc5-47f0-64f2-a7c8e3fbf964@intel.com> From: Rajesh Ravi Date: Tue, 5 Nov 2019 19:40:50 +0530 Message-ID: To: "Burakov, Anatoly" Cc: Ajit Khaparde , dev@dpdk.org, Jonathan Richardson , Scott Branden , Vikram Mysore Prakash , Srinath Mannam X-Mailman-Approved-At: Wed, 06 Nov 2019 19:17:27 +0100 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH] eal: add option --iso-cmem for external custom memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Thanks a lot Anatoly. Will the same solution work with DPDK 19.02 as well? We 're actually using DPDK 19.02 for memory allocations for SPDK 19.07. DPDK 19.11 may not be supported by SPDK 19.07 we 're currently using. I 'll definitely test if the patch works with DPDK 19.02 Thanks again for your time and help. Regards, Rajesh On Tue, Nov 5, 2019 at 5:11 PM Burakov, Anatoly wrote: > On 04-Nov-19 10:25 AM, Burakov, Anatoly wrote: > > On 30-Oct-19 7:50 PM, Rajesh Ravi wrote: > >> Thanks Anatoly. > >> Please find inline below: > >> > >> [Anatoly] vfio_mem_event_callback() is called every time memory is > >> added to a > >> heap. That includes internal and external memory > >> > >> [Rajesh] malloc_heap_add_external_memory() does call > >> eal_memalloc_mem_event_notify [ instead of vfio_mem_event_callback() ] > >> But, no callback function is getting called from inside > >> eal_memalloc_mem_event_notify() > >> execution flow is not entering inside following loop: > >> > >> /TAILQ_FOREACH(entry, &mem_event_callback_list, next) {/ > >> / RTE_LOG(DEBUG, EAL, "Calling mem event callback > >> '%s:%p'\n", > >> entry->name, entry->arg); > >> entry->clb(event, start, len, entry->arg); > >> }/ > >> > >> Do you mean to say, we are supposed to explicitly register a callback > >> which separately builds iommu tables in addition to calling > >> rte_malloc_heap_memory_add() API? > > > > Hi, > > > > No, the callback in VFIO should be registered automatically [1] at EAL > > initialization (or, more precisely, when default container is > > initialized). Does that not happen in your case? > > > > [1] > http://git.dpdk.org/dpdk/tree/lib/librte_eal/linux/eal/eal_vfio.c#n791 > > > > Hi Rajesh, > > I think i figured it out. It is a defect in design of how external > memory heaps are handled. > > When VFIO initializes, it will find first VFIO-bound device, initialize > the container, and set up DMA mappings. Then, you can add more memory > through creating custom memory regions without adding them to heap > (mmap() + rte_extmem_register() + rte_dev_dma_map()), or with adding > them to heap (mmap() + rte_malloc_heap_add_memory()). > > The problem is, memory registered through rte_dev_dma_map() will get > added into a list of user maps, while heap memory will not - the > assumption is that the DMA mapping will happen through the callback, but > there is no record left anywhere that this memory is supposed to be mapped. > > This makes it so that, if there are no VFIO-bound devices at startup, > then you create a heap, and *then* you hotplug a device, the heap will > not be mapped because (as you have correctly pointed out) type1_map() > skips it, and it's not present in a list of user mem maps either, > because it is heap memory, so EAL is supposed to handle it by itself and > not through user map list. > > There could be two fixes here. The easiest one is to just add another > flag to the memseglist - that will work for 19.11, and that's what i > intend on doing since we're breaking ABI anyway. > > For older releases, a different approach would be required (i think > scanning heaps is best we can do here) in order to keep the ABI and not > introduce new stuff into rte_memseg_list. > > I'll submit a patch shortly, it would be great if you could test it. > > -- > Thanks, > Anatoly > -- Regards, Rajesh