From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: Rajesh Ravi <rajesh.ravi@broadcom.com>
Cc: Ajit Khaparde <ajit.khaparde@broadcom.com>,
dev@dpdk.org,
Jonathan Richardson <jonathan.richardson@broadcom.com>,
Scott Branden <scott.branden@broadcom.com>,
Vikram Mysore Prakash <vikram.prakash@broadcom.com>,
Srinath Mannam <srinath.mannam@broadcom.com>
Subject: Re: [dpdk-dev] [PATCH] eal: add option --iso-cmem for external custom memory
Date: Thu, 24 Oct 2019 12:43:50 +0100 [thread overview]
Message-ID: <64edebee-3686-beca-2b30-c6ec1f26c162@intel.com> (raw)
In-Reply-To: <CAAr5zNdS37jULy6vmWsnr-uL3hP0nV8OOQSYrFOi==wM-xpHSg@mail.gmail.com>
On 22-Oct-19 8:56 AM, Rajesh Ravi wrote:
> Hi Anatoly,
> I tried calling rte_vfio_dma_map() but, it failed for me because
>
> vfio_dma_mem_map() failed with error:VFIO support not initialized
> because: default_vfio_cfg->vfio_iommu_type=NULL ( container_dma_map
> calls vfio_dma_mem_map with vfio_cfg = default_vfio_cfg )
>
> call sequence: rte_vfio_dma_map() -->container_dma_map()
> --> vfio_dma_mem_map
>
> Regards,
> Rajesh
>
> On Mon, Oct 21, 2019 at 9:16 PM Rajesh Ravi <rajesh.ravi@broadcom.com
> <mailto:rajesh.ravi@broadcom.com>> wrote:
>
> Thanks Anatoly for prompt response. Sorry for the delayed response,
> I took some time to reverify with SPDK.
>
> Infact, I do want the iommu mapping to be performed. I don't want it
> to be bypassed by type1_map()
> [lib/librte_eal/linuxapp/eal/eal_vfio.c] for external memory.
>
> Then, if I understood correctly, you suggested to call
> rte_vfio_dma_map() as an alternative.
>
> _Context & clarification_
>
> 1) We 're using DPDK to manage/allocate memory for SPDK through heap
> API.
>
> The only DPDK APIs we 're calling are:
> A)rte_malloc_heap_memory_add() to add external memory to heap.
> B)rte_malloc_heap_get_socket() & rte_malloc_socket() to allocate
> memory
>
> /Are you suggesting to make a call to rte_vfio_dma_map() from spdk,
> in addition to the APIs listed under 1)A & 1)B instead of modifying
> DPDK vfio functions?/
> Please confirm, Probably I missed the context and might not have
> understood fully.
>
>
> 2) .dma_user_map_func=vfio_type1_dma_mem_map() is called from 2
> paths in dpdk. In either case call to dma_user_map_func() is skipped.
> A) _During the startup, as you said before:_
> rte_vfio_setup_device() --> type1_map()
>
> B)During allocation event:
> vfio_mem_event_callback() (type=RTE_MEM_EVENT_ALLOC,..)
> -->vfio_dma_mem_map() -->dma_user_map_func()
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
> _Conclusion_
>
> So we have 2 alternatives:
>
> A) make additional call to rte_vfio_dma_map() API after adding
> external memory using rte_malloc_heap_memory_add() API.
>
> B) remove msl->external check which bypasses call to
> dma_user_map_func() in DPDK.
>
> I modified DPDK functions [Option B) ]. I guess you 're
> suggesting option A)
>
> Please confirm.
>
> Regards,
> Rajesh
Hi Rajesh,
Apologies for delayed response.
I'm still not sure i understand your problem.
When adding memory with rte_malloc_heap_memory_add(), this memory is, as
far as i can tell, automatically mapped for DMA mapping with VFIO. This
happens because:
rte_malloc_heap_memory_add() -> malloc_heap_add_external_memory() ->
eal_memalloc_mem_event_notify() -> vfio_mem_event_callback() ->
vfio_dma_mem_map()
VFIO registers itself for mem event callbacks in
eal_vfio.c:rte_vfio_setup_device():791.
So, there is no need to map anything for DMA when you're creating
external heaps - it is being mapped automatically by EAL.
For an example implementation have a look at testpmd, specifically
testpmd.c:mbuf_pool_create() for MP_ALLOC_XMEM type. This calls
testpmd.c:setup_extmem(), which creates an external heap. Notice how it
doesn't do anything to map the memory for DMA, because that is already
done by EAL. You can verify this working by adding --mp-alloc=xmem
argument to testpmd to force it to use external memory for its mempools.
So, what is the actual issue here, given that the memory is mapped for
DMA by EAL automatically when it is added?
--
Thanks,
Anatoly
next prev parent reply other threads:[~2019-10-24 11:43 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-15 5:30 Ajit Khaparde
2019-10-17 15:45 ` Burakov, Anatoly
2019-10-18 10:54 ` Rajesh Ravi
2019-10-18 16:53 ` Burakov, Anatoly
2019-10-21 15:46 ` Rajesh Ravi
2019-10-22 7:56 ` Rajesh Ravi
2019-10-24 11:43 ` Burakov, Anatoly [this message]
2019-10-25 12:53 ` Rajesh Ravi
2019-10-25 15:02 ` Burakov, Anatoly
2019-10-30 19:50 ` Rajesh Ravi
2019-11-04 10:25 ` Burakov, Anatoly
2019-11-05 11:41 ` Burakov, Anatoly
2019-11-05 14:10 ` Rajesh Ravi
2019-11-05 15:18 ` Burakov, Anatoly
2019-11-05 17:13 ` Burakov, Anatoly
2019-11-06 13:55 ` David Marchand
2019-11-07 15:51 ` David Marchand
2019-11-07 16:16 ` Rajesh Ravi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=64edebee-3686-beca-2b30-c6ec1f26c162@intel.com \
--to=anatoly.burakov@intel.com \
--cc=ajit.khaparde@broadcom.com \
--cc=dev@dpdk.org \
--cc=jonathan.richardson@broadcom.com \
--cc=rajesh.ravi@broadcom.com \
--cc=scott.branden@broadcom.com \
--cc=srinath.mannam@broadcom.com \
--cc=vikram.prakash@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).