From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id 5A48A559A for ; Tue, 18 Sep 2018 17:15:31 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 18 Sep 2018 08:15:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,390,1531810800"; d="scan'208";a="263654987" Received: from aburakov-mobl1.ger.corp.intel.com (HELO [10.237.220.55]) ([10.237.220.55]) by fmsmga005.fm.intel.com with ESMTP; 18 Sep 2018 08:15:19 -0700 To: Shreyansh Jain , Shahaf Shuler Cc: "dev@dpdk.org" , "laszlo.madarassy@ericsson.com" , "laszlo.vadkerti@ericsson.com" , "andras.kovacs@ericsson.com" , "winnie.tian@ericsson.com" , "daniel.andrasi@ericsson.com" , "janos.kobor@ericsson.com" , "srinath.mannam@broadcom.com" , "scott.branden@broadcom.com" , "ajit.khaparde@broadcom.com" , "keith.wiles@intel.com" , "bruce.richardson@intel.com" , Thomas Monjalon References: <9c763791-2c10-fd36-7f50-3df5efbc20ab@intel.com> From: "Burakov, Anatoly" Message-ID: Date: Tue, 18 Sep 2018 16:15:17 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH 00/16] Support externally allocated memory in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Sep 2018 15:15:32 -0000 On 18-Sep-18 1:29 PM, Shreyansh Jain wrote: > On Monday 17 September 2018 06:30 PM, Burakov, Anatoly wrote: >> On 17-Sep-18 1:16 PM, Shahaf Shuler wrote: >>> Monday, September 17, 2018 1:07 PM, Burakov, Anatoly: >>>> Subject: Re: [dpdk-dev] [PATCH 00/16] Support externally allocated >>>> memory >>>> in DPDK >>>> >>>> On 13-Sep-18 8:44 AM, Shahaf Shuler wrote: >>> >>> [...] >>> >>>>>> The responsibility to ensure memory is accessible before using it is >>>>>> on the shoulders of the user - there is no checking done with regards >>>>>> to validity of the memory (nor could there be...). >>>>> >>>>> That makes sense. However who should be in-charge of mapping this >>>> memory for dma access? >>>>> The user or internally be the PMD when encounter the first packet >>>>> or while >>>> traversing the existing mempools? >>>>> >>>> Hi Shahaf, >>>> >>>> There are two ways this can be solved. The first way is to perform VFIO >>>> mapping automatically on adding/attaching memory. The second is to >>>> force >>>> user to do it manually. For now, the latter is chosen because user >>>> knows best >>>> if they intend to do DMA on that memory, but i'm open to suggestions. >>> >>> I agree with that approach, and will add not only if the mempool is >>> for dma or not but also which ports will use this mempool (this can >>> effect on the mapping). >> >> That is perhaps too hardware-specific - this should probably be >> handled inside the driver callbacks. >> >>> However I don't think this is generic enough to use only VFIO. As you >>> said, there are some devices not using VFIO for mapping rather some >>> proprietary driver utility. >>> IMO DPDK should introduce generic and device agnostic APIs to the user. >>> >>> My suggestion is instead of doing vfio_dma_map that or vfio_dma_unmap >>> that have a generic dma_map(uint8_t port, address, len). Each driver >>> will register with its own mapping callback (can be vfio_dma_map). >>> It can be outside of this series, just wondering the people opinion >>> on such approach. >> >> I don't disagree. I don't like bus/net/etc drivers doing their own >> thing with regards to mapping, and i would by far prefer generic way >> to set up DMA maps, to which VFIO will be a subscriber. >> >>> >>>> >>>> There is an issue with some devices and buses (i.e. bus/fslmc) >>>> bypassing EAL >>>> VFIO infrastructure and performing their own VFIO/DMA mapping magic, >>>> but >>>> solving that problem is outside the scope of this patchset. Those >>>> devices/buses should fix themselves :) > > DMA mapping is a very common principle and can be easily be a candidate > for lets-make-generic-movement, but, being close to hardware (or > hardware specific), it does require the driver to have some flexibility > in terms of its eventual implementation. Perhaps i didn't word my response clearly enough. I didn't mean to say (or imply) that EAL must handle all DMA mappings itself. Rather, EAL should provide a generic infrastructure of maintaining current mappings etc., and provide a subscription mechanism for other users (e.g. drivers) so that the details of implementation of exactly how to map things for DMA is up to the drivers. In other words, we agree :) > > I maintain one of those drivers (bus/fslmc) in DPDK which needs to have > special VFIO layer - and from that experience, I can say that VFIO > mapping does require some flexibility. SoC semantics are sometimes too > complex to pin to general-universally-agreed-standard concept. (or, one > can easily call it a 'bug', while it is a 'feature' for others :D) > > In fact, NXP has another driver (bus/dpaa) which doesn't even work with > VFIO - loves to work directly with Phys_addr. And, it is not at a lower > priority than one with VFIO. > > Thus, I really don't think a strongly controlled VFIO mapping should be > EAL's responsibility. Failure because of lack of mapping is a driver's > problem. > While EAL doesn't necessarily need to be involved with mapping things for VFIO, i believe it does need to be the authority on what gets mapped. The user needs a way to make arbitrary memory available for DMA - this is where EAL comes in. VFIO itself can be factored out into a separate subsystem (DMA drivers, anyone? :D ), but given that memory cometh and goeth (external memory included), and given that some things tend to be a bit complicated [*], EAL needs to know when something is supposed to be mapped or unmapped, and when to notify subscribers that they may have to refresh their DMA maps. [*] for example, VFIO can only do mappings whenever there are devices actually attached to a VFIO container, so we have to maintain all maps between hotplug events to ensure that memory set up for DMA doesn't silently get unmapped on device detach and subsequent attach. -- Thanks, Anatoly