From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it1-f193.google.com (mail-it1-f193.google.com [209.85.166.193]) by dpdk.org (Postfix) with ESMTP id 9EB5C5F29 for ; Wed, 13 Feb 2019 12:43:22 +0100 (CET) Received: by mail-it1-f193.google.com with SMTP id p4so3219437itc.4 for ; Wed, 13 Feb 2019 03:43:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=emk+MrW5lOMLtd7wpUGlnakHuBvL2yw7ODmfmLumQcI=; b=TcPmcBLrNGCJQzBfkgfVE7FduaREno6fLCXXaSLeZF211dxxmEVRjFdcN6bK7MzUlm DY6k8+4zIjn4UHWKigzA+N/85wrZNV5Z2xr5Koo5UsjU9W8gT50O64ZkHC64lxZBF/xz QXEvSus12B1M3CJJ73kylSNjX80c/yT/KpR0tbSJAmdCFhGqCc+2SYpSm/v8KTw+xCWQ mnAdKFMXItaiBspBVxus+Nr6VCf3Hd+RaacOz4C441oO2jChP73DQ9zTXaZ99s0+E3cv UC/B4iQXNXJq8g0c+q9MPY0mWcrp1dOUk/IeeVzgtcFSu+EHWuc1ookY+QKe3NL4NRNC K7xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=emk+MrW5lOMLtd7wpUGlnakHuBvL2yw7ODmfmLumQcI=; b=S+t6GSzVIx5Kw/iulyCSM+DWB3zsCdZPyk1C0XZab/EQ5coyIRSCjy+Ia5qVCFty9O uXGUams2S+IJFmtiQZvzkUBfLcfWn3bg5+dWWbFi2jiIpW1z3gMCAnhs008xpnE1VJih pk24t1I9mFfIRSxL2VkNeWfyXL9T9hXm8azP/YM8vAp+akCJfMT9UThwWIyyMQ0fszTC dnZPZUmnwGJ2qJFKcDNcXDWRPl1Yn1aPxSu4NwY4w2yLZrI3b8WGyPCeKIy2mGJRe6BW g3ej9QnXwI+DoJld+yXmCA7cvZdIMHmNqhdpBb3U1hFAz3wsDYW9Y7O0JbE0jDcuLmBv MYaA== X-Gm-Message-State: AHQUAubYdm7uaH9YJjvyeWEK1mz5SFiDikzFPahgMDHcJRRUpedHsgvI DTRnOyHm9KLul0Tp+JblP4WgsChLYCOGRuFUMO6cLg== X-Google-Smtp-Source: AHgI3Ibi0RNi3urEhcRMFRL9wyNOxakO5kog8zuDLzimXObCn8ELn2dn2oGk/9Drg/K0+huALl5DqLsCGey2pQ2lu/I= X-Received: by 2002:a6b:c9d4:: with SMTP id z203mr33452iof.218.1550058201994; Wed, 13 Feb 2019 03:43:21 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Alejandro Lucero Date: Wed, 13 Feb 2019 11:43:11 +0000 Message-ID: To: Shahaf Shuler Cc: "Burakov, Anatoly" , Yongseok Koh , Thomas Monjalon , Ferruh Yigit , nhorman@tuxdriver.com, Gaetan Rivet , dev Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH 0/6] introduce DMA memory mapping for external memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Feb 2019 11:43:22 -0000 On Wed, Feb 13, 2019 at 9:11 AM Shahaf Shuler wrote: > This series is in continue to RFC[1]. > > The DPDK APIs expose 3 different modes to work with memory used for DMA: > > 1. Use the DPDK owned memory (backed by the DPDK provided hugepages). > This memory is allocated by the DPDK libraries, included in the DPDK > memory system (memseg lists) and automatically DMA mapped by the DPDK > layers. > > 2. Use memory allocated by the user and register to the DPDK memory > systems. This is also referred as external memory. Upon registration of > the external memory, the DPDK layers will DMA map it to all needed > devices. > > 3. Use memory allocated by the user and not registered to the DPDK memory > system. This is for users who wants to have tight control on this > memory. The user will need to explicitly call DMA map function in order > to register such memory to the different devices. > > The scope of the patch focus on #3 above. > > Why can not we have case 2 covering case 3? > Currently the only way to map external memory is through VFIO > (rte_vfio_dma_map). While VFIO is common, there are other vendors > which use different ways to map memory (e.g. Mellanox and NXP). > > As you say, VFIO is common, and when allowing DMAs programmed in user space, the right thing to do. I'm assuming there is an IOMMU hardware and this is what Mellanox and NXP rely on in some way or another. Having each driver doing things in their own way will end up in a harder to validate system. If there is an IOMMU hardware, same mechanism should be used always, leaving to the IOMMU hw specific implementation to deal with the details. If a NIC is IOMMU-able, that should not be supported by specific vendor drivers but through a generic solution like VFIO which will validate a device with such capability and to perform the required actions for that case. VFIO and IOMMU should be modified as needed for supporting this requirement instead of leaving vendor drivers to implement their own solution. In any case, I think this support should be in a different patchset than the private user space mappings. > The work in this patch moves the DMA mapping to vendor agnostic APIs. > A new map and unmap ops were added to rte_bus structure. Implementation > of those was done currently only on the PCI bus. The implementation takes > the driver map and umap implementation as bypass to the VFIO mapping. > That is, in case of no specific map/unmap from the PCI driver, > VFIO mapping, if possible, will be used. > > Application use with those APIs is quite simple: > * allocate memory > * take a device, and query its rte_device. > * call the bus map function for this device. > > Future work will deprecate the rte_vfio_dma_map and rte_vfio_dma_unmap > APIs, leaving the PCI device APIs as the preferred option for the user. > > [1] https://patches.dpdk.org/patch/47796/ > > Shahaf Shuler (6): > vfio: allow DMA map of memory for the default vfio fd > vfio: don't fail to DMA map if memory is already mapped > bus: introduce DMA memory mapping for external memory > net/mlx5: refactor external memory registration > net/mlx5: support PCI device DMA map and unmap > doc: deprecate VFIO DMA map APIs > > doc/guides/prog_guide/env_abstraction_layer.rst | 2 +- > doc/guides/rel_notes/deprecation.rst | 4 + > drivers/bus/pci/pci_common.c | 78 +++++++ > drivers/bus/pci/rte_bus_pci.h | 14 ++ > drivers/net/mlx5/mlx5.c | 2 + > drivers/net/mlx5/mlx5_mr.c | 232 ++++++++++++++++--- > drivers/net/mlx5/mlx5_rxtx.h | 5 + > lib/librte_eal/common/eal_common_bus.c | 22 ++ > lib/librte_eal/common/include/rte_bus.h | 57 +++++ > lib/librte_eal/common/include/rte_vfio.h | 12 +- > lib/librte_eal/linuxapp/eal/eal_vfio.c | 26 ++- > lib/librte_eal/rte_eal_version.map | 2 + > 12 files changed, 418 insertions(+), 38 deletions(-) > > -- > 2.12.0 > >