From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by dpdk.org (Postfix) with ESMTP id C34EE5F29 for ; Wed, 13 Feb 2019 12:44:47 +0100 (CET) Received: by mail-wr1-f67.google.com with SMTP id w17so2084435wrn.12 for ; Wed, 13 Feb 2019 03:44:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=tE6k4mjN7CKDoDomeXIqXaM4SyTt+/xk2amXz2Bers0=; b=YaA3hizzicOqQ+Z0dmQQYnrVUkp6kCK94I+zqlXo6aThSRG5i4xsvtE+gVhu2hLxeV x/F5SOQS/rfi7Fdqnp7Upqh/QtnD9SyGSraDKPOBowGPZe5z8VfM4PE3A61Lu3GtXRDU 2IST1VANfZ5nml61PGeNDIznY4H/m38NAVRetLWaCiSsdgFrIa168RaVoamvU6EOd5g/ 1vYgzOksFcR1jPjwsAjG62VYvl+BPoADxSH4pMnAo45JBnhczyglBYpF57LmAp+GOpnu 3ALXwbU/v0NzklLiQG9+beYg/DLD7TaijJUa6po+HAcM5U/2ixqVLe/nmh7xTVpPFq2b mswg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=tE6k4mjN7CKDoDomeXIqXaM4SyTt+/xk2amXz2Bers0=; b=UpO1gH1BMDURcID5HHaFxIXB3ApJYzv9m+JbjI+e1IE7vkIYsheiu1AIWZMPKUjNWb 6zlv9v7drar9eREirGU8UmBfYziZNQldC+ucgD/GpeYOCOh7pliPAV/j53Z9qlDLjwl5 qDrDtVoRA107cYZYDrV7kXiVg2mKjXPVQhaAQNR8gpnOlKITPCllbmw1gMTc5ZiqXHj1 LHSoR4VA8zntokcXm42/Rj2yFOhb1X5gCWkINDfrw5re/SQ83+giNzQfTuvK6+3ytp3i oN66j1hfizg6nZikwjuEAnOriA3AxikfDAoCQgNRbtX6nXfZ9UgPd8x0cJ4Z/L18U3OV ZD5A== X-Gm-Message-State: AHQUAualu1gGUJJx6NhvQfpOGrGR5bpnVzmIq2jKIHLsRm1msLetZERc SqUS4CXDlsUf7NBO0NCcN0m7rA== X-Google-Smtp-Source: AHgI3IZJRSOAHAJ6ZYc0LDxGVJliHK7B5On/XoQJirawxNExqrohXLiCTuyaxRH8XKJh/CrDlzVUTA== X-Received: by 2002:adf:b7c1:: with SMTP id t1mr66778wre.248.1550058287333; Wed, 13 Feb 2019 03:44:47 -0800 (PST) Received: from bidouze.vm.6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id q6sm3851256wmq.4.2019.02.13.03.44.46 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 13 Feb 2019 03:44:46 -0800 (PST) Date: Wed, 13 Feb 2019 12:44:25 +0100 From: =?iso-8859-1?Q?Ga=EBtan?= Rivet To: Shahaf Shuler Cc: anatoly.burakov@intel.com, yskoh@mellanox.com, thomas@monjalon.net, ferruh.yigit@intel.com, nhorman@tuxdriver.com, dev@dpdk.org Message-ID: <20190213114425.gsmqftnmakk4u3pj@bidouze.vm.6wind.com> References: <20190213113504.7aoc7myuacpmqo2b@bidouze.vm.6wind.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20190213113504.7aoc7myuacpmqo2b@bidouze.vm.6wind.com> User-Agent: NeoMutt/20170113 (1.7.2) Subject: Re: [dpdk-dev] [PATCH 5/6] net/mlx5: support PCI device DMA map and unmap X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Feb 2019 11:44:47 -0000 On Wed, Feb 13, 2019 at 12:35:04PM +0100, Gaëtan Rivet wrote: > On Wed, Feb 13, 2019 at 11:10:25AM +0200, Shahaf Shuler wrote: > > The implementation reuses the external memory registration work done by > > commit[1]. > > > > Note about representors: > > > > The current representor design will not work > > with those map and unmap functions. The reason is that for representors > > we have multiple IB devices share the same PCI function, so mapping will > > happen only on one of the representors and not all of them. > > > > While it is possible to implement such support, the IB representor > > design is going to be changed during DPDK19.05. The new design will have > > a single IB device for all representors, hence sharing of a single > > memory region between all representors will be possible. > > > > [1] > > commit 7e43a32ee060 > > ("net/mlx5: support externally allocated static memory") > > > > Signed-off-by: Shahaf Shuler > > --- > > drivers/net/mlx5/mlx5.c | 2 + > > drivers/net/mlx5/mlx5_mr.c | 146 ++++++++++++++++++++++++++++++++++++++ > > drivers/net/mlx5/mlx5_rxtx.h | 5 ++ > > 3 files changed, 153 insertions(+) > > > > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c > > index a913a5955f..7c91701713 100644 > > --- a/drivers/net/mlx5/mlx5.c > > +++ b/drivers/net/mlx5/mlx5.c > > @@ -1626,6 +1626,8 @@ static struct rte_pci_driver mlx5_driver = { > > .id_table = mlx5_pci_id_map, > > .probe = mlx5_pci_probe, > > .remove = mlx5_pci_remove, > > + .map = mlx5_dma_map, > > + .unmap = mlx5_dma_unmap, > > .drv_flags = (RTE_PCI_DRV_INTR_LSC | RTE_PCI_DRV_INTR_RMV | > > RTE_PCI_DRV_PROBE_AGAIN), > > }; > > diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c > > index 32be6a5445..7059181959 100644 > > --- a/drivers/net/mlx5/mlx5_mr.c > > +++ b/drivers/net/mlx5/mlx5_mr.c > > @@ -14,6 +14,7 @@ > > #include > > #include > > #include > > +#include > > > > #include "mlx5.h" > > #include "mlx5_mr.h" > > @@ -1215,6 +1216,151 @@ mlx5_mr_update_ext_mp_cb(struct rte_mempool *mp, void *opaque, > > } > > > > /** > > + * Finds the first ethdev that match the pci device. > > + * The existence of multiple ethdev per pci device is only with representors. > > + * On such case, it is enough to get only one of the ports as they all share > > + * the same ibv context. > > + * > > + * @param pdev > > + * Pointer to the PCI device. > > + * > > + * @return > > + * Pointer to the ethdev if found, NULL otherwise. > > + */ > > +static struct rte_eth_dev * > > +pci_dev_to_eth_dev(struct rte_pci_device *pdev) > > +{ > > + struct rte_eth_dev *dev; > > + const char *drv_name; > > + uint16_t port_id = 0; > > + > > + /** > > + * We really need to iterate all eth devices regardless of > > + * their owner. > > + */ > > + while (port_id < RTE_MAX_ETHPORTS) { > > + port_id = rte_eth_find_next(port_id); > > + if (port_id >= RTE_MAX_ETHPORTS) > > + break; > > + dev = &rte_eth_devices[port_id]; > > + drv_name = dev->device->driver->name; > > + if (!strncmp(drv_name, MLX5_DRIVER_NAME, > > + sizeof(MLX5_DRIVER_NAME) + 1) && > > + pdev == RTE_DEV_TO_PCI(dev->device)) { > > + /* found the PCI device. */ > > + return dev; > > + } > > + } > > + return NULL; > > +} > > Might I interest you in the new API? > > { > struct rte_dev_iterator it; > struct rte_device *dev; > > RTE_DEV_FOREACH(dev, "class=eth", &it) > if (dev == &pdev->device) > return it.class_device; > return NULL; > } > On that note, this could be in the PCI bus instead? > > + > > +/** > > + * DPDK callback to DMA map external memory to a PCI device. > > + * > > + * @param pdev > > + * Pointer to the PCI device. > > + * @param addr > > + * Starting virtual address of memory to be mapped. > > + * @param iova > > + * Starting IOVA address of memory to be mapped. > > + * @param len > > + * Length of memory segment being mapped. > > + * > > + * @return > > + * 0 on success, negative value on error. > > + */ > > +int > > +mlx5_dma_map(struct rte_pci_device *pdev, void *addr, > > + uint64_t iova __rte_unused, size_t len) > > +{ > > + struct rte_eth_dev *dev; > > + struct mlx5_mr *mr; > > + struct priv *priv; > > + > > + dev = pci_dev_to_eth_dev(pdev); > > + if (!dev) { > > + DRV_LOG(WARNING, "unable to find matching ethdev " > > + "to PCI device %p", (void *)pdev); > > + return -1; > > + } > > + priv = dev->data->dev_private; > > + mr = mlx5_create_mr_ext(dev, (uintptr_t)addr, len, SOCKET_ID_ANY); > > + if (!mr) { > > + DRV_LOG(WARNING, > > + "port %u unable to dma map", dev->data->port_id); > > + return -1; > > + } > > + rte_rwlock_write_lock(&priv->mr.rwlock); > > + LIST_INSERT_HEAD(&priv->mr.mr_list, mr, mr); > > + /* Insert to the global cache table. */ > > + mr_insert_dev_cache(dev, mr); > > + rte_rwlock_write_unlock(&priv->mr.rwlock); > > + return 0; > > +} > > + > > +/** > > + * DPDK callback to DMA unmap external memory to a PCI device. > > + * > > + * @param pdev > > + * Pointer to the PCI device. > > + * @param addr > > + * Starting virtual address of memory to be unmapped. > > + * @param iova > > + * Starting IOVA address of memory to be unmapped. > > + * @param len > > + * Length of memory segment being unmapped. > > + * > > + * @return > > + * 0 on success, negative value on error. > > + */ > > +int > > +mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, > > + uint64_t iova __rte_unused, size_t len __rte_unused) > > +{ > > + struct rte_eth_dev *dev; > > + struct priv *priv; > > + struct mlx5_mr *mr; > > + struct mlx5_mr_cache entry; > > + > > + dev = pci_dev_to_eth_dev(pdev); > > + if (!dev) { > > + DRV_LOG(WARNING, "unable to find matching ethdev " > > + "to PCI device %p", (void *)pdev); > > + return -1; > > + } > > + priv = dev->data->dev_private; > > + rte_rwlock_read_lock(&priv->mr.rwlock); > > + mr = mr_lookup_dev_list(dev, &entry, (uintptr_t)addr); > > + if (!mr) { > > + DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't registered " > > + "to PCI device %p", (uintptr_t)addr, > > + (void *)pdev); > > + rte_rwlock_read_unlock(&priv->mr.rwlock); > > + return -1; > > + } > > + LIST_REMOVE(mr, mr); > > + LIST_INSERT_HEAD(&priv->mr.mr_free_list, mr, mr); > > + DEBUG("port %u remove MR(%p) from list", dev->data->port_id, > > + (void *)mr); > > + mr_rebuild_dev_cache(dev); > > + /* > > + * Flush local caches by propagating invalidation across cores. > > + * rte_smp_wmb() is enough to synchronize this event. If one of > > + * freed memsegs is seen by other core, that means the memseg > > + * has been allocated by allocator, which will come after this > > + * free call. Therefore, this store instruction (incrementing > > + * generation below) will be guaranteed to be seen by other core > > + * before the core sees the newly allocated memory. > > + */ > > + ++priv->mr.dev_gen; > > + DEBUG("broadcasting local cache flush, gen=%d", > > + priv->mr.dev_gen); > > + rte_smp_wmb(); > > + rte_rwlock_read_unlock(&priv->mr.rwlock); > > + return 0; > > +} > > + > > +/** > > * Register MR for entire memory chunks in a Mempool having externally allocated > > * memory and fill in local cache. > > * > > diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h > > index c2529f96bc..f3f84dbac3 100644 > > --- a/drivers/net/mlx5/mlx5_rxtx.h > > +++ b/drivers/net/mlx5/mlx5_rxtx.h > > @@ -28,6 +28,7 @@ > > #include > > #include > > #include > > +#include > > > > #include "mlx5_utils.h" > > #include "mlx5.h" > > @@ -367,6 +368,10 @@ uint32_t mlx5_rx_addr2mr_bh(struct mlx5_rxq_data *rxq, uintptr_t addr); > > uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb); > > uint32_t mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr, > > struct rte_mempool *mp); > > +int mlx5_dma_map(struct rte_pci_device *pdev, void *addr, uint64_t iova, > > + size_t len); > > +int mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, uint64_t iova, > > + size_t len); > > > > /** > > * Provide safe 64bit store operation to mlx5 UAR region for both 32bit and > > -- > > 2.12.0 > > > > -- > Gaëtan Rivet > 6WIND -- Gaëtan Rivet 6WIND