From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70078.outbound.protection.outlook.com [40.107.7.78]) by dpdk.org (Postfix) with ESMTP id A2A6F5F17 for ; Wed, 13 Feb 2019 20:11:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G0QV/Youe3nnsw6l2I3nwqJctDaOAhmIVoW54wV0Xec=; b=jjldce/9wSScYRZZ/oNqFHPMiQ13mAhHuouTqLEDMS8bgGpAui9Gl/G/v0oRTmQXk5M1G6CMlsCHuJDUCKy4v75nt4G1SwhKJdu+Y1RYWrlxj7GSeeB9hhOeHeyKlVof8Mgl88Wub2PEdqzd62kOQ9KsCQo+aWkzOZoryqPbtlo= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.47.29) by AM0PR0502MB3620.eurprd05.prod.outlook.com (52.133.46.33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1622.16; Wed, 13 Feb 2019 19:11:35 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::386e:f86:50bf:31be]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::386e:f86:50bf:31be%5]) with mapi id 15.20.1622.016; Wed, 13 Feb 2019 19:11:35 +0000 From: Shahaf Shuler To: =?iso-8859-1?Q?Ga=EBtan_Rivet?= CC: "anatoly.burakov@intel.com" , Yongseok Koh , Thomas Monjalon , "ferruh.yigit@intel.com" , "nhorman@tuxdriver.com" , "dev@dpdk.org" Thread-Topic: [PATCH 5/6] net/mlx5: support PCI device DMA map and unmap Thread-Index: AQHUw5A5KoxVu+EYWUWdWXo4EvvB4aXdm8+AgAA6+FA= Date: Wed, 13 Feb 2019 19:11:35 +0000 Message-ID: References: <20190213113504.7aoc7myuacpmqo2b@bidouze.vm.6wind.com> <20190213114425.gsmqftnmakk4u3pj@bidouze.vm.6wind.com> In-Reply-To: <20190213114425.gsmqftnmakk4u3pj@bidouze.vm.6wind.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8575fa6e-a0dc-443d-15ca-08d691e712e4 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600110)(711020)(4605077)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB3620; x-ms-traffictypediagnostic: AM0PR0502MB3620: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-exchange-diagnostics: =?iso-8859-1?Q?1; AM0PR0502MB3620; 23:de7R4vIjZB3qojRctgW74f/VkKPxktTb+KxYy?= =?iso-8859-1?Q?bKd70fuzRI4JafQksE+TSPRFuZirHVqvCAHlZKxCJxx9u4q+cswlwPMb3/?= =?iso-8859-1?Q?7EJ1qAt/E9H8ZGU0KixGkMHEtDmbSa9JSKFwJ3kG8vN2i5MlokoGy/RbKu?= =?iso-8859-1?Q?mhS1Zy3Dle+opyu7YgoVDyTtxcHBNnVOhOLQIYxSLzuRa/+S0CLUGqyr9w?= =?iso-8859-1?Q?5ANW7ZKbPDttOqqrd8cw66zoY0BQ2MoTz+u9F7RfY1+ppm2dId1kbKgxUs?= =?iso-8859-1?Q?8rkC27Jqh7gCfCKbZCTGMFEEgp0bN7iZdMtJ2dQl8dxwh3YXjtKTEdsSTl?= =?iso-8859-1?Q?bk7pQEt/pr61cdwDMDrM5zKna90/KMn/GlnU5KXmF1OHoagNlqyJDaGFb/?= =?iso-8859-1?Q?XA+LMjkLNqVajyrw+8tzm9L/bn0eYAIm6zTIp0MNsR8aMm5zoDjG/KwvK+?= =?iso-8859-1?Q?VXsYMNY9lIhCmToyDsQGNtSHVsNEWeb0cMod66YPuMMQZxaZDcWYPUpXla?= =?iso-8859-1?Q?gXIUug/ftRadNx1Esmn9GGFN+vzvXS0S9+132CM0+LKOa00xeyawq9Gw3W?= =?iso-8859-1?Q?LbJCnvZ344jy4AlT9TUCoPjiAqN1tebR2wvquCvoREpQVfZc2XXCuUitdf?= =?iso-8859-1?Q?0Y8ZPDyHXyaFOfcK9hw74R5rj4DICh5xGZqDSOQDzMZrx7AE5DgKwEC5jE?= =?iso-8859-1?Q?NI3cHiAsKdJWZxo666JgGE04xKrB3HwsvPIWZJVZhlwZWKGujjr63txByW?= =?iso-8859-1?Q?X4kEdifasRdGlHRXZrCGNlrtbwGc9jrgKjdQhSdnB/E/kc8Wbw1jOwAJpm?= =?iso-8859-1?Q?aex8iHzA3W6oBy9Gy4z4NPzbeKA26HS13poS8cCO1JWU6MdGIOYwbxghpE?= =?iso-8859-1?Q?1nM3Z+1yVZ1gtQeKfPva5T/AnYREhgT91DmkFvlgTHnQVedg3ch2MFceru?= =?iso-8859-1?Q?i7LMMudE0db+Q4gRf1GPlIh/j943UZ80bUL+dsz2itpkG4vmf9kHjZdC88?= =?iso-8859-1?Q?6HMwYaYJ9HSIgyHNuGyT9wdz5oP0afs8g3eS6dAlpKnlZT5wpl96bvf1Cq?= =?iso-8859-1?Q?xJDPZTBTfR2h6v/Ef/Lzn265QinHU2/wco1ksJPunOFmFN0Z5uT/qOioXB?= =?iso-8859-1?Q?c+6Nm17EE4TGZDBwl2X36YrrYkla/5vmaw4U8wuqEVkhdxkJww9BzVNwZ2?= =?iso-8859-1?Q?kaBSLNWaC2PK0S/Y6vAVJ3vvLhfU8i98l1amol7UtGs+BO9u/L08Sr5NAY?= =?iso-8859-1?Q?JinnJaOJcsh1WvF6RA87thXRv2f6C7OlasYjQWldA=3D=3D?= x-microsoft-antispam-prvs: x-forefront-prvs: 094700CA91 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(136003)(346002)(396003)(39860400002)(376002)(366004)(189003)(199004)(7696005)(33656002)(478600001)(54906003)(2906002)(26005)(4326008)(76176011)(99286004)(93886005)(229853002)(186003)(6436002)(25786009)(14454004)(476003)(106356001)(486006)(6116002)(97736004)(55016002)(9686003)(105586002)(11346002)(446003)(3846002)(81166006)(6246003)(81156014)(7736002)(8936002)(6506007)(305945005)(86362001)(74316002)(14444005)(66066001)(256004)(53936002)(8676002)(68736007)(71190400001)(102836004)(66574012)(71200400001)(316002)(6916009); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB3620; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: +PZVjnOYa9WHouV9A4EmoA+HXEELsnhc2mF6yify3NvRJM0SYNsADppetSu2/Ae2ycdsBxCiHrxq/20PCqT/ggy68zOsC3yz+VjjkFVM+XYsxybjqr41HCKrnFSmwOwQJ2QP59cPPmdk89OU6kDhF29g+QrO69KrRmagPiZWckoyYpE2ROPYcvmLpic3xIedNWdpT3P84jNLN1z40Po9pPenC4KAqAr5w9EgEyVnJQ0k7tX7QeU+aHtN8agPHuvvD99VUahCXtJuWHySWzyylPuhifZe0LWynJq51fHIUvNsMQLPztHgbnrZuwsTiAmTmSu6/9w1Sqtef6ayaSnm6aXYcIYczoey7rLR9cgVMmbOQJ7csg6xQZBbVC81IdEDC1z3VPSWjtJRDM6rSO0yB7YMrS9XDuBkZYc55jZXqFw= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8575fa6e-a0dc-443d-15ca-08d691e712e4 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Feb 2019 19:11:35.8503 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB3620 Subject: Re: [dpdk-dev] [PATCH 5/6] net/mlx5: support PCI device DMA map and unmap X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Feb 2019 19:11:41 -0000 Wednesday, February 13, 2019 1:44 PM, Ga=EBtan Rivet: > Subject: Re: [PATCH 5/6] net/mlx5: support PCI device DMA map and unmap >=20 > On Wed, Feb 13, 2019 at 12:35:04PM +0100, Ga=EBtan Rivet wrote: > > On Wed, Feb 13, 2019 at 11:10:25AM +0200, Shahaf Shuler wrote: > > > The implementation reuses the external memory registration work done [..] > > > + > > > + /** > > > + * We really need to iterate all eth devices regardless of > > > + * their owner. > > > + */ > > > + while (port_id < RTE_MAX_ETHPORTS) { > > > + port_id =3D rte_eth_find_next(port_id); > > > + if (port_id >=3D RTE_MAX_ETHPORTS) > > > + break; > > > + dev =3D &rte_eth_devices[port_id]; > > > + drv_name =3D dev->device->driver->name; > > > + if (!strncmp(drv_name, MLX5_DRIVER_NAME, > > > + sizeof(MLX5_DRIVER_NAME) + 1) && > > > + pdev =3D=3D RTE_DEV_TO_PCI(dev->device)) { > > > + /* found the PCI device. */ > > > + return dev; > > > + } > > > + } > > > + return NULL; > > > +} > > > > Might I interest you in the new API? Good suggestion, will have a look on it in depth.=20 > > > > { > > struct rte_dev_iterator it; > > struct rte_device *dev; > > > > RTE_DEV_FOREACH(dev, "class=3Deth", &it) > > if (dev =3D=3D &pdev->device) > > return it.class_device; > > return NULL; > > } > > >=20 > On that note, this could be in the PCI bus instead? We can put it on the PCI bus, but it would mean the PCI bus will not be dev= ice agnostic.=20 Currently, I couldn't find any reference to eth_dev on the PCI bus, besides= a single macro which convert to pci device that doesn't really do type che= cks.=20 Having it in, would mean the PCI will need start to distinguish between eth= dev, crypto dev and what ever devices exists on its bus.=20 >=20 > > > + > > > +/** > > > + * DPDK callback to DMA map external memory to a PCI device. > > > + * > > > + * @param pdev > > > + * Pointer to the PCI device. > > > + * @param addr > > > + * Starting virtual address of memory to be mapped. > > > + * @param iova > > > + * Starting IOVA address of memory to be mapped. > > > + * @param len > > > + * Length of memory segment being mapped. > > > + * > > > + * @return > > > + * 0 on success, negative value on error. > > > + */ > > > +int > > > +mlx5_dma_map(struct rte_pci_device *pdev, void *addr, > > > + uint64_t iova __rte_unused, size_t len) { > > > + struct rte_eth_dev *dev; > > > + struct mlx5_mr *mr; > > > + struct priv *priv; > > > + > > > + dev =3D pci_dev_to_eth_dev(pdev); > > > + if (!dev) { > > > + DRV_LOG(WARNING, "unable to find matching ethdev " > > > + "to PCI device %p", (void *)pdev); > > > + return -1; > > > + } > > > + priv =3D dev->data->dev_private; > > > + mr =3D mlx5_create_mr_ext(dev, (uintptr_t)addr, len, > SOCKET_ID_ANY); > > > + if (!mr) { > > > + DRV_LOG(WARNING, > > > + "port %u unable to dma map", dev->data->port_id); > > > + return -1; > > > + } > > > + rte_rwlock_write_lock(&priv->mr.rwlock); > > > + LIST_INSERT_HEAD(&priv->mr.mr_list, mr, mr); > > > + /* Insert to the global cache table. */ > > > + mr_insert_dev_cache(dev, mr); > > > + rte_rwlock_write_unlock(&priv->mr.rwlock); > > > + return 0; > > > +} > > > + > > > +/** > > > + * DPDK callback to DMA unmap external memory to a PCI device. > > > + * > > > + * @param pdev > > > + * Pointer to the PCI device. > > > + * @param addr > > > + * Starting virtual address of memory to be unmapped. > > > + * @param iova > > > + * Starting IOVA address of memory to be unmapped. > > > + * @param len > > > + * Length of memory segment being unmapped. > > > + * > > > + * @return > > > + * 0 on success, negative value on error. > > > + */ > > > +int > > > +mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, > > > + uint64_t iova __rte_unused, size_t len __rte_unused) { > > > + struct rte_eth_dev *dev; > > > + struct priv *priv; > > > + struct mlx5_mr *mr; > > > + struct mlx5_mr_cache entry; > > > + > > > + dev =3D pci_dev_to_eth_dev(pdev); > > > + if (!dev) { > > > + DRV_LOG(WARNING, "unable to find matching ethdev " > > > + "to PCI device %p", (void *)pdev); > > > + return -1; > > > + } > > > + priv =3D dev->data->dev_private; > > > + rte_rwlock_read_lock(&priv->mr.rwlock); > > > + mr =3D mr_lookup_dev_list(dev, &entry, (uintptr_t)addr); > > > + if (!mr) { > > > + DRV_LOG(WARNING, "address 0x%" PRIxPTR " wasn't > registered " > > > + "to PCI device %p", (uintptr_t)addr, > > > + (void *)pdev); > > > + rte_rwlock_read_unlock(&priv->mr.rwlock); > > > + return -1; > > > + } > > > + LIST_REMOVE(mr, mr); > > > + LIST_INSERT_HEAD(&priv->mr.mr_free_list, mr, mr); > > > + DEBUG("port %u remove MR(%p) from list", dev->data->port_id, > > > + (void *)mr); > > > + mr_rebuild_dev_cache(dev); > > > + /* > > > + * Flush local caches by propagating invalidation across cores. > > > + * rte_smp_wmb() is enough to synchronize this event. If one of > > > + * freed memsegs is seen by other core, that means the memseg > > > + * has been allocated by allocator, which will come after this > > > + * free call. Therefore, this store instruction (incrementing > > > + * generation below) will be guaranteed to be seen by other core > > > + * before the core sees the newly allocated memory. > > > + */ > > > + ++priv->mr.dev_gen; > > > + DEBUG("broadcasting local cache flush, gen=3D%d", > > > + priv->mr.dev_gen); > > > + rte_smp_wmb(); > > > + rte_rwlock_read_unlock(&priv->mr.rwlock); > > > + return 0; > > > +} > > > + > > > +/** > > > * Register MR for entire memory chunks in a Mempool having external= ly > allocated > > > * memory and fill in local cache. > > > * > > > diff --git a/drivers/net/mlx5/mlx5_rxtx.h > > > b/drivers/net/mlx5/mlx5_rxtx.h index c2529f96bc..f3f84dbac3 100644 > > > --- a/drivers/net/mlx5/mlx5_rxtx.h > > > +++ b/drivers/net/mlx5/mlx5_rxtx.h > > > @@ -28,6 +28,7 @@ > > > #include > > > #include > > > #include > > > +#include > > > > > > #include "mlx5_utils.h" > > > #include "mlx5.h" > > > @@ -367,6 +368,10 @@ uint32_t mlx5_rx_addr2mr_bh(struct > > > mlx5_rxq_data *rxq, uintptr_t addr); uint32_t > > > mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb); > uint32_t mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t > addr, > > > struct rte_mempool *mp); > > > +int mlx5_dma_map(struct rte_pci_device *pdev, void *addr, uint64_t > iova, > > > + size_t len); > > > +int mlx5_dma_unmap(struct rte_pci_device *pdev, void *addr, uint64_t > iova, > > > + size_t len); > > > > > > /** > > > * Provide safe 64bit store operation to mlx5 UAR region for both > > > 32bit and > > > -- > > > 2.12.0 > > > > > > > -- > > Ga=EBtan Rivet > > 6WIND >=20 > -- > Ga=EBtan Rivet > 6WIND