From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <gaetan.rivet@6wind.com> Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by dpdk.org (Postfix) with ESMTP id 417201B14C for <dev@dpdk.org>; Thu, 14 Feb 2019 15:01:17 +0100 (CET) Received: by mail-wm1-f66.google.com with SMTP id f16so6195902wmh.4 for <dev@dpdk.org>; Thu, 14 Feb 2019 06:01:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=h7OuPpL0TC2HrwoYOLGeMgODfOpy/x8IIUTja+1zZCU=; b=YIJwM0oJpYeq4aiHjlYOmktxSttDNJ4wiH7VBi6R9aj2cyY3I/1LVa95XUyn4VPUE1 uIgKDDe/bkd2v52baD6f8emeDNQf/jVW0b/6yy+7EXNwscigKB13jDkx0GfyhH4nkr4k w/kkEKCITofb4PG4wgkjaiB1rVHM3u57GoZPxmBLbmkGKbypzFq8nsTqg83++MzsBHi8 of28Z5uXXiKfkuDhKYd7atIxzUZJp0penCBfgwEGU5VKHNpNNF3VpH/9EWNHARSnPNDB IPSLa+CZ/WN4L4hfw+9BdgCclocAzESjEfdXk9fB0VgG5e7wD9RtQgvDzzHVZxtVtIJc WcJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=h7OuPpL0TC2HrwoYOLGeMgODfOpy/x8IIUTja+1zZCU=; b=r4srOUQyQmSL63hcWa9mitbeukHetxfDpfmaV/i5QKQriOuBXhLSxYdx4hbmPMeIfG Q1xCDN2DmjPDlcQQTj8HUXnvZxORB1Br6TRHetqE3QFTj/Ka2WLeh2bA+q/Wte1zt7Bp VI0hDm3RcKKbGMpOelvqfOU+zgxkgD3HhE46q1ktIAsdYvxgR8LKAqohjLaZj3X5YHjt ujyjYSduFD7t0miCrfZ9Xga315c2VDtJ3KutX9+RTSDd1wfQqq4CyvpPPPHBLPN6ebhw k1uUJ0dXrP45Crlq8HPNdBiWVa6vi7ypz7MvJ2gOjegmM1kBj5q98meK7A6TI5ci9ibN TyyA== X-Gm-Message-State: AHQUAuYNqU2CYGZK6KxD4YkLoQiULQ/7vu9hFJwIZjT5OodGLMBgNSvM vyyKB1qGUEJ9XMX0Jy4SD74lNw== X-Google-Smtp-Source: AHgI3IYGb4535fHNLSA/CWoyvXLIwqpif462fNXM8fuXkY05XL10IlcULseYPggnsQ05hTdFtFjKLg== X-Received: by 2002:a7b:cb82:: with SMTP id m2mr3095474wmi.135.1550152876603; Thu, 14 Feb 2019 06:01:16 -0800 (PST) Received: from bidouze.vm.6wind.com (host.78.145.23.62.rev.coltfrance.com. [62.23.145.78]) by smtp.gmail.com with ESMTPSA id d15sm2586415wrw.36.2019.02.14.06.01.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Feb 2019 06:01:15 -0800 (PST) Date: Thu, 14 Feb 2019 15:00:53 +0100 From: =?iso-8859-1?Q?Ga=EBtan?= Rivet <gaetan.rivet@6wind.com> To: Shahaf Shuler <shahafs@mellanox.com> Cc: "anatoly.burakov@intel.com" <anatoly.burakov@intel.com>, Yongseok Koh <yskoh@mellanox.com>, Thomas Monjalon <thomas@monjalon.net>, "ferruh.yigit@intel.com" <ferruh.yigit@intel.com>, "nhorman@tuxdriver.com" <nhorman@tuxdriver.com>, "dev@dpdk.org" <dev@dpdk.org> Message-ID: <20190214140053.relxge6hcjcuylpy@bidouze.vm.6wind.com> References: <cover.1550048187.git.shahafs@mellanox.com> <323319abdbdc238c3586dafe9ad49dab554d6e64.1550048188.git.shahafs@mellanox.com> <20190213111702.wj3cqg6skttqduc4@bidouze.vm.6wind.com> <AM0PR0502MB379531E6C68A572768103D1AC3660@AM0PR0502MB3795.eurprd05.prod.outlook.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <AM0PR0502MB379531E6C68A572768103D1AC3660@AM0PR0502MB3795.eurprd05.prod.outlook.com> User-Agent: NeoMutt/20170113 (1.7.2) Subject: Re: [dpdk-dev] [PATCH 3/6] bus: introduce DMA memory mapping for external memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Thu, 14 Feb 2019 14:01:17 -0000 On Wed, Feb 13, 2019 at 07:07:11PM +0000, Shahaf Shuler wrote: > Wednesday, February 13, 2019 1:17 PM, Gaëtan Rivet: > > Subject: Re: [PATCH 3/6] bus: introduce DMA memory mapping for external > > memory > > > > On Wed, Feb 13, 2019 at 11:10:23AM +0200, Shahaf Shuler wrote: > > > The DPDK APIs expose 3 different modes to work with memory used for > > DMA: > > > > > > 1. Use the DPDK owned memory (backed by the DPDK provided > > hugepages). > > > This memory is allocated by the DPDK libraries, included in the DPDK > > > memory system (memseg lists) and automatically DMA mapped by the > > DPDK > > > layers. > > > > > > 2. Use memory allocated by the user and register to the DPDK memory > > > systems. This is also referred as external memory. Upon registration > > > of the external memory, the DPDK layers will DMA map it to all needed > > > devices. > > > > > > 3. Use memory allocated by the user and not registered to the DPDK > > > memory system. This is for users who wants to have tight control on > > > this memory. The user will need to explicitly call DMA map function in > > > order to register such memory to the different devices. > > > > > > The scope of the patch focus on #3 above. > > > > > > Currently the only way to map external memory is through VFIO > > > (rte_vfio_dma_map). While VFIO is common, there are other vendors > > > which use different ways to map memory (e.g. Mellanox and NXP). > > > > > > > How are those other vendors' devices mapped initially right now? Are they > > using #2 scheme instead? Then the user will remap everything using #3? > > It is not a re-map, it is a completely different mode for the memory management. > The first question to ask is "how the application wants to manage its memory" ? > If it is either #1 or #2 above, no problem to make the mapping internal on the "other vendor devices" as they can register to the memory event callback which trigger every time new memory is added to the DPDK memory management system. > For #3 the memory does not exists in the DPDK memory management system, and no memory events. Hence the application needs to explicitly call the dma MAP. > The change on this patch is just to make it more generic than calling only VFIO. > Right! I mostly used #1 ports and never really thought about other kind of memory management or how they might follow a different logic. Do you think this could be used with a lot of sequential mapping/unmappings happening? I'm thinking for example about a crypto app feeding crypto buffers, being able to directly map the result instead of copying it within buffers might be interesting. But then you'd have to unmap often. - Is the unmap() simple from the app PoV? - Must the mapping remain available for a long time? - Does the app need to call tx_descriptor_status() a few times or does dma_unmap() verify that the mapping is not in use before unmapping? -- Gaëtan Rivet 6WIND