From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E32EDA034F; Tue, 8 Jun 2021 06:10:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 569B7410EB; Tue, 8 Jun 2021 06:10:32 +0200 (CEST) Received: from mail-il1-f173.google.com (mail-il1-f173.google.com [209.85.166.173]) by mails.dpdk.org (Postfix) with ESMTP id 1608B410E7 for ; Tue, 8 Jun 2021 06:10:31 +0200 (CEST) Received: by mail-il1-f173.google.com with SMTP id t6so11223055iln.8 for ; Mon, 07 Jun 2021 21:10:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=c7JfSBUXFWIKv2kCVcSm2dRfbnaiVhxlyxBbY8ubp+U=; b=l9HHzfbayfvU39aAJtVEeSYbQ+nL84t1AKtViPNZ5UWGDbthP48XK34K+8k/IeRvgF q9rP3frWNSDnhEHa7t3/USS2j4sd8e7wBDraDziZCoo/D0uKxvU8tNYO+F3a3408TguL WYC9QfMHi46XZXvIKZ6RVR4w4bTxWphwyWQB5IAY2lEDon5rNWsed3lYtaBlqCJref60 yKsbcYWc7qpTB0r4z3TJ+075EQ2/l2pd9akVN16FcTnxFb4eAPOsptXYalm2WKgNvhTZ qaS/dPlx2lM0cgFCWpijTfN+otlB/Jt1u/G20MV+UwephH2mg7jksxhjobz3KD+TpCgK sdDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=c7JfSBUXFWIKv2kCVcSm2dRfbnaiVhxlyxBbY8ubp+U=; b=uQ0AQnwE8/7a/v8yIub8VxSJPKZpmSRZMF7n/cLtFakCfM7gR/rx8qdQSbaQR7gQNJ ePkp6JapOOnVifAhoJheLvrfF6A36A5kSB802C51eldkj7l7TMJWRdsvlz/HclAyCK7e 9Rkpa2UZu1F95li+aMkFqf04tI1C+NrDznODizeBxboiQWf+GAFQ3i1J7Z//g83vwtLs JIuWZKKnnZVc8xEnY61PZemZXYqhCz18ma2c8uZLTmyiRFEI7qQxROWGanPX8W3iRY5N pW91ktpLjy/4btK+phkQgVYoEy8JhiTFAVAre29B81kh25NmL+QRJUQNo2QEvZjVj7BU 0SCw== X-Gm-Message-State: AOAM530YWJg2YLiTQ1zkV6Tynd/Af2vJMtwvgKZRNGelC1vveN56zCuY qurt4E76Pk1cw9omidEgLyNhGd9TYFNvyvQzdQ0= X-Google-Smtp-Source: ABdhPJydbrd1SdwT5LTMeVaE/l5S90aIJmbCfOUGafMS8NgZJmbk3uFeYoQ19Ie7hKkuK2z1NgbTwBG7M9yIHwo6hDc= X-Received: by 2002:a05:6e02:1b87:: with SMTP id h7mr16838534ili.271.1623125430268; Mon, 07 Jun 2021 21:10:30 -0700 (PDT) MIME-Version: 1.0 References: <20210602203531.2288645-1-thomas@monjalon.net> <3716354.mlbyQRhZUS@thomas> <2428387.JO1QuEOxcK@thomas> In-Reply-To: <2428387.JO1QuEOxcK@thomas> From: Jerin Jacob Date: Tue, 8 Jun 2021 09:40:14 +0530 Message-ID: To: Thomas Monjalon Cc: Honnappa Nagarahalli , Andrew Rybchenko , "Yigit, Ferruh" , dpdk-dev , Elena Agostini , David Marchand , nd , "Wang, Haiyue" Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH] gpudev: introduce memory API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, Jun 7, 2021 at 10:17 PM Thomas Monjalon wrote: > > 07/06/2021 15:54, Jerin Jacob: > > On Mon, Jun 7, 2021 at 4:13 PM Thomas Monjalon wrote: > > > 07/06/2021 09:20, Wang, Haiyue: > > > > From: Honnappa Nagarahalli > > > > > If we keep CXL in mind, I would imagine that in the future the devices on PCIe could have their own > > > > > local memory. May be some of the APIs could use generic names. For ex: instead of calling it as > > > > > "rte_gpu_malloc" may be we could call it as "rte_dev_malloc". This way any future device which hosts > > > > > its own memory that need to be managed by the application, can use these APIs. > > > > > > > > > > > > > "rte_dev_malloc" sounds a good name, > > > > > > Yes I like the idea. > > > 2 concerns: > > > > > > 1/ Device memory allocation requires a device handle. > > > So far we avoided exposing rte_device to the application. > > > How should we get a device handle from a DPDK application? > > > > Each device behaves differently at this level. In the view of the > > generic application, the architecture should like > > > > < Use DPDK subsystem as rte_ethdev, rte_bbdev etc for SPECIFIC function > > > ^ > > | > > < DPDK driver> > > ^ > > | > > > > I think the formatting went wrong above. > > I would add more to the block diagram: > > class device API - computing device API > | | | > class device driver - computing device driver > | | > EAL device with memory callback > > The idea above is that the class device driver can use services > of the new computing device library. Yes. The question is, do we need any public DPDK _application_ APIs for that? If it is public API then the scope is much bigger than that as the application can use it directly and it makes it non portable. if the scope is only, the class driver consumption then the existing "bus" _kind of_ abstraction/API makes sense to me. Where it abstracts, -FW download of device -Memory management of device -Opaque way to enq/deque jobs to the device. And above should be consumed by "class driver" not "application". If the application doing do that, we are in rte_raw device territory. > One basic API service is to provide a device ID for the memory callback. > Other services are for execution control. > > > An implementation may decide to have "in tree" or "out of tree" > > drivers or rte_device implementaion. > > But generic DPDK applications should not use devices directly. i.e > > rte_device need to have this callback and > > mlx ethdev/crypto driver use this driver to implement public API. > > Otherwise, it is the same as rawdev in DPDK. > > So not sure what it brings other than raw dev here if we are not > > taking the above architecture. > > > > > > > > 2/ Implementation must be done in a driver. > > > Should it be a callback defined at rte_device level? > > > > IMO, Yes and DPDK subsystem drivers to use it. > > I'm not sure subsystems should bypass the API for device memory. > We could do some generic work in the API function and call > the driver callback only for device-specific stuff. > In such case the callback and the API would be > in the library computing device library. > On the other hand, having the callback and API in EAL would allow > having a common function for memory allocation in EAL. > > Another thought: I would like to unify memory allocation in DPDK > with the same set of flags in an unique function. > A flag could be used to target devices instead of the running CPU, > and the same parameter could be shared for the device ID or NUMA node. > >