From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 12006A0C43; Tue, 19 Oct 2021 12:00:55 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DDA1040683; Tue, 19 Oct 2021 12:00:54 +0200 (CEST) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2072.outbound.protection.outlook.com [40.107.93.72]) by mails.dpdk.org (Postfix) with ESMTP id 5D3B74003E; Tue, 19 Oct 2021 12:00:52 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EY+s0Bv2+xTYRKscivi4bng4NkhVHOvjRT4nEoW/9aE2N1+HNofHUNHqfuMdcLPHJpZ3bAE4LJTDC/Ux/TVrncMVW9ZHUN6Yp9ELDEhLJqfV93PCuuoRnT9Ajm2ic1DIEu4jy+Q3INfkZwzi6E7/jJlx1ygErKMkrwgTcivZ+C6hT1pFrMfAxHfKJhcK3Wb8cYCoazxymQxac3zIOor0ny+lgPupwiYJiO4qK8CU1X/i0SW1ro0uOMnAi+HsAuiFlP4UCxFm92o88ccM6DGWFzBxMELZHmoGuPaooz0KwzpITwYPtBvIfqFfb8UHhQUo2bK8NwlK7wzQAWezVE7kGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=l6y75HO9mPtrec1jvMAQepvZPjJsrRegj3YGLQqSVlc=; b=NmGAGt2L9VGsL/fEr/2KtC6vGbGpf9Wy6HcaR8Z4asbC6aRblECbqnaD8WdQ0vLNUc11O8QIgh6nhExt96WNJSUwOBMuoOolpi9IcyCNIVAJFvTFVWodsmgIfh7CBdJFvooQ/hbzBsZ1nBO1sD0aKPOt30T3y8xCUnHD20ubMK4CfHu1g6MJfCNDj1GTY+Q2Srs4GYGi/SwPD740FgVFBXbekgjhtOI6YibIRrrekVeH+mmB2ZN2UxlYkV3fJBCec8dREHV+kkNEEkbU3GFCYMSgxAwiejxeQVFeFcXpzgvPmjc/sAp3Eps2/kRCd+NiRz4k0FXvUBdBxDg6PYE4Jg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l6y75HO9mPtrec1jvMAQepvZPjJsrRegj3YGLQqSVlc=; b=LrNq8TALG6jAvZs+xS8l3KW676/qCoUV8eNrmmDBsW8/PUMsqVbNiHwGxiLiK5b2cs9djt544eA+YX7D8dxObTDlviRuNebk7tpBN37zKvUvxMPgpG5r5GJS8l4lcSoYtykiG/PcCo61gSoOb5omgyjwId+NaUFHer1xyar3XO3h8xWUFVHIuEpaV2n0xlohagVBJnsOEThg+MsozhMAzrKUsB3m3w8XUDD4WI6gVJTzkBdt0pl4xCuECOWip6ZNN2FO+H0ulfkeoVtA4LLYPynVTq1/WvbU9Z4e2gW47Maj+I7KB1dbgUFZ4aRped52iubTAhyHliufQd/NDHY3xQ== Received: from DM6PR12MB4107.namprd12.prod.outlook.com (2603:10b6:5:218::7) by DM6PR12MB2729.namprd12.prod.outlook.com (2603:10b6:5:43::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.18; Tue, 19 Oct 2021 10:00:45 +0000 Received: from DM6PR12MB4107.namprd12.prod.outlook.com ([fe80::1556:93f2:3c85:3769]) by DM6PR12MB4107.namprd12.prod.outlook.com ([fe80::1556:93f2:3c85:3769%4]) with mapi id 15.20.4608.018; Tue, 19 Oct 2021 10:00:45 +0000 From: Elena Agostini To: Jerin Jacob , NBU-Contact-Thomas Monjalon CC: "techboard@dpdk.org" , dpdk-dev Thread-Topic: [dpdk-dev] [PATCH v3 0/9] GPU library Thread-Index: AQHXvGwGwKHZ9Rr9JEGfZEEDC6CmhKvMBvAAgAFxRICAAAcOAIAACDwAgAAErwCAABBFAIAAFJgAgAARioCAAAzcAIACyANX Date: Tue, 19 Oct 2021 10:00:45 +0000 Message-ID: References: <20210602203531.2288645-1-thomas@monjalon.net> <68083401.ybZ649KAnY@thomas> <8296426.PREcJboN3U@thomas> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 58cc1b43-4de5-4110-223a-08d992e751a1 x-ms-traffictypediagnostic: DM6PR12MB2729: x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: DkEXeChDLqgJjGRUFIx3ngTVqWKrS6SE/qZ2eqdKmTX7Rl1LJ142lG5qIj9npOJ2XtOdAN9bql/lUzr0sgvZdQgH94zHU4jh+CGm2vAdS3zHK99gynUEoxu3TfbKXmtbELcFbBGxURU/TQyJcjooWQ52vynWrB1S9C3Mzxbt+mOyomMIsmu69hLy0Yf6vUae746UazWuZr8VWK5GM6aXLt//iE/OBgunXM1IiZS9EtRyy3FdMR3ugmIM13M5UTYC9bMVWeLK/ZEv2rt+/Y3oIHwgz71knqYimc2NT0G+aay7C3VpLR0t/4fyjbSxXerIsfzNOWVafd3pW2ECAb8GaokLs1RUh9+VMJhb9zWENPBIHtn7VSDVRUfVU81iJ6QCgvs+0AZHnLpWpH+GtuX9BeuVyMkTSNNTqxBzdlbOT/oqLi+o709L5Pxm4fC8b36qwN7TQMD3ZIuy3ynfHukWMaw+qu1pjE3fOGLM/cir1jATYYppSVaP2muXQcZqelmcMpc8643UFJNsoDA10OAkH3Xn8SkHLQFSY3O8iZQZO2WzOAFPp0j0rABT74BRDoSq+ZtPNlgfepmvh6/bONfkuZ9CDsqk1O2/gwTYuRJ349WLxgcU/lPJ915DC85H4/lsSQa7ORevsD3K2V49+vJZMmigwSzGKcQ7/dKdulnuAO5wv4EzhYoDAtbpXKsNa/0b1HcjF7avXNiezfu0MzjnIg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB4107.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(66946007)(38100700002)(86362001)(5660300002)(66556008)(66476007)(71200400001)(53546011)(122000001)(186003)(38070700005)(83380400001)(508600001)(7696005)(91956017)(2906002)(64756008)(26005)(8936002)(6506007)(76116006)(4326008)(9686003)(52536014)(55016002)(66446008)(8676002)(33656002)(110136005)(54906003)(316002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?/28lHhNm7C3HZ1PZ6YNq6xTn2xIEg0703Yrz8AUKNHyu59yb3+3/TBiq?= =?Windows-1252?Q?HjOylJvvYd7Y7Ykkm9MygZPDgMFIoF8P88JRDO9BUZZZ20c6j1Uu/CY5?= =?Windows-1252?Q?z1YoJyeg2J1XMB1/R4mS7JtQLMxbLVM0O8dZpc7Al+5pwqEddSWUKGC9?= =?Windows-1252?Q?rzFdhbogapOH3lUUDLkKfhimOju8sGexbRgUrjlWSc6WjbAt6QjxeEVr?= =?Windows-1252?Q?4iAMy11pEH0H4l8AOTpFmPwNqjtECdyzoWlWJMrh2j6P3Xpim5K5b4JP?= =?Windows-1252?Q?aF0gHOGfmHLiaf5uaKnBVwjh5jPqTQ7FyHfAS8I1oINeBbcv5EYRkBrK?= =?Windows-1252?Q?iRt+QzJzRJ+1Rnas1J1IROu/TfvNQ7lh2kRpIST/b9weK24OrCe24MtN?= =?Windows-1252?Q?i8Cn5JrWMuN2TXWzOeLBqQpqY5PqVpz+R5UjaWr9dCgxRpIJYMIBwOav?= =?Windows-1252?Q?btDLVo2RjxyqZQl7RWwYwow1L8lTujEZDyBJtlmN15n7YhM1YAbOwJrZ?= =?Windows-1252?Q?u44Yp+wV9ZXCobM1gmSlgZcAbhXHy22PPoxj3O/01Ozw8n9DMJUjYAJt?= =?Windows-1252?Q?pkP+AesUaODuJl9pII/vF5wLURGXyhaMDGKLmd0yMOle7gdvPvZ3FTZ6?= =?Windows-1252?Q?O3VqfTbjs0gf1c16A6zlAHb8Y57ZV4E7TddRyLOqQJTETkEii6xoPo06?= =?Windows-1252?Q?B/AMtqm1vj9JxnbNWxFjeVfvS92iK7Hkk4mZWk4QVoWm6/yCKN4ciKwq?= =?Windows-1252?Q?lNh4PVU+Jr1Daaj3r6D3PPWjv3cO7uLrcJAXQv/69jKN4TaYztSfc5UT?= =?Windows-1252?Q?0fCL3VmHl/fwckSizjAC7mespM0K+Ld3ZTEMaorrrD83/vRejptbPE/j?= =?Windows-1252?Q?eDCrYToCwFGOENF1WtXdXusRejCcycLiWgspI39Y4wVMZsXBrC5QKklY?= =?Windows-1252?Q?LE0ETGnw+ZumNbfN6ppz7O5YGGIeoZ0xdRUdcXLq11Lotu9chQVnAfUS?= =?Windows-1252?Q?U9h87Jmcvn8jUwl3ur6fufEIa9kH0B+4fHqL/67bwKpvblFJmFXrNr7E?= =?Windows-1252?Q?M2RRP3TdgXwuNdy71mvnt6d7BbK3pza9pqVf+MkpT78s9YMWVfr92LUM?= =?Windows-1252?Q?DDrtrTKMP0wWOnrXDATdLKM4JVr/65sjTzJe1iA4/8uC2Qz5+QdexPjK?= =?Windows-1252?Q?Xq8BvnM7Cx4Qw9O0nxHjpeVtvpBySmvjLTrClU6tvAFOO7M2tzzF/wSF?= =?Windows-1252?Q?orhlvW1lEmEUcEy4FYn0Ak2c30z6e9GGRjKk7MOZBfW0zddwPxSOnQC2?= =?Windows-1252?Q?LgAKippC016IpH0/7WYuikJ6mhj4scLJeoDU5hePRcLnSRHyIvWXmbSh?= =?Windows-1252?Q?aIB/zG0rg2/W5rwwcIefN5Gu0AllWXKpxiKsnqDLSeIHFiQvf5mGBiPp?= =?Windows-1252?Q?c8TTsUFkeijl6bhwzLy/DQ=3D=3D?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB4107.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58cc1b43-4de5-4110-223a-08d992e751a1 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Oct 2021 10:00:45.3315 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: G0Vf06HS1m5qRCvR4VCyVPx66snA3GMAB8R5KDkUN7Q2qZYkdVRt5/k1UjgihqtPQb1L4ttMlwvyKfCMnjUlAA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2729 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-dev] [PATCH v3 0/9] GPU library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Jerin Jacob Date: Monday, 11 October 2021 at 15:30 To: NBU-Contact-Thomas Monjalon Cc: techboard@dpdk.org , Elena Agostini , dpdk-dev Subject: Re: [dpdk-dev] [PATCH v3 0/9] GPU library > On Mon, Oct 11, 2021 at 6:14 PM Thomas Monjalon wro= te: > > > > 11/10/2021 13:41, Jerin Jacob: > > > On Mon, Oct 11, 2021 at 3:57 PM Thomas Monjalon = wrote: > > > > 11/10/2021 11:29, Jerin Jacob: > > > > > On Mon, Oct 11, 2021 at 2:42 PM Thomas Monjalon wrote: > > > > > > 11/10/2021 10:43, Jerin Jacob: > > > > > > > On Mon, Oct 11, 2021 at 1:48 PM Thomas Monjalon wrote: > > > > > > > > 10/10/2021 12:16, Jerin Jacob: > > > > > > > > > On Fri, Oct 8, 2021 at 11:13 PM wr= ote: > > > > > > > > > > > > > > > > > > > > From: eagostini > > > > > > > > > > > > > > > > > > > > In heterogeneous computing system, processing is not on= ly in the CPU. > > > > > > > > > > Some tasks can be delegated to devices working in paral= lel. > > > > > > > > > > > > > > > > > > > > The goal of this new library is to enhance the collabor= ation between > > > > > > > > > > DPDK, that's primarily a CPU framework, and GPU devices= . > > > > > > > > > > > > > > > > > > > > When mixing network activity with task processing on a = non-CPU device, > > > > > > > > > > there may be the need to put in communication the CPU w= ith the device > > > > > > > > > > in order to manage the memory, synchronize operations, = exchange info, etc.. > > > > > > > > > > > > > > > > > > > > This library provides a number of new features: > > > > > > > > > > - Interoperability with GPU-specific library with gener= ic handlers > > > > > > > > > > - Possibility to allocate and free memory on the GPU > > > > > > > > > > - Possibility to allocate and free memory on the CPU bu= t visible from the GPU > > > > > > > > > > - Communication functions to enhance the dialog between= the CPU and the GPU > > > > > > > > > > > > > > > > > > In the RFC thread, There was one outstanding non technica= l issues on this, > > > > > > > > > > > > > > > > > > i.e > > > > > > > > > The above features are driver specific details. Does the = DPDK > > > > > > > > > _application_ need to be aware of this? > > > > > > > > > > > > > > > > I don't see these features as driver-specific. > > > > > > > > > > > > > > That is the disconnect. I see this as more driver-specific de= tails > > > > > > > which are not required to implement an "application" facing A= PI. > > > > > > > > > > > > Indeed this is the disconnect. > > > > > > I already answered but it seems you don't accept the answer. > > > > > > > > > > Same with you. That is why I requested, we need to get opinions f= rom others. > > > > > Some of them already provided opinions in RFC. > > > > > > > > This is why I Cc'ed techboard. > > > > > > Yes. Indeed. > > > > > > > > > > > > > First, this is not driver-specific. It is a low-level API. > > > > > > > > > > What is the difference between low-level API and driver-level API= . > > > > > > > > The low-level API provides tools to build a feature, > > > > but no specific feature. > > > > > > > > > > > For example, If we need to implement application facing" subs= ystems like bbdev, > > > > > > > If we make all this driver interface, you can still implement= the > > > > > > > bbdev API as a driver without > > > > > > > exposing HW specific details like how devices communicate to = CPU, how > > > > > > > memory is allocated etc > > > > > > > to "application". > > > > > > > > > > > > There are 2 things to understand here. > > > > > > > > > > > > First we want to allow the application using the GPU for needs = which are > > > > > > not exposed by any other DPDK API. > > > > > > > > > > > > Second, if we want to implement another DPDK API like bbdev, > > > > > > then the GPU implementation would be exposed as a vdev in bbdev= , > > > > > > using the HW GPU device being a PCI in gpudev. > > > > > > They are two different levels, got it? > > > > > > > > > > Exactly. So what is the point of exposing low-level driver API to > > > > > "application", > > > > > why not it is part of the internal driver API. My point is, why t= he > > > > > application needs to worry > > > > > about, How the CPU <-> Device communicated? CPU < -> Device memor= y > > > > > visibility etc. > > > > > > > > There are two reasons. > > > > > > > > 1/ The application may want to use the GPU for some application-spe= cific > > > > needs which are not abstracted in DPDK API. > > > > > > Yes. Exactly, That's where my concern, If we take this path, What is > > > the motivation to contribute to DPDK abstracted subsystem APIs which > > > make sense for multiple vendors and every > > > Similar stuff applicable for DPU, > > > > A feature-specific API is better of course, there is no lose of motivat= ion. > > But you cannot forbid applications to have their own features on GPU. > > it still can use it. We don't need DPDK APIs for that. > > > > > > Otherway to put, if GPU is doing some ethdev offload, why not making > > > as ethdev offload in ethdev spec so that > > > another type of device can be used and make sense for application wri= tters. > > > > If we do ethdev offload, yes we'll implement it. > > And we'll do it on top of gpudev, which is the only way to share the CP= U. > > > > > For example, In the future, If someone needs to add ML(Machine > > > learning) subsystem and enable a proper subsystem > > > interface that is good for DPDK. If this path is open, there is no > > > motivation for contribution and the application > > > will not have a standard interface doing the ML job across multiple v= endors. > > > > Wrong. It does remove the motivation, it is a first step to build on to= p of it. > > IMO, No need to make driver API to the public to feature API. > > > > > > That's is the only reason why saying it should not APPLICATION > > > interface it can be DRIVER interface. > > > > > > > > > > > 2/ This API may also be used by some feature implementation interna= lly > > > > in some DPDK libs or drivers. > > > > We cannot skip the gpudev layer because this is what allows generic= probing > > > > of the HW, and gpudev allows to share the GPU with multiple feature= s > > > > implemented in different libs or drivers, thanks to the "child" con= cept. > > > > > > Again, why do applications need to know it? It is similar to `bus` > > > kind of this where it sharing the physical resouces. > > > > No it's not a bus, it is a device that we need to share. > > > > > > > > > > > aka DPDK device class has a fixed personality and it has = API to deal > > > > > > > > > with abstracting specific application specific > > > > > > > > > end user functionality like ethdev, cryptodev, eventdev i= rrespective > > > > > > > > > of underlying bus/device properties. > > > > > > > > > > > > > > > > The goal of the lib is to allow anyone to invent any featur= e > > > > > > > > which is not already available in DPDK. > > > > > > > > > > > > > > > > > Even similar semantics are required for DPU(Smart NIC) > > > > > > > > > communitication. I am planning to > > > > > > > > > send RFC in coming days to address the issue without the = application > > > > > > > > > knowing the Bus/HW/Driver details. > > > > > > > > > > > > > > > > gpudev is not exposing bus/hw/driver details. > > > > > > > > I don't understand what you mean. > > > > > > > > > > > > > > See above. > > > > We are going into circles. > > Yes. > > > In short, Jerin wants to forbid the generic use of GPU in DPDK. > > See below. Honestly I don=92t see a real problem releasing the library at application = level. It doesn=92t prevent to use it internally by other DPDK libraries/drivers i= f needed. Applications can benefit of this library for a number of reasons: * Enhance the interaction between GPU specific library and DPDK * Hide GPU specific implementation details to the =93final=94 user that= wants to build a GPU + DPDK application * Measure network throughput with common tools like testpmd using GPU m= emory Please be aware that this is just a starting point. I=92m planning to expose a number of features (at memory and processing lev= els) that can be useful to enhance the communication among GPU, CPU and NIC hiding the implementati= on details within the library/driver. > > > He wants only feature-specific API. > > To re-reiterate, feature-specific "application" API. A device-specific > bit can be > driver API and accessible to the out-of-tree driver if needed. > > IMO, if we take this path, DPU, XPU, GPU, etc we need N different librari= es to > get the job done for a specific feature for the dataplane. > Instead, Enabling public feature APIs will make the application > portable and does not > need to worry about which type of *PU it runs. > As I stated multiple times, let=92s start with something simple that works = and then think about how to enhance the library/driver. IMHO it doesn=92t make sense to address all the use-cases now. This is a completely new scenario we=92re opening in the DPDK context, let= =92s start from the basis. > > > It is like restricting the functions we can run on a CPU. > > > > And anyway we need this layer to share the GPU between multiple feature= s. > > No disagreement there. Is that layer public application API or not is > the question. > it is like PCI device API calls over of the application and makes the > application device specific. > > > > > Techboard please vote. > > Yes. > > > > >