From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2D031A0093; Tue, 4 Jan 2022 14:55:30 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D926540042; Tue, 4 Jan 2022 14:55:28 +0100 (CET) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2053.outbound.protection.outlook.com [40.107.243.53]) by mails.dpdk.org (Postfix) with ESMTP id 6103540040 for ; Tue, 4 Jan 2022 14:55:27 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MM1z+M6ikb0IJnp73aD2huMBus+mnxzIBH6yFNsBdcD1vFP7lFasfkKb88WHHORkFzx6Td81WrH8N5TjeiTYWCczgV5U7AdmTq/RK/i13GC9YroLRYNugR5o8UHU1aqQxHagpFTsW7Qk6TgEdZ6ZYoEELFY5DOqf/tlcyKPjUu9QqYMOdiNuQl6wa8liY8ik1ZVGxQ6baYIRQLQWBRmXyjopa8XHdE5VvBvHpQISMpoZQNY0i3li+X8yXkTJzq1Nf9xDXDhOMRiEzWEyEtsZRDKoWVzOFonJYl4/UrEgSrSKdoJiQT03ZJz+CCQX4Z5nGQV5P/NAEZD4IGPzXuIMEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rI1DiNHDzPX0YQ6ZOCujMJKTgMo3ryvsRbwAeyZHE2Y=; b=RN4wzwZ38ufvnqUw2vo8JDNiIhggepZUtp3Rpk2aI3+cam4UAR6IfNZGzVLS7MdjdH8QlCFmHdDH1LDH/t9oXSxzBbyhuXdoYyPFyWEkb/NqM07NzzRPMbLc321oFoBuppzx/AsM2SLfYXBz2AsLra1tzsfBb78cCq1yAO+BxnU5oO5H65Tpqyy0N0U5uEB7m6ou5msqV7zMj6VUi2fMvt1u0IPseGpsLo4COUvm0x/iAV8eFLUebzOpa1/0HZL3Xv3PDrPzOhPFFtk40dDJgc11FJS/Q1HdUyq1vgOQtBy1hn03PBwxFR/hRZgKRi/qpJGLzEH2j+DWE3M8wrUsJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rI1DiNHDzPX0YQ6ZOCujMJKTgMo3ryvsRbwAeyZHE2Y=; b=J3T0DCTqNKS6KYaoUzxAuEdDLxOSjhtrKy/AANzX3hss1pVgkWKwre1/ZIOkjZxb9OKXUlIyHZgbnHvy+LNhxHS+xq9Ja/33nVgnhAJzqgsEK5CogKaiX1BgbJcrHU/yMj13Cjsh5vx+a2U+FITLLustgDUnuba1Kn12eULB3ZODEen0FxalcggQ/sUdhsPAhEAxedVJP31HJNCZaX3pzLTBMEHsJd/i8jUbvPX9HHK8Bu/LP4tVY7YX5DsGQ77ol45jwUJwXYkZRR8HYCPXG+vPtmIGaC/nupWR7KUFvZgyw4kPHv0BPdnuGhgl97qJrLVrs3a2Gv/+c8g6SAkXPw== Received: from DM6PR12MB4107.namprd12.prod.outlook.com (2603:10b6:5:218::7) by DM6PR12MB4233.namprd12.prod.outlook.com (2603:10b6:5:210::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4844.15; Tue, 4 Jan 2022 13:55:26 +0000 Received: from DM6PR12MB4107.namprd12.prod.outlook.com ([fe80::7006:8ce7:3bc4:512]) by DM6PR12MB4107.namprd12.prod.outlook.com ([fe80::7006:8ce7:3bc4:512%4]) with mapi id 15.20.4844.016; Tue, 4 Jan 2022 13:55:25 +0000 From: Elena Agostini To: "NBU-Contact-Thomas Monjalon (EXTERNAL)" CC: "dev@dpdk.org" Subject: Re: [PATCH v2] gpudev: pin GPU memory Thread-Topic: [PATCH v2] gpudev: pin GPU memory Thread-Index: AQHYANAXuuNqv1hbAUeW28JARS7w/axS0gqAgAARxOc= Date: Tue, 4 Jan 2022 13:55:25 +0000 Message-ID: References: <20220104023408.13379-1-eagostini@nvidia.com> <20220104024100.14318-1-eagostini@nvidia.com> <12925392.uLZWGnKmhe@thomas> In-Reply-To: <12925392.uLZWGnKmhe@thomas> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 6b12f68e-be03-4bb0-9e2f-08d9cf89dc06 x-ms-traffictypediagnostic: DM6PR12MB4233:EE_ x-ld-processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 8ghjqw6zp0sQOglEhk3l79qdZOC/nPzR6vRxt925+xUQLllBAASmFSt3G/++sZWHx4CN4zd+xgKQCPVloJJPcH9D99h9XUy+RgoE2my7nGFOd82nZySTJyjo4OlLhoTv5PKJmUQap++Py93uuJaTEPOlyNTjp2YvAqHGQqUm7WTpTN6+KrVzAc7vt6y8QEI8joIR5PtxpeaYMgLjw3hMwTDRdpajvonfrsNzQyECJIWkdSZKKCRf68FQRHv9fV8CjHHIKX09ilNI2c3wOSXbtXWVi1LQ9DqtOGGH6gGlfmcWDV1n0j43VwNn0VyjyZ0rqiWWT6odv9kaXPmvroNAHImeJQX1n0iLFtZZlddZdaJJYN+4sqQn7b2qdhC92TDDOK5EumLx6xAfMK11NQ2q6BAl9B4Io5+e0fZBknRj+zscvOceQJXXO4lG0kKuVMNqEgCF0zwxefw9aHCpvhJeCfK+Z8z9g8giYKlXSaTnrlkGAMo0WfPbCiLVze2G58FCvc5v6Pwm8faCZCWlWqc7ndUvwslH27IHMedEuJCeBxPmUSjOxKoTqQnLXzzO1hz+D1Cx3rZIZB4JLTPpZTfwd47jFqtGVc+vnyoB9HjdIyWO8zsUtn+aBrRTif5Iq2R48bl7KUkvPJw6dT+ydLZlaNzGx9b0dLvVK6aSzSjT5Rlyr/cHMEe5W63vqaZwU5XrvAyMtfF1KPEEHQI+WlkRXQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB4107.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(66446008)(7696005)(508600001)(83380400001)(316002)(8936002)(122000001)(26005)(52536014)(5660300002)(186003)(71200400001)(86362001)(38070700005)(55016003)(6506007)(38100700002)(76116006)(91956017)(8676002)(9686003)(6916009)(2906002)(4326008)(64756008)(66476007)(66946007)(33656002)(66556008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?GZn4tSx3emesWUG+y9EGxo0Uqa0Pp4iwzJFsDe88YF1PICtHs5zDGnCAHnUC?= =?us-ascii?Q?SsBFHWe6g7u9aUmaEILv8O/hOQ5ScX/AQdmtukkxXWSmUWfNAszVos8yuXOv?= =?us-ascii?Q?J65PvnF5TSNtEF5OEYFg2gHDzAdUAWuoNvM4OlRsvd/6bKrLbueMwgrOytqJ?= =?us-ascii?Q?zagm6TWKD3Ti3S5lDQYlchJKoRE+xijKwuLxkkN6lQ7QPaOdhrr5ASKgP6AX?= =?us-ascii?Q?oPmJYYykALEIq/eiz9Msb5T3lwe0wXD0eOBBD1K4ankW4TP68Grt50hzUvvU?= =?us-ascii?Q?iibS6oBZlvIP1I12bX/0ee+dYTTA7MewMTcZpNh6u4M/7jhSkwW4lGLoqcjn?= =?us-ascii?Q?byd2bxuq7aXMrjZprYl46GG6MJBZ5JSCbK6Du8Y9cL5hB7aHkm59hMI0CsFH?= =?us-ascii?Q?BV9quxiDptgwdXfWyQvz/aXwqUDJ2DOMlp5Zh4vVHcg7k7pxIwFHbGQdeDqG?= =?us-ascii?Q?GXeQGEyai6K/UQ6VMQAcJepuoGY6DHqakFzg6CO37ZWCdx9n47sE+R4SDNv+?= =?us-ascii?Q?ithMO+8pJU8PplmBOQIfmSS1I0WZJK05oH6GODNla2rQuDQzJe7dpMh78zni?= =?us-ascii?Q?GEk8QfpaXaS8fIaQcZ8/j09csKJsWsQv5MMyTBZV5HPYkxkre/ns6CBs+OR5?= =?us-ascii?Q?EPsB+de9R1o8FA49X8xOaQK/h+Xl2sqXCZvr7O9QMnNigV1d4Ma9EDx0quoX?= =?us-ascii?Q?wiw5no4A6P9fV1atbhxDrOUkrUmiPicVxAfW1RfkaZ2R1aiUoH1iZ0RqrXp1?= =?us-ascii?Q?Jn/hP10cMBLS5/mWOUIftNu/UWmwhzEfdchJnRfGx4tuArAWDEn/Gxl8HoBZ?= =?us-ascii?Q?2ZEGRsah+uDEuQ3b/XGvzN8Dbzllc0KR8R/+Z/SCmNmazoOmMiLMnt8eq7eF?= =?us-ascii?Q?BZS+2PLimoFl/Lv3MorFwAcH1rK/65KktYYYPXid7+n+BpREv8XvPqUx15AX?= =?us-ascii?Q?I1q/2ZfXEeJ8+wpJ21KCgbGiDtZ6kaEL1Bqn5zqTShcaRV4HvGeQubnZ9V4H?= =?us-ascii?Q?lWDJV2I7VutzGF7AdRhwUYJYPD6c4l2brpvtcsU9VWjDJFyY7Dpa74ZRlQp/?= =?us-ascii?Q?qA1pK+WkNnWtWBz3hVcXEV4HyoGR02LuVlFJvSVFpFRkbeSuJ44RWn2S22g6?= =?us-ascii?Q?QA0x68rwBVTy7EhhZjYRkc28e3jS9VltJZ0jd64iRQDAEjZ6/YIrSbPRINHJ?= =?us-ascii?Q?fw69ssCu8HwqhBQQH47hcxCpHY+o8W5bqDyP9pKCkeW0m/hFzOPy3NXSkM09?= =?us-ascii?Q?o+42/hO5bLVYeQZgndFozqbpMdK+SJCI1nVKCw///ZQd987QA16ix1KYm3AX?= =?us-ascii?Q?Q0V5IlIwThAVLE4aVjSJVF+du90c735QaU/4eMBm64/rysmada3LVxEuVyhW?= =?us-ascii?Q?IcG1VXRqzEV3IZV2/EwWQzJXkOGshmXZ2LjRssGcSD+UtjlcAtJjqVEP6XM4?= =?us-ascii?Q?x7gcaQQli4XhvHSVu4dCA9yHaKUIelvZtHU0hxYW+BNEAYU6cjN1O4YSDybv?= =?us-ascii?Q?b2irWJkCW9xVUd0CKZX6RnpusGoLOAe6zkNhBao3+ZrUb+Krg53VF2XeA0FR?= =?us-ascii?Q?fXIv7E1zEW3+nvLhF7ffH+iWz+WDWIisFalfE1Fj4S5x4j2MLMXb4U+xzI1f?= =?us-ascii?Q?SmBeiThsiSew97Zm7H65mWkgeD9DKZhWTXSDZqlbZeBh?= Content-Type: multipart/alternative; boundary="_000_DM6PR12MB41075D97594DB4EEA4A63160CD4A9DM6PR12MB4107namp_" MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB4107.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6b12f68e-be03-4bb0-9e2f-08d9cf89dc06 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jan 2022 13:55:25.7900 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Nh7LQEMKzsLYQXwnHQMtHC7KH631WRYC1wTxMNo2XN1KcL0LH0X6gIYkA/V/C1hK27hl05wbJYK0YILmX9Gjeg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4233 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --_000_DM6PR12MB41075D97594DB4EEA4A63160CD4A9DM6PR12MB4107namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable > 04/01/2022 03:41, eagostini@nvidia.com: > > From: Elena Agostini > > > > Enable the possibility to make a GPU memory area accessible from > > the CPU. > > > > GPU memory has to be allocated via rte_gpu_mem_alloc(). > > > > This patch allows the gpudev library to pin, through the GPU driver, > > a chunk of GPU memory and to return a memory pointer usable > > by the CPU to access the GPU memory area. > > > > Signed-off-by: Elena Agostini > [...] > > +/** > > + * @warning > > + * @b EXPERIMENTAL: this API may change without prior notice. > > + * > > + * Pin a chunk of GPU memory to make it accessible from the CPU > > You should define what means "pin" exactly. > Which properties should we expect? > Thanks for reviewing, this is the kind of discussion I wanted to have. Maybe "pin" is too GDRCopy specific oriented. Here I want to make a GPU memory buffer visible from the CPU. In case of NVIDIA, this means the GPU memory address has to be pinned (virtual addr= ess doesn't change) and dma-mapped. Maybe the name should be more like rte_gpu_mem_to_cpu() that's more explicative and generic. > > + * using the memory pointer returned by the function. > > Which function should return the pointer? > rte_gpu_mem_pin is returning an int. Oversight, will fix it. > > > > + * GPU memory has to be allocated via rte_gpu_mem_alloc(). > > Why pinning is not done by rte_gpu_mem_alloc()? > Should it be a flag? rte_gpu_mem_alloc() allocate virtual memory on the GPU that doesn't have to be necessarily shared (pinned) to make it visible from CPU. > > > + * > > + * @param dev_id > > + * Device ID requiring pinned memory. > > + * @param size > > + * Number of bytes to pin. > > + * Requesting 0 will do nothing. > > + * @param ptr > > + * Pointer to the GPU memory area to be pinned. > > + * NULL is a no-op accepted value. > > + > > + * @return > > + * A pointer to the pinned GPU memory usable by the CPU, otherwise N= ULL and rte_errno is set: > > + * - ENODEV if invalid dev_id > > + * - EINVAL if reserved flags > > Which reserved flags? > > > + * - ENOTSUP if operation not supported by the driver > > + * - E2BIG if size is higher than limit > > + * - ENOMEM if out of space > > Is out of space relevant for pinning? Yes, let me add it > > > + * - EPERM if driver error > > + */ > > +__rte_experimental > > +int rte_gpu_mem_pin(int16_t dev_id, size_t size, void *ptr); --_000_DM6PR12MB41075D97594DB4EEA4A63160CD4A9DM6PR12MB4107namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

> 04/01/2022 03:= 41, eagostini@nvidia.com:

> > From: Ele= na Agostini <eagostini@nvidia.com>

> >

> > Enable th= e possibility to make a GPU memory area accessible from

> > the CPU.<= o:p>

> >

> > GPU memor= y has to be allocated via rte_gpu_mem_alloc().

> >

> > This patc= h allows the gpudev library to pin, through the GPU driver,

> > a chunk o= f GPU memory and to return a memory pointer usable

> > by the CP= U to access the GPU memory area.

> >

> > Signed-of= f-by: Elena Agostini <eagostini@nvidia.com>

> [...]

> > +/**=

> > + * @warn= ing

> > + * @b EX= PERIMENTAL: this API may change without prior notice.

> > + *<= /o:p>

> > + * Pin a= chunk of GPU memory to make it accessible from the CPU

>

> You should def= ine what means "pin" exactly.

> Which properti= es should we expect?

>

 

Thanks for reviewin= g, this is the kind of discussion I wanted to have.

Maybe "pin&quo= t; is too GDRCopy specific oriented.

Here I want to make= a GPU memory buffer visible from the CPU. In case

of NVIDIA, this mea= ns the GPU memory address has to be pinned (virtual address

doesn't change) and= dma-mapped.

 

Maybe the name shou= ld be more like rte_gpu_mem_to_cpu() that's more

explicative and gen= eric.

 

> > + * using= the memory pointer returned by the function.

>

> Which function= should return the pointer?

> rte_gpu_mem_pi= n is returning an int.

 

Oversight, will fix= it.

 

>

>

> > + * GPU m= emory has to be allocated via rte_gpu_mem_alloc().

>

> Why pinning is= not done by rte_gpu_mem_alloc()?

> Should it be a= flag?

 

rte_gpu_mem_alloc()= allocate virtual memory on the GPU that doesn't have

to be necessarily s= hared (pinned) to make it visible from CPU.

 

>

> > + *<= /o:p>

> > + * @para= m dev_id

> > + * =   Device ID requiring pinned memory.

> > + * @para= m size

> > + * =   Number of bytes to pin.

> > + * =   Requesting 0 will do nothing.

> > + * @para= m ptr

> > + * =   Pointer to the GPU memory area to be pinned.

> > + * =   NULL is a no-op accepted value.

> > +

> > + * @retu= rn

> > + * =   A pointer to the pinned GPU memory usable by the CPU, otherwise NULL= and rte_errno is set:

> > + * =   - ENODEV if invalid dev_id

> > + * =   - EINVAL if reserved flags

>

> Which reserved= flags?

>

> > + * =   - ENOTSUP if operation not supported by the driver=

> > + * =   - E2BIG if size is higher than limit

> > + * =   - ENOMEM if out of space

>

> Is out of spac= e relevant for pinning?

 

Yes, let me add it<= o:p>

 

>

> > + * =   - EPERM if driver error

> > + */=

> > +__rte_ex= perimental

> > +int rte_= gpu_mem_pin(int16_t dev_id, size_t size, void *ptr);

--_000_DM6PR12MB41075D97594DB4EEA4A63160CD4A9DM6PR12MB4107namp_--