Re: [PATCH v2] gpudev: pin GPU memory

[Elena Agostini <eagostini@nvidia.com>]

[-- Attachment #1.1: Type: text/plain, Size: 7311 bytes --]

This specific feature is not about NIC activity, it’s about make GPU memory visible and accessible to the CPU so there isn’t an asynchronous hidden invalidation procedure.
That’s the reason for my comment to change the name from rte_gpu_mem_pin() to rte_gpu_mem_to_cpu() (or similar).

The scenario you are proposing can happen but in a different context, this is not the case.

Thanks
EA

From: John Alexander <John.Alexander@datapath.co.uk>
Date: Tuesday, 4 January 2022 at 18:31
To: Elena Agostini <eagostini@nvidia.com>, NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: dev@dpdk.org <dev@dpdk.org>
Subject: RE: [PATCH v2] gpudev: pin GPU memory
External email: Use caution opening links or attachments

What happens when the Nvidia GPU driver kernel callback occurs to invalidate the pinned GPU memory region?  Doesn’t the NIC need to cease all DMA transfers to/from that region before the kernel callback can complete?

John Alexander​
Senior Software Engineer
[cid:image152483.jpg@5CD90415.402D82AB]<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.datapath.co.uk%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9b3%2FEpEiezz%2B0nLDmm2pE5ZcLkEzNbW%2F%2BJ1o61q2J3k%3D&reserved=0>
Bemrose House, Bemrose Park, Wayzgoose Drive
,
Derby
,
DE21 6XQ
[cid:image839434.png@FD4260AC.D2B1B11C]
+44 (0)1332 294 441<tel:+44%20(0)1332%20294%20441>
 |
www.datapath.co.uk<https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.datapath.co.uk%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9b3%2FEpEiezz%2B0nLDmm2pE5ZcLkEzNbW%2F%2BJ1o61q2J3k%3D&reserved=0>
[LinkedIn]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fdatapath-ltd&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=0PTscJhiHEK27PKDjq2nNgH5oRe499xubLh4S705u4A%3D&reserved=0>
[Twitter]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.twitter.com%2Fdatapathltd&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CVia5Sq%2Fi2yOAGx5d8PSsF7wJsA2oVM%2BgySwg5E7p2U%3D&reserved=0>
[YouTube]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fdatapathderby&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Obz00eF3OgTdp3YI5ux2YFRdx6Q65keqlMXVY6QNXCM%3D&reserved=0>
[cid:image974625.png@E5A3F546.B8FD2EA9]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datapath.co.uk%2Fevents-and-webinars%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MO6M5IE0mAymLgzIZwGqPPxqftcmzH7Gy43ZOGYHwAs%3D&reserved=0>
[Vote for Datapath]<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.datapath.co.uk%2Fdatapath-shortlisted-for-coveted-industry-award%2F&data=04%7C01%7Ceagostini%40nvidia.com%7Ce2fbb83b86ac4dc3571308d9cfa7b653%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637769142848844684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=D0wKDJVR%2B9kSzyIvtxt5yOdKZPCb%2Fd48biairPlYqt4%3D&reserved=0>
Datapath Ltd.  Registered Number: 1609392.  Registered in England at Bemrose House, Bemrose Park, Wayzgoose Drive, Derby. DE21 6XQ.
From: Elena Agostini <eagostini@nvidia.com>
Sent: 04 January 2022 13:55
To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: dev@dpdk.org
Subject: Re: [PATCH v2] gpudev: pin GPU memory

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 04/01/2022 03:41, eagostini@nvidia.com<mailto:eagostini@nvidia.com>:
> > From: Elena Agostini <eagostini@nvidia.com<mailto:eagostini@nvidia.com>>
> >
> > Enable the possibility to make a GPU memory area accessible from
> > the CPU.
> >
> > GPU memory has to be allocated via rte_gpu_mem_alloc().
> >
> > This patch allows the gpudev library to pin, through the GPU driver,
> > a chunk of GPU memory and to return a memory pointer usable
> > by the CPU to access the GPU memory area.
> >
> > Signed-off-by: Elena Agostini <eagostini@nvidia.com<mailto:eagostini@nvidia.com>>
> [...]
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Pin a chunk of GPU memory to make it accessible from the CPU
>
> You should define what means "pin" exactly.
> Which properties should we expect?
>

Thanks for reviewing, this is the kind of discussion I wanted to have.
Maybe "pin" is too GDRCopy specific oriented.
Here I want to make a GPU memory buffer visible from the CPU. In case
of NVIDIA, this means the GPU memory address has to be pinned (virtual address
doesn't change) and dma-mapped.

Maybe the name should be more like rte_gpu_mem_to_cpu() that's more
explicative and generic.


> > + * using the memory pointer returned by the function.
>
> Which function should return the pointer?
> rte_gpu_mem_pin is returning an int.

Oversight, will fix it.

>
>
> > + * GPU memory has to be allocated via rte_gpu_mem_alloc().
>
> Why pinning is not done by rte_gpu_mem_alloc()?
> Should it be a flag?

rte_gpu_mem_alloc() allocate virtual memory on the GPU that doesn't have
to be necessarily shared (pinned) to make it visible from CPU.

>
> > + *
> > + * @param dev_id
> > + *   Device ID requiring pinned memory.
> > + * @param size
> > + *   Number of bytes to pin.
> > + *   Requesting 0 will do nothing.
> > + * @param ptr
> > + *   Pointer to the GPU memory area to be pinned.
> > + *   NULL is a no-op accepted value.
> > +
> > + * @return
> > + *   A pointer to the pinned GPU memory usable by the CPU, otherwise NULL and rte_errno is set:
> > + *   - ENODEV if invalid dev_id
> > + *   - EINVAL if reserved flags
>
> Which reserved flags?
>
> > + *   - ENOTSUP if operation not supported by the driver
> > + *   - E2BIG if size is higher than limit
> > + *   - ENOMEM if out of space
>
> Is out of space relevant for pinning?

Yes, let me add it

>
> > + *   - EPERM if driver error
> > + */
> > +__rte_experimental
> > +int rte_gpu_mem_pin(int16_t dev_id, size_t size, void *ptr);

[-- Attachment #1.2: Type: text/html, Size: 32898 bytes --]

[-- Attachment #2: image152483.jpg --]
[-- Type: image/jpeg, Size: 5403 bytes --]

[-- Attachment #3: image158572.png --]
[-- Type: image/png, Size: 651 bytes --]

[-- Attachment #4: image416332.png --]
[-- Type: image/png, Size: 620 bytes --]

[-- Attachment #5: image447896.png --]
[-- Type: image/png, Size: 684 bytes --]

[-- Attachment #6: image509100.png --]
[-- Type: image/png, Size: 24194 bytes --]

[-- Attachment #7: image839434.png --]
[-- Type: image/png, Size: 335 bytes --]

[-- Attachment #8: image974625.png --]
[-- Type: image/png, Size: 20712 bytes --]