What happens when the Nvidia GPU driver kernel callback occurs to invalidate the pinned GPU memory region?  Doesn’t the NIC need to cease all DMA transfers to/from that region before the kernel callback can complete?

 

From: Elena Agostini <eagostini@nvidia.com>
Sent: 04 January 2022 13:55
To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>
Cc: dev@dpdk.org
Subject: Re: [PATCH v2] gpudev: pin GPU memory

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

> 04/01/2022 03:41, eagostini@nvidia.com:

> > From: Elena Agostini <eagostini@nvidia.com>

> >

> > Enable the possibility to make a GPU memory area accessible from

> > the CPU.

> >

> > GPU memory has to be allocated via rte_gpu_mem_alloc().

> >

> > This patch allows the gpudev library to pin, through the GPU driver,

> > a chunk of GPU memory and to return a memory pointer usable

> > by the CPU to access the GPU memory area.

> >

> > Signed-off-by: Elena Agostini <eagostini@nvidia.com>

> [...]

> > +/**

> > + * @warning

> > + * @b EXPERIMENTAL: this API may change without prior notice.

> > + *

> > + * Pin a chunk of GPU memory to make it accessible from the CPU

>

> You should define what means "pin" exactly.

> Which properties should we expect?

>

 

Thanks for reviewing, this is the kind of discussion I wanted to have.

Maybe "pin" is too GDRCopy specific oriented.

Here I want to make a GPU memory buffer visible from the CPU. In case

of NVIDIA, this means the GPU memory address has to be pinned (virtual address

doesn't change) and dma-mapped.

 

Maybe the name should be more like rte_gpu_mem_to_cpu() that's more

explicative and generic.

 

 

> > + * using the memory pointer returned by the function.

>

> Which function should return the pointer?

> rte_gpu_mem_pin is returning an int.

 

Oversight, will fix it.

 

>

>

> > + * GPU memory has to be allocated via rte_gpu_mem_alloc().

>

> Why pinning is not done by rte_gpu_mem_alloc()?

> Should it be a flag?

 

rte_gpu_mem_alloc() allocate virtual memory on the GPU that doesn't have

to be necessarily shared (pinned) to make it visible from CPU.

 

>

> > + *

> > + * @param dev_id

> > + *   Device ID requiring pinned memory.

> > + * @param size

> > + *   Number of bytes to pin.

> > + *   Requesting 0 will do nothing.

> > + * @param ptr

> > + *   Pointer to the GPU memory area to be pinned.

> > + *   NULL is a no-op accepted value.

> > +

> > + * @return

> > + *   A pointer to the pinned GPU memory usable by the CPU, otherwise NULL and rte_errno is set:

> > + *   - ENODEV if invalid dev_id

> > + *   - EINVAL if reserved flags

>

> Which reserved flags?

>

> > + *   - ENOTSUP if operation not supported by the driver

> > + *   - E2BIG if size is higher than limit

> > + *   - ENOMEM if out of space

>

> Is out of space relevant for pinning?

 

Yes, let me add it

 

>

> > + *   - EPERM if driver error

> > + */

> > +__rte_experimental

> > +int rte_gpu_mem_pin(int16_t dev_id, size_t size, void *ptr);