DMA Transfers to PCIe Bar Memory

DMA Transfers to PCIe Bar Memory

[Wieckowski, Jacob]

[-- Attachment #1: Type: text/plain, Size: 221 bytes --]

Hello,

Is DMA access to the bar memory of a PCIe device with DPDK possible?
If so, which example project could be used as a reference to gain a better understanding of this topic?

Kind regards
Jakob Wieckowski


[-- Attachment #2: Type: text/html, Size: 2084 bytes --]

Re: DMA Transfers to PCIe Bar Memory

[Dmitry Kozlyuk]

2024-11-06 15:50 (UTC+0000), Wieckowski, Jacob:
> Is DMA access to the bar memory of a PCIe device with DPDK possible?
> If so, which example project could be used as a reference to gain a better understanding of this topic?

Hi,

BAR (base address register) is a place in PCI configuration space,
so one really doesn't "DMA transfers to PCIe BAR memory" to be precise.
Read https://wiki.osdev.org/PCI_Express to explore the topic.

DPDK drivers write to BARs to configure devices,
specifically to setup memory-mapped IO which is used for DMA.
Take a look at this project, which is a toy version of DPDK,
and thus it is easier to grasp:
https://github.com/ixy-languages/ixy-languages

RE: DMA Transfers to PCIe Bar Memory

[Wieckowski, Jacob]

Hi Dimitry,

thank you for the quick response.

Ok, DMA in the classic sense is not possible. 
However, if you carry out a write transfer into the BAR memory from DPDK, then, as I understand it, this access should be divided into several small postage-compliant TLP packets with a maximum payload size as specified in config space. 

Can block transfers in sizes of 512 bytes be carried out with the rte memcpy? The DPDK API states that the AVX-512 memcpy parameter must be enabled for x86 platforms.

Do other special precautions have to be taken in the DPDK environment to setup this kind of transfer?

Kind regards
Jakob

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Wednesday, November 6, 2024 6:46 PM
To: Wieckowski, Jacob <Jacob.Wieckowski@vector.com>
Cc: users@dpdk.org
Subject: Re: DMA Transfers to PCIe Bar Memory

[Sie erhalten nicht h?ufig E-Mails von dmitry.kozliuk@gmail.com. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]

2024-11-06 15:50 (UTC+0000), Wieckowski, Jacob:
> Is DMA access to the bar memory of a PCIe device with DPDK possible?
> If so, which example project could be used as a reference to gain a better understanding of this topic?

Hi,

BAR (base address register) is a place in PCI configuration space, so one really doesn't "DMA transfers to PCIe BAR memory" to be precise.
Read https://wiki.osdev.org/PCI_Express to explore the topic.

DPDK drivers write to BARs to configure devices, specifically to setup memory-mapped IO which is used for DMA.
Take a look at this project, which is a toy version of DPDK, and thus it is easier to grasp:
https://github.com/ixy-languages/ixy-languages

Re: DMA Transfers to PCIe Bar Memory

[Dmitry Kozlyuk]

2024-11-07 09:16 (UTC+0000), Wieckowski, Jacob:
> Hi Dimitry,
> 
> thank you for the quick response.
> 
> Ok, DMA in the classic sense is not possible.
>
> However, if you carry out a write transfer into the BAR memory from DPDK, then, as I understand it, this access should be divided into several small postage-compliant TLP packets with a maximum payload size as specified in config space. 
> 
> Can block transfers in sizes of 512 bytes be carried out with the rte memcpy? The DPDK API states that the AVX-512 memcpy parameter must be enabled for x86 platforms.
> 
> Do other special precautions have to be taken in the DPDK environment to setup this kind of transfer?

Could you please start with the problem you're solving?

DPDK uses DMA internally (mainly) to transfer packet data from/to HW.
It puts physical address of the buffer, etc. to NIC queue descriptor,
writes to a doorbell register, then the NIC DMA-writes/reads the buffer;
PCI transfer sizes are probably selected by HW.
All of this is within PMD (userspace drivers), no API is exposed.

rte_memcpy() is intended for copy from RAM to RAM.
You can Cc: Morten Brørup <mb@smartsharesystems.com> probably,
but I doubt that rte_memcpy() is specialized for DMA in any way.
The buffer may be filled with rte_mempcy() by application,
but this is done before handling the buffer to PMD, and thus before DMA.

Are you looking for functionality of "dmadev" library?

	https://doc.dpdk.org/guides/prog_guide/dmadev.html

RE: DMA Transfers to PCIe Bar Memory

[Wieckowski, Jacob]

Hi Jacob,

2024-11-07 11:50 (UTC+0000), Wieckowski, Jacob:
> Hi Dimitry,
> 
> I am sorry, I was a bit too far ahead with my thoughts.
> 
> We have started a DPDK evaluation project to build knowledge and understanding of the DPDK framework. 
> We used an E1000 driver with a workaround to gain access to the PCIe 5.0 R-Tile on the Intel Agilex7 FPGA under Windows. 
> 
> Accesses to the PCIe BAR work in principle, but we can currently only carry out 64-bit accesses to the BAR memory.  
> In the PCIe Config Space in the Capabilities Register, a maximum payload size of 512 bytes is configured. 
> The Intel Core in the FPGA and the Root Complex also support TLPs of this length. 
> 
> We use the rte_read32 and rte_write32 functions to access the bar memory, which obviously executes accesses with max 2 DWs. 
> We were able to trigger this in the FPGA because only TLP with length 2 arrived on the RX Interface. 
> 
> How can a block transfer be initiated in DPDK so that TLPs with a length of 128 DWs are generated on the PCIe Bus?

Thanks, now I understand what you need.
Unfortunately DPDK has no relevant API; rte_read**() is for small registers.
Maybe there's some useful code in "drivers/raw", but this area is completely unknown to me, sorry.
Try replying with what you wrote above to the thread in users@dpdk.org and Cc: Intel people from MAINTAINERS file responsible for "drivers/raw".

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Thursday, November 7, 2024 10:42 AM
To: Wieckowski, Jacob <Jacob.Wieckowski@vector.com>
Cc: users@dpdk.org
Subject: Re: DMA Transfers to PCIe Bar Memory

[Sie erhalten nicht häufig E-Mails von dmitry.kozliuk@gmail.com. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]

2024-11-07 09:16 (UTC+0000), Wieckowski, Jacob:
> Hi Dimitry,
>
> thank you for the quick response.
>
> Ok, DMA in the classic sense is not possible.
>
> However, if you carry out a write transfer into the BAR memory from DPDK, then, as I understand it, this access should be divided into several small postage-compliant TLP packets with a maximum payload size as specified in config space.
>
> Can block transfers in sizes of 512 bytes be carried out with the rte memcpy? The DPDK API states that the AVX-512 memcpy parameter must be enabled for x86 platforms.
>
> Do other special precautions have to be taken in the DPDK environment to setup this kind of transfer?

Could you please start with the problem you're solving?

DPDK uses DMA internally (mainly) to transfer packet data from/to HW.
It puts physical address of the buffer, etc. to NIC queue descriptor, writes to a doorbell register, then the NIC DMA-writes/reads the buffer; PCI transfer sizes are probably selected by HW.
All of this is within PMD (userspace drivers), no API is exposed.

rte_memcpy() is intended for copy from RAM to RAM.
You can Cc: Morten Brørup <mb@smartsharesystems.com> probably, but I doubt that rte_memcpy() is specialized for DMA in any way.
The buffer may be filled with rte_mempcy() by application, but this is done before handling the buffer to PMD, and thus before DMA.

Are you looking for functionality of "dmadev" library?

        https://doc.dpdk.org/guides/prog_guide/dmadev.html

Re: DMA Transfers to PCIe Bar Memory

[Stephen Hemminger]

On Thu, 7 Nov 2024 12:42:25 +0300
Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:

> rte_memcpy() is intended for copy from RAM to RAM.
> You can Cc: Morten Brørup <mb@smartsharesystems.com> probably,
> but I doubt that rte_memcpy() is specialized for DMA in any way.
> The buffer may be filled with rte_mempcy() by application,
> but this is done before handling the buffer to PMD, and thus before DMA.

A little more detail.
rte_memcpy() was an attempt at optimization of memcpy().
With current compilers, rte_memcpy() and memcpy() end up producing close
to the same code; the only cases left where there is a difference is likely
because GCC is more pessemistic about alignment.
The plan is to get rid of rte_memcpy() all together.

Re: DMA Transfers to PCIe Bar Memory

[Stephen Hemminger]

On Thu, 7 Nov 2024 13:09:41 +0000
"Wieckowski, Jacob" <Jacob.Wieckowski@vector.com> wrote:

> Hi Jacob,
> 
> 2024-11-07 11:50 (UTC+0000), Wieckowski, Jacob:
> > Hi Dimitry,
> > 
> > I am sorry, I was a bit too far ahead with my thoughts.
> > 
> > We have started a DPDK evaluation project to build knowledge and understanding of the DPDK framework. 
> > We used an E1000 driver with a workaround to gain access to the PCIe 5.0 R-Tile on the Intel Agilex7 FPGA under Windows. 
> > 
> > Accesses to the PCIe BAR work in principle, but we can currently only carry out 64-bit accesses to the BAR memory.  
> > In the PCIe Config Space in the Capabilities Register, a maximum payload size of 512 bytes is configured. 
> > The Intel Core in the FPGA and the Root Complex also support TLPs of this length. 
> > 
> > We use the rte_read32 and rte_write32 functions to access the bar memory, which obviously executes accesses with max 2 DWs. 
> > We were able to trigger this in the FPGA because only TLP with length 2 arrived on the RX Interface. 
> > 
> > How can a block transfer be initiated in DPDK so that TLPs with a length of 128 DWs are generated on the PCIe Bus?  
> 
> Thanks, now I understand what you need.
> Unfortunately DPDK has no relevant API; rte_read**() is for small registers.
> Maybe there's some useful code in "drivers/raw", but this area is completely unknown to me, sorry.
> Try replying with what you wrote above to the thread in users@dpdk.org and Cc: Intel people from MAINTAINERS file responsible for "drivers/raw".

In general, direct access to PCI by applications is discouraged.
All PCI access should be in driver code. What you describe is a bug in the E1000
driver, and should fixed there.