DPDK usage discussions
 help / color / mirror / Atom feed
* DMA Transfers to PCIe Bar Memory
@ 2024-11-06 15:50 Wieckowski, Jacob
  2024-11-06 17:46 ` Dmitry Kozlyuk
  0 siblings, 1 reply; 7+ messages in thread
From: Wieckowski, Jacob @ 2024-11-06 15:50 UTC (permalink / raw)
  To: users

[-- Attachment #1: Type: text/plain, Size: 221 bytes --]

Hello,

Is DMA access to the bar memory of a PCIe device with DPDK possible?
If so, which example project could be used as a reference to gain a better understanding of this topic?

Kind regards
Jakob Wieckowski


[-- Attachment #2: Type: text/html, Size: 2084 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: DMA Transfers to PCIe Bar Memory
  2024-11-06 15:50 DMA Transfers to PCIe Bar Memory Wieckowski, Jacob
@ 2024-11-06 17:46 ` Dmitry Kozlyuk
  2024-11-07  9:16   ` Wieckowski, Jacob
  0 siblings, 1 reply; 7+ messages in thread
From: Dmitry Kozlyuk @ 2024-11-06 17:46 UTC (permalink / raw)
  To: Wieckowski, Jacob; +Cc: users

2024-11-06 15:50 (UTC+0000), Wieckowski, Jacob:
> Is DMA access to the bar memory of a PCIe device with DPDK possible?
> If so, which example project could be used as a reference to gain a better understanding of this topic?

Hi,

BAR (base address register) is a place in PCI configuration space,
so one really doesn't "DMA transfers to PCIe BAR memory" to be precise.
Read https://wiki.osdev.org/PCI_Express to explore the topic.

DPDK drivers write to BARs to configure devices,
specifically to setup memory-mapped IO which is used for DMA.
Take a look at this project, which is a toy version of DPDK,
and thus it is easier to grasp:
https://github.com/ixy-languages/ixy-languages

^ permalink raw reply	[flat|nested] 7+ messages in thread

* RE: DMA Transfers to PCIe Bar Memory
  2024-11-06 17:46 ` Dmitry Kozlyuk
@ 2024-11-07  9:16   ` Wieckowski, Jacob
  2024-11-07  9:42     ` Dmitry Kozlyuk
  0 siblings, 1 reply; 7+ messages in thread
From: Wieckowski, Jacob @ 2024-11-07  9:16 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users

Hi Dimitry,

thank you for the quick response.

Ok, DMA in the classic sense is not possible. 
However, if you carry out a write transfer into the BAR memory from DPDK, then, as I understand it, this access should be divided into several small postage-compliant TLP packets with a maximum payload size as specified in config space. 

Can block transfers in sizes of 512 bytes be carried out with the rte memcpy? The DPDK API states that the AVX-512 memcpy parameter must be enabled for x86 platforms.

Do other special precautions have to be taken in the DPDK environment to setup this kind of transfer?

Kind regards
Jakob

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Wednesday, November 6, 2024 6:46 PM
To: Wieckowski, Jacob <Jacob.Wieckowski@vector.com>
Cc: users@dpdk.org
Subject: Re: DMA Transfers to PCIe Bar Memory

[Sie erhalten nicht h?ufig E-Mails von dmitry.kozliuk@gmail.com. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]

2024-11-06 15:50 (UTC+0000), Wieckowski, Jacob:
> Is DMA access to the bar memory of a PCIe device with DPDK possible?
> If so, which example project could be used as a reference to gain a better understanding of this topic?

Hi,

BAR (base address register) is a place in PCI configuration space, so one really doesn't "DMA transfers to PCIe BAR memory" to be precise.
Read https://wiki.osdev.org/PCI_Express to explore the topic.

DPDK drivers write to BARs to configure devices, specifically to setup memory-mapped IO which is used for DMA.
Take a look at this project, which is a toy version of DPDK, and thus it is easier to grasp:
https://github.com/ixy-languages/ixy-languages

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: DMA Transfers to PCIe Bar Memory
  2024-11-07  9:16   ` Wieckowski, Jacob
@ 2024-11-07  9:42     ` Dmitry Kozlyuk
  2024-11-07 13:09       ` Wieckowski, Jacob
  2024-11-07 16:11       ` Stephen Hemminger
  0 siblings, 2 replies; 7+ messages in thread
From: Dmitry Kozlyuk @ 2024-11-07  9:42 UTC (permalink / raw)
  To: Wieckowski, Jacob; +Cc: users

2024-11-07 09:16 (UTC+0000), Wieckowski, Jacob:
> Hi Dimitry,
> 
> thank you for the quick response.
> 
> Ok, DMA in the classic sense is not possible.
>
> However, if you carry out a write transfer into the BAR memory from DPDK, then, as I understand it, this access should be divided into several small postage-compliant TLP packets with a maximum payload size as specified in config space. 
> 
> Can block transfers in sizes of 512 bytes be carried out with the rte memcpy? The DPDK API states that the AVX-512 memcpy parameter must be enabled for x86 platforms.
> 
> Do other special precautions have to be taken in the DPDK environment to setup this kind of transfer?

Could you please start with the problem you're solving?

DPDK uses DMA internally (mainly) to transfer packet data from/to HW.
It puts physical address of the buffer, etc. to NIC queue descriptor,
writes to a doorbell register, then the NIC DMA-writes/reads the buffer;
PCI transfer sizes are probably selected by HW.
All of this is within PMD (userspace drivers), no API is exposed.

rte_memcpy() is intended for copy from RAM to RAM.
You can Cc: Morten Brørup <mb@smartsharesystems.com> probably,
but I doubt that rte_memcpy() is specialized for DMA in any way.
The buffer may be filled with rte_mempcy() by application,
but this is done before handling the buffer to PMD, and thus before DMA.

Are you looking for functionality of "dmadev" library?

	https://doc.dpdk.org/guides/prog_guide/dmadev.html

^ permalink raw reply	[flat|nested] 7+ messages in thread

* RE: DMA Transfers to PCIe Bar Memory
  2024-11-07  9:42     ` Dmitry Kozlyuk
@ 2024-11-07 13:09       ` Wieckowski, Jacob
  2024-11-07 16:13         ` Stephen Hemminger
  2024-11-07 16:11       ` Stephen Hemminger
  1 sibling, 1 reply; 7+ messages in thread
From: Wieckowski, Jacob @ 2024-11-07 13:09 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: users

Hi Jacob,

2024-11-07 11:50 (UTC+0000), Wieckowski, Jacob:
> Hi Dimitry,
> 
> I am sorry, I was a bit too far ahead with my thoughts.
> 
> We have started a DPDK evaluation project to build knowledge and understanding of the DPDK framework. 
> We used an E1000 driver with a workaround to gain access to the PCIe 5.0 R-Tile on the Intel Agilex7 FPGA under Windows. 
> 
> Accesses to the PCIe BAR work in principle, but we can currently only carry out 64-bit accesses to the BAR memory.  
> In the PCIe Config Space in the Capabilities Register, a maximum payload size of 512 bytes is configured. 
> The Intel Core in the FPGA and the Root Complex also support TLPs of this length. 
> 
> We use the rte_read32 and rte_write32 functions to access the bar memory, which obviously executes accesses with max 2 DWs. 
> We were able to trigger this in the FPGA because only TLP with length 2 arrived on the RX Interface. 
> 
> How can a block transfer be initiated in DPDK so that TLPs with a length of 128 DWs are generated on the PCIe Bus?

Thanks, now I understand what you need.
Unfortunately DPDK has no relevant API; rte_read**() is for small registers.
Maybe there's some useful code in "drivers/raw", but this area is completely unknown to me, sorry.
Try replying with what you wrote above to the thread in users@dpdk.org and Cc: Intel people from MAINTAINERS file responsible for "drivers/raw".

-----Original Message-----
From: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> 
Sent: Thursday, November 7, 2024 10:42 AM
To: Wieckowski, Jacob <Jacob.Wieckowski@vector.com>
Cc: users@dpdk.org
Subject: Re: DMA Transfers to PCIe Bar Memory

[Sie erhalten nicht häufig E-Mails von dmitry.kozliuk@gmail.com. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]

2024-11-07 09:16 (UTC+0000), Wieckowski, Jacob:
> Hi Dimitry,
>
> thank you for the quick response.
>
> Ok, DMA in the classic sense is not possible.
>
> However, if you carry out a write transfer into the BAR memory from DPDK, then, as I understand it, this access should be divided into several small postage-compliant TLP packets with a maximum payload size as specified in config space.
>
> Can block transfers in sizes of 512 bytes be carried out with the rte memcpy? The DPDK API states that the AVX-512 memcpy parameter must be enabled for x86 platforms.
>
> Do other special precautions have to be taken in the DPDK environment to setup this kind of transfer?

Could you please start with the problem you're solving?

DPDK uses DMA internally (mainly) to transfer packet data from/to HW.
It puts physical address of the buffer, etc. to NIC queue descriptor, writes to a doorbell register, then the NIC DMA-writes/reads the buffer; PCI transfer sizes are probably selected by HW.
All of this is within PMD (userspace drivers), no API is exposed.

rte_memcpy() is intended for copy from RAM to RAM.
You can Cc: Morten Brørup <mb@smartsharesystems.com> probably, but I doubt that rte_memcpy() is specialized for DMA in any way.
The buffer may be filled with rte_mempcy() by application, but this is done before handling the buffer to PMD, and thus before DMA.

Are you looking for functionality of "dmadev" library?

        https://doc.dpdk.org/guides/prog_guide/dmadev.html

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: DMA Transfers to PCIe Bar Memory
  2024-11-07  9:42     ` Dmitry Kozlyuk
  2024-11-07 13:09       ` Wieckowski, Jacob
@ 2024-11-07 16:11       ` Stephen Hemminger
  1 sibling, 0 replies; 7+ messages in thread
From: Stephen Hemminger @ 2024-11-07 16:11 UTC (permalink / raw)
  To: Dmitry Kozlyuk; +Cc: Wieckowski, Jacob, users

On Thu, 7 Nov 2024 12:42:25 +0300
Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:

> rte_memcpy() is intended for copy from RAM to RAM.
> You can Cc: Morten Brørup <mb@smartsharesystems.com> probably,
> but I doubt that rte_memcpy() is specialized for DMA in any way.
> The buffer may be filled with rte_mempcy() by application,
> but this is done before handling the buffer to PMD, and thus before DMA.

A little more detail.
rte_memcpy() was an attempt at optimization of memcpy().
With current compilers, rte_memcpy() and memcpy() end up producing close
to the same code; the only cases left where there is a difference is likely
because GCC is more pessemistic about alignment.
The plan is to get rid of rte_memcpy() all together.

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: DMA Transfers to PCIe Bar Memory
  2024-11-07 13:09       ` Wieckowski, Jacob
@ 2024-11-07 16:13         ` Stephen Hemminger
  0 siblings, 0 replies; 7+ messages in thread
From: Stephen Hemminger @ 2024-11-07 16:13 UTC (permalink / raw)
  To: Wieckowski, Jacob; +Cc: Dmitry Kozlyuk, users

On Thu, 7 Nov 2024 13:09:41 +0000
"Wieckowski, Jacob" <Jacob.Wieckowski@vector.com> wrote:

> Hi Jacob,
> 
> 2024-11-07 11:50 (UTC+0000), Wieckowski, Jacob:
> > Hi Dimitry,
> > 
> > I am sorry, I was a bit too far ahead with my thoughts.
> > 
> > We have started a DPDK evaluation project to build knowledge and understanding of the DPDK framework. 
> > We used an E1000 driver with a workaround to gain access to the PCIe 5.0 R-Tile on the Intel Agilex7 FPGA under Windows. 
> > 
> > Accesses to the PCIe BAR work in principle, but we can currently only carry out 64-bit accesses to the BAR memory.  
> > In the PCIe Config Space in the Capabilities Register, a maximum payload size of 512 bytes is configured. 
> > The Intel Core in the FPGA and the Root Complex also support TLPs of this length. 
> > 
> > We use the rte_read32 and rte_write32 functions to access the bar memory, which obviously executes accesses with max 2 DWs. 
> > We were able to trigger this in the FPGA because only TLP with length 2 arrived on the RX Interface. 
> > 
> > How can a block transfer be initiated in DPDK so that TLPs with a length of 128 DWs are generated on the PCIe Bus?  
> 
> Thanks, now I understand what you need.
> Unfortunately DPDK has no relevant API; rte_read**() is for small registers.
> Maybe there's some useful code in "drivers/raw", but this area is completely unknown to me, sorry.
> Try replying with what you wrote above to the thread in users@dpdk.org and Cc: Intel people from MAINTAINERS file responsible for "drivers/raw".

In general, direct access to PCI by applications is discouraged.
All PCI access should be in driver code. What you describe is a bug in the E1000
driver, and should fixed there.

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2024-11-07 16:13 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-11-06 15:50 DMA Transfers to PCIe Bar Memory Wieckowski, Jacob
2024-11-06 17:46 ` Dmitry Kozlyuk
2024-11-07  9:16   ` Wieckowski, Jacob
2024-11-07  9:42     ` Dmitry Kozlyuk
2024-11-07 13:09       ` Wieckowski, Jacob
2024-11-07 16:13         ` Stephen Hemminger
2024-11-07 16:11       ` Stephen Hemminger

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).