DPDK for File Transfer and Writing

DPDK for File Transfer and Writing

[Isaac Lim]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

Dear all,
We are using DPDK to transfer files and write to SSD on the receiver side.
On the sender side, we are splitting the files into 1024-byte packets before sending over to the receiver.
Currently, on the sender side, we are using a worker lcore to retrieve the packets and storing the file data in a buffer while the main lcore writes the data to file using write().
However, we are realizing that the write speed is not fast enough to keep up with the number of packets being received.
We have tried using io_uring to aid speeding up the writing but it seems to be even slower than just writing the file normally.
We would like to check with the community if you have any DPDK compatible suggestions that would help improve file transfer and write performance?
Thank you very much in advance.
Best Regards,Isaac LIM

[-- Attachment #2: Type: text/html, Size: 1927 bytes --]

Re: DPDK for File Transfer and Writing

[Stephen Hemminger]

On Tue, 10 Jun 2025 08:49:02 +0000 (UTC)
Isaac Lim <izlim@yahoo.com> wrote:

> Dear all,
> We are using DPDK to transfer files and write to SSD on the receiver side.
> On the sender side, we are splitting the files into 1024-byte packets before sending over to the receiver.
> Currently, on the sender side, we are using a worker lcore to retrieve the packets and storing the file data in a buffer while the main lcore writes the data to file using write().
> However, we are realizing that the write speed is not fast enough to keep up with the number of packets being received.
> We have tried using io_uring to aid speeding up the writing but it seems to be even slower than just writing the file normally.
> We would like to check with the community if you have any DPDK compatible suggestions that would help improve file transfer and write performance?
> Thank you very much in advance.
> Best Regards,Isaac LIM

You could try SPDX to get faster storage, but networking is faster than storage.

At some point you need a full network stack with real protocols like TCP.

Re: DPDK for File Transfer and Writing

[Isaac Lim]

[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]

Dear Stephen,
Thank you so much.
Regarding SPDX, I did a Google search. Is it Software Package Data Exchange?
Best Regards,Isaac 

    On Tuesday, June 10, 2025 at 09:36:45 PM GMT+8, Stephen Hemminger <stephen@networkplumber.org> wrote:  
 
 On Tue, 10 Jun 2025 08:49:02 +0000 (UTC)
Isaac Lim <izlim@yahoo.com> wrote:

> Dear all,
> We are using DPDK to transfer files and write to SSD on the receiver side.
> On the sender side, we are splitting the files into 1024-byte packets before sending over to the receiver.
> Currently, on the sender side, we are using a worker lcore to retrieve the packets and storing the file data in a buffer while the main lcore writes the data to file using write().
> However, we are realizing that the write speed is not fast enough to keep up with the number of packets being received.
> We have tried using io_uring to aid speeding up the writing but it seems to be even slower than just writing the file normally.
> We would like to check with the community if you have any DPDK compatible suggestions that would help improve file transfer and write performance?
> Thank you very much in advance.
> Best Regards,Isaac LIM

You could try SPDX to get faster storage, but networking is faster than storage.

At some point you need a full network stack with real protocols like TCP.
  

[-- Attachment #2: Type: text/html, Size: 2932 bytes --]

Re: DPDK for File Transfer and Writing

[Stephen Hemminger]

On Wed, 11 Jun 2025 08:08:20 +0000 (UTC)
Isaac Lim <izlim@yahoo.com> wrote:

> Dear Stephen,
> Thank you so much.
> Regarding SPDX, I did a Google search. Is it Software Package Data Exchange?
> Best Regards,Isaac 
> 
>     On Tuesday, June 10, 2025 at 09:36:45 PM GMT+8, Stephen Hemminger <stephen@networkplumber.org> wrote:  
>  
>  On Tue, 10 Jun 2025 08:49:02 +0000 (UTC)
> Isaac Lim <izlim@yahoo.com> wrote:
> 
> > Dear all,
> > We are using DPDK to transfer files and write to SSD on the receiver side.
> > On the sender side, we are splitting the files into 1024-byte packets before sending over to the receiver.
> > Currently, on the sender side, we are using a worker lcore to retrieve the packets and storing the file data in a buffer while the main lcore writes the data to file using write().
> > However, we are realizing that the write speed is not fast enough to keep up with the number of packets being received.
> > We have tried using io_uring to aid speeding up the writing but it seems to be even slower than just writing the file normally.
> > We would like to check with the community if you have any DPDK compatible suggestions that would help improve file transfer and write performance?
> > Thank you very much in advance.
> > Best Regards,Isaac LIM  
> 
> You could try SPDX to get faster storage, but networking is faster than storage.
> 
> At some point you need a full network stack with real protocols like TCP.
>   

SPDK https://spdk.io/