[dpdk-users] What is the best threading technology when using DPDK ?

[dpdk-users] What is the best threading technology when using DPDK ?

[Antoine POLLENUS]

Hello,

I'm developing a time critical application using DPDK that require to be multithreaded. I'm wondering what threading technology I should use ?


-          Can I use the standard pthread library and if yes, is there a trade of in term of performance ?


-          I see on this page that a lthread library also exist but is kind of limited in term of functionality:
https://doc.dpdk.org/guides/sample_app_ug/performance_thread.html


-          I see also that we can launch a function on another lcore using the rte_eal_remote_launch(...)

Is there a recommendation when using DPDK to use a technology threading technology or another ?

Regards

Antoine Pollenus

Re: [dpdk-users] What is the best threading technology when using DPDK ?

[Van Haaren, Harry]

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Antoine POLLENUS
> Sent: Monday, July 1, 2019 2:20 PM
> To: users@dpdk.org
> Subject: [dpdk-users] What is the best threading technology when using DPDK
> ?
> 
> Hello,

Hi Antoine,


> I'm developing a time critical application using DPDK that require to be
> multithreaded. I'm wondering what threading technology I should use ?
> 
> -          Can I use the standard pthread library and if yes, is there a
> trade of in term of performance ?
> 
> -          I see on this page that a lthread library also exist but is kind
> of limited in term of functionality:
> https://doc.dpdk.org/guides/sample_app_ug/performance_thread.html
> 
> -          I see also that we can launch a function on another lcore using
> the rte_eal_remote_launch(...)
> 
> Is there a recommendation when using DPDK to use a technology threading
> technology or another ?

Good questions to ask, I'll bullet a few thoughts in reply;

- DPDK provides its own threading APIs, that depending on the platform calls the OS native implementation. For Linux this means pthreads. So by using DPDK's thread APIs, you're really using pthreads, but with a wrapper layer. This wrapper layer means that you can recompile against other targets (windows support is WIP for DPDK) and you won't have to change your threading code...

- Lthreads is a method of scheduling large numbers of work items on a lower numbers of "real" threads. Think of it as a scheduler implementation (like any OS has, to multiplex tasks to HW CPU cores). If you are running in a time-critical domain, general practice is to avoid multiplexing, and to dedicate resources just to the time critical work. In short; I suggest you run a DPDK lcore dedicated to the time-critical task, and do not use lthreads.

- The DPDK threading APIs use rte_eal_remote_launch() to "spawn" a worker thread to a given hardware thread on a CPU. (With hyper-threading, or running 2 "logical" threads on one "physical" core, this enumeration becomes a little more complex, but is still valid). DPDK uses this feature to do core-pinning, which means that a worker pthread is affinitized with a specific hardware-thread on the CPU. This stops the linux scheduler from moving the software-thread to a different CPU core/thread, which is desirable as you want to minimize jitter for time sensitive workloads. (And switching to a different CPU core/thread requires work, and hence takes time).

- For time sensitive processing, my recommendation would be to spawn a worker thread into a busy-loop for the time critical task. If possible it is best to dedicate that CPU for the task and not put any other work on that thread - this will minimize the jitter/latency.

- Investigate the "isolcpus" kernel boot parameter, and IRQ affinities if you have not already done so, to reduce jitter due to Linux scheduler and IRQ subsystem interference with the DPDK thread.


> Regards
> 
> Antoine Pollenus


Hope the above helps! Regards, -Harry

Re: [dpdk-users] What is the best threading technology when using DPDK ?

[Antoine POLLENUS]

Thanks a lot for your answer. I can now see clearly what I have to do in my implementation.

Regards

Antoine Pollenus

-----Original Message-----
From: Van Haaren, Harry [mailto:harry.van.haaren@intel.com] 
Sent: lundi 1 juillet 2019 17:39
To: Antoine POLLENUS <a.pollenus@deltacast.tv>; users@dpdk.org
Subject: RE: What is the best threading technology when using DPDK ?

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Antoine 
> POLLENUS
> Sent: Monday, July 1, 2019 2:20 PM
> To: users@dpdk.org
> Subject: [dpdk-users] What is the best threading technology when using 
> DPDK ?
> 
> Hello,

Hi Antoine,


> I'm developing a time critical application using DPDK that require to 
> be multithreaded. I'm wondering what threading technology I should use ?
> 
> -          Can I use the standard pthread library and if yes, is there a
> trade of in term of performance ?
> 
> -          I see on this page that a lthread library also exist but is kind
> of limited in term of functionality:
> https://doc.dpdk.org/guides/sample_app_ug/performance_thread.html
> 
> -          I see also that we can launch a function on another lcore using
> the rte_eal_remote_launch(...)
> 
> Is there a recommendation when using DPDK to use a technology 
> threading technology or another ?

Good questions to ask, I'll bullet a few thoughts in reply;

- DPDK provides its own threading APIs, that depending on the platform calls the OS native implementation. For Linux this means pthreads. So by using DPDK's thread APIs, you're really using pthreads, but with a wrapper layer. This wrapper layer means that you can recompile against other targets (windows support is WIP for DPDK) and you won't have to change your threading code...

- Lthreads is a method of scheduling large numbers of work items on a lower numbers of "real" threads. Think of it as a scheduler implementation (like any OS has, to multiplex tasks to HW CPU cores). If you are running in a time-critical domain, general practice is to avoid multiplexing, and to dedicate resources just to the time critical work. In short; I suggest you run a DPDK lcore dedicated to the time-critical task, and do not use lthreads.

- The DPDK threading APIs use rte_eal_remote_launch() to "spawn" a worker thread to a given hardware thread on a CPU. (With hyper-threading, or running 2 "logical" threads on one "physical" core, this enumeration becomes a little more complex, but is still valid). DPDK uses this feature to do core-pinning, which means that a worker pthread is affinitized with a specific hardware-thread on the CPU. This stops the linux scheduler from moving the software-thread to a different CPU core/thread, which is desirable as you want to minimize jitter for time sensitive workloads. (And switching to a different CPU core/thread requires work, and hence takes time).

- For time sensitive processing, my recommendation would be to spawn a worker thread into a busy-loop for the time critical task. If possible it is best to dedicate that CPU for the task and not put any other work on that thread - this will minimize the jitter/latency.

- Investigate the "isolcpus" kernel boot parameter, and IRQ affinities if you have not already done so, to reduce jitter due to Linux scheduler and IRQ subsystem interference with the DPDK thread.


> Regards
> 
> Antoine Pollenus


Hope the above helps! Regards, -Harry