[dpdk-users] running multiple pthreads in the same lcore id

[dpdk-users] running multiple pthreads in the same lcore id

[longtb5]

Hi Victor, 

Have you look at the Packet Framework in DPDK? The sample application ip_pipeline seems to do exactly what you've described. 

-BL 

[dpdk-users] running multiple pthreads in the same lcore id

[Victor Huertas]

Hi all,

I have developed a simple application which implements a network bridge. It
captures packets in one port (port0), sends them into a software ring
towards a second thread which receives them and sends them out via port1.

Same implementation is performed in the direction port1->port0. So there
are 4 threads in total.

[port0]-->rx thread0 -->sw ring--> tx thread1-->[port 1]
[port0]<--rx thread2 <--sw ring<-- tx thread3<--[port 1]

The thing is that this application is a first step of a functionality
expansion by inserting additional threads (all connected with software
rings) in the middle of the path so that it performs some packet
manipulation before delivering them to the destination port:

[port0]-->rx thread0 -->sw ring--> add_thread 1-->sw ring--> add_thread
N-->sw ring-> tx thread1-->[port 1]

The total number of threads can reach hundreds (much more than number of
lcores) and I would like to assign every thread to a particular lcore, no
mattering if more than one thread is being executed by the same lcore. This
assignment would be done through an XML config file, which would be loaded
at the very beginning of the application execution.

the rte_eal_remote_launch function at rte_launch.h can only launch one
thread and attach it to one and unique lcore. So imagine that I want to
launch another thread on the same lcore. DPDK doesn't seem to allow me to
do that.

I have read that using cgroups could make this possible but I haven't found
an example application where DPDK and cgroups are used toghether.

what is your recommendation on this as expert users? Is using cgroups an
acceptable option?

Is DPDK contemplating in a future release to support natively multi pthread
on one lcore on the rte_launch.h lib?

Thanks for your attention,

-- 
Victor

Re: [dpdk-users] running multiple pthreads in the same lcore id

[Van Haaren, Harry]

> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Victor Huertas
> Sent: Friday, March 16, 2018 4:24 PM
> To: users@dpdk.org
> Subject: [dpdk-users] running multiple pthreads in the same lcore id
> 
> Hi all,

Hi Victor,

> I have developed a simple application which implements a network bridge. It
> captures packets in one port (port0), sends them into a software ring
> towards a second thread which receives them and sends them out via port1.
> 
> Same implementation is performed in the direction port1->port0. So there
> are 4 threads in total.
> 
> [port0]-->rx thread0 -->sw ring--> tx thread1-->[port 1]
> [port0]<--rx thread2 <--sw ring<-- tx thread3<--[port 1]
> 
> The thing is that this application is a first step of a functionality
> expansion by inserting additional threads (all connected with software
> rings) in the middle of the path so that it performs some packet
> manipulation before delivering them to the destination port:
> 
> [port0]-->rx thread0 -->sw ring--> add_thread 1-->sw ring--> add_thread
> N-->sw ring-> tx thread1-->[port 1]
> 
> The total number of threads can reach hundreds (much more than number of
> lcores) and I would like to assign every thread to a particular lcore, no
> mattering if more than one thread is being executed by the same lcore. This
> assignment would be done through an XML config file, which would be loaded
> at the very beginning of the application execution.

Understood.


> the rte_eal_remote_launch function at rte_launch.h can only launch one
> thread and attach it to one and unique lcore. So imagine that I want to
> launch another thread on the same lcore. DPDK doesn't seem to allow me to
> do that.

Correct, DPDK uses core pinning as this enables higher performance by reducing
overhead of context-switching between threads.


> I have read that using cgroups could make this possible but I haven't found
> an example application where DPDK and cgroups are used toghether.

I would advise to consider some form of "lightweight threads", which don't actually
perform a thread-switch, but instead run a different work function on the same
DPDK lcore.

DPDK provides infrastructure for doing this - which you might like to investigate.
The "service-cores" library enables running various functions on the same DPDK lcore.
Refer to the examples/service_cores code, and you can see different work() functions being
switched between DPDK lcores dynamically (aka, runtime movement of the work between threads).


> what is your recommendation on this as expert users? Is using cgroups an
> acceptable option?
> 
> Is DPDK contemplating in a future release to support natively multi pthread
> on one lcore on the rte_launch.h lib?

I'm not aware of any such ideas.

Hope the above helps! -Harry


> Thanks for your attention,
> 
> --
> Victor

Re: [dpdk-users] running multiple pthreads in the same lcore id

[Victor Huertas]

Thanks Harry for your quick response,

Mmmm. your recommendation on lighweight threads makes me think on
suitability of using DPDK in my case. It seems to me that "lightweight
threads" is specially recommended when you have to execute in parallel many
instances of the same working function (having the same per-packet
processing load). Am I right on this assumption?

The idea that comes to mind is to break the packet processing chaining
application into several processes making then simpler (in terms of
multi-threading) and independent packet processing feature. For example:

   - one process to only handle with packet-capture and packet tx through
   NICs,: here if I want to support RSS several threads shall be run.
   - one process to classify and conform the traffic: as many threads as
   the one run in the previous process to support pipelining.
   - one process to perform scheduling in order to distribute packet into
   queues: here DPDK scheduling qos library is not an option for me because I
   have already implemented my own WFQ and Strict priority Queue Server by my
   own being able to have hundreds and thousands of independent queues. Here
   it would be a good candidate for lightweight threading (one for each queue)
   for example.

The following diagram depicts why I mean:

[Process for Tx/Rx packets NICs] --> sw ring --> [Process for
classif/conform] -->sw ring-->[Process for QoS scheculing]
                 |

           |
                 |<------------------------------------------- sw ring
<---------------------------------------------------------------|


Each process would be assigned it own lcores id thanks to the DPDK EAL
initialization options.

Despite this partition, I will have to launch a good number of phreads, so
I am afraid that I will have to use non-EAL pthread functions. However, I
use several customized packet metadata that must be preserved between
processes (I would use the userdata pointer in the rte_mbuf metadata header
in order not to loose the association: packet-->metadata along all the
application).
The key point here is to combine two things: non-EAL pthreading launching
function and the DPDK software rings and memory pools that must be shared
by ALL the proceses mentioned before.

Do you agree?

Regards,

2018-03-16 18:11 GMT+01:00 Van Haaren, Harry <harry.van.haaren@intel.com>:

> > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Victor Huertas
> > Sent: Friday, March 16, 2018 4:24 PM
> > To: users@dpdk.org
> > Subject: [dpdk-users] running multiple pthreads in the same lcore id
> >
> > Hi all,
>
> Hi Victor,
>
> > I have developed a simple application which implements a network bridge.
> It
> > captures packets in one port (port0), sends them into a software ring
> > towards a second thread which receives them and sends them out via port1.
> >
> > Same implementation is performed in the direction port1->port0. So there
> > are 4 threads in total.
> >
> > [port0]-->rx thread0 -->sw ring--> tx thread1-->[port 1]
> > [port0]<--rx thread2 <--sw ring<-- tx thread3<--[port 1]
> >
> > The thing is that this application is a first step of a functionality
> > expansion by inserting additional threads (all connected with software
> > rings) in the middle of the path so that it performs some packet
> > manipulation before delivering them to the destination port:
> >
> > [port0]-->rx thread0 -->sw ring--> add_thread 1-->sw ring--> add_thread
> > N-->sw ring-> tx thread1-->[port 1]
> >
> > The total number of threads can reach hundreds (much more than number of
> > lcores) and I would like to assign every thread to a particular lcore, no
> > mattering if more than one thread is being executed by the same lcore.
> This
> > assignment would be done through an XML config file, which would be
> loaded
> > at the very beginning of the application execution.
>
> Understood.
>
>
> > the rte_eal_remote_launch function at rte_launch.h can only launch one
> > thread and attach it to one and unique lcore. So imagine that I want to
> > launch another thread on the same lcore. DPDK doesn't seem to allow me to
> > do that.
>
> Correct, DPDK uses core pinning as this enables higher performance by
> reducing
> overhead of context-switching between threads.
>
>
> > I have read that using cgroups could make this possible but I haven't
> found
> > an example application where DPDK and cgroups are used toghether.
>
> I would advise to consider some form of "lightweight threads", which don't
> actually
> perform a thread-switch, but instead run a different work function on the
> same
> DPDK lcore.
>
> DPDK provides infrastructure for doing this - which you might like to
> investigate.
> The "service-cores" library enables running various functions on the same
> DPDK lcore.
> Refer to the examples/service_cores code, and you can see different work()
> functions being
> switched between DPDK lcores dynamically (aka, runtime movement of the
> work between threads).
>
>
> > what is your recommendation on this as expert users? Is using cgroups an
> > acceptable option?
> >
> > Is DPDK contemplating in a future release to support natively multi
> pthread
> > on one lcore on the rte_launch.h lib?
>
> I'm not aware of any such ideas.
>
> Hope the above helps! -Harry
>
>
> > Thanks for your attention,
> >
> > --
> > Victor
>



-- 
Victor