lcores clarification

lcores clarification

[Juan Pablo L.]

[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]

Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ...

I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is:

if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ?

would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ?

I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already ....

A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? ..

thank you very much!!!!


[-- Attachment #2: Type: text/html, Size: 3703 bytes --]

Re: lcores clarification

[Stephen Hemminger]

On Tue, 18 Oct 2022 22:06:04 +0000
Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote:

> Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ...
> 
> I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is:
> 
> if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ?
> 
> would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ?
> 
> I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already ....
> 
> A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? ..
> 
> thank you very much!!!!
> 

If you have multiple threads running on same lcore, you better not use locks or rings,
or mbufs, or most of the DPDK. Because these features all work like:

  while (resource is busy)
     spin wait

So if one thread has a resource and gets preempted. When another thread runs
and tries to acquire the same resource; the thread will spin until scheduler
decides to run the other thread.

This is not unique to DPDK, the same problem is described in the pthread_spinlock
man page.

Re: lcores clarification

[Juan Pablo L.]

[-- Attachment #1: Type: text/plain, Size: 3024 bytes --]

Hi Stephen, thank you for your advise, I understand that, like you said the same applies to other things not only dpdk, but I was asking to run different lcores on a CPU .. I understand that a lcore is a thread (pthread in this case) running on a CPU .. I also understand that many lcores can run on same CPU( right ?) and said lcores have sperate per-lcore variables, env, etc so one lcore is a single thread running on a while(resource is busy) NOT sharing anything with the other lcores running on same CPU that is each lcore has its own ring for example waiting for packages coming from different sockets, etc ... and the lcore is NOT running threads inside or anything like that ...

I have to admit that I am trying to understand this whole lcore,CPU, scheduling, etc in DPDK still ... thank you for your input!
________________________________
From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Tuesday, October 18, 2022 10:41 PM
To: Juan Pablo L. <jpablolorenzetti@hotmail.com>
Cc: users@dpdk.org <users@dpdk.org>
Subject: Re: lcores clarification

On Tue, 18 Oct 2022 22:06:04 +0000
Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote:

> Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ...
>
> I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is:
>
> if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ?
>
> would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ?
>
> I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already ....
>
> A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? ..
>
> thank you very much!!!!
>

If you have multiple threads running on same lcore, you better not use locks or rings,
or mbufs, or most of the DPDK. Because these features all work like:

  while (resource is busy)
     spin wait

So if one thread has a resource and gets preempted. When another thread runs
and tries to acquire the same resource; the thread will spin until scheduler
decides to run the other thread.

This is not unique to DPDK, the same problem is described in the pthread_spinlock
man page.

[-- Attachment #2: Type: text/html, Size: 4263 bytes --]

Re: lcores clarification

[Stephen Hemminger]

On Tue, 18 Oct 2022 23:04:48 +0000
Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote:

> Hi Stephen, thank you for your advise, I understand that, like you said the same applies to other things not only dpdk, but I was asking to run different lcores on a CPU .. I understand that a lcore is a thread (pthread in this case) running on a CPU .. I also understand that many lcores can run on same CPU( right ?) and said lcores have sperate per-lcore variables, env, etc so one lcore is a single thread running on a while(resource is busy) NOT sharing anything with the other lcores running on same CPU that is each lcore has its own ring for example waiting for packages coming from different sockets, etc ... and the lcore is NOT running threads inside or anything like that ...
> 
> I have to admit that I am trying to understand this whole lcore,CPU, scheduling, etc in DPDK still ... thank you for your input!
> ________________________________
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Tuesday, October 18, 2022 10:41 PM
> To: Juan Pablo L. <jpablolorenzetti@hotmail.com>
> Cc: users@dpdk.org <users@dpdk.org>
> Subject: Re: lcores clarification
> 
> On Tue, 18 Oct 2022 22:06:04 +0000
> Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote:
> 
> > Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ...
> >
> > I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is:
> >
> > if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ?
> >
> > would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ?
> >
> > I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already ....
> >
> > A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? ..
> >
> > thank you very much!!!!
> >  
> 
> If you have multiple threads running on same lcore, you better not use locks or rings,
> or mbufs, or most of the DPDK. Because these features all work like:
> 
>   while (resource is busy)
>      spin wait
> 
> So if one thread has a resource and gets preempted. When another thread runs
> and tries to acquire the same resource; the thread will spin until scheduler
> decides to run the other thread.
> 
> This is not unique to DPDK, the same problem is described in the pthread_spinlock
> man page.

Running multiple DPDK lcores on same cpu risks the scheduler hitting you and
will hurt performance.

Running a DPDK lcore with an affinity cpuset that spans multiple cpu's will
not be faster. It just will cause the scheduler to move that thread among
cpus.

The fastest is to do the default and pin the lcore to a single cpu;
and use CPU isolation so that no other process runs on that CPU.

Sure you can do lots of weird and wonderful things to rearrange lcores
but your just making the application unstable.

Re: lcores clarification

[Juan Pablo L.]

[-- Attachment #1: Type: text/plain, Size: 4004 bytes --]

Stephen, I understand, thank you!!
________________________________
From: Stephen Hemminger <stephen@networkplumber.org>
Sent: Tuesday, October 18, 2022 11:20 PM
To: Juan Pablo L. <jpablolorenzetti@hotmail.com>
Cc: users@dpdk.org <users@dpdk.org>
Subject: Re: lcores clarification

On Tue, 18 Oct 2022 23:04:48 +0000
Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote:

> Hi Stephen, thank you for your advise, I understand that, like you said the same applies to other things not only dpdk, but I was asking to run different lcores on a CPU .. I understand that a lcore is a thread (pthread in this case) running on a CPU .. I also understand that many lcores can run on same CPU( right ?) and said lcores have sperate per-lcore variables, env, etc so one lcore is a single thread running on a while(resource is busy) NOT sharing anything with the other lcores running on same CPU that is each lcore has its own ring for example waiting for packages coming from different sockets, etc ... and the lcore is NOT running threads inside or anything like that ...
>
> I have to admit that I am trying to understand this whole lcore,CPU, scheduling, etc in DPDK still ... thank you for your input!
> ________________________________
> From: Stephen Hemminger <stephen@networkplumber.org>
> Sent: Tuesday, October 18, 2022 10:41 PM
> To: Juan Pablo L. <jpablolorenzetti@hotmail.com>
> Cc: users@dpdk.org <users@dpdk.org>
> Subject: Re: lcores clarification
>
> On Tue, 18 Oct 2022 22:06:04 +0000
> Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote:
>
> > Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ...
> >
> > I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is:
> >
> > if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ?
> >
> > would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ?
> >
> > I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already ....
> >
> > A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? ..
> >
> > thank you very much!!!!
> >
>
> If you have multiple threads running on same lcore, you better not use locks or rings,
> or mbufs, or most of the DPDK. Because these features all work like:
>
>   while (resource is busy)
>      spin wait
>
> So if one thread has a resource and gets preempted. When another thread runs
> and tries to acquire the same resource; the thread will spin until scheduler
> decides to run the other thread.
>
> This is not unique to DPDK, the same problem is described in the pthread_spinlock
> man page.

Running multiple DPDK lcores on same cpu risks the scheduler hitting you and
will hurt performance.

Running a DPDK lcore with an affinity cpuset that spans multiple cpu's will
not be faster. It just will cause the scheduler to move that thread among
cpus.

The fastest is to do the default and pin the lcore to a single cpu;
and use CPU isolation so that no other process runs on that CPU.

Sure you can do lots of weird and wonderful things to rearrange lcores
but your just making the application unstable.


[-- Attachment #2: Type: text/html, Size: 5246 bytes --]