* lcores clarification @ 2022-10-18 22:06 Juan Pablo L. 2022-10-18 22:41 ` Stephen Hemminger 0 siblings, 1 reply; 5+ messages in thread From: Juan Pablo L. @ 2022-10-18 22:06 UTC (permalink / raw) To: users [-- Attachment #1: Type: text/plain, Size: 1343 bytes --] Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ... I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is: if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ? would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ? I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already .... A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? .. thank you very much!!!! [-- Attachment #2: Type: text/html, Size: 3703 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: lcores clarification 2022-10-18 22:06 lcores clarification Juan Pablo L. @ 2022-10-18 22:41 ` Stephen Hemminger 2022-10-18 23:04 ` Juan Pablo L. 0 siblings, 1 reply; 5+ messages in thread From: Stephen Hemminger @ 2022-10-18 22:41 UTC (permalink / raw) To: Juan Pablo L.; +Cc: users On Tue, 18 Oct 2022 22:06:04 +0000 Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote: > Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ... > > I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is: > > if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ? > > would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ? > > I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already .... > > A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? .. > > thank you very much!!!! > If you have multiple threads running on same lcore, you better not use locks or rings, or mbufs, or most of the DPDK. Because these features all work like: while (resource is busy) spin wait So if one thread has a resource and gets preempted. When another thread runs and tries to acquire the same resource; the thread will spin until scheduler decides to run the other thread. This is not unique to DPDK, the same problem is described in the pthread_spinlock man page. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: lcores clarification 2022-10-18 22:41 ` Stephen Hemminger @ 2022-10-18 23:04 ` Juan Pablo L. 2022-10-18 23:20 ` Stephen Hemminger 0 siblings, 1 reply; 5+ messages in thread From: Juan Pablo L. @ 2022-10-18 23:04 UTC (permalink / raw) To: Stephen Hemminger; +Cc: users [-- Attachment #1: Type: text/plain, Size: 3024 bytes --] Hi Stephen, thank you for your advise, I understand that, like you said the same applies to other things not only dpdk, but I was asking to run different lcores on a CPU .. I understand that a lcore is a thread (pthread in this case) running on a CPU .. I also understand that many lcores can run on same CPU( right ?) and said lcores have sperate per-lcore variables, env, etc so one lcore is a single thread running on a while(resource is busy) NOT sharing anything with the other lcores running on same CPU that is each lcore has its own ring for example waiting for packages coming from different sockets, etc ... and the lcore is NOT running threads inside or anything like that ... I have to admit that I am trying to understand this whole lcore,CPU, scheduling, etc in DPDK still ... thank you for your input! ________________________________ From: Stephen Hemminger <stephen@networkplumber.org> Sent: Tuesday, October 18, 2022 10:41 PM To: Juan Pablo L. <jpablolorenzetti@hotmail.com> Cc: users@dpdk.org <users@dpdk.org> Subject: Re: lcores clarification On Tue, 18 Oct 2022 22:06:04 +0000 Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote: > Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ... > > I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is: > > if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ? > > would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ? > > I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already .... > > A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? .. > > thank you very much!!!! > If you have multiple threads running on same lcore, you better not use locks or rings, or mbufs, or most of the DPDK. Because these features all work like: while (resource is busy) spin wait So if one thread has a resource and gets preempted. When another thread runs and tries to acquire the same resource; the thread will spin until scheduler decides to run the other thread. This is not unique to DPDK, the same problem is described in the pthread_spinlock man page. [-- Attachment #2: Type: text/html, Size: 4263 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: lcores clarification 2022-10-18 23:04 ` Juan Pablo L. @ 2022-10-18 23:20 ` Stephen Hemminger 2022-10-19 14:31 ` Juan Pablo L. 0 siblings, 1 reply; 5+ messages in thread From: Stephen Hemminger @ 2022-10-18 23:20 UTC (permalink / raw) To: Juan Pablo L.; +Cc: users On Tue, 18 Oct 2022 23:04:48 +0000 Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote: > Hi Stephen, thank you for your advise, I understand that, like you said the same applies to other things not only dpdk, but I was asking to run different lcores on a CPU .. I understand that a lcore is a thread (pthread in this case) running on a CPU .. I also understand that many lcores can run on same CPU( right ?) and said lcores have sperate per-lcore variables, env, etc so one lcore is a single thread running on a while(resource is busy) NOT sharing anything with the other lcores running on same CPU that is each lcore has its own ring for example waiting for packages coming from different sockets, etc ... and the lcore is NOT running threads inside or anything like that ... > > I have to admit that I am trying to understand this whole lcore,CPU, scheduling, etc in DPDK still ... thank you for your input! > ________________________________ > From: Stephen Hemminger <stephen@networkplumber.org> > Sent: Tuesday, October 18, 2022 10:41 PM > To: Juan Pablo L. <jpablolorenzetti@hotmail.com> > Cc: users@dpdk.org <users@dpdk.org> > Subject: Re: lcores clarification > > On Tue, 18 Oct 2022 22:06:04 +0000 > Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote: > > > Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ... > > > > I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is: > > > > if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ? > > > > would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ? > > > > I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already .... > > > > A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? .. > > > > thank you very much!!!! > > > > If you have multiple threads running on same lcore, you better not use locks or rings, > or mbufs, or most of the DPDK. Because these features all work like: > > while (resource is busy) > spin wait > > So if one thread has a resource and gets preempted. When another thread runs > and tries to acquire the same resource; the thread will spin until scheduler > decides to run the other thread. > > This is not unique to DPDK, the same problem is described in the pthread_spinlock > man page. Running multiple DPDK lcores on same cpu risks the scheduler hitting you and will hurt performance. Running a DPDK lcore with an affinity cpuset that spans multiple cpu's will not be faster. It just will cause the scheduler to move that thread among cpus. The fastest is to do the default and pin the lcore to a single cpu; and use CPU isolation so that no other process runs on that CPU. Sure you can do lots of weird and wonderful things to rearrange lcores but your just making the application unstable. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: lcores clarification 2022-10-18 23:20 ` Stephen Hemminger @ 2022-10-19 14:31 ` Juan Pablo L. 0 siblings, 0 replies; 5+ messages in thread From: Juan Pablo L. @ 2022-10-19 14:31 UTC (permalink / raw) To: Stephen Hemminger; +Cc: users [-- Attachment #1: Type: text/plain, Size: 4004 bytes --] Stephen, I understand, thank you!! ________________________________ From: Stephen Hemminger <stephen@networkplumber.org> Sent: Tuesday, October 18, 2022 11:20 PM To: Juan Pablo L. <jpablolorenzetti@hotmail.com> Cc: users@dpdk.org <users@dpdk.org> Subject: Re: lcores clarification On Tue, 18 Oct 2022 23:04:48 +0000 Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote: > Hi Stephen, thank you for your advise, I understand that, like you said the same applies to other things not only dpdk, but I was asking to run different lcores on a CPU .. I understand that a lcore is a thread (pthread in this case) running on a CPU .. I also understand that many lcores can run on same CPU( right ?) and said lcores have sperate per-lcore variables, env, etc so one lcore is a single thread running on a while(resource is busy) NOT sharing anything with the other lcores running on same CPU that is each lcore has its own ring for example waiting for packages coming from different sockets, etc ... and the lcore is NOT running threads inside or anything like that ... > > I have to admit that I am trying to understand this whole lcore,CPU, scheduling, etc in DPDK still ... thank you for your input! > ________________________________ > From: Stephen Hemminger <stephen@networkplumber.org> > Sent: Tuesday, October 18, 2022 10:41 PM > To: Juan Pablo L. <jpablolorenzetti@hotmail.com> > Cc: users@dpdk.org <users@dpdk.org> > Subject: Re: lcores clarification > > On Tue, 18 Oct 2022 22:06:04 +0000 > Juan Pablo L. <jpablolorenzetti@hotmail.com> wrote: > > > Hellos guys, I have a doubt that I have not been able to clarify with the docs, so maybe you can help me please ... > > > > I have a process that I want to run on a set of CPUs (performing the same task) but I want to run all of them at the same time, so my question is: > > > > if I configure something like "--lcores='2@(5-7)'" that will make my lcore with id 2 run simultaneously (at the same time in parallel) on CPUs 5,6,7 or it will run my lcore id 2 only one time at any given time on either CPU 5,6 or 7 ? > > > > would it be better if I configure different lcore id for the same and assign it to individual CPUs ? , e.g: "--lcores='2@5,3@6,4@7'" even though lcore id 2,3,4 are really the same job ? > > > > I think my question is just a matter of what is the best way to configure EAL to accomplish what I want, that is to run the same process on all CPU available concurrently, even though, I believe, option 2 seems to be the obvious way I just want to make sure I am not doing something that DPDK could do for me already .... > > > > A bonus question if its not too much, since lcores are really pthreads (in linux) and I have isolated the CPUs from the kernel scheduler, my guess is that if I have different lcores (performing the same or different tasks) running on the same CPU DPDK will be doing the scheduling right ? .. > > > > thank you very much!!!! > > > > If you have multiple threads running on same lcore, you better not use locks or rings, > or mbufs, or most of the DPDK. Because these features all work like: > > while (resource is busy) > spin wait > > So if one thread has a resource and gets preempted. When another thread runs > and tries to acquire the same resource; the thread will spin until scheduler > decides to run the other thread. > > This is not unique to DPDK, the same problem is described in the pthread_spinlock > man page. Running multiple DPDK lcores on same cpu risks the scheduler hitting you and will hurt performance. Running a DPDK lcore with an affinity cpuset that spans multiple cpu's will not be faster. It just will cause the scheduler to move that thread among cpus. The fastest is to do the default and pin the lcore to a single cpu; and use CPU isolation so that no other process runs on that CPU. Sure you can do lots of weird and wonderful things to rearrange lcores but your just making the application unstable. [-- Attachment #2: Type: text/html, Size: 5246 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-10-19 14:31 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-10-18 22:06 lcores clarification Juan Pablo L. 2022-10-18 22:41 ` Stephen Hemminger 2022-10-18 23:04 ` Juan Pablo L. 2022-10-18 23:20 ` Stephen Hemminger 2022-10-19 14:31 ` Juan Pablo L.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).