[dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?

[dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?

[Gopakumar Choorakkot Edakkunni]

Hi,

I have a requirement where one of my non-EAL app threads needs to
handoff some packets to an EAL task. I was thinking of using
rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for
the rte_ring library and it doesnt look like it has any "EAL"
dependencies, but I wanted to double confirm that there are no issues
in using it that way. Dint find much yes/no info about that on the
mailers/docs. Pls let me know your thoughts.

Rgds,
Gopa.

Re: [dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?

[Gopakumar Choorakkot Edakkunni]

rte_ring_create() needs a socket-id though and seems to be allocating
core-specific memory pools for the ring ? But my non-EAL app thread is
not bound to any core, so now I am wondering if that will work.

Rgds,
Gopa.

On Wed, Jul 1, 2015 at 10:46 AM, Gopakumar Choorakkot Edakkunni
<gopakumar.c.e@gmail.com> wrote:
> Hi,
>
> I have a requirement where one of my non-EAL app threads needs to
> handoff some packets to an EAL task. I was thinking of using
> rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for
> the rte_ring library and it doesnt look like it has any "EAL"
> dependencies, but I wanted to double confirm that there are no issues
> in using it that way. Dint find much yes/no info about that on the
> mailers/docs. Pls let me know your thoughts.
>
> Rgds,
> Gopa.

Re: [dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?

[Bruce Richardson]

On Wed, Jul 01, 2015 at 10:50:49AM -0700, Gopakumar Choorakkot Edakkunni wrote:
> rte_ring_create() needs a socket-id though and seems to be allocating
> core-specific memory pools for the ring ? But my non-EAL app thread is
> not bound to any core, so now I am wondering if that will work.
> 
> Rgds,
> Gopa.

There are no core-specific elements for rte_rings, just for mempools. Yes, you
need a NUMA node ID when creating the ring, so that DPDK knows where to allocate
the memory for it. However, once that is done, the ring can safely be used from
both EAL and non-EAL threads. There is no requirement to have an lcore-id for
the thread.

/Bruce

> 
> On Wed, Jul 1, 2015 at 10:46 AM, Gopakumar Choorakkot Edakkunni
> <gopakumar.c.e@gmail.com> wrote:
> > Hi,
> >
> > I have a requirement where one of my non-EAL app threads needs to
> > handoff some packets to an EAL task. I was thinking of using
> > rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for
> > the rte_ring library and it doesnt look like it has any "EAL"
> > dependencies, but I wanted to double confirm that there are no issues
> > in using it that way. Dint find much yes/no info about that on the
> > mailers/docs. Pls let me know your thoughts.
> >
> > Rgds,
> > Gopa.

Re: [dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?

[Gopakumar Choorakkot Edakkunni]

Thanks for the clarification Bruce. But I find this link below in the
documentation which says it should not be used in cases where the
scheduling policy is SCHED_RR because guess it can lead to an endless
spin-lock-wait by the SCHED_RR thread waiting for the lower prio
thread which got pre-empted. I am assuming this is a very unlikely
situation 32 core Xeon like cpu where a low prio thread never gets
scheduled because there are always realtime threads with work to do ?
But since the warning is in bold and pretty loud :), I decided not to
use that and just do a handoff with a standard mutex-lock :(. If my
threads were not realtime (not SCHED_RR), I could have used this!

http://dpdk.org/doc/guides/prog_guide/env_abstraction_layer.html section 3.3.4.

"The “non-preemptive” constraint means: Bypassing this constraint it
may cause the 2nd pthread to spin until the 1st one is scheduled
again. Moreover, if the 1st pthread is preempted by a context that has
an higher priority, it may even cause a dead lock."


Rgds,
Gopa.

On Thu, Jul 2, 2015 at 2:20 AM, Bruce Richardson
<bruce.richardson@intel.com> wrote:
> On Wed, Jul 01, 2015 at 10:50:49AM -0700, Gopakumar Choorakkot Edakkunni wrote:
>> rte_ring_create() needs a socket-id though and seems to be allocating
>> core-specific memory pools for the ring ? But my non-EAL app thread is
>> not bound to any core, so now I am wondering if that will work.
>>
>> Rgds,
>> Gopa.
>
> There are no core-specific elements for rte_rings, just for mempools. Yes, you
> need a NUMA node ID when creating the ring, so that DPDK knows where to allocate
> the memory for it. However, once that is done, the ring can safely be used from
> both EAL and non-EAL threads. There is no requirement to have an lcore-id for
> the thread.
>
> /Bruce
>
>>
>> On Wed, Jul 1, 2015 at 10:46 AM, Gopakumar Choorakkot Edakkunni
>> <gopakumar.c.e@gmail.com> wrote:
>> > Hi,
>> >
>> > I have a requirement where one of my non-EAL app threads needs to
>> > handoff some packets to an EAL task. I was thinking of using
>> > rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for
>> > the rte_ring library and it doesnt look like it has any "EAL"
>> > dependencies, but I wanted to double confirm that there are no issues
>> > in using it that way. Dint find much yes/no info about that on the
>> > mailers/docs. Pls let me know your thoughts.
>> >
>> > Rgds,
>> > Gopa.

Re: [dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?

[Bruce Richardson]

On Fri, Jul 03, 2015 at 08:13:47PM -0700, Gopakumar Choorakkot Edakkunni wrote:
> Thanks for the clarification Bruce. But I find this link below in the
> documentation which says it should not be used in cases where the
> scheduling policy is SCHED_RR because guess it can lead to an endless
> spin-lock-wait by the SCHED_RR thread waiting for the lower prio
> thread which got pre-empted. I am assuming this is a very unlikely
> situation 32 core Xeon like cpu where a low prio thread never gets
> scheduled because there are always realtime threads with work to do ?
> But since the warning is in bold and pretty loud :), I decided not to
> use that and just do a handoff with a standard mutex-lock :(. If my
> threads were not realtime (not SCHED_RR), I could have used this!
> 
> http://dpdk.org/doc/guides/prog_guide/env_abstraction_layer.html section 3.3.4.
> 
> "The “non-preemptive” constraint means: Bypassing this constraint it
> may cause the 2nd pthread to spin until the 1st one is scheduled
> again. Moreover, if the 1st pthread is preempted by a context that has
> an higher priority, it may even cause a dead lock."
> 
> 
> Rgds,
> Gopa.

Yes, the note is indeed correct. As a general statement, rte_rings should never
be used to pass data between two threads on the same core. At worst, it can
cause deadlock, at best it can just lead to lots of wasted time as one thread
uses its entire scheduling quantum just waiting for the other thread which it is
blocking.

/Bruce
> 
> On Thu, Jul 2, 2015 at 2:20 AM, Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> > On Wed, Jul 01, 2015 at 10:50:49AM -0700, Gopakumar Choorakkot Edakkunni wrote:
> >> rte_ring_create() needs a socket-id though and seems to be allocating
> >> core-specific memory pools for the ring ? But my non-EAL app thread is
> >> not bound to any core, so now I am wondering if that will work.
> >>
> >> Rgds,
> >> Gopa.
> >
> > There are no core-specific elements for rte_rings, just for mempools. Yes, you
> > need a NUMA node ID when creating the ring, so that DPDK knows where to allocate
> > the memory for it. However, once that is done, the ring can safely be used from
> > both EAL and non-EAL threads. There is no requirement to have an lcore-id for
> > the thread.
> >
> > /Bruce
> >
> >>
> >> On Wed, Jul 1, 2015 at 10:46 AM, Gopakumar Choorakkot Edakkunni
> >> <gopakumar.c.e@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I have a requirement where one of my non-EAL app threads needs to
> >> > handoff some packets to an EAL task. I was thinking of using
> >> > rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for
> >> > the rte_ring library and it doesnt look like it has any "EAL"
> >> > dependencies, but I wanted to double confirm that there are no issues
> >> > in using it that way. Dint find much yes/no info about that on the
> >> > mailers/docs. Pls let me know your thoughts.
> >> >
> >> > Rgds,
> >> > Gopa.