Re: [dpdk-dev] [RFC PATCH 0/7] support multi-phtread per lcore

[Bruce Richardson <bruce.richardson@intel.com>]

On Thu, Dec 18, 2014 at 04:11:12PM +0100, Olivier MATZ wrote:
> Hi,
> 
> On 12/18/2014 03:32 PM, Bruce Richardson wrote:
> > On Thu, Dec 18, 2014 at 12:20:07PM +0000, Walukiewicz, Miroslaw wrote:
> >> I have another question regarding your patch.
> >>
> >>  Could we extend values returned by rte_lcore_id() to set them per thread (really the DPDK lcore is a pthread but started on specific core) instead of creating linear thread id. 
> >>
> >> The patch would be much simpler and will work same way. The only change would be extending rte_lcore_id when rte_pthread_create() is called. 
> >>
> >> The value __lcore_id has really an attribute __thread that means it is valid not only per CPU core but also per thread.
> >>
> >> The mempools, timers, statistics would work without any modifications in that environment.
> >>
> >>  I do not see any reason why old legacy DPDK applications would not work in that model. 
> >>
> >> Mirek
> > 
> > Definite +1 here. 
> 
> One remark though: it looks that the rte_rings (and therefore the
> rte_mempools) are designed with the assumption that the execution
> units are alone on their cores.
> 
> As explained in [1], there is a risk that a pthread is interrupted
> by the kernel at a bad moment. Therefore another thread can be
> blocked, spinning on a variable to change its value.
> 
> The same could also occurs with spinlocks which are not designed
> to wakeup another pthread when the lock is held (like pthread_locks).
> 
> And finally, having several pthreads per core implies that the
> application should be designed with large queues: if a pthread is
> not scheduled during 10ms, it represents 100K packets at 10M PPS.
> 
> I don't say it's impossible to do it, but I think it's not so
> simple :)
> 
> Regards,
> Olivier
> 
> [1] http://dpdk.org/ml/archives/dev/2013-November/000714.html

Yes, completely agree, but because there are so many potential pitfalls, I
think we should take small steps without making major changes. Hence my agreement
with Mirek's suggestion as the simplest way forward that gives us good possibilities
and flexibility without major work.

BTW: For the multi-thread per core case, I think we'll need to look at threads
making extensive use of yields to help force the context switches at proper times.

/Bruce