[dpdk-dev] Does lthread_cond_wait need a mutex?

[dpdk-dev] Does lthread_cond_wait need a mutex?

[wubenqing]

Hi~
    Reference: http://doc.dpdk.org/guides-18.05/sample_app_ug/performance_thread.html?highlight=lthread
    The L-thread subsystem provides a set of functions that are logically equivalent to the corresponding functions offered by the POSIX pthread library.
    I think there is a bug with pthread_cond_wait of lthread implement.
    Look at this code, there are two lthread:

lthread1:
    pthread_mutex_lock(mutex);                 //a1
    if (predicate == FALSE) {                        //a2
        pthread_cond_wait(cond, mutex)        //a3
    }
    pthread_mutex_unlock(mutex);            //a4

int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)
{
if (override) {
pthread_mutex_unlock(mutex); //a31
int rv = lthread_cond_wait(*(struct lthread_cond **)cond, 0); //a32

pthread_mutex_lock(mutex); //a33
return rv;
}
return _sys_pthread_funcs.f_pthread_cond_wait(cond, mutex);
}

lthread2:
    pthread_mutex_lock(mutex);                //b1
    predicate = TRUE;                                //b2
    pthread_mutex_unlock(mutex);            //b3
    pthread_cond_signal(cond);                //b4


    If the sequence is:
    a1->a2->a31->b1->b2->b3->b4->a32
    Will lthread1 sleep forever?

________________________________
吴本卿(研五 福州)

Re: [dpdk-dev] Does lthread_cond_wait need a mutex?

[Wiles, Keith]

> On Jul 17, 2018, at 10:43 PM, wubenqing@ruijie.com.cn wrote:
> 
> Hi~
>    Reference: http://doc.dpdk.org/guides-18.05/sample_app_ug/performance_thread.html?highlight=lthread
>    The L-thread subsystem provides a set of functions that are logically equivalent to the corresponding functions offered by the POSIX pthread library.
>    I think there is a bug with pthread_cond_wait of lthread implement.
>    Look at this code, there are two lthread:
> 
> lthread1:
>    pthread_mutex_lock(mutex);                 //a1
>    if (predicate == FALSE) {                        //a2
>        pthread_cond_wait(cond, mutex)        //a3
>    }
>    pthread_mutex_unlock(mutex);            //a4
> 
> int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)
> {
> if (override) {
> pthread_mutex_unlock(mutex); //a31
> int rv = lthread_cond_wait(*(struct lthread_cond **)cond, 0); //a32
> 
> pthread_mutex_lock(mutex); //a33
> return rv;
> }
> return _sys_pthread_funcs.f_pthread_cond_wait(cond, mutex);
> }
> 
> lthread2:
>    pthread_mutex_lock(mutex);                //b1
>    predicate = TRUE;                                //b2
>    pthread_mutex_unlock(mutex);            //b3
>    pthread_cond_signal(cond);                //b4
> 
> 
>    If the sequence is:
>    a1->a2->a31->b1->b2->b3->b4->a32
>    Will lthread1 sleep forever?

Maybe is it possible, my brain is not working this morning. Please remember that lthreads must give up control or lthread will continue to and can not be preempted.

Does that fix the problem?

> 
> ________________________________
> 吴本卿(研五 福州)

Regards,
Keith

Re: [dpdk-dev] Does lthread_cond_wait need a mutex?

[wubenqing]

Hi~

If the lthreads run on different lcores, a race condition with lthread_mutex may occur.
Like this:
    lthread1 run on lcore=1
    lthread2 run on lcore=2
If the mutex is owned by lthread2, lthread1 try to lock the mutex that will block thread1. lthread_sched on lcore1 will insert lthread1 to the blocked queue of the mutex. lthread1 blocks until lthread2 unlock the mutex.
Is that right?

Let's go back to the previous question.

Refer to http://pubs.opengroup.org/onlinepubs/7908799/xsh/pthread_cond_wait.html
"The pthread_cond_wait() and pthread_cond_timedwait() functions are used to block on a condition variable. They are called with mutex locked by the calling thread or undefined behaviour will result.

These functions atomically release mutex and cause the calling thread to block on the condition variable cond; atomically here means "atomically with respect to access by another thread to the mutex and then the condition variable"."


So I think there is a problem with pthread_cond_wait implemented by lthread. If that is the case, could lthread fix this problem?

Regards,
Wubenqing

________________________________

From: Wiles, Keith<mailto:keith.wiles@intel.com>
Date: 2018-07-18 23:01
To: 吴本卿(研五 福州)<mailto:wubenqing@ruijie.com.cn>
CC: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: Re: [dpdk-dev] Does lthread_cond_wait need a mutex?



> On Jul 17, 2018, at 10:43 PM, wubenqing@ruijie.com.cn wrote:
>
> Hi~
>    Reference: http://doc.dpdk.org/guides-18.05/sample_app_ug/performance_thread.html?highlight=lthread
>    The L-thread subsystem provides a set of functions that are logically equivalent to the corresponding functions offered by the POSIX pthread library.
>    I think there is a bug with pthread_cond_wait of lthread implement.
>    Look at this code, there are two lthread:
>
> lthread1:
>    pthread_mutex_lock(mutex);                 //a1
>    if (predicate == FALSE) {                        //a2
>        pthread_cond_wait(cond, mutex)        //a3
>    }
>    pthread_mutex_unlock(mutex);            //a4
>
> int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)
> {
> if (override) {
> pthread_mutex_unlock(mutex); //a31
> int rv = lthread_cond_wait(*(struct lthread_cond **)cond, 0); //a32
>
> pthread_mutex_lock(mutex); //a33
> return rv;
> }
> return _sys_pthread_funcs.f_pthread_cond_wait(cond, mutex);
> }
>
> lthread2:
>    pthread_mutex_lock(mutex);                //b1
>    predicate = TRUE;                                //b2
>    pthread_mutex_unlock(mutex);            //b3
>    pthread_cond_signal(cond);                //b4
>
>
>    If the sequence is:
>    a1->a2->a31->b1->b2->b3->b4->a32
>    Will lthread1 sleep forever?

Maybe is it possible, my brain is not working this morning. Please remember that lthreads must give up control or lthread will continue to and can not be preempted.

Does that fix the problem?

>
> ________________________________
> 吴本卿(研五 福州)

Regards,
Keith

Re: [dpdk-dev] Does lthread_cond_wait need a mutex?

[Wiles, Keith]

> On Jul 18, 2018, at 7:34 PM, wubenqing@ruijie.com.cn wrote:
> 
> Hi~
> 
> If the lthreads run on different lcores, a race condition with lthread_mutex may occur.
> Like this:
>     lthread1 run on lcore=1
>     lthread2 run on lcore=2
> If the mutex is owned by lthread2, lthread1 try to lock the mutex that will block thread1. lthread_sched on lcore1 will insert lthread1 to the blocked queue of the mutex. lthread1 blocks until lthread2 unlock the mutex. 
> Is that right?
> 
> Let's go back to the previous question. 
> 
> Refer to http://pubs.opengroup.org/onlinepubs/7908799/xsh/pthread_cond_wait.html
> "The pthread_cond_wait() and pthread_cond_timedwait() functions are used to block on a condition variable. They are called with mutex locked by the calling thread or undefined behaviour will result.
> These functions atomically release mutex and cause the calling thread to block on the condition variable cond; atomically here means "atomically with respect to access by another thread to the mutex and then the condition variable”."

Yes, I believe you are correct. I was able to look at my version of lthreads and I had changed lthread_cond_wait() to have a mutex argument and removed the reserved variable. The same was done for timed wait function. As I remember I found a couple minor problems in lthreads, which I fixed. The problem is my lthread code is somewhat different then the code in DPDK and my changes would not apply to DPDK lthreads.

> 
> So I think there is a problem with pthread_cond_wait implemented by lthread. If that is the case, could lthread fix this problem?
> 
> Regards,
> Wubenqing
> 
>  
> From: Wiles, Keith
> Date: 2018-07-18 23:01
> To: 吴本卿(研五 福州)
> CC: dev@dpdk.org
> Subject: Re: [dpdk-dev] Does lthread_cond_wait need a mutex?
> 
> 
> > On Jul 17, 2018, at 10:43 PM, wubenqing@ruijie.com.cn wrote:
> >
> > Hi~
> >    Reference: http://doc.dpdk.org/guides-18.05/sample_app_ug/performance_thread.html?highlight=lthread
> >    The L-thread subsystem provides a set of functions that are logically equivalent to the corresponding functions offered by the POSIX pthread library.
> >    I think there is a bug with pthread_cond_wait of lthread implement.
> >    Look at this code, there are two lthread:
> >
> > lthread1:
> >    pthread_mutex_lock(mutex);                 //a1
> >    if (predicate == FALSE) {                        //a2
> >        pthread_cond_wait(cond, mutex)        //a3
> >    }
> >    pthread_mutex_unlock(mutex);            //a4
> >
> > int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)
> > {
> > if (override) {
> > pthread_mutex_unlock(mutex); //a31
> > int rv = lthread_cond_wait(*(struct lthread_cond **)cond, 0); //a32
> >
> > pthread_mutex_lock(mutex); //a33
> > return rv;
> > }
> > return _sys_pthread_funcs.f_pthread_cond_wait(cond, mutex);
> > }
> >
> > lthread2:
> >    pthread_mutex_lock(mutex);                //b1
> >    predicate = TRUE;                                //b2
> >    pthread_mutex_unlock(mutex);            //b3
> >    pthread_cond_signal(cond);                //b4
> >
> >
> >    If the sequence is:
> >    a1->a2->a31->b1->b2->b3->b4->a32
> >    Will lthread1 sleep forever?
> 
> Maybe is it possible, my brain is not working this morning. Please remember that lthreads must give up control or lthread will continue to and can not be preempted.
> 
> Does that fix the problem?
> 
> >
> > ________________________________
> > 吴本卿(研五 福州)
> 
> Regards,
> Keith

Regards,
Keith

Re: [dpdk-dev] Does lthread_cond_wait need a mutex?

[wubenqing]

Hi~
    I would be appreciate it if you could provide your lthread code for me. And I found that DPDK lthreads does not provide lthread_cond_timedwait API which I am looking for.
    Thanks.

________________________________
Regards,
Wubenqing

From: Wiles, Keith<mailto:keith.wiles@intel.com>
Date: 2018-07-19 13:21
To: 吴本卿(研五 福州)<mailto:wubenqing@ruijie.com.cn>
CC: dev@dpdk.org<mailto:dev@dpdk.org>
Subject: Re: [dpdk-dev] Does lthread_cond_wait need a mutex?



> On Jul 18, 2018, at 7:34 PM, wubenqing@ruijie.com.cn wrote:
>
> Hi~
>
> If the lthreads run on different lcores, a race condition with lthread_mutex may occur.
> Like this:
>     lthread1 run on lcore=1
>     lthread2 run on lcore=2
> If the mutex is owned by lthread2, lthread1 try to lock the mutex that will block thread1. lthread_sched on lcore1 will insert lthread1 to the blocked queue of the mutex. lthread1 blocks until lthread2 unlock the mutex.
> Is that right?
>
> Let's go back to the previous question.
>
> Refer to http://pubs.opengroup.org/onlinepubs/7908799/xsh/pthread_cond_wait.html
> "The pthread_cond_wait() and pthread_cond_timedwait() functions are used to block on a condition variable. They are called with mutex locked by the calling thread or undefined behaviour will result.
> These functions atomically release mutex and cause the calling thread to block on the condition variable cond; atomically here means "atomically with respect to access by another thread to the mutex and then the condition variable”."

Yes, I believe you are correct. I was able to look at my version of lthreads and I had changed lthread_cond_wait() to have a mutex argument and removed the reserved variable. The same was done for timed wait function. As I remember I found a couple minor problems in lthreads, which I fixed. The problem is my lthread code is somewhat different then the code in DPDK and my changes would not apply to DPDK lthreads.

>
> So I think there is a problem with pthread_cond_wait implemented by lthread. If that is the case, could lthread fix this problem?
>
> Regards,
> Wubenqing
>
>
> From: Wiles, Keith
> Date: 2018-07-18 23:01
> To: 吴本卿(研五 福州)
> CC: dev@dpdk.org
> Subject: Re: [dpdk-dev] Does lthread_cond_wait need a mutex?
>
>
> > On Jul 17, 2018, at 10:43 PM, wubenqing@ruijie.com.cn wrote:
> >
> > Hi~
> >    Reference: http://doc.dpdk.org/guides-18.05/sample_app_ug/performance_thread.html?highlight=lthread
> >    The L-thread subsystem provides a set of functions that are logically equivalent to the corresponding functions offered by the POSIX pthread library.
> >    I think there is a bug with pthread_cond_wait of lthread implement.
> >    Look at this code, there are two lthread:
> >
> > lthread1:
> >    pthread_mutex_lock(mutex);                 //a1
> >    if (predicate == FALSE) {                        //a2
> >        pthread_cond_wait(cond, mutex)        //a3
> >    }
> >    pthread_mutex_unlock(mutex);            //a4
> >
> > int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)
> > {
> > if (override) {
> > pthread_mutex_unlock(mutex); //a31
> > int rv = lthread_cond_wait(*(struct lthread_cond **)cond, 0); //a32
> >
> > pthread_mutex_lock(mutex); //a33
> > return rv;
> > }
> > return _sys_pthread_funcs.f_pthread_cond_wait(cond, mutex);
> > }
> >
> > lthread2:
> >    pthread_mutex_lock(mutex);                //b1
> >    predicate = TRUE;                                //b2
> >    pthread_mutex_unlock(mutex);            //b3
> >    pthread_cond_signal(cond);                //b4
> >
> >
> >    If the sequence is:
> >    a1->a2->a31->b1->b2->b3->b4->a32
> >    Will lthread1 sleep forever?
>
> Maybe is it possible, my brain is not working this morning. Please remember that lthreads must give up control or lthread will continue to and can not be preempted.
>
> Does that fix the problem?
>
> >
> > ________________________________
> > 吴本卿(研五 福州)
>
> Regards,
> Keith

Regards,
Keith

Re: [dpdk-dev] Does lthread_cond_wait need a mutex?

[Wiles, Keith]

> On Jul 20, 2018, at 8:32 PM, wubenqing@ruijie.com.cn wrote:
> 
> Hi~
>     I would be appreciate it if you could provide your lthread code for me. And I found that DPDK lthreads does not provide lthread_cond_timedwait API which I am looking for.
>     Thanks.

I took the lthread code and renamed it and did a number of changes, so it will not be easy to drop into the lthread code as it is not just these functions that have to change. I am very hesitant to give that code out as it is still a work in progress and may not even work correctly in all cases.

I will have a look at the code and see what I can do, but I make no promises.

> 
> Regards,
> Wubenqing
>  

Regards,
Keith

[dpdk-dev] 回复: Re: Does lthread_cond_wait need a mutex?

[wubenqing]

I'm really looking forward to it :)

________________________________
Regards,
Wubenqing

发件人： Wiles, Keith<mailto:keith.wiles@intel.com>
发送时间： 2018-07-21 21:49
收件人： 吴本卿(研五 福州)<mailto:wubenqing@ruijie.com.cn>
抄送： dev@dpdk.org<mailto:dev@dpdk.org>
主题： Re: [dpdk-dev] Does lthread_cond_wait need a mutex?



> On Jul 20, 2018, at 8:32 PM, wubenqing@ruijie.com.cn wrote:
>
> Hi~
>     I would be appreciate it if you could provide your lthread code for me. And I found that DPDK lthreads does not provide lthread_cond_timedwait API which I am looking for.
>     Thanks.

I took the lthread code and renamed it and did a number of changes, so it will not be easy to drop into the lthread code as it is not just these functions that have to change. I am very hesitant to give that code out as it is still a work in progress and may not even work correctly in all cases.

I will have a look at the code and see what I can do, but I make no promises.

>
> Regards,
> Wubenqing
>

Regards,
Keith

[dpdk-dev] 回复: 回复: Re: Does lthread_cond_wait need a mutex?

[wubenqing]

Hi~
    I would be appreciate it if you could provide your lthread code for me.  I want to deep into the code.

________________________________
Regards,
Wubenqing

发件人： wubenqing@ruijie.com.cn<mailto:wubenqing@ruijie.com.cn>
发送时间： 2018-07-22 20:59
收件人： keith.wiles@intel.com<mailto:keith.wiles@intel.com>
抄送： dev@dpdk.org<mailto:dev@dpdk.org>
主题： [dpdk-dev] 回复: Re: Does lthread_cond_wait need a mutex?

I'm really looking forward to it :)

________________________________
Regards,
Wubenqing

发件人： Wiles, Keith<mailto:keith.wiles@intel.com>
发送时间： 2018-07-21 21:49
收件人： 吴本卿(研五 福州)<mailto:wubenqing@ruijie.com.cn>
抄送： dev@dpdk.org<mailto:dev@dpdk.org>
主题： Re: [dpdk-dev] Does lthread_cond_wait need a mutex?



> On Jul 20, 2018, at 8:32 PM, wubenqing@ruijie.com.cn wrote:
>
> Hi~
>     I would be appreciate it if you could provide your lthread code for me. And I found that DPDK lthreads does not provide lthread_cond_timedwait API which I am looking for.
>     Thanks.

I took the lthread code and renamed it and did a number of changes, so it will not be easy to drop into the lthread code as it is not just these functions that have to change. I am very hesitant to give that code out as it is still a work in progress and may not even work correctly in all cases.

I will have a look at the code and see what I can do, but I make no promises.

>
> Regards,
> Wubenqing
>

Regards,
Keith