[dpdk-users] KNI with RTE_KNI_PREEMPT_DEFAULT=n causes CPU soft-lockup

[dpdk-users] KNI with RTE_KNI_PREEMPT_DEFAULT=n causes CPU soft-lockup

[junlee]

Hi,

We are testing KNI performance and finds the schedule_timeout_interruptible
causes extra latency for packets.

We tried to set RTE_KNI_PREEMPT_DEFAULT=n to reduce the latency, but ends
up with CPU soft-lockup.

Anybody knows how to make this work?

static int
kni_thread_single(void *data)
{
struct kni_net *knet = data;
int j;
struct kni_dev *dev;

while (!kthread_should_stop()) {
down_read(&knet->kni_list_lock);
for (j = 0; j < KNI_RX_LOOP_NUM; j++) {
list_for_each_entry(dev, &knet->kni_list_head, list) {
kni_net_rx(dev);
kni_net_poll_resp(dev);
}
}
up_read(&knet->kni_list_lock);
#ifdef RTE_KNI_PREEMPT_DEFAULT
/* reschedule out for a while */
schedule_timeout_interruptible(
usecs_to_jiffies(KNI_KTHREAD_RESCHEDULE_INTERVAL));
#endif
}

return 0;
}

Thanks.

Re: [dpdk-users] KNI with RTE_KNI_PREEMPT_DEFAULT=n causes CPU soft-lockup

[junlee]

Hi Jay,

Thanks.
We have read the thread before, increasing HZ to 1000 helps, but we are
looking for latency even less than 1ms.
The DPDK version is 17.05.

The kernel preempt setting is:
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set

Seems the watchdog/x process could not get chance to run
when schedule_timeout_interruptible() is not called.

Thanks.
BR//Wendy

2017-11-27 20:36 GMT+08:00 Jay Rolette <rolette@infinite.io>:

> That shouldn't cause soft-lockups, but IIRC, there have been KNI bugs in
> that past that would. Probably worth checking whatever version of DPDK you
> are running vs. the tree for relevant fixes.
>
> Given what you are looking into, this thread may be of interest:
>
> http://dpdk.org/ml/archives/dev/2015-June/018858.html
>
> Jay
>
> On Sun, Nov 26, 2017 at 7:27 PM, junlee <lijun422@gmail.com> wrote:
>
>> Hi,
>>
>> We are testing KNI performance and finds the
>> schedule_timeout_interruptible
>> causes extra latency for packets.
>>
>> We tried to set RTE_KNI_PREEMPT_DEFAULT=n to reduce the latency, but ends
>> up with CPU soft-lockup.
>>
>> Anybody knows how to make this work?
>>
>> static int
>> kni_thread_single(void *data)
>> {
>> struct kni_net *knet = data;
>> int j;
>> struct kni_dev *dev;
>>
>> while (!kthread_should_stop()) {
>> down_read(&knet->kni_list_lock);
>> for (j = 0; j < KNI_RX_LOOP_NUM; j++) {
>> list_for_each_entry(dev, &knet->kni_list_head, list) {
>> kni_net_rx(dev);
>> kni_net_poll_resp(dev);
>> }
>> }
>> up_read(&knet->kni_list_lock);
>> #ifdef RTE_KNI_PREEMPT_DEFAULT
>> /* reschedule out for a while */
>> schedule_timeout_interruptible(
>> usecs_to_jiffies(KNI_KTHREAD_RESCHEDULE_INTERVAL));
>> #endif
>> }
>>
>> return 0;
>> }
>>
>> Thanks.
>>
>
>

Re: [dpdk-users] KNI with RTE_KNI_PREEMPT_DEFAULT=n causes CPU soft-lockup

[junlee]

We also tried to turn off the watchdog, then we got RCU stall.
INFO: rcu_sched self-detected stall on CPU

Is there any system configuration that should be made before setting
RTE_KNI_PREEMPT_DEFAULT to n?


2017-11-27 20:49 GMT+08:00 junlee <lijun422@gmail.com>:

> Hi Jay,
>
> Thanks.
> We have read the thread before, increasing HZ to 1000 helps, but we are
> looking for latency even less than 1ms.
> The DPDK version is 17.05.
>
> The kernel preempt setting is:
> CONFIG_PREEMPT_NONE=y
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT is not set
>
> Seems the watchdog/x process could not get chance to run
> when schedule_timeout_interruptible() is not called.
>
> Thanks.
> BR//Wendy
>
> 2017-11-27 20:36 GMT+08:00 Jay Rolette <rolette@infinite.io>:
>
>> That shouldn't cause soft-lockups, but IIRC, there have been KNI bugs in
>> that past that would. Probably worth checking whatever version of DPDK you
>> are running vs. the tree for relevant fixes.
>>
>> Given what you are looking into, this thread may be of interest:
>>
>> http://dpdk.org/ml/archives/dev/2015-June/018858.html
>>
>> Jay
>>
>> On Sun, Nov 26, 2017 at 7:27 PM, junlee <lijun422@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We are testing KNI performance and finds the
>>> schedule_timeout_interruptible
>>> causes extra latency for packets.
>>>
>>> We tried to set RTE_KNI_PREEMPT_DEFAULT=n to reduce the latency, but ends
>>> up with CPU soft-lockup.
>>>
>>> Anybody knows how to make this work?
>>>
>>> static int
>>> kni_thread_single(void *data)
>>> {
>>> struct kni_net *knet = data;
>>> int j;
>>> struct kni_dev *dev;
>>>
>>> while (!kthread_should_stop()) {
>>> down_read(&knet->kni_list_lock);
>>> for (j = 0; j < KNI_RX_LOOP_NUM; j++) {
>>> list_for_each_entry(dev, &knet->kni_list_head, list) {
>>> kni_net_rx(dev);
>>> kni_net_poll_resp(dev);
>>> }
>>> }
>>> up_read(&knet->kni_list_lock);
>>> #ifdef RTE_KNI_PREEMPT_DEFAULT
>>> /* reschedule out for a while */
>>> schedule_timeout_interruptible(
>>> usecs_to_jiffies(KNI_KTHREAD_RESCHEDULE_INTERVAL));
>>> #endif
>>> }
>>>
>>> return 0;
>>> }
>>>
>>> Thanks.
>>>
>>
>>
>

Re: [dpdk-users] KNI with RTE_KNI_PREEMPT_DEFAULT=n causes CPU soft-lockup

[junlee]

Hi,

We fixed like below: use cond_schedule() to check whether there are other
tasks (watchdog, rcu, etc) to run, if no, continue to handle packets.
This can make CPU busy and performance is much better.
Do you think this can be submitted?

kni_thread_single(void *data)
{
struct kni_net *knet = data;
int j;
struct kni_dev *dev;

while (!kthread_should_stop()) {
down_read(&knet->kni_list_lock);
for (j = 0; j < KNI_RX_LOOP_NUM; j++) {
list_for_each_entry(dev, &knet->kni_list_head, list) {
kni_net_rx(dev);
kni_net_poll_resp(dev);
}
}
up_read(&knet->kni_list_lock);
#ifdef RTE_KNI_PREEMPT_DEFAULT
/* reschedule out for a while */
schedule_timeout_interruptible(
usecs_to_jiffies(KNI_KTHREAD_RESCHEDULE_INTERVAL));
#else
                /* schedule only if needed*/
cond_schedule();
#endif
}

return 0;
}

Thanks.
BR//Wendy

2017-11-28 10:24 GMT+08:00 junlee <lijun422@gmail.com>:

> We also tried to turn off the watchdog, then we got RCU stall.
> INFO: rcu_sched self-detected stall on CPU
>
> Is there any system configuration that should be made before setting
> RTE_KNI_PREEMPT_DEFAULT to n?
>
>
> 2017-11-27 20:49 GMT+08:00 junlee <lijun422@gmail.com>:
>
>> Hi Jay,
>>
>> Thanks.
>> We have read the thread before, increasing HZ to 1000 helps, but we are
>> looking for latency even less than 1ms.
>> The DPDK version is 17.05.
>>
>> The kernel preempt setting is:
>> CONFIG_PREEMPT_NONE=y
>> # CONFIG_PREEMPT_VOLUNTARY is not set
>> # CONFIG_PREEMPT is not set
>>
>> Seems the watchdog/x process could not get chance to run
>> when schedule_timeout_interruptible() is not called.
>>
>> Thanks.
>> BR//Wendy
>>
>> 2017-11-27 20:36 GMT+08:00 Jay Rolette <rolette@infinite.io>:
>>
>>> That shouldn't cause soft-lockups, but IIRC, there have been KNI bugs in
>>> that past that would. Probably worth checking whatever version of DPDK you
>>> are running vs. the tree for relevant fixes.
>>>
>>> Given what you are looking into, this thread may be of interest:
>>>
>>> http://dpdk.org/ml/archives/dev/2015-June/018858.html
>>>
>>> Jay
>>>
>>> On Sun, Nov 26, 2017 at 7:27 PM, junlee <lijun422@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are testing KNI performance and finds the
>>>> schedule_timeout_interruptible
>>>> causes extra latency for packets.
>>>>
>>>> We tried to set RTE_KNI_PREEMPT_DEFAULT=n to reduce the latency, but
>>>> ends
>>>> up with CPU soft-lockup.
>>>>
>>>> Anybody knows how to make this work?
>>>>
>>>> static int
>>>> kni_thread_single(void *data)
>>>> {
>>>> struct kni_net *knet = data;
>>>> int j;
>>>> struct kni_dev *dev;
>>>>
>>>> while (!kthread_should_stop()) {
>>>> down_read(&knet->kni_list_lock);
>>>> for (j = 0; j < KNI_RX_LOOP_NUM; j++) {
>>>> list_for_each_entry(dev, &knet->kni_list_head, list) {
>>>> kni_net_rx(dev);
>>>> kni_net_poll_resp(dev);
>>>> }
>>>> }
>>>> up_read(&knet->kni_list_lock);
>>>> #ifdef RTE_KNI_PREEMPT_DEFAULT
>>>> /* reschedule out for a while */
>>>> schedule_timeout_interruptible(
>>>> usecs_to_jiffies(KNI_KTHREAD_RESCHEDULE_INTERVAL));
>>>> #endif
>>>> }
>>>>
>>>> return 0;
>>>> }
>>>>
>>>> Thanks.
>>>>
>>>
>>>
>>
>