[dpdk-dev] running dpdk application on the same machine with other none dpdk application

[dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Shirley Avishour]

Hi,
We are trying to run a dpdk based application one the same machine with
other java based applications.
In order to do that we isolated the dpdk application cores and disabled
hyperthreading for those cores.
But sporadically we encounter misses over the interface.
We are running on a single socket cpu with 44 hyperthreaded cores.
we are using 5 cores for the dpdk application (which is in fact 10 since
hyperthreading is disabled).
My guess is that probably GC (garbage collection) running from the kernel
and perhaps over the isolated threads causing the misses.
Is there a way to solve this issue? Is there a way to isolate cores for
kernel processes/threads?
Thanks in advance,
Shirley.

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Rami Rosen]

Hi, Shirley

>In order to do that we isolated the dpdk application cores and disabled
hyperthreading for those cores.
>we are using 5 cores for the dpdk application (which is in fact 10 since
hyperthreading is disabled).

Short clarification question: did you do it with EAL paramater ?

Was it with
-l CORELIST / --lcores COREMAP EAL parameter ?

Did you try also the isolcpus Linux kernel parameter in order to isolate
cores from the general Linux scheduler?

Regards,
Rami Rosen


On Tue, Jan 29, 2019 at 6:32 PM Shirley Avishour <shirley@imvisiontech.com>
wrote:

> Hi,
> We are trying to run a dpdk based application one the same machine with
> other java based applications.
> In order to do that we isolated the dpdk application cores and disabled
> hyperthreading for those cores.
> But sporadically we encounter misses over the interface.
> We are running on a single socket cpu with 44 hyperthreaded cores.
> we are using 5 cores for the dpdk application (which is in fact 10 since
> hyperthreading is disabled).
> My guess is that probably GC (garbage collection) running from the kernel
> and perhaps over the isolated threads causing the misses.
> Is there a way to solve this issue? Is there a way to isolate cores for
> kernel processes/threads?
> Thanks in advance,
> Shirley.
>


-- 
regards,
Rami Rosen

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Shirley Avishour]

Hi Rami,
In order to isolate the required core we add isolcpus to the grub:
GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=2M hugepagesz=1G hugepages=8
isolcpus=1-5"

To disable hyperthreading for these cores we add command to the rcl.local:
(which in fact disables the peer logical core)
echo 0 > /sys/devices/system/cpu/cpu23/online
echo 0 > /sys/devices/system/cpu/cpu24/online
echo 0 > /sys/devices/system/cpu/cpu25/online
echo 0 > /sys/devices/system/cpu/cpu26/online
echo 0 > /sys/devices/system/cpu/cpu27/online

our dpdk application runs on cores 1-5: --lcores='(0,4-6)@5,1@1,2@2,3@3,7@4'

But cpu isolation does not prevent from kernel processes/threads from
running on isolated cpus.
Running java based applications on the same server with a dpdk based
application has an impact on the dpdk performance.
Probably since the JVM generates kernel based processes. It can be either
GC (garbage collection) or other operations.
We are looking for a way to prevent these kernel threads from using the
isolated cores.

Thanks,
Shirley.




On Wed, Jan 30, 2019 at 8:09 AM Rami Rosen <ramirose@gmail.com> wrote:

> Hi, Shirley
>
> >In order to do that we isolated the dpdk application cores and disabled
> hyperthreading for those cores.
> >we are using 5 cores for the dpdk application (which is in fact 10 since
> hyperthreading is disabled).
>
> Short clarification question: did you do it with EAL paramater ?
>
> Was it with
> -l CORELIST / --lcores COREMAP EAL parameter ?
>
> Did you try also the isolcpus Linux kernel parameter in order to isolate
> cores from the general Linux scheduler?
>
> Regards,
> Rami Rosen
>
>
> On Tue, Jan 29, 2019 at 6:32 PM Shirley Avishour <shirley@imvisiontech.com>
> wrote:
>
>> Hi,
>> We are trying to run a dpdk based application one the same machine with
>> other java based applications.
>> In order to do that we isolated the dpdk application cores and disabled
>> hyperthreading for those cores.
>> But sporadically we encounter misses over the interface.
>> We are running on a single socket cpu with 44 hyperthreaded cores.
>> we are using 5 cores for the dpdk application (which is in fact 10 since
>> hyperthreading is disabled).
>> My guess is that probably GC (garbage collection) running from the kernel
>> and perhaps over the isolated threads causing the misses.
>> Is there a way to solve this issue? Is there a way to isolate cores for
>> kernel processes/threads?
>> Thanks in advance,
>> Shirley.
>>
>
>
> --
> regards,
> Rami Rosen
>

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Rami Rosen]

Hi Shirley,
>Running java based applications on the same server with a dpdk based
application has an impact on the dpdk performance.
Probably since the JVM generates kernel based processes. I

This is true, but as far as I know, using isolcpus should prevent these
processes to run on the isolated cores.

Just to be on the safe side: did you make sure with cat /proc/cmdline on
the kernel you are actually running indeed
has the "isolcpus=1-5" you added in grub ? sometimes, especially in multi
OS hosts, adding entries in /etc/default/grub and running grub2-mkconfig is
not enough, if you boot from a different partition.

Regards,
Rami Rosen

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Shirley Avishour]

Hi Rami,

This is the printout for cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic
root=UUID=6f1a210b-a30f-456d-bf16-bbb210da5666 ro default_hugepagesz=2M
hugepagesz=2M hugepages=4096 isolcpus=1-5 nohz_full=1-5 rcu_nocbs=1-5

The requires cpus are in fact isolated but jvm generate some kernel threads
as well and I'm afraid that these kernel space threads eventually use all
cores. isolcpus is not applies on kernel space threads.

Shirley.



On Wed, Jan 30, 2019 at 8:57 AM Rami Rosen <ramirose@gmail.com> wrote:

> Hi Shirley,
> >Running java based applications on the same server with a dpdk based
> application has an impact on the dpdk performance.
> Probably since the JVM generates kernel based processes. I
>
> This is true, but as far as I know, using isolcpus should prevent these
> processes to run on the isolated cores.
>
> Just to be on the safe side: did you make sure with cat /proc/cmdline on
> the kernel you are actually running indeed
> has the "isolcpus=1-5" you added in grub ? sometimes, especially in multi
> OS hosts, adding entries in /etc/default/grub and running grub2-mkconfig is
> not enough, if you boot from a different partition.
>
> Regards,
> Rami Rosen
>
>
>

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Rami Rosen]

Hi Shirley,

I think you are right, isolcpus is for userspace threads.
...
"Use the isolcpus parameter on the kernel command line to isolate certain
cores from user-space tasks."
...
See:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-cpu-configuration_suggestions
So if there is no way to configure JVM to use specified kernel cores (and I
am afraid there is no such way),
I am not sure how this can be solved.

Regards,
Rami Rosen


On Wed, Jan 30, 2019 at 9:05 AM Shirley Avishour <shirley@imvisiontech.com>
wrote:

> Hi Rami,
>
> This is the printout for cat /proc/cmdline
> BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic
> root=UUID=6f1a210b-a30f-456d-bf16-bbb210da5666 ro default_hugepagesz=2M
> hugepagesz=2M hugepages=4096 isolcpus=1-5 nohz_full=1-5 rcu_nocbs=1-5
>
> The requires cpus are in fact isolated but jvm generate some kernel
> threads as well and I'm afraid that these kernel space threads eventually
> use all cores. isolcpus is not applies on kernel space threads.
>
> Shirley.
>
>
>
> On Wed, Jan 30, 2019 at 8:57 AM Rami Rosen <ramirose@gmail.com> wrote:
>
>> Hi Shirley,
>> >Running java based applications on the same server with a dpdk based
>> application has an impact on the dpdk performance.
>> Probably since the JVM generates kernel based processes. I
>>
>> This is true, but as far as I know, using isolcpus should prevent these
>> processes to run on the isolated cores.
>>
>> Just to be on the safe side: did you make sure with cat /proc/cmdline on
>> the kernel you are actually running indeed
>> has the "isolcpus=1-5" you added in grub ? sometimes, especially in
>> multi OS hosts, adding entries in /etc/default/grub and running grub2-mkconfig
>> is not enough, if you boot from a different partition.
>>
>> Regards,
>> Rami Rosen
>>
>>
>>

-- 
regards,
Rami Rosen

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Shirley Avishour]

Hi Rami,
Thanks for the quick reply. So in fact there is no way to tell the kernel
not to run kernel threads on specific cpus?
Even not by compiling the kernel with specific flags?
Do you know if the PF_NO_SETAFFINITY flag can help in any way?

Thanks,
Shirley.


On Wed, Jan 30, 2019 at 9:26 AM Rami Rosen <ramirose@gmail.com> wrote:

> Hi Shirley,
>
> I think you are right, isolcpus is for userspace threads.
> ...
> "Use the isolcpus parameter on the kernel command line to isolate certain
> cores from user-space tasks."
> ...
> See:
>
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-cpu-configuration_suggestions
> So if there is no way to configure JVM to use specified kernel cores (and
> I am afraid there is no such way),
> I am not sure how this can be solved.
>
> Regards,
> Rami Rosen
>
>
> On Wed, Jan 30, 2019 at 9:05 AM Shirley Avishour <shirley@imvisiontech.com>
> wrote:
>
>> Hi Rami,
>>
>> This is the printout for cat /proc/cmdline
>> BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic
>> root=UUID=6f1a210b-a30f-456d-bf16-bbb210da5666 ro default_hugepagesz=2M
>> hugepagesz=2M hugepages=4096 isolcpus=1-5 nohz_full=1-5 rcu_nocbs=1-5
>>
>> The requires cpus are in fact isolated but jvm generate some kernel
>> threads as well and I'm afraid that these kernel space threads eventually
>> use all cores. isolcpus is not applies on kernel space threads.
>>
>> Shirley.
>>
>>
>>
>> On Wed, Jan 30, 2019 at 8:57 AM Rami Rosen <ramirose@gmail.com> wrote:
>>
>>> Hi Shirley,
>>> >Running java based applications on the same server with a dpdk based
>>> application has an impact on the dpdk performance.
>>> Probably since the JVM generates kernel based processes. I
>>>
>>> This is true, but as far as I know, using isolcpus should prevent these
>>> processes to run on the isolated cores.
>>>
>>> Just to be on the safe side: did you make sure with cat /proc/cmdline on
>>> the kernel you are actually running indeed
>>> has the "isolcpus=1-5" you added in grub ? sometimes, especially in
>>> multi OS hosts, adding entries in /etc/default/grub and running grub2-mkconfig
>>> is not enough, if you boot from a different partition.
>>>
>>> Regards,
>>> Rami Rosen
>>>
>>>
>>>
>
> --
> regards,
> Rami Rosen
>

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Rami Rosen]

Hi Shirley,

> Do you know if the PF_NO_SETAFFINITY flag can help in any way?

AFAIK, this will not work for kernel thread.

For example, with 4.8.13-1 kernel:
Display all kernel threads with
ps aux | grep "\["n

(as kernel threads are enclosed in square brackets, [ ])
And pick one of them and try:
taskset -pc 7 521
pid 521's current affinity list: 0-15
taskset: failed to set pid 521's affinity: Invalid argument
Regards,
Rami Rosen

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Shirley Avishour]

Hi Rami,
This is correct but that is because the default value of the PF_NO_SETAFFINITY
flags does not allow to set affinity to kernel threads from user space.
I was looking for a way to change the kernel and recompile it to be able to
block kernel threads from running on the cpu isolated cores. (same as user
space threads).

BR,
Shirley.



On Wed, Jan 30, 2019 at 10:18 AM Rami Rosen <ramirose@gmail.com> wrote:

> Hi Shirley,
>
> > Do you know if the PF_NO_SETAFFINITY flag can help in any way?
>
> AFAIK, this will not work for kernel thread.
>
> For example, with 4.8.13-1 kernel:
> Display all kernel threads with
> ps aux | grep "\["n
>
> (as kernel threads are enclosed in square brackets, [ ])
> And pick one of them and try:
> taskset -pc 7 521
> pid 521's current affinity list: 0-15
> taskset: failed to set pid 521's affinity: Invalid argument
> Regards,
> Rami Rosen
>
>

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Shirley Avishour]

Perhaps by modifying the function set_cpus_allowed() which sets the kernel
threads flag with PF_NO_SETAFFINITY.
To remark it in the kernel code and to recompile the code.


On Wed, Jan 30, 2019 at 10:27 AM Shirley Avishour <shirley@imvisiontech.com>
wrote:

> Hi Rami,
> This is correct but that is because the default value of the PF_NO_SETAFFINITY
> flags does not allow to set affinity to kernel threads from user space.
> I was looking for a way to change the kernel and recompile it to be able
> to block kernel threads from running on the cpu isolated cores. (same as
> user space threads).
>
> BR,
> Shirley.
>
>
>
> On Wed, Jan 30, 2019 at 10:18 AM Rami Rosen <ramirose@gmail.com> wrote:
>
>> Hi Shirley,
>>
>> > Do you know if the PF_NO_SETAFFINITY flag can help in any way?
>>
>> AFAIK, this will not work for kernel thread.
>>
>> For example, with 4.8.13-1 kernel:
>> Display all kernel threads with
>> ps aux | grep "\["n
>>
>> (as kernel threads are enclosed in square brackets, [ ])
>> And pick one of them and try:
>> taskset -pc 7 521
>> pid 521's current affinity list: 0-15
>> taskset: failed to set pid 521's affinity: Invalid argument
>> Regards,
>> Rami Rosen
>>
>>

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Rami Rosen]

Hi Shirley,

Hi,
It could be that this will work, but as I assume you know -

This requires providing a patched kernel to end customers, some will not
like it, some
will maybe not accept it

- It could be that a kernel patch to the Kernel Mailing list for enabling
such functionality (globally or per kernel thread for example)
via a sysf entry or a kernel parameter will be accepted, though. You may try

Regards,
Rami Rosen

Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application

[Shirley Avishour]

Hi,
I will consider that.
Thanks again for your help.
Shirley.

On Wed, Jan 30, 2019 at 10:40 AM Rami Rosen <ramirose@gmail.com> wrote:

> Hi Shirley,
>
> Hi,
> It could be that this will work, but as I assume you know -
>
> This requires providing a patched kernel to end customers, some will not
> like it, some
> will maybe not accept it
>
> - It could be that a kernel patch to the Kernel Mailing list for enabling
> such functionality (globally or per kernel thread for example)
> via a sysf entry or a kernel parameter will be accepted, though. You may
> try
>
> Regards,
> Rami Rosen
>
>
>
>