* [dpdk-dev] running dpdk application on the same machine with other none dpdk application @ 2019-01-29 16:32 Shirley Avishour 2019-01-30 6:09 ` Rami Rosen 0 siblings, 1 reply; 12+ messages in thread From: Shirley Avishour @ 2019-01-29 16:32 UTC (permalink / raw) To: dev Hi, We are trying to run a dpdk based application one the same machine with other java based applications. In order to do that we isolated the dpdk application cores and disabled hyperthreading for those cores. But sporadically we encounter misses over the interface. We are running on a single socket cpu with 44 hyperthreaded cores. we are using 5 cores for the dpdk application (which is in fact 10 since hyperthreading is disabled). My guess is that probably GC (garbage collection) running from the kernel and perhaps over the isolated threads causing the misses. Is there a way to solve this issue? Is there a way to isolate cores for kernel processes/threads? Thanks in advance, Shirley. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-29 16:32 [dpdk-dev] running dpdk application on the same machine with other none dpdk application Shirley Avishour @ 2019-01-30 6:09 ` Rami Rosen 2019-01-30 6:31 ` Shirley Avishour 0 siblings, 1 reply; 12+ messages in thread From: Rami Rosen @ 2019-01-30 6:09 UTC (permalink / raw) To: Shirley Avishour; +Cc: dev Hi, Shirley >In order to do that we isolated the dpdk application cores and disabled hyperthreading for those cores. >we are using 5 cores for the dpdk application (which is in fact 10 since hyperthreading is disabled). Short clarification question: did you do it with EAL paramater ? Was it with -l CORELIST / --lcores COREMAP EAL parameter ? Did you try also the isolcpus Linux kernel parameter in order to isolate cores from the general Linux scheduler? Regards, Rami Rosen On Tue, Jan 29, 2019 at 6:32 PM Shirley Avishour <shirley@imvisiontech.com> wrote: > Hi, > We are trying to run a dpdk based application one the same machine with > other java based applications. > In order to do that we isolated the dpdk application cores and disabled > hyperthreading for those cores. > But sporadically we encounter misses over the interface. > We are running on a single socket cpu with 44 hyperthreaded cores. > we are using 5 cores for the dpdk application (which is in fact 10 since > hyperthreading is disabled). > My guess is that probably GC (garbage collection) running from the kernel > and perhaps over the isolated threads causing the misses. > Is there a way to solve this issue? Is there a way to isolate cores for > kernel processes/threads? > Thanks in advance, > Shirley. > -- regards, Rami Rosen ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 6:09 ` Rami Rosen @ 2019-01-30 6:31 ` Shirley Avishour 2019-01-30 6:57 ` Rami Rosen 0 siblings, 1 reply; 12+ messages in thread From: Shirley Avishour @ 2019-01-30 6:31 UTC (permalink / raw) To: Rami Rosen; +Cc: dev Hi Rami, In order to isolate the required core we add isolcpus to the grub: GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=2M hugepagesz=1G hugepages=8 isolcpus=1-5" To disable hyperthreading for these cores we add command to the rcl.local: (which in fact disables the peer logical core) echo 0 > /sys/devices/system/cpu/cpu23/online echo 0 > /sys/devices/system/cpu/cpu24/online echo 0 > /sys/devices/system/cpu/cpu25/online echo 0 > /sys/devices/system/cpu/cpu26/online echo 0 > /sys/devices/system/cpu/cpu27/online our dpdk application runs on cores 1-5: --lcores='(0,4-6)@5,1@1,2@2,3@3,7@4' But cpu isolation does not prevent from kernel processes/threads from running on isolated cpus. Running java based applications on the same server with a dpdk based application has an impact on the dpdk performance. Probably since the JVM generates kernel based processes. It can be either GC (garbage collection) or other operations. We are looking for a way to prevent these kernel threads from using the isolated cores. Thanks, Shirley. On Wed, Jan 30, 2019 at 8:09 AM Rami Rosen <ramirose@gmail.com> wrote: > Hi, Shirley > > >In order to do that we isolated the dpdk application cores and disabled > hyperthreading for those cores. > >we are using 5 cores for the dpdk application (which is in fact 10 since > hyperthreading is disabled). > > Short clarification question: did you do it with EAL paramater ? > > Was it with > -l CORELIST / --lcores COREMAP EAL parameter ? > > Did you try also the isolcpus Linux kernel parameter in order to isolate > cores from the general Linux scheduler? > > Regards, > Rami Rosen > > > On Tue, Jan 29, 2019 at 6:32 PM Shirley Avishour <shirley@imvisiontech.com> > wrote: > >> Hi, >> We are trying to run a dpdk based application one the same machine with >> other java based applications. >> In order to do that we isolated the dpdk application cores and disabled >> hyperthreading for those cores. >> But sporadically we encounter misses over the interface. >> We are running on a single socket cpu with 44 hyperthreaded cores. >> we are using 5 cores for the dpdk application (which is in fact 10 since >> hyperthreading is disabled). >> My guess is that probably GC (garbage collection) running from the kernel >> and perhaps over the isolated threads causing the misses. >> Is there a way to solve this issue? Is there a way to isolate cores for >> kernel processes/threads? >> Thanks in advance, >> Shirley. >> > > > -- > regards, > Rami Rosen > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 6:31 ` Shirley Avishour @ 2019-01-30 6:57 ` Rami Rosen 2019-01-30 7:05 ` Shirley Avishour 0 siblings, 1 reply; 12+ messages in thread From: Rami Rosen @ 2019-01-30 6:57 UTC (permalink / raw) To: Shirley Avishour; +Cc: dev Hi Shirley, >Running java based applications on the same server with a dpdk based application has an impact on the dpdk performance. Probably since the JVM generates kernel based processes. I This is true, but as far as I know, using isolcpus should prevent these processes to run on the isolated cores. Just to be on the safe side: did you make sure with cat /proc/cmdline on the kernel you are actually running indeed has the "isolcpus=1-5" you added in grub ? sometimes, especially in multi OS hosts, adding entries in /etc/default/grub and running grub2-mkconfig is not enough, if you boot from a different partition. Regards, Rami Rosen ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 6:57 ` Rami Rosen @ 2019-01-30 7:05 ` Shirley Avishour 2019-01-30 7:26 ` Rami Rosen 0 siblings, 1 reply; 12+ messages in thread From: Shirley Avishour @ 2019-01-30 7:05 UTC (permalink / raw) To: Rami Rosen; +Cc: dev Hi Rami, This is the printout for cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic root=UUID=6f1a210b-a30f-456d-bf16-bbb210da5666 ro default_hugepagesz=2M hugepagesz=2M hugepages=4096 isolcpus=1-5 nohz_full=1-5 rcu_nocbs=1-5 The requires cpus are in fact isolated but jvm generate some kernel threads as well and I'm afraid that these kernel space threads eventually use all cores. isolcpus is not applies on kernel space threads. Shirley. On Wed, Jan 30, 2019 at 8:57 AM Rami Rosen <ramirose@gmail.com> wrote: > Hi Shirley, > >Running java based applications on the same server with a dpdk based > application has an impact on the dpdk performance. > Probably since the JVM generates kernel based processes. I > > This is true, but as far as I know, using isolcpus should prevent these > processes to run on the isolated cores. > > Just to be on the safe side: did you make sure with cat /proc/cmdline on > the kernel you are actually running indeed > has the "isolcpus=1-5" you added in grub ? sometimes, especially in multi > OS hosts, adding entries in /etc/default/grub and running grub2-mkconfig is > not enough, if you boot from a different partition. > > Regards, > Rami Rosen > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 7:05 ` Shirley Avishour @ 2019-01-30 7:26 ` Rami Rosen 2019-01-30 7:47 ` Shirley Avishour 0 siblings, 1 reply; 12+ messages in thread From: Rami Rosen @ 2019-01-30 7:26 UTC (permalink / raw) To: Shirley Avishour; +Cc: dev Hi Shirley, I think you are right, isolcpus is for userspace threads. ... "Use the isolcpus parameter on the kernel command line to isolate certain cores from user-space tasks." ... See: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-cpu-configuration_suggestions So if there is no way to configure JVM to use specified kernel cores (and I am afraid there is no such way), I am not sure how this can be solved. Regards, Rami Rosen On Wed, Jan 30, 2019 at 9:05 AM Shirley Avishour <shirley@imvisiontech.com> wrote: > Hi Rami, > > This is the printout for cat /proc/cmdline > BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic > root=UUID=6f1a210b-a30f-456d-bf16-bbb210da5666 ro default_hugepagesz=2M > hugepagesz=2M hugepages=4096 isolcpus=1-5 nohz_full=1-5 rcu_nocbs=1-5 > > The requires cpus are in fact isolated but jvm generate some kernel > threads as well and I'm afraid that these kernel space threads eventually > use all cores. isolcpus is not applies on kernel space threads. > > Shirley. > > > > On Wed, Jan 30, 2019 at 8:57 AM Rami Rosen <ramirose@gmail.com> wrote: > >> Hi Shirley, >> >Running java based applications on the same server with a dpdk based >> application has an impact on the dpdk performance. >> Probably since the JVM generates kernel based processes. I >> >> This is true, but as far as I know, using isolcpus should prevent these >> processes to run on the isolated cores. >> >> Just to be on the safe side: did you make sure with cat /proc/cmdline on >> the kernel you are actually running indeed >> has the "isolcpus=1-5" you added in grub ? sometimes, especially in >> multi OS hosts, adding entries in /etc/default/grub and running grub2-mkconfig >> is not enough, if you boot from a different partition. >> >> Regards, >> Rami Rosen >> >> >> -- regards, Rami Rosen ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 7:26 ` Rami Rosen @ 2019-01-30 7:47 ` Shirley Avishour 2019-01-30 8:18 ` Rami Rosen 0 siblings, 1 reply; 12+ messages in thread From: Shirley Avishour @ 2019-01-30 7:47 UTC (permalink / raw) To: Rami Rosen; +Cc: dev Hi Rami, Thanks for the quick reply. So in fact there is no way to tell the kernel not to run kernel threads on specific cpus? Even not by compiling the kernel with specific flags? Do you know if the PF_NO_SETAFFINITY flag can help in any way? Thanks, Shirley. On Wed, Jan 30, 2019 at 9:26 AM Rami Rosen <ramirose@gmail.com> wrote: > Hi Shirley, > > I think you are right, isolcpus is for userspace threads. > ... > "Use the isolcpus parameter on the kernel command line to isolate certain > cores from user-space tasks." > ... > See: > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-cpu-configuration_suggestions > So if there is no way to configure JVM to use specified kernel cores (and > I am afraid there is no such way), > I am not sure how this can be solved. > > Regards, > Rami Rosen > > > On Wed, Jan 30, 2019 at 9:05 AM Shirley Avishour <shirley@imvisiontech.com> > wrote: > >> Hi Rami, >> >> This is the printout for cat /proc/cmdline >> BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic >> root=UUID=6f1a210b-a30f-456d-bf16-bbb210da5666 ro default_hugepagesz=2M >> hugepagesz=2M hugepages=4096 isolcpus=1-5 nohz_full=1-5 rcu_nocbs=1-5 >> >> The requires cpus are in fact isolated but jvm generate some kernel >> threads as well and I'm afraid that these kernel space threads eventually >> use all cores. isolcpus is not applies on kernel space threads. >> >> Shirley. >> >> >> >> On Wed, Jan 30, 2019 at 8:57 AM Rami Rosen <ramirose@gmail.com> wrote: >> >>> Hi Shirley, >>> >Running java based applications on the same server with a dpdk based >>> application has an impact on the dpdk performance. >>> Probably since the JVM generates kernel based processes. I >>> >>> This is true, but as far as I know, using isolcpus should prevent these >>> processes to run on the isolated cores. >>> >>> Just to be on the safe side: did you make sure with cat /proc/cmdline on >>> the kernel you are actually running indeed >>> has the "isolcpus=1-5" you added in grub ? sometimes, especially in >>> multi OS hosts, adding entries in /etc/default/grub and running grub2-mkconfig >>> is not enough, if you boot from a different partition. >>> >>> Regards, >>> Rami Rosen >>> >>> >>> > > -- > regards, > Rami Rosen > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 7:47 ` Shirley Avishour @ 2019-01-30 8:18 ` Rami Rosen 2019-01-30 8:27 ` Shirley Avishour 0 siblings, 1 reply; 12+ messages in thread From: Rami Rosen @ 2019-01-30 8:18 UTC (permalink / raw) To: Shirley Avishour; +Cc: dev Hi Shirley, > Do you know if the PF_NO_SETAFFINITY flag can help in any way? AFAIK, this will not work for kernel thread. For example, with 4.8.13-1 kernel: Display all kernel threads with ps aux | grep "\["n (as kernel threads are enclosed in square brackets, [ ]) And pick one of them and try: taskset -pc 7 521 pid 521's current affinity list: 0-15 taskset: failed to set pid 521's affinity: Invalid argument Regards, Rami Rosen ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 8:18 ` Rami Rosen @ 2019-01-30 8:27 ` Shirley Avishour 2019-01-30 8:30 ` Shirley Avishour 0 siblings, 1 reply; 12+ messages in thread From: Shirley Avishour @ 2019-01-30 8:27 UTC (permalink / raw) To: Rami Rosen; +Cc: dev Hi Rami, This is correct but that is because the default value of the PF_NO_SETAFFINITY flags does not allow to set affinity to kernel threads from user space. I was looking for a way to change the kernel and recompile it to be able to block kernel threads from running on the cpu isolated cores. (same as user space threads). BR, Shirley. On Wed, Jan 30, 2019 at 10:18 AM Rami Rosen <ramirose@gmail.com> wrote: > Hi Shirley, > > > Do you know if the PF_NO_SETAFFINITY flag can help in any way? > > AFAIK, this will not work for kernel thread. > > For example, with 4.8.13-1 kernel: > Display all kernel threads with > ps aux | grep "\["n > > (as kernel threads are enclosed in square brackets, [ ]) > And pick one of them and try: > taskset -pc 7 521 > pid 521's current affinity list: 0-15 > taskset: failed to set pid 521's affinity: Invalid argument > Regards, > Rami Rosen > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 8:27 ` Shirley Avishour @ 2019-01-30 8:30 ` Shirley Avishour 2019-01-30 8:40 ` Rami Rosen 0 siblings, 1 reply; 12+ messages in thread From: Shirley Avishour @ 2019-01-30 8:30 UTC (permalink / raw) To: Rami Rosen; +Cc: dev Perhaps by modifying the function set_cpus_allowed() which sets the kernel threads flag with PF_NO_SETAFFINITY. To remark it in the kernel code and to recompile the code. On Wed, Jan 30, 2019 at 10:27 AM Shirley Avishour <shirley@imvisiontech.com> wrote: > Hi Rami, > This is correct but that is because the default value of the PF_NO_SETAFFINITY > flags does not allow to set affinity to kernel threads from user space. > I was looking for a way to change the kernel and recompile it to be able > to block kernel threads from running on the cpu isolated cores. (same as > user space threads). > > BR, > Shirley. > > > > On Wed, Jan 30, 2019 at 10:18 AM Rami Rosen <ramirose@gmail.com> wrote: > >> Hi Shirley, >> >> > Do you know if the PF_NO_SETAFFINITY flag can help in any way? >> >> AFAIK, this will not work for kernel thread. >> >> For example, with 4.8.13-1 kernel: >> Display all kernel threads with >> ps aux | grep "\["n >> >> (as kernel threads are enclosed in square brackets, [ ]) >> And pick one of them and try: >> taskset -pc 7 521 >> pid 521's current affinity list: 0-15 >> taskset: failed to set pid 521's affinity: Invalid argument >> Regards, >> Rami Rosen >> >> ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 8:30 ` Shirley Avishour @ 2019-01-30 8:40 ` Rami Rosen 2019-01-30 8:45 ` Shirley Avishour 0 siblings, 1 reply; 12+ messages in thread From: Rami Rosen @ 2019-01-30 8:40 UTC (permalink / raw) To: Shirley Avishour; +Cc: dev Hi Shirley, Hi, It could be that this will work, but as I assume you know - This requires providing a patched kernel to end customers, some will not like it, some will maybe not accept it - It could be that a kernel patch to the Kernel Mailing list for enabling such functionality (globally or per kernel thread for example) via a sysf entry or a kernel parameter will be accepted, though. You may try Regards, Rami Rosen ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [dpdk-dev] running dpdk application on the same machine with other none dpdk application 2019-01-30 8:40 ` Rami Rosen @ 2019-01-30 8:45 ` Shirley Avishour 0 siblings, 0 replies; 12+ messages in thread From: Shirley Avishour @ 2019-01-30 8:45 UTC (permalink / raw) To: Rami Rosen; +Cc: dev Hi, I will consider that. Thanks again for your help. Shirley. On Wed, Jan 30, 2019 at 10:40 AM Rami Rosen <ramirose@gmail.com> wrote: > Hi Shirley, > > Hi, > It could be that this will work, but as I assume you know - > > This requires providing a patched kernel to end customers, some will not > like it, some > will maybe not accept it > > - It could be that a kernel patch to the Kernel Mailing list for enabling > such functionality (globally or per kernel thread for example) > via a sysf entry or a kernel parameter will be accepted, though. You may > try > > Regards, > Rami Rosen > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-01-30 8:45 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-01-29 16:32 [dpdk-dev] running dpdk application on the same machine with other none dpdk application Shirley Avishour 2019-01-30 6:09 ` Rami Rosen 2019-01-30 6:31 ` Shirley Avishour 2019-01-30 6:57 ` Rami Rosen 2019-01-30 7:05 ` Shirley Avishour 2019-01-30 7:26 ` Rami Rosen 2019-01-30 7:47 ` Shirley Avishour 2019-01-30 8:18 ` Rami Rosen 2019-01-30 8:27 ` Shirley Avishour 2019-01-30 8:30 ` Shirley Avishour 2019-01-30 8:40 ` Rami Rosen 2019-01-30 8:45 ` Shirley Avishour
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).