quick question about core affinity

quick question about core affinity

[이재홍]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

Hello, I'm new to DPDK

I've tried to run samples and got a query about core affinity.
As I understand, if a lcore has affinity to a CPU set, it will run only on
the CPU set.
And I thought If I run a dpdk sample with core 0-2, none process can use
the core (0-2). but when I try to run a simple app(not dpdk app) with
taskset command, it runs on 0, 1, 2 cores..

what I want was if I use cores for dpdk apps none other process can access
the cores.. but it seems possible..

I've googled to find out this but I couldn't find anything I wanted.
Is there anyone can explain about this...?


BR.
Jaehong Lee

[-- Attachment #2: Type: text/html, Size: 797 bytes --]

Re: quick question about core affinity

[Lukáš Šišmiš]

Hi,


DPDK core affinity runs your application on the selected cores. But that 
doesn't stop other applications from running on the same cores.

To get closer to your goal of really isolating the application from 
other processes you would need to add isolcpus to your boot parameters.

That instructs the scheduler to not use the mentioned cores. After 
booting with this parameter you could run your DPDK application and 
scheduler would not schedule any process to the cores that DPDK 
application would use.

However, if you run a separate application and with the taskset command 
pin it to the cores your DPDK application uses that will still run and 
will be in conflict with your DPDK app.


Best regards,

Lukas

On 26. 04. 23 13:37, 이재홍 wrote:
> Hello, I'm new to DPDK
>
> I've tried to run samples and got a query about core affinity.
> As I understand, if a lcore has affinity to a CPU set, it will run 
> only on the CPU set.
> And I thought If I run a dpdk sample with core 0-2, none process can 
> use the core (0-2). but when I try to run a simple app(not dpdk app) 
> with taskset command, it runs on 0, 1, 2 cores..
>
> what I want was if I use cores for dpdk apps none other process can 
> access the cores.. but it seems possible..
>
> I've googled to find out this but I couldn't find anything I wanted.
> Is there anyone can explain about this...?
>
>
> BR.
> Jaehong Lee

Re: quick question about core affinity

[이재홍]

[-- Attachment #1: Type: text/plain, Size: 92 bytes --]

Hello Lukas,

It was exactly what I wanted to know!
Thanks really a lot :)

BR,
Jaehong Lee

[-- Attachment #2: Type: text/html, Size: 184 bytes --]

Re: quick question about core affinity

[Jamie Fargen]

[-- Attachment #1: Type: text/plain, Size: 2525 bytes --]

Depending on the OS there maybe other ways to accomplish core isolation,
but it is documented in this page, section 7.1.3
<https://doc.dpdk.org/guides-16.04/linux_gsg/nic_perf_intel_platform.html#linux-boot-command-line>
[1].
You need to add the cores to the linux command line,

In the example you have given will need to add isolcpus=0-2 to the
linux command line.


You will likely want to inspect system cores and make sure they are on the
same numa node by ensuring that those cores are on the same 'physical core'
and you can do that by examining /proc/cpuinfo and verifying the cores have
the same 'physical id'.

You can see on this system, core 0 is on numa 0 and core 1, is on numa 1,
this would not be an optimal configuration, and instead it would be better
to use 2,4,6 or 3,5,7. You maybe asking why cores 0,1 were omitted, and
that is because some clock and timing functions of the kernel run on the
first core of the physical cpu.

$ cat /proc/cpuinfo  | grep 'processor\|physical'
processor : 0
physical id : 0
address sizes : 46 bits physical, 57 bits virtual
processor : 1
physical id : 1
address sizes : 46 bits physical, 57 bits virtual
processor : 2
physical id : 0
address sizes : 46 bits physical, 57 bits virtual
processor : 3
physical id : 1

Aother tip is you will want the cores to be on the same path as the
datapath nic. You can determine which numa node the network device is
attached to by inspecting the following file for the nic you are using for
the datapath.

# cat /sys/class/net/eno12399/device/numa_node
0



Reference Link(s):
1 -
https://doc.dpdk.org/guides-16.04/linux_gsg/nic_perf_intel_platform.html#linux-boot-command-line

On Wed, Apr 26, 2023 at 7:38 AM 이재홍 <ljh890322@gmail.com> wrote:

> Hello, I'm new to DPDK
>
> I've tried to run samples and got a query about core affinity.
> As I understand, if a lcore has affinity to a CPU set, it will run only on
> the CPU set.
> And I thought If I run a dpdk sample with core 0-2, none process can use
> the core (0-2). but when I try to run a simple app(not dpdk app) with
> taskset command, it runs on 0, 1, 2 cores..
>
> what I want was if I use cores for dpdk apps none other process can access
> the cores.. but it seems possible..
>
> I've googled to find out this but I couldn't find anything I wanted.
> Is there anyone can explain about this...?
>
>
> BR.
> Jaehong Lee
>


-- 
Jamie Fargen
Senior Engineer
jfargen@redhat.com
813-580-0718

[-- Attachment #2: Type: text/html, Size: 4211 bytes --]

Re: quick question about core affinity

[Stephen Hemminger]

On Wed, 26 Apr 2023 14:20:30 +0200
Lukáš Šišmiš <sismis@cesnet.cz> wrote:

> Hi,
> 
> 
> DPDK core affinity runs your application on the selected cores. But that 
> doesn't stop other applications from running on the same cores.
> 
> To get closer to your goal of really isolating the application from 
> other processes you would need to add isolcpus to your boot parameters.
> 
> That instructs the scheduler to not use the mentioned cores. After 
> booting with this parameter you could run your DPDK application and 
> scheduler would not schedule any process to the cores that DPDK 
> application would use.
> 
> However, if you run a separate application and with the taskset command 
> pin it to the cores your DPDK application uses that will still run and 
> will be in conflict with your DPDK app.
> 
> 
> Best regards,
> 
> Lukas
> 
> On 26. 04. 23 13:37, 이재홍 wrote:
> > Hello, I'm new to DPDK
> >
> > I've tried to run samples and got a query about core affinity.
> > As I understand, if a lcore has affinity to a CPU set, it will run 
> > only on the CPU set.
> > And I thought If I run a dpdk sample with core 0-2, none process can 
> > use the core (0-2). but when I try to run a simple app(not dpdk app) 
> > with taskset command, it runs on 0, 1, 2 cores..
> >
> > what I want was if I use cores for dpdk apps none other process can 
> > access the cores.. but it seems possible..
> >
> > I've googled to find out this but I couldn't find anything I wanted.
> > Is there anyone can explain about this...?

Look up "DPDK core isolation".

More detail in here: https://www.suse.com/c/cpu-isolation-introduction-part-1/

There are multiple ways to do this, the simplest one is to set the kernel
command line so that on boot the scheduler does not use the isolated cores.
The more complex one recommended for production is to use cgroups
and systemd.

You can't isolate CPU 0. It is special and used for system interrupts etc.
In general, don't use CPU 0 for DPDK applications.

There are other performance tuning considerations such as IRQ affinity,
nohz_full and rcu isolation that are also worth looking at.