CPU Affinity Not Working in RHEL8 with DPDK 20.05

CPU Affinity Not Working in RHEL8 with DPDK 20.05

[Rashi Agarwal]

[-- Attachment #1.1: Type: text/plain, Size: 1152 bytes --]

Hi DPDK Community,

I am facing an issue where CPU affinity is not working as expected in RHEL8 with DPDK 20.05, while it works fine in RHEL7.
Problem Statement:
      *  When running my program with numactl -C4,5 ./testRTE on RHEL7, the worker threads correctly bind to the assigned cores.
      *  However, on RHEL8, the worker threads always bind to core 1, ignoring the specified CPU set.

Environment Details:

  *   DPDK Version: 20.05.0
  *   OS: RHEL8 (Issue) | RHEL7 (Works Fine)
  *   CPU Affinity Tool: numactl -C
Sample Program:  Attached
Observations:

  *   In RHEL7, worker threads correctly bind to cores specified using numactl -C.
  *   In RHEL8, the threads always bind to core 1, ignoring the CPU set.
  *   Could DPDK be overriding the CPU affinity in RHEL8?
Questions:

  1.  Is there any known change in CPU affinity handling between RHEL7 and RHEL8 affecting DPDK?
  2.  Does DPDK override thread affinity when rte_eal_init() is called?
  3.  Is there any workaround or EAL flag to ensure that CPU affinity is respected?
Any insights or suggestions would be greatly appreciated.
Thanks,
Rashi Agarwal.

[-- Attachment #1.2: Type: text/html, Size: 11232 bytes --]

[-- Attachment #2: dpdk_test.cpp --]
[-- Type: text/plain, Size: 2469 bytes --]

#include <iostream>
#include "stdlib.h"
#include <fstream>
#include <string>
#include "unistd.h"
#include "time.h"
#include "math.h"
#include <pthread.h>
#include <csignal>

#include <rte_common.h>
#include <rte_eal.h>
#include <rte_ring.h>
#include <rte_ring_core.h>


using namespace std;
int keepRunning=1;

void signal_handler(int signum) 
{
    std::cout << "\nSignal (" << signum << ") received. Exiting..." << std::endl;
    keepRunning = 0;
}

void init_dpdk()
{
    /* DPDK initialization */
    int instId = 12;
    char procName[50] = {0} ;
    char filePrefix[50] = {0} ;
    sprintf(procName, "ipp-dpdk-opt-%d", instId) ;
    sprintf(filePrefix, "--file-prefix=ipp-inst-%d", instId) ;
    int dpdk_argc = 8; char *dpdk_argv[] = { procName, "-c", "0x2", "--no-huge", filePrefix ,"-m","64" , "--log-level=eal,8"};

    int ret = rte_eal_init(dpdk_argc, dpdk_argv);
	if(ret < 0)
    {
        cout << "DPDK initialization: rte_eal_init failed with code "<< ret << endl;
        return  ;
    }
	cout << "\n\n\n";

}

void init_ring()
{
	for (int i=0; i<25; i++)
	{
		int size = pow(2,i) ;
		cout << i+1 << " " << size -1 << endl; //" "<< rte_ring_get_memsize(size) << endl ;
		char name [50];
		sprintf (name,"test_ring-%d",i);
		rte_ring_create(name, size-1, 0, RING_F_EXACT_SZ);
	}
}

static void* thread_func1(void *arg)
{
	while(keepRunning)
	{
		printf("this is thread1\n");
		sleep(100);
	}
	return NULL;
}

static void* thread_func2(void *arg)
{
	while(keepRunning)
	{
		printf("this is thread2\n");
		sleep(100);
	}
	return NULL;
}

int main() {
	std::cout << "##################### Starting the test ################ " << std::endl;

    struct sigaction sig_action;
    sig_action.sa_handler = signal_handler;
    sigemptyset(&sig_action.sa_mask);
    sig_action.sa_flags = 0;
    sigaction(SIGINT, &sig_action, NULL);

	init_dpdk();

	// worker threads initializtion
    pthread_t thread_id1, thread_id2;
    pthread_attr_t thread_attr;
    pthread_attr_init(&thread_attr);

    pthread_create(&thread_id1, &thread_attr, &thread_func1,NULL);
    pthread_setname_np(thread_id1, "WorkerThread1");
    pthread_create(&thread_id2, &thread_attr, &thread_func2,NULL);
    pthread_setname_np(thread_id2, "WorkerThread2");

    while (keepRunning) {
        pause();  
    }

    std::cout << "##################### Edning the test ################ " << std::endl;
    pthread_join(thread_id1, NULL);
    pthread_join(thread_id2, NULL);


    return 0;
}

Re: CPU Affinity Not Working in RHEL8 with DPDK 20.05

[Rajesh Kumar]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

Hi Rashi,

Please change the argument to rte_eal_init (), in your code, -c 0x2 will
always bind it to 1st core . Change it to "-c",  "0x30" for pinning core 4
and 5.

int dpdk_argc = 8; char *dpdk_argv[] = { procName, "-c", "0x2",
"--no-huge", filePrefix ,"-m","64" , "--log-level=eal,8"};


Thanks,
*-Rajesh*


On Wed, Feb 26, 2025 at 10:25 AM Rashi Agarwal <Rashi.Agarwal@mobileum.com>
wrote:

> *Hi DPDK Community,*
>
> I am facing an issue where *CPU affinity is not working as expected in
> RHEL8* with DPDK 20.05, while it works fine in RHEL7.
>
> *Problem Statement:*
>
>       ·  When running my program with numactl -C4,5 ./testRTE on *RHEL7*,
> the worker threads correctly bind to the assigned cores.
>
>       ·  However, on *RHEL8*, the worker threads always bind to *core 1*,
> ignoring the specified CPU set.
>
>
>
> *Environment Details:*
>
>    - *DPDK Version:* 20.05.0
>    - *OS:* RHEL8 (Issue) | RHEL7 (Works Fine)
>    - *CPU Affinity Tool:* numactl -C
>
> *Sample Program:  *Attached
>
> *Observations:*
>
>    - In *RHEL7*, worker threads correctly bind to cores specified using numactl
>    -C.
>    - In *RHEL8*, the threads always bind to *core 1*, ignoring the CPU
>    set.
>    - Could DPDK be overriding the CPU affinity in RHEL8?
>
> *Questions:*
>
>    1. Is there any known change in CPU affinity handling between RHEL7
>    and RHEL8 affecting DPDK?
>    2. Does DPDK override thread affinity when rte_eal_init() is called?
>    3. Is there any workaround or EAL flag to ensure that CPU affinity is
>    respected?
>
> Any insights or suggestions would be greatly appreciated.
>
> Thanks,
>
> Rashi Agarwal.
>

[-- Attachment #2: Type: text/html, Size: 4346 bytes --]

RE: CPU Affinity Not Working in RHEL8 with DPDK 20.05

[Rashi Agarwal]

[-- Attachment #1: Type: text/plain, Size: 3095 bytes --]

Hi Rajesh,
Thanks for your response.
I appreciate the clarification regarding rte_eal_init() and core binding. I have some follow-up questions regarding the behaviour difference between RHEL7 and RHEL8.

  1.  Why does the same rte_eal_init() configuration behave differently on RHEL7 vs. RHEL8?
     *   In RHEL7, CPU binding works as expected, but in RHEL8, the application always binds to core 1. What could be causing this difference?
  2.  Process vs. DPDK Core Assignment:
     *   My goal is to assign DPDK to run on core 1 while the rest of the process threads are pinned using numactl.
     *   Does rte_eal_init() interfere with CPU affinity settings applied through numactl?
  3.  Effect of Not Specifying a Core in rte_eal_init():
     *   What happens if we don’t specify a core mask (-c option) during rte_eal_init()?
     *   Will DPDK inherit the core binding from numactl, or does it select cores dynamically?
  4.  Does rte_eal_init() override numactl settings?
     *   If numactl is used to pin the application to specific cores, does rte_eal_init() override that and reset core affinity?
Your insights on these questions would be very helpful. Thanks again for your time and support!
Regards,
Rashi Agarwal.

From: Rajesh Kumar <rajeshthepro@gmail.com>
Sent: Wednesday, February 26, 2025 4:27 PM
To: Rashi Agarwal <Rashi.Agarwal@mobileum.com>
Cc: users@dpdk.org
Subject: Re: CPU Affinity Not Working in RHEL8 with DPDK 20.05


 Caution :External
Hi Rashi,

Please change the argument to rte_eal_init (), in your code, -c 0x2 will always bind it to 1st core . Change it to "-c",  "0x30" for pinning core 4 and 5.


int dpdk_argc = 8; char *dpdk_argv[] = { procName, "-c", "0x2", "--no-huge", filePrefix ,"-m","64" , "--log-level=eal,8"};

Thanks,
-Rajesh


On Wed, Feb 26, 2025 at 10:25 AM Rashi Agarwal <Rashi.Agarwal@mobileum.com<mailto:Rashi.Agarwal@mobileum.com>> wrote:

Hi DPDK Community,

I am facing an issue where CPU affinity is not working as expected in RHEL8 with DPDK 20.05, while it works fine in RHEL7.
Problem Statement:
      •  When running my program with numactl -C4,5 ./testRTE on RHEL7, the worker threads correctly bind to the assigned cores.
      •  However, on RHEL8, the worker threads always bind to core 1, ignoring the specified CPU set.

Environment Details:

  *   DPDK Version: 20.05.0
  *   OS: RHEL8 (Issue) | RHEL7 (Works Fine)
  *   CPU Affinity Tool: numactl -C
Sample Program:  Attached
Observations:

  *   In RHEL7, worker threads correctly bind to cores specified using numactl -C.
  *   In RHEL8, the threads always bind to core 1, ignoring the CPU set.
  *   Could DPDK be overriding the CPU affinity in RHEL8?
Questions:

  1.  Is there any known change in CPU affinity handling between RHEL7 and RHEL8 affecting DPDK?
  2.  Does DPDK override thread affinity when rte_eal_init() is called?
  3.  Is there any workaround or EAL flag to ensure that CPU affinity is respected?
Any insights or suggestions would be greatly appreciated.
Thanks,
Rashi Agarwal.

[-- Attachment #2: Type: text/html, Size: 16816 bytes --]