Removing user threads from DPDK management

Removing user threads from DPDK management

[Nicolson Ken (ニコルソン ケン)]

Hi all,

I have an application framework that starts up two shared libraries in two separate threads. One of these libraries is mine, and in its initialisation I call rte_eal_init(). However, this has the unfortunate side-effect of making both threads run on the same CPU core, resulting in a throughput drop from an ideal 2.5Gbps to about 1Gbps. I've tried options like:

    --lcores='(0-19)@(0-19)'

Or

    taskset f start_framework other_lib.so my_dpdk_lib.so
then in my_dpdk_lib.so code:
    rte_eal_init(2, {"-c", "fff0"});

Or in my_dpdk_lib.so code

    rte_thread_unregister();

And various combinations of the above, but nothing works.

I've read the Programmer's Guide chapter 4.3, but I'm not getting anything to work. Am I missing something simple, or is there anywhere I can get a more in-depth explanation of this stuff?

Thanks,
Ken

RE: Removing user threads from DPDK management

[Nicolson Ken (ニコルソン ケン)]

PS: I should have added my debug info:

EAL: lib.eal log level changed from info to debug
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 4 on socket 0
EAL: Detected lcore 3 as core 4 on socket 0
EAL: Detected lcore 4 as core 8 on socket 0
EAL: Detected lcore 5 as core 8 on socket 0
EAL: Detected lcore 6 as core 12 on socket 0
EAL: Detected lcore 7 as core 12 on socket 0
EAL: Detected lcore 8 as core 16 on socket 0
EAL: Detected lcore 9 as core 16 on socket 0
EAL: Detected lcore 10 as core 20 on socket 0
EAL: Detected lcore 11 as core 20 on socket 0
EAL: Detected lcore 12 as core 24 on socket 0
EAL: Detected lcore 13 as core 24 on socket 0
EAL: Detected lcore 14 as core 28 on socket 0
EAL: Detected lcore 15 as core 28 on socket 0
EAL: Detected lcore 16 as core 36 on socket 0
EAL: Detected lcore 17 as core 37 on socket 0
EAL: Detected lcore 18 as core 38 on socket 0
EAL: Detected lcore 19 as core 39 on socket 0
EAL: Maximum logical cores by configuration: 128
EAL: Detected CPU lcores: 20
...
EAL: Main lcore 0 is ready (tid=7f345510b680;cpuset=[0])
EAL: lcore 1 is ready (tid=7f3440c08640;cpuset=[1])
EAL: lcore 2 is ready (tid=7f3423fff640;cpuset=[2])
EAL: lcore 4 is ready (tid=7f3422ffd640;cpuset=[4])
EAL: lcore 5 is ready (tid=7f34227fc640;cpuset=[5])
EAL: lcore 3 is ready (tid=7f34237fe640;cpuset=[3])
EAL: lcore 7 is ready (tid=7f34217fa640;cpuset=[7])
EAL: lcore 6 is ready (tid=7f3421ffb640;cpuset=[6])
EAL: lcore 9 is ready (tid=7f3407fff640;cpuset=[9])
EAL: lcore 10 is ready (tid=7f34077fe640;cpuset=[10])
EAL: lcore 11 is ready (tid=7f3406ffd640;cpuset=[11])
EAL: lcore 13 is ready (tid=7f3405ffb640;cpuset=[13])
EAL: lcore 12 is ready (tid=7f34067fc640;cpuset=[12])
EAL: lcore 8 is ready (tid=7f3420ff9640;cpuset=[8])
EAL: lcore 14 is ready (tid=7f34057fa640;cpuset=[14])
EAL: lcore 15 is ready (tid=7f3404ff9640;cpuset=[15])
EAL: lcore 18 is ready (tid=7f33f6ffd640;cpuset=[18])
EAL: lcore 16 is ready (tid=7f33f7fff640;cpuset=[16])
EAL: lcore 17 is ready (tid=7f33f77fe640;cpuset=[17])
EAL: lcore 19 is ready (tid=7f33f67fc640;cpuset=[19])

This is with  --lcores='4@(12,16)' , but I'm not seeing any grouping going on.

Thanks,
Ken

-----Original Message-----
From: Nicolson Ken (ニコルソン ケン) <ken.nicolson@jp.panasonic.com> 
Sent: Thursday, November 30, 2023 12:15 PM
To: users@dpdk.org
Subject: Removing user threads from DPDK management

Hi all,

I have an application framework that starts up two shared libraries in two separate threads. One of these libraries is mine, and in its initialisation I call rte_eal_init(). However, this has the unfortunate side-effect of making both threads run on the same CPU core, resulting in a throughput drop from an ideal 2.5Gbps to about 1Gbps. I've tried options like:

    --lcores='(0-19)@(0-19)'

Or

    taskset f start_framework other_lib.so my_dpdk_lib.so then in my_dpdk_lib.so code:
    rte_eal_init(2, {"-c", "fff0"});

Or in my_dpdk_lib.so code

    rte_thread_unregister();

And various combinations of the above, but nothing works.

I've read the Programmer's Guide chapter 4.3, but I'm not getting anything to work. Am I missing something simple, or is there anywhere I can get a more in-depth explanation of this stuff?

Thanks,
Ken

RE: Removing user threads from DPDK management

[Nicolson Ken (ニコルソン ケン)]

Please ignore this thread - my calling of rte_eal_init() forgot that argv[0] should be the program name, so instead I had argv[0]="--lcores=....".

Thanks,
Ken

-----Original Message-----
From: Nicolson Ken (ニコルソン ケン) <ken.nicolson@jp.panasonic.com> 
Sent: Thursday, November 30, 2023 12:27 PM
To: users@dpdk.org
Subject: RE: Removing user threads from DPDK management

PS: I should have added my debug info:

EAL: lib.eal log level changed from info to debug
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 4 on socket 0
EAL: Detected lcore 3 as core 4 on socket 0
EAL: Detected lcore 4 as core 8 on socket 0
EAL: Detected lcore 5 as core 8 on socket 0
EAL: Detected lcore 6 as core 12 on socket 0
EAL: Detected lcore 7 as core 12 on socket 0
EAL: Detected lcore 8 as core 16 on socket 0
EAL: Detected lcore 9 as core 16 on socket 0
EAL: Detected lcore 10 as core 20 on socket 0
EAL: Detected lcore 11 as core 20 on socket 0
EAL: Detected lcore 12 as core 24 on socket 0
EAL: Detected lcore 13 as core 24 on socket 0
EAL: Detected lcore 14 as core 28 on socket 0
EAL: Detected lcore 15 as core 28 on socket 0
EAL: Detected lcore 16 as core 36 on socket 0
EAL: Detected lcore 17 as core 37 on socket 0
EAL: Detected lcore 18 as core 38 on socket 0
EAL: Detected lcore 19 as core 39 on socket 0
EAL: Maximum logical cores by configuration: 128
EAL: Detected CPU lcores: 20
...
EAL: Main lcore 0 is ready (tid=7f345510b680;cpuset=[0])
EAL: lcore 1 is ready (tid=7f3440c08640;cpuset=[1])
EAL: lcore 2 is ready (tid=7f3423fff640;cpuset=[2])
EAL: lcore 4 is ready (tid=7f3422ffd640;cpuset=[4])
EAL: lcore 5 is ready (tid=7f34227fc640;cpuset=[5])
EAL: lcore 3 is ready (tid=7f34237fe640;cpuset=[3])
EAL: lcore 7 is ready (tid=7f34217fa640;cpuset=[7])
EAL: lcore 6 is ready (tid=7f3421ffb640;cpuset=[6])
EAL: lcore 9 is ready (tid=7f3407fff640;cpuset=[9])
EAL: lcore 10 is ready (tid=7f34077fe640;cpuset=[10])
EAL: lcore 11 is ready (tid=7f3406ffd640;cpuset=[11])
EAL: lcore 13 is ready (tid=7f3405ffb640;cpuset=[13])
EAL: lcore 12 is ready (tid=7f34067fc640;cpuset=[12])
EAL: lcore 8 is ready (tid=7f3420ff9640;cpuset=[8])
EAL: lcore 14 is ready (tid=7f34057fa640;cpuset=[14])
EAL: lcore 15 is ready (tid=7f3404ff9640;cpuset=[15])
EAL: lcore 18 is ready (tid=7f33f6ffd640;cpuset=[18])
EAL: lcore 16 is ready (tid=7f33f7fff640;cpuset=[16])
EAL: lcore 17 is ready (tid=7f33f77fe640;cpuset=[17])
EAL: lcore 19 is ready (tid=7f33f67fc640;cpuset=[19])

This is with  --lcores='4@(12,16)' , but I'm not seeing any grouping going on.

Thanks,
Ken

-----Original Message-----
From: Nicolson Ken (ニコルソン ケン) <ken.nicolson@jp.panasonic.com> 
Sent: Thursday, November 30, 2023 12:15 PM
To: users@dpdk.org
Subject: Removing user threads from DPDK management

Hi all,

I have an application framework that starts up two shared libraries in two separate threads. One of these libraries is mine, and in its initialisation I call rte_eal_init(). However, this has the unfortunate side-effect of making both threads run on the same CPU core, resulting in a throughput drop from an ideal 2.5Gbps to about 1Gbps. I've tried options like:

    --lcores='(0-19)@(0-19)'

Or

    taskset f start_framework other_lib.so my_dpdk_lib.so then in my_dpdk_lib.so code:
    rte_eal_init(2, {"-c", "fff0"});

Or in my_dpdk_lib.so code

    rte_thread_unregister();

And various combinations of the above, but nothing works.

I've read the Programmer's Guide chapter 4.3, but I'm not getting anything to work. Am I missing something simple, or is there anywhere I can get a more in-depth explanation of this stuff?

Thanks,
Ken