--lcores: what it does and does not do

--lcores: what it does and does not do

[fwefew 4t4tg]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]

I've made a DPDK test program that does the following on the same machine
and for the same NIC. This is a test; it does nothing practical:

* Creates 1 RX queue then reads and pretty prints contents in a loop
* Creates 1 TX queue then sends packets to a hardcoded IP address in a loop

When I run this program I include the command line arguments
"--lcores=(0)@1,(1)@2" which is passed to 'rte_eal_init'.

This means there's a lcore identified '0' run on CPU HW core 1, and lcore
'1' run on CPU HW core 2.

As I understand it, the intent of the lcores argument is that this test
program will eventually run the RX loop as lcore 0, and the TX loop as
lcore 1 (or vice-versa).

On the other hand after all the required DPDK setup is done --- memzones,
mempools, queues, NIC devices initialized and started --- here's what DPDK
has not done:

* It hasn't started an application thread for lcore 0 or lcore 1
* DPDK doesn't know the function entry point for either loop so no loops
are running.

Which is totally fine ... DPDK isn't magic. If the application programmer
wants a RX and TX application thread pinned to some CPU, it should create
the threads, set the CPU affinity, and run the loop in the NUMA aligned
way. This is trivial to do. That is, with the required DPDK setup done, all
that's left is to do this trivial work ... and the test program is up and
running: problem solved.

The --lcores doesn't and cannot do this application work. So what is the
practical result of it? What does it do?

[-- Attachment #2: Type: text/html, Size: 2685 bytes --]

Re: --lcores: what it does and does not do

[Stephen Hemminger]

On Sat, 30 Mar 2024 12:06:15 -0400
fwefew 4t4tg <7532yahoo@gmail.com> wrote:

> I've made a DPDK test program that does the following on the same machine
> and for the same NIC. This is a test; it does nothing practical:
> 
> * Creates 1 RX queue then reads and pretty prints contents in a loop
> * Creates 1 TX queue then sends packets to a hardcoded IP address in a loop
> 
> When I run this program I include the command line arguments
> "--lcores=(0)@1,(1)@2" which is passed to 'rte_eal_init'.
> 
> This means there's a lcore identified '0' run on CPU HW core 1, and lcore
> '1' run on CPU HW core 2.
> 
> As I understand it, the intent of the lcores argument is that this test
> program will eventually run the RX loop as lcore 0, and the TX loop as
> lcore 1 (or vice-versa).
> 
> On the other hand after all the required DPDK setup is done --- memzones,
> mempools, queues, NIC devices initialized and started --- here's what DPDK
> has not done:
> 
> * It hasn't started an application thread for lcore 0 or lcore 1
> * DPDK doesn't know the function entry point for either loop so no loops
> are running.
> 
> Which is totally fine ... DPDK isn't magic. If the application programmer
> wants a RX and TX application thread pinned to some CPU, it should create
> the threads, set the CPU affinity, and run the loop in the NUMA aligned
> way. This is trivial to do. That is, with the required DPDK setup done, all
> that's left is to do this trivial work ... and the test program is up and
> running: problem solved.
> 
> The --lcores doesn't and cannot do this application work. So what is the
> practical result of it? What does it do?

Did your application every start the other threads with DPDK?
It is not recommended for applications to manage their own threads.
Possible but hard to get right.

A typical application looks like:


main() {
    // do some initialization
    rte_eal_init()

    // do more initialization that has to be done in main thread

    rte_eal_mp_remote_launch(worker_func, arg, CALL_MAIN);

    // main thread worker_func has exited, wait for others
    rte_eal_mp_wait_lcore()

    // back to single main thread
    rte_eal_cleanup()
}

Re: --lcores: what it does and does not do

[fwefew 4t4tg]

[-- Attachment #1: Type: text/plain, Size: 6085 bytes --]

Stephen, thank you for the reply. It occurred to me right after I sent
the email, the salient call was rte_eal_remote_launch which you confirmed.

For the benefit of everybody else let me provide a full explanation.

The scenario is a test program,

* Creates 1 RX queue then reads and pretty prints contents in a loop
* Creates 1 TX queue then sends packets to a hardcoded IP address in a loop

The intent is to run the RX loop on one core, and the TX loop on another
core.

Now, DPDK users may have run across the '--lcores' argument to DPDK
programs.
See here for reference documentation:

https://doc.dpdk.org/guides-2.0/prog_guide/env_abstraction_layer.html
https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html

All DPDK command line arguments are processed by the DPDK application
with "rte_eal_init" first thing in the application e.g. early in 'main()'.

Now in this context, the --lcores argument is a hint or direction to DPDK
that the program
will create one or more lcores. In DPDK an "lcore" is a thread of control
pinned to a
CPU HW core. So, for example,  the fragment "--lcores=(1)@2,(2)@4" in the
command
would tell DPDK that the program will create and run two application lcores
namely
lcore '1' on CPU HW core 2, and lcore '2' on CPU HW core 4.

Then, when all the memzone, mempool, queue, and NIC initialization is done,
the program
can parlay the lcores argument above to actually launch the application
threads pinned
to the correct as instructed like this,

  int lcore_hello(__rte_unused void *arg)
  {
    unsigned lcore_id;
    lcore_id = rte_lcore_id();
    printf("hello from core %u\n", lcore_id);
    return 0;
 }
 int main(int argc, char **argv) {
    ret = rte_eal_init(argc, argv);
    if (ret < 0)
        rte_panic("Cannot init EAL\n");

    // memzone, mempool, queue, and NIC setup not shown

    // If program run with --lcores=(1)@2,(2)@4 this loop will
    // create and run two threads lcore 1 pinned to CPU 2 and lcore 2
    // pinned to CPU 4. the output will look like:
    //     hello from core 1
    //     hello from core 2
    RTE_LCORE_FOREACH_WORKER(lcore_id) {
        rte_eal_remote_launch(lcore_hello, NULL, lcore_id);
    }
    .
    .
    .

See this program for a standalone example including shutdown:
https://doc.dpdk.org/api/examples_2helloworld_2main_8c-example.html#a5
Note the referenced program sneaks in a call to lcore_hello on the main
thread
by calling 'lcore_hello(NULL)'. And that call will print 'hello from core
0'. But
do not think that output was a side effect of RTE_LCORE_FOREACH_WORKER.
See next for details.

The other detail worth mentioning is the special case of lcore 0. lcore 0
is a synonym for the main thread e.g. the thread the DPDK program runs on
starting in main. The RTE_LCORE_FOREACH_WORKER will NOT include lcore 0
even if you reference it in the command line. The salient point is the loop
launches WORKER threads, and worker threads are for lcores > 0.

Now if you provide, say, "--lcores=(1)@2,(2)@4,(0)@6" the above program
will still launch the same two worker threads pinned to the same CPUs
just as before. The only difference is that the main thread will be pinned
to CPU core 6. You can get the same effect by running:

   # task and main thread start on CPU 6, and "--lcores=(1)@2,(2)@4"
   # will start two worker threads on CPUS 2, 4 as described above
   taskset -c6 <task>  ... "--lcores=(1)@2,(2)@4" ...

> When I run this program I include the command line arguments
> "--lcores=(0)@1,(1)@2" which is passed to 'rte_eal_init'.


On Sat, Mar 30, 2024 at 12:52 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Sat, 30 Mar 2024 12:06:15 -0400
> fwefew 4t4tg <7532yahoo@gmail.com> wrote:
>
> > I've made a DPDK test program that does the following on the same machine
> > and for the same NIC. This is a test; it does nothing practical:
> >
> > * Creates 1 RX queue then reads and pretty prints contents in a loop
> > * Creates 1 TX queue then sends packets to a hardcoded IP address in a
> loop
> >
> > When I run this program I include the command line arguments
> > "--lcores=(0)@1,(1)@2" which is passed to 'rte_eal_init'.
> >
> > This means there's a lcore identified '0' run on CPU HW core 1, and lcore
> > '1' run on CPU HW core 2.
> >
> > As I understand it, the intent of the lcores argument is that this test
> > program will eventually run the RX loop as lcore 0, and the TX loop as
> > lcore 1 (or vice-versa).
> >
> > On the other hand after all the required DPDK setup is done --- memzones,
> > mempools, queues, NIC devices initialized and started --- here's what
> DPDK
> > has not done:
> >
> > * It hasn't started an application thread for lcore 0 or lcore 1
> > * DPDK doesn't know the function entry point for either loop so no loops
> > are running.
> >
> > Which is totally fine ... DPDK isn't magic. If the application programmer
> > wants a RX and TX application thread pinned to some CPU, it should create
> > the threads, set the CPU affinity, and run the loop in the NUMA aligned
> > way. This is trivial to do. That is, with the required DPDK setup done,
> all
> > that's left is to do this trivial work ... and the test program is up and
> > running: problem solved.
> >
> > The --lcores doesn't and cannot do this application work. So what is the
> > practical result of it? What does it do?
>
> Did your application every start the other threads with DPDK?
> It is not recommended for applications to manage their own threads.
> Possible but hard to get right.
>
> A typical application looks like:
>
>
> main() {
>     // do some initialization
>     rte_eal_init()
>
>     // do more initialization that has to be done in main thread
>
>     rte_eal_mp_remote_launch(worker_func, arg, CALL_MAIN);
>
>     // main thread worker_func has exited, wait for others
>     rte_eal_mp_wait_lcore()
>
>     // back to single main thread
>     rte_eal_cleanup()
> }
>
>

[-- Attachment #2: Type: text/html, Size: 12486 bytes --]

Re: --lcores: what it does and does not do

[Stephen Hemminger]

On Sat, 30 Mar 2024 17:45:41 -0400
fwefew 4t4tg <7532yahoo@gmail.com> wrote:

>     // If program run with --lcores=(1)@2,(2)@4 this loop will
>     // create and run two threads lcore 1 pinned to CPU 2 and lcore 2
>     // pinned to CPU 4. the output will look like:
>     //     hello from core 1
>     //     hello from core 2
>     RTE_LCORE_FOREACH_WORKER(lcore_id) {
>         rte_eal_remote_launch(lcore_hello, NULL, lcore_id);
>     }
>     .

You don't need that loop, that is what rte_eal_mp_remote_launch does
with more error checking.