Stephen, thank you for the reply. It occurred to me right after I sent
the email, the salient call was rte_eal_remote_launch which you confirmed.
For the benefit of everybody else let me provide a full explanation.
The scenario is a test program,
* Creates 1 RX queue then reads and pretty prints contents in a loop
* Creates 1 TX queue then sends packets to a hardcoded IP address in a loop
The intent is to run the RX loop on one core, and the TX loop on another core.
Now, DPDK users may have run across the '--lcores' argument to DPDK programs.
See here for reference documentation:
All DPDK command line arguments are processed by the DPDK application
with "rte_eal_init" first thing in the application e.g. early in 'main()'.
Now in this context, the --lcores argument is a hint or direction to DPDK that the program
will create one or more lcores. In DPDK an "lcore" is a thread of control pinned to a
CPU HW core. So, for example, the fragment "--lcores=(1)@2,(2)@4" in the command
would tell DPDK that the program will create and run two application lcores namely
lcore '1' on CPU HW core 2, and lcore '2' on CPU HW core 4.
Then, when all the memzone, mempool, queue, and NIC initialization is done, the program
can parlay the lcores argument above to actually launch the application threads pinned
to the correct as instructed like this,
int lcore_hello(__rte_unused void *arg)
{
unsigned lcore_id;
lcore_id = rte_lcore_id();
printf("hello from core %u\n", lcore_id);
return 0;
}
int main(int argc, char **argv) {
ret = rte_eal_init(argc, argv);
if (ret < 0)
rte_panic("Cannot init EAL\n");
// memzone, mempool, queue, and NIC setup not shown
// If program run with --lcores=(1)@2,(2)@4 this loop will
// create and run two threads lcore 1 pinned to CPU 2 and lcore 2
// pinned to CPU 4. the output will look like:
// hello from core 1
// hello from core 2
RTE_LCORE_FOREACH_WORKER(lcore_id) {
rte_eal_remote_launch(lcore_hello, NULL, lcore_id);
}
.
.
.
See this program for a standalone example including shutdown:
Note the referenced program sneaks in a call to lcore_hello on the main thread
by calling 'lcore_hello(NULL)'. And that call will print 'hello from core 0'. But
do not think that output was a side effect of RTE_LCORE_FOREACH_WORKER.
See next for details.
The other detail worth mentioning is the special case of lcore 0. lcore 0
is a synonym for the main thread e.g. the thread the DPDK program runs on
starting in main. The RTE_LCORE_FOREACH_WORKER will NOT include lcore 0
even if you reference it in the command line. The salient point is the loop
launches WORKER threads, and worker threads are for lcores > 0.
Now if you provide, say, "--lcores=(1)@2,(2)@4,(0)@6" the above program
will still launch the same two worker threads pinned to the same CPUs
just as before. The only difference is that the main thread will be pinned
to CPU core 6. You can get the same effect by running:
# task and main thread start on CPU 6, and "--lcores=(1)@2,(2)@4"
# will start two worker threads on CPUS 2, 4 as described above
taskset -c6 <task> ... "--lcores=(1)@2,(2)@4" ...
> When I run this program I include the command line arguments
> "--lcores=(0)@1,(1)@2" which is passed to 'rte_eal_init'.