From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E93AF43D8E for ; Sat, 30 Mar 2024 22:46:20 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 59CC84025E; Sat, 30 Mar 2024 22:46:20 +0100 (CET) Received: from mail-yb1-f181.google.com (mail-yb1-f181.google.com [209.85.219.181]) by mails.dpdk.org (Postfix) with ESMTP id 842064021D for ; Sat, 30 Mar 2024 22:46:18 +0100 (CET) Received: by mail-yb1-f181.google.com with SMTP id 3f1490d57ef6-dccb1421bdeso2628559276.1 for ; Sat, 30 Mar 2024 14:46:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711835178; x=1712439978; darn=dpdk.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=1akVcE4qqkICyXp/QEAq3QTcDvmH3/rAPoHv9ikWISc=; b=mATZ/NpGcrpKJYqiFx7AJO5F9LBInl2E9x59q4DXl6TfMcqMGsTDDDWpB1hGaS3Yru YjphBMLXBmmpbzn8b0tX0czzp7WHweNKoAeWlIph0H59pe2sGjb3ZAYJw54rN6IxEUtq N+kOVXzA5NC4Gqz8RcdH8gnx6Aj79mysL/QgURr6x7ymm9KzYB2qQuDgbeb5wnkA2Od7 bljdLJcekUlJKkSXnPpe5TsxZNIGNHOVk7edTGnaFn465Fbb/PA/wUYePW4SKr0B9R+k 3wf0DaVk3Z/4tkNlkSIO1LaWyZjbOtaNHpPL8nY71O9y6AY4bhdT4LC/Xi/nB2HDujHk ROzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711835178; x=1712439978; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1akVcE4qqkICyXp/QEAq3QTcDvmH3/rAPoHv9ikWISc=; b=JyGXjZga3/l/GaSj0ZZBpY2SSXoKM2dyhxNqwiNBIUc8xvc8ao0TbZV3zp9ImVg8Ix OxCt9ctPwA0l7fYoJbB1g+FwxK1Y6JL7U/ihL2eAIOmywc9suWihKYpC9P9eOciHdbb9 cHWViTp72V/DRMiad4xrgtPiTAs8Yt32F4aCGPxrX/eeMym0jygbC0OIGt1CEkd8NTUn ct0sjU3Ao1HnmfjbOgu7ccKINsPY2HPbi8QNOhcQKbaB49/39eVQ3rXdJPcu13wB4mwF GRv0UmCL95kiW17M+Xmy/qDJeTtSkuBCG4EGU51uUH/ELh9cT58A/NzYVNXWnXG3TEo9 ACLA== X-Gm-Message-State: AOJu0YxTBxpe7Gqa1CjcpLZ1i3SlBzDfNJ6UagP3jDeC7EjzQ5QJKvQ7 /Vk8ldWnHM5ZdcZex5Ep2X/QXxsNpEiF5xrhfcUb9qUiE36PtThT6nheHlrhKzJWo0CsYBOG3mt NQv4UZarFTbxwG6fzreXOM+NeXe2BP7XWYcw= X-Google-Smtp-Source: AGHT+IEV4ebz07gIbZ26cav9f6fTiz6EhEvLLHse9kFqCrJa0Y2URhNP8H32KjwrX/cXNc37ApZztO1cpp0xqjA53vQ= X-Received: by 2002:a25:5f0d:0:b0:dd1:7a2c:95c3 with SMTP id t13-20020a255f0d000000b00dd17a2c95c3mr4990848ybb.36.1711835177477; Sat, 30 Mar 2024 14:46:17 -0700 (PDT) MIME-Version: 1.0 References: <20240330095202.6cdf59bc@hermes.local> In-Reply-To: <20240330095202.6cdf59bc@hermes.local> From: fwefew 4t4tg <7532yahoo@gmail.com> Date: Sat, 30 Mar 2024 17:45:41 -0400 Message-ID: Subject: Re: --lcores: what it does and does not do To: Stephen Hemminger Cc: users@dpdk.org Content-Type: multipart/alternative; boundary="000000000000aaf0800614e7af1b" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org --000000000000aaf0800614e7af1b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Stephen, thank you for the reply. It occurred to me right after I sent the email, the salient call was rte_eal_remote_launch which you confirmed. For the benefit of everybody else let me provide a full explanation. The scenario is a test program, * Creates 1 RX queue then reads and pretty prints contents in a loop * Creates 1 TX queue then sends packets to a hardcoded IP address in a loop The intent is to run the RX loop on one core, and the TX loop on another core. Now, DPDK users may have run across the '--lcores' argument to DPDK programs. See here for reference documentation: https://doc.dpdk.org/guides-2.0/prog_guide/env_abstraction_layer.html https://doc.dpdk.org/guides/linux_gsg/linux_eal_parameters.html All DPDK command line arguments are processed by the DPDK application with "rte_eal_init" first thing in the application e.g. early in 'main()'. Now in this context, the --lcores argument is a hint or direction to DPDK that the program will create one or more lcores. In DPDK an "lcore" is a thread of control pinned to a CPU HW core. So, for example, the fragment "--lcores=3D(1)@2,(2)@4" in the command would tell DPDK that the program will create and run two application lcores namely lcore '1' on CPU HW core 2, and lcore '2' on CPU HW core 4. Then, when all the memzone, mempool, queue, and NIC initialization is done, the program can parlay the lcores argument above to actually launch the application threads pinned to the correct as instructed like this, int lcore_hello(__rte_unused void *arg) { unsigned lcore_id; lcore_id =3D rte_lcore_id(); printf("hello from core %u\n", lcore_id); return 0; } int main(int argc, char **argv) { ret =3D rte_eal_init(argc, argv); if (ret < 0) rte_panic("Cannot init EAL\n"); // memzone, mempool, queue, and NIC setup not shown // If program run with --lcores=3D(1)@2,(2)@4 this loop will // create and run two threads lcore 1 pinned to CPU 2 and lcore 2 // pinned to CPU 4. the output will look like: // hello from core 1 // hello from core 2 RTE_LCORE_FOREACH_WORKER(lcore_id) { rte_eal_remote_launch(lcore_hello, NULL, lcore_id); } . . . See this program for a standalone example including shutdown: https://doc.dpdk.org/api/examples_2helloworld_2main_8c-example.html#a5 Note the referenced program sneaks in a call to lcore_hello on the main thread by calling 'lcore_hello(NULL)'. And that call will print 'hello from core 0'. But do not think that output was a side effect of RTE_LCORE_FOREACH_WORKER. See next for details. The other detail worth mentioning is the special case of lcore 0. lcore 0 is a synonym for the main thread e.g. the thread the DPDK program runs on starting in main. The RTE_LCORE_FOREACH_WORKER will NOT include lcore 0 even if you reference it in the command line. The salient point is the loop launches WORKER threads, and worker threads are for lcores > 0. Now if you provide, say, "--lcores=3D(1)@2,(2)@4,(0)@6" the above program will still launch the same two worker threads pinned to the same CPUs just as before. The only difference is that the main thread will be pinned to CPU core 6. You can get the same effect by running: # task and main thread start on CPU 6, and "--lcores=3D(1)@2,(2)@4" # will start two worker threads on CPUS 2, 4 as described above taskset -c6 ... "--lcores=3D(1)@2,(2)@4" ... > When I run this program I include the command line arguments > "--lcores=3D(0)@1,(1)@2" which is passed to 'rte_eal_init'. On Sat, Mar 30, 2024 at 12:52=E2=80=AFPM Stephen Hemminger < stephen@networkplumber.org> wrote: > On Sat, 30 Mar 2024 12:06:15 -0400 > fwefew 4t4tg <7532yahoo@gmail.com> wrote: > > > I've made a DPDK test program that does the following on the same machi= ne > > and for the same NIC. This is a test; it does nothing practical: > > > > * Creates 1 RX queue then reads and pretty prints contents in a loop > > * Creates 1 TX queue then sends packets to a hardcoded IP address in a > loop > > > > When I run this program I include the command line arguments > > "--lcores=3D(0)@1,(1)@2" which is passed to 'rte_eal_init'. > > > > This means there's a lcore identified '0' run on CPU HW core 1, and lco= re > > '1' run on CPU HW core 2. > > > > As I understand it, the intent of the lcores argument is that this test > > program will eventually run the RX loop as lcore 0, and the TX loop as > > lcore 1 (or vice-versa). > > > > On the other hand after all the required DPDK setup is done --- memzone= s, > > mempools, queues, NIC devices initialized and started --- here's what > DPDK > > has not done: > > > > * It hasn't started an application thread for lcore 0 or lcore 1 > > * DPDK doesn't know the function entry point for either loop so no loop= s > > are running. > > > > Which is totally fine ... DPDK isn't magic. If the application programm= er > > wants a RX and TX application thread pinned to some CPU, it should crea= te > > the threads, set the CPU affinity, and run the loop in the NUMA aligned > > way. This is trivial to do. That is, with the required DPDK setup done, > all > > that's left is to do this trivial work ... and the test program is up a= nd > > running: problem solved. > > > > The --lcores doesn't and cannot do this application work. So what is th= e > > practical result of it? What does it do? > > Did your application every start the other threads with DPDK? > It is not recommended for applications to manage their own threads. > Possible but hard to get right. > > A typical application looks like: > > > main() { > // do some initialization > rte_eal_init() > > // do more initialization that has to be done in main thread > > rte_eal_mp_remote_launch(worker_func, arg, CALL_MAIN); > > // main thread worker_func has exited, wait for others > rte_eal_mp_wait_lcore() > > // back to single main thread > rte_eal_cleanup() > } > > --000000000000aaf0800614e7af1b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
<= font face=3D"monospace">Stephen, thank you for the reply. It occurred to me= right after I sent
the email,=C2=A0the salient call was=C2=A0rte_eal_remote_launch which you confirmed= .

For the benefit of everybody else let me provide a full explanation= .

The scenario is a test program,

* Creates 1 RX queue th= en reads and pretty prints contents in a loop
* Creates 1 TX = queue then sends packets to a hardcoded IP address in a loop
=
The intent is to run the RX loop on one core, and the TX loop on ano= ther core.=C2=A0

Now, DPDK=C2=A0users may have run across the '= ;--lcores' argument to DPDK programs.=C2=A0
See here for reference=C2= =A0documentation:=C2=A0

https://doc.dpdk.= org/guides-2.0/prog_guide/env_abstraction_layer.html
https://doc= .dpdk.org/guides/linux_gsg/linux_eal_parameters.html

<= /div>
All D= PDK command line arguments are processed by the DPDK application
with &qu= ot;rte_eal_init" first thing in the application e.g. early in 'mai= n()'.

Now in this context, the --lcores argument is a hint or = direction to DPDK that the program
will create=C2=A0one or more lcores. I= n DPDK an "lcore" is a thread of control pinned to a
=
CPU HW cor= e. So, for example,=C2=A0 the fragment=C2=A0"--lcores=3D(1)@2,(2)@4&qu= ot; in the command
would tell DPDK that the program will create and run t= wo application lcores namely=C2=A0
lcore '1' on CPU=C2=A0<= span style=3D"font-family:monospace">HW core 2, and lcore '2' on CP= U HW core 4.

Then, when all the memzone, mempool, queue, and NIC i= nitialization is done, the program
can parlay the lcores argument above t= o actually launch the application threads pinned
to the correct as instru= cted like this,

=C2=A0 int lcore_hello(__rte_unused void *arg)
=C2= =A0 {
=C2=A0 =C2=A0 unsigned lcore_id;
=C2=A0 =C2=A0 lcore_id =3D rte= _lcore_id();
=C2=A0 =C2=A0 printf("hello from core %u\n", lcor= e_id);
=C2=A0 =C2=A0 return 0;
=C2=A0}
=C2=A0int main(int argc,= char **argv) {
=C2=A0 =C2=A0 ret =3D rte_eal_init(argc, argv);
=C2=A0= =C2=A0 if (ret < 0)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 rte_panic("Cann= ot init EAL\n");
<= font face=3D"monospace">
=C2=A0 =C2=A0 // memzone, mempool, queue, an= d NIC setup not shown
<= font face=3D"monospace">
=C2=A0 =C2=A0 // If program run with=C2=A0--= lcores=3D(1)@2,(2)@4 this loop will
=C2=A0 =C2=A0 // create and run two t= hreads lcore 1 pinned to CPU 2 and lcore 2
=C2=A0 =C2=A0 // pinned to CPU= 4. the output will look like:
=C2=A0 =C2=A0 //=C2=A0 =C2=A0 =C2=A0hello = from core 1
=C2=A0 =C2=A0 //=C2=A0 =C2=A0 =C2=A0hello from core 2=
=C2= =A0 =C2=A0 RTE_LCORE_FOREACH_WORKER(lcore_id) {
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 rte_eal_remote_launch(lcore_hello, NULL, lcore_id);
=C2=A0 =C2=A0 }<= br>
=C2=A0 =C2=A0 .
<= font face=3D"monospace">=C2=A0 =C2=A0 .
=C2=A0 =C2=A0 .

See this= program for a standalone example including shutdown:
https://doc= .dpdk.org/api/examples_2helloworld_2main_8c-example.html#a5
Note the referenced program sneaks= in a call to=C2=A0lcore_hello on the= main thread
by calling 'lcore_hello(NULL)'. = And that call will print 'hello f= rom core 0'. But
do not think that output was a side eff= ect of=C2=A0RTE_LCORE_FOREACH_= WORKER.
See next for details.

The other detail worth mentioning is the special case of lcore 0. = lcore 0
is a synonym for the main thread e.g. the thread the DPDK program= runs on
starting in main. The=C2=A0RTE_LCORE_FOREACH_WORKER will NOT include lcore 0
ev= en if you reference it in the command line. The salient point is the loop
launches WORKER threads, and worker threads are for lcores > 0.<= /div>

<= /font>
Now if you provide, say, "--lcores=3D(1)@2,(2)@4,(0)@6" the above program
will= still launch the same two worker threads pinned to the same CPUs
just as before. The only difference is that the main thread will be p= inned
to CPU core 6. You can get the same effect by running:=
=C2=A0 =C2=A0
=C2=A0 =C2=A0# task and main thread start = on CPU 6, and "--lcores= =3D(1)@2,(2)@4"
=C2=A0 =C2=A0# will start two worker th= reads on CPUS 2, 4 as described above
=C2=A0 =C2=A0taskset -c6 <task&= gt;=C2=A0 ...=C2=A0"--lcores=3D(1)@2,(2)@4" ...

> When I run this program I include= the command line arguments
> "--lcores=3D(0)@1,(1)@2= " which is passed to 'rte_eal_init'.


On Sat, Mar 30, 2024= at 12:52=E2=80=AFPM Stephen Hemminger <stephen@networkplumber.org> wrote:
On Sat, 30 Mar 2024 12:06:15 -0400
fwefew 4t4tg <7= 532yahoo@gmail.com> wrote:

> I've made a DPDK test program that does the following on the same = machine
> and for the same NIC. This is a test; it does nothing practical:
>
> * Creates 1 RX queue then reads and pretty prints contents in a loop > * Creates 1 TX queue then sends packets to a hardcoded IP address in a= loop
>
> When I run this program I include the command line arguments
> "--lcores=3D(0)@1,(1)@2" which is passed to 'rte_eal_ini= t'.
>
> This means there's a lcore identified '0' run on CPU HW co= re 1, and lcore
> '1' run on CPU HW core 2.
>
> As I understand it, the intent of the lcores argument is that this tes= t
> program will eventually run the RX loop as lcore 0, and the TX loop as=
> lcore 1 (or vice-versa).
>
> On the other hand after all the required DPDK setup is done --- memzon= es,
> mempools, queues, NIC devices initialized and started --- here's w= hat DPDK
> has not done:
>
> * It hasn't started an application thread for lcore 0 or lcore 1 > * DPDK doesn't know the function entry point for either loop so no= loops
> are running.
>
> Which is totally fine ... DPDK isn't magic. If the application pro= grammer
> wants a RX and TX application thread pinned to some CPU, it should cre= ate
> the threads, set the CPU affinity, and run the loop in the NUMA aligne= d
> way. This is trivial to do. That is, with the required DPDK setup done= , all
> that's left is to do this trivial work ... and the test program is= up and
> running: problem solved.
>
> The --lcores doesn't and cannot do this application work. So what = is the
> practical result of it? What does it do?

Did your application every start the other threads with DPDK?
It is not recommended for applications to manage their own threads.
Possible but hard to get right.

A typical application looks like:


main() {
=C2=A0 =C2=A0 // do some initialization
=C2=A0 =C2=A0 rte_eal_init()

=C2=A0 =C2=A0 // do more initialization that has to be done in main thread<= br>
=C2=A0 =C2=A0 rte_eal_mp_remote_launch(worker_func, arg, CALL_MAIN);

=C2=A0 =C2=A0 // main thread worker_func has exited, wait for others
=C2=A0 =C2=A0 rte_eal_mp_wait_lcore()

=C2=A0 =C2=A0 // back to single main thread
=C2=A0 =C2=A0 rte_eal_cleanup()
}
--000000000000aaf0800614e7af1b--