From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 99AA3429E4; Tue, 25 Apr 2023 04:59:38 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2B2CE40A7E; Tue, 25 Apr 2023 04:59:38 +0200 (CEST) Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by mails.dpdk.org (Postfix) with ESMTP id 8821A400D7 for ; Tue, 25 Apr 2023 04:59:36 +0200 (CEST) Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-63b62d2f729so4292227b3a.1 for ; Mon, 24 Apr 2023 19:59:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=damore-org.20221208.gappssmtp.com; s=20221208; t=1682391575; x=1684983575; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=hjLJkjN+gWufbMBkA/ILyWIB3J9QNVdH5dd+JiLDOxk=; b=Gx3MffIRlXxeyjEwlKhB4MnMbk4mlshj5Z5wQbLlf+t8bf5CbX5MarBA1rXZaplh8V G74ynqmFW0rAAP04bk9qt2NDr5BsUHVBJvzMnDMKzCbt/222+UO1yxfOEUKdMQQfLpbE 5NZDHdMLYEcupOABYvh8nrKr8HWkRSVIatvToF5hYWRmbGOnKvN/DqOYPnCzx0LeNb2o aNz6/vS8r1BCx6g6Ou+eDJXm9O2b8jRr4QOi5P2Q8VMjMGLfV2bn2jARNRIwJOnJ8PV3 kDYij+D2F8xnnjj4KiRdVnvFmmHFSDl8YqrENBJM8WVvDT6Xrr/7XEwPGLup9tbqCnHc HruQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682391575; x=1684983575; h=mime-version:subject:references:in-reply-to:message-id:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hjLJkjN+gWufbMBkA/ILyWIB3J9QNVdH5dd+JiLDOxk=; b=MjAr3Z1d3moW+e9TrQY3pHLBMJszRdvloawHsXKZPFUl3JjzMnWMBbl1XpHj3taZSP /6npbyaDvIBiU4dA+NC6pkMud2JPelw38+H7veOHloDt3xdwbiMBL8hMg086g+mhHL/H Ejhu6MnZ0fQGvWSBP2k9TNK3z0H5a68YTYAzGGr6NzMyiswIfueoWPXIXY0XiMOoF1EC sXQ6s9fLdorL1zj4r1XhdCsR0CoRK3FPVx5iL0GEBOgroFVV9wzNSlDBI48Oupplrx4/ TwMy1nY/0iGMq9gXOmpt0eLYT6w/hNZ/hAw59LXtrOGY8gb/fApg9qd4+1J3+vOnJeS5 WBew== X-Gm-Message-State: AAQBX9e++MczAJSldZ16cOm5HFtLZBKQrx4H45P2o+dMn8cf8zpfyt3u UxZThNTvU2P8Zvj7XSWv5awPhQ== X-Google-Smtp-Source: AKy350YU62uCfL21H0cYgINQ9fElKjV1z66sKVSqQDs4/zDfvGM1XwlVOdClsAtRcjtSUckYG37MhA== X-Received: by 2002:a05:6a21:348a:b0:f0:ec12:7bdf with SMTP id yo10-20020a056a21348a00b000f0ec127bdfmr17965876pzb.32.1682391575515; Mon, 24 Apr 2023 19:59:35 -0700 (PDT) Received: from [10.0.4.28] (ip68-6-176-252.sd.sd.cox.net. [68.6.176.252]) by smtp.gmail.com with ESMTPSA id g1-20020a056a00078100b0063b675f01a2sm7963979pfu.26.2023.04.24.19.59.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 24 Apr 2023 19:59:35 -0700 (PDT) Date: Mon, 24 Apr 2023 19:59:27 -0700 From: Garrett D'Amore To: Stephen Hemminger , fengchengwen Cc: thomas@monjalon.net, ferruh.yigit@amd.com, dev@dpdk.org Message-ID: <9659343a-97c5-49e1-a71c-060cea098111@Spark> In-Reply-To: References: <20230424130208.9517-1-fengchengwen@huawei.com> <20230424090836.0550e14b@hermes.local> <20230424191621.2075bf31@hermes.local> Subject: Re: [RFC 0/3] introduce coroutine library X-Readdle-Message-ID: 9659343a-97c5-49e1-a71c-060cea098111@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="64474215_1190cde7_29c" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --64474215_1190cde7_29c Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline =46irst time poster here: I worry a bit about a coroutine approach as it may be challenging for som= e uses like ours.=C2=A0=C2=A0We have a purely event driven loop with a Re= actor model written in D.=C2=A0=C2=A0The details are not specifically nee= ded here, except to point out that an approach based on ucontext.h or som= ething like that would very likely be utterly incompatible in our environ= ment.=C2=A0=C2=A0While we don=E2=80=99t currently plan to integrate suppo= rt for your hns3 device, I would have grave reservations about a general = coroutine library making it=E2=80=99s way into DPDK drivers =E2=80=94 it = would almost certainly cause no end of grief for us at Weka. I=E2=80=99m doubtful that we=E2=80=99re the only DPDK users in this situa= tion. =E2=80=A2 Garrett On Apr 24, 2023 at 7:50 PM -0700, fengchengwen , wrote: > On 2023/4/25 10:16, Stephen Hemminger wrote: > > On Tue, 25 Apr 2023 10:11:43 +0800 > > fengchengwen wrote: > > > > > On 2023/4/25 0:08, Stephen Hemminger wrote: > > > > On Mon, 24 Apr 2023 13:02:05 +0000 > > > > Chengwen =46eng wrote: > > > > > > > > > This patchset introduces the coroutine library which will help = refactor > > > > > the hns3 PMD's reset process. > > > > > > > > > > The hns3 single function reset process consists of the followin= g steps: > > > > > 1.stop=5Fservice(); > > > > > 2.prepare=5Freset(); > > > > > 3.delay(100ms); > > > > > 4.notify=5Fhw(); > > > > > 5.wait=5Fhw=5Freset=5Fdone(); // multiple sleep waits are invol= ved. > > > > > 6.reinit(); > > > > > 7.restore=5Fconf(); > > > > > > > > > > If the DPDK process take over multiple hns3 functions (e.g. 100= ), > > > > > it's impractical to reset and restore functions in sequence: > > > > > 1.proc=5Ffunc(001); // will completed in 100+ms range. > > > > > 2.proc=5Ffunc(002); // will completed in 100=7E200+ms range. > > > > > ... > > > > > x.proc=5Ffunc(100); // will completed in 9900=7E10000+ms range.= > > > > > The later functions will process fail because it's too late to = deal with. > > > > > > > > > > One solution is that create a reset thread for each function, a= nd it > > > > > will lead to large number of threads if the DPDK process take o= ver > > > > > multiple hns3 functions. > > > > > > > > > > So the current hns3 driver uses asynchronous mechanism, for exa= mples, it > > > > > use rte=5Feal=5Falarm=5Fset() when process delay(100ms), it spl= its a serial > > > > > process into multiple asynchronous processes, and the code is c= omplex > > > > > and difficult to understand. > > > > > > > > > > The coroutine is a good mechanism to provide programmers with t= he > > > > > simplicity of keeping serial processes within a limited number = of > > > > > threads. > > > > > > > > > > This patchset use to build the coroutine framework= , and it > > > > > just provides a demo. More APIs maybe added in the future. > > > > > > > > > > In addition, we would like to ask the community whether it it p= ossible > > > > > to accept the library. If not, whether it is allowed to provide= the > > > > > library in hns3 PMD. > > > > > > > > > > Chengwen =46eng (3): > > > > > lib/coroutine: add coroutine library > > > > > examples/coroutine: support coroutine examples > > > > > net/hns3: refactor reset process with coroutine > > > > > > > > Interesting, but the DPDK really is not the right place for this.= > > > > Also, why so much sleeping. Can't this device be handled with an = event based > > > > model. Plus any complexity like this introduces more bugs into al= ready fragile > > > > interaction of DPDK userspace applications and threads. > > > > > > A event base model will function as: > > > event-handler() =7B > > > for (...) =7B > > > event =3D get=5Fnext=5Fevent(); > > > proc=5Fevent(); > > > =7D > > > =7D > > > The root cause is that the proc=5Fevent() take too many time, and i= t will lead to other > > > function can't be processed timely. > > > > > > =46or which proc=5Fevent() may wait a lot of time, the coroutine co= uld also used to optimize > > > it. > > > > > > > > > > > Not only that, coroutines add to the pre-existing problems with l= ocking. > > > > If coroutine 1 acquires a lock, the coroutine 2 will deadlock its= elf. > > > > And someone will spend days figuring that out. And the existing a= nalyzer > > > > tools will not know about the magic coroutine library. > > > > > > Analyzer tools like lock annotations maybe a problem. > > > > > > Locks in DPDK APIs are mostly no-blocking. We can add some restrict= ions(by reviewer), such > > > as once holding a lock, you can't invoke rte=5Fco=5Fyield() or rte=5F= co=5Fdelay() API. > > > > > In addition, any technology has two sides, the greatest advantage o= f coroutine I think is > > > removes a large number of callbacks in asychronous programming. And= also high-level languages > > > generally provide coroutines (e.g. C++/Python). With the developmen= t, the analyzer tools maybe > > > evolved to support detect. > > > > > > > > > And one more, if not acceptable as public library, whether it is al= lowed intergration of this > > > library in hns3 PMD =3F Our internal evaluation solution (use corou= tine refactor) is feasible, > > > but the code needs to be upstream, hope to listen to community's co= mments. > > > > > > The standard DPDK architecture is to have dedicated threads. > > Unless you convert the user application to some other model, there re= ally is no other > > Instead of adding a new running model, this coroutine library just adap= ts to the current > DPDK framework. > My visions: > 1. DPDK launch a default thread run coroutine service, this thread just= like interrupt thread, > it could provide server for PMD drivers. > 2. Application could launch coroutine scheduler in lcore (just like 2/3= commit) if it want to > use this library. > > > useful work that can be done while waiting for your driver. > > > > There was a previous DPDK library for lightweight threading, but it n= ever got any > > usage and was abandoned and dropped. Why is this better=3F > > DPDK lightweight threading =3F I didn't know before, but will take a cl= oser look at. > > This patchset is caused by a problem in the our driver reset process. T= he reset process is > complex and difficult to understand. We think the coroutine implementat= ion is practical to > solve this problem, so we try to send this R=46C. > > > . > > --64474215_1190cde7_29c Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
=46irst time poster here:

I worry a bit about a coroutine approach as it may be challenging for som= e uses like ours.&=23160;&=23160;We have a purely event driven loop with = a Reactor model written in D.&=23160;&=23160;The details are not specific= ally needed here, except to point out that an approach based on ucontext.= h or something like that would very likely be utterly incompatible in our= environment.&=23160;&=23160;While we don=E2=80=99t currently plan to int= egrate support for your hns3 device, I would have grave reservations abou= t a general coroutine library making it=E2=80=99s way into DPDK drivers =E2= =80=94 it would almost certainly cause no end of grief for us at Weka.&=23= 160;

I=E2=80=99m doubtful that we=E2=80=99re the only DPDK users in this situa= tion.
  • Garrett
On Apr 24, 2023 at 7:50 PM -0700, f= engchengwen <fengchengwen=40huawei.com>, wrote:
On 2023/4/25 10:16, Stephen Hemminger wrote:
On Tue, 25 Apr 2023 10:11:43 +0800
fengchengwen <fengchengwen=40huawei.com> wrote:

On 2023/4/25 0:08, Stephen Hemminger wrote:=
On Mon, 24 Apr 2023 13:02:05 +0000
Chengwen =46eng <fengchengwen=40huawei.com> wrote:

This patchset introduces the coroutine libr= ary which will help refactor
the hns3 PMD's reset process.

The hns3 single function reset process consists of the following steps: 1.stop=5Fservice();
2.prepare=5Freset();
3.delay(100ms);
4.notify=5Fhw();
5.wait=5Fhw=5Freset=5Fdone(); // multiple sleep waits are involved.
= 6.reinit();
7.restore=5Fconf();

If the DPDK process take over multiple hns3 functions (e.g. 100),
it's impractical to reset and restore functions in sequence:
1.proc=5Ffunc(001); // will completed in 100+ms range.
2.proc=5Ffunc(002); // will completed in 100=7E200+ms range.
...
x.proc=5Ffunc(100); // will completed in 9900=7E10000+ms range.
The later functions will process fail because it's too late to deal with.=

One solution is that create a reset thread for each function, and it
will lead to large number of threads if the DPDK process take over
multiple hns3 functions.

So the current hns3 driver uses asynchronous mechanism, for examples, it<= br /> use rte=5Feal=5Falarm=5Fset() when process delay(100ms), it splits a seri= al
process into multiple asynchronous processes, and the code is complex
and difficult to understand.

The coroutine is a good mechanism to provide programmers with the
simplicity of keeping serial processes within a limited number of
threads.

This patchset use <ucontext.h> to build the coroutine framework, an= d it
just provides a demo. More APIs maybe added in the future.

In addition, we would like to ask the community whether it it possible to accept the library. If not, whether it is allowed to provide the
= library in hns3 PMD.

Chengwen =46eng (3):
lib/coroutine: add coroutine library
examples/coroutine: support coroutine examples
net/hns3: refactor reset process with coroutine

Interesting, but the DPDK really is not the right place for this.
Also, why so much sleeping. Can't this device be handled with an event ba= sed
model. Plus any complexity like this introduces more bugs into already fr= agile
interaction of DPDK userspace applications and threads.

A event base model will function as:
event-handler() =7B
for (...) =7B
event =3D get=5Fnext=5Fevent();
proc=5Fevent();
=7D
=7D
The root cause is that the proc=5Fevent() take too many time, and it will= lead to other
function can't be processed timely.

=46or which proc=5Fevent() may wait a lot of time, the coroutine could al= so used to optimize
it.


Not only that, coroutines add to the pre-existing problems with locking.<= br /> If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself.
And someone will spend days figuring that out. And the existing analyzer<= br /> tools will not know about the magic coroutine library.
=
Analyzer tools like lock annotations maybe a problem.

Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(b= y reviewer), such
as once holding a lock, you can't invoke rte=5Fco=5Fyield() or rte=5Fco=5F= delay() API.

In addition, any technology has two sides, = the greatest advantage of coroutine I think is
removes a large number of callbacks in asychronous programming. And also = high-level languages
generally provide coroutines (e.g. C++/Python). With the development, the= analyzer tools maybe
evolved to support detect.


And one more, if not acceptable as public library, whether it is allowed = intergration of this
library in hns3 PMD =3F Our internal evaluation solution (use coroutine r= efactor) is feasible,
but the code needs to be upstream, hope to listen to community's comments= .


The standard DPDK architecture is to have dedicated threads.
Unless you convert the user application to some other model, there really= is no other

Instead of adding a new running model, this coroutine library just adapts= to the current
DPDK framework.
My visions:
1. DPDK launch a default thread run coroutine service, this thread just l= ike interrupt thread,
it could provide server for PMD drivers.
2. Application could launch coroutine scheduler in lcore (just like 2/3 c= ommit) if it want to
use this library.

useful work that can be done while waiting = for your driver.

There was a previous DPDK library for lightweight threading, but it never= got any
usage and was abandoned and dropped. Why is this better=3F

DPDK lightweight threading =3F I didn't know before, but will take a clos= er look at.

This patchset is caused by a problem in the our driver reset process. The= reset process is
complex and difficult to understand. We think the coroutine implementatio= n is practical to
solve this problem, so we try to send this R=46C.

.

--64474215_1190cde7_29c--