[RFC 0/3] introduce coroutine library

[RFC 0/3] introduce coroutine library

[Chengwen Feng]

This patchset introduces the coroutine library which will help refactor
the hns3 PMD's reset process.

The hns3 single function reset process consists of the following steps:
    1.stop_service();
    2.prepare_reset();
    3.delay(100ms);
    4.notify_hw();
    5.wait_hw_reset_done(); // multiple sleep waits are involved.
    6.reinit();
    7.restore_conf();

If the DPDK process take over multiple hns3 functions (e.g. 100),
it's impractical to reset and restore functions in sequence:
    1.proc_func(001); // will completed in 100+ms range.
    2.proc_func(002); // will completed in 100~200+ms range.
    ...
    x.proc_func(100); // will completed in 9900~10000+ms range.
The later functions will process fail because it's too late to deal with.

One solution is that create a reset thread for each function, and it
will lead to large number of threads if the DPDK process take over
multiple hns3 functions.

So the current hns3 driver uses asynchronous mechanism, for examples, it
use rte_eal_alarm_set() when process delay(100ms), it splits a serial
process into multiple asynchronous processes, and the code is complex
and difficult to understand.

The coroutine is a good mechanism to provide programmers with the 
simplicity of keeping serial processes within a limited number of
threads.

This patchset use <ucontext.h> to build the coroutine framework, and it
just provides a demo. More APIs maybe added in the future.

In addition, we would like to ask the community whether it it possible
to accept the library. If not, whether it is allowed to provide the
library in hns3 PMD.

Chengwen Feng (3):
  lib/coroutine: add coroutine library
  examples/coroutine: support coroutine examples
  net/hns3: refactor reset process with coroutine

 drivers/net/hns3/hns3_ethdev.c    | 217 ++++++++++++++++++++++++++++++
 drivers/net/hns3/hns3_ethdev.h    |   3 +
 drivers/net/hns3/hns3_intr.c      |  38 ++++++
 drivers/net/hns3/meson.build      |   2 +-
 examples/coroutine/main.c         | 153 +++++++++++++++++++++
 examples/coroutine/meson.build    |  10 ++
 examples/meson.build              |   1 +
 lib/coroutine/meson.build         |   8 ++
 lib/coroutine/rte_coroutine.c     | 190 ++++++++++++++++++++++++++
 lib/coroutine/rte_coroutine.h     | 110 +++++++++++++++
 lib/coroutine/rte_coroutine_imp.h |  46 +++++++
 lib/coroutine/version.map         |  11 ++
 lib/meson.build                   |   1 +
 13 files changed, 789 insertions(+), 1 deletion(-)
 create mode 100644 examples/coroutine/main.c
 create mode 100644 examples/coroutine/meson.build
 create mode 100644 lib/coroutine/meson.build
 create mode 100644 lib/coroutine/rte_coroutine.c
 create mode 100644 lib/coroutine/rte_coroutine.h
 create mode 100644 lib/coroutine/rte_coroutine_imp.h
 create mode 100644 lib/coroutine/version.map

-- 
2.17.1

[RFC 1/3] lib/coroutine: add coroutine library

[Chengwen Feng]

This patch adds coroutine library. The main elements are:
1. scheduler: container of coroutines, which is responsible for
scheduling coroutine.
2. coroutine: Minimum scheduling unit, it should associated to one
scheduler.

In the coroutine callback, application could invoke rte_co_yield() to
give up the CPU, and invoke rte_co_delay() to delay the specified
microsecond.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 lib/coroutine/meson.build         |   8 ++
 lib/coroutine/rte_coroutine.c     | 190 ++++++++++++++++++++++++++++++
 lib/coroutine/rte_coroutine.h     | 110 +++++++++++++++++
 lib/coroutine/rte_coroutine_imp.h |  46 ++++++++
 lib/coroutine/version.map         |  11 ++
 lib/meson.build                   |   1 +
 6 files changed, 366 insertions(+)
 create mode 100644 lib/coroutine/meson.build
 create mode 100644 lib/coroutine/rte_coroutine.c
 create mode 100644 lib/coroutine/rte_coroutine.h
 create mode 100644 lib/coroutine/rte_coroutine_imp.h
 create mode 100644 lib/coroutine/version.map

diff --git a/lib/coroutine/meson.build b/lib/coroutine/meson.build
new file mode 100644
index 0000000000..2064fb1909
--- /dev/null
+++ b/lib/coroutine/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 HiSilicon Limited.
+
+sources = files('rte_coroutine.c')
+headers = files('rte_coroutine.h')
+indirect_headers += files('rte_coroutine_imp.h')
+
+deps += ['ring']
diff --git a/lib/coroutine/rte_coroutine.c b/lib/coroutine/rte_coroutine.c
new file mode 100644
index 0000000000..07c79fc901
--- /dev/null
+++ b/lib/coroutine/rte_coroutine.c
@@ -0,0 +1,190 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 HiSilicon Limited
+ */
+
+#include <pthread.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <ucontext.h>
+
+#include <rte_alarm.h>
+#include <rte_ring.h>
+
+#include "rte_coroutine.h"
+#include "rte_coroutine_imp.h"
+
+#define FATAL(fmt, args...) printf("[FATAL] %s() %d: " fmt "\n", __func__, __LINE__, ##args)
+
+static __thread struct rte_schedule *co_schedule;
+
+struct rte_schedule *
+rte_schedule_create(const char *name, uint32_t max_coroutines)
+{
+	struct rte_schedule *s = calloc(1, sizeof(struct rte_schedule));
+	if (s == NULL)
+		return NULL;
+
+	s->ring = rte_ring_create(name, max_coroutines, rte_socket_id(),
+				  RING_F_SC_DEQ);
+	if (s->ring == NULL) {
+		free(s);
+		return NULL;
+	}
+
+	s->max_coroutines = max_coroutines;
+
+	return s;
+}
+
+static void
+co_run_func(uint32_t low, uint32_t hi)
+{
+	uintptr_t ptr = (uint64_t)low | ((uint64_t)hi << 32);
+	struct rte_cocontext *co = (struct rte_cocontext *)ptr;
+	co->cb(co->arg);
+	/* Run complete, so free it. */
+	free(co->stack);
+	free(co);
+}
+
+int
+rte_schedule_run(struct rte_schedule *s)
+{
+	struct rte_cocontext *co = NULL;
+	uintptr_t ptr;
+
+	/* Set local thread variable as input argument. */
+	co_schedule = s;
+
+	while (!rte_ring_empty(s->ring)) {
+		rte_ring_dequeue(s->ring, (void **)&co);
+		if (co->state == COROUTINE_READY) {
+			getcontext(&co->ctx);
+			co->ctx.uc_stack.ss_sp = co->stack;
+			co->ctx.uc_stack.ss_size = co->stack_sz;
+			co->ctx.uc_link = &s->main;
+			co->state = COROUTINE_RUNNING;
+			s->running = co;
+			ptr = (uintptr_t)co;
+			makecontext(&co->ctx, (void (*)(void))co_run_func, 2,
+				    (uint32_t)ptr, (uint32_t)(ptr >> 32));
+			swapcontext(&s->main, &co->ctx);
+		} else if (co->state == COROUTINE_SUSPEND) {
+			co->state = COROUTINE_RUNNING;
+			s->running = co;
+			swapcontext(&s->main, &co->ctx);
+		} else {
+			FATAL("invalid state!");
+		}
+	}
+
+	while (s->yield_head != NULL) {
+		co = s->yield_head;
+		s->yield_head = co->yield_next;
+		if (co->state == COROUTINE_YIELD) {
+			co->state = COROUTINE_RUNNING;
+			s->running = co;
+			swapcontext(&s->main, &co->ctx);
+		} else {
+			FATAL("invalid yield state!");
+		}
+	}
+
+	return 0;
+}
+
+int
+rte_co_create(struct rte_schedule *s, coroutine_callback_t cb, void *arg, uint32_t stack_sz)
+{
+	struct rte_cocontext *co = calloc(1, sizeof(struct rte_cocontext));
+	int ret;
+	if (co == NULL)
+		return -ENOMEM;
+
+	co->owner = s;
+	co->state = COROUTINE_READY;
+	co->cb = cb;
+	co->arg = arg;
+	if (stack_sz < MIN_STACK_SIZE)
+		stack_sz = MIN_STACK_SIZE;
+	co->stack_sz = stack_sz;
+	co->stack = calloc(1, stack_sz);
+	if (co->stack == NULL) {
+		free(co);
+		return -ENOMEM;
+	}
+
+	ret = rte_ring_enqueue(s->ring, co);
+	if (ret != 0) {
+		free(co->stack);
+		free(co);
+	}
+
+	return ret;
+}
+
+static inline void
+co_addto_yield_list(struct rte_schedule *s, struct rte_cocontext *co)
+{
+	co->yield_next = NULL;
+	if (s->yield_head == NULL) {
+		s->yield_head = s->yield_tail = co;
+	} else {
+		s->yield_tail->yield_next = co;
+		s->yield_tail = co;
+	}
+}
+
+void
+rte_co_yield(void)
+{
+	struct rte_schedule *s = co_schedule;
+	struct rte_cocontext *co;
+	if (s == NULL) {
+		FATAL("thread co_schedule is NULL!");
+		return;
+	}
+	co = s->running;
+	if (co == NULL) {
+		FATAL("running is NULL!");
+		return;
+	}
+	co->state = COROUTINE_YIELD;
+	s->running = NULL;
+	co_addto_yield_list(s, co);
+	swapcontext(&co->ctx, &s->main);
+}
+
+static void
+co_delay_imp(void *arg)
+{
+	struct rte_cocontext *co = (struct rte_cocontext *)arg;
+	int ret;
+
+	ret = rte_ring_enqueue(co->owner->ring, (void *)co);
+	if (ret != 0)
+		FATAL("enqueue failed!");
+}
+
+void
+rte_co_delay(unsigned int us)
+{
+	struct rte_schedule *s = co_schedule;
+	struct rte_cocontext *co;
+
+	if (s == NULL) {
+		FATAL("thread co_schedule is NULL!");
+		return;
+	}
+
+	co = s->running;
+	if (co == NULL) {
+		FATAL("running is NULL!");
+		return;
+	}
+
+	rte_eal_alarm_set(us, co_delay_imp, (void *)co);
+	co->state = COROUTINE_SUSPEND;
+	s->running = NULL;
+	swapcontext(&co->ctx, &s->main);
+}
diff --git a/lib/coroutine/rte_coroutine.h b/lib/coroutine/rte_coroutine.h
new file mode 100644
index 0000000000..71ac9488b6
--- /dev/null
+++ b/lib/coroutine/rte_coroutine.h
@@ -0,0 +1,110 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 HiSilicon Limited
+ */
+
+#ifndef RTE_COROUTINE_H
+#define RTE_COROUTINE_H
+
+#include <stdint.h>
+
+#include <rte_compat.h>
+
+/**
+ * Callback prototype of coroutine.
+ *
+ * @param arg
+ *   An arg pointer coming from the caller.
+ */
+typedef void (*coroutine_callback_t)(void *arg);
+
+struct rte_schedule;
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Create coroutine scheduler.
+ *
+ * The scheduler is a coroutines container, which could schedule coroutine
+ * running.
+ *
+ * @param name
+ *   The unique name of scheduler.
+ * @param max_coroutines
+ *   Maximum number of coroutines that can be processed by the scheduler.
+ *
+ * @return
+ *   Non-NULL on success. Otherwise NULL value is returned.
+ */
+__rte_experimental
+struct rte_schedule *rte_schedule_create(const char *name, uint32_t max_coroutines);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Run the coroutine scheduler.
+ *
+ * This function will schedule all associated coroutine, it will return zero if
+ * no coroutines are active after scheduled.
+ *
+ * @param s
+ *   The pointer of the scheduler.
+ *
+ * @return
+ *   0 on success. Otherwise negative value is returned.
+ */
+__rte_experimental
+int rte_schedule_run(struct rte_schedule *s);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Create one coroutine.
+ *
+ * This function will create one coroutine which associated target scheduler
+ * 's'.
+ *
+ * @param s
+ *   The pointer of the scheduler.
+ * @param cb
+ *   The callback function.
+ * @param arg
+ *   The argument parameter for callback function.
+ * @param stack_sz
+ *   The stack size which associated to coroutine. The value zero indicates that
+ *   the default size (which is 8KB) is used.
+ *
+ * @return
+ *   0 on success. Otherwise negative value is returned.
+ */
+__rte_experimental
+int rte_co_create(struct rte_schedule *s, coroutine_callback_t cb, void *arg, uint32_t stack_sz);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Coroutine yield the CPU.
+ *
+ * This function yield the cpu of current coroutine.
+ */
+__rte_experimental
+void rte_co_yield(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Coroutine delay function.
+ *
+ * This function support delay microseconds of current coroutine.
+ *
+ * @param us
+ *   The time in microseconds which need delay.
+ */
+__rte_experimental
+void rte_co_delay(unsigned int us);
+
+#endif /* RTE_COROUTINE_H */
diff --git a/lib/coroutine/rte_coroutine_imp.h b/lib/coroutine/rte_coroutine_imp.h
new file mode 100644
index 0000000000..70b4f19670
--- /dev/null
+++ b/lib/coroutine/rte_coroutine_imp.h
@@ -0,0 +1,46 @@
+#ifndef RTE_COROUTINE_IMP_H
+#define RTE_COROUTINE_IMP_H
+
+#include <stdbool.h>
+#include <stdint.h>
+#include <ucontext.h>
+
+#include "rte_coroutine.h"
+
+#define MIN_STACK_SIZE	8192
+
+enum rte_costate {
+	COROUTINE_INVALID,
+	COROUTINE_READY,
+	COROUTINE_RUNNING,
+	COROUTINE_YIELD,
+	COROUTINE_SUSPEND,
+};
+
+struct rte_cocontext {
+	struct rte_schedule *owner; /**< Which scheduler this coroutine belongs. */
+	int state; /**< The current coroutine state. */
+	ucontext_t ctx;
+
+	coroutine_callback_t cb; /**< The coroutine callback function. */
+	void *arg; /**< The coroutine callback function's input argument. */
+
+	struct rte_cocontext *yield_next;
+
+	void *stack; /**< The allocated stack pointer. */
+	uint32_t stack_sz; /**< The allocated stack size. */
+};
+
+struct rte_schedule {
+	uint32_t max_coroutines; /**< Max coroutines which this scheduler supports. */
+
+	ucontext_t main;
+	struct rte_cocontext *running; /**< Current running coroutine. */
+
+	struct rte_ring *ring; /**< Command ring for schedule. */
+
+	struct rte_cocontext *yield_head; /**< Yield coroutine list for schedule. */
+	struct rte_cocontext *yield_tail;
+};
+
+#endif /* RTE_COROUTINE_IMP_H */
diff --git a/lib/coroutine/version.map b/lib/coroutine/version.map
new file mode 100644
index 0000000000..393b8979a6
--- /dev/null
+++ b/lib/coroutine/version.map
@@ -0,0 +1,11 @@
+EXPERIMENTAL {
+	global:
+
+	rte_co_create;
+	rte_co_delay;
+	rte_co_yield;
+	rte_schedule_create;
+	rte_schedule_run;
+
+	local: *;
+};
diff --git a/lib/meson.build b/lib/meson.build
index dc8aa4ac84..50e41f1511 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -64,6 +64,7 @@ libraries = [
         'flow_classify', # flow_classify lib depends on pkt framework table lib
         'graph',
         'node',
+        'coroutine',
 ]
 
 optional_libs = [
-- 
2.17.1

Re: [RFC 1/3] lib/coroutine: add coroutine library

[Ferruh Yigit]

On 4/24/2023 2:02 PM, Chengwen Feng wrote:
> This patch adds coroutine library. The main elements are:
> 1. scheduler: container of coroutines, which is responsible for
> scheduling coroutine.
> 2. coroutine: Minimum scheduling unit, it should associated to one
> scheduler.
> 
> In the coroutine callback, application could invoke rte_co_yield() to
> give up the CPU, and invoke rte_co_delay() to delay the specified
> microsecond.
> 
> Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>

<...>

> +++ b/lib/coroutine/rte_coroutine.c
> @@ -0,0 +1,190 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2023 HiSilicon Limited
> + */
> +
> +#include <pthread.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <ucontext.h>
> +
> +#include <rte_alarm.h>
> +#include <rte_ring.h>
> +
> +#include "rte_coroutine.h"
> +#include "rte_coroutine_imp.h"
> +
> +#define FATAL(fmt, args...) printf("[FATAL] %s() %d: " fmt "\n", __func__, __LINE__, ##args)
> +

No 'printf' for logging please.

<...>

> +int
> +rte_schedule_run(struct rte_schedule *s)
> +{
> +	struct rte_cocontext *co = NULL;
> +	uintptr_t ptr;
> +
> +	/* Set local thread variable as input argument. */
> +	co_schedule = s;
> +
> +	while (!rte_ring_empty(s->ring)) {
> +		rte_ring_dequeue(s->ring, (void **)&co);
> +		if (co->state == COROUTINE_READY) {
> +			getcontext(&co->ctx);
> +			co->ctx.uc_stack.ss_sp = co->stack;
> +			co->ctx.uc_stack.ss_size = co->stack_sz;
> +			co->ctx.uc_link = &s->main;
> +			co->state = COROUTINE_RUNNING;
> +			s->running = co;
> +			ptr = (uintptr_t)co;
> +			makecontext(&co->ctx, (void (*)(void))co_run_func, 2,
> +				    (uint32_t)ptr, (uint32_t)(ptr >> 32));

Why passing 'co' address as two 32bit values? Why not just pass address
of it?

Also can it be possible to set context to 'co->cb()' here and do the
cleanup after context returned to 's->main' (below)?

> +			swapcontext(&s->main, &co->ctx);
> +		} else if (co->state == COROUTINE_SUSPEND) {
> +			co->state = COROUTINE_RUNNING;
> +			s->running = co;
> +			swapcontext(&s->main, &co->ctx);
> +		} else {
> +			FATAL("invalid state!");
> +		}
> +	}
> +
> +	while (s->yield_head != NULL) {
> +		co = s->yield_head;
> +		s->yield_head = co->yield_next;
> +		if (co->state == COROUTINE_YIELD) {
> +			co->state = COROUTINE_RUNNING;
> +			s->running = co;
> +			swapcontext(&s->main, &co->ctx);
> +		} else {
> +			FATAL("invalid yield state!");
> +		}
> +	}
> +

As coroutines in 'ready' state stored in a ring, same can be done for
the coroutines in 'yield' state, instead of using a linked list, to
simplify the flow. Just a question, both works fine.

> +	return 0;
> +}
> +
> +int
> +rte_co_create(struct rte_schedule *s, coroutine_callback_t cb, void *arg, uint32_t stack_sz)
> +{
> +	struct rte_cocontext *co = calloc(1, sizeof(struct rte_cocontext));
> +	int ret;
> +	if (co == NULL)
> +		return -ENOMEM;
> +

I guess somewhere here should check the 's->max_coroutines';

[RFC 2/3] examples/coroutine: support coroutine examples

[Chengwen Feng]

This patch adds coroutine example, usage:
1. start examples: dpdk-coroutine -a 0000:7d:00.2  -l 10-11
2. will output:
Start yield coroutine test!
	I am in yield coroutine 111!
	I am in yield coroutine 222!
	I am in yield coroutine 333!
	I am in yield coroutine 111!
	I am in yield coroutine 222!
	I am in yield coroutine 333!
...
Start delay coroutine test!
	I am in delay coroutine 111!
	I am in delay coroutine 222!
	I am in delay coroutine 222!
	I am in delay coroutine 111!
	I am in delay coroutine 222!
	I am in delay coroutine 222!
...
3. use ctrl+c to exit example.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 examples/coroutine/main.c      | 153 +++++++++++++++++++++++++++++++++
 examples/coroutine/meson.build |  10 +++
 examples/meson.build           |   1 +
 3 files changed, 164 insertions(+)
 create mode 100644 examples/coroutine/main.c
 create mode 100644 examples/coroutine/meson.build

diff --git a/examples/coroutine/main.c b/examples/coroutine/main.c
new file mode 100644
index 0000000000..2704ad1dc9
--- /dev/null
+++ b/examples/coroutine/main.c
@@ -0,0 +1,153 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2023 HiSilicon Limited
+ */
+
+#include <stdint.h>
+#include <stdlib.h>
+#include <getopt.h>
+#include <signal.h>
+#include <stdbool.h>
+#include <unistd.h>
+
+#include <rte_common.h>
+#include <rte_coroutine.h>
+#include <rte_lcore.h>
+#include <rte_launch.h>
+
+static volatile bool force_quit;
+
+static struct rte_schedule *target_s;
+
+static void
+yield_coroutine_1(void *arg)
+{
+	RTE_SET_USED(arg);
+	int i = 10;
+	while (i--) {
+		printf("\tI am in yield coroutine 111!\n");
+		rte_co_yield();
+	}
+}
+
+static void
+yield_coroutine_2(void *arg)
+{
+	RTE_SET_USED(arg);
+	int i = 10;
+	while (i--) {
+		printf("\tI am in yield coroutine 222!\n");
+		rte_co_yield();
+	}
+}
+
+static void
+yield_coroutine_3(void *arg)
+{
+	RTE_SET_USED(arg);
+	int i = 10;
+	while (i--) {
+		printf("\tI am in yield coroutine 333!\n");
+		rte_co_yield();
+	}
+}
+
+static void
+yield_coroutine_test(void)
+{
+	printf("Start yield coroutine test!\n");
+	rte_co_create(target_s, yield_coroutine_1, NULL, 0);
+	rte_co_create(target_s, yield_coroutine_2, NULL, 0);
+	rte_co_create(target_s, yield_coroutine_3, NULL, 0);
+	sleep(1);
+}
+
+static void
+delay_coroutine_1(void *arg)
+{
+	RTE_SET_USED(arg);
+	int i = 10;
+	while (i--) {
+		printf("\tI am in delay coroutine 111!\n");
+		rte_co_delay(100 * 1000);
+	}
+}
+
+static void
+delay_coroutine_2(void *arg)
+{
+	RTE_SET_USED(arg);
+	int i = 20;
+	while (i--) {
+		printf("\tI am in delay coroutine 222!\n");
+		rte_co_delay(50 * 1000);
+	}
+}
+
+static void
+delay_coroutine_test(void)
+{
+	printf("Start delay coroutine test!\n");
+	rte_co_create(target_s, delay_coroutine_1, NULL, 0);
+	rte_co_create(target_s, delay_coroutine_2, NULL, 0);
+	sleep(1);
+}
+
+static int
+co_main_loop(void *arg)
+{
+	RTE_SET_USED(arg);
+	while (!force_quit)
+		rte_schedule_run(target_s);
+	return 0;
+}
+
+static void
+signal_handler(int signum)
+{
+	if (signum == SIGINT || signum == SIGTERM) {
+		printf("\n\nSignal %d received, preparing to exit...\n",
+			signum);
+		force_quit = true;
+	}
+}
+
+int
+main(int argc, char **argv)
+{
+	uint32_t lcore_id = rte_lcore_id();
+	int ret;
+
+	/* Init EAL. 8< */
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n");
+
+	force_quit = false;
+	signal(SIGINT, signal_handler);
+	signal(SIGTERM, signal_handler);
+
+	/* Check if there is enough lcores for all ports. */
+	if (rte_lcore_count() < 2)
+		rte_exit(EXIT_FAILURE,
+			"There should be at least one worker lcore.\n");
+
+	target_s = rte_schedule_create("co-sched-test", 128);
+	if (target_s == NULL)
+		rte_exit(EXIT_FAILURE,
+			"Create target scheduler failed!\n");
+
+	lcore_id = rte_get_next_lcore(lcore_id, true, true);
+	rte_eal_remote_launch(co_main_loop, NULL, lcore_id);
+
+	yield_coroutine_test();
+	delay_coroutine_test();
+
+	/* force_quit is true when we get here */
+	rte_eal_mp_wait_lcore();
+
+	/* clean up the EAL */
+	rte_eal_cleanup();
+
+	printf("Bye...\n");
+	return 0;
+}
diff --git a/examples/coroutine/meson.build b/examples/coroutine/meson.build
new file mode 100644
index 0000000000..c3576fe2f3
--- /dev/null
+++ b/examples/coroutine/meson.build
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 HiSilicon Limited
+
+allow_experimental_apis = true
+
+deps += ['coroutine']
+
+sources = files(
+        'main.c',
+)
diff --git a/examples/meson.build b/examples/meson.build
index 6968c09252..111d628065 100644
--- a/examples/meson.build
+++ b/examples/meson.build
@@ -11,6 +11,7 @@ all_examples = [
         'bbdev_app',
         'bond',
         'cmdline',
+	'coroutine',
         'distributor',
         'dma',
         'ethtool',
-- 
2.17.1

[RFC 3/3] net/hns3: refactor reset process with coroutine

[Chengwen Feng]

This patch adds reset mode 1 which use coroutine to refactor the
reset process. And this just a demo which only function at PF driver.

Using the coroutine will make the reset process more intuitive.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/net/hns3/hns3_ethdev.c | 217 +++++++++++++++++++++++++++++++++
 drivers/net/hns3/hns3_ethdev.h |   3 +
 drivers/net/hns3/hns3_intr.c   |  38 ++++++
 drivers/net/hns3/meson.build   |   2 +-
 4 files changed, 259 insertions(+), 1 deletion(-)

diff --git a/drivers/net/hns3/hns3_ethdev.c b/drivers/net/hns3/hns3_ethdev.c
index 36896f8989..06ff0bcae1 100644
--- a/drivers/net/hns3/hns3_ethdev.c
+++ b/drivers/net/hns3/hns3_ethdev.c
@@ -6487,7 +6487,224 @@ static const struct eth_dev_ops hns3_eth_dev_ops = {
 	.eth_tx_descriptor_dump     = hns3_tx_descriptor_dump,
 };
 
+#include <rte_coroutine.h>
+
+static const char *reset_string[HNS3_MAX_RESET] = {
+	"flr", "vf_func", "vf_pf_func", "vf_full", "vf_global",
+	"pf_func", "global", "IMP", "none",
+};
+
+static void
+hns3_clear_reset_level(struct hns3_hw *hw, uint64_t *levels)
+{
+	uint64_t merge_cnt = hw->reset.stats.merge_cnt;
+	uint64_t tmp;
+
+	switch (hw->reset.level) {
+	case HNS3_IMP_RESET:
+		hns3_atomic_clear_bit(HNS3_IMP_RESET, levels);
+		tmp = hns3_test_and_clear_bit(HNS3_GLOBAL_RESET, levels);
+		merge_cnt = tmp > 0 ? merge_cnt + 1 : merge_cnt;
+		tmp = hns3_test_and_clear_bit(HNS3_FUNC_RESET, levels);
+		merge_cnt = tmp > 0 ? merge_cnt + 1 : merge_cnt;
+		break;
+	case HNS3_GLOBAL_RESET:
+		hns3_atomic_clear_bit(HNS3_GLOBAL_RESET, levels);
+		tmp = hns3_test_and_clear_bit(HNS3_FUNC_RESET, levels);
+		merge_cnt = tmp > 0 ? merge_cnt + 1 : merge_cnt;
+		break;
+	case HNS3_FUNC_RESET:
+		hns3_atomic_clear_bit(HNS3_FUNC_RESET, levels);
+		break;
+	case HNS3_VF_RESET:
+		hns3_atomic_clear_bit(HNS3_VF_RESET, levels);
+		tmp = hns3_test_and_clear_bit(HNS3_VF_PF_FUNC_RESET, levels);
+		merge_cnt = tmp > 0 ? merge_cnt + 1 : merge_cnt;
+		tmp = hns3_test_and_clear_bit(HNS3_VF_FUNC_RESET, levels);
+		merge_cnt = tmp > 0 ? merge_cnt + 1 : merge_cnt;
+		break;
+	case HNS3_VF_FULL_RESET:
+		hns3_atomic_clear_bit(HNS3_VF_FULL_RESET, levels);
+		tmp = hns3_test_and_clear_bit(HNS3_VF_FUNC_RESET, levels);
+		merge_cnt = tmp > 0 ? merge_cnt + 1 : merge_cnt;
+		break;
+	case HNS3_VF_PF_FUNC_RESET:
+		hns3_atomic_clear_bit(HNS3_VF_PF_FUNC_RESET, levels);
+		tmp = hns3_test_and_clear_bit(HNS3_VF_FUNC_RESET, levels);
+		merge_cnt = tmp > 0 ? merge_cnt + 1 : merge_cnt;
+		break;
+	case HNS3_VF_FUNC_RESET:
+		hns3_atomic_clear_bit(HNS3_VF_FUNC_RESET, levels);
+		break;
+	case HNS3_FLR_RESET:
+		hns3_atomic_clear_bit(HNS3_FLR_RESET, levels);
+		break;
+	case HNS3_NONE_RESET:
+	default:
+		return;
+	};
+
+	if (merge_cnt != hw->reset.stats.merge_cnt) {
+		hns3_warn(hw,
+			  "No need to do low-level reset after %s reset. "
+			  "merge cnt: %" PRIu64 " total merge cnt: %" PRIu64,
+			  reset_string[hw->reset.level],
+			  hw->reset.stats.merge_cnt - merge_cnt,
+			  hw->reset.stats.merge_cnt);
+		hw->reset.stats.merge_cnt = merge_cnt;
+	}
+}
+
+static void hns3_reset_coroutine(void *arg)
+{
+	struct hns3_hw *hw = (struct hns3_hw *)arg;
+	struct hns3_adapter *hns = HNS3_DEV_HW_TO_ADAPTER(hw);
+	struct timeval tv, tv_delta;
+	int ret, i;
+
+	/*
+	 * calc reset level.
+	 */
+	hns3_clock_gettime(&hw->reset.start_time);
+	hw->reset.level = hns3_get_reset_level(hns, &hw->reset.pending);
+	if (hw->reset.level == HNS3_NONE_RESET)
+		hw->reset.level = HNS3_IMP_RESET;
+	hns3_warn(hw, "start reset level: %s", reset_string[hw->reset.level]);
+
+
+	/*
+	 * stop service.
+	 */
+	ret = hns3_stop_service(hns);
+	hns3_clock_gettime(&tv);
+	if (ret) {
+		hns3_warn(hw, "Reset step1 down fail=%d time=%ld.%.6ld",
+			  ret, tv.tv_sec, tv.tv_usec);
+		return;
+	}
+	hns3_warn(hw, "Reset step1 down success time=%ld.%.6ld",
+		  tv.tv_sec, tv.tv_usec);
+
+
+	/*
+	 * yield CPU to schedule other function's reset.
+	 */
+	rte_co_yield();
+
+
+	/*
+	 * prepare reset.
+	 */
+	ret = hns3_prepare_reset(hns);
+	hns3_clock_gettime(&tv);
+	if (ret) {
+		hns3_warn(hw,
+			  "Reset step2 prepare wait fail=%d time=%ld.%.6ld",
+			  ret, tv.tv_sec, tv.tv_usec);
+		return;
+	}
+	hns3_warn(hw, "Reset step2 prepare wait success time=%ld.%.6ld",
+		  tv.tv_sec, tv.tv_usec);
+
+
+	/*
+	 * delay 100ms which refer the manual of hardware.
+	 */
+	rte_co_delay(100 * 1000);
+
+
+	/*
+	 * notify IMP reset.
+	 */
+	/* inform hardware that preparatory work is done */
+	hns3_notify_reset_ready(hw, true);
+	hns3_clock_gettime(&tv);
+	hns3_warn(hw,
+		  "Reset step3 request IMP reset success time=%ld.%.6ld",
+		  tv.tv_sec, tv.tv_usec);
+
+
+	/*
+	 * Wait hardware reset done.
+	 */
+	for (i = 0; i < HNS3_RESET_WAIT_CNT; i++) {
+		rte_co_delay(HNS3_RESET_WAIT_MS * USEC_PER_MSEC);
+		if (is_pf_reset_done(hw))
+			break;
+	}
+	if (i >= HNS3_RESET_WAIT_CNT) {
+		hns3_clock_gettime(&tv);
+		hns3_warn(hw, "Reset step4 hardware not ready after reset time=%ld.%.6ld",
+			  tv.tv_sec, tv.tv_usec);
+		return;
+	}
+	hns3_clock_gettime(&tv);
+	hns3_warn(hw, "Reset step4 reset wait success time=%ld.%.6ld",
+		  tv.tv_sec, tv.tv_usec);
+
+
+	/*
+	 * yield CPU to schedule other function's reset.
+	 */
+	rte_co_yield();
+
+
+	/*
+	 * Devinit.
+	 */
+	rte_spinlock_lock(&hw->lock);
+	if (hw->reset.mbuf_deferred_free) {
+		hns3_dev_release_mbufs(hns);
+		hw->reset.mbuf_deferred_free = false;
+	}
+	ret = hns3_reinit_dev(hns);
+	rte_spinlock_unlock(&hw->lock);
+	hns3_clock_gettime(&tv);
+	if (ret) {
+		hns3_warn(hw, "Reset step5 devinit fail=%d retries=%d",
+			  ret, hw->reset.retries);
+		return;
+	}
+	hns3_warn(hw, "Reset step5 devinit success time=%ld.%.6ld",
+		  tv.tv_sec, tv.tv_usec);
+
+
+	/*
+	 * yield CPU to schedule other function's reset.
+	 */
+	rte_co_yield();
+
+
+	/*
+	 * Start service.
+	 */
+	/* IMP will wait ready flag before reset */
+	hns3_notify_reset_ready(hw, false);
+	hns3_clear_reset_level(hw, &hw->reset.pending);
+	rte_spinlock_lock(&hw->lock);
+	hns3_start_service(hns);
+	rte_spinlock_unlock(&hw->lock);
+	hns3_clock_gettime(&tv);
+	timersub(&tv, &hw->reset.start_time, &tv_delta);
+	hns3_warn(hw, "%s reset done fail_cnt:%" PRIu64
+		  " success_cnt:%" PRIu64 " global_cnt:%" PRIu64
+		  " imp_cnt:%" PRIu64 " request_cnt:%" PRIu64
+		  " exec_cnt:%" PRIu64 " merge_cnt:%" PRIu64,
+		  reset_string[hw->reset.level],
+		  hw->reset.stats.fail_cnt, hw->reset.stats.success_cnt,
+		  hw->reset.stats.global_cnt, hw->reset.stats.imp_cnt,
+		  hw->reset.stats.request_cnt, hw->reset.stats.exec_cnt,
+		  hw->reset.stats.merge_cnt);
+	hns3_warn(hw,
+		  "%s reset done delta %" PRIu64 " ms time=%ld.%.6ld",
+		  reset_string[hw->reset.level],
+		  hns3_clock_calctime_ms(&tv_delta),
+		  tv.tv_sec, tv.tv_usec);
+	hw->reset.level = HNS3_NONE_RESET;
+}
+
 static const struct hns3_reset_ops hns3_reset_ops = {
+	.coroutine = hns3_reset_coroutine,
 	.reset_service       = hns3_reset_service,
 	.stop_service        = hns3_stop_service,
 	.prepare_reset       = hns3_prepare_reset,
diff --git a/drivers/net/hns3/hns3_ethdev.h b/drivers/net/hns3/hns3_ethdev.h
index 9acc5a3d7e..4029bea133 100644
--- a/drivers/net/hns3/hns3_ethdev.h
+++ b/drivers/net/hns3/hns3_ethdev.h
@@ -379,6 +379,7 @@ struct hns3_wait_data {
 };
 
 struct hns3_reset_ops {
+	void (*coroutine)(void *arg);
 	void (*reset_service)(void *arg);
 	int (*stop_service)(struct hns3_adapter *hns);
 	int (*prepare_reset)(struct hns3_adapter *hns);
@@ -396,6 +397,8 @@ enum hns3_schedule {
 };
 
 struct hns3_reset_data {
+	struct rte_schedule *reset_sched;
+	uint32_t mode;
 	enum hns3_reset_stage stage;
 	uint16_t schedule;
 	/* Reset flag, covering the entire reset process */
diff --git a/drivers/net/hns3/hns3_intr.c b/drivers/net/hns3/hns3_intr.c
index 44a1119415..327d548bc6 100644
--- a/drivers/net/hns3/hns3_intr.c
+++ b/drivers/net/hns3/hns3_intr.c
@@ -2393,10 +2393,36 @@ hns3_handle_error(struct hns3_adapter *hns)
 	}
 }
 
+#include <rte_coroutine.h>
+
+static uint32_t thread_run(void *arg)
+{
+	struct rte_schedule *s = (struct rte_schedule *)arg;
+	int ret;
+
+	rte_thread_setname(pthread_self(), "co-sched-hns3");
+
+	while (1) {
+		ret = rte_schedule_run(s);
+		if (ret != 0)
+			break;
+	}
+
+	return 0;
+}
+
 int
 hns3_reset_init(struct hns3_hw *hw)
 {
 	rte_spinlock_init(&hw->lock);
+	hw->reset.mode = 1; /* mode 1 means use coroutine. */
+	static struct rte_schedule *s;
+	if (s == NULL) {
+		rte_thread_t thread;
+		s = rte_schedule_create("co-sched-hns3", 128);
+		rte_thread_create(&thread, NULL, thread_run, (void *)s);
+	}
+	hw->reset.reset_sched = s;
 	hw->reset.level = HNS3_NONE_RESET;
 	hw->reset.stage = RESET_STAGE_NONE;
 	hw->reset.request = 0;
@@ -2427,6 +2453,12 @@ hns3_schedule_reset(struct hns3_adapter *hns)
 	if (hw->adapter_state >= HNS3_NIC_CLOSED)
 		return;
 
+	if (hw->reset.mode == 1) {
+		rte_co_create(hw->reset.reset_sched, hw->reset.ops->coroutine,
+			      (void *)hw, 128 * 1024);
+		return;
+	}
+
 	/* Schedule restart alarm if it is not scheduled yet */
 	if (__atomic_load_n(&hw->reset.schedule, __ATOMIC_RELAXED) ==
 			SCHEDULE_REQUESTED)
@@ -2453,6 +2485,12 @@ hns3_schedule_delayed_reset(struct hns3_adapter *hns)
 		return;
 	}
 
+	if (hw->reset.mode == 1) {
+		rte_co_create(hw->reset.reset_sched, hw->reset.ops->coroutine,
+			      (void *)hw, 128 * 1024);
+		return;
+	}
+
 	if (__atomic_load_n(&hw->reset.schedule, __ATOMIC_RELAXED) !=
 			    SCHEDULE_NONE)
 		return;
diff --git a/drivers/net/hns3/meson.build b/drivers/net/hns3/meson.build
index 33f61f9883..7abb3d1986 100644
--- a/drivers/net/hns3/meson.build
+++ b/drivers/net/hns3/meson.build
@@ -37,7 +37,7 @@ require_iova_in_mbuf = false
 
 annotate_locks = false
 
-deps += ['hash']
+deps += ['hash', 'coroutine']
 
 if arch_subdir == 'arm' and dpdk_conf.get('RTE_ARCH_64')
     sources += files('hns3_rxtx_vec.c')
-- 
2.17.1

Re: [RFC 0/3] introduce coroutine library

[Stephen Hemminger]

On Mon, 24 Apr 2023 13:02:05 +0000
Chengwen Feng <fengchengwen@huawei.com> wrote:

> This patchset introduces the coroutine library which will help refactor
> the hns3 PMD's reset process.
> 
> The hns3 single function reset process consists of the following steps:
>     1.stop_service();
>     2.prepare_reset();
>     3.delay(100ms);
>     4.notify_hw();
>     5.wait_hw_reset_done(); // multiple sleep waits are involved.
>     6.reinit();
>     7.restore_conf();
> 
> If the DPDK process take over multiple hns3 functions (e.g. 100),
> it's impractical to reset and restore functions in sequence:
>     1.proc_func(001); // will completed in 100+ms range.
>     2.proc_func(002); // will completed in 100~200+ms range.
>     ...
>     x.proc_func(100); // will completed in 9900~10000+ms range.
> The later functions will process fail because it's too late to deal with.
> 
> One solution is that create a reset thread for each function, and it
> will lead to large number of threads if the DPDK process take over
> multiple hns3 functions.
> 
> So the current hns3 driver uses asynchronous mechanism, for examples, it
> use rte_eal_alarm_set() when process delay(100ms), it splits a serial
> process into multiple asynchronous processes, and the code is complex
> and difficult to understand.
> 
> The coroutine is a good mechanism to provide programmers with the 
> simplicity of keeping serial processes within a limited number of
> threads.
> 
> This patchset use <ucontext.h> to build the coroutine framework, and it
> just provides a demo. More APIs maybe added in the future.
> 
> In addition, we would like to ask the community whether it it possible
> to accept the library. If not, whether it is allowed to provide the
> library in hns3 PMD.
> 
> Chengwen Feng (3):
>   lib/coroutine: add coroutine library
>   examples/coroutine: support coroutine examples
>   net/hns3: refactor reset process with coroutine

Interesting, but the DPDK really is not the right place for this.
Also, why so much sleeping. Can't this device be handled with an event based
model. Plus any complexity like this introduces more bugs into already fragile
interaction of DPDK userspace applications and threads.

Not only that, coroutines add to the pre-existing problems with locking.
If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself.
And someone will spend days figuring that out. And the existing analyzer
tools will not know about the magic coroutine library.

Bottom line: please no

Re: [RFC 0/3] introduce coroutine library

[fengchengwen]

On 2023/4/25 0:08, Stephen Hemminger wrote:
> On Mon, 24 Apr 2023 13:02:05 +0000
> Chengwen Feng <fengchengwen@huawei.com> wrote:
> 
>> This patchset introduces the coroutine library which will help refactor
>> the hns3 PMD's reset process.
>>
>> The hns3 single function reset process consists of the following steps:
>>     1.stop_service();
>>     2.prepare_reset();
>>     3.delay(100ms);
>>     4.notify_hw();
>>     5.wait_hw_reset_done(); // multiple sleep waits are involved.
>>     6.reinit();
>>     7.restore_conf();
>>
>> If the DPDK process take over multiple hns3 functions (e.g. 100),
>> it's impractical to reset and restore functions in sequence:
>>     1.proc_func(001); // will completed in 100+ms range.
>>     2.proc_func(002); // will completed in 100~200+ms range.
>>     ...
>>     x.proc_func(100); // will completed in 9900~10000+ms range.
>> The later functions will process fail because it's too late to deal with.
>>
>> One solution is that create a reset thread for each function, and it
>> will lead to large number of threads if the DPDK process take over
>> multiple hns3 functions.
>>
>> So the current hns3 driver uses asynchronous mechanism, for examples, it
>> use rte_eal_alarm_set() when process delay(100ms), it splits a serial
>> process into multiple asynchronous processes, and the code is complex
>> and difficult to understand.
>>
>> The coroutine is a good mechanism to provide programmers with the 
>> simplicity of keeping serial processes within a limited number of
>> threads.
>>
>> This patchset use <ucontext.h> to build the coroutine framework, and it
>> just provides a demo. More APIs maybe added in the future.
>>
>> In addition, we would like to ask the community whether it it possible
>> to accept the library. If not, whether it is allowed to provide the
>> library in hns3 PMD.
>>
>> Chengwen Feng (3):
>>   lib/coroutine: add coroutine library
>>   examples/coroutine: support coroutine examples
>>   net/hns3: refactor reset process with coroutine
> 
> Interesting, but the DPDK really is not the right place for this.
> Also, why so much sleeping. Can't this device be handled with an event based
> model. Plus any complexity like this introduces more bugs into already fragile
> interaction of DPDK userspace applications and threads.

A event base model will function as:
  event-handler() {
    for (...) {
        event = get_next_event();
        proc_event();
    }
  }
The root cause is that the proc_event() take too many time, and it will lead to other
function can't be processed timely.

For which proc_event() may wait a lot of time, the coroutine could also used to optimize
it.

> 
> Not only that, coroutines add to the pre-existing problems with locking.
> If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself.
> And someone will spend days figuring that out. And the existing analyzer
> tools will not know about the magic coroutine library.

Analyzer tools like lock annotations maybe a problem.

Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such
as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API.


In addition, any technology has two sides, the greatest advantage of coroutine I think is
removes a large number of callbacks in asychronous programming. And also high-level languages
generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe
evolved to support detect.


And one more, if not acceptable as public library, whether it is allowed intergration of this
library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible,
but the code needs to be upstream, hope to listen to community's comments.

> 
> Bottom line: please no
> 
> .
> 

Re: [RFC 0/3] introduce coroutine library

[Stephen Hemminger]

On Tue, 25 Apr 2023 10:11:43 +0800
fengchengwen <fengchengwen@huawei.com> wrote:

> On 2023/4/25 0:08, Stephen Hemminger wrote:
> > On Mon, 24 Apr 2023 13:02:05 +0000
> > Chengwen Feng <fengchengwen@huawei.com> wrote:
> >   
> >> This patchset introduces the coroutine library which will help refactor
> >> the hns3 PMD's reset process.
> >>
> >> The hns3 single function reset process consists of the following steps:
> >>     1.stop_service();
> >>     2.prepare_reset();
> >>     3.delay(100ms);
> >>     4.notify_hw();
> >>     5.wait_hw_reset_done(); // multiple sleep waits are involved.
> >>     6.reinit();
> >>     7.restore_conf();
> >>
> >> If the DPDK process take over multiple hns3 functions (e.g. 100),
> >> it's impractical to reset and restore functions in sequence:
> >>     1.proc_func(001); // will completed in 100+ms range.
> >>     2.proc_func(002); // will completed in 100~200+ms range.
> >>     ...
> >>     x.proc_func(100); // will completed in 9900~10000+ms range.
> >> The later functions will process fail because it's too late to deal with.
> >>
> >> One solution is that create a reset thread for each function, and it
> >> will lead to large number of threads if the DPDK process take over
> >> multiple hns3 functions.
> >>
> >> So the current hns3 driver uses asynchronous mechanism, for examples, it
> >> use rte_eal_alarm_set() when process delay(100ms), it splits a serial
> >> process into multiple asynchronous processes, and the code is complex
> >> and difficult to understand.
> >>
> >> The coroutine is a good mechanism to provide programmers with the 
> >> simplicity of keeping serial processes within a limited number of
> >> threads.
> >>
> >> This patchset use <ucontext.h> to build the coroutine framework, and it
> >> just provides a demo. More APIs maybe added in the future.
> >>
> >> In addition, we would like to ask the community whether it it possible
> >> to accept the library. If not, whether it is allowed to provide the
> >> library in hns3 PMD.
> >>
> >> Chengwen Feng (3):
> >>   lib/coroutine: add coroutine library
> >>   examples/coroutine: support coroutine examples
> >>   net/hns3: refactor reset process with coroutine  
> > 
> > Interesting, but the DPDK really is not the right place for this.
> > Also, why so much sleeping. Can't this device be handled with an event based
> > model. Plus any complexity like this introduces more bugs into already fragile
> > interaction of DPDK userspace applications and threads.  
> 
> A event base model will function as:
>   event-handler() {
>     for (...) {
>         event = get_next_event();
>         proc_event();
>     }
>   }
> The root cause is that the proc_event() take too many time, and it will lead to other
> function can't be processed timely.
> 
> For which proc_event() may wait a lot of time, the coroutine could also used to optimize
> it.
> 
> > 
> > Not only that, coroutines add to the pre-existing problems with locking.
> > If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself.
> > And someone will spend days figuring that out. And the existing analyzer
> > tools will not know about the magic coroutine library.  
> 
> Analyzer tools like lock annotations maybe a problem.
> 
> Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such
> as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API.

> In addition, any technology has two sides, the greatest advantage of coroutine I think is
> removes a large number of callbacks in asychronous programming. And also high-level languages
> generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe
> evolved to support detect.
> 
> 
> And one more, if not acceptable as public library, whether it is allowed intergration of this
> library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible,
> but the code needs to be upstream, hope to listen to community's comments.


The standard DPDK architecture is to have dedicated threads.
Unless you convert the user application to some other model, there really is no other
useful work that can be done while waiting for your driver.

There was a previous DPDK library for lightweight threading, but it never got any
usage and was abandoned and dropped. Why is this better?

Re: [RFC 0/3] introduce coroutine library

[fengchengwen]

On 2023/4/25 10:16, Stephen Hemminger wrote:
> On Tue, 25 Apr 2023 10:11:43 +0800
> fengchengwen <fengchengwen@huawei.com> wrote:
> 
>> On 2023/4/25 0:08, Stephen Hemminger wrote:
>>> On Mon, 24 Apr 2023 13:02:05 +0000
>>> Chengwen Feng <fengchengwen@huawei.com> wrote:
>>>   
>>>> This patchset introduces the coroutine library which will help refactor
>>>> the hns3 PMD's reset process.
>>>>
>>>> The hns3 single function reset process consists of the following steps:
>>>>     1.stop_service();
>>>>     2.prepare_reset();
>>>>     3.delay(100ms);
>>>>     4.notify_hw();
>>>>     5.wait_hw_reset_done(); // multiple sleep waits are involved.
>>>>     6.reinit();
>>>>     7.restore_conf();
>>>>
>>>> If the DPDK process take over multiple hns3 functions (e.g. 100),
>>>> it's impractical to reset and restore functions in sequence:
>>>>     1.proc_func(001); // will completed in 100+ms range.
>>>>     2.proc_func(002); // will completed in 100~200+ms range.
>>>>     ...
>>>>     x.proc_func(100); // will completed in 9900~10000+ms range.
>>>> The later functions will process fail because it's too late to deal with.
>>>>
>>>> One solution is that create a reset thread for each function, and it
>>>> will lead to large number of threads if the DPDK process take over
>>>> multiple hns3 functions.
>>>>
>>>> So the current hns3 driver uses asynchronous mechanism, for examples, it
>>>> use rte_eal_alarm_set() when process delay(100ms), it splits a serial
>>>> process into multiple asynchronous processes, and the code is complex
>>>> and difficult to understand.
>>>>
>>>> The coroutine is a good mechanism to provide programmers with the 
>>>> simplicity of keeping serial processes within a limited number of
>>>> threads.
>>>>
>>>> This patchset use <ucontext.h> to build the coroutine framework, and it
>>>> just provides a demo. More APIs maybe added in the future.
>>>>
>>>> In addition, we would like to ask the community whether it it possible
>>>> to accept the library. If not, whether it is allowed to provide the
>>>> library in hns3 PMD.
>>>>
>>>> Chengwen Feng (3):
>>>>   lib/coroutine: add coroutine library
>>>>   examples/coroutine: support coroutine examples
>>>>   net/hns3: refactor reset process with coroutine  
>>>
>>> Interesting, but the DPDK really is not the right place for this.
>>> Also, why so much sleeping. Can't this device be handled with an event based
>>> model. Plus any complexity like this introduces more bugs into already fragile
>>> interaction of DPDK userspace applications and threads.  
>>
>> A event base model will function as:
>>   event-handler() {
>>     for (...) {
>>         event = get_next_event();
>>         proc_event();
>>     }
>>   }
>> The root cause is that the proc_event() take too many time, and it will lead to other
>> function can't be processed timely.
>>
>> For which proc_event() may wait a lot of time, the coroutine could also used to optimize
>> it.
>>
>>>
>>> Not only that, coroutines add to the pre-existing problems with locking.
>>> If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself.
>>> And someone will spend days figuring that out. And the existing analyzer
>>> tools will not know about the magic coroutine library.  
>>
>> Analyzer tools like lock annotations maybe a problem.
>>
>> Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such
>> as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API.
> 
>> In addition, any technology has two sides, the greatest advantage of coroutine I think is
>> removes a large number of callbacks in asychronous programming. And also high-level languages
>> generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe
>> evolved to support detect.
>>
>>
>> And one more, if not acceptable as public library, whether it is allowed intergration of this
>> library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible,
>> but the code needs to be upstream, hope to listen to community's comments.
> 
> 
> The standard DPDK architecture is to have dedicated threads.
> Unless you convert the user application to some other model, there really is no other

Instead of adding a new running model, this coroutine library just adapts to the current
DPDK framework.
My visions:
1. DPDK launch a default thread run coroutine service, this thread just like interrupt thread,
it could provide server for PMD drivers.
2. Application could launch coroutine scheduler in lcore (just like 2/3 commit) if it want to
use this library.

> useful work that can be done while waiting for your driver.
> 
> There was a previous DPDK library for lightweight threading, but it never got any
> usage and was abandoned and dropped. Why is this better?

DPDK lightweight threading ? I didn't know before, but will take a closer look at.

This patchset is caused by a problem in the our driver reset process. The reset process is
complex and difficult to understand. We think the coroutine implementation is practical to
solve this problem, so we try to send this RFC.

> .
> 

Re: [RFC 0/3] introduce coroutine library

[Garrett D'Amore]

[-- Attachment #1: Type: text/plain, Size: 6365 bytes --]

First time poster here:

I worry a bit about a coroutine approach as it may be challenging for some uses like ours.  We have a purely event driven loop with a Reactor model written in D.  The details are not specifically needed here, except to point out that an approach based on ucontext.h or something like that would very likely be utterly incompatible in our environment.  While we don’t currently plan to integrate support for your hns3 device, I would have grave reservations about a general coroutine library making it’s way into DPDK drivers — it would almost certainly cause no end of grief for us at Weka.

I’m doubtful that we’re the only DPDK users in this situation.

• Garrett

On Apr 24, 2023 at 7:50 PM -0700, fengchengwen <fengchengwen@huawei.com>, wrote:
> On 2023/4/25 10:16, Stephen Hemminger wrote:
> > On Tue, 25 Apr 2023 10:11:43 +0800
> > fengchengwen <fengchengwen@huawei.com> wrote:
> >
> > > On 2023/4/25 0:08, Stephen Hemminger wrote:
> > > > On Mon, 24 Apr 2023 13:02:05 +0000
> > > > Chengwen Feng <fengchengwen@huawei.com> wrote:
> > > >
> > > > > This patchset introduces the coroutine library which will help refactor
> > > > > the hns3 PMD's reset process.
> > > > >
> > > > > The hns3 single function reset process consists of the following steps:
> > > > > 1.stop_service();
> > > > > 2.prepare_reset();
> > > > > 3.delay(100ms);
> > > > > 4.notify_hw();
> > > > > 5.wait_hw_reset_done(); // multiple sleep waits are involved.
> > > > > 6.reinit();
> > > > > 7.restore_conf();
> > > > >
> > > > > If the DPDK process take over multiple hns3 functions (e.g. 100),
> > > > > it's impractical to reset and restore functions in sequence:
> > > > > 1.proc_func(001); // will completed in 100+ms range.
> > > > > 2.proc_func(002); // will completed in 100~200+ms range.
> > > > > ...
> > > > > x.proc_func(100); // will completed in 9900~10000+ms range.
> > > > > The later functions will process fail because it's too late to deal with.
> > > > >
> > > > > One solution is that create a reset thread for each function, and it
> > > > > will lead to large number of threads if the DPDK process take over
> > > > > multiple hns3 functions.
> > > > >
> > > > > So the current hns3 driver uses asynchronous mechanism, for examples, it
> > > > > use rte_eal_alarm_set() when process delay(100ms), it splits a serial
> > > > > process into multiple asynchronous processes, and the code is complex
> > > > > and difficult to understand.
> > > > >
> > > > > The coroutine is a good mechanism to provide programmers with the
> > > > > simplicity of keeping serial processes within a limited number of
> > > > > threads.
> > > > >
> > > > > This patchset use <ucontext.h> to build the coroutine framework, and it
> > > > > just provides a demo. More APIs maybe added in the future.
> > > > >
> > > > > In addition, we would like to ask the community whether it it possible
> > > > > to accept the library. If not, whether it is allowed to provide the
> > > > > library in hns3 PMD.
> > > > >
> > > > > Chengwen Feng (3):
> > > > > lib/coroutine: add coroutine library
> > > > > examples/coroutine: support coroutine examples
> > > > > net/hns3: refactor reset process with coroutine
> > > >
> > > > Interesting, but the DPDK really is not the right place for this.
> > > > Also, why so much sleeping. Can't this device be handled with an event based
> > > > model. Plus any complexity like this introduces more bugs into already fragile
> > > > interaction of DPDK userspace applications and threads.
> > >
> > > A event base model will function as:
> > > event-handler() {
> > > for (...) {
> > > event = get_next_event();
> > > proc_event();
> > > }
> > > }
> > > The root cause is that the proc_event() take too many time, and it will lead to other
> > > function can't be processed timely.
> > >
> > > For which proc_event() may wait a lot of time, the coroutine could also used to optimize
> > > it.
> > >
> > > >
> > > > Not only that, coroutines add to the pre-existing problems with locking.
> > > > If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself.
> > > > And someone will spend days figuring that out. And the existing analyzer
> > > > tools will not know about the magic coroutine library.
> > >
> > > Analyzer tools like lock annotations maybe a problem.
> > >
> > > Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such
> > > as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API.
> >
> > > In addition, any technology has two sides, the greatest advantage of coroutine I think is
> > > removes a large number of callbacks in asychronous programming. And also high-level languages
> > > generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe
> > > evolved to support detect.
> > >
> > >
> > > And one more, if not acceptable as public library, whether it is allowed intergration of this
> > > library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible,
> > > but the code needs to be upstream, hope to listen to community's comments.
> >
> >
> > The standard DPDK architecture is to have dedicated threads.
> > Unless you convert the user application to some other model, there really is no other
>
> Instead of adding a new running model, this coroutine library just adapts to the current
> DPDK framework.
> My visions:
> 1. DPDK launch a default thread run coroutine service, this thread just like interrupt thread,
> it could provide server for PMD drivers.
> 2. Application could launch coroutine scheduler in lcore (just like 2/3 commit) if it want to
> use this library.
>
> > useful work that can be done while waiting for your driver.
> >
> > There was a previous DPDK library for lightweight threading, but it never got any
> > usage and was abandoned and dropped. Why is this better?
>
> DPDK lightweight threading ? I didn't know before, but will take a closer look at.
>
> This patchset is caused by a problem in the our driver reset process. The reset process is
> complex and difficult to understand. We think the coroutine implementation is practical to
> solve this problem, so we try to send this RFC.
>
> > .
> >

[-- Attachment #2: Type: text/html, Size: 7055 bytes --]

Re: [RFC 0/3] introduce coroutine library

[Stephen Hemminger]

On Mon, 24 Apr 2023 19:59:27 -0700
Garrett D'Amore <garrett@damore.org> wrote:

> > > There was a previous DPDK library for lightweight threading, but it never got any
> > > usage and was abandoned and dropped. Why is this better?  
> >
> > DPDK lightweight threading ? I didn't know before, but will take a closer look at.


https://doc.dpdk.org/guides-21.11/sample_app_ug/performance_thread.html

Re: [RFC 0/3] introduce coroutine library

[Ferruh Yigit]

On 4/25/2023 3:11 AM, fengchengwen wrote:
> On 2023/4/25 0:08, Stephen Hemminger wrote:
>> On Mon, 24 Apr 2023 13:02:05 +0000
>> Chengwen Feng <fengchengwen@huawei.com> wrote:
>>
>>> This patchset introduces the coroutine library which will help refactor
>>> the hns3 PMD's reset process.
>>>
>>> The hns3 single function reset process consists of the following steps:
>>>     1.stop_service();
>>>     2.prepare_reset();
>>>     3.delay(100ms);
>>>     4.notify_hw();
>>>     5.wait_hw_reset_done(); // multiple sleep waits are involved.
>>>     6.reinit();
>>>     7.restore_conf();
>>>
>>> If the DPDK process take over multiple hns3 functions (e.g. 100),
>>> it's impractical to reset and restore functions in sequence:
>>>     1.proc_func(001); // will completed in 100+ms range.
>>>     2.proc_func(002); // will completed in 100~200+ms range.
>>>     ...
>>>     x.proc_func(100); // will completed in 9900~10000+ms range.
>>> The later functions will process fail because it's too late to deal with.
>>>
>>> One solution is that create a reset thread for each function, and it
>>> will lead to large number of threads if the DPDK process take over
>>> multiple hns3 functions.
>>>
>>> So the current hns3 driver uses asynchronous mechanism, for examples, it
>>> use rte_eal_alarm_set() when process delay(100ms), it splits a serial
>>> process into multiple asynchronous processes, and the code is complex
>>> and difficult to understand.
>>>
>>> The coroutine is a good mechanism to provide programmers with the 
>>> simplicity of keeping serial processes within a limited number of
>>> threads.
>>>
>>> This patchset use <ucontext.h> to build the coroutine framework, and it
>>> just provides a demo. More APIs maybe added in the future.
>>>
>>> In addition, we would like to ask the community whether it it possible
>>> to accept the library. If not, whether it is allowed to provide the
>>> library in hns3 PMD.
>>>
>>> Chengwen Feng (3):
>>>   lib/coroutine: add coroutine library
>>>   examples/coroutine: support coroutine examples
>>>   net/hns3: refactor reset process with coroutine
>>
>> Interesting, but the DPDK really is not the right place for this.
>> Also, why so much sleeping. Can't this device be handled with an event based
>> model. Plus any complexity like this introduces more bugs into already fragile
>> interaction of DPDK userspace applications and threads.
> 
> A event base model will function as:
>   event-handler() {
>     for (...) {
>         event = get_next_event();
>         proc_event();
>     }
>   }
> The root cause is that the proc_event() take too many time, and it will lead to other
> function can't be processed timely.
> 
> For which proc_event() may wait a lot of time, the coroutine could also used to optimize
> it.
> 
>>
>> Not only that, coroutines add to the pre-existing problems with locking.
>> If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself.
>> And someone will spend days figuring that out. And the existing analyzer
>> tools will not know about the magic coroutine library.
> 
> Analyzer tools like lock annotations maybe a problem.
> 
> Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such
> as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API.
> 
> 
> In addition, any technology has two sides, the greatest advantage of coroutine I think is
> removes a large number of callbacks in asychronous programming. And also high-level languages
> generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe
> evolved to support detect.
> 
> 
> And one more, if not acceptable as public library, whether it is allowed intergration of this
> library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible,
> but the code needs to be upstream, hope to listen to community's comments.
> 

Hi Chengwen,

For having library in hns3 PMD, my concern is ucontext.h portability.

If this breaks the build in a specific platform, a user even doesn't
have an intention to use hns3 driver will be impacted and will need to
debug/fix this issue.

If the dependency properly managed in the meson and for unsupported
platforms coroutine can be disabled, I don't see any reason to not have
this in the driver (except than the reasons already mentioned to not
have coroutine as a public library applies to here).

But this means you will need to maintain and support reset method
without coroutine, or disable driver on the platform that ucontext.h not
supported, are you OK with this?


And if you continue continue to have coroutine in the driver can you
please group into a subfolder etc in the driver and document it (again
in a subsection of the driver) to help users that found it useful and
try for themselves.


I put a few comments to the implementation as well.


Thanks,
ferruh

Re: [RFC 0/3] introduce coroutine library

[fengchengwen]

On 2023/4/26 19:27, Ferruh Yigit wrote:
> On 4/25/2023 3:11 AM, fengchengwen wrote:
>> On 2023/4/25 0:08, Stephen Hemminger wrote:
>>> On Mon, 24 Apr 2023 13:02:05 +0000
>>> Chengwen Feng <fengchengwen@huawei.com> wrote:
>>>
>>>> This patchset introduces the coroutine library which will help refactor
>>>> the hns3 PMD's reset process.
>>>>
>>>> The hns3 single function reset process consists of the following steps:
>>>>     1.stop_service();
>>>>     2.prepare_reset();
>>>>     3.delay(100ms);
>>>>     4.notify_hw();
>>>>     5.wait_hw_reset_done(); // multiple sleep waits are involved.
>>>>     6.reinit();
>>>>     7.restore_conf();
>>>>
>>>> If the DPDK process take over multiple hns3 functions (e.g. 100),
>>>> it's impractical to reset and restore functions in sequence:
>>>>     1.proc_func(001); // will completed in 100+ms range.
>>>>     2.proc_func(002); // will completed in 100~200+ms range.
>>>>     ...
>>>>     x.proc_func(100); // will completed in 9900~10000+ms range.
>>>> The later functions will process fail because it's too late to deal with.
>>>>
>>>> One solution is that create a reset thread for each function, and it
>>>> will lead to large number of threads if the DPDK process take over
>>>> multiple hns3 functions.
>>>>
>>>> So the current hns3 driver uses asynchronous mechanism, for examples, it
>>>> use rte_eal_alarm_set() when process delay(100ms), it splits a serial
>>>> process into multiple asynchronous processes, and the code is complex
>>>> and difficult to understand.
>>>>
>>>> The coroutine is a good mechanism to provide programmers with the 
>>>> simplicity of keeping serial processes within a limited number of
>>>> threads.
>>>>
>>>> This patchset use <ucontext.h> to build the coroutine framework, and it
>>>> just provides a demo. More APIs maybe added in the future.
>>>>
>>>> In addition, we would like to ask the community whether it it possible
>>>> to accept the library. If not, whether it is allowed to provide the
>>>> library in hns3 PMD.
>>>>
>>>> Chengwen Feng (3):
>>>>   lib/coroutine: add coroutine library
>>>>   examples/coroutine: support coroutine examples
>>>>   net/hns3: refactor reset process with coroutine
>>>
>>> Interesting, but the DPDK really is not the right place for this.
>>> Also, why so much sleeping. Can't this device be handled with an event based
>>> model. Plus any complexity like this introduces more bugs into already fragile
>>> interaction of DPDK userspace applications and threads.
>>
>> A event base model will function as:
>>   event-handler() {
>>     for (...) {
>>         event = get_next_event();
>>         proc_event();
>>     }
>>   }
>> The root cause is that the proc_event() take too many time, and it will lead to other
>> function can't be processed timely.
>>
>> For which proc_event() may wait a lot of time, the coroutine could also used to optimize
>> it.
>>
>>>
>>> Not only that, coroutines add to the pre-existing problems with locking.
>>> If coroutine 1 acquires a lock, the coroutine 2 will deadlock itself.
>>> And someone will spend days figuring that out. And the existing analyzer
>>> tools will not know about the magic coroutine library.
>>
>> Analyzer tools like lock annotations maybe a problem.
>>
>> Locks in DPDK APIs are mostly no-blocking. We can add some restrictions(by reviewer), such
>> as once holding a lock, you can't invoke rte_co_yield() or rte_co_delay() API.
>>
>>
>> In addition, any technology has two sides, the greatest advantage of coroutine I think is
>> removes a large number of callbacks in asychronous programming. And also high-level languages
>> generally provide coroutines (e.g. C++/Python). With the development, the analyzer tools maybe
>> evolved to support detect.
>>
>>
>> And one more, if not acceptable as public library, whether it is allowed intergration of this
>> library in hns3 PMD ? Our internal evaluation solution (use coroutine refactor) is feasible,
>> but the code needs to be upstream, hope to listen to community's comments.
>>
> 
> Hi Chengwen,
> 
> For having library in hns3 PMD, my concern is ucontext.h portability.

Yes, after in-depth evaluation, it was found that this was indeed a problem.

And ucontext_t were deprecated in Posix v6 and removed in v7. See:
https://stackoverflow.com/questions/33331894/why-does-ucontext-have-such-high-overhead

Although it is possible to implement an assembly version (which replace ucontext_t),
but maintaining assembly code in DPDK is difficult.

> 
> If this breaks the build in a specific platform, a user even doesn't
> have an intention to use hns3 driver will be impacted and will need to
> debug/fix this issue.
> 
> If the dependency properly managed in the meson and for unsupported
> platforms coroutine can be disabled, I don't see any reason to not have
> this in the driver (except than the reasons already mentioned to not
> have coroutine as a public library applies to here).
> 
> But this means you will need to maintain and support reset method
> without coroutine, or disable driver on the platform that ucontext.h not
> supported, are you OK with this?

Yes, if two set are maintained at the same time, the complexity is increased.

Combine the opinions of all reviewer, the introduction of coroutine will cause
a lot of problems, and I will *stop* this patchset.

@Ferruh @Setphen @Garrett @Mattias
Thanks for your review and feedback.

> 
> 
> And if you continue continue to have coroutine in the driver can you
> please group into a subfolder etc in the driver and document it (again
> in a subsection of the driver) to help users that found it useful and
> try for themselves.
> 
> 
> I put a few comments to the implementation as well.

@Ferruh, thanks for the review.

> 
> 
> Thanks,
> ferruh
> 
> .
> 

Re: [RFC 0/3] introduce coroutine library

[Mattias Rönnblom]

On 2023-04-24 15:02, Chengwen Feng wrote:
> This patchset introduces the coroutine library which will help refactor
> the hns3 PMD's reset process.
> 
> The hns3 single function reset process consists of the following steps:
>      1.stop_service();
>      2.prepare_reset();
>      3.delay(100ms);
>      4.notify_hw();
>      5.wait_hw_reset_done(); // multiple sleep waits are involved.
>      6.reinit();
>      7.restore_conf();
> 
> If the DPDK process take over multiple hns3 functions (e.g. 100),
> it's impractical to reset and restore functions in sequence:
>      1.proc_func(001); // will completed in 100+ms range.
>      2.proc_func(002); // will completed in 100~200+ms range.
>      ...
>      x.proc_func(100); // will completed in 9900~10000+ms range.
> The later functions will process fail because it's too late to deal with.
> 
> One solution is that create a reset thread for each function, and it
> will lead to large number of threads if the DPDK process take over
> multiple hns3 functions.
> 
> So the current hns3 driver uses asynchronous mechanism, for examples, it
> use rte_eal_alarm_set() when process delay(100ms), it splits a serial
> process into multiple asynchronous processes, and the code is complex
> and difficult to understand.
> 
> The coroutine is a good mechanism to provide programmers with the
> simplicity of keeping serial processes within a limited number of
> threads.
> 

Coroutines (or anything with a stack, really) are generally too slow as 
a vehicle for concurrency in data plane applications, I would argue.

They might help you solve this particular slow path task, but that alone 
doesn't seem like anything close to a rationale why a new concurrency 
library should be accepted.

DPDK does need a deferred work mechanism, but I don't think coroutines 
are the answer. Currently, RTE timer is the closest thing you have in 
DPDK. To solve your issue (and many other, including things in the fast 
path), I would rather look in that direction, maybe extending to 
something Linux' tasklets.

> This patchset use <ucontext.h> to build the coroutine framework, and it
> just provides a demo. More APIs maybe added in the future.
> 
> In addition, we would like to ask the community whether it it possible
> to accept the library. If not, whether it is allowed to provide the
> library in hns3 PMD.
> 
> Chengwen Feng (3):
>    lib/coroutine: add coroutine library
>    examples/coroutine: support coroutine examples
>    net/hns3: refactor reset process with coroutine
> 
>   drivers/net/hns3/hns3_ethdev.c    | 217 ++++++++++++++++++++++++++++++
>   drivers/net/hns3/hns3_ethdev.h    |   3 +
>   drivers/net/hns3/hns3_intr.c      |  38 ++++++
>   drivers/net/hns3/meson.build      |   2 +-
>   examples/coroutine/main.c         | 153 +++++++++++++++++++++
>   examples/coroutine/meson.build    |  10 ++
>   examples/meson.build              |   1 +
>   lib/coroutine/meson.build         |   8 ++
>   lib/coroutine/rte_coroutine.c     | 190 ++++++++++++++++++++++++++
>   lib/coroutine/rte_coroutine.h     | 110 +++++++++++++++
>   lib/coroutine/rte_coroutine_imp.h |  46 +++++++
>   lib/coroutine/version.map         |  11 ++
>   lib/meson.build                   |   1 +
>   13 files changed, 789 insertions(+), 1 deletion(-)
>   create mode 100644 examples/coroutine/main.c
>   create mode 100644 examples/coroutine/meson.build
>   create mode 100644 lib/coroutine/meson.build
>   create mode 100644 lib/coroutine/rte_coroutine.c
>   create mode 100644 lib/coroutine/rte_coroutine.h
>   create mode 100644 lib/coroutine/rte_coroutine_imp.h
>   create mode 100644 lib/coroutine/version.map
>